Virtual Reality in Gaming. Technology & Patent Landscape

Transcription

1 Virtual Reality in Gaming Technology & Patent Landscape R

2 Contents Executive Summary 03 Introduction Technology Overview Virtual Reality Ecosystem Technology & Patent Landscape Analysis Data Processing Interactive Hardware Internal Chipsets Feedback & Control Content Major Patent Portfolios Seminal Patent Analysis Key Players in the Landscape Microsoft Immersion Corporation Sony Samsung Alphabet Key Litigation & Licensing Deals Conclusion 37 Appendix 38 Authors 41 02

3 EXECUTIVE SUMMARY The debate surrounding the sustainability of Augmented Reality (AR) versus Virtual Reality (VR) is gaining heat. Every major company, including Microsoft, Intel, Nvidia, Samsung, Apple, Google, Facebook, HTC, Sony and Valve are pouring millions of dollars into developing both concepts. While VR and AR have found application in various fields like education, entertainment, military training, healthcare, space training, architecture, e-commerce, etc., gaming remains its largest space with ~33% market share. mobile-based VR market, strengthening its IP portfolio with 60 of its 906 IP assets pertaining to gaming. irunway also identified and analyzed ~300 granted patents as seminal, or strong, with immense potential for monetization. 92% of Broadcom s granted patents were found seminal, followed by Mq Gaming with 77% of its granted patents of high value. About 58% of Nantworks and 31% of Immersion s granted patents were identified as seminal in nature. VR and AR first witnessed a physical dimension in the 1980s with the Sega Master System 3D glasses. However, it was in 1995 that it gained traction; post 2005, the technology has grown exponentially into a billion-dollar industry marked a leap point with about 10 million VR/AR products shipped worldwide. Patenting trends suggest that the technology and its consumer market is expected to witness a spike in , with a strong focus on untethered devices. irunway s research identified 3,720 IP assets in VR and AR technology for gaming registered with the USPTO. Majority IP assets relate to Data Processing. Microsoft, Immersion and Sony are pioneers in this space with a cumulative IP asset share of 25%. The demand for greater immersive experience is encouraging R&D in haptic feedback mechanisms. Immersion Corporation and Microsoft lead this space with 150 and 46 IP assets respectively. Alphabet is making significant headway in both the product and IP space. It has marked its presence in the R&D in this space witnessed a slack between the late 1990s up to 2005 owing to slow progress in motion control and high peripheral cost. However, renewed innovation is witnessing a string of high profile acquisitions by Facebook, Hewlett Packard, Apple and Google. After a hiatus in the early 2000s, patenting in VR gaming witnessed a spike after While companies such as Microsoft and Sony are charting a leadership path, early patentees such as Immersion, Nantworks and Broadcom have a lage set of patents expiring in the next few years. This may result in active monetization activities. AR and VR for gaming is becoming more affordable, and competition is only going to make it even more consumer-friendly. Patenting trends suggest that the technology is still on the lower end of the digital capability curve, with much left to explore. While speculations are rife on the dominance of which of the two concepts, patentees are looking to develop both simultaneously, with a strong focus on the capability of merging the two concepts into one of mixed reality for a more futuristic consumer market. 03

4 Cancer Immunotherapy: Antibody Therapy Virtual Reality in in Gaming: US Patent U.S. Patent Landscape INTRODUCTION The virtual reality industry has witnessed a strong evolution phase over the last two decades, developing interactive 3D graphics, visual and aural stimulation, and user interfaces. The technology is largely influenced by video games with a greater portion of the R&D focused on developing virtual reality for gaming technology. The terms virtual reality (VR) and augmented reality (AR) differ vastly. While AR overlays the digital environment on a real environment, VR transports a user into an immersive virtual environment. However, since a large section of the underlying technology is the same for both, this irunway research considers both terms together for analysis. The concept of taking gaming into the virtual world was the idea of naval radar technician Douglas Engelbart in the 1950s. Since then, it has intrigued researchers to create an illusionary digital world, especially for gamers. The 1990s witnessed a fairly peaking market in 3D graphics for gaming. In 1993, the Sega wraparound prototype VR glasses were introduced with features such as head tracking, stereo sound and LCD screens in the visor. In 1995, 3D gaming console Nintendo Virtual Boy (originally known as VR-32) was launched. But the technology witnessed a setback in the consumer market towards the end of the 1990s owing to high costs. Since 2000, however, there has been a revival and rapid advancement in VR technology innovation for gaming. In 2001, researchers at Sony, University of Washington and Carnegie Mellon University published an early version of a head mounted display, Tiles, for an augmented reality interface 1. The system included a Sony Glasstron PLMS700 headset with a miniature NTSC Toshiba camera that offered a wide-angle lens. They concluded that developers of tangible AR interfaces must focus on the interface s physical design and virtual icons computer graphics design. Advances in computer technology, followed by inventions of the smartphone era and high data speeds through 4G and LTE networks, has enhanced demand for high-density displays and 3D graphics capabilities, encouraging growth of the VR market. The potential of VR technology to impact varied application areas and reach out a wide consumer base has attracted several technology companies such as Facebook, Google, Samsung, Sony and Intel. Though researchers expect gaming as the most promising application area for virtual and augmented reality, live events and video entertainment are other major consumer-focused applications. The potential of new technologies extends way beyond the consumer space. According to a Goldman Sachs estimate 2, nearly half the industry s revenue will be generated in the enterprise and public sector, with healthcare and engineering as the most promising areas. Several market studies indicate that the AR/VR market in the U.S. is expected to gross at $80 billion by 2025, growing from a predicted $11.6 billion market in This indicates that patenting in VR in gaming is poised to grow at a 25% CAGR in the U.S. from 2017 through

5 Technology Overview VR/AR technology requires a compilation of multiple technologies to work in synchronization to achieve an immersive or sub-immersive experience. Hardware and Software form two major and complementary components of VR/AR technology, powering an immersive experience. Hardware components primarily comprise of head mounted displays (HMDs), input and tracking hardware, sensors, graphics cards, processors and memory devices. Software includes methods of image rendering, data processing, content creation and sharing. The Virtual Reality Ecosystem Multiple technology companies are collaborating to develop commercially available AR/VR devices. Facebook Oculus, Sony PlayStation VR, Samsung Gear VR, Google Cardboard, Daydream, Glass, HTC Vive and Microsoft HoloLens are notable devices equipped with cameras, sensors, controllers and input hardware like audio input, gesture recognition, motion tracking, eye and head movement. Nokia s GoPro includes cameras, Leap Motion provides gesture recognition and motion tracking, and Valve has controllers incorporated. Feedback mechanisms are an integral aspect of a VR gaming device, where tactile and force feedback are commonly provided by existing haptics technology. Enhancing battery life, communication with remote systems (sensors, servers or co-players in a multi-player environment), hands-free mobility and enhanced immersive mechanisms for a more realistic experience are other major areas of research in this space. Graphics processors and central processors are the backbone of AR/VR devices. Nvidia, AMD, Intel and Qualcomm provide GPU and CPU chipsets. Feedback and haptics are emerging as an attractive feature to provide users a realistic experience. Immersion leads in haptics. A software platform provides the underlying framework for processing, recognition and content rendering. Figure 1 depicts the leaders in the VR infrastructure. Virtual Reality & Augmented Reality Devices PC/ Console-based VR devices Mobile-based devices AR/ Mixed Reality-based devices Sony Playstation VR Oculus HTC Vive Google Cardboard Samsung Gear VR Google Daydream Google Glass Microsoft HoloLens Vuzix Smart Glasses Input Devices Graphics Processors Processors Feedback & Haptics Go Pro Nokia Valve Leap Motion Nvidia AMD Qualcomm Intel Qualcomm Texas Instruments Immersion Corp. Texas Instruments Software Platforms Unity OSVR Unreal Engine Microsoft SDK Oculus SDK PlayStation SDK Google SDK Steam VR Figure 1: Virtual Reality Technology Ecosystem irunway 2016 Public 3 05

6 irunway conducted an in-depth analysis of the U.S. patent landscape in VR and AR for gaming. The following sections present insights analyzed in a broader landscape of VR in gaming. While there are companies who lead a particular space, they may or may not feature in irunway s analysis owing to their patent portfolio pertaining to gaming technology. The VR in gaming patent landscape can be categorized into five major technology segments as shown in Figure 2. Level 1 Data Processing Level 2 Data Manipulation Data Storage Simulation 3D Modelling Image rendering/ Processing Interactive Hardware Display devices I/O devices Feedback & Control Motion tracking Feedback mechanisms/ Haptics Internal Chips Content Processing devices Communication/ Interface devices Content access Content sharing Avatar/ Character features Software plugins Figure 2: Virtual Reality Technology Taxonomy* * All granted and active patents and patent-pending applications are referred to as IP assets in this report. irunway 2016 Public 4 06

7 TECHNOLOGY AND PATENT LANDSCAPE ANALYSIS irunway identified 3,720 IP assets related to VR in gaming technology in the U.S. patent landscape and categorized them across the technology taxonomy, as represented in Table 1, for further analysis: Data Processing (2,423) Data manipulation Data storage Simulation 3D Modelling Image rendering/ Processing ,460 Interactive Hardware (976) Display devices I/O devices Internal Chips (422) Processing Devices Communication/ Interface devices Feedback and Control (607) Motion tracking Feedback mechanisms/ Haptics Content (875) Content access Content sharing Avatar/ Character features Software plugins Table 1: Distribution of IP assets across the VR in Gaming Technology Taxonomy Location-based entertainment found an accepting consumer market in the 1990s. A growing consumer base for personal computers and more responsive gaming consoles promised a boom phase for VR technology in gaming. Arcade game machines were packaged with stereoscopic 3D visuals, and visors accounted for head mounted displays (HMDs) offering two displays with resolutions of 276px X 372 px. However, high deployment costs stalled the technology s growth during this period. VR technology for gaming started witnessing interest in the 2000s again with players such as NVidia and ATi who brought in cost-effective and competitive hardware. Xbox by Microsoft and PlayStation by Sony brought VR in gaming into the limelight once again. Mid 2000s marked renewed interest in this technology and resulted in a rise in patenting activity, with significant increase in patent filing starting A growing market for gaming consoles and rising demand for a more immersive experience is likely to have contributed to a drastic increase in filing of patents in data processing and interactive hardware categories starting

8 Head mounted displays (HMD) are a key hardware component. From the bulky complicated device invented by Ivan Sutherland in 1968 for The Sword of Damocles, to the lightweight Google Cardboard launched in 2015, HMDs have evolved and emerged to make 3D visualization powerful, compact and convenient. Of the 3,720 IP assets that irunway identified in the VR/AR gaming space, 2,837 are granted patents and 883 are patent-pending applications. Figure 3 charts the overall patent filing and grant trend in this technology domain. Over 65% of IP assets constitute data processing technologies, which is the most essential aspect of VR. Hardware and software components aim at rendering realistic 3D experience. Image data obtained from the environment or generated through artificial models are processed to enable 3D immersive and semi-immersive visualization. Research in VR started in the early 1990s aligned to the 1991 launch of Virtual Reality Arcade in London. Nintendo was among the early players, and released the Virtual Boy in 1995 which was touted as the first console capable of displaying stereoscopic 3D graphics. This bolstered research in VR. That same year, a Seattle group publicly demonstrated the Virtual Environment Theater, a 270 degree immersive room. Patenting activity increased steadily starting 2005, two years prior to the launch of Google Street View in However, there was a sudden spike in filings in 2009 owing to the development of the first Oculus Rift prototype in The next landmark year was 2014 when Facebook acquired Oculus, and Sony announced Project Morpheus (codename for PlayStation VR). Google introduced Cardboard, a DIY stereoscopic viewer for smartphones bringing VR to the consumer market Figure 3: Patent filing and grant trends in VR in gaming in the U.S. region. 08

9 Analysis of Level 1 Categories Until 2007, research was focused on developing interactive hardware for seamless input and output of data, and internal chips to support a 3D experience. However, the urge to provide a virtually real experience through advanced feedback and haptics mechanisms, and management of content displayed for a VR experience has resulted in increased patent filing in feedback and control technology. The number patents filed to support VR content has outnumbered hardware IP assets in the last five years. However, since hardware internal chips-related IP assets aim to address data and graphics processing, it can be comprehended that a portion of the data processing IP assets includes hardware related claims. Figure 4 charts the patent filing trend across Level 1 categories Data processing Interactive hardware Feedback and control Internal chips Content Figure 4: Patent filing trend across Level 1 categories in the U.S. region DATA PROCESSING The VR gaming industry enjoys a strong consumer demand for a more realistic experience. Data processing technologies are required to address the growing need for data compression, data storage and manipulation, image compression and 3D modeling. Methods of efficiently projecting 2D images in 3D for an immersive experience requires complex 3D image rendering techniques. A prime reason for VR in gaming failing to attract the consumer market in the 1990s was its inability to resolve motion sickness that users experienced after playing VR games. In the last few years, however, there has been a strong R&D focus to improve image rendering and refresh rate to resolve motion sickness. About 65% (2,423 IP assets) of the IP assets have been 09

10 Virtual Reality in in Gaming: U.S. Patent Landscape 1,600 1,460 1, Data manipulation Data storage Simulation 3D modeling Image rendering/ processing Figure 5: Distribution of IP assets across Level 2 categories under Data Processing Data manipulation Data storage 3D modelling Simulation Image rendering /processing Figure 6: Patent filing trend across Level 2 categories under Data Processing (Source: irunway analysis based on patent data from Innography.) 10

11 categorized under data processing. Efficient distortion aware image-based techniques are being developed for special effects to ensure effective rendering of 3D images. There is significant research interest to combine eye tracking with distortion-aware image rendering algorithms for eye tracking cameras. Technologies are being developed to adjust camera projection to the angle of eye movement and pupil dilation to render clear images for users with vision problems. Bloom, defocus blur and motion blur caused by rapidly changing frames required in gaming resulted in significant research in image rendering and processing as seen in Figure 5. The filing trend of key Level 2 technologies represented in Figure 6 shows an early and significant increase in patent filings in data processing. Until 2009, data manipulation and image rendering were key research areas. After that, while image rendering witnessed significant increment in patent grants, there was a steady growth in patenting in data storage, 3D modelling and data manipulation. Increased research in data storage can be attributed to Samsung, Google, HTC and Apple that are bringing VR to gaming in mobile devices. Mobile devices come with limited space and demand enhanced data storage and processing capabilities. Applications and software in areas such as image stitching (for panoramic images and VR photography) and Google s Street View (for an AR experience) started gaining research popularity after INTERACTIVE HARDWARE Head mounted displays have become synonymous to VR gadgets. Most commercially available VR products such as Samsung VR, Oculus Rift, Google Cardboard and Microsoft HoloLens provide an immersive experience while allowing untethered mobility in an interactive space. To enhance user experience, VR devices also enable man-machine interactions such as audio input Display devices I/O devices Figure 7: Distribution of IP assets across Level 2 categories under Interactive Hardware 11

12 Virtual Reality in in Gaming: U.S. Patent Landscape and output, gesture and touch input, eye and head movement tracking. With advances in VR headsets and interoperability of various technologies, companies are aiming to develop display devices that are compatible with multiple platforms. HTC Vive and Lenovo 3 are developing standalone VR headsets for Google Daydream. These headset aim to operate without a mobile phone or PC. Interactive hardware constitutes 26% (976 IP assets) of the total IP assets in the VR gaming patent portfolio. Figure 7 shows that 620 IP assets (16%) are related to displays devices and 446 IP assets (11%) to I/O devices. The launch of the first prototype of Oculus Rift in 2010 increased patenting activity in display devices exponentially. Thereafter, companies have been racing to launch VR headsets. Google Cardboard and Sony PlayStation VR were announced in 2014 followed by the release of Samsung Gear VR in 2015, and Oculus Rift and HTC Vive in Figure 8 charts the patent filing trend in interactive hardware technologies. INTERNAL CHIPSETS A growing demand for enhanced graphic technologies that offer an immersive VR gaming experience has bolstered R&D by chipmakers. There is an increasing emphasis on GPU manufacturers to develop processors that can render 3D content more realistically. With VR devices getting more compact and independent, there is a growing need for processors and chipsets to be included in a single device. This demands that the chipsets are compact and light in weight, exhibit complex processing capabilities and consume less power. The VR in gaming industry today is reinforcing visual computing and graphic processors to provide a visually realistic experience to users. A combination of graphics Display devices I/O devices Figure 8: Patent filing trend across Level 2 categories under Interactive Hardware

13 Display devices Communication/ Interface devices Figure 9: Distribution of IP assets across Level 2 categories under Chipsets Communication/ Interface devices Processing devices Figure 10: IP Filing trend of VR in gaming IP assets across Level 2 categories under Chipsets 13

14 Virtual Reality in in Gaming: U.S. Patent Landscape processor units (GPUs), central processor units (CPUs) and processors and communication interfaces. display devices such as HMDs are essential to render a VR experience. Since VR demands real time calculations, companies and coders are moving workloads off the main CPU to a wide range of alternative processors. This helps avoid lag time caused by remote communication. Because complex gaming requirements can no longer be handled on CPUs alone, companies such as Microsoft, Google and Amazon are shifting processing loads to FEEDBACK AND CONTROL Human-machine interface is key to VR technology. The Telesphere mask introduced in the 1960s was the first head-mounted display 5. It included stereoscopic 3D and offered wide vision, but lacked motion tracking features. Lack of portability and comfort of use, coupled with bulkiness, resulted in a failing market 6. GPUs, field programmable gate arrays (FPGAs), and custom-built AI processor 4 chips. Neural networks and other forms of AI are the driving forces behind this shift. Among the many VR gaming interactive hardware, Microsoft s Kinect was a breakthrough. Launched in 2010, this motion sensing gaming system created a digitized Communication and Interface devices can couple with I/O devices and processors to bridge the gap between the physical and digital world. Sensors and corresponding recognition processors play key roles in leveraging interactive hardware to provide better feedback and control. Figure 10 charts the patent filing trend in version of the user to interact in a virtual environment. Soon after, Apple s iphone Virtual Reality Viewer, Oculus Rift, Tactical Haptics VR, Motion Controller, Google Cardboard, Samsung Gear VR and Microsoft HoloLens have focused on creating better interactive hardware products and systems Motion tracking Feedback mechanisms/ Haptics Figure 11: Distribution of IP assets across Level 2 categories under Feedback and Control

15 Feedback mechanisms/ Haptics Motion tracking Figure 12: Patent filing trend across Level 2 categories under Feedback and Control (Source: irunway analysis based on patent data from USPTO) There is a rapid emergence of new display technologies customized to VR gaming requirements. Haptic feedback is being actively researched to provide gamers with a more immersive experience. Direct retinal stimulation, head motion tracking and body movement monitoring are important for a complete sensory experience. There is significant R&D effort in haptics and feedback mechanism technologies. Immersion (150 IP assets) and Microsoft (55 IP assets) are key contributors towards developing this technology. Feedback and Control constitutes 16% (607 IP assets) of the landscape. Figure 12 charts the patent filing trend in these two categories. CONTENT Early VR content was coded to suit a specific hardware and software. The advent of multiple VR devices has increased the demand for content that is independent of a platform or device. Growing consumer demand for improved hardware has also promoted research for better visual content. The adoption of AR and VR headsets will be driven by the introduction of less expensive content powered by smartphones. While sale of VR headsets involves a large portion of the VR market share, a sustainable market is likely to be dependant on VR content that can attract consumers to a device. According to Technavio 7, the global VR content market is expected to grow at a CAGR of close to 128% during 2016 through irunway s analysis found Content constituted 23% (875 IP assets) of the landscape, with 15% of the patent filing activity in content sharing technologies

16 The launch of the Oculus Rift prototype in 2010 accelerated the creation and sharing of content to enhance virtual reality efforts. Figures 14 charts the patent filing trend across Content technologies Content access Content sharing Avataar/ Character features Software plugins Figure 13: Distribution of IP assets across Level 2 categories under Content Content access Software plugins Avatar /character features Content sharing Figure 14: Patent filing trend across Level 2 categories under Content 16

17 Major Patent Portfolios Microsoft, Immersion and Sony are vying for a larger share of the VR patent landscape, leading the list of the top 10 patent portfolio owners. Microsoft is heralding the development of its HoloLens technology, and leads with 545 IP assets (475 granted patents and 70 pending applications). Immersion Corp. owns the second largest patent portfolio with 201 IP assets (192 granted patents and 9 pending applications), leading in feedback and control technologies. With the third largest portfolio comprising 185 IP assets (153 granted and 32 pending U.S. patent applications), Sony is focusing on multiple technology areas. Microsoft owns the largest number of patents across all categories, except feedback and control. It also has significant patents related to its Kinect motion sensing and gaming devices, which HoloLens traces its lineage to. The company invented a third processor: a holographic processing unit or HPU. However, HoloLens also includes a CPU and GPU to render a 3D environment in front of a user. Since the HoloLens is an untethered device and doesn't need a PC connection or phone to operate, its battery life is important to offer an uninterrupted experience. Microsoft s IP assets are distributed across 431 INPADOC families, with major R&D in data processing and hardware. Sony has a strong R&D focus in display devices. It has been developing 3D modelling and simulation technology for PlayStation VR, and HMD technologies. Despite being a late entrant, Alphabet has garnered a fair market share with the Google Cardboard. Alphabet, through Google, started patenting in 2007 and is developing a balanced portfolio across data processing, display devices and content sharing Microsoft 195 Immersion Corp Sony Nintendo Intellectual Ventures Samsung Qualcomm Apple Alphabet Nokia Granted Patents Applications Figure 15: Top 10 Patent owners in VR in Gaming patent landscape in the U.S. region 17

18 Data processing Interactive hardware Internal chips Feedback & control Content Microsoft Sony Qualcomm Apple Alphabet Immersion Corp. Samsung Nintendo Nokia Intellectual Ventures Figure 16: Distribution of IP assets across Level 1 categories in the portfolios of top 10 assignees in the U.S. region Level 2 Microsoft Immersion Corp. Sony Nintendo Intellectual Ventures Samsung Qualcomm Apple Alphabet Nokia Data manipulation Data storage Simulation 3D Modeling Image rendering/ processing Display devices I/O devices Processing devices Communication/ Interface devices Motion tracking Feedback mechanisms/ Haptics Content access Content sharing Avataar/ Character features Software plugins Table 2: Distribution of Patents across Level 2 categories of the Top 10 Assignees in the U.S. region 18

19 In 2012, Microsoft filed 111 patent applications, 79 of which were related to image rendering and processing. This can be traced to Microsoft s research in image processing and projection to achieve early virtual realization of digital objects as seen in its Holodesk 8 and MirageTable 9. Figure 17 charts the patent filing of the top five patent portfolio owners in Virtual Reality in gaming Microsoft Immersion Corp. Nintendo Intellectual Ventures Sony Figure 17: Patent filing trend of the Top 5 Assignees in the U.S. (Source: irunway analysis baased on patent data from USPTO)

20 SEMINAL PATENT ANALYSIS irunway analyzed the VR gaming patent landscape to identify high value patents with immense potential for monetization. The 2,837 granted U.S. patents in VR in gaming were ranked using irunway s proprietary portfolio analysis solution COMPASS SM to identify seminal (or, strong) patents. The patent set was assessed on multiple parameters, including number of independent and dependent claims, backward and forward references, patent age, etc. The top ~10% comprising 300 granted patents were found to be seminal and considered for further analysis. Immersion owns the largest number of seminal VR gaming patents with a 20% share. Four of the top 10 IP asset owners - Microsoft, Immersion, Sony and Intellectual Ventures - feature in the top 10 seminal patent holders list and own a cumulative 44% of the total seminal patents. Figure 18 charts the top seminal patent holders. Mq Gaming, Broadcom and Nantworks own a fair share of seminal patents despite their relatively small portfolio size. 23 of Broadcom s 25 VR gaming patents are seminal. Most of Broadcom s VR gaming IP assets relate to image rendering and processing, processing devices and display devices. Mq Gaming has 41 granted patents distributed across technologies related to feedback and control, data processing and interactive hardware. Of these, 33 are seminal patents. 23 of Nantworks 30 granted patents are seminal, as shown in Table 3. 32% of Immersion s VR in gaming patents, and 20 of Broadcom s seminal patents will expire by Figure 19 charts the share of seminal patents that will expire by 2020 for the top 10 seminal patent holders Immersion Mq Gaming Microsoft Sony Broadcom Nantworks Intellectual Ventures LG Electronics Blast Motion Qualcomm Figure 18: Leading Seminal Patent Owners in the VR Gaming Landscape in the U.S. (Source: irunway analysis based on patent data from USPTO) 20

21 Assignee Total IP Assets Number of Seminal Patents Percentage of Seminal Patents Immersion Corp % Mq Gaming % Microsoft % Sony % Broadcom % Nantworks % Intellectual Ventures % LG Electronics % Blast Motion % Qualcomm % Table 3: Percentage of Seminal Patents within the portfolios of the Top 10 Seminal Patent Owners Immersion Mq Gaming Microsoft Sony Broadcom Nantworks Intellectual Ventures LG Electronics Blast Motion Qualcomm Expiry by 2020 Expiry after 2020 Figure 19: Health Analysis of the Top 10 Seminal Patent Portfolios in the U.S. VR in Gaming Landscape 21

22 irunway analyzed the seminal patents of the top 10 assignees across the taxonomy and found that more than 75% of Immersion s seminal patents are related to feedback mechanisms/haptics. Over 50% of Mq Gaming s and Microsoft s seminal patents, and about 80% of Broadcom s and Nantworks seminal patents are related to image rendering/processing technologies. Table 4 charts the distribution of seminal patents across Level 2 categories in the portfolios of the top 10 seminal patent owners. Level 1 Level 2 Immersion Corp. Mq Gaming Microsoft Sony Broadcom Nantworks Intellectual Ventures LG Electronics Blast Motion Qualcomm Content Data processing Interactive hardware Content access Content sharing Avatar/character features Software plug-ins Data manipulation Data storage Simulation D Modelling Image rendering/processing Display devices I/O devices Internal chips Feedback and control Processing devices Communication/Interface devices Motion tracking Feedback mechanisms/haptics Table 4: Distribution of Seminal Patents across Level 2 categories in the portfolios of the Top 10 holders 22

23 KEY PLAYERS IN THE LANDSCAPE Microsoft Prior to the Hololens, Microsoft researched several projection systems that could lay the foundation for advanced virtual reality systems that they own today. In 2011, the company introduced Holodesk 10 that used a Kinect to allow interaction with virtual 3-D objects. Kinect is a motion sensing device that uses depth sensing technology to recognize parts of a user s body such as hands and head, coupling it with virtual objects to render a virtual effect. This involved image projection and analysis. Microsoft also introduced the MirageTable 11 in 2012 that used a 3D-video projector to beam images onto a sheet of curved white screen in front of a user. These early attempts at realizing digital objects required R&D in image rendering and processing. Microsoft is the largest patent owner in VR gaming technology, with 475 granted patents and 70 pending applications. Its granted patents comprise of 31 seminal patents, mostly related to image rendering and processing. The company has also built a strong portfolio of inventions for HMDs and content generation. Microsoft has filed several patent applications related to augmented reality glasses. Patent application US titled Multiplayer game invitation system indicates Microsoft s interest to integrate AR into gaming by combining it with the Xbox or for use in its HoloLens. Microsoft has a large number of patents which relate to holographic processing units (HPUs) and graphic processing units (GPUs). One such patent, US , describes a method which can be used to improve GPU efficiency. The disclosed technology accelerates general matrix operation by partitioning processes into a sequence of passes through a macro step which can be used in VR gaming related GPUs. In October 2014, Microsoft s researchers revealed a concept of RoomAlive to transform any room into an immersive and augmented Figure 20: Patent US describes a method to invite potential players in a multiplayer gaming environment using voice commands, eye tracking information, depth information, facial recognition, and a potential player s HMD device information. This image from the patent describes the HMD. Figure 21: Patent US discloses programmable pixel shaders and parallel architecture of a GPU. Pixel shaders are used to implement operations in machine learning algorithms and non-graphic applications. Parallel architecture of the GPU hardware allows training and execution of algorithms faster on GPU

24 Data processing Interactive hardware Internal chips Feedback & control Content Data manipulation Data storage Simulation 3D modelling Image rendering/ Processing Display devices I/O devices Processing devices Communication/ Interface devices Motion tracking Feedback mechanisms/ Haptics Content access Content sharing Avatar/ character features Software plugins Figure 22: Distribution of Microsoft s VR in Gaming Patents across Level 1 and 2 categories entertainment experience using a Kinect sensor and projectors 12. In April 2015 Microsoft made a RoomAlive developer tool kit available but since then it has been silent about its progress or launch intentions. Of the 545 patent assets, a major share of Microsoft s IP assets pertain to image rendering and processing software (236 assets). It also has a strong portfolio in data manipulation (112 assets) and 3D modelling patents (92) aligned to its research. Microsoft owns 170 patent assets in interactive hardware technology, primarily involving HMDs (99 IP assets). Figure 22 charts the distribution of Microsoft s patent portfolio in VR in gaming technology. Microsoft has engaged in active in-house R&D to develop its patent portfolio. Of the 545 patents, it acquired only 14 patents; the rest were developed in-house. In 2014, Microsoft purchased wearable computing patents worth $150 million from Osterhout Design Group (ODG) 13, of which only six are related to VR in Gaming technology. Microsoft has also engaged in an IP licensing deal with Edge3 Technologies 14. Edge3 Technologies provides patent licenses related to gestural and movement tracking technology using time-of-flight (TOF) camera sensors essential to VR in gaming. Microsoft s patent filing trends in Level 1 categories as seen in Figure 23 shows marked activity in data processing technology. Content sharing and interactive

25 hardware has attracted research interest in recent years. Development of holographic content for Hololens and enhancing HMDs to incorporate user-friendly features appears to be other areas of focus Data processing Interactive hardware Internal chips Feedback and control Content Figure 23: Patent Filing Trend of Microsoft across Level 1 Categories 25

26 Immersion Corporation Immersion is a market leader in touch feedback (haptics) technology. Founded in 1993, it has collaborated with granted patents as specifically relevant to VR in gaming. Inventor and founder, Louis B. Rosenberg, spearheaded these inventions with 56 patents awarded to him. multiple VR product developers and researchers to provide a holistic immersive experience. In 1997, Immersion collaborated with Microsoft to integrate TouchSense technology into Microsoft's DirectInput API for DirectX The companies also worked on DirectX 6 and 7, and signed an agreement in 1999 to share each other's "feel simulation" technology. Immersion also collaborated with Nintendo to provide haptic technology to Nintendo Switch, which is envisioned to work with a VR headset. Immersion owns 204 patents and leads the feedback and control category with 73% of its portfolio focusing on haptics technology. While its patent US dealt with simple force feedback, a more recently granted patent, US , discloses a system and method to provide complex haptic stimulation for interacting in a virtual environment. The patent claim includes a first stimulation determined in response to the performance of the initial portion of a control gesture, and a second Despite its stronghold in haptics feedback, Immersion has been aggressive in asserting its patents against potential competition. It has initiated patent litigation against stimulation that is different from the first stimulation and is determined responsive to performance of the first intermediate portion of the control gesture. Microsoft, Sony, Apple, Nintendo and HTC since Immersion s patent filing trend suggests aggressive Focusing on force and tactile feedback technology, Immersion built a strong patent portfolio to protect its innovations. Immersion has more than 597 granted patents in the U.S. irunway identified 195 of these 597 patenting activity till Post 2005, while Immersion s litigation activities increased, its patenting activities slowed down. 61% (126 patents) of Immersion s VR in Gaming patents are expected to expire by Figure 24: Patent US discloses a method to provide haptic feedback using an actuator in response to a controller receiving input signals from an input device. Figure 25: Patent US discloses haptic stimulation that includes a first stimulation in response to an initial portio of a control gesture. A different second stimulation is determined based on the response of an intermediate portion of the control gesture

27 Data processing Interactive hardware Internal chips Feedback & control Content Data manipulation Data storage Simulation 3D modelling Image rendering/ Processing Display devices I/O devices Processing devices Communication/ Interface devices Motion tracking Feedback mechanisms/ Haptics Content access Content sharing Avatar/ character features Software plugins Figure 26: Distribution of Immersion s VR in Gaming Patents across Level 1 and 2 categories Data processing Interactive hardware Internal chips Feedback and control Content Figure 27: Patent Filing Trend of Immersion across Level 1 categories 27

28 Sony On 14 March 2014, Sony announced the development of Project Morpheus as an add-on to the PlayStation 4. Morpheus was envisioned to provide an immersive experience to gamers 17. The system includes an HMD which works seamlessly with the PlayStation camera to track head orientation and movement and allow the virtual world to move in tandem. Sony s attempt to make HMDs (Tiles) 18 and handheld displays (MagicBook) 19 for VR and AR in the early 2000s laid the foundation for the PlayStation VR, which it launched in October This console comprising a full-hd LCD display and an optical unit with about a 100 o field of view. The device consists of a high sensitivity accelerometer and gyroscope to detect the user s head movement and provide 3D video and surround sound,. Sony owns a patent, US , titled Head mountable display, which describes an HMD device and its operation. The device is fitted with a camera to detect and capture the viewer s eye and head movements. This data is used to estimate the relative distance between the viewer s eyes and the viewed frame. Morpheus also allows players to use a PlayStation Move (PS Move) Motion Controller as an object, such as a sword within the game to provide a physical feel. Sony has also filed patents describing technologies that assist in detecting and tracking hand movements. Patent application US describes a hand held input device and works based on users hand movements. Sony is an early researcher in VR gaming, with patenting activity dating to Figure 31 charts the distribution of Sony s patents across the taxonomy. Of the total 185 patents, a major share of 93 patents belongs to image rendering and processing software. Sony has 60 patents in interactive hardware, of which 28 patents describe display devices and 39 describe I/O interfaces. Sony s patent filing trend across Level 1 categories as seen in Figure 30 shows marked activity in data processing related technologies since There has also been healthy R&D in content sharing and interactive hardware. Development of PlayStation 4 to enable a VR experience through a HMD required rendering of image data to 3D projections. Sony s focus to provide enhanced audio I/O devices, camera and sensors for an immersive experience can be gauged from its interactive hardware patents. Figure 28: Patent US discloses a head mounted display arrangement for automatically determining an interoccular distance of a user

29 Data processing Interactive hardware Internal chips Feedback & control Content Data manipulation Data storage Simulation 3D Modelling Image rendering/ Processing Display devices I/O devices Processing devices Communication/ interface devices Motion tracking Feedback mechanisms/ Haptics Content access Content sharing Avatar/ character features Software plugins Figure 29: Distribution of Sony s VR in Gaming Patents across Level 1 & 2 categories Data Processing Interactive Hardware Internal Chips Feedback and Control Content Figure 30: Patent Filing Trend of Sony s VR in Gaming Patents across Level 1 categories 29

30 Samsung Samsung actively started researching virtual reality in the early 2010s. In 2010, Samsung s researchers published a work-in-progress project 20 related to 3D user interface that included gaze and hand gesture tracking. It described a virtual tour application that allowed users to navigate through the virtual avataar of popular places. Samsung owns a patent, US , which describes a smartphone with an HMD. It describes a compact information terminal combining a HMD with a smartphone. It enables tracking a user's motion as key input without any separate input device. This notion emerged as the primary concept for Samsung Gear VR and Google Cardboard. It also created a buzz in the technology world, similar to what Oculus Rift did in Samsung released the Gear VR headset in September , building it around the Samsung Galaxy Note 4 Smartphone. Two built-in lens focus the user s vision onto the Galaxy Note 4's screen 22. The device also comes with a rocker that can adjust focal length for the convenience of users with vision errors. Patent US powers this technology. Gear VR includes the Oculus Home, a program that allows a user to stay inside a virtual world while browsing applications and videos. The user can navigate using a touchpad on the side of the device, enjoying a seamless experience. Samsung has also been researching actively in facial image rendering. This technology plays a key role in Avatar creation and in game features dependent on facial movements. Samsung s patent application US describes a technology for managing communication using an HMD, which generates data based on the user s facial movements. Figure 35 charts the distribution of Samsung s patent portfolio across the taxonomy. Of the 105 patents, 18 patents are related to HMD technology. Samsung s patenting trends suggest a balanced focus on interactive hardware, internal chips and content, apart from data processing observed a peak in patenting trend marked by the launch of Samsung Gear VR in Figure 31: Patent US discloses a combined HMD information terminal (PDAs and Smartphones) to sense a user s motion as input, without the use of a separate input device. It leverages a single image generating unit in an HMD to simulate a large screen due to increased angular magnification

31 Data processing Interactive hardware Internal chips Feedback & control Content Data manipulation Data storage Simulation 3D modelling Image rendering/ processing Display devices I/O devices Processing devices Communication/ interface devices Motion tracking Feedback mechanisms/ Haptics Content access Content sharing Avatar/character features Software plugins Figure 32: Distribution of Samsung s VR in Gaming Patents across Level 1 & 2 categories Data processing Interactive hardware Internal chips Feedback and control Content Figure 33: Patent Filing trend of Samsung across Level 1 categories (Source: irunway analysis based on patent data from.) 31

32 Alphabet Alphabet, the parent company of Google, has emerged as PHASE 1 a new competitor in the VR market. The company has developed key VR gaming products like Daydream, Jump, Expeditions and Cardboard. Google has also invested in Project Tango where devices can track motion and understand distances and their position in the real world. It can also be used to enable smartphone AR experiences. American toy manufacturer Mattel Inc. unveiled its iconic View Master at the Toy Industry Association's Toy Fair It adopted Google s Cardboard technology to create its stereoscope View-Master to provide an immersive 360 o digital experience. Users can virtually explore historical monuments, nature and any place by pairing the View Master s experience reel with an Android smartphone 23. Google has collaborated with LG Electronics to release a Cardboard-based headset for the LG G3, known as VR for G3. Released in February , it was distributed as a free accessory with new G3 models and was perceived to be a competitor to the Samsung Gear VR accessory. Alphabet s portfolio comprises 60 IP assets, of which 45 are granted patents. The company acquired many patents from Outland Research 25, a company run by inventor Louis B. Rosenberg. Patent US titled Method and apparatus for a verbo-manual gesture interface, describes an interface system that is responsive to a user s verbal commands and performs physical manipulations based on speech commands. Figure 35 charts the distribution of Alphabet s VR gaming patent portfolio. Of its 60 IP assets, 26 IP assets relate to image rendering and processing. Interesting content is key to retain consumers and forms an important area of research focus for Google. To encourage content developers, Google provides three software development kits to espouse Cardboard applications: a Java-based kit for Android OS, C# for game engine Unity, and one for the ios. Despite being a late entrant, Alphabet has scaled up quickly, building a strong IP portfolio and impressive product line. As seen in Figure 36, Alphabet has been strengthening its portfolio in data processing with aggressive patent filings, while consolidating its VR device and content through steady patenting activity. Figure 34: Patent US describes a wand-shaped manipulandum that senses gesture motions and voice commands

33 Data manipulation 15 Data Processing Data storage Simulation 3D modelling Image rendering/ processing Interactive Hardware Display devices I/O devices 10 9 Internal Chips Processing devices Communication/ Interface devices 3 5 Feedback & Control Motion tracking Feedback mechanisms/ Haptics 1 10 Content Content access Content sharing Avatar/ character features Software plugins Figure 35: Distribution of Alphabet s VR in Gaming Patents across Level 1 categories (Source: irunway analysis based on patent data from Innography.) Data processing Interactive hardware Internal chips Feedback and control Content Figure 36: Patent Filing trend of Alphabet across Level 1 categories (Source: irunway analysis based on patent data from Innography.) 33

34 KEY LITIGATION AND LICENSING Of the 2,837 granted U.S. patents in the VR in gaming U.S. patent landscape, 41 patents have been involved in 85 litigation, 26 inter parte reviews (IPRs) and 10 ITC cases. Ten of the 41 patents are due to expire by Virtual Immersion Technologies filed 10 patent litigation against multiple defendants on the strength of its patent US Five of these 10 cases have settled. Abarta, LLC sued eight defendants on the basis of its acquired patent US , and settled all cases. Broadcom has settled seven of its nine VR gaming related patent litigation. An analysis of the top 10 litigated patents in the VR in gaming domain revealed that five patents are owned by Immersion Corporation alone. These five patents have been involved in 9 PTAB appeals and 4 patent litigation. Immersion has settled all four litigation filed against Apple, Plaintiff Virtual Immersion Technologies Abarta LLC Broadcom Immersion Corp. Aleksandr L. Yufa No. of cases Table 5: Top 5 plaintiffs of VR gaming technology patents in the U.S. (Source: irunway analysis based on patent data from Innography.) HTC, Lenovo and AT&T; and two PTAB appeals filed by Apple. Broadcom s patent US is seminal and the second largest litgated patent in this space. This patent underwent reexamination in 2009 which strengthend the patentability of claims 1 to 22. Though Broadcom securitized this patent to Bank of America in 2016, it released the security in 2017 post its acquisition by Avago and used it to file six patent litigation against Sigma Designs, LG Electronics, SiRF Technology, MStar Semiconductor, MediaTek, Vizio Inc., and Funai Electric Company Ltd. Major Litigation 2002 Immersion sued Sony and Microsoft for infringing on its patents US and US These are related to vibration functions in gaming controllers 26. While Microsoft settled out of court purchasing a 10% share in Immersion 27, Sony continued to defend the case and subsequently lost. It had to compensate Immersion with $97.2 million as per the court judgment, along with royalties totaling to about $53.1 million Motiva 29 filed a lawsuit against Nintendo accusing the company for violating its technology disclosed in US related to remote controllers used in the Wii video game system. Nintendo won a U.S. appeals court decision which will allow it to keep importing Wii system into the United States

35 2010 IA Labs filed a suit against Nintendo claiming the company violated its patented technology disclosed in US in two of its products the Wii Fit and Wii Fit Plus 31. The court ruled in favor of Nintendo in In January 2014, Nintendo acquired the entire patent portfolio of IA Labs Tomita Technologies filled a lawsuit against Nintendo accusing Nintendo 3DS of infringing on its 3D camera technology. This technology allows users to view 3D images without the need of special glasses. It is disclosed in Tomita Technologies patent US The court ruled in favor of Tomita Technologies, and Nintendo had to pay royalty fee of 1.82% of the wholesale price of each 3DS device sold 33. It also had to pay an additional amount of over $200,000 in supplemental damages and prejudgment interest. Nintendo filed an appeal to the judge s decision on the grounds of incorrect interpretation of a key element in the patent. In April 2016, it was ruled by the New York Federal Court that Nintendo had not infringed on Tomita's patent In March 2012, World INC. sued Activision Blizzard for infringing four of its patents - US , US , US and US related to systems and Publication no. Plaintiff No. of cases Level 2 category Expiry year US Virtual Immersion Tech. 10 3D modeling 2020 US Broadcom 9 Processing device 2019 US Abarta LLC 8 3D modeling 2025 US Immersion Corp. 5 Feedback mechanisms/ haptics 2022 US Skyline Software Systems 5 3D modeling 2019 US Skyline Software Systems 4 3D modeling/ content sharing 2017 US Immersion Corp. 4 Feedback mechanisms/ haptics 2024 US Immersion Corp. 4 Feedback mechanisms/ haptics 2024 US Immersion Corp. 4 Feedback mechanisms/ haptics 2023 US Immersion Corp. 3 Feedback mechanisms/ haptics 2020 Table 6: Top 10 VR in Gaming patents under litigation in the U.S. (Source: irunway analysis based on patent data from Innography.)

36 methods for users to interact in a virtual space 34. Activision Blizzard s World of Warcraft and Call of Duty video games have been identified in the complaint as infringing on Worlds patents 35. Markman hearing for this case was completed in US , US , US , US and US Immersion Corp. has also alleged Apple s infringement of its patent US in its Retina display technology. In November 2012, Virtual Reality Feedback Corp. filed a lawsuit against Amazon for infringing two of its tactile sensation generator patents US and US The company has also sued Microsoft, Sony, Target Brands and Walmart for infringing patent US However, all the patent litigations are closed In 2013 Skyline Software Systems accused Apple for infringing its media-related patents used in the Apple Flyover feature. The patents in question were US and US The company also sued Microsoft and Alphabet. However, all the patent litigations are closed In 2015, Convergent Media Solutions accused Netflix for infringing its video playback related technology protected by patents US and US This technology features in the Netflix application. The company also sued AT&T for U-Verse application In , Immersion Corp. filed a lawsuit against Apple over its patented haptic technology used in 3D Touch functionality and Force Touch trackpad used in iphones, MacBook and Apple Watch. The patents in question were

37 CONCLUSION In the last 20 years, virtual reality has emerged as a game changer. It has defined a new dimension to experience gaming and entertainment. Virtual reality has attracted interest from various technology companies and encouraged collaborations between them to provide an ecosystem, with gaming being the most promising application area. Data processing is the backbone of virtual reality systems. Image and graphics processing, three dimensional image rendering, and algorithms and memory management for low latency processing is at the core of data processing. Interactive hardware, including HMDs and controllers, were the most obvious areas of focus of virtual reality device manufacturers. Haptics provides the required sensorial and kinesthetic feedback for an immersive experience. The VR in gaming patent landscape is dominated by Microsoft. Focusing on mixed reality, Microsoft has developed its own holography-based HMD device which allows a sub-immersive experience. This focus is expected to strategically position Microsoft in the virtual reality landscape with equal opportunities to delve either into AR or VR, based on market growth. Also, unlike Oculus and Sony, Microsoft's research and product (Hololens) is aimed to find widespread application beyond gaming; for example, in healthcare, military and engineering. Microsoft has developed a proprietary holographic processing unit (HPU) for its Hololens and is collaborating with Asus, Dell, Lenovo and HP to develop cost-effective VR headsets to dominate the VR market. Though a late entrant in the virtual reality domain, Alphabet has emerged as a strong competitor owing to its affordable low cost VR product, Google Cardboard. Making a strong presence in the mobile-based VR market, it has strengthened its patent portfolio with 906 patents and pending applications, 60 of which pertain to gaming. Alphabet is advancing its technology from Google Glass and Cardboard, and is paving its way to develop Google Daydream and Jump. In recent years, litigation in the VR gaming space has been dominated by Immersion, a leader in haptic feedback technology. It has filed several cases against Microsoft, Motorola, HTC and Apple. In the years to come, researchers will aim to make the technology more consumer friendly. Low cost HMDs, mobile-based virtual reality devices and DIY kits will encourage the use of VR technology in everyday life. Advancement in telecommunication, high efficiency graphics processors and cheaper display technologies can expedite the growth of virtual reality. 37

38 APPENDIX Content: Content in virtual reality includes the operations performed to make VR related content available to the user. This category head includes technologies required to process avatar or in game character features, and software plugins necessary to run the content. Several technologies describe the processes undertaken in order to make the content accessible to users, including technologies related to data transfer from the server, access rights, authentication, etc, which have been included under this taxonomy head. Technologies related to content sharing between users, like in multiplayer gaming, chat rooms, etc., are also included under the Content category. Avatar/character features: This category includes patents describing methods/processes adopted to develop in game character features. It also includes patents disclosing development of 3D Avatars for virtual reality games. Content access: Content Access comprises technologies and patents that describe processes to make VR content accessible to users, including data transfer from the server, access rights, authentication, etc. Software plug-ins: Software Plugins in virtual reality includes technologies related to program codes that run on processing systems in order to make them compatible for VR applications. The plugins are additional components which add modules to existing code, that in turn add certain features in order to run VR applications. Data processing: Data processing is a key aspect of virtual reality. This category includes patents disclosing technologies related to processing of different forms of data that enable content for users. Virtual environments are rendered by processing 3D image data for a user to interact within the space for gaming applications. Image data can be rendered/ processed for different applications to enable users to see and interact in a virtual space. A large part of this category includes patents that describe methods to store data efficiently. Data processing in VR gaming is required to generate, store and modify 3D images generated to project a virtual environment. For example, data collected by various sensors in virtual reality systems is sent to a processor, which converts the data into 3D images for a VR experience; sometimes, 2D images are taken as input and modified to complete a virtual reality environment. Content sharing: Content Sharing technologies in virtual reality discuss sharing of content between users in scenarios such as multiplayer gaming, chat rooms, etc. Technologies disclosing methods/processes for the transfer of notifications/ text/ graphic content between users dynamically are also included under this sub category. 3D modelling: 3D modelling in virtual reality includes technologies that disclose methods/ processes for designing/ rendering virtual environments. The methods are used to build virtual spaces and process objects within them for gaming applications. In some applications, objects in the real world are rendered in a virtual world for user interaction. 38

39 Data manipulation: This category includes patents related to data manipulation algorithms including, but not limited to, sorting, selecting, merging or comparing data. Patents that discuss concepts related to analysis of content and indexing are also included in this category. Data storage: This category contains technologies which disclose methods related to data storage for VR gaming applications. This category comprises technologies relating to operation of databases and related methods. Image rendering/ processing: Image rendering/ processing in virtual reality is a key component of VR applications. This category comprises IP assets that disclose methods related to image data processing, with primary applications in AR/VR devices. Simulation: This category includes technologies which disclose methods for simulating different environments/ scenarios for the users. Technologies like real time Avatar simulations which creates scenarios that can be interacted with by the users are included. Feedback and control: Feedback and control in virtual reality primarily revolves around gaming set-ups. This category includes IP assets that disclose methods/ processes for audio control based on feedback received from users. Motion tracking and haptic control are two major sub categories. Motion tracking predominantly comprises methods to track user movement enabling the system to capture feedback and make relavent changes in the virtual environment. Haptic control includes technologies used to detect user feedback in the form of touch by gaming systems. These technologies are also used to deliver system response to changes in the virtual environment to the user. Feedback mechanisms/ haptics: This category discusses concepts related to feedback mechanisms, including methods/ systems used to capture and deliver feedback to users. This category also includes patents related to haptic control, i.e. technologies used to detect user feedback through touch in gaming systems. It is also used to deliver the system s response to changes in the virtual environment to the user. Motion tracking: Motion tracking includes methods to track user movement for feedback that may be captured by the system to make changes in the virtual environment, which in turn may provide feedback to users interactively. Chipsets: This category includes patents related to processors and communication interface devices. Processors include technologies related to graphics processing units/chips or processors in general. Communication interface devices include all VR gaming systems that are involved in data transfer between a local device/ terminal and an external system, or between two end components. Processors: Processor technologies are related to graphics processing units/ chips. IP assets categorized here discuss methods to increase processor efficiency. They also describe the arrangement and construction of processors and materials used, primarily to increase efficiency. 39

40 Communication interface: Patents categorized here disclose communication devices involved in data transfer between a local device/ terminal and an external system. These may include mobile phones used in head mounted virtual reality displays, gaming devices and cameras that transmit and receive data. This category also includes patents related to interface devices that are involved in the transmission of information between two end components. For example, a device which receives user input and transmits it to a server will be classified as an interface device. Interactive hardware: Interactive hardware includes all systems that run applications and code to enable user interaction in the virtual world. Display devices deliver image/ video feedback to users. This category also comprises technologies related to head mounted displays and headsets. Input hardware includes cameras, and devices that are involved in receiving data/ feedback from users. Audio devices comprise devices used to deliver audio/speech feedback. It also includes 3D surround sound speaker technologies. time frame, such as integrating multiple cameras into a single camera to capture high quality image/ video content. This category also includes patents related to audio devices that deliver audio/speech feedback to users. It also comprises patents that describe methods to improve efficiency of audio devices with the usage of improved materials that emit sounds. Active patent: This is a granted patent that is currently alive and enforceable. It is a patent whose maintenance fee has been paid and the patent has not crossed its life term. Display devices: Display devices deliver image/ video feedback to users in a virtual reality environment. It includes head mounted displays and headsets. It also comprises IP assets that disclose methods of decreasing weight and size of displays to increase efficiency of content viewing. I/O devices: Input hardware includes cameras and devices that receive data/ feedback from users. It also includes IP assets that describe methods/ processes used to capture higher quantity and quality of data in a given 40

41 AUTHORS Harish Daiya Harish is Vice President at irunway where he leads the Solutioning function and enables strategic decision making in IP monetization both from technology and business standpoint. He advises companies and legal entities on excavating licensing opportunities, and conducts technical and business due-diligence to build patent portfolio monetization programs. He has been part of multi-million-dollar licensing deals, settlements and jury-verdicts. Aditi Das Aditi is an Assistant Vice President at irunway. She provides key technical insights that help IP attorneys profoundly improve licensing and litigation outcomes. She specializes in data mining, technology and patent landscape study, and patent infringement and validity analyses. Allada Revanth Kumar Revanth is a technology consultant at irunway, specializing in data mining. He provides in-depth analyses of patent portfolios to provide lead indicators that support complex IP litigation. Shivam Gupta Shivam is an Associate at irunway. He specializes in technology and patent landscape analyses, with special interests in the networking, telecom and data management technologies. The authors acknowledge the contributions of Ashish Gupta and Purnima Lodha towards shaping this report. 41