Head Tracking for Google Cardboard by Simond Lee (slee74@student.monash.edu) Virtual Reality Through Head-mounted Displays A head-mounted display (HMD) is a device which is worn on the head with screen displays in front of the user s eyes. It is hardware that can take someone into an immersive environment called virtual reality (VR). In 2012 a company named Oculus VR successfully received $2.4 million via crowdfunding (Oculus Rift: Step Into, n.d.) to produce a HMD called the Oculus Rift (Rift) that would provide VR using stereoscopic vision (figure 1). Stereoscopy is the illusion of depth by providing each eye different corresponding images. VR provided by HMD has been a popular idea and goal for a long time, appearing in films such as the 1983 Brainstorm (Passafiume, n.d.). However, technology was not ready to serve the idea until recently and after the release of Rift development kits to the public, many companies have raced to release their own VR HMDs. Google Cardboard (Cardboard) (figure 2) is a cardboard foldout with focal lenses that turns a mobile phone into a cheap accessible VR HMD (Cardboard, n.d.). Figure 1 Oculus Rift Development Kit 2. From File:Oculus Rift, (n.d.) Figure 2 Google Cardboard. From Cardboard (n.d.) Head Tracking The Rift has a head tracking system that offers six degrees of freedom (6DoF) (figure 3) and the latency is good enough that it can offer a convincing VR environment. However, it requires a computer to run simulations and it is tethered to it - this limits the physical movements of the user. Cardboard has no such physical limitations because it is running software on the phone. Unfortunately, Cardboard does not offer 6DoF because it cannot track the position (translation) of the phone (up-down, left-right and backforward). Figure 3 The six degrees of freedom. From File:6DOF en.jpg (n.d.) The use of fiducial markers is a common technique for head tracking. The most popular type of markers are QR codes and these are used for the implementation of augmented reality (AR). AR is the augmentation of the real-world, achieved by modifying the view of reality with a computer.
Figure 4 displays an example of achieving AR and is a screenshot of the user s view observing a marker modified by AR software. The Project Figure 4 Screenshot using AR software. From ARToolKit Documentation (n.d.) The goal of the project is to add head positional tracking for Cardboard software with the support of AR fiducial markers. Although the Cardboard software development kit (SDK) has a trackposition property, the effect is virtually minimal. The SDK mainly uses the phone s gyroscope to track head rotation. The gyroscope works great for tracking head rotation, although the biggest limitation is that the phone can have drift problems. Drift problems are most obvious when the user is stationary. The SDK has a drift correction setting, however the problem is not entirely eliminated. Integrating AR into Cardboard involves using the phone s camera to point at a special marker and set an object relative to the marker s position on the screen. Figure 5 demonstrates the setup of Cardboard pointing at a single marker. The tracking data from the AR can be applied to the translation of an object. ARToolKit, an open source tracking library will be used to integrate AR. Figure 6 is an example of a pattern marker supplied by ARToolKit that is printed on paper. An ARToolKit package for Unity, a cross-platform game engine, is available to add AR functionality to applications. C# is the language used for scripting in Unity. Figure 5 Google Cardboard integrating AR for head tracking Figure 6 Pattern marker. From Download ARToolKit (n.d.)
Marker setup The location of markers determines the software logic of the tracker. There are two main approaches: outside-looking-in and inside-looking-out. Figure 7 illustrates these two setups. Figure 7 Head tracking setups. From Welch, et al. (2001) We will attempt to replicate the two setups in Unity applications. Examples of Integrating AR for Head Tracking AR tracking with markers is a cheap way to test tracking software and has been used in prototypes by companies. StarVR, a HMD by Starbreeze, uses the outside-in method (figure 8). Valve Corporation has a room where there are many markers on the walls using the inside-out method (figure 9). Figure 8 StarVR HMD. From StarVR (2015) Figure 9 SteamVR room. From Speculation: HTC Vive's (2015)
Outside-Looking-In (Outside-In) Outside-in requires placing markers on the cardboard itself, attaching an external camera to the phone and pointing at the markers. It is called outside-in because the tracking device (camera) is placed on a location external to the HMD and tracks location by looking into the HMD. ARToolKit includes a configuration for a simple cube marker that has markers on the outside layers (figure 10). Markers are placed on the sides of the cardboard one on the front, two on the sides and two above and beneath. The camera tracks the markers as the user rotates and moves their head. Figure 10 A cube with markers The camera can be either attached with a USB On- The-Go cable or WiFi/Bluetooth by having the camera connected to a computer and the phone communicating to it - using WiFi/Bluetooth introduces wireless latency. Utilising the cube marker requires inversing translations up-down, and the rotations pitch and yaw. A similar system is used by the Oculus Rift an external camera tracks installed LEDs installed on the Rift. The camera only needs to find one of the markers to track the head. Inside-Looking-Out (Inside-Out) Inside-out involves using the phone s rear camera and pointing at markers placed on the wall or surface. It is the most direct method to integrate AR for head tracking. It is called inside-out because the tracking device is placed on the HMD and tracks location by looking out. Markers are either placed flat on a surface or hanging on the wall having the markers directly facing the user (figure 5). In the spirit of Cardboard - cheap accessible VR - this option will be explored most and applied. The only extra resource this setup requires is to print markers on paper. Limitations ARToolKit uses and provides different types of marker tracking. Traditional makers use contrasting black and white patterns, enclosed in a black square. Custom markers can be made and used, the constraint is it must be in black and white. Barcode markers (figure 11) are similar to pattern markers (figure 6), the difference is the pattern is predetermined black and white squares in a regular matrix (Using 2D Barcode, n.d.). Barcode marker recognition have a complexity of constant time, so adding multiple barcode markers requires little computational costs. The less complicated a pattern, the better range of tracking is available (About Traditional, n.d.) Barcode markers were mainly used in the research to test because anything that reduces CPU usage of the phone will be practical. Figure 11 Barcode marker. From Download ARToolKit (n.d.) Head tracking with the support of AR is able to track both rotation and translation. In practice, camera tracking requires certain conditions to work nicely lighting, movement and field of view not
exceeding camera capabilities. This is a problem of finesse and is a possible area of improvement. A possible solution is the idea of client-side prediction a network programming technique used in video games such as QuakeWorld (Client-side prediction, n.d.). Testing The main methods of testing are: Running test runs on Unity using an attached webcam, Deploying on Android and using the camera viewfinder on the screen to test AR tracking, Deploying the Cardboard SDK demo scene with AR functionality. The use of multiple markers requires them to be in predetermined fixed positions. Figure 12 is an example of a multi marker configuration. Multi-markers only require one marker to be in view for active tracking. The use of 8x6 multiple markers have not caused any CPU usage problems. Figure 13 illustrates the successful head tracking result of integrating AR tracking. The combination of using gyroscope and AR have been experimented. Two techniques are possible: Using gyroscope to track head rotation and AR to track head translation only. When markers are not visible to the camera, turn off AR tracking and use the gyroscope only. When a marker is visible, turn off gyroscope tracking and use AR to track both rotation and translation. Tests for both methods above have been successful - refinement is required to align the AR marker object positions with the default camera attached to the gyroscope. Using AR to track head translation only requires refinements also. The gyroscope provides very fast and accurate head rotation tracking, however its biggest disadvantage is that it introduces drift. Tracking with AR does not have this problem, but its constraints are related to the camera imaging variables. Figure 12 A multi-marker configuration. From Download ARToolKit (n.d.) Figure 13 Example application integrating AR for head tracking
Unity Implementation For the application, when a marker is in view the camera in Unity is setup to point relative to the marker s appearance in the real-world camera view (figure 14). Marker objects are reference points in Unity that display the relationship between the markers in Unity and the realworld. Figure 15 shows the Unity marker reference a square surrounding the cube marked as BARCODE0. For static scenes, for example, an application that allows a viewer to observe a 3D model, the user will only need to place the 3D object relative to the Unity marker - the marker object is in a fixed position. Figure 14 Unity editor - AR tracking Figure 15 Unity editor - AR marker object surrounding 3D cube Figure 16 shows a cube marker scene with a cube placed on one side. Gear VR Figure 16 Unity editor - AR cube marker objects with cube on one marker Oculus VR in collaboration with Samsung released Gear VR, a HMD designed specifically for its highend phones (figure 17). It adds touch input for navigation. VR-wise it is functionally the same as Cardboard and it also shares its limitations no positional head tracking. Figure 17 Gear VR. From Samsung Gear VR (n.d.)
To Be Solved Head tracking with fiducial markers, such as QR codes, for Google Cardboard works great (provided conditions are optimal) and can be cheap to setup, however setting up the markers and placing them on certain locations adds an extra installation step to VR. Initiatives for marker-less head tracking have been explored and the current well-known solution is Project Tango (Google s Project Tango, 2014). It is a project by Google for smartphones and tablets to support full head tracking by using a combination of a motion tracking camera and an integrated depth sensor (figure 18). Acknowledgements I want to thank my supervisors Dr. Maxime Cordeil and Dr. Karsten Klein, for their guidance and support. Without them, this report would not have been possible. Professor Jon McCormack, for his hospitality at sensilab, and Dr. Julian Garcia, for giving a first-year student the opportunity to undertake this research. References Figure 18 Project Tango design. From Google's Project Tango (2014) Oculus Rift: Step Into the Game. (n.d.). Retrieved October 23, 2015, from https://www.kickstarter.com/projects/1523379957/oculus-rift-step-into-the-game Cardboard Google Developers. (n.d.). Retrieved October 23, 2015, from https://developers.google.com/cardboard/ Passafiume, A. (n.d.). Brainstorm. Retrieved October 23, 2015, from http://www.tcm.com/thismonth/article/309186 309216/Brainstorm.html ARToolKit Documentation (Tutorial 1: Tracking Stability). (n.d.). Retrieved October 23, 2015, from http://www.hitl.washington.edu/artoolkit/documentation/tutorialhistory.htm The world's most widely usedtracking library for augmented reality. (n.d.). Retrieved October 23, 2015, from http://artoolkit.org/ Create and Connect with Unity 5. (n.d.). Retrieved October 23, 2015, from https://unity3d.com/ Download ARToolKit SDKs. (n.d.). Retrieved October 23, 2015, from http://www.artoolkit.org/download-artoolkit-sdk Using 2D Barcode Markers. (n.d.). Retrieved October 23, 2015, from http://www.artoolkit.org/documentation/doku.php?id=3_marker_training:marker_barcode
About Traditional Template Marker Tracking. (n.d.). Retrieved October 23, 2015, from http://www.artoolkit.org/documentation/doku.php?id=3_marker_training:marker_about Welch, G., Bishop, G., Vicci, L., Brumback, S., Keller, K., & Colucci, D. N. (2001). High-performance wide-area optical tracking: The hiball tracking system. presence: teleoperators and virtual environments, 10(1), 1-21. Client-side prediction. (n.d.). Retrieved October 23, 2015, from https://en.wikipedia.org/wiki/clientside_prediction Speculation: HTC Vive's Tracking Uses Inivisible, Projected Markers for 360 Degree Inside-Out Tracking - Road to VR. (2015, March 2). Retrieved October 23, 2015, from http://www.roadtovr.com/speculation-htc-vives-tracking-uses-inivisible-projected-markers-for-360- degree-inside-out-tracking/ Step into the Rift. (n.d.). Retrieved October 23, 2015, from https://www.oculus.com/en-us/ File:Oculus Rift development kit 2.jpg. (n.d.). Retrieved October 23, 2015, from https://en.wikipedia.org/wiki/file:oculus_rift_development_kit_2.jpg Wheatstone, C. (1838). Contributions to the physiology of vision.--part the first. On some remarkable, and hitherto unobserved, phenomena of binocular vision. Philosophical transactions of the Royal Society of London, 371-394. Craig, J. J. (1986). Introduction io Robotics, Mechanics and Control. File:6DOF en.jpg. (n.d.). Retrieved October 23, 2015, from https://en.wikipedia.org/wiki/file:6dof_en.jpg Graham, M., Zook, M., & Boulton, A. (2013). Augmented reality in urban places: contested content and the duplicity of code. Transactions of the Institute of British Geographers, 38(3), 464-479. Samsung Gear VR Browser - Pocket Lifestyle. (n.d.). Retrieved October 23, 2015, from http://www.pocket-lifestyle.com/portfolio/samsung-gear-vr-browser/ StarVR 210-Degree VR Headset Receiving Tobii Eye-Tracking Support. (2015, September 24). Retrieved October 23, 2015, from http://www.geeky-gadgets.com/starvr-210-degree-vr-headsetreceiving-tobii-eye-tracking-support-24-09-2015/ Google's Project Tango is a smartphone with sensors to map the world around you. (2014, February 20). Retrieved October 23, 2015, from http://www.engadget.com/2014/02/20/google-projecttango-smartphone-sensors/