Proseminar - Augmented Reality in Computer Games Jan Schulz - js@cileria.com Contents 1 What is augmented reality? 2 2 What is a computer game? 3 3 Computer Games as simulator for Augmented Reality 3 3.1 Why are computer games a good simulator for AR?.............. 3 3.2 Interfaces and controls............................... 3 3.3 Selection....................................... 4 3.4 Highlighting and Annotation........................... 4 3.5 Visualization modes................................ 5 3.6 Conclusion..................................... 5 4 Computer games using components of AR related technology: Nintendo WII 6 4.1 Nintendo WII: Tracking by Infrared Technology................. 6 4.2 Highlighting and Annotation........................... 6 4.3 Nintendo WII: Motion Sensoring by Accelerometers.............. 7 4.4 Conclusion..................................... 7 5 A game designed for Augmented Reality: ARQuake 7 5.1 What is ARQuake?................................. 7 5.2 Required hardware................................. 7 5.3 Movement and interaction............................. 8 5.4 Tracking....................................... 8 5.5 The campus level.................................. 9 5.6 Monsters...................................... 10 5.7 Conclusion..................................... 10 6 Discussion: AR games - A big market? 11 6.1 The vision...................................... 11 6.2 The requirements.................................. 11 7 Conclusion 12 1
Figure 1: example of Augmented Reality 1 What is augmented reality? This article explains computer gaming via Augmented Reality. It deals with the combination of real world and computer generated data. This mostly happens by digital processed video imaginery which is augmented by the addition of computer generated graphics. Ron Azuma, one of the first scientist in the field of AR, said that Augmented reality has basically three functions: (Taken from [1]) combines real and virtual interactive in real time registered in 3d Augmented Reality (AR) is a variation of Virtual Environments (VE), or Virtual Reality as it is more commonly called. VE technologies completely immerse a user inside a synthetic environment. While immersed, the user cannot see the real world around him. In contrast, AR allows the user to see the real world, with virtual objects superimposed upon or composited with the real world. Therefore, AR supplements reality, rather than completely replacing it. - Ron Azuma, Introduction to AR (2001) Taken from [1] Figure 1 shows an example of what this might look like. It shows a real coast with a real beach and a real sky. But the lines and symbols across the image aren t real. They are augmented and have the task to inform the user about dangers like submerged mines and about distances or other geographical informations. 2
2 What is a computer game? In order to define what computer game it is necessary first to define what a game is. French sociologist Roger Caillois defined a game as an activity that must have the following characteristics: (Taken from [9]) fun: people enjoy it separate: circumscribed in time and place uncertain: the outcome of the activity is unforeseeable non-productive governed by rules: the activity has rules that are different from everyday life fictinous: it as accompanied by the awareness of a different reality Once its clear what a game is its logical that a computer game is a game that involves interaction with a computer user interface to generate visual feedback on a computer screen. Consequently, a AR game is a game that involves interaction with a AR user interface to generate visual feedback on a AR head-mount-display. 3 Computer Games as simulator for Augmented Reality This section illustrates some AR techniques used in computer games. (Taken from [7]) 3.1 Why are computer games a good simulator for AR? Applications in AR have to face many technically difficulties, e.g. unreliable tracking. Consequently, AR applications get stuck in the how to implement phase rather progressing to the what to show phase. The visualization of today s 3d game engines is fairly realistic and therefore offers a good testing environment using real-time processing. Developers therefore can save costs and time during the development process of their AR application. 3.2 Interfaces and controls The control and interface of an AR application have to be simulated by a computer game. A computer has a stationary monoscopic screen, keyboard, mouse and a gamepad. In AR there would be a head mounted display which here has to be simulated by the screen. With respect to the fact that today s games don t support 3d tracking devices (except the Nintendo WII - more about that in chapter 4), all interactions will be at a distance and use plane techniques, e.g. methods like ray-picking (Figure 2) to select and manipulate objects. Another point is that in AR the user can t freeze the camera movement entirely, while in computer games one can by just stopping movement. 3
Figure 2: plane technique: ray-picking 3.3 Selection Selection means selecting an 3d object in a 3d environment usually using the mouse. A good example for that gives a strategy game like Warcraft 3 : If one clicks on a unit there appears a sign around it. This game element uses a technique called ray-picking. If one wants to select more units one draws a rectangle around them and there appears a sign around every unit. This game element uses a technique called rubberbanding which is a extension to raypicking. AR also uses these techniques to establish interaction between the user and objects, therefore it is important to use them while simulating AR. 3.4 Highlighting and Annotation In AR applications it is often necessary to get a user s attention to specific-context dependent information. A technique which highlights an object by brighten it up or make it glowing can help. This technique is very often used by computer games and also can contribute to AR applications, e.g. if a tracked physical object for which the silhouette or 3d model is known can be highlighted by attenuating the background video buffer. For example: One sees a teapot on the table and the AR system recognizes the correct 3d model of the teapot. After the highlighting process the teapot appears brighter now because a virtual invisible teapot has been rendered on the same virtual spot by your AR system, while the rest of the field of view was darkened. While highlighting changes the appearance of an 3d object, annotation gives information about it. This means the tracked object gets an text above it s top and the user gets information about the object. This technique is used by the example strategy game Warcraft 3 aswell: if one clicks on an unit one sees the unit type and it s energy and damage level. Another form of annotation involves symbolic objects added to the scene. In a racing game it 4
Figure 3: The follow-me -car driving the perfect line. Taken from [7] is e.g. the follow-me car driving the perfect line on the street (Figure 3), in a role playing game it is the navigation map which leads the player to the next quest. AR applications can benefit from such direction manipulating techniques during information gathering. (See [4] for more info) 3.5 Visualization modes AR has the power to completely change the user s perception. If a 3d model of the environment is available, an AR system could help the user to navigate through the reality by displaying information using the mentioned techniques (selection, highlightning, annotation). Another technique is multiviewing which can provide viewing from a remote viewpoint. The user now can see his world from different viewpoints at the same time while walking around. Multiviewing is often used by computer games like e.g. Battlefield or Counterstrike. 3.6 Conclusion Today s computer game techniques can contribute a lot to AR applications. Everything which is possible in AR is possible in a computer game, because of that computer games are good simulators for AR applications. 5
Figure 4: The sensor bar of the Nintendo WII (Taken from [5]) 4 Computer games using components of AR related technology: Nintendo WII The user of an AR application usually needs a pointing device in order to interact with augmented reality objects. With reference to this case, games, the user of a e.g. virtual tennis game needs a virtual tennis bat if he wants to hit the virtual tennis ball and defeat his (maybe) virtual opponent. The virtual tennis bat is controlled by a real pointing device which is a component of Tangible Augmented Reality. Because one needs to see the position of the tennis bat he needs to find out the relative position of our real pointing device, so the tennis bat s position is displayed correctly in our augmented reality. The technique which makes it possible is called tracking. Furthermore he needs to know which rotation angle the tennis bat has in order to find out how he hits the ball. Does it hit our bat in wide angle, he probalby sliced the ball. If it hits the ball in a sharp angle, the ball got more drive. This technique is called motion sensoring and captures the motion of our pointing device itself. 4.1 Nintendo WII: Tracking by Infrared Technology 4.2 Highlighting and Annotation First you have an infrared output from the sensor bar (Figure 4, that is seen by receivers in the front of the WII-contoller. Like a remote control for your TV, so long as it is within range of the infrared reader, it can tell where about you are pointing it on the screen. If you are too far or too close it will lose your signal. 6
4.3 Nintendo WII: Motion Sensoring by Accelerometers The second technology that brings it all together is the use of motion sensors. Nintendo uses for the WII a device that detects motion by way of acceleration, an so called accelerometer (Figure 5). An accelerometer measures its own motion (locomotion) in contrast to a device based on remote sensing, in our case the sensors of the Nintendo WII. (Taken from [8]) 4.4 Conclusion The Nintendo WII uses infrared tracking and motion sensoring and therefore AR related technology. The customer gets a new feeling because of having even more control over the game using both motion sensors and infrared tracking. 5 A game designed for Augmented Reality: ARQuake This section describes ARQuake [See 2] which is an AR game developedd by Wayne Piekarski, Bruce Thomas, Nicholas Krul and Benjamin Close of the University of South Australia. (Taken from [6], [3], [2]) 5.1 What is ARQuake? ARQuake is an AR-extension of the popular first-person shooter Quake developed by ID Software. In Quake, the player runs around a virtual world, shooting at monsters, collecting objects, and completing objectives. The game is desktop-based, with the user interacting with it using a monitor, keyboard, and mouse. Although the game is relatively old, the graphics engine is very powerful and runs on a wide range of computing hardware. The AR-version game is played in the physical world, with the user able to freely move around real the world. The view is determined solely by the orientation and position of the users head. The game is experienced as augmented reality using a transparent optical see-through. 5.2 Required hardware Due to the fact that the player has to move in an outdoor environment it would be difficult to have a stationary computer. Thats why ARQuake uses a Tinmith-4 wearable computer system (Figure 5) which is all mounted on a rigid backpack. The processing is performed by a Toshiba 320CDS notebook using a Pentium-233 and 64 Mb Ram running LinuxOS. The limited I/O capabilities of the single serial port are augmented with the use of a four serial port Quatech QSP100 communications card. Connected to the laptop are a Precision Navigation TCM2-80 digital compass for orientation information, a Garmin 12XL GPS receiver for positioning, and a DGPS receiver for improved accuracy. For the head mounted display (HMD), one uses alternately the i-glasses unit from I-O Display Systems, and the Sony Glasstron PLM- S700E. Various other devices are present as well, such as a small forearm keyboard for data entry, power converters for the different components, and necessary connection cabling and adaptors. The construction of the backpack was directed with ease of modifications in mind, at the sacrifice of wearability and miniaturisation. 7
Figure 5: Tinmith outdoor backpack computer (Taken from [2]) Figure 6: vision based tracking: fiducial markers 5.3 Movement and interaction Once the system works and is running the user can move through the level by walking and changes his view by looking around. Logically, the user sees the game and the physical world through the HMD. In the lower part of the screen he sees information about armor, health, ammo and his type of weapon. In general, actions which are not easily reflected in the physical world are removed from the game such as secret and locked doors. Therefore, the tracking of the user s position and orientation of the user s head handles the majority ot the interaction for the user. Another interaction for the user is to shoot or change the current weapon. This happens through a two-button hand-held device as a physical input device for these actions. The direction the weapon fires is the center of the current view of the HMD. 5.4 Tracking The system also has the goal to provide continous indoor and outdoor tracking. This is realized by combining GPS and a compass system with a vision based system (Figure 6). Therefore the tracking needs were categorized into three areas: 8
Figure 7: The campus level: rendered grid patterned mode and in black mode (invisible walls) Outdoors far from building - if the player is at a large distance (more than 50 m) from a building - uses GPS/compass system Problem: inaccuracy of GPS-tracking Outoors near building - if the player is at a short distance (less than 50 m) from a building - uses GPS/compass system (if very close then +vision based tracking) Problem: higher inaccuracy of GPS-tracking Indoors - if the player walks into a building - uses vision based tracking by fiducial markers and patterns Problems: marks have to be placed on many walls; at least one pattern must be visible Each of these need a different approach, while maintaining information in a common format of WGS 84/UTM positioning information and heading/pitch/roll angles for orientation information. Another thing is the usage of visual landmarks in order to improve registration: allow the system to correct the final image by aligning the landmark with a known position in the graphical image use landmarks to extract a relative position and orientation of the camera from the landmarks 5.5 The campus level The team created a Quake level representing a part of the Levels campus of the University of South Australia. For testing reasons the level was rendered in two modes: black (invisible walls) for game mode and a grid patterned for testing mode (Figure 7). Consequently, the Quake engine only renders monsters, items on the ground (health packs, weapons, etc.) and regions of interest. The campus level was derived from architecturial drawings of the campus provided by the university. Measures: East/West: 94 metres; North/South: 156 metres [=size of virtual quake level] 9
Figure 8: A monster: skin color and texture changed (Taken from [3]) 5.6 Monsters In the Quake world, there are sixteen different types of monster. The normal Quake game was designed to fight monsters with super powers, fast movement and unrealistic jumps. Therefore the team chosed seven types of monsters which are all land-based creatures using weapons from a distance and all seem well suited for the system. The monster s skin colour and texture were changed (Figure 8) to make them easier to perceive and distinguish from the physical world (due to contrast reasons). 5.7 Conclusion The implementation of ARQuake was never really completed. Nevertheless many interesting results were generated: user interface issues for AR outdoor/indoor games architecture for low cost moderately accurate indoor/outdoor 6DOF tracking combination of GPS- and optical tracking issues for converting a desktop application into an AR application 3d model of an outdoor section of an university campus 3d model of an indoor section of an university successful mapping of keyboard and mouse interaction of ARQuake application to head/body movements and simple two-button input device to ARQuake application shown another way of architectural visualisation 10
6 Discussion: AR games - A big market? This section illustrates the economic perspective of AR games from my point of view. 6.1 The vision Imagine the following situation: You and three friends are playing paint-ball. After three hours you get a bit bored and now you think it would be completely amazing, if it was possible to play a cooperative match together in team against a lot of monsters or zombies, robots or enemy soldiers. Another option would be just to go hunting in the woods, without really killing animals (or just think about a non-violent game concept, e.g. golf). These new ways of gaming requires Augmented Reality. Augmented Reality offers a completely new way of gaming because it combines the virtuality and the reality so therefore the player can do things he can t do in reality games like paint-ball, like shooting monsters, and he now can do things he can t do in virtuality, like perceiving the environment with his whole body: while shooting monsters he now can crouch, jump or climb. I think I don t have to make clear that its much more entertaining than using the mouse or keyboard. 6.2 The requirements In order to establish Augmented Reality gaming, the product Augmented Reality Game has to fulfill the following requirements: Hardware features: HMD s size should decreased; a HMD with size of normal glasses would be perfect (ubiquitous devices) tracking should be more accurate tracking should be possible on every spot on the planet: indoor and outdoor the HMD should be connected to a wearable mini computer which has the power to render photorealistic 3d graphics including current tracking data the mini computer should be able to connect to other mini computers via some kind of WLAN in order to request the other player s position to enable multiplayer games also access to the internet, to get information about a current game being played in nearer location there should be two easy-to-use hand-held devices with accurate tracking for completely new gaming experience (firing two guns at the same time in two different directions) all devices should be stable and mortgageable to environment influences like sun rays, rain, snow, heat, cold Software features: interesting game concepts, i.e. hunting, shooters, economy simulation, adventure games, sport games 11
easy calibration of trackers and 3d models there is an API required which combines a state-of-the-art 3d engine with AR technology good documentation for further developers good usability - installing, starting, playing games; no complicated HUDs or GUIs Economic features: low price: ca. 300 EURO per HMD + mini computer good support community building intensive marketing high investment for development, promotion and production; popular technology investors with connections and money 7 Conclusion This article presented a short introduction to AR in games as simulator, AR related technology like Nintendo WII and a game designed for AR named ARQuake. AR has many faces. AR gaming is still less developed, but has in my oppinion a big chance to become huge and sucessfull. The problems include hardware, tracking, software and of course the game itself. Creating testing environments in games is a good way to decrease costs while testing AR gaming related concepts. Nintendo proves the need of new gaming experiences by selling lots of WIIs and ARQuake demonstrates impressively a first sign of AR gaming coming true. In my oppinion, AR gaming will be successfull in 3-4 years when tracking, hardware and software are higher developed. 12
References [1] Ronald T. Azuma. A survey of augmented reality. In A Survey of Augmented Reality, August 1997. [2] Benjamin Close Bruce Thomas, Nicholas Krul and Wayne Piekarski. Arquake: An outdoor/indoor augmented reality first person application. In ARQuake: An Outdoor/Indoor Augmented Reality First Person Application, February 2003. [3] Benjamin Close Bruce Thomas, Nicholas Krul and Wayne Piekarski. Usability and playability issues for arquake. In Usability and Playability Issues for ARQuake, July 2003. [4] Prof. Gudrun Klinker Marcus Toennis, Verena Broy. A survey of challenges related to the design of 3d user interfaces for car. In A Survey of Challenges Related to the Design of 3D User Interfaces for Car, November 2006. [5] Michael Pauly. Wiimote technology exposed! how nintendo wii controller works. In Wiimote Technology Exposed! How Nintendo Wii Controller Works, February 2007. [6] Wayne Piekarski and Bruce Thomas. Arquake: Gaming system (2002). In ARQuake: Gaming System (2002), April 2002. [7] Dieter Schmalstieg. Augmented reality techniques in games (2005). In Augmented Reality Techniques in Games (2005), November 2005. [8] Unknown. Wiki: Accelerometer. In Wiki: Accelerometer, September 2000. [9] Unknown. Wiki: Computer game. In Wiki: Computer game, March 2007. 13