Augmented Board Games Peter Oost Group for Human Media Interaction Faculty of Electrical Engineering, Mathematics and Computer Science University of Twente Enschede, The Netherlands h.b.oost@student.utwente.nl ABSTRACT This paper presents the results of a design study in the usage of ARToolKit in the development of an Augmented Reality board game. ARToolKit is used to develop a board game with a tangible interface. This allows for visually attractive games, potentially equal to modern day computer games, while still allowing the game to be played identically to traditional board and tabletop games. This paper presents methods with which the task of creating an augmented board game becomes no more difficult than creating a normal computer board game without the requirement of expensive equipment. Keywords Augmented and mixed reality, tangible and physical interfaces, 3D interfaces, intelligent interaction, virtual gaming, ARToolKit. 1. INTRODUCTION Mixed environments are real world environments on which interactive virtual objects are overlaid. Mixed reality research aims to integrate these virtual objects seamlessly into the real world to allow for an augmented environment. Most games played in an augmented reality environment require expensive and often cumbersome equipment [Tho01]. These games require not only the minimal requirement of a headmounted display (HMD), but also motion trackers and other specialized input devices and a portable computer to allow the user to move through the real, and therefore, the virtual environment. Classical board games however limit the user in several ways. The game environment is limited to the game board and the input is limited to interacting with board pieces on the game board. The size of the game board is limited and therefore the user does not have to move through a large game environment and he does not need a portable computer to play a board game in an augmented environment. However, current research projects on board games still use rare and cumbersome equipment for input [Bal00, SEG98, Sta00]. A solution to this may be found in the research area of tangible interfaces. Elements of these interfaces exist in the real world Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission. 2nd Twente Student Conference on IT, Enschede 21 January, 2005 Copyright 2005, University of Twente, Faculty of Electrical Engineering, Mathematics and Computer Science and can be easily manipulated by the user. The interfaces presented in [KB99, DNH03, Pou01] use cards within the real environment on which they project parts of the virtual world. Because these cards exist within the real world they can be manipulated without the use of specialized input devices. These systems require only a camera and a HMD, both of which can be reasonably comfortably worn. This paper presents a design study on tangible interfaces based on [KB99, DNH03, Pou01], for use in augmented board games. This paper presents a design with which the development of an AR board game will be no more difficult than the development of a standard computer board game. Augmented board games based on this design will be as easy to use as a classical board game while offering the graphical attraction of modern computer games which can not otherwise be added to a classical board game. The methods described here allow the game programmer to continuously know the three dimensional position of each object within the game. This allows the game logic to be applied as usual and thereby eliminates a major dependence on expensive and cumbersome equipment which is generally used to determine these positions. This allows for many developers to experiment with AR with relatively low cost. 2. AUGMENTED GAMING To better explain the concept of Augmented Reality or AR this part of the paper will contain a motivation to continue research in this area and a short overview of current gaming attempts in AR. A players game experience can be divided into four parts, physical, mental, social and emotional [NLL04]. Neither real world board games or computer board games are likely to be physically demanding. But in order to give an overview of current game attempts a few examples of physical AR games will be presented. Many real world games are physically challenging; paint ball, basketball, soccer and most other sports. The players are free to use their body and perhaps physical artifacts in the game environment. Some computer games do rely on the players physical skill. First Person Shooter games for example which rely on the players hand-eye coordination. The range of physical interaction is however limited by the availability of input devices. AR presents the user with more possibilities in this area. It may extend the users input capabilities as in [LBK04, Ish99, Sta00] which present a more physical means of interaction in an AR environment. Also the research area of Exertion Interfaces is of interest. Other AR game projects focus on expanding the game environment. In ARQuake [Clo00] gamers get in full gear to shoot monsters through out a city. In Pirates! [Bjo01] players use a hand held to receive game information while walking around.
Mental challenges in games come in various forms. Some games offer the player challenges in the form of puzzles, other games rely on complex rule systems. Both classic games and computer games can offer many of the same puzzles. However a computer can support complex rule based games by calulating the numbers involved in complex rule systems automatically. Another benefit for computer games is the possibility to add artificial opponents to the game. AR games share these advantages with normal computer games. Classic games can provide a strong social aspect due to natural interaction methods. Computer games have mixed benefits. The social interaction is more limited because the player interaction has to pass through the computer input devices but due to the availability of network systems its communications have a longer range. AR board games can potentially utilize both the natural methods of interaction [SEG98] and the network capabilities of computer games [BFS04]. The emotional aspect of games is complex to define. Important are to take an interest in the story and the game world. The capabilities for this are independent of the medium. Audio and visual effects also stimulate the emotional aspect. These effects are common in computer games. Real world games rely often on more physical effects, like a theme park ride. In an augmented environment it is possible for the player to experience all of these effects. A more thorough overview on this subject is presented in [SHN02]. 3. ARTOOLKIT ARToolKit [Kat04] is a software package which calculates the position and orientation of physical markers relative to the camera. It can be used with one, or multiple cameras and performs all calculations in real time. ARToolKit is designed to output to a Head Mounted Display (HMD) but can also output to a normal computer screen. Figure 1 gives an example of ARToolKit at work on a single marker. It is primarily been developed by Hirokazu Kato and other researchers of the Human Interface Technology Lab of the University of Washington. ARToolKit is distributed free for non-commercial or research applications. ARToolKit website [Kat04]. The workings of ARToolKit are best described in [KB99]. In short, ARToolKit detects the location and transformation of markers created by the user. ARToolKit detects the marker by thresholding the camera input image. Regions whose outline contour can be fitted by four line segments are extracted. This region is normalized and a sub image within this region is compared by template matching with specific patterns selected by the user. This will result in the identification of the specific marker. The final result of these operations is the identity of the located marker and a homogeneous transformation matrix representing the rotation and translation of the marker from the perspective of the camera. ARToolKit provides the functions to process the video input stream and a few basic functions to work with the transformation matrix it returns. In addition the latest version of the toolkit also contains a VRML renderer for the rendering of VRML objects. ARToolKit is platform independent and uses the OpenGL API for its 3D rendering. ARToolKit is currently in widespread use in various research applications. Among these are a virtual desktop [DNH03], an indoor tracking system [Pie03], an authoring interface [Pou01] and games [NLL04] and many others. ARToolKit is even making its way to hand held devices such as PDAs and mobile cellphones [WS03]. 4. DESIGN The motivation behind the design study presented here is to make augmented board game development similar to normal computer game development. This means the game creator should have access to normal two or three dimensional coordinates for each object within the game environment. This is accomplished with the use of an internal representation of all markers involved in the game. This internal representation presents us with various extra benefits. Among these is an option for an additional type of interaction. 4.1 Internal Representation The internal representation is constructed by calculating the relative positions of all markers detected by ARToolKit to a special fiducial marker tile, or preferably a series of fiducials. This creates a special internal coordinate system with the fiducial as origin. The positions of the other markers in the internal representation are the relative positions of these markers towards the fiducial marker in the real world. ARToolKit gives the user a transformation matrix for each marker it identifies. To calculate the difference between marker α, with transformation matrix A, and marker β with transformation matrix B one only needs to apply the following formula: The resulting matrix C is the transformation matrix to 'go' from α to β. This formula is applied to calculate the difference between the fiducial marker and all other markers visible by ARToolKit. Because we have set the fiducial marker as the origin of our internal coordinate system all marker positions are now transformed to the internal representation. Figure 1. A wireframe model of a teapot on my desk Documentation for ARToolKit is still in development but the toolkit comes with a collection of examples and there is a mailing list for additional assistance. The most recent version of the toolkit and news related to ARToolKit is available on the 4.2 Three Cases The method described above only works when the fiducial marker is seen by ARToolKit. To increase the odds of a fiducial marker being visible we spread a number of fiducials evenly over the game board. The exact distance between these markers is known because we place them ourselves and they do not
move. ARToolKit is able to use the knowledge of these marker positions to calculate a single transform for the entire set of fiducials. Instead of the transformation matrix of a single fiducial we use this combined matrix to calculate the relative positions of all other markers. Despite this precaution it is still possible for no fiducials to be visible. This gives a special case in which additional steps are necessary. There are four different situations in total: 1. One or more fiducials are visible. 2. No fiducials are visible but there are one or more markers visible which also have an internal representation. 3. No fiducials are visible and the only markers visible do not correspond to any markers currently in the internal representation. 4. No fiducials or markers are visible. In the first situation we do as described above. Calculate all positions relative to the fiducial and update the internal representation. For the second situation it is necessary to make an extra step. We estimate the position of the fiducial by backtracking from the internal representation of one of the visible markers. Say ARToolKit has detected marker α and has calculated the transformation matrix A. Internally the position of α is given by matrix A int. The position of the fiducial can be calculated as follows: The resulting matrix C is the transformation matrix for the fiducial. Because of a large chance for error in the resulting matrix C we do not use this matrix to update the internal representation. We can however use this to draw the game board and the other pieces, more about this later. If the precision is improved or it is determined that the position of a tile as determined by ARToolKit is precise enough it may be recommendable to update the internal representation anyway. In the third situation we basically do nothing besides drawing the markers in the standard manner of ARToolKit. It may be possible to build another (temporary) internal representation with one of the markers visible as a fiducial. However tests with the prototype demonstrate that occasions like these occur only rarely and even then mostly before ARToolKit had a first glimpse at the fiducial. The fourth situation obviously does not require us to do anything at all. 4.3 Benefits In the first two cases we are presented with an option: do we draw the objects directly with the transformation matrix given to us by ARToolKit or do we draw the objects at the position in the internal representation and transform this position with the transformation matrix of the fiducial marker. The second option holds a few advantages over the first: The objects are always drawn on their correct spot on the game board. Objects represented by markers that are not detected by ARToolKit but should still be visible can be drawn correctly. The first advantage is especially important in board games, it is often important for a player to see correctly where his pieces are on the game board. This also makes it possible to adjust the position of the drawn object. For example when a soldier tile stands on a mountain the height of the soldier is easily altered in the internal representation. The second advantage is important for the coherence of the different frames. It is often difficult for ARToolKit to detect each marker every frame and with this technique it no longer needs too. With the same technique it is also possible to draw a small bit of the object even though the marker is not within the camera image at all. This can be useful to draw the tip of the cannon of a tank which sticks out while the rest of the tank remains out of view. This effect is visible in figure 2. Figure 2. Closeup view of the prototype 4.4 Additional Interaction Besides the internal representation of the objects there is another ability which will be useful in many games. Besides the movement of the pieces and determining their location on the game board, some pieces may be able to interact with each other. For example a soldier may be selected to fire at an enemy monster. To facilitate this it is possible to check if a specially assigned marker touches, or is very close to, another marker. When a player would try to touch a marker with the special marker it is very likely for one of the markers to become obstructed. When this happens ARToolKit is no longer able to determine the position of this marker. Thanks to the internal representation this does not need to be a problem. We can check for the distance between the special marker and the position of the touched marker in the internal representation. This makes it fairly simple to design special interactions between different pieces which would be very clumsy with the normal use of ARToolKit. 5. IMPLEMENTATION The design and prototype has been developed on a 1.8Ghz Pentium IV computer with a single Philips PCVC730K web cam. It runs at up to 15 frames per second, which is the maximum supported frame rate of the web cam. To achieve this, the application requires the full processing power of the CPU. The application was developed and tested under Linux with a 2.6.8 kernel and ARToolKit v2.6.8. with a patch by Uwe Woessner for improved camera support. The game board consists of wooden board of 45 by 48 centimeters in size. The board is marked by nine evenly spread paper markers of 6x6 centimeters. There are four 8x8 markers which serve as game pieces and a single selection marker of the same size. The main problem encountered with the implementation was caused by the image format of my web cam not being recognized by ARToolKit. The Philips PCVC730K and many
other web cams which use the pwc driver 1 returns an image in a YUV format (YUV420P to be exact). A patch by Uwe Woessner converts the images of the web cam to the RGB image format which ARToolKit uses. The latest version of ARToolKit (version 2.70) includes this patch. The image returned by the Philips PCVC730K is also inverted horizontally. This was fixed by adding a simple function to the ARToolKit which flips the image. The coding style for the prototype is mostly the same as the coding style for the examples which come with ARToolKit. This is done in the hope of someday adding this code as an extra ARToolKit example to ease the experimentation with AR game design. The result of this is a program in plain C and no use of any library besides the standard C libraries, OpenGL and Glut. ARToolKit itself does not provide a method to easily organize data. The ARToolKit examples provide an object structure to organize some of the marker data. For the purpose of this prototype I added a similar structure to organize the internal representation of all the pieces in a similar manner. The game specific elements can be completely separate from the basic design if the game mechanics are rather simple. When pieces need to interact it is likely to be necessary to alter the contents of either the structures for the real world markers or for the internal representation. 6. PROTOTYPE With the use of the above algorithms I developed a prototype application. As as a game the prototype is extremely underdeveloped. However, it is a nice demonstration of the possibilities of the algorithms presented above and it makes good use of ARToolKit. There are four game pieces on a square game board. The player controls all pieces. The pieces are represented by four different marker tiles. The game board consists of nine markers placed evenly on a wooden board. Figure 3 gives an overview of the entire game board and all pieces. Each piece may fire at each other piece in turn. Once a piece is hit, it 'dies' and is no longer displayed. The player may use a specially marked tile to first select a piece and second a piece to be fired upon by the first piece. The player may also move any of the pieces around the board, this however gives no result besides the obvious movement of the piece. Figure 3. Overview of the prototype showing the game board and all four pieces 1 http://www.saillard.org/linux/pwc/ This obviously very simple application makes use of all possibilities presented thus far in this paper; consistency of the game board and pieces and interactivity with the use of a special marker tile. This prototype also gives a robust and natural impression of the interface. This prototype shows the algorithms to actually work in a real time augmented board game. A full game is not a simple thing to create and is far more complex than this prototype but this application shows the development of an augmented board game, with the use of the algorithms presented in this paper, does not need to be any more difficult than the creation of a standard computer board game. With the use of ARToolKit and this design the benefits AR can be grasped by all developers. 7. FUTURE WORK The prototype makes use of all the functions of the design. It is however far from a complete game. The completion of such a game would truly show the capabilities of ARToolKit and of Augmented Reality in general. The development of a complete game in C is more time consuming than development in an object-oriented language such as C++. It would therefore be beneficial to create a C++ version of the prototype. There are a few additions I would like to look into further and possibly add them to the prototype. First most of these is making use of the knowledge that all tiles, when they are not being moved, lay on the game board. By projecting the transformation matrix for the marker as detected by ARToolKit on the board plane the error in the detection of the transformation matrix may be reduced. This will likely give an increased precision in determining an objects position and rotation. Another improvement in precision in the second case, as mentioned in the Design chapter, may come from using the marker with the highest accuracy to estimate the position of the fiducial. Currently there are no checks on which marker to use and it can be improved with relatively simple metrics such as: the size of the marker or the distance to the camera. The processor usage of ARToolKit increases rapidly with the number of markers to match. Each possibly detected marker has to be checked against all known templates in order for ARToolKit to know which marker it has actually detected. When ARToolKit is used for a complete board game it may have to check for dozens, or even hundreds of markers. Even this limited prototype which uses only 14 different markers needs the full capabilities of my processor to run at 15fps. Daniel Wagner has developed an ID Based Marker Extension [Wag04] for ARToolKit to circumvent this. His extension uses binary encoded markers instead of the templates normally used by ARToolKit. This extension may drastically improve performance when many markers are used and it supports up to 512 different markers simultaneously. Since the start of this project there have been two new releases of ARToolKit. Before any other steps are taken the prototype will be ported to the latest version of ARToolKit. 8. CONCLUSION The resulting prototype is very limited in its capabilities. However it does make use of the tangible interface offered by ARToolKit and the usability of this interface has been extended by the here presented design. Overall the interface for the prototype feels very natural and robust. With the development of the prototype the design presented here has shown its applicability in the development of Augmented Reality board games. With the use of ARToolKit
and this design the task of developing an AR board game has been reduced to the task of creating a computer board game. The programmer continuously knows the position of each object in the game and all game logic can therefore be applied as normal. The design does this without the use of expensive and cumbersome equipment which are not available for many developers interested in AR. That is not to say these devices are without their use. They are able to operate in far less limited environments, such as entire urban environments, where as my design only operates in the vicinity of the game board. ARToolKit makes it possible to relatively easily create AR applications which require very few devices. Only a single camera connected to a computer is required to create an augmented image. To view this image a head-mounted display is not required. A standard computer screen can be used to view the augmented image. Though this will not give the full AR experience this is already enough to experience the many possibilities of Augmented Reality. This ability alone gives many developers, myself included, an opportunity to experiment with AR. REFERENCES [Bal00] S. Balcisoy et. al. Augmented Reality for Real and Virtual umans. In Proceedings of the International Conference on Computer Graphics, 2000, 303. [BFS04] I. Barakonyi, T. Fahmy and D. Schmalstieg. Remote Collaboration using Augmented Reality Videoconferencing. In Proceedings of the 2004 conference on Graphic Interfaces. London, Ontario, Canada, 2004, 89-96. [Bjo01] S. Bjork et. al. Pirates! Using the Physical World as a Game Board. In Proceedings of Interact, 2001. Tokyo, Japan. [Clo00] B. Close et. al. ARQuake: An Outdoor/Indoor augmented reality first person application. In 4 th Int'l Symposium on Wearable Computers. Atlanta, GA, USA, 2000, 139-146. [DNH03] S. DiVerdi, D. Nurmi, T. Höllerer. ARWin A Desktop Augmented Reality Window Manager. In Proceedings of the Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003, 298-299. [Ish99] H. Ishii et. al. PingPongPlus: Design of an Athletic- Tangible Interface for Computer-Supported Cooperative Play. In Proceedings of CHI, 1999, 394-401. [Kat04] H. Kato et. al. Augmented Reality Toolkit. http://artoolkit.sourceforge.net, Nov. 8, 2004. [KB99] H. Kato and M. Billinghurst. Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferencing System. In Proceedings of the 2nd International Workshop on Augmented Reality, San Francisco, USA, October, 1999, 85-94 [LBK04] G.A. Lee, M. Billinghurst and G.J. Kim. Occlusion based Interaction Methods for Tangible Augmented Reality Environments. In Proceedings of the ACM Siggraph International Conference on Virtual Reality Continuum and its Applications in Industry, 15-18th June, Singapore, 2004, ACM Press, New York, 419-426. [NLL04] T. Nilsen, S. Linton and J. Looser. Motivations for AR Gaming. In Proceedings of the First International Workshop on Foundations of Unanticipated Software Evolution, New Zealand Game Developers Conference, Dunedin, New Zealand, 26-29 June, 2004, 86-93. [Pie03] W. Piekarski et. al. Hybrid Indoor and Outdoor Tracking for Mobile 3D Mixed Reality. In 2nd Int'l Symposium on Mixed and Augmented Reality, Tokyo, Japan, Oct 2003. [Pou01] I. Poupyrev et. al. Tiles: A Mixed Reality Authoring Interface. INTERACT 2001 Conference on Human Computer Interaction, 2001. [SEG98] Z. Szalavâri, E. Eckstein and M. Gervautz. Collaborative Gaming in Augmented Reality. In Proceedings of the ACM symposium on Virtual reality software and technology, 1998, 195-204. [SHN02] C.B. Stapleton, C.E. Hughes and M. Moshell. Mixed Reality and the Interactive Imagination. Swedish American Simulation Conference,2002. [Sta00] T. Starner et. al. MIND-WARPING: Towards Creating a Compelling Collaborative Augmented Reality Game. In Proceedings of the 5 th international conference on Intelligent user interfaces, 2000, 256-259 [Tho01] B.H. Thomas. Challenges of Making Outdoor Augmented Reality Games Playable. The University of South Australia, 2001. [Wag04] D. Wagner. ID Based Marker Extension. http://www.ims.tuwien.ac.at/~daniel/download/idbased Markers.zip, Nov. 8, 2004. [WS03] D. Wagner and D. Schmalstieg. First steps towards Handheld Augmented Reality. In Proceedings of the 7th International Conference on Wearable Computers, White Plains, NY, USA, Oct. 21-23, 2003.