Integration of jartoolkit and enjine: extending with AR the potential use of a didactic game engine

Transcription

1 Integration of jartoolkit and enjine: extending with AR the potential use of a didactic game engine Fernando Tsuda Paula M. Hokama Thiago M. Rodrigues João L. Bernardes Jr. Escola Politécnica da Universidade de São Paulo São Paulo, SP (11) {fernando.tsuda, paula.hokama, thiagomoreira13}@gmail.com joao.bernardes@poli.usp.br Figure 1. The "We A.R. Dancin' " Game ABSTRACT This paper describes the integration of a didactic game engine (enjine) and an Augmented Reality API (jartoolkit) to increase the potential uses of the engine both as a teaching tool and as a testbed for new technologies, in this case, low cost AR games. The tools, methodology and the implemented solution are analyzed and several tests are described, showing results regarding the feasibility and playability of such a system. Keywords Augmented Reality, Electronic Games, Game Engines, jartoolkit. 1. INTRODUCTION Augmented Reality (AR) is a technology that allows the combination of real and virtual elements in a scene, with which the user can interact in real time with the use of 3D registration [4]. To make this interaction possible, the use of high cost equipment is common, such as HMDs (Head Mounted Displays) to show the scene with virtual objects and position sensors such as GPS (Global Positioning System) or magnetic sensors that indicate the user s localization in the environment. However, it is possible to use AR with low cost devices such as conventional video monitors, webcams, PDAs or cell phones. In this case, the user is often detected in the environment with the use of computer vision techniques. This technology can be used in several applications such as medicine, buildings and equipment maintenance or the entertainment area in general. This work s goal fits in this last category, specifically the electronic games area using AR, whose development can be facilitated integrating a game engine with an AR library. In this work, the term Augmented Reality is used to designate any system that satisfies Azuma s definition [4], without the distinctions presented by Milgram [11]. The electronic games area has considerable social, economic and technological importance. In the social area, they allow the interaction of different people even in distant locations, a fact strengthened with the growth of multiplayer online games where thousands of players interact in a common virtual environment. Augmented Reality allows a richer interaction between these people, either introducing the real image of the player in the virtual environment or augmenting the real world with virtual elements of the game. In the economic area, the world-wide electronic games segment has, in the past years, generated yearly revenues of approximately US$ 26 billion, exceeding the profit from ticket sales by the Hollywood movie industry [8]. And in the technological area, games motivate the creation of new hardware, software and architectures with the capability to deal with the increase of its complexity that occur due to de use, for instance, of higher definition graphics and sound, which strengthen the realistic aspect of current games. As examples of technological advances in hardware greatly motivated by games, the increase of GPU (Graphics Processing Unit) capabilities and the creation of a unit responsible by processing the physical calculations inside a game (called PPU or Physics Processing Unit) can be mentioned. Games are also one of the most interesting areas to explore Augmented Reality. They not only do not usually demand the high precision required in medical or maintenance areas, for instance, they also benefit from AR through the development of games with original playability and with greater player immersion. Combining games and AR not only allows better (more natural, for instance) interfaces with the player but also stimulate the use of AR technology, so it can be more easily popularized and used in other areas, including applications that require more precision. The ludic characteristic of games

2 facilitates the acceptance and tests with innovative user interfaces, which may even be still in prototype stages, much like many AR systems. [17] Electronic games also have a great potential of application in the didactic area either in form of educational games or with the development of games as a pedagogical strategy. This later paradigm has been successfully applied at Escola Politécnica da USP to teach Computer Graphics courses [20]. For this end, a didactic game engine, called enjine [12], was developed, based on Java and the Java 3D graphics API. Currently, enjine has as secondary objective, aside from its didactic use, which is to serve as a test platform for new technology in games. Its simple and structured architecture aids in both objectives. However, like the most existing game engines, enjine does not offer Augmented Reality resources. An alternative to develop AR games with enjine is to use the jartoolkit library [7]. Integrating these two tools, however, is not a simple task, since the architecture of one ignores the other, generating conflicts of access to graphic resources and difficulties to share data between them. Thus the work presented here has as its objective the integration of jartoolkit and enjine, to simplify the development of games that use the concept of Augmented Reality. This project brings two major contributions, each related to one of enjine's main objectives. The first is to increase the potential of didactic applications of enjine, for instance to teach concepts of Augmented Reality and Image Processing, subjects that can be part of an undergraduate Computer Graphics course or even constitute postgraduate courses by themselves, as is the case in Escola Politécnica. The other is to supply a framework that facilitates the exploration of new alternatives to create games based on AR with low cost, increasing the enjine s potential as a test platform for new technologies in games (in this case, AR). To test the integration of these tools and at the same time to build a proof of concept of the viability of low cost AR games, some prototypes have been developed, among which the dance game We AR Dancin (some images of this game are shown in Figure 1) was the most elaborate and deserves a more detailed analysis in the next sections. In the following sections some similar works and the tools used in this project will be detailed. After that the specification and methodology used on the integration of enjine with jartoolkit will be discussed. Later the results obtained will be shown and discussed and the We AR Dancin game will be more detailed. 2. RELATED WORK One of the aspects that makes the entry of new consumers in the games market difficult is the growing complexity to handle their traditional input devices [16]. Games have an increasingly larger quantity of commands to input through a keyboard or a joystick with a great number of buttons currently they have 8 action buttons and 3 directional buttons whose functions the player must learn and memorize for each game [9]. Thus, new ways of input have been used as an attempt to create more instinctive playability. Some examples from the game industry can be mentioned: the use of controllers that look like musical instruments such as electric guitars or bongo drums; the use of carpets where the user performs a "dance", controlling the game with his feet; cameras such as the EyeToy, an accessory to Sony's Playstation videogame system where the user plays making moves in front of the camera; and the WiiMote, the Nintendo Wii controller, which is a recent attempt from Nintendo to attract new players through the use of a innovative way to play videogames, where the controller s movement is detected and converted into a correspondent game action. It must also be mentioned that the EyeToy can be used in AR games with low cost, in a similar way to the work presented in this text since, aside from being capable to detect movements, it can also capture the real image of the user and insert it in the virtual environment of the game using the camera, the TV and the console. Figure 2 shows the EyeToy and the WiiMote. Figure 2. The EyeToy, a camera for Sony's Playstation console and the WiiMote, Nintendo Wii s controller Most academic works resembling what is proposed here for Augmented Reality games have a peculiarity: they have an architecture based around the requisite of AR first and the characteristics of games come in second place, if at all. In this project, the starting point was the game engine, the enjine, and AR is treated as another form of input (exploring enjine's abstract input layer) and output (integrating the video stream from the camera with the virtual elements) to the game. An example of architecture that also uses the ARToolKit and has been used to develop Augmented Reality games, but is actually a platform for general AR applications, is the StudierStube [18]. Although initially this platform was limited to specific forms of data input and output like HMDs and a Personal Interaction Panel [15], it now supports a greater variety of interaction devices and the use of computer vision. It has even been adapted to be used in mobile devices [22]. Another framework to develop games using AR which supports the use of ARToolKit (used in the first application created with this library) and is even oriented to the development of educational games is described by Santiago et al. [14]. This framework is limited to the development of adventure games with AR and uses the concept of hypermedia graphs to link game events and entities and represent its evolution. Focusing in AR games instead of the tools for their development, there are several works in the related literature. Bernardes et al. [5] lists and tries to classify some of them and also discusses tangible interfaces (where a specific physical object may be associated with each virtual object and, handling this physical counterpart directly, the user may handle the virtual object in an instinctive manner). Most AR games developed with ARToolKit use this kind of interface, with the markers acting as the tangible real elements. Among the games that use a conventional video monitor as display two can be mentioned: a game that aids in learning Kanji [21] and CamBall [24], a Ping-Pong game where

3 the marker is located at the paddles. Another game where the markers are used as a tangible interface augmented with the use of buttons is a Chinese Checkers game [6], but this game cannot be considered as an AR game, since it consists of only virtual elements. Other games use the markers as a non-tangible interfaces, to facilitate the identification of features of interest. An example is the Invisible Train [22], which uses the StudierStube in a mobile device. Another example is the GeFighters [19] (which, for the same reason as the Chinese Checkers game, cannot be considered an AR application, but that also uses fiducial markers and computer vision as means of player interaction), a fighting game where the movements of the markers attached on the player s hands controls its avatar (while, curiously, the foot movements controls the fighting movements such as punches). GeFighters uses a platform to manage the input devices called CIDA, which translates the actions from different devices as a game actions, similarly to what happens in enjine's abstract input layer. The integration of enjine with jartoolkit allows the use of the markers as either tangible interfaces or features of interest; however none of the prototypes developed for the testing in this work use markers as tangible interfaces yet. 3. SOFTWARE TOOLS 3.1 enjine The enjine [10][12] is a game engine developed at the Interactive Technologies Laboratory (Interlab-USP) with the use of the Java language and Java 3D. Its main goal is to serve as a didactic tool to aid in teaching subjects such as Computer Graphics and Software Engineering through the quick implementation of games. Thus, despite its simple structure (because the students have little time to learn it), enjine allows the creation of relatively complex games in short periods of time. A second purpose for enjine is to act as a test platform to new technologies in games, since its simple architecture has proven suitable for this function. This work contributes with both purposes, as will be discussed later. Conceptually, the enjine can be divided into three layers, as shown in Figure 3. Classes from the Java API and other libraries such as Java 3D are located in the lowest tier. The middle layer contains several "core" enjine modules, responsible for services such graphical rendering of objects, command input, sound output and others functions related to game management. The topmost layer contains a framework that makes the implementation of enjine s services more transparent to the programmer for specific types of games. The class SinglePlayerGame, for instance, facilitates the creation of games that will run in a common desktop computer. Figure 3. enjine layers EnJine's different modules are implemented as packages, as shown in Figure 4. The Core package contains the Game, GameState and GameObject classes which represents game entities with different characteristics and can aggregate the classes Updater, to update the object, Collidable, to detect collisions between objects, Viewable, responsible for the object's visual representation and Messenger to allow the exchange of messages between objects. Figure 4. enjine packages The IO package contains the Graphics, Input and Sound packages. The Graphics package is responsible for the rendering of game objects, using the Java 3D API. The Sound package is responsible for the basic functionality for sound in the game. The Input package handles user input in enjine and is based on an abstract layer that separates input devices from the correspondent game actions. Each device (such as a joystick, mouse or keyboard) is represented by a subclass of InputDevice which has a series of InputSensor objects representing device elements such as a joystick s buttons or a keyboard s keys. To link these sensors to a game action ( InputAction ) such as jump or move right, for instance, there is a class called InputManager. Figure 5 illustrates the relation between the classes that belong to Input module of enjine.

4 for the allocation and initialization of the main instances used, such as JARToolKit and JARFrameGrabber. Other important classes are ARPatternTransforfGroup, which extends Java 3D's TransformGroup class and is responsible for marker information and represents the transform group that will be influenced by the marker movement; and the ARBehavior class that extends the Behavior from Java 3D and is responsible for updating the images obtained from the camera, for identifying the marker patterns and for rendering virtual objects on them. Figure 5. enjine input system 3.2 ARToolKit and jartoolkit The ARToolKit [2] is a library that provides functions to facilitate the development of Augmented Reality applications. Its works based on computer vision techniques to calculate the real position of the camera in relation to fiducial markers and their orientation, allowing the programmers to show virtual objects on them. Figure 6 shows how an usual application using ARToolKit works. Figure 6. Operation of an application developed with ARToolKit However, the use of ARToolKit is restricted to users of C and C++ languages. To allow the use of ARToolKit s functions with the Java language, jartoolkit was developed. It consists of classes that access ARToolkit's functions using the JNI (Java Native Interface). jartoolkit's architecture divides the ARToolKit's functions in two main classes: jartoolkit the class that encapsulates the functions of ARToolKit responsible for tracking. For the time being, only the basic functions are implemented, focusing on the Windows version of the library. JARFrameGrabber Class that encapsulates all the functions necessary to access the video input from the camera, using the DirectShow API. Another important information is that the integration between Augmented Reality functionality and Java 3D is made through a component called "jartoolkit for Java 3D", with several classes that facilitate the inclusion of AR-related scene graph nodes in Java 3D. The main class, called JARToolKit3D, is responsible 4. METHODOLOGY To develop this project, the UML (Unified Modeling Language) was used to model its components, making it possible to visualize the work s products and services with the use of standardized diagrams. The integration with jartoolkit gives enjine the capacity to recognize commands inserted with the use of fiducial markers and a video camera, aside from reproducing the video stream obtained from the camera, characteristics that allow the creation of games that use the Augmented Reality technology. To specify this part of the project, it doesn t make sense to use regular use cases, because its functionality is not presented directly to actors but instead to the games that use the integrated platform. Thus, an abstract actor was created to represents the game that will be developed using the enjine. Initially, it was necessary to study Augmented Reality libraries. Besides the already mentioned ARToolKit and jartoolkit, other available Augmented Reality libraries were studied, such as the ARToolKit Plus [3] and ARTag [1] as well Intel's computer vision library, OpenCV [13]. The choice of jartoolkit was based mainly on its implementation in the Java language, which facilitates the integration with enjine. The study of demo applications supplied with jartoolkit (such as the one shown in Figure 7) and their respective source code showed that the position and orientation of the markers are stored in a 4x4 transformation matrix with the three first lines and columns containing the information about the rotation (and about the scale too, but in this case it is not used) and the last column containing the marker's translation. This matrix is normally used to apply the same transformation to some virtual object, giving the user the sensation that the object is on the marker. Figure 7. Demo application of jartoolkit

5 With the demo applications, some tests were carried out to validate the use of jartoolkit in games with the measurement of values such as the maximum distance between the markers and the camera and the maximum speed of movement of the marker that would still allow its position to be registered by the system. To aid in this task, the apparatus shown in Figure 8 was built. An electric motor pulls the marker by a string with controlled speed in front of the camera and the camera height can be adjusted. Despite some problems, described ahead, jartoolkit proved adequate to the project. branch of scene graph that recognizes the marker on the image and to transfer it to the ARInput device in intervals defined by the game s programmer; ARDisplay : Extending enjine's class GameDisplay, it was created to allow the reproduction of the video stream from the camera as a background and to add the scene graph that represents the AR devices, with the use of ARInstance methods. Figure 8. Apparatus mounted to validate jartoolkit Regarding the enjine, the study was focused in the classes located in the Input and Graphics packages. 5. ARCHITECTURE The studies described earlier revealed the need to add to enjine's scene graph jartoolkit s branch graph responsible for detecting the marker on the scene, to allow the acquisition of the marker transformation matrix. But just adding that branch to enjine s scene graph is not sufficient for the values within that matrix to be recognized as a command inputs. Another need identified was the reproduction of the video stream obtained from the camera in the game. Those problems were solved with the creation of a middleware component, containing the following classes and functionalities: ARInstance : Class that initializes and accesses the main functions of the JARToolKit3D class. Among its main functions are: the registration of the markers that will be used in the game and the creation of the scene graph responsible by the capture and reading of the images obtained from the camera. Only one instance of this class is allowed to exist (it is a Singleton); ARInput : Class that represents an input device from the images obtained from camera. It inherits from InputDevice. In this implementation, each marker is considered as a distinct device and must have an ARPatternTransformGroup that will be added to the scene graph of the game allowing its identification on the captured image. This class is also responsible for converting the values from the transformation matrix into the six commands that will be used in the game (translation in the x, y and z axes and rotation around those axes, representing, in this way, the six degrees of freedom of an object), as shown in Figure 9; ARTimerTask : Extending the Java s class TimerTask, is responsible for getting the transformation matrix from the Figure 9. Sequence of value transfer from the transformation matrix to the AR device To facilitate the creation of Augmented Reality games, a class called ARGame was created inside the Framework package, inheriting from Game. It was developed with to include the attributes and methods necessary to instantiate the interface between enjine and jartoolkit correctly and to make this task transparent to the programmer. Figure 10 shows a class diagram representing the classes of the middleware developed in this work and its relation with jartoolkit and enjine. Figure 10. Middleware classes and their relations to the enjine and jartoolkit

6 During the initial tests (which will be described later) to validate the integrated tools, the need was perceived to develop routines for the calibration of the camera, to identify the position of the marker on the scene in a correctly. Until then, the virtual objects generated on the markers was incorrectly registered, causing the following effects: 1) In the perspective image obtained from the camera, the values of coordinates on the x and y axis vary according to the distance between the marker and the camera. For instance, if at a distance of 1 meter the value of x in the position matrix varies from -100 to +100 for each extremity of the screen, for 3 meters these values will proportionally proportionally to and +300; 2) For a given distance, the camera is not correctly calibrated for all points in the xy plane. For points near the center of the screen the object stays on the marker but if the marker moves away from the center, the object will move with a different speed. To solve these two problems, the second one was focused on first. Analyzing the virtual object displacement in relation to the marker in the tests, it was noticed that this variation was linear in each of the x and y axes. With the tests, the values of the constants (the angular coefficient, k1, and the displacement of the line, k2) in the equation of the line (equation 1) that represents this variation could be determined empirically. y k x + = (1) 1 k 2 This function, however, can only realize the calibration on the xy plane located at the distance used on the tests (z=1600 mm, adopted as a reference value). Thus the solution that permits to adjust this calibration function to any distance is to project (during the calibration of the z axis) all points obtained from the camera to the reference distance before the calibration of x and y axes. Figure 11 shows the necessary sequence to obtain the calibration of each point. Figure 11. Complete process of calibration This method of calibration was satisfactory to adjust virtual objects located in places with fixed z values. However, the tests with the variation of this value (detailed ahead) showed that the process is not satisfactory: the object does not follow the marker correctly in this case. Thus, to solve this new problem, it was created a new class extending the class Camera from enjine and the transformation values used by the camera in jartoolkit was adopted. This solution supplies a result with least failures and deviation and reveals itself satisfactory. The cost of its adoption, however, was the limitation of allowing the creation of a single instance of class Camera in the game because of the use of fixed values applied in the camera calibration. This limits the enjine's functionality of creating several instances of the camera and positioning them in different points. 6. RESULTS AND TESTS The results obtained with the integration were verified with several tests, which are described below. 6.1 jartoolkit Tests The tests with jartoolkit were accomplished with the aid of the equipment shown in Figure 8. Initially, the results of these tests were not satisfactory, because the marker was being recognized only when moving in very low speeds (around 0.1 m/s). In addition, small variations in lighting on the marker affected the tests results, as can be verified by the presence or not of people near the marker, even if to the naked eye the lighting was not altered (there were no shadows directly on the marker, for instance). As the configuration of the computer that was used is recent (Pentium M, with 1.73 GHz, 512 MB of RAM, on-board video card and Java version 5.0 Update 5 and Java3D 1.3.1), the first conclusion was that the problem was caused by the jartoolkit, which was developed over an old version of ARToolKit. jartoolkit latest version s date and consequently of the ARToolKit version used is July 2004, while ARToolKit s newest version, while the tests were made, dated from June However, two factors showed that jartoolkit could be used in the project: The use of dedicated video cards instead of on-board cards. The execution of the same program in Pentium IV HT computers, with 3.40 GHz, 1 GB of RAM and an ATI FireGL V3100 GPU with 128 MB of dedicated memory showed the possibility of high velocity while moving the marker. This configuration allowed the virtual object to follow the marker that was moving with the velocity of approximately 1 m/s; The use of newer versions of the same tools. For instance, using Java version 5.0 Update 8 and Java3D in the first computer where the tests were made allowed the movement of the marker in greater speeds (approximately 0.4 m/s). This factor contributed decisively for the use of jartoolkit in the project. 6.2 jartoolkit integration with enjine The integration tests with jartoolkit and enjine were accomplished in two ways: Command input tests with the markers, using games already developed with enjine; The creation of a game prototype, using this new type of interaction. In the first test, the game City Run was used, a racing game developed by students in the computer graphics course, and available in enjine s website [10]. In that game, the playability was changed so that it could accept commands from a marker in the video stream. This way, the car could be manipulated by moving the marker up, down, left and right and disregarding the movement in depth and the rotations of the marker.

7 One of the interesting results of the tests made with this prototype was that the presence of many graphic elements (in this case, the game environment) in the screen, apparently can confuse the user s control, because it covers the video and, consequently, the perception of the marker s position on screen, which serves as a feedback to the user. Additionally, the need for the horizontal inversion of the video proceeding from the camera was determined, so that the user could act as if he were facing a mirror. This showed a marked improve in usability during the tests. With regard to the marker s movement, in this prototype, when it moved up, the system detected this movement, however its amplitude was not stored, that is, the car would move forward always with the same intensity. In this moment the need for different interpretations marker s movement was identified. Interaction should possibly be made in several ways: according to the absolute marker s position, according to the marker s position relative to some point of interest or according to the position relative to the marker s former position (i.e., analyzing its movement). This approach allows the creation of games that use the camera and the markers as a flexible form of input, and that do not even need to be Augmented Reality applications. Figure 12 shows one of the prototype s screens. bounce the ball back move in the corresponding direction, and aiming to hit the ball, so that it will be sent to the adversary. In the tests made with this prototype, the goals were to verify the use of more than one marker at the same time in a game, and to learn the process of game creation using the enjine. Even though the rules of the game were not fully implemented the score, for example the goal of moving the objects with different markers was accomplished. In this test it also was possible to test the rotation through of the values of the angles that were obtained from the matrix, but without practical effects on the game rules implemented so far. However, the attempt to apply the rotations using one TransformGroups for each angle for each marker was not accomplished successfully, due to the difficulty to determine the right order in which to apply the transformations. It is worth mentioning that this prototype was already developed using the absolute position of the markers as a form of input. Figure 13 shows a screenshot of the Pong prototype. Figure 12. Test using adaptations of a game already developed with enjine Therefore, with this first test, it could be identified the following facts: Detection of the user movement s validation and their association, in fact, to a specific action in a developed game; The video stream insertion in the game screen s validation, joined with the rest of the scenery; Identification of the necessity to invert the images from the camera; The identification of the superposition of the virtual scenery over the video stream real scenery and the possible disturbance that it can be to the user relating the loss of a marker s movement visibility. For the second prototype, a game similar to Pong was created. It chosen because of the need of having few graphic elements on the screen and its easy development. Its functioning consists of moving the marker up and down, making the cubes that are representing the platforms which in the original game are used to Figure 13. Pong prototype In this second test, the following facts could be identified: Validation of the different interpretations that were developed for the marker movement (in this case, the virtual object follow the absolute value of the y coordinate of the marker but remains at a fixed x coordinate); Validation of the use of more than one marker at the same time on screen; Validation of the association between the marker and the object rotations, but in this particular application this still presented some problems. At this point, a more complete test was judged necessary to verify the behavior of the 3D object in relation to the marker movement on the z axis (depth). To this end, a test application similar to jartoolkit s demo (Figure 7) was developed, in which the cube should follow the marker and enjine should be able to, from the position matrix given by jartoolkit, interpreted the marker movement and thus the cube movement. In this test, it was identified that the camera calibration procedure developed at first and discussed earlier, in which the translation values from the position matrix were adapted to the reference plane, was functional only for games which had actions without depth variation for the markers. This is why new solutions were sought, that would allow the creation of 3D games, attending the minimum requisites of Augmented Reality.

8 As discussed before, the solution adopted was the creation of a class extending the Camera class and using jartoolkit's calibration values, an alternative that, despite being limited, allows the scope of the project to remain without drastic changes. A better calibration procedure is planned as a future work. 6.3 We A.R. Dancin This game was developed as a second stage of the project, to put to the test some Game Design concepts applied to an AR game, and to validate the integrated tools. First, analyzing the results from a brainstorm undertaken by the project's participants, a dance game was chosen for implementation, since this idea would allow focusing the work on the game logic and diminishing the efforts on more artistic areas (in which the development team lacked expertise) such as plot development and modeling of complex graphical objects. Another advantage resulting from the tests is the possibility to offer to the player the sensation of immersion in an environment that contains virtual objects without these objects interfering with the visual feedback to the player, (i.e. not occluding the video stream severely). The following requirements were defined for the game: The player must use only the markers to select all objects shown by the game on the screen, touching them, thus avoiding the use of keyboard or mouse; The possibility of two players dividing the screen, allowing multiplayer matches, besides single player games; Use of the hands and legs to execute the movements. Thus, each player must use four markers, one for each member, on which three-dimensional objects are rendered; The game presents objects that must be touched by the player with the specific member (one of the hands or one of the legs). For this, different colors and symbols are used, to indicate to the player which member he must use. Figure 14. Screen from game "We A.R. Dancin'" To facilitate the playability, the virtual objects presented by the game have its depth and scale adjusted dynamically accordingly to the marker s position. For instance, if the marker that represents the player s left hand is at 2 meters of distance from the camera, the objects related to this member will have their positions adjusted to this depth, but this is imperceptible to the player. In this manner, the game can be played independently of the distance between the player and the camera. Although it does not influence the playability, the problem with the rotation faced previously was solved with the use of a matrix already built with the angles obtained from the middleware on the TransformGroup. The matrix used is the same used internally by ARToolKit. During informal tests with several users (between 20 and 30 users), including users whose ages varied widely and with the use of a better equipment than used during the previous tests (Intel Core 2 Duo with 2.0 GHz, 2 GB of RAM and a video camera with higher resolution) the game was received with enthusiasm, with the possibility to play with nearly imperceptible delay and with correct recognition of the markers. During the game, up to 8 markers could be used, allowing the participation of two players simultaneously. While in a few frames some marker might not have been detected (or not detected with great precision), this happened rarely enough to be nearly imperceptible to the players. No cases of incorrect marker recognition (recognizing one marker as another) was detected during these tests. 7. CONCLUSIONS The integration of the jartoolkit and enjine did increase the potential use of enjine to teach several Augmented Reality concepts through the development of original games. It can illustrate, for instance, the concept of 3D registration or the merging of real and virtual elements in a rather practical way. More complex games may even be developed to study concepts of Image Processing, for instance to remove the background from an image. The implementation of the game We A.R. Dancin has validated the proposed solution satisfactorily. The creation of games with Augmented Reality offers to the players the possibility to interact with the game in more instinctive (and original) ways and was very well received in this case. Despite the success of the implementation, the project offers several opportunities for future works. Among the various possible improvements identified, some are related with new internal functions for the enjine, such as the possibility remove the background from the image obtained from the camera (as mentioned earlier), allowing the integration of the image of the player obtained in real time with the game, as the player s avatar or the game's main character, with the player immersed into the virtual game world. In the case of We A.R, Dancin, it is the virtual objects that are included in the real environment of the player instead. Another possibility is the capacity to render the real elements with depth (currently the video stream is just an background image, always occluded by the virtual objects) allowing the occlusion of virtual objects and even reducing the caused when complex game worlds occlude the real elements and reduce the feedback to the user and thus playability. In the final solution adopted, the possibility to create various instances of cameras inside the game by the enjine was limited, because of the chosen solution to calibrate the camera. An

9 important future work, therefore, is the improvement of this calibration. Regarding the use of jartoolkit itself, after the intense study of its architecture and the analysis made to identify the more reasonable solution to allow its integration with the enjine with, it was possible to identify points to improve the performance of the system. As an example, the interface between the jartoolkit and Java 3D (implemented by the component jartoolkit for Java 3D ) could be removed and implemented directly as part of the enjine, aiming optimize its performance, especially considering the fact that jartoolkit is no longer updated. The development of new Augmented Reality games using the integrated tools and exploring new game designs and forms of interaction forms is, obviously, an important possibility for future works, too. REFERENCES [1] ARTAG. Available at artag/. Last visit in December, [2] ARTOOLKIT 2.33 manual. Available at: ufrj.br/grva/realidade_aumentada. Last visit in December, [3] ARTOOLKIT PLUS. Available at: handheld_ar/artoolkitplus.php. Last visit in December, [4] AZUMA, R.T. A Survey of Augmented Reality. Teleoperators and Virtual Environments, vol. 6, p , Aug [5] BERNARDES, J.; Dias, J. e Tori, R. Exploring Mixed Reality User Interfaces for Eletronic Games. In: BRAZILIAN GAMES AND DIGITAL ENTERTAINMENT WORKSHOP, 4, Anais... SBC, v. 1, p [6] COOPER, N.; Keatley, A.; Dahlquist, M.; Mann, S.; Slay, H.; Zucco, J.; Smith, R. & Thomas, B. Augmented Reality Chinese Checkers. In: ACM SIGCHI ADVANCES IN COMPUTER ENTERTAINMENT, Anais... ACM, p [7] GEIGER, C. et al. JARToolKit A Java Binding for ARToolKit. Augmented Reality Toolkit, The First IEEE International Workshop [8] HAUSE, K. What to Play Next: Gaming Forecast , in Conference On Human Factors In Computing Systems, p , [9] INFOEXAME. News Article on Stagnation of the Worldwide Market of Games, Info Exame Magazine, Feb [10] INTERLAB. enjine: Engine for Games in Java. [11] MILGRAM, P.; Takemura, H.; Utsumi, A. & Kishino, F. Augmented Reality: A Class of Displays on the Reality- Virtuality Continuum. SPIE Vol. 2351, Telemanipulator and Telepresence Technologies, 1994, p [12] NAKAMURA, R.; Bernardes, J. & Tori, R. enjine: Architecture and Application of an Open-Source Didactic Game Engine. SBGames [13] OPENCV. Avaliable at logy/computing/opencv/. Last visit in December, [14] SANTIAGO, J.; Romero, L. & Correia, N. A Framework for Exploration and Gaming in Mixed Reality. in Pervasive Computing Environments Workshop, Pervasive 2004 Conference, [15] SCHMALSTIEG, D., Fuhrmann, A., Hesina, G., Szalavari, Zs., Encarnação, L. M., Gervautz, M., Purgathofer, W.: The Studierstube Augmented Reality Project. PRESENCE - Teleoperators and Virtual Environments, Vol. 11, No. 1, MIT Press. [16] SIGGRAPH 2005: Beyond the Gamepad panel session. Transcription available at features/ / kane_01.shtml. Last visit in December, [17] STARNER, T. et al. Mind-Warping: Towards Creating a Compelling Collaborative Augmented Reality Game. In: INTERNATIONAL CONFERENCE ON INTELLIGENT USER INTERFACES, 5, Anais... ACM, p [18] STUDIERSTUBE Augmented Reality Project. studierstube.icg.tu-graz.ac.at/. Last visit in December, [19] TEIXEIRA, J.; Farias, T.; Moura, G.; Lima, J.; Pessoa, S.; & Teichrieb, V. GeFighters: an Experiment for Gesture-based Interaction Analysis in a Fighting Game. SBGames [20] TORI, R.; Bernardes Jr., J. L.; Nakamura, R. Teaching Introductory Computer Graphics Using Java 3D, Games and Customized Software: a Brazilian Experience, Anais do ACM SIGGRAPH 2006 Educator s Program, [21] WAGNER, D. & Barakonyi, I. Augmented Reality Kanji Learning. In: IEEE AND ACM INTERNATIONAL SYMPOSIUM ON MIXED AND AUGMENTED REALITY, 2, Anais... ACM, [22] WAGNER, D.; Pintaric, T.; Ledermann, F. & Schmalstieg, D. Towards Massively Multi-User Augmented Reality on Handheld Devices. In: INTERNATIONAL CONFERENCE ON PERVASIVE COMPUTING, 3, Anais [23] WAGNER, D. & Schmalstieg, D. First Steps Towards Handheld Augmented Reality. [24] WOODWARD, C.; Honkamaa, P.; Jäppinen, J. & Pyökkimies, E. CamBall Augmented Networked Table Tennis Played with Real Rackets. In: ACM SIGCHI ADVANCES IN COMPUTER ENTERTAINMENT, Anais... ACM, p