AR Cannon. Multimodal Interfaces. Students: Arnaud Durand 1, Léonard Stalder 2, Thomas Rouvinez 3 Professors: Dr. Denis Lalane 4, May 23, 2014

Transcription

1 AR Cannon Multimodal Interfaces Students: Arnaud Durand 1, Léonard Stalder 2, Thomas Rouvinez 3 Professors: Dr. Denis Lalane 4, May 23, 2014 University of Freiburg 5 // Freiburg University of Freiburg Msc. Computer Science 1700 Freiburg Switzerland 1. arnaud.durand@unifr.ch 2. leonard.stalder@unifr.ch 3. thomas.rouvinez@unifr.ch 4. denis.lalane@unifr.ch 5. University of Freiburg,

2 i Table of Content 1 Game overview 1 2 Game interactions and workflow Gameplay overview Moving cannons Firing Game workflow Technologies and tools Android Unity Vuforia Android Speech recognition API Custom-built controller Architecture 5 5 Modalities CASE model CARE model Evaluation Quantitative evaluation Survey Result interpretation Qualitative evaluation Improvements 9 8 Conclusions 10

3 1 Abstract AR Cannon is a modern remake of the game Bang Bang 1 which is a simple turn by turn 2D ballistic game. Each player controls a single cannon and can select its elevation and power. With AR Cannon we take the same concept but adapt it to the current technological state of human-machine interfaces. The battlegrounds are made of augmented reality to exploit players imagination. The keyboard is replaced by speech recognition or a custom controller (built for this game). We exploit multiple modalities to enhance players immersion in the game. We use this opportunity to experiment with multiple types of modalities for the same actions. Our results indicate that speech recognition deepens the experience while the use of the custom controller results in faster turns and more intensity action-wise. 1 Game overview Bang Bang is a ballistic game with a glorious past. Developed for Windows 3.0 its concept is rather simple : allow two players to fight turn by turn with cannons. Each player can control the elevation and power of his/her cannon and attempt to use ballistic to destroy the enemy s cannon. The concept of Bang Bang is clear, easy to learn and fun to play with. Many other games extended this concept and created new games based on it. However most (if not all) these new implementations are platform specific and use the types of controls they provide. Note also the evolution of gaming as tower PCs are less and less the main platform for gaming. With the emergence of new smaller devices such as smartphones and tablets that have the graphical performances required to run games many markets have focused on these new platforms. With centralized online distribution of software smartphones have become one of the most used gaming device. It however breaks with the tradition of playing with a mouse and a keyboard. This leaves room for new types of inputs and modalities. Such devices nowadays feature multi-touch screens, microphones and gesture recognition. Figure 1 Augmented reality is used as a battleground. During this project, we attempted to recreate a game exploiting the neat design of Bang Bang while using the latest technologies. AR Cannon stands for Augmented Reality Cannon. Many aspects of the original game can be extended upon. We chose to improve on the following gameplay elements : Diversity of battlegrounds : in the original game the battlegrounds were fixed 2D drawn images. The cannons would always be placed at the same locations and would allow players 1. See

4 2 to know with experience what inputs would guarantee one-shot wins. To remedy to this issue we based the whole game on augmented reality to enable players to use the real world as a battlefield. By doing so we guarantee unlimited possibilities and no two identical games. Freedom of movement : the original cannons were displayed in 2D, thus forcing the gameplay into a 2D perspective also. The only controls available were elevation of the cannons and the firing power. Since we use the world as a battlefield, we need to use 3D cannons. Therefore we let the player control elevation, orientation and firing power of the cannons. This added complexity results in more turns being played before the match is over and more excitement for the players. Controls : the original game was built for PC devices and relied only on keyboard inputs. Our wish for AR Cannon is to create a broader experience by using more natural modalities to be used to control the cannons. Real artillery is commanded from the battlefield by an officer providing coordinates. The orders are relayed by radio and executed behind the battlefront. We want to follow the same workflow in AR cannon to provide a more immersive experience for the players. Using augmented reality, the player may observe the battlefield and come up with coordinates to attack. By pushing the transmission button, the player can transmit by radio spoken orders that the artillery will carry out. Finally, controlling the firing power of the cannons is achieved by capturing the intensity at which the firing order is spoken. Yell and the cannon shall release its full power. As previously said we also created a custom built controller for alternate controls over the cannons. The controller is made of three potentiometers that represent the movements the cannons can carry out. There is one knob for elevation, one for orientation and one for firing power. Finally the user can use the single button available to actually trigger the strike. From a multimodal point of view, we create mixed interactions between all these modalities. We rely on augmented reality, automatic speech recognition, voice intensity, touch screen and an external controller. In the next section we present all the technologies used to enable each of these modalities within the game. 2 Game interactions and workflow 2.1 Gameplay overview The game starts by setting the playground : the players agree on the game physical location and each player place its augmented reality (AR) image target on the playground. As soon as the targets are detected by the game, the players see the cannons appearing on top of their tracker. As the game is turn-by-turn, only the current player can do an action. Turn time is limited, so the player should choose one of these actions as soon as possible : Move the cannon Fire When the current player fire or its time is up, the other players take the control. The objective is to hit the other player with a cannonball. 2.2 Moving cannons Players have to adjust their cannon in the opponent direction. Two modalities are available for this action : speech recognition or physical interaction with the dedicated game controller. Using speech recognition, the player can give a voice order like Turn 20 north! or Turn 5 up and 65 right! directly to the microphone. Using the game controller, the player can adjust the cannon by moving the knobs. These modalities are redundant, so player can choose one of them, or even use both at the same time (see sect. 4 for implementation details). When the cannon is moved, the user is rewarded by a visual feedback of the cannon rotating directly on the playground. If the player give an order using speech, the game reply to the user if the order has been accepted or if it was not recognized using pre-recorded audio messages.

5 3 2.3 Firing When the current player is happy with the cannons position, he/she tries to hit his/her opponent by firing a cannonball. Firing cannon can be achieved using two concurrent modalities : analog controller input or voice meter. By using the analog controller, the player is able to precisely adjust the power by moving a knob and then hitting the fire button. If the player choose to rather use the voice meter, he can adjust the power by shouting on the microphone. Current player firepower is always visible through a 3D power bar indicator. When the cannon fire, there is both a visual feedback (we can see the cannonball animation) and an audio feedback (fire and then explosion sound effect when hitting the ground or the opponent). 2.4 Game workflow The workflow is summarized in fig 2. Figure 2 Game workflow. 3 Technologies and tools 3.1 Android The game is targeted at mobile platform, specifically Android. After a basic technology analysis, we evaluated that targeting Android was a better option for this game than building a PC game for the following reasons : Mobile devices are well suited for AR applications as they generally have cameras and are light enough to be moved around the physical playground. Vuforia is the best AR library currently available and its dedicated to mobile devices.

6 4 3.2 Unity Mobile devices have microphone and Android provide a built-in speech recognition module. We had some personal devices already available for testing. Unfortunately, working on mobile platforms is not as flexible as working on PCs. Indeed we cannot access raw devices and we need to rely on built-in API (e.g. game controller). Also compiling, exporting and testing on external device is time consuming. Unity is a generic multi-platform game engine. Application can be tested on the PC using built-in player while targeting other platforms. Unity enable us to easily develop 3D game for Android platform for free, as the application does not need any of the Pro version features. It also supports a wide range of 3d models formats for game assets. Game logic is programmed using C#, UnityScript or Boo language scripts. Our application is fully programmed using C#, with the notable exception of Java for native Android calls. 3.3 Vuforia Vuforia 2 is an augmented reality (AR) library for mobile devices. It is available both as SDK for Android or IOS and as Unity extension (the later was used). Vuforia functionalities include image tracking, cylinder targets tracking, extended tracking (tracking of objects outside of Field of Vision abbrev. FoV), tual buttons and more. Our application mainly use image tracking and extended tracking. Our targets (cannons) are each mapped to an image target. Tracking quality is dependant to image complexity, image size, camera quality and device processing power. To achieve a good level of tracking, we used full-sized A4 complex images. On top of that, we enabled extended tracking : using image s motion, the library is capable to locate targets outside of camera FoV. On the context of our game, this feature is important as players device may not continuously view both trackers. 3.4 Android Speech recognition API To enable the game to detect and recognize spoken orders, we rely on the built-in Android Speech API. This API allows developers to use the recognition service from Google 3. Any Android application may request the Google recognition service to perform online speech recognition. To do so a program must create a recognizer object and initialize it. The microphone of the device is then enabled after a short audio beep to notify the user that speech recognition is listening. Once the user has stopped to speak, the recognizer releases the microphone s resource and sends as feature vector to Googles servers. The reply from these servers is a string with the recognized textual information. The Android Speech API is not directly available through Unity scripts. Therefore we developed a native plugin to specifically handle speech recognition and capture microphone decibel readings. This plugin is then triggered from C# script, executes the native statement blocks in Java and finally return the results back to the C# script. This process is fast enough to provide the illusion of simple radio delays in the game and does not affect the player s experience. 3.5 Custom-built controller The custom-built controller is based on an small-form factor Arduino Leonardo. It has three analogue inputs (potentiometers) and one digital input (button). The ATmega32u4 microcontroller integrates a USB controller, enabling standard HID device emulation. This was a requirement to communicate with an Android device. The controller is connected to the Android device through OTG and the Android device act as master. HID device emulation is achieved through the unojoy 4 library. Potentiometer inputs are mapped to x,y and z axis and are sent as 8-bit values, enabling 256 levels of precision. There is no preprocessing (smoothing, dead zones...) directly on the controller. The controller is then detected as a standard gamepad by the Android device and provides access to it through its APIs. The controller is then accessed by Unity through Android APIs. 2. See 3. See 4. See

7 5 Figure 3 The custom-built controller. 4 Architecture The game is split in logical objects (called GameObject in unity terminology) with attached behaviours. Cannons are objects with their own behaviour, enabling them to rotate or fire a cannonball projectile (which is also an object with its own behaviour). The cannons are enabled and disabled every turn by a manager object. Augmented reality through Vuforia is permanently enabled. At every frame, target positions are calculated. Conceptually cannon s transform (the combination of position, rotation and scale) data fusion occur in two steps : calculate base transform using AR targets and then applying rotation of game inputs (voice and game controller). In practice the first step is done automatically by the Vuforia SDK and parent/children mechanisms enable relative transforming without explicit coding. Every cannon has two quaternions : one for the the rotation relative to speech orders (initialized as identity and calculated during orders) and one related to the controller axes position (calculated every frame). Obviously cannons rotations are always relative to the rotation of the target tracked by Vuforia. Final position is calculated by interpolating those two quaternions. Using these techniques players have the ability to move their cannons using both voice commands and the game controller simultaneously. Figure 4 Transform data fusion. Sequential events (GUI interaction, fire button and sound meter) are directly called by the manager. Speech recognition is called by the manager via an asynchronous method. When the speech has been analysed, it is returned to the manager with a callback and the order is analysed using regular expressions. Unfortunately, audio analysis with restricted vocabulary is not available on Android. If the order is accepted the cannon start moving by interpolating between the current position and the target position by t, where

8 6 t is the movement progress. This enables cannons to move smoothly during several frames. The user can either use the fire button or the sound meter to launch the cannonball. If the user presses the fire button, the cannonball speed will be set by the controller z axis value (one of the potentiometer). If the user chooses to use the sound meter, he/she presses a GUI button, then the controller is disabled and the sound volume smoothed using a circular buffer. When the user releases the button the cannonball is launched at the speed of the last smoothed value. 5 Modalities Multimodal interfaces are subject to evaluations and formalizations. Specific models have been designed to bring standardization to this evaluation process. To conceptualize the relationships between the inputs and outputs we use the Case/Care model. The Case describes the multimodal communication types on the machine-side whereas the Care model focuses on the human-machine level. Figure 5 Simplified sequence of a turn. 5.1 CASE model The case model focuses on four different types of machine-side interactions between modalities. We present these four types with respect to AR Cannon s implementation : Concurrent : two modalities are applied in parallel, but independent of each other. As previously explained AR Cannon can be used with an external controller to interact with the elevation, orientation, fire-power and trigger of the cannons. The same interactions performed with the potentiometers can also be performed using voice commands. For example a spoken order to modify the orientation of the cannon can be given while the player is using the elevation potentiometer on the controller. Same applies for all combinations of elevation, orientation and fire-power adjustments. Alternate : in order to fulfil a task two modalities must be used one after each other. AR Cannons uses the real world as a battleground and relies on augmented reality to do so. Augmented reality requires trackers be placed in the world to be able to place virtual objects and update their position according to the user s movements. In order to get a game going, the player must first place the markers in his/her environment, then use either the controller or spoken orders to interact with the cannons. Once the cannon is in the desired state and the user is using voice commands, he/she must then press a touch button to start recording the voice intensity levels to control the fire-power. When the desired fire-power is input, the player has to press

9 7 again the touch button to actually fire the cannon. In AR Cannon the whole game flow is based on alternate modalities. Synergistic : different modalities have to be applied in parallel and at the same time to reach a goal. AR Cannon features two cases of synergistic modalities : the first case involves the spoken orders and the controller. Each spoken order given by the user takes time and as each turn is time constrained, the user may want to quickly make small adjustments to the cannon s elevation and orientation before firing it. For example a spoken order to control elevation can be input at the same time the user corrects the orientation using the controller. The second case is used to fire the cannon by using voice intensity levels as power adjustment modality. The player has to adapt his/her voice intensity to input the firing power and click on the touch button to fire the cannon when he/she is happy with the fire-power level acquired. Exclusive : non-complementary and non-simultaneous usage of modalities. AR Cannon uses only one exclusive modality. When the game is first launched, the augmented reality needs to scan the chosen battleground for the markers. During this phase, no other modality is used nor can be used. 5.2 CARE model The case model focuses on four different types of human-side interactions between modalities. We present these four types with respect to AR Cannon s implementation : Complementarity : use of multiple modalities required to reach a goal. Technically speaking, all the actions in AR Cannon use complementarity to some level. Indeed all actions are based on the augmented reality for visualization, feedback and modality use. For example one cannot give preview the current position and orientation of a cannon or see the feedback of voice intensity levels capture without the cannons displayed on the screen. Assignment : only one modality can be used to reach a given state. AR Cannon has two assignment modalities : both as linked to the use of buttons. The first button starts the automatic speech recognition and the second one starts the capture of voice intensity levels. Redundancy : two modalities can be used to achieve the same goal when used independently in time. The best example of redundancy in AR Cannon is the use of the controller to perform the same actions as the voice can. Equivalence : use one, another or multiple modalities at the same time to execute an action. The best example in AR Cannon for equivalent modalities is the adjustment of the firing power of the cannons. The user can use just as much voice intensity levels or the respective potentiometer on the controller. To some extent the controller and spoken order achieve the same purpose but with different sensitivities and responses.

10 8 6 Evaluation 6.1 Quantitative evaluation Survey Having our application finished, we evaluated and compared which modality is the most adapted to the game and the user. To perform a T-test we asked the users to play a game and we measured the time it took to achieve a game. The goal is to know which modality is the most efficient for such a game. We tested our two modalities with 10 users.the following results we found : Protocol A The user plays a game with speech recognition in X seconds Protocol B The user plays a game with the custom-built controller in X seconds Protocol A in sec Protocol B in sec The following graph shows the distributions for both approaches : Figure 6 Simplified sequence of a turn. We did an unpaired t-test to determine if the two sets of data are significantly different from each other. We see that both protocols follow a normal distribution and we determine the following hypothesis test : H0 the average of our two sets is equal. µ A = µ B H1 µ A > µ B The goal of this test is to reject H0 and accept H1. Once a t-value is determined, a p-value can be found using a table of values from Students t-distribution. The results are the following : t = , df

11 9 = 18, standard error of difference = 9.707, P-value = , x A x B = 27.9, 95% confidence interval of this difference = From 7.51 to Result interpretation Group Protocol A Protocol B Mean SD SEM N The calculated p-value is below the threshold chosen for statistical significance (usually the 0.10, the 0.05, or 0.01 level). Therefore we can reject the null hypothesis (H0) and accept the alternative hypothesis (H1) and so the second modality is more efficient than the first on. With the custom-built controller people complete a game faster than the voice recognition controller. 6.2 Qualitative evaluation During the testing phase with the ten users we also asked them questions about their general impressions on the game and observed them during the rounds. The following remarks and observations emerged : We observed that the users have more difficulties to define the power of the cannon with the voice controller. Having to define the power of the cannon with the power of the user s voice is takes more as to define power with the potentiometers. The surveyed users prefer and find it more fun to control the cannons with the voice but they find the potentiometers more efficient. We also observed having to repeat the sentences into the microphone causes time wasting. To conclude, the surveyed people prefer and find it more fun to control the cannon with the voice. The augmented reality is usually much appreciated by the users and makes the game really interesting. 7 Improvements In its current state the game is playable and fun. The modalities are used by the players and the learning curve is steep. However the first valuable improvement would be to polish the GUI of the game. Indeed the current buttons are barely readable and could be placed in a better way to improve reachability. Globally the whole HUD should be a single frame with the buttons and functionalities well implemented in it. Another improvement would be to insert an explicative screen during the loading of a new match. Since it is a mandatory screen every player would have to watch it. Currently players are familiar with the gameplay but not with the modalities. Many users tend to keep the button for automatic speech recognition pushed while they speak for example.since we developed our own game for this project, we did not focus too much on the game itself but more on the mixing of modalities within a single screen. From the modalities point of view, the controller could be improved. Currently it is hard to use as because it requires the player to carry both the tablet and the controller at the same time. Moreover the wire to the controller is not convenient as the players may move to observe the battleground under different perspectives. Also the potentiometers could be placed in a more intuitive way and the whole controller would benefit from a miniaturization. Speech recognition worked better than we first anticipated. We currently recognize up to two orders and cancel all noisy words not useful for the game. Since this system is not depending on our implementation, the only improvements possible would be better user feedback. At the moment we use the responses of the soldiers in game to acknowledge if an order was understood or not. We have noticed that all the errors happen on the number of degrees to move. For every order to be correctly recognized, the player has to really articulate well the numbers.

12 10 8 Conclusions This project turned out to be a two-way success. First the game we created is fun to play and works surprisingly well. We expected the game to reach a decent level of playability but the game is actually fun and easy to use. Players who tested AR Cannon needed only one game to learn the mechanics and largely used speech recognition. Also using augmented reality amazed our testers as it really opens a new dimension which is largely unexplored in gaming. Secondly the multimodal interaction with the game truly deepens the gameplay experience. Speech is a more natural way to give inputs and orders in the context of the game. Players with more experience become fast to input orders even with speech recognition. This is possible thanks to the Google Speech API which is fast to return results, giving the impression of reactivity. In the end the whole systems ends up being natural to use and simplicity allows for more fun. The player feels more immersed into the game with speech recognition than traditional inputs like keyboards or even controllers. Designing and implementing a multimodal interface requires a clear original concept and a well thought integration of all the modalities. Using a game as a support application helps with being imaginative but also give opportunities of using synergistic modalities that require a lot of care during the implementation. Overall this project was very interesting to work on and opened many new ideas for other developments.