A Mobile Device Interface Using Orientation Sensing For Electronic Games and Digital Cinema

Size: px

Start display at page:

Download "A Mobile Device Interface Using Orientation Sensing For Electronic Games and Digital Cinema"

Hester Lester
5 years ago
Views:

1 A Mobile Device Interface Using Orientation Sensing For Electronic Games and Digital Cinema Julian Bleecker Assistant Professor, Interactive Media Division, School of Cinema-TV & Research Fellow, Annenberg Center for Communication University of Southern California University Park, LUC-310B Los Angeles, CA julian at techkwondo com Peter Brinson Adjunct Professor, Interactive Media Division, School of Cinema-TV University of Southern California University Park, LUC-310B Los Angeles, CA peter at pabit org Abstract This paper describes a work-in-progress research project that investigates the use of orientation sensors that measure roll, pitch and azimuth as an interface modality for mobile, handheld devices. We focus on the use of orientation to control the point-of-view in visual applications, specifically computer games and a speculative form of cinematic and visual storytelling experience. Overview Pervasive electronic games have been attracting the attention of researchers and artists for the last several years [5, 6, 9-11, 16]. Such electronic games bring together mobile computing platforms, locationsensing technology, wireless networking systems and designed game scenarios in which the physical environment, space and architecture become components of the game s play field and part of the logic behind the game s design. The relative affordability of PDAs and tablet computers, GPS devices, and WiFi networking systems makes it possible to begin exploring their use as prototype entertainment and gaming experiences. Our research project, called Vis-à-Vis, is meant as a prototype framework for investigating how orientation sensing azimuth, roll and pitch might create a more compelling or immersive interaction modality. The research anticipates near-future usage scenarios in which mobile, handheld electronics, such as cellular phones and portable electronic game devices, are sensitive to the direction they are pointing, and more contextually aware of the surrounding physical environment. For example, in the near future devices might know where they are physically in the world, as well as where they are in relationship and proximity to other objects such as buildings, other devices, and people.

2 Figure 1. View of the current prototype device. Extending this scenario, the physical motion and orientation of the mobile device, together with location and proximity-based awareness might integrate to create a new kind of interface paradigm appropriate for a wide range of applications, including location-based cinema and electronic games, explored herein. Technology Design The Vis-à-Vis prototype consists of a three axis digital compass device developed by Honeywell. The device uses 3 MEMS (Microelectromechanical Systems) accelerometers, which are combined to provide compass azimuth, pitch and roll angles. The device includes firmware that produces a data stream over a conventional serial port interface. This feature facilitates integrating the module into custom application designs. For our usage scenarios we used a Motion Computing LS800 TabletPC, chosen for its bright outdoor viewable display, small size and suitable processing and graphics rendering capabilities [1]. We made a simple rig for mounting the TruePoint device to the back of the Motion Computing platform so that sensor readings corresponded to the panning and tilting movement of the TabletPC.

3 Prototype Designs Our research design is composed of two prototype concepts. The first is designed to investigate electronic game usage scenarios, while the second investigates visual or cinematic usage scenarios. We will describe each in turn. Prototype Design Electronic Game The electronic game prototype is built for the Microsoft Windows platform, and is composed of two software components. The first software component is a small custom application on the TabletPC that communicates over a serial link to the TruePoint device. This custom application disseminates the roll, pitch and azimuth readings from the TruePoint device to other applications, obviating the need for every application to have custom code to handle this communications step. Thus, this one custom application brokers communication with the hardware module, allowing other applications to easily obtain the data readings produced by the device. The second software component is a game engine called Torque, chosen because its license allows us access to the source code and it facilitates relatively easy creation of a first-person perspective game [2]. With access to the source code, we were able to make modifications to the game engine so that the conventional mouse or joystick interface code used to change the first-person perspective was controlled by the TruePoint device. Thus, panning the TabletPC in the azimuth, or horizontal plane moved the first-person perspective left and right. Tilting the device in the vertical plane moved the first-person perspective up and down. (For this prototype, roll was not used to change any aspect of the first-person perspective.) We developed a simple prototype game that exercised the main interface syntax of panning and tilting. The game, a variation of the children s playground game Red Light, Green Light, Go! is multiplayer and networked [19]. It requires the player manipulating the TabletPC (player one) to turn around in a circle and observe the approach of seven monsters, formed in a distant circle around player one. One of the monsters is controlled by a second player, who is at a conventional networked PC. Using voice recognition technology, player one, handling the TabletPC, says green light, at which time the monsters begin running toward player one. Player one then says red light, and all the monsters are supposed to stop. The objective of the game is for player one to infer which of the monsters is controlled by player two, by observing subtle differences in the reaction time and motion between the game s AI engine controlled monsters versus the monster controlled by player two. In this way, the game is a kind of reverse Turing Test, applied to AI bots and their movement within the game environment. We have produced video documentation of this game experience [8]. Through a number of experiments and informal user trials, we found that the interface design was compelling for users. It suggested a number of game design scenarios, including ones in which location, as derived by GPS, could play an integral part in a location-based electronic game design. Prototype Design Cinematic Experience Our second prototype is designed to investigate how manipulating the orientation of a mobile media device might be a component of a mobile cinematic experience. We worked from a basic usage scenario

4 in which physically manipulating the orientation of the device changed the point-of-view within a panoramic cinematic experience. Our initial prototype made use of the Quicktime VR video format, which allows the point-of-view in a 360 degree panoramic scene to be freely altered. For this design, we used the same TruePoint device and TabletPC as used in the electronic game design. For rendering cinematic visuals, we used Cycling74 s Max/MSP+Jitter, a robust visual programming environment suitable to creating custom applications that make heavy use of graphics [3]. Max/ MSP+Jitter was chosen because it runs on both Windows and OS-X, has native features that support the required serial port communication, and has the ability to programmatically manipulate Quicktime VR movies. In this prototype, pan and tilt of the TabletPC allows one to pan and tilt within a Quicktime VR movie. We experimented with several Quicktime VR movies and found that the experience created a sense of immersion that went beyond what one felt while viewing Quicktime VR scenes at a desktop computer. We have produced video documentation of this experience [7]. Implications for Computer-Human Interaction This research is largely motivated by a desire to create more kinesthetic interface modalities that are well-suited to handheld, mobile computing platforms such as the nearly pervasive cellular phone. The proliferation of mobile devices that are largely manipulated with one s hands and arms makes it important to expand prevailing interface designs that were relevant for tethered usage scenarios, such as fixed desktop PCs, or even laptop computers [12, 13, 17, 20]. Future Directions for Investigation Our objective with this prototype was to create a technology framework that would allow us to experiment with a variety of game designs and cinematic experiences that made use of orientation as an interface component. Future directions for this prototype under consideration include integrating location sensing (via GPS, for example). Such a framework would allow for the creation of location-based content experiences, for example, in which media is triggered both by one s position in the world, as well as by the direction in which one is orienting the device. This framework as described would facilitate so-called geospatial web experiences in which network-based data on the web is geographically tagged and, with a WiFi networked mobile device, accessed and experienced in the geographic context for which it is relevant [4, 18]. Other possible future directions include orienteering, way-finding and mapping systems in which a digital map is sensitive to orientation so that holding the TabletPC map up in front of one s line of sight allows one to orient in the proper direction to follow a path or to identify buildings, roads and landmarks [14, 15]. We are presently investigating a much smaller mobile device, the OQO, to develop a prototype that anticipates the use of this kind of interface design for handheld devices such as mobile phones.

5 Acknowledgments This research project is under development at the Mobile and Pervasive Laboratory at the University of Southern California. This laboratory is an interdisciplinary facility jointly run by the School of Cinema- TV and the Annenberg Center for Communication. We gratefully acknowledge the continuing support of the Annenberg Center for Communication and the School of Cinema-TV. References [1] Motion Computing LS800 TabletPC. [2] Torque Game Engine. [3] Cycling74 Max/MSP+Jitter. [4] The New Sense of the Web. [5] Benford, S., Fraser, M., Reynard, G., Koleva, B. and Drozd, A., Staging and evaluating public performances as an approach to CVE research. in Proc. of the 2002 Int'l Conf. on Collaborative virtual environments. [6] Benford, S., Rowland, D., Flintham, M., Drozd, A., Hull, R., Reid, J., Morrison, J. and Facer, K., Life on the Edge: Supporting Collaboration in Location-Based Experiences. in Proc. ACM SIGCHI [7] Vis-a-Vis Cinematic Experience Documentation. [8] Vis-à-Vis Games. [9] Cheok, A.D. ACM SIGCHI international conference on advances in computer entertainment technology. Comput. Entertain., 2 (1) [10] Cheok, A.D., Goh, K.H., Liu, W., Farbiz, F., Teo, S.L., Teo, H.S., Lee, S.P., Li, Y., Fong, S.W. and Yang, X., Human Pacman: a mobile wide-area entertainment system based on physical, social, and ubiquitous computing. in Proc. ACM SIGCHI [11] Cheok, A.D., Yang, X., Ying, Z.Z., Billinghurst, M. and Kato, H. Touch-Space: Mixed Reality Game Space Based on Ubiquitous, Tangible, and Social Computing. Personal Ubiquitous Comput., 6 (5-6) [12] Daniel, W. and Ravin, B., /TiltText/: using tilt for text input to mobile phones. in Proceedings of the 16th annual ACM symposium on User interface software and technology. [13] Harrison, B.L., Fishkin, K.P., Gujar, A., Mochon, C. and Want, R., Squeeze me, hold me, tilt me! An exploration of manipulative user interfaces. in Proc. SIGCHI 1998, Pages. [14] Ito, M., Furuichi, Y., Nakazawa, J. and Tokuda, H., mpath View: An Interactive Behavior History Viewer for Enhancing Communication. in Adjunct Proceedings of the 3rd Int'l Conf. on Pervasive Computing, [15] Ito, M., Nakazawa, J. and Tokuda, H., mpath: An Interactive Visualization Framework for Behavior History. in The IEEE 19th Int'l Conf. on Advanced Information Networking and Applications, Pages. [16] Koleva, B., Taylor, I., Benford, S., Fraser, M., Greenhalgh, C., Schnädelbach, H., Lehn, D.v., Heath, C., Row-Farr, J. and Adams, M., Orchestrating a mixed reality performance. in Proc. ACM SIGCHI 2001,

6 [17] Levin, G. and Yarin, P., Bringing sketching tools to keychain computers with an acceleration-based interface. in Ext. Abst. CHI '99, Pages. [18] The Geospatial Web: A Call To Action. [19] Opie, I. and Opie, P. Children's Games in Street and Playground : Chasing, Catching, Seeking, Hunting, Racing, Dueling, Exerting, Daring, Guessing, Acting, and Pretending. Oxford University Press, [20] Poupyrev, I., Maruyama, S. and Rekimoto, J., Ambient touch: designing tactile interfaces for handheld devices. in Proc. of the 15th annual ACM symposium on User interface software and technology.

The Mixed Reality Book: A New Multimedia Reading Experience

The Mixed Reality Book: A New Multimedia Reading Experience Raphaël Grasset raphael.grasset@hitlabnz.org Andreas Dünser andreas.duenser@hitlabnz.org Mark Billinghurst mark.billinghurst@hitlabnz.org Hartmut