WiiInteract: Designing Immersive and Interactive Application with a Wii Remote Controller Jee Yeon Hwang and Ellen Yi-Luen Do Georgia Institute of Technology Atlanta, GA 30308, USA {jyhwang, ellendo}@gatech.edu ABSTRACT We designed and implemented WiiInteract to provide a tangible interface that links digital interactivities directly to children s real life experiences. WiiInteract consists of four interactive applications that support playful and tangible interactions for children. WiiInteract utilize physical manipulations and movements of digital data through a sensing device, the Nintendo Wii Remote Controller. WiiInteract has immersive interactions in the natural environment and aims to offer a potential creative play environment through tangible interaction. In this paper, we report the design and implementation of WiiInteract applications and conclude with discussion and future work. Author Keywords Tangible interaction for children, Wii Remote Controller, immersive interaction processes. ACM Classification Keywords D.2.2 [Design Tools and Techniques]: User interfaces; H.5.2 [User Interfaces]: Input devices and strategies, Interaction styles, Screen design, Style guides. INTRODUCTION The tangible user interface (TUI) has emerged as a powerful method through which to embed digital elements in the real world. We have a recent trend that applies TUI applications to playful learning systems for children to join the physical and digital worlds [13]. In order to facilitate realistic interaction, WiiInteract connects physical movements and screen-based digital information with applying a design approach for TUIs. The design process of WiiInteract involves considering how to motivate children so that they will interact and engage with the tangibility as interactive artists do with their artwork [1]. The existing tangible interfaces, like I/O Brush [10], StoryMat, a soft storytelling interface [11], and the Tabletop interface [9] are creative examples of tangible interfaces, but they require extra interaction techniques and devices, such as sensors, 15th International Conference on Computer Games: AI, Animation, Mobile, Interactive Multimedia, Educational & Serious Games, July 28-31, Galt House Hotel, Louisville, Kentucky, USA. cameras, and LEDs. WiiInteract, on the other hand, needs only a Wii Remote Controller (Wii Remote), which integrates novel interaction techniques and motion sensing capabilities into a tangible remote control device. Thus, WiiInteract enables children to achieve embodied interactions with its wireless and sensing technology. With its applications and flexible interaction methods, WiiInteract can offer expressive and non-competitive interactions. The educational technology for children has been constrained by two-dimensional exploration [8]. To improve screen-based interactions, WiiInteract utilizes its three-dimensional physical controls and playful applications, which model the natural environment. RELATED WORK There are two categories of related works based on system components: Tangible interfaces using the Wii Remote, and other technical devices. The Pinocchio project builds a system to enable children to experience and learn classical music by conducting a virtual orchestra with a Wii Remote [2]. WiiArt project is primarily for artistic and expressive interactions. It includes prototype applications, such as Illumination (drawiing), Beneath (Waldo), Time Ripples, and WiiBand, to compose both images and sounds for art experiences [7]. One of the non-wii interfaces is StoryMat, a tangible interface covered with soft quilt. The software in the mat works with an ultrasonic transmitter embedded in a small stuffed animal. The StoryMat mediates a conversation between children plays the role of a playmate by responding to the child s story with a similar story [11]. Another example is I/O Brush, which contains a video camera, lights, and touch sensors. The I/O Brush is a design and development tool that allows users to create visual art with elements extracted directly from their personal objects and their immediate environment [10]. In comparison to these interfaces, WiiInteract uses one simple controller to manage multimedia options and enables children to experience diverse connected activities rather than working with one constrained feature. WIIINTERACT: APPLICATIONS FOR A TANGIBLE INTERFACE Interaction Control with a Wii Remote Controller The Wii Remote contains an accelerometer, infrared camera, and built-in 4-point tracking blob, all of which support motion-sensing technology. After connecting the Wii Remote
to the computer via Bluetooth wireless communication, the user can start interacting by launching WiinRemote, an existing program that uses the controller on the computer. The program provides cursor options and a button-assigned window. Each application s inputs or the user s movements were assigned to the Wii Remote s buttons (up/down/left/ right, +/-, A/B, etc.) and the motion parameters through the Wii Remote. WiiInteract uses a motion sensor instead of an infrared (IR) sensor for its cursor movement and smooth navigation in its Flash-based applications. Flash and Action Scripts were used in the design of WiiInteract s visual interface and its immersive applications. The Glove Programmable Input Emulator (GlovePIE) Script, which initially was developed for Virtual Reality Gloves, is another development option to control the remote controller. However, WiiInteract focuses on Flash and Action Script components to keep the applications easy-to-execute in any case or environment. For example, users can make interactive waves from LakeForest with light tapping motions, and they can easily make constellation patterns in StarLight with simple drag-and-drop motions. Further, they can feel the different aspects of Earth instantly when they select one of the view options from GlobalSteps. According to the previous observations on Wii tennis game plays of 14 researchers for 3 days, Wii Remote is an effective modality to support immersive and playful interactions. Wii Remote enables users to control screenbased interface with dynamic motions like waving their arms in similar to their real-life actions. Thus, when playing the game, some of the researchers said that their body movements such as serve and backhand stroke lead them to have more playful and interactive experiences. WiiInteract also utilizes the motion sensing methods and interaction options of the Wii Remote, but the difference is, beyond the typical Wii games, WiiInteract provides tangible interaction for children to explore immersive experiences of play in the real world. In this way, WiiInteract provides tangible interactions, not for the augmented virtual environment but for the augmented real environment. maps on the right that the children can select to see displayed on the globe. Users can zoom into specific area when pushing the Wii Remote Controller s + button, and Figure 1. GlobalSteps enable users to explore various aspects of earth using its interactive globe. manages exploration speed by moving the controller horizontally such as a page flipping motion. In addition to the unique features on manipulating size or speed, GlobalSteps has segmented multiple views that connect to the organized map, Google Earth, to improve its interaction and navigation. It can be enhanced to a mixed reality tour game by manipulating some parts of its visual data representation to zoom in on some aspects of the Earth. This "mixed reality tour" can give children the opportunity to tour the place around the world where they have never visited with zero carbon footprint. WIIINTERACT INTERACTIVE APPLICATIONS WiiInteract gives pre-defined menu options and enables children to set their preferences and change interaction types. WiiInteract consists of four interactive applications to support its tangible interaction: GlobalSteps, StarLight, LakeForest, and TravelWorld. GlobalSteps Children can learn various aspects of earth with GlobalSteps. There are four options to view and rotate the threedimensional globe. Playing with GlobalSteps will help children grasp the concepts of environmental pollution, energy consumption, and explore aspects around the globe. The application also includes additional objects, such as asteroids and stars that change their colors according to children s mood or interacting status [5]. Figure 1 shows the display of an ever evolving globe and the various status Figure 2. StarLight users can match stars to the constellation picture or create new ones. StarLight StarLight encourages the study of constellations by children with direct mapping of stars to the constellation picture and also enables them to create new ones using the flexible controller s movement. Being able to change the connecting network diagram of stars and to make their own constellations can help them reflect and understand the
constellations. This constellation-making procedure would be useful for other applications that a diagram structure may be useful. For example, computational thinking tools, such as a tangible state machine game, Escape Machine [12], and a science game of DNA sequences can also use this constellation creation structure. In addition, iterative steps in constellation creation can enable users to engage in interactive learning and storytelling based on the mythology of the constellations. In this manner, StarLight has the potential to visualize the children s symbolic perception, expression, and imagination. TravelWorld TravelWorld is a dynamic word and picture matching game. When children select and drag the message box with while holding the Wii Remote s B (Backside) button, they can rotate, shuffle, or move it in any direction. After reading the essential information from the message, the user looks and matches each picture with the message. If the graphical or textual information is insufficient, then the user can search for additional information from Wikipedia.com by following its link. TravelWorld is also applicable to a communicative and social interaction to share not only messages, ideas, and pictures, but also immersive physical activities. Further, TravelWorld can be enhanced to work for an interactive message center or a discussion room for design collaborations with its flexible message control options and gesture data representation. Figure 3. TravelWorld has interactive options for users to travel around the world and learn value-added geographical information. LakeForest LakeForest application is inspired by both the peaceful enjoyment and the playful acts of ripple making people often engage in lakeside scenery. On the lake surface, interactions with the Wii Remote are marked as traced images of wave and ripple movements. LakeForest displays realistic and natural images for the context of the play. Users can move the controller in a horizontal direction, such as smoothly moving the sunlight, touching the surface, and feeling the water flow. In addition, by tapping the lake s surface, the user can play with reflective ripples. This interaction may inspire and increase users participation on natural environment. The authors plan to combine sounds with LakeForest s actions to increase children s immersion and participation. The research project, Elysian Fields, shows that the combination of sound and action enables users to perceive of being physically located in the place and actually engaging in these activities. This sense of having a physical effect may also give participants a strong pleasure from creation [1]. Figure 4. LakeForest presents water waves and ripple movements reflecting the Wii Remote s motions and touches. FUTURE WORK WiiInteract offers selective interactions and navigations by obtaining inputs from the Wii Remote s buttons and motion sensors. This interface offers sophisticated motions, such as pitching, rotating, and rumbling to enable children to interact more actively with the virtual environments that simulate our natural environments. Future work of this project could enhance the sensory aspects of the system similar to what Johnny Chung Lee did for his Wiimote Projects. His work connected digital sensors to the real world objects are demonstrated in Wiimote DesktopVR, Whiteboard with IR pen, and Finger Tracking using the Wii Remote Controller [6]. Further extension of WiiInteract could augment the play interaction with a shape or an avatar to represent children s activities on screen in the virtual world, or display previous play interaction data (e.g., images of prior journey and explorations, travel logs, etc. from themselves or other children around the world) to motivate children to reflect and explore the various activities. Tangibility and the Child-User Benefits During the development process of WiiInteract, we were interested in about how the integration of the representation of natural environment with tangibility in application design can help and enhance children s interaction activities. For approaches of user studies, we drew from three particular methodologies that we would briefly describe here. The first method is surveying each participant about their experiences and feelings with detailed classifications of choice options, such as the 13 pleasure categories of play [1]. The framework of the pleasure categories was developed by
the six theorists, all of whom approach play and pleasure from different perspectives. According to Costello, B., and Edmonds, E., of A Study in Play, Pleasure, and Interaction Design [1], the framework can enable the interactive designers to focus more on the type of playful experiences that they want their work to elicit. Likewise, we can get feedback on what children feel and experience from the tangible applications and compare their responds to the first assumed results, like creation and fantasy, which were made during the design process. This process can bolster the weak points and find their unexpected positive effects. The second option is to conduct play testing of WiiInteract several times using different sizes of user groups. When they form a team, children can participate in interaction more cooperative and social ways. As shown in the paper, People, Places, and Play: Player Experience in a Socio- Spatial Context [3], the games frequently gives meaningful social interactions in the real and tangible world of gamer. Moreover, WiiInteract s applications provide more educational and pragmatic experiences. The group activity is, therefore, effective to obtain social and empirical evidence to support the assumed benefits of tangibility. Lastly, asking caregivers (i.e., parents and teachers) may be an efficient way to assess benefits of tangibles for children. We can also provide a questionnaire using pictures, short sentences, and diary options to let children express their opinions comfortably and conveniently. CONCLUSION In this paper, we present WiiInteract, a tangible interface with four applications that provide immersive, expressive, and creative experiences with the Wii Remote. Believing that engaging in well-designed game enhances the user s social and other abilities in real life [3], WiiInteract focuses on offering children educational and playful benefits through its interconnection between physical controls and the realistic natural environment. Studies of playful behavior report that the player switch back and forth between the explorative goal, What can this object do and the playful goal What can I do with this object [1]. The user also becomes easily bored if they cannot find new features or possibilities to play with. Thus, in support of tangible interface that provides responsive and interactively changing environment, WiiInteract seeks to find a balance between those goals while providing customizable and renewable content with its applications to support children s preferences and frequent activities. WiiInteract encourages children to the design of interactive applications to explore the potential of the Wii Remote for playful learning and interaction. In this way, it focuses on helping children become more active with their bodily actions and specific controls of playful applications such as the modeling and visualizing of the natural environment. The design ideas and applications of WiiInteract offer a new approach to the tangible interface using the Wii Remote. Exploring a variety of application properties, this design pattern may provide interesting properties and insights that may benefit design and implementations of future tangible interfaces. REFERENCES 1. Brigid Costello, B., and Edmonds, E. A study in play, pleasure and interaction design. Proc. conf. on Designing Pleasurable Products and Interfaces, DPPI 2007, ACM Press (2007), 76-91. 2. Bruegge, B., Teschner, C., Lachenmaier, P., Renzl, El, Schmidt, D., and Bierbaum, S., Pinocchio: conducting a virtual symphony orchestra. Proc. ACE 2007, ACM Press (2007), 294-295. 3. De Kort, Y.A.W., and Ijsselsteijn, W.A. People, places, and play: player experience in a socio-spatial context. Proc. Computers in Entertainment (CIE), ACM Press (2008), Article No. 18. 4. Florida, R. The Rise of the Creative Class: And How It s Transforming Work, Leisure, Community and Everyday Life. Basic Books, New York, 2003. 5. Fullerton, T., Chen, J., Santiago, K., Nelson, E., Diamante, V., Meyers, A., Song, G., and DeWeese, J. That Cloud Game: Dreaming (and Doing) Innovative Game Design. Proc. SIGGRAPH 2006, ACM Press (2006), 51 59. 6. Johnny Chung Lee's Wiimote Projects Including Wiimote DesktopVR, Whiteboard, and Finger Tracking. http://www.wiimoteproject.com/. 7. Lee, H., Kim, H., Gupta G., and Mazalek, A. WiiArts: Creating collaborative art experience with WiiRemote interaction. Proc. Tangible and Embedded Interaction (TEI) 2008, ACM Press (2007), 33-36. 8. Revelle, G., Zuckerman, O., Druin, A., and Bolas, M. Tangible user interfaces for children. Proc. of ACM CHI 2005, ACM Press (2005), 2051-2052. 9. Rick, J., Harris, A., Marshall, P., Fleck, R., Yuill, N. and Rogers, Y. Children designing together on a multi-touch tabletop: An analysis of spatial orientation and user interactions. Proc. Interaction Design and Children (IDC), ACM Press (2009), 106-114. 10. Ryokai, K., Marti, S., and Ishii, H. Designing the World as Your Palette. Proc. CHI 2005, ACM Press (2005), 1037-1049. 11. Ryokai, K., and Cassell, J., Computer Support for Children's Collaborative Fantasy Play and Storytelling. Proc. Computer Supported Collaborative Learning (CSCL), ACM Press (1999), 510-517. 12. Weller, M.P., Do, E.Y.-L., and Gross, M.D. Escape Machine: teaching computational thinking with a tangible state machine game. Proc. Interaction Design and Children (IDC), ACM Press (2008), 282-289. 13. Xie, L., Antle, A. N., and Motamedi, N. Are tangibles more fun? Comparing children's enjoyment and engagement using physical, graphical and tangible user
interfaces. Proc. Tangible and Embedded Interaction (TEI) 2008, ACM Press, (2008), 191-198.