Authors: Bill Tomlinson, Man Lok Yau, Jessica O Connell, Ksatria Williams, So Yamaoka

Transcription

1 9/10/04 Dear Sir/Madam: We would like to submit an interactive installation to the CHI 2005 Interactivity program. Authors: Bill Tomlinson, Man Lok Yau, Jessica O Connell, Ksatria Williams, So Yamaoka Contact information: Bill Tomlinson 430A Computer Science Building University of California, Irvine Irvine, CA Tel: Cell: Fax: wmt@uci.edu Title of Submission: The Protohuman Project Category of Submission: Creative and Artistic Innovation Thank you for your consideration! Best regards, Bill Tomlinson and the Social Code Group at UCI

2 The Protohuman Project: A Mobile Interface for Interacting with Communities of Autonomous Characters Bill Tomlinson, Man Lok Yau, Jessica O Connell, Ksatria Williams, So Yamaoka ACE (Arts Computation Engineering) program University of California, Irvine 430A Computer Science Building / Irvine, CA wmt@uci.edu ABSTRACT This paper presents a novel and intuitive paradigm for interacting with autonomous animated characters. This paradigm utilizes a mobile device to allow people to transport characters among different virtual environments. The central metaphor in this paradigm is that virtual space is like land and real space is like water for virtual characters. The tangible interface described here serves as a virtual raft with which people may carry characters across a sea of real space from one virtual island to another. By increasing participants physical engagement with the autonomous characters, this interaction paradigm contributes to the believability of those characters. Author Keywords Tangible interfaces, Intuitive interfaces, Mobile devices, autonomous characters, interactive animation ACM Classification Keywords H5.2. Information interfaces and presentation (e.g., HCI): User Interfaces; I.3.7 Three Dimensional Graphics and Realism: Animation INTRODUCTION The Protohuman Project is a multidisciplinary undertaking seeking to create communities of believable autonomous characters (Fig. 1) that inhabit heterogeneous networks of computational devices. In particular, the project is interested in allowing the characters to break the plane of the traditional desktop screen. To this end, we have designed an interactive installation featuring a novel tangible paradigm for interacting with the characters. This paradigm involves the use of a mobile device, such as a Tablet PC or handheld computer, as a virtual raft (Fig. 2) by which a character may be transported among several Figure 1: A community of autonomous characters gathers around a camp fire in a screen-based virtual island. virtual worlds. By enabling the character on the raft to react in real time to the raft s motion in real space, this installation encourages participants to become physically engaged with virtual characters. We believe that this physical engagement can lead to an increase in the believability of the characters. Figure 2: These characters may jump from a desktop screen onto a mobile device with a "virtual raft" on it and be carried around in physical space. 1

3 INTERACTION In the interactive installation, each of three fixed computer screens displays a virtual environment inhabited by a small community of animated characters. These characters exhibit simple autonomous behavior such as gathering around a central campfire or approaching people who come up to their screen. The three screens serve as islands the members of one community roam freely on their own screen. The islands are separated by a gulf of real space that is not traversable by synthetic agents without assistance. The primary interaction for human participants is to help the characters migrate among the three islands. People can do this by means of a virtual raft a Tablet PC that can be moved in physical space by a human interactor. When this virtual raft is brought near one of the islands, a character may jump off the island onto the raft (Fig. 3). a form of tangible interface [3], allowing a participant to physically transport a virtual character. The interface is also inspired by toys such as Tamagotchi [1], where virtual characters are embodied on handheld devices, and computer games such as Animal Crossing [5], where characters may be transported from one virtual environment to another on a non-interactive memory stick. The project described here is unique from these predecessors in that it uses 3D graphics, accelerometer data and high-speed networking to create believable characters on the mobile device itself. The characters in this installation use simple machine vision [2] to detect humans and the virtual raft when they approach their island. The characters autonomous behavior is derived from the work of Bruce Blumberg s Synthetic Characters Group at the MIT Media Lab [4]. The system is written in Java and uses JOGL for the graphics [6]. INNOVATION AND RELEVANCE A great deal of research has contributed to the creation of believable animated characters. Researchers have made characters look more believable with lifelike hair, skin, clothes etc., sound more believable through expressive speech synthesis, and act more believable through artificial intelligence and autonomous agent techniques. The goal of the project described here is to create an interface to autonomous characters that will contribute to their believability as well. By allowing the character to jump instantaneously from desktop to mobile device and to animate similarly on both sides of the jump, the system maintains the illusion of believability across these heterogeneous devices. Figure 3: When the raft comes close to an island, characters may jump on or off it. Once the character jumps on, it then needs to balance on the raft. The raft contains accelerometers that detect the motion of the raft in two axes. The participant may give the character a smooth ride by carrying the raft carefully, or may simulate stormy seas by tilting and swaying the raft. The smoothness of the ride on the raft affects the emotional state of the characters, which is expressed through the style of their animation. When the person brings the raft up to another island, the character may then jump off. Through this mechanism, the person may determine which characters should inhabit each of the different islands. Once a character has been transported to another island, it begins to interact with the other individuals on that island. Currently these interactions are very simple; a goal of the project to date has been to create an interactive platform through which researchers may explore autonomous agent communication and learning as individuals are introduced to and removed from different communities. TECHNOLOGY The technology in this installation builds on a variety of previous research projects. The core interface technology is ACKNOWLEDGMENTS We thank the ACE (Arts Computation Engineering) program, the Donald Bren School of Information & Computer Science and the Claire Trevor School of the Arts at UCI and Cal-(IT)2 for supporting this project. REFERENCES 1. Bandai, "Tamagotchi." Japan, S.-C. Cheung and C. Kamath, "Robust techniques for background subtraction in urban traffic video," Video Communications and Image Processing, SPIE Electronic Imaging, San Jose, H. Ishii and B. Ullmer, "Tangible bits: towards seamless interfaces between people, bits and atoms," in Proceedings of the SIGCHI conference on Human factors in computing systems: ACM Press, 1997, pp D. Isla, R. Burke, M. Downie, and B. Blumberg, "A Layered Brain Architecture for Synthetic Creatures," Proceedings of the International Joint Conferences on Artificial Intelligence (IJCAI), Seattle, WA, Nintendo, "Animal Crossing." Japan, D. Twilleager, J. Kesselman, A. Goldberg, D. Petersen, J. C. Soto, and C. Melissinos, "Java technologies for games," Comput. Entertain., vol. 2, pp ,

4 The Protohuman Project Bill Tomlinson, Man Lok Yau, Jessica O Connell, Ksatria Williams, So Yamaoka ACE (Arts Computation Engineering) program University of California, Irvine 430A Computer Science Building Irvine, CA wmt@uci.edu DESCRIPTION OF THE WORK The Protohuman Project is an interactive installation in which human participants interact with several communities of autonomous animated characters. These protohumans live on three virtual islands one on each of three computer screens arranged around the installation space (see Figure 1). The protohumans have a variety of autonomous behaviors. When no real people are near their screen-based island, they gather around the camp fire and drum together (see Figure 2). When one or more human participants walk in front of their island, the characters stop interacting with each other and will instead approach the people (see Figure 3). In addition to interacting with the protohumans directly through their computer vision systems, participants are able to carry a mobile device, which represents a virtual raft, around the installation space. The protohumans are not able to migrate among the three islands without human assistance. When a participant brings the virtual raft near one of the islands, a protohuman from that island may jump onto it (see Figure 4). While the character is on the raft, it has to try to balance and keep from falling over when the participant tilts the mobile device (see Figures 5, 6 and 7). The participant can then carry the character to a different island. Each island has its own fire color (see Figure 8); the crowns of the protohumans correspond to the fire on their birth island. When the raft is close enough to this other island, the character can jump off (see Figure 9). It will now begin to interact with the other characters already residing on that island (see Figure 10). Figure 1: A schematic of the installation space. Three virtual islands run on the three monitors, and a human participant carries a mobile device the virtual raft. In the Chamber at CHI 2005, this installation will be presented in its fully functional form so that attendees are able to interact with it. The installation is sufficiently robust that it will not require the creators to supervise it. Rather, student volunteers will simply need to switch Figure 2: A community of autonomous animated protohumans gathers around a camp fire on one of the screen-based islands.

5 the battery in the mobile device every few hours, and insure that the device is not removed from the installation space. WHY IT IS APPROPRIATE The goal of this project is to demonstrate that the interface between people and autonomous characters can have a significant impact on the believability of the characters. Live demonstrations in the Chamber will fully exhibit the interactivity and believability in a way that would otherwise be difficult to show in a normal presentation environment. DESCRIPTION OF SYSTEM This system presents a novel and intuitive paradigm for interacting with autonomous animated characters. This paradigm utilizes a mobile device to allow people to transport characters among different virtual environments. The central metaphor in this paradigm is that virtual space is like land and real space is like water for virtual characters. The tangible interface serves as a virtual raft with which people may carry characters across a sea of real space from one virtual island to another. By increasing participants physical engagement with the autonomous characters, this interaction paradigm contributes to the believability of those characters. PROBLEM IT ADDRESSES A great deal of research has contributed to the creation of believable animated characters. Researchers have made characters look more believable with lifelike hair, skin, clothes etc., sound more believable through expressive speech synthesis, and act more believable through artificial intelligence and autonomous agent techniques. The goal of the project described here is to create an interface to autonomous characters that will contribute to their believability as well. By allowing the character to jump instantaneously from desktop to mobile device and to animate similarly on both sides of the jump, the system maintains the illusion of believability across these heterogeneous devices. Figure 3: When participants walk up to an island, the protohumans approach them. Figure 4: When the raft is brought near one of the islands, a protohuman may jump onto it. Figure 5: The character then needs to balance on the raft.

6 RELEVANCE TO CHI COMMUNITY This installation presents a novel interaction paradigm that is relevant to the CHI community for several reasons. The paradigm involves a tangible interface, through which participants may become physically engaged with the characters. The interaction paradigm is intuitive, drawing on participants understanding of the land/water dichotomy, a central theme in many different cultures. The project demonstrates that an interface may have a substantial impact on the believability of autonomous characters, making them seem more real because they share an awareness of the same physical space that people inhabit. This project draws inspiration from a range of previous research in human computer interaction, in particular regarding tangible interfaces and mobile devices. This project is unique from its predecessors in that it uses 3D graphics, accelerometer data and high-speed networking to create believable characters on the mobile device itself. COMMERCIAL STATUS This installation is an academic prototype, and has not been commercialized. It was developed with the support of the ACE (Arts Computation Engineering) program, the Donald Bren School of Information & Computer Science and the Claire Trevor School of the Arts at the University of California, Irvine. EQUIPMENT/SUPPORT FOR INSTALLATION This installation runs on the following equipment: 3 desktop PCs (Windows 2000/XP, >=2.4GHz, >=512MB RAM, OpenGL compatible graphics card with >= 64MB video RAM, speakers) 3 flat panel displays (>=18 ) or projectors and screens (rear projection only) 3 USB webcams (with 3 USB extension cables, at least 6 feet each) 1 Toshiba M200 Tablet PC Wired network to the three desktops & wireless network to the Tablet PC Assorted cables, keyboards, mice, etc. Figure 6: A participant tilts the virtual raft. Figure 7: A screenshot of a protohuman balancing on the raft. Figure 8: The camp fire on each island is a different color red, green and blue.

7 The authors will provide the Tablet PC and webcams. We request that the conference provide the three desktop machines, monitors and networking equipment. If this equipment request presents a significant problem, please contact Bill Tomlinson (wmt@uci.edu) to discuss other options. DESCRIPTION OF PRESENTERS The presentation will be delivered by the first and second authors of the project Bill Tomlinson and Man Lok Yau. Bill Tomlinson is an Assistant Professor of Informatics and Drama at the University of California, Irvine, where he teaches in the ACE (Arts Computation Engineering) graduate program. He is a researcher and animator of autonomous computational characters. Previous interactive projects have been shown at SIGGRAPH (97, 98, 99, 01, 02), Ars Electronica (02), the ZKM Future Cinema exhibition (02-03) and other venues, and have been reviewed by CNN, the Wall Street Journal, Sculpture Magazine, Scientific American Frontiers, the LA Times, Wired.com and the BBC. In addition his animated film, Shaft of Light, screened at the Sundance Film Festival and was distributed by the Anti-Defamation League in its AntiBias/Diversity Catalog. He holds an A.B. in Biology from Harvard College, an M.F.A. in Experimental Animation from CalArts, and S.M. and Ph.D. degrees from the MIT Media Lab. Figure 9: When the raft is close enough, the protohuman jumps off of it onto the blue island. Figure 10: The protohuman from the red island now begins to interact with the residents of the blue island. Man Lok (Simon) Yau is a Computer Science undergraduate at the University of California, Irvine. His research interests are artificial intelligence and human-computer interaction. He is a fellow of the Summer Undergraduate Research Program (SURP) at UC Irvine. EQUIPMENT/SUPPORT FOR PRESENTATION The presenters will provide their own notebook computers. We request that the conference provide a projector and external speakers that may be hooked up to the notebooks.

8 The Protohuman Project: Presentation Outline Bill Tomlinson, Man Lok Yau, Jessica O Connell, Ksatria Williams, So Yamaoka ACE (Arts Computation Engineering) program University of California, Irvine 430A Computer Science Building / Irvine, CA wmt@uci.edu 1. Overview of the Work Presenter: Tomlinson Length: 6 minutes 1.1. Description of Installation 1.2. Land/Water Metaphor 1.3. Relevance to CHI community 2. Video Length: 3 minutes 3. Technical Details 3.1. How the Virtual Raft Works Presenter: Tomlinson Length: 3 minutes Accelerometers Animation Water effect 3.2. How the Jump from Desktop to Mobile Device Works Presenter: Yau Length: 3 minutes Networking Animation Timing 4. Demonstration of Character Balancing on Raft Length: 5 minutes Presenters: Tomlinson & Yau 5. Question & Answer Length: 10 minutes Presenters: Tomlinson & Yau