Attention Meter: A Vision-based Input Toolkit for Interaction Designers

Transcription

1 Attention Meter: A Vision-based Input Toolkit for Interaction Designers Chia-Hsun Jackie Lee MIT Media Laboratory 20 Ames ST. E Cambridge, MA USA jackylee@media.mit.edu Ian Jang Graduate Institute of Architecture National Chao Tung Univ. Hsin-Chu, Taiwan yvainj@yahoo.com Ting-Han Daniel Chen Graduate Institute of Architecture National Chao Tung Univ. Hsin-Chu, Taiwan daniel@arch.nctu.edu.tw Yang-Ting Bowbow Shen Information Architecture Lab. National Cheng Kung Univ. Tainan, 700 Taiwan bowbowshen@yahoo.com.tw Jon Wetzel MIT Media Laboratory 20 Ames ST. E Cambridge, MA USA jonwzl@mit.edu Ted Selker MIT Media Laboratory 20 Ames ST. E Cambridge, MA USA selker@media.mit.edu Abstract This paper presents Attention Meter- a vision-based input toolkit for visual artist and designers to develop interactive art installation. This toolkit creates a text interpretation of human body gestures that can trivially be interpreted by any application to read this text file to make interactive use. This toolkit enables multiple head movements and expressions as input methods for Adobe Flash movie. A four day workshop demonstrated that Flash designers with no other user interface experience could competently make interactive spaces (i.e. TaiKer-KTV and ScreamMarket) based on body motion. Keywords Multimodal interaction, visual attention, interactivity ACM Classification Keywords H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. Copyright is held by the author/owner(s). CHI 2006, April 22 27, 2006, Montreal, Canada. ACM 1-xxxxxxxxxxxxxxxxxx. Introduction Visual artworks are often sitting quietly inside galleries or museums. It is important to understand how people react in front of the artwork and provide feedback at the right time. In [1], Krueger presented artificial reality approach for digital art installation to interact

2 2 with visitors. Providing active interactivity to the visitors is important. However, understanding how people visually experience artwork is useful to know what makes people interested in. Since attention is a limited resource [2], artifacts inside galleries or museums are all having their stories. How people perceive the information from the artifacts need to be designed carefully. Visual attention can be tracked and measured by understand the patterns of eye gestures [3]. In [4], attention-based augmentations were deployed as a digitally-switchable domestic environment. Monitoring a group of people s attention and behavior is helpful to provide relevant visual or audio feedbacks depending on how people are engaged. By providing a system of ways to understanding people s intention and reactions, artists or designers will be able to create artworks that can effectively engage people. Attention Meter Painting walking browsing staying Attention level reading engaging Figure 1: We can measure the attention level of people by monitoring their head movement, eye blinking frequency and proximity using computer vision systems This paper presents Attention Meter demonstrating attention-based input methods implemented in VC++. In Figure 1, different levels of attentive engagement (i.e. passing by, glancing, standing and watching, reading carefully, and engaging) can be observed by monitoring pattern of people s behavior using a camera. Figure 2. Attention Meter show the group of attention level as the green bar and reasoning human behaviors from head and eyes movements. The input is a video stream from a camera mounted near or within the target of attention. The camera is positioned such that subjects attending to the target are looking almost directly at it, allowing the attention meter to analyze their attention based on cues from their faces. Face Tracking Each frame taken from the video stream is run through a face detection algorithm from the Intel Open Computer Vision (OpenCV) library, as shown in Figure 2. This algorithm gives us the location and sizes of all faces in the image that are turned towards the display. Tracking faces from frame to frame is accomplished

3 3 simply by assuming that faces move very little from frame to frame, and then matching faces with the nearest coordinates within a small threshold. Nodding, Shaking Heads and Eye Blinking By keeping track of the motion of individual faces from frame to frame, the Attention Meter not only detects when faces move closer or farther, but also recognizes the smaller gestures of nodding and shaking. In addition, by using basic knowledge of the structure of the face and looking for the distinctive brightness gradient of the eye, the Attention Meter can detect eyes in faces, and over several frames measure the face s blink rate. Attention Scores Every face being tracked is given an attention score which varies over time. The score starts at 0 and increases up to some predefined maximum as the face exhibits more attention. The individual scores are then summed together to form a group attention score. Remaining still allows the score to increase, while lateral motion halts it. Nodding and/or shaking, moving closer to the target (becoming larger), and blinking less often (eyes visible more often) will also increase the attention score. The scoring system is developed through experimentation. The user may also change constants affecting the motion recognition, such as the number of times a face must alternate directions to be considered nodding or shaking. Thus users may further customize the Attention Meter to better meet their requirements. Interfacing with Adobe Flash Movie The Attention Meter also outputs a summary of its collected data into a.txt file, which can be read by many other programs, including Adobe Flash. This data (i.e. attentionlevel=30 & face=2 & nodding=1 & shaking=1 & moving=1) includes the group attention score, number of faces, and for each individual face: coordinates, attention score, size (proximity), blink rate, and whether the face is moving laterally, nodding, or shaking. A Design Workshop and User Experience The Attention Meter system was demonstrated as a toolkit for a four-day design workshop. Ten students who have design-related background (i.e. architecture, design, and art) without any formal training of computer science were divided into groups for quick prototyping ideas. Students were given one three-hour lecture with tutorial for understanding immersive and interactive spaces and how to use the tool to integrate visual attention and multimodal interaction. Their assignment is to explore and implement interactive installations in a context of night marketplace in the Taiwanese culture. Two groups of students quickly integrated the Attention Meter system into their proposals, such as ScreamMarket and TaiKer-KTV. First Example: ScreamMarket ScreamMarket transformed the Taiwanese traditional night market experience into a virtual and simulated space. An attention-based triggering mechanism was developed using Attention Meter. The virtual night marketplace image becomes clearer when people pay attention to it. If more people are gathering in front of the installation, the performers show up, as show in Figure 3. Audience has the ability to respond to the installation and get visual feedbacks. People can give off sound captured by the microphone

4 4 to interact with the performer. The higher volume the microphone detects from crowd, the more responding and entertaining actions the performer presents to crowd. Second Example: TaiKer-KTV TaiKer-KTV is trying to enhance the joyful surrounding and social communication in the designed small space. Karaoke (KTV) is very popular in Asia for entertainment and social events. Figure 4. TaiKer characteristics were implemented into this music Flash MTV for people to actively interact with the karaoke environment. Figure 3. The Scream Market presents an animation of two Taiwanese girls if the audiences are paying attention to the stage. While the crowd is interested in the show and scream, the virtual girls will dance and entertain the audience. The process of interaction is similar to the behavior that we watch the interactive show or bargain with the hawker in the night market. According to the method of interaction, the users are not only the visitors, but also become the performers. In addition, the atmosphere of the interaction is a simulation and a recurrence of night market. 30 people had played with the ScreamMarket. It has constrains from the environment noise, so that people need to scream very loudly to interact with the Flash movie. By expressing themselves with own figures and body movements, communication can be encouraged here. For this reason, an interactive KTV system is presented from traditional karaoke context, called Tai-Ker KTV (TKTV), as shown in Figure 4, which exploits headshaking dance to enrich the environmental projection as a way to support group interaction. The purpose is to amplify the group participation phenomenon in KTV and to create an interactive way to enhance the joyful and relaxing atmosphere as well as to enrich the KTV experience with fun. Tai-Ker as Taiwanese-style guest in literal translation is one particular kind of culture on lower civic level to a trend that native Rock stars claiming themselves as. In dancing pool, Tai-Ker always has strong visual images and vivid outfits. They like techno music, patterned design shirt, black suits, white socks on black shoes,

5 5 blue and white slippers, betel nut and screaming while dancing...etc. Shaking and nodding heads along with the beats of the techno, it s also the time when people get high and higher in KTV. In the implementation, a linear but interactive music video was created in Adobe Flash and was projected on a wall. The lyrics go with a nodding head indicator to lead singing and dancing. Video will be kept still and dull if it doesn t get enough attention. It gets fancier whenever people all dance like Tai-Kers. Furthermore, the more people are engaged, the more special visual effects are applied. Some general rules have been defined for the techniques of the music video: If one moves his/her body, the image gets clear. If one nods his/her head, image switches faster. If one shakes his head, the environmental light flashes more dramatically. If there are more people participate, the media elements (i.e. symbols, visual effects, texts and recorded screaming voices) gets mixed. The T-KTV system contains a webcam, a video projector, Flash music video and the Attention Meter system, as show in Figure 5. The webcam is used to collect participants image streaming and project it on the wall for singer-machine-audience interaction. And the contextual data, including number of participants attention, whether they are moving, nodding, or shaking their head can be also retrieved from the streaming. The Attention Meter system is for analyzing the data from webcam and determining the level of movements, especially for Tai-Ker dancing. A typical rave-party anthem- Mei-Fay-Se-Wu by Sammi Cheng is selected as the featured song. A Flash movie was implemented based on the song and get the interaction parameters from the Attention Meter and reflect it as the wall projection and environmental visual effects. This system was completely compelling. Around 100 people came up to it and immediately began making strong movements to make the people on the screen dance. The ease and success at creating an feeling of inhibition in the user was striking. Discussion This experience demonstrates the ability for modern tools to allow visual designers to make innovative arts based on new interface technologies in a very short time. Visual artists and designers usually have limited tools to develop art installations that interact with the audience. ScreamMarket and TaiKer-KTV were built upon the Attention Meter system and were all done in a four-day workshop. These two examples demonstrated value and opportunity of giving visual designers good understandings, without any technological background, they can still succeed to make interactive art installations. The Attention Meter system shows its capability for a single camera to interpret attentive actions and transform the head, face, and eyes movements into computational systems. It also demonstrates the extensibility in interfacing with other software system. In the future, the Attention Meter system will be extended with modulated sensors as a complete toolkit for designers to quickly prototype ideas. This toolkit demonstrates that research grade user interface tools can be put in a form to allow novices to use them in innovative ways.

6 6 Visual graphics Attention Meter Screen Figure 5. Taiwanese Karaoke responses to the atmosphere of the party. Whenever people are nodding or shaking their heads, it enhances the visual experience in the party environment with rotating and blinking lights. Acknowledgements We thank Sheunn-Ren Liou, Mao-Lin Chiu, Sheng-Fen Chien for their comments and Kristy Liao, Scottie, Dang from the Asian Reality workshop 2005 in Taiwan. Reference [1] Krueger, M., Gionfriddo, T., Hinrichsen, K., Videoplace- An Artificial Reality, Proc. Of CHI 85, ACM Press (1985), [2] Pashler, H., The Psychology of Attention. Bradford Books. Reprint edition, 1999 [3] Selker, T., Visual Attentive Interfaces. In BT Technology Journal, Vol 22 No 4, October (2004), [4] Bonanni, L., Lee, C.H., Selker, T., Attention-Based Design of Augmented Reality Interfaces, Ext. Abstracts CHI 2005, ACM Press (2005)