Experiences of Research on Vision Based Interfaces at the MIT Media Lab

Transcription

1 HELSINKI UNIVERSITY OF TECHNOLOGY Telecommunications Software and Multimedia Laboratory Tik Seminar on content creation Autumn 2003: Aspects of Interactivity Experiences of Research on Vision Based Interfaces at the MIT Media Lab Johnny Biström 21548C

2 Experiences of Research on Vision Based Interfaces at the MIT Media Lab Johnny Biström HUT, Telecommunications Software and Multimedia Laboratory Abstract This paper describes some interesting research projects conducted at the MIT Media Lab. The paper starts with an analysis of the need of visual interfaces that do not limit the movements of the body but still permit advanced communication in what is called perceptive spaces. The paper continues with the description of some of the most promising technologies proposed, tested and evaluated for the purposes mentioned above. A number of innovative applications depending on the vision based technologies are next presented. Finally the results of the experiments are evaluated and some guidelines for future investigations are presented. 1 INTRODUCTION Computer based interactive art and entertainment applications have always fascinated the more expressive programmers. The ability to track the movements and sounds of the human body and to let them produce multimedia based content or interaction with virtual environments have been the goal of several research projects at the MIT Media Laboratory during the last decade. The focus have been set on analyzing body movements and sounds and from those extract control signals to produce interesting audio, light, graphics, animations or video experiences for the actor and the viewers. The keyboard and the mouse are generally considered as insufficient input devices for artistic computer applications. The expressiveness of the human body is much greater and thus the interest in reading and analyzing the actions of the human body has increased. Computer applications that rely on interaction with the human body have often depended on equipment that the user wears. Examples of such equipment are data gloves and special movement detecting suits. These are often very expensive, difficult to put on and limit the movement of the human being trying to express herself. Dancers and children that use their bodies to express deep feelings often see these equipments impossible to use. Lately the research projects have been concentrating on creating body movement information from video camera input by analyzing and processing the digital image stream. Thus the movement of head, hands, feet and torso can be identified and used as control signals for different multimedia or used for interaction in storytelling applications. The MIT Media Laboratory has conducted several research projects and 1

3 studies where different interface technologies for visual interfaces have been evaluated. The technologies have generally been tested on visual applications that require the user input as body movements and sounds. This report describes, compares and evaluates some of the technologies and applications that have been tested at the MIT Media Laboratory and gives some guidelines for further research in this area. 2 VISUAL INTERFACE TECHNOLOGIES There are a number of interesting visual technologies that can be used for detecting body movements. In this chapter some of the most promising technologies are reported from the simplest use of one camera with color vision approach to stereo vision and multiple cameras. In cases where the lightning changes or when projected background is used the solution with infrared cameras give better result. Finally solutions with IR-emitters are analyzed. 2.1 Color Vision with One Frontal Camera The simplest way for an application to communicate with the user is the one camera color vision concept. The color is needed to be able to separate the parts of the human body from the background. The camera is installed in the front of the viewer. The needed equipment is cheap to buy and can be installed on any personal computer. Disadvantages are that it has high requirements on unchanged background and unchanged lightning. This concept can not be used when a background or floor projection is used. Another limitation is that only one person can be identified at a time. A typical one camera interactive space is presented by Wren et al. (1999, p.2) in figure 1. Figure 1: An Interactive Virtual Environment with One Frontal Color Camera The MIT Media Laboratory has developed a one camera system called Person Finder (Pfinder). Sparacino (2001, p.3) describes that the system uses a multi-class statistical model of color and shape to segment a person from a background scene and then track 2

4 the body parts and their movements. Pfinder first builds the scene model by observing the scene without a person. When a person enters the scene Pfinder constructs a multiblob model of the person s body based on the colors in the image. Blobs are generated for the head, hands, feet, shirt and pants. The process is driven by a 2D contour shape analysis. The performance of the Pfinder system is reported by Wren et al. (1999, p.3) and the results are shown in table 1. Table 1: Pfinder Estimation Performance This environment has been tested on thousands of persons in installations all over the world and it has proved to be very functional and stable. An application called DanceSpace has been developed based on this concept. DanceSpace tracks the movements of dancer and generates graphics and music based on the dancer movements. DanceSpace will be described later in this paper. 2.2 Multi Camera Color Vision Environment To make interactive stories where the can move around in the room and use furniture or appliances for different purposes the one camera approach is not sufficient as it cannot detect the user movements in the axis perpendicular to the projection screen. There might also be positions in the room where the furniture hides the user from the camera. For this purpose additional cameras can be used and they can be placed over head to track the movements of the user or users. Such an interactive space was used in the development of the applications KidsRoom at MIT. The idea of KidsRoom will be described in detail later. In this case no frontal camera was used but all three cameras were placed in the ceiling. There is however a possibility to use a frontal camera here as well if the background and lightning conditions can be kept constant. The several camera room built for KidsRoom is described in Pinhanez et al (2000, p. 441) in figure 2. Two cameras are used to track the gestures (recognition camera) of the users while one camera (overhead camera) is used to track the position of the users. This construction enables the users to move around the room and to fulfill different tasks. 3

5 Figure 2: An Interactive Space with Several Cameras The KidsRoom has been experienced by thousands of children and adults in London as part of the Millennium Dome Project. The solution has proved to be functional and reliable. 2.3 Stereo vision To improve the abilities of the tracking system and to make more complex applications where the user can interact in three dimensions with the application stereo vision, based on two or even three cameras, have been developed. Stereo vision makes it possible to produce 3-D blobs instead of the 2-D blobs described earlier. By using cameras which are much more apart than the human eyes a three dimensional model of the user can be produced. The stereo pair described by Wren et al. (1999, p.4) in figure 3 shows the configuration in which the research at MIT was done. By using two cameras a 3-D estimate of the user s body can be constructed and a blob produced. The accuracy of the estimation in the 3-D case is not as good as in the 2-D case because the estimation along the z axis is not so well conditioned mathematically. This is a result of the positioning of the cameras. The measured performance for the 3-D estimate is shown in table 2. 4

6 Figure 3: 3-D Estimation of the Position of One and a 3-D Blob of another User Table 2: Stereo Estimation Performance, Wren et al. (1999, p.4) Stereo vision has also been used in other projects at MIT. A special solution with three cameras in front of the scene was used in the It/I project described by Pinhanez et al (2000, p. 444). It/I is a computer play with two characters. One real actor plays himself while the other actor is played by the computer and has a non-human appearance which is composed of computer graphics (CG) representing technology in the form of clocks, cameras, televisions and switches e.g. The ideas behind It/I will not be described in this paper. Readers interested in It/I are referred to Pinhanez and Bobick (1998). The technical reason for having three cameras instead of two is that a background screen is needed for the play and it can be used for the viewers and then be eliminated from the process of identifying the movements of the user. The stage used for It/I is shown in figure 4. 5

7 Figure 4: Physical Setup of It/I with tree cameras. Pinhanez et al (2000, p. 444) 2.4 Infrared Sensitive Cameras and Infrared Light The problem of having varying lightning conditions and the need of projecting behind or below the camera can be solved by using infrared sensitive cameras instead of normal cameras. The infrared sensitive cameras used produce a black and white image and they are insensitive for projections made by data projectors. The lack of color information on the other hand makes it impossible to identify different body parts and to produce accurate blobs from these. Only the body silhouette can be observed which makes it difficult to identify small movements of arms, hands, head or feet. The user thus has to use fairly large gestures or body movements to interact with the computer. Another advantage is that the equipment needed is cheap and it can be connected to any personal computer. An application called City of News at MIT takes advantage of the infrared camera environment as it projects a map on the floor where the user stands. Infrared cameras observe the user from the front and from above. City of News is an interactive environment where the user can explore a virtual world, move and trig actions based on movements. The concept of City of News will be penetrated later in this paper. 6

8 If the need of accuracy is grater than what can be achieved with infrared sensitive cameras an array of infrared light emitters and a camera can be used. This was done to realize the Personal Aerobic Trainer (PAT) where a projected screen was needed both in front of and at the back of the user to make the user interface convenient. The array of IR emitters generates a IR floodlight from which the silhouette of the user can be identified. The arrangement can be seen in figure 5. Figure 5: IR Emitters behind Projected Screen. Pinhanez et al (2000, p. 447) The PAT application, which led to the development of the IR-emitter grid, needed greater accuracy to be able to identify the exact position of the user in order to correct errors in the pose or the aerobic movement the user exercised. The PAT application will be described later in this paper. Developing of a full 3D perceptive space or room would need the projection of all four walls, the floor and the ceiling. This approach is a challenge for visual identification. Normal color cameras can not used because of the changes in lightning and background. The infrared vision cameras used so far have been black and white cameras. Today color infrared cameras are available at a reasonable price. These color IR-cameras definitely have a future in visual identification as different levels in the form of colors can be identified. We must however remember that the coloring of infrared pictures is based on temperature. Identifying hands and head can benefit from this as they normally are warmer than clothes. There is however no guarantee that a persons arms, body or pants are identified as the same color. Future research should be put on the possibilities of color IR identification. 7

9 3 APPLICATIONS THAT USE VISUAL INTERFACE TECHNOLOGIES There are a number of research projects at MIT Media Laboratories that use the visual interfaces mentioned above. Only a few of them will be presented here. The purpose here is to show what kind of stories, applications and environments the technologies can be used for. The examples below address different audiences and have different type of stories and interaction. 3.1 Kids Room The KidsRoom is a multi-user experience in a fantasy world where children in the age of 6 to 12 years can interact with computer produced monsters and take part in an adventure play which starts and ends in a bedroom. The children are taken to a mystical forest, on a boat ride on a river and finally to the monster world. To advance in the story the children have to perform tasks as shouting, walking, running, hiding, paddling and dancing. The whole adventure takes about 12 minutes if the children follow the narrator s advices. The children have no control of the overall story development; they just interact in the scenes and perform the given tasks as smoothly as possible. The detailed story is described by Bobick et al (1999). Figure 5: Users Experiencing the KidsRoom. Pinhanez et al (2000, p. 443) KidsRoom takes place in the multi camera color vision environment described above. This environment is needed as the children are supposed to move around the room and their tasks are observed as they do them in different physical locations. As the environment is a closed rectangular room of 8 * 6 meters it is simple to keep the lightning conditions constant and projection is done only on the two front walls. It is however not possible to dim the lights in the mystical forest to achieve greater excitement or to make a projected path for children to follow on the floor. 8

10 3.2 Personal Aerobic Trainer An application created mainly for the adult audience is the Personal Aerobic Trainer (PAT). PAT is a virtual aerobic trainer that lets the user select which aerobic moves, which music and which instructor personality the user prefers and then guides the user through a workout while the trainer supervises and motivates the user. The application can be compared with a workout video but it has real interactivity in the same way as it would be to workout in a gym with a personal trainer. The application is described in detail by Davis and Bobick (1998). Figure 6: Interaction with PAT. Pinhanez et al (2000, p. 449) The first version of this application was based on a camera below the TV-set configuration as shown in figure 6. This was a product that aimed for the commercial market using only one cheap color camera. Further development of PAT at MIT led to a configuration with one projected screen in front of the user and one behind the user as described earlier in this paper. To realize this configuration some changes in the principles of the space had to be done. This application has special demands on the visual interface. The camera must exactly identify how the user performs the movements shown by the PAT. The large scale movements are easy to interpret but the ones which are more static or require only small movements are more complex to detect. Still these static and small movements are necessary to exercise in the right way to ensure a proper workout. The infrared camera used, because of the projected background, was not accurate for the purpose so an infrared emitter grid was developed and placed behind the user to improve the interpretation of the user movements. This setup is however not usable for home users but might be an alternative for gym users. 9

11 3.3 DanceSpace Several efforts have been made in the last decades to create interactive stages to produce graphical and musical effects from the movements of the human body. By identifying a dancer s hands, head, feet and torso and following the movements of these graphical and acoustic effects can be realized. DanceSpace was designed to visualize the movements of a dancer and to map the movements to music and sound. The movements of the parts of the body leave a multicolored trail. This is realized by drawing two Bezier curves to represent the dancer s body. The first curve is drawn according to the placement of the left foot, head and right foot. The second curve is drawn from the position of the left hand, center of torso and right hand. In this way a shadow of the dancer can be produced. The duration of the shadow can be prolonged and as time flows the color of the shadow changes producing a multicolored trail of the dancer. (Sparacino (2001, p.3)) Figure 7: User Dancing with Generated Shadow. Wren et al. (1999, pp.1 and 9) The equipment used for this application is the Pfinder, single frontal color camera solution, described earlier in this paper. Using this equipment it is naturally possible to realize any visualization the programmer is able to calculate. 10

12 3.4 City of News This application is actually an immersive, interactive web browser that enables the user to use his capabilities in 3D spatial layout. This means that most people have a capability to remember where things a situated in their homes. This ability is not used as people browse the Internet. City of News connects the addresses in the web to physical representations in a virtual 3D world. The user starts from a freely chosen home page which has links to other websites. This home page is converted to a city where each link is represented by a building. This building is identified by pictures or text from the webpage. As the user navigates his way through different links new buildings are raised in the city. Streets and alleys separate the buildings and the user can move in the streets and find his way to the buildings and to the sites they are linked to. The 3D city will form a virtual reality of the links visited and thus the user can use his ability to navigate in city to create a 3D representation of the websites visited. The mapping of the buildings takes place according to certain simple rules that associate the links to areas of the city. Each time a new building is raised the user is moved in front of that building. City of News is described in detail in Sparacino et al (1999). The navigation in the city takes place using previously specified gestures and sounds for different actions. The user can move around the city by raising left or right arm to point straight out. Raising both arms straight up lifts the user from the plane and makes it possible to observe the city from above to get a total picture of the city. By pointing at a building and saying there the user opens a new webpage. He can scroll down the page by pointing down and scroll up the page by pointing up. Every needed action has a corresponding gesture. Figure 8: User Navigating in the City of News. Wren et al. (1999, p.7) The user can navigate in the City of News by sitting at a desktop, as shown in figure 8, or by standing in a room with projection in the front wall and the floor. The equipment used for identifying the user gestures is one color camera in front of the user. 11

13 3.5 SURVIVE SURVIVE (Simulated Urban Recreational Violence Interactive Virtual Environment) is an application where the users movements, detected by color vision cameras, are mapped directly to the game controls of the popular shooting game Doom. The user stands in a perceptive space in front of the screen holding a large toy gun. The Doom game is played by moving, turning, looking up or down and pointing with the gun instead of using the keyboard controls or a game pad in a virtual environment. Some instructions are given with voice commands as the change of weapon and the firing of the weapon chosen. The user moves around the 3D worlds and tries to kill as many enemies as possible. Figure 9: Playing Doom with SURVIVE. Wren et al. (1999, p.5) The user interface in much more intuitive than using the keyboard and it forgives small mistakes better than the finger based keyboard. The risk of making erroneous moves is also reduced. The game is more immersive and physically demanding than the original keyboard controlled one according to Wren et al. (1999, p.5) SURVIVE has also been played as a multiplayer game between MIT and the British Telecom's Research Center in the U.K. 12

14 4 CONCLUSIONS The above illustrated examples show that perceptive spaces, interactive virtual environments or smart rooms as they are called today are challenging environments with innumerous possibilities in realizing visual communication between computer and the human being. They have been proved to be more intuitive and more immersive giving a intense, artistic or realistic possibility of interaction with virtual worlds. All of the projects above have been very successful and they have been presented at conferences, exhibitions and museums all over the world. Many of the projects have been developed during the years from the first prototypes produced. The equipment used for the projects are still in research and demonstrational use. The projects have however still been left on this demonstrational stage and no noteworthy commercial products based on this projects have been released on the market. It seems as the breakthrough of perceptive spaces are still to come. The technologies on infrared color cameras seem to be a promising new area in the detection of the gestures of the human body. One can extrapolate that these cameras also can be used as stereo cameras giving possibilities to produce 3D models of the person observed. In the near future it will be possible to map the gestures of a person to control 3D avatars in virtual 3D environments in real time. The possibility to fully understand, interpret and model the actions of a human body is a very complex task which leads to a very large number of possible states according to Pinhanez (1999). We are far from that goal at the moment. We must however remember that the screen, keyboard and mouse used for interaction with the user today easily, with the use of cameras, can be extended in the direction towards better understanding of the intensions of the human being. REFERENCES Bobick, A.; Intille, S.; Davis, J.; Baird, F.; Pinhanez, C.; Campbell, Y.; Ivanov, Y.; Shutte, A. and Wilson, A The KidsRoom: A Perceptually-Based Interactive Immersive Story Environment. PRESENCE: Teleoperators and Virtual Environments 8, n. 4. Cambridge, MA, USA. Davis, J. and Bobick, A Virtual PAT: A Virtual Personal Aerobics Trainer. Proceedings of Workshop on Perceptual User Interfaces, San Francisco, CA, USA. Pinhanez, C. S.; Davis, J. W. ; Intille, S. ; Johnson, M. P. ; Wilson, A. D. ; Bobick, A. F.; Blumberg, B Physically Interactive Story Environments. IBM Systems Journal. Vol. 39. Nos. 3&4. IBM. Pinhanez, C. S Representation and Recognition of Action in Interactive Spaces, Doctor Thesis. Media Arts and Sciences, Massachusetts Institute of Technology, MA, USA Pinhanez, C. S. and Bobick, A. F It/I : A Teater Play Featuring an Autonomous Computer Graphics Character. Proceedings of the ACM Multimedia 98 Workshop on Technologies for Interactive Movies. ACM. New York. USA 13

15 Sparacino, Flavia (Some) computer vision based interfaces for interactive art and entertainment installations. INTER_FACE Body Boundaries, issue editor Emanuele Quintz, Anomalie, n.2. Paris, France. Anomos. Sparacino, F.; DeVaul, R.; Wren, C.; MacNeil, R.; Davenport, G.; Pentland, A (1999). City of News. SIGGRAPH 99, Visual Proceedings, Emerging Technologies, Los Angeles, USA Wren, C. R.; Sparacino, F. ; Azarbayejani, A. J.; Darrell, T. J.; Davis, J. W.; Starner, T. E.; Kotani, A.; Chao, C. M.; Hlavac, M.; Russell, K. B.; Bobick, A.; Pentland, A. P Perceptive Spaces for Performance and Entertainment. Perceptual Computing Section. The MIT Media Laboratory. Cambridge, MA, USA. WEBREFERENCES MIT Media: Interactive Cinema at MIT Media: Vision and Modeling at MIT:Media: Smart Rooms: Bobick, A.: Pinhanez, C.: Sparacino, F.: Wren, C.: Sensing Places: 14