Skeleton Video that Allows Users to Interact with Images

Transcription

1 Skeleton Video that Allows Users to Interact with Images Osamu Morikawa*1, Yoko Sakata*2, and Takanori Maesako*3 National Institute of Advanced Industrial Science and Technology(AIST) *1 Faculty of Communication, Aichi Shukutoku University*2 Faculty of Human Sciences, Osaka University*3 *1 *2 *3 Abstract A skeleton video is a new type of video that provides a framework and outline (skeleton) of the video contents and allows viewers to modify the video themselves in such a way that individual viewers create different adaptations of the video by viewers participation as subjects. We produce skeleton video teaching materials for young children (five years of age), conduct demonstrations, and examine the effectiveness of the skeleton video. 1. Introduction Electronic media, including TV, video devices, and PCs, have recently come to be commonplace throughout daily life. Electronic media are used as learning tools in education sites, such as schools and kindergartens, and introduce major changes in the forms of instruction and learning. Information is dispersed to students unidirectionally through learning using TV or video. For that reason, there are few interaction elements in which information is transmitted through experience, and which enable the students to solve problems independently. However, interactions [1] and experiences [2][3] are important methods for young children in particular to reach their solutions. We propose a new type of video called a skeleton video in this report that allows interactions between participants and images and examine its effectiveness through the production and demonstration of skeleton video teaching materials. Section 2 describes the skeleton video, Section 3 cites related studies, Section 4 reports on the demonstration experiments, and Section 5 discusses the target age and consistency of the video images. 2. Skeleton Video A skeleton video is a new type of video that provides the framework and outline (skeleton) of the video contents and allow viewers to modify the video themselves as a subject in such a way that individual viewers create different adaptations of the video. Images in conventional videos represent the finished work of the producers; further modifications of the video contents cannot be performed. In contrast, a skeleton video is produced from the initial stage under the assumption of modification by viewers. A skeleton video has a structure similar to that of karaoke. A karaoke song becomes a finished musical creation only after it is combined with a human voice. Karaoke provides new ways to enjoy music through the enjoyment of singing songs, learning songs, and creating music through collaboration, as well as the enjoyment of listening to the final musical creation. Video Mixer Figure 1. Skeleton video where viewers participate as subjects by overlapped his/her image into the recorded video. A skeleton video becomes complete when the viewer is included in the image of his or her friends, who then travel to scenic areas and places of historical interest as if they were together (Fig.1,2). A complete image is created by incorporating the viewer into the image, for example, as if he or she were dancing with performers on a stage. A skeleton video provides a new way to enjoy video beyond simply viewing images, by allowing viewers to

2 participate in the process of completing the video contents. A: skeleton video B: viewer X C: viewer Y D: bad example Figure 2. Playing of skeleton video. As B and C are natural image, viewers feel as if they are there. In D, balance is bad, so she feels unconformable. 3. Related Studies Studies that enable viewers to become subjects in the image and synthesize images to achieve interaction, a sense of co-existence, and impression of integration with the virtual reality of the image include HyperMirror [4] by Morikawa, et al., Reflection of Presence [5] by Stefan et al., and the See-through Video Avatars system [6] by Yasuda et al. These are remote interaction systems, in which viewers interact by creating an image of their behavior as observed from an overhead point of view. This image is synthesized, and thus the viewer can interact by pointing out objects in their partner's space. The sense of being at the site is obtained in HyperMirror by creating an image of the viewer and partner as if they were in the same virtual room. Reflection of Presence and the See-through Video Avatars system provide virtual space for interactions that cannot exist in reality by changing the display sizes, display positions, and transparency of the interaction partner. The viewer is displayed as an video image in ALIVE [7] by Maes et al., while the partner interacts with an avatar in the computer. In contrast, the partner in VirtualActor [8] by Ishii et al. is displayed as a video image while the viewer interacts using an avatar. Both the viewer and partner become avatars in a virtual space that cannot exist in reality and interactions take place by moving avatars using a mouse or other device in DIVE [9] by Fahlen et al., InterSpace [10] by Sugawara et al., MASSIVE [11] by Greenhalgh et al., and GreenSpace [12] by Mandeville et al. 4. Experiments We anticipate that using a skeleton video will change learning through viewing only into learning through a video in which the viewer interacts with the image using his or her whole body. We produced skeleton video teaching material for young children (five years of age) and performed experiments using it. It must be noted that what viewers interact with is not recoded video material, but synthesized images with viewers. The recoded video material itself does not synchronize with viewers' action. The synthesized images with viewers, however, varies following to the viewers themselves. The experiments enable us to examine whether learning using a skeleton video is a joyful experience for young children. Learning through play is known to be effective in childhood [13]. It is important to have fun during learning. We believe that it is necessary to first examine whether the proposed form of learning is fun and easily accepted by children before we focus on the effects of learning and the ability to accomplish the objective. Skeleton Video Teaching Materials Camerawork is effective for increasing the power of expression in the video images. Similar effects can also be expected in a skeleton video; however, obtaining video of the viewer involves the camerawork being synchronized with the video, which requires sophisticated equipment. Poor camerawork will cause the viewers to easily perceive the image as being synthesized, which will hinder their progress into the world of the skeleton video. The authors believe that realizing a skeleton video that requires camerawork is presently too complex a process, and considered producing skeleton video teaching material using a fixed camera. We produced skeleton video teaching material using an "Algorithm Gymnastic Exercise," which is performed in NHK's educational program, "Pitagoraswitch," as the subject. This gymnastic exercise contains scenes in which two people clap their partner's hands and alternately stand and sit down, making it a gymnastic exercise that requires interactions with others. The skeleton video images of the completed version consist of a scene in which fourteen

3 kindergarten children perform the "Algorithm Gymnastic Exercise" by forming pairs. Therefore, video was recorded of each individual scene of thirteen children in which either the male or female child of the pair at the top was omitted and was used as the skeleton video for female and male children(fig. 3). The video was recorded through cooperation with fourteen five-year-old kindergarten children as the subjects of the experiment of "Algorithm Gymnastic Exercise." This exercise was conducted frequently in the same kindergarten class, and thus most children in that kindergarten knew the routine and were able to perform the exercise. We presumed that the children would attain a sense of co-existence and integration in the image with children familiar to them from the kindergarten by becoming part of the video content. Furthermore, it is also possible to incorporate elements of learning by experience since the children act in synchronization with the image. Figure 3. Skeleton Video Teaching Materials. A participant (top) and the synthesized image with her (bottom). Experiment Equipment A large screen and camera were placed in front of the classroom for the experiment, as illustrated in Fig. 4, and the video contents were projected on an area 120 cm wide and 90 cm high, located 30 cm above the floor. The resolutions of the camera and screen were on the level of NTSC TV signals. A stage was provided for the children to stand in the rear part of the classroom. A blue carpet was placed on the floor of the stage, and a blue curtain was hung to cover the entire wall. The image of the skeleton video teaching material and the image of the children were synthesized by Chromakey and input to the projector. The projector reversed the video contents laterally and then projected the result on the screen in the front portion of the classroom. camera screen projector blue stage Figure 4. Equipment. The child on the blue stage dances in the recorded video image on the screen. Scoring Procedures It is difficult to objectively measure to what extent children are feeling joyful. Questionnaires, sensation evaluations, or introspection methods can be applied to adults, but the verbal abilities of young children are undeveloped and it is difficult for them to express subjective impressions in an objective manner. We measured the concentration of metabolic-type immunity globulin A (s-iga) and that of cortisol in sputum in this study and examined the changes in concentrations during viewing of the skeleton video. s-iga is an immunity component that increases with relaxation and positive emotions [14], while cortisol increases under stress [15]. Method Our experiment subjects included a total of eleven five-year-old children, consisting of five male and seven female children (ranging in age from 5.5 to 6.3 years, with an average age of 5.8). The first sample of sputum was collected from the subject children after they had rinsed their mouths with water. The subject children were instructed to stand roughly at the center of the stage. Their positions were not strictly specified. Instead, the camera position was adjusted so that the skeleton video could be completed using the children's chosen positions. An ordinary video that was not a skeleton video was projected on the front screen. The contents of the video were the completed version of the images in which fourteen children, divided into pairs, performed the "Algorithm Gymnastic Exercise." The children participating in the experiment simply watched this video on the screen in the first session. The skeleton video was projected with the participating children synthesized into it in the second session, and the children were instructed to perform the same exercise while watching. The

4 second sample of sputum was collected after the exercise. Samples of sputum were collected using a alivette, which was developed for collecting sputum (made by SARSTEDT, Inc.), after the children rinsed their mouths with the supplied water. The salivette is composed of a centrifugal tube body, suspended insert, cylindrical absorbent cotton, and cap. A sample of sputum is collected when the subject child bites the sterilized cylindrical absorbent cotton. The subject children were very young, and thus a piece of string was threaded through the cylindrical absorbent cotton, and the end of the string was held while the subject children bit the cotton, to prevent them from swallowing it. The cylindrical absorbent cotton soaked with sputum was returned to the tube body and immediately refrigerated so that the intact components could be extracted later. The intact components were analyzed by SRL cooperation in Japan. s-iga(µg/ml pre-test post-test Figure 5. Changes in s-iga concentration in pre-post test. cortisolµ/ pre-test P<.059 P<.01 post-test Figure 6. Changes in Cortisol concentration in pre-post test. Results The samples of sputum collected before and after viewing the skeleton video were extracted and their concentrations of s-iga and cortisol were measured (Fig.5 and 6). We conducted t-test (one-sided test) before and after the experiment, which revealed that the concentration of s-iga increased significantly (t=1.707, df=10, p<.059) while the concentration of cortisol decreased significantly (t=3.444, df=10, p<.01). Our initial hypothesis, described above, was supported by the analysis results of the two components. It was evident that the skeleton video is a system that can be utilized joyfully and positively, and not a system that makes young children feel stress or a sense of incongruity. 5. Discussion This form of learning using a skeleton video, which incorporates interactions with images and elements of learning by experience, is effective for teaching young children. The experiment results indicated that the skeleton video can be joyfully and positively utilized by young children. However, one issue remains unresolved. The children must be active while watching the image in which they are synthesized. Therefore, the students must always look at the image of themselves. They must move appropriately to synchronize their action with the image. This is seldom experienced in daily life except when one looks into a mirror. The five-year-old children moved appropriately during the experiment by recognizing the images on the screen as themselves. However, it is questionable whether this can be achieved without difficulty by four-year-old or younger children. Effects of Displaying One's Own Image Adults recognize their own mirrored image and feel no sense of incongruity. However, this recognition of an image of oneself is not a innate behavior. Lewis & Brooks-Gunn [16] reported that normal children become able to discriminate between themselves and others at the age of one and a half, and become aware of themselves in the mirror. They pass the mark test, a technique used in the field of developmental psychology, at about the age of two. In this test, a colored lipstick is applied to an area of the subject child that he or she cannot look at directly, such as the nose. The child is then allowed to look at a mirror and the psychologist observes whether the child touches his or her nose. It can be assumed that if the child touches his or her nose that he or she recognizes that the image in the mirror is that of him- or herself. The image of a child with a lipstick mark was recorded on video in more recent studies, and the image was shown to the child after roughly three minutes to assess his or her ability to recognize an image of him- or herself [17][18]. It was reported that recognition of oneself becomes possible at the age of four or older under conditions involving such a delay. Thus, the ability to recognize an image of oneself develops from the age of two to four, depending on the experiment conditions. These facts suggest that

5 Figure 7. The camera work for participant has to synchronize with the camera work of the recorded video. The third flame is unnatural image created by unsuitable one.. there is no guarantee that the present experiment results for five-year-old children are applicable to children at the age of four or younger. It is necessary to consider the developmental stage when utilizing skeleton video for young children, such as those in kindergarten. Consistency of Images Morikawa [19] demonstrated that there are at least three methods for interpreting synthesized images that differ qualitatively. Let's consider that two images, A and B, are synthesized and displayed in a single image. There is a positional relation between A and B on the screen. There is also a mode to interpret that they are not related to each other (0) and two modes to interpret whether they are related to the contents of the display (1, 2). These are further divided into two modes: mode 1 to interpret whether A and B are in separate spaces and mode 2 to interpret whether they are in the same 3D space. For example, on a TV news screen, while character string A that represents the current time is in the relation of mode 0 with background image B, character string A that represents the name of a crime suspect is in the relation of mode 1 with the image of person B in the background, in the sense that A is an explanation of B. When weather forecaster A points to weather chart B, some may interpret that the relation is mode 2 while others interpret it as mode 1. Special effects images in a movie are produced under the assumption that they will be interpreted as being in mode 2, but some may interpret them as being in mode 1. Special effects images with poor consistency are difficult to interpret as being in mode 2, so viewers will interpret them as being in mode 1. Skeleton videos generally aim to produce synthesized images containing the viewer with good consistency, though this depends somewhat on their purpose. For example, skeleton videos for traveling to scenic locales and places of historical interest must produce a synthetic image that enables the viewers to feel merged into the world of the video image as if they were actually traveling. This is necessary for viewers to interpret the video work as being in mode 2. However, it is rare for video contents to consist of a single image taken by a fixed camera. There are generally many instances in which the contents involve various types of camerawork, such as camera zooms, pans, and switching. Techniques to synthesize the image of viewers in synchronization with the camerawork used when taking the original video are necessary when the video contents involve camerawork. For example, virtually dancing together with performers on a stage requires synthesizing that is synchronized with the camerawork used on the stage when the original video was recorded. It may become necessary to prepare multiple cameras to record video of the viewer and mechanisms for the skeleton video to control the camerawork, such as switching cameras (Fig.7). We used skeleton video taken by a fixed camera for our experiment, and therefore the videotape itself constituted the skeleton video. However, a skeleton video that involves camerawork must be a combination of video images with related camerawork control data for the viewer's camera and various control data for image-synthesizing positions. Moreover, extreme accuracy is required for the control data. Viewers will easily detect that the images are synthesized if the accuracy of the camerawork is poor, and they will be unable to merge themselves into the world of the skeleton video by abandoning the interpretation in mode 2. Further research and technical developments are required since realizing a skeleton video that involves camerawork is presently too complex a process. 6. Conclusion We proposed here a skeleton video in which viewers participate as subjects. We produced teaching material for young children and examined

6 its effectiveness. Interactions in a skeleton video are interactions with video contents using one's whole body. We conducted a experiment using five-year-old children. The results confirmed that the skeleton video was accepted and greatly enjoyed. The content at this time was Algorithm Gymnastic Exercise; skeleton videos will ultimately be applicable to various types of learning. Utilization of skeleton video teaching materials for four-year-old or younger children whose mirror image selfrecognition is not yet fully developed requires further study. In contrast, many more applications can be expected if the targets are expanded to elementary school and junior-high school students, university students, and adults. The authors have conducted research and development of an operation training system that utilizes skeleton video for use by medics performing operations using an endoscope [20]. Operations by skilled doctors are converted into video teaching material; viewers as medics go into the world of the video and mimic the operations by skilled doctors to learn their techniques. The subjects are not limited to skilled doctors. Skeleton videos can be applied to transfer the expertise of more common workers, which may be difficult to explain verbally. Skeleton videos are recorded teaching material, and thus they are reproducible and can be enjoyed repeatedly. For example, autistic children who are poor at communication can attempt to improve their symptoms by utilizing a skeleton video and can communicate in the world of the video by using their whole body. References [1] Sakata,Y., & Kawai,N. Does a communicative video enhance preschool children's memorization? The Japanese Journal of Developmental Psychology, 15, (2004) [2] Nakamura,K. The effect of manual operation of a moving target on young children's perception of causality. The Japanese Journal of Developmental Psychology, 7, (1996) [3] Nakano,S. The effects of pre-play experience on problem-solving activities in preschool children. The Japanese Journal of Educational Psychology, 29, (1981) [4] Morikawa,O. & Maesako,T. HyperMirror: Toward Pleasant-to-use Video Mediated Communication System, CSCW'98, (1998) [5] Stefan Agamanolis, Alex Westner, & V. Michael Bove, Jr. Reflection of Presence:Toward more natural and responsive telecollaboration, Proc SPIE Multimedia Networks,MIT97, 3228A (1997) [6] Yasuda,K., Sugita,K., Ushida,K., Naemura,T. & Harashima,H., A Mutual Communication-Support Using See-through Video Avatars, Proc. Virtual Reality Society of Japan, pp (2002)(in Japanese) [7] Maes, P., Darrell, T., Blumberg, B. & Pentland. A., The ALIVE System: Wireless, Full-Body Interaction with Autonomous Agents, Multimedia Systems 5(2): (1997) [8] Ishii,Y., Watanabe,T., Developmrnts of an embodied video communication system in which own VirtualActor is superimposed for virtual face-to-face scene (in Japanese), HIS2002,99-102(2002) [9] Fahlen,L., Brown,C., Stahl,O., Carlsson,C.: A Space Based Model for User Interaction in Shared Synthetic Environments, INTERCHI'93,43-48(1993) [10] Sugawara,S., et al., InterSpace: Networked Virtual World for Visual Communication,IEICE transactions on Information and Systems,E77-D(12), (1994) [11] Greenhalgh,C. & Benford,S., Massive: A collaborative virtual enviroment for teleconferencing. ACM Transactions on Computer-Human Inteaction,2.3, (1995). [12] Mandeville,J. et al., GreenSpace: Creating a Distributed Virtual Environment for Global Applications, In Proceedings of IEEE Networked Virtual Reality Workshop (1995) [13] Sylva, K., Bruner, J. S., & Genova, P. The role of play in the problem solving of children 3-5 years old. In Bruner, J. S., Jolly, A. and Sylva, K. (Eds.), Play. Penguin Books (1974) [14] Tsujita,S., & Morimoto,K. Secretory IgA in saliva can be a useful stress marker. Environmental Health and Preventive Medicine, 4, 1-8 (1999) [15] Suga,K., Sato,M., Yonezawa,H., Naga,S., & Shimizu,J. Effects of robot-assisted activity on senior citizens. -Indicators of HVA, MHPG, and CS concentrations in saliva-, Journal of Analytical Bio- Science. 26, (2003) [16] Lewis, M., & Brooks-Gunn, J. Social cognition and the acquisition of self. New York: Plenum Press(1979) [17] Kinoshita,Y. Judgement of legitimacy of group decision-making over indivisual concerns. A developmental study. The Japanese Journal of Developmental Psychology, 12, (2001) [18] Povinelli, D. J., Landau, K. R., & Perilloux, H. K. Self-recognition in young children using delayed versus live feedback: Evidence of a developmental asynchrony. Child Development, 67, (1996) [19] Morikawa,O. Explanation of Togetherness in Video Communication System by Cognitive Modes (In Japanese), Information Processing Society of Japan,Vol.46-7, (2005) [20] Morikawa,O., Kumagai,T., Yamashita,J., Yokoyama, T. An endoscopic surgical skill training system using HyperMirror(In Japanese), Proc. of 14th Annual Meeting of Japan Society of Computer Aided Surgery, (2005)