Shadow Communication:

Transcription

1 Shadow Communication: System for Embodied Interaction with Remote Partners Yoshiyuki Miwa Faculty of Science and Engineering, Waseda University #59-319, 3-4-1,Ohkubo, Shinjuku-ku Tokyo, , Japan Chikara Ishibiki Faculty of Science and Engineering, Waseda University #59-319, 3-4-1,Ohkubo, Shinjuku-ku Tokyo, , Japan ABSTRACT Shadow Communication system has been developed allowing others to communicate by means of their shadows in remote space. This is designed so that each participant s shadow enters into a space through a semi-transparent screen and interacts with another subject s shadow. This interaction occurs at a common stage (a co-existing space) that can be self-organized. The interaction takes place in a spatial relationship ( ma - perceptive distance or Maai in Japanese) and generates a situation where, seemingly, the participants are actually talking to each other in a face-to-face manner. Experiments on collaborative drawings or remote lecturing conducted with various subjects resulted in the different groups being properly positioned in their co-existing space, thus suggesting that the co-creative activities between the groups were successfully established. Categories and Subject Descriptors H.5.3 [Information Systems]: Group and Organization Interfaces Synchronous interaction, Computer-supported cooperative work. General Terms Design, Human Factors, Experimentation. Keywords Co-existing space ( Ba ), Shadow, Spatial relationship ( ma - perceptive distance), Embodied interaction, Inter-group communication, Collaborative drawing. 1. INTRODUCTION When a typical dialogue is considered as an improvisational drama, since the embodiment plays a very important role for the communication in the drama, it is obvious that both participants need to be on the same drama s stage. Furthermore, each participant s embodiment mutually transmits something and this something is shared by the participants, so that they can continue to act the drama without misunderstanding. However, it is not an easy task for people who are present in remote spaces to act and continue their drama while creating a consistent and smooth Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CSCW 4, November 6 1, 24, Chicago, Illinois, USA. Copyright 24 ACM /4/11 $5.. action. The difficulty is due to the fact that, since the self exists on the drama s stage, there exists an inseparable relationship, in other words, a subject-object inseparable relationship between the self and the stage. Hence, in order for people who exist in different actual fields to create a virtual stage, their actual fields and the virtual stage must be integrated inseparably. Furthermore, it is also necessary to appropriately position each participant on the integrated stage. The conventional subject-object separation theory which has been used in information technology tends to separate the self and the actual field, and express the drama s stage as a space which is not accompanied with the actuality of a field. As a result, conventional subject-object separation is not sufficient to position each participant on the drama s stage. Therefore, when each participant exists on the stage as an actor in a drama, it is necessary to express the situation in front of him/her i.e., the Ba notes_1, via the media and to transmit the situation to each actual field. With regard to this problem, Shimizu [1][2] advocated the Improvisatorial drama model of communication, and proved theoretically that it used a dual-domain function of self to generate the drama. Namely, there are two domains in self; the conscious domain which responds to one s individual consciousness and the unconscious domain which responds to the common feeling of existence. The former supports an explicit expression, and the latter supports an implicit (tacit) expression. Furthermore, the latter expression corresponds to the expression of Ba, which is mediated and is shared by the self-organization of the embodiment. This model was proposed as the Dual-domain model of self. Moreover, Shimizu hypothesizes that the co-sharing of context (situational meaning) can be attained through this dual-domain media expression. Therefore, with these models, in order to express the stage of a drama, it is necessary to investigate how an embodied media expression works on the unconscious domain of self [1][3]. Moreover, by co-sharing the drama s stage, it is expected that an inseparable subject-object relationship can be created, and the relationship can generate a united feeling of co-existence. The authors treated the CSCW as an improvisational drama and sought to develop an expression method for supporting co-creative activity among many people on co-shared stage ( Ba ). To achieve this specific aim, it was first necessary to Note_1. Ba is defined as a kind of space - not just a physical space, but rather an entire situation of the actual field. In other words, the concept of Ba allows a person to recognize unconsciously his/her situation in relation to his/her perception of time and space [1]. Volume 6, Issue 3 467

2 discover a method to position remote participants on a common stage (namely, co-existing space). In all, the objective of this study was to develop a system where each remote participant could communicate within a ma - perceptive distance note_2 (or Maai in Japanese). Proxemics by Hall [4] suggested that the term ma - perceptive distance connotes an unconsciously felt spatial relationship amongst individuals. This implies that ma - perceptive distance can be generated directly through each embodiment [1][5]. In order to generate this ma - perceptive distance at the remote locations, it is mandatory to develop the media of expression of existence by which each participant can position their existence at a common stage. The present authors have thought of an idea where shadow can serve as the media of expression and have developed a co-existing communication system WSCS-I (Waseda Shadow Communication System Type I) using the self shadow as the participant s agent [6][7][8]. Below, the design requirements and methods for a co-existing communication system that uses shadow as the media of expression of existence are compared and contrasted to those reported by others. Following which, the efficiency and efficacy of the WSCS-I for effecting communication while taking the ma - perceptive distance between the remote distances will be demonstrated. Based on these results, the new WSCS-II, which can be applied to co-existing communication between remote groups, will be introduced. Furthermore, results on employing WSCS-II when drawing common pictures between remote distances, and the efficiency of this method for creating a co-existing stage between remote groups will be discussed. 2. DESIGN OF CO-EXISTING COMMUNICATION SYSTEM 2.1 Bodily Expression Method for Sharing Space Research on communication systems through which space can be shared between remote actual fields is very important to support the creation of a feeling of co-existence and mutual trust. It has been pointed out that it is also important to perform media expression of the bodily actions of each participant [9]. Consequently, several works have been reported on the communication systems using video image media. For example, systems such as VideoDraw(Tang et al.) [1] or ClearBoard (Ishii et al. )[11] can be found. Using these media, each participant made improvised drawings on a board. Moreover, expressions of each hand or body movement, as well as eye contact were attempted. In these examples, the remote participants were separated by the board. Therefore, unlike our design, the other partner s position cannot be created in the actual field where the self exists. Thus, it can be interpreted that with these examples, the co-existing virtual stage was realized in the restricted board space. Moreover, there are also other videoconferencing systems such as the Hydora Unit (Sellen) [12] or MAJIC (Okada et al.) [13] that use the bodily arrangements of a plurality of participants. There is also Agora (Kuzuoka et al.) [14] through which the indication operation of real object can be realized from a remote location. Similarly, with these systems, each participant recognizes the other partner s existence through the display of intentional actions. Consequently, the systems do not provide the successful creation of a co-existing stage where positional relationship and distance ( ma - perceptive distance) can be generated unconsciously. Moreover, in examples like CVEs [15][16][17] and FreeWalk(Nakanishi et al.)[18], avatars or agents are introduced to express the participants actions in a 3-D virtual space via CGs and the relative positional relationship is expressed by manipulating the avatars or agents. However, under these systems, the participant s virtual space and real space are separated, so a situation where a co-existing feeling which can be conveyed directly via the embodiment of participants is not created. The HyperMirror (Morikawa et al.)[19]. appears to be a similar system to our system, under which the self image is incorporated into a video imaged screen. It was reported that under this system it is possible to take ma - perceptive distance between partners. However, since the ma - perceptive distance is measured and recognized in the video image by using the self video image as a reference, the stage where each participant can exist is limited on the screen. Moreover, there are trials that use robots as self agents to express the embodiment of participants. However, as yet, with these systems the stage can not be shared with other partners [2][21]. Furthermore, there are several research endeavors such as VideoWhiteboard (Tang et al.) [22] and others [23][24] that try to express the other partner s existence not by the video image but by shadow, in order to create a feeling of co-existence. Although a shadow possesses more uniqueness than a video image, as discussed later in more detail, since the shadow is explicitly expressed as a silhouette by the VideoWhiteboard system, the system can be considered to be close to that of a video image expression. Although under the system, partner s actions can be expressed, the feeling of distance and ma - perceptive distance between the participant and the other partner can not be generated. Accordingly, there are no current research endeavors being conducted on inseparable subject-object co-existing communication systems which share the stage required for an improvisational drama as CSCW does. 2.2 Design Method of Co-existing Communication System In the co-existing communication system which we try to produce, different actual fields (existing spaces) are self-organized into a co-existing stage (or space). And while the self is positioned in a space organized in such a manner, each participant is able to interact. This idea is based on the above-mentioned Dual-domain model of self proposed by Shimizu [1], which has also been referred to as dual-domain communication. Hence, an inseparable subject-object media is necessary to express the self and the actual field where the self exists in an inseparable manner. The creation of a co-existing stage through self expression which acts directly on the other partner s embodiment using such media is also required. Based on these requirements, the authors noticed the following unique characteristics associated with shadow. a) A person and their shadow are absolutely inseparable. b) The existence of a person can be evoked by their shadow. c) A shadow has the function of expanding a body image. d) A shadow changes its movement or location proportionally in response to the bodily movement or position of the shadow s owner. Note_2. Maai can be defined as the relationship between each entity positioned in the Ba. It is also called Ma in Japanese. Maai, in this study, possesses concepts of spatial distance, which is unconsciously expressed by individual feelings and can be identified as ma - perceptive distance. 468

3 Room A Shadow A Room B Room A Shadow B Room B Shadow A as an agent of s existence passing through the screen Shadow B as an agent of s existence passing through the screen Shadow A Integration of actual field Integration of actual field Shadow of Shadow A Creation of common stage (co-existing space Ba ) (a) Shadow passing through the screen (b) Use of space by shadow passing through the screen the shoji Figure 1. Design method of the co-existing communication system The characteristics of a) and b) are due to our ability to share the recognition that the self exist in a world where your shadow is also present. Hence, it is expected that a shadow can inseparably connect the self and actual fields. The characteristic of c) is on the verge of being verified in the area of brain science, and is currently receiving a great deal of attention [25]. Although the amount of the explicit information concerning shadow is very small when compared to that regarding video images, shadow can play an important role in integrating all detailed body actions. Moreover, the characteristic of d) suggests that there is a possibility to transmit an actual field s condition through another partner s shadow. In the next step, it is necessary to develop an interface for integrating participants actual fields using their shadows characteristics, and to express an integrated virtual stage and each actual field in a manner that has an inseparable relationship. As seen in Figure 1(a), a Shoji screen (a traditional Japanese paper-sliding-door) is adopted as a metaphor to realize the aforementioned expression media, which plays the role of the interface to connect the remote spaces. A Shoji screen, which is different from a common block wall, has functions that evoke the situation and atmosphere outside the space and allows the existence of a person to be imagined due to being able to see the shadow through the Shoji screen. Thus, by sending shadow back and forth as agents of the participants through a screen which has a similar function to a Shoji screen, a method has been proposed to create a co-existing virtual stage as if a person in space A (room A) is co-existing with a person in space B (room B), and vice versa. As shown in Figure 1(b), it can be made to evoke the existence of communication participants in the space outside the screen or on the inner side by projecting the mutual shadows of the participants who are present in the remote places on the screen of each room. In recognition of the design method, it was named the separated Using inside and outside the screen < Separated space type > Sharing space inside the screen < Shared space type > space type when the other communication partner s shadow was over the screen as a silhouette, i.e., it existed outside the screen and the space was separated; and it was named the shared space type when the other partner s shadow was extended from his/her feet, i.e., it existed in the same actual space and the space inside the screen was shared. Although the former type is similar to the VideoWhiteboard [22] expression method, a type similar to the latter has not yet been seen. Other noticeable characteristics of this method are that the position, dimensions and contrast of the shadow can be varied according to the other partner s actions. Particularly with the latter shared space type, since the participant s shadow and the other partner s shadow are simultaneously displayed on the screen, the participants feel as if they are present in the same space through each other s shadow. 3. WSCS-I 3.1 Construction of WSCS-I WSCS-I, which has been previously reported [6][7][8], is constructed with two separated rooms which make communication possible (or simply called a communication space) and are installed with a semi-transparent screen at the front Space A Mic Speaker Mat type touch sensor H8 Projector (source of light) Shadow image projection Capturing thermal distributional image Thermal vision camera Ethernet (1Mbps) Figure 2. Structure of WSCS-I Space B Sound output Data of standing position (1 byte serial data) Sound input Volume 6, Issue 3 469

4 of each room. The dimensions of the room are 1.8 [m] wide, 1.8[m] deep, and 2. [m] high (see, Figure 2). The image of a person entering this space is converted to a thermo-distribution image by the thermal vision camera, and the converted image is processed to his/her shadow and extracted by the control PC. The processed image is then modified in response to the standing position of a participant in the remote communication space. Consequently, the touch-type sensor mat installed in a grit pattern on the floor of communication spaces detect the standing position of the participant. In both the separated space type and the shared space type, the shadow made in the above manner is projected on the screen by a projector installed at the back side of each room. The screen possesses a certain level of transparency so that in case of the shared space type the shadow projected on the screen can be seen with almost the same depth as the shadow projected on the floor. Moreover, since this transparency was used in the separated space type, changing the size and contrast of the partner s shadow resulted in a creation of a spatial depth feeling by the participant position in the space outside the screen with a similar sense with the Shoji screen. Furthermore, we attempted to connect consistently separated space type and shared space type using the screen as a boundary between outside and inside spaces. Figure 3 demonstrates sequential movements of the other partner s shadow while entering the space (where the participant is present) from the other side of the screen and then returning back to its room through the screen. The details of the extraction technique and the projection technique for making the shadow image have been previously reported [7]. The dialogue which occurs when the system is in use is recorded by a directional microphone, and is reproduced using a speaker and a telephone line or voice chat software. The time delay of the above system is approximately 7 [ms], at 15 [frames/sec] which is the processing rate of the two-way motion image processing communication software. 3.2 Communication Experiments As described previously, the uniqueness of this system is that the remote participant s existence can be positioned in the self space and the ma - perceptive distance can be shared by the shadow s function. Focusing on this point, the following comparative experiments were conducted for the following two communication types; (1) the communication mode was set as shared space type, and (2) the communication mode was set as separated space type, using a life size video image or shadow. All participants were allowed to interact and speak freely. A questionnaire was administered after the experiment to evaluate the extent of the generation of the ma - perceptive distance during the experiments. Figure 4 shows some parts of the experiments, and Figure 5 shows the participants comments from the shared space type experiment. When we performed statistical analysis to identify the significant differences, as shown Figure 6, it was found that the shared space type shadow chat makes it easier and Figure 3. Shadow appearance passing back and forth the screen B s Shadow A s Shadow Communication space A Communication space B (a) Shared space type shadow chat Communication space A B s Shadow (b) Separated space type shadow chat Communication space A B s video image Communication space B A s Shadow Communication space B A s video image (c) Separated space type video chat Figure 4. Appearance of communication experiments - I felt as if I was talking to my partner on the same stage. - I sensed the presence of my partner. - I felt that I tried to keep the distance between my partner. - I felt as if I would like to move around. - I felt easily to synchronize my body movements with my partner. - When my shadow was touched, I felt as if my body was touched. - When I stepped on my partner s shadow, I felt some sort of guilty. - When the shadow is approaching me, I felt as if I was restricted. - I felt as if I was talking to myself. - It was easy to speak to. - I sensed some sort of equality. - No tired, nor tension. (Sample: 15 adult males and 3 adult females.) Figure 5. Comments on the shared space type communication tests smoother than the separated space type video chat to position oneself and the other partner in his/her space, and to communicate while taking ma - perceptive distance. 47

5 Significant difference * p<.5 ** p<.1 Shadow Video image ** * * * * * ) There is a feeling of existence of my partner. 2) There is a feeling as if I am on the same stage. 3) I am talking while taking a distance from my partner. 4) I feel to move my body spontaneously. 5) I try to adjust the partner s rhythm. 6) When my partner touches my shadow, I felt as if he touches me directly. 7) I feel that my partner is watching me. 8) My partner's feeling comes across me. 9) My partner s intention is conveyed. 1) There is an equal feeling and anonymousness. Figure 6. Questionnaire results on creation of ma -perceptive distance ( Separated space type video chat - shared space type shadow chat) Figure 7 shows an example of time-related changes and a movement trace of the standing positions of the conversation participants. As seen clearly from the figure, it was found that each participant stopped their movements soon after they started the video chat experiments. This might be due to the fact that their attention was drawn to observing the video images. On the other hand, with the shared space type shadow conversation, it can be said that they continued the conversation while changing their positions as if they are using the space effectively. At the same time, during shared space type conversation, entrainments [26] were often observed which synchronized the movement speeds of the participants. As seen in Figure 7(b), participants moved frequently, with small movements to the screen along the horizontal axis (X-axis), a phase relationship was found in the changes in their speed, and large movements to the screen along the vertical axis (Y-axis). Since the entrainment phenomenon is an unconscious synchronizing of body rhythms which can be recognized during actual tête-à-tête communication, the above results suggested that both participants felt as if they were communicating in the same space. Moreover, these suggestions are in close agreement with the comments listed in Figures 5 and 6. Hence, these results also suggested that it is possible to sense and feel the other partner s existence by projecting the other partner s shadow from the floor to the screen. If a video image was transposed on the shadow (see Figure 4(b)), what would happen? In this case, although the following comments were reported, Because of seeing the shadow image, the feeling of being seen by the other partner became much weaker, so that it was much easier to talk him/her. or When a shadow was shown, I felt as if someone else was present behind the screen, there were very few supportive comments for the generation of a spatial relationship ( ma - perceptive distance) like the situation of video chat. Therefore, in this mode it also appears to be difficult to position the other partner in your own space. From the above, it was found that the shared space type shadow expression method was very effective in creating a virtual drama s stage where the participant could co-exist with the other partner in the same space, in a manner that can be used to support the generation of ma - perceptive distance. (X-axis) (Y-axis) (X-axis) (Y-axis) Velocity[mm/s] Subject A Subject B ~ 6[sec] Time [s] 6 X-axis A tracing Time [s] 6 X-axis B tracing < Separated space type video chat > 18 Subject A Subject B 12 6 Time [s] Time [s] 6 X-axis B tracing < Shared space type shadow chat> (a) Time-related changes of positions and tracing Time [sec] 5 X-axis A & B tracing (b) Synchronization of the moving speed rates and tracing during the shared space type shadow chat Figure 7. Changes in standing positions during the communication tests 4. WSCS-II From the communication experiment using WSCS-I, it was found that the shadow expression technique of the shared space type effectively helped to position the existence of the other partner, who was present in own actual field, in the common stage, and to create a co-existing space. Based on these positive and encouraging findings, the present authors extended this method from a one-to-one communication system to an inter-group communication system. By expressing the shadows of a group comprised of a plurality of people into a group in which the participant was present, the creation of a co-existing space between an individual and a group or between two groups engaged in a communication activity while taking ma - perceptive distance were tested and evaluated. The system details (WSCS-II) are described below. 4.1 System Construction of WSCS-II The required design elements for developing WSCS-II by expanding WSCS-I system are listed below; a) Extraction of shadows of a plurality of people b) Shadow projection, corresponding to each member s standing point c) Extension and adaptability of the communication space d) Installation of a collaborative work application usable by a plurality of people e) Easy setting-up of the system Y-axis Y-axis Y-axis X-axis A tracing Y-axis Y-axis ~ 6[sec] 4 ~ 5[sec] Volume 6, Issue 3 471

6 Hanging scroll (drawing surface) Projector Encoder Line heater 35 Thermal vision camera Stepping motor 45 Making of drawing line (plotting of heat source) Heat insulation cover (a) Hanging scroll-drawing application Unit [mm] Capturing drawing data (a) Moving base of thermal vision camera and projector Extraction only heat part Tatami floor (b) Extended communication space Figure 8. Communication space, shadow acquisition and projection equipment With regard to point a), although the extraction technique of the shadow remains the same as that for the WSCS-I system, the viewing range was extended from 26. x 19.6 [deg] to 52. x 39.2 [deg] by attaching a view magnification lens to the thermal vision camera, so that the extraction of people s shadows could be conducted in a much wider range. Concerning point b), since a mat-shaped touch sensor array of 1.8 [m] x 1.8 [m] was installed in the WSCS-I system in order to detect a participant s standing position, it was impossible to detect the positions of a plurality of people. Accordingly, we devised a technique to match the view angle of the thermal vision camera for shadow extraction which captures the thermal distribution image of the communication space and the view angle of the projector which projects the shadow. Now a participant s shadow can be displayed in the remote space without the need to detect each participant s standing point by sensors. Furthermore, the image beyond the viewing range can be also covered by synchronously moving the thermal vision camera and projector, so that the communication space can be expanded for point c). The appearance of the newly developed device is shown in Figure 8 (a). Additionally, since the thermal vision camera and the projector are attached at the base which controls the installation angle, requirement e) for the design is satisfied. Due to these modifications the communication space can be expanded to a floor space about 3 times larger; 2.7[m] deep and 3.6[m] wide, as shown in Figure 8(b). Furthermore, the thermal vision camera and the projector can be driven by two stepping motors which have a range of pitch ±113 [deg] and a yaw ±6 [deg], and a maximum working speed of 16.6 [deg/sec] with an error range of.5 [deg]. These mechanisms make the communication space more flexible. In order to realize point d), a collaborative drawing application was installed in WSCS-II which uses a drawing pen. We installed two types of drawing systems; a hanging scroll-drawing application and a black and white drawing ( Sumi-e in Japanese) drawing application (see Figure 9). The former uses a hanging scroll with a motif as the area on which participants can 472 Displaying drawing data A pixel displays the overlapping portion deeply (b) Black and white drawing application Figure 9. Collaborative drawing application, i nstalled in the WSCS-II Thermal vision camera h θv L1 Ly x[pixel] hl1+h2tanθv Ly= h-l1tanθv (θv=yθth-v/24) θth-v=39.2[deg]: Vertical viewable angle y[pixel] Transform of vertical position with screen Thermal vision showing Thermal vision plots of ankles camera Battery box θh Lx Ly Lx=Lytanθh (θh=xθth-h/32) θth-v=52.[deg]: Horizontal viewable angle Line heater Transform of horizontal position with screen (a) Heat source (b) Conversion rule of the detected coordinates equipments into the actual standing position Figure 1. Detecting technique of participant s standing position using heat sources draw together, and the latter enables simultaneous black and white drawing by a plurality of people throughout the entire space. A silicone line heater was attached to the nib of each drawing pen, and was heated to 5~8 [ C]. The temperature distribution was displayed in color by a specific temperature display function of the thermal camera, so that the nib coordinates could be acquired separately from the human body in the thermal distribution image. In the hanging scroll-drawing application, when the nib was in the drawing area of the scroll drawing, the control PC extracted the nib coordinates. It connected the current nib position and the last nib position with a line to display the drawing line. In the black and white drawing application, since we adopted a system where the pixel of the heat source of the nib captured by the thermal vision camera can be kept in the frame of the projection image as

7 Shadow B & C Mic Projector Movable base Speaker Space A Thermal vision camera LAN(1Mbps) Telephone line (a) Dialogue appearance when chairs are used. Shadow A Space B Mic Human B & C Speaker Control PC Figure 11. Structure of WSCS-II Display a gray drawing line, when the drawing pen is in the same position as a subsequent frame, the gray color can overlap and make the line darker. As a result, such lines can provide more Sumi-e -like expression. Moreover, with this modified system, it is possible to simultaneously draw pictures using a plurality of pens. This technique was also employed for measuring the standing positions in the communication experiment, as will be described later. In this case, a heat source was attached to the ankles of participants as shown in Figure 1(a). The coordinates of this heat source within the frame were detected using a specific temperature detection function, which is the same way that was used in the previous drawing application. The detected coordinates were converted into the actual standing position in the communication space according to the conversion rule shown in Figure 1(b). A measurement accuracy within 7[mm] at a point 7[m] away from the thermal vision camera s lens surface was achieved. The construction of the whole system with the integration of the above details is shown in Figure 11. Control software for the shadow extraction processing and drawing line processing was written using C++ Builder4 and installed on the control PCs. The PCs were connected by 1 [Mbps]LAN. Data of the extracted shadow and the angle data of the thermal vision camera and projector were communicated. Similar to the WSCS-I system, voice was gathered by a directional microphone installed in the upper part of the communication space, and was reproduced by a speaker in the other communication space via a telephone line. The time delay of the whole system was about 7[ms]. 4.2 Emergence of Spatial Relationship ( MA - Perceptive Distance) While Conducting Dialogue Using WSCS-II First of all, we have conducted experiments on the generation of a spatial relationship ( ma - perceptive distance) using the WSCS-II system. The experiment was set up in such a way that two remote participants entered the communication space from outside the system, then the detailed movements of the two participants before and during their dialogue were observed and measured. Two different conditions were pre-determined; (1) they were asked to sit down on chairs when they start to talk, and (2) they were asked to converse while standing (no chairs were provided). Each participant was not informed about the other (b) Dialogue appearance while standing Figure 12. Creation of ma - perceptive distance during the Dialogue start dialogue tests Subject A Subject B Time [sec] Enter into Before sitting After sitting WSCS-II (a) Distance among dialogue in WSCS-II (using chair) 2 1 Dialogue start Subject C Subject D Enter into Time [sec] WSCS-II (b) Distance among dialogue in WSCS-II Dialogue start 3 Subject E Subject F Enter into Time [sec] WSCS-II (c) Distance among dialogue in actual meeting Figure 13. Changes in standing positions of dialogue tests partner prior to the tests. The general appearance of each experiment is demonstrated in Figure 12, and an example of the results of changes in the relative distances of both participants is shown in Figure 13. For the purposes of comparison, Figure 13 also shows the results when two participants entered the same communication space of this system and talked to each other. As a result of this series of experiments, regardless of whether or not chairs were used, 11 out of 12 different pairs demonstrated getting closer to facing each other and the dialogue started after the emergence of a certain distance between the participants. This agrees very well with the case when two people actually have a face-to-face encounter and their dialogue begins after each scrambles to generate ma - perceptive distance, as shown in Figure 13(c). More interestingly, the relative distance that emerges in this case, quite corresponds well with the relative distance that emerges in shared space type situation. What this means is that this system assists participants to create the dialogue space as a stage ( Ba ) which can be aware to the mutual existence of both participants through positioning the other partner s existence in the space where the self exists. Therefore, it can be said that this Volume 6, Issue 3 473

8 Shadow D & E B E C Human C & C Drawing tool E D A D D E A B A C C B Chairs Figure 14. Creation of ma - perceptive distance by a plurality of people Drawing line Shadow B Shadow C Figure 15. Collaborative drawing picture by a plurality of people using WSCS-II (Improvisational drawing of some trees, rivers, and mountains, etc.) Communication space A (lecturer side) Communication space B (student side) Figure 16. Remote mock lecture by collegians system may be able to realize dialogue between remote localities that is accompanied by a sense of distance such as in the Proxemics, which Hall [4] proposed. 4.3 Inter-group Communication Using WSCS-II Next, in order to investigate the generation of ma - perceptive distance among a plurality of people, we conducted an experiment in which each participant takes their own chair in the communication space, and was asked to determine the position where they could sit down freely by evaluating the surrounding situation. The appearance of the experiment, when three participants are in one communication space and two participants are in the other space, is shown in Figure 14. This figure shows the situation of the communication space in which three participants were present. As a result, even with a plurality of people, it was found that the overall spatial relationship was created through an interaction with the shadow of the other partner s existence. Namely, each participant did not sit on a pre-determined chair, and the bodily arrangement as reported by Kendon [27] was created through interaction between the remote groups. Therefore it can be considered that this system integrated the remote groups actual fields and made it possible for them to co-exist in the same common stage. This also suggests that this system is supportive of co-sharing the stage between remote groups comprising a plurality of people and successfully interacting while taking a comfortable ma - perceptive distance. Hence, a collaborative drawing by a plurality of people using a drawing tool was performed. The appearance of the experiment is shown in Figure 15. As a result, all participants responded with positive comments indicating that they enjoyed drawing pictures Felt a strong existing-feeling on a main person. - I felt as if I were teaching. - I felt that all of us are sharing a same feeling. - I felt that we all spending the same time. - I could take ma - perceptive distance very nicely. - I sensed the tension, eagerness, understanding and puzzling of the partner by judging his shadows. - I naturally tried to yield my space to a shadow entering into my space or a shadow asking questions. - I felt jealously on a group of people which is united. - While drawing pictures, I felt as if there is no partition between the next room and would like to move around. - Compared to Internet, telephone conversation, or communication, I felt and sensed much easier about what my partner was going to say or feel. - The impression when we communicated well remained strongly even after the communication was over. - Since questions from the shadow were much more, I felt as if my partner listened to me very well. - Because of just the shadow, I could explain it much easier even I met the shadow for the first time. - I felt that more than number of peoples being present are gathering in my space. (Sample: 15 adult males and 3 adult females) Figure 17. Comments on remote group communication tests Furthermore, if participants began to draw a picture without first deciding the theme, it turned out that they could draw an organized picture extemporaneously because the role assignment took place between participants intra-temporally. These results suggest that the intentions of each participant s act in a group are shared, because the participant who is present in the remote location enters into the other partner s space, and thus the existence of each participant is positioned on a common stage. If this is true, this system can support remote groups by helping them sharing their context. The following experiments were conducted on this point. To begin with, a remote lecture experiment was performed in which a lecturer was located on one side and students on the other side. The students were selected from a group whose members are familiar to each other, such as students working in the same laboratory. Using one lecturer, students participated in this test by either being in the same room as the lecturer or as a shadow

9 Communication Space A Communication Space B Each group is acting individually Onset of exchanging activities between groups Development of unified movement as a big group Walking following former human is easy Walking is difficult with mask Figure 19 Appearance of an experiment when black colored mask cover on the floor (When floor is masked participants cannot emerge spatial relationship ( ma - perceptive distance)) projected from a distant place. At the same time, a similar test was performed for groups in which each member was not familiar with the other members. The appearance of the experiment is shown in Figure 16, and the comments obtained from participants are demonstrated in Figure 17. From the results, it was found that using shadows the stage was shared even between groups, thus an inseparable subject-object relationship between groups was created. Next, as a trial to connect two different groups, an experiment was conducted in which two dancing schools were connected. Two professional dance instructors were assigned at respective dance lesson studios in remote locations, as shown in Figure 18. Initially, instructors were giving different lessons in their own localities. However, in time instructors came to give lessons to students of the other group through their own shadow projected from the remote location. Finally, it was also recognized that the two dancing schools created one dance activity together by mutually practicing lessons. This series of results described above suggest that the shared space type shadow expression integrated the different groups actual fields into a co-existing stage, resulting in co-creative activity while co-sharing context. Of further interest is the fact that, when a mask was used to entirely cover the floor so that only the projected shadow was displayed on the screen, it was noticed that the smooth dialogue and co-creative activity between remote locations which was described above, was no longer observed (see Figure 19). In this situation, the participant s shadow was displayed only on the screen, meaning that the participant s self and the participant s Figure 18. A trial for connecting tow different dance lesson studios shadow were separated on the floor. Hence, this situation became the same as the previous separated space type where the display included the participant s own shadow. As a result, the shadow image completely lost its function as a shadow, and made it seem as if the symbolized image of the participant was displayed in a different space from the actual participant. Because the participants cannot position their own existence in their own space, and the co-existing stage of the integrated group was disordered. In other words, it is a sine qua non to project shadows in the other partner s space according to the positions of the participant s actual field in order to create a co-existing stage. Moreover, integrating spatially the participant s shadow in his/her actual field and the shadow of the other partner, and displaying these shadows simultaneously are two important design elements in order to generate ma - perceptive distance. 5. SUMMARY AND CONCLUSIONS In order to create feelings of co-existence and mutual trust between remote participants in the CSCW area, it is important for them to share a common stage ( Ba ) and produce co-creative activity. In order to create such a stage in an inseparable subject-object manner, a unique communicable expression method was proposed by employing shadow as an implicit media which expresses each participant s existence. Based on this expression, a co-existing type communication system was also developed which makes it possible to create a common stage (an integrated virtual field) which consists of a participant s actual fields and his/her position within the space. We have presented successful experimental works demonstrating that collaborative works and conversation between remote locations while participants are taking ma -perceptive distance and the drama -like communication can progress while participating groups are taking actions coherently. This system, as seen in Figure 2, can be considered as a new type of inseparable subject-object communication system which operates on a co-existing stage created by the functions of shadows. Namely, there is a possibility that each participant is able to exchange messages while sharing their actual field, in the manner of normal communication in a real space. In conclusion, the authors believe that once this system is further developed and advanced, it can surpass the one domain communication system Volume 6, Issue 3 475

10 Space A Ma - perceptive distance Shadows as agents of existence Emergence of common stage (co-existing space Ba ) by the Shadow s function Integrated virtual field Space B Ma - perceptive distance Figure 2. Co-existing communication using shadows under which only symbolized messages are exchanged based on Shannon s theory, and can realize the dual domain communication transmitting context proposed by Shimizu [1]. Furthermore, various quarters expect this system to have practical applications in a range of fields including education, welfare, health care, psychological therapy, and community creation. Although all results presented in this study were based on projecting shadows in the same dimensional space, another expression method to create a common stage by projecting shadows in different dimensional spaces, is in progress. 6. ACKNOWLEDGMENTS This study was funded in part by the RISTEX (the Research Institute of Science and Technology for Society) Program Chaos and Its Controls in Highly Automized Social System, the Japan Science and Technology Agency (former Japanese Science and Technology Corporation), We would like to express our sincere thanks to Dr. H. Shimizu, Professor Emeritus of Tokyo University (and a research director for the aforementioned research project) for his valuable comments. Thanks should also go to our graduate students, A. Shinohara, T. Watanabe and undergraduate students, H. Itoh and T. Fumino for their assistants and collaborations for conducting delicate experiments. 7. REFERENCES [1] Shimizu, H.: The Thoughts of "Ba": Creative stage of life" (in Japanese), University of Tokyo Press, (23). [2] Shimizu, H., Kume, T., Miwa, Y., Miyake, Y.: Ba and co-creation" (in Japanese), NTT publishing, (2). [3] Miwa, Y.: Communication technology of ba in co-creation (in Japanese); Systems/Control/Information, 45-11, pp , (21). [4] Hall, E.T.: The hidden dimension, Garden City, N.Y., (1966). [5] Itai, S., Kudo, A., Miwa, Y., Aizawa Y.: Creation and co-share of timing in an actual communication, Proc. of 22 IEEE International Conference on Systems, Man and Cybernetics, CD-ROM, (22). [6] Miwa, Y., Ishibiki, C., Watanabe, T., Itai, S.: Design of co-existing space by shoji interface showing shadow, Human-Computer Interaction, Vol.2, Proceedings of HCI International 23, pp , (23). [7] Miwa, Y., Ishibiki, C., Shinohara, A., Watanabe, T.: Development of Co-Existing Communication System using the Shadow as an Agent of Self (in Japanese); Technical report of The Institute of Electronics, Information and Communication Engineering, pp.35-41, (23). [8] Miwa, Y., Ishibiki, C.: Development of Co-existing Communication system using shadow for creation of Ba (in Japanese), Proceedings of interaction 24, pp , (24). [9] Heath, C. C., Luff. P. K.,: Disembodied Conduct: Asymmetries in video mediated interaction in an office environment", CHI'91: Reaching Through Technology, pp , (1991). [1] Tang J. C., Minneman, S. L.: VideoDraw: A video interface for collaborative drawing, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, pp , (199). [11] Ishii, H., Kobayashi, M., Grudin, J.: Integration of Inter-Personal Space and Shared Workspace: ClearBoard design and experiments", Proceedings of CSCW 92, pp.33-42, (1992). [12] Sellen, A.: Speech patterns in video-mediated conversations", Proceedings of CHI 92, ACM Press, pp.49-59, (1992). [13] Okada, K., Maeda, F. Ichikawa, Y., Matsushita, Y.: Mulyiparty videoconferencing at virtual social distance: MAJIC design, Proceedings of CSCW 94, pp , (1994). [14] Kuzuoka, H., Yamashita, J., Yamazaki, K., Yamazaki, A.: Agora: A remote collaboration system that enables mutual monitoring", Proceedings of CHI 99, pp , (1999). [15] Benford, S., Greenhalgh, C., Rodden, T., & Pycock, J.: Collaborative virtual environments". Communications of the ACM, 44 (7), pp.79-85, (21). [16] Greenhalgh, C., Benford, S.: Massive: A collaborative virtual environment for teleconferencing", ACM Transactions on Computer-Human Interaction, 2.3, pp , (1995). [17] Hagsand, O.: Interactive multiuser VEs in the DIVE system, IEEE Multimedia 3, 1, IEEE Computer Society, pp.3-39, (1996). [18] Nakanishi, H., Yoshida, C., Nishimura, T., Ishida, T.: FreeWalk: Supporting casual meetings in network ; Proceedings of CSCW 96, ACM Press, pp , (1996). [19] Morikawa, O., Maesako, T.: HyperMirror Toward Pleasant-to-use video mediated communication system", CSCW 98 pp , (1998). [2] Kuzuoka, H., Kosaka, J.; Designing a Robot as Communication Media --The effect of head and body movement on co-present s impression, Proceeding of IEEE International Symposium on Computer Intelligence in Robotics and Automation, pp. 5-54, (23). [21] Paulos, E., Canny, J.; Social Tele-Embodiment: Understanding presence, Autonomous Robots vol.11, pp.87-95, (21). [22] Tang J. C., Minneman, S. L.: VideoWhiteboard: Video shadows to support remote collaboration, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, pp , (1991). [23] Apperley, M., McLeod, L., Masoodian, M., Paine, L., Phillips, M., Roger, B., Thomson, K.: Use of video shadow for small group interaction awareness on a large interactive display surface, Proceedings of AUIC 23, Vol. 18, pp.81-9, (23). [24] Okamoto, M., Nakanishi, H., Nishimura, T., Ishida, T.: Silhouettell: Awareness support for real-world encounter, In Toru Ishida Ed., Lecture Notes in Computer Science 1519, pp , (1998). [25] Pavani, F., Castiello1, U.: Binding personal and extrapersonal space through body shadows", journal of Nature Neuroscience, January 24 Vol.7, No. 1, pp , (24). [26] Condon, W. S., Sander. L. W.: Neonate movement is synchronized with adult speech"; Science, No.183, pp.99-11, (1974). [27] Kendon, A.: Conducting Interaction: Patterns of behavior in focused encounters", Cambridge University Press, (199). 476