Abstract FACILITATING REAL-TIME INTERCONTINENTAL COLLABORATION with EMERGENT GRID TECHNOLOGIES: Dancing Beyond Boundaries James Oliverio, Andrew Quay and Joella Walz Digital Worlds Institute University of Florida Gainesville, Florida 32611-5800 USA {james, andy, joella}@digitalworlds.ufl.edu Up to this point, remote collaboration over high-speed networks has typically consisted of talking heads occasionally embellished with PowerPoint presentations. We herein present a working system that has enabled a much higher level of interaction and collaboration. Using the Access Grid and Internet2, a ten-minute work of modern dance and music was created in three days with a team of more than 125 artists and engineers in four locations across two continents, and subsequently premiered at the global SuperComputing Conference. Creative Collaboration with Emergent Technologies The Access Grid (AG) 1 is an ensemble of resources used to support human interaction across the Internet. It consists of multimedia displays, multipoint bi-directional audio, cameras, and shared spaces for presentations and interactions. It supports large-scale distributed meetings, collaborative work sessions, seminars, lectures, tutorials and training. The system is designed for group-to-group communication (differentiating it from desktop to desktop based tools that focus on individual communication). Thus, the environment enables both formal and informal group interactions. Access Grid nodes are "designed spaces" that explicitly contain the high-end audio and visual technology needed to provide a high-quality compelling user experience. Currently, there are well over one hundred fully operational sites internationally. The majority of these are used for face-to-face teleconferencing, whether one-to-one, one-to-many or group-to-group. While there have been networked performing arts projects [1], the use of AG multipoint technology in the performing arts has not yet been widely deployed. In an effort to create a truly interdisciplinary and collaborative work of art with tools including the AG, the Digital Worlds Institute 2 designed and facilitated a collaborative environment where artists and engineers interacted at a high level in real time across North and 1 http://www.accessgrid.org 2 http://www.digitalworlds.ufl.edu
South America. The resultant project, entitled Dancing Beyond Boundaries 3, joined master percussionists in Brazil, musicians and dancers at the University of Florida, a choreographer and dancers in Minneapolis and dancers with a three-walled display at the 2001 SuperComputing Conference in Denver (SC2001) 4. A key feature of this system is the ability of each of the AG nodes (locations) to capture up to four video images simultaneously and transmit them to all of the other collaborating locations (Figure 1). This multi-cast capability not only enabled each of the artists to see and speak to each other (as in a standard teleconference) but to actually create original music and dance as fluidly as if all of the performers were in the same physical space. Yet the participants were spread across North and South America, and never graced the same stage except in the virtual studio. FIGURE 1. A Basic Access Grid (AG) node configuration 5 The notion of distributed collaboration has inspired a number of excellent papers and projects that utilize the potential for iterated design decisions to be made by participants at a distance, including recent work by Sawhney and researchers at the MIT Media Lab [2.]. Dancing Beyond Boundaries (DBB) used the multiple perspectives captured at each of the AG nodes as a palette of possibilities for the choreographer to create not only the dance, but choose the shots and angles displayed in the final performance of the piece on the last day of SC2001. In addition to the AG technologies utilized, a number of other tools were used including Internet Protocol (IP) 3 http://www.digitalworlds.ufl.edu/sc2001 4 http://www.sc2001.org/ 5 Schematic found at http://aspen.ucs.indiana.edu/users/ggundez/accessgrid.html
phones and multiple Minerva video codecs 6. While the majority of the connectivity was provided by the Internet2 7, it is notable that the commodity Internet was used for a portion of the connection between Brazil and Florida, and latency issues were not a problem, much to the surprise of the network engineers. Engineers and Artists: Strange Bedfellows? To facilitate the distributed collaboration of DBB, a host of diverse specialists were called upon including network engineers, computer scientists and engineers, audio and video engineers, television producers and directors, camera operators, digital animators, theater designers, a composer and choreographer, five musicians and over 40 dancers. In all, more than 125 people across North and South America collaborated over a period of four days to create a ten-minute work of modern dance and world music from scratch. It is not typical that engineers and artists work together on a daily basis, whether in the same city or across two continents. Indeed, most of the participants in the project had never met or seen each other before the first day of rehearsals, and the South American musicians spoke only Portugese, while all but one of the North American artists spoke English. Despite these initial challenges, we found that the AG and related technologies became almost transparent, and that the artists were able to articulate their needs to the engineers, and the engineers were able to reciprocate with descriptions of what the system could and could not do. Because of their familiarity of video cameras and projections, microphones, headphones and telephones, the artists were able to concentrate on making the actual content of the piece, while the engineers monitored the system to maintain the best possible quality of service. The choreographer was able to see nearly a dozen different camera shots, and to articulate to the video producers when and where the camera should move as the piece unfolded. As a result, the selected camera shots were switched into the high quality Minerva connection and appeared in full screen, full motion video on the large display system at SC2001. The biggest technical problems encountered were the occasional loss of network connectivity on the conference floor, and the intermittent audio feeds. And we found that we had underestimated the number of necessary communication back channels (i.e. background communications between participants including the camera operators, engineers, choreographer and assistants, musicians, dancers and video producers.) However, for a system of this complexity, the technologies worked well in facilitating the needs of the artists as they created the piece over the course of three days and at a distance of some 5,000 miles of separation (Figure 2). 6 http://www.minervanetworks.com/ 7 http://www.internet2.org
FIGURE 2. A Schematic of the Dancing Beyond Boundaries Distributed Collaboration
Where does a Distributed Performance Piece Exist? An interesting question resulting from this preliminary work is: Where does one best experience this piece? Although the display system on the floor of SC 2001 provided a featured location with three live dancers and the images of the remote artists via telepresence, each of the remote locations also had live performers and audio and video of the other locations. Thus people in each city had a different perspective of the exact same event. This is, however, true of each audience member in a traditional performance venue (i.e. large theater, symphony hall, opera, etc.), the primary difference being that the traditional event happens in one physical space. Dancing Beyond Boundaries happened in four physical spaces, each with their own audience, plus the fifth space of the Internet, where people from around the world watched with a host of different perspectives. So where did this piece happen? One could argue the piece really happened in the interconnection between the four physical locations, and that this connectivity added a large number of fifth spaces as people experienced it over the Internet. Thus the combination of multi-modal information from the four nodes created a virtual studio [3 ] that could also be termed a distributed virtual environment, though perhaps not in the usual sense [4]. It may also be possible to consider the experience at each of the nodes as a kind of mixed reality [5] or even augmented reality [6], although again not in the traditional sense of head mounted displays and computer-generated imagery. This piece had a high level of real-time interactivity, and the telepresence generated by multicast perspectives accompanying the live artists created a distributed collaborative environment where the organic and the electronic points of presence interacted seamlessly. Future Research Trajectories While a high level of collaboration between engineers and performing artists has been demonstrated, we intend to augment this approach with the addition of real-time computer graphics and multi-channel audio. In the initial iteration of DBB, an audience member could only discern the location of the remote performers by reading descriptive text in the upper left-hand corner of a video display box. And both the North and South American audio were combined into a composite stereo mix. While this approach ameliorated the physical separation between performers, it did not help the audience to visualize the intercontinental scope of the project. We plan to experiment with audio localization as a means of signifying the different physical locations of remote performers, and real-time video/graphics compositing to further enhance the perceived physical location of the distributed participants. Thus the audience will be able to intuitively discern the variety of remote locations while appreciating the quality of the artists real-time interaction.
Acknowledgments The authors wish to recognize the contributions of a number of individuals and institutions, including Mestre Boca, Dr. Leonardo Mendez and his lab at the University of Campinas, Brazil, Shapiro and Smith and their dancers and technicians at the University of Minnesota, Minneapolis, Welson Tremura, Kelly Cawthon, K. Kay Spengler, John Polefko, Greg Goddard, Dave Pokorney, Frank Counts and the dancers and technicians at the University of Florida. References: [1.] Malloch and Owen. 1999.Networking Moving Images: the work of the Performing Arts Data Service, University of Glascow. Journal of the Moving Image Society, Vol 81 No 8 [2] Sawhney, Griffith, Maguire and Prestero. 2001. ThinkCycle : Sharing Distributed Design Knowledge for Open Collaborative Design. Draft Paper for TechKnowLogia Journal. http://www.thinkcycle.org/tc-filesystem/download/collaborative_opensource_design_platforms/thinkcycle:_sharing_distributed_design_knowledge_for_ope n_collaborative_design/tc_techknowlogia_draft.pdf?version_id=21716 [3] Cascone, 2002. NEXT@CNN. Transcript of television feature broadcast internationally April 27/28, 2002. http://www.cnn.com/transcripts/0204/27/nac.00.html [4] Strauss, Fleischmann, et al. 1999. Staging the space of mixed reality: Reconsidering the concept of a multi user environment. Paper from the German National Research Centre for Information Technology. http://www.c-lab.de/vrml99/vrml99papers/strauss.pdf [5] Billinghurst, Kato. 1999. Collaborative Mixed Reality. Proceedings of the First International Symposium of Mixed Reality. ISMR Mixed Reality- Merging Real and Virtual Worlds, pp. 261-284. Berlin: Springer Verlag. http://www.hitl.washington.edu/publications/r-98-36/r-98-36.pdf [6] Drascic and Milgram. 1996. Perceptual Issues in Augmented Reality. SPIE Volume 2653: Stereoscopic Displays and Virtual Reality Systems, pp 123-134. http://gypsy.rose.utoronto.ca/people/david_dir/spie96/spie96.full.html