Human-Bandwidth and the Design of Internet2 Interfaces: Human Factors and Psychosocial Challenge

Transcription

1 Human-Bandwidth and the Design of Internet2 Interfaces: Human Factors and Psychosocial Challenge Frank Biocca Ameritech Professor of Telecommunication Director, M.I.N.D. Labs Michigan State University Abstract This article poses the question: What interface design techniques can amplify human bandwidth to make optimal use of the increased transmission and information bandwidth of Internet2? Emerging design approaches are identified. A set of design mandates growing out of users experiences with Internet1 has become the promise of Internet2. Three challenges facing Internet2 interface designers are introduced: the human-bandwidth challenge, the interface optimization challenge, and the psychosocial environment challenge. Key human-factors and human computer interaction design issues associated with teleimmersive interfaces and networked social computing are reviewed. 1. Introduction Looked at from the point of view of users, the purpose of a transmission system, including the very high-bandwidth systems envisioned for Internet2, is to connect minds to vast pools of information, and to connect minds to each other. The communication interface is the place where minds and bodies, our senses and motor systems, are plugged into the telecommunication system via elaborate representational systems that structure the interface and information. The pipes, routers, and fibers of the transmission systems and the sensors and effectors of communication interfaces are two sides of the telecommunication system. Both are part of the vision for Internet2. Dramatic changes in transmission bandwidth capacity open up numerous possibilities for changes in the design, interactivity, and the sociability of the interface. It is likely that dramatic increases in transmission will fuel significant changes in the interface. High-information bandwidth immersive interfaces such as immersive virtual reality, room VR (CAVES), and wide-screen panoramic VR systems (e.g., SGI Reality Center) give us a glimpse as to what kind of interface can make use of higher-bandwidth. But it is fair to say that no current interface exists that can make optimal use of the 2.5 Gigabit capacity of early Internet2 experiments or of terabit transmission systems under research and development by DARPA and other US institutions. This Internet2 white paper outlines some of the human factors challenges to creating interfaces that can make optimal use of increased information bandwidth of Internet2. To make optimal use of the increased bandwidth, these interfaces would not only have to improve human-to-machine communication, but support intensive collaboration and knowledge networking among large, distributed, and heterogeneous communities of networked users. This Internet2 white paper reviews the human factors opportunities, challenges, and trends in the design of interface infrastructures. It addresses the following key issues: The promise of Internet2: Human factors and the vision of a high-bandwidth interface Human factors challenges in the design of Interfaces for Internet2 Emerging Approaches to Interface Design for Internet2 Potential psychosocial issues associated with advanced interface usage in Internet The promise of Internet2: Visions of the interface 67

2 The success of Internet1 has helped sow the seeds for Internet2. U.S. research and development (R&D) in computing, communications, and information technologies has enabled unprecedented scientific and engineering advances, transforming workplace products and processes, and benefiting society and individuals [17, p. 1]. The dramatic and quick success of Internet1 has helped fuel the desire for greater information bandwidth. The key flaw in Internet 1 can be summarized in one sentence: Internet1 information transmission speeds lag behind the information processing capacity of users. As more and more users experienced the Internet in the late 1990s, more were exposed to frustration with the delays in moving from page to page or downloading multimedia content. Because interfaces and transmission systems are linked, interfaces have been designed to carry less information than older transmission systems (e.g., physical distribution through CD) to fit limitations in transmission. The limitations in download time drive interface guidelines as designers struggle to design pages not for maximum, psychological impact or communication effectiveness, but for minimal download times. Users expectations of what the interface might achieve have risen as they have experienced attempts to overcome interface limitations imposed by bandwidth restrictions. For example, the implementation of sensory rich data types like the current generation streaming video, spatialized sound, 3D graphics, and collaborative multi-user environments have given users a taste for what might be possible with more bandwidth. These interface issues and the potential range of applications possible in a higher bandwidth environment have helped drive the business and institutional imperative to develop network infrastructure that can increase transmission bandwidth. The technical goals and specifications for Internet2 and other high bandwidth initiatives are being socially defined by research, discussion, and actions in the government, business, academic, and user communities. How do we satisfy these user-driven design mandates? What human factors challenges must be addressed in reaching these goals? The input of social scientists, interface designers, interface manufacturers, policy makers, and research funders can help layout paths to reach these goals and mobilize research, financial, and policy resources to achieve them. This social process helps coordinate the behavior of many individuals and organizations in the major effort to dramatically increase the capability of the nation s telecommunication infrastructure. The visions of the future interfaces that emerge from this institutional interaction define the ultimate destination of this national engineering effort and can have a strong if not formative effect on the design process by: Defining the future goals and targets Influencing the perceived value of different design paths, Affecting the distribution of design funds, and Preparing the ground for social acceptance and technology diffusion Table 1: Interface design mandates resulting from user needs from increased transmission bandwidth User Need or Desire Need to reduce delays in real time interaction and information retrieval Desire for more sensory immersive data types Need for greater remote sensing and monitoring Need for high bandwidth real time collaboration Need for more efficient allocation of bit rates to real time and timecritical applications Design Mandate Improved interactivity and responsiveness Improved sensorimotor bandwidth Improved sense of telepresence Improved social presence Prioritized bandwidth allocation This white paper and the discussion surrounding the Internet2 Sociotechnical summit seek to contribute background to this discussion. A number of visions of what the next generation high bandwidth telecommunication system will do are being articulated and circulated. Most of these visions link the engineering specifications of higher bandwidth and new protocols to a description of potential applications and benefits. These visions of applications help define expectations and requirements for interfaces for Internet2. The following features emerge in a number of discussions of Internet2 interfaces: Teleimmersion Ubiquitous Video Streaming Collaborative social virtual environments 68

3 Remote Sensing Interfaces that support telepresence and high levels of process control Immersive Digital Libraries and Search Techniques Advanced, interactive visualizations and multi-sensory simulations These are described in the sections that follow. 2. Three fundamental challenges for the design of Internet2 interfaces There are three issues that urge the humancomputer interaction community to explore human factors and interface designs that are more suited to high bandwidth environments The human bandwidth challenge: Can transmission bandwidth exceed human bandwidth for communication or information processing? The currency that is part of the discussion of Internet2 is bandwidth. Because our focus is the user and interface, we can introduce the idea of human-bandwidth, which is often considered in the design of interface input and output devices. We can define human bandwidth as the optimal capacity of the human mind and body to process a full range of sensory information and to initiate the body s full motor range of communication actions and information and object manipulations. In most situations, Internet1 transmission bandwidth and interfaces have lagged significantly behind human bandwidth. In Internet1, the major bottleneck in information processing has been the limitations of transmission systems. At the end of 1990s, the range of available content was large, but information delivery was slow, modest in sensory bandwidth, limited in responsiveness, and in ability to sense and transmit complex motor input and communication behavior. The user s attention can wilt in a system that does not yet occupy the user s full capacity to absorb information in real time. The goal of Internet2 and high bandwidth initiatives is to eventually increase transmission bandwidth by at least 1000 times. So we may have an interesting change in the nature of the problem. As transmission systems increase their capacity by a thousand fold and higher, the capacity for the human user to efficiently find, use, and absorb the potential flow of information may be exceeded. The user s limited ability to absorb information in real time may become the bottleneck in the telecommunication system. How might Internet2 content challenge the user s ability to process information? Content is likely to be highly processed and information dense. Consider the demands of real time processing on the human user. Judging from attention, memory, and information processing studies of video, film, and multimedia interfaces, Internet2 information density and presentation rates may easily overwhelm the user s ability to fully filter and absorb information in real time. The sheer increased volume of information available on the expanding Internet will compound demands on the user s ability to absorb information in real time. A wide screen or immersive interface may be able to present to the user multiple edited video streams, massive 3D data environments, multi-user environments, etc. Increasingly, the transmission system may support real time data rates that may stress or exceed the processing capacity of the human user. On the other hand, there is reason to doubt that human bandwidth will be a problem. For example, we can point to the real world environment, such as a city street in New York. Real environments like this are teaming with information: a constant flow of faces, thousands of signs, video streams in windows, constant demands on navigation, etc. People seem to be able to navigate these environments by filtering and selectively attending to relevant information. While this does point to the ability of individuals to absorb information, it may be a better analogy for experience foraging in TV channel surfing than for information processing in most Internet2 environments. The specter of information overload is more vivid when we consider a user in an environment for which there is greater task responsibility and more real time processing. An analogy is a pilot in an advanced fighter plane. Information flows in these environments are quite critical. Controls and displays compete for the user s attention. Filtering of read outs from key equipment systems can be difficult, confusing, and detrimental to performance. Information may be condensed. The user s ability to find, process, and absorb information may be severely challenged by a system that can deliver a gushing pipeline of information from a reservoir of content that may include most of the world s videos, images, text 69

4 reports, 3D models, audio records, and other pools of data types. One scenario portends a user lost, overwhelmed, and over stimulated in the emerging, worldwide Alexandrian library that will be Internet2. I2 interface designs must somehow help guide the user floating on this turbulent sea of information that the continued growth of data speed and digital content promises The interface optimization challenge: How can a high bandwidth interface optimally be designed to connect the user s mind and body to a high-speed, interactive, high-bandwidth telecommunication system? The most common interface at the end of the 1990s was a small computer monitor displaying information in a windows-based metaphor controlled by a keyboard and mouse. The keyboard and white paper screens are obvious technological descendants of the typewriter; they are clearly designed for word processing. The introduction of the mouse in the 1960s was a clear step in the direction of direct manipulation of information objects. Through the mouse, the hand was reaching into the information space. But adoption of more powerful or task specific I/O devices has been limited (e.g., drawing pads, data gloves, etc.). The sensory range of the common Internet1 interface is limited. The screen is very small, so only a small part of the visual field is used to carry information that may include models of building, information landscapes, etc. The addition of quality audio and streaming video has helped move the common computer interface into a position that sometimes matches but often lags behind the display richness of even the average non-interactive medium such as the radio, television set, and film. Is the current interface optimal for a large range of human interaction in a high bandwidth environment? Consider some of the tasks that could be better supported by the Internet2 interface: Monitoring a distant environment: plant or environment surveillance Engineering, plant design, and architecture: Building a product, building, or other design Teleconferencing: Mediated face-to-face interaction in work groups or meetings Medical application: Joint consultation with medical teams and patients and 3D CAT, MRI, and other date Casual social interaction, gaming, etc. If we consider the broad range of tasks and social environments potentially supported by networked media, it appears that the common interface is far less than optimal. But is it adequate? Here, there is less agreement, but the preponderance of opinion appears to be no. It is likely that most commonly used interfaces are inadequate to make use of the human communication and object manipulation in high bandwidth environments. It also suggests that if the current interfaces continue to be the most commonly diffused, then the human communication opportunities provided by a high bandwidth telecommunication system may not be realized The psychosocial environmental challenge: What are the psychological and social consequences of long-term use and human interaction via a teleimmersive, high-bandwidth telecommunication system? The interfaces and networked environments of Internet2 will be more than simple hardware and information and more than just a communication system. Internet2 will also be a psychosocial environment that will be the locus of much human attention, communication, and action. People born into the television generation were likely to spend as much as seven years of their life watching television [9]. The interface that connects individuals to Internet2 may well become what Allan Kay called a metamedium : a medium that mimics and displaces the amount of time we spend with the range of existing communication appliances including the present computer, television, radio, newspaper, book, etc. If we add up how much time individuals spend with each of these older interfaces, we come to a chilling realization. Individuals born into the world of Internet2 interfaces may come to spend as much as 20 years of their waking lives connected to the interface that links them to the information network. This presents us with the psychosocial environment challenge: What are the psychological and social consequences of long-term use and human interaction via a teleimmersive, high-bandwidth telecommunication system? 70

5 2.4. The challenges: A summary The challenge to make better use of human information processing bandwidth and the realization that current common interface is inadequate to make use of transmission bandwidth gives rise to what may emerge as the fundamental interface challenge for Internet2: What interface design techniques can amplify human bandwidth to make optimal use of the increased transmission and information bandwidth of Internet2? 3. Emerging human factors research and design issues for Internet2 Below we review a number of the human factors research issues and interface techniques related to Internet2 interfaces. These interface techniques are emerging to meet the fundamental challenge of amplifying human bandwidth to make optimal use of increased information and transmission bandwidth of Internet Teleimmersion: attempting to match human bandwidth to telecommunication bandwidth One of the goals for Internet2 is to support high levels of human collaboration via Teleimmersive applications. Increased bandwidth for teleimmersion has been identified as one of the key areas for Internet2 application development. In this section we discuss what teleimmersive interfaces are, how they might assist users of Internet2 applications, and some interface design issues related to teleimmersion Designing more natural telecommunication interfaces that better support the processes of human body and mind Researchers in a number of national interface design labs have called for more natural interfaces. For example, calls from DARPA and NSF also refer to natural interfaces. Human computer interaction technology focus [es] on design methods and enables technology for more natural interaction between people and computers [17, p. 17]. But what is a natural interaction with teleimmersive environments? Natural interaction in an interface can be defined as interaction techniques that simulate in the virtual environment the ways users interact with the physical world: how users (1) move and navigate through physical space (2) interact with objects in the physical environment, and (3) interact with humans and other intelligent beings (e.g., animals, etc.). Interfaces with more natural interaction support more of the processes by which the human body and mind interact with the natural world. To achieve more natural interaction, an interface must be built upon knowledge of human perception, cognition, action, and communication. Learning how to better connect the human body to the virtual environments of Internet2 is an essential route to making interfaces simulate how the body interacts with the physical environment. The body is the primordial communication medium. To communicate information to the mind of a user, all interfaces must transmit information to the body. All communication to and from the human mind must be filtered through the human senses or expressed via human motor channels: seeing a scene, expressing emotion through facial expressions, etc. The senses and motor channels make use of evolutionary driven affordances in the environment as well as socially constructed communication codes. The body, therefore, must inevitably be a major part of any future communication interface. Do the current interfaces communicate naturally along the full bandwidth of human communication capabilities? The present generation of telecommunication interfaces vary dramatically in the range of human bandwidth used to communicate information to a user. For example, the traditional telephone handset uses far less of the user s senses and motor systems than does a video teleconferencing system or an immersive virtual reality system. One approach to expanding human ability to communicate and absorb information might use more of the human body s bandwidth to make better use of transmission bandwidth and must somehow better connect the human body to information carried by the high bandwidth transmission systems. Research that adopts this approach to interface design tends to explore how to use more of the human sensory and motor channels as a way of communicating more information in and out of the user s mind. The fast moving evolution of media has created various families of interfaces that use various combinations of input and output devices to connect the interface to the human body. The family of virtual reality [3, 5] and augmented reality interfaces [1]is an example. 71

6 What is a teleimmersive interface? Towards a definition. One of the goals of Internet2 is to support teleimmersive interfaces. But what are teleimmersive interfaces? It might be helpful to a discussion of Internet2 interfaces to define what this means. Teleimmersive interfaces surround more of the user s body, their senses and motor channels, in stimulation and interaction with the virtual environment. The word tele in teleimmersive points to the fact that the individuals are immersed in the environment that is distant, carried by the telecommunication system and different from the immediate physical environment in which the computer interface is located. This environment can be a distant physical environment or a purely virtual one. The immersive part of teleimmersive refers to the degree to which the body is immersed in the interface. The trend in the development of interfaces is towards increasingly embodying the user in the interface: (1) connecting the interface to the body of the user, and (2) representing the user s body and actions inside 2D or 3D virtual environments. Immersion is believed to help make the user feel more present in the distant environment and to provide more natural, multisensory, and multi-modality interaction with the virtual objects. The degree to which an interface is teleimmersive can be defined as a combination of the following: Level of sensory immersion Sensory immersion is defined as the degree to which the senses are immersed with stimuli originating in the virtual environment rather than the physical environment. Specifically, this is composed of: SE 1: The number of sensory channels engaged by the virtual environment. Media, like pictures, communicate information about a virtual environment using only one sensory channel. Increasingly embodied media include visual, aural, tactile, nasal, and proprioceptive stimuli. SE 2: Sensory fidelity of displays for each sensory channel. Each sensory channel can be defined by the static and dynamic energy bandwidth of stimulation that the channel can sense or absorb. Interfaces for each sensory channel, for example, a computer monitor or cinemascope screen, vary greatly in how much of the sensory bandwidth they can address. This is most commonly referred to as the sensory fidelity of the interface (See [5] for a specification of the capacity of each sensorimotor channel viewed from a VR engineering perspective). SE 3: The level of relative immersion in the virtual as opposed to the physical environment. The relation of the sensory channels to the medium cannot be considered in isolation of evaluating how the senses are connected to the physical environment. In an effort to fully immerse the user s communication bandwidth, the capacity of the senses engaged by the system must be immersed in the representation of the virtual world. Saturation of a sensory channel is defined as the percentage of the channel occupied by stimuli from the virtual as opposed to the physical environment. For example, when a user looks at a typical computer monitor only a fraction of the visual field is occupied by stimuli from the virtual environment. But this is beginning to change as monitors are becoming progressively larger, and VR head mounted displays are moving to larger fields of view with the ambition of saturating the field-of-view of the user with the virtual environment. The use of display systems during communication often also includes the suppression of stimuli from the physical environment. A good example is the movie theater. Dimming the lights so that the screen (the virtual environment) is dominant diminishes visual information from the physical environment. Sound volume and social rules about making noise suppress sound from the ambient environment. Soft comfortable seats suppress awareness of the haptic channel. The physical environment is set up to allow the users to immerse their senses in the virtual environment of the movie screen. Communication flows to senses outside of the virtual environment are decreased and the sensory stimuli originating from the virtual environment dominates our perception The level of immersion of the body s actions, expressions, and states We act upon the world and communicate by moving our bodies: smiling, grabbing, speaking, gesturing, etc. Increasingly, sensors on the computer incorporate more of the body s motion and actions into the interface [3,5]. The computer increasingly maps the body and its actions. Historically, the mapping of the body into the computer began with the mouse, which recorded the 2D movement of the user s hand. 72

7 This was different and more natural than sensing actions on the keyboard, which is primarily a less natural symbolic input device for textual conversation with the computer. In the physical world we do not carry out conversations with keyboards. ME 1:The number of user bodily actions, expressions, and autonomic states monitored by the interface (i.e., the degree to which motor and autonomic channels are sensed) The user is usually tracked by various sensors and input devices such as joysticks, head trackers, eye trackers, facial motion systems, etc. ME2: Resolution of body sensors. Sensors, like displays, capture finer and finer resolutions of body motion, user action, and physiological activity Sensorimotor coordination and integration One of the most important factors in creating teleimmersion is the way human senses and motor channels are coordinated and integrated by the interface. As Held and Durlach (1991) succinctly point out, "the information received through all channels should describe the same objective world" (p. 110). Sensorimotor coordination and integration refers to the degree to which sensations in the sensory channels (e.g., seeing your hand move in the virtual environment) is coordinated with motor action (e.g., the movement of your hand in the physical environment). Changes in body position should map consistently and coherently to changes in sensory feedback. The coordination of the sensory and motor channels is the essence of feedback, especially the kind of feedback we experience in our natural, everyday interaction with the physical environment. For example, the presence of lag in immersive virtual reality systems between motor movement and sensory feedback is a significant source of simulation sickness and decrements in human performance [2, 8] HCI and psychological research issues related to Teleimmersion There are a number of interesting issues in human computer interaction and interface design relevant to Internet2. We will not address all of them here. Rather, we will focus on those that bear some light on what is the fundamental challenge for interfaces designed for broadband communication and Internet2: What interface design techniques can amplify human bandwidth to make optimal use of the increased transmission and information bandwidth of Internet2? Increasing interface bandwidth by immersing the human body: Research and development of a broader range of high fidelity body displays and sensors The general assumption of immersive interfaces is that greater immersion increases the amount of sensory and motor bandwidth that can be used in human-to-machine communication and human-to-human mediated communication. As we mentioned above, the most widely used interfaces for Internet1 fall far short of making use of the full capabilities of human sensorimotor systems. For example, screens are typically small, display resolutions at less than human vision, and display information in only a fraction of the visual field Research on human-factors of multisensory displays Can multisensory displays increase the Internet2 user s ability to process information and make use of a greater amount of human bandwidth? Most would agree that adding video to audio allows the user to process more information about a remote scene. Should other sensory interfaces become common in Internet2? Internet2 and broadband technologies can potentially offer enough bandwidth to stream a broad spectrum of sensory data to distributed end users who have interface output devices (effectors) that can display the information. There is some evidence that tactile displays can provide addition information about the spatial structure of data and touch the data, including representations of things that are too small or too large to grab hold of in the real world. For example, tactile displays have been used to explore data visualizations and molecular modeling. Tactile feedback can be very valuable in the control of remote equipment, virtual surgery, and other likely Internet2 applications where motor control is desired. For example, Internet2 will allow for more telepresence applications where an operator may control a robotic device at a distant location, remote surgery, machine maintenance, etc. Reliable, robust, and timely tactile feedback may be necessary to efficiently manipulate an object. For example, it might allow the user more efficient manipulation of objects such as turning a screw, handling a scalpel, picking up a delicate object 73

8 such as a computer board, or systematically squeezing pliable objects like a tube of adhesive, etc. There is some evidence that haptic feedback may be valuable in visualization applications. The sense of touch can provide the user with more sensory information about 3D form. A model application area is the Nanotechnology project at UNC. It allows users to feel the forces or form of very small objects such as molecular structures, the nanometer surface of gold atoms, and other micro or macro shapes. In direct manipulation applications the user can increase their sense of the spatial structure as well as the properties of an object or material Research on increased display fidelity and its value to information processing. The sensory fidelity of most displays does not allow the user to make use of the full range of perceptual cues available during observation in the physical environment. Even the highest fidelity displays fall far short of the full range and capacity of the senses. Visual and auditory displays are the most mature, but the highest fidelity displays (e.g., 3D IMAX) are often not interactive, by no means ubiquitous, and are not connected to advanced transmission systems. Is higher fidelity needed to improve human performance in Internet2 applications? The human performance value of increased fidelity may vary with user tasks and applications, and is not likely to improve performance across all domains and across various users. We can derive a general principle and call it selective fidelity. For example, determining the color of a streetlight in an image may not require a highresolution image. But for some applications, such as some highly detailed scientific visualizations or radiology images, more fidelity may be essential. A significant value of teleimmersive interfaces and more natural interaction techniques may come from leveraging the powerful capabilities of the human perceptual system by manipulating the environment to maximize the ways in which the senses experience the data. This is especially true in an interactive system where the user can directly manipulate the data and observe subtle changes in model. For example, the direct hands on manipulation of the 3D model of fluid flow through a plastics mold may allows the user s visual system to detect patterns in the data that could not be detected by looking at the raw data, summary tables, or other non-iconic, and noninteractive models. Increasing the range and fidelity of displays for Internet2 may give users greater ability to use and manipulate the certain increase in the volume and density of networked information. High fidelity, interactive visualization systems, can leverage the power of the perceptual system to process information in parallel. Users abilities to extract value from data can be enhanced through: Pattern perception (e.g., detect that a series of objects appear to form a line) Anomaly detection (e.g., find the letter T in a field of B s) Trend extraction (e.g., observe that a two variable graph based on 1,000,000 observations is going up over time) Let us take a very simple example. The visual angle of a display is one aspect of visual fidelity. The yardstick for visual fidelity is unmediated vision. Typical visual displays occupy only a small part of the visual field, so they use only a small part of the spatial range of active human vision. Consider a possible Internet2 display similar to the modern monitor but only bigger, for example a high resolution, wall size display occupying 180 degrees. By moving their eyes over this large display, a user could more easily monitor and process multiple video streams, correlate dynamic graphs with a video observation, or monitor the minute facial expressions of multiple teleconferencing participants involved in a negotiation Increasing sensorimotor interaction The level of teleimmersion and, potentially, the cognitive amplifying power of the interface may call for a significant improvement in interfaces. Some of the greatest value may come from interface designs that improve not only fidelity by itself, but link it to the motor actions of a user. Research with virtual reality systems suggests that fidelity by itself may not be the key to increases in human performance, but rather sensorimotor coordination. The human ability to make use of affordances, for example in visual flow, comes from the link between users movement and action and the experience of sensory feedback during the actions Increasing computer sensing of human motion and action Gestures, facial expressions, eye movements, leaps, and struts - the moving body carries tremendous information about the user s mental and emotional states, intentions, and their likely 74

9 actions, especially in 3D virtual environments. Very little of this information is used by the typical Internet1 interface although much of this data can be sensed and used in lab-based virtual reality and other interfaces. But teleimmersive applications may require more information about the user s body to control and display information in 3D virtual environments. Body motions, such as reaching, pointing, and staring are a source of information for applications and for other users Increasing a sense of presence in the I2 virtual environment: Research on the effect of teleimmersive interfaces on telepresence and human performance Transmission systems collapse space. From Web cams to television news, users experience human actions in distant locations. High-end display systems (e.g., IMAX film) can give the user the experience of being in another remote place or in a virtual space. Space is collapsed by transmitting sensory impressions to the remote user, e.g., video, sounds, tactile forces, etc. When an interface truly transmits the users senses and remote actions to a distant location, the user may be temporarily unaware of the medium, and may feel as if they are in the remote location rather than in the room with the interface. In a way, the interface may psychologically disappear. This psychological sensation of being there is called telepresence, or simply presence. If Internet2 is to deliver more sensory information to the user, we should expect users to report higher levels of presence. This higher level of presence may be related to improved human performance and greater satisfaction with the mediated environments carried by Internet2. But the psychology of presence is not well understood. Nor is it certain that increasing the sense of presence will help human performance in all settings and tasks. But, nonetheless, it appears that one of the fundamental reasons for increased transmission bandwidth in Internet2 is to carry more information about a distant location, a remote collaborator, etc. so that the user feels as if they are there in the remote space with the remote other. This would suggest that more research is needed in understanding the psychology of presence, because it appears to be a fundamental - if not well understood - goal of communication systems Just in time bandwidth: Research on the modulated used of bandwidth Internet2 offers more bandwidth, but is high bandwidth always necessary? Internet2 high bandwidth interfaces may come to offer a wide variety of sensory data and immersive interfaces. It is fair to assume that maximum or even higher bandwidth transmission and interface capability is not necessary at all times, even if this were possible. When does a user need more bandwidth? What tasks require it? What kinds of collaborative communication? What kinds of content most benefit for higher fidelity, more immersive interaction, and maximum transmission bandwidth? Can better measurement of the user s behavior, estimates of the cognitive task, or physiological state tell the system when to decrease or increase information flow even in a teleimmersive application? There are no well-researched guidelines on when and for what applications high bandwidth transmission and teleimmersive interfaces are most valuable in improving or supporting human performance. HCI studies in this area could provide some guidelines but also provide insight into possible bandwidth modulators that might increase or decrease bandwidth allocation dynamically based on user tasks or cognitive states Multi-sensory representational techniques: Research on new interactive multisensory techniques to represent data and abstract information Most information carried by media is not raw; it is processed and organized by representational systems and communication conventions. Words are chosen. Video is edited. Representational conventions allow information to abstracted, condensed, and organized. Recent developments in computer graphics have shown the power of visualizations to make use of the ability of the human visual system to detect and process visual patterns. What representational systems can best make use of the increased sensory and motor bandwidth potentially allowed by Internet2? With Internet2 a number of issues are potentially raised regarding representational systems. Leveraging spatial cognition: Virtual environments, especially 3D virtual environments, present the user with a 3D space. How can the spatial properties of a teleimmersive virtual environment be leveraged to make greater use of human spatial cognition 75

10 for information navigation, representation, and memory? Leveraging multisensory displays: If multisensory, tactile stimuli are added to teleimmersive systems, how can information be best represented to the tactile sense? Can we do more than represent just tactile replicas of the physical environment? Can we develop an abstract representational system that can do for the tactile sense what visualization and sonification do for representing abstract information to the visual and auditory senses? Leveraging the psychology of interactive narratives and rituals: For thousands of years, information has been transmitted and coordinated in groups via narratives and participatory rituals. There are as yet few examples of powerful interactive narratives that have the same psychological impact as film and book narratives. The medium is still young. Research and development is needed to support the exploration of the use of participatory, interactive networked narratives and rituals that facilitate the communication of values, norms, goals, and roles in networked organizations Sensing non-intentional user communication and user states: Research on the use of autonomic responses and behavior mapping for affective and adaptive computing When individuals communicate information to one another, they may monitor the receiver s body to evaluate how the individual is processing the information, for example, their level of attention, signs of agreement, evidence of emotional reactions such as anger, puzzlement, etc. When the computer interface is assisting the user in absorbing information from a high bandwidth teleimmersive environment, the interface need not be limited to using immediately observable behavior. Autonomic responses such as heart rate, blood pressure, skin conductance, can potentially be used to evaluate the user s affective state. Output from the processing of this information might be used to guide interface responses such as information flow and rate or the behavior of an agent. While such interface processing have been proposed [13] and limited experimentation exists, there are not enough examples of effective use of this data in interfaces that support of high level cognitive processing in the user, guide interface modulation of information flow and presentation, or amplify interpersonal communication of affective states and moods. Such information might be used in Internet2 applications to modulate information presentation styles and bandwidth to the user or to communicate individual or groups cognitive states in collaborative teleimmersive applications Embedding interpersonal communication techniques into the interface: Research on anthropomorphic agents Humans are hard wired to interact with other humans. No manual is necessary. Interpersonal communication is highly practiced. It is perhaps not surprising that since the advent of the computer, the most common vision of the interface has been a vision of the intelligent, humanoid other either in the form of a talking voice, a typing intelligence, or an animated robotic entity. Few interfaces take this form. But recent research has shown that individuals tend to use interpersonal communication models in their interaction with their computers even though such interfaces many not be designed as social [15]. Can anthropomorphic agents provide a means of taming and handling the increased information flow possible in Internet2? The use of anthropomorphic agents represents a potentially powerful means of structuring interaction in various applications, acting as proxies for the individual user in networked processes and interaction, and in representing networked states. A number of research areas can potentially support interface development for Internet2: Interpersonal communication techniques to represent bandwidth requests and allocations. The use of anthropomorphic forms to represent application and network states The use of agent interactions and morphology to visualize more abstract information (e.g., extension of the use of C- faces in statistical graphs) Monitoring the body and mind for unintended psychophysical effects: Research on the short term and long term psychosocial effects of extended use of teleimmersive interfaces There has been much speculation and some research on the short term and long-term psychosocial effects of specific media. Some of it deals with the larger social phenomena associated with adoption of a medium (for example, the effects of Internet1 on economic organizations) or the effect of content categories such as violence or pornography on social 76

11 behavior. While these are important, they do not pertain specifically to the effects of the structure of the interface: input and output devices, the logic of the interaction, information presentation and organization. In dealing with this broad category here, we will focus on interface issues only, and especially on teleimmersive interfaces that may be distinctive to Internet2. A fundamental difference in high-bandwidth telecommunication systems is likely to be the way the body is connected to information. To increase the information flow to the human user in a high bandwidth telecommunication system it may be desirable to tightly couple the user s body (sensory, motor, and autonomic channels) to the interface. But interfaces that presently attempt this tight coupling such as virtual reality and simulation technologies are immature. The degree of coordination between motor actions and feedback via sensory displays or across different sensory displays may be delayed, out of synchronization, or produce various conflicting sensory cues. The result is simulation sickness [8, 7] or, worse, perceptual adaptation and other performance distorting aftereffects [4, 16]. As predicted [2], this problem has severely inhibited the adoption and diffusion of immersive, tightly coupled virtual reality technology. These psychophysical problems are not insurmountable. But it is clear that a great deal more research on the short term and long term effects of these media is necessary prior to the widespread use of tightly-coupled, teleimmersive interfaces. 4. Networked minds and social computing: Looking for ways to leverage social communication and human collaboration via teleimmersive environments The movement towards I2 telecommunication bandwidth levels dramatically increases the speed and amount of real time information that can be exchanged between two people, within a work group, distributed in an organization, or diffused socially and internationally. Boosts in I2 bandwidth dramatically increases the amount of information that can be exchanged, but just as importantly, the amount and types of data that can be exchanged between individual and groups in real time. This includes support for the kinds of interactions that have to this point in time not been common: continuous flow of thousands of interactive video data streams, multi-user teleimmersive environments supporting thousands of simultaneous users. Most significantly, many of the advancements in human productivity have come from better social coordination of human labor or exchange of thought processes. Since the arrival of electronic communication systems there has been a vision of higher levels of social intelligence as one mind is connected to another, and human minds are increasingly networked into a flow of thought processes. We can see versions of this idea at the turn of the century in the work of Theilhard DeChardin, H.G. Wells, and others. Later in the century we see echoes of this vision in the work of Marshall McLuhan, and more recently in Pierre Levy s work. Common to these visions of networked intelligence are notions of a human communication threshold that is crossed by the development of telecommunication bandwidth and interfaces that can make use of the bandwidth. But one additional element is necessary and often seen as emergent in the social system: ways of distributing and coordinating human cognition and intellectual labor across multiple individuals. What is envisioned for networking computing at the turn of this century is a system for distributing and organizing intellectual labor and human creativity. The hope is that the advanced Internet will leverage the power of coordinated intellectual labor, the way the assembly line leveraged the power of coordinated human physical labor. Internet2 is a social communication system. It will touch many aspects of human communication and social organization. We will not address all of them here. Some are addressed in other white papers growing out of the Internet2 Sociotechnical summit. In this section, we focus on human factors and interface design issues relevant to networked teleimmersive environments HCI and psychosocial research issues on interfaces for networked minds and social computing The most interesting and powerful content in any communication medium is other people. This is especially true of interactive environments. A lot of the traffic on the net deals with interactions of one person with another: , teleconferencing, chat, etc. , an expression of the desire for people to interact, built Internet1. 77

12 Some of the additional bandwidth for teleimmersive systems will be allocated to increase the copresence of others by carrying more information about networked interactants: facial expressions, gestures, etc. The most common Internet1 interfaces provide modest support for interpersonal interaction between two individuals (e.g., audio, low bandwidth video), and only weak interaction capabilities for large teams or groups (e.g., chat rooms, audio teleconferencing, video teleconferencing, etc.). Typically these interfaces are designed as net meeting systems, that is the support of some verbal communication (e.g., comments, opinions, etc.) and some limited non-verbal communication. The limitations of the typical interface is most apparent when interactants must jointly manipulate a common object, even simple objects like text, but especially 3D objects such as an engine design, or an MRI model. Cues such as gesturing, eye contact, gaze awareness, etc. are missing, or weakly transmitted. New proposed interface designs have mostly concentrated on reproducing the typical business meeting or the work team. The current interfaces are most severely limited when interactions must involve large groups of 10, 100, or more. Interfaces such as audio, single camera video, or text-based interfaces cannot support turn taking, subgroup organization, and large group object manipulation. Purely audio teleconferencing systems are most sensitive to the effect of group size. Among the exceptions are early experiments with networked virtual environments using 3D avatars in VRML and distributed military simulation [11]. The capability of these systems has been severely limited by current bandwidth limitations and transmission unpredictability. But what issues emerge when bandwidth increases? Let's put our heads together : Research on interfaces that support distributed cognition and coordinate networked intelligence Research on distributed cognition has explored how cognitive processes are distributed in groups, organizations, and cultures [6]. The computational parallels of perception, information processing and filtering, decision making, and motor action can be found in organized groups. In almost all cases, including the model example of ship based navigation; technologies play a large role in facilitating the distribution of cognition. Computational processing of representations of the world or organizational states is critical to holding together an efficient distributed cognitive system. The question for Internet2 interfaces is how increased bandwidth can: Better support distributed cognition Allow the growth of larger and more complex distributed cognitive networks Support more metacognitive awareness of the distributed cognitive processes Human factors and interface research in this area would need to explore some of the following: Process capture and awareness: The ability to record and playback collaborative experiences in small and large group interactions Division of computation: Ability to support more minute division of real time information and cognitive processing across more individuals Filtering: Ability to filter group processes, information processing status, and information flow in the distributed cognitive network Visualization and abstraction of group processes: Ability to condense and present various representations, visualizations, and views of distributed cognitive processes to support group metacognitive awareness Dynamic reconfiguration of the distributed cognitive network: Ability to dynamically add individuals and organizations to information flows and locations in the distributed network Living with Virtual Teams, Crowds, and Communities: Research on the development of large, multi-user, teleimmersive environments Collaboration and cognition involves more than small work teams. It involves whole organizations and cultures. How can interfaces for Internet2 support large group collaboration? What are some of the issues? It is clear that research and development work is needed for interfaces that can support very large group interactions already visible in Internet1 and increasingly possible with increases in Internet2 bandwidth. It has been a goal of networked virtual environments to support 1000 or more users in fully embodied interaction. Early models are most developed in military simulations [11, 14]. What are appropriate interfaces for large-scale interaction? Many large scale interaction systems 78