Breaking the Keyhole in Human-Robot Coordination: Method and Evaluation Martin G. Voshell, David D. Woods

Transcription

1 Breaking the Keyhole in Human-Robot Coordination: Method and Evaluation Martin G. Voshell, David D. Woods Abstract When environment access is mediated through robotic sensors, field experience and naturalistic studies show robot handlers have difficulties comprehending remote environments - they experience what domain practitioners often call a `soda straw'. This illustrates the keyhole effect in HRI, a phenomena studied in the context of large virtual data space interfaces and the current research seeks to reduce this effect. A simulation for human-robot search and rescue was created in a virtual NIST arena based on WTC response experiences. Pilot studies showed that traditional measures of performance were inadequate to analyze control and exploration tasks in these environments. New measures were developed based on fractal path analysis describing the tortuosity of these goal-directed paths. New concepts for helping remote observers pick up the environment affordances were tested using the simulation and evaluation measures. These studies look to concepts based on Gibsonian principles to reduce keyhole effects, enhancing functional presence. Introduction: The Keyhole Effect In HRI There are many challenges in designing successful human-robot systems (Woods, Tittle, Feil, Roesler, 2004). When the robot is a stand in for a remote human observer, the natural dynamic relationship between properties of the scene being explored and the human perceptual system is broken. The decoupling undermines the remote observers perception of affordances in the scene (Gibson, 1986) which is illustrated by recent cases of HRI where remote observers experience various difficulties in understanding the environment being traversed by a robotic system (Murphy, 2003, 2004). For example, practitioners often complain about what they call the `soda straw effect' due to the limited angular view available (Casper and Murphy, 2003). This is an example of the keyhole effect. Typical consequences of the keyhole effect include missing new events, increased difficulty in navigating novel environments, gaps or incoherent models of the explored space (Woods and Watts, 1997). Keyhole problems arise from the fact that typical virtual environments sever the coordination between the foveal field of view/focal attention and the orienting perceptual functions that help people fluently know where to look next, despite the potential for new interesting events to intrude on ongoing activities. Thus, keyhole problems cannot be solved simply by expanding the size of the field of view. When stuck looking through the `soda straw' operators have a difficult time understanding spatial layout. Operators easily miss alleys, landmarks are tough to discern, and the handler is usually forced to manually switch and integrate multiple views. Safe and successful navigation is more than just looking where one is going. In the natural

2 world humans tend to sample everywhere around where they are going in a context sensitive way. This reminds us that the mechanisms that allow people to coordinate direction of gaze and direction of movement as they move in a changing scene are removed when viewing a scene through a robot s camera (Hughes and Lewis, 2004). Human gaze control is tuned to anticipate future movements and conditions of interest. Contrast how you would direct gaze as you turn to climb a stairs with scattered debris on it and with various items or activities of interest at the top of the stairs versus how robotic platforms position their cameras during the same maneuver. Generally, the robot camera either points at each step one at a time or remains pointed at the ceiling as the robot climbs, whereas people direct and shift gaze in tight coordination with the affordances present in the situation given their purposes and context (e.g., when to look at the activities heading for the top of the stairs and when to look at potential obstacles along the stairs). The challenge for HRI is how to breakdown the keyhole and enhance a remote observer s understanding of the environment being traversed by the robotic system? This research addresses the question by creating a simulation of HRI in a search and rescue situation, developing new measures of HRI performance based on fractal path analysis, and testing new concepts for breaking down the keyhole in HRI. From Field work to Simulation Field experience in search and rescue tasks have shown that keyhole effects and other problems in remote perception significantly reduce the potential benefits of rescue robots. For example, see Casper and Murphy's experiences with rescue robots at the World Trade Center (Casper and Murphy, 2003). The first step in the research program on keyhole effects in HRI was to develop a simulation that captured the difficulties encountered in the field and allow for detailed testing of HRI performance. To accomplish this, a virtual human-operated robot assisted search and rescue simulation was created based on Murphy s experiences from the WTC response and set within a virtual reconstruction of the NIST physical HRI test arena (National Institute for Standards and Technology). Michael Lewis started using Unreal Tournament videogame based simulations for USAR applications citing the benefits of realistic physics and high-fidelity graphics (Wang, Lewis, and Gennari, 2003). We collaborated with Lewis and started work with a model of the NIST Orange Reference Test Facility for Autonomous Robots. We heavily modified this environment to add many new obstacles, ambiguities, and world geometry. Within the simulation, operators had to perform multiple exploratory tasks analogous to actual search and rescue missions while facing many of the same physical and visual constraints seen in the field. The C/S/E/L Unreal code allows an investigator to introduce various robotic platforms, sensors, and the ability to rapidly develop and test new interfaces and camera arrangements. A highlevel quantitative analysis framework was built into the code so that event information and three dimensional position data could be exported for evaluation.

3 Novel Analysis Methods What became evident from pilot studies with simulated search and rescue tasks was that traditional measures for analyzing the robot handler s performance in a complex environment with multiple tasks was not very informative). Gathering completion times and items collected simply did not capture the intricacies and problems inherent in search and rescue in remote exploration. We needed much richer ways of looking at goal-oriented paths in relation to the environments characteristics and challenges. To meaningfully evaluate HRI activity, we borrowed techniques from two diverse fields ecological perception and physiological entymology. Visual guided translation is well researched in perceptual psychology and the most valuable data to focus on are the environment affordances (Gibson, 1986). Optic flow during translation contributes greatly in virtually any visually guided behavior (Warren, Morris, and Kalish, 1988) and one immediate application of this in robot control is the handler/robot's ability of perceiving aperture affordances. For this, we chose to look at path approach velocity transitions in relation to the `pasability' of apertures and obstacles in the remote environment. The other metric of analysis looks at the fractal dimension of the handler/robot's path. This method was first used by Dicke and Burrough (Dicke and Burrough, 1988) to characterize the tortuosity of spider mite spatial exploration. Deviations in the fractal dimension of the robot/handler's goal directed path through a complex environment in relation to various ambiguities and obstacles provide a metric that captures a very rich set of descriptive behaviors. Changes in the fractal dimension of the path provide significant insight into the handler/robot's search efficiency, path tortuosity, and overall space utilization in relation to handler goals and overall characteristics of the environment (see Figure 2). Bringing Affordances to the Remote Observers The ability to expand the keyhole is the goal of the research program. Using the knowledge gained from the simulation and these ecological measures, the next step was to discover new interface concepts for helping remote observers pick up the affordances in the environment and re-couple perception and action cycles. One design concept proposed to help overcome the keyhole and make these affordances more salient is Perspective Folding (see Figure 1). By folding the display screen around the remote observer and mapping camera sensors output, we re-embed the observer in the scene and for example re-introduce peripheral optic low cues that signal movement relative to the environment. We have developed a multi-camera version of Perspective Folding (illustrated in Figure 1) and a continuous spherical folding based on a single fixed camera called the Dynamic Perceptor Sphere (Feil et al., 2004). In contrast to a single wide angle camera or an interface to switch among different camera feeds, in the 5-fold version of Perspective Folding an array of five cameras, each oriented at different angles, provides the wrap around effect. By doing this, we are attempting to perceptually better integrate multiple views into a global reference frame. The camera orientations are preserved on the robot handler's display, presented

4 in depth, and do not just provide a larger field of view but serve as a frame of reference around the robot attempting to heighten spatial awareness and robot-body size without image distortion. By integrating each of the local cameras the operator immediately sees that areas usually in peripheral vision now explicitly surround the `robot' in the display. By doing this, we are attempting capture and present some of the many cues that peripheral vision and optic flow contribute to locomotion, perception of selfmotion, information about other moving objects, and the general 3D environment (Warren, 2003). The focus of expansion is mimicked in the arrangement of the planes folded inward and the ground surface is continuously uninterrupted with the fixed camera pointing downward. In a Gibsonian sense, all of this information is fundamentally necessary (and available in the raw data) for the handler/robot ensemble to exhibit successful visually controlled behavior in starting and stopping, steering toward goals, slowing down, and avoiding obstacles in the remote environment (Gibson, 1986, Warren, 1988). For example, in tele-operation, the ground plane provides flow cues to motion that enhance the ability to perceive approach to an obstacle and smoothly steer around it while picking up other relevant aspects of the environment. In supervisory control the question becomes does the new concept help the remote observer sense when the robot algorithms are going to have difficulty traversing the environment. Study Environments were rendered using the C/S/E/L modified version of Unreal Tournament 2004 on a Pentium 4 based computer using an ATI RADEON graphics accelerator card. The virtual environment was displayed on a 19'' monitor at a resolution of 1600 x 1200 pixels. Observers sat in front of the monitor (approx. 57 cm) and used a dual-analog joystick to navigate the environments as a tele-operator of the simulated robot. Interface trials were randomly selected and data was then collected and analyzed using an Apple Macintosh G4 series computer. Participants navigated a virtual robot through a modified version of the National Institute of Standards and Technology s Orange reference test arena for autonomous mobile robots. A series of obstacles, ambiguities, and world geometry were added to make the navigation more substantial. Participants were inserted into a training environment and given one of three interfaces (single camera at \degrees{45}, or five single `flat' cameras at \degrees{45} each totaling \degrees{135} HV, or five single cameras slanted in perspective, `Perspective Folding', each at \degrees{45} totaling \degrees{135} HV) to familiarize them with the interface, teach them how to identify goal items (`hazardous objects'), and learn the joystick control. Upon completion they were brought into the modified NIST arena with the same interface they were trained on and were instructed to find as many `hazardous objects' as quickly as possible. Subjects had 1 hour to complete the task before the `robot' lost all battery power. Fifteen Ohio State University students and staff participated in the study. All individuals were paid volunteers. All volunteers were informed the purpose of the study was to

5 learn how individuals navigate and how different arrangements of cameras in interfaces could affect navigation. Upon an auditory cue signifying the end of the trial, participants were asked about their experience, were given a debriefing statement, and had the chance to ask questions. Findings and Discussion A graphical overview of the results for a portion of the study are shown in Figure 2. The entire three dimensional path is displayed for each participant in the single-flat and the perspective folding interfaces. This is just a snap shot from an interactive graphic. Mean velocities were computed for each participant and the arena was divided into separate transit and `special' stages. Velocity transitions translating toward goals and obstacles were extracted (starting/stopping). Fractal tortuosity was also calculated to indicate how efficient or tortuous a path (lower fractal number the more efficient or direct the path to a target or around an obstacle). The two measures should be complementary if they are tapping into the quality of navigation, i.e., efficient paths as determined by fractal number should also have smooth velocity transitions near targets and obstacles. As an example of the value of the new measures consider this aspect of the simulated environment. One specific area consists of chairs and rubble in the center of a passageway. To one side is a glass wall that can be seen through. In the back corner there is a small opening allowing access to a small unexplored area. Participants who determined that the transparent wall was in fact a wall tended to show efficient fractal score, smooth linear paths around the central obstacles, and maintain velocity toward the small opening. Other individuals had a significant amount of trouble realizing the see-through wall was impassable. Their robots tended to start and stop abruptly, reorient multiple times, and this control difficulty was reflected in convoluted paths and tortuous fractal scores. Traditional measures were not informative either in general or in identifying patterns within a trial. On the other hand, the fractal tortuosity scores coupled with velocity transitions showed how well different observers handled the difficulties in the environment. The study succeeded in demonstrating the value of these new measures. It also provided an opportunity to begin to examine new concepts to break down the keyhole. The new measures revealed different behavioral signatures for observers using the flat single camera view versus the 5-fold view. Focusing on the more difficult portions of the environment, single camera viewers tended to move quickly to the next visual cue, stop, identify the next point to move to and again move quickly to that visual reference. This pattern was shown as a few rapid velocity transitions translating toward goals, around obstacles, and approaching apertures. The fractal score showed these participants followed more tortuous paths for exploring the environment. Participants using the 5-fold view showed more gradual velocity transitions as they approached targets,

6 obstacles, and apertures indicating a more natural navigation pattern. They also had somewhat lower fractal scores. These results are pointers to further studies. We are at work adding features to the environment to introduce new classes of perceptual ambiguities, for example, related to navigating around stairways, and with extended sightlines where the observer can see areas but it is not obvious how to reach those distant parts of the space. The simulation is being expanded to enable more complicated situations and tasks which allow testing of how well participants build up a model of the space they are traversing. For example, after the robot has moved through a portion of the space, a collapse will occur which blocks some passages, and introducing new goals that require a return to previously encountered key points. With these expanded capabilities, future studies will be able to examine the difference between tele-operating the robot and supervisory control schemes. The measures and use of the simulated search and rescue environment provide a means to examine the perception of affordances through video feeds form a robotic platform. Measures sensitive to these attributes show potential to help test new concepts for reducing ambiguities and breaking down the keyhole in human-robot navigation. Acknowledgments Prepared through collaborative participation in the Advanced Decision Architectures Consortium sponsored by the U. S. Army Research Laboratory under the Collaborative Technology Alliance Program, Cooperative Agreement DAAD References Casper, J., and Murphy, R. R. (2003). Human-Robot Interaction during the RobotAssisted Urban Search and Rescue Response at the World Trade Center. IEEE SMC B, 33, Murphy, R. R. (2004). Human-Robot Interaction in Rescue Robotics. IEEE SMC Part C, 34(2), Dicke, M., Burrough, A. P. (1988). Using fractal dimensions for characterizing tortuosity of animal trails. Physiological Entymology, Vol. 13, Gibson, J. J. (1986). The ecological approach to visual perception. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. (Original work published 1979). Hughes, S., Lewis, M. (2004). Robotic camera control for remote exploration In CHI '04: Proceedings of the 2004 conference on Human factors in computing systems, (pp. ( ). Vienna, Austria: ACM Press. Wang, J., Lewis, M., and Gennari, J. (2003). USAR: A Game-Based Simulation for Teleoperation. Proceedings of the 47th Annual Meeting of the Human Factors and Ergonomics Society Denver, CO, Oct Warren, W. H. (1988). Action modes and laws of control for the visual guidance of action. In O. Meijer K. and Roth (Eds.) Movement behavior: The motor-action controversy (pp ). Amsterdam: North- Holland.

7 Warren, W. H. (2003) Optic Flow In L. Chalupa and J. Werner (Eds.) The Visual Neurosciences. Cambridge, MA: MIT Press. Warren, W. H., Morris, M. W., Kalish, M. (1988). Perception of translational heading from optical flow Journal of Experimental Psychology: Human Perception and Performance Vol. 14 (4), Warren, W. H., and Whang, S. (1987). Visual guidance of walking through apertures: Body scaled information for affordances. Journal of Experimental Psychology: Human Perception and Performance, 13, Woods, D. D., Tittle, J., Feil, M., Roesler, A. (2004) Envisioning Human-Robot Coordination in Future Operations IEEE Transactions on Systems, Man, and Cybernetics- Part C, 34(2): Woods, D. D. and Watts J. C. (1997). How Not To Have To Navigate Through Too Many Displays. In Helander, M.G., Landauer, T.K. and Prabhu, P. (Eds.) Handbook of Human-Computer Interaction, 2nd edition. Amsterdam, The Netherlands: Elsevier Science, Figure 1 Remote Environment Handler Display Handler Display Handlers Perform Action At a Distance The simplified perception action cycles looking at different sensor feeds from two robots in a remote environment. The robot on the left has a single camera and is sending back information to the operator on

8 the left. The robot on the right is utilizing five cameras to send information back to a Perspective Folding display and is handled by the operator on the right. Figure 2 Velocity field transitions and fractal path. The blown up section shows a specific transit stage and an optimal path taken through it with a fractal dimension of L(δ) = kδ 1 D We are using a similar dividers method as Dicke and Burrough [3] as developed by Dr. Flip Phillips (Skidmore College), where L is the path length, δ is the step size, k is a constant, and D is the fractal dimension.