Analysis of Perceived Workload when using a PDA for Mobile Robot Teleoperation

Transcription

1 Analysis of Perceived Workload when using a PDA for Mobile Robot Teleoperation Julie A. Adams EECS Department Vanderbilt University Nashville, TN USA julie.a.adams@vanderbilt.edu Hande Kaymaz-Keskinpala EECS Department Vanderbilt University Nashville, TN USA hande.kaymaz@vanderbilt.edu Abstract A Personal Digital Assistant (PDA) based interface has been developed to provide teleoperation of a mobile robot. The interface provides three different screen designs, all of which employ touch (finger) based interaction rather than stylus based interaction. The interface provides general interaction capabilities for driving the robot based upon the information display. The Vision-only screen provides the forward facing camera image, the Sensory-only screen provides the on-board ultrasonic sensors and laser range finder information, while the Vision with sensory overlay screen integrates all three data sets. A user evaluation was conducted to evaluate the usability and perceived workload required for each screen. Thirty participants completed the quantitative evaluation. The focus of this paper is the obtained perceived workload results. Keywords-Human-Robot Interaction; PDA; User Evaluation I. INTRODUCTION Personal Digital Assistants (PDAs) have become devices many of us use on a daily basis. They are relatively inexpensive; offer a broad spectrum of applications while also being small, portable, and fairly easy to use. Therefore, it seems relatively obvious that PDAs may represent a good platform for interacting with robots. Their small size, processing capability, and portability make them rather attractive human-robot interaction (HRI) devices. Many standard PDA interfaces have been developed for a wide range of applications. In addition to standard applications such as word processing, calendar management, and calculators, some more specialized systems have been developed. For example, character recognition [1,2,3], educational tools [4,5,6,7], face recognition [8], and mobile surveillance [9]. Robotics has not lagged in the development of PDA based applications. Fong [10] developed stylus based interfaces to provide an operator the ability to interact with a robot via his collaborative control architecture. The general idea is to provide the capability for the operator and the robot to collaborate during the task execution. Perzanowski et al. [11] have developed a multimodal interface incorporating a PDA, gestures, and speech interaction. This interface is used to interact with single and multiple robots. The focus is the development of multimodal humanrobotic interfaces. Huttenrauch and Norman [12] implemented PocketCERO to provide screens to a service robot. They focused on providing a PDA-based HRI because they felt that a mobile robot should have a mobile interface. Their interface provided three screens; two required stylus based interaction and a third permits touch (finger) based interaction. Skubic, Bailey and Chronis [13, 14] have developed a PDA-based sketch interface to provide a path to a mobile robot. The user employs the stylus to provide landmarks as well as a path through the landmarks. This information can then be translated into commands that the mobile robot can execute. Each of these groups had a slightly different motivation for their development efforts and this motivation directed their HRI design and implementation. This work has a different motivation. First, the domains under consideration for this interface include military operations. Therefore, these domains potentially require the operator to be running or walking while operating the robot. This limits the operator s ability to focus on small buttons, menus choices, or writing in the graffiti language. Second, the military personnel that we spoke to stated: the stylus will be lost in the first five minutes. Given these first two considerations, the design was intended to provide touch-based interactions via a finger, thus eliminating the stylus. Third, the domains in question may require the operator to wear bulky gloves. Therefore, the interface must provide interaction capabilities that can accommodate the additional bulk. Fourth, the interface should provide sensory information rather than raw sensory data. Finally, the interface should be usable both when the robot is in view as well as when the operator is at a remote location. The work that most closely resembles our work is that of Lundberg et al. [15]. Their system appears to be based upon many of the design considerations outlined above. It is designed for use with military and search and rescue personnel. They too focused on a finger touch based interface, but do have interaction capabilities that require a stylus. In general, their system is more extensive as it includes capabilities such as collision avoidance, person following, and autonomous exploration. On the other hand, their interface limits the sensory feedback presentation to the operator. A limitation that

2 requires operators be able to directly view the robot s environment during teleoperation. The focus of our work was to develop the system using standard user centered design principles. Therefore, we focused on basic screens and quantitative feedback before further system development. Their system uses a similar PDA and robot as this work. The purpose of this paper is to provide a brief description of the interface design as well as a report the perceived workload results from the quantitative user evaluation. Section II provides the interface design background. Section III describes the experimental design and Section IV reports the evaluation results. Finally, Section V provides the conclusions. II. INTERFACE DESIGN The task domain is military operations. The intention was to provide a small, lightweight, portable, and usable. As one military interviewee stated, It needs to be Private proof (when referring to soldiers within the Private rank). This implied a very simple interface was needed. Given these constraints, the decision was made to pursue the use of a Personal Digital Assistant (PDA) for the teleoperation interface. The domains in question require that the interaction techniques do not rely on fine-grained interaction capabilities. Rather, the preferred interaction mechanism is via touch-screen interaction using a finger. The interviewed military personnel indicated that a stylus would be quickly lost in the field. As well, they did not want to consider an interaction device that required fine-grained interactions. Therefore, the designed interface does not require the stylus. PDAs inherently provide the necessary touch-screen interaction capabilities. The interface design had to accommodate touch-based interactions while operators are wearing gloves, such as chemical protection gloves. This requirement implied larger than normal touch screen buttons for commanding the robot. This constraint conflicts with attempting to maximize the ability to view information on the screen. PDA s offer a small display screen real estate. Therefore, the design had to provide the ability to maximize the screen usage while also accommodating the large command buttons. Providing transparent command buttons satisfied this constraint. An additional requirement, as suggested by the interviewed military personnel, was that the interface should be attached to their person. They should not have to remove the device from a pocket. Therefore, the typical PDA interface interactions were rotated 90º counter-clockwise so that the PDA could be strapped to the operator s arm. This work employs a Toshiba E740 PDA to control an ATRV-Jr robot. The PDA operating system is Microsoft PocketPC The PDA is equipped with 64 MB of RAM, and an Intel PXA MHz XScale Processor. The robot has a forward facing camera, a forward facing laser range finder, seventeen ultrasonic sonar sensors, and an Intel Pentium III TM processor. The ultrasonic sonar sensors are located around the robot with five sonar facing forward, five facing out from each side (a total of ten), and two mounted on the rear of the robot. Wireless Ethernet provides the communication capabilities between the PDA and the robot. Embedded Visual Basic was used to program the interface. A. General Interaction Capabilities Three screens were designed, each providing a different level of sensory feedback (Complete design details can be found in [16, 17]). The robot command buttons are consistent across all three screens. The operator can command the robot to move forward, backward, left, right or a combination of forward and turning or backward and turning. The move forward button is the upward facing button in Figure 1. An emergency stop button is provided in the lower right corner of the screen, as shown in Figure 1. This particular position was chosen for two reasons. The initial objective was to provide tactile feedback to the operator as to the button s location based upon the physical PDA structure. The current button position is the only position that ensures the operator is unable to accidentally activate operating system functionality rather than issue the command. B. Teleoperation Interface Screens Three screens were designed to provide varying feedback levels. It is expected that military operators will use the screens in situations where they can directly view the robot and the environment, as well as when the robot is located in a remote environment and is not directly visible. The Vision-only screen provides the forward facing camera image along with the general robot command buttons. The current design does not provide the capability to pan or tilt the camera. As can be seen in Figure 1, the image is visible through the buttons, thus permitting the operator to view the information behind the buttons. A limitation of this screen is the operator s inability to view the area surrounding the robot when it is located in a remote environment. The operator is required to command the robot to physically rotate which can be cumbersome and time consuming. Figure 1. The Vision-only screen. The Sensor-only screen provides the operator with the ultrasonic sonar and laser range finder sensory information. The ultrasonic sensors are located around the body of the robot, and therefore provide the operator with feedback from the entire area within their field of view. The laser range finder

3 provides a 180º field of view at the front of the robot. Figure 2 provides a screen shot of the Sensor-only screen. A disadvantage is the inability to view the visual feedback from the environment while viewing this interface. This screen is best used when the operator is able to directly view the environment and the robot. Laser Range-Finder Information Sonar Information Figure 2. The Sensor-only screen. The Vision with sensory overlay screen provides the operator with the fused camera image and sensory feedback. The forward facing ultrasonic and laser range finder information is overlaid on top of the standard camera image, thus permitting the operator to view the available information on one screen. Figure 3 provides an example of the Vision with sensory overlay screen. While this screen attempts to combine all available sensory information, it is limited to the forward facing information. If additional information is required, the robot must be rotated to view the surrounding areas. Laser Range-Finder Data Figure 3. The Vision with sensory overlay interface. III. EXPERIMENTAL DESIGN Sonar Data A user evaluation was conducted to assess the perceived workload while teleoperating a robot in an indoor environment containing obstacles. The evaluation was designed to determine which screen the evaluation participants preferred for these tasks. A. Participants Thirty male and female participants completed the evaluation. All participants were unpaid volunteers from the Vanderbilt University community. A majority of the participants were pursuing either Master s or Ph.D. degrees. All participants completed the entire study and all participants had experience using PDAs but no experience with mobile robots. B. Design and Procedure The experiment was a with-in subjects repeated-measures design. Each participant completed two trials with each of the three screens. The screen presentation was randomized across participants in order to counterbalance learning effects. The screen presentation remained the same between trials. Three task environments were set-up for the three different screens. The environment remained constant for a particular screen across trials. The participants navigated the robot through the environment to a designated goal position while avoiding all obstacles and not running into walls. They were to complete the tasks as quickly as possible. All obstacles were easily identifiable within the environment. The participants received a thirty-minute training session during which they were introduced to the interface and robot capabilities. They also practiced driving the robot. The participants were not permitted to view or practice in the actual task environments. The participants were placed in a remote location from the robot and environment. The participants then completed trial one of each task. At the completion of each task, participants completed a questionnaire, which asked them to rate their perceived workload along the 100-point scales across all six factors of the NASA Task Load Index (TLX) tool [18]. The questionnaire also contained five-point Likert scale usability questions. At the completion of each trial, the participants completed another questionnaire that included the fifteen paired factor comparisons of the NASA TLX tool. The questionnaire also contained a series of questions that ranked the three interfaces from best to worst based on a series of factors. The participants then completed a second trial with tasks and questionnaires administered as in the first trial. Additional data collection included the time required to complete each task, the number of errors, the location and number of screen touches, and the participants ability to obtain the final goal position. There was one difference between the trials. The participants were permitted to view the environment and the robot during the second trial while completing the Sensor-only task. The Vision-only and Vision with sensory overlay tasks were completed from the remote location as in the first trial. IV. EXPERIEMENTAL RESULTS A complete statistical analysis of the NASA TLX, usability, and interface ranking questions was conducted. Additionally, an analysis of the task completion times, number of errors, the location and number of screen touches, and accuracy of goal achievement was conducted. This paper

4 focuses on the perceived workload results, the analysis of all evaluation data can be found in [17]. Hart and Staveland [18] define workload as representing the cost incurred by a human operator to achieve a particular level of performance. NASA TLX is a widely accepted perceived workload assessment methodology that entails two components. The first requires participants to rate their responses on a scale from 0 to 100 across six factors: mental demand, physical demand, temporal demand, frustration, effort, and performance. The participants select the factors that contributed the most to perceived workload across all fifteen paired-comparisons. This information was employed to calculate the overall perceived workload. Table 1. Perceived workload results by screen and trial. Trial one Trial two Mean St. Dev. Mean St. Dev. Vision-only Sensor-only Vision with sensory overlay The hypothesis for the perceived workload analysis was: The Vision with sensory overlay and Sensor-only interface screens induce higher workload levels for the defined user group than the Vision-only interface screen. Table 1 provides the overall mean perceived workload values by screen and trial. Figure 4 provides the overall mean workload values by screen and trial. The raw data indicates that the hypothesis should hold for trial one but may not hold for trial two. Figure 4. Mean perceived workload by screen and trial. As Figure 4 indicates, the perceived workload increased slightly for the Vision-only and the Vision with sensory overlay screens between trials. The Sensor-only screen perceived workload dramatically dropped between trials. This result is due to permitting the participants to directly view the environment and robot while completing this task during the second trial. Based upon this result a within subjects, repeated measures Analysis of Variance (ANOVA) was conducted across screens and trials. The result indicate that the difference between trials across the three screens was statistically significant, F(1, 29) = 10.52, p < 0.01 (p < 0.05 is considered significant). Further analysis comparing task pairs using the repeated measures ANOVA was conducted. A. Vision-only versus Sensor-only Analysis The first analysis set compared the Vision-only and Sensoronly screens. The ANOVA across screens and trials indicated that this relationship was statistically significant, F(1,29) = 13.74, p < The ANOVA directly comparing the Visiononly screen to the Sensor-only screen across both trials was insignificant, F(1, 29) = 0.14, p = This result is primarily due to the participants ability to directly view the environment and robot during the second trail of the Sensoronly task. The result was that the mean workload value for that screen over both trials was while the mean for the Vision-only screen over both trials was A t-test was conducted to compare the individual screens within a particular trial in order to understand the differences between the data sets while considering the change to the Sensor-only task during the second trial. The t-test comparing the two screens during trial one found a significant result, t(29) = -2.5, p = The workload value for the Vision-only screen was significantly lower than that of the Sensor-only screen. Similar analysis for the same screens during trial two found a significant result, t(29) = 2.48, p = 0.02, however this result is the reverse of the trial one result. The workload value for the Vision-only screen was significantly higher than that of the Sensor-only screen. These results indicate that when the participants were unable to directly view the environment and robot while using the Sensor-only screen, their perceived workload was significantly higher than when using the Vision-only screen. This result is reversed when participants are permitted to directly view the environment and robot while using the Sensor-only screen. Their perceived workload is significantly lower when using the Sensor-only screen. An ANOVA of the combined mean workload for the Vision-only and Sensory-only screens during trial one versus trial two was found to be statistically significant, F(1,29) = 8.231, p < This result indicates that the combined mean workload values of both screens were statistically different across the trials. The mean perceived workload for the Visiononly and Sensor-only screens during trial one was (std. dev. = 16.72). This mean is higher than the combined mean during trial two (mean = 40.92, std. dev. = 16.94). The individual screen results indicate (based on Figure 4) that the value increased slightly for the Vision-only screen during the second trial. It was anticipated that the Vision-only screen perceived workload values between trials would decrease as the participants gained more experience. The resulting aggregate reduction between trials is not related to this factor. Rather the reduction is primarily due to the participants ability to directly view the environment while completing the Sensor-only task. During the first trial participants were located in a remote

5 location. This difference in the task conditions created a dramatic decrease in the perceived workload values for the Sensory-only screen resulting in a statistically significant result across trials. B. Sensor-only versus Vision with sensory overlay Analysis An ANOVA comparing the perceived workload across screens and trials for Sensor-only and Vision with sensory overlay screens found a significant relationship, F(1,29) = 15.33, p < Additionally, the ANOVA comparing the mean workload values for the Sensor-only screens across both trials to those for the Vision with sensory overlay screen was significant, F(1,29) = 12.15, p < This result does not tell the entire story due to the change in the Sensor-only task treatment during the second trial. The mean workload values for these two screens during trial one were relatively similar, as seen in Table 1 therefore; the significant result is attributed to the change in Sensor-only screen task treatment during trial two. A t-test was conducted on the relationship between the individual tasks within a trial. The results comparing the Sensor-only and the Vision with sensory overlay screens during trial one found an insignificant result, t(29) = -0.53, p = It was anticipated that a lower overall workload would be found with the Vision with sensory overlay screen than with the Sensor-only screen when both were used in the remote location. Not only was the result insignificant, the Sensor-only screen received the lower overall workload value. This result may be attributed to the time delay between issuing a robot command via the Vision with sensory overlay screen and the robot beginning to execute the command. The delay was about five seconds and is due to the large amount of PDA based processing. The t-test for the same screens during trial two was statistically significant, t(29) = -5.57, p < This result is a direct effect of the task change during the second trial. These results indicate that when the participants were able to directly view the environment and robot while using the Sensor-only screen, their perceived workload was significantly lower than when using the Vision with sensory overlay screen. This implies that the Sensor-only screen in this condition requires much lower workload to complete the task. When participants are required to complete both tasks while located at a remote location, both interfaces require relatively the same workload. An ANOVA of the combined mean workload for both the Vision with sensory overlay and Sensor-only screens across trials was statistically significant, F(1,29) = 8.93, p < This result indicates that the perceived workload values were statistically different across the trials. The mean perceived workload for both screens during trial one was (std. dev. = 17.44). This mean is higher than the combined mean for trial two (mean = 46.55, std. dev. = 16). The lower mean workload across both screens during the second trial is due to the participant s ability to directly view the environment and robot. Again, the individual task results indicate, based on Figure 4 that perceived workload increased slightly for the Vision with Sensory overlay screen during the second trial. C. Vision-only versus Vision with sensory overlay Analysis An ANOVA analyzing the perceived workload across trials and screens for the Vision-only and Vision with sensory overlay screens found an insignificant relationship, F(1, 29) = 0.001, p = An ANOVA comparing the mean workload values for the Vision-only screen across both trials to those for the Vision with sensory overlay screen was found to be significant, F(1, 29) = 27.4, p < As Figure 4 shows, the change between trials for each of these tasks was minor. The figure also indicates that the Vision with sensory overlay screen had much higher mean workload (mean = 55.16) than the vision-only screen (mean = 43.95). The t-test further analyzing the screen data within a trial found significant relationships. The t-test comparing the Vision-only and Vision with sensory overlay screens during trial one found t(29) = -3.45, p < The same comparison during trial two found t(29) = -4.75, p < These results indicate that the perceived workload is significantly lower when using the Vision-only screen independent of trial, and therefore requires lower workload. An analysis of the combined mean workload for both the Vision and the Vision with sensory overlay screens during trial one versus trial two was found to be insignificant, F(1,29) = 0.33, p = This result indicates that the combined perceived workload values across the two screens did not dramatically change between trials. The combined mean perceived workload for both screens during trial one was (std. dev. = 15.98). This mean was lower than the combined mean for trial two (mean = 50.21, std. dev. = 17.33). It was anticipated that the workload values for these screens during trial two would decrease. It is believed that this effect existed due to the dramatic drop in workload when participants were able to directly view the environment and robot during Sensoronly task. This may have caused participants to rate their perceived workloads for the Vision-only and Vision with sensory overlay screens slightly higher during the second trial. It is not clear from the data that this is the cause of this result. These results indicate that while there was little change in the workload values for a particular screen between trials, the difference between the overall workload values for the Visiononly and the Vision with sensory overlay screens was significant. Therefore, the participants felt that their perceived workload was always significantly lower with the Vision-only screen. This result may be attributed to the participant s inexperience with ultrasonic sonar and laser ranger finder information. It may be the case that with further training, the difference in perceived workload would decrease. An additional confounding factor is the time delay between issuing a command to the robot when using the Vision with sensory overlay screen and when the robot begins the command execution. No time delay occurs with the Vision-only screen; therefore the participants noticed the delay. This time delay is due to the additional processing required on the PDA to process and display all information. It is feasible that this time delay heavily influenced the perceived workload ratings. An additional evaluation is required to better understand how operators use the interfaces when they are able to directly view the environment and robot. It is unclear from this study if

6 the operators focused more on the robot rather than the provided interface during the second trial of the Sensor-only task. An element of such a study would be to compare the three screens in this situation to determine if there are any differences between them. Further analysis should be completed with more dedicated operators, such as military personnel in order to understand exactly how they would use such a system and which screens prove most beneficial. Such users would have longer training periods and perhaps a better understanding of the screens and their uses. In addition to further evaluations, there are system components that require improvement and modification. The current Vision with sensory overlay screen implementation adversely affected the evaluation results. An implementation revision to eliminate the delays is necessary. Further refinements are required to better integrate the ultrasonic and laser range finder data for the Sensory-only and the Vision with sensory overlay screens. V. CONCLUSIONS A PDA-based teleoperation interface composed of three touch-based screens has been described. The focus of the paper was the presentation of the perceived workload results obtained during the quantitative user evaluation. A with-in subjects repeated measures evaluation was conducted with thirty volunteers. All volunteers had experience using PDA s but no experience working with mobile robots. The original hypothesis related to perceived workload was: The Vision with sensory overlay and Sensor-only interface screens induce higher workload levels for the defined user group than the Vision-only interface screen. This hypothesis was found to be true when participants were unable to directly view the robot and environment for all three tasks. When participants were permitted to directly view the environment and robot while working with the Sensor-only screen, their perceived workload values were significantly lower than the results for the Vision-only and the Vision with sensory overlay screens. If the participants were able to directly view the environment and robot while working with these two screens, it possible that the original hypothesis would stand. Unfortunately, our results do not permit us to draw any firm conclusions on that particular question. ACKNOWLEDGMENT The authors wish to thank the Center for Intelligent Systems at Vanderbilt University for the use of the PDA and ATRV-Jr robot. The authors also thank Mary Dietrich for her statistical analysis guidance. REFERENCES [1] Z. Luo and C.H. Wu, A Chinese Character Recognition Interface for Mobile Communication Devices Using Fuzzy Logic and Unit Extraction, Proceedings of the IEEE IECON 22nd International Conference on Industrial Electronics, Control, and Instrumentation, Vol.1, pp , August [2] H. Kang and H.J. Kim, Design of an Interface on PDA for Korean, IEEE Transactions on Consumer Electronics, Vol.46, Issue: 3, pp , August [3] J. Zhang, X. Chen, J. Yang and A. Waibel, A PDA-based Sign Translator, Proceedings of IEEE International Conference on Multimodal Interfaces, pp , October [4] J. Breitbart, D. Balakrishnan, and A. Ganz, Pocket-IMPACT Software for Delivering Online Courseware on a PDA: Challenges, Design Guidelines and Implementation, Proceedings of IEEE Frontiers in Education Conference, Vol.1, pp. T3F-5, November [5] X. Vila, A. Riera, E. Sanchez, M. Lama, D. L. Mureno, A PDA-based Interface for a Computer Supported Educational System, Proceedings of the 3 rd IEEE International Conference on Advanced Learning Technologies, pp.12-16, July [6] Y.-W. Chen, Z.-J. Yan, J.-C. Huang, I-H. Peng, J.-W. Zhan, Implementation of a PDA/GPS Based Development Platform and Its Application in Native Education, Proceedings of the IEEE International Conference on Communications, Circuits and Systems and West Sino Expositions, Vol. 2, pp , June - July [7] R. Avanzato, Student Use of Personal Digital Assistants in a Computer Engineering Course, Proceeding of the Frontiers in Education Conference, Vol. 2, pp. F1B-F19, October [8] J. Yang, X. Chen, and W. Kunz, A PDA-based Face Recognition System, Proceedings of the Sixth IEEE Workshop on Applications of Computer Vision, pp , December [9] S.-T. Li, H.-C. Hsieh, L.-Y. Shue, and W.-S. Chen, PDA Watch for Mobile Surveillance Services, Proceedings of IEEE Workshop on Knowledge Media Networking, pp , July [10] T. Fong, Collaborative Control: A Robot Centric Model for Vehicle Teleoperation, Technical Report CMU-RI-TR-01-34, Ph.D. Thesis, Robotics Institute, Carnegie Mellon University, November [11] D. Perzanowski, A.C. Schultz, W. Adams, E. Marsh, and M. Bugajska, Building a Multimodal Human Robot Interface, IEEE Intelligent Systems, Vol. 16, No. 1, pp , January/February [12] H. Huttenrauch and M. Norman, PocketCERO Mobile Interfaces for Service Robots, Proceeding of the International Workshop on Human Computer Interaction with Mobile Devices, France, September [13] M. Skubic, C. Bailey, and G. Chronis, A Sketch Interface for Mobile Robots, Proceedings of the 2003 IEEE International Conference on Systems, Man, and Cybernetics, pp , October [14] C. Bailey, A Sketch Interface For Understanding Hand-Drawn Route Maps, Master s Thesis, Computational Intelligence Lab, University of Missouri-Columbia, December [15] C. Lundberg, C. Barck-Holst, J. Folkeson, and H. Christensen, PDA Interface for a Field Robot, Proceedings of the 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp , October [16] H. Kaymaz-Keskinpala, J.A. Adams, and K. Kawamura, PDA-Based Human-Robotic Interface, Proceedings of the International Conference on Systems, Man and Cybernetics, pp , October [17] H. Kaymaz-Keskinpala, PDA-Based Teleoperation Interface for a Mobile Robot, Master s Thesis, Vanderbilt University, May [18] S. Hart and L. Staveland, "Development of NASATLX (Task Load Index): Results of Empirical and Theoretical Research," in Human Mental Workload, P.A. Hancock, N. Meshkati, Ed., pp , 1988.