Bridging the gap between users' expectations and system evaluations

Transcription

1 Bridging the gap between users' expectations and system evaluations Manja Lohse Abstract What users expect of a robot strongly influences their ratings of the interaction. If the robot satisfies the expectations, the users are usually pleased. If not, their experience of the interaction is negative. We as robot designers strive to design for positive interaction experiences. Therefore, users expectations need to be taken into account. However, in this paper we argue that not all expectations are equally important. Based on a semantic differential questionnaire, we show a correlation between the perceived importance of users expectations and the difficulty to satisfy them in interaction. The paper argues that this correlation allows to identify the most important improvements that should be made to robot systems but also to find attributes which the robot is quite good at already. E I. INTRODUCTION XPECTATIONS have been shown to have a crucial influence on how users behave in human-robot interaction (HRI) [8]. Moreover, they influence how users feel about the interaction. Usually, if their expectations are not satisfied, the user experience will be negative [12]. Moreover, the flow of the interaction will be disrupted as the users will constantly need to review their expectations and search for more appropriate behaviors. Therefore, the goal of system design should be to meet the users expectations. However, we argue that not all expectations have an equally strong influence on the interaction. Rather we claim that some are more important than others and that it is crucial to know about the perceived importance of the expectations in order to identify the most important changes that need to be made to the system and to determine what the robot is already doing quite well. Analyzing the interaction based on, for example, video data and measuring the user experience after the interaction cannot answer the question of how important certain expectations are. However, such measures are needed. Therefore, we conducted a study to acquire them. The expectations were rated using a semantic differential questionnaire. The perceived importance of the expectations was acquired by asking the participants how important they rate each of the items. In a user study with a robot, the same questionnaire could then be used to measure the performance of the robot after the interaction. We propose that the gap between ratings of the robots performance and user expectation multiplied by the perceived importance of the expectations can be used as a criterion to estimate the Manuscript received March 17, M. Lohse is with the Research Institute for Cognition and Robotics (CoR-Lab), Bielefeld, Germany (phone: ; fax: ; mlohse@techfak.uni-bielefeld.de). strength of problems and the necessity for improving them. How we received these values will be described in the course of the paper. In Section II we shortly introduce the concept of expectations in order to explain its relevance for HRI. Moreover, some semantic differential questionnaires that have been developed or used for HRI will be described. Thereafter, our questionnaires are introduced in Section III. Section IV focuses on a study that showed the relationship between expectations and their perceived importance. In a next step, section V links the expectations and their perceived importance to performance ratings of a robot in a user study. The paper closes with some concluding remarks. A. Expectations in HRI II. RELATED WORK The term expectation is used in many contexts in everyday life: employees have expectations toward their employers, we expect a door to open if we push it in the right direction, and people have expectations towards robots. A first meaning of the term expectation refers to what happens in the case of uncertainty [11], for example, if the action of pushing the door would actually open it. Given that the door is a kind of product, we can relate this example to Norman s concept of affordances [10]. Norman differentiates physical, logical and cultural constraints. A physical constraint would be that the door is locked and, therefore, cannot be opened, a logical constraint is the fact that one needs a key to unlock it, and the cultural constraint concerns the way in which doors are usually designed in certain cultures (for example, in which direction do you have to push or pull it). Cultural constraints are strongly interlinked with the second meaning of the term expectations which concerns norms [11]. Relating this example to HRI, the question is which are the constraints that drive user behavior in the interaction with a robot (which also is a case of uncertainty) to achieve a certain effect and how their behavior can be predicted. In this context, the perceived affordances subsume what users believe a system can do and the expectation concept goes further by also describing how the system does it. Also in the field of HRI some research has been conducted on user expectations, for example, on the normative distances a robot has to keep from the user (for example, [12], [17]). Other studies discussed which tasks would be normative for intelligent service robots [6] and how the robots tasks shape the users attitudes (for example, [1], [14]). In previous studies, we have also researched

2 which tasks the users expect a robot to be suitable for based on its appearance [9]. This research was informed by the theory of mental models ([5], [13]). It showed that appearance certainly is one important factor. However, during the interaction, behavior is just as important. In this context, Komatsu, Kurosawa, and Yamada define the relation between the users expectations towards the robot before the interaction and the perceived function after the interaction as adaptation gap [7]. The goal of design should be that the adaptation gap is not negative, in other words, the robots function should equal or exceed the users expectations in order to influence the users evaluation of the robot and their behavior in a positive way. As neither the robot nor the user can be expected to actually adapt to the other, we here prefer to use the term expectationperformance gap. Moreover, Komatsu et al. do not focus on concrete behavioral attributes of the robot that influence the adaptation gap to a certain degree. This is what we want to concentrate on here. The question in the following will be which behavioral attributes people expect the robot to have, how important these are to them, and how they rate a robot s performance regarding these attributes. B. Semantic differentials in HRI We chose to research the expectations towards the robot and its performance with a semantic differential questionnaire (see Section III). Therefore, this kind of questionnaire shall be introduced here. It is based on ratings on scales between two bipolar adjectives, for example, warm : cold or beautiful : ugly [15]. We decided to use such ratings because often the expected robot behavior lies between poles such as funny or serious. Thus, adequate ratings could hardly be acquired using Likert scales. Other semantic differential questionnaires have already been proposed for HRI, such as the Godspeed scales of Anthropomorphism, Animacy, Likeability, Perceived Intelligence, and Perceived Safety of Robots [2]. These were developed in order to measure the perception of the factors named above and have also been used by different researchers. The same is true for the AttrakDiff questionnaire [3] which has been applied to measure user experience in HRI [18]. However, these questionnaires measure concepts that do not cover the questions we had regarding the interaction with our robot. That is why we decided to develop our own questionnaire. III. QUESTIONNAIRES Our semantic differential questionnaire has been designed for use in studies of the home tour scenario. This scenario and the robot have certainly influenced the choice of adjectives used in the semantic differential questionnaire. We do not aim to propose one questionnaire for all kinds of systems as we believe that this questionnaire would be too general to actually guide system design in a specific domain. Therefore, it would probably be necessary to adapt the questionnaire for other scenarios and robots. In the home tour scenario, the robot is guided through an apartment by the user and learns about rooms and objects. This is necessary because it is not possible to pre-program the robot for every possible environment. Hence, it has to learn about its surroundings with the help of the user. Once the robot has acquired the knowledge about its surroundings, it can serve as a kind of butler providing personal services (for example, laying the table or cleaning rooms). The robot we use to research the scenario is the Bielefeld Robot Companion (BIRON). BIRON is based on the GuiaBot TM1 research platform, customized and equipped with sensors that allow for the analysis of the current situation in a human-robot interaction. The robot base is a PatrolBot TM. More information about BIRON can be found in [16]. Based on the scenario, requirements that arose in the work with our robot, and on the theoretical thoughts discussed in Section II.A a questionnaire was developed. The questionnaire items were derived from an expert discussion and pre-tested with a group of 24 people. Items that the participants did not understand or for which they believed that the adjectives were not clearly bipolar were removed. The resulting questionnaire contains 17 items which have been found to cover the important attributes of the robot in the home tour scenario. These items are: talkative : quite, funny : serious, diversified : boring, interested : indifferent, independent : dependent, active : passive, friendly : unfriendly, polite : impolite, autonomous : not autonomous, intelligent : dumb, cooperative : uncooperative, attentive : inattentive, fast : slow, predictable : unpredictable, adaptive : not adaptive, practical : impractical, useful : useless, and obedient : disobedient. All items are rated on a five-point scale ranging from 4 (first word) to 0 (second word). They can be used to rate both the expectations of the users before the interaction and the performance of the robot after the interaction. The same items are also rated in the perceived importance questionnaire on a five-point Likert scale ranging from very important to not important. This procedure is based on the assumption that future ratings of the robot will depend on the relation of expectations and their perceived importance. The perceived importance describes the strength of the users need that the actual system matches their expectations. The higher the perceived importance, the more urgently should the expectation be satisfied. This is equally true for expectations with high and low values. For example, the users could expect the robot to be quiet, which corresponds to a low expectation value on the scale of talkative : quite, but at the same time this expectation could be very important to them. In other words, the ratings of the expectations are not necessarily correlated with the ratings of the perceived importance, as high expectation values do not necessarily refer to positive attributes of the robot. 1

3 The questionnaires were distributed in German. For the German translation of the items see TABLE 1. In the following, we want to show how this particular questionnaire was applied in our research. IV. EXPECTATIONS AND PERCEIVED IMPORTANCE A first study with the questionnaire was conducted in order to determine people s expectations towards a robot in the above-mentioned scenario and to get ratings of the perceived importance of each item. A. Study Procedure The participants were handed a questionnaire which contained the following description of the scenario (in German): Please imagine the following scenario: You have bought a robot that shall support you in the household. Today, this robot has been delivered. In principal, it is ready to complete its tasks. For example, it could get your cup from the kitchen while you stay seated on the couch. However, the robot needs to learn where the kitchen is located (as well as all other rooms where it will have to complete tasks) and how the objects (for example, your cup) look like and where they are at. After reading this text, the participants filled in the semantic differential questionnaire to rate their expectations towards the robot. Thereafter, they rated how important they found the attributes of the robot. In this study, the participants did not interact with a robot but only answered the questionnaire. We chose to proceed this way in order to validate the questionnaire before using it in actual user studies with the robot, because our user studies usually take quite long and we did not want to have participants answer additional questionnaires. That this approach was valid from a statistical point of view will be shown in Section V.A. B. Participants The study was conducted with 31 participants (16 female, 15 male). Their age ranged between 17 and 45 years with an average of 24.2 years (SD=5.09). All participants were German native speakers which was relevant to ensure the understanding of the semantic differential scale. C. Results TABLE 1 depicts the results of the questionnaire study ordered by the perceived importance ratings of the items. First of all, it shows that the participants indeed were ready to rate their expectations and the perceived importance of the items based on the short description of the scenario. Moreover, items with low, medium, and high perceived importance could be differentiated. For some items the means also show that adjective pairs which can intuitively be seen as being closely related indeed are: high perceived importance: practical : impractical, useful : useless; adaptive : not adaptive, obedient : disobedient; autonomous : not autonomous, independent : dependent, predictable : unpredictable medium perceived importance: active : passive, cooperative : uncooperative, attentive : inattentive friendly : unfriendly; polite : impolite intelligent low perceived importance: interested : indifferent, talkative : quite funny : serious, diversified : boring TABLE 1. RATINGS OF EXPECTATIONS AND PERCEIVED IMPORTANCE (PI) item (German) item (English) expectation mean (SD) PI mean (SD) low PI lustig : ernst funny : serious 1.84 (1.32) 1.19 (1.28) abwechslungsreich: diversified : 2.19 (1.22) 1.23 (1.09) langweilig boring interessiert : interested : 1.84 (1.32) 1.52 (1.24) desinteressiert indifferent gesprächig : talkative : 1.13 (1.28) 1.77 (1.26) ruhig quite medium PI freundlich : friendly : 2.94 (1.00) 2.16 (1.57) unfreundlich unfriendly höflich : polite : 3.07 (1.03) 2.19 (1.52) unhöflich impolite aktiv : active : 2.32 (1.40) 2.65 (1.14) passiv passive kooperativ : cooperative : 3.23 (0.85) 2.71 (1.24) unkooperativ uncooperative aufmerksam : attentive : 3.19 (0.98) 2.74 (1.09) unaufmerksam inattentive intelligent : intelligent : 2.90 (1.14) 2.81 (1.17) dumm dumb high PI selbstständig : autonomous : 2.39 (1.33) 3.00 (0.86) unselbstständig not autonomous unabhängig : independent : 1.68 (1.60) 3.03 (0.71) abhängig dependent vorhersagbar : predictable : 3.23 (1.12) 3.03 (1.05) unvorhersagbar unpredictable schnell : fast : 3.07 (1.00) 3.16 (0.82) langsam slow lernfähig : adaptive : 3.42 (0.67) 3.65 (0.55) lernunfähig not adaptive gehorsam : obedient : 3.81 (0.40) 3.65 (0.55) ungehorsam disobedient praktisch : practical : 3.65 (0.76) 3.68 (0.60) unpraktisch impractical nützlich : useful : 3.71 (0.64) 3.68 (0.54) nutzlos useless mean 2.75 (0.78) 2.66 (0.82)

4 The most important items were practical : impractical and useful : useless (both mean perceived importance [mpi] values being 3.68 on the scale of 0 to 4) which is certainly due to the scenario which is strongly task-driven. The participants also had high expectations with this respect (mean expectations [me]=3.65 and 3.71, respectively). Furthermore, it was important to them that the robot was highly adaptive (mpi=3.65; me=3.42), obedient (mpi=3.65, me=3.81), fast (mpi=3.16, me=3.07) and predictable (mpi=3.03, me=3.23). Also the items independent : dependent (mpi=3.03), and autonomous : not autonomous (mpi=3.00) were rated as being very important. The expectation of the users was that the robot would show a medium degree of autonomy (me=2.39), and independence (me=1.68). The group of items with medium perceived importance encompasses attributes that describe how the robot should cooperate with the user. Among the items with medium perceived importance, active : passive (mpi=2.65) was the only one with also a medium expectation (me=2.32), in other words the robot should neither be too active nor too passive. For the other items in this group, the expectation values are rather high, i.e., the robot shall be rather cooperative (mpi=2.71, me=3.23), attentive (mpi=2.74, me=3.19), friendly (mpi=2.16, me=2.94), polite (mpi=2.19, me=3.07), and intelligent (mpi=2.81, me=2.90). The group of items with low perceived importance basically consists of robot attributes that should please the user but are not merely important for the task and the functional aspects of the robot. This again reflects the taskdriven scenario. The item talkative : quite (mpi=1.77) has the lowest expectation value (me=1.13). Thus, the users want a rather quite robot and do not think that a lot of talking is required for the task. The expectation values for funny : serious (mpi=1.19, me=1.84), diversified : boring (mpi=1.23, me=2.19), and interested : indifferent (mpi=1.77, me=1.84) are in the medium range of the scale. This questionnaire study has shown that the robot is expected to have certain characteristics which have different degrees of perceived importance. Task related attributes can be seen as being more important than social attributes. If assuming that it is crucial to satisfy users expectations that are highly important to them, system design should mainly focus on the task related attributes or in other words on the robots ability to fulfill its function, i.e., its affordances. With this in mind, we will now look at how our system performed and satisfied the users expectations. V. LINKING EXPECTATIONS, PERCEIVED IMPORTANCE, AND SYSTEM EVALUATIONS The first study has determined what attributes a robot should have in a home tour scenario and how important it is that the robot actually fulfills the users expectations regarding the attributes. The logical next step is to research how our robot performs with respect to the expectations. In other words, the expectations, their perceived importance, and the actual evaluation of the system need to be linked. Therefore, the questionnaire was distributed in a user study which proceeded as follows. A. Study Procedure The participants answered the semantic differential questionnaire twice: once before the interaction (after getting an introduction very similar to the one above) to rate their expectations and once after the interaction to evaluate the system. The ratings before the interaction were taken to ensure that they are similar to the ratings in the first study made by people that answered the questionnaire in a different situation (i.e., without having to interact with the robot). Statistical analysis showed that this was actually true (U-test (two-tailed): n 1 =17, n 2 =17, U=154, p>.05). Therefore, in future work for similar populations and a similar scenario the ratings acquired in the first study can be used. The interaction with the robot consisted of the following steps. First, the participants practiced the interaction in order to reduce hesitant behaviors. They were handed a tutorial script which contained all commands the subjects would need later on (for example, Hello, Follow me ). After the tutorial session, the participants carried out the main task. The instruction for this main task was: guide the robot from the living room to the dining room via the corridor show and label the living room and the dining room show the bookshelf in the living room and the floor lamp in the dining room ask the robot to go back to one of the rooms or objects that had been learned previously More information about the procedure can be found in [4]. B. Participants 14 participants (7 male, 7 female) took part in the study. Their age ranged from 18 to 54 years (average 38.9 years, SD=11.83). Their experience with robots was very limited. All participants were German native speakers and interacted with BIRON in German. They were recruited at a public event of the university. C. Results TABLE 2 shows the results of the performance ratings of the robot. Overall, the performance of the robot with respect to the attributes was 2.09 on a scale of 0 to 4. This value does not tell us a lot without taking into account how the users expected the robot to be, because again high ratings on the semantic differential questionnaire do not correlate with good performance. Rather, the gap between the users expectations and the robot s performance should ideally be minimal. Therefore, the table also shows the gap between the performance of the robot and the users expectations. The overall difference between the items was Thus,

5 on average the performance values were smaller than the expectation values. The minimum difference was on the item funny : serious which means that the robot was only slightly less funny than the users expected it to be. The maximum difference was for the item fast : slow. Thus, the robot was much slower than the users expected it to be. The overall expectation-performance gap could be calculated by adding up the absolute values of all items (here14.57). This result can simply be compared to ratings of other studies with the same or different robots. Expectation-performance gap values already tell a lot about the performance of the robot in relation to the expectation of the users. However, as has been argued above, the perceived importance of the items needs to be taken into account in order to judge how severe the gap between the performance and the expectation actually is. TABLE 2. ROBOT PERFORMANCE, PERFORMANCE-EXPECTATION GAP (PEG), LINK TO PERCEIVED IMPORTANCE (PI) OF EXPECTATIONS item performance PEG PEG * PI mean (SD) low PI funny : 1.71 (0.91) serious diversified : 2.00 (0.93) boring interested : 2.50 (0.94) indifferent talkative : 1.40 (0.99) quite medium PI friendly : 3.20 (0.56) unfriendly polite : 3.20 (0.56) impolite active : 2.07 (0.88) passive cooperative : 2.73 (0.70) uncooperative attentive : 2.60 (0.74) inattentive intelligent : 1.93 (0.80) dumb high PI autonomous : 1.33 (1.11) not autonomous independent : 1.07 (1.16) dependent predictable : 1.73 (0.96) unpredictable fast : 0.93 (0.70) slow adaptive : 2.87 (0.64) not adaptive obedient : 2.67 (0.72) disobedient practical : 1.79 (0.80) impractical useful : 1.93 (0.83) useless mean 2.09 (0.69) Therefore, the table also shows the values of the expectation-performance gap multiplied by the perceived importance. These values are also depicted in Figure 1 which clearly shows the relation between the measures. As in the table, in the figure, the items are ordered by perceived importance (from lowest perceived importance on the left to highest perceived importance on the right). It can be seen that the seven items on the left with the lowest perceived importance have values between -1 and 1. In other words, the users expectations with respect to these items were rather well satisfied. Due to the low gap between expectations and performance and the low perceived importance the attributes do probably not lead to a negative user experience. The expectations connected to them should be processed heuristically and they should not interrupt the flow of the interaction. However, as can be seen in the figure, the values for items with above average perceived importance decrease in an almost linear fashion. This means that with increasing perceived importance also the gap between performance and expectations increases (correlation of r=-.77). The items with the lowest values are fast : slow, practical: impractical, and useful : useless. Hence, the robot was too slow and did not seem very useful. This was probably due to the fact that the robot is a research prototype that lacks a lot of abilities which it would need if it was an off-the-shelve product. However, the results also have positive implications. Given their high perceived importance, the robot performs pretty well with respect to the right degree of independent : dependent and adaptive : not adaptive behavior. As the expectation value of adaptivity was rather high, the participants believed that the robot was able to learn and to better adapt to the task over time. For comparison with other systems or trials, the sum of the expectation-performance gap values multiplied by the perceived importance can again be calculated by adding up the absolute values for all items (here 44.95). VI. CONCLUSION AND FUTURE WORK In this paper we have measured users expectations about a robot in a home tour scenario with the help of a semantic differential questionnaire. Moreover, we evaluated the performance of the robot with respect to these expectations or, in other words, determined the expectation - performance gap. While this gap alone carries some information about the interaction, we argue that we also need to take the perceived importance of the expectations into account. We could show that not all expectations are equally important to the users. In the case of our scenario the task related, functional attributes of the robot turned out to be more important than social attributes. A preliminary conclusion from this is that the robot should in the first place function well and complete its tasks and only in the second place show nice and enjoyable behaviors in interaction. Unfortunately, the task related attributes are the ones the robot was not perceived to perform really good at in the moment. Therefore, the robot

6 Figure 1. Values for expectation performance gap multiplied with the perceived importance of the expectations for all items needs to be improved with this respect. In the process of designing the robot, the questionnaire introduced here should in future work be used to measure improvements by decreasing the expectation-performance gap and the value of the expectation-performance gap multiplied by the perceived importance of the expectations. Moreover, these measures should also be taken and compared for other systems and tasks. In this context, we also need to address the question of how the appropriate and expected system behavior can be designed, for example, which system behaviors lead to the desired degree of intelligence or diversity and how are these interrelated. Answering these questions will certainly depend on more user studies and an inherently iterative design process. ACKNOWLEDGMENT I thank the participants for their time and effort. Many thanks also go to Frederic Siepmann and Sascha Hinte for their help in conducting the studies and analyzing the data. REFERENCES [1] K. O. Arras and D. Cerqui, Do we want to share our lives and bodies with robots? A 2000-people survey. Technical Report Nr Autonomous Systems Lab Swiss Federal Institute of Technology, EPFL, [2] C. Bartneck, E. Croft, D. Kulic, and S. Zoghbi, Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International Journal of Social Robotics, 2009, 1(1), pp [3] M. Hassenzahl, The thing and I: Understanding the relationship between user and product. In: M. Blythe, C. Overbeeke, A. F. Monk and P. C. Wright (Eds.) Funology. From Usability to Enjoyment, Kluwer, Dordrecht, 2003, pp [4] S. Hinte and M. Lohse, The Function of Off-Gaze in Human-Robot Interaction, submitted. [5] P. Johnson-Laird. Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Cambridge University Press, [6] Z. Khan, Attitudes towards intelligent service robots. Nr. TRITA- NA_P9821, IP-Lab IPLab, Royal Institute of Technology (KTH), Sweden, [7] T. Komatsu, R. Kurosowa, and S. Yamada, How does difference between users expectations and perceptions about a robotic agent (adaptation gap) affect their behaviors?. In Proc. of HRI2011 Workshop on The role of expectations in intuitive human-robot interaction. [8] M. Lohse, Investigating the influence of situations and expectations on user behavior: empirical analyses in human-robot interaction. PhD Thesis. Faculty of Technology. Bielefeld University, [9] M. Lohse, F. Hegel, and B. Wrede, "Domestic Applications for social robots - a user study on appearance and function", Journal of Physical Agents, 2:2, 2008, pp [10] D. A. Norman, Affordances, Conventions and Design. Interactions 6(3), ACM Press, 1999, pp [11] J. Olson, N. Roese, and M. Zanna, Expectancies. In E. Higgins & A. Kruglanski (Eds.). Social Psychology: Handbook of Basic Principles. New York: Guilford Press, 1996, pp [12] E. Pacchierotti, H. I. Christensen, and P. Jensfelt, Human-Robot Embodied Interaction in Hallway Settings: a Pilot User Study. Proceedings IEEE International Conference on Robot and Human Interactive Communication (RO-MAN 05), 2005, pp [13] A. Powers, A. and S. Kiesler, The Advisor Robot: Tracing people's mental models from a robot's physical attributes. Proceedings of Human-Robot Interaction Conference: HRI 2006, [14] C. Ray, F. Mondada, and R. Siegwart, What do people expect from robots?. Proceedings 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, Nice, France, 2008, pp [15] J. G. Snider and C. E. Osgood, Semantic Differential Technique: A Sourcebook. Chicago: Aldine, [16] S. Wachsmuth, F. Siepmann, D. Schulze, A. Swadzba, ToBI - Team of Bielefeld: The Human-Robot Interaction System for RoboCup@Home Tech. rep., RoboCup Singapore, [17] M. L. Walters, K. Dautenhahn, K. L. Koay, C. Kaouri, R. Te Boekhorst, C. Nehaniv, I. Werry, and D. Lee, Close encounters: Spatial distances between people and a robot of mechanistic appearance. Proceedings of the IEEE-RAS International Conference on Humanoid Robots, [18] A. Weiss, Validation of an Evaluation Framework for Human-Robot Interaction. The Impact of Usability, Social Acceptance, User Experience, and Societal Impact on Collaboration with Humanoid Robots. PhD thesis, University of Salzburg, February 2010.