BEYOND R2D2. The design of nonverbal interaction behavior optimized for robot-specific morphologies. Daphne Karreman

Transcription

1 BEYOND R2D2 The design of nonverbal interaction behavior optimized for robot-specific morphologies Daphne Karreman

2

3 BEYOND R2D2 Daphne Karreman

4 Ph.D Graduation Committee Chairman and Secretary Promotor Assistant-promotors Members Prof. dr. P.G.M. Apers, University of Twente, NL Prof. dr. V. Evers, University of Twente, NL Dr. ir. G.D.S. Ludden, University of Twente, NL Dr. E.M.A.G. van Dijk, University of Twente, NL Prof. dr. D.K.J. Heylen, University of Twente, NL Prof. dr. ir. P.P.C.C. Verbeek, University of Twente, NL Prof. dr. ir. D.M. Gavrila, Delft University of Technology, NL Prof. dr. F. Eyssel, Bielefeld University, GE Dr. W. Ju, Stanford University, USA The research reported in this thesis was carried out at the Human Media Interaction group of the University of Twente. CTIT Ph.D. Thesis Series ISSN: , No Centre for Telematics and Information Technology P.O. Box 217, 7500 AE Enschede The Netherlands SIKS Dissertation Series No The research reported in this thesis has been carried out under the auspices of SIKS, the Dutch Research School for Information and Knowledge Systems. The research reported in this thesis received funding from the European Community s 7th Framework Programme under Grant agreement ( eu/). Daphne Karreman, Enschede, The Netherlands Layout in InDesign. Printed by CPI-Koninklijke Wöhrmann- Zutphen. ISBN: DOI: / All rights reserved. No part of this work may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without prior permission from the copyright owner.

5 BEYOND R2D2 THE DESIGN OF NONVERBAL INTERACTION BEHAVIOR OPTIMIZED FOR ROBOT-SPECIFIC MORPHOLOGIES PROEFSCHRIFT ter verkrijging van de graad van doctor aan de Universiteit Twente, op gezag van de rector magnificus, Prof. dr. H. Brinksma, volgens besluit van het College voor Promoties in het openbaar te verdedigen op woensdag 14 september 2016 om 12:45 uur door DAPHNE ELEONORA KARREMAN geboren op 23 februari 1984 te Delft

6 This dissertation has been approved by: Promotor: Assistant-promotors: Prof. dr. V. Evers, University of Twente, The Netherlands Dr. ir. G.D.S. Ludden, University of Twente, The Netherlands Dr. E.M.A.G. van Dijk, University of Twente, The Netherlands

7 Acknowledgment A When I was finishing my master s degree, it became clear to me that I like to do research and that doing a PhD would be an interesting and promising next step. Today, over four years later, I am sure that I made the right decision. Looking back, even though it was not always easy, I would definitely do it again. Luckily, I did not have to do everything by myself. I want to thank many people for their support. Even though I can only mention a few of them here, I want to thank everyone who helped me in some way. I would like to thank Vanessa and Betsy for giving me the opportunity to start the PhD in human-robot interaction, and for supervising me. Vanessa, thank you for giving me the freedom to go in the direction in which I wanted to go, and for your interesting ideas and support to improve the research and the presentation of the results. Betsy, thank you for your patience, discussing research setups and results, reading and improving papers and deliverables, and helping me out with stupid questions. Even though you were not as closely involved the last years, I very much appreciate our meetings and the feedback you gave me. A special thanks goes to Geke, who became my daily supervisor when I had been working on my PhD for two years. Thank you for diving into the world of human-robot interaction, for your support with exploring new opportunities for human-robot interaction, and of course for the encouraging moments which I needed when I was finishing the first draft. Many more people helped me during the years. There were weekly meetings with the robot research group, which increased in size over the years. Thank you Michiel (also, thank you for all the other things you did for me), Maartje, Daniel, Vicky, Cristina, Jorge, Roelof, Jered, Jaebok, Manja, Khiet, Dennis, and others who joined over the years, for giving nice insights into your research and helping me with mine. Also, thanks to the students Gilberto, Sandra and Lex, who collaborated in my research. I want to thank Lynn for her effort put into improving my writing in papers, deliverables, and of course this thesis. Moreover, I enjoyed traveling together for FROG very much. Also, I would like to thank Charlotte and Alice for all their help with arranging and organizing everything. And of course, thanks to all other colleagues at HMI for the pleasant work environment. Iris and Randy, I want to thank you for being my paranymphs. Iris, even though there were many differences, we followed the same path. Thank you for sharing experiences, peer pressure during writing, and the fun time we had. Randy, thank you for always being willing to help with FROG, and the help with all questions regarding the process of finishing a PhD. A special thanks goes to Veronique who helped and encouraged me to do what I really wanted V

8 Acknowledgment to do. Without her help, I probably would not have been at the University of Twente. I want to thank the partners of the FROG project for the nice time we had in Seville and Lisbon. The working days in the Zoo and the Royal Alcázar were long, but we achieved a nice result. Furthermore, thank you for showing me around the city and for helping me find the best places to eat! I found out that other activities were as important for doing my PhD as was doing the work itself. Therefore, I thank my hockey teams in Delft and Enschede, and everyone on the violin making course, for their interest in my PhD, and also for giving me a chance to think about something completely different and to relax after a long day/week of work. To my friends from Delft, I promise to have more time during the weekends to meet with you again! I want to thank my parents for their support in whatever choice I have ever made. Hans, I enjoyed the discussions we had about everything. We do not always agree, but that is the interesting part. It gave me insights in a lot of things. Ellen, thank you for always being ready to help me, both at any moment I came to Delft, as well as when I just had a question on the phone. Japke, I want to thank you for just being there, for listening, and for sharing experiences. Also, I want to thank Lammie and Theo for their interest in my thesis and the carefree weekends. Last but not least, I want to thank Tim. Thank you for everything in the last four years. It must have been frustrating that even though we live in Dordrecht, I was in Enschede for half of the week. I appreciate that you never complained, and gave me the chance to do what I wanted. During the last year, I often needed the weekends to work, but you kept things running and supported me when I was tired. I did not always show appreciation for that, but you know that I am thankful for all your support. Things will become less complicated, I promise. Daphne Karreman, Enschede, August 2016 VI

9 Abstract (EN) It is likely that in the near future we will meet more and more robots that will perform tasks in social environments, such as shopping malls, airports or museums. However, design guidelines that inform the design of effective nonverbal behavior for robots are scarce. This is surprising since the behavior of robots in social environments can greatly affect the interaction between people and robots. Previous work has shown that people have the tendency to project motivations, intentions and emotions on non-living objects, such as robots, when these objects start to move. Moreover, it has been shown that people tend to react socially to computers, televisions and new media. Similarly, people will react socially to robots. Currently, in the field of human-robot interaction the focus is on creating robots that look like humans and that can use humanlike behavior in interaction. However, such robots are not suitable for all tasks, and humanlike robots are complex, vulnerable and expensive. Moreover, people do not always prefer humanlike robots over robots with other appearances. This indicates that there are good reasons to develop low-anthropomorphic robots; robots that do not resemble people in appearance. A A challenge in designing nonverbal behavior for low-anthropomorphic robots is that these robots often lack the abilities to imitate humanlike behavior correctly. These robots could lack the specific modalities to perform the behavior, for example, they may not have arms and fingers to point with. When a robot does have the right modalities to perform a specific behavior, these modalities might not have similar degrees of freedom to imitate recognizable nonverbal behavior. Yet, contrary to people, low-anthropomorphic robots may use screens, projections, light cues or specific movements to communicate their intentions and motivations. To optimize the use of robot-specific modalities and morphology, we developed robot-optimized behavior. This was done in the context of developing guide robots for tourist sites. To understand the effects of both imitated humanlike behavior and robot-optimized behavior for low-anthropomorphic robots on peoples perception, controlled lab studies and in-the-wild studies have been performed. First, we analyzed the effect and effectiveness of imitated humanlike guide behavior on low-anthropomorphic robots. These studies showed that humanlike behavior is preferred over random behavior, but that it is also more distracting. This means that humanlike behavior may not the best solution to design behavior for low-anthropomorphic robots. An important question now was what would be a promising alternative to imitating human behavior. In follow-up studies we compared the effect and effectiveness of imitated humanlike behavior for the robot to robot-optimized behavior. From these studies we learned that robot-optimized behavior is a good alternative VII

10 Abstract for low-anthropomorphic robots instead of imitated humanlike behavior. To effectively perform the studies mentioned above, we developed and introduced DREAM, Data Reduction Event Analysis Method. This method allowed us to analyze video data of in-the-wild human-robot interaction. Such a method did not yet exist, and is essential in order to gain insight in effective behaviors for robots. Because many more aspects than only the behavior of the robot and the person interacting with the robot play a role in the final user experience, such as context and other people in that context, it is necessary to add real world field studies to studies in controlled (lab) environments. Therefore, in this thesis, DREAM, is introduced to analyze the rich and unstructured data of in the wild human-robot interactions in a fast and reliable manner. This method turned out to be effective to use for the guide context human-robot interaction data. The work presented in this thesis is a first step towards understanding 1) the effect of nonverbal behavior for low-anthropomorphic robots, 2) which design approaches can be effectively used to design this behavior and 3) how nonverbal robot behavior can be studied and evaluated in the wild. The tangible outcomes of this thesis: 1) the robot-optimized approach to design behavior for low-anthropomorphic robots, and 2) DREAM to evaluate human-robot interaction in the wild. These results can well serve as a starting point to further develop more diverse and effective design approaches in human-robot interaction. VIII

11 Abstract (NL) Het is waarschijnlijk dat we steeds vaker robots zullen tegenkomen die taken uitvoeren in sociale omgevingen zoals winkelcentra, vliegvelden of musea. Desondanks zijn er (nog) geen richtlijnen die sturing geven aan het ontwerpen van effectief non-verbaal gedrag voor robots. Dit is opvallend, omdat juist gedrag van robots in sociale omgevingen veel invloed kan hebben op de interactie tussen mens en robot. Eerder onderzoek heeft aangetoond dat mensen de neiging hebben om motivaties, intenties en emoties te projecteren op niet levende dingen, zoals robots, wanneer deze dingen bewegen. Daarnaast is het ook aangetoond dat mensen de neiging hebben om sociaal te reageren op gedrag van computers, televisies en nieuwe media. Daarom zullen mensen ook sociaal reageren op robots. Op dit moment is onderzoek in het mens-robot interactie veld vooral gericht op het ontwikkelen van robots met een menselijk uiterlijk, die menselijk gedrag kunnen gebruiken in interactie. Echter, zulke robots zijn niet geschikt om alle taken uit te voeren, omdat menselijke robots complex, kwetsbaar en duur zijn. Daarnaast is het gebleken dat mensen niet per definitie voorkeur hebben voor robots die er menselijk uit zien. Er zijn dus goede redenen om laag antropomorfe robots te ontwerpen; robots die uiterlijk juist niet op mensen lijken. A Een uitdaging bij het ontwerpen van non-verbaal gedrag voor laag antropomorfe robots is dat deze robots vaak de modaliteiten missen om menselijk gedrag correct te imiteren. Dit kan komen doordat deze robots niet de modaliteit hebben om het gedrag uit te voeren, de robot heeft bijvoorbeeld geen armen en kan daarom niet wijzen. Als de robot wel vergelijkbare modaliteiten heeft, kan het zijn dat deze niet dezelfde bewegingsvrijheid hebben als mensen om herkenbaar gedrag te tonen. Het is echter wel mogelijk dat zulke robots, in tegenstelling tot mensen, schermen, projectors, licht effecten, en specifieke bewegingen gebruiken om intenties en motivaties te communiceren. Om het gebruik van deze robot specifieke modaliteiten en uiterlijk van de robot te optimaliseren, hebben we geoptimaliseerd robot gedrag ontwikkeld. Dit hebben we gedaan in de context van de ontwikkeling van gids robots voor toeristische plaatsen. Om te begrijpen welke effecten zowel imitatie van menselijk gedrag, als robot geoptimaliseerd gedrag hebben op de perceptie van de interactiepartners, zijn gecontroleerde studies en studies in de echte wereld uitgevoerd. Allereerst hebben we geanalyseerd wat het effect en de effectiviteit van geïmiteerd menselijk gedrag is voor laag antropomorfe robots. In deze studies vonden we dat menselijk gedrag de voorkeur heeft boven random gedrag, maar dat het ook erg afleidend is. Dit betekent dat menselijk gedrag niet de beste oplossing is om gedrag voor laag antropomorfe robots te ontwerpen. Een belangrijke vraag was nu wat een veelbelovend alternatief zou kunnen zijn voor het gebruik van gekopieerd menselijk IX

12 Abstract gedrag. In opvolgende studies hebben we effect en effectiviteit van gekopieerd menselijk gedrag voor de robot vergeleken met gedrag dat was geoptimaliseerd voor het uiterlijk en de modaliteiten van de robot. Van deze studies hebben we geleerd dat geoptimaliseerd robot gedrag een goed alternatief is in plaats van het imiteren van menselijk gedrag voor laag antropomorfe robots. Om de bovengenoemde studies goed uit te kunnen voeren, hebben we DREAM, Data Reduction Event Analysis Method, ontwikkeld en geïntroduceerd. Deze methode maakt het mogelijk om video data van spontane (niet gecontroleerde) mens-robot interacties te analyseren. Een dergelijke methode bestond nog niet, maar is wel noodzakelijk om inzicht te verkrijgen in effectief gedrag voor robots. Omdat veel meer aspecten dan gedrag van de robot en de persoon die interacteert met de robot een rol spelen, zoals fysieke omgeving en de andere mensen in de omgeving, is het nodig om ook echte wereld veld studies naast gecontroleerde (lab) studies uit te voeren. DREAM is geïntroduceerd om rijke en ongestructureerde interacties tussen mensen en robot op een snelle en betrouwbare manier te analyseren. Het is gebleken dat deze methode effectief is om spontane mensrobot interactie in een gids/museum situatie te analyseren. Het werk dat in dit proefschrift gepresenteerd wordt is een eerste stap in het begrijpen 1) van effectief non-verbaal gedrag voor laag antropomorfe robots, 2) welke design methodes gebruikt kunnen worden om dit gedrag te ontwerpen en 3) hoe non-verbaal gedrag voor robots in spontane situaties bestudeerd kan worden. Dit heeft geresulteerd in de volgende resultaten: 1) Het creëren van robot geoptimaliseerd gedrag om gedrag voor laag antropomorfe robots te ontwerpen, en 2) DREAM, om spontane mens-robot interactie te analyseren. Deze resultaten kunnen als startpunt dienen om meer diverse en effectieve ontwerpmethoden voor mens-robot interactie te ontwikkelen. X

13 Content Chapter 1 - Introduction The influence of nonverbal behavior on communication Interacting with robots in everyday environments Approaches to design robots Research questions Research context Application areas for the FROG project Overview of chapters 11 C Chapter 2 - Background and related work Main concepts and definitions Important theories for human-robot interaction and social robotics Previous research on tour guides robots Studies involving the first generation tour guide robots Studies involving second generation tour guide robots Previous research into specific tour guide robot behaviors Framework to design behavior for low-anthropomorphic robots Conclusion 35 Chapter 3 - Contextual analysis; A robot in a tourist site? Motivation Related work Strategies in tour guiding Factors influencing the visitor experience Study 1: Contextual analysis of human tour guide behaviors Study design Results Discussion Study 2: Understanding visitor experiences Study design Results Discussion General discussion Understanding the context Phenomena influencing the design of behavior of a tour guide robot 59 XI

14 Content Focus on user experience in human robot interaction 60 Chapter 4 - Translating human nonverbal behaviors to robots Motivation Related work; How humanlike nonverbal cues influence humanrobot interaction Influencing the interaction with robot gaze Influencing the interaction with robot hand gestures Influencing the interaction with robot bodily movement and orientation Study 3: The influence of a low-anthropomorphic robot s eye-gaze on interactions with multiple users Hypothesis Study design Results Discussion Study 4; How does a tour guide robot s orientation influence visitors orientations and formations? Research question Study design Results Discussion General discussion 92 Chapter 5 - Beyond R2D2; Designing nonverbal interaction behaviors for robot-specific morphologies Motivation Related work; Designing nonverbal interaction behaviors for lowanthropomorphic robots Research question Two approaches towards the design of multimodal robot behavior Approach 1: human-translated behavior Approach 2: robot-optimized behavior Methods for evaluation Manipulation of the robot s behavior in both studies Method of the online video study (study 5) Method of the in-the-wild study (study 6) Results Attention Attitude towards the robot 116 XII

15 5.7. Discussion Discussion of the results Design approach to develop robot behavior Use of mixed methods for evaluation Conclusion 120 Chapter 6 - Experiences with a low-anthropomorphic tour guide robot; A case study with FROG in the wild Motivation Visiting cultural heritage with a tour guide robot: A user evaluation study in the wild (study 7) Study design Results Discussion 133 C Chapter 7 - Introducing DREAM; Bringing back video analysis to humanrobot interaction in the wild Motivation Related work DREAM: annotating videos using thin slices The pillars of DREAM Dataset to validate DREAM Study design Use of DREAM Validation of the method Results of the comparison of DREAM and counting predefined events Experiences of annotators Discussion Conclusion and future work 154 Chapter 8 - Discussion and conclusions Discussion of the main findings To what extent are humanlike nonverbal behaviors effective when transferred to low-anthropomorphic robots that interact with people? (RQ1) What is the best approach to design a consistent set of nonverbal communication behaviors for low-anthropomorphic robots? (RQ2) In what way should the effect of robot behavior on people s perception and experience be studied? (RQ3) 160 XIII

16 Content How should low-anthropomorphic robots display nonverbal behaviors in social interactions with people so that they are easy to understand and lead to positive experiences? (Main research question) Implications for theory Approaches to design robot behavior Recognize overlap in the fields of design and human-robot interaction Implications for future development of social robots Future work Conclusion 170 Appendix 173 Appendix A - Observation scheme guide behavior - study Appendix B - Questionnaire - study Appendix C - Annotation of participant s attention - study Appendix D - Coding scheme - study Appendix E - Questionnaire - study Appendix F - Interview questions - study Appendix G - Coding scheme - study Appendix H - Interview questions - study Bibliography 199 List of publications 213 Relevant for this thesis 213 Additional publications 213 SIKS Dissertation Series 215 XIV

17 1 Introduction In this chapter I will introduce the main goal of this thesis and provide the motivation for studying nonverbal behaviors for low-anthropomorphic robots. Furthermore, I will present the research questions that have guided the research. Next, I will describe the application context of the EU funded project FROG. Finally, I will give an overview of chapters for the remainder of the thesis as well as an overview of the research methods used. 1 1

18 CHAPTER 1 Introduction 1.1. The influence of nonverbal behavior on communication R2D2 is a famous robot from the Star Wars movies. If we were to encounter R2D2 in the real world, we would probably not literally understand the beeps and movements it makes. This is because R2D2 does not closely resemble people in the way it looks or behaves. However, we might be able to deduce R2D2 s intentions from the way it moves, beeps, drives and otherwise behaves. This implicit understanding may be due to the tendency of people to project humanlike behavior onto non-human things such as robots (Duffy, 2003; Fink, 2012), their social responses to computers and technology in general (Nass & Moon, 2000; Reeves & Nass, 1996) and their inclination to interpret behavior of non-human things in a humanlike manner (Heider & Simmel, 1944; Ju & Takayama, 2009). In the example of R2D2, it is not the verbally given information that people are able to understand, but people interpret the nonverbal communication signals to understand the situation. This nonverbal communication includes body movement, body posture, proxemics, gestures, facial expressions and tone of voice. It is that part of the communication that does not use actual words to explain concepts. It is known that nonverbal communication plays an important role in interaction. Several researchers have put forward that in human-human interaction between 60% and 93% of communication is nonverbal (Birdwhistell, 1990; Mehrabian, 1972; A. Pease, & B. Pease, 2004). Also in human-product interaction, product and interaction designers develop nonverbal cues to inform users of how to use the product (Preece, Sharp, & Rogers, 2002). For example, a coffee machine uses blinking or continuously shining light to show whether the coffee is ready. A ticket machine s lights blink to indicate what the user should focus on next. By using nonverbal communication in machines, extensive written instructions are rendered unnecessary and first time users can interact with the technology intuitively. Although it is unlikely for us to encounter R2D2 in an everyday situation, it is expected that we will encounter similar service robots that perform tasks to support us in various settings. This could be a laundry robot in a hospital, a delivery robot in an office, or a guide robot in a museum. Because these machines will need to operate in populated environments and understand and interact with people intensively, the interaction becomes more social rather than purely functional. For example, people expect robots to behave according social rules and get annoyed if the robot does not (Mutlu & Forlizzi, 2008). This indicates that interaction has a social component when robots have to react to social situations and their behavior most likely will be interpreted (a-) socially by people. As a result, this inherently social component of interaction places human-robot interaction between the highly social human-human interactions and the more functional human-product interactions. Therefore, in order to work effectively in a typically human environment that is full of social encounters 2

19 and social situations, robots, just like people, might benefit from adopting nonverbal communication behaviors Interacting with robots in everyday environments Robots that perform tasks in everyday social environments include service robots for professional use or personal use. As was reported by the International Federation of Robotics (2015), in 2014, the total number of professional service robots sold (such as medical robots or milking robots) increased by 11.5% compared with 2013 to 24,207 robots. The number of service robots meant for personal and domestic use (such as robot vacuum cleaners or handicap assistance robots) that was sold in 2014 was 4.7 million, which was an increase of 28% compared with The increase of sales in entertainment robots for personal use (such as LEGO Mindstorms or Furbies) between 2013 and 2014 was 40%, to about 1.3 million robots sold in Moreover, the predictions for are that about 152,400 service robots in the professional sector and about 35 million service robots in the personal sector will be installed by 2018 (International Federation of Robotics, 2015). This indicates an enormous growth of the service robots market in the coming years. 1 All of these robots, will to some extent encounter people when performing their tasks. Even the simplest robots that perform tasks in everyday work or home environments will need to be handled, fixed and worked or interacted with. Robots that people encounter or have to interact with in daily social environments will be held to the prevalent social norms in the environment of use. When a delivery robot passes people in a hallway, we might well expect it to move to the right to let us pass. We may get frustrated or annoyed if it positions itself in the middle or waits for a long time because it is stuck (Mutlu & Forlizzi, 2008). We might not understand if and how we can pass the robot or we might find the robot annoying. Alternatively, when such a robot stands in line, it should maintain a proper distance from the other people in line (Nakauchi & Simmons, 2002). The robot should not come too close to the person in front of it and its behavior needs to unequivocally show that it is queuing. These examples illustrate that getting nonverbal behavior right is challenging and requires insight into and understanding of local social norms and customs. Social behavior for robots is even more important and more complex when the interaction with people is more involving then passing or queuing. This is the case for example, for a robot that guides people to their gate in a busy airport (Joosse et al., 2015), a robot that helps children to engage in tasks, such as tidying their rooms (Zaga, Lohse, Truong, & Evers, 2015) or a robot that autonomously guides people through a museum (e.g. (Jensen et al., 2002)). In order for these robots to offer effective, efficient, safe and enjoyable services to people, their behaviors need to be easy to interpret and the interaction needs to be natural. 3

20 CHAPTER 1 Introduction Research into nonverbal communicative behaviors for social robots (discussed in Chapter 2) is often concerned with the development of humanlike robots that display humanlike behaviors to leverage familiarity and ease the interaction. However, there are limitations to this approach. While humanlike appearances and behaviors for a robot evoke natural interaction (Bischoff & Graefe, 2002) and people prefer to interact with robots in a way similar to the way they interact with other people (Fong, Nourbakhsh, & Dautenhahn, 2003), current capabilities of robots are limited. These technical limitations constrain the full imitation of human behavior (Duffy, 2003). Additionally, highly humanlike robots are complex, vulnerable, and expensive. Furthermore, for many tasks, a humanoid form for robots is probably not effective or efficient (e.g. a swimming lifeguard robot, dog walking robot). Therefore, it is imperative to consider robots that do not closely resemble people and explore alternative ways to design the nonverbal behavior of these robots beyond copying human behaviors. Although an increasing number of social robots will enter our society in the coming years and these robots are envisioned to behave according to human social norms in order to ease the human-robot interaction in everyday environments, robots often lack humanlike modalities. Moreover, it is unknown what effective nonverbal behaviors for robots that do not closely resemble people should be. This means that important questions arise around the appropriateness of copying human behavior to robots and that new forms of robot interactive behavior need to be defined Approaches to design robots In this thesis, I will disconnect the robot s behaviors from its appearance. It is not essentially necessary that a biologically inspired robot only shows humanlike behavior, neither should a functional robot show only functional behavior. The terms that I will use in this thesis, describe the appearance of the robot, while the focus of this thesis is on the exploration of effective nonverbal behaviors that is optimized for the robot s morphology. To avoid indistinctness, I will adopt the term humanlike robot for highly anthropomorphic robots that closely resemble people in appearance and interaction modalities. Humanlike robots have features such as arms, a head, and facial features. They are able to imitate humanlike gestures, show humanlike facial expressions and full body pose behaviors as well as realistic humanlike head and gaze cues. I will use the term low-anthropomorphic robot for robots with bodies that are very different from human bodies and subsequently lack physical properties to closely imitate humanlike behaviors. However, these robots may instead have interaction modalities that people do not have, such as screens to communicate information, colored lights to show intentions or beyond-human sensor capabilities. 4

21 This classification is comparable to the classification as proposed by Fong et al. (2003), who distinguished between two main robot-types. According to them robots can be designed either based on biological inspiration or on functional design (Fong et al., 2003). Biologically inspired robots are highly anthropomorphic robots that are developed to physically resemble biological entities such as people or animals. As a result, such robots are designed and developed to simulate human physical, cognitive and social capabilities. The androids of Professor H. Ishiguro are good examples of biologically inspired robots (Ishiguro, 2007; Minato, Shimada, Itakura, Lee, & Ishiguro, 2006; Sakamoto, Kanda, Ono, Ishiguro, & Hagita, 2007). However, the development of Robovie V1, and its successors (See Figure 1.1), also lead to quite humanlike robots (Ishiguro et al., 2001). Functionally designed robots are robots that are not inspired by the human form. Contrarily, their design is functionality driven and the physical as well as interaction design is optimized for interaction with people. Examples of functionally designed robots are the HUG (DiSalvo, Gemperle, Forlizzi, & Montgomery, 2003) and the Care-O-Bot 3 (See Figure 1.2). 1 Nevertheless, most robots will not neatly fit into the biologically inspired or the functionally designed class. In most cases influences from both classes are visible. For example a biologically inspired head and facial features on to the functionally designed robot body of Snackbot (See Figure 1.3) (Lee et al., 2009). This shows that the classification of Fong et al. (2003) does not cover the full range of robots. Therefore, I will refer to functionally designed robots that are provided with some biologically inspired features as low-anthropomorphic robots as well, because initially they are functionally designed and in most cases the seemingly biologically inspired features applied to those robots do not have similar functionality as people have. Figure 1.1: Robovie R V2; a humanlike robot (taken from robots/robovie-pc) Figure 1.2: Care-O-Bot 3; a low-anthropomorphic robot (taken from: (Mast et al., 2012)) Figure 1.3: Snackbot; a lowanthropomorphic robot with some humanlike features (taken from: (Lee et al., 2009)) 5

22 CHAPTER 1 Introduction Moreover, in their paper, Fong et al. (2003) focused mainly on biologically inspired robots, which according to them seems to be the most effective way to develop robots. This approach is strongly based on knowledge from Computer Science, and Behavior Sciences and Psychology. Nevertheless, Fong et al. (2003) acknowledge that other design approaches such as taken from Industrial Design and Interaction Design are focused on fewer. This indicates that robot used for social human-robot interaction are mainly biologically inspired and knowledge on how to develop effective behaviors for functionally designed robots is scarce. In this thesis, I will address this shortcoming by focusing more on design approaches suitable for functionally designed low-anthropomorphic robots. Social robots that do not have an anthropomorphic or low-anthropomorphic appearance such as those resembling animals (zoomorphic) or mechanical platforms will not be considered in the remainder of this thesis. While the existence of other biologically inspired robots such as zoomorphic is acknowledged, this thesis mainly focuses on humanlike aspects of robot and behavior design. Moreover, research in human robot interaction is often carried out on early prototypes, that look mechanical or machinelike (e.g. see (Hinds, Roberts, & Jones, 2004; Walters, Syrdal, Dautenhahn, te Boekhorst, & Koay, 2007)). While results from such studies are considered, these robot platforms are not included as examples of either humanoid or low-anthropomorphic robot types for the reason that they are unfinished concepts, and would not be able to perform tasks in real world environments Research questions This thesis concerns the challenge of designing effective behaviors for robots that intensively interact with people in social environments. More specifically, in this thesis I focus on the exploration of effective nonverbal behaviors for low anthropomorphic robots (robots that do not closely resemble people) in the context of guiding visitors at an outdoor tourist site. It stems from the premise that effective methods to develop nonverbal behavior for lowanthropomorphic robots are scarce and therefore, need to be studied. As previous studies have shown, people have the tendency to interpret movements of non-living objects, such as robots, in humanlike terms. This means that if a robot moves, but behavior is not or poorly designed, it will be interpreted as intended behavior and influence people s reactions in interaction. Therefore, it is imperative to understand the effects of nonverbal behaviors of low-anthropomorphic robots on human-robot interaction. This led to the main research question: How should low-anthropomorphic robots display nonverbal behaviors in social interactions with people so that they are easy to understand and lead to positive experiences? 6

23 To answer the main research question, several sub-questions also need to be answered. These are first of all born out of the lack of research into the effects of human behaviors transferred to low-anthropomorphic robots on the user s responses and experiences. Leading to the following research question: RQ1: To what extent are humanlike nonverbal behaviors effective when transferred to low-anthropomorphic robots that interact with people? The most commonly used approach to design social robot behaviors is to imitate human social behavior. However, it is as yet unknown what design approach is preferable for the design of nonverbal behaviors for low-anthropomorphic robots. Furthermore, while previous work addressed isolated robot behaviors, it is unclear how multimodal behaviors influence human-robot interaction. This leads to the following research question: 1 RQ2: What is the best approach to design a consistent set of nonverbal communication behaviors for low-anthropomorphic robots? Finally, the design of behaviors for robots that are to be deployed in real world everyday environments requires methodological approaches beyond laboratory studies or technology probe-type explorations. This leads to the final research question: RQ3: In what way should the effect of robot behavior on people s perception and experience be studied? Throughout the work presented in this thesis, these questions have guided the direction of the research. Additionally, the research design was influenced by the real world context in which the research was performed. In the next section, I will explain how the context influenced the research presented in this thesis Research context The studies presented in this thesis were carried out in the context of the EU 7th Framework project FROG (Fun Robotic Outdoor Guide), which ran from 2011 to The goal of the FROG project was to develop an outdoor guide robot with an engaging personality for the exploration of outdoor attractions. In order to inform the design of the robot s interactive behaviors, specific functionality, navigation behaviors and synthetic personality, several fields of expertise were involved in the FROG-project. These were vision-based detection, robotics design and navigation, human-robot interaction, affective computing, intelligent agent architecture, and dependable autonomous outdoor robot operation. All partners contributed with their expertise and combined their knowledge to create the robot platform FROG. 7

24 CHAPTER 1 Introduction The robot used in three of the empirical studies of this thesis was the FROG robot that was developed within the FROG project. Even though this robot turned out to be green, had big eyes and was called FROG, it was designed as a functional robot to guide visitors and not biologically inspired on frogs. Moreover, the robot s role was unrelated to being a frog, only name and appearance slightly referred to frogs, but it did not display any frog-like behaviors. Therefore, I did not expect a zoomorphic effect, which means that I did not expect people to expect frog-like behaviors of the robot. Although the FROG project was the starting point for this thesis, the studies performed and topics used were carefully selected to find a best way to develop effective multimodal nonverbal behavior for a low-anthropomorphic robot, such as FROG. Through this research line, I informed the nonverbal behavior design of the robot in the FROG project. Moreover, by doing these studies, I gained a lot of knowledge on behavior design practices and realworld research methods in human-robot interaction and how these might be improved, which goes beyond the scope of the FROG project. The development of an autonomous robot tour guide that engages visitors in a fun exploration of a tourist site offers a relevant case study for the application of nonverbal robot behaviors. This context of the FROG project provided a real world context, a low-anthropomorphic platform, and relates to a body of previous work on effectiveness of humanlike behaviors, which made this context very suitable to explore the nonverbal communication behavior for a social robot Application areas for the FROG project Within the FROG project, two areas of application and related real world test locations were chosen. The two environments that combined an outdoor nature with many visitors were a Zoo and a famous Historical site. The test locations were the Lisbon City Zoo in Lisbon, Portugal and the Royal Alcázar, a historical royal residence in Seville, Spain. Both sites offer interesting and challenging opportunities for having robot guides guiding visitors. Moreover, both sites attract international visitors, which enables to generalize the findings for other places as well. The Lisbon City Zoo is a park showing several species of wild animals to the public that aims to educate people about nature and animals. Yearly, approximately 800,000 visitors (national and international) visit the Zoo. The website of the Lisbon City Zoo ( zoo.pt/) provides information about the history and the mission of the Zoo. The Zoo was founded in 1884, and since 1905 it has been at its present location. The mission of the Lisbon City Zoo is to educate visitors, but also to develop both a zoological and botanical park, as a center for conservation, breeding and reintroduction of endangered species through scientific research and environmental enrichment programs. Accordingly, in

25 a Teaching Center opened where the Zoo provides free education to school classes to raise awareness of the problems of the populations of species and the environment. Currently, the Zoo houses over 2000 animals of 300 different species, including mammals, birds, reptiles and amphibians. Where animals were showcased in the earlier years, now the Zoo takes an active role in the protection and conservation of nature. This means that animals live in enriched environments to stimulate the natural behaviors of each species. The guides are employees of the Lisbon City Zoo, which means that the guides who gave the tours in the Lisbon City Zoo are trained and employed by the Zoo. Visitors of the Zoo are mainly families with one or two (young) children, couples with or without children, school classes and groups of friends. See Figure 1.4 for an impression of the Lisbon City Zoo. 1 Figure 1.4: Impression of the Lisbon City Zoo The second site selected to perform studies was the Royal Alcázar in Seville (Spain). The Royal Alcázar is a royal home, however the royal family does not live in the buildings of the Royal Alcázar. Yearly about 1200,000 visitors visit the Royal Alcázar, 5500 visitors a day in busiest periods in May to 1500 visitors a day in January. Visitors stay on average for 90 minutes in the site and the most popular location to visit is the Mudéjar Palace. The first building was built in the ninth century and during the ages Christians and Muslims have built, destroyed and rebuilt the buildings on the site. Here, visitors can see how the Christian 9

26 CHAPTER 1 Introduction and the Muslim architectural styles are mixed. Guides who offer tours in the Royal Alcázar work for independent agencies or are selfemployed. All guides must have certification, but the board of the Royal Alcázar does not regularly control the guides. The guides who give tours here do not only guide visitors through the Royal Alcázar, but also offer guided city tours through Seville. Usually, a visit to the Royal Alcázar is part of such a city tour. Visitors of the Royal Alcázar are mostly couples (with older children), groups of tourists and school classes. See Figure 1.5 for an impression of the Royal Alcázar. Figure 1.5: Impression of the Royal Alcázar 10

27 1.6. Overview of chapters In this section I will describe the content of the chapters and the methods used for the studies. In chapter 2, I will discuss the state-of-the-art in social robot research and design of social robots that informed the studies in this thesis. First, I will focus on theories and frameworks that influence the design of appearance and behaviors of social robots. Afterwards, I will review studies that have been performed with tour guide robots. This led to insights into how the design of behavior for robots is currently influenced as well as to insights on the state of the art in effective behaviors for guide robots. In chapter 3, I will focus on understanding the context of tourist sites. To this purpose, two studies have been carried out. The first study was to gain insight into the effect of human tour guide behaviors (study 1). In the second study, the focus was on understanding the visitor experiences of visiting a site with and without tour guides (study 2). This was carried out to gain rich insights into effective behavior strategies in human guiding and current visitor s experiences of the sites. 1 In chapters 4 6, I will present five iterative studies of human-robot interaction. In chapter 4, I will present two studies with low-anthropomorphic tour guide robots that used copied humanlike behavior to explain about points of interest. For these robots one singled out modality was manipulated. In the first study, gaze behavior was manipulated, while in the second study orientation behavior was manipulated. These studies led to insights on whether and how humanlike behavior could be used for low-anthropomorphic robots. In chapter 5, I will explore an approach to develop behavior for low-anthropomorphic robots. By using this approach to develop nonverbal behavior for low-anthropomorphic robots, the behavior will be optimized for the robot s morphology and the modalities that the robot has. Results of the two studies presented in this chapter give insights on how people react to robot-optimized behavior compared to human-translated behavior in terms of attention towards the robot and attitude towards the robot. In chapter 6, I will describe a user evaluation of tours given by FROG. Results provide insight into how people experienced the robot-optimized behaviors designed for FROG in real world robot guided tours. Two of the studies presented in chapters 4 and 5 were performed in controlled settings. These were a controlled lab study into the effects of robot gaze behavior on the responses of participants (study 3) and a video study to assess the influence of robot multimodal behavior on participants understanding of the behavior (study 5). These controlled 11

28 CHAPTER 1 Introduction studies allowed me to study user responses to specific nonverbal behaviors of the robot in a controlled way and compare participants evaluations of two conditions of robot behavior. The other two studies of these chapters (study 4; study 6) were a semi-controlled comparison between two conditions and performed in an in-the-wild setting. These were performed to gain insights into the effects of robot orientation on user responses and experiences (study 4) and to assess the effects of different sets of multimodal behavior on participants experiences (study 6). The in-the-wild approach allowed the observation of natural and original responses to the robot. The data from these studies was less controlled, but offered rich insights into peoples experiences. The study presented in chapter 6 (study 7) was performed in an in-the-wild setting with a fully autonomous robot. This allowed the observation of actual interaction between the robot and real visitors and interviewing of visitors who interacted with the robot. See Table 1.1 for an overview of the studies that I performed in relation to the research methodologies. Table 1.1: Overview of studies presented in this thesis Single-modality studies Multi modality studies Collected data Efficiency of behavior (fully controlled two conditions) Gaze behavior in the lab (chapter 4 study 3) Online video study of multimodal behavior for FROG (chapter 5 study 5) Questionnaire data on participants perception of robot behavior Experience of behavior (unstructured two conditions) Orientation behavior in the wild (chapter 4 study 4) In-the-wild study of multimodal behavior for FROG (chapter 5 study 6) Observation of visitor reactions and short interviews with visitors DREAM Experience of a full FROG tour (uncontrolled one condition) FROG evaluation in the wild (chapter 6, study 7) Visitor speech during tour, observations and extensive interviews with invited participants and short interviews with spontaneous visitors. In chapter 7, I will introduce DREAM, the Data Reduction Event Analysis Method. In this chapter, I will describe DREAM, a method I developed and evaluated to analyze the rich and unstructured interactions that were video recorded during the semi-controlled in-the-wild studies. I found that using DREAM resulted in reliable findings for objectively observable actions and changes in situations. This offers opportunities to learn from rich and valuable real life data to create even better future robots. Finally, in chapter 8 I will answer the research questions as proposed in this chapter and 12

29 discuss the results. Furthermore, I will discuss what implications the findings presented in this thesis have on theory and practice. I will close the thesis with directions for future work. 1 13

30

31 2 Background and related work In this chapter I will describe the essential background information for the overall thesis and offer working definitions for the main concepts in social robotics. Then, I will describe relevant theories in human-robot Interaction. Next, I will give an overview of tour guide robot platforms developed to date and a review of previous research into the effects of tour guide robot behavior design on human-robot interaction. Finally, the main methods towards approaching the design of social robot behavior will be discussed and the main framework of practice for this thesis will be provided. 2 15

32 CHAPTER 2 Background and Related Work 2.1. Main concepts and definitions This thesis concerns the challenge of designing effective behaviors for robots that intensively interact with people in social environments (See Chapter 1). The robots this thesis focuses on in general have the ability to detect and interpret human behavior in typical everyday environments such as offices, shopping malls, museums, and hospitals. Moreover, these robots are able to operate in such inherently social environments: they can detect and interpret people s behavior, they can navigate autonomously while taking into account the people in the environment, they are able to adapt their behaviors to people s responses and situations, and they can engage in prolonged interactions with people. Because of the abilities of such robots to operate in, deal with and adapt to the social environments in which they are deployed, these robots are considered social robots. Several human-robot interaction researchers have proposed definitions to describe the concept of social robots. For example, Breazeal (2003) describes social robots as the class of robots to which people attribute humanlike characteristics in order to interact with them. Fong, Nourbakhsh and Dautenhahn (2003) use the term socially interactive robots to describe robots for which social interaction plays a key role. Furthermore, Hegel, Muhl, Wrede, Hielscher-Fastabend, and Sagerer (2009) argue that people attribute social qualities to a robot if the robot contains a robot and a social interface. From these definitions, we can conclude that social robots are robots that have interactions with people in social situations. What the definitions do not describe is how the robot should communicate and what the flow of social interaction between a robot and a person should be like. Therefore, these definitions do not address the main concerns of this thesis. Bartneck and Forlizzi (2004) also proposed a definition for social robots in which they focused more on social situations. They define a social robot as follows: A social robot is an autonomous or semi-autonomous robot that interacts and communicates with humans by following the behavioral norms expected by the people with whom the robot is intended to interact. This indicates that in order to have meaningful interactions with people, a social robot should be able to read and process social behavior of people as well as react to it in normative and social ways. A limitation of this definition is that the concepts of social and normative behavior are not further defined. Also, this definition does not address how people might experience interacting with a social robot. Taking the above definitions into account, the working definition of social robots for this thesis should focus on adhering to the behavioral norms and achieving effective and fun interaction with people in a social context. Regardless of the robot s morphology and regardless of the interaction modalities it has at its disposal, it should effectively engage with people to achieve specific outcomes related to the service it offers and elicit positive 16

33 user experiences. Therefore the working definition proposed for this thesis is: Social robots are robots that engage and interact with people, effectively display multimodal behaviors and take the social context into account to achieve a desired interaction outcome and user experience in everyday social environments. Interestingly, several authors have suggested that in order for a robot to be social, it requires humanlike features and imitations of human communication behavior (e.g. (Bartneck & Forlizzi, 2004; Breazeal, 2003; Fong et al., 2003; Hegel et al., 2009)). According to this, only robots that resemble people in appearance and that can imitate human communication behavior should be used as social robots. However, the current state of the art in robot technology constrains opportunities to fully imitate people with robots (Duffy, 2003). Therefore, in this thesis I argue against this notion that human-likeness is essential to create an effective social robot. Instead, I propose that careful design of the behavior of low-anthropomorphic robots can offer an equal or even better human-robot interaction experience. 2 The definition that I propose is not limited to humanlike robots but includes robots that do not look like people at all. As long as the robot offers an effective interaction and positive user experience and is able to interact with people in an environment which is inherently social, it is considered a social robot. Consequently, this definition does not restrict social robots to robots that look like and/or behave like people. Rather, it leaves room to explore alternative manifestations of both appearance and behavior in social robots. To better understand the processes that determine people s perceptions of, attitudes towards and responses to social robots and social robot behavior, in the next section I will discuss the most relevant theories from Computer and Social Sciences. Also, I will discuss design approaches from Product and Interaction Design, to broaden the view of humanrobot interaction from mainly biologically inspired to other forms of social interactive robots Important theories for human-robot interaction and social robotics In this section I will discuss 5 relevant theories and frameworks that can inform the design of social robots and their behaviors. First, I will briefly describe the notion of Anthropomorphism and several studies that explore animacy in non-living objects. Second, I will discuss Mori s Uncanny Valley Theory. Third, I will discuss Reeves and Nass Media Equation and its consequences for the design of a robot s interactive behavior. Fourth, I will discuss the theory of Product Personality and the product personality scale that Mugge, Govers and Schoormans developed. Finally, I will describe Norman s Levels of Design in 17

34 CHAPTER 2 Background and Related Work relation to human robot interaction and the design of robot behaviors. Anthropomorphism is the act of attributing humanlike characteristics, motivations, intentions, and emotions to non-human organisms and objects and is considered to be an inherent tendency of human psychology (DiSalvo & Gemperle, 2003; Duffy, 2003; Epley, Waytz, & Cacioppo, 2007). This phenomenon can be leveraged by purposefully designing humanlike characteristics on to objects. Nevertheless, this tendency is also present if human characteristics, such as intentions and emotions have not been designed or programmed for a robot. An example of anthropomorphism is to feel sad for a robot that is being dismantled or to think a robot feels sad because you insulted it. Anthropomorphism seems to be evoked even by the simplest of objects. For instance, Heider and Simmel (1944) observed that participants assigned intentions and emotions to moving triangles and dots in a simple animation. When people only saw a still image of the triangles and dots no intentions or emotions were attributed (Heider & Simmel, 1944). Michotte (1963) performed studies with two moving balls and showed that people described some of the movements of the balls as factual, while for other movements people attributed motivations, emotions, relationships and even gender and age to the two balls. Furthermore, Ju and Takayama (2009) showed that people interpreted movements of doors that open automatically as intentional gestures. This suggests that people read intentions, motivations and emotions in the behavior of non-living things that do not look like people. For the field of human-robot interaction this means that not only a robot s humanlike features or overall humanlike appearance may elicit anthropomorphism, the behavior of robots can evoke anthropomorphism too (Disalvo, Gemperle, Forlizzi, & Kiesler, 2002). The above studies also strengthen the argument made previously that robots do not need to have humanlike characteristics in appearance to evoke attributions of emotion or intentions. Nevertheless, several researchers have argued that human-likeness is essential for social robots. Epley et al. (2007) argue that the purposeful use of humanlike behavior and appearance for technological agents can benefit the interaction. They state, for example, that human-likeness can enable a sense of efficacy with technological agents, and that it may increase social bonding with technology which in turn might increase liking of the technology (Epley et al., 2007). Moreover, Waytz, Cacioppo and Epley (2010) state that anthropomorphic technology lead to people follow social rules in interaction with the technology. Further, the technology was perceived as more understandable, predictable, intelligent and credible. It was found to increase engagement, appeared more effective in collaborative decision-making tasks, and was preferred because it could express emotions in a humanlike fashion (Waytz et al., 2010). However, they also state that humanlike appearance and behavior for computers and robots has its limitations. For instance they found that people were more likely to blame humanlike technology when it malfunctioned, 18

35 people felt less responsible for success, and it generated inappropriate expectations of the capabilities of the technology (Waytz et al., 2010). An important question therefore is: how humanlike should a robot s morphology be? This question is partly answered by Lee, Šabanović and Stolterman (2016) who interviewed potential users about their responses to 27 robots. Some of those robots had humanlike appearances, while others were minimalistic in form. They found that participants did not refer to robots as being humanlike or non-humanlike. Instead, they referred to parts of the robot as being humanlike (e.g. the hands, the eyes or the body). More importantly, participants evaluated how well humanlike features would fit in their everyday use contexts. This could indicate that humanlike appearances of robots are evaluated positively by users only in specific contexts of use, and that humanlike appearance in not necessary to create social robots (Lee et al., 2016). Duffy (2003) emphasized that it is important to separate the use of humanlike features from anthropomorphic tendencies. The use of humanlike features influences the robot s form and function, such as humanlike posture, and expression of emotions through a dynamic face. People s tendencies to anthropomorphize influence how these features are perceived, and how people interpret form and behavior of the robot. This tension between form and function is further addressed in the Uncanny Valley theory. 2 Mori (1970) argues in his Uncanny Valley theory that the more a robot resembles a human, the more people will feel familiar with it (see Figure 2.1 for a graphical representation of the uncanny valley). This familiarity in turn would benefit the interaction with the robot. However, he also poses that robots which resemble people so closely that they are difficult to distinguish from people are perceived as uncanny, because they are not quite real humans. This effect is called the uncanny valley (Mori, MacDorman, & Kageki, 2012). Moreover, the effect is thought to exacerbate when robots start moving, because movement shows even more clearly the limitations of a robot that closely resembles a human (Mori et al., 2012). MacDorman (2006) has shown through multiple studies that there are likely more, as well as more complex factors than familiarity that influence the comfort or discomfort people feel when interacting with a robot. A major impact is thought to be the discrepancy between the way the robot looks and the way it moves or behaves. For example, a mismatch between auditory and visual stimuli can elicit this feeling of eeriness (Meah & Moore, 2014). Similarly, an extremely humanlike robot will become creepy when its movements are jerky. In contrast, a simple box-like robot may seem surprisingly intelligent if it shows abilities beyond people s expectations. This notion of aligning form and function can also be found in product design where a product needs to display behaviors that correspond to user s expectations. For example, 19

36 Background and Related Work CHAPTER 2 Figure 2.1: Graphical representation of the uncanny valley (Mori, 1970) a tiny vacuum cleaner that generates an incredible amount of noise might not meet the user s expectations. The discrepancy between visual and auditory characteristics of the product might lead to confusion or even irritation. Instead, an optimal match where product characteristics correspond with the intended, overall experience may lead to more preferred products (Hekkert, 2006; Ludden & Schifferstein, 2007). Closely connected to people s tendency to assign human traits to non-human entities is the Media Equation: People respond to media as they would to a person (Reeves & Nass, 1996). Reeves and Nass (1996) argue that Individuals interactions with computers, television, and new media are fundamentally social and natural, just like interactions in real life, even though people are not aware of these responses. Moreover, in another study was shown that even when robots are not humanlike in appearance, people tend to interpret the movements of the robots as social actions (Terada, Shamoto, Mei, & Ito, 2007). While researchers have argued that people are likely to respond socially to robots because of their anthropomorphic form (Fong et al., 2003; Hinds, Roberts, & Jones, 2004; Mirnig, Riegler, Weiss, & Tscheligi, 2011; Sardar, Joosse, Weiss, & Evers, 2012; Siegel, Breazeal, & Norton, 2009; Syrdal, Dautenhahn, Woods, Walters, & Koay, 2007; Walters et al., 2007; Wang et al., 2010), Nass and Moon (2000) have stated that these social responses are not triggered by the anthropomorphic form of technology. They instead argue that people 20

37 seem to react socially to several characteristics in the technology, such as a voice for output, interactivity and taking on certain roles that were traditionally carried out by people (Nass & Moon, 2000). Also, the desktop PC s used by Nass and colleagues in their studies into politeness and reciprocity to computers, did not have a humanlike appearance while people did react to these computers socially (Nass, Fogg, & Moon, 1996). This again strengthens the argument that robots do not need to look nor do they need to behave in a strictly humanlike manner to be perceived as and responded to socially. It has been suggested that the behavior of a social robot should be consistent over time, so that people can predict what will happen next (e.g. (Duffy, 2003; Kim, Kwak, & Kim, 2008; Walters, Syrdal, Dautenhahn, te Boekhorst, & Koay, 2008; Woods, Dautenhahn, Kaouri, Boekhorst, & Koay, 2005). Moreover, these researchers state that to ensure that robot behavior is perceived as consistent over time and to facilitate predictable and understandable actions, a personality profile can be developed for the robot. The personality profiles that are created for robots these days are mainly based on human personality profiles such as the Big Five inventory (Digman, 1990) and the MBTI (Myers-Briggs Type Indicator) (Myers, Kirby, & Myers., 1993) measures, and inventories that are based on these. However, a disadvantage of the use of measures from human psychology for research into robot personality is that not all human characteristics can be meaningfully transformed to robots. Even though one of the Big Five Dimensions, extraversion, is clearly recognizable in robots, the other four dimensions (conscientiousness, neuroticism, openness and agreeableness) were found more difficult to apply to and to recognize in robots (Lippa & Dietz, 2000). 2 An alternative approach to create a consistent behavior and personality for a robot could be to look at how people experience the personalities of products. People use human personality characteristics to describe their impression of a product: this notion has been used to define and study product personality (Govers, 2004). Product personality describes the overall impression that a product makes on someone and can be influenced by different product characteristics such as its sensory properties, the perceived quality and the product s behavior. Theory on product personality has been used as a tool to design for effective and specific experiences (Govers, Hekkert, & Schoormans, 2003; Janlert & Stolterman, 1997). Moreover, Mugge, Govers and Schoormans (2009) developed a scale to define a product personality and to evaluate the product personality. They found that a product s personality (including appearance, behavior, sounds, smell, interaction; the entire experience of a product) can be described by 20 distinguishable dimensions, such as cheerful, relaxed, provocative and honest. These dimensions can be used by designers to design a personality for a product, and they can be used to test whether the designed personality is recognized by the users of the product. All these dimensions are applicable to products, it could therefore be 21

38 CHAPTER 2 Background and Related Work expected that they are applicable to robots as well. The 20 scales of product personality might therefore be more beneficial to use in defining and designing robot personality than the Big Five Personality Dimensions. Using the product personality scale could lead to a specific, pre-defined product experience for appearance and behavior of a robot. Moreover, the product personality scale can also be used to evaluate the user s experience of the interaction with the product (Desmet, Ortíz Nicolás, & Schoormans, 2008). To better understand what defines someone s experience of a product I use Norman s (2004) categorization of levels of design. Within these Three Levels of Design, Norman proposes three levels of experience of products that together define how people experience the interaction with technology: the visceral level, the behavioral level and the reflective level (Norman, 2004). The visceral level concerns fast affective reactions, the first basic reaction people have towards a product, such as I like this or It looks nice. The behavioral level considers people s responses in interactions. This level is more cognitive than the visceral level and comes into play when someone interacts with the product. This leads to reactions such as this is effective to use. The reflective level explains how people attribute meaning to a product and how they reflect on their experience of using the product. A reaction on the reflective level could for example be I felt really confident while using this product. The design of technology, such as robots, can trigger experiences on all three levels. For example, on the visceral level, a person might think that robot looks cute and I want to interact with it or that robot looks creepy and I don t want to get any closer to it. On the behavioral level, the right functionality and effective behavior of a robot might evoke reactions like this robot supports me very well. Finally, when people encounter a guide robot, the appearance and behavior of this robot together contribute to the evaluation on a reflective level of a person that her or his visit to a tourist site has been a memorable experience partly because of the positive experience with the tour guide robot. Considering these three levels of design could aid robot developers in translating a desired interpretation and reaction to actual designs of robot appearance and behavior. To create a consistent and effective product experience, robot developers should think about how to create a pre-defined personality on all three levels of design. In this way, the overall product experience link back to the theories that were discussed at the beginning of this section. A robot that has consistent characteristics (appearance, behavior, sounds, smell, interaction) that correspond with the intended functionality might be liked by people (Hekkert, 2006; Ludden, Schifferstein, & Hekkert, 2006), while inconsistent appearance and behaviors might lead to a robot that falls in the uncanny valley (Mori et al., 2012). To gain an understanding of how these theories and frameworks influence the actual development of tour guide robots, the next section will provide an overview of the most important studies with tour guide robots in human-robot interaction. 22

39 2.3. Previous research on tour guides robots This section reports a review of previous studies into human-robot interaction for tour guide robots. For this review of the existing literature, I focused on studies that involved tour guide robots that gave actual guided tours to real visitors. The resulting studies included evaluations of robot guided tours in university labs, in museums, at cultural heritage sites and at fairs. Furthermore, I included more controlled (laboratory) studies into the effects of specific robot behaviors on the interaction, such as head movement or pointing behavior. The resulting approximately 50 papers could be categorized in roughly three categories: research on autonomous tour guide robots in museums, fairs or exhibitions; research into the effectiveness of a specific tour guide robot behavior or modality; and research on navigation and localization in populated environments. In this section, I will focus on the first two categories. Even though in numerous papers robust navigation and localization in unpredictable dynamic and otherwise challenging environments was studied, I will not further report on this direction of work. I acknowledge that the consideration of localization and navigation is essential for effective autonomous tour guides, however, in this thesis the focus is on papers that have studied and report the effects of robot behaviors on humanrobot interaction. 2 The robot platforms that were involved in the studies that were included in the review could be described as first generation and second generation tour guide robots. The first generation tour guide robots had limited sensing capabilities and mostly detected people as dynamic objects. The second generation tour guide robots had more extensive sensing capabilities, for instance because of more evolved computer vision. These robots were able to detect and interpret more complex human behavior and social situations and had more adaptive behaviors. The robots that were used in these studies to evaluate specific behaviors were often remote controlled, but mainly had similar capabilities to those of the second generation tour guide robots. In the remainder of this section, I will first describe the research into first generation tour guide robots (2.3.1), then research into second generation tour guide robots (2.3.2). Finally, I will discuss the findings from previous work concerning the effects of specific robot design and behaviors on the users perception and experience (2.3.3) Studies involving the first generation tour guide robots Tour guide robot research probably started with the development of Polly, a robot that gave lab tours at the 7th floor of the AI lab at MIT (Horswill, 1993). Visitors to the lab could wave their feet in front of the robot to indicate that they wanted to join a tour. Then Polly guided the visitors to several locations in the lab. This research mainly focused on creating a low cost vision system for mobile robots and did not yet focus on effective human-robot 23

40 CHAPTER 2 Background and Related Work interaction between Polly and the visitors. The first tour guide robot deployed in an actual museum setting was Rhino (Burgard et al., 1999). The faceless robot Rhino (See Figure 2.2) guided hundreds of visitors in the Deutsches Museum Bonn (Bonn, Germany) during a six-day experiment in Even though the main focus of the research seemed to be on safe navigation and localization, some interaction capability was realized. For example, information displayed on a screen, using integrated graphics, sound and motion. Visitors could request further information from Rhino by pressing a button. Interestingly, the authors found that visitors continuously blocked Rhino s path to make it blow a horn that indicated it wanted to pass (Burgard et al., 1999). One explanation for this robot abuse by visitors of the museum may be that even though Rhino did a good job navigating and localizing, visitors had a desire to influence its actions or wanted to interact more with the tour guide robot. With their behavior they may have tried to evoke responses from the robot. After Rhino, Minerva (Thrun et al., 1999) and the Mobot Museum Installations (Willeke, Kunz, & Nourbakhsh, 2001) were developed. Minerva (See Figure 2.2) was installed at the Smithsonian s National Museum of American History (Washington, US) for a period of 14 days in 1998 (Thrun et al., 1999). The Mobot Museum Installations was a series of iteratively improved robots, Chips (See Figure 2.2), Sweetlips, Joe Historybot and Adam, that were deployed in several museums or fairs (Willeke et al., 2001). Mounted on a cylindrical base, they had dynamic faces that were able to show simple emotions (happy, neutral, sad and angry) to indicate intentions and internal states and to smoothen the human-robot interaction. For these robots, research effort was spent on developing the human-robot interaction. For example, the tours of the robots were composed dynamically to keep visitors engaged during the tours (Thrun et al., 1999). Moreover, while Chips simply showed movies at each stop, its successors were increasingly interactive and engaged with the user in a dialogue (Willeke et al., 2001). Apart from robots for museums, several robots were developed for fairs. For example, Diligent (See Figure 2.2), was developed to welcome and guide visitor to the LAAS stands at the conferences at the Cité de l Espace and the SITEF 2000 Exhibition (Alami, 2002). Blacky was developed to entertain and guide visitors at three different trade fairs in 2001 (Rodriguez- Losada, Matia, Galan, & Jimenez, 2002). Urbano gave lab tours and demonstrations at the Universidad Politecnica de Madrid laboratory and several fairs between (Rodriguez-Losada et al., 2008). The aim of these robots was to guide visitors to their desired destinations at the fairs. Also here, researchers observed that people liked to block the path of the robots. 24

41 Related to that, Alami (2002) observed that visitors often did not notice that Diligent had reached its final destination, as they were mostly interested in playing with it by blocking its path to see how the robot would deal with it. In their field studies they observed that people covered the cameras of the robot in order to block its path. However, this did not work, because the robot did not rely on the cameras but used laser range finders to navigate and avoid collisions (Alami, 2002). For Blacky, the effect of behaviors to react appropriately to this abuse was studied. When the path of the robot was blocked, the robot would pause the content of the tour, ask visitors to clear the way and then resume the tour again (Rodriguez- Losada et al., 2002). However, they found that visitors, specifically children, still liked to block the path of the robot. This might be explained by the notion that people, in their interaction with the robot, attempted to understand how the robot worked. By blocking its path and covering its cameras, they came to an understanding of what the robot reacted to and what its limitations were. While this could be a sense-making process, on the other hand, it could also simply be a matter of people having fun teasing the robot. Three entertainment robots, the Inciting, the Instructive and the Twiddling (See Figure 2.2), were developed for the Museum für Kommunikation (Berlin, Germany), installed in 2000 and still in operation (Graf & Barth, 2002) have a designed morphology but lack facial features. Even though these robots do not resemble people, it is easy to anthropomorphize their appearance. The robot Hermes, that was not designed as a tour guide, but deployed at the Hannover Fair in 1998 at the Heinz Nixdorf MuseumsForum in Paderborn (Germany) (Bischoff & Graefe, 2002), looked more humanlike than the previously described robots, even though is its body was quite basic and rectangular and its head lacked facial features as well. These four robots communicated their expression of intentions and emotions with movements and tone of voice rather than dynamic facial expressions. These behaviors were designed as imitations of human behavior. 2 In their studies, Bischoff and Graefe (2002) found that the humanlike form of the robot, even though it lacked a dynamic face, encouraged people to interact with the robot and assume human-human interaction roles. While overall appearance and communicative behavior of these robots was carefully designed to study human-robot interaction, visitors still liked to block the path of the robots, push the emergency buttons or touch the tactile bumpers to stop the robot (Bischoff & Graefe, 2002; Graf & Barth, 2002). This suggests that even with a more humanlike posture and carefully designed interactive dialogue, people still tend to test the robot physically. First generation tour guide robots were overall simple of appearance, though some did have more sophisticated humanlike shapes and facial features. What these robots have in common is that technical challenges as well as challenges in navigation and localization were the main focus of the research group deploying the robot. However, some investigations 25

42 CHAPTER 2 Background and Related Work into people s responses to the robots were carried out. Overall, these observations showed people s tendency to abuse or try out the robots by blocking their path, pressing buttons and so on. These studies showed that when robots enter public environments, people will actively engage and interact with these robots. These observations indicate the importance of carefully designing the human robot interface, the robot s input and output modalities and behaviors. The first generation tour guide robots in social environments were a novelty. People will still be unfamiliar with the abilities of robots and the services they may provide for the coming years. This indicates that in order to guide people through a robust sensemaking process, and to ensure optimal usage, robots in public places require careful design. Figure 2.2: Impression of the first generation tour guide robots. From left to right: Rhino (taken from: (Buhmann et al., 1995)), Minerva (taken from ( realprogress/), Chips (taken from: Diligent (taken from: (Alami, 2002), and the three entertainment robots (taken from: com/2006/03/14/robot-trio-of-museum-fur-kommunikation-2/) Studies involving second generation tour guide robots While the first generation tour guide robots was limited to a few types, appearance and behavior of second generation tour guide robots showed much more variation. Robots like Robox (Siegwart et al., 2003), Rackham (Clodic et al., 2006) and SCITOS G5 (Poschmann & Donner, 2012) still showed similarities with the first generation tour guide robots in their appearance; they lacked arms and they used screens and buttons to interact with the visitors. Contrarily, robots like Robovie (Shiomi, Kanda, Ishiguro, & Hagita, 2006), and Robotinho (Nieuwenhuisen, Gaspers, Tischler, & Behnke, 2010) had humanlike interaction capabilities, such as arms, a head and eyes. Even though the interaction opportunities of these robots were more elaborated than those of the first generation tour guide robots, when visitors become interested in the technology, they were found to test the system, for instance, by blocking its path (Drygajlo, Prodanov, Ramel, Meisser, & Siegwart, 2003). The robots of the RoboX series (See Figure 2.3), a series that contained 10 robots that guided visitors through the Swiss National Exhibition Expo.02 for 5 months in 2002 (Siegwart et al., 2003) and Rackham, which guided visitors through the Cité de l Espace in Toulouse France every three months for two weeks between March 2004 and November 2006 (Clodic et al., 2006) are not extremely humanlike, but their heads look like rather simplified human 26

43 faces. Also, the existing platform SCITOS G5 (See Figure 2.3), which was used to guide visitors during 5 days in the museum Technische Sammlungen Dresden (Poschmann & Donner, 2012), had a simplified humanlike appearance. Additionally, these robots were equipped with buttons and screens to support the interaction. Results from observations showed that also for these robots, visitors would immediately start to play with the robots, without waiting for an explanation on how to use it. Visitors figured out how to control the robot by trying or by looking how others interacted with the robot (Jensen et al., 2002). For robot Rackham, researchers found that visitors needed the robot to show its status and intentions Clodic et al., 2006). Robovie and Robotinho (for both see Figure 2.3) resembled people even more in appearance and communication behavior. Robovie guided visitors through the Osaka Science museum (Osaka, Japan) during a 2 month trial (Shiomi et al., 2006) and Robotinho guided visitors through the Deutsches Museum Bonn (Nieuwenhuisen et al., 2010). To create the behavior for these robots, human behavior was imitated as closely as possible. Behaviors that could not be performed due to limitations in the robots appearance were omitted. As a result, some interactions that human guides perform were not possible, for example natural language conversation. This was either because the feature was not implemented in the robot (Shiomi et al., 2006), or speech recognition failed in noisy environments (Nieuwenhuisen et al., 2010). This shows that also the humanlike robots were not yet capable of initiating smooth and natural interactions with the visitors. 2 Figure 2.3: Impression of second generation tour guide robots. From left to right: RoboX (taken from: (Tomatis et al., 2002), SCITOS G5 (taken from: (Poschmann & Donner, 2012)), Robovie V1 (taken from: (Shiomi et al., 2006), Robotinho (taken from: (Faber et al., 2009). Even though all second generation tour guide robots were equipped with more capabilities than their predecessors, the actual interactions with visitors in the museum did not seem 27

44 CHAPTER 2 Background and Related Work to improve significantly compared to the first generation tour guide robots. It is interesting that while robots had increasingly humanlike features and more sophisticated capabilities, the second generation tour guide robots still evoked boundary-searching or bullying behaviors from users. This could indicate that while tour guide robots have been endowed with more interaction capabilities, insufficient effort has been spent designing appropriate and effective communication behaviors Previous research into specific tour guide robot behaviors In the previous section we have learned that for the first and second generation tour guide robots, the interactive behaviors of the robots have not always been the focus of research when developing the platforms. In order to better understand the effects of specific social robot behaviors on the user experience, researchers have been studying the effectiveness of robot behaviors in human robot interaction as well as the people s experience with the robot. In the remainder of this section we will therefore, particularly address studies on physical behaviors such as robot gestures, gaze and head movement, and body orientation and positioning. These behaviors are the nonverbal behaviors that are found to be important for guiding people. Moreover, most of these behaviors can be displayed by many robots and are not restricted to humanoid robots that have more sophisticated expressive abilities such as facial expressiveness. The influence of robot gestures Pointing is an effective behavior to direct the attention of visitors. Therefore, this behavior was developed for the robot Alpha (Bennewitz, Faber, Joho, Schreiber, & Behnke, 2005). When Alpha informed visitors at a point of interest, it simultaneously moved its head and eyes in the direction of the object of interest, and pointed in its direction with a corresponding arm gesture. When pointing, the robot s gaze followed the pointed arm, and the top of its fingers were in line of sight for the robot, as if Alpha looked at the exhibit. This was thought to be comparable to human pointing (Bennewitz et al., 2005). People found the robot s eyegaze, gestures, and facial expressions humanlike. Furthermore, people were attracted by its vivid humanlike eye movement (Bennewitz et al., 2005). Fritz (See Figure 2.4), the successor of Alpha used not only pointing gestures, but also some symbolic gestures such as greeting and come closer gestures. It was argued that visitors understood how to interact with Fritz, because the robot used humanlike behavior (Bennewitz, Faber, Joho, & Behnke, 2007). Interestingly, while these authors observed that people at times did not move closer when Fritz made come closer gestures, they argued that this happened to human tour guides as well. They noticed that, after a while, when visitors got used to the robot, they did move closer and treated the robot as a communication partner (Bennewitz et al., 2007). The authors argued that even though the robot was limited 28

45 in its capabilities, its imitated human gestures did positively impact the interaction. In contrast, Pitsch and Wrede (2014) found that robots pointing gestures could also lead to confusion. In their study, a Nao robot pointed to an object with its head orientation and arm for a moment when starting to inform visitors. They found that when visitors were not watching the robot, they did not observe the gestures. If visitors looked back at the robot after it had returned to a more neutral position, visitors were confused and did not know which object it was referring to (Pitsch & Wrede, 2014). Therefore, they concluded that guide robots need to solve this issue by using head direction or arm pointing gestures for recurring referencing (Pitsch & Wrede, 2014). The influence of robot head movement and gaze Kuno, Sekiguchi and Tsubota (2006) found that a tour guide robot could shift the focus of attention of people by turning its head towards the object of attention. This shift occurred when the robot turned its head in a way similar to human tour guides when they explain about an exhibit. Human tour guides turn their head towards the object of interest multiple times when emphasizing information, and when providing unfamiliar information. When this behavior was copied to the robot, participants reacted to the robot similarly as to human tour guides (Kuno, Sekiguchi, & Tsubota, 2006). However, one could argue that people responded to action in general (i.e. the robots movement) rather than to robot head movement in particular. Therefore, in a follow-up study (n=16), head movement of Robovie-R v2 (See Figure 2.4) at interactional significant points was compared to random head turning. Results indicate that participants nodded more and engaged in mutual gaze more when the robot moved its head purposefully (Kuno et al., 2007b). To explain this behavior, the authors base themselves on (Sacks, Schegloff, & Jefferson, 1974) and argue that nodding and mutual gaze of the participant indicate engagement and understanding (Kuno et al., 2007b). 2 Similar results were found again in a later study with a larger sample (n=46) (Kuno et al., 2007a; Sadazuka, Kuno, Kawashima, & Yamazaki, 2007). Moreover, similar results were found by (A. Yamazaki et al., 2008). And also for a similar study conducted in a real life museum (K. Yamazaki et al., 2009). Additionally, in a follow-up study was found that a robot nodding and gazing at specific moments in the story made the visitor gaze and nod as well (A. Yamazaki, K. Yamazaki, Burdelski, Kuno, & Fukushima, 2010). These findings suggest that humanlike behaviors such as head movement and gaze may lead to increased engagement and visitor feedback behaviors. Additionally, these visitor responses to robot gaze behaviors were used by Kobayashi et al. (2010a, 2010b) and Sano et al. (2015) to identify engaged visitors (those who are nodding and engaged in mutual gaze) to answer a question. Moreover, Gehle, Pitsch and Wrede 29

46 CHAPTER 2 Background and Related Work (2014) studied how visitors used gaze to signal that they had difficulties in understanding the robot. They found that the gaze pattern of a group of visitors followed a stepwise pattern of interrupting gaze with the robot, establish mutual gaze among each other, focus on a different point of interest and finally return the gaze to the robot (Gehle, Pitsch, Dankert, & Wrede, 2015). Surprisingly, after such a failure when visitors were successfully engaged again by the robot, this caused a similar gaze pattern between visitors (Gehle et al., 2015). These results indicate that not only is humanlike gaze performed by a robot important to successfully engage and direct visitor s attention, the visitor s gaze behavior is an important indicator for the quality of the human robot interaction as well. While the above shows the strengths of using humanlike robot gaze behaviors for robots, it was found that robot gaze behavior was not always interpreted similarly to human gaze behavior. In their study, participants found that a Nao (See Figure 2.4) that is searching by moving its head, is less available for communication compared to a Nao that focused on the entrance of the room (Pitsch, Wrede, Seele, & Süssenbach, 2011). Moreover, Pitsch, Gehle and Wrede (2013) found that autonomous dynamic shifts in head orientations were interpreted by visitors as meaningful gaze strategies. Furthermore, they observed the gaze strategies of four human tour guides and concluded that there were differences in how visitors reacted to human tour guides compared with robot tour guides, even when humanhuman interaction strategies were applied to a robot. Therefore, they question the use of human-human interaction to model human-robot interaction (Pitsch et al., 2013). The influence of robot orientation and positioning Lab studies on how close the humanoid robot Robovie (v1) should stand to a participant when explaining about an object revealed that for listening to the robot and for speaking to the robot participants preferred the robot s positioning leaving an O-space (empty space surrounded by the people and objects involved in a social interaction, where every participant looks inward into it) (Yamaoka, Kanda, Ishiguro, & Hagita, 2008). This way, the proximity to listener, the proximity to the object, the listener s field of view, and the presenter s field of view were taken into account (Yamaoka et al., 2008; Yamaoka, Kanda, Ishiguro, & Hagita, 2010). In studies with the Gestureman robot (See Figure 2.4) was found that this robot was able to reconfigure the spatial arrangement of one participant by changing its own body orientation (Kuzuoka, Suzuki, Yamashita, & Yamazaki, 2010). It was found that the robot was able to reshape the formation by changing its body orientation, for which the upper body reconfiguration and the full body reconfiguration had a stronger influence on the spatial arrangement of the participant than movement of the head only (Kuzuoka et al., 2010). Moreover, Ghosh and Kuzuoka (2013, 2014) found that the body movement at the end of explanations influenced responses of the visitors. Within this study, the robot guided in 30

47 four different conditions: 1) used summative assessment and lean back movement, 2) used summative assessment only, 3) used lean back movement only, 4) did not use any specific behavior. Furthermore, they found that a robot that used both the summative assessment and the lean back movement was more effective in disengaging visitors from one point of interest to prepare them for the next, compared to the other conditions (Ghosh & Kuzuoka, 2014; Ghosh & Kuzuoka, 2013). Inspired by work of Yamaoka et al. (2008) on how close to position to visitors and the work of Kuzuoka et al. (2010) on the effect of robot body orientation on the orientation of a visitor, Yousuf, Kobayashi, Kuno, K. Yamazaki and A. Yamazaki (2012b) studied how a robot could configure the spatial formation of multiple visitors. They found that participants rated the behavior of a robot that showed stepwise behavior to create an appropriate formation as effective for creating an F-formation. An F-formation arises when two or more people and/ or robots form a spatial and orientational relationship in which the space between them is one to which they have equal, direct, and exclusive access (Kendon, 1990). This behavior also was effective for catching the participants attention (Yousuf, Kobayashi, Kuno, A. Yamazaki, & K. Yamazaki, 2012a; Yousuf et al., 2012b). In a follow-up study, Yousuf, Kobayashi, Kuno, A. Yamazaki and K. Yamazaki (2013) investigated the effects of body orientation on initiating interaction with the visitors. They found that body orientation influenced the visitors perception of the appropriateness of the robot s greeting and attending to the visitors (Yousuf et al., 2013). 2 Figure 2.4: Impression of robots used for studies to effective behavior for tour guide robots. From left to right: Fritz (taken from: (Bennewitz et al., 2007), Robovie R. V2 (taken from: (K. Yamazaki et al., 2009), Nao, and Gestureman (taken from: (Kuzuoka et al., 2010). Most studies performed with humanlike tour guide robots evaluated the use of specific behavior and showed that people recognized and responded naturally to the humanlike behavior that the robot performed. This indicates that singled out behaviors can be transferred to robots. However, as Pitsch et al. (2013) already discussed, when behavior that the robot had to show became more complex, the interaction between the robot and 31

48 CHAPTER 2 Background and Related Work the visitors became less smooth. This was also found for the humanlike second generation tour guide robots, for which the interaction was less smooth during the whole tour. I believe this might have been due to the reason that these robots had to show complex behavior to fulfill several tasks (keep visitors engaged, explain about exhibits, explain intentions and motivations, and bring visitors to the next exhibit). In these cases, the visitors had to interpret the not perfectly transferred humanlike behavior and might have been confused at times. Thus, even though singled out humanlike behaviors evaluated with humanlike robots evoked expected responses from people, more complex behavior did not. This indicates a tension between the ability to perform complex human behaviors during interaction and the different ways in which people respond to behavior in robots compared to behavior in humans. In the following section I will further elaborate on this tension Framework to design behavior for lowanthropomorphic robots From this overview of studies with tour guide robots, we can conclude that in most studies, robot behavior was created by copying human behaviors as closely as possible to the robot s modalities. However, some researchers noticed that not all humanlike behavior for the increasingly anthropomorphic robots evoked responses that were similar to responses to human tour guides. Therefore, this shows a tension between on the one hand, the ability to leverage familiar human behaviors to smooth the interaction and on the other hand, the different ways in which people respond to behavior in robots compared to behavior in humans. This tension raises an important question, specifically when designing nonverbal basic robot behaviors such as gesturing and orientation: Should we adopt the approach of copying human behaviors or should we design purpose built behaviors for the robot? In this section, I will further elaborate on this tension and propose a framework to structure the design of low-anthropomorphic robot behaviors. When a humanlike robot is not able to perform behaviors in a way people would expect from the robot, this can lead to misunderstanding. For instance, slow head turning might be perceived as change of focus, while the robot intended to shake its head. Moreover, jerky shaking head movements might become creepy. As another example, low-anthropomorphic robots that are not able to turn their head, but instead, turn their full body, might be misunderstood. Although the resulting behavior may seem similar (a copy of human behavior adapted to the robot s abilities) people might interpret the behavior very differently. This is the tension that was described in the last section. Moreover, as we have seen in the previous section, when behavior becomes complex, it seems more difficult to transfer correctly to the robot. Thus, copying humanlike behavior to a robot might not be the most effective way to develop behavior for that robot, specifically when the robot lacks humanlike modalities. 32

49 Alternatively, I argue that robot behavior can be optimized for the robot s morphology and modalities. This concept is not new to the field of human-robot interaction. For example, Embgen et al. (2012) already proposed to use abstracted robot-specific behavior (consisting of body movement and display of colored lights), to show a robot s emotion. Moreover, Ju & Leifer (2008) proposed a framework that explains implicit interactions for non-living objects, which might include robots. Also, it was showed that social behavior, which was not a copy of humanlike behavior, for minimalistic robots was preferred over alarm type behavior (Šabanović, Reeder, & Kechavarzi, 2014). However, guidelines or approaches on how such behaviors should be designed for robots, to my knowledge has not been proposed before. Therefore, I present a framework that explains the implications of the different approaches to develop nonverbal behavior for a low-anthropomorphic robot. Figure 2.5 presents two approaches to develop nonverbal behavior for low-anthropomorphic robots and shows how both approaches effect user experience. Currently, the design approach that most researchers use to design behavior for any kind of robot is to apply human behavior to the robot, this approach follows the arrows on the right in Figure 2.5. Behavior that the robot cannot perform due to limitations in modalities or lack of specific modalities is usually left out (e.g. pointing behavior is left out when a robot does not have arms). As a result, behavior for robots can become confusing and inconsistent. As an alternative, the effects of human behavior can be analyzed and robot specific behavior can be developed that has a similar interactional outcome, while the behavior is optimized for the modalities that the robot has at its disposal. This approach follows the arrows on the left in Figure In this approach, an important step is included that concerns the analysis of the (intended) effect of human behavior. Communicative behavior of the robot is based on this analysis (that leads to interactional outcomes) and is subsequently optimized for the modalities of the robot. This approach is expected to minimize the tension between using humanlike behaviors for robots that cannot perform these behaviors and optimizes the use of the robot s capabilities. The resulting behavior is expected to be understandable because it leads to similar interactional outcomes, while the behavior is not confusing to people because it is not a poor or incomplete copy of humanlike behavior. To conclude, on the one hand the theories presented in this chapter show that anthropomorphic appearance and humanlike behavior for robots are one solution to create natural interaction and familiarity in human-robot interaction. On the other hand, consistency between appearance and behavior is necessary to create positive interaction experiences. Current research on human-robot interaction has a strong focus on determining what effective interactions are. While this is certainly important, because it influences the behavioral level of experience (Norman, 2004), research in this area is often concerned with 33

50 CHAPTER 2 Background and Related Work isolated characteristics of the robot and does not cover the overall experience of using the robot. Experiences at the visceral and reflective level are just as important because they also partly determine how people feel about their interaction with the robot. When it comes to tour guide robots, initial reactions of people (visceral level) may even determine whether or not someone decides to initiate or engage in interaction with the robot. As Norman states, attractive things work better (Norman, 2004). Figure 2.5: Framework of the tension between appearance of the robot and the possible behaviors that it can perform This indicates that not only what is designed, but also how people perceive it plays a role in human-robot interaction. Therefore, I argue that in the design of social robots, researchers should not limit their options to creating humanlike robots only. A way to define whether or not a humanlike robot would be the best solution would be to evaluate whether a humanlike robot can perform the functional tasks and evoke the desired interaction outcomes and user experiences. If the answer to one of these questions is negative, it might be more appropriate to consider a low-anthropomorphic robot. The framework that I propose, might help researchers to find alternatives for anthropomorphic appearance and behaviors for robots. 34

51 2.5. Conclusion The overview of human robot interaction research with first and second generation tour guide robots showed that over time, robots were increasingly equipped with humanlike appearance and modalities. Also, research into the effects of specific robot behaviors such as gaze, gestures and orientation showed that humanlike behavior copied to robots evoked positive responses in visitors. However, previous research also shows that people responded differently to behaviors displayed by robots compared with humans. This is likely caused by robot form and capabilities that limit the possibilities to effectively copy human behavior. One solution to this is to develop extremely lifelike humanoid robots that have the same or very similar action capabilities to people. However, it is unlikely that tour guide robots will become fully humanoid in the near future. Many technical breakthroughs are needed to ensure robust operation in real world environments, for instance, in bipedal walking, facial expressiveness, natural language dialogue and multimodal behavior alignment many developments are still needed. A limitation then is that such robots will become complex, expensive and vulnerable, which in turn will limit the opportunities to use these robots unsupervised in public and semi-public spaces. 2 Nevertheless, robotic services will continue to be developed and expanded to serve in public and cultural places. The overview of important theories shows that the tendency of people to respond socially to robots is decoupled from their anthropomorphic appearance. Moreover, when robots resemble humans too closely the increase in familiarity heightens the expectation leading to discomfort and uncanniness. Contrarily, people tend to react socially to minimalistic robots, when these robots use words for output, show interactivity and take over roles of people. The overview of previous research into human robot interaction for tour guide robots focused on robot behaviors that were observed in human tour guides as well. By focusing on a humanlike appearance and modalities in the development of social robots, such as tour guide robots, the researchers had the opportunity to copy humanlike behavior on the robots to see how this would influence the course of the interaction. And vice versa; the choice to imitate human behavior for robots, led to the use of humanlike robots. However, as previously stated, androids and highly humanlike robots often are too expensive and vulnerable to use for unsupervised interaction. To make social robots available for public spaces, they should be robust, fool-proof, and affordable. A low-anthropomorphic, functionally designed robot might help to achieve this. This tension is related to research question 2 of this thesis as formulated in section 1.4. Literature shows that humanlike behavior for humanlike robots can be functional. However, 35

52 CHAPTER 2 Background and Related Work this approach has limitations in interaction, specifically when the robot misses humanlike modalities, or the modalities are limited in their functionality. This indicates that further research is needed on how to develop effective behavior for a low-anthropomorphic robots. The contrast between the successful application of humanlike behavior and the need to adopt robot-specific alternatives to optimize human-robot interaction is the core goal of this thesis. Concretely, this thesis raises the question of what approach should be preferred: Copying human behaviors to low anthropomorphic robots or to design robot behaviors optimally to the robots morphology and interaction capabilities? In the following chapters of this thesis I will present studies into the effects of nonverbal behavior on human-robot interaction for low-anthropomorphic robots. However, as these robots often lack human modalities, research is needed on how to design nonverbal behavior for low-anthropomorphic robots. Therefore, I will first discuss what the effects on people s responses and experiences are when humanlike behavior is applied to lowanthropomorphic robots. Next, I will explore an alternative approach to design behavior taking into account the opportunities and limitations of the robot platform. 36

53 3 Contextual analysis; A robot in a tourist site? This chapter will be dedicated to gaining an understanding of how and why human tour guides use specific guide behaviors and strategies during a tour and how visitors experience the tourist sites of the FROG project. In this chapter, two studies will be reported. The first study offers a description of tour guide behavior on location as well as how the behavior of the tour guides influences the pace of a guided tour. The second study sets out to identify the factors that influence visitor experiences at tourist sites and discusses how these findings might impact the development of a robot for a tourist site. In the discussion of this chapter, the focus will be on how observed phenomena of visitor experience and effective tour guide behavior might influence the development of behaviors for a low-anthropomorphic tour guide robot. 3 This chapter is largely based on: Karreman, D. E., van Dijk, E. M. A. G., & Evers, V. (2012, September). Using the visitor experiences for mapping the possibilities of implementing a robotic guide in outdoor sites. In Proceedings of the 21st IEEE International Symposium on Robot and Human Interactive Communication (pp ). IEEE. & Karreman, D. E., van Dijk, E. M., & Evers, V. (2012, October). Contextual analysis of human non-verbal guide behaviors to inform the development of FROG, the fun robotic outdoor guide. In Proceedings of International Workshop on Human Behavior Understanding (pp ). Springer Berlin Heidelberg. 37

54 CHAPTER 3 Contextual Analysis 3.1. Motivation Before creating a tour guide robot for a tourist site, it is important to understand the context the robot will function in, and how visitors experience the current situation. In Chapter 2, we described effective behaviors for previously developed tour guide robots and the experience of visitors of joining robot guided tours. However, not only previous created tour guide robots and the tours they gave are a source of information, also information about the behaviors of human tour guides and the strategies they use to engage visitors might inform behavior for a tour guide robot. Not only the use of strategies to engage visitors or effective interaction behaviors are needed for a successful tour guide robot, the robot should also be able to create a positive visitors experience over time. Only when it is clear what factors influence the experience of visitors in indoor/outdoor sites, can functionality and behavior be developed for a robot to optimize the experience of the visitors. Therefore, in this chapter the focus is on effective human tour guide behaviors to communicate information about the site, and current visitor experiences at the sites and how these can be improved Related work Strategies in tour guiding In general, a guide role has two key dimensions: pathfinder; guiding visitors around a site in a logical route, and mentor; providing information about the site to inform the visitors (Cohen, 1985). The literature suggests that the most important strategies to gain and keep visitors interest for tour guides are interacting with the visitors (Uyen Tran & King, 2007) and adjusting the tour to the personal interests of the visitors (Best, 2012). Before these strategies became common, tours were like lectures, with more or less a monologue from the tour guide (Uyen Tran & King, 2007). Nowadays the tours have become more adjusted to the interests of the visitors which has the advantage that visitors are more involved in the tour and supposedly will like the tour better (Best, 2012). To adapt the tour to the interest of the visitors, the guide needs to be able to tell flexibly about everything they encounter, so visitors do not notice the adaptations in the tour when the guide makes changes in content, for example, in cases where some places of the tour not being available (Wynn, 2010). However, these strategies are too general to develop robot behaviors for specific situations, thus more specific information on tour guide behavior is required to inform the robot guide behavior. 38

55 Factors influencing the visitor experience In this section definition of visitor experience, or the user experience of visitors, and its importance when developing an interactive robot tour guide are presented. According to the ISO definition user experience (UX) is a person s perceptions and responses that result from the use or anticipated use of a product, system or service (ISO ). This definition is in line with the view of researchers and practitioners in the field of user experience, as was found in a study on how to define UX (Law, Roto, Hassenzahl, Vermeeren, & Kort, 2009). However, the definition is still very broad and subjective, and allows anyone to fill in their own ideas about UX (Hassenzahl, 2008). Moreover, Desmet and Hekkert (2007) used the term product experience when they referred to all possible affective experiences involved in human-product interaction. In their user experience white paper, Roto, Law, Vermeeren, and Hoonhout (2011) described the user experience rather than proposing a definition. Their description makes the concept of user experience clear. Moreover, they distinguish three factors that influence it. They state that user experience is personal, contextual and dynamic (Roto et al., 2011). Furthermore, Hassenzahl (2008) states that a series of experiences can lead to an attitude towards something, such as in our case the robot in the tourist site. 3 To understand the complexity of the concept of user experience, the effect of being personal, contextual and dynamic within the setting of a tourist site will be described. First, a person who likes ancient ruins and cultures will have a different personal opinion about a remainssite than a person that does not like old buildings and cultures, but instead likes modern art. Second, when the weather is very nice, visitors might experience the outdoor site more positively than when it rains. Third, visitors experiences are dynamic and can change over time, for example, when it is quiet in the morning, visitors experiences will probably be more positive than in the afternoon when it is busy and the exhibits are occupied by others and therefore hardly visible. Even though the experience of visitors might differ for each individual person that visits a site, a general user experience can be defined by investigating what a group of visitors like and do not like while visiting a site (Roto et al., 2011). Therefore, it is valuable to study the perceptions, interpretations and emotions people have of a visit of a (small) group of people, to gain a feel for general user experiences of user groups of the site. Furthermore, when people visit a tourist site, they are most likely not alone. They will likely be with a small group of people, and therefore they will have a shared experience. Together these people can reflect on this shared experience later in time, without the need to explain everything they experienced among each other. Battarbee and Koskinen call this co-experience, in which co-experience is the seamless blend of user experience of products and social interaction (Battarbee & Koskinen, 2005). Shared experience is important for 39

56 CHAPTER 3 Contextual Analysis FROG tours, because exactly this is what following a tour guide robot distinguishes from visiting a site while all individually listening to an audio guide. Thus, understanding the shared experiences people have, can help to create a positively rated FROG tour. The definition of user experience that combines all these aspects and that I will adopt in this thesis is the one of Hassenzahl and Tractinsky (2006): UX is a consequence of a user s internal state (predispositions, expectations, needs, motivation, mood, etc.), the characteristics of the designed system (e.g. complexity, purpose, usability, functionality, etc.) and the context (or the environment) within which the interaction occurs (e.g. organisational/social setting, meaningfulness of the activity, voluntariness of use, etc.). Thus, for the development of a tour guide robot for a cultural heritage site, it is important to understand current visitors experiences in representative sites. This is because an environment or a tour guide can influence individual as well as shared experiences. Therefore, a study of the current visitors experiences in the two selected sites will be reported Study 1: Contextual analysis of human tour guide behaviors The goal of this study was to fully understand the behavior patterns of human tour guides and the strategies they use, to inform the behavior of a future tour guide robot. For the analysis the Lisbon City Zoo in Lisbon, Portugal and the Royal Alcázar in Seville, Spain were selected. Both sites offer interesting and challenging opportunities for robot guides Study design Participants For the study, a total of four tours with four different guides were observed and videorecorded. In the Lisbon City Zoo two guides participated. The first guide Tiago, a male who had ten years experience - guided a group of seven adult visitors. The second guide Sylvia, a female who had some years experience - guided a school class of 19 children aged 9-10 years old. The guides in the zoo were employed by the Lisbon City Zoo. In the Royal Alcázar two guides participated, both were female. The first guide Paloma, who had ten years experience - guided a group of eight adult people, and the last guide Mariana, who had several years experience- guided a group of twelve adults (and two small kids). The guides in the Royal Alcázar who contributed to this study were certified tour guides. Procedure Two researchers joined the four tours. One of the researchers recorded the whole tour on video. In the video, the guide and his/her expressions and some visitors or the whole group 40

57 of visitors were visible. The story the guide told is not always clearly audible on the tape, but nonverbal behavior of the guide and the visitors responses were very well visible. The second researcher followed the guide at a close distance (as member of the group being guided) and made notes on the content given by the guide, on notable guide behavior and on the events that happened during the tour. After the tours, the two researchers interviewed the guides together. At the start of each tour all the visitors and the guide were informed about the study and that they were to be video recorded for analysis purposes. After finishing the tour the group was led to a room and all adults, including the guides, completed consent forms. After signing the consent forms, the researchers interviewed the guides and both took notes of the answers. After the guide left, the researchers completed the notes of the observation of the tour and the interview. Data collection All tours were videotaped. Moreover, notes were taken during and after the tour, on things that stood out about guide behavior. One of the researchers followed the guide closely and took notes during the tour, the second researcher filmed during the tour and then took the notes at the end of the tour. 3 After the tour, the two researchers interviewed the guides (approximately 15 minutes) about the tour they had just given, their experiences guiding different kinds of groups, use of strategies and how they would like to improve the visitor experience at the sites. The following questions guided the interview. What is the purpose of your tour? What is the main exhibit in the site you are guiding? Do you notice differences between groups? How do you deal with that? Is guiding children different from guiding adults? If yes, in what way? How do you gain and keep attention of the visitors? What do you want to change to improve the visitor experience? Data analysis The video data was annotated. This provided material to analyze the behavior and strategies of the guides. As well as the general analysis of the full recordings, two film fragments (approximately 2-4 minutes) were taken from all guides to annotate in detail. Aspects that were analyzed: the orientation of the guides, the story the guides told, the movement and gestures the guides made, the gaze behavior of the guides and the guides behavior when they finished the story at exhibits (See Appendix A for the observation scheme for the annotation of guide behaviors). 41

58 CHAPTER 3 Contextual Analysis The annotated video data and the notes from the different researchers were combined in the analysis using an affinity diagram. This method is based on the grounded theory method; themes and results of the research emerged from the data (Corbin & Strauss, 1990). Using an affinity diagram is very useful when large amounts of qualitative data have to be analyzed, from which the results are complex and not easy to grab (Courage & Baxter, 2005). The affinity diagram helps to order the information and to find logical and natural relationships between the parts of the data. To study human tour guide behavior, textual fragments taken from all data sources (observation, interview, video analysis and literature) were written on small cards. The fragments were color coded by resource (e.g., all fragments taken from the first observation were written in blue and fragments taken from the interviews were written in purple). When all cards were completed, the researcher started to cluster the cards containing the fragments on a large wall. During this clustering the placement of the cards was not fixed and cards were removed and replaced if necessary. Finally, all cards were stuck on the wall, names for the clusters were determined, and some clusters with sub-clusters were determined. An affinity diagram can get very large, which makes it unsuitable to present the results. Therefore, we created a connection diagram to communicate the results, based on the affinity diagram. The connection diagram is a graphical presentation of the data, which we used to show the relations between findings of the study. The clusters and sub-clusters and the relations between them became visible through this presentation. In the next section, the results will be presented based on this connection diagram. Note that I made the cards containing the fragments and the affinity diagram. However, we discussed the notes on the observations and the video analysis together before the cards were written. After making the affinity diagram and the connection diagram, the researchers discussed the results and added relations, connections and changed names for the clusters to get the best overview of the results Results The connection diagram, presented in Figure 3.7, shows the names of the identified clusters, sub-clusters and their connections. The clusters were connected with lines when there was a relation between them. For example, the connection between distraction and taking pictures is that when visitors get distracted during the tour, they often start to take pictures. Or, vice versa, visitors become distracted because they are taking pictures. Moreover, the colors present the strategies and implicit behaviors guides used and the visitor behaviors. 42

59 Clusters Ten clusters with sub-clusters appeared, which are briefly described (detailed description and explanation of nonverbal communication cues will follow in the next paragraphs): Attention of the visitors is needed for a tour guide to give a tour. Guides used lots of strategies to get and keep the visitors attention, such as interacting with the visitors, adjusting the content of the tour to the visitors interests, breaking eye contact and starting to move in the next direction at the end of an exhibit. Interaction during a tour is common nowadays, in contrast to tours in the past. All guides knew that a tour of two hours was too long for visitors to just listen to a guide. Therefore, the guides tried to interact with the visitors by addressing them in different ways, such as asking and answering questions, showing visuals, letting visitors experience (touch, smell) the site, pointing at objects, searching for differences together, asking visitors what interested them and referring to that later, and having a chat with the visitors during the walk. Information about the site is conveyed by the guide. The guide can usually tell flexibly about everything they encounter at the site, and answer all the questions of the visitors. Visitors particularly like to hear curiosities, information specific to the site that cannot be obtained somewhere else. Sometimes a guide brings visuals, such as pictures or objects to support the story. 3 Adaptation of the tour is always done to fit the tour to the different types of visitors. But guides also adapt the tour on the flow to keep visitors interested. The guides usually try to shape the tour around the visitors interests, because when telling more about a subject that visitors are already interested in, it was easy for the guide to keep their attention. Guides can adapt the content, speed or route of a tour. At the exhibits guides started and finished with less important sentences, hence, when the latest visitors arrived, they would not have missed important information. To keep the tour going, the guides did not wait until the group was complete before starting the story. They started to talk when only the nearby visitors had arrived. Gaze-behavior is quite intuitive. Guides used gaze, among other cues, to keep the attention of the visitors by alternating their gaze between them. Moreover, they were able to see from the visitors gaze if the visitors were still interested. Based on this information, the guides adjusted the story to the objects of interest. Gestures made by the guide helped the guide to tell the story or helped the visitor understand the story. All guides made a lot of arm gestures, varying from supporting or depicting the story to pointing. 43

60 CHAPTER 3 Contextual Analysis Orientation is the way guides orientated their body towards visitors and to the exhibit. This was always in a way visitors were able to see both the exhibit and the guide. Taking pictures was something lots of visitors did when they were in an attractive place. Taking pictures either indicated that visitors were really interested, taking pictures of the subject of interest. Or it indicated that the visitors were distracted, when they started to walk around and to take pictures of everything. Visitors that took pictures during the tour often got lost, because the guide did not wait for them. Therefore, these visitors missed much of the content. The procedure of the tour was determined by the tour guide, following a tight schedule. The guides were not strict to adults, because they could choose themselves whether they wanted to listen. Strategies and effective nonverbal behavior From the observations and interviews we found that guides were fully aware of effective behavior to influence the attention of the visitors. They called these behaviors strategies. Nevertheless, they also used behaviors that they did not explicitly mention, which we call implicit behavior. The strategies used by the tour guides were similar for all four guides. However, there were slight variations in the implicit behaviors, which might have been caused by personality aspects. As the scope of this thesis is on nonverbal behavior, in this chapter we only extensively describe the use of nonverbal strategies by tour guides. All strategies used by the tour guides used are summarized in Table 3.1. Gaze-behavior The guides reported in the interviews that they alternated their eye contact evenly between the visitors, as a strategy to keep their attention. The guides did indeed alternate their gaze, but in our analysis of the video material, the guides seemed to choose one visitor at each exhibit to talk to. Most of the time this visitor got the attention and the guide sometimes shifted attention to other visitors, but always got back to the one they looked at initially. However, for different exhibits, the guides seemed to choose different visitors to address. The visitor the guide was looking at often nodded, turning their head towards the exhibit and back and looked at the guide. This provided the guide with information about engagement of the group. When pointing at an object of interest, the guides also looked at the exhibit for a while. For the guide, this was to check where to direct the group s attention, but sometimes visitors reacted to it by looking in the same direction. When the guides invited the visitors to look 44

61 at an exhibit or visual, the guides decided when to go on with the story. Most of the time, the guides waited for most visitors to indicate they had seen it (by nodding or gazing at the guide again). To keep the story going, the guides did not wait for all visitors to look back. On the other hand, the guides sometimes provided extra time to look at something for a particular visitor, by adding unimportant sentences to fill the time. Gestures and movements All guides used a lot of gestures, which can be categorized in pointing, depicting the story and supporting the story. While telling the story all guides used their arms to depict and point to the subject. For depicting the story and pointing at objects, the guides knew how, why and at which moment to perform the action in support of the story. Depicting the story and pointing to exhibits helped the visitors to understand the story the guide was telling. The guide only depicted parts of the story if the subject was not visible at the moment. Otherwise the guide would point at the exhibit to make clear what he/she was talking about. Furthermore, visual support could be in the site itself. In these cases, the guides were able to point to it, or touch it. However, the guides could also show visuals that they brought. Furthermore, the guides were able to depict the story if the subject they were talking about was not visible at the moment. 3 When the guides walked away from an exhibit, they moved slightly in the next direction during the last sentence of the story. This made clear for the visitors that the story was finished and which direction to go next. Sometimes the guides made a follow-me signal. We observed that visitors always followed the guide in a chain reaction. The nearby visitors starting to walk first and the furthest visitors following last. When looking at the pace of the arm and head gestures and the movements the guides made, there are no commonalities. The gestures and movements were made unconsciously and might help the guide telling the story, however, from our observations, we could not determine how they influenced the transfer of knowledge. These gestures and movements were personal for each guide and fitted with the overall personality of the guides. Orientation In most cases the guides oriented themselves facing the visitors in such a way that all visitors could stand in a semi-circle around the guide. Also, the guides stood close to the exhibit of interest, but not in front of it, with their backs towards the exhibit of interest. This allowed the guides to change their focus from the group of visitors to the exhibit by turning a bit on the spot. Due to this orientation of the guides, the visitors could easily change their focus from guide to exhibit and back without making large movements. 45

62 CHAPTER 3 Contextual Analysis In the case of Sylvia the orientation was different several times. At these moments she allowed the children to stand directly in front of the exhibit and stood behind the group of children. The children could not see Sylvia, but they had a free view into the exhibit. As Sylvia was taller than the children, she still could see what happened in the exhibit. During walks towards the next exhibit, the guides walked in front or at the front of the group, guiding the group to the next exhibit. However, in this way the guides also set the pace of the tour. During the walks the guides talked to the visitors at the front of the group, about the exhibits, or about personal interests of the visitors to gain insight in how to adapt the tour to the interests of the group. Cluster Attention Interaction Differences in tours Adaptation Subcluster Addressing visitors Addressing visitors Addressing visitors Addressing visitors Children s games Table 3.1: Strategies used by human tour guides Action guide Interactional outcome The guide receives all kind of feedback from the visitors about their engagement with the tour. Guides encourage visitors to ask questions. Guides encourage visitors to experience (feel/smell) the site. Guides encourage visitors to discuss about the site. Guides encourage visitors to search for commonalities and differences in specific places in the site. Guide has some games for children, e.g., jump like a kangaroo, or bend like a giraffe. (See Figure 3.1). Distraction When visitors get distracted, the guide is not paying attention to these visitors anymore, but focusing on the others. Flexibility The guides had an impressive knowledge of the site, so they could easily change to another subject and they were able to talk freely about everything they encountered. Adaptation before starting the tour Adaptation during tour If a guide knows before the tour what kind of group will join, the guide can prepare (e.g. an easier tour for children and a more scientific tour for adults). Change tour to interests of group of visitors. The guide chooses how to go on with the story and to focus on which visitor. Create interaction to keep the visitors attention. Create interaction to keep the visitors attention. Create interaction to keep the visitors attention. Create interaction to keep the visitors attention. Keep children s attention to the story. Give information to interested visitors. Keep visitors attention to the story. Keep visitors attention to the story. Keep visitors attention to the story. 46

63 Cluster (continued) Information Subcluster Story Action guide The guide knows exactly what to tell. Interactional outcome Create an interesting story, fits the interest of the visitors. Purpose The guide has a clear purpose with giving the tour. Keep the visitors attention. Curiosities Visuals The guides give facts that are special for the site or for the region. The guides show pictures or objects. (See Figure 3.2). Catch (again) and keep the visitors attention. Catch (again) and keep the visitors attention. At exhibits Start The guide starts with nonimportant words or sentences. Catch visitors attention again. Start Start Start The guide starts the story when only a few visitors arrived (See Figure 3.3). The guide starts with a bit louder voice when he/she want to say something important. The guide turns towards the visitors. Keep pace in the tour. Catch visitors attention again and indicate that story at new exhibit has started. Catch visitors attention again. 3 End The guide makes a closing remark about the exhibit. Indicate that the story at the exhibit had ended. Gaze behavior End End Alternating gaze Eye contact Eye contact Break eye contact The guide changes direction of focus to somewhere else in the room after a long talk at an exhibit. The guide is already walking into the direction of the new exhibit during the last sentence of the current exhibit. The guide is looking at all visitors and alternating gaze. The guide is looking at all engaged visitors and alternating gaze between them. The guides focus on one visitor that is close while telling a story, while just sometimes alternating gaze. The guides look away from the visitors (mainly to the ground). Prepare group to move attention. Indicate direction of next exhibit. Check engagement of the visitors. Address the visitors that are listening. Receive implicit feedback and base story on interests of one close person. Indicate story at exhibit had ended. Gaze direction The guides look (and point) into an exhibit. Direct the visitors attention to the exhibit (creating mutual gaze). 47

64 CHAPTER 3 Cluster (continued) Gestures Subcluster Pointing Action guide The guide is pointing into an exhibit. Interactional outcome Direct the visitors attention to the exhibit. Contextual Analysis Orientation Procedure Depicting Supporting gestures Towards visitors Towards exhibit Behind visitors Walk to next exhibit Walk to next exhibit Route Time schedule Authority The guides use their hands/arms or whole body to show something that is not visible in the exhibit. (See Figure 3.4). The guide makes a lot of gestures while they are talking. The guide is standing in front or next to the exhibit. The guide is standing so that he can easily point into the exhibit he wants to tell about. (See Figure 3.5). If visitors are children (small) the guide stands behind them to not block the sight. The guide is walking (slightly) in front of the group. The guide is talking to visitors during the walk. (See Figure 3.6). The guide is able to choose a route that corresponds to the story. The guide has to stick to a time schedule, as the visitors paid for or agreed to a tour of specific time span. The guide is the leader of the group, leads the way and decides on the time schedule. However, the guide does not make the visitors listen. Explain something that is not clearly visible in the exhibit. Help to tell the story, and makes guide focus of attention. Keep the visitors attention to the story and address the visitors. Direct the visitors attention to the exhibit. Direct the visitors attention to the exhibit. Indicate direction of next exhibit and guide visitors to next exhibit. Creating a relationship with the visitors. To keep the tour as close to the interests/ previous gained knowledge of the visitors as possible. Create a short tour that covers all important information. Keep the attention of the visitors. 48

65 Impression of guide behaviors during the tour Figure 3.1: Children jumping like kangaroos. Figure 3.2: Paloma showing a picture to clarify something Figure 3.3: Tiago starting at the exhibit while only one visitor arrived Figure 3.4: Tiago explaining the stripes of the cheetah on his own face 3 Figure 3.5: Paloma carefully chooses where to stand Figure 3.6: Tiago talking to the visitors at the front during the walk Discussion While only four guides were observed, their behavior was analyzed thoroughly to identify important strategies and behavior cues. Strategies for orientation, gaze behavior and gesturing that we found in our analysis were also reported by Best (2012). This indicates that these strategies are more general for tour guides in any situation. Moreover, many commonalities in the guide s behaviors and strategies were found for both sites. The results of this observation study should impact the design of nonverbal behavior for low-anthropomorphic robots. First of all, low-anthropomorphic robots might use some 49

66 CHAPTER 3 Contextual Analysis anthropomorphic modalities and therefore might use some humanlike behavior cues. This was, for example, the case for the first generation tour guide robots described in chapter 2. These low-anthropomorphic robots were provided with (dynamic) faces to make the interaction with visitors intuitive and natural. However, to be able to use humanlike modalities effectively, knowledge of strategies and behaviors that the human tour guides perform is necessary. For example, behavior and strategies such as telling curiosities, addressing the visitors, breaking eye contact and moving in the next direction during the last sentence are important as they might be effective in a future tour guide robot as well. Hence, understanding of human tour guide behaviors and strategies might positively influence development of future tour guide robot behavior. The observations in this study emphasize the importance of guides gaze behavior, as was already was found in other studies with human tour guides (Best, 2012) and with tour guide robots (e.g. Yamazaki et al., 2009). We specifically highlight that the gaze cues used in human-human interaction can be successfully applied to robots and elicit the same effect in understanding and recall of the story (Kuno et al., 2007; Mutlu, Forlizzi, & Hodgins, 2006). Therefore, when designing robot behaviors, examining human gaze behaviors and translating them to a robot s capabilities might help the robot to effectively convey information. Furthermore, the orientation and position of a tour guide might influence the design of the behavior for a tour guide robot, as the orientation of a tour guide influences the orientation of the visitors. Currently, a guide often stands at the side of an exhibit, and steps forward when he/she shows visuals. While, the robot tour guide will be smaller and visitors can look over it, but more important, the robot might make use of screens or projections to show visuals. As a result, a robot position that is similar or adjusted from a tour guide s orientation might influence the behavior of the robot to direct the visitors attention. Therefore, the proximity, orientation and position, and way of showing visuals should be extensively tested with the low-anthropomorphic robot to identify the most effective behavior. As the examples above show, the physical form of the robot will have a large influence on how these robot behaviors can be performed. Nevertheless, understanding effective tour guide behaviors and strategies is valuable, because it also indicates what moments are important for gaining, keeping or losing the visitors attention. Therefore, the behaviors and strategies presented in this chapter will form a basis for further development of behavior of the tour guide robot. 50

67 Figure 3.7: Connection diagram; All clusters are blue. The strategies tour guides uses are marked by a red frame, the implicit behavior that influences the interaction are marked by a purple frame, and the visitor behaviors are marked with a yellow frame.

68 Figure 3.8: Visitor Experience map. Explanation of color codes: Central words on a white background are names of the main clusters; Blue (italic) terms describe the visitor experiences from non-guided visitors; Purple (sans serif) terms describe the visitor experiences from visitors that followed a guided tour; Green (serif) terms describe the visitor experiences abstracted from observation; Green (light) area of a cluster is positive; Red (darker) area of a cluster is negative; Sizes of the words show the importance (bigger is important, i.e. was more often mentioned); Orange circles cluster similar experiences into a secondary cluster; Orange (solid) lines relate secondary clusters to each other; Yellow (dashed) lines relate experiences to each other.

69 3.4. Study 2: Understanding visitor experiences The goal of this study was to find out what phenomena influence the current visitors experiences in cultural heritage sites without a robot being present. The visitors experiences were investigated at the Lisbon City Zoo in Lisbon (Portugal) and at the Royal Alcázar in Seville (Spain) Study design Participants Six adults (a subset of the participants of study 1) participated in the visitor experience workshop. These participants were project partners of the FROG project. They were all men and between years of age. All had at least some experience with robotics. However, this workshop concerned only the evaluation of positive and negative aspects of visiting a tourist site while being guided and not guided. Data collection To design for positive user experiences, in Interaction Design, nowadays, it is common to actively involve the end users early in the design process. In this process, the users are not only observed, interviewed or asked to fill in a questionnaire, but they are actively involved in the first idea generation, for example, by following a participatory design process (Stappers, van Rijn, Kistemaker, Hennink, & Sleeswijk Visser, 2009), or context mapping process (Visser, Stappers, van der Lugt, & Sanders, 2005). Just asking end users to tell what they want to have improved or what kind of product or service will fulfill a described task does not give satisfying results, because the end users imagination is often limited to solutions that are on the market already (Visser et al., 2005). It is the designers task to use the users remarks, observations and analysis of the situation as input for the design process and go beyond these. 3 A workshop on experiencing the sites was organized with six participants, who visited both sites with and without tour guides. Within the workshop we followed context mapping principles (Visser et al., 2005). Context mapping follows the following steps: Preparation of the study; Sensitization: trigger, encourage and motivate participants to think, reflect and wonder about the context of study to prepare them for the group session; Session in which participants are invited to generative exercises; Analysis of the input provided by the participants; Communication of the results to the design process (Visser et al., 2005). In this workshop the participants were invited to share their experiences of visiting the sites (the Lisbon City Zoo and the Royal Alcázar in Seville) with and without guides. First, to sensitize them, they were provided a visit to both sites, both being guided and non-guided. 53

70 CHAPTER 3 Contextual Analysis That way they acquired enough experiences to tell about. Then, to map the test visitor experiences, a brainstorm and discussion was organized. The brainstorm and discussion were videotaped. During the brainstorm the participants were asked to write down their experiences, feelings, remarks and observations of the two sites on post-its. Two different colors of post-its were used: pink for their experiences being guided and yellow for their experiences exploring the site by themselves. Then all post-its were collected and during a group discussion the post-its were ordered and clustered on a large wall to create an affinity diagram (Courage & Baxter, 2005). During the discussion, the participants were referring to previous visits to other sites to explain their experiences. When the participants had more inspiration, they had the opportunity to write down more notes on post-its and these notes also became part of the ordering and clustering process. No constraints were given beforehand to fit the notes in. At the end of the brainstorm and discussion, a photograph of clustered and ordered notes was made. We also observed the participants during tours and during their free visit. During the tour and the free visits, family members or friends of the participants also joined. These observations were performed to gain understanding of their behaviors and experiences in the sites. At both sites the participants joined at least one. All tours were videotaped. Additionally, at both sites the researchers observed and videotaped the behavior of visitors that were not guided. These observations lasted approximately five hours. The observers did not talk to the visitors. Data analysis The videotapes and notes of observations of visitors and guides and the results of the workshop were analyzed. For this analysis the affinity diagram created during the workshop was the starting point and the other observations and interviews were used to complete and clarify findings of the affinity diagram. When necessary, clusters defined during the visitor workshop were (re-) named Results We found that the participants were always looking for information about what they encountered in the two tourist sites. This was one of the fun experiences of visiting a tourist site. Actually, it was best appreciated when the information could be obtained somewhat passively. Also, the participants liked the visit because they had a nice day together. Based on the affinity diagram, a diagram showing the positive and negative visitors experiences and the connections between these experiences was created. This resulted in the visitor experience map (see Figure 3.8, at the beginning of this study). The visitor experience map 54

71 shows all other factors and their relations with each other that influenced the participants experiences positively or negatively. The map gives the information and connections between the clusters schematically, to understand the participants actions, interactions and experiences. The main findings of visitor experiences are described below. Context of site The participants did know beforehand what kind of tourist site they were visiting and in what kind of context they would end up. But as we learned, the background information they had, was not always up to date and certainly not complete. So the participants were interested to find more information about the sites. Social dynamics When the participants visited a site they experienced the site in a particular way. They experienced the City Zoo as a (family) day out. This is because they went there and took a walk with family, friends and children and in the mean time they talked about everything, including the animals every now and then. From observation and interviews with the guides we learned that when a group containing children visited the City Zoo, the main goal of the visit was that the children would have a fun time. In the Royal Alcázar visitors searched for information more explicitly. Mainly couples of all ages, couples with children (mainly older than 10) and school classes visited the Royal Alcázar. In the site they were looking at exhibits, discussing them and taking time to relax in the gardens. This indicates that a visit to both sites was a social experience. 3 The number of cameras observed in the City Zoo as well as in the Royal Alcázar is remarkable. All groups of visitors carried at least one camera. Most of the visitors not following a guided tour had their cameras ready and took pictures of every exhibit they passed. Visitors also often posed or let friends or their children pose for the camera. Information Several participants had the feeling that too little information was given, walking around the site on their own. In the City Zoo and in the Alcázar only one small information board in just two languages per species or room was given. The participants indicated that they often needed to search for these information boards, therefore, their need for information was not satisfied. In the Royal Alcázar there was little written information available. Other sources of information in the Royal Alcázar were to choose for an audio guide or read an information book they bought to gain information about the site. 55

72 CHAPTER 3 Contextual Analysis In the City Zoo some more information about the tigers and the primates was given in the tigerhouse and the temple of primates, which were specially designed to show lots of information. Participants were positive about the richness and the visual presentation of the information in these information houses. But participants stated that at the same time they could become overloaded by the richness of information boards and the other information given in the compact spaces. Yet, being offered too much information was preferred, because participants could then choose for themselves whether they had consumed enough. Contrarily, when the information was not available at all, the participants were disappointed. Thus, too much and too little information were both experienced negatively, however getting too much information was preferred. In both sites visitors could join guided tours. The participants joined a guided tour in both sites. In both cases, the guides gave a lot of information on their tours, but the amount of information given did not always satisfy the participants. On the one hand, some found the duration of a guided tour too long. As a result, the participants got distracted in the end because of an overload of information. On the other hand, participants perceived a guided tour usually to pass the points of interest in a too fast pace. As a result, the visitors were not able to have a proper look at the points of interest, fell behind if they wanted to take pictures of the animals and often missed things because the guide already started to talk while the group was not yet complete. The sub-cluster fun experience of the social dynamics cluster is closely related to the information cluster. Listening actively to the guide telling funny stories and curiosities about the site was one of the fun experiences for participants, because these stories cannot be obtained anywhere else. The clusters social dynamics and information seem to contradict. The participants liked to go around and take pictures at their own pace, but then they experienced a lack of information both at the City Zoo and at the Royal Alcázar. When guided around during a tour, participants liked the information they received, especially the curiosities, but at the same time they did not like the speed of the tour, the tight time schedule, the tight story line and lack of time for taking pictures. Overall, the participants preferred a guided tour over wandering around. When participants happen to fell behind, they could always follow some parts of the story later in the tour because guides tend to repeat some information. Moreover, the participants mentioned that after the guided tour was finished they could go and visit the site at their own pace and socialize together. 56

73 Human tour guides As presented in section 3.3, human tour guides in the sites used several behaviors and strategies to influence the behaviors of the visitors. We observed and the participants mentioned that they had all adopted some specific behavior and used several common strategies to influence the attention of the visitors. These behaviors and strategies influenced how the participants of the study experienced their visit to the site. In general, participants experienced being guided as positive, because the tour guides were enthusiastic and could tell about everything they encountered. Furthermore, guides provided structure to the visit, balanced the amount of information at the different points of interest and adjusted the tour to the interests of the visitors. Hence, when joining a tour, participants had the impression of learning a lot about the site. However, some actions of the guide were experienced as negative by the participants. For example, repeatedly explaining concepts to remind visitors of relevant background information was sometimes experienced negatively by participants who had the idea they heard some things over and over again. When the guide addressed one person for too long, which could make the person feel embarrassed, which the participants also perceived negatively. Moreover, as there was no room for social dynamics during a tour, some participants stated they would prefer not to join a tour. 3 Track finding Participants liked guided tours or a predefined route to follow on their own. This offered structure to the visit and led them through a site in a logical order. Moreover, the tour or predefined routes provided them with background context and curiosities at the right time and at the right place. For the participants it was positive that they did not need a map of the site and did not have to figure out where they were Discussion Based on the study, three important phenomena that influenced the visitors experience were identified. These are that the site is usually visited by small groups of people who like to have a fun time together, visitors like to take a somewhat passive role, and the amount of information given to the visitors influences their experience. In this section I will explain how these three phenomena might affect the design of a robot tour guide. First of all, the tourist sites are social environments that (most) people visit together. Visitors experience the visit as a day out and like to have fun with friends or family. Even when they visit the site with a tour guide, we observed that visitors interacted with each other during the walks and sometimes during the explanations as well. Visitors did not prefer to use audio guides, because the experiences were gained individually. Also, visitors did not like 57

74 CHAPTER 3 Contextual Analysis the tight schedule of a guide, because it left no room for socializing together. Last, visitors did not want to use books, as these required an active role in obtaining information. Based on these visitor experiences it becomes clear that when employing a robot tour guide there needs to be room for social interactions among the group members during tours. Second, we found that visitors visited the tourist sites during a day out or during their holidays. Therefore they did not appreciate the rush of a guided tour. However, all visitors liked the content given by the tour guide. Even though these results seems contradictory, it shows that the visitors who participated in the study like to be a bit passive in their visit. Most visitors experienced a visit as positive when they were guided around, or did not have to search for the route. Furthermore, they liked to passively hear the information and not search for the information boards or find the corresponding information in books. Thus, for a robot tour guide it is recommended that information is presented in an active way. Third, visitors liked that the tour guides balanced the amount of information at each point of interest to the visitors interests and attention. In contrast, visitors who explored the site on their own did not like the little information they got at most points of interest and specifically the large amount of information they had to read in the tigerhouse and the temple of primates. Thus, the amount of information given by a robot tour guide should be balanced for different points of interest. Consequently, these findings have the following implications for a future tour guide robot for indoor/outdoor tourist sites: A tour guide robot should be aware of the visitors and the group dynamics and take these into account when giving a tour. A tour guide robot can take a somewhat leading role, without giving the visitors the impression they cannot influence the pace. A robot can balance the amount of information at several stops and base the length of information on the interests of the visitors. Hence, I will use these recommendations to evaluate the suitableness of the proposed tour guide robot through the remainder of this thesis General discussion Understanding the context In this chapter I focused on the tourist sites and behaviors and strategies of guides in these sites. Based on these observations, I found challenges a robot tour guide faces in these environments. These are not only because of technical aspects, for example how would computer vision deal with sunlight and shadow conditions. There are also challenges in 58

75 human-robot interaction, for example how will visitors react when they have to walk quite a distance. Noteworthy is that most tour guide robots were deployed in museums, which I explained in Chapter 2. However, for the FROG project we chose to deploy a tour guide robot in an indoor/outdoor tourist site. Differences between museums and indoor/outdoor sites that most likely influence the human-robot interaction are: the area in which a guide robot will guide visitors, and the placement of points of interest the robot will present. First of all because the area of the tourist site is much larger than a museum. Also, the way exhibits or animals are presented is different from how this is done in a museum. For example, an animal is not always visible in its exhibit and points of interest in a cultural heritage site are sometimes in places that are difficult to see, while in a museum the exhibits are mostly presented in a manner that visitors can see these optimally. This indicates that the choice for context to deploy a tour guide robot will have impact on how the robot should behave Phenomena influencing the design of behavior of a tour guide robot Similar to studies with guides in museum settings, gaze was found to be an important phenomenon in the interaction between guides and visitors. On the one hand we found that guides used gaze to communicate information to the visitors, such as where to look or that the story at an exhibit was finished. On the other hand, guides also gained information based on the gaze behavior of visitors. These findings are similar to the findings of Best in museum settings (Best, 2012), and were also observed by and effectively applied to humanlike robot tour guides (Kobayashi et al., 2010b; Yamazaki et al., 2009). Even though there are no previous results for low-anthropomorphic robots, the results presented in this chapter strengthen that gaze behavior is important for tour guides. 3 Another important phenomenon observed with human tour guides is that their movements and orientation in the site have great influence on how visitors form formations around a tour guide. In general, when a small group gathers around one person giving them information, they usually form a sort of (semi-) circle. In that way all group members can listen to the person who has the word (Marshall, Rogers, & Pantidi, 2011). The semi-circle formation is also recognizable when a human tour guide guides a (small) group of visitors and when people gather around a point of interest to all have the chance to see it. At this point, our observations are in line with the results presented in (Best, 2012). Yet, studies with low-anthropomorphic tour guides must show whether visitors react in similar manners when guided by such a robot. Previous work shows that visitors formations when gathering around static objects are similar to what we see happen with human tour guides. In these cases a lot of visitors get 59

76 CHAPTER 3 Contextual Analysis a chance to see the object at the same time, because people give each other space to see the exhibit. However, when gathering around interactive objects (often including a screen or interactive buttons), fewer people can see the object at the same time (Heath, vom Lehn, & Osborne, 2005), because people tend to stand closer to see the details shown on the screen or to directly interact with the (touch) screen. Even though these interactive exhibits introduce more interactivity to the exhibition, they decrease the social interactions and collaborations between visitors (Heath et al., 2005). This subtle difference in visitor formation with tour guides or interactive exhibits emphasized that attention should be given to how a robot is positioned and what its specific functionality is. This relates to research question 2, which asks how to create a consistent set of nonverbal behavior for a lowanthropomorphic robot. Based on the results of the studies presented in this chapter, the answer should include that focus on humanlike behavior only is too limited Focus on user experience in human robot interaction From the studies presented in this chapter it becomes clear that the way in which tour guides guide a group of visitors has great influence on how visitors experience a site. Therefore, it also might impact how a robot should guide visitors. However, this is a direction that was little studied in previous human-robot interaction studies. For example, for the first generation tour guides, the focus was on safe navigation and localization. For the second generation tour guides intuitive and natural interaction was important. And in the studies focused on specific guide behavior for robots, the effectiveness of the used behavior was of importance. Yet, the user experience and how a robot can be used to improve the visitor experience in tourist sites was not evaluated. Contrarily, in the field of interaction design, the user experience is deemed a factor that strongly influences the design of products. Design for user experience goes beyond design for usability. Nielsen and Molich (1990) stated that using a product should minimize the user s memory load, be consistent over the interface and provide feedback and so on. Even though these guidelines were written for the evaluation of web interfaces, they are also relevant for more advanced interfaces such as tour guide robots. However, using a product should also be pleasant and engaging (Hummels, 1999; Norman, 2004). Therefore, the design for experience seems an important step to make in human-robot interaction. Based on the results presented in this chapter, some directions to improve the visitors experiences were found. In general, the experience of visitors who have the impression of missing out on information needs to be enriched. To create a better experience for visitors, a tour guide robot should present short pieces of information about the main point of interest in the site, the robot should provide a structured route through the site, and the robot should recognize and react to the social behavior of the visitors. These recommendations will be taken into account while developing behavior for the tour guide robot. 60

77 The focus on user experience when designing behavior for a low-anthropomorphic tour guide robot has implications for the studies to be set out. First of all, we need to evaluate what is effective behavior for low-anthropomorphic robots, because knowledge on this topic is scarce. The design method mostly used to design behavior for robots in the field of human-robot interaction is to copy humanlike behavior to the robot. It is obvious to try this method first, before searching for alternative approaches. However, an open mind for other approaches is essential as well. Second, it is impossible to measure real visitor experience in the lab. This is because in addition to the guide behaviors and the information presented by the guides, being present in a tourist site influences the visitor s experience as well. This aspect of visiting tourist sites is very difficult to script in controlled lab settings. Furthermore, in the wild visitors will react differently to a robot than they might do in the lab (Shiomi, Kanda, Koizumi, Ishiguro, & Hagita, 2007). Thus, in-the-wild studies and evaluations of the robot behaviors are essential to understand actual visitor experiences of joining a tour guide robot. This need for the inthe-wild evaluation relates to research question 3 which asks in what way the effect of robot behavior on people s perception and experience should be studied. The studies in this chapter provide important insights into how visitor experience can be improved at tourist sites. Effective tour guide behaviors were identified in the contextual analysis. These provide the basis for studying the effects of translating human tour guide behaviors to low-anthropomorphic robots in the next chapter. Furthermore, the findings in this chapter identify important interactional outcomes during visitor-guide interaction, such as creating mutual gaze. These form the basis for designing behavior optimized for the morphology of a robot as studied in chapter

78

79 4 Translating human nonverbal behaviors to robots This chapter will be about the opportunities and limitations of the use of imitated human nonverbal cues for low-anthropomorphic social robots. First I will motivate why it is important to study effective nonverbal behavior for low-anthropomorphic robots. Second, an overview will be given on how nonverbal humanlike behaviors are currently imitated on robots. In this overview the focus will be on gaze behaviors, gestures and orientation behaviors. Next, two studies on the effects of humanlike nonverbal behavior on human-robot interaction will be presented. In the first study, the effects the translated humanlike gaze behaviors have on the responses and attitudes of participants will be explored. In the second study the effects of humanlike and functional orientation for a low anthropomorphic robot on the orientations and group formations of visitors will be explored. Finally, the chapter closes with a discussion on how the results of these two studies influence the development of nonverbal behaviors for a low-anthropomorphic robot. 4 This chapter is largely based on: Karreman, D. E., Sepúlveda Bradford, G. U., van Dijk, E. M. A. G, Lohse, M., & Evers, V. (2013, November). Picking favorites: The influence of robot eye-gaze on interactions with multiple users. In Proceedings of 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp ). IEEE. & Karreman, D. E., Ludden, G. D., van Dijk, E. M. A. G., & Evers, V. (2015, April). How can a tour guide robot s orientation influence visitors orientation and formations?. In Proceedings of 4th International Symposium in New Frontiers in Human-Robot Interaction. 63

80 CHAPTER 4 Single Modality Studies 4.1. Motivation For robots that look a lot like people, it seems logical to use humanlike nonverbal cues in their interaction behavior. This allows people to interpret the robot s nonverbal cues in the same way they would interpret nonverbal cues of people (Okuno, Kanda, Imai, Ishiguro, & Hagita, 2009). Furthermore, people might expect such behavior from humanlike robots, because the humanlike appearance strongly influences people s expectations of the interaction capabilities of the robot (Lohse et al., 2007). Lastly, humanlike behavior has a positive impact on the way people perceive the humanlike robot in interaction (Salem, Rohlfing, Kopp, & Joublin, 2011). However, is humanlike behavior also effective for lowanthropomorphic robots? From the observation of human tour guides as presented in chapter 3 it became clear that gaze behavior, gestures, and full body movements and orientation behavior have large effects on the interaction between guide and visitors. Therefore, in this chapter I will focus on the use of these three different nonverbal behaviors to understand how they influence the human-robot interaction Related work; How humanlike nonverbal cues influence human-robot interaction Influencing the interaction with robot gaze There are several examples of how gaze cues of social robots influence the physical and mental responses of people who interact with a robot. As has been presented in chapter 3, gaze is very influential for interaction between tour guide robots and people who interact with the robot. Changes in the gaze direction of the robot in guiding situations were interpreted as cues in the interaction, for example to change gaze direction to the robot or to shift gaze towards a point of interest. In other contexts than guiding the effects of gaze behavior of robots were studied. For example, Sidner, Kidd, Lee and Lesh (2004) found that a penguin-like robot wearing glasses that talked and moved its head to focus on the subject of interest, captured the attention of participants more often than a robot that only talked. They also found that people changed their own head direction to the subject of interest when the robot changed its head direction to it (Sidner et al., 2004). Moreover, it was found that leaking gaze cues of the robots Geminoid and Robovie lead to better performances from participants in a game where people had to pick out the item the robot had in mind from several items on a table (Mutlu, Yamaoka, Kanda, Ishiguro, & Hagita, 2009). In the game the robots were leaking information by looking at the item they had in mind. This example shows that these specific 64

81 gaze cues can be read as a cue to effectively shift the focus of interlocutor s attention. This means that also in other contexts than guiding, people seem to perceive the changes in head direction and gaze behavior of the robot as cues to focus on a point of attention. Robot gaze behavior does not only affects the physical reaction of people. People seem to derive the robots thoughts and intentions from its gaze behavior just as people do with the gaze cues of other people. Andrist, Tan, Gleicher and Mutlu (2014) found that intentional gaze aversion of a NAO robot, that told a story to one participant at a time, was perceived as intentional. They concluded that robots can use gaze aversions to appear more thoughtful and effectively manage the conversation floor (Andrist et al., 2014). Moreover, Sano et al. (2015) found that a robot (Robovie v3 with an adjusted head) which turned its head and used its eye blinks gave a friendlier impression, as compared with a robot that turned its head without eye blinks. The robot without eye blinks was more suitable for making people shift their attention towards the robot s gaze direction (Sano et al., 2015). These are remarkable and interesting findings, especially given that the robots nonverbal gaze cues are not as rich as the gaze cues of people. This is caused by the limitations in the design of robots, such as static or less dynamic eyes. However, these studies show that even eyes with limited functionality are efficient for communicating intentions. The previously described studies were performed with anthropomorphic to highly anthropomorphic robots to evaluate the effect of gaze behavior on the responses of people. Nevertheless, some studies on the effects of robot appearance on the effectiveness of robot gaze cues were performed as well. For example, Kidd and Breazeal (2004) studied how human eyes, robot eyes and animated eyes were perceived in terms of credibility, enjoyableness of the interaction, fairness, reliability, and informativeness. They did not find that people experienced the interaction between robot and human eyes differently, while they found that the interaction experience with animated eyes was lower compared to the interaction with robot and human eyes (Kidd & Breazeal, 2004). Therefore, they hypothesize that this might be due to the fact that the human and the robot eyes in this study were seen as real, while the animated eyes were seen as being human-made (Kidd & Breazeal, 2004). 4 In contrast to that, Mutlu et al. (2009) found that a less perfect copy of a human (in this case Robovie) did elicit less positive responses in the, previously described, object picking game than the more perfect copy of the human (Geminoid) did. Although the cues for both robots were designed to be of a similar level, the gaze cues of Robovie were less dynamic and as a result less effective in showing rich cues compared to those of Geminoid (Mutlu et al., 2009). Even though the studies of Kidd and Breazeal (2004) and Mutlu et al. (2009) present different results, I conclude that this suggests that robot eyes of less anthropomorphic robots are less effective than the eyes of people or highly anthropomorphic robots, but are still effective. 65

82 CHAPTER 4 Single Modality Studies In contrast with both above described studies, Admoni and Scassellati (2012) found in a study in which they evaluated peoples low-level responses to gaze cues of several faces, that while human faces (also includes line drawings of human faces) and arrows cued reflexive attention shifts, robot gaze (Zeno and Keepon) did not. Furthermore, it seems that all robots perform less well in gaze cues to convey directional information on a reflexive level compared with human faces, and arrows (Admoni, Bank, Tan, Toneva, & Scassellati, 2011). They found that all stimuli they used (human face, face of Zeno, face of Keepon and an arrow) were able to convey the directional information, but both robots failed to elicit attentional cueing effects (shift attention to the object of attention of the robot) that were evoked by non-robot stimuli. This effect was regardless of robot anthropomorphism, as they used the highly anthropomorphic face of Zeno and the lower anthropomorphic Keepon (Admoni et al., 2011). They hypothesize that even though people might interpret robots gaze as social cues, on the reflexive level, they need more time to interpret the gaze cues. Although the low-level response to read gaze cues from robots in peoples minds is different from processing human gaze cues (Admoni & Scassellati, 2012), the high-level response of people seems to be the same. As I have shown, even though the eyes of robots are not as dynamic as human eyes and they cannot produce as rich cues as people can, people still read the intentions of the robots via the (intentional) communication cues given with the robots eyes. This is in line with the findings that people tend to anthropomorphize robots and their behavior (Duffy, 2003; Fink, 2012). To gain deeper insight into the effect of gaze on attention, Mutlu, Forlizzi and Hodgins (2006) performed a study in which they specifically measured the recall of details of a story told by an ASIMO robot to two interlocutors at the same time. For this study, they based their hypotheses on findings from psychology that indicate that people recalled details better when there was eye contact between storyteller and listener (Fry & Smith, 1975; Otteson & Otteson, 1979; Sherwood, 1987). Supporting their hypotheses, they found that participants performed better in recalling the story told by ASIMO, when the robot looked at the participants more (Mutlu et al., 2006). Thus, with this study, Mutlu et al. (2006) showed that gaze effects of human-human interaction are transferable to human-robot interaction. Contrarily, van Dijk, Torta and Cuijpers (2013) found that there was no effect for gaze behavior of the NAO on the retention of a message whether it looked at participants or not. This indicates that the previously presented effect as found by Mutlu et al. (2006) is not generally supported in human-robot interaction. Therefore, I hypothesize that the contradictory results between the two previously described studies can be explained by actions of the robot beyond gaze behavior. The previously presented studies indicate that apart from nonverbal gaze behavior only, other nonverbal behaviors also influence the interaction, in which a subtle interplay 66

83 between all nonverbal cues occurs. Subsequently, in studies in which the focus is on one main modality, actually the effectiveness of multimodal behavior for the robots is evaluated. This happened, for example, in a study in which participants were asked to teach robot Leonardo the names of buttons and afterwards press them all. In this study, the robot Leo could respond explicitly only, this meant that it responded directly to questions and looked at the button prior to pointing or pressing, or Leo could respond implicitly, which implied that it responded with gaze direction and head changes, showed confusion and showed some liveliness (Breazeal, Kidd, Thomaz, Hoffman, & Berlin, 2005). Results showed that participants understood the robot better when it gave nonverbal information implicitly as well as explicitly and that participants had a better mental modal of the robot than when it gave information only explicitly (Breazeal et al., 2005). In a similar vein, Chidambaram, Chiang and Mutlu (2012) found that participants complied more with the suggestions of a Wakamaru robot when the robot used nonverbal cues, such as head gestures and gaze, proxemics and hand gestures, than when the robot did not so. Hence, as a robot s gestures appear to have a significant impact on the perceived gaze behaviors, I will focus on studies evaluating gesture behavior and orientation behavior of robots in the next sections Influencing the interaction with robot hand gestures The design of hand and arm gestures for humanlike robots is just as a robot s gaze behavior mainly inspired by human-human interaction. McNeill (1992) and Alibali, Heath and Myers (2001) show that people in human-human interaction use four types of gestures to support their speech: deictic gestures, iconic gestures, metaphoric gestures and beat gestures. 4 Deictic gestures are pointing gestures that refer to people, objects or locations around. Iconic gestures are gestures that refer to concrete actions, objects or people. Metaphoric gestures are gestures that refer to abstract ideas. Beat gestures do not represent content, but support the story of the speaker rhythmically. In chapter 3, I defined the gestures that tour guides used as follows; pointing gestures (deictic gestures), depicting gestures (iconic gestures and metaphoric gestures) and supporting gestures (beat gestures). As human tour guides use a combination of these four types of gestures (explained in chapter 3), this indicates that these gestures might be effectively used by tour guide robots and robots performing tasks in other contexts. Several studies in human-robot interaction have shown that the use of gestures for robots influence the human-robot interaction positively (e.g. (Huang & Mutlu, 2013; Lohse, Rothuis, Gallego Peréz, Karreman, & Evers, 2014; Salem, Eyssel, Rohlfing, Kopp, & Joublin, 2013; van Dijk et al., 2013)). In the following paragraphs I will explain this effect. Furthermore, the way that gestures are made also influences the interaction, as it may show some emotion or personality of the robot (Kim, Kwak, & Kim, 2008), but this aspect of gesturing goes beyond 67

84 CHAPTER 4 Single Modality Studies the scope of this thesis. Studies on robots using gestures have often been performed with two or more of the four types of gestures integrated, because it is very difficult to measure the effect of one type only. For example, Salem et al. (2013) found that the use of iconic gestures, deictic gesture and what they call pantomimic gestures (a specific form of iconic gestures) by ASIMO congruent with its speech resulted in that participants anthropomorphized the robot more, perceived the robot as more likeable, reported greater shared reality with the robot and showed increased future contact intentions than when ASIMO gave instructions without gestures. Also, they found that non-congruent gestures of ASIMO had negative effect on the task performance of the participants (Salem et al., 2013). Further, Huang and Mutlu (2013) found that all types of gestures affect user perceptions of the Wakamaru robot s performance as a narrator positively. Nevertheless, in the following paragraphs I describe the effects of several gestures individually: deictic gestures, iconic and metaphoric gestures, and beat gestures. In general, pointing gestures of a robot are easy for people to understand. This was for example shown by Shiomi et al. (2008), who conducted a study with a robot (Robovie v1) at a train station pointing people to several points of interest. Furthermore, Huang and Mutlu (2013) found that pointing gestures of a Wakamaru robot consistently predicted users information recall. Also, pointing gestures of a Nao have a positive effect on the task performance of difficult route direction tasks, as Lohse et al. (2014) found that robot gestures increased user performance and decreased perceived workload. Even if a robot does not have arms, it can point with other body parts effectively, such as its beak (Sidner et al., 2004). For human tour guides the gaze and pointing gestures seemed to be interconnected (Best, 2012), which seems to be the case for anthropomorphic robots as well. This is for example shown in a study in which several ways of giving instructions with a Nao robot were compared. Results showed that gaze alone was not very effective, but that the combination of gaze and pointing was most effective to give instructions (Häring, Eichberg, & André, 2012). Although Okuno et al. (2009) only applied deictic gestures to their Robovie (v1) robot, they argue that for example the use of iconic gestures would improve the effect of the gestures. Their reason to leave these kind of gestures out of the set of gestures made by the robot, is that most robots does not have dynamic hands and therefore are not able to make detailed iconic or metaphoric gestures. As a result, they argue that using such gestures when the robot lacks some interaction modalities only would make the interaction unnatural and more difficult to understand for participants (Okuno et al., 2009). Nevertheless, Huang and Mutlu (2013), who used a Wakamaru robot that was not able to perform dynamic hand gestures, found that iconic/metaphoric gestures performed with the robots arms helped participants 68

85 to retell a story the robot told them earlier. For males these robot arm gestures positively affected the narration length, while it influenced the use of gestures for females (Huang & Mutlu, 2013). Also, Van Dijk et al. (2013) found the use of iconic gestures of Nao indeed aided retention, but only of the verb to which the action-depicting gestures pertain. This shows that iconic and metaphoric gestures are of importance in human-robot interaction. As beat gestures in human-human interaction mainly seem to support the speaker and not the understanding of the listener, I would argue that beat gestures are not necessary for the robot. This is because a robot does not think in the same manner as people do. A reason why beat gestures could be useful for social robots is to give them some liveliness. Sidner et al. (2004) used beat gestures for the wings of a penguin robot, and found that the moving robot was better in keeping the attention of people than the robot that did not make any movements. Furthermore, added arm gestures to the storytelling ASIMO robot gave it some liveliness (Mutlu et al., 2006). Thus even though beat gestures have no added value for the speech of the robot, they might have a positive effect on the human-robot interaction. To conclude, the imitation of human gestures has primarily been applied to anthropomorphic robots, because low-anthropomorphic robots might not have arms or hands. Limited degrees of freedom in the hands and arms of the robots may cause a limitation for humanlike robots to perform hand and arm gestures (Okuno et al., 2009). This can make it impossible for the robot to perform a certain action, to get the timing right or to recognize the desired gesture. However, as I showed in this section, gestures of robots can have large influence on the effectiveness of human-robot interaction. Therefore, designers of social robot should keep in mind the effect (leaving out) gestures can have, even if they design no or simplified arms for a robot Influencing the interaction with robot bodily movement and orientation With a change in body orientation, an ASIMO robot can elicit a change in the formation between itself and a person, such as was described by Kuzuoka, Suzuki, Yamashita and Yamazaki (2010). With this study, they explicitly found that not head movement alone, but full body movement of the robot was more effective to reconfigure the formation (Kuzuoka et al., 2010). Furthermore, Shiomi, Kanda, Ishiguro and Hagita (2010) found that a Robovie (v1) robot that guided people to specific shops attracted more spontaneous visitors when it drove backwards when guiding people to a shop entrance. This indicates that humanlike orientation and posture is effective for social robots as well. Positioning also influences how people perceive a robot. Joosse, Poppe, Lohse and Evers (2014) used a questionnaire to ask 181 participants from China, U.S. and Argentina how close a robot could approach a group of a father a mother and a child, and found that 69

86 CHAPTER 4 Single Modality Studies participants preferred a robot that stayed out of their intimate zone. However, there are cultural differences in these preferences, for example, Chinese participants believed that closer approaches by the robot were appropriate, in contrast to U.S. participants (Joosse et al., 2014). Moreover, they found against their expectations that the robot approach between mother and child was appropriate. They hypothesize that this is believed to be appropriate because then the robot faces the man directly (Joosse et al., 2014). This indicates that positioning of the robot influences how people perceive the interaction. Even though orientation and positioning of the robot can influence visitor s responses, it is difficult to attribute these responses of people to the orientation of the robot only. This is due to the strong connection between orientation and other modalities (e.g. eyes or screens) of the robot. For the examples discussed above, the appearance of the robot, which was mainly humanlike, dictated the orientation of the robot. For example, in order to look at people, the robot had to orientate with the front side towards the people. A disadvantage in human-robot interaction research is that in most of the cases it is not clearly described why certain robot modalities and appearances were chosen for the studies. This is especially the case when non-humanlike robots are used. Therefore, the effects of body orientation for low-anthropomorphic robots should be analyzed better to fully understand them. In the following sections I will address: (1) the effect of humanlike gaze behavior of a low-anthropomorphic robot on the effectiveness and intuitiveness of the human-robot interaction, (2) and the effect of low-anthropomorphic robots orientation behavior on the orientation behavior of people. In order to study the effectiveness of the robots nonverbal behaviors, I used knowledge from human-human interaction as a way to start the exploration to obtain new information. Therefore, I used the observed human tour guide behaviors that I described in chapter 2 as a starting point to design nonverbal behaviors for a social tour guide robot. To understand the effects of the use of humanlike behaviors in lowanthropomorphic robots, I compared the effectiveness of these cues based on the (desired) responses of participants Study 3: The influence of a low-anthropomorphic robot s eye-gaze on interactions with multiple users When people interact, nonverbal communication plays an important role. Here is an example in the museum context: when visitors follow a guided tour and are not interested in the story anymore, they start looking around. In contrast, interested visitors direct their attention towards the point of interest or the guide (Best, 2012). Another example shows even more subtle nonverbal communication: when a human tour guide is about to finish a story at one exhibit, the guide already orients him/herself a little bit towards the direction of the next exhibit while giving the last bits of information. By doing this, the guide communicates to 70

87 the visitors that the story will finish soon and in which direction they will go next (Karreman, van Dijk, & Evers, 2012). In the following study we explored how subtle humanlike gaze cues can be transferred to low-anthropomorphic robots Hypothesis For the purpose of this study, we focused on robot gaze behavior as a way of indicating or engaging in shared attention towards an object of interest. As discussed in section 4.2.1, gaze cues by humanlike robots evoke responses similar to human gaze cues (Mutlu et al. 2009). Moreover, the robot that showed gaze cues was perceived as more reliable than the robot that only talked (Sidner et al., 2004). Therefore, we expected that a robot could effectively use its gaze to direct a person s attention towards a particular object. We expected that for a robot guide, directing a visitor s attention towards an object with its gaze behavior would result in more attention towards the object of interest. We expected that if a robot steered the visitors attention towards an object then the visitors would pay more attention to the information the robot conveyed about the object. This lead to the first hypothesis: Hypothesis 1: When a robot displays object-oriented gaze behavior (looks at the participants as well as to the object of interest), participants will pay more attention to the object and the robot, will remember more about the object and the story, and will have a more positive attitude towards the robot compared with a situation in which a robot displays person oriented gaze behavior (looks at the participants only). 4 As well as gaze behavior towards the object, the robot s gaze behavior towards the persons may influence the interaction. When interacting with groups of people, human tour guides distribute their attention between the members of groups, however, they do not do this evenly (see Chapter 3). Human guides gaze at some people more than at others. Research on gaze cues in person-to-person interaction shows that the amount of gaze expresses and evokes liking as well as attraction (Kleinke, 1986). People who are positively attracted to each other tend to gaze at each other more. Being favored by robot gaze behavior might therefore as well influence the perception of the robot. Further studies report that in situations where gaze is interpreted positively rather than as an act of aggression, participants tend to comply more to requests from experimenters that gaze at them than to requests from experimenters that do not (Kleinke, 1986). Another effect of robot gaze that has been found by Mutlu et al. (2006) is that people tend to have better recall of a story told by a robot when the robot looked at them, even when the robot does not have dynamic eyes. Such an effect was derived from person-to-person interaction (Otteson & Otteson, 1979). 71

88 CHAPTER 4 Single Modality Studies This lead us to consider that a robot guide that interacted with a small group would inevitably distribute its gaze unequally between the visitors regardless of the behavioral algorithm implemented. We were interested to find out whether this would be perceived as getting more attention from the robot. We expected that favored visitors would like the robot more and would pay more attention to its story about the object of interest. We expected that when the robot displayed object-oriented gaze, favored people would be more inclined to engage in shared attention towards the artwork. This lead to the second hypothesis: Hypothesis 2: When a robot pays more attention to one participant in a small group, the favored person will pay more attention to the object and the robot, will remember more about the story, and will have a more positive attitude towards the robot compared with the not favored participants. Also, the favored person will be more prone to engage in shared attention with the robot compared with the not favored participants Study design We conducted a 2 (object-oriented vs. person-oriented gaze) x 2 ( favored vs. not favored by the robot) mixed factorial design study, where a robot guide talked about two artworks to groups of three participants. Participants 57 Participants, students and staff members of the University of Twente, took part in the experiment (mean age was 25.6, SD = 7.59; 41 male, 16 female participants). Of the participants 79% studied or worked in IT. However, 91% of the participants reported having little or no experience with robots. Slightly more than half (56%) of the participants had some previous knowledge about one or both artworks. Robot The platform used in the experiment was a Magabotrobot. It came with a table-structure and a custommade shell was used. The custom-made shell was very basic and had no anthropomorphic features. A laptop showing anthropomorphic eyes was placed on top of the structure of the platform. The robot turned its front towards the point of interest it was looking at as well as moving its eyes to that side to indicate it was looking at a specific participant or at one of the artworks. See Figure 4.1 for an impression of the robot. Figure 4.1: The Magabot as tour guide robot 72

89 Manipulations During each session of the experiment, the guide robot stood in front of two posters of famous artworks; The Mona Lisa, and, The Girl with the Pearl Earring. The robot told a story about both artworks to groups of three participants. Favored vs. not-favored condition: While giving information about the artworks, the robot looked towards each of the participants and alternated its gaze between them. However, the robot focused on the person on the left, and alternated its attention at eight moments during the story to the persons in the middle and on the right for 1 to 2 seconds. Person-oriented vs. object-oriented gaze condition: In the person-oriented gaze condition, the robot alternated its gaze between the three participants, as described above. In the object-oriented gaze condition, the robot also turned towards ( gazed at ) the artwork it was talking about. The duration of this gaze cue was 2 to 6 seconds. The robot looked towards the artworks at moments when a human tour guide would point at the object. When the robot looked at the objects, it asked participants explicitly to look at the artwork ( please take a look at her hands ), or it was implicit about looking at the artwork ( the Mona Lisa is perhaps the most famous artwork in western art history ). 30 Participants in 10 groups interacted with the robot in the object-oriented condition, 27 participants in 9 groups interacted with the robot in the person-oriented condition. Procedure The experiment was performed in a lab at the University of Twente. The participants were welcomed in groups of three and informed about the study; that they would join a robot that would explain two famous paintings as if it was a museum guide. All participants signed a consent form before entering the experiment room. Figure 4.2 shows the setup of the room. The participants were asked to position themselves on one of the three lines. The position of the participants was partly predefined, as they had to stand on a line drawn on the floor. However, the participants were able to choose their proximity to the robot. Also the lines were designed to place participants around the robot tour guide as they would be in real exhibition areas where they would stand common-focus gathering: in small groups a slight difference between speaker and group (oriented in semi-circle) becomes visible (Marshall, Rogers, & Pantidi, 2011). The robot was remote controlled by one of the experimenters in a Wizard-of-Oz setting. After listening to the robot s story about the two artworks, the participants filled out an online questionnaire individually on laptops provided to them. Each session took approximately 30 minutes. 4 Measures To measure attitudes towards the robot, a questionnaire was developed using several 73

90 Single Modality Studies CHAPTER 4 Figure 4.2: Schematic top-view of experiment setup validated scales. To measure Trust in the robot we used the sociability, competence, composure and character subscales of the 15 item Source Credibility Scale of McCroskey (Rubin, Palmgreen, & Sypher, 2004). In addition, the subscales Anthropomorphism, Likeability, and Perceived Safety of the Godspeed Scale (Bartneck, Kulić, Croft, & Zoghbi, 2008) were used. The subscales Attentional Allocation and Co-Presence of the Social Presence Measure of Harms and Biocca (Harms & Biocca, 2004) were used to measure perceived attention received from the robot and given to the robot and perceived co-presence with the robot. Table 4.1 shows the items and the reliability (Cronbach s α) of the subscales. The items of the Source Credibility Scale and of the Godspeed Scale were measured on a 7-point semantic differential scale and the items of the scales were presented in a random order in the survey. The items of the Social Presence Measures were measured on a 5-point Likert scale (for all questions answers were ranged strongly disagree strongly agree ), such as in the original questionnaires. Some items were added to mask the intention of the questionnaire; only one of the added items (safe-threatening) was used in the analysis, as replacing the original item quiescent-surprised by this one improved the reliability of the perceived safety sub scale (from α=0.43 to α=0.70). (See Appendix B for the full questionnaire presented to the participants). Besides the questionnaires, we used video recordings to measure the visitors attention more objectively. We annotated and analyzed the videos to examine where the participants looked at specific moments during the experiment. We annotated all seven instances of explicit compliance (people were expected to look at the artwork when they were explicitly asked to look), and all seven instances of implicit compliance (people were expected to 74

91 look at the artwork because the robot turned and looked even though the robot only looked at the artwork in the object-oriented mode, we annotated the same moments in the person-oriented mode for comparison) and 20 times general attention (number of times people looked at the artwork or at the robot, measured at fixed times during the experiment, time between those moments was 30 seconds). Participants who looked at the robot or the artworks were considered to be paying attention to the story and participants who looked elsewhere were considered to be distracted. (See Appendix C for the annotation scheme of participant s attention). Measurement Table 4.1: Items and reliability of scales used α Source Credibility Scale 0.83 Good-natured Irritable Cheerful-Gloomy Friendly-Unfriendly * Expert-Inexpert Unintelligent-Intelligent Intellectual-Narrow Poised-Nervous Tense-Relaxed * Calm-Anxious Dishonest-Honest Unsympathetic-Sympathetic Good-Bad 4 Godspeed: Anthropomorphism 0.82 Fake-Natural Machinelike-Humanlike Unconscious-Conscious Artificial-Lifelike Moving rigidly-moving elegantly Godspeed: Likeability 0.83 Dislike-Like Unfriendly-Friendly * Unkind-Kind Unpleasant-Pleasant Awful-Nice Godspeed: Perceived Safety 0.70 Tense-Relaxed * Agitated-Calm Safe-Threatening 75

92 CHAPTER 4 Single Modality Studies Measurement (continued) Social Presence Measure: Co-Presence I noticed the robot The robot s presence was obvious to me My presence was obvious to the robot The robot caught my attention I caught the robot s attention Social Presence Measure: Attentional Allocation I was easily distracted from the robot when other things were going on The robot was easily distracted from me when other things were going on I remained focused on the robot throughout our interaction The robot remained focused on me throughout the interaction The robot did not receive my full attention I did not receive the robot s full attention α * Only asked once, but used in both scales To measure likeability objectively we also used proxemics: participants chose their own position (on a line) and could move freely towards or away from the robot. The lines had marks every 25 cm starting from the robot, so the researcher was able to observe and write down the distance between participants and the robot. In accordance with Mumm and Mutlu (2011) we assumed that the closer the participants stood, the more they liked the robot. The distance between participants and robot was measured three times through the experiment. The first occurrence was when the participants positioned themselves at the start of the narrative. Second, halfway during the story, when the robot finished it story about the Mona Lisa and started explaining about the Girl with the Pearl Earring. The last measurement was taken at the end of the story, when the robot said goodbye. To obtain insight into the recall of details of the stories and the artworks, the participants had to answer some knowledge questions individually, as part of the questionnaire. There were three kinds of questions. 1) questions about details mentioned in the story and visible in the artwork, 2) questions about details mentioned in the story only and 3) questions about details visible in the artworks only. For each knowledge question three possible answers and the option I can t remember were presented. Data analysis The questionnaire data were analyzed using SPSS. All scales were first tested for reliability, using the Cronbach s α coefficient. After tests for normality, two-way ANOVA s were run to test the main hypotheses. Video data was pre-processed and only parts that showed participants listening to the 76

93 robot s story were annotated. The length of the videos ranged from minutes to minutes. The videos were annotated by two researchers, these annotations were tested for inter-rater reliability. The annotators scored attention-direction of the participants; that is whether the participants were looking at the robot or the artwork (showing attention), or the participants were looking away (showing no attention). We annotated three different events - explicit compliance, implicit compliance and general attention. The overall similarity of all annotations was 91%. However, the inter-rater reliability Kappa turned out to be 0.51, which means a moderate agreement (Sim & Wright, 2005). This low Kappa is likely due to the Boolean scoring (no attention-attention), so the chance of choosing one of them is 50%. Also, we expected people to have attention most of the time, and rated it more than participants having no attention. Therefore, the likeliness for attention to happen is high and the chance that both researchers rate attention accidentally the same is also higher, which decreases the inter-rater reliability Kappa. For the analysis of the video annotations, the annotations of one of the researchers were used without preference Results The manipulation of favoritism was checked with one survey item that asked who the robot had looked at most. The majority of the participants had indeed noticed that the robot looked more at one of the participants, indicating that the manipulation was successful. 44 of 57 people (77%) responded correctly that the participant on the line at the side of the Girl with the Pearl Earring was favored. Only 4 of 57 (7%) responded the participant on the line in the middle was favored, and the remaining 9 (16%) responded that they did not know. None of the participants responded that the participant in front of the Mona Lisa was favored. 4 The manipulation check for gaze condition was done by analyzing the video data. We expected that when the robot looked at the artwork, participants would also look at the artwork and thus engage in shared attention. However, we did not find this. Instead, we found that participants in the object-oriented mode paid more attention to the robot at explicit compliance moments, F(1,35) = p <.001, and at implicit compliance moments, F(1,35) = 21.90, p <.001, than participants in the person-oriented mode. Instead of directing attention to the object, it seems that the robot s (gaze) movement drew attention to the robot. This means we successfully drew people s attention with the object-oriented gaze behavior but were not successful in creating shared attention towards the art object. Hypothesis 1 concerned the expectation that participants in the object-oriented gaze condition (the robot looked at the participants and also at the artwork) would pay more attention to the object and the robot, would remember more about the object and the story, and would have a more positive attitude towards the robot. In fact, participants interacting with the robot in the object-oriented gaze condition tended to perceive the robot as more 77

94 CHAPTER 4 Single Modality Studies humanlike, F(1,53) = 3.84, p =.055, than participants that were exposed to the robot looking at participants only. Also, these participants (M = 1.39m, SD = 0.32m) stood significantly closer to the robot halfway through the interaction, F(1,53) = 4.17, p <.05, than participants in person-oriented gaze condition (M = 1.58m, SD = 0.35m). Overall, during the story, the difference in proximity between participants in object-oriented mode (M = 1.39m, SD = 0.32m) and person-oriented mode (M = 1.58m, SD = 0.35m) was marginally significant. The participants in the object-oriented mode tended to stand closer to the robot F(1,53) = 3.60, p = 0.063, which indicates they liked the robot more than participants in person-oriented mode. No differences were found for either recall of story or artwork details between the two groups. Also, no differences in general attention between groups were found. With these results hypothesis 1 is partly supported. Hypothesis 2 concerned the effect of a robot paying more attention to one of three participants. Expectations were that the favored person would pay more attention to the object and the robot, would remember more about the story the robot told, and would have a more positive attitude towards the robot. Also, favored persons were expected to be prone to engage in shared attention in the object-oriented condition. Results showed that favored participants perceived to receive more attention from the robot and to give more attention to the robot, F(1,53) = 91.74, p <.001, than not-favored participants. Also, favored participants found the robot more present and felt the robot to be more aware of their presence, F(1,53)=37.79, p <.001, than their not-favored colleagues. Furthermore, we found that favored participants tended to like the robot more, F(1,53) = 3.74, p =.059, compared to not-favored participants. A trend for the main effect being favored at implicit compliance moments, F(1,35) = 3.39, p =.074, was found. Favored participants tended to look less at the robot and more at the artwork than not-favored participants in both gaze conditions. Results also showed a surprising interaction effect between gaze condition and being favored at explicit compliance moments. In the person-oriented mode, the participants looked more at the artwork when the robot told them explicitly to do so than participants in the object-oriented mode, F(1,35) = 5.22, p <.05. This effect more strongly influenced participants that were favored by the robot (object-oriented mode M = 0.83, SD = 0.21; person-oriented mode M = 1.00, SD = 0.00) than the not-favored participants (objectoriented mode M = 0.95, SD = 0.07; person-oriented mode M = 0.97, SD = 0.06). The values indicate the probability that a participant is looking at the artwork. This shows that contrary to our expectation, specifically favored participants had more attention for the artworks in the person-oriented mode rather than the object-oriented mode. At moments of explicit compliance another interaction effect showed that participants looked more at the robot and/or artwork in the person-oriented mode, F(1,35) = 4.48, p < 78

95 0.05 rather than in the object-oriented mode. Favored participants in the person-oriented mode looked significantly more at the artwork or the robot than favored participants in object-oriented mode. For not-favored participants this trend seemed to be the other way around, but was not significant Discussion Even though the previously presented results only lead to partial acceptance of hypothesis 1, we found that participants in the object-oriented mode rated the robot more humanlike and the participants attention indeed was attracted towards the robot. However, participants in the object-oriented gaze condition did not pay more attention to the artwork compared with the person-oriented condition. Also, there were no differences in remembering the story between participants in object-oriented mode and person-oriented mode. These results seem to indicate that robot movements related to gaze behavior do indeed draw attention as indicated by Sidner et al. (2004) but do not necessarily direct the visitors attention to the object of interest as expected. We found that people did not engage in shared attention with the robot but instead focused more on the robot. We know that people very effectively direct other people s attention with directed gaze behavior (Kendon, 1967), however, in this study the same behavior copied one-to-one to a robot resulted in more attention towards the robot rather than the object of interest. This makes us conclude that implementing functional gaze behavior in a robot to direct visitors attention (as an effective alternative for pointing) does not carry across easily from person-to-person communication to human-robot interaction and low-anthropomorphic robots specifically. 4 The findings only partially support hypothesis 2. Favored participants did indeed perceive the robot as paying attention to them and felt the robot was aware of them, indicating a more positive attitude towards the robot. They tended to like the robot more than the notfavored participants. However, favored participants did not show a better recall of details and did not stand closer to the robot than not-favored participants did. The results indicate differences between being favored or not, but not as we expected. Favored participants did not remember the story better, while they did in the study of Mutlu et al. (2006), which might be due to the distraction of the moving robot. Also, favored participants tended to like the robot better, but they did not stand closer than the not-favored participants, while we would expect that result based on previous findings of Mumm and Mutlu (2011). However, this effect was probably caused by influences of the other group members. Even though we artificially created a situation where three people interacted with a robot, their orientation was comparable to the orientation of visitors around a human tour guide. 79

96 CHAPTER 4 Single Modality Studies Therefore, we can use the study findings that clearly show differences in attitudes and behavioral responses towards the robot. Future research should be carried out in the wild to explore whether more spontaneous encounters between people and robots yield the same results. A limitation of the study could be that 79% of the participants studied or worked in IT. These people have higher interest in and better understanding of the possibilities of computers, therefore these people might have a different attitude towards interacting with robots than a more general group of people. Because, 91% of them had little to no experience with robots at the time of the study. We feel this group is still reasonably representative. For the remaining studies of this thesis we ensured a more mixed ans representative sample. Overall, we found that people responded to movements of the robot, as Sidner et al. (2004) stated in previous work. However, our findings are not completely in line with the results of Mutlu et al. (2009) who found that people understand nonverbal communication cues intuitively when applied to robots. In our study, people did not interpret the robot s gaze cue as a signal to look at the artwork. We found the robot less effective in creating shared attention with gaze. Perhaps the robot s movements distracted people and drew their attention towards the robot rather than engaging them in shared attention towards the object. This could indicate that a typical social gaze cue ( look at this object ) carried out by human guides is interpreted differently when displayed by a low-anthropomorphic robot. Furthermore, in the person-oriented gaze condition, the robot kept its gaze on the participants throughout the interaction. We know from previous work that people tend to comply when they are being gazed at as a request is made (Kleinke, 1986). Perhaps this explains participants compliance to the robot s request to look at artworks in the personoriented mode. In general, this study shows that more research to effective gaze behavior for lowanthropomorphic robots is needed. We have showed that the effect of gaze behavior for this low-anthropomorphic robot is influenced by movements of the robot. Therefore, the effect of multimodal behavior should be studied as well Study 4; How does a tour guide robot s orientation influence visitors orientations and formations? Several robots have been developed to give guided tours in museum-like settings, as I presented in chapter 2. While giving the tours, these robots captured the attention of visitors, had interactions with visitors and guided the visitors through smaller or larger parts of exhibitions. Studies reporting information about visitors responses to the robot s actions 80

97 have led to knowledge on specific responses of people to the modalities of these robots and behavior shown by these robots. In this section I will focus on the formation and orientation of visitors as a response to the robot orientation behavior. We use the term formation to indicate the group structure, distance and orientation of the visitors who showed interest in the robot and/or the point of interest the robot described. In human guided tours, people generally stand in a commonfocus gathering, a formation in which people give each other space to focus on the same point of interest, often a semi-circle which is a specific F-formation (Kendon, 2010). For robot guided tours, we expected to find similar formations. However, previous research has indicated that also individual persons or pairs of visitors only joined the tour (Jensen, Tomatis, Mayor, Drygajlo, & Siegwart, 2005, Thrun et al., 1999). Therefore, we considered the combination of distance and orientation of these individuals or pairs as formations as well. We assumed that people would be engaged with the robot or the explanation when they were oriented towards the robot or the point of interest for a longer period of time. Hence, we also use the terms formation, orientation and engagement separately from each other in order to be specific in the description of the results Research question The question we wanted to answer with this study was: how does the robot orientation behavior influence the orientations of the visitors, as well as the type of formations that (groups of) visitors form around the robot? Study design The goal of this study was to determine how orientation behavior of a low-anthropomorphic robot influenced visitors orientation and the formations groups of visitors formed around the robot. The orientation behavior of the robot was manipulated, while other interaction capabilities were limited due to the minimalistic robot platform. To evaluate how visitors responded to the robot, we performed a study in a real world environment in the Royal Alcázar in Seville (Spain). The robot gave short tours with four explanations at points of interest in the Hall of Festivities in the Royal Alcázar (See Figure 4.4). Participants Participants of the study were visitors to the Royal Alcázar. At both entrances of the room, all visitors were informed with signs that a study was going on. By entering the room, visitors gave consent to participate in the study. It was up to them if they wanted to join the short tour given by the robot or not. Approximately 500 people (alone or in groups ranging from 2 to 7 visitors) interacted with the robot during the study, which was conducted over a period of 2 days in May

98 CHAPTER 4 Single Modality Studies Robot The robot used for this study was low-anthropomorphic. We chose this particular robot to be able to determine the effects of body orientation on visitors responses without being influenced by other factors in robot design and behavior (such as aesthetics of the robot, pointing mechanisms, visualizations on a (touch-) screen or active face modifications). The robot was a four-wheeled data collection platform (see Figure 4.3). The body of the robot was covered with black fabric to hide the computers inside. A bumblebee stereo camera was visible at the top of the robot, as well as a Kinect below the bumblebee camera. The robot was remotely operated. The operator was present in the room, even though he was not in the area where the robot gave tours, he was visible for the participants. The robot was operated using a laptop. The laptop screen was used to check the status of the robot, while the keyboard was used to actually steer the robot. The interaction modalities of the robot were limited; the robot was able to drive through the hall, change its orientation and play pre-recorded utterances. The instruction follow me was visible on the front of the robot, and signs informing people about the research (in English and Spanish) were fixed to the sides of the robot. During the study we used a user-centered iterative design approach (Gould, Watson, & Lewis, 1985) for the behavior of the robot. When the robot charged in between sessions, we discussed robot behaviors that had the intended effect and behaviors that did not work well. During the study we modified the explanation of the robot after session one, because it became clear that visitors did not understand where to look. A total of three iterations were tested. In all iterations changes were only made to the explanation of the robot, however the content about the points of interest remained the same. Procedure The tour given by the robot took about 3 minutes and 10 seconds. Visitors were not beforehand informed or invited to join a tour, only limited information about the research and consent for filming was showed on three information boards in the hall. The points of interest chosen were all visible on the walls of the room (no exhibits were placed anywhere in the room), however the position of the points of interest on the walls differed in height. During a tour, the distance to drive in between the points of interest also differed, from approximately two meters up to approximately twelve meters. This was done so we could see if there were different visitor behaviors when following the robot. However in this thesis I will not focus on the results of visitors following the robot. When visitors entered in the Hall of Festivities, the robot stood at the starting place (1) (see Figure 4.4) and began the tour by welcoming the visitors and giving some general information about the room. When the robot had finished this explanation, it drove to the 82

99 next point of interest (about 7 meters away), asking the visitors to follow. At the next point of interest (2) the robot told the visitors about the design of the figures on the wall that were all made with tiles, after which it drove the short distance (about 2.3 meters) to the next exhibit. At the third point of interest (3) the robot told the visitors about the banner that hung high above an open door. At the end of this story the robot asked the visitors to follow after which it drove the long distance to the last point of interest (about 11.8 meters). Here (at point of interest 4) it gave information about the faces visible on the tiles on the wall. Before ending the tour the robot drove back to the starting place (about 5.5 meters), informed the visitors the tour had finished and wished them a nice day. After a while, when new visitors entered the room, a new tour was started. During the study the robot tried to persuade visitors to follow it with the sentences please follow me and don t be afraid, if the robot operator noted visitors were hesitant. In all cases it was up to the visitors to decide whether they followed the robot or not. Visitors were never instructed to follow the robot by researchers who were present in the room. As the study was performed in a real life setting, with uninformed visitors, at times we had to deviate from the procedure. The robot had defined places for explanations at points of interest. Specifically at times the robot could not stop at the exact way-point, because people walked or stood in front of the robot. Another reason to deviate was when the robot lost the attention of all people who followed the tour. In those case the robot drove back to the starting place and started over again. If visitors lost interest and left, but other visitors stayed to listen to the robot, it continued the tour. 4 If all visitors left the hall, or did not show any attention towards the robot, the tour was aborted, and restarted when new visitors entered the hall. Therefore the number of times the robot presented at each of the four exhibits decreased. The robot started the tour 87 times at the first exhibit, continued 70 times at the second exhibit. At the third exhibit the robot started its presentation 63 times and it finished the story only 58 times at the fourth exhibit. A total of 278 complete explanations at points of interest were performed (see Table 4.2 for a specification of the actions per point of interest). Manipulations During the study, we manipulated the robot s orientation behavior. Either the robot was orientated towards the point of interest or the robot was orientated towards the visitors. When it was orientated towards the point of interest, the front of the robot faced in the direction of the point of interest. The points of interest were all located a few meters apart from each other. When the robot was oriented towards the visitors, its front was directed towards a single visitor or towards the middle of the group of visitors. See Table 4.2 for a specification of the orientation of the robot per iteration and per point of interest. 83

100 Single Modality Studies CHAPTER 4 Figure 4.3: The robot and visitors in the site Figure 4.4: Layout of the tour Table 4.2: Specification of actions per Point of Interest (POI) Robot actions POI 1 POI 2 POI 3 POI 4 Iteration 1 To exhibit To people Excluded Iteration 2 To exhibit To people Excluded Iteration 3 To exhibit To people Excluded Total To exhibit To people Excluded In between the three iterations, some changes were made to the explanation by the robot. The explanations for the robot were developed in such a way that they could be used for both orientations of the robot. During the first iteration we observed that these explanations worked fine when the robot was oriented towards the points of interest. However, we found that it seemed unclear where to look when the robot was oriented towards the visitors. Therefore, for the second iteration, the explanations of the robot when oriented towards 84

101 the visitors at points of interest two, three and four were modified. Information was added about where visitors had to look exactly to find the point of interest the robot explained about. As a result, the robot explained more clearly to the visitors to look behind the robot when it was orientated to the visitors and to look here when it was oriented towards the point of interest. Also, the sentences please follow me and don t be afraid were added to try to convince people to follow the robot to the next point of interest. In the third iteration another modification was made to the explanation of the robot when it was oriented towards the visitors. The sentences were ordered in such a way that the robot would capture the attention of the visitors with something trivial, so people would not miss important parts of the explanations. Each of the iterative sessions took about 1 hour and 40 minutes. Data collection During the study, the visitors were recorded with two cameras: a fixed camera that recorded the whole tour and a handheld camera that was used to record the facial expressions of the visitors close to the robot. Also, several visitors who followed (a part of) the tour were interviewed about their experiences. The interviews were sound recorded. The results of the interviews are not included in this chapter. For this study only the data collected with the fixed camera was used, because the data from this camera gave a good overview of the room and the actions, orientation and formations of the visitors. We decided not to use recordings from the cameras that were fixed on the robot, because their angle of view was limited to only the front of the robot. Using these recordings would not give us opportunities to study the behavior of visitors who were next to or behind the robot (for example when the robot was oriented towards the exhibit), which in this study would lead to the loss of a lot of information on visitor orientation and formations. The proximity of the visitors towards the robot was measured based on the number of tiles they stood away from the robot. A square pattern of tiles was 0.83*0.83 m, the small tiles were 0.28*0.28 m and the bigger tiles were 0.28*0.56 m. Consequently, by counting the square patterns and individual tiles an estimation could be made about the distance visitors kept from the robot. 4 Data analysis For the analysis, 236 out of a total of 278 robot actions were used. 42 cases were excluded from analysis because no visitors were in the room or no robot was visible, because it was out of the angle of view of the camera, or the view was blocked by large numbers of visitors (for example a group with a human tour guide that did not show any interest in the robot). This resulted in 236 robot actions in 3 iterations that were left for the analysis. The robot 85

102 CHAPTER 4 Single Modality Studies was oriented towards the exhibit while it explained 127 times, and the robot was oriented towards the visitors while it presented 109 times. We were interested in the responses of the visitors that might be influenced by the robot orientation during each of these 236 complete explanations at the points of interest. However, exact visitor behavior to search for was not defined before the study. We therefore performed a content analysis of the recordings from the fixed camera. We isolated robot actions -the moments that the robot stood close to a point of interest and presented about it- in the data for coding purposes. Coding of the data was done by using DREAM. We developed DREAM, Data Reduction Event Analysis Method, to annotate rich video data in a focused and fast manner and to anticipate for too early interpretations. DREAM is based on thin slices of behavior (Ambady & Rosenthal, 1992), the grounded theory method (Corbin & Strauss, 1990) and coding with multiple coders. For data analysis sequences of three stills from the video data were created. Each sequence showed the begin, the middle and the end of an explanation of the robot at a point of interest. Only the collection of sequences, and not the video data, were reviewed for analysis. The advantage of using DREAM over other video analysis methods is that only the moments of interaction are analyzed, which lead to a large reduction of the data. Also, not videos are watched over and over again, but sequences of three stills per action are observed. These sequences give a good impression of the changes in visitor position and formation during the chosen length of the interaction. As a result, the analysis of the video data becomes fast, but reliable, as we further explain in chapter 7. Following DREAM, no exact codes were defined before the start of the analysis. We defined the codes based on the actions of the visitors found in the video recordings. In total 19 different codes were defined. Some examples of codes are: standing very close to the robot and oriented towards each other, visitors standing in a semi-circle and robot oriented towards the exhibit, visitors losing interest during the robot story and robot oriented towards the visitors, visitors walking towards the robot and robot oriented towards exhibit. We used a count method to compare the responses of the visitors during the robot actions between the different robot orientations and the different points of interest. (See Appendix D for a full overview of the codes used for annotation). 10% of the data was double coded and we found an overall inter-rater reliability of Cohen s κ = 0.66 (Cohen s Kappa), which indicates a substantial agreement between the coders (Sim & Wright, 2005). Hence, one coder finished the coding of the dataset that was used for analysis. 86

103 Results We found that visitors stood further away more often when the robot was oriented towards the visitors (31 times, 24% of all cases in this condition) than when the robot was oriented towards the point of interest (17 times, 16% of all cases in this condition). Further, no differences were found in formations of the visitors between both conditions. However, when the robot was oriented towards the visitors, just 18 times (14% of all cases in this condition) visitors walked towards the robot, while when the robot was oriented towards the point of interest visitors walked towards the robot 25 times (23% of all cases in this condition). In both conditions and at all explanations at points of interest, a lot of people (78% of all cases) were just walking by, showing no attention for the robot at all. However, most of the time one or a few visitors had already joined the robot by then. A few times we observed that visitors waited until the robot was free again and then followed the tour. Also, when some of the visitors left the robot, others stayed to hear the rest of the explanation about the point of interest. We found more differences between visitor formations when we focused our analysis on the interactions in point of interest two, three and four, while excluding explanation in point of interest one. We decided to exclude point of interest one from our analysis, because at that point of interest the robot was always oriented towards the visitors and it was not explaining about a specific point of interest in the room. We found that when the robot provided information about points of interest two, three and four, more people lost interest when the robot was oriented towards the point of interest (22 times, 21% of all cases in this condition) than when the robot was oriented towards the visitors (8 times, 13 % of all cases in this condition). Also, 6 times (9.4 % of all cases in this condition) visitors did not have a clue where to look when the robot was oriented towards the visitors. This was never the case (0% of all cases in this condition) when the robot was oriented towards the point of interest. 4 The number of visitors standing at 30 cm or closer to the robot was comparable in both conditions (5 times, 4% of all cases with orientation towards the visitors and 6 times, 6% of all cases with orientation towards the exhibit). Nevertheless, a difference was observed in visitors action that varied based on the robot s orientation. Only at points of interest one and two, did visitors stand really close to the robot when the robot was oriented towards the visitors. However, in the condition where the robot was oriented towards the point of interest people stood close to the robot at all explanations at points of interest. From reviewing the video, we observed that when people stood very close to the robot and the robot was oriented towards them, visitors only seemed to focus on the robot, while visitors focused on the point of interest when the robot was oriented towards the point of interest. We also found differences in visitor responses between the different explanations at points 87

104 CHAPTER 4 Single Modality Studies of interest. The least visitors walked towards the robot at point of interest three (5 times; 9% of the cases in this condition), most did so at point of interest four (16 times, 30% of the cases in this condition). Visitors lost interest in the story and the robot most often at point of interest three (14 times; 26% of all cases in this condition) and least often in point of interest four (6 times; 11% of all cases in this condition). Looking only at the differences between the explanations at points of interest over both conditions, we found that many more single visitors and pairs joined the robot for at least one explanation at a point of interest (86 times, 36% of all cases) than that people gathered around the robot in any group formation (38 times, 16% of all cases). We found that during 11 robot actions (5% of all cases) visitors stood less than 30 cm away from the robot. During 48 robot actions (20% of all cases) people stood more than 3 meters away from the robot. In 131 robot actions (56% of all cases) visitors stood between the 30 cm and 3 meters from the robot. Note that these cases can overlap, because there could be more than one visitor at the same time. In the rest of the cases no visitors or no robot were in the field of view or the visitors did not join the robot tour Discussion Influences of robot orientation We found that visitors who stood far away from the robot more often when the robot was oriented towards the visitors than when it was oriented towards the point of interest. Furthermore, we found that visitors tended to walk towards the robot more often when the robot was oriented towards the point of interest than when the robot was oriented towards the visitors. One possible explanation for this visitor response might be that visitors could not hear the robot well enough. Another explanation could be that the visitors felt that a distance was created by this specific orientation of the robot. This may have caused that people felt safer to approach the robot when it was oriented towards the point of interest. Possibly, the robot kept people at a distance with its eyes when it was oriented towards the visitors. This finding is in line with findings from other studies that people walked closer to a robot that did not gaze at them rather than when the robot increased gaze at them (mutual gaze), as shown by Mumm and Mutlu (2011). Remarkable was that more people lost interest when the robot was oriented towards the point of interest than when the robot was oriented towards the visitors. As we argued before, the orientation of the robot towards the point of interest might have felt safer for people, at the same time, it might also have given them the feeling of being excluded, which made them leave the robot. At points of interest one and two, several people walked towards the robot, because the robot captured their attention and they were curious to see what it was for. Fewest visitors 88

105 walked towards the robot at point of interest three, most did at point of interest four. Visitors probably did not have to walk to the robot in point of interest three, as it was very close to point of interest two (approximately 2.3m). From point of interest three to point of interest four was the longest walk (approximately 11.8m). Visitors who walked towards the robot in point of interest four were probably a bit reserved following the robot and therefore just walked to the robot when it had already started the next explanation. Apart from that, point of interest three was close to an open door, the entrance to the next room, therefore people who lost interest could easily walk away from the robot into the next room. When visitors followed to point of interest four, the last point of interest of the tour, they were likely to follow the robot the whole tour. We assume these visitors liked to hear the explanations of the robot and stayed with the robot until the final explanation, therefore fewer of them left the robot in point of interest four. Visitor actions that were coded with losing interest showed that most of the time not all visitors lost their interest at the same moment. If one visitor of a pair or group walked away, the other(s) either followed the leaving person directly, stayed until the end of the explanation at that point of interest or stayed until the end of the tour. This indicates that visitors of pairs or groups gave each other the time to do what they liked and that they did not have to leave together at the same moment. An advantage was that for most people it was clear that the robot just gave a short tour, so the people who left did not have to wait for a long time if the others stayed. In some cases we observed pairs of visitors discussing if they would follow the robot and in the end they decided that one would follow the tour, and that the other would wait outside the research area. It was important for the robot that when one visitor lost interest, most of the time the robot had other visitors (either close or far) who were still interested in the robot and the story, so it went on with the story. 4 We found a difference in the distance people kept from the robot and the orientation of the robot. Only at points of interest one and two, did visitors stand really close to the robot when the robot was oriented towards the visitors. However, when the robot was oriented towards the point of interest, visitors stood very close in all four points of interest. It seemed that when visitors stood very close to the robot and the robot was oriented towards them, visitors only had interest in the robot as an object and they tried to make contact with the robot (by waving at the robot or bringing their eyes on the same height as the lenses of the camera of the robot). We observed this visitor behavior mainly occurred at points of interest one and two, which might be because at these moments the robot captured people s attention and joining a robot tour was still new to them. At point of interest three and four only visitors who were already following the tour seemed to be present and people who were only interested in the robot as an object did not disturb the robot guide and its visitors at these points. When visitors stood close and the robot was 89

106 CHAPTER 4 Single Modality Studies oriented towards the point of interest, the visitors probably could not hear the voice of the robot well enough to follow the story in the crowded area, while they were interested in the point of interest the robot presented about and wanted to hear the explanation. Visitors who were interacting with the robot oriented towards them in iteration 1 appeared to have no clue where to look at times. This indicates that visitors were sensitive to the orientation of the robot. Therefore, we added more verbal cues to the explanation of the robot in iterations 2 and 3. However, during these iterations, we still observed that when the robot was oriented towards them visitors got the clue where to look later than they expected. So, even though we changed the explanation of the robot to make more clear where to look and started with something trivial, just as human tour guides do (See chapter 3), visitors did not readily understand where to look. A reason for this might be the length of the explanations of the robot. These were much shorter than explanations given by a human tour guide at a point of interest usually are. So, in general visitors had less time to focus again before they would miss something. The robot orientation towards the point of interest avoided this problem. Visitor responses to the eyes of the robot Our observations showed that visitors were aware of the lenses of the camera on the robot and responded to them as if they were the eyes of the robot. This can for example be seen from the observation that some visitors waved at the camera when they arrived or when they left the robot. People also stood in front of the camera when they wanted to make contact with the robot. The observation that people are sensitive to the camera of a robot and orient in front of it was also made by Walters et al. (2005). These examples make clear that visitors respond to the orientation of the robot and probably see the lenses of the camera as the eyes of the robot. Another observation that strengthens these conclusions is that visitors most often lost their interest in point of interest three. In this point of interest the explanation was difficult to understand because the story was about a banner that hung high in the room, above an open door. When the robot was oriented towards the exhibit, it seemed as if it was looking at a point of interest in the other room because it was not able to tilt its orientation upwards. This confused the visitors, even when the robot was clear in its explanation about where to look. Differences between robot guide and human tour guide We found that visitors responded differently to the robot tour guide than we would have expected from observed responses to a human tour guide. First of all fewer groups and more individual visitors or pairs of visitors joined the robot tour guide. Also, visitors seemed 90

107 not inclined to join strangers, but rather waited till the tour was finished and they could join a new tour. Most visitors stood between 30 cm and 3 meters from the robot. When there were visitors standing very close or far away from the robot, there could also be visitors who stood at average distance (between 30 cm and 3 m) from the robot. While most visitors stood at an average distance, standing really close or staying at a distance differs from visitor behavior shown when they follow a human tour guide. Most of the time visitors of a group with a human tour guide do not show such large differences in proxemics to a guide and often stand in a semi-circle to give everyone a chance to see the guide (Best, 2012). Also, Walters et al. (2006) and Joosse, Sardar and Evers (2011) showed in controlled experiments that people allowed different approach distances and appropriate proxemics for a robot than they allow for confederates. This leads to the conclusion that we cannot assume that people respond the same to robot tour guides as to human tour guides. Implications of study set-up The study was performed in the wild which influenced the execution of the study and the manner of analysis. We used Wizard-of-Oz to control the robot in the real world environment, because at the moment of the study, interaction abilities of the robot were not that far developed yet to be able to put a trustworthy robot in front of the visitors. Moreover, due to the real world setting the guided tours could not be strictly controlled. However, the advantage of performing in the wild does compensate for this disadvantage. Furthermore, less information of the visitors could be obtained. For example, we could not have extended questionnaires because people would likely not want to spend their time filling these in. 4 A major advantage of the in-the-wild set-up of this study was that we observed the responses of the visitors the way they would probably be if an autonomous tour guide robot were to be installed in the Royal Alcázar, which is important in human-robot interaction according to Šabanović, Michalowski and Simmons (2006). The findings of this research were an important step for the development of FROG, because with in-the-lab studies with small groups of users, it would be difficult to create a similar environment including people who are acquaintances and strangers, as is also underlined by Shiomi, Kanda, Koizumi, Ishiguro and Hagita (2007). Probably we would not have found how people respond when the robot is already occupied by strangers, whereas in this set-up we did find interesting responses of visitors in the real-world context. Therefore, the in-the-wild environment suited this study best. We chose to perform the study in several iterations in which we modified the explanation of the robot. This led to the following differences between the iterations. In iteration one the robot was mainly oriented towards the point of interest. In iteration two the modification of 91

108 CHAPTER 4 Single Modality Studies the explanation seemed insufficient, so the robot was mainly oriented towards the points of interest. In iteration three the robot was mainly oriented towards the visitors. Without these modifications to the explanations, we would not have been able to perform the manipulation of the orientation of the robot, because with the original explanation visitors did not seem to know where to find the point of interest when the robot was oriented towards them. To analyze the video data we used DREAM (See chapter 7). The advantage of using this method is that it is fast compared to other video analysis methods. Also, it gave the opportunity to ground all found phenomena in the data. This was especially important for this study, as previously defined expectations might lead to overlooking unexpected situations that are nevertheless important for a tour guide robot like FROG General discussion Already in the overview of the related work in this chapter, I have shown that similar behaviors transferred to different robots can lead to different effects on the human-robot interaction. This is because the behavior design for robots is multimodal. For example, when gaze behavior is designed and some small movements to add liveliness are also applied to the robot, this behavior can influence the participants responses to the gaze behavior that is controlled. This is also what I observed in both single modality studies I performed with the low-anthropomorphic robots. In the first study I focused on robot gaze behavior, but the participants were distracted by the body movement of the robot. Whereas, in the second study, I controlled the orientation behavior, but the participants interpreted the gaze behavior of the robot. Thus even when single modalities for robots are studied, other behaviors might influence the effect of the controlled behavior. Nevertheless, really influencing one behavior (e.g. gaze) in a robot should be possible. A robot only performs what has been programmed and therefore one behavior can be used and all other behaviors can be left out. However, this might lead to an unrealistic simulation. Firstly, because human behavior is multimodal and has redundancies. And secondly, because leaving out other behavior does not lead to a complete view of reality. Therefore, I argue to develop robot behavior as multimodal, even if this does make it difficult to find what exactly causes some effects. Furthermore, I found that it is not possible to just copy humanlike behavior to lowanthropomorphic robots and expect the same visitor responses as visitors would show to human tour guides. In the controlled lab study I found that the participants were distracted by the full body movement of the robot when it turned its body where a human tour guide would only turn their head. Another head turning behavior was not possible with 92

109 the chosen appearance of the robot. Even though the robot showed more liveliness and mimicked as closely as possible the humanlike behavior, the condition that was preferred by the participants, it negatively influenced the human-robot interaction. Also, in the inthe-wild study I found that the orientation behavior of visitors was not comparable when they joined a tour guide robot or a human tour guide. Therefore, I argue that, for lowanthropomorphic tour guide robots, behaviors that fit the appearance of the robot should be used instead of a copy of humanlike behavior. Moreover, in the in-the-wild study in the Royal Alcázar, a stereo bumblebee camera and a Kinect were clearly visible on the robot. My experience in this study provided the insight that visitors see the stereo camera on top of the robot as the eyes of the robot. Therefore, when the camera cannot be hidden, the camera should be designed as eyes, including the design of gaze cues and gaze direction. Using these cues, especially when people expect them already, will probably smoothen the human-robot interaction. In our case, the FROG robot was not a humanoid robot, as the camera was visible. Therefore, I argued that a visible camera should be used as eyes of the robot, because this would support the mental model users will create of the robot. Overall, these studies provide insights that help answer research question 1 about to what extent humanlike behaviors effective when transferred to low-anthropomorphic robots. The most striking finding is that humanlike behavior might transfer quite well to humanlike robots, but may not be effectively copied to low-anthropomorphic robots to create humanrobot interaction that is similar to human-human interaction. First of all, because the use (or non-use) of multiple modalities influences the effect of a behavior performed with a modality, as for example overall liveliness of a robot influences the effectiveness of robot gaze behavior on the recollection of details of a story. Second, it might be difficult to use humanlike behavior for low-anthropomorphic robots because these robots often lack the right modalities to perform certain behaviors. In the current studies for instance the robots were not able to swivel heads, while this might have been more useful when designing gaze behavior. Thus, multimodal behavior for low-anthropomorphic robots probably should not be a copy of humanlike behaviors. 4 93

110

111 5 Beyond R2D2; Designing nonverbal interaction behaviors for robot-specific morphologies This chapter will be about defining approaches to develop effective behavior for low-anthropomorphic robots. First I will motivate why imitating humanlike behavior might not the best manner to develop behavior for low-anthropomorphic robots. Second, current approaches to design robot behavior will be discussed. Third, the two design approaches that were used to design the two behavior sets that we compare in this chapter will be described. Next, I will present the methods that were used the studies that were preformed to evaluate the different sets of robot behavior. The results of both studies will presented and discussed. Moreover, the approaches to design robot behavior and the use of mixed methods to evaluate these behaviors will discussed. Finally, I will present conclusions and directions for future work. 5 This chapter is largely based on: Karreman, D. E., Ludden, G. D. S., Evers, V. Beyond R2D2: Designing multimodal interaction behavior for robot specific morphology (in review). 95

112 CHAPTER 5 Multimodality Studies 5.1. Motivation In the introduction of this thesis, I already emphasized the importance of nonverbal behavior for social robots. So far, a large body of work in human-robot interaction has been focused on designing behavior for humanlike social robots (humanoids), however, few researchers have studied the design of behavior for low-anthropomorphic social robots. As a result, little knowledge is available on how to successfully design multimodal behaviors for social robots that lack humanlike modalities. Therefore, in this chapter, I will set out to report and evaluate an alternative approach to develop behavior for low-anthropomorphic social robots Related work; Designing nonverbal interaction behaviors for low-anthropomorphic robots Humanoid robots are often envisioned to perform tasks in everyday social situations. These humanoids have a similar basic physical structure and kinetic capabilities to those that people have, and thus are physically able to behave as people would do. For example, these robots are able to walk, use their arms and hands to manipulate things and are able to swivel their heads. Therefore, these robots seem to be suitable to perform social tasks in everyday environments such as schools, shopping malls and museums. Examples of such robots are: Nao (e.g. used in studies by (Gehle, Pitsch, & Wrede, 2014; Lohse, Rothuis, Gallego Peréz, Karreman, & Evers, 2014)) or ASIMO (e.g. used in studies by (Mutlu, Forlizzi, & Hodgins, 2006; Salem, Eyssel, Rohlfing, Kopp, & Joublin, 2013)). A common strategy to create the behavior for these humanlike robots is to copy humanlike behavior. There are several good reasons to do so; people are already familiar with these behaviors (Breazeal, 2003), people can read this behavior naturally (Okuno, Kanda, Imai, Ishiguro, & Hagita, 2009), people seem to expect it from humanlike robots (Salem, Rohlfing, Kopp, & Joublin, 2011), and this way of developing behavior works very well for humanlike robots (e.g. (Kuno et al., 2007; Kuzuoka, Suzuki, Yamashita, & Yamazaki, 2010)). However, humanoid appearances and behaviors for robots might also lead to disadvantages. For instance, being highly humanlike in appearance makes robots more complex, expensive and vulnerable. Moreover, anthropomorphic appearance and behavior can also lead to disadvantages in social interactions between robots and people. For example, an extremely humanlike appearance can in some cases lead to a frightening robot, specifically when it starts moving (Mori, 1970). Another disadvantage can be that a humanlike appearance can raise high expectations of the abilities of the robot (DiSalvo, Gemperle, Forlizzi, & Kiesler, 2002; Duffy, 2003). This can lead to disappointment when the robots are not as skillful and advanced as people expect them to be (Choi & Kim, 2009; Fink, 2012). 96

113 Furthermore, the technical state of the art or functional design requirements can in some cases limit the possibilities of creating highly anthropomorphic behavior and appearance. For example, in the design of the Snackbot the functional requirements were reflected in the final design of the appearance of the robot (Lee et al., 2009). In this design, the width of the head was dictated by the technical components, in this case the Bumblebee stereo camera that was used. Also, the robot did not walk, but it was based on an existing platform to drive around, and had bumpers for safety reasons. As a result, choices for technical components partly define the appearance of the robot, which in this case led to a less humanlike design. Another option for service robots for social environments would be to deliberately create robots that do not resemble people closely in appearance or behavior. Creating a robot that does not look like a human does not necessarily limit the capabilities of this robot to interact with people. Several sources (e.g. (DiSalvo et al., 2002; Duffy, 2003)) state that anthropomorphism is not the only solution to create effective social robots. According to DiSalvo et al. (2002), a social robot should leverage robot-ness and product-ness, as well as humanlike-ness, to avoid false expectations of the machine s capabilities and to make people feel comfortable using the product, while the interaction is socially engaging. The robotness, product-ness and humanlike-ness need to be balanced and should be recognizable in appearance and behavior of the robot. Even though a non-humanlike appearance might be an advantage for robots, there is not yet a distinct approach to design behavior for such robots. Currently, the approach to develop behavior for humanoids is also used to develop behavior for low-anthropomorphic robots. However, we question whether it is indeed logical to use this human-behavior-translatedto-robot strategy for all kinds of social robots, specifically for robots that lack humanlike modalities. 5 As our focus was on a tour guide robot, we evaluated the behavior of previously developed tour guide robots that lacked several humanlike modalities. Based on several studies on tour guide robots, we deduced that most tour guide robot behavior is based on how human tour guides would behave in a certain situation. Examples we found of tour guide robots were: the robots Rhino and Minerva which were both equipped with interfaces that resemble people in behavior to anticipate the diverse modes of interactions in public spaces (Thrun et al., 1999), Rackham which shows a humanlike face on a screen to ease the human-robot interaction (Clodic, Fleury, Alami, Herrb, & Chatila, 2005), three entertainment robots that were given humanlike roles to make their functions recognizable (Graf & Barth, 2002), and a simple guide robot that only had limited movement capabilities to perform humanlike behavior to explain the content of two artworks (Karreman, Sepúlveda Bradford, van Dijk, Lohse, & Evers, 2013). From these studies, we observed two main limitations that occur when humanlike behavior 97

114 CHAPTER 5 Multimodality Studies is transferred to robots that do not look like people. First, oftentimes if robots still have some humanlike modalities, there are limitations in the movements these can perform. This is because the robot modalities most likely do not have similar degrees of freedom to human modalities. Second, the robot morphology may limit the extent to which behaviors can be exactly copied to the robot. This is because these robots usually lack humanlike modalities such as arms or dynamic necks (Karreman et al., 2013). Thus, humanlike behavior is usually not suitable to apply to low-anthropomorphic robots. An alternative approach to design behavior for robots that do not look like people might solve these problems and could even lead to more effective and/or understandable robot behavior. The design of appearance, interaction and behavior of social robots finds many parallels in the design of smart interactive products. The last domain is increasingly studied in the fields of product design and interaction design. Similar to what social robots do, some smart products interact with people in a social and personalized manner. However, to design appearance and behavior of smart products, industrial designers generally do not rely on copying human behavior or appearance. Instead, they often take a desired user experience as the starting point for their design. Several researchers from the fields of interaction design and product design provide design guidelines on how to implement user experience in the design of products, which may find their use in the field of human-robot interaction. For example, Forlizzi and Ford (2000) state that to design a successful product, the focus should be on the user, the product and the context of use. Similarly, Hummels (1999) states that not only a product, but a context for experience should be designed. She states that interaction does not always have to be as simple as possible (ease of use), but that the interaction with the product should be engaging and that the product should encourage users to explore the context with all senses to gain rich experiences (e.g. pressing the play button on a cd-player is less engaging than cleaning an LP and putting the stylus in the groove to play the music). In line with these arguments, Hekkert, Mostert and Stompff (2003) used the user experience of a dance as the starting point to design the interaction with a photo copier to stimulate several senses in a surprising way. Furthermore, Desmet, Ortíz Nicolás and Schoormans (2008) successfully created intended interaction experiences, dominant vs. elegant, for two similarly looking devices developed for their study. These experiences were not visible in the device, they were only perceived through interaction style, indicating that it is possible to design interaction devices with different product personalities. In these examples, the designers did not start from a copy of humanlike behavior for their product. Instead, they focused on the desired experiences to influence and fine-tune the designed interaction. Therefore, based on the approaches used in interaction and product design, we expect that taking a desired user experience and a robot s functional goals 98

115 as starting points, might be a promising alternative to copying human appearances and behavior in the design of socially interactive robots Research question The context chosen for this study was a tour guide setting. This is an interesting context for research in human-robot interaction, because it provides a subtle combination between verbal and nonverbal communication and usually is short-term in duration. Furthermore, so far guiding tasks have mainly been performed by people. However, we think that guiding people is a task that can be well performed by robots. Until now, behavior for tour guide robots had mainly been a (poor) copy of humanlike behavior, as it was the best and only example existing. As a result, it seemed logical to copy humanlike behavior for tour guide robots, however we know little about the effects of specific robot behavior in this context. Notably, in this context, not only people s understanding of what the robot explains but also the user experience of the tour is of importance. We hypothesized that visitors in general would be more positive towards a lowanthropomorphic robot with multimodal behavior that had been specifically designed for the modalities of the robot than towards a robot with humanlike behavior. However, only part of the success of a tour guide robot is based on the most effective behavior to convey information; the user experience of interacting with that kind of behavior will also account for the success. We expected that human-translated behavior would be appreciated for highly anthropomorphic robots, but that this behavior might be perceived uncanny or not understandable when copied to low-anthropomorphic robots. Therefore, we assume that robot-optimized behavior that is specifically designed for the morphology of the robot will be perceived as a better fit for these robots. 5 Based on the expectations described above, our main research question is: Do people respond more positively to robot-optimized behavior compared with human-translated behavior? Because this research question is general, we focus on participant s attention and participant s attitudes towards the robot s behavior to measure its effectiveness. To test our question, we created two hypotheses. H1: Robot behavior optimized for the modalities of a low-anthropomorphic tour guide robot will lead to longer periods of attention and higher recall of the information provided by the robot. H2: Robot behavior optimized for the modalities of a low-anthropomorphic tour guide robot will lead to a more positive attitude towards the robot. 99

116 CHAPTER 5 Multimodality Studies 5.4. Two approaches towards the design of multimodal robot behavior To test the assumption outlined above, we developed two sets of behaviors for the FROG robot following two different design approaches. We state that copying humanlike behavior to a low-anthropomorphic robot limits the options for behavior design, while our newly proposed approach optimizes the use of the robot modalities. The robot we used in our studies was the FROG robot (see Figure 5.1). FROG was developed within the EU FP7 project FROG, which stands for Fun Robotic Outdoor Guide. FROG was developed as a tour guide robot to guide small groups of visitors and engage and educate them at tourist sites. The robot does not look like a person, but it does have some humanlike characteristics. Therefore, this robot was deemed suitable for the studies we wanted to perform. In Figure 5.2 we present how our newly proposed approach relates to the conventional approach to design multimodal behavior for socially interactive robots. As can be seen from Figure 5.2, both design approaches start from the observed human tour guide behavior. This is especially valuable when the robot will perform social interaction tasks that previously were only performed by people, as is the case for guiding. The right side of Figure 5.2 shows how the robot morphology may limit the opportunities for behavior when humanlike behavior is translated to the robot. This is due to the fact that the robot might not have the modalities or technical features to perform behavior exactly the way that people do. This approach we call human-translated. On the left side, we present the robot-optimized approach. This shows how analysis of the effects of human behavior lead to interactional outcomes that can be translated to behavior that is optimized for the modalities of the robot. This approach we call robot-optimized. Figure 5.1: FROG robot giving short tours 100

117 Figure 5.2: Schematic view of the two different design approaches, robot-optimized behavior and human-translated behavior; both stem from observed human behavior Approach 1: human-translated behavior The human-translated approach was to translate human behavior cues as similarly as possible to the robot. As explained previously, this is an approach that is commonly used in the field of Human-Robot Interaction to design social robot behavior. 5 The resulting set of behavior cues mimicked the observed behaviors of human tour guides studied in (Karreman, van Dijk, & Evers, 2012) as much as possible (see Table 5.1 for an impression). The actions of the robot were timed at similar moments as the actions of human tour guides would happen. For example, when human guides want to go to the next exhibit, they were observed to break eye-contact with the group of visitors before they started to move to the next exhibit. We translated this behavior as closely as possible to the robot: When a robot finished providing information about an exhibit, the animation in the eyes of the robot looked downwards, indicating a break of eye-contact before it moved to another exhibit. 101

118 CHAPTER 5 Multimodality Studies Approach 2: robot-optimized behavior As well as this approach, we proposed an alternative robot-optimized approach to design behavior for robots based on principles that are common in product design and interaction design. Starting points for this approach are the robot functional specifications and the desired user experience of the human-robot interaction. To design this set of behaviors, we focused on interactional outcomes and we designed behaviors that fitted the modalities of the robot. Interactional outcomes are the effects of multimodal actions that human tour guides aim for with specific guide behaviors during their tour. For example, focus visitor attention on the exhibit is the interactional outcome of a guide s action gaze towards the exhibit while pointing towards the exhibit. See study 1 in chapter 3 for details on guide behaviors and the accompanying interactional outcomes. Moreover, we have studied the visitor experiences of following a human guided tour. By designing behavior for the robot tour guide we emphasized the parts of behavior that evoked positive experiences and limited the negative experiences. To do so, we selected the interactional outcomes to design multimodal behavior for based on expected positive visitor experiences. For example, visitors did not like too long explanations, thus the explanations would be kept short. For details on visitor experiences in tourist sites, see study 2, chapter 3. Here, we will mainly focus on the functional specifications of the robot as starting point for the design of the robot s behavior. To explore different options for the robot-optimized behavior set, we used a morphological chart (Jones, 1992). This is a method taken from Industrial Design to systematically come to design options. By using this method, several ideas on how to use a modality to communicate the intention of the robot (interactional outcome) were generated. In the morphological chart (see Figure 5.3), for each interactional outcome options for ways to use each modality of the robot were mapped out. In the next step, promising ideas on how to use the different modalities were combined in one set of behaviors for the robot. Examples of robot-optimized behavior are: the robot scans for visitors with its pointer, instead of making a full body movement; the robot uses two different types of animations for the eyes, one attention-attracting for the drive-mode and a more subtle one for the explain-mode; and the screen of the robot was not used to show a face, but rather to show additional information, instructions or intentions Methods for evaluation To check whether participants responded differently towards the sets of behaviors, we prepared an online video study (study 5) and an in-the-wild interaction study (study 6). The online video study allowed us to collect a more controlled response to the two sets 102

119 Figure 5.3: Example of morphological chart of behavior of the robot and the in-the-wild observations were less controlled by nature but offer richer insights into peoples experiences when encountering a service robot in a semi-public place, as we will explain below. Furthermore Ju and Takayama (2009) suggested that the use of different experimental methods can address or mitigate potential limitations inherent in each method. Video studies in human-robot interaction research are an efficient way to collect data on visitors perception of the robot, understandability of the robot behavior sets and visitors attitudes towards the robot (Evers, Brodecki, & Hinds, 2008; Lohse et al., 2007; MacDorman, 2006; S. N. Woods, Walters, Koay, & Dautenhahn, 2006). A video of isolated tour guide behaviors was not expected to provide an experience similar to experiencing an actual robot in an actual cultural site. However, by showing videos of the different sets of robot behaviors we were able to find whether and which differences participants observed between the sets of robot behavior and how participants perceived and evaluated the different sets of behaviors in terms of (product) characteristics. 5 In-the-wild studies in human-robot interaction elicit valuable insights into how people will interact with robots in unstructured settings (Šabanović, Michalowski, & Simmons, 2006). In our study, we observed reactions from visitors that encountered and joined the robot tour guide. Furthermore, some of these visitors were randomly invited to participate in a short interview, in which they were asked for their experiences. As a result, the in-the-wild study gave us the opportunity to find out how spontaneous visitors reacted to the robot and how they experienced a tour of FROG. 103

120 CHAPTER 5 Multimodality Studies Manipulation of the robot s behavior in both studies We developed human-translated behavior and robot-optimized behavior for the FROG robot. To control for the number of different behaviors, we developed the two sets of behaviors for the same interactional outcomes. In Table 5.1 we present two examples of interactional outcomes, how human tour guides perform them, and how we translated and optimized them for the robot. Note that these images only give an impression of the differences in the behavior sets, because the behaviors of the guide and the robot are very dynamic and difficult to grasp in stills. Table 5.1 presents the behaviors using images from the video study only, the behaviors in the in-the-wild study were similar. In this section the methodology of each study will be reported separately. Study 5 and study 6 are highly related and are mostly different in methodological approach. Therefore, in the remainder of this chapter, we will first report the methodology of study 5, followed by the methodology of study 6. Then the results of study 5 and study 6 will be jointly reported in the results section, followed by a discussion and conclusion Method of the online video study (study 5) The video study had a between-subjects design with robot behavior design as the independent variable. For this purpose, we developed two videos, one with the humantranslated robot behavior, and the other with the robot-optimized behavior set. The robot The robot that we used as a tour guide in both videos was a small 3D model (approximately 30 cm in height) of the actual robot developed in the EU FP7 project FROG. We made this model to closely resemble the design of the robot used in the project since we specifically wanted to understand the effect of behavior rather than that of appearance. The robot was made of cardboard, foam, colored paper and plastic sheets. This model allowed us to use the following modalities of the robot simulate different behaviors in the videos: animated eyes, pointer on top, visuals on screen, whole body movement and driving. The robot itself would not talk, but in the video a voice-over was used for the explanations. The voice-over was a computer voice (female) with speech generated by a text to speech engine (NaturalReader with US Crystal voice). See Table 5.2 for an overview of the differences between the human-translated and the robot-optimized behavior used for the robot in the online video study. Participants A total of 204 participants (132 males, 58 females, 14 preferred not to disclose, aged 15-58, mean age 31.6 years) evaluated the stimuli. All participants were recruited through 104

121 Table 5.1: Overview of differences between human-translated and robot-optimized behavior sets Interactional outcome Human behavior cues Human-translated (Direct copy of human tour guide behavior) Robot-optimized (Optimal use of modalities to achieve interactional outcomes) Keep visitors engaged and get implicit feedback Gaze at one person and change to the others sometimes Stand close to exhibit, but not in front of it (in front of pillar etc.) Orientation towards visitors Keep gazing at one person, and by turning on the spot alternate to others (not able to swivel head only) Stand close to exhibit but not in front of it Orientation towards visitors Slightly turn towards exhibit Show supporting terms/visuals on screen Scan visitors every once in a while with pointer Stand close to exhibit but not in front of it Orientation towards visitors Direct attention to the exhibit Point into exhibit, supporting while saying there etc. Gaze to exhibit when pointing Point into exhibit Gaze into exhibit Point into exhibit with arrow and pointer Finish at exhibit Break eye-contact Animation of eyes look downwards Eyes start pulsing Look in the direction of next exhibit Turn slightly on the spot, as if looking at all visitors, then find small gap in between visitors to drive through On screen please follow me Point towards next exhibit Start moving in direction of next exhibit Point towards next exhibit with pointer Start moving in direction of next exhibit Pointer turns from visitors towards next direction Start moving in direction of next exhibit Keep pointing to next exhibit On screen please follow me

122 Main characteristics of human-translated behaviors set Table 5.2: Overview of differences between human-translated and robot-optimized behavior sets The robot shows a smiling mouth on the screen and looks towards the visitors. When the robot points at the painting, it uses its pointer, turns in that directon and the eyes indicate that they look to the painting. Sometimes the robot shows a picture on the screen, that it shows to all visitors by turning slightly on the spot. When the robot drives to the next point, the pointer is in rest, and the screen shows a smiling mouth. Main characteristics of robot-optimized behaviors set The robot is always oriented towards one point (the middle of a group of visitors) and scans for visitors with its pointer. The robot points by using its pointer and an arrow on the screen. It keeps its original orientation (does not turn to all visitors). The robot often shows details of objects and other pictures on the screen. It keeps its original orientation (does not turn to all visitors). While driving, the eyes of the robot are animated in another way than during the explanations. Also, the pointer points at the next exhibit.

123 Crowdflower, a crowdsourcing site, which allowed and stimulated the involvement of a wide range of participants. The participants came from different continents (most from Europe 42% and Asia 31%). Of the participants 69% had no previous experience with social robots, 16% had little previous experience with social robots, 7% had much previous experience with social robots, 1% answered other and 7% did not answer the question. Of all participants, 146 participants (72%) rated their level of English good to very good, 40 participants (20%) rated their level of English as average, and only 5 participants (2%) rated their level of English as bad to very bad and 6% of the participants did not answer the question. Measures In line with our research question and hypotheses, we prepared a questionnaire that focused on participants attitudes towards the robot behavior and a part that focused on participants attention. Measures that focused on participants experiences included constructs of the Godspeed questionnaire (Christoph Bartneck, Kulić, Croft, & Zoghbi, 2008) and the Source Credibility Scale (Rubin, Palmgreen, & Sypher, 2004) as well as four items that were added to mask the intention of the questionnaire (obviousness, novelty, being qualified and reliability of the robot). The constructs of the Godspeed that were used were Anthropomorphism, Likeability, Animacy and Perceived Intelligence. The constructs of the Source Credibility Scale that we used were Sociability, Extraversion, Competence and Character. Three items ( artificial-lifelike, unfriendly-friendly and unintelligent-intelligent ) were asked once in the questionnaire, but used in two different constructs during analysis, because they appeared in constructs of the Godspeed and the Source Credibility Scale. Furthermore, the participants were asked to rate the behavior of the robot on the product personality scale (Mugge, Govers, & Schoormans, 2009). 5 Questions that focused on participants attention included nine questions that evaluated whether participants had understood and/or were distracted by the behavior of the robot and four questions to measure the recollection of details. We used information related to two paintings which were the Mona Lisa by da Vinci and the Girl with the Pearl Earring by Vermeer. These paintings were chosen because many people are familiar with them but would be interested to learn details about them that are not widely known. This way, the information provided would be new and interesting. Also, it allowed us to test for recall of details afterwards. The actual recollection of details was measured by presenting the participants with four multiple choice questions to test whether people had remembered the details of the story that the robot told (4 options, of which one was always I can t remember ). The story and questions were based on the story used for the study presented in chapter 4, study 1. Participants who observed both videos were asked to answer four questions. One was a question on preference for one of both robot behavior sets, the other three were open 107

124 CHAPTER 5 Multimodality Studies questions examining why participants preferred a specific set of behavior and what differences they observed between the behaviors. Last, all participants were asked some demographical details, such as age, gender, education level, profession, and experience with social robots. (See Appendix E for the full questionnaire as presented to the participants). Procedure Participants were asked to fill out a questionnaire about the behavior of a robot that gave a tour in a small art gallery. They were informed that the questions would be about the behavior of the robot and the information the robot gave. Two short stop-motion videos (1:37 minutes and 1:39 minutes) were prepared; one for the human-translated condition and one for the robot-optimized condition. Both videos showed the robot presenting two artworks in a museum-like setting. For both videos, the story line was kept the same and the same sound file (a voice offering explanations about the paintings in English) was used. Only the multimodal behavior sets were different. Furthermore, the videos were controlled for robot activity to ensure that the robot was not more active in one of both conditions, as this seemed to be the cause of the differences found in a previous study (Karreman et al., 2013). The videos and online survey were distributed through a link on Crowdflower that led to Surveymonkey. On the first page of the questionnaire, the participants were asked for their consent. After that, the online questionnaire started with a video of either the robot in the human-translated condition or of the robot in the robot-optimized condition. The participants were randomly assigned to one of the two conditions. After seeing the video, the participants were asked to evaluate the behavior of the robot and how they experienced the behavior of the robot. The items in the questionnaire were presented on 5-point Likert scales or 5-point semantic differentials, unless stated otherwise. The items of the constructs and other questions in the questionnaire were randomized per page, unless order of the questions was important. After the participants had finished the evaluation, they could decide to see the video of the other condition as well. Participants who decided to see the second video were asked to indicate which of the two sets of robot behaviors they preferred, and they answered three open questions about why they preferred a specific behavior, what the main differences between the robot s behaviors in the two videos were and whether they had any suggestions for the behavior of the robot. They were not asked to answer all previously answered questions again, but they were asked to answer 4 open questions. After seeing the second video and responding to the 4 items, participants were directed to the last page. When 108

125 participants chose not to see the other video, they were directed straight to the last page. On the last page of the questionnaire, all participants were asked demographical information. The questionnaire contained 83 questions in total (participants had to fill in 79 questions when they chose to not see the other condition). After participants completed the questionnaire they received a code to obtain a small payment for their participation. Data preparation and analysis Rather than performing an outlier analysis, participants were removed when their responses seemed to be fake. By doing so, 29 participants who filled out the questionnaire in less than 7 minutes, as completion took minutes on average; by doing so the fake responses (people that gave the same answers for every item) were removed. Moreover, 18 participants who did not give consent or who did not start the questionnaire after giving consent were removed,. After preparing the data, a total of 204 participants were left (from the original 251 who started the questionnaire). The participants were originally evenly divided over the two conditions. However, some participants were removed from analysis, because they either did not complete the questionnaire or they responded with the same answer to all items, for example they always chose 5. Hence, due to the fact that 47 participants were removed from analysis, we had 114 cases for the human-translated condition, and 90 cases for the robotoptimized condition. Out of the remaining 204 participants, 166 (81% of all participants) compared both sets of robot behaviors. Data was analyzed with SPSS. All scales were first tested for reliability, using the Cronbach s α coefficient. As the data was not normally distributed, we performed Mann-Whitney tests. 5 Reliability of measures We used Cronbach s Alpha to check the reliability of our scales. A Cronbach s Alpha above 0.7 is considered reliable (Nunnally & Bernstein, 1967). Therefore, we were able to use the combined scales for the constructs Likeability, Perceived Intelligence, Sociability and Understandability of robot behavior in the analyses. See Table 5.3 for an overview of the found Cronbach s Alphas for each of the constructs. Furthermore, we analyzed results for all individual items of the questionnaire. Table 5.3: Cronbach s Alpha for constructs Measurement # items Found Alpha α Godspeed Anthropomorphism Godspeed Animacy Mechanical - Organic was left out to increase the Cronbach s Alpha. 109

126 CHAPTER 5 Multimodality Studies Measurement # items Found Alpha α (Continued) Godspeed Likeability Godspeed Perceived Intelligence Source Credibility Scale Sociability Source Credibility Scale Extraversion Source Credibility Scale Competence Source Credibility Scale Character Understandability of robot behavior Manipulation check As a manipulation check for differences between both behavior sets, one item asked the subset of participants who observed both sets of robot behavior, whether they found any differences between both robot behaviors and if so, which. After seeing both videos, 95 participants (58% of the participants who saw both videos) stated they saw differences in the robot behavior, 34 participants (21%) stated that they did not see any differences, and 35 (21%) gave an answer unrelated to the question. The condition, age or nationality was not found to influence these results. Therefore, we concluded that the manipulation was merely successful. In the following section we describe the method of the in-the-wild study, before we report on the combined results of both studies Method of the in-the-wild study (study 6) In the in-the-wild study the robot gave short tours to spontaneous visitors in the Royal Alcázar in Seville, Spain. The robot The robot used for the in-the-wild study was the FROG platform, see Figure 5.1. The different sets of behaviors were programmed on this platform. To drive around, FROG was controlled remotely. The controller of the robot was the same for all sessions performed during the study. FROG had enough battery power to give tours for approximately one and a half hours. Afterwards the robot needed to recharge for several hours. The robot was designed to attract and keep the attention of the visitors using its body movements, the eyes (sequences played with the LED-lights in the eyes), the pointer on top of the robot (pointing at things and sequences played with the LED-lights in the pointer), a touchscreen that was used to show pictures and movies, and prerecorded synthesized speech. In this experiment the visitors were not asked to use the touchscreen. Furthermore, the robot would not understand the visitors if they should talk to it. See Table 5.4 for an 110

127 Table 5.4: Overview of the differences in one explanation at one point of interest of the tour between human-translated and robot-optimized behavior in the wild Example of one explanation at one point of interest using the human-translated behavior set The robot turned its whole body slightly towards the different visitors while giving information. On the screen a mouth was visible and one explaining picture was shown for a few seconds. When the robot pointed at the exhibit, the pointer and eyes pointed there and the robot turned its whole body slightly in that direction. Example of one explanation at one point of interest using the robot-optimized behavior set The robot stood in a position and used the pointer to search for people. Before the explanation started, the screen showed come closer. During the explanation the screen was mainly black, only one explaining picture was shown for a few seconds. When the robot pointed at the exhibit, the pointer pointed in that direction, but the orientation of the robot did not change. After the explanation, the screen showed follow me. overview of the differences between the human-translated and the robot-optimized behavior used for the robot in the in-the-wild study. 5 Participants All visitors of the Royal Alcázar were potential participants for the in-the-wild study, because they were all free to join or to ignore the robot. Based on chance, visitors participated in the study. We estimated that in each condition about 300 participants joined the robot at some point during the tours (approximately 15 tours in human-translated condition, and approximately 14 tours in robot-optimized condition). We did this by counting participants who joined the FROG tour and interacted with the robot. Interactions with the robot could range from joining the robot for the explanation of one point of interest to joining the robot for an entire tour of explanations at five points of interest or longer. Only visitors who joined the tour for at least one explanation were counted in this number. People who stood close to the robot, but also people who followed the tour from a distance were counted. 111

128 CHAPTER 5 Multimodality Studies Procedure One short tour was prepared for the study, while two different sets of behaviors were prepared. In the first condition the robot communicated its intentions with humantranslated behaviors, in which the observed human behaviors were mimicked as closely as possible. In the second condition the robot showed robot-optimized behavior, in which the behavior was optimized for the modalities of the robot. The sets of behaviors were largely similar to the sets of behaviors used in the online video study. The only difference was that they were used in a different cultural heritage setting and thus the multimodal behaviors were performed at other moments in time than in the video study. The two sets of behaviors were controlled for activity of the robot to ensure that the robot was not more active in one of both conditions. The tour consisted of 5 explanations at 5 different points of interest. At all points of interest the explanations took about 20 seconds. The total duration of the tour was 3-5 minutes. Deviations in duration of the tours occurred because the robot had to wait for visitors to step aside or to navigate around visitors present in the room to reach the next point of interest. The images in Table 5.4 show the same explanation at a point of interest in the tour, but in different conditions. Here is visible that in the human-translated condition the robot moves its body to show attention towards the visitors, while it uses its pointer in the robotoptimized condition. Due to chosen camera positions the visitor reactions are clearly visible, while the differences in robot eye and screen behavior are less well observable. Before the robot started the guided tours, the researchers set up the camera and placed signs to give information about the study and the consent. The robot was brought into the room at three time-slots on two different days. A clearly visible camera was used to record the visitor s reactions. The camera was placed at a height of 2 meters. When visitors entered the room during the performance of the study, signs clarified that an experiment was going on. Visitors who joined the tour of the robot were not further informed about the study before the tour started. If people decided to enter anyway, they automatically gave their consent for the use of the obtained video data for academic research. It was stated that this included use of the material for publications. If one of the visitors wished not to be recorded, they could contact one of the researchers who were recognizable by their FROG project badge. In such cases, the camera would be stopped and if necessary the recordings would be destroyed. However, this never happened. People who joined the robot tour for at least one full explanation at one point of interest were randomly chosen to participate in a short interview. The interviews were with groups of one to three visitors who visited the Royal Alcázar together and joined the robot together. 112

129 While the robot performed in the human-translated condition, 6 interviews (10 people) were conducted. 11 interviews (16 visitors) were conducted when the robot performed in robot-optimized condition. The interviews had a semi-structured set-up and questions that were asked included: Which aspect of the robot or the tour got your attention first? Can you describe your experience of following the robot guided tour, as if you are telling it to friends and family at home? Can you describe the robot in three words? In what way is the robot guided tour different from information boards, audio guides or tour guides? Imagine the Royal Alcázar would decide to use this kind of robots in other areas of the site as well, what would you think of that? Do you have suggestions for the robot or other remarks? The interviews took approximately 2-5 minutes. The interviews were recorded using a voice recorder when people gave consent to record, otherwise only notes were taken, which was the case for only one interview. The interviews were performed in English or occasionally in Dutch. (See Appendix F for the interview questions). Data preparation and analysis We analyzed the video data by using DREAM. DREAM (Data Reduction Event Analysis Method) is a method we developed to annotate rich video data in a focused and fast manner and to anticipate for too early interpretations. The method is based on thin slices of behavior (Ambady & Rosenthal, 1992), the grounded theory method (Corbin & Strauss, 1990) and coding with multiple coders. For the analysis of the video data of in-the-wild human-robot interaction, only thin slices, sequences of three stills of the video of each of the robots explanation at a point of interest, were used to analyze the data. Going from video data to sequences of stills imply a large reduction of data, which speeds up the data analysis. Nevertheless, the outcomes still are reliable. See chapter 7 for a full description about the development and evaluation of DREAM. 5 The advantage of using this method is that only the moments of interaction are analyzed, by the use of three images per action. This led to a series of sequences of three images to be annotated. The time that the robot was going from one point to the next was not taken into account in this analysis. To analyze the sequences, two researchers (#1 and #2) discussed the definition of the codes and created an affinity diagram (Courage & Baxter, 2005), on which the final code scheme was based. The found codes (53) were clustered into 10 categories, of which each category contained several codes. For example the category FORMATIONS contained 8 codes: 1 person as close as possible, 2 people as close as possible, 2 different groups of 2 or more people, group resembles a large guided group with more than 11 people, people stand far away, formation is a line, formation is a semi-circle, formation of group is unstructured 113

130 CHAPTER 5 Multimodality Studies (note: each of these codes should be used once when visitors perform these actions, and more than one code of this category can be applied to a sequence). (See Appendix G for a full overview of the codes used for annotation). Atlas.ti (version , 2015) was used to code the data and the inter-rater reliability was calculated with Coding Analysis Toolkit (CAT) (2010). The main researcher (#1) annotated the dataset with the codes from the code scheme. Afterwards a third researcher (#3) was asked to annotate 25% of the sequences using the same codes. The annotations of researchers #1 and #3 were compared to calculate the inter-rater reliability. The inter-rater reliability was κ=0.61 (Fleiss Kappa), which indicates that the codes applied were considerably consistent between researchers (Sim & Wright, 2005). Overall, the robot gave a similar number of explanations at points of interests in both conditions (73 explanations at points of interest in human-translated condition and 71 explanations at points of interest in the robot-optimized condition), so, we could compare the numbers found with the annotation of the data. For the current study, we used the cumulative numbers of annotations of all explanations at points of interests taken together, because we wanted to find the general differences in visitors actions and reactions between the robot behavior sets. Manipulation check To check for successful manipulation of the behavior sets, we searched for observable differences in the behavior of the visitors between conditions that we will present in the results section. As we found differences between the behavior sets in the video study, and the behavior sets for the video study and the in-the-wild study were similar of set up, we assumed visitors would observe these differences as well. Moreover, we do not have indications that these observed differences in visitor s behaviors were introduced due other factors than the two sets of robot behavior. Therefore, based on the results of the online video study and the in-the-wild study we state that we were largely successful in creating two different types of robot behavior Results In order to answer the main research question, we focused on two aspects. These two aspects are the participant s attention during the explanation of the robot at an exhibit and participant s attitudes towards the robot during the explanations at points of interest of the FROG guided tour. We analyzed these aspects through the questionnaire data of the online video study and observations as well as interviews of the in-the-wild study. In this section, we will present the combined results of both studies. 114

131 Attention H1: Robot behavior optimized for the modalities of a low-anthropomorphic tour guide robot will lead to longer periods of attention and higher recall of the information provided by the robot. First, to gain insight into how visitor attention was influenced by the different behavior sets, we analyzed the quantitative data of the online video study. To check whether participants felt distracted by one of both behavior sets, we performed Mann-Whitney tests with the items of the attention scale as dependent variables and the condition as independent variable. We found no significant differences in self-reports on how participants understood the robot s stories or participants distraction or attention to the artworks between conditions. Furthermore, we analyzed to what extent participants remembered correctly the details provided by the robot. We did this by summing up correct answers per participant, incorrect answers per participant, and I can t remember answers per participant. For each of the categories participants could score between 1 and 4 as there were 4 questions. Note that the sum of the three categories for each participant also was 4 (unless they did not fill out all questions). We used these items in Mann-Whitney tests to check for differences between conditions. We found that participants who observed the human-translated behavior set (M = 2.18, SD = 1.44) gave significantly more correct answers than participants in the robotoptimized condition (M = 1.78, SD = 1.34), U = 4280, Z = -1.98, p <.05. And vice versa, we found that participants in the robot-optimized condition (M = 1.32, SD = 1.25) gave significantly more incorrect answers than participants who observed the robot with the human-translated behavior set (M = 0.91, SD = 1.03) U = , Z = , p < Second, we analyzed the actions of the visitors based on the coded video data of the in-thewild study. To check which of both robot behavior sets kept the user attention for a longer time span we counted the number of explanations at points of interest that visitors joined the robot in each condition. In the human-translated condition, we observed more often that people left the tour after one explanation at one point of interest (23 times), compared with the robot-optimized condition (16 times). Contrarily, in the human-translated condition, we observed less often that people left the tour after two explanations at two point of interest (8 times), compared with the robot-optimized condition (15 times). Furthermore, in the robot-optimized condition we observed visitors following the robot for more than five explanations at points of interest, which was more than one full tour (5 times), while we only found this one time in the human-translated condition. No differences between conditions were found for joining the robot for 3-5 explanations at different points of interest. To check whether one of both behavior sets was more understandable to visitors, we counted in which condition visitors complied better with the requests of the robot to come closer. 115

132 CHAPTER 5 Multimodality Studies Those requests were verbally asked by the robot, but in the robot-optimized condition the request was also visible on the screen of the robot. We evaluated the compliance of the visitors by checking how visitors reacted to the robot s request to come closer. We found that when one visitor followed the robot, this visitor in the robot-optimized condition complied better to the request please come closer (23 times of 71 explanations at points of interest in this condition) than this visitor in the human-translated condition (6 times of 73 explanations at points of interest in this condition). We did not find differences for compliance of groups of 2 or more people to come closer to the robot. Third, we analyzed the interview data of the in-the-wild study. To check whether the different behavior sets had influence on the understanding of the visitors, we analyzed whether visitors reacted differently to the question did you understand the behavior of the robot? We found that in both conditions visitors stated that they understood the behavior. However, in 5 interviews out of 11 in the robot-optimized condition, visitors explicitly added to their answers what they understood the robot s intentions and instructions were. In contrast, visitors in human-translated condition never added such remarks. As a result, H1 is partly supported. Even though visitors recalled fewer details of the story of the robot correctly when the robot used robot-optimized behavior, it was better able to keep the visitor s attention for a longer time span, people seemed to comply more to the robot with robot-optimized behavior and people seemed to understand the robot-optimized behavior better Attitude towards the robot H2: Robot behavior optimized for the modalities of a low-anthropomorphic tour guide robot will lead to a more positive attitude towards the robot. To gain insight into the aspects that influence visitors attitudes towards the robot, we again started with the analysis of the online questionnaire data. The data was not normally distributed. Hence, to test whether one of both conditions was evaluated more positively, we performed Mann-Whitney tests with condition as independent variable and the reliable constructs of the Godspeed and Source Credibility scales as dependent variables. We did not find any statistical differences for the constructs of the Godspeed and the Source Credibility scale. We also performed Mann-Whitney tests with condition as independent variable and the individual items of the Product Personality scale, and all individual items of the Godspeed and the Source Credibility scales as dependent variables. We found that the robot with the human-translated behaviors (Mdn = 4, SD = 0.98) scored significantly higher on the item serious from the product personality scale than the robot with the robot-optimized 116

133 behaviors (Mdn = 3, SD = 1.04) U = , Z = 2.28, p <.05. Moreover, we found statistical differences on the item Mechanical - Organic from the Godspeed scale U = , Z = -5.8, p <.001. Participants rated the robot with the human-translated behavior (Mdn = 2, SD = 0.95) as more mechanical and the robot with the robot-optimized behavior (Mdn = 3, SD = 1.14) as significantly more organic. Furthermore, to check which robot behavior set participants preferred, we compared the answers to the corresponding question. We found that of the participants who observed both behavior sets, 76 participants (46%) preferred the robot with the human-translated behaviors, while 90 participants (54%) preferred the robot with robot-optimized behaviors. We did not find that age, gender, nationality or education level had influence on these outcomes. Moreover, to check what factors play a role in the preference for one of both behavior sets, we analyzed the answers to the open questions of the questionnaire data. We found no clear distinction for participants preferences based on the answers to the open questions. Second, we used the coded video data of the in-the-wild study to analyze the visitors attitudes towards the robot behaviors. To check whether one of both behavior sets elicited visitors to behave more attracted to the robot, we counted the times that visitors took pictures of the robot. We found that in robot-optimized condition, visitors took a picture 20 times (of 71 explanations at points of interest in this condition), while in human-translated condition, visitors took a picture of the robot only 6 times (of 73 explanations at points of interest in this condition). Finally, we analyzed the interviews to gain insight into the attitude of visitors towards the robot. To check whether the robot attracted the same kind of visitors between conditions, we asked them if they would prefer a human tour guide over the robot tour guide. We found that more frequently during interviews in the human-translated condition (2 out of 6 interviews), than during interviews in the robot-optimized condition (1 out of 11 interviews) visitors stated they would prefer a human tour guide. Even though we did not specifically ask, in five interviews out of 11 interviews, visitors in the robot-optimized condition mentioned that they really did not like human tour guides and the large groups, against no remarks like this during the interviews out of 6 with the visitors in the human-translated condition. Also, we found that in four interviews out of 11, visitors in the robot-optimized condition mentioned that they liked the fact that they could leave the robot when they wanted to go somewhere else or when they wanted to focus on something else, without offending the guide, while in none of 6 interviews visitors in the human-translated condition made this remark. 5 Thus, H2 is partially supported. The robot-optimized behavior was slightly more preferred 117

134 CHAPTER 5 Multimodality Studies over human-translated behavior, but did not lead to a significantly more positive attitude towards the robot Discussion In this section we will first discuss the results of both studies, and then go on to discuss the implications of using the robot-optimized design approach to design behavior for lowanthropomorphic robots. Finally we will discuss the use of mixed methods for the evaluation of robot behavior Discussion of the results The combined results of our studies show that even though we only found small differences between both conditions, participants seemed to be slightly more positive towards the tour guide robot with robot-optimized behavior. Moreover, visitors that did not like human tour guides, enjoyed joining the robot with robot-optimized behavior to improve their experiences at the site. Therefore, we argue that using a robot-optimized approach to develop behavior for low-anthropomorphic robots is more effective than using a humantranslated approach. In the studies, we did not find major differences on effectiveness of either the humantranslated behavior set or the robot-optimized behavior set in terms of participants attention and attitudes towards the robots. This could be explained by the degree of human-likeness of FROG. Possibly, the modalities of FROG were not perceived as very distinct from human modalities (e.g. because FROG has dynamic eyes, and a pointer that could be interpreted as one arm). Hence, FROG s characteristics might have led to behavior sets that were perceived as being quite similar. Based on this explanation, we expect that perceived differences between the behavior sets we applied may be more extreme when they are applied to robots that look much less humanlike and use modalities that are very distinct from human modalities. For example, the use of light, sound and projection might give more options to create a distinct design for robot-optimized behavior. Furthermore, the use of modalities that are not a replacement for human modalities might lead to more creative behavior design for robots. Compared to other studies in which robot behavior was developed and evaluated, the differences we found in responses towards the two sets of behavior may seem marginal. However, there are two main differences between our study and previously performed studies. First, although multimodal behavior strongly influences human-robot interaction, knowledge 118

135 of how to effectively design a set of intuitive and unambiguous multimodal behaviors for social robots is limited. In previous research, behavior cues have been mostly studied in isolation. For instance, studies have been carried out on how gaze (Andrist, Tan, Gleicher, & Mutlu, 2014), gestures (van Dijk, Torta, & Cuijpers, 2013) or body movement (Kuzuoka et al., 2010) separately influence human-robot interaction. This has led to valuable insights, however, the interplay between modalities has not been addressed in these studies. When more modalities are used at the same time, which is almost always the case in natural behavior, the effectiveness of specific behaviors might be different than suggested in these kinds of studies. Nevertheless, this manner of studying effectiveness of behavior also introduces a limitation as it becomes more difficult to understand what causes the differences influenced by the different behavior sets. Thus a combination of both approaches might be necessary to fully understand the effect of robot behavior on human-robot interaction. Second, with the current study we venture beyond previous studies on behavior design for social robots (e.g. (Karreman et al., 2013; Kuzuoka et al., 2010; Sidner, Kidd, Lee, & Lesh, 2004)), because in these studies humanlike robot functional interaction behavior is compared to random or no interaction behavior (control condition). As a result, the developed functional interaction behavior was in all situations perceived more positively than the lack of behavior or randomly performed behavior. Therefore, the only conclusion that can be drawn from these studies is that designing behavior for a robot is better than not designing behavior for a robot. In contrast, in this chapter we explored behavioral cues in multimodal form and evaluated two carefully and functionally designed alternatives to find which set of behaviors would be preferred Design approach to develop robot behavior The field of human-robot interaction is a relatively young field. Because of this, methods and knowledge from other fields are used extensively. Human-robot interaction strongly relies on knowledge from social sciences, behavioral sciences, psychology and human-computer interaction. So far, the fields of product and interaction design have had a modest influence on human-robot interaction, although there are some good examples of the contribution of industrial design on human-robot interaction. Examples are the development and evaluation of the Hug (DiSalvo, Gemperle, Forlizzi, & Montgomery, 2003), the development of the Snackbot (Lee et al., 2009) and a study in which new behavior for a Roomba vacuum cleaner was designed starting from a defined user experience (Hendriks, Meerbeek, Boess, Pauws, & Sonneveld, 2011). These studies were all performed by, or in collaboration with product design or interaction design researchers. By presenting our alternative design approach to design robot behavior we contribute to transferring knowledge from product and interaction design to human-robot interaction and show what opportunities these fields offer. 5 During our studies of the literature, we found that usually no methodological approach is 119

136 CHAPTER 5 Multimodality Studies used to create behavior for social robots. As a result, the multimodal behavior of the robot might be inconsistent and unpredictable. This is a problem, because several researchers state that robot behavior should be consistent over time (Duffy, 2003; Kim, Kwak, & Kim, 2008; Walters, Syrdal, Dautenhahn, te Boekhorst, & Koay, 2007; S. Woods, Dautenhahn, Kaouri, Boekhorst, & Koay, 2005). Currently, creating consistent robot behavior is usually done by adding a humanlike personality to the robot. This personality profile informs how behaviors should be performed (e.g. the use of a large amplitude for gestures for an extravert robot). Alternatively, in the robot-optimized approach we used the morphological chart to systematically develop behavior for the different modalities of the robot. The morphological chart makes the design choices visible which makes it easier for designers to discuss the best options. Additionally, the use of images instead of text only to describe behavior of the robot, helps designers to agree on what the behavior of the robot will look like and it will make early interpretation on how the behavior will be experienced more feasible. Furthermore, this way of working ensures the documentation of design choices throughout the project and more systematic improvements for the behavior. For these reasons, we found that the creation of the morphological chart and all possible options for robot behaviors was worth the effort and we would use this method again when developing behaviors for a robot that lacks humanlike modalities Use of mixed methods for evaluation To be able to find results on the value of our alternative design approach to design behavior for robots, we used a combination of study methods. We are aware that both studies introduced some limitations. First, the videos in the video study only lasted about 1.40 minutes. Therefore, people may not have had enough time to interpret and understand certain behaviors. This might explain why people recalled more details correctly in the robot with humanlike behavior condition, because people were more familiar with this behavior. A limitation of the in-the-wild study is that the sessions of this study were performed in different time-slots during two days. In the robot-optimized condition the experiment was stopped due to technical issues and was continued for 45 minutes in the afternoon of the same day. This might have had some influence on the visitors reactions towards the robot. However, as we found that the visitors opinions about the robot behaviors did not change over sessions, we decided to use the interviews of all sessions in the analysis Conclusion To conclude, we answer the question: Do people respond more positively to robot- 120

137 optimized behavior compared with human-translated behavior? Based on the results of both studies we argue that the proposed robot-optimized approach lead to slightly more positive attitudes and responses towards the robot. Therefore, this approach is promising to design behavior for robots. Although the differences we found between conditions were small, participants understood the robot-optimized behavior and accepted the robot-optimized behavior. Thus, with these studies we showed that when a robot does not have a humanlike appearance, people can understand behavior specifically designed and optimized for a robot s modalities. Therefore, further exploration of the robot-optimized design approach seems valuable. Our alternative approach to develop robot behavior was based on methods used in product design and interaction design. While this field is developing methods to design smart interactive products that people can intuitively interact with and establish relationships with, the field of human-robot interaction is increasingly focusing on similar aims in the development of social robots. Therefore, we would like to argue that these fields can mutually learn from one another. As a result, the difference between functionally designed smart products, which include robots such as FROG, and biologically inspired social robots as was proposed by Fong, Nourbakhsh and Dautenhahn (2003) is narrowing. This is because knowledge and approaches to develop robots and smart products become more widely accepted, understood and used. On the one hand, consumer products become more and more robotic, in a sense that they become more autonomous, take over tasks from people and communicate in a humanlike manner, for example with speech. Consumer products that have become more robotic are robotic vacuum cleaners such as the Roomba, which people might perceive as social (Forlizzi & DiSalvo, 2006) and autonomous cars, which were originally designed in the fields of product design and interaction design, but now need more and more knowledge from the field of human-robot interaction. On the other hand, a growing number of robots that were developed with knowledge from the field of Human-Robot Interaction become interactive consumer products. Think, for example, of the introduction of Aldebaran s Pepper 1 or C. Breazeal s Jibo 2 ; robots that are designed to assist, engage and entertain people in their homes or in public places as a day-to-day companion. Consequently, this enriches the view on the development of social robots and interactive smart products. 5 We expect that more specific modalities, such as light, sound and movement, will lead to better distinguishable robot-optimized behavior. Future studies should focus on these aspects. Moreover, a focus on developing more robot specific modalities will lead to consumer robots that do not all closely resemble people. Therefore, we argue that developing robot- 1 For more information see: 2 For more information see: 121

138 CHAPTER 5 Multimodality Studies optimized behavior specifically for the modalities of a low-anthropomorphic robot is the future for robot behavior design. Especially, study 6 shows that robot optimized-behavior for low-anthropomorphic robots, that is designed following a method from Industrial Design, lead to positive responses. However, differences between both behavior sets as presented in study 5 and 6 were small. Moreover, the participants of study 6 reported to generally not prefer guided tours and may be prone to liking the robot-optimized behavior. While the result is not yet generalizable to other contexts of use, these results are a first step to answering research question 2 on what the best approach is to design a consistent set of nonverbal communication behavior for low-anthropomorphic robots. Moreover, the combination of a video study and an in-the-wild study allowed combining rich context information with more statistic insights. Even though it was impossible to use similar methods in both studies, the results of the studies complement each other. This combination of methods - controlled and in-the-wild - allows analysis of peoples perceptions, attitudes and responses in a more scripted situation as well as observing free and spontaneous interactions with the robot. Thus, the results presented in this chapter, contribute to research question 3 (in what way the effect of robot behavior on people s perception and experience should be studied), by showing the benefits of combining controlled and in-the-wild studies. 122

139 6 Experiences with a lowanthropomorphic tour guide robot; A case study with FROG in the wild In this chapter I will present the evaluation of the experiences of invited and spontaneous visitors who joined a fully autonomous FROG tour. The chapter will start with a description of the fully functional FROG platform and a description of interaction abilities. Next, the results of the evaluation study that focused on the user experiences of visitors joining a FROG tour will be described. Finally, in the discussion I will focus on how visitors reacted to robot-optimized behavior for FROG, how high expectations of robots in general might have influenced visitors experiences of the FROG tours, and how understanding of social situation should influenced development of technology for social robots in the future. This chapter was largely based on: Karreman, D. E., Ludden, G. D. S., & Evers, V. (2015, October). Visiting cultural heritage with a tour guide robot: A user evaluation study in the wild. In Proceedings of International Conference on Social Robotics (pp ). Springer International Publishing

140 CHAPTER 6 Case Study 6.1. Motivation Creating a socially interactive tour guide robot does not only require a robot that can drive around autonomously and can start to present information in the right place, but it needs a robot that can understand and react to real interactions between and with uninformed people. This is exactly what we have done within the FROG project. The knowledge of several partners was combined to create a socially aware robot that could guide visitors around the Royal Alcázar. The experiences of the visitors were collected by means of interviews and observations and provided a basis to improve FROG tours even further. Note that FROG gave full tours, but that my approach to design robot-optimized behavior described in this thesis only applies for moments that a robot explains at an exhibit. For the moments that the robot traveled from one exhibit to the next, we had to add behavior as well. Even though the traveling behavior for FROG was not directly a result of the research presented in this thesis, it was based on observations of failures and successes of previous studies. Nevertheless, I present results of the full tour in this chapter, to gain a good overview of visitors experiences of joining a robot tour guide Visiting cultural heritage with a tour guide robot: A user evaluation study in the wild (study 7) In order to gain rich insights into the behaviors, attitudes, responses and experiences visitors have of the FROG robot, we performed a user evaluation study with FROG at the Royal Alcázar in Seville, Spain. Participants followed FROG for a fully autonomous tour through indoor and outdoor spaces in the site. The least intrusive way to gather data about how people experienced the tours would be through observation. However, using only observations would give too little insight into people s understanding and experience of the robot guided tour. Therefore, as well as observing visitors, we interviewed them after completing a FROG tour. As we could not rely on spontaneous visitors (n=18) to comply with the request for an interview after a tour, we also invited participants (n=8) to join a FROG tour and participate in the interview. These participants were not instructed beforehand about the tour or the robot. Scheduling participants also offered us the opportunity to equip them with a microphone to record their speech during the tour and the interview. In this way we were able to collect rich data on user experience, attitude, responses and behaviors from our sample of scheduled participants. 124

141 Study design FROG; a fully autonomous tour guide robot FROG is equipped with technical features that enable the robot to perform autonomous tours. FROG is able to drive around autonomously. It avoids collisions with people and objects by taking into account basic social conventions, such as driving around people that stand in its way. This has been described by Ramón-Vigo, Perez-Higueras, Caballero and Merino (2014). A bumper around the base of the robot ensures that the robot will stand still immediately when it touches an object or person. Furthermore, FROG can search for groups, estimate their orientation and drive towards them, details of this have been described by Flohr, Dumitru-Guzu, Kooij and Gavrila (2015). Moreover, FROG can adjust the content of a tour to the interest of visitors by calculating the interest of the visitors based on their facial expressions. The techniques used for this have been described by Marras, Tzimiropoulos, Zafeiriou and Pantic (2014). These three described features are state-of-the-art innovations created by the technical partners of the FROG project. The FROG platform was designed by IDMind (choice and placement of technical components, appearance design) and YDreams (appearance design). The appearance of the robot was designed to attract the visitors, but also to be functional. The front of the robot has two eyes in which the cameras for group detection were placed. On top of the robot, a pointer arm (3 degrees of freedom) with extra camera and LED lights was placed to enable FROG to point to places of interest. The touch screen on the front, which has full sunlight capacity in order to allow outdoor usage, enables visitors to make contact and interact with FROG. See Figure 6.1 for the FROG robot in action. The content was developed by the University of Twente. The content was carefully chosen to give visitors a brief but rich insight into the history of the site. Two criteria were used to choose the points of interest FROG would visit. First, the point of interest had to be accessible for FROG, which was not able to climb stairs. The second criteria was how often human guides visited the points of interest. The content (narrations and visuals) that FROG provided for each of the points of interest was based on information given in the room by the Royal Alcázar complemented with information given in one of the official books sold by the Royal Alcázar (Hernández Núñez & Morales, 1999). Also, as far as possible, curiosities were added to the narrations. Curiosities Figure 6.1: FROG in action 6 125

142 CHAPTER 6 Case Study are pieces of information that are special for one location only. An earlier study revealed that according to human tour guides, visitors really like to obtain such information (Karreman, Dijk, & Evers, 2012). Last, University of Twente worked on the software integration, to connect the input and output of the different partners. As a result, the consortium created FROG that was a fully autonomous robot which was able to give tours through the main sightseeing locations of the Royal Alcázar. Multimodal robot-optimized behavior The nonverbal multimodal behavior performed by FROG at exhibits was based on results of earlier performed studies. However, the robot would now drive longer distances than in previously presented studies. Moreover, more content had to be presented in the current tour than in previously presented studies. Therefore, the choices for behavior beyond previously studied nonverbal behavior are described in the following section. Guide/narrator To present FROG as an engaging and fun robot, and at the same time to have it narrate the more serious content, we introduced a dual personality for the robot s interface. This dual personality allowed FROG to switch from robot guide mode to narrator mode. As a guide, FROG used a robot voice that guided the visitors from one point to the next. To ensure realistic expectations of FROG s intelligence, the robot voice consisted of repetitive, prerecorded standard sentences, used only to convey the status of the robot and to indicate that the robot was not processing speech input. At the points of interest, FROG gave information about the site as the narrator. As narrator, FROG used a prerecorded human voice-over to offer narrations. Use of visuals Narrations of FROG were supported by visuals. These visuals were presented either on FROG s screen or projected on nearby walls through the onboard projector. Furthermore, FROG was equipped with a pointer that was used to point to several points of interest. However, the pointer also had another function, namely to search for participants when the robot stood at the starting point. This was only an interactional feature, because the pointer could not actually sense people around FROG; localizing people was done by using the laser sensors in the base of the robot. Orientation Although FROG s main interaction features were positioned on its front, it drove forward 126

143 during transitions to a next point, which meant that visitors had to follow the robot facing its back. By facing forwards, FROG would indicate to following visitors and other surrounding visitors its direction. We expected that it would be most natural for visitors to follow the robot in this way, because this is what happens when people follow a human tour guide as well. Behavior during travel Even though the focus of this thesis is not on robot-optimized behavior while driving, FROG needed designed behavior for the moments it traveled. Therefore the behavior of the robot during its movement from one point of interest to the next is based on observations of successes and failures during driving, obtained in earlier performed studies. During travel, the screen was used to visualize a face for FROG. Moreover, in the bottom right corner of the screen a small detail of the map of the site was shown, indicting where the robot was. Even though at points of interest FROG did not have added anthropomorphic features on the screen, we decided to give it some character while driving to gain and keep the visitors interest. Thus, during travel, the screen mostly showed a smile. Additionally, information on the status of the robot was added, such as a small map to show its destination or messages such as loading location data. The pointer was not used to indicate driving direction when the robot traveled to the next location. This was for the reason that extensive moment would be applied to the pointer caused by the uneven floors of the Royal Alcázar. Therefore, during travel the pointer was in rest orientation. Furthermore, FROG did not turn towards visitors before it started an explanation about a point of interest, because it needed time to take in a position that was most advantageous for visitors to see the content. Nevertheless, we expected that participants would have enough time to gather around to face the front of the robot again at the new location. 6 Participants During the FROG tours, invited participants as well as spontaneous visitors joined the robot tours. Invited participants were people we recruited in advance to follow the whole tour and they participated in a long (approx. 30 minutes) interview. A total of eight participants were recruited in four separate groups; two groups comprised of a Dutch male and a Dutch female; one group consisted of one Spanish female and her baby; and one group comprised of three Spanish students. All participants gave consent to participate in the study. As compensation for their participation, invited participants were allowed to visit the site by themselves after they had finished their participation in the study. 127

144 CHAPTER 6 Case Study The invited participants were often joined by spontaneous visitors during the tour. Spontaneous visitors were people who visited the Royal Alcázar by chance and who joined one of the FROG tours spontaneously. A total of 18 spontaneous visitors who followed 8 different tours were interviewed. The compositions of these groups varied. There was one big group of five adults and two children (<8 years), a pair of adult men, two couples, a mother with her daughter (<10 years), and three visitors who visited individually. All participants gave consent to sound record the short interviews. In general, more people than the interviewed spontaneous visitors followed the tour. We observed their interactions with the robot, but those visitors were not interviewed. Moreover, we did not ask them for consent, therefore this data is not used in this chapter. The invited participants had little or no experience with social robots. One of the spontaneous visitors was a technician and one was a robotics lecturer, others had no previous experience with robots. Most participants had previous experience with human tour guides or audio guides. All participants spoke English as a first or second language. Procedure During a single week in June 2014 FROG gave one to three autonomous tours a day through the Royal Alcázar. The tours always started close to the entrance gate. At this starting point the robot searched for groups of visitors who had just entered. When the robot located (a group of) visitors, it asked whether they were interested in a guided tour. The visitor groups were either the invited participants, the spontaneous visitors or comprised of both. The robot traveled to six points of interest. During the week that the robot gave tours in the Royal Alcázar, small changes were made to the behavior of the robot to iteratively improve the tour. Even though FROG performed the tours autonomously, 7-10 researchers followed each tour from a distance to monitor progress of the various technical onboard systems. Also, one researcher carried a remote control stop, to stop the robot in any case of emergency; this did not occur. The complete tour took about 25 minutes, depending on the number of obstacles and the number of people the robot encountered in small hallways. The invited participants were asked to think aloud during the tour. For each group with invited participants, one participant wore a small microphone, which could generally pick up the speech of the whole invited group. During the walks between points of interest, the researcher asked the participants some questions to gather first reactions on their experience of being guided by the robot. Invited participants were requested to indicate the moment they had the desire to leave the robot tour in case this should occur. However, they were asked to follow the tour till the end for data collection purposes. Spontaneous visitors could join or leave the robot whenever 128

145 they wanted. After a tour with invited visitors, the participants were interviewed about their experiences with the robot. Also, several groups spontaneous visitors were asked to answer some questions when they left the robot. Map of the tour through the Royal Alcázar The exact length of the tour depended on the number of obstacles and the groups of people it encountered in small hallways, and whether or not the story at the second Point of Interest was extended for interested visitors. The route FROG drove is visible on the map in Figure 6.2. The red line (darker) is the route FROG drove during the tour. The stops where FROG gave a presentation are indicated in green (dots). The orange line (lighter) is the route FROG drove when it had ended the tour and drove back to the starting point. 1. Starting point in the Lion s Courtyard 2. First Point of Interest in the Lion s Courtyard 3. Point 3a; Second Point of Interest in the Hunters Courtyard before hrs. in shadow Point 3b; Second Point of Interest in the Hunters Courtyard after hrs. in shadow 4. Third Point of Interest in the Courtyard of the Cruise 5. Fourth Point of Interest inside the Gothic Palace 6. Fifth Point of Interest inside the Vault Room 7. Last Point of Interest and end of tour close to the entrance to the gardens 6 Figure 6.2: Route of the FROG tour through the Royal Alcázar 129

146 CHAPTER 6 Case Study Data collection The interview with the invited participants took approx. 30 minutes and included topics such as: 1) their experience of the tour, 2) the things they liked about the robot or the tour, 3) what they would change about the robot or the tour, 4) how they experienced the interaction with the robot and 5) how they experienced the way the robot guided them to the next point. An example of a question is: How did you experience the length of the tour? Why? (See Appendix D for all interview questions). The interviews were semi-structured, there was no specific order of the topics, and participants were able to expand on what they found important to discuss. Also, since invited participants were interviewed together after each tour, they were able to comment on each other s remarks. This happened a lot and as a result this provided insight into their expectations of the robot. Spontaneous visitors were asked if they would answer a few questions right after they left the robot. The interviews with the spontaneous visitors took about two to five minutes. These visitors were asked about their impression of the robot, their experience of being guided by the robot and if they would have any suggestions for improvement. An example of a question is: How would you describe this experience of following the robot to people at home who did not see the robot? Data analysis The data collected consisted of voice recordings of invited participants while they followed the tour, interviews with the invited participants, and short interviews with the spontaneous visitors. All interviews and the voice recordings made during the tour with the invited participants were transcribed. The transcribed recordings were coded using the NCT (Noticing Things, Collecting Things and Thinking about Things) method as described by Friese (2014), using the qualitative analysis software Atlas.ti (version , 2015). One researcher reviewed all the coded data and grouped it per code. Subsequently, she read the remarks given per code carefully, searching for commonalities and remarkable statements of the participants. These findings were combined in a summary of the general experience, details on what participants liked and did not like in the tour Results In general, participants responded positively to being guided by the robot. This is likely to have been influenced by a willingness to please the interviewer, as well as by the novelty experience of being guided by a robot. Moreover, spontaneous visitors who agreed to take part in an interview were more likely to be those people who had a positive experience. Otherwise, they would have left the robot earlier or not be willing to tell about their 130

147 experiences. The reports of first impressions of the robot were generally positive. The reason that was most mentioned (10 times) was that it was seen as an easy way to obtain information about the site. Four of the invited participants said that even though they could get the same information themselves from books, the experience of following the robot was much more fun to them than reading the guidebook would have been. Six of the spontaneous visitors mentioned that they liked to get information (in English). Further results on what the invited participants and spontaneous visitors experienced as positive or negative will be presented in table 6.1. Table 6.1 presents those themes that were most mentioned by participants, indicating how many of the invited visitors and how many of the spontaneous visitors made similar remarks. The number and quality of the remarks that people gave during the tours brings us closer to understanding how people will experience robot guided tours in real life. However, these are not yet a valid comparison to real representation of the real world. Partly because in the interviews with invited participants, the participants got room to discuss what they thought was important to them, even though there was a topic list and questions that were asked to all. Furthermore, not all themes were addressed in the interviews with spontaneous visitors. See Figures for impressions of the FROG guided tour. 6 Figure 6.3: Visitors oriented in semicircle around FROG Figure 6.4: Visitors following FROG to the next point of interest Figure 6.5: The robot explains what is visible on the wall Figure 6.6: The quiz attracts a lot of people (point of interest 6) Figure 6.7: Look to the object the robot is pointing at 131

148 CHAPTER 6 Case Study Themes Table 6.1: Influenced the visitor experience positively Remarks of visitors on factors that influenced their experience It was fun to join a robot tour, because it is innovative and cannot be found somewhere else, yet, so it was an experience in itself. It enriched the interaction with the environment more effectively than for example audio guides or books would do. Even though it was clear to participants that the robot could not hear or understand them, they talked to the robot, but only when the robot used the robot-voice. Invited participants (n=8) 7 (87.5%) 4 (50%) 7 (87.5%) Spontaneous visitors (n=18) 13 (72.2%) 6 (33.3%) - Length of the total tour as well as the length and the number of stops was OK. It gave the information needed to understand the history of the place. Maybe one more stop would be OK. It is OK when strangers join, but it can be a problem when the new people stand in front of initial visitors or talk too loudly while the robot explains something. The robot guide is helpful and fun for children/young people to explain the history of the site. Influenced the visitors experience negatively 8 (100%) 8 (100%) 4 (50%) 2 (11.1%) - 5 (27.8%) The movements of the robot were jerky and therefore made unclear what its intentions were. 7 (87.5%) - The robot was unclear about where it wanted to go or whether visitors were standing in its way. 3 (37.5%) The robot drove too slowly. 6 (75%) The robot did not turn towards the visitors once it arrived at the location before starting the explanation, this made people feel ignored by the robot. 5 (62.5%) 1 (5.6%) 7 (38.9%) 1 (5.6%) After the explanation, the robot did not allow visitors to look around; it went to the next point immediately. 3 (37.5%) The robot voice was too repetitive. 3 (37.5%) The robot did not make clear how long the tour would take and where the robot would bring them, which was a problem when spontaneous visitors had only limited time to visit the Royal Alcázar. 1 (5.6%) 2 (11.1%) - 9 (50%) 132

149 Clearly visible from the results presented in Table 6.1, is that some themes were only discussed by invited participants, some were discussed by invited participants as well as by spontaneous visitors and one was only discussed by spontaneous visitors. This strengthens the choice for the combined interview approach of invited participants and spontaneous visitors that we used for this study Discussion This user evaluation study of the FROG robot in a real world environment offered us insights into how we can improve the functionality and perceived experience of robot guided tours. In the area of research on tour guide robots, this is a first real world study in an indoor/ outdoor environment that was performed with a low-anthropomorphic, fully autonomous robot, that used a set of nonverbal, multimodal behaviors optimized for its morphology, just as morphology was optimized to perform certain behaviors. Even though it is difficult to generalize the findings to other situations and environments that socially interactive robots may be used for these days, I think that my findings and experiences can help other researchers and designers to influence human-robot interaction positively by creating clear robot behaviors and functionality. There are three main points that I deduced from the results of this user evaluation inthe-wild case study with the tour guide robot FROG. These are that 1) behavior for lowanthropomorphic robots does not have to be a copy of humanlike behavior in order to be understandable and enjoyable, 2) the robot should clearly communicate its abilities and 3) that the technology has to work perfectly, as was already mentioned by the developers of the very first tour guide robots (Arras & Burgard, 2002; Horswill, 1993). Firstly, this study clearly confirmed that robot behavior optimized for the modalities of a low-anthropomorphic robot is accepted and understood by a variety of users. The behavior I designed for FROG (as was described in chapter 5) was intuitive to understand, even though it was not a copy of humanlike behavior. For example, the dual personality that was introduced was clear to the visitors and did not lead to any trouble in understanding. Also, the function of the pointer was clear; participants found out that it was meant to point at points of interest, but that it was used to make contact as well. Nevertheless, participants mentioned that they would have liked to receive more feedback. Participants mentioned that more information about its status could be presented when the robot traveled to the next point. For example, by adding red or green blinking or constantly shining lights that would turn on and off to indicate the status (waiting, arrived) of the robot. Such remarks emphasize that a low-anthropomorphic robot s behavior does not have to be an imitation of human tour guide behavior to convey information naturally and intuitively

150 CHAPTER 6 Case Study Secondly, people that encounter socially interactive robots in everyday environments base their expectations of the robot not only on the capabilities the robot shows. Even though two invited participants stated that they were flabbergasted after they found out at the end of the tour that the robot had given the tour totally autonomously, in this study I found that people had high expectations of robots in general. This might be due to their expectations of robots, which mainly originate from robots they see in science fiction movies. However, most robots designed for public spaces have not reached that level of perfection yet. Therefore, the robot should implicitly make clear to spontaneous users what it is able to do and how to interact with it. Only during the interview of 1 group out of 12 groups of invited and spontaneous participants, 3 participants (out of a total of 26 participants) commented on FROG s name. They mentioned they would have found a bull more appropriate for Spain. One other group of participants (2 people) mentioned that a futuristic robot did not quite fit the ancient Royal Alcázar. This leads me to believe that the name FROG, printed on the robot and its somewhat zoomorphic appearance did not lead to biased responses. However, in future research, it would be interesting to compare different morphologies of low-anthropomorphic robots. Thirdly, participants mainly name technical limitations when they were asked what they experienced negatively. For example, when there were issues with timing due to the projector turning on or off, participants mentioned that something seemed to go wrong. Also, participants mentioned the jerky driving style and jerky movements of the pointer. Although these issues are known and can be solved, visitors will unnecessarily respond to it and not experience the optimal tour. Therefore, it is very important that the technology works flawlessly when a robot is released into semi-public or public places. Moreover, even when technology is flawless, it still might face problems in real world interactions. For example, detecting interest was only possible when the robot could detect a face and read the facial expressions of a visitor. In order to do so, a visitor had to stand right in front of the robot and at a distance of 1 to 1.5 meters. However, from our observations in this in-the-wild study we noticed that groups larger than two or three people would form a semi-circle around the robot in order to allow everybody to see its screen. This is in line with findings of Heath and vom Lehn on visitor behavior in museums (Heath & vom Lehn, 2008). As a result, visitors stood further away from the robot than the system was developed for. As FROG would monitor the facial expressions of a person close by to detect interest or disinterest, and would adapt the content accordingly, the semi-circle formation made it impossible to read the facial expressions. Thus in this case, technology functioned well, however, in interaction the technology could barely be used, because it did not detect people in the places they would naturally stand. Therefore, it is important to understand social situations before technology is developed to be able to positively 134

151 influence the visitor s experiences. Overall, the most important finding of this study was that visitors of the Royal Alcázar are ready to accept a robot tour guide. Most of the invited visitors were not scared of the robot, and they liked the way in which the robot presented the content. Furthermore, several visitors also joined spontaneously and followed the robot, even though they often left early because they were unsure about the duration of the tour. Nevertheless, visitors did not leave the robot for the reason that the behavior of the robot was unclear. This indicates that robot-optimized behavior is understandable for invited and spontaneous visitors. Moreover, from interviews I learned that visitors did not see the robot as a replacement for human tour guides, but as a more fun and socially aware replacement for audio guides

152

153 7 Introducing DREAM; Bringing back video analysis to human-robot interaction in the wild In this chapter I will describe the need for, the development of and the evaluation of a new video analysis method for the analysis of human-robot interaction. Video recordings of human-robot interaction offer rich and valuable data to study human-robot interactions in natural and unconstrained settings. However, analysis of this rich video data is often time consuming and subject to early interpretation. To avoid these problems we developed and evaluated DREAM (Data Reduction Event Analysis Method). The aim of DREAM is to speed up the objective analysis of rich interaction video data. We based the new method on the notion of thin slices of behavior, inspired it by grounded theory methods and realized it through multiple annotators. To check effectiveness, reliability, validity and generalizability of the method, we compared the annotations produced with DREAM to annotations produced with a traditional video annotation method. Also, we compared both methods on effort and experience of the annotators. This chapter is largely based on: 7 Karreman, D. E., Ludden, G. D. S., Evers, V. Introducing DREAM: Bringing back video analysis to human-robot interaction in the wild. (manuscript in preparation). 137

154 CHAPTER 7 Data Reduction Event Analysis Method 7.1. Motivation One of the major goals of social robotics is to create social robots that function in real world environments. This makes evaluation of such robots in unstructured real world settings necessary and even inevitable (Šabanović, Michalowski, & Simmons, 2006). Such evaluations will provide insights into the patterns in human-robot interaction that emerge in less structured real-world social settings. In these settings the interaction between people and the robot is not only influenced by actions of the robot, but also by contextual factors such as other people or environmental factors. These contextual factors will be present when people interact with future social robots, and by learning from real-life trials now, better future robots can be designed. Data on how people react to robots in real world situations can be collected in different ways such as through observation notes, interviews and video recordings. Making video recordings is especially valuable because through video all the information about interactions can be captured and secured for analysis. Yet, as real world video data is rich and unstructured, analysis of this data is often laborious. Furthermore, current methods of analysis are limited in their opportunities because they only support working with predefined codes, counting of events or timing of durations. Approaches that offer more explorative means of analysis stem from grounded theory. These seem to be more suitable to use with rich video data, because they aim to find themes and theories in the data itself. However, although grounded theory methods work well with written data, they are more difficult to use with video data. This is because the use of video recordings asks for interpretation through observation and lacks explanations from the subjects of study. Therefore, a method is needed to analyze rich interaction video data in a fast and reliable way. For the FROG project we collected video data of responses of visitors to the tour guide robot FROG. Our aim was to better understand and positively adapt human-robot interactions. We developed, applied and tested new behavior for FROG in several iterations and observed and recorded the interactions between visitors and FROG. From these studies we obtained large amounts of video data. As the observed interactions between robot and visitors were unconstrained, it was challenging and time consuming to analyze the data through traditional video analysis. Therefore, we developed DREAM to analyze rich video data that includes unstructured and less predictable situations Related work In this section, we give an overview of methods used in human-robot interaction to analyze 138

155 video data of controlled studies and in-the-wild studies, of which we describe the benefits and limitations. Driven by the limitations of these methods, we explain the need for a new method to analyze rich and unpredictable in-the-wild human-robot interaction data. Many researchers recognize the value of video data for human-robot interaction, and therefore video data is used for several reasons. Firstly, video is used to get an idea of what is going on, without performing an extended analysis. This procedure was for example followed by Vroon et al. (2015) who only viewed the videos of a robot approaching a group that were very positively rated or very negatively rated. Subsequently, they summarized the phenomena they observed in these approaches. In this way they gained a good understanding of what happened during these human-robot interactions, which they used to support their other results that were based on quantitative analysis of questionnaires. Second, in many cases video data is annotated using predefined codes that were based on previous research or literature. As well as this, the duration of predefined events is annotated, for example to compare between conditions. This method can be called counting predefined events. To illustrate the various types of research that this method can support we will provide some examples. Using annotated video data was for example used in a study in which the researchers counted the occurrence of fourteen criteria, such as eye gaze and touch of an autistic child in interaction with a robot (Dautenhahn & Werry, 2002), it was also used in a study in which the researchers annotated when children spoke, touched the robot, turned their head away from it, and laughed when in interaction with an expressive furniture robot and his robot lamp sidekick (Vázquez, Steinfeld, Hudson, & Forlizzi, 2014) and in a study in which the researchers observed and analyzed guide behaviors of a human tour guide while he explained an exhibit using conversational analysis methods (Yamazaki et al., 2008). Third, in human-robot interaction studies that involve in-the-wild unstructured video data the counting predefined events method using predefined codes or measuring length of predefined events is also used to analyze the data. This was for example done by Rosenthalvon der Pütten et al. (2014) in a study in which the length of the interaction between a robot and naïve people was measured. These people were confronted with Geminoid HI-1 and comparisons were made between a robot in moving condition and a robot in static condition. Moreover, Shiomi, Kanda, Ishiguro and Hagita (2010) analyzed whether naïve people in a shopping mall overheard the robot s conversation more often when the robot was driving backwards or if it was driving forwards by annotating the visitor reactions. In these studies predefined codes were used to analyze the rich dataset. 7 Even though often used, the use of predefined codes is not always desired, because this limits the exploration of unexpected actions and interactions between robot and person. Thus, when one is searching for new patterns when no previously gained information can 139

156 CHAPTER 7 Data Reduction Event Analysis Method be found about a topic, and no previous codes can be provided, a traditional counting predefined events method is not practicable. In contrast, a grounded theory method or NCT-method seems to be more suitable than a counting predefined events method to search for new patterns. In the grounded theory method (Corbin & Strauss, 1990), the starting point for data analysis is not a hypothesis, but theories and results are viewed as being grounded in the data. The method consists of three steps, which can be performed iteratively to come to the best results (open coding, axial coding and selective coding). Furthermore, in the grounded theory method not all data have to be collected before the analysis starts. Moreover, the data collection can be stopped when the researcher is under the assumption that all necessary information is there. This is, for example, when in the open coding step no new codes can be found in newly collected data (Corbin & Strauss, 1990). Another method to annotate and analyze qualitative data is the NCT-method; Noticing Things, Collecting Things and Thinking about Things (Friese, 2014). This method does allow for the use of images or video data. Furthermore, this method is especially suitable to use with computer-aided qualitative data analysis software, such as the software packages Atlas.ti (version , 2015) or NVIVO (Richards, 1999), because it supports the user to go through the data over and over again, which is not the case for a traditional Grounded Theory analysis (Friese, 2014). Even though using these methods for annotation makes the process laborious and time consuming, these methods are most suitable to analyze in-thewild unstructured data. Note that the use of computer-vision programs to automatically perform the analysis is only possible when predefined codes are used. The search for new patterns in the rich video data using computer-vision programs is impossible, because the interaction patterns are unstructured, unpredictable and initially unknown to the researchers. Consequently, the computer-vision program does not have any requirements to search for. Therefore, the use of computer-vision programs is not suitable to use for the analysis of rich and unstructured in-the-wild video data. A good example of the use of the grounded theory method in human-robot interaction is the work by Mutlu and Forlizzi (2008). They used the method to annotate and analyze interviews and observation notes to identify the cause for the different experiences of employees of two different departments in a hospital where a robot was implemented. Furthermore, the grounded theory method was used in a study on Roomba robots in the homes of people performed by Forlizzi & Disalvo (2006), in a study by Siino and Hinds (2005) on how sex segregation structures in different groups of workers impact the way in which they make sense of a mobile autonomous robot in their work environment and in a study in which Kanda, Miyashita and Osada (2008) identified the extent to which the appearance of 140

157 a humanoid robot affects human behavior towards it. Note that all above mentioned studies used transcribed interviews and field or observation notes, but did not use the grounded theory method to analyze video observation data. In contrast, Sabelli, Kanda and Hagita (2011) used the grounded theory method to analyze video data of a study in which they placed a Robovie robot in an elderly care center for 3.5 months. As a preparation step to analyze the video data, even before annotating the videos, one researcher transcribed all video recordings to be able to explore them in detail (Sabelli et al., 2011). We would like to argue that a downside of this protocol is that when rich video data is available, transcriptions of it lead to a too early interpretation by only one researcher. Further, by making these transcriptions rich data might get lost, because the researcher is likely to overlook something. Therefore, by following this protocol, Sabelli et al. (2011) actually used the grounded theory method to analyze observation notes. Possibly, the protocol of making transcriptions of the video data was done because these researchers felt that using the grounded theory method did not fully support analysis of the initial video data. Even though the grounded theory method for video analysis is rarely used, Salinger, Plonka and Prechelt (2008) state that qualitative analysis is needed to find what is really going on in interaction situations. They tried to use the method for video data they recorded on pair programming, however, they experienced problems in using the grounded theory for the analysis of rich video data (Salinger et al., 2008). In their paper they describe these problems as follows: a lack of predefined focus, no predefined level of detail for coding, no predefined level of acceptable interpretation of events (subjectivity), too many topics to code, lack of concept grouping due to diversity of events to code and misjudgments of importance due to the large number of concepts (Salinger et al., 2008). This indicates that the grounded theory method has severe limitations when working with video data. To overcome limitations of the counting predefined events method for analysis of in-thewild data, as well as the limitations of the grounded theory method or the NCT-method for use with video data, in this chapter I introduce DREAM, Data Reduction Event Analysis Method. For the creation of the method, we found inspiration in the grounded theory method (Corbin & Strauss, 1990), the Thin Slices of Behavior theory of Ambady and Rosenthal (1992) and the use of the Affinity Diagram (Courage & Baxter, 2005). In the remainder of this chapter I will explain the concept, intensive testing and evaluation of DREAM DREAM: annotating videos using thin slices In this section we will present DREAM. In order to highlight the differences between DREAM and traditional methods to analyze human-robot interaction video data, we will first describe 141

158 CHAPTER 7 Data Reduction Event Analysis Method the traditional method that we evaluate DREAM against later in the chapter. Next, we will describe the pillars of DREAM while explaining the methods they are based on. The traditional video analysis method that comes closest to DREAM is to count predefined events. As explained in the previous section, this method to analyze interactions from the actual video data is used regularly in the analysis of human-robot interaction. Examples of the use of this method are described in (Dautenhahn & Werry, 2002; Vázquez et al., 2014; Yamazaki et al., 2008). Counting predefined events is suitable to annotate and thus analyze the occurrence of predefined actions or reactions, for example to analyze differences in subject behavior between two test conditions. This is specifically valuable when studies are performed in controlled settings and when differences between conditions should be analyzed. However, due to the pre-defined codes it does not allow annotators to score unexpected events because these are by definition not part of the pre-defined code set. Specifically for in-the-wild studies, unexpected events are likely to occur and analyzing them can be of great value for human-robot interaction researchers. Moreover, looking back and forth through video data to apply codes, makes the process laborious and time consuming. Thus, using the counting predefined events method limits the chances to find unexpected events or surprising participant actions that are grounded in the data, while it is time consuming as well. With DREAM we want to introduce an alternative method that overcomes the disadvantages of the counting predefined events method. DREAM is based on thin slices of events; essential parts of video data are clipped into sequences of three images. Only these sequences are used to define the codes for annotation and are subsequently annotated with the codes found. As a result, not the video data, but only the sequences are annotated, which speeds up the analysis process. Moreover, DREAM uses the raw video data without previous interpretation of the data to create a code scheme. As a result, the codes are grounded in the data, but are defined before the annotation of the data-set starts. Therefore, we expect that DREAM will allow for out-of-the-box findings, while also speeding up the data-analysis The pillars of DREAM In human-robot interaction studies it is inevitable that for study purposes only parts of the full interaction are of interest. For example, when participants should interact with a robot to hand over a product, there is a moment of establishing joint attention with the robot to the product, the actual handover and the moment of leaving the robot. For study purposes, only the actual handover might be interesting to analyze, even though the other moments are recorded as well. Moreover, when a human-robot interaction study is performed in the wild, a full experience should be offered to people that spontaneously interact with the robot. As a result, not all collected data is interesting for analysis. Subsequently, a reduction of data is possible to answer the research question. In this section we describe how DREAM 142

159 reduces the data and how in DREAM the data is prepared for analysis. Large reduction of data due to the use of thin slices In DREAM the video data will be reduced to multiple sequences of images. Based on the Thin Slices of Behavior theory of Ambady and Rosenthal (1992), we expect that not all data is needed to gain a good understanding of the context compared to counting predefined events. Ambady and Rosenthal (1992) have shown that people are able to label the right emotions when seeing only short (fragments of) videos. This was also true for videos with a language that annotators did not understand, and for short audio-files in a language that annotators did not understand. Apparently people are good at understanding the context, without seeing the whole picture. Based on these results, we wanted to find out if these thin slices would be good indicators for interactions between people and robots. We decided not to work with short videos but just a sequence of images of visitor responses to an action of the robot to understand the context and interpret the interactions during the tour. To clip the sequences (e.g. three images in row), timeframes for the sequences need to be chosen. The duration of a piece of video to be clipped to a sequence of images depends on the level of detail that needs to be visible in the sequences to be analyzed. Which in turn depends on the level of detail that is needed to be able to answer the research question(s). As the search for events is clearly defined with the research question, we assume it is acceptable to largely reduce the data. For example, we described how this works with the handover, which was presented at the start of this section. To understand what behavior of the robot influences the actual handover, a valid question would be: what robot behavior influences interaction during the handover? Such a question requires us to review the full interaction. With this question it is unclear what exactly should be focused on and it is likely that information will be overlooked. However, more specific would be a question such as: how does robot gaze behavior influence the handover? Such a question would require us to focus on the change in behavior of the participant when the robot changes its gaze behavior, while information on other actions of the robot and reactions of the participants does not have to be analyzed. This indicates that a more focused research question can lead to a larger reduction of data to analyze. 7 Nevertheless, in DREAM a more radical reduction of data is performed by making sequences of stills from the video data. Note that while going from video to stills, a lot of data is left out. Hence, this leads to a large reduction of the data to analyze. Yet, this also means that data will be lost. In a large dataset, if an event will be lost, it presumably will occur more often and will be found in the other sequences. Otherwise, if an event only occurs once, it probably is not that important and therefore OK to loose, as in any case such a finding would 143

160 CHAPTER 7 not significantly influence the results. To prove this, we will compare DREAM to a counting predefined events methods and we will show that the large reduction of data with DREAM is of little influence on results found. Data Reduction Event Analysis Method Codes are grounded in the data, but defined before the annotation process starts In general, there are two ways to create codes schemes that will be used to annotate qualitative data. These are: a) use of existing or predefined codes or b) use of codes created during the annotation based on patterns in the data. For traditional video analysis methods, usually existing or predefined codes are used. As a result, during the analysis phase there is less room to focus on unexpected, not predefined themes. For grounded theory methods, codes, theories and results are based on the dataset, and thus grounded in the data. In DREAM the clipped sequences are used to create the codes, before these will be annotated. Consequently, the codes are based on patterns in the data and thus allow us to include findings that were impossible to think of beforehand. Yet, the code set is known before the actual annotation starts, while the patterns to code are fully grounded in the data. Multiple annotators to come to reliable results We propose that in DREAM, at least two annotators (1 and 2) together create the coding scheme. A discussion between annotators will help to objectively define distinctive categories and the right textual descriptions for the categories during the sorting process. This was also experienced by Salinger et al. (2008), who conducted a study on the use of grounded theory to analyze video data. Based on their findings, we presume that the participation of two annotators will lead to a coding scheme with higher internal reliability than when it is created by one annotator. Figure 7.1: Impression of the sorting process 144

161 Categorization of the sequences can be performed by the Affinity Diagram method (Courage & Baxter, 2005), of which the result will be the coding scheme. To use this method, all sequences should be printed on separated sheets. Subsequently, the annotators can physically sort all sequences into categories. See Figure 7.1 for an impression of the creation of the affinity diagram for the human-robot interaction sample study as described in section 7.5. When all sequences are sorted, the annotators should check categories for misplacement and disagreement. At this stage issues with misplaced sequences should be solved, by discussing the options. Next, large categories can be split into smaller and more specific categories, while small categories can be joined into one. Finally, a textual description of the codes, appropriate instructions on how to apply the codes, and additionally a set of code cards (see Figure 7.2) are created. During analysis phase, the first and a third annotator will annotate part of the data individually, to define inter rater reliability of the codes applied. It is important to ask a third annotator to join, because the first and second annotators are already familiar with the data and the codes. This is done to ensure the generalizability and reliability of the results. Code cards (additional) To assist annotators in the actual coding of the data, code cards can be created. Code cards are cards with a visual representation of the textual description of the codes. As annotators will be asked to code images, a visual code is probably easier to apply to a sequence of images than a textual code. In the code cards not all variations of one category can be visualized, therefore, schematic and simplified visualizations should be used. See Figure 7.2 for examples of code cards, as used in the analysis in the study described in section Dataset to validate DREAM To try out DREAM, we first used it to analyze the video data of a semi controlled real world human-robot interaction study. In this study, in which a robot tour guide gave short tours to spontaneously joining visitors, we focused on the group formations the visitors made and how the orientation of the robot influenced these. The results of this study are presented in (Karreman, Ludden, van Dijk, & Evers, 2015) and chapter 4, study 2 of this thesis. By using DREAM to analyze the video data that was gathered in this study, we wanted to gain a first impression of the effectiveness and applicability of the method. 7 The research question that we defined for this study was: how does the robot orientation behavior influence the orientations of the visitors, as well as the type of formations that (groups of) visitors form around the robot? This led us to only use the data of the moments 145

162 CHAPTER 7 Data Reduction Event Analysis Method that the robot gave explanations about points of interest. Thus, we used sequences of the moments that the robot stopped at an exhibit to analyze the changes in visitor formations and the differences in visitor formations between two robot orientation conditions. Our first impressions of effectiveness and efficiency of using DREAM to annotate and analyze the dataset affected three major aspects of video analysis. It speeded up data analysis, we assumed we found generalizable results, and results were not subject to too early interpretations. Thus, after this first trial, we concluded that the results were positive. Therefore, based on these results, we decided to further validate the method and compare it to traditional methods. Figure 7.2: Examples of code cards In the remainder of this section, we will first give an overview of the set-up of the study that we used to validate DREAM. Then we will describe how DREAM was used for this dataset. Note that in the results section our focus is on the comparison between DREAM and the counting predefined events method. Therefore, we only present results on the effectiveness, reliability, validity and generalizability of the method Study design The study was performed with the FROG robot tour guide in the Hall of Festivities in the Royal Alcázar in Seville (Spain). The tour in the study was a prerecorded explanation, remote controlled, English spoken, guided tour given to visitors who were interested in the robot and the story told by the robot. The goal of the study was to gain insight into the visitors reactions to different types of robot behavior (to simplify the study in this chapter, we call the different behaviors: behavior type A and behavior type B). The tour given by the robot consisted of 5 explanations at points of interest in the Hall of Festivities in the Royal Alcázar in Seville (Spain). When visitors entered in the hall, the robot stood in the starting place and started the tour by welcoming the visitors and gave some general information about the room. Then the robot visited three other points of interest in the room and gave information about these points. After the fourth explanation at the point of interest, the robot drove back to its starting point, informed the visitors that the tour had finished and wished them a nice day. The tour was started again when new visitors had entered the room. 146

163 The study was performed in a real life setting with uninformed naïve visitors, which implied that we had to sometimes slightly deviate from the research procedure. Although, the robot had predefined places to give explanations, it sometimes had to stop close to the defined places. This was because people walked or stood in front of the robot. Another reason to deviate was when the robot lost the attention of all the visitors who followed the tour. In such situations it drove back to the starting place and started a new tour when new visitors joined. A fixed camera recording from a mid-shot perspective was used to collect the rich humanrobot interaction data. The camera recordings gave an overview of all events that occurred in the research area. On these recordings, the whole tour of the robot was visible. The cameras mounted on the robot were not used to collect data, because these would not give a good overview of what happened in the room, for instance when visitors stood behind the robot. By using a fixed camera we could observe the reactions of visitors who stood next to or behind the robot and visitors who stood quite far away from the robot while they kept showing interest in the robot. Further details about the study and the results of the study are presented in chapter 5 of this thesis Use of DREAM Creation of sequences Our main research question for the study was: Do visitors respond more positively towards robot behavior type A compared with behavior type B? First, based on this research question we decided to focus on the moments that the robot offered information about an exhibit. This would provide us information about how changes in orientation of the robot might affect orientations and formations of the visitors. Next, we determined the level of detail we needed to analyze the data. To be able to yield a good impression of the (change in) attention of the visitors and orientation and formation of the visitors during the story, not all video data was needed. Within this study we were, for example, not interested in the moments that the robot drove from one point of interest to the next. Thus, to create the sequences, we decided to use stills of the beginning, the middle and the end of a 20 second explanation of the robot at an exhibit. 7 From the video recordings, we made sequences for each explanation at a point of interest of the robot. These sequences consisted of an image of the beginning, the middle and the end of the explanation. Due to the real world unstructured setting, some deviations in the tours occurred. When sequences were incomplete as a result of these deviations, we left them out of analysis. A total of 144 sequences were created, 73 sequences in the condition that the robot used behavior type A and 71 sequences in the condition that the robot used behavior type B. See Figure 7.3 for an example of a sequence that was created for the analysis. 147

164 CHAPTER 7 Data Reduction Event Analysis Method Figure 7.3: Example of sequence to be analyzed Creation of codes All sequences of screenshots were printed, each sequence on one sheet. These sheets were used by two annotators to discuss and compare them. In this discussion, the annotators verified whether their interpretations of the sequences were similar. Then the annotators started to physically categorize the sequences that showed similar events. During categorization, the annotators sometimes had to discuss their interpretations to come to an agreement for a category or to create a new category. Together the two annotators checked all categories for accidentally misplaced sequences. Further, large categories were split into several smaller categories, and two or more small categories were combined in some larger categories. This resulted in 53 categories. Subsequently, the categories were given names and short descriptive and distinctive textual explanations, which converted them into codes. These codes we clustered in 10 code categories, to collect all codes that had similar meanings (e.g. formation of the group). Annotating the data Before annotating the complete set of sequences, the codes were used by the primary annotator and a third annotator to calculate an inter-rater reliability over a sample of 25% of the sequences. Before starting to apply annotations to the data, the third annotator was informed and trained in a discussion with the primary annotator. From discussions between the primary annotator and the third annotator was found that some of the codes were not yet clear, complete or distinctive. These issues were solved and the first and the third annotator annotated the same 25% of the sequences to be able to calculate an inter-rater reliability. The found inter-rater reliability was κ=0.61 (Fleiss Kappa), which indicates that the codes applied were considerably consistent between annotators (Sim & Wright, 2005). To prepare the full dataset for analysis, the primary annotator annotated the full dataset. In total, 993 annotations were applied to the full dataset. Note: We expect that beginners with DREAM will need some iterations to define the codes, while experts will more easily be able to define the clear and distinctive codes. Moreover, the creation of code cards and the use of them while annotating the data were very helpful for the application of the codes by other annotators than the ones who created them. 148