An On-Going Evaluation of Domestic Robots

Transcription

1 An On-Going Evaluation of Domestic Robots Position paper ABSTRACT Gabriella Cortellessa Institute for Cognitive Science and Technology Italian National Research Council Via S. Martino della Battaglia 44, Rome, Italy In this position paper we describe an on-going effort to provide an in-depth and cross-cultural evaluation of how elderly users perceive robotic systems for domestic cognitive support. Our work is grounded on two implemented smarthome prototypes, namely the RoboCare Smart Home developed in Italy, and the PEIS Home developed in Sweden. The former project has provided a testbed for an a- posteriori evaluation of smart home technology with Italian user groups. The presence in Sweden of the PEIS Home, a system which shares numerous commonalities with the RoboCare Smart Home, gives us the opportunity to extend these results by (1) providing a cross-cultural perspective on the perception of smart home technology, and (2) lay the foundations for a live, Wizard of Oz based evaluation within the PEIS Home. 1. INTRODUCTION In this brief paper, we report our current strategy to push the envelope in the development of acceptable and effective HRI schemes for social assistive robots. Specifically, we summarize an on-going evaluation effort which started last year within a project called RoboCare [4] developed at ISTC- CNR with Italian user groups. The availability of a prototypical smart home (which was the primary outcome of the RoboCare project see made it possible to study aspects related to the interaction between users and a set of intelligent artifacts in the smart home. These artifacts include a mobile robot, which is the principal mediator of all human-system interactions. After providing some points of discussion on the results obtained so far in the RoboCare evaluation, we describe our current effort in extending this evaluation with a cross-cultural assessment performed on Swedish user groups. This further evaluation is currently in set-up phase, and is grounded on another robotically-rich environment, namely the PEIS Home [11], which focuses on the development of an ecology of Physically Embedded Intelligent Systems (PEIS). Both RoboCare and PEIS were initially conceived to focus primarily on technology development. The use of similar AI and robotics based technology for supporting elderly people at home has been addressed in various other research projects in the last years [8, 9]. More recent research has been increasingly focusing on Cognitive Systems to produce aids that enhance human cognition capabilities [7]. The state-of-the-art in robotics today is bringing about an in- Amy Loutfi and Federico Pecora AASS Mobile Robotics Lab Dpt of Technology, Örebro University S Örebro, Sweden fpa@aass.oru.se, amy.loutfi@tech.oru.se creasing focus on human-robot interaction in general and on social assistive robotics in particular. Indeed, as the robotic, sensory and software infrastructure began to enter a more stable stage of development in RoboCare and PEIS, the question of evaluation naturally came to our attention. In RoboCare, a team of psychologists was employed to drive technology development since its early stages, although its recommendations and the technological possibilities appeared to be misaligned in numerous occasions. In later stages of development, the availability of a complete prototypical system was employed to produce an a-posteriori evaluation with Italian user groups. This evaluation underscored several interesting indications concerning the perception of an assistive robot and the smart home s capabilities. At the same time, the issue of evaluation emerged in the PEIS Home, and an investigation into possible evaluation strategies and methodology began. This lead to the idea of performing the same a-posteriori evaluation that was carried out on the Italian user groups with Swedish users, in order to assess the extent to which the findings could be generalized across different European cultures. In the following paragraphs, we briefly summarize the principal findings of the RoboCare evaluation, and put forth some hypotheses on the results of the latter evaluation. We conclude with a brief discussion on further possibilities for continuing the evaluation effort in both the PEIS and the RoboCare contexts. 2. EVALUATING USER PERCEPTIONS OF SOCIAL ASSISTIVE ROBOTS The emphasis in social assistive robotics is to support human users through social rather than physical interaction [5]. A key aspect in the development of social assistive robots is the evaluation of social interaction between human users and robotic agents. A number of guiding principles exist in the literature. For example, [10] highlights how observation and behavioral analysis of human-robot social interaction in real environments is necessary in order to take into consideration all the divergent factors pertaining to the design of social robots. The design of social robots also raises a number of ethical issues that need to be discussed within the research community to provide guidance to system designers. [14] considers some of the ethical implications of humanrobot interaction, mainly related to the kind of authenticity we require in our technology as well as to the choice of the most appropriate relationship between children/elders and

2 relational artifacts. 2.1 Previous Results: the RoboCare Evaluation The RoboCare project examines some of the relevant factors for the design of assistive robots. RoboCare has involved research groups with different backgrounds with the goal of investigating how state of the art AI and robotics techniques can be combined to create new domestic services for elderly people. The project has produced a prototype of integrated home environment (the RoboCare Smart Home) consisting in a robotic interactive agent, some vision-based sensors for continuous monitoring, and additional intelligent systems that store and reason upon knowledge about the assisted elder s scheduled activities. A multi-agent coordination algorithm guarantees the coherence of the behavior of the whole environment. This provides a functional cohesive which invokes the smart home s services so as to preserve the person s safety and to provide suggestions. The details of how these different interactive functionalities are obtained and integrated are described in [4]. The RoboCare Smart Home includes a mobile robotic platform with interaction capabilities 1. The multi-agent environment interacts with the assisted person through this robot, which we have called a robotic mediator (see Figure 1). The base on top of which the robotic mediator Figure 1: The RoboCare Smart Home and the robotic mediator. is built consists in a Pioneer platform. The mobile platform is equipped with additional sensors, namely a laser range finder, a stereo camera and an omni-directional camera, as well as additional computational resources consisting in two laptops, one for on-board sensor processing and navigation and one for human-robot interaction. The robot is endowed with verbal user interaction skills: speech recognition is achieved with the Sonic speech recognition system (University of Colorado) 2, while speech synthesis is driven by a simple text-to-speech system. Since its beginning, RoboCare has raised a number of user acceptability challenges. In particular one, also re- 1 In the spirit described in [5] the Smart Home is an example of Social Assistive Robot, a concept which can be distinguished from Social Interactive Robot [6] because its main task is to monitor and assist the person rather than simply interacting with him/her. 2 For details, see cslr.colorado.edu/beginweb/speech_ recognition/sonic.html ported in [13] has been paramount in our work: what are the circumstances in which people accept an assistive robot in their environment?. Other important questions we have strived to answer (or at least investigated) are how should an elder user communicate with a robot?, should the robot look like a human being?, and, last but not least, are robots useful in the domestic environment?. RoboCare set out to provide a first investigation into the above questions. The full results of this investigation as well as the details of the methodology are reported in [2, 3]. The evaluation was performed through eight short films (ranging from about 30 seconds to little more than one minute) depicting potential interactions occurring in a real domestic environment between an elderly person and the robotic mediator. Various experimental manipulations of the features of the robotic agent were employed (e.g., two variants of the robot, one in which it had a 3D representation of a face, and one with no anthropomorphic traits). In the eight scenarios we presented common everyday life situations in which the robot provides cognitive support to the elderly person. Scenarios referred to critical areas of every day living, including (a) management of personal/environmental safety, (b) healthcare, (c) reminding events/deadlines, (d) support to activity planning, (e) suggestions. A sample movie is available at After viewing the videos, respondents were asked to answer a questionnaire on the eight scenarios regarding (a) the likelihood of the eight situations, (b) the physical aspect of the robot, (c) an emotional scale for evaluating the possible presence of the robot in their own home, and (d) the perceived health of the interviewee. The evaluation was performed on forty male and female users, aged between 56 and 88. Among the most interesting results, we highlight the following. Perceived usefulness. The proposed scenarios were evaluated as significant in the everyday life of respondents, and the Smart Home s support was evaluated as both useful and appreciated. In particular, we found a higher level of perceived utility for personal safety scenarios, and a lower level for instances in which the system suggests non-critical activities, such as going for a walk or reminding someone s birthday. General evaluation of the robot. Mean scores referring to key features of the robot were calculated. On the whole, the robot emerged to be positively evaluated with respect to physical aspect, interactive behavior and communication modalities, level of integration with the domestic environment, absence of perceived intrusion/disturbance in everyday life and routines, and personal advantages of having such a device at home. Similarity to human beings. A comparison between experimental conditions showed a stronger preference for the least anthropomorphic version of the robot. In particular, a significant difference in favour of this version of the robot emerged with respect to physical aspect, integration with the home environment, and advantages provided by the robot at home. No significant difference between the different versions emerged with respect to interactive behavior and communication modalities and level of perceived intrusion/disturbance.

3 Influence of psychological variables. Our analyses outlined a clear influence of both perceived health and worry about loss of cognitive competence on evaluations expressed by elderly people. Elderly people perceiving better health conditions expressed more positive evaluations about the integration of the robot in the domestic environment, and perceived the robot as less intrusive in everyday life than did those who complained about worse health conditions. No significant difference emerged with respect to physical aspect, interactive behavior and communication modalities, nor did poorer health affect the perception of the advantages provided by having the robot at home. Overall, although the perceived utility of and the expressed preference towards a proactive robot intervening in case of an emergency is higher than in other scenarios, a positive reaction to different interactive situations was undoubtedly found. This is in line with current research in aging, in which the key role of compensation strategies to manage the loss of personal resources is underscored. In addition, even though elderly people do not think that a robotic agent living in the domestic environment can be as useful for uncompelling activities as it is perceived to be in safety-related contexts, they do not perceive it as completely out of place when supporting the former. This result shows an openness of elderly people towards a variety of functionalities, at least to some extent: they may be rather unfamiliar with some of the smart home s technology, but certainly not against it a-priori. Lastly, we should mention that a further evaluation with Italian users is currently in progress, and is aimed at assessing the perception user have of the RoboCare Smart Home with no robotic mediator. Specifically, the eight scenarios are presented in a smart home without an embodied relational agent, the interaction between the assisted person and the system occurring through environmental microphones and loud speakers. Given that the RoboCare robot is not endowed with manipulatory capabilities, we are interested in understanding the value of embodiment in the interaction between users and the intelligent cognitive support system. Although the evaluation is currently in its final stages, preliminary results show that the embodied version of the RoboCare Smart Home maintains overall preference. Specifically, the overall preferred configuration among the proposed scenarios is the one in which the robot is present and has less anthropomorphic traits. 2.2 Evaluation in PEIS The concept of PEIS, originally proposed in [11], combines insights from the fields of ambient intelligence, autonomous robotics and sensor networks (see Figure 2) to generate a new approach to the inclusion of robotic technology into smart environments. In this approach, advanced robotic functionalities are achieved through the cooperation of many simple robotic, sensory and software components. PEIS builds upon the principle that any device incorporating some computational and communication resources, and possibly able to interact with the environment via sensors and/or actuators, is included as a building block within a unifying framework. A PEIS can be as simple as a toaster and as complex as a humanoid robot. In general, we define a PEIS-Ecology to be a set of inter-connected software components, called PEIS-components, residing on distributed Figure 2: The PEIS approach combines insights from several fields. physical entities. Each component may include links to sensors and actuators, as well as input and output ports that connect it to other components in the same or another PEIS- Ecology. The framework connecting all PEIS is a uniform communication and cooperation model. This allows the exchange of information among PEIS, and can cope with them joining and leaving the ecology dynamically. Also, a common cooperation schema allows each participating PEIS to use functionalities from other PEIS in the ecology in order to compensate or to complement its own. Figure 3: Two views of the PEIS environment. The approach underlying user interaction implemented in the PEIS Home is based on the idea that the human is just another PEIS in the PEIS-Ecology. As any other PEIS, the human can be made aware of which functionalities can be provided by the environment, and decide to use any one of those. In the reverse direction, the PEIS-Ecology may rely on the human user to provide certain functionalities, like choosing between alternative courses of action. In practice, this intuitively appealing viewpoint is complicated by a number of issues. First, the set of tasks that the whole PEIS-Ecology could perform is, in general, very large. These tasks can be at very different levels of abstraction and complexity, ranging from basic tasks which are performed by a single PEIS, like dimming the lights in the bedroom; to tasks which involve the coordinated activity of several PEIS, like collecting all the expired items from the fridge and putting them in the garbage. Since the set of possible tasks is so large, the human user cannot be presented with the whole set at all times. Instead, an effective interface should filter the tasks to be presented to the user according to their relevance to the current context. In addition to the possibility of starting a large number of tasks, the human user may also be interactively involved in the execution of a task: the ecology may ask the user to

4 provide some functionalities which are needed for the performance of the task. Typically, these will be cognitive decision making functionalities, like deciding which ones of the expired items in the fridge should be reordered. A preliminary user interface design which attempts to overcome these issues in the PEIS Home is reported in [1]. As in RoboCare, the elementary functionalities provided by the PEIS Home can be combined to obtain high-level cognitive support services. While there exists a strong difference in infrastructure, the two systems are both conceived for providing support services in safety and healthcare management, memory-aides, activity planning support and providing suggestions on daily activities. For this reason, an evaluation of the scenarios originally scripted on the Robo- Care system would provide results that can be used to drive further high-level service implementation in the PEIS Home. In addition to this, such an evaluation provides a rare opportunity to compare the bias of different cultural backgrounds on the perceived benefit of the various features which can be implemented in the smart home. The evaluation will focus on a user group with the same characteristics as the RoboCare sample, most importantly with respect to parameters such as number of participants, age, educational and social background. Also, the administration and evaluation methodology will be kept consistent. Specifically, all material has been reproduced exactly to be as coherent as possible between the two evaluations: the questionnaires were translated and the order of questions preserved; the videos were subtitled 3 in Swedish, and the order of presentation of the eight scenarios will be preserved. 3. DISCUSSION The anticipated outcome of the cross-cultural evaluation is a set of indications that disprove or confirm the general idea that the perceived advantage of employing current state-ofthe-art smart home technology does not vary significantly among Italian and Swedish societies. There are, indeed, reasons to suspect either result. On one hand, the uptake of relatively new technologies is very different in the two countries. In Italy, just 2.8% of households of people aged over 65 has access to the Internet, while 45% of them own a mobile phone; in Sweden, the percentage of households having access to the Internet at home was 73% in 2005, and the percentage of individuals in the age range using the Internet at least once a week is 51% 4. This may suggest a very different level of confidence in smart home technology, and possibly different expectations with respect to the potential benefit of living in a smart home. However, there are two factors that add complexity to the comparative analysis. First, the fact that the analysis is concerned with very specific aspects of acceptability eight different scenarios, in which the system supports the elderly person in a variety of specific tasks. The level of appreciation of the different 3 A short pre-evaluation on one elderly user was performed to ascertain whether subtitling or dubbing would be a preferred method for conveying the spoken language in the videos. As dubbing is virtually absent in Sweden, the user reported that reading subtitles seemed positively more natural than watching a dubbed scene, thus subtitling was chosen as the method of translation. 4 Data taken from the Supporting Policy Development for e- Inclusion project see the the EU country reports section on features most probably depends on different aspects of the user s background which are not necessarily related to their acquaintance with ICT. Secondly, the user base in both experiments represents a small sample of a very specific part of society, namely middle class elderly people with an adequate level of schooling and previous jobs in the service sector (e.g., teaching). Lastly, another factor which may distinguish Swedish users perception of smart home technology lies in different national elderly care culture and policy [12]. In Sweden, 98% of elderly people lived either alone or with their spouse in 2002, while 35% of the Italian elderly lived with other family members and/or within communities in the year Even more interestingly, only 17% of the elderly in Nordic countries (Sweden, Denmark, Finalnd and Norway) received care from family members within their own household in 1992, while in 67% of the elderly residing in Southern European countries (Italy, Portugal, Spain, Greece, Ireland and Austria) found themselves in the same situation. It is reasonable to assume that these and many other social differences probably affect Swedish and Italian users perception of smart home technology in different ways. Video-based vs. live evaluation On the basis of the results from the cross-cultural evaluation, we are investigating the possibility to conduct user evaluation within the physically instantiated environment of the PEIS Home. Specifically, a possibility is to employ the Wizard of Oz (WoOz) technique, an experimental condition in which subjects interact with a system that is perceived as autonomous, but which is actually being operated or partially operated by an unseen human being. This study can contribute to overcoming the inevitable limitations of video-based evaluation: to get an accurate feel of being in the same room of a robot requires all senses (e.g., the robot makes noise, the height of the robot may be intimidating, etc.) Most importantly, comparing results of a WoOz evaluation can provide a critical assessment of video-based vs. live evaluation methods [15]. Wizard of Oz Evaluation Video sessions as preparatory material Video Based Evaluation Figure 4: Using video-based evaluation to prepare users for live evaluation in the PEIS. Finally, a third possible form of evaluation we are investigating involves the possibility to include some of the users employed for the video-based evaluation also in the live WoOz sessions (see Figure 4). Specifically, this would add the possibility to assess whether the video material and the fact that the user has thought about the situations by answering the questions can serve as a preparatory stage to curb the potentially disturbing presence of live components.

5 Acknowledgement The work referenced in this paper on evaluation in Robo- Care was performed jointly with Maria Vittoria Giuliani, Massimiliano Scopelliti and Lorenza Tiberio of ISTC-CNR. The Authors are grateful to Alessandro Saffiotti and Gion Koch Svedberg for their support. 4. REFERENCES [1] M. Broxvall, A. Loutfi, and A. Saffiotti. Interacting with a robot ecology using task templates. In Proceedings of the 16th IEEE Int Symp on Robot and Human Interactive Communication (RoMan), [2] A. Cesta, G. Cortellessa, M. Giuliani, F. Pecora, R. Rasconi, M. Scopelliti, and T. Tiberio. Proactive Assistive Technology: An Empirical Study. In Proceedings of the Eleventh International Conference on Human-Computer Interaction (INTERACT 07), [3] A. Cesta, G. Cortellessa, M. Giuliani, F. Pecora, M. Scopelliti, and L. Tiberio. Psychological implications of domestic assistive technology for the elderly. PsychNology Journal, 5(3): , [4] A. Cesta, G. Cortellessa, F. Pecora, and R. Rasconi. Supporting Interaction in the RoboCare Intelligent Assistive Environment. In Proccedings of AAAI Spring Symposium on Interaction Challenges for Intelligent Assistants, [5] D. Feil-Seifer and M. Matarić. Defining Socially Assistive Robotics. In Proceedings of IEEE International Conference on Rehabilitation Robotics (ICORR-05), Chicago, IL, pages , [6] T. Fong, I. Nourbakhsh, and K. Dautenhahn. A survey of socially interactive robots. Robotics and Autonomous Systems, 42(3 4): , [7] K. Myers. Calo: Building an intelligent personal assistant. Invited Talk. The Twenty-First National Conference on Artificial Intelligence and the Eighteenth Innovative Applications of Artificial Intelligence Conference (AAAI-06), [8] J. Pineau, M. Montemerlo, M. Pollack, N. Roy, and S. Thrun. Towards Robotic Assistants in Nursing Homes: Challenges and Results. Robotics and Autonomous Systems, 42(3 4): , [9] M. Pollack. Intelligent Technology for an Aging Population: The Use of AI to Assist Elders with Cognitive Impairment. AI Magazine, 26(2):9 24, [10] S. Sabanovic, M. P. Michalowski, and R. Simmons. Robots in the wild: observing human-robot social interaction outside the lab. In Proceedings of the 9th International Workshop on Advanced Motion Control (AMC 2006), pages IEEE, March [11] A. Saffiotti and M. Broxvall. Peis ecologies: ambient intelligence meets autonomous robotics. In soc-eusai 05: Proceedings of the 2005 joint conference on Smart objects and ambient intelligence, pages , New York, NY, USA, ACM. [12] G. Sundström and L. Johansson. The changing balance of government and family in care for the elderly in Sweden and other European countries. Australasian Journal on Ageing, 24(s1):S5 S11, [13] A. Tapus, M. Matarić, and B. Scassellati. The grand challenges in socially assistive robotics. IEEE Robotics and Automation Magazine, 14(1):35 42, [14] S. Turkle, W. Taggart, C. Kidd, and O. Daste. Relational artifacts with children and elders: The complexities of cybercompanionship. Connection Science, 18(4): , [15] S. Woods, M. Walters, K. Koay, and K. Dautenhahn. Comparing human robot interaction scenarios using live and video based methods: Towards a novel methodological approach. In Proceedings of the Ninth International Workshop on Advanced Motion Control (ACM 2006), Istanbul, March 27-29, pages , 2006.