Task-Based Dialog Interactions of the CoBot Service Robots

Transcription

1 Task-Based Dialog Interactions of the CoBot Service Robots Manuela Veloso, Vittorio Perera, Stephanie Rosenthal Computer Science Department Carnegie Mellon University Thanks to Joydeep Biswas, Brian Coltin, Tom Kollar, Daniele Nardi, Robin Soetens, and Yichao Sun Abstract As we incorporate autonomous service robots into our environments, we realize the need and opportunity for them to interact with humans through speech and language. Our CoBot robots incorporate such natural interaction within their taskoriented behaviors. Our robots combine perception, planning, execution, natural spoken interaction, and learning to accomplish a variety of service tasks. We believe the combination of these multiple features to be well tuned with the varied interests of this JGC60 Festschrift celebration, and are very pleased to present a brief overview of the dialog interactions in particular. 1 Introduction These days, in the Gates-Hillman building, it is common to see a CoBot robot roaming around executing requested service tasks (see Figure 1). Our research and continuous indoor deployment of the CoBot robots in multi-floor officestyle buildings (Biswas and Veloso, 2013) provides multiple contributions, including: robust real-time autonomous localization (Biswas et al., 2011), based on WIFI data (Biswas and Veloso, 2010), and on depth camera information (Biswas and Veloso, 2012); symbiotic autonomy in which the deployed robots can overcome their perceptual, cognitive, and actuation limitations by proactively asking for help from humans (Rosenthal et al., 2010; Rosenthal et al., 2011b), and, in ongoing experiments, from the web (Kollar et al., 2012; Samadi et al., 2012), and from other robots (Aguero and Veloso, 2012; Hristoskova et al., 2013); human-centered planning in which models of humans are explicitly used in robot task and path planning (Rosenthal et al., 2011a); semi-autonomous telepresence, enabling the combination of rich remote visual and motion control (a) CoBot-1 (b) CoBot-2 Figure 1: Two of the CoBot robots. (The hardware was designed and built by Michael Licitra.) with autonomous robot localization and navigation (Coltin et al., 2011a); web-based user task selection and information interfaces, also as input to creative multi-robot task scheduling and execution (Coltin et al., 2011b). Our robots purposefully include a modest variety of sensing and computing devices, including the Microsoft Kinect depth-camera, vision cameras for telepresence and interaction, and a small Hokuyo LIDAR for obstacle avoidance and localization comparison studies (no longer needed or present in our most recent CoBot-4). Finally, and of particular relevance to this paper, the robots include a touch-screen and speech-enabled tablet, with its microphones and speakers, to support the dialog-based human-robot interaction. The CoBot robots perform multiple task types: A single-destination task, in which the user asks the robot to go to a specific location the Go-To-Room task and, in addition, possibly to deliver a given spoken message the Deliver-Message task;

2 An item-transport task, in which the user requests that the robot retrieve an item from a location, and deliver it to a destination. This Transport task also acts as the task to accompany a person between locations. A person-guiding task, the Escort task, in which the robot waits for a person at the elevator hall on the floor of the destination location, and guides the person to the location; and the Visitor Companion task, in which the robot accompanies an all-day visitor. A semi-autonomous Telepresence task, in which a user remotely selects destinations on a displayed robot view or map, to which CoBot autonomously navigates. The robot s motion and camera is controlled through a rich web interface (Coltin et al., 2011a). The above tasks are equivalent from a navigational point of view, as they are achieved by the same navigation planner generating plans to reach destinations in the building and to pickup or deliver items at locations (see Figure 2). (a) Navigation (b) Object Delivery Figure 2: Snapshots of CoBot s execution For the performance of service tasks over their long lifetime, the robots need to interact with humans in their environments. We have researched and developed multiple modalities for the human-robot interaction, namely through a touch-screen based interface, through restrictedvocabulary speech interaction via a dedicated headset, and through a general microphone-based, open-environment, and unrestricted-vocabulary input. All the interactions are assumed to be related to the tasks the robot can perform. We briefly present three aspects related to language and speech. We first present the interaction within the Visitor Companion task. We then present the learning of grounding locations through speech interaction with human task requesters. Finally, we discuss our ongoing work on the processing of complex command sentences. 2 Companion for All-Day Visitors The All-Day Visitor-Companion task, developed as CoBot s first complete task (Rosenthal, 2012), offers an interesting opportunity for the robot to naturally interact with a human, as a visitor does not know or has limited knowledge of the overall environment. The robot therefore has the chance to inform the visitor about locations, people, and research, producing an interaction useful to the human, consisting of two main modules: the core interaction to (i) provide information about the day s schedule and visit, (ii) inform the visitor about the next appointment, and (iii) escort the visitor to meeting locations. the elective interaction to (i) offer extra unsolicited information, (ii) respond to requests from the visitor, and (iii) handle schedule delays or changes. The core interaction algorithm is a predefined parameterized policy, which gets instantiated with the specific given schedule and hosts. The policy includes states and interaction actions, where the states are detected by the robot s localization and its internal timing, and the actions consist of informative acts. The robot proceeds by executing the interaction per the instantiated policy (Rosenthal et al., 2010). The elective interaction is also a parameterized policy, but gets instantiated from spoken input from the visitor. This All-Day Visitor-Companion task uses a predefined command-like language. The input affects both the underlying task policy and future interactions with the visitor. An announcement of a delay, in particular, is handled through a lack of the robot receiving that the visitor is ready to continue its visit within the expected meeting end time. The robot could take the initiative of interrupting the meeting, though it currently does not. Further elective interactions include providing unsolicited information that matches the visitor s gathered research interests, modeled as broad keywords matching the robot s knowledge base of locations. Finally, the visitor can initiate requests for more information about a meeting, as well as for other services such as printing papers and retrieving coffee between meetings. The robot proceeds, adding tasks to its own schedule in accordance with both the requests and the visitor s meeting schedule (Rosenthal et al., 2010).

3 Figure 3: Example trace of the All-Day Visitor-Companion path. Snapshots: a) CoBot leads Alice to her first meeting. b) CoBot requests assistance from a staff member to prepare a cup of coffee. c) Alice graciously accepts her cup of coffee during her meeting. d) CoBot notifies Alice of a lab of interest on their way to a meeting. In this task, we have successfully used a headset for the All-Day Visitor, as the voice remains the same for the duration of the visit. In our tests with state-of-the-art speech recognition systems, we also use a brief initial training phase for the visitor with a limited vocabulary confined to the theme of the visit. Table 1 shows an illustrative trace of CoBot s interaction with an All-Day Visitor, as also shown in Figure 3. This trace represents one of many possible sequences of events and interactions. In general, both the visitor and CoBot can initiate dialog. The robot cannot predict most of the visitor s needs, so the visitor makes requests explicitly. However, CoBot maintains state about the visitor (Rosenthal et al., 2010). Using the dialog history, CoBot can determine when it is appropriate to offer an amenity such as coffee if the visitor has not requested it recently. If the visitor makes a request that CoBot cannot answer, CoBot has the ability to perform Internet searches and to request help from other humans through dialog or (Samadi et al., 2012; Coltin et al., 2011b). We have also researched having the robot generate an interesting dialog, during its all-day interaction with the visitor. We have, in particular, investigated two different techniques to keep the interaction engaging by reducing repetitiveness. First, rather than repeating the same notifications about the same locations throughout the day, CoBot increases the level of detail of the information given Initial State - CoBot and the visitor, Alice, start in the initial location Room 3412, and both know of the meeting schedule. Alice spends 5mn training CoBot s speech recognizer. CoBot explains that it will help her get to each meeting. Visitor: Alice; Interests: Robotics; Schedule: 9:00-10:00 AM - Pat Smith, Room :00-11:00 AM - Chris Jones, Room :00-12:00 PM - Dana Adams, Room :50 AM - Alice asks CoBot for a cup of coffee. Elective: Visitor Request - CoBot plans the subtask of getting coffee after navigating Alice in time to her meeting in Room 3123 (Figure 3a). 9:00 AM - CoBot and Alice arrive at Room Elective: Task Execution - CoBot executes the plan to get coffee while Alice is in the meeting. It dialogs with a human for help to get coffee. (Figure 3b). 9:55 AM - Alice tells CoBot she s ready. Elective: Task Execution - CoBot announces it has the coffee and the Visitor takes it. Core: Schedule - CoBot announces the next meeting room and navigates to Room :00 AM - Alice asks about host Chris Jones. Elective: Visitor Request - CoBot stops and displays the host s website. 10:05 AM - CoBot notifies Alice about a robot lab. Elective: Unsolicited Notification - Based on Alice interests (Robotics), CoBot describes a robot lab on the way (Figure 3d). CoBot and Alice arrive in time to Room :00 AM - CoBot s Dana that Alice is late. Elective: Task Delay - CoBot reached a timeout of waiting for Alice. CoBot s Dana that Alice is late. When she is ready, CoBot navigates to Dana s office. Table 1: Illustrative human-robot interaction at specific locations, as a function of the number of times it visits the locations. Second, our future work in CoBot includes probabilistically choosing an utterance from a set of synonyms so that similar

4 notifications do not sound repetitious, a technique we have used previously for the robot soccer CMCast commentators (Veloso et al., 2008). As there are clearly many ways for CoBot to respond to a request, offer help, and ask for help, it can vary the utterances to ensure that the dialog does not become boring and predictable, by weighing the responses by multiple factors, e.g., the frequency and recency of the utterance, and possibly feedback from the visitor. 3 (a) (b) (c) (d) (e) (f) Learning from Spoken Interaction In speech-based interaction, users solicit tasks and interact with the robot through open speech. In addition to the clear speech understanding challenges, the main issue is to extract a task request and its arguments from the spoken language. To attack such a challenge, the robot dialogs with the human and is capable of learning groundings of language to action types and arguments, namely locations of language references. Such groundings are learned and accumulated in a knowledge base. Figure 4 shows a sample of the learned knowledge base, which contains the learned groundings (currently thousands) for language references to actions and to arguments (Kollar et al., 2013). Not shown in the snapshot are the learned object groundings acquired from accessing the web to determine the most probable location of an object, as requested to be transported (Samadi et al., 2012). Figure 4: A sample snapshot of the knowledge base learned from interaction with humans. The knowledge base is used by CoBot to infer the action and parameters of the action from the spoken dialog. When the language cannot be resolved by using the learned facts, the robot engages in further dialog or access to the web, and learns new facts. Figure 5 shows snapshots of the spoken interaction with the robot. The CoBot Task Executor executes the actions to ask for help in performing Figure 5: The user requests a task through spoken language, and the action parameters are identified and processed by the Task Planner and Task Executor with the corresponding Ask Help actions. transport tasks, solicited through speech-based requests. In Figure 5(a), the user requests a transport task of an object to a specific location. In Figure 5(b) and Figure 5(c), after speech understanding, access to its knowledge base for action and parameter grounding, and actions to ask for help for parameters, including confirmation from the user on the robot s understanding, the robot has filled in all the parameters of its action primitives, and proceeds planning and executing the route to its pickup location. In Figure 5(d) and Figure 5(e), the robot asks for help for actuation to put the object in its basket and asks for help for the condition of detecting that the object is in the basket; and in Figure 5(f), the robot autonomously navigates to the destination location and delivers the object, asking for help for the actuation of removing the object from its basket. The speech-based interaction allows for users to request tasks and for any person to interrupt the robot and have the robot answer common questions about its status and capabilities. Moreover the speech interface enables the scheduling of a task without the need to access a computing device, and provides the ability to dialog to clarify the spoken request.

5 4 Requests with Complex Sentences As discussed, CoBot is able to process task requests as single spoken commands, e.g., Go to the lab or Bring me a soda from the kitchen. Moreover, the robot can engage in a dialog with the user if some part of the command is not clear or unknown, as illustrated in Figure 6. USER: Take me to Jaime s office. COBOT: I did not understand, should I go somewhere? USER: Yes COBOT: Ok, where should I go? USER: To Jaime Carbonell s office COBOT: Where can I find it? Please spell the room number USER: 6271 COBOT: I am going to room Is it correct? USER: Yes Figure 6: Direct-request task dialog CoBot is able to understand single commands, but we understand that spoken interactions about tasks and requests may be more complex. A user may want to request that the robot perform a sequence of tasks, or a user may not be sure about some part of the command, or the user may want to give different options to the robot. For instance, a user could ask Look for Professor Carbonell in his office or in the meeting room. We are pursuing research to handle this kind of complex command, concretely as sentences with conjunction and disjunction. To understand a spoken command, our goal is to map input sentences to one or more semantic frames representing the robot s task. When a command is given to the robot, it is analyzed using a free-form speech recognizer. The output of the ASR is a set of strings representing possible transcriptions of the command. To recover a frame from these strings, the algorithm needs to parse each sentence and extract an action and a variable number of arguments. For actions, we are interested in the referring expressions for tasks that the robot can execute such as go to or bring me ; the arguments can be of three different types: object, person, and location. An example of a parsed sentence is shown in Figure 7. The model to obtain such parses was trained using a conditional random field (CRF); the features used to train this model are binary and include both the words and the part of speech for current, previous, and next word. Bring me Action a cup of coffee Object from the kitchen Location Figure 7: Parsed sentence This model enables the algorithm to recover the role of each chunk in a sentence, but it also needs to recover the frame structure and identify how many frames there are in a sentence. To do this, we trained a second model, still using a CRF, where in addition to features used to train the first model, we used the label from the first parse. The goal of this second model is to extract a common root (RR) in a sentence and a set of conjunctions ( ) or disjunctions ( ). An example of the result of this second parse is shown in Figure 8. Look for Professor Carbonell RR in his office or in the meeting room Bring me to the lab and then go back to my office Figure 8: Parsed disjunctive sentence Once the input sentences have been parsed using both models, a partial frame is filled using the frame elements contained in the part of the sentence labeled as root. This partial frame, potentially empty, is then filled once for each or chunk. Figure 9 shows both parses for a sentence, the partial frame, and the fully filled frames. [Bring me] Action [a cup of coffee] Object or [some tea] Object [Bring me] RR [a cup of coffee] or [some tea] (a) Parses Frame: Bringing Object:... Frame: Bringing Object: cup of coffee Frame: Bringing Object: some tea (b) Partial Frame (c) Frame One (d) Frame Two Figure 9: Examples of steps to parse and extract task parameters of complex sentences Once the full frames are recovered the robot can execute the required task or, if more options are available, pick one according to its scheduler. If, after analyzing all the conjuncts or disjuncts, no frame is entirely filled, the robot can still engage in a dialog similarly to the one illustrated in Figure 6.

6 5 Conclusion Our CoBot robots include task-based dialog to allow users to request tasks and have general spoken interactions with people. We have researched different levels of vocabulary, from menu-based to free speech interpreted within a task. The robot is capable of learning mappings from language to its known task components, such as locations, actions, and objects. We have also devised a solution to process complex sentences composed of conjunctions and disjunctions. Our goal is to continue enriching the understanding of spoken interaction, with the next steps being to process conditional commands, e.g., Bring me a cup of coffee, only if it s freshly brewed and time expressions, e.g., Go to the lab and wait there until someone arrives. Our overall approach to dialog understanding and generation is strongly based on the fact that the human-robot interaction and the underlying dialog are bounded by the known predefined capabilities of the robot, the grounded physical information known and perceived, and the actions the robot can perform. Such a task-based framework for the robot enables the contribution of rich spoken interaction, as bounded by the finite robot s built-in or learned task parameters. References C. Aguero and M. Veloso Transparent Multi-Robot Communication Exchange for Executing Robot Behaviors. In Int. Conference on Practical Applications of Agents and Multi-Agent Systems (PAAMS 2012), pages J. Biswas and M. Veloso Wifi localization and navigation for autonomous indoor mobile robots. In Robotics and Automation (ICRA), 2010 IEEE International Conference on, pages IEEE. J. Biswas and M. Veloso Depth camera based indoor mobile robot localization and navigation. In Proceedings of ICRA 12, the IEEE International Conference on Robotics and Automation. J. Biswas and M. Veloso Localization and navigation of the CoBots over long-term deployments. International Journal of Robotics Research, 32(14): , December. B. Coltin, J. Biswas, D. Pomerleau, and M. Veloso. 2011a. Effective semi-autonomous telepresence. Proceedings of the RoboCup Symposium, pages , July. B. Coltin, M. Veloso, and R. Ventura. 2011b. Dynamic user task scheduling for mobile robots. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence. A. Hristoskova, C. Aguero, M. Veloso, and F. De- Turck Heterogeneous Context-aware Robots Providing a Personalized Building Tour. International Journal of Advanced Robotic Systems, January. DOI: / T. Kollar, M. Samadi, and M. Veloso Enabling robots to find and fetch objects by querying the web. In Proceedings of AAMAS 12, the Eleventh International Joint Conference on Autonomous Agents and Multi-Agent Systems. T. Kollar, V. Perera, D. Nardi, and M. Veloso Learning Environmental Knowledge from Task- Based Human-Robot Dialog. In Proceedings of ICRA 13, the IEEE International Conference on Robotics and Automation, June. S. Rosenthal, J. Biswas, and M. Veloso An effective personal mobile robot agent through symbiotic human-robot interaction. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, volume 1, pages S. Rosenthal, M. Veloso, and A.K. Dey. 2011a. Is someone in this office available to help me? Journal of Intelligent & Robotic Systems, pages S. Rosenthal, M. Veloso, and A.K. Dey. 2011b. Task behavior and interaction planning for a mobile service robot that occasionally requires help. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence. S. Rosenthal Human-Centered Planning for Effective Task Autonomy. Ph.D. thesis, Computer Science Department, Carnegie Mellon University, Pittsburgh, PA, May. Available as technical report CMU- CS M. Samadi, T. Kollar, and M. Veloso Using the Web to Interactively Learn to Find Objects. In Proceedings of the Twenty-Sixth Conference on Artificial Intelligence (AAAI-12), Toronto, Canada, July. M. Veloso, N. Armstrong-Crews, S. Chernova, Crawford. E., C. McMillen, M. Roth, D. Vail, and S. Zickler A team of humanoid game commentators. International Journal of Humanoid Robotics, 5(3): J. Biswas, B. Coltin, and M. Veloso Corrective gradient refinement for mobile robot localization. In Intelligent Robots and Systems (IROS), 2011 IEEE International Conference on. IEEE.