Human Robot Dialogue Interaction. Barry Lumpkin

Transcription

1 Human Robot Dialogue Interaction Barry Lumpkin

2 Robots

3 Where to Look: A Study of Human- Robot Engagement Why embodiment? Pure vocal and virtual agents can hold a dialogue Physical robots come with many technical issues What is gained through a physical presence? This paper focuses on conversational gaze behaviors Does gaze contribute towards user engagement? Engagement: the process by which two (or more) participants establish, maintain, and end their perceived connection

4 Where to Look: A Study of Human- Robot Engagement Meet Mel the robotic penguin Open/close beak (synced with dialogue) Nod/turn head (head/object tracking) Flap wings up/down (beat gestures) Collagen (collaborative agent software) IBM ViaVoice (speech recognition) Single user interaction

5 Where to Look: A Study of Human- Robot Engagement The scenario was designed from human-human interactions Demonstrating and observing equipment at MERL (Mitsubishi Electric Research Labs) Videos used to develop rules for the robot gestures and dialogue script Dialogue designed to be robot controlled to help predict responses Mel designed to demonstrate IGlassware under two conditions Mover: Mel will use facial tracking to look at the user in addition to shifting gaze to the table when appropriate. Talker: Mel will only track faces until it has engaged a user in dialogue. At this point, Mel no longer shifts gaze. 37 subjects split into two groups (20 mover / 17 talker) using between-subjects design

6 Where to Look: A Study of Human- Robot Engagement M: Hi, I'm Mel a robotic penguin. U: Hi. M: What's your name? U: Sam. M: Sam, I'd like to show you a demo. OK? U: No. M: But it's really interesting. Come on. Try it! U: Okay. M: It would be really nice to know what kind of progress people are making in their dining. So Paul Dietz created a new product called I Glassware for this. By the way, Sam, do you know Paul? U: No. M: Well, Paul is a member of MERL. This demo is one of his creations. U: Okay. M: I Glassware stands for Instrumented glassware! Sam, right there <robot turns head to table> is the I Glassware cup <robot turns head to user>. And near it, is the table readout. The word re-fill on the readout means that it is empty. U: Ok.

7 Where to Look: A Study of Human- Robot Engagement Results from Questionnaires No significant effects Liking of Mel Knowledge of the demo Confidence of knowledge of the demo Significant effects Engagement in the interaction (borderline for female 4.84 to 4.48) Reliability of Mel (high for talker 5.19 to 3.84) Appropriateness of movements (significant for mover 4.99 to 4.27)

8 Where to Look: A Study of Human- Robot Engagement Results from Behavioral Observations Measure Mover Talker Test/Result Significance Interaction Time Shared Looking Mutual Gaze Talk directed to Mel Look backs overall Table Look 1 Table Look seconds seconds Single-factor, ANOVA: F(1,36)= % 35.9% Single factor ANOVA: F(1,36)= % 36.1% Single-factor, ANOVA: F(1,36) = % 73.1% Single-factor, ANOVA, F[1,36]= looks; median looks; median 12 12/19, 63% 6/16, 37.5% Single-factor, ANOVA: F[1,36]= t-tests, t(33)= 1.52 Significant difference: p < 0.01 Significant difference: p < 0.01 No significant difference: p = 0.40 No significant difference: p=0.71 Highly significant difference: p < Weak significance: One-tailed: p= /20, 55% 9/16, 56% t-tests, t(34)= No significance: One-tailed: p = 0.47

9 Where to Look: A Study of Human- Robot Engagement Conclusions Gesturing talking robots capture the user s attention more often Users respond to changes in gaze by changing their own gaze Users engage in mutual gaze regardless of condition Users direct their gaze to the robot when taking a turn regardless of condition Users aware that simply a talking head is not what they expect from a 3D robot

10 Where to Look: A Study of Human- Robot Engagement Questions on the paper?

11 On Natural Language Dialogue with Assistive Robots Defining Assistive Robots Using Yanco and Drury s 8 categories Autonomy level (provides support for the disabled or caregivers) Amount of intervention (typically designed to receive directions) Human robot ratio (typically 1:1) Type of shared human robot interaction (typically one human one robot) Decision support for operators (not used: Too varied to be useful) Criticality (High due to assisting caregivers) Time/space (synchronous and collocated) Composition of robot teams (possibly not applicable to assistive technologies)

12 On Natural Language Dialogue with Assistive Robots Why bother with Natural Language Dialogue? Graphical User Interfaces work well in other areas of HRI NLD is natural for the human and requires little to no training GUI is not always appropriate when access to hardware devices is impractical Building robots capable of NLD yields insights into human cognition

13 On Natural Language Dialogue with Assistive Robots Weaknesses in the NLD architecture Link between speech recognition and NLP ASR errors present a safety concern for assistive technologies Cannot guarantee high criticality when misrecognizing 5 out of 100 commands Their robotic guide for the blind had several failures in ASR Users engaging in side conversations Coughing and throat clearing Shared vocabulary Do target users have impaired speech that ASR cannot handle? Can the user base provide the training needed for the ASR vocabulary? Are they incentivized to undergo training?

14 On Natural Language Dialogue with Assistive Robots Weaknesses in the NLD architecture Link between NLD and speech synthesis Speech beacons can be less desirable than non-speech beacons Visually impaired users may opt for audio of water bubbles instead of speech to signify a water fountain Speech beacons do not need to be learned, whereas nonspeech beacons require training Non-speech beacons use less cognitive processing and are more easily perceived in noisy environments Hard to hold a dialogue with a third party when being given speech beacons Very little data available on the perception of speech vs. nonspeech beacons

15 On Natural Language Dialogue with Assistive Robots Weaknesses in the NLD architecture Link between DMS and the robot hardware Hardware reliability is lacking The DMS must be aware of the state of the robot to function The probability that the internal state of the DMS reflects the actual hardware depends on its reliability Lack of self awareness from various NLD design practices in HRI 1. All NLD components are developed on simulated robot hardware 2. NLD capabilities are added after the hardware is designed, developed, and deployed Lack of self awareness from dialogue management techniques State-based, frame-based, and plan-based are largely deterministic while the hardware is stochastic by nature Mismatches between what is said and the actions performed

16 On Natural Language Dialogue with Assistive Robots Example Systems Perzanowski et al. developed Coyote to interpret commands such as turn 30 degrees left/right. Concluded that numbers are difficult to understand for the ASR Huttenrauch et al. developed an office delivery robot. They claim that the speech interface is the primary interaction mode when in close proximity, however in the actual evaluation a GUI is used instead with no explanation Huttenrauch and Eklundh mention speech as an interface for a coffee service robot, it is only used for a scripted output request. Input is handled by a button to confirm the coffee is ready Billard developed the Robota doll robot for educational purposes on creating robots with social skills. IBM s ViaVoice and Microsoft s SAPI are both mentioned, but it is not clear which one is used if not both. No statistics are given on the usability of speech as input or output

17 On Natural Language Dialogue with Assistive Robots Example Systems Montemerlo et al. describe Pearl, a robotic guide for the elderly. They report problems with both speech synthesis and speech recognition, but do not give statistics on the types of errors or how they affected performance or perception. Gockley et al. describe Valerie, a Roboceptionist that uses speech output and keyboard input to avoid ASR issues. Breazeal et al. use the Leonardo platform to teach the robot the names and locations of buttons. Subjects are taught the list of allowed phrases prior to interaction to eliminate the shared vocabulary problem. The two main sources of error are speech recognition failures and failures of the vision system to recognize a gesture.

18 On Natural Language Dialogue with Assistive Robots Should a system be dialogical? Does the target user have any disabilities to prohibit use of natural language? Does speech misrecognition undermine the criticality of the assistive robot? Can speech misrecognition be overcome through user training or a hardware device? Can the target user acquire shared vocabulary? Is the use of NLD economically justified? How reliable is the robot hardware? Are speech beacons appropriate?

19 On Natural Language Dialogue with Assistive Robots When is NLD appropriate for HRI in general A = autonomy, I = operator interventions, P = how much potential exists for NLD Conjecture 1: As A goes to 1, P goes to 0 As the robot becomes fully capable of making its own decisions automatically, there is less need to interact with it through NLD Conjecture 2: As I goes to 1, P goes to 0 As the level of intervention required increases, the potential for NLD decreases since language is a slow medium. A high amount of intervention would be expressed through formal languages used by knowledgeable operators Conjecture 3: P>0, when A [a 1,1-δ 1 ], where 0<δ 1 <1, 0<a 1 <1 δ 1, and I [a 2,1 δ 2 ], where 0<δ 2 <1, 0<a 2 <1 δ 2 Potential for NLD exists when the robot is partially autonomous and requires some intervention The robot should be competent, but also interesting both physically and cognitively

20 On Natural Language Dialogue with Assistive Robots Questions on the paper?

21 Situated Reference in a Hybrid Human-Robot Interaction System This paper focuses on the detection and explanation of errors made while constructing objects with a wooden toy set The robot uses a goal inference model to select its actions based on the user s actions and utterances When it detects an error, it can explain not only what the error was, but also what can be done to correct the mistake This system is capable of reference generation to clarify which building blocks are being discussed 1. System First we will build a 5. System To build a windmill, you windmill. first need to build a tower. 2. User Okay. 6. System [picking up and holding 3. User {picks up a yellow cube, out red cube] To build the tower, unnecessary piece for a insert the green bolt through the Windmill} end of this red cube and screw it 4. System You don t need a yellow into the blue cube. cube to build a windmill. 7. User [takes cube, performs action] Okay.

22 Situated Reference in a Hybrid Human-Robot Interaction System Intell (15.6) Task 72.7 (10.4) Feelin g Constant Adaptive M-W 66.9 (15.9) Conv (13.6) Overa ll 72.1 (11.2) 74.9 (12.7) 71.1 (8.3) 66.8 (14.2) 75.2 (10.7) 71.8 (9.1) p = 0.19, n.s. p = 0.69, n.s. p = 0.51, n.s. p = 0.036, sig. p = 0.68, n.s Measure Constant Adaptive M-W Duration (s.) (62.8) Duration (turns) 29.8 (5.02) Rep requests 0.26 (0.45) Explanations 2.21 (0.63) Successful trials 1.58 (0.61) (94.6) 31.2 (5.57) 0.32 (0.78) 2.41 (0.80) 1.55 (0.74) p = 0.90 p = 0.44 p = 0.68 p = 0.44 p = 0.93

23 Situated Reference in a Hybrid Human-Robot Interaction System Questions on the paper?

24 Dragonbot