RELATIONSHIP BETWEEN TRUST AND ENTRAINMENT IN SPEECH. FA , PI Stefan Benus

Transcription

1 RELATIONSHIP BETWEEN TRUST AND ENTRAINMENT IN SPEECH FA , PI Stefan Benus AFOSR Program review, Arlington, June 13-17, 2016

2 MOTIVATION Background Speech entrainment (adaptation of communicative behavior to the behavior of the conversational partner) correlates with positive feelings of conversational in human-human dialogues Objective Test (in novel game designs) a hypothesized causal link between entrainment & trust in human-machine interactions Impact/Applications: Better understanding of entrainment-trust in human-machine dialogues In distinct populations culturally and linguistically incl. non- Germanic) In non-native English as ever increasing means of communication Probing both dynamic and static measures Affect/calibrate trust of humans toward machines by manipulating machine s prosody 2

3 OUTLINE Entrainment in human-human dialogues Entrainment & trust in human-machine dialogues Pilot findings Novel experimental designs in progress 3

4 PEOPLE Štefan Beňuš (PI) Constantine the Philosopher University in Nitra, Slovakia Agustín Gravano (co-pi) University of Buenos Aires, Argentina Linguistics, computer science Forms functions of prosody in conversation 4

5 TEAMS & COLLABORATORS Slovakia Student team at the Slovak Technical University (Computer Science) designing a computer game Research team at Slovak Academy of Sciences (Speech processing department) designing Slovak TTS & ASR system with entrainment capabilities Argentina 1 PhD student, 3 MA students USA Julia Hirschberg, Rivka Levitan 5

6 CORPUS STUDIES OF HUMAN-HUMAN SPOKEN INTERACTIONS 6

7 COLUMBIA GAMES CORPUS Describer describes the position of a target object with respect to other fixed objects on her screen; Placer tries to move his representation of the target object to the same position on his own screen. Describer Placer

8 SPEECH (DIS)ENTRAINMENT What kind of entrainment - what kind of social behavior? Cross-correlation; signed synchrony Linear regression with social variable labels from MTurk Perez, Galvez, Gravano (2016) Disentrainment may be a positive thing: A novel measure of unsigned acoustic-prosodic synchrony, and its relation to speaker engagement. Proceedings of Interspeech. 8

9 SPEECH ENTRAINMENT (ENGLISH) Results No relationship between the raw measure of entrainment (signed synchrony) and social variables Robust relationship between social variables and a measure combining entrainment and dis-entrainment (unsigned synchrony) Social variables linked to a positive perception of the speakers interaction (contributes-to-successfulcompletion, making-self-clear, engaged-in-game) show significantly positive correlations, especially for features related to pitch and intensity of the voice. Labels do not mention trust specifically, but they are consistent with the perception of trustworthiness Perez, Galvez, Gravano (2016) Disentrainment may be a positive thing: A novel measure of unsigned acoustic-prosodic synchrony, and its relation to speaker engagement. Proceedings of Interspeech. 9

10 SPEECH ENTRAINMENT (ENGLISH & SPANISH/SLOVAK) Comparable corpora of task-oriented dialogues Global and local entrainment, identical methodology Similarities Entrainment on intensity (global & local) Absence of global entrainment on pitch Differences Tendency for more disentrainment in Slovak/Spanish than in English More local entrainment on pitch for English than Slovak/Spanish Levitan, Benus, Gravano, Hirschberg (2015). Acoustic-prosodic entrainment in Slovak, Spanish, English and Chinese: A cross-linguistic comparison. Proceedings of SIGdial. 10

11 BACKCHANNELS AS ENTRAINMENT SIGNALS Backchannels (okay, mhm, ) indicate conceptual alignment (we are on the same page, go on) In English, more similar to preceding turns than other turn types on f0 & intensity (Heldner et al. 11, Levitan et al. 15). In Slovak, overall lower intensity and f0 shape similarity are used to cue alignment prosodically Benus (2016). The prosody of backchannels in Slovak. Proceedings of Speech Prosody. 11

12 FROM HUMAN-HUMAN CORPUS STUDIES OF ENTRAINMENT TO TESTING THE EFFECT OF ENTRAINMENT ON TRUST 12

13 ENTRAINMENT -?- TRUST: GOFISH 13

14 ENTRAINMENT -?- TRUST: GOFISH Originally implemented for English (last year demo), now reimplemented in python for Arg. Spanish and Slovak Some differences Entraining on speech rate & intensity (EN), only speech rate (SK/ES) Matching vs. random (EN) & relative adaptation vs. stable/disentrain (SK/ES) Levitan, Benus, Galvez, Gravano, Savoretti, Trnka, Weise, Hirschberg (2016).Implementing acoustic-prosodic entrainment in a conversational avatar. Proceedings of Interspeech. 14

15 GOFISH RESULTS English Subjects were more likely to ask the entraining avatar for advice than the dis-entraining avatar(p < 0.001) Subjects liked the entraining voice better (p = 0.053) and were less likely to call it annoying (p = 0.002). Arg. Spanish & Slovak Subjects were less likely to ask the entraining avatar for advice than the dis-entraining avatar (Spanish; p < 0.1, Slovak; p < 0.05). No effect for stable vs. entraining, and no effect of avatar identity. Levitan, Benus, Galvez, Gravano, Savoretti, Trnka, Weise, Hirschberg (2016).Implementing acoustic-prosodic entrainment in a conversational avatar. Proceedings of Interspeech. 15

16 GOFISH: DISCUSSION Different results in English vs. Spanish/Slovak might be due to Differences in methodology & implementation In non-english environment, greater prosodic variability might be valued more than plain synchrony in developing trust toward the avatar Next Implement entraining on speech rate, pitch, and intensity in a unified design for English, Spanish, and Slovak and collect data Analyze possible links between personality and trustentrainment Levitan, Benus, Galvez, Gravano, Savoretti, Trnka, Weise, Hirschberg (2016).Implementing acoustic-prosodic entrainment in a conversational avatar. Proceedings of Interspeech. 16

17 CURRENTLY UNDER DEVELOPMENT Interactive adventure game TRASPI Question-answer game GuessWho 17

18 TRASPI: OBJECTIVE Employ the basic rationale from GoFish User faces decisions, gets advice from 2 avatars, one entrains the other does not Entrainment-trust on both local and global levels Does the user follow the advice of the entraining or disentraining avatar (cf. GoFish which avatar does the user ask for advice) More natural spoken interaction Induce the user to use less constrained language when communicating with the avatar GoFish relied on 5-6 wh-questions from the user and 5-6 short responses from the system Vary the stakes when making the decisions Make the game engaging, dynamic and fun to play 18

19 TRASPI DEMO 19

20 TRASPI: STATUS Already implemented Story design Communication with the ASR/TTS engine Speech rate, intensity, pitch adaptation of TTS to the user All basic functionality in place Left to do Increase the number of decision points Increase the perception that the choices matter (although in fact they do not) Pilot-test, de-bug Plan to start collecting pilot data in September 20

21 GUESS WHO: OBJECTIVE Use the Wizard of Oz paradigm to increase the flexibility of interaction while keeping relatively tight control (intermediate between GOFISH & TRASPI) circumvent ASR deficiencies Allows for richer inventory of turns, no restriction on syntax & referring expressions 21

22 GUESS WHO: OBJECTIVE More fine-grained focus on the entrainment-trust link In question-answer pairs, test entrainment within and across the dialog act (Q-Q & A-A vs. Q-A & A-Q) Entrainment is highly complex (evidence of complementary entrainment, recent findings on dialogue acts (Cole & Reichel, 2016) 22

23 SUMMARY Entrainment-trust link is rather complex in humanhuman interactions We employ several scenarios for understanding different aspects of entrainment-trust link Possible factors to explore Human/non-human avatars Different features to entrain Cue avatar s ability overtly Other than monetary benefits for trust (maze with noise) Move beyond naïve general entrainment-trust to more structured (dialog acts) 23

24 THANK YOU! Štefan Beňuš, Benus (2016). The prosody of backchannels in Slovak. Proceedings of Speech Prosody. Gravano, Benus, Levitan, Hirschberg (2015). Backward Mimicry and Forward Influence in Prosodic Contour Choice in Standard American English, in Proc. of Interspeech. Levitan, Benus, Galvez, Gravano, Savoretti, Trnka, Weise, Hirschberg (2016).Implementing acoustic-prosodic entrainment in a conversational avatar. Proceedings of Interspeech. Levitan, Benus, Gravano, Hirschberg (2015). Acoustic-prosodic entrainment in Slovak, Spanish, English and Chinese: A cross-linguistic comparison. Proceedings of SIGdial. Levitan, Benus, Gravano, Hirschberg. (2015). Entrainment and Turn- Taking in Human-Human Dialogue. AAAI Spring Symposium on Turn- Taking and Coordination in Human-Machine Interaction. Perez, Galvez, Gravano (2016) Disentrainment may be a positive thing: A novel measure of unsigned acoustic-prosodic synchrony, and its relation to speaker engagement. Proceedings of Interspeech.