Virtual Human Toolkit Tutorial

Transcription

1 Virtual Human Toolkit Tutorial Arno Hartholt 2015 The work depicted here was sponsored by the U.S. Army. Statements and opinions expressed do not necessarily reflect the position or the policy of the United States Government, and no official endorsement should be inferred.

2 World Class Researchers & Staff Currently ~ 170 staff members Doctorates: 34 M.D.s: 1 Masters: 40 Bachelors: 50 USC Faculty Members: 14 Graduate Students: 16

3 Major Research Areas Immersion Graphics Learning Sciences Social Simulation Virtual Humans

4 LEEP / Oculus Rift FAAST FOV2GO Immersion Graphics Learning Sciences Social Simulation Virtual Humans

5 Immersion Graphics Learning Sciences Social Simulation Virtual Humans

6 Immersion Graphics Learning Sciences Social Simulation Virtual Humans

7

8 Immersion Graphics Learning Sciences Social Simulation Virtual Humans

9 Immersion Graphics Learning Sciences Social Simulation Virtual Humans

10 Virtual Humans Vision Simulated world with characters that are almost human, who Are autonomous Fully perceive their environment Interact naturally, both verbally and nonverbally Model their own and other s believes, desires and intention Exhibit emotion Are efficient, coherent and integrated 10

11 Related Work Capabilities Automated Speech Recognition: Sphinx (CMU) Perception: Computer Expression Recognition Toolbox (CERT) (Littlewort et al., 2011) Task Modeling: DTask (Bickmore et al., 2009) Natural Language Generation (Stone, 2003) Text-To-Speech: Festival (Taylor et al., 1998) 11

12 Related Work Frameworks SAIBA Framework Function Markup Language (FML) (Heylen et al., 2008) Behavior Markup Language (BML) (Kopp et al, 2006) BML Realizers LiteBody (Bickmore et al., 2009) Greta (Poggi et al., 2005) Elckerlyc (Van Welbergen et al., 2010) EMBR (Heloir & Kipp., 2009) SmartBody (Shapiro 2011) 12

13 Integration Research opportunities Which integrated abilities are essential for which type of interactions? Which are desired? Which minimal and preferred dependencies exist between them? How do abilities affect each other? How can they best leverage each other to increase effectiveness? Larger effort with all VH and other ICT groups, as well as the research and development community (BML, PML, etc.) 13

14

15 Virtual Human Architecture Agent Nonverbal Behavior Understanding Natural Language Understanding Natural Language Generation Nonverbal Behavior Generation Audio-Visual Sensing Speech Recognition Speech Generation Behavior Realization User Renderer 15

16 Twins Architecture Question / Answering Agent NPCEditor Agent Nonverbal Behavior Understanding Natural Language Understanding Natural Language Generation Nonverbal Behavior Generation Audio-Visual Sensing OtoSense Prerecorded Speech SmartBody User Gamebryo 16

17 Rapport Architecture Virtual Listener Rule-Based Action Selector Agent Nonverbal Behavior Understanding Natural Language Understanding Natural Language Generation Nonverbal Behavior Generation Watson Speech Recognition Prerecorded Speech SmartBody User Unity 17

18 SimSensei Architecture FLoReS Nonverbal Behavior Understanding mxnlu Natural Language Generation Cerebella MultiSense PocketSphinx / BBN Prerecorded Speech SmartBody User Unity 18

19 VHToolkit

20 Toolkit Main Modules Audio-Visual Sensing: MultiSense Natural Language Processing: NPCEditor Nonverbal Behavior Generation: NVBG Behavior Realization: SmartBody 20

21 Toolkit Remaining Components ASR: AcquireSpeech, w/ PocketSphinx (Google, AT&T) TTS: Festival, MS SAPI, CereVoice Renderer: Unity, Ogre ActiveMQ-based messaging (VHMsg) Authoring tools Debug tools Supporting libraries 21

22 Toolkit Provided Examples Question-answering agents Brad & Rachel Cognitive Science movie maker 22

23 Toolkit Implementation vrperception (PML) NPCEditor vrexpress (FML + BML) NVBG nvb_parser NVB-Parser nvb_result vrspeech vrspeak (BML) RemoteSpeechCmd MultiSense (GAVAM, CLM, FAAST) AcquireSpeech PocketSphinx SmartBody DLL RemoteSpeechReply TTSRelay Google ASR Festival Framework Human User AT&T ASR Unity MS SAPI 23

24 Main Modules

25 The Multimodal Challenge: From Signals to User State Audio signals Voice pitch Visual signals Head pose Behaviors / Gestures Head nod Pitch accent Perceived User State Distress Engagement Frustration Sensing Recognition Understanding Temporal Trimodal Interpersonal Multiplicity Verbal Auditory Visual Human Communication Dynamics 25

26 MultiSense: Multimodal Perception Library (Scherer et al., 2012) CONSUMER TOOLS\MODULES PROVIDERS: Audio Capture Webcam Capture Mouse Kinect( Depth/Intensity/IR image/skeleton) TRANSFORMERS: Facetrackers Gavam CLM/CLMZ Okao Shore Real-time hcrf ActiveMQ VHMessenger EmoVoice CONSUMERS: Image Painter Signal Painter PML: Perception Markup Language Sensing Layer <person id= subjecta > <sensinglayer> <headpose> <position z="223" y="345" x="193" /> <rotation rotz="15" roty="35" rotx="10" /> <confidence>0.34<confidence/> </headpose>... </sensinglayer> </person> Behavior Layer <person id= subjectb > <behaviorlayer> <behavior> <type>attention</type> <level>high</level> <value>0.6</value> <confidence>0.46<confidence/> </behavior>... </behaviorlayer> </person> 26

27 NPCEditor (Leuski & Traum 2008) Statistical text classifier Authoring GUI Basic DM Groovy scripting 27

28 NPCEditor Novel question: what do you do here, girls? Rank Similar questions Relevant answers Relevant words Appropriate answers 28

29 NVBG (Lee & Marsella 2006) Cognitive Module Reasoning, Emotion, Language Communicative Function Derivation Function Rules Behavior Suggestion NVB Rules Behavior Realization Behavior Description Behavior Description to Animation Controller Surface Text Cache Parse Tree Natural Language Parser 26 speech related rules 29 rules for gaze behaviors 20 rules for listener reactions Other rules: negotiation stance, idle behaviors Rules are modifiable and extensible for different projects, animations, characters, etc. 29

30 NVBG Rules mapped to semantic structure Derivation Function Behavior No, not, nothing, Head shakes on Negation cannot, none phrase Really, very, quite, great, absolutely, gorgeous Yes, yeah, I do, We have, It s true, OK Intensification Affirmation Head nod and brow frown on word Head nods and brow raise on phrase Everything, all, whole, several, plenty, full Inclusivity Lateral head sweep and brow flash on word Rules mapped to syntactic structure <NP> (noun phrase) First <VP> (verb phrase) Life-like Life-like Head nod at the start of noun phrase Head nod and beat gesture on start of the first verb phrase 30

31 NVBG Info on cognitive state Info on listener s reaction Info on negotiation stance Input Message Function Behavior Status: speaker & planning_speech Status: speaker and seek social support Status: listener and speech comprehension low Status: listener and emotion surprise Negotiation stance: delay Hold Conversation Turn Show desire for social support Show confusion Show emotional reaction Show defensive stance Gaze aversion from addressee Glance at other entity Eyebrow lowered, head tilt Eyebrow raised and pulled together, eyelid open Change posture to crossed arms on chest 31

32 SmartBody Controllers that generate motion via: Procedural algorithm Example data Some controllers used in hierarchy Generalized movements -> Specialized movements Control via BML <gaze target= john /> Emotion Speech Gestures Gazing Lip Syncing Posture Head Movements Reaching / Touching Locomotion Steering Eye Movements Breathing 32