Lecture 4: Dialogue system architectures & humanrobot

Transcription

1 Lecture 4: Dialogue system architectures & humanrobot interaction Pierre Lison, Language Technology Group (LTG) Department of Informatics Fall 2012, September Outline Dialogue system architectures Three hot research topics Human-robot interaction Summary 2

2 Outline Dialogue system architectures Basic components Architectural models Three hot research topics Human-robot interaction Summary 3 Dialogue system architectures Dialogue is a very flexible medium of communication... but is also a quite complex one to process! SDS depend on a wide range of NLP technologies: automatic speech recognition (ASR); syntactic parsing & semantic interpretation; dialogue management; natural language generation; speech synthesis 4

3 A simple schema Extra-linguistic environment Language understanding Interpreted utterance ãu Intended response am Generation Recognition hypotheses ~ uu Utterance to synthesise um Speech recognition Dialogue management Speech synthesis input speech signal (user utterance) User output speech signal (machine utterance) 5 List of basic components Automatic speech recognition (ASR) is responsible for converting the raw acoustic signal into a list of possible string hypotheses about what the user said This list is often called the N-best list, and is typically accompanied by some kind of confidence scores ~ uu = this legend on spoken data system it really interesting dont you sink this lecture on spokane log systems is really interesting doesn t using these lectures on *UNK* dialogue systems is really interesting doesn t think 6

4 List of basic components (2) Natural language understanding (NLU) covers a range of processing tasks devoted to extracting the meaning of a given user utterance Can include a (usually shallow, domain-specific) semantic parser, a reference resolution engine, disfluency and error correction tools, contextual interpretation, etc this legend on spoken data system it really interesting dont you sink 0.7 Assert(lecture, interesting) Topic(lecture, SDS) FeedbackRequest ~ uu = this lecture on spokane log systems is really interesting doesn t using these lectures on *UNK* dialogue systems is really interesting doesn t think ~ au = 0.2 Assert(lecture, interesting) Topic(lecture, logging) using(logging) 0.1 JUNK 7 List of basic components (3) Dialogue management (DM) is in charge of controlling the conversation, and make decisions to say/do things at a given time, depending on the inputs Usually based on some representation of the current dialogue state ~ au = Assert(lecture, interesting) Topic(lecture, SDS) FeedbackRequest Assert(lecture, interesting) Topic(lecture, logging) using(logging) am = 4.6 Confirm 2.2 AskRepeat 0.8 DoNothing 0.1 JUNK 8

5 List of basic components (4) Natural language generation (NLG) is the reverse task of NLU: given a high level representation of the response, find the rights words to express it How to express (or realise) the given intention might depend on various contextual factors am = Confirm um = 2.0 Yes, I agree! 1.3 Yes, I already love this class! 0.8 Sure! 9 List of basic components (5) Finally, speech synthesis (TTS, for «text-to-speech») is the task of generating a synthesis speech signal corresponding to the selected system reply Can be modulated in various ways (voice, intonation, accent, etc.) um = Yes, I agree! 10

6 Architectures Outlined schema can be adapted in various ways Integration or removal of specific modules Each component can itself be decomposed: E.g. understanding can include parsing, reference resolution, etc. Similarly, the DM needs components for updating the dialogue state, planning the best action, predicting the next turn, etc. These components must be tightly coupled and continuously exchange information 11 Architectures Most basic architecture: pipeline The components are connected in a processing chain Each component is a black box getting inputs from its predecessor, and generating an output Can also operate in a distributed mode ASR NLU DM NLG TTS 12

7 Architectures Shortcoming of pipeline architectures: No feedback loop between modules Rigid information flow Difficult to operate incrementally Poor turn-taking behaviour (Typically) no anytime processing ASR NLU DM NLG TTS 13 Architectures Blackboard (or «information-state») architectures: Architectures revolves around a dialogue state, and a set of modules reading/writing to this state The modules listen to the state for relevant changes, in which case they do some processing and update the state with the result ASR NLU DM Dialogue state NLG TTS 14

8 Architectures Advantages of blackboard architectures: More flexible information flow (can exploit all information contained in the state for processing) Possibility to integrate feedback loops and process information in parallel Allows for more complex turn-taking behaviour But more complex software design [S. Larsson and D. R. Traum (2000), «Information state and dialogue management in the TRINDI dialogue move engine toolkit» in Natural Language Engineering] 15 Architectures The processing modules can be distributed on several machines and possibly run in parallel (if permitted by information flow) In addition to the shared repository (e.g. information state), each module might also have its own, private repository Can lead to complex setups with multiple OSs, programming languages and encoding formats Middlewares such as CORBA or ICE might be required to handle the communication between modules 16

9 Architectures Central architectural issues: Reusability: can a specific module (e.g. a reference resolver) be reused in another other dialogue systems? Domain portability: can the system handle other dialogue domains, or is the domain «hardwired» in the system? Adaptivity: can the system learn and adapt itself (to its user, its environment) from experience? Robustness: can the system cope with input/output errors and module breakdowns? Efficiency: can the system run fast enough to handle real-time interactions? Can the system run in anytime mode? 17 Outline Dialogue system architectures Three hot research topics: Incrementality Statistical approaches Situated interaction Human-robot interaction Summary 18

10 Incrementality Humans process and produce language incrementally one phoneme at a time, one word at a time Gradual expansion and refinement of interpretations Continuous creation of expectations about what the user is going to say next 19 Incrementality However, most SDS operate in «batch mode» For instance, NLU will typically wait for the sentence to be completed to start parsing it Similarly, TTS will wait for the system utterance to be fully generated to start synthesising it This is why dialogue systems often exhibit a «ping-pong» turn-taking behaviour Can we build incremental dialogue systems? 20

11 Incrementality Advantages of incremental processing: More reactive turn-taking behaviour The system can provide feedback on its understanding (or lack thereof) while the user is speaking More flexible handling of e.g. utterance fragments, interruptions Performance gains: the system can start processing as soon as the user begins to talk 21 Incremental processing model David Schlangen s generic incremental model of dialogue processing: Network of interconnected processes, transferring information via input and output buffers Incremental Unit (IU) = basic representational unit IUs are interconnected via various relations, forming a full network within & across processing levels 3 basic operations on IUs: update, purge and commit [Schlangen, D. and Skantze G. (2009) «A General, Abstract Model of Incremental Dialogue Processing», in Proceedings of EACL 2009.] 22

12 NUMBERS dialogue system 23 Statistical approaches Another hot topic in spoken dialogue system is statistical learning Spoken dialogue can be hard to design manually: User behaviour can be difficult to predict Need to consider many alternatives Presence of noise and uncertainty (ambiguities etc.) 24

13 Statistical approaches An alternative is to rely on machine learning techniques (i.e. statistical modelling from data) Such techniques can (and have been) be applied for all components of SDSs: Speech recognition (acoustic and language modelling) Dialogue understanding (data-driven parsing etc.) Dialogue management (statistical optimisation of policies) Generation & speech synthesis (stochastic NLG and TTS) [S. Young (2002), «The statistical approach to the design of spoken dialogue systems»] 25 Statistical approaches Advantages of statistical approaches: Better account of uncertainties (ASR errors, ambiguities) More natural & flexible interaction styles Can encode complex trade-offs between objectives (i.e. ensure understanding vs. keep dialogue short) Adaptation of the system to its users Reduced design time, reusability of components 26

14 Statistical approaches Major bottleneck: good in-domain data is scare and expensive to acquire! Scalability to complex domains is also challenging: the state space grows exponentially with the problem size For some domains, the system developer needs to enforce specific constraints (e.g. business rules) into the model 27 Situated interaction As already noted (lecture 2), dialogue is full of deictics - that is, references to external entities in the world Especially the case for dialogue systems operating in situated environments To make sense of situated dialogue, the system must first make sense of the situation, and the actions that can be performed in it Language Situated reality Action 28

15 Situated interaction Symbol grounding: linguistic symbols must ultimately be grounded in other modalities Bridge to perception and bodily experience Linguistic symbol door must be linked to the prototypical image of a door Role of affordances (what can be typically done with a door, and how) Note: symbol grounding dialogue grounding (feedback etc.) [M. Anderson (2003), «Embodied Cognition: a Field Guide», Artificial Intelligence] 29 Situated interaction Most artificial agents have a very brittle understanding of their environment Detecting objects, persons, places, or events in natural environments is really, really hard In addition to dialogue-related uncertainties (ASR errors etc.), the system must also deal with uncertainties about its perception of the context Grounding strategies (feedback, clarification requests) become even more important 30

16 Situated interaction Another interesting aspect of situated interactions is engagement: «the process by which [ ] participants establish, maintain and end their perceived connection during interactions they jointly undertake» Situated systems must determine when a person wants (or does not want) to interact And produce the appropriate kind of verbal and non-verbal signals to keep the person engaged [C.L. Sidner et al. (2005), «Explorations in engagement for humans and robots», Artificial Intelligence] 31 Outline Dialogue system architectures Three hot research topics Human-robot interaction Summary 32

17 Talking robots? 33 Talking robots? Our long-term aim: Hi! I am C-3PO, human-cyborg relations! (and he speaks 6 million languages...) For the time being, we ll have to scale down our expectations... 34

18 What is a robot? Concept of «robot» present for centuries in religion, mythology, philosophy and fiction mechanical creature working for (and sometimes against) humans The word robot itself originates from the Czech robota, which means drudgery or servitude. first use in Karel Chapek s 1920 s play «Rossum s Universal Robots». Mary Shelley s Frankenstein (1818) 35 What is a robot? A robot is a physical artificial agent It can perceive the world around him, via its sensors (camera, lasers, microphones, etc.) It can also act on its environment, via its actuators (usually motordriven mechanical devices) Some robots are also able to interact with other agents (human or robotic) in their environment 36

19 What is a robot? Robots are typically developed to accomplish particular goals in their environment Most industrial robots today are either remotely controlled, or preprogrammed to execute only a fixed sequence of actions Others (so-called cognitive robots) integrate reasoning capabilities to autonomously cope with new situations 37 Robot sensors Sensors allow the robot to perceive its external environment: Imaging sensors (e.g. monocular/stereo cameras) Sonar, laser (distance measurements) Tactile sensors (whiskers, bumps) Proprioceptive sensors (inform the robot of its own internal state) Force and torque sensors And many others 38

20 Robot actuators Actuators allow the robot to act on its environment to achieve its goal They are the muscles of the robot, used for manipulation and locomotion Typically mechanical devices converting energy it into motion Usual source of power: electric motor (but pneumatic or hydraulic actuation also possible) 39 Types of robots Robots can be classified based on their actuators: Static robots (e.g. robotic arms) have manipulation actuators Mobile robots have actuators for motion control And finally, hybrids combine both manipulation and motion (for instance, humanoids robots) Mobile robots Static robots Kuka robotic arm (LWR Leica T-Mac) Hybrid robots Big Dog from Boston Dynamics Underwater snake robot, H. Fukushima Lab Asimo humanoid robot from Honda 40

21 Types of robots Another factor is the number of interacting agents in the environment «Multi-agent» can be cooperative, competitive, or both «Swarm intelligence»: large number of interacting agents forming an emerging behaviour Single-agent Multi-agent Nao robots participating to RoboCup Swarm Nao robot (Aldebaran Robotics) Simple swarm robots 41 Robot control How does the robot decides which action to perform? Model-based deliberative control: try to build a model of the environment and use it to plan the optimal action Reactive control: derive the action directly, without prior model 42

22 Robot control Each approach has its strengths and weaknesses: Reactive control more appropriate for low-level decisionmaking in real-time, bottom-up perception Deliberative control more appropriate for high-level planning, top-down attention and control Existence of hybrid cognitive architectures integrating both reactive & deliberative control [P. Langley et al. (2005). «Cognitive architectures: Research issues and challenges»] 43 Human-robot interaction Human-robot interaction (HRI) is «the field of study dedicated to understanding, designing, and evaluating robotic systems for use by or with humans» Objective of HRI: develop principles, techniques and algorithms for natural and effective communication between humans and robots. [M. Goodrich & Schultz (2007), «Human-robot Interaction: A survey», Foundations and Trends in Human-Machine Interaction] 44

23 Human-robot interaction Communication in all its aspects Verbal- and non-verbal behaviours, including gesture, posture, affective display,... at various interaction ranges (proximal, distant), with reference to varying spatio-temporal contexts 45 Uncanny valley? Conjecture developed in the 70 s by Masariso Mori: «when human replicas look and act almost like humans, it causes some revulsion among human observers» Possible, but... In any case, robotics is concerned with building useful, interactive robots, not human replicas Relevance for practical research is more anecdotal Henrik Scharfe s Geminoid-DK 46

24 Uncanny valley? But there might still be a lesson to learn: One problem with «human replicas» is the mismatch between their sophisticated appearance and their actual (simplistic) behaviour The appearance triggers high expectations in the minds of human interlocutors, which fall short as soon as they interact with it When designing a robotic or virtual agent, we should make sure that its appearance are «in line» with the functionalities integrated in it The icat (Philips Research) 47 Human-robot interaction Example of application: assisting elderly or mentally impaired persons at home Physical and cognitive assistance, rehabilitation Social companionship Monitoring for health & safety hazards Can lead to improved well-being and autonomy (empowerment) Ethical challenges to address [A. Sharkey and N. Sharkey (2011), «Granny and the robots: ethical issues in robot care for the elderly», Ethics and Information Technology] 48

25 Human-robot interaction Example of application: search-and-rescue Work in hostile or hazardous environments Exploring areas too dangerous for humans Team collaboration between humans (e.g. firefighters) and robots Varying levels of autonomy 49 Outline Dialogue system architectures Three hot research topics Human-robot interaction Summary 50

26 Summary Architectures for spoken dialogue systems: Integration of several components: speech recognition (ASR), dialogue understanding (NLU), dialogue management (DM), generation (NLG) and speech synthesis (TTS) Pipeline or blackboard architectures to connect them ASR NLU DM Dialogue state NLG TTS 51 Summary Three hot research topics: Incrementality: how do we make dialogue systems that are able to process inputs as soon as they are available, instead of in «batch mode»? Statistical approaches: how can we use machine learning to automate part of the system design? Situated interaction: how should we build smart dialogue systems operating in situated environments? 52

27 Summary Robots are physical agents that can perceive their environment and act upon it to achieve their goals Research in HRI seeks to develop robots endowed with communicative abilities To this end, robots need to be able to: Be aware of their surrounding context Ground their linguistic symbols in the external reality 53 Next Monday Next Monday, we ll have an exercise session devoted to the following topics: Analysis of spoken dialogue Software tools for phonetics (Praat) In addition, I ll present you Lenny, a small humanoid robot we use for our research 54

28 Next Friday Next Friday, we ll talk about probabilistic modelling Probabilistic models are used in virtually every dialogue system component, from speech recognition to synthesis We ll explain what these probabilistic models are, how they are constructed, and how to use them to perform inference 55