Introduction to Talking Robots. Graham Wilcock Adjunct Professor, Docent Emeritus University of Helsinki

Size: px

Start display at page:

Download "Introduction to Talking Robots. Graham Wilcock Adjunct Professor, Docent Emeritus University of Helsinki"

Chester Price
5 years ago
Views:

1 Introduction to Talking Robots Graham Wilcock Adjunct Professor, Docent Emeritus University of Helsinki 2/12/2016 1

2 Robot Speech and Dialogues Graham Wilcock 2/12/2016 2

3 Speech Recognition: Open or Closed Vocabulary? Closed vocabulary Example: city names (c. 50 flight destinations) [ Helsinki London Paris Berlin Bern Dublin ] How many words in a language? English: estimated 300,000 different words Open vocabulary Example: words that sound like house [ grouse browse blouse ploughs louse ] 2/12/2016 3

4 Spoken Dialogue Systems: Closed-Domain Dialogues Example: Flight reservations Using finite state transitions Easily implemented by finite state machine (FSM) Fixed-domain database Flights, cities, days Easy to add new flights, new cities, Difficult to switch domains 2/12/2016 4

5 FSM for a Closed Domain 2/12/2016 5

6 Spoken Dialogue Systems: Annual Conferences SIGDIAL (since 2000) Special Interest Group on Discourse and Dialogue IWSDS (since 2009) International Workshop on Spoken Dialogue Systems 2/12/2016 6

7 Spoken Dialogue Systems: Books K. Jokinen and M. McTear Spoken Dialogue Systems Morgan & Claypool, 2009 K. Jokinen Constructive Dialogue Modelling Wiley, /12/2016 7

8 Robot Dialogue Systems: Books K. Jokinen and G. Wilcock (editors) Dialogues with Social Robots Springer, /12/2016 8

9 WikiTalk: Talking Robots and Wikipedia 2/12/2016 9

10 History of WikiTalk (1): Hungary 2011 Visits to MTA-SZTAKI, Budapest 2011 Hungarian Academy of Sciences Advanced 3-D Virtual Reality with Nao robot It would be nice if it could talk about Shakespeare How could that be possible? By using Wikipedia (key idea) I made a prototype with Python in 2011 Windows speech (Windows XP) 2/12/

11 New Direction: Open-Domain Dialogues Need open-domain knowledge source Get information from the web Wikipedia has articles on almost any topic Example: to talk about Shakespeare Get Wikipedia article about Shakespeare Start reading out the first paragraph 2/12/

12 2/12/

13 FSM for Open-Domain (draft) 2/12/

14 2/12/

15 Topic Shifts and Topic Chains Hyperlinks -> topic shifts -> topic chains Robot: Shakespeare was born and raised in Stratford-upon-Avon (link) Human: Stratford-upon-Avon? (topic shift) Robot: Stratford-upon-Avon is a market town and civil parish in Warwickshire, England Human: Warwickshire? (topic shift) Otherwise, continue same topic 2/12/

16 2/12/

17 FSM for Open-Domain (WikiTalk) 2/12/

18 2/12/

19 Speech Recognition in WikiTalk Open Domain but not Open Vocabulary Open vocabulary speech recognition basically does not work (about 300,000 words in English) Recognize phrases from small phrase sets Dynamically predicted from Wikipedia links Recognize small set of navigation commands Repeat, Continue, Enough, Previous, etc. 2/12/

20 History of WikiTalk (2): France 2012 International summer school project, Metz 2012 Led by Prof. Kristiina Jokinen (Tartu U.) and me 5 PhD students, 4 Nao robots, 4 hot weeks in July Transferred Python WikiTalk code to Nao robots Added robot gesturing and multimodal interaction Demos in Metz and Budapest /12/

21 International Summer Workshop enterface 2012, Metz, France 2/12/

22 Making WikiTalk on 4 Nao robots at enterface /12/

23 First Demo of WikiTalk on Nao (Metz, France 2012) 2/12/

24 ATIS Spelling Alphabet (used with Alphabet command) 2/12/

25 History of WikiTalk (3): Multilingual versions Finnish WikiTalk, Helsinki 2014 Prof. Jokinen, PhD student Niklas Laxström and me 2 robots borrowed from SASKY, Sastamala Demo at Robottiviikko 2014 exhibition Japanese WikiTalk, Kyoto 2015 Prof. Yamamoto, MSc student Okonogi and me 3 robots in Prof. Yamamoto s lab, Doshisha University Demo at SIGDIAL 2015 conference 2/12/

26 EU Robotics Week 2014 at AiRo Island Helsinki 2/12/

27 Multilingual WikiTalk: Finnish and English 2/12/

28 Multilingual WikiTalk: English and Japanese 2/12/

29 Open-Domain Talking: WikiTalk Talking Robot + Wikipedia Instant talking on any topic (open-domain) Community-validated talking Extend robot communicative capabilities 2/12/

30 Open-Domain Listening: Towards WikiListen Anticipated developments in ASR Continuous listening capability Better cloud-based speech recognition Open vocabulary speech recognition Wikipedia-based listening Open-domain named entity recognition Wikification: Wikipedia reference linking Extend robot communicative capabilities again 2/12/

31 Speech Technology for Under-resourced Languages Example: North Sami Spoken by 20,000 people in Lapland DigiSami project To support revitalization of North Sami language using modern language technologies Prof Kristiina Jokinen (University of Helsinki) Funded by Academy of Finland 2/12/

32 The Sami Languages 2/12/

33 DigiSami Corpus of North Sami Conversational Speech 2/12/

34 Speech Synthesizer and Recognizer for North Sami 2/12/

35 First Demo of SamiTalk (IWSDS 2016 in Lapland) 2/12/

Introduction to Talking Robots

Introduction to Talking Robots Graham Wilcock Adjunct Professor, Docent Emeritus University of Helsinki 1/9/2016 1 Speech and Robot Dialogues Graham Wilcock 1/9/2016 2 Speech Recognition: Open or Closed