Evaluation of In-Car Voice Services: Tasks for the New Q.12/12 Sebastian Möller

Transcription

1 Internatinal Telecmmunicatin Unin Evaluatin f In-Car Vice Services: Tasks fr the New Q.12/12 Sebastian Möller IKA, Ruhr-University Bchum, Germany C-Rapprteur Q.12/12 The Fully Netwrked Car, A Wrkshp n ICT in Vehicles PALEXPO, Geneva, 2-4 March 2005

2 In-Car Vice Services Rely n Speech Technlgy: Speech recgnitin (ASR), speaker recgnitin Natural language understanding, dialgue management Speech synthesis (TTS, CTS) Applicatin Examples: Vice dialling Navigatin Office applicatins (dictatin, reading) Infrmatin services Cntrl f remte devices,... 2

3 Factrs Influencing Service Quality??? Speech Underst. Speech Recgn. Attenuatin / Frequency Distrtin Cder/Decder Backgrund Nise Dialgue Management Spken Dialgue System PSTN / ISDN Mbile Netwrk IP Netwrk Respnse Generatin Speech Output Cder/Decder Uncrrelated Nise Acustic Cupling Delay Ech Transmissin Errrs Frame / Packet Lss Vice Activity Detectin Technlgy Perfrmance Perceived Quality 3

4 Q.12/12: Perfrmance Evaluatin f Services Based n Speech Technlgy Technlgy Perfrmance Impact f acustical cnditins and transmissin impairments n ASR perfrmance Measurement f speech synthesis perfrmance Perceived Service Quality and Usability Quality evaluatin methds Usability inspectin methds Quality predictin mdels 4

5 Transmissin Channel Impact ASR Perfrmance under Circuit Nise see ITU-T COM (2001) and Möller (2005) 5

6 Transmissin Channel Impact ASR and TTS with IP Packet Lss see Möller, Krebber & Raake (2004) 6

7 Predicting the Perfrmance f Speech Technlgy ASR Perfrmance Adequate estimatins may be btained with (mdified) netwrk planning mdels, e.g. the E-mdel (Rec. G.107, 2003) Other appraches: Signal-based mdels, e.g. Rec. P.862 (see Möller, 2005) TTS Quality Can nly be measured in auditry experiments (see Rec. P.85, 1994) Predictin perhaps pssible with singleended mdels (e.g. Rec. P.563, 2004)? 7

8 Envirnmental Factrs Speech I/O Quality Dialgue Cperativity User Factrs Quality f Service Agent Factrs Dialgue Symmetry Task Factrs Cntextual Factrs Perceived Quality: Taxnmy f Quality Aspects Cmmunicatin Efficiency Cmfrt Task Efficiency Usability User Satisfactin Service Efficiency Ecnmical Benefit Utility Acceptability see Möller (2002), Rec. P.851 (2003) 8

9 Subjective Evaluatin f Service Quality Rec. P.851 (2003) Definitin f quality aspects and system/service characteristics Descriptin f experimental prcedure (test set-up, subjects, scenaris, etc.) Questinnaire example: Hw well did yu feel understd by the system? extremely bad pr fair gd excellent ideal bad First applicatin examples have been reprted 9

10 Usability and Acceptability Evaluatin Methds still need t be defined Cgnitive effects f the driving task? New guidelines needed? Smeele et al. (2004) 10

11 Service Quality Predictin Parametric Apprach: PARADISE Mdel Dialgue system user satisfactin Instrument duratin, # turns,... Expert WER, # quest.,... Interactin parameters κ, c i user satisfactin = α N(κ ) n i = 1 w N( i c i ) Walker et al. (1997) 11

12 Q.12/12: Current Tasks Quantify impact f channel impairments and acustic cnditins n ASR /speaker recgnitin perfrmance TTS quality Overall service quality Predictin pssible? Define measurement methds fr TTS quality: Subjective methd: Update Rec. P.85? Instrumental methd: Rec. P.563? Service quality: Update f Rec. P.851? Usability and Acceptability: Rec. P.QVS 12

13 Q.12/12: Current Tasks (2) Predict Perceived Quality Define infrmative input parameters: Suppl. t P.85x Series (expected 10/2005), Rec. P.PST (expected 2006), see e.g. ITU- T D.030, 2005) Mdeling apprach: PARADISE r nn-linear algrithm? Several new Recmmendatins will be prduced in Yur cntributins are very welcme! Please cntact the Rapprteurs: sebastian.meller@rub.de alexander.raake@limsi.fr 13

14 References ITU-T Cntributin COM (2001). Influence f the transmissin channel n speech recgnizer and spken dialgue system perfrmance. Federal Republic f Germany (Authr: S. Möller), ITU-T SG12 Meeting, Feb. 2001, CH-Geneva. ITU-T Delayed Cntributin D.XXX (2005). Prpsal fr parameters describing the perfrmance f speech technlgy devices. Federal Republic f Germany (Authr: S. Möller), ITU-T SG12 Meeting, January 2005, CH-Geneva. ITU-T Rec. G.107 (2003). The E-mdel, a cmputatinal mdel fr use in transmissin planning. Internatinal Telecmmunicatin Unin, CH-Geneva. ITU-T Rec. P.563 (2004). Single-ended methd fr bjective speech quality assessment in narrw-band telephny applicatins. Internatinal Telecmmunicatin Unin, CH-Geneva. ITU-T Rec. P.85 (1994). A methd fr subjective perfrmance assessment f the quality f speech vice utput devices. Internatinal Telecmmunicatin Unin, CH-Geneva. ITU-T Rec. P.851 (2003). Subjective quality evaluatin f telephne services based n spken dialgue systems. Internatinal Telecmmunicatin Unin, CH-Geneva. Möller, S. (2005). Quality f telephne-based spken dialgue systems. Springer, US-New Yrk NY. Möller, S. (2002). A new taxnmy fr the quality f telephne services based n spken dialgue systems. In: Prc. 3 rd SIGdial Wrksh. n Discurse and Dialgue, US-Philadelphia, Möller, S., Krebber, J., Raake, A. (2004). Perfrmance f speech recgnitin and synthesis in packet-based netwrks. In: Prc. 8 th Int. Cnf. n Spken Language Prcessing (Interspeech ICSLP), KR-Jeju Island, Vl. 2, Smeele, P., Möller, S., Krebber, J. (2004). Evaluatin f the speech utput f a smart-hme system in a car envirnment. In: Prc. 8 th Int. Cnf. n Spken Language Prcessing (Interspeech ICSLP), KR-Jeju Island, Vl. 3, Walker, M.A., Litman, D.J., Kamm, C.A., Abella, A. (1997). PARADISE: A framewrk fr evaluating spken dialgue agents. In: Prc. 35th Meet. Assc. Cmp. Ling., ES-Madrid,