End-to-End Speech Quality Testing in a Complex Transmission Scenario

Transcription

1 End-to-End Speech Quality Testing in a Complex Transmission Scenario F. Kettler*, H.W. Gierlich*, J. Berger**, H. Klaus**, I. Kliche**, K.-D. Michael**, T. Scheerbarth**, R. Scholl***, J.-L. Freisse**** * HEAD acoustics GmbH, Herzogenrath, ** T-Nova Dt. Telekom Innovationsgesellschaft mbh Berkom, *** European Telecommu. Standards Institute ETSI (now Siemens AG) **** European Telecommunications Standards Institute ETSI

2 Overview n Introduction 1 st ETSI Speech Quality Test Event, Oct n Test Setup for End-to-end Speech Quality Testing n Test Methods, Parameters and Results n One-way Speech Quality n Echo and Double Talk Performance n Background Noise Transmission Quality n Summary

3 Introduction IP network designed for pure data transmission Telephone network real time transmission properties delay is high, unpredictable and time-variant influence on echo and conversational quality speech coders specific kind of disturbances (packet loss, jitter...) + differentiation between terminal and network is no longer possible!

4 ETSI Speech Quality Test Event Organized by ETSI, Sophia Antipolis, France, Oct Sponsored by Texas Instruments, Alcatel, T-Nova and HEAD acoustics Test labs T-Nova and HEAD acoustics n Evaluation of end-to-end (mouth to ear) speech quality using real equipment in a controlled VoIP environment n One exclusive testing day for each manufactures n Objective tests and subjective test (offline) n Detailed test report and an anonymous report containing all results n Goals : identification of system improvements and current status of achievable VoIP quality

5 Overview n Introduction 1 st ETSI Speech Quality Test Event, Oct n Test Setup for End-to-end Speech Quality Testing n Test Methods, Parameters and Results n One-way Speech Quality n Echo and Double Talk Performance n Background Noise Transmission Quality n Summary

6 Typical Signal Processing noise reduction EC level switching VAD coder microphone NR NLP VAD - Coder Buffer, Packetization IP H(f ) delay, jitter, packet loss + Decoder Buffer, PLC IP loudspeaker EC CN comfort noise decoder jitter buffers, PLC

7 Speech Quality Parameters from the user s perspective talking situation speech quality listening situation conversational situation

8 Listening Situation IP terminal A terminal B voice activity detection speech coders packet loss packet loss concealment...

9 Talking Situation...echo? IP terminal A terminal B round trip delay echo level and echo characteristic implementation of speech echo cancellers quality of background noise transmission

10 Conversational Situation IP terminal A terminal B propagation delay - dynamics impairments double talk detection (EC implementation) echo during double talk level variations during double talk

11 Test Set-up IP NISTNet Packet Loss DELAY IP Packet MONITOR Acoustical acoustical test set-up IP-Terminal PC Input IP-Phone HEAD acoustics Testsystem CAS Output with IP terminals mounted to the HATS (Head And Torso Simulators)

12 Test Set-up NISTNet Packet Loss DELAY IP Packet MONITOR IP Gateway Acoustical electrical test set-up IP T1 / E1 / BRI IP Terminal (Phone or PC) ALCATEL PABX 4400 ISDN DSS1 with one IP phone and one gateway (accessed through a 4-wire ISDN line) Input HEAD acoustics Testsystem CAS Output

13 Overview n Introduction 1 st ETSI Speech Quality Test Event, Oct n Test Setup for End-to-end Speech Quality Testing n Test Methods, Parameters and Results n One-way Speech Quality n Echo and Double Talk Performance n Background Noise Transmission Quality n Summary

14 Listening Situation IP terminal A feeding terminal B recording and analysis Analysis methods: perceptual speech quality models

15 Instrumental Measures Typical Processing Steps (Schematic): Results of Listening Tests! Speech Signal Test Object Adaptation processed signal reference signal Hearing Model Hearing Model Comparison, Reference, Reduction, Signal Value S index Q Realizations ΠPESQ (P.862) ΠTOSQA ΠTOSQA 2001 Perceptual Evaluation of Speech Quality Telecommunications Objective Speech Quality Assessment Extension to acoustical measurements

16 1 st ETSI SQTE - Test Conditions IP terminal A terminal B c1: clear condition c2: 1%packet loss c3: 2%packet loss c4: 3%Packet loss c5: 5%packet loss c6: 1%packet loss + 20 ms jitter

17 1 st ETSI SQTE - Results 5 TMOS 4 G.711 reference 3 G.729 ref. 2 G.723.1@6.3 ref. 1 C1 C2 C3 C4 C5 C6 C1 C2 C3 C4 C5 C6 C1 C2 C3 C4 C5 C6 G.711 G G.711 VAD on, PLC on/off PL= ms G.723.1@6.3 VAD on, PLCon/off PL=30ms G.729 VAD on, PLC on/off PL= ms G.723.1@6.3 MNRU References direkt

18 Analysis of packet loss and PLC Occurrence of packet loss (20 ms packet length) Packet loss concealment (typical implementation)

19 Detailed analysis of PLC implementations Cross correlation analysis vs. time Relative Approach analysis vs. time and frequency

20 Overview n Introduction 1 st ETSI Speech Quality Test Event, Oct n Test Setup for End-to-end Speech Quality Testing n Test Methods, Parameters and Results n One-way Speech Quality n Echo and Double Talk Performance n Background Noise Transmission Quality n Summary

21 Talking Situation...echo? IP terminal A terminal B feeding, recording and analysis Analysis methods: echo measurements based on Composite Source Signals (CSS) under single and double talk conditions

22 Composite Source Signal Noise part Voiced part Pause Signal description in ITU-T P.501 Used e.g. in G.168 (Echo Cancellers) Sequence length 700 ms

23 Echo under Single Talk Conditions Requirement on echo loss depends on transmission delay SLR JLR JLR RLR 7dB 0dB 0dB 3dB TELR a(echo) a(echo) TELR RLR JLR JLR SLR 3dB 0dB 0dB 7dB OLR = 10dB

24 Conversational Situation IP terminal A terminal B feeding feeding, recording and analysis Analysis methods: double talk measurements based on two de-correlated Composite Source Signals

25 Combination of two CSS Signal description in ITU-T P.501 Analysis methods in ITU-T P.502

26 Comparison to Real Speech Combination of CSS incl. single and double talk periods Typical double talk sequence (speech) English, male and female voices

27 Double Talk Evaluation: Result #1 measured signal nearly perfectly matches original test signal green: measured signal red: original test signal yellow: overlap

28 Double Talk Evaluation: Result #2 short term echo components during pauses and jitter echo jitter green: measured signal red: original test signal yellow: overlap

29 Double Talk Evaluation: Result #3 strong echo components (occurred only during double talk!) strong echo green: measured signal red: original test signal yellow: overlap

30 Double Talk Evaluation: Result #4 clipping at beginning and end of most bursts and comfort noise during pauses clipping green: measured signal red: original test signal yellow: overlap comfort noise

31 Overview n Introduction 1 st ETSI Speech Quality Test Event, Oct n Test Setup for End-to-end Speech Quality Testing n Test Methods, Parameters and Results n One-way Speech Quality n Echo and Double Talk Performance n Background Noise Transmission Quality n Summary

32 Test Signal for Noise Transmission Noise signal (Hoth spectrum) with increasing level vs. time Signal description in ITU-T P.800

33 Comparison of Test Results Comparison of three typical implementations 1 3 (1) red: original test signal (2) yellow: VAD threshold (3) blue: AGC gain (4) magenta: comfort noise injection 2 4

34 Overview n Introduction 1 st ETSI Speech Quality Test Event, Oct n Test Setup for End-to-end Speech Quality Testing n Test Methods, Parameters and Results n One-way Speech Quality n Echo and Double Talk Performance n Background Noise Transmission Quality n Summary

35 Summary n Identification of improvements and current status of achievable VoIP quality - success n Combination of various test procedures! n Overall quality rating given by the participants excellent n A short report can be found under etsi.org/plugtests n ETSI strongly recommended to continue the process of end-to-end speech quality tests n 2 nd SQTE planned for April 2002