International Telecommunication Union. Speech Quality Testing for VoIP Terminals and Gateways: Input from ETSI Plugtest

Transcription

1 International Telecommunication Union Speech Quality Testing for VoIP Terminals and Gateways: Input from ETSI Plugtest Plugtest Speech Quality Test Events H. W. Gierlich HEAD acoustics GmbH Geneva, June 26

2 Overview o Introduction - 1 st, 2 nd and 3 rd SQTE o Discussion of Results Gateways IP phones o Summary & Outlook Geneva, June 26 2

3 Overview o Introduction - 1 st, 2 nd and 3 rd SQTE o Discussion of Results Gateways IP phones o Summary & Outlook Geneva, June 26 3

4 Introduction The ETSI Speech Quality Test Events 1 st SQTE October 2 organized and hosted by the ETSI Bake-off Service 6 participating companies test labs HEAD acoustics, Deutsche Telekom 2 nd SQTE April 22 organized and hosted by the ETSI Plugtests Service 5 participating companies test labs HEAD acoustics, Deutsche Telekom 3 rd SQTE June and September 24, twin event EU, US organized by the ETSI Plugtests Service and HEAD acoustics, hosted by HEAD acoustics 11 participating companies, test lab HEAD acoustics 4 th SQTE June 26, EU just ongoing Geneva, June 26 4

5 The Goal Evaluate all conversational aspects Listening speech quality ( status quo ) Detailed parameter tests Echo measurements, EC implementation Double talk performance tests Quality of background noise transmission In order to Compare different implementations Document the state of the arte of technology Give input to standardization Help companies to optimize their implementations Geneva, June 26 5

6 Test Setup Gateways Gateway IP NISTNet IP Packet MONITOR IP Gateway Packet Loss Delay E1 E1 PBX ISDN DSS1 ISDN DSS1 Input HEAD acoustics Test System ACQUA Output Condition Packet Loss (Equal) Additional Delay 1 Delay Variation 1a No 2a 1% No 3a 2% No 4a 3% No Condition Packet Loss (Equal) Additional Delay 1 Delay Variation 1b No 2b 5% No 3b 5 ms 2 ms (2) 4b 5% 5 ms 2 ms(2) 5a 5% No 6a 1% 5 ms 2 ms (2) Geneva, June 26 6

7 Test Setup IP-Phones NISTNet IP Packet MONITOR IP Gateway IP Packet Loss Delay E1 PBX IP Terminal (Phone or PC) ISDN DSS1 Input HEAD acoustics Test System ACQUA Output Condition Packet Loss (Equal) Additional Delay 1) Delay Variation 1c 1 ms No 2c 1 ms 2 ms(2) 3c 1% 1 ms No 4c 1% 1 ms 2 ms(2) Condition Packet Loss (Equal) Additional Delay 1) Delay Variation 1d 1 ms No 2d 3% 1 ms No 3d 1 ms 2 ms(2) 4d 3% 1 ms 2 ms(2) 5c 3% 1 ms No Geneva, June 26 7

8 Overview o Introduction - 1 st, 2 nd and 3 rd SQTE o Discussion of Results Gateways IP phones o Summary & Outlook Geneva, June 26 8

9 Listening Speech Quality PESQ P.862 (MOS-LQO) max. MOS-LQO 5 PESQ delay 6 average min MOS-LQO 3 3 delay / ms 2 min max. delay average %, ms, VAD on %, ms 1%, ms 2%, ms 3%, ms 5%, ms 1%, 2ms, r=.9 (fix delay 5ms) 1%, 2ms, r=.5 (fix delay 5ms) network conditions G.711 Geneva, June 26 9

10 Listening Speech Quality TOSQA21 (TMOS) TOSQA delay TMOS 3 3 delay / ms %, ms, VAD on %, ms 1%, ms 2%, ms 3%, ms 5%, ms 1%, 2ms, r=.9 (fix delay 5ms) 1%, 2ms, r=.5 (fix delay 5ms) network conditions G.711 Geneva, June 26 1

11 Listening Speech Quality (MOS-LQO vs. TMOS) PESQ TOSQA 5 4 TMOS, MOS-LQO 3 2 For further SQTE results (G.729, G.723) see: Anonymized Report Gateways SQTE Results IP Phones: Anonymized Report IP Phones 1 %, ms, VAD on %, ms 1%, ms 2%, ms 3%, ms 5%, ms 1%, 2ms, r=.9 (fix delay 5ms) 1%, 2ms, r=.5 (fix delay 5ms) network conditions Geneva, June 26 11

12 Packet Loss Concealment Highly influencing MOS score - optimization criteria Test signal (5s) Recommendation P.51 periodical repetition of a voiced sound pitch frequency approx.. 33 Hz Relative Approach hearing model based forward estimation comparison to actual signal displays estimation error Cross correlation vs. Time analysis of signal phase periodical pattern (approx.. 3 ms) Geneva, June 26 12

13 Packet Loss Concealment t/s k.4.2 U - V -.2 MOS-LQO 5% packet loss t/s k.2 U - V t/s cp/cpa m t/s % k f - Hz 3 3m t - s -3m t/s cp/cpa m t/s % k f - Hz 3 3m t - s -3m t/s k U - V k t/s U -V k 2k t/s cp/cpa m f - Hz 3 3m t - s -3m t/s cp/cpa m f - Hz 3 3m t - s -3m t/s t/s % % Geneva, June 26 13

14 VAD and Comfort Noise Gateway NISTNet Gateway ISDN ISDN Simulator Simulator PBX PBX IP Packet Loss Delay IP PBX PBX ISDN ISDN Simulator Simulator Voice Activity Detection silence suppression silence packets comfort noise generation Geneva, June 26 14

15 VAD und Comfort Noise activation thresholds,, hysteresis hysteresis voiced sound, decreasing and increasing level vs. time Level vs. time Manual(5, ms) L/dB[V] Level vs. time Manual(5, ms) L/dB[V] measured signal level test signal level t/s t/s Level vs. time Manual(5, ms) L/dB[V] -3 Level vs. time Manual(5, ms) L/dB[V] -3 Level vs. time Manual(5. ms) L/dB[V] t/s t/s t/s Geneva, June adaptive comfort noise -65-7

16 VAD und Comfort Noise II comfort noise level slightly too high Realitsic noise scenario (students cafe) Level vs. time Manual(5, ms) L/dB[V] -1 Level vs. time Manual(5, ms) L/dB[V] noise contrast t/s t/s Level vs. time Manual(5, ms) L/dB[V] Level vs. time Manual(5, ms) L/dB[V] Level vs. time Manual(5. ms) L/dB[V] t/s t/s t/s Geneva, June quickly adapting comfort noise -7-8

17 Setup for Echo Canceller Testing Receive direction ISDN ISDN Simulator Simulator PBX PBX Gateway IP NISTNet Packet Loss Delay IP Gateway Send direction PBX PBX ISDN ISDN Simulator Simulator infinite ERL 4 db and ERL 6 db near end far end EC under test EC off (Echo path realized in MFE VI) In RCV Out SND USB In SND MFE VI Test system ACQUA Out RCV Geneva, June 26 17

18 Double Talk Performance ISDN ISDN Simulator Simulator PBX PBX Gateway IP NISTNet Packet Loss Delay IP Gateway PBX PBX ISDN ISDN Simulator Simulator EC under test EC off In RCV Out SND USB In SND MFE VI Out RCV infinite ERL 4 db and ERL 6 db Testsystem ACQUA Geneva, June 26 18

19 Double Talk Performance 6 db ERL DT_VAR_S.DAT p/pa U/V.15 U/V echo 5 1 t/s echo components 5m.1 5m -5m -5m t/s U/V t/s clipping -.15 U/V m 5m -5m -5m t/s t/s Geneva, June 26 19

20 Summary Results: Gateway Pie listening speech quality conversational aspects like echo behaviour, double talk performance and background noise transmission Geneva, June 26 2

21 Summary Results: Gateway Pie ( P.55) Each slide represents one parameter listening speech quality conversational aspects like echo behaviour, double talk performance and background noise transmission Geneva, June 26 21

22 Summary Results: Gateway Pie ( P.55) Assumption: Independency of parameters, no interaction aspects listening speech quality conversational aspects like echo behaviour, double talk performance and background noise transmission Geneva, June 26 22

23 Summary Results: Gateway Pie ( P.55) Minimum requirement (red circle) represents average performance of 3 rd SQTE listening speech quality or recommended numbers conversational aspects like echo behaviour, double talk performance and background noise transmission Geneva, June 26 23

24 Example Results (4 of 1) G.711 PLC balanced implementation EC inserts high BGN modulation (NLP/CN) BGN modulation caused by VAD/CN jitter buffer performance G.711, significant disturbances violation of requirement only caused by CN level (echo attenuation high enough) Geneva, June 26 24

25 Overview o Introduction - 1 st, 2 nd and 3 rd SQTE o Discussion of Results Gateways IP phones o Summary & Outlook Geneva, June 26 25

26 Testing IP 3rd SQTE Tests in handset and hands-free mode 2 N 13 N For handset Frequency response, TMOS using TOSQA21 Echo measurements Double talk performance Quality of background noise transmission Geneva, June 26 26

27 Testing IP 3rd SQTE Receiving frequency responses and TMOS 3.1 TMOS 2N 3.6 TMOS 8N 3.7 TMOS 13N L/dB[Pa/V] f/hz L/dB[Pa/V] TMOS 2N 3.7 TMOS 8N 3.9 TMOS 13N L/dB[Pa/V] f/hz L/dB[Pa/V] TMOS 2N 3.5 TMOS 8N 3.8 TMOS 13N f/hz TMOS 2N 3.8 TMOS 8N 4. TMOS 13N f/hz Geneva, June 26 27

28 Listening Speech Quality TOSQA21 (TMOS) TOSQA delay TMOS delay / ms sending 2N 8N 13N %, ms %, 2ms, r=.9 network conditions 1%, ms 1%, 2ms, r=.9 G.711, sending & receiving direction 3%, ms 1%, 2ms, r=.5 Geneva, June 26 28

29 Double talk test: Double Talk Test Hands-Free Terminals Level vs. time Manual(5, ms) Ch. 1 - Ch.2 L/dB 1-1 Level vs. time Manual(5, ms) Ch.1 - Ch.2 L/dB Attenuation range in sending direction (P.34) t/s Level vs. time Manual(5, ms) Ch.1 - Ch L/dB t/s Level vs. time Manual(5, ms) Ch.1 - Ch L/dB t/s Reference t/s Geneva, June 26 29

30 Summary Results: IP-Phone Pies ( P.55) conversational aspects hands-free - echo, double talk, BGN transmission - handset listening speech quality conversational aspects handset - echo, double talk, BGN transmission - Geneva, June 26 3

31 Summary Results: IP-Phone Pies BGN modulation T P.34 type 3 implem. BGN modulation T P.34 type 3 implem. balanced handset Implementation balanced handset implementation BGN modulation T P.34 type 3 implem., Violation caused by noise level (echo attenuation high enough) G.729 and hands-free not tested Geneva, June 26 31

32 Some quotes from the 3rd SQTE event: o o o Michael Metzger, Executive Director of Marketing, Mindspeed Technologies: "... bringing together the world leaders in VoIP technology in a co-operative and neutral environment, this prestigious event continues to improve the quality of VoIP... Peter Faxel, member of the executive board, AVM; "... The ETSI event has helped us towards achieving our goals, and given us valuable information about the market trend and technological developments." Daniel Hartnett, Infineon Technologies: Plugtests(tm) service of ETSI is an invaluable exercise for a company that sees voice quality as a key element for the success of our telephone terminal product families. Infineon will continue to "The VoIP Speech Quality Test Event (SQTE) organised by the work closely with HEAD acoustics and ETSI to benchmark their products for voice quality excellence" Overall quality of the event: : Score 4.6 out of 5 Geneva, June 26 32