Quality comparison of wideband coders including tandeming and transcoding

Transcription

1 ETSI Workshop on Speech and Noise In Wideband Communication, 22nd and 23rd May Sophia Antipolis, France Quality comparison of wideband coders including tandeming and transcoding Catherine Quinquis (catherine.quinquis@orange-ftgroup.com) France Telecom Research & Development, France research & development

2 Wideband codecs Last year, wideband extensions of narrowband codecs have been standardised G providing high packetized wideband voice quality with scalability and interoperability with existing G.729 based VoIP networks and terminals EVRC-WB providing wideband in 3GPP2 networks using the same rate set as the current EVRC. Other ITU-T wideband codecs G.722 Mainly used in conference call, but introduced also in VoIP networks G providing wideband in 3GPP networks (also called AMR-WB) tandeming and transcoding p 2 research & development France Telecom Group

3 summary Comparison of Wideband codecs G.729.1, G.722 & G EVRC-WB, AMR-WB & VMR mode0 G.722 PLC Impact of transcoding and tandemming Self tandeming transcoding Comparison of wideband codecs with narrowband codecs tandeming and transcoding p 3 research & development France Telecom Group

4 Comparison of Wideband codecs tandeming and transcoding p 4 research & development France Telecom Group

5 Compared Subjective WB quality of G.729.1, G.722 & G Extract from the characterisation phase (step 2) Experiment 1 narrowband Experiment 2 Purpose : evaluate the performance of G729.1 algorithm with respect to well known references, in wide band clean speech (free of background noise) conditions with a variety of input levels and frame error rates. Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW). Languages : French (Canada) & English (US) Subjects : 32 naïve listeners Experiment 3 Wideband Music tandeming and transcoding p 5 research & development France Telecom Group

6 Compared Subjective WB quality of G.729.1, G.722 & G (no FER) Experiment 2 - clean speech MOS LQSW Dynastat VoiceAge Direct G k G k G k G k AMR- WB 23.85k AMR- WB 12.65k tandeming and transcoding p 6 research & development France Telecom Group

7 Compared Subjective WB quality of G.729.1, G.722 & G (FER) Exp 2 clean speech + FER - Lab E Exp 2 clean speech + FER - Lab F G k G k G k AMR-WB 12.65k AMR-WB 23.85k G k G k G k AMR-WB 12.65k AMR-WB 23.85k FER = 0% FER = 3% FER = 6% FER = 10% 1 FER = 0% FER = 3% FER = 6% FER = 10% tandeming and transcoding p 7 research & development France Telecom Group

8 Compared Subjective EVRC-WB quality with AMR-WB and VMR Mode-0 Extract from EVRC-WB Characterization test Experiment 1 Purpose : evaluate the performance of EVRC-WB algorithm with respect to well known references, in wide band clean speech (free of background noise) conditions with a variety of input levels and frame error rates. Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW). Languages : English (US) Subjects : 32 naïve listeners Experiment 2 Purpose : evaluate the performance of EVRC-WB algorithm with respect to well known references, in wide band noisy speech, and VAD/DTX scheme Methodology : P.835 Experiment 3 & 4 Narrowband tandeming and transcoding p 8 research & development France Telecom Group

9 Compared Subjective EVRC-WB quality with AMR-WB and VMR Mode-0 CT1 Results 5 EVRC-WB EVRC-WB with DTX kbps with DTX VMR Mode 0 4 MOS db level -32 db level -12 db level 1% FER with HRPD Rev A mask 2% FER with HRPD Rev A 3% FER 6% FER 1% D&B+1% packet level signaling* tandeming and transcoding p 9 research & development France Telecom Group

10 Performance of G.722 with packet loss concealment Extract from G.722 PLC Selection test Experiment 1a &1b Purpose : evaluate the performance of PLC algorithm with respect to well known references, in wide band clean speech (free of background noise) conditions with a variety of frame error rates (random for exp1a, burst for exp 1b). Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW). Languages : Japanese, French & English (US) Subjects : 32 naïve listeners Experiment 2a &2b Purpose : evaluate the performance of PLC algorithm with respect to well known references, in noisy speech, (random for exp2a, burst for exp 2b). tandeming and transcoding p 10 research & development France Telecom Group

11 Performance of G.722 with packet loss concealment Clean peech, Random FER Clean speech, Bursty FER PLC A PLC C PLC0 G k PLC A PLC C PLC0 G k % 1% 3% 6% 3%+0.1% RBER 1 0% 1% 3% 6% 3%+0.1% RBER tandeming and transcoding p 11 research & development France Telecom Group

12 Conclusion on Wideband Quality All these codecs provide high wideband quality that can be roughly divided into 2 categories: Maximum wideband quality for the most recent codecs at their maximum bit rates: very close to "direct" quality in the test conditions Slightly lower quality for these codecs when operating at reduced bit rates around kbit/s and for G.722 at 64 kbit/s but for much reduced complexity tandeming and transcoding p 12 research & development France Telecom Group

13 Impact of transcoding and tandemming tandeming and transcoding p 13 research & development France Telecom Group

14 Impact of transcoding/tandeming G & G.722 Extract from the characterisation phase of AMR-WB Experiment 1 Purpose : evaluate the performance of AMRWB algorithm, in wide band clean speech (free of background noise) tandeming conditions with a variety of input levels. Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW). Languages : Finnish & English Subjects : 32 naïve listeners Experiment 2 Purpose : evaluate the performance of AMRWB algorithm, in wide band clean speech (free of background noise) conditions in transcoding with other wideband standards Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW). Languages : French & English (US) Subjects : 32 naïve listeners tandeming and transcoding p 14 research & development France Telecom Group

15 Compared Subjective WB quality of G.722 & G in self tandeming Lab = Nokia Lab = BT single coding self-tandem single coding self-tandem single coding self-tandem Direct G kbit/s G kbit/s Mode 2 (12.65 kbit/s) Mode 2 (12.65 kbit/s) Mode 8 (23.85 kbit/s) Mode 8 (23.85 kbit/s) tandeming and transcoding p 15 research & development France Telecom Group

16 Compared Subjective WB quality of G.722 & G in transcoding Lab = FT Lab = LMGT G G Direct G G kbit/s Mode 2 (12.65 kbit/s) Mode 2 (12.65 kbit/s) Mode 8 (23.85 kbit/s) Mode 8 (23.85 kbit/s) tandeming and transcoding p 16 research & development France Telecom Group

17 Conclusion on tandemming and transcoding Codecs self tandemings produce quite limited quality degradations of around 0.2 MOS-LQSW. Transcodings between different wideband formats produce more significant degradation : G722 AMR-WB transcoding quality score 0.2 to 0.4 MOS-LQSW below G k quality. tandeming and transcoding p 17 research & development France Telecom Group

18 Comparison od wideband codecs with narrowband codecs tandeming and transcoding p 18 research & development France Telecom Group

19 Comparison of wideband codecs with narrowband codecs (1) Extract from the G characterisation phase (step1) Experiment 1a Narrowband Experiment 1b Purpose : evaluate the performance of G algorithm, in wide band clean speech (free of background noise) with a variety of input levels. Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW). Languages : French & English (US) Subjects : 32 naïve listeners tandeming and transcoding p 19 research & development France Telecom Group

20 Comparison of wideband codecs with narrowband codecs (2) MOS-LQSM Lab: VoiceAge Lab : FT Direct NB Direct WB G.729A -8k (NB) G k (WB) G k (WB) G k (WB) G k (WB) G k (WB) tandeming and transcoding p 20 research & development France Telecom Group

21 Conclusion WB versus NB Results show that wideband voice, even coded at the lowest bit rates of G.722 (48 kbit/s), gets better score than direct narrow band quality with a gap up to +0.5 MOS-LQSM MOS-LQSM difference between narrow band and wideband direct speech is greater than 1 MOS-LQSM and remain between 0.5 MOS-LQSM and 1 MOS-LQSM between direct narrow band and high quality wideband coded speech. tandeming and transcoding p 21 research & development France Telecom Group

22 References G Characterization step 2 references ITU-T-SG12-TD42rev3(WP1/12), " G.729EV Characterization phase step 2 Quality Assessment Test Plan ", Source: Rapporteur for Question 7/12, Geneva, 5-13 June 2006 ITU-T-SG16-TD258(GEN/16), "LS on testing issues", Source: Rapporteurs Q7/12, Geneva, November 2006 ITU-T-SG16-TD258(GEN/16)-Attachment 2, " Executive summary of G729.1 Characterisation step 2 Experiments 1, 2 & 3. ", Source: France Télécom, Geneva, November 2006 G Characterization step 2 references ITU-T-SG12-TD22rev2(WP1/12), " G729EV Characterisation/Optimisation step1 Test plan ", Source: Rapporteur for Question 7/12, Geneva, October 2005 ITU-T-SG16-TD202(GEN/16), " LS on audio issues ", Source: Rapporteurs Q7/12, Geneva, 3-13 April 2006 ITU-T-SG16-TD202(GEN/16)-Attachment 1, " G729EV Characterisation/Optimisation step1: Summary of results ", Source: France Télécom, Geneva, 3-13 April 2006 AMR-WB Characterization references ETSI TR V6.0.0 ( ), " Performance characterization of the Adaptive Multi-Rate Wideband (AMR-WB) speech codec", Tdoc S4 (01)0351R1, " AMR Wideband Characterisation Phase 1 Listening Tests, Experiment 1 - BT Results", Source: BT, June 4-8, 2001, Naantali, Finland Tdoc. S4 (01)0326, " Executive Summary from France Télécom R&D for the ETSI AMR-WB Characterisation Phase Results for Experiments 2 & 5", Source: France Telecom, June 4-8, 2001, Naantali, Finland Tdoc S4 (01)0321, " AMR WB Characterization Experiments 2A and 6A - LMGT Results ", Source: LMGT, June 4-8, 2001, Naantali, Finland Tdoc S , " Nokia report for AMR-WB characterisation experiments 1A & 6B ", Source: Nokia, June 4-8, 2001, Naantali, Finland EVRC-WB Characterization references ITU-T-SG16-TD291(GEN/16) "Updated LS reply on follow-up on embedded extension to G and media coding summary database " Source: Chairman SG 16 (on behalf of 3GPP2 TSG-C) Geneva, November 2006 G.722 Packet Loss Concealment references ITU-T-SG16-TD217(WP3/16), " Report of Question 10/16 Software tools for signal processing standardization activities and maintenance and extension of existing voice coding standards ", Source: Rapporteur for Question 10/16, Geneva, November 2006Extract from the G characterisation phase (step1) tandeming and transcoding p 22 research & development France Telecom Group