ETSI Workshop on Speech and Noise In Wideband Communication, 22nd and 23rd May 2007 - Sophia Antipolis, France Quality comparison of wideband coders including tandeming and transcoding Catherine Quinquis (catherine.quinquis@orange-ftgroup.com) France Telecom Research & Development, France research & development
Wideband codecs Last year, wideband extensions of narrowband codecs have been standardised G.729.1 providing high packetized wideband voice quality with scalability and interoperability with existing G.729 based VoIP networks and terminals EVRC-WB providing wideband in 3GPP2 networks using the same rate set as the current EVRC. Other ITU-T wideband codecs G.722 Mainly used in conference call, but introduced also in VoIP networks G.722.2 providing wideband in 3GPP networks (also called AMR-WB) tandeming and transcoding p 2 research & development France Telecom Group
summary 1 2 3 Comparison of Wideband codecs G.729.1, G.722 & G.722.2 EVRC-WB, AMR-WB & VMR mode0 G.722 PLC Impact of transcoding and tandemming Self tandeming transcoding Comparison of wideband codecs with narrowband codecs tandeming and transcoding p 3 research & development France Telecom Group
Comparison of Wideband codecs tandeming and transcoding p 4 research & development France Telecom Group
Compared Subjective WB quality of G.729.1, G.722 & G.722.2 Extract from the characterisation phase (step 2) Experiment 1 narrowband Experiment 2 Purpose : evaluate the performance of G729.1 algorithm with respect to well known references, in wide band clean speech (free of background noise) conditions with a variety of input levels and frame error rates. Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW). Languages : French (Canada) & English (US) Subjects : 32 naïve listeners Experiment 3 Wideband Music tandeming and transcoding p 5 research & development France Telecom Group
Compared Subjective WB quality of G.729.1, G.722 & G.722.2 (no FER) Experiment 2 - clean speech 5.0 4.5 4.0 MOS LQSW 3.5 3.0 2.5 Dynastat VoiceAge 2.0 1.5 1.0 Direct G.722 64k G.729.1 32k G.729.1 24k G.729.1 14k AMR- WB 23.85k AMR- WB 12.65k tandeming and transcoding p 6 research & development France Telecom Group
Compared Subjective WB quality of G.729.1, G.722 & G.722.2 (FER) Exp 2 clean speech + FER - Lab E Exp 2 clean speech + FER - Lab F 5.00 5 4.50 4.5 4.00 4 3.50 3.00 2.50 G.729.1 14k G.729.1 24k G.729.1 32k AMR-WB 12.65k AMR-WB 23.85k 3.5 3 2.5 G.729.1 14k G.729.1 24k G.729.1 32k AMR-WB 12.65k AMR-WB 23.85k 2.00 2 1.50 1.5 1.00 FER = 0% FER = 3% FER = 6% FER = 10% 1 FER = 0% FER = 3% FER = 6% FER = 10% tandeming and transcoding p 7 research & development France Telecom Group
Compared Subjective EVRC-WB quality with AMR-WB and VMR Mode-0 Extract from EVRC-WB Characterization test Experiment 1 Purpose : evaluate the performance of EVRC-WB algorithm with respect to well known references, in wide band clean speech (free of background noise) conditions with a variety of input levels and frame error rates. Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW). Languages : English (US) Subjects : 32 naïve listeners Experiment 2 Purpose : evaluate the performance of EVRC-WB algorithm with respect to well known references, in wide band noisy speech, and VAD/DTX scheme Methodology : P.835 Experiment 3 & 4 Narrowband tandeming and transcoding p 8 research & development France Telecom Group
Compared Subjective EVRC-WB quality with AMR-WB and VMR Mode-0 CT1 Results 5 EVRC-WB EVRC-WB with DTX AMR-WB @ 12.65 kbps with DTX VMR Mode 0 4 MOS 3 2 1-22 db level -32 db level -12 db level 1% FER with HRPD Rev A mask 2% FER with HRPD Rev A 3% FER 6% FER 1% D&B+1% packet level signaling* tandeming and transcoding p 9 research & development France Telecom Group
Performance of G.722 with packet loss concealment Extract from G.722 PLC Selection test Experiment 1a &1b Purpose : evaluate the performance of PLC algorithm with respect to well known references, in wide band clean speech (free of background noise) conditions with a variety of frame error rates (random for exp1a, burst for exp 1b). Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW). Languages : Japanese, French & English (US) Subjects : 32 naïve listeners Experiment 2a &2b Purpose : evaluate the performance of PLC algorithm with respect to well known references, in noisy speech, (random for exp2a, burst for exp 2b). tandeming and transcoding p 10 research & development France Telecom Group
Performance of G.722 with packet loss concealment Clean peech, Random FER Clean speech, Bursty FER PLC A PLC C PLC0 G.729.1-32k PLC A PLC C PLC0 G.729.1-32k 5 5 4 4 3 3 2 2 1 0% 1% 3% 6% 3%+0.1% RBER 1 0% 1% 3% 6% 3%+0.1% RBER tandeming and transcoding p 11 research & development France Telecom Group
Conclusion on Wideband Quality All these codecs provide high wideband quality that can be roughly divided into 2 categories: Maximum wideband quality for the most recent codecs at their maximum bit rates: very close to "direct" quality in the test conditions Slightly lower quality for these codecs when operating at reduced bit rates around 12-14 kbit/s and for G.722 at 64 kbit/s but for much reduced complexity tandeming and transcoding p 12 research & development France Telecom Group
Impact of transcoding and tandemming tandeming and transcoding p 13 research & development France Telecom Group
Impact of transcoding/tandeming G.722.2 & G.722 Extract from the characterisation phase of AMR-WB Experiment 1 Purpose : evaluate the performance of AMRWB algorithm, in wide band clean speech (free of background noise) tandeming conditions with a variety of input levels. Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW). Languages : Finnish & English Subjects : 32 naïve listeners Experiment 2 Purpose : evaluate the performance of AMRWB algorithm, in wide band clean speech (free of background noise) conditions in transcoding with other wideband standards Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW). Languages : French & English (US) Subjects : 32 naïve listeners tandeming and transcoding p 14 research & development France Telecom Group
Compared Subjective WB quality of G.722 & G.722.2 in self tandeming 5 4.5 4 3.5 3 2.5 Lab = Nokia Lab = BT 2 1.5 1 - single coding self-tandem single coding self-tandem single coding self-tandem Direct G.722-64 kbit/s G.722-64 kbit/s Mode 2 (12.65 kbit/s) Mode 2 (12.65 kbit/s) Mode 8 (23.85 kbit/s) Mode 8 (23.85 kbit/s) tandeming and transcoding p 15 research & development France Telecom Group
Compared Subjective WB quality of G.722 & G.722.2 in transcoding 5.0 4.5 4.0 3.5 3.0 2.5 Lab = FT Lab = LMGT 2.0 1.5 1.0 - - - - G.722-64 - G.722-64 Direct G.722-64 G.722-48 kbit/s Mode 2 (12.65 kbit/s) Mode 2 (12.65 kbit/s) Mode 8 (23.85 kbit/s) Mode 8 (23.85 kbit/s) tandeming and transcoding p 16 research & development France Telecom Group
Conclusion on tandemming and transcoding Codecs self tandemings produce quite limited quality degradations of around 0.2 MOS-LQSW. Transcodings between different wideband formats produce more significant degradation : G722 AMR-WB transcoding quality score 0.2 to 0.4 MOS-LQSW below G.722 64 k quality. tandeming and transcoding p 17 research & development France Telecom Group
Comparison od wideband codecs with narrowband codecs tandeming and transcoding p 18 research & development France Telecom Group
Comparison of wideband codecs with narrowband codecs (1) Extract from the G.729.1 characterisation phase (step1) Experiment 1a Narrowband Experiment 1b Purpose : evaluate the performance of G.729.1 algorithm, in wide band clean speech (free of background noise) with a variety of input levels. Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW). Languages : French & English (US) Subjects : 32 naïve listeners tandeming and transcoding p 19 research & development France Telecom Group
Comparison of wideband codecs with narrowband codecs (2) 5.0 4.5 4.0 MOS-LQSM 3.5 3.0 2.5 Lab: VoiceAge Lab : FT 2.0 1.5 1.0 Direct NB Direct WB G.729A -8k (NB) G.722 48k (WB) G.722 56k (WB) G.729.1-14 k (WB) G.729.1-24 k (WB) G.729.1-32 k (WB) tandeming and transcoding p 20 research & development France Telecom Group
Conclusion WB versus NB Results show that wideband voice, even coded at the lowest bit rates of G.722 (48 kbit/s), gets better score than direct narrow band quality with a gap up to +0.5 MOS-LQSM MOS-LQSM difference between narrow band and wideband direct speech is greater than 1 MOS-LQSM and remain between 0.5 MOS-LQSM and 1 MOS-LQSM between direct narrow band and high quality wideband coded speech. tandeming and transcoding p 21 research & development France Telecom Group
References G.729.1 Characterization step 2 references ITU-T-SG12-TD42rev3(WP1/12), " G.729EV Characterization phase step 2 Quality Assessment Test Plan ", Source: Rapporteur for Question 7/12, Geneva, 5-13 June 2006 ITU-T-SG16-TD258(GEN/16), "LS on testing issues", Source: Rapporteurs Q7/12, Geneva, 14-24 November 2006 ITU-T-SG16-TD258(GEN/16)-Attachment 2, " Executive summary of G729.1 Characterisation step 2 Experiments 1, 2 & 3. ", Source: France Télécom, Geneva, 14-24 November 2006 G.729.1 Characterization step 2 references ITU-T-SG12-TD22rev2(WP1/12), " G729EV Characterisation/Optimisation step1 Test plan ", Source: Rapporteur for Question 7/12, Geneva, 17-21 October 2005 ITU-T-SG16-TD202(GEN/16), " LS on audio issues ", Source: Rapporteurs Q7/12, Geneva, 3-13 April 2006 ITU-T-SG16-TD202(GEN/16)-Attachment 1, " G729EV Characterisation/Optimisation step1: Summary of results ", Source: France Télécom, Geneva, 3-13 April 2006 AMR-WB Characterization references ETSI TR 126 976 V6.0.0 (2004-12), " Performance characterization of the Adaptive Multi-Rate Wideband (AMR-WB) speech codec", Tdoc S4 (01)0351R1, " AMR Wideband Characterisation Phase 1 Listening Tests, Experiment 1 - BT Results", Source: BT, June 4-8, 2001, Naantali, Finland Tdoc. S4 (01)0326, " Executive Summary from France Télécom R&D for the ETSI AMR-WB Characterisation Phase Results for Experiments 2 & 5", Source: France Telecom, June 4-8, 2001, Naantali, Finland Tdoc S4 (01)0321, " AMR WB Characterization Experiments 2A and 6A - LMGT Results ", Source: LMGT, June 4-8, 2001, Naantali, Finland Tdoc S4-010353, " Nokia report for AMR-WB characterisation experiments 1A & 6B ", Source: Nokia, June 4-8, 2001, Naantali, Finland EVRC-WB Characterization references ITU-T-SG16-TD291(GEN/16) "Updated LS reply on follow-up on embedded extension to G.722.2 and media coding summary database " Source: Chairman SG 16 (on behalf of 3GPP2 TSG-C) Geneva, 14-24 November 2006 G.722 Packet Loss Concealment references ITU-T-SG16-TD217(WP3/16), " Report of Question 10/16 Software tools for signal processing standardization activities and maintenance and extension of existing voice coding standards ", Source: Rapporteur for Question 10/16, Geneva, 14-24 November 2006Extract from the G.729.1 characterisation phase (step1) tandeming and transcoding p 22 research & development France Telecom Group