INTERNATIONAL TELECOMMUNICATION UNION. SERIES P: TELEPHONE TRANSMISSION QUALITY Methods for objective and subjective assessment of quality

Transcription

1 INTERNATIONAL TELECOMMUNICATION UNION ITU-T P.861 TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (08/96) SERIES P: TELEPHONE TRANSMISSION QUALITY Methods for objectve and subjectve assessment of qualty Objectve qualty measurement of telephoneband ( Hz) speech codecs ITU-T Recommendaton P.861 (Prevously CCITT Recommendaton)

2 ITU-T P-SERIES RECOMMENDATIONS TELEPHONE TRANSMISSION QUALITY Vocabulary and effects of transmsson parameters on customer opnon of transmsson qualty Seres P.10 Subscrbers' lnes and sets Seres P.30 Transmsson standards Seres P.40 Objectve measurng apparatus Seres P.50 Objectve electro-acoustcal measurements Seres P.60 Measurements related to speech loudness Seres P.70 Methods for objectve and subjectve assessment of qualty Seres P.80 P.300 P.500 P.800 Audovsual qualty n multmeda servces Seres P.900 For further detals, please refer to ITU-T Lst of Recommendatons.

3 ITU-T RECOMMENDATION P.861 OBJECTIVE QUALITY MEASUREMENT OF TELEPHONE-BAND ( Hz) SPEECH CODECS Summary Ths Recommendaton descrbes an objectve method for estmatng the subjectve qualty of telephone-band ( Hz) speech codecs. Ths Recommendaton specfes the producton of source speech for objectve qualty measurement, codec and reference condtons for whch the objectve qualty measurement method has been shown to provde vald results, the calculaton of objectve qualty based on the objectve qualty measure called the Perceptual Speech Qualty Measure (PSQM), the estmaton of the subjectve qualty from the objectve measurement results and an analyss of the results. Ths Recommendaton can be appled when evaluatng the effects on subjectve qualty of speech codecs of speech nput levels, talkers, bt rates and transcodngs. Source ITU-T Recommendaton P.861 was prepared by ITU-T Study Group 12 ( ) and was approved under the WTSC Resoluton No. 1 procedure on the 30th of August Keywords Objectve speech qualty measurement, Subjectve speech qualty evaluaton.

4 FOREWORD ITU (Internatonal Telecommuncaton Unon) s the Unted Natons Specalzed Agency n the feld of telecommuncatons. The ITU Telecommuncaton Standardzaton Sector (ITU-T) s a permanent organ of the ITU. The ITU-T s responsble for studyng techncal, operatng and tarff questons and ssung Recommendatons on them wth a vew to standardzng telecommuncatons on a worldwde bass. The World Telecommuncaton Standardzaton Conference (WTSC), whch meets every four years, establshes the topcs for study by the ITU-T Study Groups whch, n ther turn, produce Recommendatons on these topcs. The approval of Recommendatons by the Members of the ITU-T s covered by the procedure lad down n WTSC Resoluton No. 1 (Helsnk, March 1-12, 1993). In some areas of nformaton technology whch fall wthn ITU-T s purvew, the necessary standards are prepared on a collaboratve bass wth ISO and IEC. NOTE In ths Recommendaton, the expresson Admnstraton s used for concseness to ndcate both a telecommuncaton admnstraton and a recognzed operatng agency. ITU 1996 All rghts reserved. No part of ths publcaton may be reproduced or utlzed n any form or by any means, electronc or mechancal, ncludng photocopyng and mcroflm, wthout permsson n wrtng from the ITU. Recommendaton P.861 (08/96)

5 CONTENTS 1 Scope References Abbrevatons Defntons Conventons Summary of objectve measurement procedure Source speech materal preparaton Real voces Artfcal voces Selecton of expermental parameters Calculaton of objectve qualty Global ntalzatons Tme algnment Global scalng Global calbraton Tme-frequency mappng Wndowng Sampled Spectral Power Densty (SPD) Frequency warpng and flterng Sampled Ptch Power Densty Local scalng Telephone-band flterng Hoth nose Intensty warpng Cogntve modellng Loudness scalng Sampled nose dsturbance densty Asymmetry processng Nose dsturbance ncludng slent nterval processng Transformaton from the objectve qualty scale to the subjectve qualty scale Mean opnon scores Equvalent-Q values Analyss of results Bblography Page Recommendaton P.861 (08/96)

6 Page Appendx I Contents of floppy dskette Accompanyng Recommendaton P I.1 Introducton I.2 \src drectory I.3 \test drectory v Recommendaton P.861 (08/96)

7 Introducton Subjectve qualty assessment of speech codecs s one of the key technologes n desgnng dgtal telecommuncaton networks. Recommendaton P.830 defnes subjectve testng methodologes for speech codecs. Snce subjectve qualty assessment s tme-consumng and expensve, t s therefore desrable to develop an objectve qualty assessment methodology to estmate the subjectve qualty of speech codecs wth less subjectve testng. The most wdely-used objectve speech qualty measure demonstratng the performance of speech codecs s the Sgnal-to-Nose Rato (SNR = S/N). However, t s ponted out that the SNR does not adequately predct subjectve qualty for modern network components. Ths s especally true for recent low bt-rate codecs. Therefore, a varety of more sophstcated objectve qualty measures, such as the LPC Cepstrum Dstance Measure (CD) [1], Informaton Index (II) [2], Coherence Functon (CHF) [3], Expert Pattern Recognton (EPR) [4], and Perceptual Speech Qualty Measure (PSQM) [5] were developed. The performance of these systems, n terms of ablty to gve accurate estmates of subjectve qualty, has been nvestgated n ITU-T snce the 1980s. After careful comparsons among these objectve qualty measures, t was concluded that the PSQM best correlated wth the subjectve qualty of coded speech. Therefore, ths Recommendaton descrbes objectve qualty assessment wth the PSQM as the objectve qualty measure [12]. In order to assst the readers of ths Recommendaton n the development of ther own mplementaton of the PSQM, a floppy dskette has been ncluded wth ths Recommendaton. A descrpton of the contents of ths dskette can be found n the README fle on the dskette and n Appendx I. Recommendaton P.861 (08/96) v

8

9 Recommendaton P.861 OBJECTIVE QUALITY MEASUREMENT OF TELEPHONE-BAND ( Hz) SPEECH CODECS (Geneva, 1996) 1 Scope Subjectve qualty assessment of speech codecs can be made n lstenng-only (one-way) tests or n conversatonal (two-way) tests. The objectve qualty measurement descrbed n ths Recommendaton estmates the subjectve qualty n lstenng-only tests. To demonstrate the subjectve performance of a codec, the effects of a varety of qualty factors should be nvestgated (see Recommendaton P.830). The accuracy of the objectve qualty measurement descrbed n ths Recommendaton has not been verfed for examnng all of the factors specfed n Recommendaton P.830. Table 1 s ntended to be a gude to facltate the readers' determnaton of the test factors, codng technologes and applcatons to whch ths Recommendaton apples. TABLE 1/P.861 Relatonshp of codng technologes, expermental factors and applcatons to ths Recommendaton Test factors Note Speech nput levels to a codec 1 Lstenng levels n subjectve experments 2 Talker dependences 1 Multple smultaneous talkers 2 Transmsson channel errors 2 Bt rates f a codec has more than one bt-rate mode 1 Transcodngs 1 Bt-rate msmatchng between an encoder and a decoder f a codec has more than one bt-rate mode 2 Envronmental nose n the sendng sde 2 Network nformaton sgnals as nput to a codec 2 Musc as nput to a codec 2 Delay 3 Short-term tme warpng of audo sgnal 2 Long-term tme warpng of audo sgnal 4 Temporal clppng of speech 2 Ampltude clppng of speech 2 Codng technologes Waveform 1 CELP and hybrds 4 kbt/s 1 CELP and hybrds <4 kbt/s 2 Recommendaton P.861 (08/96) 1

10 TABLE 1/P.861 (concluded) Relatonshp of codng technologes, expermental factors and applcatons to ths Recommendaton VOCODERs 2 Other coders 2 Applcatons Coder optmzaton 1 Coder evaluaton 1 Coder selecton 2 Network plannng 5 Lve network testng 6 In-servce non-ntrusve measurement devces 3 NOTES 1 The objectve measure has demonstrated acceptable accuracy n the presence of ths varable. 2 Insuffcent nformaton s avalable about the accuracy of the objectve measure wth regard to ths varable. 3 The objectve measure s known to provde naccurate predctons when used n conjuncton wth ths varable, or s otherwse not ntended to be used wth ths varable. 4 The objectve measure s known to provde naccurate predctons when there s a sgnfcant amount of wander (more than 10% of the frame length). The applcablty of the measure when there s a small amount of wander s for further study. 5 Wth cauton, the objectve measure mght be used for some network plannng purposes. The reader should note that there are mportant factors n network plannng to whch ths Recommendaton s not applcable (see the "Test factors" secton of ths table). 6 Wth cauton, the objectve measure mght be used for some lve network testng. The reader should note that there may be factors or technologes n a lve network connecton to whch ths Recommendaton s not applcable (see the "Test factors" and "Codng technologes" sectons of ths table). When comparng a codec wth another codec or wth a reference condton based on subjectve expermental results, statstcal tests that take the dstrbutons of subjectve votes nto account are often used. Snce the objectve measurement n ths Recommendaton estmates only the mean of subjectve votes (e.g. MOS, DMOS), such statstcal tests cannot be appled to the results of objectve measurement. Predcton of per cent poor or worse (%PoW) and per cent good or better (%GoB) are currently under study. 2 References The followng Recommendatons and other references contan provsons whch, through reference n ths text, consttute provsons of ths Recommendaton. At the tme of publcaton, the edtons ndcated are vald. All Recommendatons and other references are subject to revson; all users of ths Recommendaton are therefore encouraged to nvestgate the possblty of applyng the most recent edton of the Recommendatons lsted below. A lst of the currently vald ITU-T Recommendatons s publshed regularly. ITU-T Recommendaton P.50 (1993), Artfcal voces. ITU-T Recommendaton P.800 (1996), Methods for subjectve determnaton of transmsson qualty. ITU-T Recommendaton P.810 (1996), Modulated Nose Reference Unt (MNRU). 2 Recommendaton P.861 (08/96)

11 ITU-T Recommendaton P.830 (1996), Subjectve performance assessment of telephoneband and wdeband dgtal codecs. CCITT Recommendaton G.711 (1988), Pulse Code Modulaton (PCM) of voce frequences. CCITT Recommendaton G.726 (1990), 40, 32, 24, and 16 kbt/s Adaptve Dfferental Pulse Code Modulaton (ADPCM). CCITT Recommendaton G.728 (1992), Codng of speech at 16 kbt/s usng low-delay code excted lnear predcton. ITU-T Recommendaton G.729 (1996), Codng of speech at 8 kbt/s usng Conjugate Structure Algebrac Code-Excted Lnear-Predcton (CS-ACELP). CCITT Supplement No. 13 (1994) to the P-Seres Recommendatons. 3 Abbrevatons For the purposes of ths Recommendaton, the followng abbrevatons are used: ACR Absolute Category Ratng CELP Code Excted Lnear Predcton DCR Degradaton Category Ratng DMOS Degradaton Mean Opnon Score MOS Mean Opnon Score PSQM Perceptual Speech Qualty Measure 4 Defntons For the purposes of ths Recommendaton, the followng defnton apples: 4.1 dbov: db relatve to the overload pont of a dgtal system 5 Conventons Subjectve evaluaton of speech codecs may be conducted usng lstenng-only or conversatonal methods of subjectve testng. For practcal reasons, lstenng-only tests are the only feasble method of subjectve testng durng the development of speech codecs, when a real-tme mplementaton of the codec s not avalable. Ths Recommendaton dscusses an objectve measurement technque for estmatng subjectve qualty obtaned n lstenng-only tests. 6 Summary of objectve measurement procedure Fgure 1 llustrates the objectve measurement procedure. Recommendaton P.861 (08/96) 3

12 Parameters n PSQM lstenng level weghtng on slent ntervals envronment nose n the recevng sde characterstcs of hearng threshold sendng and recevng characterstcs of handset Source speech (real and/or artfcal) Sgnal processng to smulate codec/ reference condtons Coded speech Objectve qualty measurement by PSQM Transformaton from objectve qualty scale to subjectve qualty scale Estmated mean subjectve score T FIGURE 1/P.861 Objectve qualty measurement procedure Objectve qualty measurement of speech codecs requres a number of steps: 1) preparaton of source materals,.e. recordng of talkers and/or generaton of the artfcal voces conformng to Recommendaton P.50; 2) selecton of expermental parameters that wll exercse the salent features of the codec and are able to be tested by objectve measurement; 3) producton of coded/reference speech; 4) calculaton of the objectve speech qualty based on the Perceptual Speech Qualty Measure (PSQM), usng source and coded speech; 5) transformaton from the objectve qualty scale to the subjectve qualty scale, f necessary; 6) analyss of results. Each of these steps s descrbed below. 7 Source speech materal preparaton Source sgnals for objectve measurement may be real voces or the artfcal voces specfed n Recommendaton P.50, dependng on the goals of the experment. Snce the artfcal voces defned n Recommendaton P.50 reproduce the mean characterstcs of human speech over varous languages, they are useful n objectvely estmatng the mean subjectve qualty of a codec over these languages. When the talker-dependency of a codec or the performance of a codec for partcular languages s concerned, t s recommended that real voces be used. In ether case, no envronmental nose should be added. 7.1 Real voces When real voces are used n objectve measurement, they should be produced, recorded and levelequalzed n accordance wth clause 7/P Recommendaton P.861 (08/96)

13 It s recommended that a mnmum of two male talkers and two female talkers should be used for each testng condton. If talker dependency s to be tested as a factor n ts own rght, t s recommended that more talkers be used as follows: 8 male; 8 female; 8 chldren. 7.2 Artfcal voces When the artfcal voces conformng to Recommendaton P.50 are used n objectve measurement, t s recommended that both male and female artfcal voces be used. These sgnals should be passed through a flter wth approprate frequency characterstcs to smulate sendng frequency characterstcs of a telephone handset, and level-equalzed n the same manner as real voces (see Recommendaton P.830). ITU-T recommends the use of the Modfed Intermedate Reference System (IRS) sendng frequency characterstc as defned n Annex D/P Selecton of expermental parameters To demonstrate the performance of a codec, the effects of varous qualty factors on the performance of the codec should be examned. Recommendaton P.830 provdes gudance on subjectvely assessng the followng qualty factors: 1) speech nput levels to a codec; 2) lstenng levels n subjectve experments; 3) talkers (ncludng multple smultaneous talkers); 4) errors n the transmsson channel between an encoder and a decoder; 5) bt rates f a codec has more than one bt-rate mode; 6) transcodngs; 7) bt-rate msmatchng between an encoder and a decoder f a codec has more than one bt-rate mode; 8) envronmental nose n the sendng sde; 9) network nformaton sgnals as nput to a codec; 10) musc as nput to a codec. Snce the objectve qualty measure descrbed n ths Recommendaton assumes: 1) source speech s "clean" (.e. wthout added envronmental nose n the sendng sde); and 2) there are no channel degradatons such as transmsson bt errors, frame erasures (e.g. as n moble rado applcatons), or cell loss (e.g. as n ATM networks), the qualty factors to whch ths Recommendaton apples are speech nput levels, talkers (excludng multple smultaneous talkers), bt rates and transcodngs. NOTE 1 Objectve measurement for qualty factors other than those specfcally noted as applcable n ths Recommendaton s stll under study. Therefore, these factors should be measured only after the accuracy of an objectve measure s verfed n conjuncton wth subjectve tests conformng to Recommendaton P.830. Recommendaton P.861 (08/96) 5

14 NOTE 2 - Although there are some ndcatons that the objectve measure can accurately predct qualty under channel degradaton condtons [10][11], the applcablty of the measure to those condtons s stll under study. In addton to the codec condtons, Recommendaton P.830 recommends the use of reference condtons n subjectve tests. These condtons are necessary to facltate the comparson of subjectve test results from dfferent laboratores or from the same laboratory at dfferent tmes. Also, when expressng the objectve test results n terms of equvalent-q values, reference condtons usng the narrow-band Modulated Nose Reference Unt (MNRU) as specfed n Recommendaton P.810 should be tested. NOTE 3 Includng other standard codecs such as G kbt/s PCM, G kbt/s ADPCM, G kbt/s LD-CELP, and G kbt/s CS-ACELP as well as MNRU n objectve qualty measurement may help demonstrate the relatve performance of the codec under test and standardzed codecs. Detaled explanatons of these expermental parameters are found n Recommendaton P Calculaton of objectve qualty Ths clause descrbes a method for measurng the qualty of telephone-band ( Hz) coded speech usng the Perceptual Speech Qualty Measure (PSQM). The objectve of PSQM s to mmc the sound percepton of subjects n real-lfe stuatons [6]. The PSQM smulates experments n whch subjects judge the qualty of speech codecs. It does ths by comparng a coded sgnal to a source sgnal (Fgure 2). Although ths basc prncple of comparson makes t especally suted for Degradaton Category Ratng (DCR) testng, Absolute Category Ratng (ACR) experments can be smulated as shown n the valdaton tests [12]. To the extent that PSQM s a fathful representaton of human percepton and judgement processes, naudble dfferences between nput and output wll receve the same PSQM score. In partcular, f the nput and the output are dentcal, PSQM wll predct perfect qualty rrespectve of the qualty of the nput sgnal. Wthn PSQM, the physcal sgnals consttutng the source and coded speech are mapped onto psychophyscal representatons that match the nternal representatons of the speech sgnals (the representatons nsde our heads) as closely as possble. These nternal representatons make use of the psychophyscal equvalents of frequency (crtcal band rates) and ntensty (Compressed Sone). Maskng s modelled n a smple way: only when two tme-frequency components concde n both the tme and frequency domans, maskng s taken nto account. Wthn the PSQM approach, the qualty of the coded speech s judged on the bass of dfferences n the nternal representaton. Ths dfference s used for the calculaton of the nose dsturbance as a functon of tme and frequency. In PSQM, the average nose dsturbance s drectly related to the qualty of coded speech. The transformaton from the physcal (external) doman to the psychophyscal (nternal) doman s performed by three operatons: tme-frequency mappng; frequency warpng; ntensty warpng (compresson). Besdes perceptual modellng, the PSQM method also uses cogntve modellng [7] n order to get hgh correlatons between subjectve and objectve measurements. Fgure 3 shows a block dagram of the PSQM algorthm. All the parameters and varables n ths clause are summarzed n Tables 2, 3 and 4. 6 Recommendaton P.861 (08/96)

15 nput x[t] Speech codec output y[t] subject propertes lstenng condtons subject model output y[t] Perceptual Model Internal Representaton of the Output subject propertes lstenng condtons Dfference n Internal Representaton Determnes the Audble Dfference Cogntve Model qualty nput x[t] Perceptual Model Internal Representaton of the Input subject propertes T FIGURE 2/P.861 Overvew of the basc phlosophy used n the development of the PSQM A computer model of the subject, consstng of a perceptual and a cogntve model, s used to compare the output of the speech codec wth the nput TABLE 2/P.861 Lst of parameters n PSQM Name Descrpton Value Nb number of bands n crtcal band (Bark) doman (see Table 4) Nf number of samples n tme frame 512 for 16-kHz samplng frequency 256 for 8-kHz samplng frequency F[j] handset recevng frequency characterstcs IRS from Recommendaton P.830 (Table 4 contans the IRS power transfer functon) H[j] Hoth characterstcs (Table 4 contans the addtve power of the Hoth characterstc) P 0 [j] absolute threshold of hearng (Table 4 contans the equvalent power representaton of P 0 [j]) f[j] bandwdth of band j n Hertz (see Table 4) z bandwdth of each subband n crtcal band doman γ exponent of compresson functon W sl weghtng factor on slent frames 0.2 (provsonal) W sp weghtng factor on actve speech frames W sp = (1- W sl )/W sl = 4.0 (provsonal) Recommendaton P.861 (08/96) 7

16 x [n] y [n] Hannng wndow Hannng wndow xw [n] FFT X [k] FFT yw [n] Y [k] Px [k] frequency warpng Px ' [j] calculate local scalng factor Py [k] frequency warpng Py ' [j] S Py " [j] flter wth recevng characterstcs of handset PFx [j] PHx [j] ntensty warpng Lx [j] Hoth nose calculate loudness scalng factor flter wth recevng characterstcs of handset PFy [j] PHy [j] ntensty warpng Ly [j] Cogntve subtracton Sl Ly ' [j] T N [j] asymmetry processng N slent nterval weghtng Nwsl FIGURE 3/P.861 Block dagram of PSQM algorthm 8 Recommendaton P.861 (08/96)

17 TABLE 3/P.861 Varables n PSQM Name m n j k x[m] y[m] S global S p S l x[n] y[n] xw [n] yw [n] X [k] Y [k] Px [k] Py [k] I f [j] I l [j] Px ' [j] Py ' [j] Px ' Py ' Py'' [j] PFx [j] Pfy [j] Phx [j] Phy [j] S Descrpton ndex n tme doman ndex n tme doman n a frame (n: 1, 2, 3,, Nf) ndex for frames ndex n warped-frequency doman (crtcal band doman) (j: 1, 2, 3,, Nb) ndex n frequency doman (Hz) (k: 1, 2, 3,, Nf/2) tme-algned and global-calbrated verson of sampled source speech sgnal tme-algned, global-scaled, and global-calbrated verson of sampled coded speech sgnal scalng factor n global scalng ptch power calbraton factor ptch loudness calbraton factor x[m] n frame I y[m] n frame I wndowed verson of x [n] wndowed verson of y [n] FFT of xw [n] FFT of yw [n] SPD of xw [n] SPD of yw [n] FFT ndex of frst value of k of Px [k] and Py [k] n band j FFT ndex of last value of k of Px [k] and Py [k] n band j sampled Ptch Power Densty of xw [n] sampled Ptch Power Densty of yw [n] power of source speech sgnal n frame power of coded speech sgnal n frame local-scaled verson of Py' [j] telephone-band fltered verson of Px' [j] telephone-band fltered verson of Py'' [j] PFx [j] plus Hoth nose as envronmental nose (recevng) PFy [j] plus Hoth nose as envronmental nose (recevng) scalng factor n local scalng n frame S av Lx [j] Ly [j] average (arthmetc mean) of S sampled Compressed Loudness Densty of source speech sgnal n frame and band j sampled Compressed Loudness Densty of coded speech sgnal n frame and band j Recommendaton P.861 (08/96) 9

18 TABLE 3/P.861 (concluded) Varables n PSQM Name Lx Ly Sl Ly' [j] N [j] C [j] N N wsl M sp M sl N spav N slav Descrpton momentary Compressed Loudness of source speech sgnal n frame momentary Compressed Loudness of coded speech sgnal n frame scalng factor n loudness scalng n frame loudness-scaled verson of Ly [j] sampled nose dsturbance densty n frame and band j asymmetry-effect factor n frame and band j nose dsturbance n frame average nose dsturbance wth weghtng on slent frames number of actve speech frames number of slent frames average of N over actve speech frames average of N over slent frames TABLE 4/P.861 Crtcal band frequency allocatons and flter characterstcs (Based on a 16 khz samplng rate) Band number, j Upper frequency [Hz] Frst FFT Bn n Band j, I f Last FFT Bn n Band j, I l Recevng Characterstc, F Hearng Threshold, P 0 Hoth nose, H dscarded n processng E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E Recommendaton P.861 (08/96)

19 TABLE 4/P.861 Crtcal band frequency allocatons and flter characterstcs (Based on a 16 khz samplng rate) Band number, j Upper frequency [Hz] Frst FFT Bn n Band j, I f Last FFT Bn n Band j, I l Recevng Characterstc, F Hearng Threshold, P 0 Hoth nose, H E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E+02 Recommendaton P.861 (08/96) 11

20 TABLE 4/P.861 Crtcal band frequency allocatons and flter characterstcs (Based on a 16 khz samplng rate) Band number, j Upper frequency [Hz] Frst FFT Bn n Band j, I f Last FFT Bn n Band j, I l Recevng Characterstc, F Hearng Threshold, P 0 Hoth nose, H E E E E E E E E E E E E+02 NOTES 1 The absolute threshold, P 0, uses the calbraton 0 db SPL= The frst upper frequency (15.6 Hz) s equvalent to of a crtcal band. The bandwdth z s of a crtcal band. 9.1 Global ntalzatons Before startng the computaton of the nose dsturbance, whch s the output of the PSQM algorthm, the followng global ntalzatons should be carred out for each par of source and coded speech: tme algnment; global scalng for compensaton of the system gan; global calbraton for settng the loudness of the speech. Snce telephone-band speech codecs usually adopt an nput samplng frequency of 8 khz, ths Recommendaton assumes both the source and coded speech have a samplng frequency of 8 khz or 16 khz (.e. up-sampled by a factor of 2) Tme algnment The frst global ntalzaton that should be carred out s the tme algnment of the source sgnal x[m] and the coded sgnal y[m]. If the sgnals are not algned properly, PSQM cannot be appled. When the tme lag n the coded sgnal relatve to the source sgnal s unknown theoretcally, the tme lag that gves the maxmum of the cross-correlaton between source and coded sgnals can be used as an estmate. For sgnals that show group delay dstorton, the delay that leads to the mnmum PSQM value s the correct one. In the processng, leadng and tralng zeros n the speech fle are dscarded and the start pont and stop pont are calculated by detectng speech actvty usng only the source sgnal. The algorthms for the determnaton of the frst and last actve speech sample are as follows. When determnng the start of actve speech n a fle, the frst sample to be declared actve s the one n whch the magntude (.e. absolute value) of that sample, plus the magntudes of the four precedng samples total 200 or more. (For the purposes of testng the frst four samples for the start of speech actvty, samples precedng the frst sample are consdered to have a value of 0.) When determnng the end of actve speech n a fle, the last sample to be declared actve s the last sample for whch the magntude (.e. absolute value) of that sample, plus the magntudes of the four followng samples total 200 or more. (For the purposes of testng the last four samples for the end of speech actvty, samples followng the last sample are consdered to have a value of 0.) 12 Recommendaton P.861 (08/96)

21 9.1.2 Global scalng After the tme-algnment process, the coded sgnal y[m] s scaled n order to compensate for the overall gan of the system. The scalng factor S global s defned by: S global = stop pont start pont stop pont start pont x y 2 2 [ m] [ m] The coded sgnal y[m] s then multpled by S global Global calbraton In order to ensure optmum accuracy of the objectve measure, t s necessary to provde a calbraton between the lstenng level and the compressed loudness. The values n Table 4 are based on the assumpton that 0 db SPL s equvalent to a maxmum value of 1.0 n the ptch power doman as computed n [.e. max( Px ' [ j]) = 1.0 for a gven frame]. Also assumed s that the optmum j lstenng level of 78 db SPL s used n conjuncton wth speech fles that have an actve speech level of 26 dbov, as ndcated n Recommendaton P.830. The calbratons are performed wth a 1 khz sne wave at a level of 40 db SPL (.e. 64 dbov). Ths s best performed wth a real (.e. non-nteger) sne wave, to avod quantzaton artefacts n the calbraton functon. A level of 40 db SPL corresponds to a zero-to-peak ampltude of The frst calbraton s to scale the maxmum value of the ptch power representaton of the calbraton tone to [.e. f the max( Px ' [ j]) = 1.0 for 0 db SPL, the max( Px ' [ j]) j j = for 40 db SPL]. Ths calbraton factor, S p, s calculated by: S p = max( Px' [ j]) j when Px' [j] (see 9.3.1) s calculated for the calbraton tone. For an mplementaton of PSQM where the FFT s scaled by n, as n the commercally avalable routne "four1" from Numercal Recpes n C [13], S p = e The second calbraton sets the compressed loudness of the calbraton tone, as calculated n 9.4, to 1.0 Compressed Sone. The calbraton factor s calculated by: Sl = 1 Lx when Lx s calculated for the calbraton tone. If the frst calbraton s performed correctly, S 1 = NOTE 1 The calbraton tone should not be fltered through the recevng characterstc, F, nor should Hoth nose be added to the calbraton tone pror to the computaton of Lx and S l. Ths excepton s for calbraton purposes only. If the actve speech level n the dgtal fle s not 26 dbov, or the lstenng level s not 78 db SPL, the nput data should be scaled accordngly. 06 Recommendaton P.861 (08/96) 13

22 NOTE 2 In a 16-bt dgtal fle, 0 dbov s represented by a DC level of Therefore, a snusod wth a 0-to-peak ampltude of would have an RMS level of 3.01 dbov. Wth the assumptons n ths subclause, that would correspond to approxmately 101 db SPL. 9.2 Tme-frequency mappng The mappng from tme doman to tme-frequency doman s mplemented wth a short-term Fourer transform wth a Hannng wndow resultng n a tme-frequency representaton wth a constant resoluton n both tme and frequency domans Wndowng The source sgnal x [n] and coded sgnal y [n] n frame are wndowed usng a Hannng (sn 2 ) wndow: xw [n] = w[n] x [n] yw [n] = w[n] y [n] wth w[n] as the wndow functon. The wndowng functon can be computed as follows: 2π wn [ ] = cos for 0 Nf 1 Nf Throughout clause 9, all calculatons are defned on a frame-by-frame bass. A frame length of 256 samples for 8-kHz samplng and 512 samples for 16-kHz samplng, whch approxmately correspond to the wndow length of the ear, should be used and adjacent frames should overlap each other by 50% Sampled Spectral Power Densty (SPD) The sampled SPDs of xw [n] and yw [n], denoted by Px [k] and Py [k], are calculated by usng the Fast Fourer Transforms (FFTs): xw[ n] FFT X [ k] yw[ n] FFT Y[ k] 2 2 Px[ k] = (Re X [ k]) + (Im X [ k]) 2 2 Py [ k] = (Re Y [ k]) + (Im Y [ k]) 9.3 Frequency warpng and flterng Ths subclause frst descrbes the warpng from the Hertz scale to the crtcal band scale, leadng to a sampled ptch power densty representaton wthn each frame. The sampled ptch power densty of the coded sgnal s scaled wthn each frame after whch both the source and coded sgnals are telephone-band fltered and Hoth nose s added to smulate the lstenng envronment. Fnally, the sgnals are fltered wth the transfer functon from the outer- to nner-ear characterstc Sampled Ptch Power Densty The frequency ndex k n Hz s transformed to a ptch ndex j n the crtcal band doman by a frequency-scale warpng. Frst the crtcal band scale s dvded nto equal nterval bands, and for each band a ptch power densty value (a sample) s computed from the samples (usually more than 1) of the spectral power densty n the correspondng band on the Hertz scale. The sampled ptch power denstes Px' [j] and Py' [j] for band j n frame are found by: 14 Recommendaton P.861 (08/96)

23 I f l [ j] ' j 1 Px[ j] = Sp Px [ k] z I [ j] I [ j] + 1 l l f f If [ j] I j f l [ ] ' j 1 Py[ j] = Sp Py [ k] z I [ j] I [ j] + 1 where I f [j] s the ndex of the frst and I l [j] s ndex of the last sample on the Hertz scale for band j, wth f j as the bandwdth n band j n Hertz, z as the bandwdth of each subband n the crtcal band doman, and S p as the ptch power calbraton factor as ndcated n Local scalng The source and coded sgnals should be scaled wthn each frame, as a compensaton for slow gan varatons. Only audble tme-frequency components are taken nto account (above the absolute threshold of audblty for each band P 0 [j] defned n Table 4). The total powers of the source and coded sgnals n a frame, Px' and Py', are computed usng the warped frequency representaton: Nb Px j j=1 ' Px = '[ ] Py ' Nb = Py [ j] j=1 wth Nb as the total number of bands. When both the power of the source sgnal Px' and the power of the coded sgnal Py' are above 40 db SPL, the power of the coded sgnal for band j, Py' [j], s multpled by a scalng factor S : where: ' Py [ j] = S Py [ j] " ' S ' Px = ' Py When ether the power of the source sgnal Px' or the power of the coded sgnal Py' s below 40 db SPL, the power of the coded sgnal for band j, Py' [j], s multpled by a scalng factor S av whch s the average of all factors S calculated earler Telephone-band flterng Px' [j] and Py'' [j] should be fltered usng recevng characterstcs approprate for a telephone handset: ' PFx [ j] = F[ j] Px [ j] '' PFy [ j] = F[ j] Py [ j] where F[j] s the frequency response n band j of the recevng characterstcs of a handset. ITU-T recommends the use of the modfed IRS recevng characterstcs defned n Annex D/P.830 as recevng frequency characterstcs of a telephone handset. The values of F[j] for these characterstcs are gven n Table Hoth nose In normal telephone use, the speech sgnal s dsturbed by surroundng sounds n the recevng envronment. Wthn PSQM, ths effect s modelled by addng Hoth nose to both the source and If [ j] Recommendaton P.861 (08/96) 15

24 coded sgnals. The Hoth nose [8] s added to the sampled ptch power densty for every value of j, usng the spectral power densty functon as gven n Recommendaton P.800: PHx [ j] = H[ j] PFx [ j] PHy [ j] = H[ j] PFy [ j] where H[j] s the power of Hoth nose n band j gven n Table 4. NOTE All valdatons of the PSQM method wthn the ITU-T were performed usng Hoth nose at a level of 45 dba. 9.4 Intensty warpng After calculatng the sampled ptch power denstes that take nto account telephone-band flterng and Hoth nose, the ntensty scale s warped to a loudness scale leadng to a sampled compressed loudness densty functon. From the ptch power denstes PHx [j] and PHy [j], the sampled compressed loudness denstes Lx [j] and Ly [j] are calculated usng a compresson functon gven by Zwcker [9]: P[ j] PHx[ j] Lx[ j] = Sl γ γ P0 [ j] P[ j] PHy[ j] Ly[ j] = Sl γ γ P0 [ j] wth P 0 [j] as the nternal threshold as gven n Table 4, and S as the ptch loudness calbraton factor as explaned n Negatve values of Lx [j] and Ly [j] are set to zero. The optmal value of γ found n optmzatons usng databases that resulted from dfferent speech qualty evaluaton experments s The (total) momentary compressed loudnesses Lx and Ly (n Compressed Sone) are computed by summaton of the sampled compressed loudness denstes Lx [j] and Ly [j]: Lx = Lx [ j] z Nb j= 1 Ly = Ly [ j] z Nb j= 1 wth z as the bandwdth n the crtcal band doman. The momentary compressed loudnesses Lx and Ly are used n the cogntve modellng. 9.5 Cogntve modellng Wthn the PSQM context, all operatons that cannot be performed on the source sgnal alone or on the coded sgnal alone are defned as cogntve operatons. Four cogntve effects are dscussed n ths subclause: loudness scalng; nternal cogntve nose; asymmetry processng; slent nterval processng. 16 Recommendaton P.861 (08/96)

25 9.5.1 Loudness scalng Wthn PSQM, the sampled compressed loudness densty of the coded sgnal s scaled, wthn each frame, relatve to the loudness of the source sgnal: Ly ' [ j] = Sl Ly [ j] where the scalng factor Sl s computed from the (total) momentary compressed loudnesses Lx and Ly : Lx Sl = Ly When Lx or Ly s below 0.02 Compressed Sone, Sl s set equal to Sampled nose dsturbance densty The sampled nose dsturbance densty N [j] n band j n frame s computed as the absolute dfference between Lx [j] and Ly' [j]: ' N [ j] = Ly [ j] Lx [ j] 001. where the factor 0.01 Compressed Sone represents the nternal cogntve nose. If, because of ths factor, N [j] becomes negatve, then N [j] s set equal to zero Asymmetry processng When a new tme-frequency component s ntroduced n the speech sgnal, the subjectve qualty turns out to be more degraded than when an equally loud component s left out by the speech codec. Ths asymmetry s most promnent durng the slent ntervals. Nose that s present n the source sgnal may be suppressed, leadng to an ncrease n qualty. If there s no nose durng the slent ntervals n the source sgnal, any dfference between source and coded speech leads to a decrease n qualty. Furthermore, when a tme-frequency component s left out of the source sgnal (not coded by the codec), the remanng sgnal s stll one coherent audtory event. If a new unrelated tme-frequency component s ntroduced nto the coded sgnal (a dstorton), the newly-formed sgnal can be decomposed n two parts, the orgnal sgnal and the dstorton. Ths decomposton of the audtory stream makes the nose more annoyng. The asymmetry effect s quantfed by C [j] and taken nto account n the nose dsturbance n frame, N : where: Nb N = N [ j] C [ j] z j = 1 PHy C [ j] = PHx [ j] + 1 [ j] + 1 wth PHx [j] and PHy [j] as the powers of the source and coded sgnals (after IRS flterng and the addton of Hoth nose), respectvely, wthn frame and band j. When PHx [j] and PHy [j] are less than 20 db above the absolute threshold of audblty n band j (.e. PHx [j] and PHy [j] are less than 100 P 0 [j]), C [j] s set equal to 1. The maxmum value of C [j] must be lmted to Recommendaton P.861 (08/96) 17

26 9.5.4 Nose dsturbance ncludng slent nterval processng In PSQM, the slent ntervals are taken nto account usng a weghtng factor, W sl, that depends on the context of subjectve experments. Slent frames are defned as frames for whch the source sgnal has a total power Px' (.e., ' Px[ j] ) below 70 db SPL. If the global calbraton factor, S p, has j been computed correctly the slence threshold s Px' = Frames wth Px' less than ths value are consdered slent. The average nose loudnesses, N spav and N slav, can now be computed over actve speech frames and over slent frames, respectvely: N spav 1 = M sp for actve speech frames N N slav 1 = M sl for slent frames N where M sp s the number of actve speech frames and M sl s the number of slent frames. The nfluence of slent ntervals depends drectly on the length of these ntervals. If the source speech does not contan any slent ntervals, the nfluence s zero. If the source speech contans a certan percentage of slent frames, the nfluence s proportonal to ths percentage. Usng a set of trval boundary condtons, one can show that the correct weghtng s: N w = sl Wsp psp W p + p sp sp sl psl N spav + W p + p sp sp sl wth p sl as the fracton of slent frames, p sp as the fracton of actve speech frames (p sl + p sp =1.0), W sl Wsl as the weghtng factor on slent ntervals, Wsp = 1, and N wsl as the nose dsturbance corrected Wsl wth a weght factor W sl for the slent nterval. Ths nose dsturbance N wsl, called the PSQM value n the followng subclauses, can be used to predct the subjectvely perceved speech qualty obtaned n the Absolute Category Ratng (ACR) method usng the Lstenng Qualty scale. NOTES 1 The value of N wsl should have an upper lmt of For speech materal havng long perods of slent ntervals, the weghtng s dfferent from that for speech materal havng only short perods of slent ntervals. Furthermore, the nose n the recordng of the source materals also has an nfluence on the slent nterval weghtng. For speech materal havng no slent ntervals, the weghtng s not relevant and N wsl becomes equal to N spav. A number of speech databases were examned for determnng the optmal weghtng on the slent ntervals. These databases conssted of speech materal wth about 50% slent ntervals. The optmal weghtng that was found vared between 0.0 and 0.5 [10] [11] [12]. Determnaton of the value of W sl for speech wth slent ntervals s stll under study. Provsonally, a weghtng of 0.2 s recommended for speech materals wth about 50% slent ntervals. N slav 10 Transformaton from the objectve qualty scale to the subjectve qualty scale The output of the algorthm descrbed n clause 9, whch s called the PSQM value, ndcates the degree of subjectve qualty degradaton due to speech codng. Therefore, when estmaton of subjectve qualty on a specfc scale s not necessary, e.g. n optmzng parameters of a codec or n smply comparng the performance of codecs, the PSQM value tself s qute useful. To estmate 18 Recommendaton P.861 (08/96)

27 subjectve qualty on qualty scales such as Mean Opnon Scores (MOSs) and the equvalent-q values, however, the PSQM value s transformed as descrbed below Mean opnon scores In subjectve assessment of the performance of codecs, the ACR method usng the Lstenng Qualty scale specfed n Recommendaton P.800 s often used, gvng subjectve qualty n terms of MOS. Snce the relatonshp between the MOS and PSQM values s not necessarly the same for dfferent languages or even for dfferent subjectve tests wthn a language, t s dffcult to determne a unque functon whch transforms the PSQM value to the estmated MOS value. In practce, therefore, t s necessary to derve such transformaton functons for ndvdual languages and ndvdual subjectve tests n advance. NOTE The absolute value of the MOS depends on the context of the subjectve experment. The estmated MOS obtaned by a predetermned transformaton functon predcts the subjectve qualty n the subjectve experment wth context equvalence to those used n dervng the transformaton functon. When the results are presented n the estmated-mos doman, the transformaton functon from the PSQM value to the MOS value should be reported Equvalent-Q values It s dffcult to compare the MOSs obtaned n dfferent subjectve experments snce subjectve judgement s affected by the expermental settngs, e.g. the range of speech qualty n the experment. Therefore, the equvalent-q value s sometmes used as a subjectve qualty scale. The equvalent-q value s determned as the Q value of MNRU defned n Recommendaton P.810 for whch the MOS s equvalent to that of coded speech. In the objectve measurement, the equvalent-q value can be estmated drectly from the PSQM values for coded speech and MNRU condtons, wthout transformng the PSQM value to the MOS (see Fgure 4). When the results are presented n the estmated equvalent-q doman, the Q versus PSQM-value characterstcs llustrated n Fgure 4 should be reported. NOTE The equvalent-q value becomes relatvely unrelable n the regons of hgh- and low-q value because the Q versus PSQM curve becomes almost flat n these regons. Accordngly, care should be taken when workng n the Q doman wth very hgh- and very low-qualty speech. Recommendaton P.861 (08/96) 19

28 PSQM value for coded speech Increasng PSQM value Equvalent-Q value for coded speech T Increasng Q FIGURE 4/P.861 Determnaton of equvalent-q value of coded speech 11 Analyss of results The analyss of objectve measurement results should be carred out based on the PSQM value, the estmated MOS, or the estmated equvalent-q. For each testng condton, the mean scores over male talkers, female talkers, and ther average should be calculated separately and reported. Calculaton of separate standard devatons for each testng condton s not recommended. Confdence lmts should be evaluated by takng nto account the varaton of objectve qualty over talkers and sentences and sgnfcance tests performed by conventonal analyss-of-varance technques. NOTE The statstcal analyses descrbed here are dfferent from those n subjectve assessment where the means of subjectve qualty are statstcally evaluated by takng nto account the varatons over subjects as well as talkers and sentences. Snce the PSQM cannot estmate the dstrbutons of subjectve votes but only the mean of them, t s mpossble to perform the analyss over subjects. Estmatng the dstrbutons of subjectve votes s stll under study. Therefore, when the analyss over subjects s necessary, subjectve experments conformng to Recommendaton P.830 should be conducted. Bblography [1] NTT: Transmsson performance objectve evaluaton model for fundamental factors, CCITT Contrbuton COM XII-174, November [2] LALOU (J): The nformaton ndex: an objectve measure of speech transmsson performance, Annales des Télécommuncatons, Volume 45, No. 1-2, pp , CNET/France, [3] BNR: Evaluaton of non-lnear dstorton va the coherence functon, CCITT Contrbuton COM XII-60, Aprl Recommendaton P.861 (08/96)

29 [4] KUBICHEK (R.F.), QUINCY (E.A.), KISER (K.L.): Speech Qualty Assessment Usng Expert Pattern Recognton Technques, Proceedngs of the IEEE Pacfc Rm Conference on Computers, Communcaton, and Sgnal Processng, June [5] Royal PTT, Netherlands: Measurng the qualty of audo devces, CCITT Contrbuton COM XII-114, Geneva, December [6] BEERENDS (J.G.), STEMERDINK (J.A.): A Perceptual Speech-Qualty Measure Based on a Psychoacoustc Sound Representaton, J. Audo Eng. Soc., Vol. 42, No. 3, pp , March [7] BEERENDS (J.G.): Modellng Cogntve Effects that Play a Role n the Percepton of Speech Qualty, Speech Qualty Assessment, Workshop papers, Bochum, pp. 1-9, November [8] CCITT Supplement No. 13 to P-Seres Recommendatons, Nose Spectra clause 2, Blue Book, Fasc. 5, ITU, Geneva [9] ZWICKER (Feldtkeller): Das Ohr als Nachrchtenempfänger, S. Hrzel Verlag, Stuttgart, [10] Royal PTT, The Netherlands: Correlaton of a perceptual qualty speech measure wth the subjectve qualty of the CCITT LD-CELP (G.728) speech codec, ITU-T Contrbuton COM 12-10, Geneva, March [11] Royal PTT, The Netherlands: Correlaton between the PSQM and the subjectve results of ITU-T 8 kbt/s 1993 speech codec test, ITU-T Contrbuton COM 12-31, Geneva, September [12] NTT: Revew of valdaton tests for objectve speech qualty measures, ITU-T contrbuton COM 12-74, Geneva, May [13] PRESS (W.H.) et al.: Numercal Recpes n C, The Art of Scentfc Computng, Cambrdge Unversty Press, Cambrdge, Massachusetts, Appendx I Contents of floppy dskette accompanyng Recommendaton P.861 I.1 Introducton In order to assst the readers of ths Recommendaton n the development of ther own mplementaton of the PSQM, a floppy dskette has been ncluded wth ths Recommendaton. Ths appendx provdes a descrpton of the fles on that dskette. I.2 \src drectory Ths drectory contans an example of a "C" code mplementaton of the algorthm descrbed n ths Recommendaton. The code s desgned to work wth 16-bt lnear PCM speech fles sampled at 16 khz and stored wth the least sgnfcant byte (LSB) frst. The followng fles are ncluded n ths drectory: psqm.c psqm.h psqmprot.h psqmvals.c readsamp.c spchopn.c spchread.c The contents of the fles are descrbed below. Recommendaton P.861 (08/96) 21

30 psqm.c psqm.h psqmprot.h psqmvals.c readsamp.c spchopn.c spchread.c Ths fle contans the majorty of the logc for the computaton of the PSQM. The FFT algorthm used was not ncluded due to copyrght restrctons. That algorthm was from Numercal Recpes n C. (Press, W.H., et al. [1988] Numercal Recpes n C, The Art of Scentfc Computng, Cambrdge Unversty Press, Cambrdge, Massachusetts.) Ths fle contans the declaratons for the globally avalable varables used n computng the PSQM. Ths fle contans the functon prototypes for all functons used n computng the PSQM. Ths fle contans the value assgnments for the constants and arrays used n computng the PSQM. Integer functon read_samples. Ths functon reads n a sngle frame of speech for processng. Ths fle has a functon whch opens the speech fle and determnes the start and stop ponts based on the algorthm descrbed n 9.1.1/P.861. Ths fle contans nteger functon speech_read. Ths serves as an nterface to the functon read_samples. I.3 \test drectory Ths drectory contans fles for use n testng an mplementaton of the PSQM for accuracy. The fles contaned n ths drectory are: longs.cod longs.src outlong.txt outshort.txt shorts.cod shorts.src The contents of the fles are descrbed below. longs.cod longs.src outlong.txt A coded speech fle ntended to be used n calbratng the PSQM. Stored LSB frst and delayed by 22 samples from the source speech fle. The source speech fle used to create longs.cod. Stored LSB frst. Frame-by-frame output for the long test vector. In addton to the calbraton varables (S p and S l ), the followng nformaton s ncluded n the fle: calculated global scalng factor (S global ) ; start and stop ponts for both source and coded speech fles; frame-by-frame nose dsturbance (N ); frame-by-frame ndcator of frame slence (1=slent, 0=not slent). outshort.txt Step-by-step varable values for the short test vector. Requres the followng values n the program: delay = 0; start pont = 0; stop pont = 511; S global = 1.0. The followng ntermedate values are provded n the fle: nput sequence (x [n], y [n]); wndowed verson of nput sequence (xw [n], yw [n]); sampled Spectrum Power Densty (Px [k], Py [k]; 22 Recommendaton P.861 (08/96)