Auditory Processing of Speech: The COG Effect. Student Researcher: Daniel E. Hack. Advisor: Dr. Ashok Krishnamurthy

Transcription

1 Auditory Proessing of Speeh: The COG Effet Student Researher: Daniel E. Hak Advisor: Dr. Ashok Krishnamurthy The Ohio State University Department of Eletrial and Computer Engineering Abstrat The COG effet is an auditory phenomenon in whih a human listener an pereive the spetral enter of gravity (COG), or entroid, of a frequeny band up to 3.5 Bark wide. Engineering appliations of this effet inlude extrating entroid features for use in speeh reognition algorithms. This work investigates the representation of the spetral entroid in the auditory system using a omputational model of the auditory pathway. Speifially, a model of the representation of sound in the primary auditory ortex (the termination of the auditory pathway in the temporal lobe of the ortex) is used to derive a spetral entroid measure. This measure is then used to predit the results of two COG effet listening experiments, the first a vowel mathing task and the seond a pith mathing task. By demonstrating that the properties of a subset of the ortial representation of sound math those of the COG perept, this work onludes that ortial proessing and the resulting ortial representation may represent the mehanism underlying the COG effet. Introdution and Objetives The human auditory system is superior to mahines in nearly all speeh related listening tasks suh as speeh reognition and speaker identifiation. In light of this fat, researhers have looked to the auditory system for insight in how to improve their algorithms. This trend has led to the inorporation of priniples of auditory proessing into state-of-the-art speeh proessing algorithms. For example, melfrequeny epstral oeffiient (MFCC) feature vetors are routinely used to enode the spetral features of speeh signals as the first stage in speeh reognition and speaker identifiation algorithms. The MFCC is based on two priniples of auditory proessing: first, the frequeny analysis of the ohlea (organ of the human inner ear) may be simulated as a bank of bandpass auditory filters, the output of whih is typially alled the auditory spetrum; and seond, the auditory spetrum is subjet to a seond layer of proessing in whih properties of the spetral profile are extrated. Aordingly, the MFCC is alulated by passing the power spetrum of an input signal through an auditory filterbank (generate auditory spetrum), then applying logarithmi ompression and a osine transform (extrat features of the auditory spetral profile). Thus, by imitating the proessing of the auditory system, the MFCC has enabled inreased performane in speeh reognition and speaker identifiation algorithms. This motivates the study of auditory proessing itself, speifially its speeh proessing properties, as a means to inspire novel speeh proessing algorithms. One suh avenue of researh investigates spetral integration: how the auditory system ombines information aross frequeny. A speeh waveform is a broadband signal, ontaining frequeny ontent from roughly 1 to 6 Hz. The spetral profile of a speeh signal typially ontains several prominent peaks, alled formants, whih orrespond to the resonant frequenies of the voal trat at the time of prodution. The formant frequenies are labeled F1, F2, F3, et., in order of inreasing frequeny. The idea of spetral integration in speeh pereption researh has been formalized into the enter of gravity (COG) effet, whih states that two losely spaed vowel formants are effetively merged into a single spetral prominene whose COG (mean frequeny) determines the phoneti quality of the vowel. Phoneti quality refers to the properties of the vowel whih influene the listener s determination of vowel identity, i.e., the phonemi deision. In other words, in vowels with losely spaed formants, the COG of the two formants is a salient ue whih plays a signifiant role in vowel, and thus speeh, reognition. Independently, the engineering literature has reently inorporated a similar idea into speeh reognition algorithms. A line of researh led by Kuldip Paliwal (Paliwal 1998, Chen et. al. 24) is investigating the 1

2 use of spetral subband entroids (SSCs) as features in speeh reognition algorithms. SSCs are alulated by filtering the speeh signal through a fixed number of spetral subbands, and alulating the entroids of the resulting power spetra. Results show that SSCs an ahieve omparable reognition performane to MFCCs with lean speeh, and better performane than MFCCs with noisy speeh, validating their effiay as speeh reognition features. However, the evolution of SSC alulation has progressed heuristially, through engineering intuition, rather than inorporating knowledge of the auditory proessing of speeh. It is reasonable that better performane ould be ahieved by expliitly inorporating the parameters governing the COG effet into the alulation of SSCs. However, there is dispute in the speeh pereption literature regarding the details of the COG effet: what is the bandwidth of spetral integration; where is the COG enoded in the auditory nervous system (peripherally or entrally); is the COG effet a fundamental property of the auditory system, or exlusive to speeh pereption? This work onsiders suh questions in order to lay the ground work for future appliations of the COG effet in speeh proessing. Methodology The monaural auditory pathway onveys information as a time-varying pattern of neural exitation. It onsists of a omplex network of serial and parallel neural onnetions, onverging in several major neural nulei, and terminating in the primary auditory ortex. It is generally divided into two setions: peripheral proessing (ear and auditory nerve); and entral proessing (brainstem, midbrain, and ortex). Reent work investigating the physiology of the ferret auditory ortex has led Shamma and olleagues to propose a omputational model of auditory proessing, omposed of peripheral (Yang et. al. 1992) and entral stages (Wang et. al. 1995). The peripheral stage simulates the mehanial to neural transdution of auditory information in the ohlea and a lateral inhibitory spetral sharpening proess in the ohlear nuleus. The output of the peripheral stage is the auditory spetrum, a spatial pattern of neural exitation at the output of the ohlear nuleus, whih resembles a sharpened version of the aousti input spetrum. The frequeny axis of the auditory spetrum is alled the tonotopi axis beause in the auditory system frequeny is mapped to loation. Figure 1 shows the spetrum of an input signal and its orresponding auditory spetrum. The entral stage performs a spetral shape analysis, implemented as a multi-sale filtering of the auditory spetrum, resulting in the final ortial representation. This is a simplifiation of the entral stages of the auditory pathway, but is funtionally valid in that it has been shown to aurately predit ortial neural responses to vowel stimuli (Versnel et. al. 1998). The entral stage models the auditory ortex as a population of neurons with reeptive fields (RFs) parameterized along three dimensions: best frequeny x (otaves re. 1 khz), the RF s enter frequeny along the tonotopi frequeny axis; sale Ω (yles/ot), whih ontrols the RF s bandwidth; and symmetry φ (rad), whih ontrols the RF s symmetry w.r.t. the best frequeny. A reeptive field is defined as: RF(x,x,Ω,φ ) = h(x x )osφ h ˆ (x x )sinφ, where h(x) is an even seed funtion, defined as the seond derivative of a Gaussian, and h ˆ (x) its odd Hilbert transform. These are shown in figure 2. Sample RFs are shown in figure 3 for different sales and symmetries. Intuitively, the RF desribes a neuron s exitatory and inhibitory regions. The response of a neuron with reeptive field RF is given as the inner produt of RF with the input auditory spetrum y(x): r( x, Ω, φ ) = y( x), RF( x; x, Ω, φ ) = a( x, Ω )os( Ψ( x, Ω ) φ ) As this equation shows, the maximum response a(x,ω ) is ahieved for symmetry valueφ =Ψ(x,Ω ). If we take this value of φ for all (x,ω ) pairs, the response an be displayed as a 2D funtion of x and Ω. An example ortial response r(x,ω ) is shown in figure 4, for the same aousti signal as figure 1. 2

3 Effetively, the ortial response is alulated by integrating energy over the tonotopi frequeny axis with a variety of RF weighting funtions. A subset of these neural RFs, entered around sale Ω =.34 y/ot, have exitatory bandwidths approximately equal to the bandwidth of spetral integration hypothesized to underlie the COG effet. Fousing on the profile of the ortial response along Ω =.34 y/ot, we define the ortial pereptual COG as the best frequeny x whih gives the maximum response, COG = arg max r( x, Ω ) Ω =.34 / o x Pereptual COG alulation is illustrated in figure 4, where the dotted line represents the ortial response profile along the Ω =.34 y/ot ontour (solid line), and the dashed vertial line represents the COG frequeny. It is important to point out that the auditory spetrum is typially time-varying. Hene, the ortial response and pereptual COG are time varying as well. In all modeling, the input stimuli is split into 1 ms frames, and a ortial response and COG are alulated for eah frame. The COGs are then averaged to produe an average COG, µ og. Average COG values were used to model the results from two listening experiments. The first, reported in Assmann (1991), was a vowel pereption experiment in whih listeners mathed the phoneti quality of multi-formant syntheti vowels. In this experiment, the referene stimuli were 6 formant vowels whih had harmonis in the region of F2 and higher saled by -2, -1,, +1, and +2 db. This had the effet of lowering (-2 and -1) or raising (+1 and +2) the pereptual COG in the F1-F2 formant region, a region ruial in determining phoneti quality. Three different referene (F1,F2) onditions were tested: (35, 5); (45, 7); and (55, 8) Hz. The mathing stimuli were 6 formant vowels with equal F1 and F2 amplitudes, where F2 = F Hz. Listeners adjusted F1 in order to math the phoneti quality of the referene vowel under onsideration. The seond experiment, reported in Feth (1982), was a psyhoaoustis experiment in whih listeners mathed the pith of two-tone signals. In this experiment, the referene stimuli were omposed of two sinusoids at frequenies F1 and F2, where F1 = f f/2 and F2 = f + f/2, with amplitude ratios (A2/A1) of -3, -1, -.5,.5, 1, and 3 db. For A2/A1 >, the X H stimuli, the listeners hear a pith loser to F2, while for A2/A1 <, the X L stimuli, listeners hear a pith loser to F1. This is indiative of pith being determined by some sort of COG mehanism. Fixed enter frequenies f equal to 5, 1, and 2 Hz were used, along with f values equal to 1, 2, 5, and 1 Hz. The mathing stimuli were omposed of two equal amplitude sinusoids, at frequenies f f/2 and f + f/2 Hz. For the mathing stimuli listeners hear a pith equal to f Hz. Listeners adjusted the mathing stimuli f to math the pereived pith of the referene stimuli. Model preditions were obtained for both experiments by mathing referene stimuli µ og to mathing stimuli µ og. In both experiments, the mathing stimuli µ og were monotonially inreasing with inreasing stimulus parameter (F1 in Assmann, f in Feth). For eah referene stimulus, the mathing stimulus with equal µ og was taken as the model predition. Results and Disussion The results of the Assmann experiment are shown in figure 5. The plots show the experimental mathed F1 frequenies for the subjets, along with model preditions. Dashed lines indiate the referene F1 and F2 frequenies. The results indiate that for the db ondition, listeners mathed the referene and mathing F1 frequeny, as expeted. In the +1 and +2 db onditions, listeners inreased the mathed F1 frequeny, as expeted if listeners were mathing the pereptual COG of the F1/F2 region. However, in the -1 and -2 db onditions, listeners mathed the referene and mathing F1 frequenies, rather than dereasing the mathed F1 frequeny to ompensate for an expeted derease in the referene F1/F2 pereptual COG. Assmann interpreted this result as inonsistent with the COG hypothesis. However, the model predits the same pattern of results in all onditions. This suggests that the referene stimuli pereptual COGs, in fat, did not derease in the -1 and -2 db onditions relative to the db ondition. While the COG does derease if alulated as the mean frequeny of a 3.5 Bark region entered between F1 and F2, it does not derease when alulated using the pattern of ortial responses as desribed in the previous setion. We therefore onlude that a COG mehanism is not inonsistent with the results. 3

4 A subset of the results of the Feth experiment is shown in figure 6. The plots show the average differene between X H and X L mathed f values for eah f, f, and I ombination. For example, onsider the upper-left-most data point (f = 5 Hz, f = 1 Hz, I = 3dB), where the mathed frequeny differene is equal to 5.5 Hz. This means that the mathed f for the +3dB (X H ) stimulus was 5.5 Hz higher than the mathed f for the 3dB (X L ) stimulus. The model preditions are shown by the solid and dashed lines. As shown, the model preditions math the trends of the experimental results. These results support the following onlusions. First, the mehanism underlying the COG effet might be a plae oding mehanism in whih the pereptual COG is represented as the peak exitation in a subset of ortial neurons. More speifially, the COG may be enoded as the best frequenies of the olletion of neurons, tuned to a sale of approximately.34 y/ot and maximum symmetry, whih have maximum exitation in response to an input stimulus. This onlusion is supported by the modeling results presented in this work. Seond, the pereptual COG effet involves a COG alulation more involved than simply alulating the mean frequeny of a region of the input stimulus spetrum, as demonstrated by the Assmann experimental and modeling results. This is signifiant beause it suggests that SSC features, whih rely upon alulating simple moments of the stimulus power spetrum, do not losely model the underlying pereptual proess, and that they may be improved by inorporating the ortial COG alulation proposed in this work. Third, the COG effet is not a speeh-speifi effet, but a fundamental property of the auditory system, as demonstrated by the Feth results. Figures Magnitude db Aousti Spetrum Frequeny (Hz) Relative Neural Exitation db Auditory Spetrum Tonotopi Frequeny x (Hz) Figure 1 Comparison of Aousti Spetrum (left) and auditory spetrum (right) of a 1 ms syntheti vowel. Amplitude x (otave) x (otave) Figure 2 Plot of h(x) (left) and h ˆ( x ) (right), used to generate reeptive fields. Amplitude x (otave) x (otave) Figure 3 Example RFs. The left RF is tuned to x = ot, Ω =.3 y/ot, and φ = π/6 rad. The right RF is tuned to x = ot, Ω = 1 y/ot, and φ = -π/4 rad. 4

5 Sale Ω (y/ot) Tonotopi Frequeny (Hz) Figure 4. Example Cortial Response. The solid line indiates f = Assmann Data Model F1 = 35, F2 = 6 Hz 3 2 I = 1 db I = 3 db freq (hz) freq (hz) F1 = 45, F2 = 7 Hz Frequeny Differene X H - X L (Hz) f = f = 2 1 F1 = 55, F2 = 8 Hz freq (hz) weight (db) Frequeny Separation f (Hz) FIG. 5 Assmann (1991). Experimental results show the mean mathed F1 frequenies, with bars representing +/- 1std. Model preditions are shown by solid lines. FIG. 6 Feth (1982). Experimental results show the mean mathed frequeny differene between orresponding X H and X L stimuli. Model preditions are shown by the solid and dashed lines. 5

6 Aknowledgments The author would like to thank OSGC for support, Ashok Krishnamurthy and Larry Feth for guidane, and Shihab Shamma and Taishi Chi for help in understanding the model. Referenes 1. Peter F. Assmann. The Pereption of Bak Vowels: Centre of Gravity Hypothesis. Quart. Journal Exp. Psyh., 43A(3): , Jingdong Chen et al. Reognition of Noisy Speeh Using Dynami Spetral Subband Centroids. IEEE Signal Proessing Letters, 11(2): , Feb Lawrene F. Feth et al. Pith of Unresolved, Two-omponent omplex tones. J. Aoust. So. Am, 72(5): , Nov Kuldip K. Paliwal. Spetral Subband Centroid Features for Speeh Reognition. Pro. of ICASSP 98. 2:617-62, May Huib Versnel and Shihab A. Shamma. Spetral-ripple Representation of Steady-State Vowels in Primary Auditory Cortex. J. Aoust. So. Am., 13(5): , May Kuansan Wang and Shihab A. Shamma. Spetral Shape Analysis in the Central Auditory System. IEEE Trans. Speeh Audio Proess., 3(5): , Sept Xiaowei Yang and Shiha A. Shamma. Auditory Representations of Aousti Signals. IEEE Trans. Info. Theory, 38(2): , Marh