1-4. A Neurornorphic Microphone for Sound Localization I; J

Transcription

1 A Neurornorphic Microphone for Sound Localization Chiang-Jung Pu, John G. Harris, and Jose C. Prinlcipe Coputational Neuro-Engineering Laboratory University of Florida ECE Dept., 453 EB Gainesville, FL Abstct- We propose a localization odel which uses onaural spectral cues for localizing a sound source in a l-d plane. A neuroorphic icrophone is constructed to ipleent this odel. We use the ter neuroorphic because the icrophone s operating principles take advantage of biologically-based onaural cues. By concentrating on onaural cues, we hope to better understand huan sound localization and also build low-cost stereo capabilities into a single icrophone. The Head Related Transfer Function (HRTF) plays a critical role for huan onaural localization since the shape of the external ear (pinna) spectrally shapes the sound differently for each sound source direction. Using HRTFs, huans can perceive the difference between front and back and the sound source s different elevation positions using only a single ear. The neuroorphic icrophone relies on a specially shaped reflecting structure that allows echo-tie processing to localize the sound. Since our recorded signal is coposed of the direct sound and its echo, the sound is a siplified version of actual HRTF recordings which are coposed of the direct sound and its reflections fro the external ear, head, shoulder, and torso. The recorded signal fro our special icrophone is first processed using a gaa filter. The gaa filter generalizes the standard transversal filter by adding the ability to choose an optial tie-scale. Our studies have shown that the gaa filter solutions require on the order of five paraeters while the ore typical FIR filter solutions require hundreds of paraeters. A ultilayer perceptron neural network is then used to learn the elevation angle of the sound-allowing the icrophone to correctly localize sounds. With the successful building of the special icrophone, a full hardware version of a neuroorphic icrophone is possible. I. INTRODUCTION In recent years, increasing attention has been focused on localization by huan listeners. Huan three-diensional auditory localization has been regarded as a special case of passive localization by a syste consisting of two sensors, two ears. In ost huan sound localization studies, the sound sources have been restricted to one of two planes, either the horizontal plane or the vertical (id-sagittal) plane. The distinction between horizontal and vertical localization also appears to be justified by differences in the principal spatial cues for horizontal and vertical localization (i.e. interaural difference cues vs spectral cues). Sound n-aves incident onto the ears of a listener are reflected by the head on the side facing the incident wave, diffracted to the ear on the shadowed side of the head, and transitted to the eardru via the pinna. These reflections and diffractions produce interaural tie difference (ITD) and interaural intensity difference (IID), which are well-known interaural difference cues. The pinna cavity syste, via relative wave otion within the cavity syste, produce i Sound Input 1-4 Recording Prcproccssor 1 Sound Sourcc Location Localization Model 1 with Long-Tcr lntcgration Frcqucncy Gaa odcl Sound input I; J Sound Source Location Neuroorphi c Mi crophone Fig. 1. (a) Proposed 1ocal:ization odel (b) A neuroorphic icrophone- ipleentation of the localization odel in (a) direction-dependent sound spectra at the ears. The socalled head related transfer function (HRTF) describes the sound signal in the ilddle ear at each location. This iddle ear s signal includes, a direct sound fro the source and several delayed sounds ostly caused by the external ear s reflections. These spectra are known to be significant for sound localization in the vertical plane. We are concentrating 011 onaural spectral cue localization. A localization odel is proposed in Figure l(a). In general, our proposed localization odel includes the recorded sound (e.g. actual HRTFs or our neuroorphic icrophone recordings), a preprocessor used to process ore realistic sound signals (not only ipulse responses), a feature extractor, and a ultilayer perceptron neural network which is trained to learn HRTFs. A neuroorphic icrophone is constructed to ipleent this localization odel. We use the ter neuroorphic because the icrophone s operating principles take advantage of biologicallj-based onaural cues. By concentrating on onaural cues, we hope to better understand huan sound localization and also build low-cost stereo capabilities into a single icrophone. This syste could be used in low EEE 1469

2 cost teleconferencing applications. The proposed neuroorphic icrophone is depicted in Figure l(b). The neuroorphic icrophone relies on a specially shaped reflecting structure that allows echo-tie processing to localize the sound. Since our recorded signal is coposed of the direct sound and its echo, the sound is a siplified version of actual HRTF recordings which are coposed of the direct sound and its reflections fro the external ear, head, shoulder, and torso. The recorded signal fro our special icrophone is first processed using a gaa filter[6] to extract features fro the recordings. A ultilayer perceptron neural network is then used to learn the elevation angle of the sound-allowing the icrophone to correctly localize sounds. Section I1 presents a brief introduction of sound localization cues. Section I11 discusses our icrophone recordings and preprocessing techniques. Section IV discusses feature extraction using the gaa filter and the localization result of a siple ultilayer perceptron output. Section V concludes this work. 11. SOUND LOCALIZATION CUES It is well known that sounds incident in the horizontal plane can be localized on the basis of phase-derived interaural tie difference. It can be shown that at low frequencies the interaural tie difference (ITD) can be approxiated by: [91 3a. ITD 2 -S2qinc) (1) C where c is the abient speed of sound, a is the radius of the head, and Oinc is the angle of incidence. At high frequencies, the priary localization cue in the horizontal plane is interaural intensity difference (IID). The IID is defined as IID(dB) = 2OlogP~ - 2ologPR (2) where PL and PR are pressure agnitudes at the left and right ears. For frequencies below 1000Hz, the sound wavelength can be several ties larger than the head, the aount of shadowing (which depends on the wavelength of the sound copared with the diensions of the head) is negligible. At higher frequencies, interaural phase would be abiguous by ultiples of 360. Therefore, the accuracy of localization of low frequencies would be attributed to ITD and that at high frequencies to IID. In the range fro 1500 to 3000Hz, stiuli are too high in frequency to provide a usable phase cue and too long in wavelength to provide adequate IID. It is not suprising that localization perforance is worst for sound signals fro Hz. To the extent that the head and the ears are syetrical, a stiulus presented at any location on the edian plane should produce no interaural differences and thus, interaural differences should provide no cue to the vertical localizations. Batteau[l] was one of the first to ephasize that the external ear, specifically the pinna, could be a. source of spatial cues that ight account for vertical localization. He easured the ipulse response of an enlarged pinna replica and concluded that the physical structure of 1 - f Ll De ay 1OOto300 Tv p s w Fig. 2. Batteav external ear odel the external ear introduced two significant echoes in addition to the original sound. One echo varies with the aziuthal position of the sound source, having a latency in the 0 to 8Ops range, while the other varies with elevation in the loops to 300ps range. Batteau s odel is depicted in Figure 2. The output y(t) is related to the input x(t) as y(t) = z(t) + a1z(t - 7,) + azz(t - TV) (3) where raru refer to aziuth and elevation echoes respectively; al, a2 are two reflection constants. The frequency response of this odel is Watkins[7] used Batteau s results to synthesize directionencoded sound, and he proved the odel effective. Hiranaka et al. easured the ipulse response of the pinna fro nine subjects. Their results showed that a huan pinna works as a copound sound reflector which produces ajor reflected coponents within 35Ops[4]. One of their conclusions fro their easureents is: The tie interval between the first peak and those ajor reflections increases as the source oves downward. That is, the first echo s delay tie after the direct signal is increased as the sound source is oved fro above to below. By transforing this tie doain response to frequency doain, the echo s delay tie deterines a notch s frequency in frequency doain. Larger echo delay eans a notch igrates toward the lower frequencies in frequency doain. Butler et al. ade siilar observations for frequency doain spectral cues[2]. The tie interval can give us the key to deterine a sound source s elevation angle. A convenient and atheatically tractable geoetry for siulating the pinna s curved surfaces is the parabola. If the concha wall and helix are cobined to for one effective parabola and if the ear canal is placed off the parabola s axis, this can approxiately siulate the pinna. The precise locations of notches can be adjusted systeatically by adjusting the parabola s focal length, and the off-axis placeent of the ear canal entrance[g]. 1470

3 ~ (ear ~ Abpve (k,,,/@\ / t\ Microphone (cxtcal ear) ~ Below canal) Sound source location Fig. 3. Recording of the ezternal ear odel 111. MICROPHONE RECORDINGS AND PREPROCESSING As discussed in the previous section, a parabolic curved surface is a siple geoetry for siulating the external ear. The concha wall and helix are cobined to for the parabola, the ear canal is placed off the parabola's axis. A single reflection produces the interference pattern which encodes the sound source's location in the front half plane. Since our goal is to design a single icrophone for localizing a sound source in the 1-D plane, we do not attept to copare the accuracy of the parabolic odel's response with the easured external ear's response. Figure 3 shows our parabolic external ear odel which is used to record the sound signal. The icrophone (which iics the ear canal in the external ear odel) is placed off the parabola's axis. The purpose of the icrophone's off-axis position is to provide asyetric response as a sound source oves fro above to below. Since our recorded signal is coposed of the direct sound and its echo, the sound is a siplified version of actual HRTF recordings which are coposed of the direct sound and its reflections fro the external ear, head, shoulder, and torso. Coparing our siplified HRTF with actual HRTF, both functions provide spatial inforation, but there exist two significant differences. The first difference is our siplified HRTF is recorded in real-world surroundings instead of HRTF which is easured fro huan or KEMAR's' pinna in the laboratory. The second difference is the diension of our reflector is uch larger than the pinna. The frequency response of our recording odel is where At is the echo delay tie and a1 is the reflection constant. For the ideal case, this reflection odel give us a single dip at FN (z A). We do not attept to copare the accuracy of our recordings with actual HRTFs. Our large (in reference to the size of the pinna) reflector provides observable echo delays at various sound locations. Our parabolic reflector is 21c in height and loc in width. Roughly estiated, the echo tie delay fro this parabolic reflector is at the range 'KEMAR is Knowles Electronic Manikin for Acoustic Research 6 frequency response of the recordings I I I I I I sound signal: a gun-shot ri I I I I I I I I 1 location at 10 Deg I I I I I 30 $50 - location at 170 Deg. 30- ' I I I I I Fig. 4. Frequency responses of our recordings at various sound source directions. (a) a gun shot signal response [b) source locates at 10' (c) source locates at 90 [d) source locates at 170' of 600ps. Thus, the dip frequency caused by the reflector is about 833Hz. In our further signal processing techniques, we only dealt with frequencies between 550 to 1450Hz. This frequency range is deterined by the range of possible reflection distances fro the set-up of the parabolic reflector. This frequency bandwidth provides inforation of reflection caused by the reflector and gets rid of the unwanted reflections caused by the surroundings. The setup of our recording is depicted in Figure 3. A gun shot was used as an ipulse sound signal. Sound pickup was a condenser icrophone (B&K 4133) which had been calibrated and had relative flat frequency response up to 10KHz. The recordings were digitized at 44.1KHz sapling frequency. Figure 4l:a) is the frequency response of a gun shot signal. Three different directions (O=lOo, go", 170", 0 is defined in Figure 3) were tested in the recording. Because the icrophone was placed off the parabolic surface's axis, the reflection path is longer than O=170". Therefore, the notch frequency igrates toward the lower frequencies as the source is oved fro above toward below. The recordings are consistent with our earlier discussions. Since one of our goals is to build a practical sound localizer, we choose speech as a ore useful testing sound signal for our recordings. The tie duration for each syllabi in the speech signal is at least hundreds of illiseconds. Be- 1471

4 - 701, 65 % 60 v 2 e 55 U sec speech with no reflection 45' ld00 lib0 ' Frequency(Hz) 60 sec speech with reflection I frequency response of speech.- c a I I I I 6 50' ' I I I % t g60 $55 U 50 45' I00 2AO EA lob0 lib0 ' Frequency(Hz) Fig. 5. (a)integrate 60 second speech (b) sae as (a), with a reflected echo cause the delay tie of the reflections fro our parabolic reflector is around ls, it would be very difficult (if not ipossible) to observe the echoes fro recorded speech signal without further preprocessing. To do the preprocessing, we integrate the recorded speech signal with a long tie window and create a Fourier transfor. For speech signals, the frequency coponents in the longer tie window cover any ore frequencies copared to a shorter tie window. The long tie integration of the speech signal approxiates the frequency response of an ipulse. After transforing into the frequency doain, the echo of the speech would be found in the proper frequency range. The preprocessing is a long-ter integration which integrates the signal in a uch longer period of tie (e.g. 20 seconds). Figure 5 shows the frequency response of a preprocessed 60-second speech signal. The 60-second speech was recorded at an IlkHz sapling frequency in a typical graduate student office. A subject was seated in a fixed position. A B&K 4133 condenser icrophone was placed 10c away fro the subject on the height level of the subject's outh. A DSC240 audio aplifier was used to aplify the recorded signal fro the icrophone. Figure 5(a) shows the result when the subject spoke without the presence of the reflector. Figure 5(b) is siilar to (a) except a reflector was placed 15c away fro the recording icrophone. A dip at 580Hz is expected because of the reflection caused by the reflector. With the sae experiental setup described in gun-shot experient previously, the subject was asked to change three different directions at 8 = 10",90", and 170" (8 is defined in Figure 3). Figure 6(a), (b), and (c) are the three recordings which the subject changed 8 directions. Siilar to the gun-shot recording, the notch frequency igrates toward the lower frequencies as the subject changed directions fro 6' = 10" to i9 = 170". We have recorded the siplified HRTFs which include spatial inforation in a 1-D plane. Copared to actual HRTFs, our siplified HRTF provides a systeatic change of the dip frequency as a sound source oves in the 1- i3 E location at 90 Deg Frequency(Hz) $ E" a 1 I I I 55' ' I 4 I I I I 500 Frequency(Hz) Fig. 6. Frequency responses of our recordings at various sound source locations for speech signal. (a)source locates at 10' (b)source locates at 90' (c)source locates at 170' Fig. 7. The generalized feedforward filter D plane. For actual HRTFs, because of ore reflections coing fro the pinna, torso, and shoulder, the change of each HRTF is far ore coplicated even when a sound source is oved in a 1-D plane. Therefore, it takes ore effort to extract spatial inforation fro actual HRTFs than our siplified HRTFs. In the following section, the gaa filter is applied to extract the spatial inforation fro actual HRTFs[8]. IV. FEATURE EXTRACTION, AND LEARNING HRTF Fro the previous descriptions of sound localization cues, the external ear provides extreely iportant inforation for spatial hearing. Physical odels seek to represent the physical echanis by which the external ear transfors the incident sound signal. Various physical odels have been proposed and evaluated in the past few decades. Batteau's tie doain odel (Figure a), was based on the external ear's structure. The liitation of I 1472

5 x 10' Fig. 8. gaa odel to fit easured HRTFs L-4 Gaa, Elev-70 Tap No Fig. 10. Use diflerent gaa taps to fit HRTF The gaa transfer function for each tap in the Z-doain is therefore cl L=4 Gaa, Elev-0 1 I : : tieh" Fig. 9. Use 4-tap gaa filter to fit HRTF for [a) elevation=7o0 (b) e1evation=o0. Botto curue at each plot is the difference between the easured HRTF and the gaa approxiation Batteau's odel is that it does not give an accurate representation of the external ear's response copared to easured HRTFs which are ade possible because of the odern recording techniques. Other external ear odels such as the beaforer odel cobine FIR filters and updating weights to represent the external ear's transfer function[3]. Even though great accuracy has been reported, the beaforer odel requires hundreds of free paraeters which hinder its useful application. In order to siplify HRTFs, the question is could we use fewer variables to represent the external ear's transfer function? To siplify the odeling of the external ear's function, we would like to decrease the nuber of free paraeters. Replacing FIR filters with IIR filters which include feedback signals can decrease the nuber of variables. However, one of IIR filters ajor concerns is stability. To utilize IIR filters efficiently and to avoid typical IIR filters' coplicated stability probles, we propose to use the gaa filter which is a class of IIR filter with restricted feedback[6]. The gaa filter is depicted in Figure 7. The output of the gaa filter is defined in the tie doain as k=l where the xk(rl) are recursively coputed using a(.) = (1 - p)"k(n - 1) + Pk-l(n - 1) (7) I The stability of the gaa filter is guaranteed if the poles are located within the unit circle. As a result, the gaa filter is stable when O< p <2 [6]. The applied gaa odel is depicted in Figure 8. The desired function, d(n), is a actual HRTF[8] (or siplified HRTFs in our recordings). The tested HRTFs in the following are fro +80" to -40" in steps of -10' in elevation and zero aziuth (which is straight ahead of the head). The output of the gaa odel is the su of the product of each tap output and its weight. It is expressed as equation(6), where is the tap nuber. The error function at tie n is e(n) = d(n) - y(n) = d(n) - cwk(n)zk(n) (9) k=l The weight values, wk(n), are updated to iniize the ean-square error between the gaa odel response and the easured response. The proble is to iniize in llti - w k~k1)~ (10) The tie constant, p, in the gaa odel is also updated to iniize the ean-square error. With the flexibility of variable tie constant p and updating weights, gaa odel fits the easured HRTFs well in our study. Thus far we have applied the gaa odel to fit easured HRTFs. With only 4-tap gaa filters, each HRTF is represented well by only 6 paraeters (tie constant p, and 5 weight values). This gaa odel siplifies HRTF representation fro the hundreds of variables necessary in the FIR case. Our next step is to use the 6-paraeter HRTF to localize the sound source. An iportant question is whether each HRTF is represented by the 6 paraeters uniquely? Since the standard gaa odel continuously updates the weights and its tie constant p, the ean-square error function is not quadratic. That is, there ay exist several local inia in the error function. It is possible that

6 - I the search of the iniu error becoes trapped in a local inia during the iniization of the error function. Therefore, for each HRTF, we have several different solutions based on the initial weight values. Sall changes in the tie constant ay lead to large variations in the weights. Localizing a sound source based on these nonunique solutions would be disastrous. The reason of gaa filter's powerful perforance is its ability to choose an optial tie-scale (p), but the adaptation of the tie constant p contributes to a coplicated error function. To overcoe this proble, we tested a fixed tie constant p for all HRTFs. Fortunately, the gaa odel fits to actual HRTFs reains excellent. With a constant p, the error function is greatly siplified to a quadratic forat. There are no local inia in the error function. That is, the solution to each HRTF is not affected by the initial weight values. The fitting results with a fixed p is shown in Figure 9. A four-tap gaa is applied to fit a HRTF with a constant p value. Figure 9 (a) and (b) are HRTFs at two different elevations. The botto curves of plot(a) and (b) are the differences between the easured HRTFs and its gaa odel responses. These easured and odeled differences show that the gaa odel is able to fit HRTFs well even if the tie constant p is fixed. The advantage of the gaa odel is to siplify the HRTF representation fro hundreds of coponents to a unique 5-paraeter (five weight values) solution to each HRTF. We have applied a 4-tap gaa odel to fit HRTFs and generate a unique 5-paraeter solution to each HRTF. Our experiental data in Figure 10 shows that a 4-tap gaa is able to fit HRTFs well under a constant p value. We applied the gaa odel to fit 13 HRTFs which are at zero aziuth, ranging fro +80" to -40" in steps of -10" in elevation. Each HRTF is encoded to a fiveparaeter solution. Even though we could not find any systeatic changes of these HRTFs by coparing any two neighboring HRTFs, huans are able to perceive the sound source's position correctly within liited error (5"). It sees that our auditory syste has soe "black box" to decode these HRTFs. Neural networks odels have been proposed to solve this "black box" [5]. With the networks' nonlinearity and proper configuration, it is not difficulty to classify these HRTFs after training of the networks. Instead of classifying these HRTFs, we are ore interested to see the network's interpolation (learning) ability after training. That is, after proper training of HRTFs, how is the networks' localization perforance based on gaa odel's 5-paraeter representation? We split these 13 5-paraeter HRTFs to 7 training and 6 testing data sets. These HRTF data are all fro 0" aziuthal angle. The training data are HRTFs fro +80" to -40" in steps of -20" in elevation. The testing data are HRTFs fro +70 to -30" in steps of -20" in elevation. The network's training is stopped as the training error drops below a threshold. The following table suarizes the network perforance based on various configurations: single hidden layer, two hidden layers, various of hidden units. etwork Configuratiod Training Error I Testing Error Single, H=5 I 7.7", A = 11.6" I 10.9", A = 11.6" Single, H=12 TWO, Hl=H2=5 TWO,Hl=H2=12 Two, H1=5, H2=12 7.3", A = 11.3" 6.6", A = 11.7" 6.5", A = 11.5" 7.1", A = 11.6" 14.5", A = 13.9" 7.9", A = 9.7" 8.7", A = 10.1" 10.0", A = 10.4" dard deviation of the error's absolute value. Fro this table, two hidden layers network is able to localize the 5- paraeter HRTFs within the errors of 6.6" and 7.9" in training and testing data respectively. The standard deviations are about 11". The error is siilar to the huan perception error in elevation[g]. V. CONCLUSION We proposed a localization odel which used onaural spectral cues to localize a sound source in the 1-D plane. A neuroorphic icrophone based on this localization odel is constructed. The neuroorphic icrophone relies on a specially shaped reflecting structure that allows echo-tie processing to localize the sound. Our recording signals show a systeatic change of feature as a sound source (gun shot and speech) is oved along a 1-D plane. The gaa odel is applied to extract spatial features efficiently fro the recordings (actual HRTFs or siplified HRTFs). With the efficient extractor, a siple ultilayer perceptron localizes a sound source within 8". With the successful building of the special icrophone, a full hardware version of a neuroorphic icrophone is possible. REFERENCES D. W. Batteau. The role of the pinna in huan localization. Proc. E. Soc. London, Ser. B, 168: , R. Butler and K. Belendiuk. Spectral cues utilized in the localization of sound in the edian sagittal plane. J. Acoust. Soc. A., 61: , J. Chen, B. Van Veen, and K. Hecox. External ear transfer function odeling: a beaforing approach. J. Acoust. Soc. A., 92~ , Y. Hiranaka and H. Yaasaki. Envelop representations of pinna ipulse responses relating to three-diensional localization of sound sources. J. Acoust. Soc. A., 73: , C. Neti, E. Young, and M. Schneider. Neural network odels of sound localization based on directional filtering by the pinna. J. Acoust. Soc. A., 92: , J.C. Principe, B.de Vries, and P.G. de Oliveira. The gaa filtera new class of adaptive iir filters with restricted feedback. IEEE Trans. on Signal Processing, 4(2): , Feb A.3. Watkins. Psychoacoustical aspects of synthesized vertical locale cues. J. Acoust. Soc. A., 63: , F. Wightan and D. Kistler. UW HRTF data. Head-Related Transfer Functions fro the laboratories of Drs. Wightan and Kistler at the University of Wisconsin. W.A. Yost and G. Gourevitch. Directzonal Hearing. Springer- Verlag,