Implementation of Digital Hearing Aid as a Smartphone Application

Transcription

1 Interspeech September 208, Hyderabad Implementaton of Dgtal Hearng Ad as a Smartphone Applcaton Saketh Sharma, Ntya Twar, Prem C. Pandey Department of Electrcal Engneerng, Indan Insttute of Technology Bombay, Mumba, Inda sakethgsharma@gmal.com, ntya@ee.tb.ac.n, pcpandey@ee.tb.ac.n Abstract Hearng ads for persons wth sensorneural hearng loss am to compensate for degraded speech percepton caused by frequency-dependent elevaton of hearng thresholds, reduced dynamc range, abnormal loudness growth, and ncreased temporal and spectral maskng. A dgtal hearng ad s mplemented as a smartphone applcaton as an alternatve to ASIC-based hearng ads. The mplementaton provdes userconfgurable processng for background nose suppresson and dynamc range compresson. Speech enhancement technque usng spectral subtracton based on geometrc approach and nose spectrum estmaton based on dynamc quantle trackng s mplemented to mprove speech percepton. To compensate for reduced dynamc range and frequency-dependent elevaton of hearng thresholds, a sldng-band dynamc range compresson technque s used. Both processng blocks are mplemented for real-tme processng usng sngle FFT-based analyss-synthess. Implementaton as a smartphone applcaton has been carred out usng Nexus 5X wth Androd 7. Nougat OS. A touch-controlled graphcal user nterface enables the user to fne tune the processng parameters n an nteractve and real-tme mode. The audo latency s 45 ms, makng t sutable for face-to-face communcaton. Index Terms: dynamc range compresson, hearng ad, nose suppresson, smartphone applcaton. Introducton Sensorneural hearng loss s assocated wth loss of sensory har cells n the cochlea or degeneraton of the audtory nerve. It may be nherted genetcally or may be caused by excessve nose exposure, agng, nfecton, or ototoxc drugs. It s characterzed by frequency-dependent elevaton of hearng thresholds, abnormal growth of loudness known as loudness recrutment, ncreased temporal and spectral maskng, and wdenng of audtory flters leadng to degraded speech percepton []-[6]. Several sgnal processng technques have been reported for mprovng the speech percepton by patents sufferng from sensorneural hearng loss. Frequency selectve amplfcaton and dynamc range compresson are the prmary processng technques used n hearng ads [6]-[8]. Sngle band dynamc range compresson leads to reduced hgh-frequency audblty and multband dynamc range compresson may lead to perceptble dstorton due to a transton of speech formants across the band boundares. These problems can be addressed by usng sldng-band dynamc range compresson [9]-[0]. The compresson parameters can be tuned to ft the frequency dependent thresholds and loudness recrutment of the patent. Persons wth sensorneural loss experence dffculty n understandng speech n a nosy envronment. Processng for nose suppresson n a hearng ad can mprove speech audblty and qualty. Spectral subtracton [], a snglechannel speech enhancement technque usng an estmate of the nose spectrum, s sutable for such applcatons as t has low algorthmc delay and computatonal complexty. Dynamc quantle trackng based nose estmaton [2]-[3] s reported to track statonary and non-statonary nose effcently and can be used for real-tme nose suppresson. Hearng ads are generally desgned usng ASICs due to power and sze constrants, leadng to prohbtve costs n development and testng of new processng technques. Use of smartphone-based applcaton (app) as a hearng ad can provde user-confgurable settngs and a greater flexblty to hearng ad users and developers. A smartphone app wth sldng-band dynamc range compresson has been developed earler [4]. Incorporaton of processng for suppresson of background nose s needed to extend the usefulness of the app under dfferent lstenng condtons. Ths paper presents an mplementaton of sgnal processng for nose suppresson based on computatonally effcent dynamc quantle trackng along wth the earler reported sldng-band dynamc range compresson as a smartphone app. The sgnal processng technques and mplementaton detals for dynamc range compresson and nose suppresson are presented n the second secton. Implementaton of the hearng ad app s presented n the thrd secton, followed by test results n the fourth secton and conclusons n the last secton. 2. Sgnal processng Sgnal processng for dynamc quantle trackng based nose suppresson [2] and sldng-band dynamc range compresson [9] use a DFT-based analyss-synthess. The two processng technques are descrbed brefly n the followng subsectons. 2.. Dynamc quantle trackng based nose suppresson Dynamc quantle trackng based nose estmaton [2] s used along wth spectral subtracton for nose suppresson. For each frequency bn of the spectrum, the most frequently occurrng value, obtaned as the peak of the hstogram, s reported to be representatve of the nose value [5]. The nose estmaton technque dynamcally estmates hstogram usng the dynamc quantle trackng wth low memory and computaton requrements [6]. The peak of the hstogram s used as the adaptve quantle for estmatng the nose at each spectral sample. The hstogram s estmated by dynamcally trackng multple quantle values for a set of probabltes chosen to be evenly spaced. The desred quantle correspondng to the peak of the hstogram s obtaned by fndng quantle for whch the dfference between neghborng quantle values s mnmum. The estmate of the nose spectrum at the nth frame and kth spectral sample s obtaned as D( n, k) arg mn [ q ( n, k) q ( n, k)]; 2,3,..., J () q ( n, k ) /Interspeech

2 where J s the number of quantles tracked. The estmate of quantle, q ( n, k ), s obtaned by applyng an ncrement or a decrement on ts prevous estmate as q ( n, k) q ( n, k) d( n, k) (2) The change d (n, k) s gven as Δ ( k ), X ( n, k) q (, ) n k d( n, k) (3) ( k ), otherwse where Δ ( k) and Δ ( k) are selected to be approprate fractons of the range R(n, k) as ( k) R( n, k) p ( k) (4) ( k) R( n, k)( p( k)) (5) The range s estmated usng dynamc peak and valley detectors for updatng the peak P(n, k) and the valley V(n, k) usng the followng equatons: pp( n, k) ( p) X ( n, k), X ( n, k) P( n, k) P( n, k) (6) pp( n, k) ( p) V ( n, k), otherwse vv ( n, k) ( v) X ( n, k), X ( n, k) V( n, k) V ( n, k) (7) vv ( n, k) ( v) P( n, k), otherwse R(n, k) = P(n, k) V(n, k) (8) The dynamc quantle trackng to estmate quantle q ( n, k) as gven by (2), (3), and (8) can be wrtten as the followng: q ( n, k) p ( k) R( n, k), X ( n, k) q( n, k) q ( n, k) (9) q( n, k) ( p( k)) R( n, k), otherwse Spectral subtracton based on geometrc approach [7]-[8] s used for suppresson of background nose, as t results n smaller resdual nose Sldng-band dynamc range compresson The processng for sldng-band compresson [9]-[0] comprses the steps of short-tme spectral analyss, spectral modfcaton, and sgnal re-synthess. To compensate for ncreased hearng thresholds and reduced dynamc range, a frequency-dependent gan functon s calculated n accordance wth the desred levels for soft, comfortable, and loud sounds (referred to as SL, CL, LL, respectvely). For each spectral sample k, the spectral modfcaton s carred out usng a pecewse lnear relatonshp between the nput power and the output power on db scale. The relatonshp s specfed by the values of POdBSL( k ), POdBCL( k ), and POdBLL( k ) whch are the output sgnal levels correspondng to soft, comfortable, and loud sounds, respectvely, for the hearng ad user and by the values of PIdBSL( k ) and PIdBLL( k ) whch are the nput sgnal levels correspondng to soft and loud sounds, respectvely, for a normal-hearng lstener. The relatonshp s defned n three regons wth the compresson rato as CR =, CR >, and CR = n the frst, second, and thrd regon respectvely. Wth G ( k) P ( k) P ( k), the LdB OdBSL IdBSL target gan for the spectral sample k n the th frame n the three regons s gven as GLdB( k), PIdB (, k) POdBCL ( k) GLdB( k) GLdB( k) P IdB (, k) POdBCL ( k) CR( k), (0) GTdB(, k) CR( k) POdBCL ( k) GLdB( k) PIdB (, k) PIdBLL( k) POdBLL( k) PIdB (, k), PIdB (, k) PIdBLL( k) For the spectral sample k n the th frame, the nput level P (, ) IdB k s calculated as the sum of squared magntudes of the spectral samples n the band centered at k and wth the bandwdth correspondng to the audtory crtcal bandwdth 2 f k 0.69 BW( k) ( ) () where f(k) s the frequency, n khz, correspondng to the kth spectral sample. For spectral modfcaton, the target gan s converted to a lnear scale. The gan appled to the kth spectral sample n the th frame s obtaned usng the desred attack and release rates by updatng the gan from the prevous value towards the target value, as gven n (0). It s gven as max G(, k) / a, GT (, k), GT (, k) G(, k) (2) G(, k) mn G(, k) r, GT (, k), GT (, k) G(, k) The number of steps durng the attack and release phases are controlled usng gan ratos ( G G ) sa and a max mn s ( ) r r Gmax Gmn, respectvely. Here G max and G mn are the maxmum and mnmum possble values of the target gan. The number of steps sa durng attack and the number of steps sr durng release are selected to set the attack tme as Ta Sa S fs and the release tme as Tr Sr S fs, where f s = samplng frequency and S = number of samples for frameshft. A fast attack avods the output level from exceedng the uncomfortable level durng transents, and a slow release avods amplfcaton of breathng. 3. App mplementaton The smartphone app for real-tme processng has been developed and tested usng Nexus 5X wth Androd 7. Nougat OS due to ts relatvely small audo I/O delay and hgh processng capablty. Lke the earler app for dynamc range compresson [4], t has a touch-controlled graphcal nterface, enablng the user to adjust the processng parameters n an nteractve and real-tme mode. In addton to the faclty for settng the processng parameters for nose suppresson and dynamc range compresson, there s a provson for the parameters for addtonal processng blocks of the future versons. Fgure shows a block dagram of the mplementaton of the hearng ad app. The setup comprses a handset wth ts headset. The headset conssts of a mcrophone and a par of earphones wth assocated wres and swtchng. The handset conssts of the codec, the processor, and the touch screen for user nterface. The nput sgnal acqured from the mcrophone of the headset s amplfed and s converted to dgtal samples by ADC of the codec. These samples are buffered, processed, and buffered by the processor. The resultng samples are output through DAC of the codec and amplfed. The resultng sgnal s output through the earphones of the headset. The nput samples acqured n an S-word buffer and the prevous samples stored n a 3S-word buffer form the L-sample nput wndow for FFT-based analyss-synthess. The processng for nose suppresson and dynamc range compresson s carred out usng N-pont FFT of the nput wndow. The N-pont IFFT of the modfed complex spectrum s calculated and the output sgnal s re-syntheszed usng overlap-add. The analysssynthess uses 20-ms frames wth 75% frame overlap and 024-pont FFT. The processng s carred out usng samplng frequency = 24 khz, L = 480, S = 20, and N = 024. The 76

3 program was wrtten usng a combnaton of C++ and Java, wth Androd Studo as the development envronment. The screenshot of the home screen of the app s shown n Fgure 2. The play/stop button s for control of the output. All processng modules have ndvdual on/off and settngs buttons. The on/off button can be used for togglng the processng and the settngs button can be used for settng the processng parameters graphcally. Fgure 3a shows screenshot of the settngs screen for nose suppresson and dynamc range compresson modules. Settngs for nose suppresson module provdes user nterface (UI) wth touch control of ponts, called thumbs, for selectng the values of oversubtracton factor α as functon of frequency. The values of α can be set as 5 for up to 0 frequences and the values for all the ntermedate frequences are obtaned by smooth curve fttng. The screenshot of the settngs screen for dynamc range compresson showng graphcal controls for the SL, CL, and LL values, s shown n Fgure 3b. The UI conssts of three touch-controlled curves to set the values of SL, CL, and LL across frequences. Control ponts called as thumbs are provded to adjust the curves. Each curve conssts of 0 thumbs. Provson s provded to store and retreve up to 4 parameter settngs. The UI also conssts of undo and redo button to access recent thumb movements. The mplementaton enables the user to adjust the processng parameters n an nteractve and real-tme mode, to save them as one of the profles, or to select the most approprate profle from the saved ones. Fgure : Implementaton of hearng ad app wth nose suppresson and dynamc range compensaton. Fgure 2: Screenshot of the home screen of the app. 4. Test results The processng modules were tested on the handset model Nexus 5X wth Androd 7. OS. Qualtatve evaluaton was carred out usng the headset of the handset for speech nput through ts mcrophone and audo output through ts earphone. For objectve evaluaton, a PC sound card was used for applyng the audo nput and acqurng the processed output usng a 4-pn TRRS connector to the headset port of the handset. The tme taken for computaton per frame was measured usng Androd devce montor, a proflng tool for Androd OS. The average CPU tme per frame for audo loopback (nclusve of FFT and IFFT operatons) wthout any processng modules was measured as 0.45 ms. Total audo latency comprses the processng delay (sum of algorthmc delay and computatonal delay) and the I/O latency. The algorthmc delay s 25 ms (frame length of 20 ms and frameshft of 5 ms). The total audo latency was measured by applyng a khz tone burst of 200 ms from a functon generator as the nput and observng the delay from onset of the nput tone burst to the correspondng onset n the output, usng a dgtal storage osclloscope. It was found to be 20 ms for audo loopback and 45 ms for the app. The test results for two processng modules are gven n followng subsectons. 4.. Results for nose suppresson Informal lstenng and objectve evaluaton usng perceptual evaluaton of speech qualty (PESQ) measure [9] was used for evaluaton of the nose suppresson module. The processng was also tested usng the 30 sentences from Fgure 3: Screenshot of the settngs screen for nose suppresson and dynamc range compresson. NOIZEUS database [20] added wth noses from AURORA database [2]. Arport, babble, car, street, and tran staton noses from AURORA database and whte nose are added at SNR of 5, 2, 9, 6, 3, 0, and 3 db to form nosy speech. The nosy speech was output from the PC soundcard and nput to the handset usng TRRS connector. The processed output was captured usng Focusrte Scarlett 22 soundcard to avod excessve nose observed whle capturng audo usng PC soundcard. The degradaton n the PESQ score because of loopback through the devce s less than 0.0. The processng parameters for nose suppresson were set as λ = /256, τ p = τ v = 0. and σ p = σ v = (0.9) /024. The hstogram was estmated by trackng eght quantles correspondng to p = 0.25, 0.30, 0.35,..., Table shows the PESQ mprovement for dfferent noses for 0, 3, and 6 db SNR. It can be seen that the mprovement n PESQ scores s n the range The average tme taken for computaton per frame s measured as the dfference of the average CPU tme taken per block wth and wthout nose suppresson module. The average CPU tme per block for nose suppresson wth spectral subtracton based on geometrc approach was measured as. ms. - LL - CL - SL 77

4 Table : PESQ scores for unprocessed speech and mprovement n scores by nose suppresson for dfferent types of noses and SNRs (test materal: 30 sentences from NOIZEUS). Arport nose Babble nose 6 db db db db db db Car nose Street nose 6 db db db db db db Tran staton nose Whte nose 6 db db db db db db c) Fgure 4: Example of processng for dynamc range compresson: ( nput sgnal of ampltude modulated tone of khz, ( GUI parameters set for constant gan of 2 db and compresson rato of 2, (c) processed output Results for dynamc range compresson An example for dynamc range compresson wth ampltude modulated nput s shown n Fgure 4. Input s an ampltudemodulated tone of khz and processng parameters are set as shown n Fgure 4 ( wth compresson rato of 2. The processng gves hgher gans at lower values of the nput level. Spkes n the ampltude envelope of the output sgnal n response to step changes n the ampltude envelope of the nput sgnal, as seen n the fgure, are typcal of the dynamc range compresson wth a fnte frame shft and can be elmnated by usng one-sample frameshft but wth a sgnfcantly ncreased computaton load. The app was further tested for speech modulated wth dfferent types of ampltude envelopes. An example of the processng s shown n Fgure 5, for an ampltude modulated concatenaton of speech sgnals. The nput conssts of three solated vowels, a Hnd sentence, and an Englsh sentence, (- /a/-//-/u/- aaye aap ka naam kya ha? where were you a Fgure 5: Example of processng for dynamc range compresson: ( nput; ampltude modulated VHSES speech, and ( processed speech wth parameters as shown n 3(. year ago? ) referred to as VHSES materal. An nformal lstenng test was carred out wth dfferent speech materals, musc, and envronmental sounds wth large varaton n the sound level as nputs. The outputs exhbted the desred amplfcaton and compresson wthout ntroducng perceptble dstortons. The average CPU tme taken per block wth compresson module s measured to be.3 ms. 5. Concluson To enable the use of smartphone as a hearng ad, ntegraton of the sgnal processng for dynamc quantle trackng based nose suppresson and sldng-band dynamc range compresson has been mplemented usng LG Nexus 5X runnng Androd 7.. The processng parameters can be set by the user n an nteractve and real-tme mode usng a graphcal touch nterface. The audo latency of the app s 45 ms, whch s much less than the detectablty threshold of 25 ms for audo-vsual delay [22] and hence may be consdered as acceptable for a hearng ad durng face-to-face conversaton. Implementaton of the app on other smartphones and ts use by a large number of hearng-mpared lsteners s needed for real-lfe evaluaton and further enhancement. Acknowledgments The research s supported by 'Natonal Program on Percepton Engneerng Phase-II' sponsored by the Department of Electroncs & Informaton Technology, Government of Inda. 78

5 References [] H. Levtt, J. M. Pckett, and R. A. Houde (eds.), Senosry Ads for the Hearng Impared. New York: IEEE Press, 980. [2] J. M. Pckett, The Acoustcs of Speech Communcaton: Fundamentals, Speech Percepton Theory, and Technology. Boston, Mass.: Allyn Bacon, 999, pp [3] B. C. J. Moore, An Introducton to the Psychology of Hearng, London, UK: Academc, 997, pp [4] S. A. Gelfand, Hearng: An Introducton to Psychologcal and Physologcal Acoustcs. New York: Marcel Dekker, 998, pp [5] D. O' Shaughnaessy, Speech Communcatons: Human and Machne. Hyderabad, Inda: Unv. Press, 200, pp [6] H. Dllon, Hearng Ads. New York: Theme Medcal, 200. [7] R. E. Sandln, Textbook of Hearng Ad Amplfcaton. San Dego, Cal.: Sngular, 2000, pp [8] D.Byrne, W. Tonnson, Selectng the gan of hearng ads for persons wth sensorneural hearng mparments, Scandnavan Audology, vol. 5, pp. 5 59, 976. [9] N. Twar and P. C. Pandey, A sldng-band dynamc range compresson for use n hearng ads, n 20th Natonal Conference on Communcaton (NCC 204), Kanpur, Inda, do: 0.09/NCC [0] P. C. Pandey and N. Twar, "Dynamc range compresson wth low dstorton for use n hearng ads and audo systems," US Patent No , 207. [] S. F. Boll, Suppresson of acoustc nose n speech usng spectral subtracton, IEEE Transactons on Acoustcs, Speech and Sgnal Processng, vol. 27, no. 2, pp. 3-20, 979. [2] N. Twar and P. C. Pandey, "Speech enhancement usng nose estmaton based on dynamc quantle trackng for hearng mpared lsteners," n 2st Natonal Conference on Communcaton (NCC 205), Mumba, Inda, 205, paper no [3] P. C. Pandey and N. Twar, Method and system for suppressng nose n speech sgnals n hearng ads and speech communcaton devces, US Patent applcaton publcaton no. US 207/ A, 207. [4] S. Sharma, N. Twar, and P. C. Pandey, Implementaton of a dgtal hearng ad wth user-settable frequency response and sldng-band dynamc range compresson as a smartphone app, n 8th Internatonal Conference on Intellgent Human Computer Interacton (IHCI 206), Plan, Inda, Dec. 2-3, 206, paper no. 8. [5] H. Hrsch, C. Ehrlcher, Nose estmaton technques for robust speech recognton, n Internatonal Conference on Acoustcs, Speech, and Sgnal Processng (ICASSP 995), Detrot, Mchgan, 995, pp [6] N. Twar and P. C. Pandey, A technque wth low memory and computatonal requrements for dynamc trackng of quantles, Journal of Sgnal Processng Systems, Epub. 208 Jan 08, DOI: 0.007/s [7] Y. Lu and P. C. Lozou, "A geometrc approach to spectral subtracton," Speech Communcaton, vol. 50, pp , [8] Y. Lu and P. C. Lozou, Speech enhancement code. [onlne]. Avalable: [9] ITU, Perceptual evaluaton of speech qualty (PESQ): an objectve method for end-to-end speech qualty assessment of narrow-band telephone networks and speech codecs, ITU-T Rec., P.862, 200. [20] A. Varga and H. J. M. Steeneken, Assessment for automatc speech recognton: II. NOISEX-92: A database and an experment to study the effect of addtve nose on speech recognton systems, Speech Communcaton, vol. 3, no. 3, pp , 993. [2] H. Hrsch and D. Pearce, "The AURORA expermental framework for the performance evaluaton of speech recognton systems under nosy condtons," n ISCA Tutoral and Research Workshop on Automatc Speech Recognton: Challenges for the New Mllenum 2000, (ASR 2000), Pars, France, pp [22] Internatonal Telecommuncaton Unon: Relatve tmng of sound and vson for broadcastng, ITU Rec. ITU-R BT