Acoustic echo cancellers for mobile devices

Transcription

1 Dr. Nazarov A.G, IntegrIT Acoustic echo cancellers for mobile devices Broad market development of mobile devices and increase their computing power gave new opportunities. Now handset mobile gadgets incorporate new communication functions like audio/videoconferencing which were previously only available in the professional or premium grade devices. Popular services like FaceTime, Skype, Oovoo, Qik come to the mobile market as well. Component cost reduction and expansion of ARM based processors is moving the market to use them in a variety of those applications (dispatcher control, radios, etc.) which traditionally used specialized cores or DSP. On the other hand, users are becoming more particular about the quality of communications. Traditional systems with a bandwidth of standard telephone channel ( Hz) can no longer satisfy the people. It s time to wideband IP telephony. It gives people superior speech quality and increased bandwidth ( Hz) at a reasonable cost. Acoustic echo canceller is a vital key component of communication platforms and IP telephony. In the modern terminology it is more than just a canceller, but rather speech preprocessor that combines all functions for echo and noise removal, adjusting speech level and overall equalization of a signal from the microphone. One of the major problems encountering in the use of cancellers is the lack of standard specifications and test methods to compare products from different manufacturers. International community is constantly searching for methods of standardization. A set of international recommendations (P.501, P.862, and so on) already describe the quality assessment procedures, and some metrics including speech sources, synthesized speech, acoustic rooms and many others. Some time ago there was recommendation ITU T G.167 devoted to acoustic cancellers. However, it was more focused on the almost ideal installation conditions and too far from the nowadays mobile applications. This article discusses the main requirements for acoustic echo cancellers and draws attention to the problems faced by software developers in their use. The range of problems associated with echo cancellation task is rather broad, so we will discuss only the most important issues delay optimization the features and audio properties of real mobile handsets and effect to echo cancellation 1

2 typical signal distortions joint echo/noise suppression resource constraints for ARM class CPUs Delay Delay is an important characteristic of the audio subsystem. It is known that the subjective speech quality (MOS quality) and the perception for two way communication become much worse with increasing delays. Recommendation G.114 provides some experimental results illustrating this effect. These pictures show serious deterioration of quality due to the growing delay. However total delay also affects the susceptibility of the human ear to the echo. Recommendation G.131 gives some guidelines for required echo suppression level depending on the delay. It is evident that the increased delay in the audio path significantly tightens the requirements for residual echo level. That is why we should discuss the how to minimize the delay. 2

3 At first glance, the delay is not a problem of echo canceller. This is true, but only a part of truth. Let's look how canceller is integrated into the audio path and non obvious sources of delay. In contrast to the speech codecs the echo canceller has an additional input (called Rin) which is passed to the speaker as well. It uses this input for echo analysis and synthesis. Echo canceller microphone Noise, interference audio driver Sin Sout Adaptive filter internet Remote talker echo Rin audio driver Local talker speaker Figure 1: Simplified echo cancellation diagram 3

4 It should be noted that the classical echo cancellers are very sensitive to an asynchrony between signals Rin and Sin. The presence of even a small time drift (often unaudible) makes canceller completely unworkable. This fact creates problems in integration with non realtime operating systems like Linux, Android. They are not optimized for making synchronous audio input and output. Needed synchronization may be achieved either by writing the special kernel mode driver or extra bufferring which smoothes the asynchrony. Thus, the total delay compared to the asynchronous drivers can grow up significantly (by milliseconds). Meanwhile, such the delay may become unacceptable for a high quality communication. For example, Recommendation ITU T G.114 specifies a one way delay of 150 milliseconds as the maximum acceptable for the human perception. One of possible solutions is small modification of the hardware. We may add one extra audiocodec (or use stereocodec instead of mono for microphone input). This codec digitizes the output coming directly to the speaker. The audio samples coming from it may be used as Rin signal. Since they will be read together with the microphone signal, the output might be asynchronous with input. Thus, we may use asynchronous driver with minimal delay. Echo canceller microphone Noise, interference ADC Sin Adaptive filter ADC internet Rin echo DAC Local talker Sout audio driver audio driver speaker Figure 2: An alternative echo cancelling scheme However, for most cases such opportunity is not exist because the software runs on existing hardware platforms the mobile phone, tablet PC and so on. In that case we need cancellation algorithms which are robust to delay variations. In any case, the delay in the audio driver should be measured and does not change a lot, and echo canceller should be able to use the knowledge of the constant delay term to minimize consumption of resources reducing the equivalent echopath. In addition, you should pay attention the echo canceller delays signal (the so called algorithmic delay). Typical values for this delay lie in the range of ms. Larger values are unacceptable. 4

5 Distortions and the echo In the mobile devices the echo canceller is exposed to substantially greater level of echo, noise and distortion. Local talker active Near end signal (microphone input) Far end signal (speaker output) Remote talker active Figure 3: Signal + echo + noise on microphone input and the speaker output Some portion of echo occurs due to the mechanical coupling between speaker and microphone. Although manufacturers taking steps to damping speakers and microphones, it is not possible to make high level of acoustic isolation in small enclosures due to its size, propagation via plastic and metal, etc. Besides its fairly high level, it is characterized by a high level of distortion, and in some cases, the presence of distinct resonant peaks. The second source of echo is reflection of surrounding elements. Picture looks like a sum of many scattered echoes over a big echopath interval and some concentrated reflections. It is not static picture it is fluctuating rapidly in phase and amplitude due to the movement. Total echopath becomes large even for small places. Normally, 256 msec is considered as good enough for most areas, but for large rooms it may reach 512 msec or even more. Generally, mobile devices shows high level of intermodulation and nonlinear distortions. They are caused by properties of small microphones and speakers, using 5

6 non linear amplifiers with better P.A.E., the resonances in enclosure, and other factors. The nonlinear distortions of order 10 db and intermodulation about 15 db are pretty typical. Looks strange but in some cases, additional distortions are introduced by audio drivers supplied by hardware manufacturers. They trying to compensate acoustic properties by software means and adding extra processing into the drivers, i.e. automatic gain control, noise suppression, equalizing, etc. However, this can drastically affect the echo canceller, since it should compensate the changes in the AGC as changes in the echo. For canceller, any non linear processing in the audio drivers looks like an additional nonlinear distortion. So, make the manufacturer s AGC or noise reduction can significantly degrade the echo suppression level or cause undesirable speech distortions. Thus, summarizing all above, the echo canceller should: have means to compensate nonlinear and intermodulation distortions support echopath up to 512 msec have tools to work on different sampling rates have fast adaptation rate to minimize the impact of the built in AGC have means to combat with unwanted resonances, and also have the possibility of correcting frequency response of the microphone Joint echo and noise cancellation In the mobile world, the environment is characterized by the high ambient noise level and its rapid changes. The sources of noises are diverse in nature and can be as broadband and frequency selective. At a first glance, the noise level in offices is not so bad. However, there are additional noise factors, such as ac harmonics produced by fluorescent or energy saving lamps. They are visible even at frequencies above 1 khz. Speech to noise level of order db is very typical. In such circumstances, echo canceller must provide for the joint echo and noise reduction, otherwise, users may experience the following adverse effects: slowing the rate of convergence of the echo suppressor deterioration of speech quality, especially through swallowing short phonemes, sizzling sounds and distortion of weak speech audible echo in a double talk conditions or the appearance of residual echo in the pauses slowing the adaptation to changes of ambient noise in the presence of a significant echo or double talk the appearance of the so called musical noise in the pauses or metallic speech 6

7 A significant change in the tembre We should also note that the noise suppressor usually has its own algorithmic delay typically not less than 30 msec. And so, the combining of the noise and the echo cancellation reduces total algorithmic delay. Pictures below show two typical environments with high noise level. Both in car and on the street, the noise situation is unstable and requires quick reactions as to frequency selective noise, as well to changes of background noise. Near end signal (microphone input) Near end signal (microphone input) Output of acoustic echo canceller Output of acoustic echo canceller Figure 4: Spectrograms of typical signals at the inputs of echo cancellers (left street, right in the car) and result of processing The figures below show the typical pattern of signals at the inputs of echo canceller in the office room. Echo strongly "smeared" in time domain, its level is comparable to speech of the local speaker. The most noise is frequency selective with many spectrum lines. 7

8 Far end signal (speaker output) Near end signal (microphone input) Output of acoustic echo canceller Figure 5: Spectrograms of typical signals at the inputs of echo cancellers (office), and the result of processing echo canceller CrystalSpeech Resource constraints Understanding the resource constraints is really important for real applications. The limited scope of the article can not present complete survey of all the methods used for echo cancellation. However, below we give some basic metrics for resource consumption of echo cancellers. 8

9 In a certain sense, all the classical methods echo are variations of the LMS method. It allows, taking the estimate of the echopath impulse response, synthesize the echo and subtract it from the input signal, thus leaving only the local speech. Noise, interference microphone Sin Sout Synthesized echo Local talker Echopath Adaptive filter Rin speaker Figure 6: LMS algorithm idea From a computational point of view, resources are determined by two factors: filtering procedure finding the optimal impulse response and tracking algorithm Filtering can not be avoided and it is most consuming part. Tracking and update algorithm is much more mathematically complicated but may be made less consuming because it might be run block wise and adapts coefficients not each sample. Usually, period of adaptation should not be longer than 5 to 10 milliseconds due to the variation of a speech and slowing the convergence rate. Roughly speaking, we may assume the tracking takes 25% resources in comparison with filtering. Filtering can be done both in time and in frequency domains. In a literature, the frequency domain methods are considered as silver bullet solving all the resource consumption problems. Below we show that it is not absolutely true. First, look to the time domain method. For wideband echo cancellation we have 16 khz sample rate and total echopath 512 msec. This gives equivalent filter with 8192 taps and the total number of multiple accumulate operations (MAC) about 130 millions. ARM architecture does not provide data loads in parallel with computations so we have to account the overhead for data/coefficients load. Taking this, ARM9e processors are capable to make around 0.75 MAC per cycle and ARM11 makes twice more. Additionally, we have to take into account 30% performance loss due to the caching and 25% for tracking algorithm. This gives: 9

10 Platform MIPS ARM9E 280 MIPS ARM MIPS For frequency domain method the performance is determined by Fast Fourier Transform (FFT). ARM11 architecture promotes variety of SIMD operations that can effectively utilized in the FFT (SHADD16, etc.). It is indeed true; however echo cancellation requires large FFT length together with low level of algorithmic noise. All ARM SIMD operations are 16 bits and the large 16 bit FFTs should have extra intermediate scaling stages to prevent the loss of accuracy. This causes additional overhead and the total gain of ARM11 over ARM9e is not as big as expected originally. The measured FFT cycle counts (real FFT 8192) are presented below. These numbers are given for real processor, so caching overhead is already taken into account: Platform Cycles ARM9E ARM For frequency domain method we should make at least 3 FFTs (one to calculate signal spectrum, another one for conversion of impulse response to the frequency response and, the last, inverse FFT for signal restoration). Filtering is done blockwise and processing block should not exceed 10 milliseconds as mentioned above. Thus, taking into account the 25% overhead for tracking algorithm the frequency domain method yields: Platform MIPS ARM9E 188 MIPS ARM MIPS As we can see, the benefits of frequency domain method are not as big as anticipated. Potentially, the gain would be more but this requires increasing of the block size which may affect to the convergence, residual echo level, double talk detection and so on. Real echo canceller includes extra modules such as noise suppressor, nonlinear distortion processor, gain control, voice activity detection, etc. They normally add MIPS for ARM11 and MIPS for ARM9e. 10

11 Thus, we see that classical methods of wideband echo cancellation demand high CPU load (about 150 MIPS for the ARM11 and 200 MIPS for ARM9e). Therefore, algorithms should use sophisticated techniques reducing computational complexity while keeping the speech quality. There are several obvious reasonable approaches acceptable for many applications use narrowband mode instead of wideband or reducing maximum echopath to 256 msec. But in any case, the computational complexity is high enough and considering as an important factor for using the echo canceller. Memory requirements are also important. Usually, amount of data memory requested by the canceller is around kbytes. Additionally, canceller may require extra temporary data storage of kilobytes. This scratch memory may be reused between applications but it is updated each time causing strong caching overhead. This may result deterioration of performance on the processors with small cache (16 or 32 kbytes). Also, performance loss may be observed if other memory consuming applications are executed concurrently in the multithreading environment. For example, video applications are hungry for memory and cause large cache traffic as well. Therefore, software designers should pay attention to the accurate sharing of scratch memory between applications when using acoustic cancellation. Concluding all above we see that wideband acoustic echo cancellation is computationally intensive task and ARM11 processors fits better than ARM9e. The use of DSP processors like C64xx/OMAP may provide significant performance gain up to 3 5 times compared to ARM11. IntegrIT CrystalSpeech cross-platform VoIP Engine All the problems described above already resolved in a software product IntegrIT's CrystalSpeech. Its technical characteristics are summarized in the table below: Mode Specification Modes narrowband (sample rate 8 khz), wideband (sample rate 16 khz), optionally supplied with resamplers to support most popular sample rates Robustness to delay variation or time drift Nonlinear/intermodulation compensation distortion Additional AC harmonic suppression Comfort noise generator (CNG) Maximum echopath up to 512 msec 11

12 Compensation of audio drivers delay up to 512 msec Total algorithmic delay 60 msec Equalizer with resonance blocking Voice activated (AGC) automatic gain control Noise suppression (NS) joint echo noise cancellation Musical noise removal Typical resource consumption 44 MIPS (narrowband mode), 75 MIPS (wideband mode), ARM11 core Maximum data memory requirements Supported CPUs Operating systems worst case: 120 kbytes data RAM + 50 kbytes scratch data (wideband mode) ARM9e, ARM11, Cortex, Marvell Kirkwood/Armada, Texas Instruments C64xx, DaVinci, OMAP, Tensilica HiFi2/ConnXD2, x86 Linux, Maemo/MeeGo, Windows, Windows CE/Mobile, Android, DSP BIOS IntegrIT CrystalSpeech is a new step in the real time speech processing software enabling hands free full duplex communications. Technology is ideally suited for conferencing terminals, smartphones, communicators, videophones, dispatcher boards, speech synthesis and recognition systems, VoIP solutions, etc. Speech enhancement technology includes intellectual echo and noise suppressors providing natural speech quality that selectively recognizes active speaker even in noisy places. This allows conversations in a wide range of conditions with extremely high echo level and environmental noise. Echo cancellation technology adapted for use in mobile devices such as notebooks, communicators and gadgets where audio quality is limited due to the mechanical resonances, small speakers and high level of microphone speaker acoustic feedback. Unique low consumption of resources, even on mobile processors and cross platform implementation enables the use of technology in a variety of applications from multimedia gadgets to high performance conferencing servers. IntegrIT software solutions are used by telecommunication equipment manufacturers. Broad selection of VoIP components optimized for most popular modern processors allows to cut the time to market, reduces development cost, guaranties the quality and compatibility. See detailed datasheets of CrystalSpeech and other product information on the web site: /products. 12