A Novel Speech Controller for Radio Amateurs with a Vision Impairment

Transcription

1 IEEE TRANSACTIONS ON REHABILITATION ENGINEERING, VOL. 8, NO. 1, MARCH A Novel Speech Controller for Radio Amateurs with a Vision Impairment Chih-Lung Lin, Bo-Ren Bai, Li-Chun Du, Cheng-Tao Hu, Shuenn-Tsong Young, and Te-Son Kuo Abstract This paper describes a portable speech controller system for persons with a vision impairment to adjust the channel frequency of a radio set via speech commands. The speech commands are recognized on a general-purpose digital signal processor using a hidden Markov model (HMM), and are used to remotely control radio channel changes. Index Terms Hidden Markov model, radio-digital signal processor, speech controller, visually impaired. I. INTRODUCTION PEOPLE with a vision impairment have difficulty using machines. Voice communication offers a potentially user friendly interface. Machines that talk are now quite common. For example they are used to identify landmarks [1] where a small prerecorded vocabulary is required. Or, alternatively as outputs for computers where a synthesized voice is required [2]. Voice communication in the other direction, from the person to the machine, however is not yet extensively used. This paper deals with this latter topic; the recognition of human speech by machines. Amateur radio can provide communication support for emergencies, fairs, and other activities in a local area. Persons who use amateur radio can also reach anywhere around the world. They can thus acquire much interesting and useful information. Most commercial radios have a digital display for the channel frequency. Persons with a vision impairment usually cannot read the digital display, and therefore cannot conveniently adjust the channel frequency. Lacquet et al. [3] proposed a digital-display-to-natural-voice converter to help radio amateurs with visual impairments. This converter could convert the channel frequency of a radio set into speech for tuning the radio. This converter, however, was operated passively and just announced the selected channel frequency. A more convenient method would be a human voice recognition device which selects the desired channel frequency. Manuscript received November 16, 1998; revised March 30, 1999 and Septemnber 15, This work was supported by the National Science Council of the R.O.C. under Grant NSC E C.-L. Lin, B.-R. Bai, and L.-C. Du are with the Department of Electrical Engineering, National Taiwan University, Taipei 106, Taiwan, R.O.C. C.-T. Hu is with the Department of Biomedical Engineering, National Defense Medical Center, Taipei 106, Taiwan, R.O.C. S.-T. Young is with the Institute of Biomedical Engineering, National Yang-Ming University, Taipei 106, Taiwan, R.O.C. T.-S. Kuo is with the Department of Electrical Engineering, National Taiwan University, Taipei 106, Taiwan ROC and the Graduate Institute of Biomedical Engineering, National Taiwan University, Taipei 106, Taiwan, R.O.C ( kuo@ccms.ntu.edu.tw). Publisher Item Identifier S (00) Speech is one of the most natural methods of communication for humans, and it should be a good human-device interface. However, the speech signal is highly variable and difficult to distinguish [4]. A dynamic time warping (DTW)-based approach has been widely used for speech recognition using speech-chips for consumer products. The ability to increase the recognition rate is limited by the algorithms. The other well known method is the hidden Markov model (HMM) [4] [8]. The HMM retains more statistical information about the speech pattern, and performs much better than the DTW-based approach [4], [6]. Complex HMM algorithms demand and consume large amounts of memory space and computation power. It is difficult to implement the HMM speech recognition algorithm into a single speech-chip. The general-purpose digital signal processing (DSP) chips have considerable computation power, which can be extended with external memory devices. DSP chips are good candidates for managing complex HMM algorithms. There are many successful systems for the recognition of Mandarin Chinese speech using the HMM approach by a personal computer (PC) [8] [10]. Recently, DSP chips have been widely adopted for sound processing in rehabilitation engineering [11] [13]. The DSP devices are programmable and flexible. New processing schemes can easily be developed, evaluated, and modified within the DSP devices. This paper proposes a portable speech controller system based on a DSP chip to control the channel frequency of a radio set to aid persons with a vision impairment. II. METHODS A. Architecture of the Proposed System The hardware block diagram of the portable speech controller system is shown in Fig. 1. The system was composed of a microphone, a speech-recognition device, a voice generator and a remote controller. The human voice was received by a microphone that collected the sound of voices. The input speech was amplified and passed into an anti-aliasing filter with a cutoff frequency of 8 khz. An analog-to-digital converter (ADC) sampled the input speech at a rate of 16 khz, and converted it into a series of 12-bit data. The core of the speech-recognition device was a 32-bit DSP integrated circuit (Texas Instruments TMS320C31-50 MHz), 4-Mbit flash ROM, and 8-Mbit highspeed SRAM. A programmable array logic (PAL) device decoded the output via the DSP data bus and selected peripheral devices to be controlled, such as the flash ROM, high-speed SRAM, voice output chip, and the remote controller /00$ IEEE

2 90 IEEE TRANSACTIONS ON REHABILITATION ENGINEERING, VOL. 8, NO. 1, MARCH 2000 Fig. 1. Hardware block diagram of the speech controller system. A voice output chip (Information Storage Devices ISD 2590) saved analog voice signals as long as 90 s. The analog voice signals were associated with some speech commands. These voice signals can be played back on request through an external 8 loudspeaker by activating the associated addresses. The speech commands were prerecorded by saving the user s voice in the voice output chip. The voice output chip generated the associated speech signals according to the speech commands recognized by the DSP chip. The keypad of the remote controller was used to set the channel frequency and to adjust the voice volume. The digital signals that were associated with the functions of the remote controller were precoded. When the portable speech controller system recognized a speech command, the DSP chip transferred the associated digital signal to the remote controller, which then tuned the channel frequency or adjusted the voice volume. B. Description of Speech Command Recognition A detailed description of the speech recognition processes is presented in this section. First, the input speech signal of a speech utterance was divided into short segments and blocked into 16-ms frames with 8-ms overlap per frame. A Hamming window and fast Fourier transformation (FFT) were then applied to each frame of the input speech. Considering the nonlinear characteristics of the human ear, a mel-scale filter bank was applied to each frame to get the mel-cepstrum coefficients [6], [14] as the input feature. The mel-cepstrum was calculated by (1) (1) Fig. 2. Block diagram of the speech recognition process. where number of filter bank channels; output of the th mel-scale filter bank channel; dimension of mel-cepstrum coefficient. In this paper, we chose and. For each frame obtained from the input speech, the 12-dimension mel-cepstrum, and the 12-dimension regressive mel-cepstrum (which is a linear combination of the mel-cepstrum between adjacent frames [15]) were produced as the feature vector for model training and recognition. The HMM is primarily described by the state transition probabilities and state observation probabilities. The state transition probabilities are used to describe the vocal tract transition between different articulatory configurations, while the state observation probabilities are used to describe the distribution of the acoustic features in different articulatory configurations. When using an HMM, the reference command

3 LIN et al.: A NOVEL SPEECH CONTROLLER FOR RADIO AMATEURS WITH A VISION IMPAIRMENT 91 Fig. 3. Flow diagram of the speech controller system. models are represented by subword models [9], [16]. For example, the spoken Mandarin Chinese command /Da-Sheng/ (increase the volume) was represented by the two subword models /Da/ and /Sheng/. The subword models provided the flexibility to add to or change the command set. In our experiments, each subword model had 7 states that individually had five Gaussian distributions. To recognize an input speech command, the HMM-based keyword spotting technique shown in Fig. 2 was used to detect a speech command embedded in the input speech utterance. While the DSP of the speech controller system performed the keyword spotting, each segment of input speech utterance should be mapped to a pretrained model. Since the input speech utterance was unnecessary only including commands (i.e., the keywords), a filler model was trained for the nonkeyword portion of received voice signal (including background noise and nonkeyword). This filler model was used for dealing with noncommand speech [5], [17]. While the recognition processes were performed, the sequence of the obtained feature vectors were matched with all of the models by a dynamic programming algorithm to find the model with the highest score, i.e., the model that most closely matched the input speech. Since extensive computations were required to calculate the likelihood scores of all frames with respect to all distributions in all models, considerable computation power was necessary. In order to improve reliability, utterance verification techniques were used to adjust the scores of the keyword candidates. All utterances were verified to insure that the keyword candidates existed in the input speech utter-

4 92 IEEE TRANSACTIONS ON REHABILITATION ENGINEERING, VOL. 8, NO. 1, MARCH 2000 ance. Two conditions were considered: 1) the scores of the highest and second candidates were too close and differed to a lesser degree than a predefined value or 2) the highest score was smaller than a predefined value, which meant that the discrimination capability was weak. If any one condition occurred, the recognition processing failed, i.e., the keyword recognition could not be accepted, and had to be attempted again. These methods can provide better results. The cepstrum mean subtraction (CMS) technique was applied to reduce the background noise effect. The cepstrum average was taken over the speech frames, which included the background noise. This average was then subtracted from each cepstral feature vector [18]. This method attempted to remove the noise embedded in the input speech on the feature level, and thus improve the performance to a certain degree. C. System Software The flow diagram of the speech controller system is shown in Fig. 3. The boot loader fetched the program codes from ROM to the SRAM and initialized the hardware of the speech controller system. The system then downloaded the predefined command models that were acquired from the trained speech space of a computer. These models were obtained from a speaker-independent model used at the speech lab, and all of the models were trained from 150 male speakers. The system then analyzed the environmental noise to prevent noise from interfering with the speech command. The zero-crossing rate and short time energy methods were adopted for detection of the endpoint of the received voice signal. The decided speech frame was transformed into a sequence of mel-cepstrum coefficients and was applied to CMS algorithm to reduce the noise interference. Then the recognition processes were then performed by comparing the obtained sequence of mel-cepstrum with pretrained word models. The recognized speech command was enunciated by the voice output chip. The person with a vision impairment could then check whether the speech command was correctly recognized. The Mandarin Chinese spoken words One through Nine, Zero, Point, Mega, Up, Down, Reset, and Hertz were recognized to state a desired channel frequency on the radio set. Within this system, the word Hertz was used for the termination of a command for channel frequency setting. When the word Hertz was recognized, the system treated the previous input speech as a completed command. The Up and Down words were used to adjust the voice volume of the radio set. The Reset command was used to renew the incorrect recognition to avoid mistaken command operation. A recognized speech command was converted into a suitable format for the remote controller to tune the channel frequency or adjust the voice volume of the radio set. III. EXPERIMENTAL RESULTS AND CONCLUSION Five volunteers took part in this experiment. Each subject selected ten desired channel frequencies randomly and then adjusted the volume up and down once on each channel. The portable speech controller system recognized the speech commands in succession and drove the peripheral interface. In the effective test of the noise reduction technique, we adjusted the background noise to create three test environments having about 30, 25, and 20 db signal-to-noise ratios (SNR s). The test environment was in a laboratory. The background noise included air-conditioner noise, multitalker babble, music-weighted noise, etc. The correct rate of speech command recognition before background noise reduction was 72 4% (mean SD) with 30 2 db SNR, 63 4% with 25 db SNR, and 56 % with 20 2 db SNR, respectively. After adopting the CMS technique to reduce the background noise, the correct rate was 90 2% with 30 2 db SNR, 88 2% with 25 2 db SNR, and 86 4% with 20 3 db SNR. However, with the application of the Reset command, all desired commands could be correctly recognized and executed. This paper has presented a portable speech controller system that consists of a speech recognition device, a voice generator, and a remote controller. The HMM-based keyword spotting technique, a set of subword models, and noise reduction techniques were used to recognize the speech commands. Although this system is based on Mandarin Chinese speech recognition, it could be applied to other languages by storing the associated command features into the ROM. Some modifications of the recognition software may also be necessary. This speech controller system is a high recognition rate, portable and low cost system. It should be very useful for radio amateurs with a vision impairment. This system can also be used by quadriplegics. Quadriplegics are usually unable to operate consumer electronics with their hands. The system has the potential to be a good human device interface for quadriplegics. This system can easily be retrofitted to commercial microwave ovens, televisions, air conditioners, and washing machines with remote controllers. ACKNOWLEDGMENT The authors would like to thank L.-S. Lee for providing a Mandarin Chinese speech database. REFERENCES [1] P. Blenkhorn and D. G. Evans, A system for enabling blind people to identify landmarks The sound buoy, IEEE Trans. Rehab. Eng., vol. 5, pp , Sept [2] P. Blenkhorn, Designing products that speak Lessons from talking systems for blind people, Comput. Contr. Eng. J., vol. 5, no. 4, pp , Aug [3] B. M. Lacquet and F. H. Baird, An affordable digital-display-to-nature-voice converter for visually impaired radio amateurs, IEEE Trans. Rehab. Eng., vol. 4, pp , Dec [4] B. R. David and J. G. Wilpon, Whither speech recognition The next 25 years, IEEE Commun. Mag., pp , Nov [5] J. G. Wilpon, L. R. Rabiner, C. H. Lee, and E. R. Goldman, Automatic recognition of keywords in unconstrained speech using hidden Markov models, IEEE Trans. Speech Signal Processing, vol. 38, pp , Nov [6] L. R. Rabiner and B. H. Juang, Fundamentals of Speech Recognition. Englewood Cliffs, NJ: Prentice-Hall, [7] L. R. Rabiner, A tutorial on hidden Markov models and selected applications in speech recognition, Proc. IEEE, vol. 77, pp , Feb [8] L. S. Lee, C. Y. Tseng, H. Y. Gu, F. H. Liu, C. H. Chang, Y. H. Lin, Y. Lee, S. H. Hsieh, and C. H. Chen, Golden Mandarin (I) A real-time Mandarin speech dictation machine for Chinese language with very large vocabulary, IEEE Trans. Speech Audio Processing, vol. 1, pp , Apr

5 LIN et al.: A NOVEL SPEECH CONTROLLER FOR RADIO AMATEURS WITH A VISION IMPAIRMENT 93 [9] H. M. Wang, T. H. Ho, R. C. Yang, J. L. Shen, B. R. Bai, J. C. Hong, W. P. Chen, T. L. Yu, and L. S. Lee, Complete recognition of continuous Mandarin speech for Chinese language with very large vocabulary using limited training data, IEEE Trans. Speech Audio Processing, vol. 5, pp , Mar [10] B. R. Bai, C. L. Chen, L. F. Chein, and L. S. Lee, Intelligent retrieval of dynamic networked information from mobile terminals using spoken natural language queries, IEEE Trans. Consum. Electron., vol. 44, pp , Feb [11] H. McDermott, A. E. Vandali, R. J. M. van Hoesel, C. M. McKay, J. M. Harrison, and L. T. Cohen, A portable programmable digital sound processor for cochlear implant research, IEEE Trans. Rehab. Eng., vol. 1, pp , June [12] Y. Nejime, T. Aritsuka, T. Imamura, T. Ifukube, and J. Matsushima, A portable digital speech-rate converter for hearing impairment, IEEE Trans. Rehab. Eng., vol. 4, pp , June [13] H. McDermott, A programmable sound processor for advanced hearing aid research, IEEE Trans. Rehab. Eng., vol. 6, pp , Mar [14] S. B. Davis and P. Mermelstein, Comparison of parametric representation for monosyllabic word recognition in continuously spoken sentences, IEEE Trans. Speech Audio Processing, vol. 28, pp , Apr [15] S. Furui, Speaker-independent isolated word recognition using dynamic features of speech spectrum, IEEE Trans. Acoust., Speech, Signal Processing, vol. 34, pp , Feb [16] C. H. Lee, B. H. Juang, F. K. Soong, and L. R. Rabiner, Word recognition using whole word and subword models, in Proc. Int. Conf. Acoust., Speech, and Signal Processing, 1989, pp [17] R. C. Rose and D. B. Paul, A hidden Markov model based keyword recognition system, in Proc. Int. Conf. Acoust., Speech, and Signal Processing, 1990, pp [18] B. S. Atal, Effectiveness of linear prediction characteristics of the speech wave for automatic speaker verification, J. Acoust. Soc. Amer., vol. 55, pp , Chih-Lung Lin was born in Tainan, Taiwan, R.O.C., in He received the M.S. and Ph.D. degrees in electrical engineering from National Taiwan University, Taiwan, R.O.C., in 1993 and 1999, respectively. His research interest are medical instrumentation, medical signal processing, and neuron modeling. Bo-Ren Bai received the B.S. and Ph.D degrees in electrical engineering from National Taiwan University, Taiwan, R.O.C., in 1992 and 1998, respectively. His research interests include speech processing and information retrieval. Li-Chun Du was born in Taipei, Taiwan, R.O.C., in He received the B.S. degree and is working toward the M.S. degree in electrical engineering from National Taiwan University, Taiwan, R.O.C. His research interests are speech processing and biomedical hardware design. Cheng-Tao Hu was born in Taiwan, R.O.C., in He received the Ph.D. degree in electrical engineering from National Taiwan University, Taiwan, R.O.C., in His research interests include medical signal processing. Shuenn-Tsong Young was born in Taiwan, R.O.C., in He received the M.S. and Ph.D. degrees from the Department of Electrical Engineering, National Taiwan University, Taiwan, R.O.C., in 1987 and 1989, respectively. Since 1994, he has been a Professor at the Institute of Biomedical Engineering and Director of the Research and Development Center of Biomedical Engineering, National Yang-Ming University. His research interest is focused on medical instrumentation, medical informatics, and medical electronics. Te-Son Kuo was born in Taiwan, R.O.C., on January 8, He received the B.S. degree in electrical engineering from National Taiwan University, Taipei, Taiwan, in 1960, and the M.S. and Ph.D. degrees in electrical engineering from Georgia Institute of Technology, Atlanta, in 1967 and 1970, respectively. From 1963 to 1964, he had one year study on digital computers in the Philips International Institute of Technological Studies, Eindhoven, The Netherlands. He was a Visiting Assistant Professor in the Electrical Engineering Department at Texas A & M University, College Station, TX, in From 1970 to 1973, he was an Associate Professor and became a Full Professor in 1973, then he served as the Department Head from 1975 to 1981, all in the Department of Electrical Engineering, National Taiwan University, where he is now a Full Professor. He has been also holding a joint appointment with the center for biomedical engineering, National Taiwan University since February His fields of interest are computer-aided design, control system, and biomedical engineering.