ON THE IMPLEMENTATION OF MELODY RECOGNITION ON 8-BIT AND 16-BIT MICROCONTROLLERS

Size: px

Start display at page:

Download "ON THE IMPLEMENTATION OF MELODY RECOGNITION ON 8-BIT AND 16-BIT MICROCONTROLLERS"

Anissa Joseph
5 years ago
Views:

1 ON THE IMPLEMENTATION OF MELODY RECOGNITION ON 8-BIT AND 16-BIT MICROCONTROLLERS Jyh-Shing Roger Jang and Yung-Sen Jang Dept. of Computer Science, National Tsing Hua University, Taiwan {jang, ABSTRACT This paper presents practical issues and considerations when implementing melody recognition on 8-bit and 16-bit microcontrollers. The underlying melody recognition system (also known as query-by-singing/humming system) allows the user to sing or hum a segment of a melody to the microphone and the system can retrieve the intended song in a timely manner. Performance evaluation based on 192 clips shows that the system has a satisfactory top-1 recognition rate of 92% out of 20 candidate songs in the database. This system demonstrates the feasibility of microcontroller based melody recognition for music retrieval, which can be used extensively in consumer electronics such as melody-activated interactive toys, query engines for MP3/VCD/DVD players and karaoke machines, and so on. KEYWORDS Melody Recognition, Content-based Music Retrieval, Dynamic Time Warping, Audio Signal Processing, Query by Singing, Query by Humming, Embedded Systems, Microcontrollers 1. INTRODUCTION This paper presents practical issues and technical tradeoffs when implementing melody recognition on both 8-bit and 16-bit microcontrollers. These issues include precision/computation tradeoffs, effective simplification in algorithms, performance evaluation and optimization, hands-free user interface, rejection mechanism, and so on. As far as we know, this system is the world s first successful attempt of porting melody recognition from desktop PCs to both 8-bit and 16-bit microcontrollers. The system allows the user to retrieve an intended song via her/his acoustic input of singing or humming. The melody recognition algorithm is based on several papers presented previously for desktop PC (see [1][2][3]). For microcontroller implementation, we have to adapt the previously reported algorithms since the computing power and available memory are much less. Moreover, we have to consider other issues for real-world applications, such as melody-activated interactive toys. These practical considerations include quick response time (preferably less than 2 seconds), hands-free operation, duly rejection (to reject unintended input), and appropriate feedbacks (after successfully retrieving the intended song). This paper will discuss how these issues are handled in our system. The successful implementation of the prototype demonstrates the feasibility of using such low-cost system for the massive market of consumer electronics, including melody-activated interactive toys, music query engines for MP3/VCD/DVD players or karaoke machines, and so on. 2. SYSTEM OVERVIEW We have implemented melody recognition on both 8-bit and 16-bit platforms: 8-bit platform: Century Semiconductor Inc. CS3012 microcontroller (which is compatible with 8052) 16-bit platform: Sunplus Inc. SPCE061 microcontroller 2.1 Pitch Tracking The acoustic input to the system should be monophonic, which could be a clip of singing, humming, or single instrument playing. An omni-directional microphone on the demo board is used for recording, with duration of 5 seconds, sample rate of 8000 Hz, and 8-bit resolution for each sample point. The acoustic input is first put into frames of 256 points with no overlap. And every other frame is converted to a pitch in semitone using SMDF (sum magnitude difference function), which is a straightforward extension of AMDF (average magnitude difference function, see [5]) without the averaging operation. More specifically, the SMDF used in our implementation can be expressed as: 128 smdf ( n) = i= 1 y( i) y( n 1+ i), n = 1,..., 128 To save computation, we only choose the first half of the frame as the moving part to compute SMDF. The following figure shows a typical frame and the corresponding SMDF, where the small circle in the second plot denotes the correct minimum point for computing the fundamental period.

2 required to derive the SMDF. The following plot shows the waveform after passing the sign function and the corresponding SMDF: One disadvantage of the above arrangement is that, if the frame containing the onset of a voice, the SMDF curve is not likely to have a clear minimum point due to the low energy of the moving part. This is obvious in the following figure, where plot (a) is the original frame that is almost silent during the first half of the frame. As a result, the SMDF curve, shown in plot (c), does not have a clear minimum. However, if we flip the frame (plot (b)) before computing SMDF, the resulting curve will have a correct minimum point, as shown in the (d) plot of the following figure. All the operations involved from each frame to its fundamental frequency computation is based on either 8- or 16-bit fixed-point of addition/subtraction. However, the fundamental frequency to semitone conversion, as shown in the next equation, is based on table lookup: frequency semitone = * log For 8-bit platform, the pitch resolution is 1 in semitone due to the allowable minimum and maximum number representation. For 16-bit platform, the pitch resolution is 0.1 in semitone since we have much larger range for number representation. The conversion curve of the lookup table for the 16-bit platform is shown next. Therefore we flip the frame before computing SMDF if the energy of the first half frame is less than that of the second half. We choose SMDF instead of ACF (autocorrelation function) for pitch tracking since SMDF does not require multiplication operations. Moreover, the intermediate variables used in SMDF require less bit representation than those used in ACF. However, on 8-bit platform, the computation of SMDF cannot keep up with the recording speed. Moreover, the SMDF is likely to cause overflow with 8-bit representation. As a result, we have to devise a simplified version in order to simplify the computation and representation. This is achieved by passing the original input waveform through a sign function: 1if y( n) > 0, y ( n) = 0, otherwise. In the above equation, y (n) is the original acoustic waveform and y (n) is the simplified waveform after passing the sign function. The computation based on y (n) is much simpler since only 1-bit operations are From the above plot, it is obvious that the min. and max. pitch frequencies are 55 Hz (33 in semitone) and 1661 Hz (92 in semitone), respectively. Thus each pitch point corresponds to (2*256)/8000 = 1/ seconds and the pitch vector has a length of about 5*16.625=78 points. It is imperative to have an efficient pitch tracking algorithm since we would like to have the pitch vector ready for comparison right after the user finishes singing/humming. Therefore the pitch tracking is performed simultaneously while recording. This on-line pitch tracking can ensure a minimum

Use initial several frames to estimate noise level and set a pitch point to zero if it is silent, namely, its energy is less than the threshold. 2. Delete leading and trailing silence. 3.

3 response time that is mostly due to song comparisons, as explained in the following section. After a pitch vector of 78 points is obtained, we still need to perform the following post-processing to ensure a smooth and reliable pitch vector for song comparison: 1. Use initial several frames to estimate noise level and set a pitch point to zero if it is silent, namely, its energy is less than the threshold. 2. Delete leading and trailing silence. 3. For silence between voicing frames, replace the zero-pitch with the previous nonzero pitch. 4. Pass the pitch vector through a median filter of order 5. of the host platform, we cannot perform 5 DTW (as used in [5]) when comparing with each candidate song. For more information about DTW for melody recognition and related issues, the readers are referred to [1][2]. Once the DTW distance to each candidate song is computed, the song with the minimum distance is returned as the most likely song and the system will play the song immediately. 2.3 Specs of the System A snapshot of the demo board of the 8-bit platform (Century Semiconductor Inc. CS3012 microcontroller) is shown next: In summary, to speed up the pitch tracking such that each pitch point can be computed within 1/ = second, we have adopted the following strategy: 1. Use SMDF instead of AMDF or ACF. 2. Apply sign function to simplify the input waveform (for 8-bit platform only). 3. Flip the frame, if necessary, to keep as much pitch information as possible. 4. Take half the frame as the moving part for computing SMDF. 5. Use lookup table to do the conversion from frequency (in Hz) to pitch (in semitone). 2.2 Song Comparisons Once the input pitch vector is identified, we need to compare it with those of the candidate songs in the database. Since the major applications are for toys, we assume the user always sings/hums from the beginning of a song. The generalization to match anywhere in a song is possible but requires more computation. To have effective comparison, we need to deal with two problems: 1. Tempo variation 2. Key variation The specs of the 8-bit platform together with some other related information are shown as follows. Hardware specs: 3.6 MHz oscillator, PLL output frequency 36 MHz Avg. response time: 0.12 sec/song for 5-sec input Data/code size Code memory: 2.75 KB RAM: 2.5 KB ROM: 256 B/song A snapshot of the demo board of the 16-bit platform (Sunplus Inc. SPCE061 microcontroller) is shown next: To account for tempo variation of each individual user, we use dynamic time warping (DTW) [5] to compute the warping distance between the input pitch vector and that of each song in the database. To avoid possible error in note segmentation, we use the whole pitch vector for DTW. The advantage is that the user can present the song using natural singing and the recognition rate is higher. However, the cost to pay is more computation. To deal with key variation, we use a simple heuristics to shift the candidate songs before each DTW computation. The shift amount is chosen such that the first few notes (having a total length equal to that of the input pitch vector) should have the same average semitone as that of the input pitch vector. Due to limited computing power The specs of the 16-bit platform together with some other related information are shown as follows.

Hardware specs: System clock: 24 MHz CPU clock: 24, 12, 6 MHz Avg. response time: 0.06 sec/song for 5-sec input (when CPU clock = 12 MHz.) Data/code size: Code size: 1.

4 Hardware specs: System clock: 24 MHz CPU clock: 24, 12, 6 MHz Avg. response time: 0.06 sec/song for 5-sec input (when CPU clock = 12 MHz.) Data/code size: Code size: 1.14 K word (in flash memory) Song size: 256 word/song (in flash memory) RAM: 1 K word 3. PERFORMANCE OPTIMIZATION AND EVALUATION To evaluate the recognition rate of the system, we have implemented an equivalent software system on PC. We have carefully checked every intermediate results to make sure the PC software has exactly the same behavior as that of the 16-bit platform. So the discussion presented in this section only applies to the 16-bit platform. 3.1 Performance Optimization/Evaluation We collected 192 clips from 19 subjects, where each clip is from the beginning of one of the 20 candidate songs. Some of the clips were taken with the microphone to be two to three meters from the user to mimic the real-world adverse recording situation. We then define a parameterized score function: 1 l score = a * + (1 a)*, d p 1+ b * l where d is the DTW minimum distance, l is the length of minimum DTW path, p is the length of the input pitch vector, and a and b are parameters to be determined. The range of a is 0 a 1 and the range of b is b > 0. The optimum values of a and b were obtained from an offline exhaustive search. That is, we wrote an automatic testing program that applied different values of a and b in our score function to find which combination results in the best recognition rate. The final recognition rate is 91.67%, which is higher than 90.63%, the recognition rate based on minimum DTW distance alone. Based on the above score function, the following plot shows the pie chart of percentages for several ranking categories. 3.2 Rejection Criteria For toy applications, it would be nice to have a duly rejection mechanism by returning a piece of pre -specified music indicating the user s input does not correspond to any of the songs in the system. The purpose of this rejection mechanism is two fold: 1. Prevent from responding to accidental and unintended input. 2. Make the system (sound) smarter since the accepted input should have a better chance of high recognition rate. To implement such a rejection mechanism, we have the following two sequential rejection criteria: 1. Reject the input if the pitch vector (after leading/trailing silence removal and before intermediate silence replacement) has less than 40 nonzero elements. 2. Reject the input if the top-1 score (x ) and the margin between the top-1 and top2 score ( y ) does not satisfy the following inequality: x + y (, y We can plot the x ) pair for each of the 192 clips on a two-dimensional space, together with the above rejection line, as shown next:

5 In the above plot, a circle denotes a hit (correctly recognized clip) while a cross denotes a miss (incorrectly recognized clip). The rejection line is designed such that the system can only take reasonably good input for comparison. The rejection line shown above lead to the following statistics: 1. Acceptance rate is 162/192 = % while rejection rate is %. 2. Among the accepted clips, the top-1 recognition rate is 161/162 = %. 3. Among the rejected clips, the top-1 recognition rate is 15/30 = 50%. Obviously, the accept clips have a very high recognition rate. However, the rejected clips also have a 50% recognition rate. These two percentages are complementary and they should be designed according to the requirement of the actual target applications. 3.3 Error Analysis Among the 16 clips that are not correctly recognized, 12 of them can be attributed to the following reasons: 1. Pitch tracking error: Double/triple pitch occurs, indicating a more sophisticated minimum picking mechanism for SMDF curves should be designed. 2. Endpoint detection error: This is likely to happen when the user is far away from the microphone. 3. Pitch shifting error: This is likely to happen if the tempo of the user s input is quite different from those in the candidate songs. However, we still have 4 songs that cannot be diagnosed to find the cause of error. A possible way to find the reason is to plot and observe the corresponding optimum DTW paths, which is a time-consuming and tedious task. 3.4 Other Considerations For commercial applications such as toys, we must consider more issues that can make the toys much more fun and interesting. These issues are: 1. The response time should be short, preferably less than 2 seconds. This is achieved by allow less songs for low-end platforms. For instance, we have 10 songs in the 8-bit platform to achieve an average response time of 1.2 second. For 16-bit platform (operated at 12 MHz), we have 20 songs to achieve an average response time of also 1.2 second. 2. Instead of button pushing, the recording activation should be as simple and natural as possible. To achieve this goal, we have implemented a mechanism to activate recording based on clapping hands twice. As a result, the operation of the system is totally hands free and there is no need to push a button or the likes to activate recording. 3. Once the correct song is retrieved, the toy should be able to respond with interesting actions other than playing the retrieved song alone. We have respond to this concern by playing the retrieved song from the point where the user s singing is likely to be. This is achieved by considering the end point of the DTW path, plus the comp arison time to extrapolate the position of the user s singing. As a result, the user can keep on singing all along and the system, after identify the melody, can play the melody in harmony with the user s singing. 4. CONCLUSIONS AND FUTURE WORK The successful implementation of the system together with its high recognition demonstrate the feasibility of its use for commercial applications of consumer electronics, such as melody-activated interactive toys, vocal training devices, query-by-singing/humming engines for MP3/VCD/DVD players or karaoke machines, and so on. Since we only use simple sequential search, the response time of the system is proportional to the number of candidate songs. As a result, the response time could be unbearably slow if we have a mo re and more candidate songs for comparison. Currently we are working on new comparison algorithms that allow effective indexing over DTW. 5. REFERENCES [1] Jang, J.-S. Roger and Gao, Ming-Yang "A Query-by-Singing System based on Dynamic Programming", International Workshop on Intelligent Systms Resolutions (the 8th Bellman Continuum), PP , Hsinchu, Taiwan, Dec [2] Jang, J.-S. Roger and Lee, Hong-Ru "Hierarchical Filtering Method for Content-based Music Retrieval via Acoustic Input", The ninth ACM Multimedia Conference, Ottawa, Ontario, Canada, Sept., [3] Jang, J.-S. Roger and Lee, Hong-Ru, "Query by Tapping: A New Paradigm for Content-based Music Retrieval from Acoustic Input", The Second IEEE Pacific-Rim Conference on Multimedia, Beijing, China, October [4] Jang, J.-S. Roger, Chen, Jiang-Chun, and Kao, Ming-Yang, "MIRACLE: A Music Information Retrieval System with Clustered Computing Engines," International Symposium on Music Information Retrieval, Bloomington, Indiana, Oct, [5] Proakis, J. R. J. G. and Hansen, J. H. L. "Discrete-time processing of speech signals," New York, Macmillan Pub. Co., 1993.

Query by Singing and Humming

Abstract Query by Singing and Humming CHIAO-WEI LIN Music retrieval techniques have been developed in recent years since signals have been digitalized. Typically we search a song by its name or the singer