DSP First Laboratory Exercise #11 Extracting Frequencies of Musical Tones This lab is built around a single project that involves the implementation of a system for automatically writing a musical score by analyzing the frequency content of a recording (a sampled signal). A primary component of such a system is the spectrogram which produces a time-frequency representation of the recorded waveform. However, to make a working system, several other processing components are needed after the spectrogram to extract the important information related to the notes. The design of these additional blocks will lead naturally to a deeper understanding of what the spectrogram actually represents. 1 Overview In Chapter 9 we introduced the spectrogram as an important tool for time-frequency analysis. We also gave two viewpoints that aid in understanding the spectrogram: a filter bank structure and a sliding-window FFT structure. The filter bank view helps to explain frequency resolution, which can be related to the bandwidth of the channel filters. The FFT view is more useful when discussing computational issues. For music signals the spectrogram tends to produce an image with only a few peaks. Finding these peaks and identifying their frequencies and durations is the main issue in this lab. A Matlab GUI for showing the spectrogram along with musical notation is available for experimentation. In order to make the project manageable, we will progress through several different signal types while testing. These include: 1. Sine waves at a specific frequency. 2. Sine waves that make up a C-major scale. Music GUI 3. Sinusoids that create the tune for Twinkle, Twinkle Little Star. 4. A piano rendition of Twinkle, Twinkle Little Star. 5. Other recorded songs are available for processing: Jesu, Joy of Man s Desiring, Minuet in G, Beethoven s Fifth and Für Elise. 2 Warm-up: System Components In the warmup we will investigate several M-files needed to build the complete processing chain. The instructor verification sheet is included at the end of this lab. 2.1 Spectrogram Computation In Matlab there is already a function for calculating the spectrogram, but it is not always available since it is part of the Signal Processing Toolbox. A similar function is included as part of the DSP First Toolbox, and we have preserved the same list of arguments. The calling format for either specgram is the following: [B,T,F] = specgram( xx, Nfft, fs, window, Noverlap ) Recorded Songs 1
where xx is the input signal, Nfft is the FFT length, fs is the sampling frequency, window is a column vector containing the coefficients of the window, and Noverlap is the number of points in the overlapped part of consecutive sections. The outputs are the spectrogram matrix B, a vector T containing the time locations of the windowed segments, 1 and a vector F which contains the list of scaled frequencies corresponding to the spectrogram analysis. The vectors T and F are useful for labeling plots. Both are scaled by the sampling frequency so the units are seconds for T, and hertz for F. The spectrogram B contains complex values and its size is such that it has a column length (number of rows) equal to length(f) and a row length equal to length(t). IntheDSP First implementation, the calling program must provide all of the arguments. The Matlab function, on the other hand, allows the caller to omit arguments, but that just adds complexity in the programming. In this section, we will present the steps in a spectrogram calculation, so that you could write your own function. The preferred viewpoint for calculation is that of a sliding-window FFT. In this implementation, we take a segment of the signal of length L, multiply the segment by a window, and then compute a zero-padded N-point FFT to form one column of the spectrogram matrix. Then the starting point of the data segment is moved over by an amount L N overlap and the process is repeated. Eventually, we run out of data and the spectrogram is complete. For a Matlab program, we need to write a while loop that tests whether any signal remains. The following example shows all the code needed for the inner loop: B = zeros( Nfft/2 + 1, num_segs ); %- Pre-allocate the matrix L = length(window) %-- assuming a user generated window iseg = 0; while( ) %<==== FILL IN THE TEST CONDITION nstart = 1 + iseg*(l-noverlap); xsegw = window.* xx( nstart:nstart+l-1 ); %-- xx is a column XX = fft( xsegw, Nfft ); iseg = iseg + 1; B(:,iseg) = XX(1:Nfft/2+1); end specgram.m Explain how each of the steps in the spectrogram are being calculated. Explain how to calculate the number of segments num segs ahead of time. Also, explain the purpose of the last line in the while loop. And finally, determine a test that can be used to terminate the while loop. Instructor Verification (separate page) 2.2 Generating the Window The call to specgram requires a window of length L. One possibility is the rectangular window consisting of all ones, but a better window is that for the Hann filter. The rectangular window corresponds to a running-sum filter. The definition of the Hann window is ( ) 2πn w[n] = 1 2 1 2 cos n=1,2,...l (1) L +1 1 There are several conventions for defining the time: (1) start of the segment, (2) middle of the segment, or (3) end of the segment. The middle choice probably makes the most sense, but it really doesn t matter in this project because only relative times will be significant. 2
There are some variations on this definition, but the one given here omits end points that would be zero. Write a function that will generate the Hann window, making sure that it returns a column vector. Then you can use this when calling your specgram function. Make a plot of the Hann window for L = 64. Instructor Verification (separate page) 2.3 Display the Spectrogram The display of the specgram output can be done with the Matlab function imagesc or with the DSP First function show img. On a computer monitor the spectrogram display can use color so that low-level details can be seen, but the conventional printout is a gray scale image with black indicating large values. In addition, if the gray level is proportional to the magnitude of B, small details may be lost, so it might advantages to convert to a logarithmic scale covering 30 or 40 db (called log mag ). Finally, the default orientation in Matlab is a matrix orientation with the origin in the upper left-hand corner. To change this orientation so that the origin is in the lower left-hand corner, use axis xy. The following code fragment summarizes the display: if (LOG) %-- assume LOG is a true/false variable B = 20*log10( abs(b) ); %-- ignore log(0) warnings dbmax = 30; B = B - max(abs(b(:))) + dbmax; B = B.*(B>0); %-- db range is now 0 <= B <= dbmax. else B = abs(b); end imagesc( T, F, B ); colormap(1-gray(256)); axis xy; show img.m 2.4 Finding Peaks Although it might be easy to spot peaks in the spectrogram visually, it is much harder to write a computer program to extract the same peaks reliably. This first step in the process, however, is to just extract all the peaks. Then we can follow this up with an editing program that removes extraneous peaks. The peak-picking function only needs to do one-dimensional picking along the frequency axis because the music spectrogram has a definite horizontal bias the tones last for a long duration along the time (horizontal) axis. If we scan each column of the B(k, l) matrix for peaks, we can merge peaks from neighboring columns to see if a note is present and also determine how long it lasts. A one-dimensional peak-picker is available in the function pkpick.m whose help comments are given below: function [peaks, locs] = pkpick( xx, thresh, number ) %PKPICKER pick out the peaks in a vector % Usage: [peaks,locs] = pkpick( xx, thresh, number ) % peaks : peak values % locs : location of peaks (index within a column) % xx : input data (if complex, operate on mag) % thresh : reject peaks below this level % number : max number of peaks to return 3 pkpick.m
% Test that pkpick.m works as you expect by generating a cosine wave and finding its peaks. Instructor Verification (separate page) An unexpected problem with peak picking is the quantization of the frequency axis. The peakpicking function will give an output that is on the grid of possible frequencies. If we need to estimate the peak location between these grid points, interpolation is needed. The function pkinterp.m is available for that purpose. 3 Design of the Music Writing System The complete system for writing the music is quite complicated, so we follow the engineering practice of breaking the system down into smaller, more manageable, components. 3.1 Block Diagram for the System x[n] Spectrogram IMAGE Peak Picking LIST Editing & Merging KEY # Write Notes SHEET MUSIC Figure 1: Block diagram of major components in music writing system. Figure 1 shows the major sub-systems needed to extract enough information from a musical recording to write the sheet music for that input. Each of these should be implemented as a separate Matlab function. 3.2 Write a Spectrogram Function Use the code fragment above as the basis for writing your own specgram function. Test your function by having it compute the spectrogram of a sine wave. The display should be one horizontal line at the frequency of the sinusoid. 3.3 Parameters of the Spectrogram Window Length: Derive a resolution requirement for separating notes, so that you can specify a window length. The resolution must be converted from continuous-time frequency (in Hz) to discrete-time frequency: ˆω =2π f f s FFT Length: Use a power of two FFT for efficiency. A long FFT will give more frequencies and reduce the gridding problem for peak interpolation. Overlap: Determine the time spacing needed to find the duration of notes. Be careful when making the time spacing very small because the amount of computation will increase dramatically. 4
3.4 Peak Picking & Editing The peak picking operation is relatively straightforward to implement only the number of peaks and a threshold need to be specified. If the threshold is too low, the editing phase will have to deal will many extraneous peaks. The peak picker should generate a list consisting of triplets (frequency, time, amplitude). The function pkpick.m provided in Section 2.4 will only find peaks with one vector, so it must be modified to find peaks as a function of both time and frequency. Editing is the crucial step and also the hardest to specify. Unlike the spectrogram which is a well-defined calculation with only a couple of parameters to adjust, the editing process can take many different forms. The editing system must take a list of frequency-time-amplitude triplets generated by the peak-picker and eliminate many of them based on rules that are derived from common sense. The following issues should be considered: 1. Frequency: (a) How close is the frequency to one of the allowable frequencies of the piano keys. (b) The harmonics must be eliminated, but there are cases where an octave is played, so the second harmonic might be allowed. In addition, when the song has both bass and treble sections, note frequencies can be 4 or 8 times each other. 2. Time: (a) Check the duration; is it a half note, quarter note, etc? This requires that peaks be merged and tracked along the time axis. (b) Timing of the notes. We expect the notes to start at regular times because the music has a rate, such as 2/4 time, or 4/4 time. (c) In fact, an interesting sidelight project would be to extract the beat of the music. This might help to establish a time base for the song, and help set the parameters of the expected durations. (d) There is a minimum duration unless we have a piece with lots of special effects, trills, grace notes, etc. 3. Amplitude: (a) Keep the strongest ones, but how many? (b) If we also look in the time domain, the attack could be found. This is the sharp rise in amplitude at the beginning of a note. 3.5 Writing the Musical Score The output of the editing process should be a list of key numbers and durations that define the music. We have provided a function wrinotes.m that will create a Matlab image that has the notes in the musical score. Consult the help for wrinotes.m to learn the data structure that is needed for its input. wrinotes.m 4 Testing the Music Extraction Program This project is relatively complicated and testing will not be easy. However, several test files are available, progressing very easy cases to difficult ones. You should run your program on the following four test cases: 5
1. Sine waves at a specific frequency. 2. Sine waves that make up a C-major scale. 3. Sinusoids that create the tune for Twinkle, Twinkle Little Star. 4. A piano rendition of Twinkle, Twinkle Little Star. In each case, you know what the true answer should be, so you can assess the capabilities of your music writer. All the piano songs are sampled at 11.025 khz. Alternate songs are: Jesu, Joy of Man s Desiring, Minuet in G, Beethoven s Fifth, and Für Elise. Each of these will be quite difficult and challenging, unless your editing logic is very sophisticated. Remember that the objective of the lab is to make a working system containing the major components listed in Fig. 1. Even with a few simple tests, you should learn quite a bit about the spectrogram, its strengths and its shortcomings. 6
Lab 11 Instructor Verification Sheet Staple this page to the end of your Lab Report. Name: Date: Part 2.1 Complete and explain spectrogram code: Verified: Part 2.2 Write a function to return a Hanning window: Verified: Part 2.4 Test the peak picking function: Verified: 7