HCS / ACN 6389 Speech Perception Lab

Transcription

1 HCS / ACN 6389 Speech Perception Lab Course Requirements Matlab problems & lab assignments (40%) Oral presentations (10%) Term project paper (50%) Dr. Peter Assmann Fall Term project: important dates Sep 7: Submit project topics (title only) Sep 21: Submit project outline Sep 28: Present project idea in class Nov 16 / Nov 30: Project presentations Dec 14: Final project paper due Interactive MATLAB Tutorial (requires login) Start Matlab doc Matlab Click on Getting Started with Matlab This launches a video in your browser 4 Command Window enter commands at the prompt >> Create some numeric variables: >> x = 5; % 1x1 scalar >> y = 1:12; % 1x12 vector >> z = rand(4,4); % 4x4 matrix 5 Create character variables by enclosing text in single quotes: >> a = 'c'; % 1x1 character >> b=dir('*.m'); % directory listing >> b=char(b.name); % character array The semicolon tells Matlab not to echo the contents of the variable on the screen. This is important for large vectors (e.g. 10 seconds of speech). 6 1

2 Concatenate variables using square brackets >> a = [ ]; >> b = [ ]; >> c = [a b]; >> disp(c) c = Concatenate character variables using square brackets >> a = 'Matlab is fun! '; >> b = 'Speech perception is interesting.'; >> c = [a b]; >> disp(c) c = Matlab is fun! Speech perception is interesting. 8 Concatenate character variables using square brackets Horizontal concatenation: [a b] Vertical concatenation: [a; b] Combined: [a b; c d] Note: a=1; b=2; c=3; d=4; [a b; c d] ans = a=1; b=2; c=3; d=4; [a b; c] Error using vertcat Dimensions of matrices being concatenated are not consistent. 9 Matlab workspace: variables and data created in the current session Type who to find out what variables exist in the workspace; whos provides details about those variables. >> whos 10 >> whos Name Size Bytes Class Attributes a 1x1 2 char b 16x char x 1x1 8 double y 1x12 96 double z 4x4 128 double Type cd to list the current directory >> cd C:\Users\assmann\Documents\MATLAB Use cd <path> to change to another directory: >> cd C:\Users\assmann\Documents

3 Digital representations of signals Digital representations of signals amplitude time Sampling frequency (e.g khz) Nyquist frequency Effects of discrete-time sampling on bandwidth Quantization rate (16 bits) 16 bits =2 16 quantization steps Effects of discrete-level quantization on dynamic range sampling quantization Vector representation of speech In Matlab speech signals are represented as row or column vectors (e.g., N rows x 1 columns, where N is the number of samples in the waveform). First, download or record a.wav file and store it in the current directory (e.g., wheel.wav on the course web page). Check to make sure the file is in the current directory: >> dir wheel.wav 'wheel.wav' not found. 15 Vector representation of speech In Matlab speech signals are represented as row or column vectors (e.g., N rows x 1 columns, where N is the number of samples in the waveform). First, download (or record) a.wav file and put it in the current directory. >> [y, fs]=audioread('wheel.wav'); % load waveform audioread.m is a Matlab function to read data from.wav files Left side = output variables; right side = input variables The variable y contains the speech signal. The variable fs contains the sample rate in Hz. 16 Vector representation of speech > [y, fs]=audioread('wheel.wav'); % load waveform >> size( y ) ans = Vector representation of speech Load the waveform and plot it: >> [y,fs]=audioread('wheel.wav'); % load waveform >> t=(1:length(y) )./ (fs/1000); % set up time axis >> plot( t, y ); % use plot command >> size( fs ) ans = 1 1 The variable y has 3200 rows x 1 column (row vector). The variable fs has 1 row x 1 column (scalar)

4 Annotating plots >> axis( [ ] ); % set axis limits >> xlabel('time (ms)'); % x-axis label >> ylabel('amplitude'); % y-axis label >> title('waveform plot'); % axis title Try this: >> axis tight >> axis auto Max range Time (x) axis in milliseconds (1000 ms = 1 second) Amplitude (y) axis ~ sound pressure level quantized to 16 bits and range set to ±1. 19 Exercise1 Use built-in Matlab functions to find various properties of the waveform:» length ( y ) % vector length» min ( y ) % minimum value» max ( y ) % maximum value» mean ( y ) % mean value» std ( y ) % standard deviation» plot ( y ) % plot waveform» sound ( y, fs ) % listen to waveform Spectral analysis in Matlab Fourier spectrum of a vector: >> X= fft (y); >> help fft FFT Discrete Fourier transform. FFT(X) is the discrete Fourier transform (DFT) of vector X. If the length of X is a power of two, a fast radix-2 fast- Fourier transform algorithm is used. If the length of X is not a power of two, a slower non-power-of-two algorithm is employed. For matrices, the FFT operation is applied to each column. FFT(X,N) is the N-point FFT, padded with zeros if X has less than N points and truncated if it has more.» soundsc ( y, fs ) % safe version of sound Spectral analysis in Matlab Log magnitude (amplitude) spectrum: >> X= fft (y); >> m = 20 * log10 ( abs ( X ) ); >> help abs ABS Absolute value. ABS(X) is the absolute value of the elements of X. When X is complex, ABS(X) is the complex modulus (magnitude) of the elements of X. Spectral analysis in Matlab Log magnitude (amplitude) spectrum: >> plot(20*log10(abs(fft(y))))

5 » help fp Plotting amplitude spectra FP: function to compute & plot amplitude spectrum Usage: [a,f]=fp(y,fs,window); y: input waveform fs: sample rate in Hz (default Hz) window options: 'hann', 'hamm', 'kais', or 'rect' (default=hamming) [a,f]: log magnitude (db re:1), frequency (Hz) Plotting amplitude spectra» [a,f]=fp(y,fs,window);» [a,f]=fp(y,fs,'hann'); Amplitude (db) p Frequency (khz) Source properties In voiced sounds the glottal source spectrum contains a series of lines called harmonics. The lowest one is called the fundamental frequency (F 0). F 0 Source properties: Pitch Fundamental frequency (F 0) is determined by the rate of vocal fold vibration, and is responsible for the perceived voice pitch. Relative Amplitude (db) Frequency (Hz) Harmonicity and Periodicity Period: regularly repeating pattern in the waveform Period duration T0 = 6 ms Waveform Source-filter independence Effects of F0 changes Harmonics are integer multiples of F 0 and are evenly spaced in frequency Amplitude (db) F 0 = 1000 / 6 = 166 Hz Amplitude Spectrum F 0 = 1 / T Frequency (khz)

6 Effects of formant frequency changes Source-filter independence Exercise: Harmonics in recorded vowels Use Matlab s audiorecorder function to record a vowel, plot the spectrum and inspect the harmonic structure. Produce sustained vowel, /a/ (as in hot ) Exercise: Harmonics in recorded vowels >> fs = 10000; % set sampling rate >> nbits=16; % amplitude quantization (bits) >> nchan = 1; % single channel (mono) >> recobj = audiorecorder(fs, nbits, nchan); >> recordblocking(recobj, 3); % 3 seconds Exercise: Harmonics in recorded vowels >> play(recobj); % listen to the recording >> y = getaudiodata(recobj); % vector of recorded samples with 16-bit double precision >> audiowrite('ah.wav',y,fs); % option: save to file Exercise: Harmonics in recorded vowels Plot amplitude spectrum of your recording >> fp( y, fs ); What is the fundamental frequency (approximately)? What frequency is the strongest harmonic? Frequency axis is in khz (1 khz =1000 Hz)

7 Uniform tube model (schwa) Vocal tract model / ə / Quarter-wave resonator: Fn = ( 2n 1 ) c / 4 L Fn is the frequency of formant n in Hz c is the velocity of sound (about cm/sec) L is the length of the vocal tract (17.5 for adult male) Vocal tract model Acoustic vowel space 3000 Quarter-wave resonator: i heed Fn = ( 2n 1 ) c / 4 L F 1 = (2(1) 1)*35000/(4*17.5) = 500 Hz F 2 = (2(2) 1)*35000/(4*17.5) = 1500 Hz F 3 = (2(3) 1)*35000/(4*17.5) = 2500 Hz Second formant, F2 frequency (Hz) u who d Ə 600 ɑ hod First formant, F1 frequency (Hz) 40 Spectral analysis Amplitude spectrum: sound pressure levels associated with different frequency components of a signal Power or intensity Amplitude or magnitude Log units and decibels (db) Phase spectrum: relative phases associated with different frequency components Degrees or radians Frequency domain representation Why perform spectral analyses of speech? The ear+brain carry out a form of frequency analysis The relevant features of speech are more readily visible in the amplitude spectrum than in the raw waveform BUT: the ear is not a Fourier analyzer. Fourier analysis provides amplitude and phase spectrum; speech cues have been mainly associated with the amplitude spectrum. However, the ear is not "phase-deaf"; many phase changes are clearly audible. Frequency selectivity is greatest for frequencies below 1 khz and declines at higher frequencies

8 » help fp Plotting amplitude spectra FP: function to compute & plot amplitude spectrum Usage: [a,f]=fp (y, fs, window); wave: input waveform rate: sample rate in Hz (default Hz) window options: 'hann', 'hamm', 'kais', or 'rect' (default=hanning) Output: a = log magnitude, f=frequency Plotting amplitude spectra» [a,f]=fp( y, fs, 'hann'); Exercise1 Step 3: Find vowel midpoint; define a range of sample points to extract from the waveform.» nfft = 512;» start = ( length (y) / 2 ) (nfft/2 1);» stop = ( length (y) / 2 ) + nfft/2; % y ( start : stop ) Exercise1 Step 4: Use the function fp.m to compute and plot the amplitude spectrum of the vowel segment:» fp ( y( start : stop ), fs, 'Hamm' ); input vector (waveform segment) input arguments sample rate type of window function 47 Function M-file: fp.m There are two types of M-files: scripts and functions. To display the contents of an M-file, type the following:» type fp.m Function M-files start with a function statement (see next page) and a series of comment lines. The comment lines are included to provide online help and are optional (but very useful!). The next five slides illustrate and explain the contents of the function fp.m 48 8

9 Function M-file: fp.m % FP: function to compute & plot amplitude spectrum % Usage: [a,f]=fp(wave,rate,window); comment lines % wave: input waveform % rate: sample rate in Hz (default Hz) % window options: 'hann', 'hamm', 'kais', 'rect' (default=hanning) % [a,f]: log magnitude, frequency function [ a, f ] = fp ( x, rate, window ) ; optional output arguments a=log magnitude spectrum f=corresponding frequencies function statemen t 49 Function M-file: fp.m % set reasonable defaults for optional variables if ~exist ( 'rate', 'var' ), rate=10000; set defaults end; if ~exist ( 'window', 'var' ), window = 'hamm' ; end; 50 Function M-file: fp.m x = x ( : ) ; n = length ( x ) ; % convert x to column vector % length of data vector Variables defined inside a function are local. In other words, they are not accessible on the command line, outside the function itself. 51 Function M-file: fp.m % illustration of if-else statements: window=lower(window); % window must be lower case if window=='rect', % rectangular window = [ ] x=x.*ones(n,1); % multiplying x by 1 does nothing! elseif window=='hamm', x=x.*hamming(n); % multiply x by Hamming window elseif window=='hann', x=x.*hanning(n); % multiply x by Hanning window else, x=x.*hamming(n); % default case: Hamming window end; 52 Function M-file: fp.m Function M-file: fp.m m=fft(x,n); % Fast Fourier Transform (fastest if n = power of 2) no2=round(n/2); % n/2 samples: FFT is symmetrical a=20*log10( abs ( m ) / n); % convert linear magnitude to db f=rate/n*(0:no2)/1000; % frequency scale: DC = 0 to fs/2 freq = f (1:no2); % retain only the first n/2 samples amp = a (1:no2); % retain only the first n/2 samples % plot amplitude spectrum: frequency vs. amplitude plot ( freq, amp ) ; % frequency = x-axis, amplitude=y-axis axis( [ 0 rate/2000 -Inf Inf ] ) ; % axis range: [ xl xh yl yh ] % ****** End of function fp.m ******

10 Exercise1 Annotate graph: >> xlabel ( 'Frequency (khz)' ); % x-axis label >> ylabel ( 'Amplitude (db)' ); % y-axis label >> title ( filenames ( i, : ) ); % graph title Turn off the axis labels by inserting an empty string: Exercise: modify fp.m % modify fp.m to compute phase spectrum phase = unwrap ( angle (m) ) ; p = 180 / phase; % convert from radians to degrees % plot phase spectrum: frequency vs. phase plot ( freq, phase ) ; % frequency = x-axis, phase=y-axis axis( [ 0 rate/ ] ) ; >> ylabel ( ' ' ); % null axis label Annotations >> xlabel ( 'Frequency (khz)' ); % x-axis label >> ylabel ( 'Amplitude (db)' ); % y-axis label >> title ( filenames ( i, : ) ); % graph title Modifying axes properties Modify default axes properties: >> gca % get current axes = axes handle >> set ( gca, 'XLim', [ 0 4 ] ); % x-axis range >> set ( gca, 'YLim', [ ] ); % y-axis range >> set ( gca, 'TickDir', 'Out' ); % tick mark dir Amplitude spectrum Phase spectrum Amplitude (db) Frequency (khz) 59 Phase (deg) Frequency (khz) 60 10

11 Speech spectrograms What is a speech spectrogram? Display of amplitude spectrum at successive instants in time ("running spectra") How can 3 dimensions be represented on a twodimensional display? Gray-scale spectrogram Waterfall plots Animation Why are speech spectrograms useful? Shows dynamic properties of speech Includes frequency analysis Speech spectrograms in Matlab» help specgram SPECGRAM Calculate spectrogram from signal. B = SPECGRAM(A,NFFT,Fs,WINDOW,NOVERLAP) calculates the spectrogram for the signal in vector A. SPECGRAM splits the signal into overlapping segments, windows each with the WINDOW vector and forms the columns of B with their zero-padded, length NFFT discrete Fourier transforms Speech spectrograms in Matlab» help sp sp: create gray-scale spectrogram Usage: h=sp(wave,rate,nfft,nsampf,nhop,pre,drng); wave: input waveform rate: sample rate in Hz (default 8000 Hz) nfft: FFT window length (default: 256 samples) nsampf: number of samples per frame (default: 60) nhop: number of samples to hop to next frame (default: 5 samples) pre: preemphasis factor (0-1) (default: 1) drng: dynamic range in db (default: 80) title: title for graph (default: none) Making spectrograms >> [y,fs] = audioread)( wheel.wav ); % Load waveform >> sp (wheel, 8000); % Use defaults for other variables >> colormap(copper.*hot); % determines color scheme waveform waveform spectrogram spectrogram F4 F3 F2 F

12 F1 American English vowel space Advancement F2 front center back i heed u who d high ɩ hid ʊ hood Assignment 1 Part 1: (Matlab code, plots, brief summary) Make a set of digital recordings (WAV files) of the 12 vowels of American English: e hayed mid ɛ head Ə schwa o hoed ɔ hawed Height "heed" "hid" "hayed" "head" "had" "hud" "hod" "hawed" low ʌ hut "hoed" "hood" "who d" "herd" æ had ɑ hod Assignment 1 Load waveforms into Matlab; make 12 subplots of the amplitude spectra of the vowels, sampled near the midpoint. Assignment 1 Plot the amplitude spectra of the vowels. Place all 12 plots in a single figure window using the subplot command:» [ y, fs ] = wavread ('heed.wav');» subplot (4,3,1);» start = ( length (y) / 2 ) - 256;» stop = ( length (y) / 2 ) + 256;» fp ( y ( start : stop ), 512, fs, 'heed.wav', 'Hamming'); >> subplot ( 3, 4, 1); >> plot ( x, y ); >> subplot ( 3, 4, 2); >> plot ( x, y ); / / "heed" / / "hid" / / "hayed" / / "head" / / "had" / / "hud" / / "hod" / / "hawed" / / "hoed" / / "hood" / / "who d" / / "herd" Assignment 1 Step 1: Make a list of the filenames as a character array: >> filenames = char ( 'heed', 'hid', 'hayed', 'head', 'had', 'hud', 'herd', 'hod', 'hawed', 'hoed', 'hood', 'whod' ) ; >> deblank ( filenames ( 3, : ) ) Assignment 1 Step 2: Load the waveform of each vowel from the disk: >> for i=1:12, >> [ y, rate ] = audioread ( deblank ( filenames ( i, : ) ) ); >> y = y * 2^15; % scale signal to 16-bit range (±2 15 ) >> end; ans = hayed

13 TrackDraw: a graphical speech synthesizer Fundamental Frequency (F0) window Amplitude of voicing (AV) window TrackDraw: finished tracks Saving, printing and re-loading tracks >> specsynth; % when finished tracking click on exit button >> savetr % save tracks in file; enter name xxheedtr % savetr will append the.mat extension >> load xxheedtr.mat % To re-load track files and run statistics >> plottracks >> print -Pljhd Fourier analysis and synthesis D.P.W. Ellis (2009). An introduction to signal processing for speech. In The Handbook of Phonetic Science, 2 nd edition, edited by Hardcastle, Laver, and Gibbon. chapter 22, pp , Blackwell

14 Ellis (2009, p. 12) 14