Introduction 1 Course goals Introduction 2 SGN 14006 Audio and Speech Processing Lectures, Fall 2014 Anssi Klapuri Tampere University of Technology! Learn basics of audio signal processing Basic operations and their underlying ideas and principles Give basic skills although all the latest cutting edge algorithms cannot be covered! Learn fundamentals of speech processing Speech production and its computational modeling Acoustic features to represent speech signals Some applications: speech coding, synthesis! Learn the basics of acoustics and human hearing These form the foundation for technical applications Introduction 3 Introduction 4 Lecture timeline (some changes may still take place) What is not covered by this course! Sound, audio signals, acoustics! Hearing! Basic audio signal processing operations AD/DA-conversion, filters and filter banks, dynamic control, etc.! Sound synthesis! Audio coding! Speech recognition, audio content analysis, and acoustic pattern recognition " Course SGN-24006 Analysis of Audio, Speech and Music Signals (period 4)! Analog audio Electroacoustics, microphone and loudspeaker design " See the course Akustiikan mittaukset! Speech production anatomy, phonetics! Linear prediction and cepstrum! Speech coding! Speech synthesis! Hardware implementations
Introduction 5 Introduction 6 Practical arrangements Exercises! Course homepage: http://www.cs.tut.fi/~sgn14006! Lectures Mondays 12-14 in TB219 Thursdays 14-16 in TB219 Anssi Klapuri, anssi.klapuri @ tut.fi! Lecture slides will be available as pdf on the course page Course is not based on any individual textbook. Lectures, lecture notes and exercises will be sufficient to take the exam. Some recommended textbooks are mentioned at the end of this introduction! Requirements: exam and project work! 5 cr! Exercises start one week after the lectures (2.9.2014)! Assistants: Aleksandr Diment! Contents: math and Matlab exercises related to the lectures! Two alternative groups Tuesday 10-12 in TC407 Friday 12-14 in TC303 Register to either group on-line at www.tut.fi/pop! Math problems are to be solved in advance, Matlab exercises are done during the exercises! Active completion of the exercises and participation in the exercises is credited up to 3 points in the exam (equivalent to one mark)! Project work will be discussed at the exercises too Introduction 7 Introduction 8 Project work Reference material! Implementing an audio signal processing algorithm in Matlab In two-person groups! Topic(s) will be introduced later during the lectures! Requirements: Choosing the topic Implementing the algorithm Final report by 28.10.! More detailed instructions will appear on the course home page! If you do not have a user account to Birdland Unix / Linux environment (domain *.cs.tut.fi), please apply for one! Gold, Morgan, Ellis, Speech and audio signal processing, Wiley, 2011.! Zölzer. Digital audio signal processing, Wiley&Sons, 2nd ed. 2008. Including AD/DA-conversion, dynamic control, equalization, filter banks! T.F. Quatieri: "Discrete-Time Speech Signal Processing: Principles and Practice", Prentice Hall PTR, 2002.! Rossing. The science of sound, Addison-Wesley, 1990. Acoustics, hearing! Brandenburg, Kahrs. (1998). Applications of digital signal processing to audio and acoustics, Kluwer Academic Publishers Chapter on Perceptual audio coding
Introduction 9 Audio signals Introduction 10 Introduction to audio signals and their representation! Audio = related to sound or hearing! The word sound may mean 1. a sensation perceived by the auditory system, or 2. longitudinal pressure waves in a material medium (such as air) that may cause a hearing sensation Due to human hearing, we usually consider the frequency range 20 Hz 20 khz and air as the medium (although hearing works also underwater for example)! Sound signal audio signal Numerical representation of sound Sound pressure level as a function of time, measured using a microphone for example! Note: audio signal is often understood as non-speech audio signal, although speech signals are audio too Audio and speech processing Introduction 11 Audio signal representations Introduction 12! Where is audio and speech processing needed?! Examples: Convert a musical piece into compressed mp3 format and store it on a hard disc for playback later (audio coding) Encode a speech signal on a mobile phone before transmission Add reverberation to a sound, correct the pitch of a singer (studio technology) Enhance the quality of a speech signal (denoising, echo cancell.) Compensate for loudspeaker non-idealities by digital equalization! Typical digital signal processing system: 1. Digitize a signal (sampling, quantization) 2. Process in digital form (store, manipulate, etc) -digital representation enables a variety of algorithms 3. Convert back to an analog signal! Different applications employ different representations Time domain representation Frequency domain representation Time-frequency domain representation! On this course we consider mainly music and speech Music signals involve a wide variety of sounds, billions of people listen to music worldwide Speech signals are an important special category of sound signals due to their importance for communication
Time domain signal Introduction 13 Time domain signal (1) Introduction 14! Air pressure level as a function of time (zero level = normal air pressure) is a natural representation for audio An analog signal is easy to record using a microphone and play back using a loudspeaker! For music, typical sampling rates are 44.1 or 48 khz Allows for representing the frequency range of human hearing (approximately 20 Hz 20 khz)! For speech 8 khz is the conventional telephone rate (sibilants /s/, /f/ distorted) 16 khz: wideband speech (voice over IP, bandwidth extension)! Other rates are also widely used: 96, 32, 22.05 khz etc.! Most of the energy (and information) of natural sounds is at low frequencies (around 200 Hz 5 khz)! Analog signal (solid line) can be represented with discrete samples (dots) without loss of information, if the sampling frequency 2 * highest frequency component in the signal Remember from introductory signal processing courses Time domain signal (2) Introduction 15 Time domain signal (3) Introduction 16! Large time scale illustrates the sound amplitude envelope! Example signal: one note from the oboe Amplitude is zero before the sound starts The oboe has continuous excitation, therefore the sound s amplitude envelope remains nearly constant throught it duration! Zoom-in of the same oboe signal at time t = 0.45 s! 90 ms frame illustrates the periodic waveform Many sounds are periodic, for example most musical instrument sounds and vowels in speech
Frequency domain representation spectrum Introduction 17 Consider log-frequency and db-magnitude Introduction 18! Obtained by computing discrete Fourier transform (for example) of the time-domain signal, usually in a short frame! Many perceptually important properties are more clearly visible in the frequency domain! Decibel scale for amplitude is useful from the viewpoint of the human hearing and the dynamics of natural sounds! Phases are perceptually less important often omitted! Linear scale usually hard to see anything! Log-frequency each octave is approximately equally important perceptually! Log-magnitude perceived change from 50dB to 60dB about the same as from 60dB to 70dB Time-frequency representation spectrogram Introduction 19 Example audio signals: guitar Introduction 20! Shows sound intensity as a function of time and frequency! Obtained by blocking the signal into short analysis frames and by computing their spectra! For audio, the frame size is typically 10 100 ms: sound spectra are often nearly stationary at that time scale! Sound decays gradually after the onset! Instantaneous excitation: string is plucked at onset! Periodic sound (vibrating string, covered on Acoustics lecture)
Introduction 21 Introduction 22 Example audio signal: snare drum Example audio signals: snare drum (2)! Instantaneous excitation, exponentially decaying amplitude envelope! Zoom-in of the snare drum waveform! The signal contains also non-periodic components Introduction 23 Introduction 24 Example audio signals: snare drum (3) Example audio signals: snare drum (4)! Spectrum is noise-like too: not as clear structure as that in oboe s spectrum! Spectrogram
Polyphonic music (1) Introduction 25 Polyphonic music (2) Introduction 26! Polyphonic music consists of a mix of several sound sources (linear superposition)! Spectrogram reveals e.g. the rhythmic structure Speech: time domain signal (1) Introduction 27 Speech: time domain (2) Introduction 28! One sentence ( He knew what taboos he was violating. )! Speech can be viewed as a sequence of phonemes! Zooming in to different phonemes Left: vowel e in He (voiced: periodic) Right: t in taboos (unvoiced: noisy )
Speech spectrogram Introduction 29! Each phoneme has its characteristic spectral shape! Transitions between phonemes are continuous rather than step-like