SGN 14006 Audio and Speech Processing Introduction 1 Course goals Introduction 2! Learn basics of audio signal processing Basic operations and their underlying ideas and principles Give basic skills although all the latest cutting edge algorithms cannot be covered Lectures, Fall 2015 Pasi Pertilä Tampere University of Technology (slides by Anssi Klapuri)! Learn fundamentals of speech processing Speech production and its computational modeling Acoustic features to represent speech signals Some applications: speech coding, synthesis! Learn the basics of acoustics and human hearing These form the foundation for technical applications Lecture timeline (some changes may still take place) Introduction 3 What is not covered by this course Introduction 4! Sound, audio signals, acoustics! Hearing! Basic audio signal processing operations AD/DA-conversion, filters and filter banks, dynamic control, etc.! Sound synthesis! Audio coding! Speech production anatomy, phonetics! Linear prediction, MFCCs, and cepstrum! Speech coding! Speech synthesis! Speech recognition, audio content analysis, and acoustic pattern recognition " Course SGN-24006 Analysis of Audio, Speech and Music Signals (period 4)! Analog audio Electroacoustics, microphone and loudspeaker design " See the course Akustiikan mittaukset! Hardware implementations
Practical arrangements Introduction 5! Course homepage: http://www.cs.tut.fi/~sgn14006! Lectures Mondays 12-14 in TB219 Thursdays 14-16 in TB222 Pasi Pertilä, pasi.pertila @ tut.fi! Lecture slides will be available as pdf on the course page Course is not based on any individual textbook. Lectures, lecture notes and exercises will be sufficient to take the exam. Some recommended textbooks are mentioned at the end of this introduction! Requirements: exam and project work! 5 cr Exercises Introduction 6! Exercises start one week after the lectures (2.9.2015)! Assistants: Shriram Nandakumar, Emre Cakir! Contents: math and Matlab exercises related to the lectures! Two alternative groups Tuesday 10-12 in TC303 (updated!) Friday 12-14 in TC303 Register to either group on-line at 14:00 today www.tut.fi/pop! Math problems are to be solved in advance, Matlab exercises are done during the exercises! Active completion of the exercises and participation in the exercises is credited up to 3 points in the exam (equivalent to one mark)! Project work will be discussed at the exercises too Introduction 7 Introduction 8 Project work Reference material! Implementing an audio signal processing algorithm in Matlab In two-person groups! Topic(s) will be introduced later during the lectures! Requirements: Choosing the topic Implementing the algorithm Final report by 28.10.! More detailed instructions will appear on the course home page! Gold, Morgan, Ellis, Speech and audio signal processing, Wiley, 2011.! Zölzer. Digital audio signal processing, Wiley&Sons, 2nd ed. 2008. Including AD/DA-conversion, dynamic control, equalization, filter banks! T.F. Quatieri: "Discrete-Time Speech Signal Processing: Principles and Practice", Prentice Hall PTR, 2002.! Rossing. The science of sound, Addison-Wesley, 1990. Acoustics, hearing! Brandenburg, Kahrs. (1998). Applications of digital signal processing to audio and acoustics, Kluwer Academic Publishers Chapter on Perceptual audio coding! Pulkki, Karjalainen, Communication acoustic,2015, Wiley
Introduction 9 Audio signals Introduction 10 Introduction to audio signals and their representation! Audio = related to sound or hearing! The word sound may mean 1. a sensation perceived by the auditory system, or 2. longitudinal pressure waves in a material medium (such as air) that may cause a hearing sensation Due to human hearing, we usually consider the frequency range 20 Hz 20 khz and air as the medium (although hearing works also underwater for example)! Sound signal audio signal Numerical representation of sound Sound pressure level as a function of time, measured using a microphone for example! Note: audio signal is often understood as non-speech audio signal, although speech signals are audio too Audio and speech processing Introduction 11 Audio signal representations Introduction 12! Where is audio and speech processing needed?! Examples: Convert a musical piece into compressed mp3 format and store it on a hard disc for playback later (audio coding) Encode a speech signal on a mobile phone before transmission Add reverberation to a sound, correct the pitch of a singer (studio technology) Enhance the quality of a speech signal (denoising, echo cancell.) Compensate for loudspeaker non-idealities by digital equalization! Typical digital signal processing system: 1. Digitize a signal (sampling, quantization) 2. Process in digital form (store, manipulate, etc) -digital representation enables a variety of algorithms 3. Convert back to an analog signal! Different applications employ different representations Time domain representation Frequency domain representation Time-frequency domain representation! On this course we consider mainly music and speech Music signals involve a wide variety of sounds, billions of people listen to music worldwide Speech signals are an important special category of sound signals due to their importance for communication
Time domain signal Introduction 13! Air pressure level as a function of time (zero level = normal air pressure) is a natural representation for audio An analog signal is easy to record using a microphone and play back using a loudspeaker! For music, typical sampling rates are 44.1 or 48 khz Allows for representing the frequency range of human hearing (approximately 20 Hz 20 khz)! For speech 8 khz: Narrowband the conventional telephone rate (sibilants /s/, /f/ distorted) 16 khz: Wideband voice over IP, bandwidth extension! Other rates are also widely used: 96, 32, 22.05 khz etc.! Most of the energy (and information) of natural sounds is at low frequencies (around 200 Hz 5 khz) Time domain signal (1) Introduction 14! Analog signal (solid line) can be represented with discrete samples (dots) without loss of information, if the sampling frequency 2 * highest frequency component in the signal Remember from introductory signal processing courses Introduction 15 Introduction 16 Time domain signal (2) Time domain signal (3)! Large time scale illustrates the sound amplitude envelope! Example signal: one note from the oboe Amplitude is zero before the sound starts The oboe has continuous excitation, therefore the sound s amplitude envelope remains nearly constant throught it duration! Zoom-in of the same oboe signal at time t = 0.45 s! 90 ms frame illustrates the periodic waveform Many sounds are periodic, for example most musical instrument sounds and vowels in speech
Introduction 17 Frequency domain representation spectrum! Obtained by computing discrete Fourier transform (for example) of the time-domain signal, usually in a short frame! Many perceptually important properties are more clearly visible in the frequency domain! Decibel scale for amplitude is useful from the viewpoint of the human hearing and the dynamics of natural sounds Due to Fechner s law (subjective sensation is proportional to the logarithm of the stimulus intensity)! Phases are perceptually less important often omitted Consider log-frequency and db-magnitude! Linear scale usually hard to see anything! Log-frequency each octave is approximately equally important perceptually! Log-magnitude perceived change from 50dB to 60dB about the same as from 60dB to 70dB Introduction 18 Time-frequency representation spectrogram Introduction 19 Example audio signals: guitar Introduction 20! Shows sound intensity as a function of time and frequency! Obtained by blocking the signal into short analysis frames and by computing their spectra! For audio, the frame size is typically 10 100 ms: sound spectra are often nearly stationary at that time scale! Sound decays gradually after the onset! Instantaneous excitation: string is plucked at onset! Periodic sound (vibrating string, covered on Acoustics lecture)
Introduction 21 Introduction 22 Example audio signal: snare drum Example audio signals: snare drum (2)! Instantaneous excitation, exponentially decaying amplitude envelope! Zoom-in of the snare drum waveform! The signal contains also non-periodic components Introduction 23 Introduction 24 Example audio signals: snare drum (3) Example audio signals: snare drum (4)! Spectrum is noise-like too: not as clear structure as that in oboe s spectrum! Spectrogram
Polyphonic music (1) Introduction 25 Polyphonic music (2) Introduction 26! Polyphonic music consists of a mix of several sound sources (linear superposition)! Spectrogram reveals e.g. the rhythmic structure Speech: time domain signal (1) Introduction 27 Speech: time domain (2) Introduction 28! One sentence ( He knew what taboos he was violating. )! Speech can be viewed as a sequence of phonemes! Zooming in to different phonemes Left: vowel e in He (voiced: periodic) Right: t in taboos (unvoiced: noisy )
Speech spectrogram Introduction 29! Each phoneme has its characteristic spectral shape! Transitions between phonemes are continuous rather than step-like AD#1 NERDS MEET ART ISTS 2015-2016 Joint Course Module of Signal Processing, School of Architecture and Civil Engineering GOAL: Introduction 30 Help signal processing engineers to understand needs of urban design and help architects and civil engineers to understand potential of modern ICT in quantitative analysis of urban spaces. With the help of camera and microphone systems automatic analysis is provided for quantitative urban space monitoring. The quantitative data is used for boosting architectural and civil engineering design of future urban spaces. COURSE: SGN-81006 S ignal Processing Innovation Project PARTICIPATION: Enroll to the above course and come to the O pening Session August 25 2015 10:00-12:00 RO104 where the overall description is given and the project groups will be formed. The works will be supervised by the researchers from Department of Signal Processing, School of Architecture and Department of Civil Engineering. FOR MORE INFORMATION: Harry Edelman (School of Architecture / Dept. of Civil Engineering) Joni Kämäräinen (Dept. of Signal Processing - video processing) Tuomas Virtanen (Dept. of Signal Processing - audio processing) Invitation to Data Collection Campaign AD#2, Participate in a study, get a movie ticket! Introduction 31 I A project in Department of Signal Processing needs speech data for research purposes. I Your task is to read out simple English sentences from a script. Takes 25 minutes. I Reward: a movie ticket. I How to participate? I We need two persons per recording.! come with a friend. If you are alone, we could try to pair you. I Sign-up via email aleksandr.diment@tut.fi I The sessions take place on 24-28.8 during office hours, or at a different time upon agreement.