Advanced Audiovisual Processing Expected Background

Transcription

1 Advanced Audiovisual Processing Expected Background As an advanced module, we will not cover introductory topics in lecture. You are expected to already be proficient with all of the following topics, which will be crucial to understanding lectures and preparing suitable coursework. If you are not completely comfortable with any of the topics listed below, please spend time outside of class to improve your proficiency. I suggest visiting the URLs listed here for online resources related to each topic. I also suggest the following resources from the Year 2 module Perception & Multimedia Computing: Home page at contains the lecture slides, labs (with solutions), and occasionally other resources from the offering. All of the non-programming topics listed below are covered in the slides. This is a good first place to go for information. The Vol. 1 subject guide at vol1.pdf covers signals and systems (including spectra, FFT, filters, convolution) in detail, and the Vol. 2 guide at vol2.pdf covers aspects of human perception, some of which are relevant here. Again, between the two subject guides, all of the non-programming topics below are covered in these guides. This is a good second place to go for information. Another good general-purpose resource for audio is BACKGROUND TOPICS YOU SHOULD KNOW Java/Processing programming, object-oriented programming concepts o You should be very comfortable with:! Object-Oriented Programming: Instantiating objects, calling functions (methods) of objects Writing new functions Writing new classes Looking at code for an existing class and knowing how to instantiate and use objects of that class! Understanding the difference between primitive types and objects in Java/Processing, specifically the differences between how they are stored in memory and passed as method arguments (function parameters)

2 Array manipulation ( o Other resources: also any good Processing/Java intro book Basics of sound o What is sound? o What is the relationship between frequency and period? What does it mean to say that a waveform has a frequency of 30Hz? o What is a sine wave?! Define frequency, amplitude, and phase, and relate to the equation y = A*sin(2*pi*f*t + phi)! Be able to draw a sine wave for a specific equation, e.g. 2*sin(2*pi*100*t + pi/2)! Explain how amplitude, frequency, phase affect the sound of a sine wave (e.g., pitch, volume)! Be able to draw a square wave, triangle wave, sawtooth wave given a known amplitude & frequency o What happens when two waves are added together? (e.g., given a sketch of Wave A and Wave B over time, draw the sum of Wave A + Wave B over time) o What is the harmonic series? Why is it important?! in a nutshell, a set of harmonics will each have a frequency equal to N*f for some integer N and some fundamental frequency f, for example {100, 200, 400, 500} or {150, 300, 450, 1500}.! Acoustic pitched instruments (e.g., air tubes such as flutes, bowed or plucked strings such as violins or guitars) will vibrate at many frequencies simultaneously, and the frequencies with the most energy tend to be harmonically-related frequencies! Any synthesised periodic waveform, for example a square wave that repeats at 30Hz, is also a sum of harmonically-related frequencies, with the fundamental being the frequency of the complex waveform (30Hz in this case)! We tend to hear waves that are sums of harmonics as a single pitch, rather than multiple note o How is sound represented on the computer (in uncompressed formats, especially PCM formats)?! Sampling: measuring an analog wave (such as a sound picked up by a microphone) at different, equally-spaced points in time (e.g., times per second) What is the sample rate of a file? What are common sample rates, and why would we choose a higher or lower sample rate?

3 ! Quantization: Choosing a finite number of bits to represent each sample What is the quantization level of a file? What are common quantization levels for PCM audio, and why would we choose a higher or lower quantization level?! Pulse-code modulation (PCM) representations, such as.wav and.aiff files: What is stored in these files? o Resources for sound & waveform basics:! See lessons 1 and 2 of html,! o Resources for sampling, quantization: Time-domain and frequency-domain; spectral analysis o What is the time domain representation of a wave?! This is the representation we are used to, e.g. when drawing a waveform or looking at a DAW: we represent the amplitude of the wave for every point in time (at every sample, if working with a digital waveform) o Understand that any wave can be represented as a sum of sinusoids of specific frequencies, amplitudes, and phases! Pitched sounds are sums of harmonically-related frequencies (called partials )! Unpitched sounds are sums of non-harmonically related frequencies (in the extreme, white noise = all frequencies) o The spectral domain representation of a wave is the representation wherein we specify the frequencies, amplitudes, and phases of each sinusoid that, summed together, produce our waveform! We talk about the spectrum of a particular sound at one moment in time, where the sound is not changing in a perceptible way! Doesn t usually make sense to talk about the spectrum of a whole song (though sometimes it does )! More often, look at a very short snippet of audio (e.g., 100ms), in which the sound isn t changing too much, and discuss the spectrum at that moment o Fourier transform / FFT: What is it?! computational tool that allows us to compute the spectrum for a snippet of audio o Inverse Fourier transform: What is it?! Allows us to compute a snipped of audio from a spectrum o Magnitude spectrum: What is it?

4 ! Because phase often isn t too important to our hearing, we will usually just talk about the relative strengths of different frequencies in the spectrum, or make a plot showing the relative strength of each frequency.! Human hearing o What is the frequency range of human hearing? (roughly 20Hz - 20,000Hz) o If a sound is composed of only harmonically-related frequencies, what does that tell us about the pitch? (see above) o How does spectral content relate to pitch?! If partials are harmonically related, then we ll usually hear the pitch corresponding to the fundamental o How does spectral content relate to timbre?! More high-frequency content generally leads to brighter sounds; less to warmer / darker sounds o What is the relationship between two pitches that are heard as one octave apart?! The higher pitch is twice the frequency of the lower pitch o What is the relationship between to pitches that are heard as one semitone (one note ) apart?! The higher pitch is equal to the lower pitch raised to the power of the twelfth root of 2! Lesson 2 of Basic signals & systems o A signal is an audio waveform, image, or video (for our purposes; usually easiest to think of audio) o A system is a processing component that changes one signal into another signal! e.g., volume knob on a stereo changes the amplitude of the incoming signal o We ll focus on linear time-invariant (LTI) systems, which have some nice properties:! If we want, we can think about the effect of a system in terms of its effect on the spectrum of the input signal (e.g., filtering out certain frequencies)! We can implement a system by taking the FFT of a signal, multiplying the magnitude of each frequency by some value specific to that system, and then take the IFFT of the modified spectrum

5 ! Equivalently, we can implement the system by convolving the input with a special signal called the impulse response of the system Convolution is simply a mathematical operation that can be applied to two signals, similar to addition or multiplication: chapter2.htm the impulse response is the signal that is output by the system when the impulse signal ([1, 0, 0, 0, ]) is input o Filters! In audio, what are low-pass, high-pass, band-pass, band-stop, resonant filters? Specifically, what are the effects on the spectrum of a signal?! In image/video, what are blur, edge detection kernels? Specifically, what are the effects on an image? How do these relate to low-pass and high-pass filters in audio?! p