3D audio overview : from 2.0 to N.M (?)

Transcription

1 3D audio overview : from 2.0 to N.M (?) Orange Labs Rozenn Nicol, Research & Development, 10/05/2012, Journée de printemps de la Société Suisse d Acoustique "Audio 3D" SSA, AES, SFA

2 Signal multicanal 3D audio in a nutshell sound rendering broadcast sound recording Downmix Extraction des paramètres Codeur Décodeur mono/stéréo Paramètres spatiaux Synthèse Paramétrique CODEUR DECODEUR spatial audio coding sound synthesis format conversion upmix stéréo > 5.1 downmix 5.1 > stéréo downmix 5.1 > binaural Orange Labs - Research & Development Speech & Audio coding, 3D Audio 2012/02/02

3 Summary 3D natural listening what are the (main) acoustic cues used by the auditory system to localize real sound sources? ITD (Interaural Time Difference), ILD (Interaural Level Difference), SC (Spectral Cues) 3D audio for Virtual Auditory Space (VAS) to create spatialized virtual sound sources various technologies stereo and multichannel audio (5.1, 6.1, 7.1, 10.2, 22.2 ) binaural (2.0) holophony & Wave Field Synthesis, WFS (N.M) ambisonics & Higher Order Ambisonics, HOA (N.M) Orange Labs - Research & Development - presentation title date

4 acoustic cues for sound localization - titre de la présentation date

5 How do we localize sounds? acoustic cues used by the auditory system localization in the horizontal plane Duplex theory (Lord Rayleigh): interaural cues time difference IPD (Interaural Phase Difference) ITD (Interaural Time Difference) level difference ILD (Interaural Level Difference) localization in the median plane experiments of hearing phantoms suggest that the [Bloom 1977] perception of sound elevation is based on the spectral content [Blauert ] introduction of the concept of Spectral Cues (SC)

6 IPD/ ITD originated from the difference of the travel path of the acoustic wave between the left and right ears low frequency cue Woodworth's theory ITD c a sin Left Plane wave Right

7 ILD originated from the diffraction of the acoustic wave by the head head's shadow effect on the contralateral ear high frequency cue Pressure level ratio with vs without head (db) Frequency (khz)

8 SC originated from diffraction/ reflection/ resonance by the ear pinna cf. the magnitude variation of transfer functions as a function of elevation

9 Stereo & multichannel sound

10 Stereophonic system virtual source: spatial location controlled by level and/or time difference between the 2 loudspeakers crosstalk not a full 3D spatialization, nor even 2D, but rather "1D 1/6" direct path

11 Stereophonic microphone pairs for natural recording time difference level difference XY Stereosonic MS time and level difference AB system

12 Multichannel sound Originally defined as an audio format dedicated to movies today extended to music, television, games and any multimedia application Typically: 5.1 Front: 3 channels [3Hz-20kHz] right center left Rear: 2 surround channels [3Hz-20kHz]: right surround left surround ".1": 1 LF channel [3-120Hz] Spatialization focused in the front not so much more than stereo "general use": rear channels mainly confined to sound effects Future evolution(s) 6.1, , 22.2 effect of sound elevation with several layers of loudspeakers

13 Multichannel sound recording: 5.1 "trees" IRT OCT DECCA [AFSI, 2008]

14 7.1 set-up

15 10.2 set-up

16 22.2 set-up [Hamasaki, 2007]

17 Binaural technology - titre de la présentation date

18 Binaural technology The sound spatialization the closest to natural listening to mimic 3D spatial hearing with only 2 ears by recording or synthesizing (binaural synthesis) the signals at the entrance of the auditory canal "Full 3D" spatialization the virtual sound source is localized in any direction, all around the listener both azimuth and elevation control [Nicol, 2010 (AES monograph)]

19 Spatial encoding spatial encoding based on the acoustic cues of sound localization ITD, ILD, SC but this encoding depends on the listener's morphology as the result of the interaction between the acoustic wave and the head, the pinna and the torso consequence: the encoding is highly individual

20 Binaural recording recording by a pair of microphones inserted in the ear canal either "natural" head or an artificial head, ie a dummy head Neumann KU100 Head Acoustics Orange Labs - Recherche HSMIII & Développement KEMAR Brüel & Kjær HATS

21 Binaural synthesis virtual recording How? by synthesizing the 2 signals which would have been recorded by 2 microphones inserted at the entrance of the auditory canal by filtering a monophonic signal (1 signal per source) by a pair of binaural filters Each binaural filter is derived by the transfer function between the sound source and the listener's ear This transfer function is defined as a HRTF (Head Related Transfer Function)

22 HRTF Transfer function between the 3D located sound source and the listener's ears Interaction between the acoustic wave and the listener's morphology: reflection & diffraction by the head / pinna / torso Depends on location of the sound source fine sampling of the 3D sphere to account for all spatial variations individual morphology of the listener measuring individual HRTF is required to ensure a highquality of spatialization and immersion Individual HRTF are commonly obtained by acoustic measurement numerical modelling (BEM) [Katz, 1998] [Kahana, 2000]

23 Measurement of individual HRTF TNO laboratory [Bronkhorst, 1995] Headtracking Microphone (blocked-meatus measurement)

24 HRTF: spatial variation horizontal plane median plane

25 HRTF: inter-individual variation 8 individuals for the same direction (Jean-Marie Pernaux database) HRTF (-45, 0 ) Frequency (Hz)

26 Rendering over headphones Headphones are the preferred way to render binaural signals but vibrations perceived by the listener's body are missing Pre-processing is required to compensate for the transfer function between the headphones and the listener's ears HPTF: Head-Phone Transfer Function But the compensation depends on both : the listener's morphology individual compensation is required the positioning of the headphones it is impossible to compensate for the exact (ie current ) positioning average compensation is a commonly adopted compromise What type of headphones? recommendation: to be the closest to free-field condition FEC (Free-air Coupling Equivalent) PDR (Pressure Division Ratio) [Möller & al, 1995]

27 Rendering over louspeakers [Gardner, 1997] HRTF DBB virtual source crosstalk cancellation Binaural signals may be rendered by a pair of loudspeakers provided that the crosstalk is cancelled Various methods transaural system stereo dipole 4-loudspeaker set-up [Atal & Schroeder 1966] [Cooper & Bauck 1988] [Kirkeby, 1997] [Guastavino 2007]

28 Holophony - titre de la présentation date

29 Huyghens' principle n S wavelets E 1 r 0 R E 2 listener(s) r sound source(s) the acoustic pressure in the listening area is expressed as the soundfield produced by a set of sound sources (= wavelet) along the boundary jkr jkr e e p(r, ) R 4 R Kirchhoff- Helmholtz integral p n p cos 1 jkr ds velocity monopole microphone pressure microphone dipole

30 Application to spatial sound reproduction recording: microphone array reproduction: loudspeaker array

31 Soundfield reconstruction advantages the acoustic wave is reproduced identical (at least in theory ) to the original wave temporal and spatial properties are preserved the listener localizes virtual sources in the same way as real sources extensive listening area multi-listener the listener is free to move limitations high number of transducers and signals spatial aliasing the aliasing frequency is imposed by loudspeaker spacing ex: 10 cm spacing aliasing for f f al 1700 Hz but the sound localization is not altered provided that f al 1500 Hz

32 WaveField Synthesis (W.F.S) [Berkhout 1988] One example of a holophonic system in fact, the only one now! Holophony becomes a reality thanks to a set of simplifications validated by early work at Delft University focussing the reproduction on the horizontal plane 1D loudspeaker-array is sufficient, which gives 2D WFS however 3D WFS is still possible (with 2D array ) virtual recording concept of "notional source" the source location is controlled by amplitude and phase

33 WFS: application to a videoconferencing system [Nicol, 1999] video screen sound recording sound reproduction

34 WFS: summary no system for natural sound recording [de Vries, 2009 (AES monograph)] only virtual recording rendering by a loudspeaker array (eg 48.1) subjective assessment points out: the quality of spatialization spectral distorsion multichannel equalization MAP (Multi Actuator Panel) a alternative and promising way to render WFS IST project CARROUSO Sonic Emotion WFS for movie theater IOSONO [Corteel, 2004] [Rébillat, 2010]

35 Ambisonics - titre de la présentation date

36 Step 1: Ambisonics order 1 Soundfield microphone [Gerzon & Craven, 1977] [Gerzon, 1980] spatial audio reproduction based on the recording of: the acoustic pressure: W the 3 components of the acoustic velocity: X,Y,Z the velocity conveys the spatial information B format link with the wave vector (cf. plane wave) W = LF + LB + RF + RB X = LF - LB + RF - RB Y = LF + LB - RF - RB Z = LF - LB - RF + RB A format

37 Step 2: Spherical harmonics 1-order components = X,Y,Z 0-order component = W 2-order components B format what are the spherical harmonics? - they are eigenfunctions which allow one to represent any acoustic wave (3D spherical coordinates) - analysis of the spatial information - hierarchical expansion link with Ambisonics? 3-order components - the 4 signals (W, X, Y, Z) of B format are the 1rst order expansion of the acoustic wave [Bamford 1995] [Daniel 2000]

38 Step 3: Higher Orders Spherical harmonics expansion p( r ) m 0 (2m 1) j B coefficients = Ambisonics signals B m mn 0 n m, 1 Y mn (, ) j m ( kr) spherical harmonics: azimuth and elevation variation Y mn (, ) P mn spherical Bessel functions: radius variation cos( n ) si 1 (sin ) sin( n ) si 1 Higher Order Ambisonics (HOA) extension of the 1rst order expansion to take into account higher order components (M 1)

39 Higher order's benefits the lowest orders provide an accurate reconstruction of the soundfield only at the center of reproduction set-up for an off-centered listener LF: accurate recontruction is preserved for a limited bandwidth for HF: the reconstruction is altered the highest orders provide: a widening of the listening area Synthesis of a plane wave (60 ) an improvement of the rendering of high frequencies for off-centered listeners M = 4, f = 250Hz M = 4, f = 1kHz M = 19, f=1khz

40 Towards HOA Format? a universal representation of a sound scene the spherical harmonics expansion is an exact description for any acoustic wave a scalable format the first components provide a full description of the sound scene higher components only improves spatial accuracy the scalability allows to truncate higher components to adapt to the limitation of the transmission bitrate or the listening set-up a generic format independent of both the recording format (ie microphone signals) the reproduction format (ie loudspeaker signals)

41 Set-up for HOA recording Prototypes from research laboratories [Moreau, Daniel, Bertet], [Meyer, Elko] [Rafaely][Farina] Orange Labs 32 mic. 4-order Maryland Univ. [Lee, Duraiswami] 64 mic. 6-order Orange Labs 20 mic. 3-order Orange Labs 12 mic. 2-order On the market EigenMike (mh-acoustics) 32 mic 4-order B&K (acoustic measurement) 36/50 mic. 5-6 order

42 Set-up for HOA reproduction A specific & attractive property Ambisonics is not dedicated to a particular layout of loudspeakers in theory, any loudspeaker array may be convenient which allows arbitrary choice of the number and the position of the loudspeakers however the simpler the geometry, the robuster the preprocessing Use of a decoding matrix the HOA signals (from the recording step) are pre-processed (decoding step) to derive the loudspeaker signals pre-processing performed by the decoding matrix takes into account the loudspeaker layout for a proper reconstruction of the soundfield various laws of decoding are available to follow various set of constraints for the reconstruction (physical and/or psycho-acoustic)

43 Spatial decoding N loudspeakers W shelf-filter g0 X shelf-filter g1 Y shelf-filter g1 Z shelf-filter g1 Decoding matrix M d shelf-filter gm g M BF MF HF f

44 Conclusion a wide set of technologies are available for spatial sound reproduction including tools for 3D audio recording dummy head HOA microphone array(s) including tools for 3D audio rendering loudspeaker array including MAP headphones "mobile" 3D audio so it's up to us to use these 3D audio tools! to record 3D audio content to listen to to correct and optimize the associate processing(s) to study crossmodal interaction (vision, tactile ) Orange Labs - Research & Development - presentation title date

45 Thank you Any question?... Orange, the Orange mark and any other Orange product or service names referred to in this material are trade marks of Orange Personal Communications Services Limited. Orange Personal Communications Services Limited. France Telecom Group restricted.