Principles and applications of 3D reproduction using Ambisonics. Alois Sontacchi

Transcription

1 Principles and applications of 3D reproduction using Ambisonics

2 @ the focus of Art and Technology Institute of Electronic Music and Acoustics University of Music and Performing Arts Graz AUSTRIA Graz - Inffeldgasse 10/3 2

3 artistic/scientific technical/scientific Professors 1 1 Academic non-professorial teaching staff (scientists) 6 12 Academic non-professorial teaching staff (lectors) 2 7 Other staff members 5 3

4 Education Musicology (Bachelor, Master) in cooperation with KFU Graz Composition - Computer Music (Master + NEW: Bachelor) Electrical and Audio Engineering (Bachelor, Master) in cooperation with TU Graz Doctoral school 4

5 Research Areas Sound & Music Computing Artistic Research sound & space embodiment algorithmic composition Signal Processing and Acoustics spatial audio audio signal processing psychoacoustics Computer Music sonification interaction design 5

6 Motivation What is the necessity for 3D Audio production? Games & Entertainment Music Performance / Composition Listening Test: In-Situ Simulation - speech intelligibility in noise - evaluation of hearing-aids - improvements of ANC-applications Immersive Telepresence LoRA A Loudspeaker-Based Room Auralization System S. Favrot, J. M. Buchholz, Acta Acoustica Vol. 96, p ,

7 Outline Introduction Ambisonics Theory Optimization criteria for reproduction quality Applications 7

8 Introduction Psychologists/psychophysicists discover phantom source around phantom sound localised, where there s nothing physical medical term such as phantom pain 8

9 Introduction X/Y uses first-order transduer characteristics to produce psychoacoustic effect 9

10 Introduction X/Y uses first-order transduer characteristics to produce psychoacoustic effect 10

11 Introduction X/Y uses first-order transduer characteristics to produce psychoacoustic effect -15dB 11

12 Introduction M/S is useful for modifying coincidence stereophony image Encoding

13 Introduction M/S is useful for modifying coincidence stereophony image Encoding

14 Introduction M/S is useful for modifying coincidence stereophony image Decoding Encoding But don t increase S: out-of-phase Advancements of Ambisonics 14

15 Introduction Soundfield microphone A equilateral triangle is a regular 2D simplex; for 3D: tetrahedron FLD BLU FRU BRD Front/Back,Left/Right,Up/Down z BLU FRU y x Encoding y x z w FLD BRD 15

16 Introduction Decoding w by sampling Simplistically

17 Introduction Decoding x by sampling Simplistically

18 Introduction Decoding y by sampling Simplistically

19 Introduction The entire encoding and decoding signal chain instead of array recording, mono signals can be encoded by vector. Ambisonics format Matrix (Encoder) Mic. array config. And properties Matrix (Decoder) Lspk. setup 19

20 Introduction The entire encoding and decoding signal chain Experimental evidence: what is heard? instead of array recording, mono signals can be encoded by vector. How to arrange loudspeakers optimally? Ambisonics format Matrix (Encoder) Mic. array config. And properties Matrix (Decoder) Lspk. setup Higher-order recording unclear before 2000 It won t work anymore using 2D/3D direction vectors How to best shape the decoder? Is it sound field synthesis? What s its theory? 20

21 Theory of sound field synthesis, see Schenk (CHIEF point method) or Burton-Miller solution method The theory behind sound field synthesis lies in the Kirchhoff-Helmholtz integral (KHI) equation 21

22 Theory of sound field synthesis, see Schenk (CHIEF point method) or Burton-Miller solution method The theory behind sound field synthesis lies in the KHI equation and equivalent scattering problem d/dn p Γ is the surface normal derivative of sound pressure (proportional to normal component of particle velocity) 22

23 The theory behind sound field synthesis lies in the KHI equation and equivalent scattering problem For simplicity (no encoding of source distance): Considering amplitude panning, the driving function only selects active sources on the boundary. g is a virtual panning function: beause it is continuous (loudspeakers will be discrete sources) 23

24 Panning and decoding [Franz Zotter, Matthias Frank, All-Round Ambisonic Panning and Decoding, JAES, October 2012] Virtual panning function (with height) enc red>0 blue<

25 Panning and decoding [Franz Zotter, Matthias Frank, All-Round Ambisonic Panning and Decoding, JAES, October 2012] Virtual panning function (with height) enc red>0 blue<0 25

26 Panning and decoding [Franz Zotter, Matthias Frank, All-Round Ambisonic Panning and Decoding, JAES, October 2012] Virtual panning function (with height) red>0 blue< Advancements of Ambisonics 26

27 Panning and decoding [Franz Zotter, Matthias Frank, All-Round Ambisonic Panning and Decoding, JAES, October 2012] Virtual panning function (with height) red>0 blue<0 How to optimally arrange with loudspeakers? Virtual panning function must be decoded to discrete loudspeakers How can they represent it well? Advancements of Ambisonics 27

28 Panning and decoding [Zotter, Pomberger, Frank, An Alternative Ambisonic Formulation, 126th AES Conv, May, 2009.] Trying to do sound field synthesis seriously, with spherical functions:

29 Panning and decoding [Franz Zotter, Matthias Frank, All-Round Ambisonic Panning and Decoding, JAES, October 2012] Trying it more psychoacoustically: Gerzons measures for the decoding problem psychoacoustic measures were considered for decoding. Based on theoretical consideration f<1.5khz loudness direction of localisation f>1.5khz loudness direction of localisation Makita theory. What is relevant for broad-band sounds? What can we demand? 29

30 Panning and decoding [Jérôme Daniel, PhD thesis, 2000] Jérôme Daniel s shapes for virtual panning functions: which is best? Insertion of order weights a n basic Max-r E in-phase (a bit more complicated ) 30

31 Panning and decoding [Franz Zotter, Matthias Frank, All-Round Ambisonic Panning and Decoding, JAES, October 2012] Gerzons measures applied on virtual panning function loudness linear measures direction of localisation loudness quadratic measures direction of localisation 31

32 Panning and decoding [Franz Zotter, Matthias Frank, All-Round Ambisonic Panning and Decoding, JAES, October 2012] Virtual panning functions have nice properties: panning-independent properties of Ambisonic panning functions: symmetric around panning direction ϑ s amplitude integral is constant (V) directional amplitude concentration: - constant magnitude of concentration r V - always points towards panning direction ϑ s =r V / r V energy integral is constant (E) directional energy concentration: - constant magnitude of concentration r E - always points towards panning direction ϑ s =r E / r E 32

33 Panning and decoding [Franz Zotter, Matthias Frank, All-Round Ambisonic Panning and Decoding, JAES, October 2012] Nice properties might get lost after decoding: Which are relevant? Psychoacoustically? symmetric around panning direction ϑs amplitude integral is constant (V) directional amplitude concentration: - constant magnitude of concentration r V - always points towards panning direction ϑ s =r V / r V energy integral is constant (E) directional energy concentration: - constant magnitude of concentration r E - always points towards panning direction ϑ s =r E / r E 33

34 Psychoacoustically relevant factors [Matthias Frank not yet published.] Vector r E is most useful measuring localisation direction horizontal-only level panning broad-band sounds VBAP VBIP 34

35 Panning and decoding [Franz Zotter, Matthias Frank, All-Round Ambisonic Panning and Decoding, JAES, October 2012] Which sampling preserves all energetic measures? 35

36 Panning and decoding [Franz Zotter, Matthias Frank, All-Round Ambisonic Panning and Decoding, JAES, October 2012] Virtual t-design Ambisonics using VBAP VBAP Virtual t-dsgn. Ambi. N = Fig. 6: Triangulation, energy measure, and spread measure as a function of the virtual source direction. [Zotter, Frank, Sontacchi, EAA 2010] [Zotter, Frank, JAES 2012], PhD-Thesis Matthias Frank,

37 Panning and decoding [Franz Zotter, Matthias Frank, All-Round Ambisonic Panning and Decoding, JAES, October 2012] VBAP and Ambisonics compared Triplet-wise panning (VBAP) + constant loudness + arbitrary layout -- varying spread Ambisonic Panning ~+ constant loudness + arbitrary layout ~+ invariant spread 37

38 Psychoacousticians are sceptical about Ambisonics with height. Two vertically stacked loudspeakers do not yield a stable phantom source; should not be more than 30 separated A loudspeaker triplet does not yield stable phantom sources in all orientations, and results are individual However: What if we do not understand Ambisonics as a simple pair or triplet? How diffuse must it be to obtain stable playback? [Results Braun/Frank/Pomberger 2011] Advancements of Ambisonics 38

39 [Sabastian Braun, Matthias Frank, Localization of 3D Ambisonic Recordings ICSA, 2011] Advancements of Ambisonics 39

40 [Sabastian Braun, Matthias Frank, Localization of 3D Ambisonic Recordings ICSA, 2011] Advancements of Ambisonics 40

41 Application for Loudspeaker Reproduction position position rotation 1 2 Encoder Ambisonics Bus 3 4 Encoder Ambisonics Recording Ambisonics Decoder Encoder position Encoder position Live electronics in modern operas with Ambisonics: 2002 Thomas Musil Markus Noisternig 2004 Thomas Musil Peter Plessas 2010 Pomberger, Zotter: An Ambisonics Format for flexible playback layouts, 1 st Int. Ambisoncis Symposium, Advancements of Ambisonics 41

42 Application for Headphone Reproduction Sources Encoding Room Head Rotation Filter Reproduction Advancements of Ambisonics 42

43 I would particularly like to thank Franz Zotter Matthias Frank Hannes Pomberger Advancements of Ambisonics 43

44 Available software tools Tools: CUBEmixer (IEM) spat (ircam) AmbDec, Amb plugins (Fons Adriaensen) ICST Max/MSP plugins plogue bidule Soon more to come. If you are a mixing desk manufacturer Advancements of Ambisonics 44

45 Format Format International Conference on Ambisonics and Spherical Acoustics Advancements of Ambisonics 45

46 Panning and decoding [Franz Zotter, Matthias Frank, All-Round Ambisonic Panning and Decoding, JAES, October 2012] How to record higher order? upscaling methods from first order - DirAC: Fraunhofer / V. Pulkki builds intensity vector from p, vx, vy, vz at each frequency to assign narrow direction determines diffuseness - HARPEX: S. Berge can isolate even two signal directions at each frequency Advancements of Ambisonics 46

47 Higher order microphone arrays Meyer, Elko Rafaely&Park, Petersen, Hald, Moller-Juhl, B&K, Takashima, Nakagawa, Williams, 01db, 2008 Li, Duraiswami, O Donavan, Grassi, visisonics, Jin, v.schaik, Pausch, Zotter Pomberger, 2013 Advancements of Ambisonics 47

48 Time-line 40 years of coincidence recording yield Ambisonics 1930s - Blumlein s coincident directional microphone pair (X-Y, M/S) - Olson s ribbon microphones (figure-of-eight) 1940s - microphones with acoustically switchable characteristics - Bauer s supercardioid has maximal front/back ratio 1950s - Neumann s U57 and AKG s C12 (double membrane) studio microphones allow electronic switching - AKG s logo - Vinyl and tape records available in stereo - (many non-coincident A/B) 1960s - BBC broadcasts in stereo - Psychoacoustics on stereo 1970s - Quadraphony - Matrix stereophony: encoding/decoding in azimuthal harmonics (Cooper, Shiga) - Ambisonics (Gerzon/Fellgett/Craven): recording: CALREC-triplet, Soundfield microphone, decoding/encoding circuits - Panning in higher orders

49 Time-line 40 years later: a hand-full of higher orders, enriched by sound field synthesis theory, psychoacoustic experiments, and a community 1970s to 1990s: - theoretical psychoacoustic decoding, electronics (Gerzon) V,E,rV,rE 1990s: - binaural Ambisonics (Jot) - Sursound list - Pulkki s papers and experiments about 3D stereophony/vbap 2000s: - Daniel (300 pages: theory), - Ward/Abhayapala truncation error - Malham (energetic distribution), - Sontacchi/Höldrich/Noisternig/Z mölnig (IEM-Bin-Ambi, IEM-CUBE - Poletti ( Mode-matching ), More in 2000s: - Meyer/Elko ( Eigenmike, 4th-order recording) End of 2000s: - experimental psychoacosutics: Bertet, Solvang, Kratschmer, Frank, Bates, Marentakis - Advancement in theory (concepts, sampling, truncation): Ahrens/Spors, Fazi, Li, Rafaely - First Ambisonics Symposium: HRTFs, musical instrument radiation, embedding of Ambi into scene description, - Zotter/Pomberger/Frank