BINAURAL SOUND GENERATION CORRESPONDING TO OMNIDIRECTIONAL VIDEO VIEW USING ANGULAR REGION-WISE SOURCE ENHANCEMENT

Transcription

1 BINAURAL SOUND GENERATION CORRESPONDING TO OMNIDIRECTIONAL VIDEO VIEW USING ANGULAR REGION-WISE SOURCE ENHANCEMENT K. Niwa, Y. Koizumi, K. Kobayashi, and H. Uematsu NTT Media Inteigence Laboratories, Japan ABSTRACT Web appications for watching omnidirectiona video through headmounted dispays (HMDs) or smartphones have been widey distributed. The goa of this study was to generate binaura sounds corresponding to the user viewpoint. Assuming that a microphone array is used for sound recording, the enhanced signa for each anguar region can be extracted. By convoving head-reated transfer functions (HRTFs) and enhanced and re-synthesizing them, binaura sounds corresponding to the user viewpoint can be virtuay generated. In this paper, we propose a method for achieving anguar region-wise source enhancement by generating a mutichanne Wiener fiter based on the power spectra density (PSD)-estimationin-beamspace method. To measure user ocaization when watching omnidirectiona video through an HMD, we used a system that enabes the generation of binaura sounds corresponding to the user viewpoint in rea time. Through subjective tests, we confirmed that sound ocaization corresponding to the user viewpoint can be obtained when appying about a 40-degree anguar region-wise source enhancement. Index Terms omnidirectiona camera, microphone array, binaura sound, head-reated transfer functions (HRTFs), source enhancement 1. INTRODUCTION In virtua reaity (VR), simuating presence as if being in a remote pace has been activey studied. Because many approaches have been investigated to address this issue, many types of omnidirectiona cameras and head-mounted dispays (HMDs) have been reeased recenty. Capturing omnidirectiona video is possibe through web video streaming services or smartphone appications. Despite the fact that the user viewpoint seamessy changes, the streamed sounds through headphones are fixed even when the user viewpoint changes. Thus, the goa of this study was to generate binaura sounds corresponding to the user viewpoint. Severa methods for virtuay generating binaura sounds corresponding to user contro, such as those invoving seective istening point (SLP) audio [1], an interactive controer [2], and instrument equaizer [3], have been studied. The SLP audio method was deveoped for generating binaura sounds corresponding to free viewpoint TV [4, 5]. Sounds are captured by pacing many microphones to surround the acoustic fied and a sound separation agorithm based on frequency-domain independent component anaysis (FD-ICA) [6]- [11] is appied. After generating fixed inear fiters to extract separate sources grouped into severa ocaized regions and convoving them with head-reated transfer functions (HRTFs) [12] from their positions to the istener s ears, virtuay generating binaura sounds corresponding to the user contro (e.g. mouse dragging) is possibe. However, since FD-ICA is a batch-based agorithm, it is difficut to adaptivey update fiters to foow the sound source movement. To generate binaura sounds independenty of the position/number changes on sound sources, a source enhancement method for dividing the acoustic fied eveny into severa areas (e.g. equianguar regions) shoud be appied. This is preferabe for appication to omnidirectiona video viewing since separated are resynthesized after convoving with HRTFs. We previousy deveoped the PSD-estimation-in-beamspace method, which may be an effective method for enhancing the arriving from the identified anguar region [13, 14]. By using mutipe beamforming output, the PSD of the sound sources arriving from the identified anguar region and that of surrounding noise can be estimated. By designing Wiener fiter using estimated PSDs, it may be possibe to segregate arriving from the identified anguar region from surrounding noise. Thus, by adjusting PSDs among adjacent anguar regions, the mutichanne Wiener fiters for anguar region-wise source enhancement can be generated. For this study, we (i) expanded the PSD-estimation-in-beamspace method for anguar region-wise source enhancement and (ii) evauated our rea-time omnidirectiona video viewing system. When a user wears an HMD equipped with a gyro sensor, his/her head orientation can be captured in rea time. By convoving the separated and HRTFs corresponding to the head motion and re-synthesizing them, the binaura sounds corresponding to the user viewpoint can be generated. Since the received sounds and dispayed images vary with head motion, the ocaization of each user woud adapt to the viewing content. For such a situation, we investigated the reationships between user ocaization with/without our anguar region-wise source enhancement method through subjective tests. This paper is organized as foows. In Sec. 2, we give a system overview of generating binaura sounds and detais of our anguar region-wise source enhancement method. In Sec. 3, we expain our omnidirectiona video viewing system. After discussing the subjective tests we discuss our evauation of our omnidirectiona video viewing system in Sec. 4 and concude this paper in Sec BINAURAL SOUND GENERATION CORRESPONDING TO USER VIEWPOINT 2.1. System overview Let us assume that K sound sources are observed with a microphone array composed of M sensors. The k-th source signa and m-th observation signa in the frequency ω and frame-time τ are denoted as S k, and X m,, respectivey. When the transfer function between them is denoted as A m,k,ω, M observation x are modeed as x = A ω s + n, (1) /16/$ IEEE 2852 ICASSP 2016

2 Transfer functions A User contro (Head motion was Observed measured using HMD) Pre-enhanced Separated Binaura x, y, z, X b, 1,, Y 1,, Z 1,, (Left) X 2,, Directivity forming Y 2,, Anguar region-wise Z Convouting of B, 2,, (e.g. beamformings or source enhancement enhanced (Right) receiving using (e.g. ex X M,, shotgun microphones) Y and HRTFs, and B xpanded PSD estimation, L,, Z in beamspace method) L,, re-synthesing them Fig. 1. Signa processing fow to generate binaura sounds corresponding to user viewpoint where x =[X 1,,...,X M,] T, (2) a k,ω =[A 1,k,ω,...,A M,k,ω ] T, (3) A ω =[a 1,ω,...,a K,ω], (4) s =[S 1,,...,S K, ] T, (5) n =[N 1,,...,N M,] T, (6) Here, T and N m, denote the transposition and incoherent background noise at the m-th microphone, respectivey. We now expain the generation of binaura sounds corresponding to the user viewpoint, which are denoted as b = ] T. The user head orientation at τ is represented in poar coordinates as Ψ τ = [Ψ τ (Hor), Ψ (Ver) τ ] T and can be obtained from the gyro sensor instaed in the HMD or smartphone. The directivity of the sound source and background noise are ignored. The binaura sounds corresponding to the user viewpoint are outputted by convoving HRTFs with the sound source as [B (Left),B (Right) where H (Left) k,ψ τ,ω B (Left) B (Right) k,ψ τ,ω H (Left) k,ψ τ,ω S k,, (7) H (Right) k,ψ τ,ω S k,, (8) are the HRTFs between the k-th source position and user eft/right ear, respectivey. By taking into account that HRTFs do not drasticay vary with sight position change, grouping source in a oca anguar region (ocaized signa) wi have no appreciabe effect on user ocaization. Thus, we aimed to extract L ocaized whose shaft centers are respectivey toward the direction Θ =[Θ (Hor), Θ (Ver) ] T ( =1,...,L) instead of extracting source signa individuay. Athough a method of extracting ocaized, which are denoted as Z Θ, ( =1,...,L), is expained in Sec. 2.2, those are assumed to be aready prepared in this section. In such a situation, the binaura sounds corresponding to the user viewpoint are approximatey cacuated as B (Left) B (Right) =1 =1 H (Left) Θ,Ψ τ,ω ZΘ,, (9) H (Right) Θ,Ψ τ,ω ZΘ,, (10) where H (Left) Θ,Ψ τ,ω Θ,Ψ τ,ω are the HRTFs from the representative direction of the -th region to the user s eft/right ear, respectivey. An overview of generating b is shown in Fig. 1. Athough HRTFs vary with the room reverberation time, individuaity of the aurice/head structure, and distance from source to receiver [12], we ignored this for this study. This makes it possibe to simpy represent H (Left) Θ,Ψ τ,ω Θ,Ψ τ,ω.specificay, they are seected from a database [15] composed of impuse responses measured by discretey pacing a oudspeaker and a head-and-torso simuator (HATS) in a ow reverberation room PSD estimation in beamspace To achieve anguar region-wise source enhancement, we expanded the PSD-estimation-in-beamspace method [13]. By appying beamforming to x or simpy observing with, e.g., shotgun microphones, the source arriving from Θ are assumed to be pre-enhanced, and L pre-enhanced y = [Y Θ1,,...,Y ΘL,] T are obtained. Assuming that the source are uncorreated, the PSD of Y Θ, is modeed as φ YΘ,ω = Y Θ, 2 D Θ,k,ω 2 φ Sk,ω, (11) where and φ Sk,ω denote the expectation operator and PSD of the k-th sound source, respectivey. Since the reationships, expressed by Eq. (11), can be satisfied between pre-enhanced and ocaized, φ YΘ,ω is approximatey represented as φ YΘ,ω D Θ,Θ i,ω 2 φ SΘi,ω, (12) i=1 where D Θ,Θ i,ω denotes the average sensitivity of the -th preenhancement to the anguar region whose shaft center is Θ i,and φ SΘi,ω represents the PSD of the i-th ocaized signa. The reationships between φ SΘi,ω and φ YΘ,ω can be modeed in matrix form as φ YΘ1,ω D Θ1,Θ 1,ω 2 D Θ1,Θ L,ω 2 φ SΘ1,ω. = φ YΘL,ω D ΘL,Θ 1,ω 2 D ΘL,Θ L,ω 2 φ SΘL,ω }{{}}{{}}{{} Φ Y,ω D ω Φ S,ω (13) To estimate L ocaized, the inverse probem of Eq. (13) can be soved. For this study, assuming that sparseness of the sound 2853

3 Tabe 1. Conditions for HRTFs measurement [15] Samping frequency 44.1 khz (Up-samping to 48 khz) Room reverberation time 310 ms # of horizonta anges 72 (5 interva) # of vertica anges 28 (5 interva) (from -45 to 90 degrees) Z Θ, = G Θ,Y Θ,. (18) By appying the inverse fast Fourier transform (FFT) to Z Θ,,the time-domain ocaized are obtained. Fig. 2. (a) User watches omnidirectiona video through HMD and headphones. (b) Stereoscopic view of HMD sources in the time-frequency domain is adequatey high, the PSD of the target region and that of surrounding noise can be estimated frame by frame as ˆΦ S, = D 1 ω Φ Y,. (14) Athough the effect of incoherent background noise is ignored in Eq. (14) for simpicity, its PSD is separatey estimated and subtracted from each PSD of the ocaized signa. The detais of how to cacuate the PSD of incoherent background noise is described in our previous work [14]. In this study, we estimated the PSD of background noise assuming that it is temporay stationary Anguar region-wise source enhancement by adjusting estimated PSDs of ocaized To expand the PSD-estimation-in-beamspace method for anguar region-wise sound enhancement, we adjusted mutichanne PSDs of ocaized between adjacent anguar regions. When simpy appying Eq. (14), the estimated PSDs may incude estimation errors since the same sound source can be emphasized among adjacent anguar regions. This may degrade the sound ocaization of binaura. To reduce the PSD estimation error among adjacent anguar regions, the region index whose estimated PSD is the highest, described by ξ τ, is cacuated for each frame as ξ τ =argmax ˆφ, (15) SΘ, ω Ω By suppressing ˆφ SΘ,, whose region index is adjacent to Θ ξ τ, emphasizing the same sound source in different regions can be avoided by using { ˆφSΘ ˆφ =, ( = ξ τ ) SΘ, 0 ( φ, (16) τ) where φ τ denotes the index set of the adjacent anguar region of Θ ξτ. Finay, by using adjusted PSDs, a Wiener fiter to enhance arriving from the -th anguar region is generated using G Θ, = The ocaized are cacuated as ˆφ SΘ, L i=1 ˆφ SΘi,. (17) 3. REAL-TIME OMNIDIRECTIONAL VIDEO VIEWING SYSTEM The rea-time omnidirectiona video viewing system, which incudes an HMD and headphones, as shown in Fig. 2, was used [18]. The Ocuus Rift DK2 (Ocuus VR Inc.) was used as the HMD. With Ocuus Rift DK2, wide perspective fish-eye enses, whose view ange is from 100 to 110 degrees, are mounted in front of the screen with a graphica resoution of pixes. To compensate for the optica distortionoriginating from the fish-eye enses, the inverse characteristics of the enses are mutipied to the screen-dispayed images for each frame. Aso, Ψ τ can be observabe in rea time using the gyro sensor and triaxia sensor instaed in Ocuus Rift DK2. Since the image to cover the visua fied is generated and corresponds to the user s head motion, immersive video viewing is possibe for each user. To sove the probem of arge-capacity image-data transmitting and rendering, we used the H. 264 (MPEG-4 AVC)-based omnidirectiona image streaming system [16, 17, 18]. The high-resoution image captured with an omnidirectiona camera is too arge to render to correspond to Ψ τ. Instead of streaming uncompressed omnidirectiona images, both the entire ow-resoution omnidirectiona image (around 0.5 Mbps) and partia high-resoution image corresponding to the user overooking region (around Mbps) are transmitted in parae. To foow quick head motion, ow-resoution images are outputted to the screen after modifying the optica distortion of the fish-eye enses. When the user s head is fixed to an identified direction for around seconds, the resoution of the user viewpoint is switched from ow to high. For outputting cacuated binaura sounds, the ASIO audio interface (Roand Octa-capture) and headphones (SONY MDR- CD900ST) are used. The L ocaized are assumed cacuated beforehand. By convoving HRTFs and ocaized, as in Eq. (10), binaura sounds corresponding to the user viewpoint are generated in rea time. The impuse responses measured by pacing HATS (B&K 4128C) in a soundproof chamber were used as HRTFs [15]. Since the anguar interva of the HRTF database is degrees, as shown in Tabe 1, seamess ocaization contro corresponding to a user s head motion can be achieved. 4. EXPERIMENTS 4.1. Recording setup Two kinds of futsa (a modified form of soccer) actions were recorded using an omnidirectiona camera (Point Grey Ladybug3) and an icosahedra microphone array, as shown in Fig. 3. The icosahedra microphone array was composed of M =20shotgun microphones positioned for each face center. The diameter of the 2854

4 (a) Sound ocaization #1: Practice match #2: Ba jugging (b) Sound quaity #1: Practice match #2: Ba jugging MO OS score MO OS score MOS score MOS score Fig. 3. Recording system composed of omnidirectiona camera and icosahedra microphone array Tabe 2. Experimenta parameters Samping frequency 48.0 khz Number of microphones, M 20 (shotgun microphone) Diameter of array 0.4 m Number of beamspaces, L 20 FFT window ength 5.30 ms FFT shift 2.65 ms array was 0.4 m. The camera and microphone array were paced on the futsa court. To prevent the array from being caught on camera, it was paced 0.5 m beow the camera. The recorded actions were (i) a practice match: payers chasing a ba to score a goa and (ii) ba jugging: four payers jugge a ba aternatey whie paying word chain. Ba kicking, running around the fied, payers shouting, audience cheering, and air conditioning noise were incuded in the observed. Since the motion of futsa payers was fast during the practice match, the arriva direction of sounds varied from frame to frame. Since the payer position was neary fixed for ba jugging, the viewer was abe to associate the arriva direction of sounds whie viewing the images. By dividing the acoustic fied into L =20equianguar regions whose center shaft extends from the array center through each microphone, ocaized sounds were estimated. The anguar difference between adjacent regions was 41.8 degrees. Since shotgun microphones were used in this experiments, the pre-enhanced y were equa to x. Since non-stationary sound sources, such as ba kicking, were mixed with the observed, they were anayzed in a short window ength (5.3 ms). The adjacent regions defined in Eq. (16) were determined as a hemisphere; thus, the number of simutaneous ocaized sounds was up to 2. The other experimenta parameters are isted in Tabe Subjective evauations We conducted subjective evauations using our omnidirectiona video viewing system. We found that appying our anguar regionwise source enhancement method affected (i) sound ocaization corresponding to the overooking image and (ii) overa sound quaity, which incudes the degradation caused from our mutichanne Wiener fitering. As a comparison method, pre-enhanced M =20 observed were simpy convoved with HRTFs corresponding to the user s head motion. The participants were six maes and the sequence number for each action and the sound processing method W/O W W/O W W/O W W/O Fig. 4. Subjective evauation resuts, (a) sound ocaization, and (b) sound quaity. (W/O): Without source enhancement processing, (W): With our anguar region-wise source enhancement method. was five. The ength of actions was imited to 20.0 seconds and we asked each user to rotate his head. The mean opinion scores (MOS) (5: Exceent, 4: Good, 3: Fair, 2: Poor, 1: Bad) were used as an evauation measure. Fig. 4 shows the MOS scores. The bod horizonta ines indicate the average scores of the six participants and the horizonta narrow ines indicate the minimum/maximum score. As shown in Fig. 4(a), the MOS score of sound ocaization increased by appying our anguar region-wise source enhancement method. However, some participants rated high score to the comparison method. This was due to the fact that effective cues for sound ocaization were incuded in the observation since many directivity microphones (M =20) were used for observation. As shown in Fig. 4(b), the sound quaity improved a bit with our method. This was due to the fact that stationary air conditioning noise was suppressed with our method. 5. CONCLUSION We proposed an anguar region-wise source enhancement method for our immersive omnidirectiona video viewing system. To generate binaura sounds corresponding to the user viewpoint, we expanded the PSD-estimation-in-beamspace method as an anguar region-wise source enhancement method. By convoving enhanced ocaized and HRTFs corresponding to a user s head motion, binaura sounds were generated in rea time. Through subjective tests using our omnidirectiona video viewing system, we confirmed that sound ocaization corresponding to the user viewpoint coud be obtained. Some issues remain for future work such as investigation of (i) the reationships between sound ocaization and anguar width to be separated and (ii) the effect of HRTF individuaity. 6. ACKNOWLEDGEMENTS We woud ike to express our gratitude to Shinnosuke Iwaki (DWANGO Co. Ltd.) for providing the omnidirectiona image viewer appication. We speciay thank Kazuya Takeda (Nagoya University) and Takanori Nishino (Mie University) for aowing us to use the HRTF database. We received generous support from Daisuke Ochi, Akio Kameda, and Yutaka Kunita (NTT Media Inteigence Laboratories) for video coding and content recording. W 2855

5 7. REFERENCES [1] K. Niwa, T. Nishino, and K. Takeda, Encoding arge array into a 3D sound fied representation for seective istening point audio based on bind source separation, in Proc. ICASSP2008, pp , [2] N. Kamado, H. Nawata, H. Saruwatari, K. Shikano, and T. Nomura, Interactive controer for audio object ocaization based on spatia representative vector operation, in Proc. IWAENC2010, [3] K. Itoyama, M. Goto, K. Komatani, T. Ogata, and H. Okuno, Instrument equaizer for query-by-exampe retrieva: improving sound source separation based on integrated harmonic and inharmonic modes, in Proc. ISMIR 2008, pp , [4] T. Fujii and M. Tanimoto, Free-viewpoint TV system based on the ray space representation, in Proc. SPIE ITCom, vo , pp , [5] ISO/IEC JTC1/SC29/WG11 (N9168), Juy 2007(Lausanne, Switzerand). [6] A. Be and T. Sejnowski, An information-maximization approach to bind separation and bind deconvoution, in Proc. Neura Computation, vo. 7, pp , [7] S. Choi, S. Amari, A. Cichocki, and R. Liu, Natura gradient earning with a nonhoonomic constraint for bind deconvoution of mutipe channes, in Proc. Internationa workshop on ICA and BSS, pp , [8] P. Smaragdis, Bind separation of convoved mixtures in the frequency domain, in Proc. Neurocomputing, vo. 22, pp , [9] H. Saruwatari, S. Kurita, K. Takeda, F. Itakura, T. Nishikawa, and K. Shikano, Bind source separation combining independent component anaysis and beamforming, EURASIP J.Appied Sig. Proc., pp , [10] F. Asano, S. Ikeda, M. Ogawa, H. Asoh, and N. Kitawaki, Combined approach of array processing and independent component anaysis for bind separation of acoustic, IEEE Trans. on Speech and Audio Proc., vo. 11, no. 3, pp , [11] H. Sawada, S. Araki, R. Mukai, and S. Makino, Soving the permutation probem of frequency-domain BSS when spatia aiasing occurs with wide sensor spacing, in Proc. ICASSP 2006, vo. V, pp , [12] J. Bauert, Spatia hearing (revised ed.), [13] Y. Hioka, K. Furuya, K. Kobayashi, K. Niwa, and Y. Haneda, Underdetermined sound source separation using power spectrum density estimated by combination of directivity gain, IEEE Trans. on Audio, Speech, and Language Proc., vo. 21, pp , [14] K. Niwa, Y. Hioka, and K. Kobayashi, Post fiter design for speech enhancement in various noisy environments, in Proc. IWAENC 2014, pp , [15] Nagoya University, Head Reated Transfer Functions Database, [16] D. Ochi, S. Iwaki, Y. Kunita, J. Hirose, K. Fujii, and A. Kojima, HMD viewing spherica streaming system, in Proc. ACM MM 2014, [17] D. Ochi, S. Iwaki, A. Kameda, Y. Kunita, and A. Kojima, Live streaming system for omnidirectiona video, in Proc. IEEE VR 2015, [18] D. Ochi, K. Niwa, A. Kameda, Y. Kunita, and A. Kojima, Dive into remote events: omnidirectiona video streaming with acoustic immersion, in Proc. 23rd ACM Mutimedia, pp ,