Investigation on the Quality of 3D Sound Reproduction

Transcription

1 Investigation on the Quality of 3D Sound Reproduction A. Silzle 1, S. George 1, E.A.P. Habets 1, T. Bachmann 1 1 Fraunhofer Institute for Integrated Circuits IIS, Erlangen, Germany, andreas.silzle@iis.fraunhofer.de Abstract Standard audio reproduction formats assume that the loudspeakers are positioned in the horizontal plane. The last missing dimension height offers many opportunities and challenges. In this paper we focus on one challenge, i.e., the evaluation of reproduction systems with additional height channels. First, the results of an expert survey regarding the importance of 3D sound for different applications are presented. Secondly, the overall quality of 3D sound reproduction is evaluated by listening tests. Because the selection of sound items is critical for the evaluation a broad selection of different sound items is used. To classify sound items, we extend an existing 2D sound categorisation concept to 3D. The test signals were derived from original 22-channel sound items. The 2-, 5-, and 9-channel sound items were generated by passive down-mixing algorithms and selective muting of reproduction channels. In addition, we investigate the influence of an external reference. Therefore, one listening test with a reference and a second one without a reference were conducted. A companion paper focuses on controlling physical relevant parameters of a 3D audio setup. 1. Introduction The current standard for a five channel reproduction setup [1] was defined in The requirements to improve two channel stereo reproduction, as well as related investigations, are summarized by Theile in [2] and [3]. He also provided good insight into the requirements for future recording/production setups using height loudspeakers [4]. Because it is not possible to reproduce the physical wavefield of the recording room in three dimensions using for example 9 or 22 loudspeakers, it is important to differentiate between the perceptual and the physical domain. Important are the relations between physical sound field elements (quality elements) and perceptual attributes (quality features), as summarized in Table 1. An in-depth investigation of these relations for auditory virtual environments in the context of binaural reproduction for different applications is given in [5]. Quality Features Direct sound Quality Elements Early reflections Reverberation Direction, elevation Distance, depth Reverberance Atmo Envelopement Timbre Table 1: Relation between quality elements and quality features for 3D sound field, after [4]. The number of black dots indicates the importance. The requirements in the physical domain for a good 3D sound reproduction and their evaluation is described in the companion paper [6]. The content of this paper is organised as follows. To investigate the importance of 3D sound for different applications a survey was conducted among experts. Section 2 reports the results of the survey. Followed by this, details of two different listening tests conducted in the context of 3D sound reproduction are presented. Section 3 discusses the selection of audio items. Section 4 specifies the selection of the different reproduction conditions. Finally, the results of these 3D sound listening tests are given in Section Applications for 3D Sound Results of an Expert Survey Humans judge the quality of e.g. a sound reproduction system by measuring the difference between their perception and their expectation, [7]. The expectation of a sound reproduction system is application-dependent. Clearly, the expectation of the sound reproduction of a mobile phone is different compared to the one of a home cinema system. It is therefore important to know for which kind of application the sound reproduction system should be designed. For the definition of a 3D audio system with height reproduction, the combination of the reproduced content and the target application will have a strong influence on the quality judgement. Figure 1 presents the sorted results of an expert survey regarding this question. The answers were collected among 29 international developers of 3D sound reproduction systems and sound engineers working in this area. The results indicate that news presented in radio or TV doesn t require currently 3D audio reproduction. Conversely, nearly all of the international experts agree that 3D (video) cinema, (standard) cinema and movie reproduction from TV/DVD will gain a lot from 3D audio reproduction. The participants of the survey were given the opportunity to provide a wish list for such a future reproduction system. A summary is given below. Proceedings of ICSA

2 Figure 1: Sorted results of the expert survey on applications for 3D audio reproduction. Presented are median (red line in the middle of the box) and quartile values (upper and lower border of the box), red crosses mark outliers. Item Name Counterpoint Donut Hakodate Jet Toccata A Toccata D Waves Length [s] Producer Theegarten Bachmann Hamasaki Hamasaki Hamasaki Hamasaki Hamasaki Composer St. Reich Donuts N/A N/A J.S.Bach J.S.Bach N/A Source IIS IIS NHK NHK NHK NHK NHK Genre Experimental music Rock music Atmo Effect Classical music Classical music Atmo Classification F-F/F-F F-F/B-B F-B/F-B F-F/F-F B-B/F-B B-B/F-B F-F/F-F Table 2: 22-channel items used in the tests, classifications explained in Section 3 Expert wish list for a future 3D sound reproduction system: No coloration, precise localisation, precise vertical localisation for the frontal area Reproduction of distance Significant improvement to the existing system Quality improvement should scale with the number of loudspeakers Large sweet spot Scalability, backward compatibility Should work for different loudspeaker set-ups Should not be critical regarding loudspeaker positions Good effort to benefit ratio Keep it simple, stupid Not too expensive for home use Acceptable setup in a living room The average number of desired loudspeakers was Selection of Audio Items for 3D Sound Evaluation The selection of audio items has a significant influence on the results of a listening test. It is important to have a good selection of critical and representative material related to the research question. At the moment, the number of 3D sound items is quite limited due to the lack of production tools and availability of the reproduction setups. For this test a selection of around 25 items in different formats such as 5+4 (5 loudspeaker in the horizontal plane and 4 elevated loudspeakers), 7+5 and 22 channels are available. More details regarding the different formats can be found in [6]. Seven selected items with a broad categorisation of different genres are shown in Table 2. In a 3D extension to the classification of spatial characteristics representing basic audio scenes, given in [8], the following classes are introduced: F foreground content (with direct sound sources) B background content (ambient sound) These two classes are specified for four main directions: 1. Front horizontal 2. Back horizontal 3. Front height 4. Back height With this classification, a recording of an organ in a church gets the characterisation: B-B/F-B, where the organ as a elevated direct source is front and the rest around is ambient Proceedings of ICSA

3 sound, like the two organ recordings (Toccata A and Toccata D) listed in Table 2. An experimental music item with moving sound sources through all channels is categorised under F-F/F-F e.g., Counterpoint in Table Design of Listening Tests Hamasaki conducted a number of experiments [9] to evaluate the quality of 22-channel sound system. In one of their tests, they used the semantic difference scale technique where listeners evaluated 29 low-level attributes of audio quality. The conclusion from the principle component analysis was that a 22-channel sound system provides clearly distinguishable quality difference compared to 5- and 2- channel systems. presented conditions, like in this case. For an analysis with more detailed quality features more experience among the subjects is necessary. The graphical user interface used in Experiment 1 is shown in Figure 2. In recent years, many manufacturers, for e.g., Galaxy studios (Auro 3D) [10, 11] and Samsung (7+3) [12] have proposed systems with fewer number of channels. Therefore, we intend to analyse the preferences and the quality differences of systems that use reproduction channels between 2 and 22. Recently, Kim et. al. [12] conducted experiments to evaluate directional quality and overall quality of a 22-channel sound system and some systems with a smaller number of reproduction channels. They concluded that the perceptual quality of a recently proposed 7+3 system is similar to that of a 22-channel system. The sound scene of the three program materials used in their study were mixed on a 22- channel sound system. They have also included two program materials produced using the Directional Audio Coding (DirAC) algorithm. The sound scenes of their program materials were focused mainly on the top layer of the system. A number of down-mixes were created for different loudspeaker set-ups. The test methodology used in their study was ITU-R BS Recommendation (MUSHRA) [13]. In a MUSHRA listening test, listeners always compare the test signals with a reference signal. Therefore results from a MUSHRA listening test provide the difference between the reference and other test signals. The reference, which is repeated in the test as a hidden reference, has to be rated with the highest score of 100. All other scores are relative to this. With this given reference, the MUSHRA test gives reliable and repeatable test results. It was developed for the evaluation of audio codecs. For these tests the original (the unprocessed audio signal) is the reference signal. Figure 2: User interface for Experiment 1 without reference 4.2 Experiment 2 To analyse the influence of the usage of the reference signal, Experiment 2 was a multi-stimulus test with the reference for comparison, i.e., the standard MUSHRA test. The reference signal was the original 22-channel mixes of the sound items. In a MUSHRA test, it is recommended to include a low anchor signal filtered using a low-pass filter with cut-off frequency at 3.5 khz. In ITU-R BS.1534 recommendation, it is stated that other types of anchors showing similar types of impairments as the system under test can be used. Here a standard 2-channel stereo downmix was used as a low anchor. The graphical user interface used in Experiment 2 is shown in Figure Experiment 1 For Experiment 1, it was the intention to measure the absolute preference of the different conditions. The judgement about the Overall Sound Quality is only aligned to the verbal anchors of the scale from Bad to Excellent not to a reference signal. The listeners were asked to make their ratings on a continuous 100-point grading scale. The labels used in the grading scale were identical to that in ITU-R BS recommendation. The listeners can decide freely about their preferences. Additionally, a multi-stimulus test gives the listeners the opportunity to compare directly between the different conditions. This is an advantage, when the listeners do not have so much experience with the Figure 3: User interface for Experiment 2 with reference Proceedings of ICSA

4 Figure 4: Active loudspeakers in a 22-channel sound system Figure 5: Active loudspeakers and signal flow of condition M5+4 (see Table 3 for coefficients used) Figure 6: Active loudspeakers and signal flow of condition D5+4 (see Table 4 for coefficients used) Figure 7: Active loudspeakers and signal flow of condition M5 (see Table 5 for coefficients used) Figure 8: Active loudspeakers and signal flow of condition D5 (see Table 6 for coefficients used) Figure 9: Active loudspeakers and signal flow of condition D2 (see Table 7 for coefficients used) Proceedings of ICSA

5 4.3 Test Conditions Two types of test conditions were used in the listening tests: passive down-mixing algorithms and selective muting of reproduction channels. The motivation behind selecting mutes and down-mixing algorithms as conditions is described in the companion paper (see [6] Section 6). The Auro 3D (5+4) format was chosen because of its popularity in the film industry. 5-channel and 2-channel stereo formats were chosen in order to get a comparison to these standards. The conditions are graphically illustrated in Figure 4 to Figure 9. The coefficients used for the conditions are provided in Table 3 to Table 7. Two expert listeners carefully aligned all conditions of each item in loudness. The listening tests were conducted in the standard listening room that conforms to ITU-R BS Recommendation. The user interfaces for the listening tests were developed using Pure Data (Pd Extended ). Some considerations while developing a software tool for listening test are given in [6]. The software tool used for the listening test had the capability to switch instantly between the different test conditions. The signals were played back over the loudspeakers mounted on the rings. As described in our companion paper [6], loudspeaker corrections were applied. FLc = FLc FRc = FRc FC = FC LFE = LFE1 LS = BL RS = BR TpFL = TpFL TpFR = TpFR TpLS = TpBL TpRS = TpBR Table 3: Coefficients for 22 to M5+4 FLc = FL (FLc + SiL) + BtFL FRc = FR (FRc+SiR) + BtFR FC = FC (FLc+FRc) + BtFC LFE = (LFE1+LFE2) LS = BL (SiL + BC) RS = BR (SiR+BC) TpFL = TpFL (TpSiL + TUFC) +0.5TpC TpFR = TpFR (TpSiR + TpFC) +0.5TpC TpLS = (TpBC+TpSiL) + TpBL + 0.5TpC TpRS = (TpBC+TpSiR) + TpBR + 0.5TpC FLc = FLc FRc = FRc FC = FC LFE = LFE1 LS = BL RS = BR Table 4: Down-mix coefficients for 22 to D5+4 Table 5: Coefficients for 22 to M5 FLc = FL (FLc + SiL) + BtFL + TpFL (TpSiL) FRc = FR (FRc + SiR) + BtFR + TpFR (TpSiR) FC = FC (FLc + FRc) + BtFC + TpFC (TpC) LFE = (LFE1+LFE2) LS = BL (SiL + BC) + TpBL (TpSiL) + 0.5(TpC) TpBC RS = BR (SiR + BC) + TpBR (TpSiR) + 0.5(TpC) TpBC Table 6: Down-mix coefficients for 22 to D5 FL = FL FC LS LFE FR = FR FC RS LFE Table 7: Down-mix coefficients for 22 to D2. 22-channel signal is first down-mixed to D5 using coefficients in Table 6 and then the coefficients above are applied. 4.4 Procedure of Listening Tests In both listening tests, 17 listeners participated. The listeners were experienced in sound quality evaluation, but not specifically in 3D sound quality evaluation. Every listener completed Experiment 1 before they started Experiment 2. This was done intentionally such that the reference signals used in Experiment 2 did not bias the listener when taking part in Experiment 1. Written instructions for the tests were given to the subjects. Before the first experiment, each listener had to take part in familiarisation and training sessions. In the familiarisation session, the listeners were given three 22-channel sound items to get familiarised with the new type of sound system. In the training session, listeners have done a short listening test, to get familiarised with the graphical user interface and grading scale. After the training session, the blind grading phase started and on an average the tests lasted for 45 minutes to 1 hour. The listeners were instructed to take a mandatory break of 10 minutes if they felt that the test takes longer than 30 minutes. Since the same group of listeners participated in Experiment 2, the listeners were not given a second familiarisation, but a short training phase to get used to the reference signal. The listeners were also given a suggestion that they could move their head vertically in addition to the horizontal movements. 5. Results of Listening Tests 5.1 Post-Screening Subjects reported that in general, both tests were challenging. Several subjects mentioned that they were not familiar with the new 3D reproduction system and the sound items. For Experiment 1, they distinguished between the different conditions, but it was not always clear for them how to judge them. Some subjects reported that the level of difficulty reduced when they progressed themselves during the listening tests. For these reasons no listeners were postscreened. To evaluate the performance of the listeners some analysis was done as described below. For Experiment 1, the repeated test scores of an item (ToccataD and ToccataD2 in Figure 10) were used for the Proceedings of ICSA

6 analysis. The absolute differences between the first and second trials were calculated. The average of the differences across the listeners for Experiment 1 was 10.1 and for Experiment 2 was 8.2. For Experiment 2, additional calculations were conducted for evaluating the reliability of listeners for identifying the hidden references. It was found that some listeners found it difficult to identify the hidden references for few program items, see Figure Results of Experiment 1 Mean opinion scores of overall sound quality with 95% confidence intervals are plotted in Figure 10. From the figure it can be seen that the 22-channel conditions (original) have a clearly distinguishable preference among the listeners when all items are considered. They are categorised between good and excellent (78 points) on the grading scale. The next preferred conditions are M5+4 and D5+4 (approx. 70 points). They are not distinguishable from each other. They are preferred over conditions with five channels (D5 62 points and M5 56 points). Finally, the standard stereo reproduction D2 is rated in the poor area (34 points) of the scale. The 28 points difference in preference rating between D2 and D5 is nearly double as the distance between D5 and the original 22-channel signals. The intermediate formats with 9 reproduction loudspeakers have only an advantage of 8 points compared to D5. From inspecting the scores of different sound items, it can be seen that some items do not follow the general trend. Examples are Jet and ToccataD. It is clear from the graphs that for these two items condition M5+4 have sometimes graded better than the original signals. Both these items contain predominantly direct signals compared to the other items. Therefore it is difficult to judge them without knowing reference. For example, for the Jet item, the jet planes fly in different directions for different conditions, but it could not be decided which is the original one. Possibly, this could be the reason why listeners gave similar ratings to conditions M5+4 and the original. For the Donut item, it can be seen that D5+4 is rated lower than M5+4. This is a live recording from a festival and has surrounding crowd noise. Because of down-mix problems, the item M5+4 with muted channels is rated higher. For the rest of the items the down-mix problems are not relevant compared to the missing content of the mute conditions. 5.3 Results of Experiment 2 The result of the MUSHRA test with a reference signal is plotted in Figure 11. The original signal is rated with 98 points, meaning that the original was not always recognised as original. The condition D5+4 is rated as the next best with 76 points in the upper good area. Unlike the results in Experiment 1, the confidence intervals of the conditions D5+4 and M5+4 (76 and 68 points) are clearly distinct. The perceptual distance (i.e., the difference in points) of these two conditions to D5 is smaller than that to the original. The condition D5 gets almost the same rating (63 points) as in Experiment 1. The condition D2 also has similar score (30 points) as in Experiment 1. The condition M5 is graded with 50 points. There is a clear difference between both mute versions compared to the down-mix versions with same number of channels in this experiment with an explicit reference. The results in Figure 11 are item-dependent, as in Experiment 1. In general, for all items the same ranking of the conditions is observed as in Experiment 1. The difference increases in the presence of the reference signal. In particular, M5+4 is not equally rated to the original for items ToccataD, Jet and Donut. 5.4 Discussion and Conclusions Since there was no explicit reference given to the listeners in Experiment 1, the results obtained are related to the personal preferences of the listeners. The ranking of conditions in general are inline with the results where the 22-channel reference signals are given, although the perceptual distances between the conditions are larger and clearly distinct. A comparison between the results from Experiment 1 and Experiment 2 for the average values over all items is provided in Figure 12. Interestingly there are nearly no differences between the D2, D5 and M5+4 scores in the two experiments. However there are larger differences for D5+4 and 22-channel original when the reference is given. Only in Experiment 1 without a reference the step size between different conditions can be compared, because we have an absolute judgement for the preference and not a comparison regarding the reference like in Experiment 2. Nevertheless, from the results of both experiments, it can be concluded that there is a preference of the listeners for conditions with height reproduction and that this preference is, to some extent, item-dependent. The perceptual difference between the 5-channel reproduction D5 and the original 22- channels is about half the distance as to the 2-channel stereo reproduction D2 for the experiment without explicit reference. The quality improvement of an intermediate condition D5+4 is not very large. In Figure 12 a virtual condition called MD5+4 is introduced. The corresponding scores for this condition are calculated by choosing the maximum score between conditions D5+4 and M5+4. This condition represents the best possible sound quality using 9 loudspeakers. From the figure it can be seen that there was no significant difference between 22-channel original items and 9-channel conditions in Experiment 1. However, this observation cannot be validated without conducting experiments with active down-mixing algorithms. The sound items represent a good average of the present 3D audio materials. With more experience in 3D production and improved production tools, a larger perceptual difference could probably be achieved. Proceedings of ICSA

7 Figure 10: Mean opinion scores with 95% confidence intervals for Experiment 1. Figure 11: Mean opinion scores with 95% confidence intervals for Experiment Summary and Future Work In this paper, descriptions of listening tests in the context of 3D audio reproduction are given. Two multi-stimulus listening tests are conducted and listeners evaluated overall sound quality. In the first test (Experiment 1), listeners are not given a reference for comparison, whereas in the second test (Experiment 2), the listeners are given a reference for comparison. The reference signals consisted of 22-channels. The other signals used in the tests are derivations of these reference signals, constructed either by passive down-mixing or muting selected channels. These different conditions have 2, 5 and 9 (5+4) reproduction channels. When an explicit reference is given to the listeners the relative differences between the test conditions increased (see Figure 12). General observations from both tests are that the listeners prefer sound with height reproduction and that the results are item-dependent. Figure 12: Comparison between the results of Experiment 1 and Experiment 2 Although the results from Experiment 1 are obtained without a given reference, the personal preference of the listeners is influenced by the other conditions present in the test. Therefore, a pair comparison test could lead to more unbiased Proceedings of ICSA

8 results. Additionally, a multidimensional analysis could provide better insight into the quality differences between different conditions. The improvement by applying an active down-mix will also be explored. Acknowledgements The authors like to thank Oliver Scheuregger, Matthias Lang, Wolfgang Hörlbacher, Julian Popp, Stefan Varga, Kimio Hamasaki, Akio Ando and Jean-Marie Geijsen for their technical support and suggestions during the course of this work. References [1] ITU-R Recommendation-BS (1994). Multichannel stereophonic sound system with and without accompanying picture. Intern. Telecom Union: Geneva, Suisse [2] Theile, G. (1990). Further developments of loudspeaker stereophony. 89th AES Convention, Los Angeles. preprint #2947 [3] Theile, G. (1991). HDTV sound systems: How many channels? 10th AES Conference, London [4] Theile, G. and H. Wittek (2011). Die dritte Dimension für Lautsprecher-Stereofonie. VDT Magazin(02): pp [5] Silzle, A. (2007). Generation of quality taxonomies for auditory virtual environments by means of systematic expert surveys. Doctoral dissertation. Institut für Kommunikationsakustik: Ruhr-Universität Bochum [6] Silzle, A., S. George, and T. Bachmann (2011). Experimental setups for 3D audio listening tests. International Conference on Spatial Audio (ICSA), Detmold, Germany [7] Jekosch, U. (2004). Basic concepts and terms of "quality", reconsidered in the context of product-sound quality. Acustica - acta acustica. 90(6): pp [8] Zielinski, S.K., F. Rumsey, and S. Bech (2002). Subjective audio quality trade-offs in consumer multichannel audio-visual delivery systems. Part I: Effects of high frequency limitation. 112th AES Convention, Munich, Germany. preprint #5562 [9] Hamasaki, K., et al. (2006). Effectiveness of height information for reproducing presence and reality in multichannel audio system. 120th AES Convention, Paris, France. preprint #6679 [10] Wisse, E. (2011). Auro 3D - ein neuer Standard? Ein Interview mit Wilfried van Baelen, Galaxy Studios. VDT Magazin(02). [11] (2011). Auro 3D. [12] Kim, S., Y.W. Lee, and V. Pulkki (2010). New channel vertical surround system (10.2-VSS); Comparison study of perceived audio quality in various multichannel sound systems with height loudspeakers. 129th AES Convention, San Francisco, USA. preprint #8296 [13] ITU-R BS (2003). Method for subjective assessment of intermediate quality level of coding systems. Intern. Telecom Union: Geneva, Switzerland Proceedings of ICSA