Audio Engineering Society. Convention Paper. Presented at the 116th Convention 2004 May 8 11 Berlin, Germany

Transcription

1 Audio Engineering Society Convention Paper Presented at the 116th Convention 2004 May 8 11 Berlin, Germany This convention paper has been reproduced from the author's advance manuscript, without editing, corrections, or consideration by the Review Board. The AES takes no responsibility for the contents. Additional papers may be obtained by sending request and remittance to Audio Engineering Society, 60 East 42 nd Street, New York, New York , USA; also see All rights reserved. Reproduction of this paper, or any portion thereof, is not permitted without direct permission from the Journal of the Audio Engineering Society. Advanced 3D Audio Algorithms by a Flexible, Low Level Application Programming Interface Aleksandar Simeonov 1, Giorgio Zoia 1, Robert Lluis-Garcia 1, and Daniel Mlynek 1 1 Signal Processing Institute, EPFL, Lausanne, CH-1015, Switzerland ABSTRACT The constantly increasing demand for a better quality in sound and video for multimedia content and virtual reality compels the implementation of more and more sophisticated 3D audio models in authoring and playback tools. A very careful and systematic analysis of the best available development libraries in this area was carried out, considering different Application Programming Interfaces, their features, extensibility, and portability among each other. The results show that it is often difficult to find a tradeoff between flexibility, efficiency, quality and speed. In this paper we propose a low level, modular DSP library, which can be used to implement advanced 3D audio models; it is based on reconfigurable primitive methods required by most 3D algorithms and it provides fast development and good flexibility. 1. INTRODUCTION The last generations of multimedia applications are characterized by the constantly increasing demand for a better quality in sound and video for enhanced and virtual reality. Some advanced 3D audio models, proposed in the past through their implementation in well-known and notable tools (e.g. DIVA [1] and Spat [2]) recently became standard interfaces for high quality 3D audio (MPEG-4 [3]). However, due to the high complexity of many 3D rendering algorithms (which often increases linearly with the number of audio sources to be processed), it is still quite hard to build real-time systems without affecting considerably the resources of the computing devices. The constant progress in hardware technologies made the real-time implementation of these methods quite feasible, and opened the doors to the development of new, even more complex, 3D algorithms and their implementation in recent multimedia applications and tools. However new kinds of requirements are always appearing: on one hand, the distributed nature of these new generations of systems, where the different authoring/processing components of the multimedia

2 system chains are possibly located in different sites and integrated with cross-media contents, requires the development and implementation of robust, low latency synchronization algorithms. On the other hand, the need of possibly fast and easy integration of the advanced 3D audio algorithms in multimedia applications implies very high, flexibility and configurability of the available development tools. 2. ANALYSIS OF CURRENT 3D AUDIO SOLUTIONS Having in mind all the above mentioned requirements, we performed a thorough analysis of some of the available development frameworks (libraries and APIs), offering 3D audio functionality. During this analysis, also their extensibility has been tested, adding to some of them new methods required by novel and advanced 3D audio algorithms [6]. This resulted, in some cases, in extended APIs and libraries, which have been tested (among other things) for the implementation of MPEG- 4 3D Audio models, geometrical and perceptual. The next subsections briefly describe the APIs and frameworks that we ve explored, moving from the most popular and consumer oriented ones to higher quality and professional DirectSound and DirectSound3D DirectSound is one of the most popular low level APIs, which has turned into a de-facto standard, supported by almost all manufacturers of PC-based sound boards. DirectSound provides direct access to the sound device and possibly (according to the level of its support) automatic use of hardware acceleration for functionality such as mixing sounds, and automatic conversion of input wave formats to match the output format, hiding existing low level device drivers. It works with buffers, which are its basic unit of work. Secondary buffers are used by applications to play wave data. Each wave occupies one secondary buffer. Primary buffer is the output buffer for DirectSound. DirectSound mixes data from secondary buffers into the one primary buffer, with a determined output format. Static buffers hold sound in memory. Sound can be written to the static buffer in a single operation. Static buffers can be mixed by sound cards. Streaming buffers contain a portion of sound and are useful to play long sounds without storing all in memory. The DirectSound Mixer is responsible for mixing the samples from all of the secondary buffers and performing operations such as volume scaling, panning, and pitch shifting. The DirectSound mixer places most of the load on the system CPU. Figure 1 shows the relationship between the mixer and the primary and secondary buffers. Secondary Buffer Secondary Buffer Secondary Buffer Secondary Buffer Mixer Primary Buffer Figure 1: DirectSound API Mixing Mechanism DirectSound3D provides basic 3D audio algorithms and allows for the hardware acceleration of those algorithms, just like the basic DirectSound. Many sound card companies have various algorithms that perform 3D audio processing and the number of manufacturers providing DirectSound3D accelerated hardware is considerable. However, due to some characteristics of the DirectSound API, it is not easy to use it in true (hard) real-time interactive applications. Actually, the latency is relatively high. This is due to the mixing model and to the underlying operating system, which require long secondary buffers in order to keep the sound fluent and without gaps. Of course, one can always write directly to the primary buffer, but in this case the built-in 3D capabilities cannot be used. Another problem is that the mixing of the secondary buffers in the DirectSound mixer cannot be controlled when the number of channels of the secondary buffers is different than that of the primary buffer. Finally, DirectSound3D is not platform independent, in the sense that its implementation is tied to the Windows operating system, and to soundboards that support it. Page 2 of 9

3 Moreover, this API is not extensible, and thus not giving any possibility of customization of the 3D room models OpenAL API The Open Audio Library [4] was designed to provide an object oriented open source cross-platform solution for 2D and 3D audio programming. It is licensed under the GNU LGPL, with current implementations supporting Windows, Mac OS, Linux, FreeBSD, OS/2, and BeOS. The OpenAL API is conceived for portability of applications, particularly games and other multimedia applications, in a way similar to OpenGL for graphics. It uses extensions compatible with the IA-SIG 3D Level 1 and Level 2 rendering guidelines [5], [6] to handle sound-source directivity and distance-related attenuation and Doppler effects, as well as environmental effects such as reflections, obstruction, transmission, and reverberation. It also offers the same model for logarithmic attenuation relative to the source-listener distance, parameterized by a roll-off factor and a reference distance as in DirectSound3D API. The three basic objects classes of the OpenAL API are the Listener, sound Sources, and their corresponding Buffers. Each Source has a queue of Buffers, which are processed sequentially. Moreover, some of the Buffers may be declared as shared for several Sources, thus simplifying the control of the audio sources in a 3D audio scene. The user application should handle the buffer management and queue them to the appropriate Source Buffer queue. The library then mixes and plays the buffers automatically. The biggest advantage of OpenAL is that it is independent from the used operating system. It can be deployed on virtually any soundcard usable in the supported platforms, though of course its potential will be most fully realized on cards with multichannel audio output. The API is a relatively high-level interface that provides a communication protocol with the sound card driver. Of course, adding such an additional layer over the platform s sound interface considerably increases the overall latency ASIO The Audio Streaming Input/Output interface [5] is a platform independent driver. The idea of creating this low level API came from the necessity to implement an interface to overcome the two main disadvantages of other audio APIs like the two described earlier - the relatively high latency, and the difficulty to provide a good synchronization between the audio input and output. Both features are critical for true real-time media applications; the high latency and the not so well synchronized audio input are even more important when user interaction is requested and added. Being a rather low-level interface, ASIO provides no 3D audio functionality, but just the necessary functions and callback mechanisms to handle two-layer buffers: one for the audio input, and one for the audio output. While the user application writes to the output buffer 0, the hardware driver is supposed to write to input buffer 1; after that, starting from the following frame, the buffers are switched to output buffer 1 and input buffer 0 respectively. The length of the buffers could be as low as 50 ms, which makes this API suitable for many realtime audio applications VS3 API The VS3 API is an object oriented development framework that has been developed to help customer applications to interface a professional audio DSP board [8], which permits a considerable number of low latency, hardware accelerated DSP algorithms. From a programmer s point of view the VS3 API is like a digital audio mixing console with a programmable number of inputs, outputs and summing busses. The interface comprises the following main objects: The Manager is the main object that is responsible for creating Studios and Effects; it handles all physical audio inputs/outputs, and runs time code synchronization objects. Studios are the objects that are responsible for the actual audio playback by processing audio data by a mixer connecting through Manager s inputs and outputs. They are also responsible for plugging and unplugging the available Effects. TimeCodes are synchronization signal generating objects that can drive Managers and Studios. Sources and Targets are the objects generating or consuming sound samples, while the pluggable Effects implement all EQs, compressors, expanders, delay lines, or any other custom DSP algorithm. Page 3 of 9

4 2.5. First Conclusions The analysis of the frameworks described in the previous section (including work done to extend some of the APIs [8]) revealed some important drawbacks. First of all, it is in some cases necessary to develop completely new code to achieve the desired results, due to a lack of flexibility in the algorithm configurations or to unavailable functionality; if the integration of different solutions and APIs is pursued, running at different hardware and software levels, this causes some problems with the synchronization between them, since high level APIs often introduce high latencies. The result is in some cases the system instability with consequent degraded quality, or even loss, of audio content for networked scenarios. On the other hand, if an API is extended through new models and algorithms, the source code of this extended API s is not easily portable to another platform without putting significant effort in it, since the additional code need to be developed and/or integrated and compiled from scratch. In the end, it was noted that if the APIs for 3D sound are higher level and developer-friendly they normally bring with them high latencies, sometimes rigid 3D model solutions or high portability costs. If on the other hand they are lower-level solutions conceived for platformindependent low latency development and true real-time performance, the development of existing and new 3D models may result long if good and robust source code is not available. Of course this is valid especially in the case of the multimedia and communication domains, where the implementer is often not free to choose the preferred model(s) but instead they are agreed with the corresponding parameter sets and the compatibility must be assured. Our analysis and research has been done under assumption that in the near future standardized models will increase and they will need to be implemented on a large range of different platforms (computers, set-top boxes, portable devices, etc.) with strong requirements on portability, scalability and short development times. Table 1 summarizes the comparison results that came out of our analysis: Feature DirectSound3D OpenAL VS3 ASIO Extensibility No Yes Yes Yes Latency High (> 100ms) High (>150ms) Low (depends on PCI bus load, typically < 20 ms) Low, depends on buffer size, could be < 20 ms 3D Audio Yes Yes No No Yes, supported by Hardware Yes, dedicated DSP most of the recent Not available Yes acceleration hardware audio boards Portability No, supported only by Windows OS Yes 3. STRUCTURE OF THE PROPOSED LOW LEVEL 3D AUDIO API As mentioned above, the analysis work has been completed without finding an ideal solution able to offer a good tradeoff among flexibility, efficiency, quality and speed; in fact, when the developer focuses on one 3D Audio model or this latter can be chosen with a good degree of freedom, valuable development tools are available for different levels of experience, knowledge, requirements in terms of development time and performance. Our starting point was someway different, since it focused on media or communication engineers Table 1: API comparison table No, implemented only for Windows platforms Yes dealing with development of complex multimedia or cross-media applications with standardized or precise models and important requirements on scalability, complexity, synchronization (especially for the case of multiple different devices in the target platform). From this point of view, which is gaining importance in the last years, an easy to use but non configurable model is not acceptable and a too low level approach may require a considerable implementation effort and knowledge of the 3D Audio domain. A new approach has been then conceived to develop an independent middle-to-low level DSP library, which could provide a mid-layer development tool to easily Page 4 of 9

5 configure new 3D algorithms in a precise way, simplifying at the same time the implementation task. The input elements to the conception of this approach are mainly two. On one side it is important to consider the low-level software/hardware modules that are generally available since they represent rather general purpose methods in the field of signal processing and media streams; on the other side it is fundamental to leave to the developer enough flexibility and reconfigurability so that the modules he may have at disposal allow to implement with the required precision and quality the models in a way that they can be easily adapted to a wide range of different platforms and qualities in a short time. These two elements have been applied to all the 3D Audio models and algorithms that were considered during our preliminary investigation work, in order to identify different layers of basic and fundamental macro instructions common to different cases and whose combination can allow a fast realization of existing and new 3D audio methodologies. Finally an object-oriented, two-level hierarchical structure has been adopted for the library. As a root of the hierarchy, some common properties for all primitives have been defined (e.g. number of input/output channels, single sample and blocks processing functions, etc.) and assigned to an object, common for all the DSP primitives (Parent Node). The first level (Atomic Level) contains the atomic DSP primitives, i.e. primitives that cannot be built by the others: multiplier, adder, delay line, etc. Ideally, this layer corresponds to an instruction set and memory structures of a virtual machine designed for signal processing, possibly in a way that these objects and methods match a physical device as far as possible; this permits to easily implement this layer based on the instruction set of a device, exploiting when possible a processing by blocks of samples. At this layer fast lowlevel interfaces like ASIO and VS3 may be used for some basic functionality. The second level (Basic DSP Level) contains generic DSP primitives, e.g. low-pass filter, all-pass, nested allpass, double nested all-pass, generic FIR and IIR filters, geometric and trigonometric operators (on vectors), FDN Matrices [10], etc. This level is build by primitives previously mentioned in the Atomic Level, and it ideally corresponds to the instruction set and memory structures of a virtual device built on top of the primitive one to support 3D Audio specific functionality. It is exactly at this level that the performed analysis and profiling of the 3D Audio models has been exploited, according to the input elements described earlier. It is still present at this layer an effort to match commonly available DSP libraries for physical devices, so that these DSP primitives might be as far as possible available in an optimized way through normal software development kits. But in addition, additional functionality and flexibility are required that characterize 3D Audio models in particular. This additional functionality represents the added value to a low-level API or driver (such as ASIO or VS3) that provides to the developer ad-hoc methods to easily implement and configure 3D Audio models without at the same time loosing contact with the low level configuration of the device I/Os, of the synchronization and of some important quality aspects. This is possible since the Basic DSP Level still interfaces to the Atomic Level. As test application, a 3D Audio layer has been implemented containing 3D algorithms compatible first with the standardized MPEG-4 room models (mentioned in section 1) and in addition with several other algorithms like: Schroeder reverberator, Dattorro network reverberator, Gardner reverberator, etc. New algorithms could be easily added and configured, using the already available DSP building blocks. The library, and the test 3D Audio layer, have been implemented in C++ (that assures the availability of SDKs for most platforms) and successfully ported, optimized and run on two different hardware platforms (Pentium IV running Windows OS and embedded Trimedia processor) with reduced effort. The implementation and optimization of the library require of course to reduce the purely object-oriented approach with its typical overhead, so that the most computationally intensive parts can exploit for instance the architecture parallelism (some specific optimizations of the source code have been introduced in order to improve the overall performance of the library especially for the Trimedia processor). This leads to some changes in the initial structure of the API. For example, as well-known execution speed very often relies on processing by blocks; but this is not possible when methods that are not primitives include some form of intrinsic feedback [11]. In our case, for instance, the relatively complex iterative structure of the of the FDN matrix structure may introduce too much overhead because of the many function calls (in case of being realized by Atomic primitives, as it was initially intended, that would have to be used in a sample by Page 5 of 9

6 sample way). To overcome this and other smaller problems, it was at some time decided to decrease the software abstraction and encapsulation level, and implement e.g. the FDN as a single DSP primitive directly including the necessary Atomic methods. Parent Node Atomic Level Delay Line, Summer, multiplier, FDN Basic DSP Level low-pass filter, all-pass filter, nested, double nested AP, FIR, IIR, etc. Advanced 3D Audio Layer Schroeder Reverb, Dattorro Reverb, Gardner Reverb, Geometrical Spat, Perceptual Spat Figure 2: The 3D Audio API structure 4. IMPLEMENTATION AND TEST OF THE 3D AUDIO LIBRARY The designed interface and implemented library have been tested in different applications, partially developed within the scope of the MAP (MPEG-4 Audio Platform) project. MAP is a Swiss project, run in collaboration between the authors Laboratory and Merging Technologies SA. The main scientific and technical objective of the project MAP is twofold: to consolidate and enhance software/hardware existing tools at both industrial (Mykerinos board, Pyramix Virtual Studio) and experimental (MPEG-4 AAC, SA codecs) level to provide mutual effective integration, and at the same time to add and/or improve 3D Audio capabilities in professional market products. to implement an MPEG-4 Audio Systems decoder by means of mixed general-purpose/fpga/multimedia- DSP technology to obtain true real-time performance and associated standard user interaction. For both objectives the developed API has been exploited to easily include custom or standardized 3D Audio models into existing applications. For the first objective 3D object-oriented Audio capability has been integrated into a professional sound production toolbox, providing at the same time the possibility to monitor locally multi-channel output and to generate/encode object-oriented audio content (compatible to MPEG-4 BIFS format). Working by objects is more intuitive in many situations and by this approach the content can be coded in a platform-independent way by means of appropriate 3D reconstruction models [12, 13]. For the second objective, MAP provided the implementation of two different 3D audio rendering configurations. Both are based on the development of an MPEG-4 compatible Audio and Systems player [9] partially developed in the context of the European project CARROUSO. The first configuration includes a considerable part of the proposed library/api (all the necessary for the MPEG-4 3D Audio models) running in the Mykerinos board (based on the TriMedia multimedia processor) and exploiting at the low-level the VS3 API; in the second configuration the same MPEG-4 compatible player is running with the library completely compiled on a conventional Pentium based workstation. In the first case the practical result is a complete MPEG- 4 player, including video, graphic functionality and interaction, which exploits the Mykerinos board as hardware acceleration engine for what concerns 3D audio spatialization and rendering; output is available as stereo output as well as 4.1 surround format. The block diagram is shown in Figure 3. The media (MPEG-4) bitstream and objectbased description decoding is performed in the host platform (general-purpose workstation), including the task to decode the video and audio sequences by the appropriate tools (in the case of audio this can be for Page 6 of 9

7 example: AAC, MP3, PCM, SA, etc.), and attached to the scene. Next, the decoded audio samples together with the spatialization parameters described by the standardized room models, are sent to the 3D Audio library (Mykerinos platform), which performs the 3D audio signal processing and rendering. MPEG-4 Bitstream and for maximum compatibility, the DirectSound API is used only as interface to the sound board to output the processed sound. If the platform sound device supports ASIO, then low latency can be obtained for I/O and interaction; moreover output could be configured for multichannel configurations. MPEG-4 Bitstream MPEG-4 DMIF Decoder MPEG-4 DMIF Decoder Video Stream Decoder AudioBIFS Tree Decoder Raw Monophonic Sources AudioBIFS Tree Decoder Video Output Host Platform Advanced 3D Audio Layer DirectSound/ASIO Video Stream Decoder Philips Trimedia Advanced 3D Audio Layer Sound Output Video Output VS3 Mixer Figure 4: The PC Rendering Architecture 5. CONCLUSION AND FUTURE WORK ADAT Output Figure 3: The VS3 Rendering Architecture The second architecture ( Figure 4) is hosted entirely by a Pentium compatible platform. In this case the Advanced 3D Audio Layer is compiled directly in the same MPEG-4 player executable as static library. If nothing else is available, The 3D Audio library has been successfully ported on several different hardware architectures, with maximum source code reusability and minimum efforts for system rebuild without compromising the quality of the audio. The results are promising: in this sense the proposed DSP library may be considered as a significant step towards a wider-range and systematic approach to the problem, since it offers very easy and fast 3D audio reconfigurability without affecting the low latency that is needed for professional audio applications. Now, given that we have already a reference platform for the development and implementation of 3D algorithms, we are ready to explore new directions and Page 7 of 9

8 application areas. The new audio and media technology being introduced by MAP can be furthermore successfully brought to market as soon as the acceptance of MPEG-4 as a hybrid coding scheme (media and BIFS scene description for higher level representations) is broadening; on the other hand, in the professional sector, sound authoring and production applications for cinemas, electronic cinemas, theaters and so on can be envisaged, as soon as the new format is spreading and generating demand. A step-by-step introduction into the professional market should be planned in such a way that products will remain compatible to existing sound and surround sound formats. For instance, it will be possible to save and deliver media and project information in the usual way or possibly to save and stream MPEG-4 compatible information using its standardized format. In this way the authored content will continue to be compatible with all existing tools as today, adding new functionality and universal compatibility to compliant multimedia player first, and to possible future extensions and plug-ins for professional toolsets later. In this sense we think that the implementation of an authoring tool, offering such kind of functionality, is indispensable. 6. ACKNOWLEDGEMENTS Most of the research work described in this paper has been done within the framework of the project p MAP, funded by the Swiss Federal Office of Science and Education (CTI). There are several persons that we re particularly grateful to Mr. Claude Cellier, CEO of Merging Technologies, Mr. Bertrand Van Kempen and Mr. Patrick Bovay our industrial counterpart team during the MAP project. Additional thanks to Mr. Ruohua Steve Zhou with which we shared some test phases of our work. 7. REFERENCES [1] L. Savioja, T. Lokki,, J. Huopaniemi, Auralization applying the parametric room acoustic modeling technique - The DIVA auralization system, Proc. 8th International Conference on Auditory Display (ICAD2002), Kyoto, Japan, pp , July 2-5, [2] Jean-Marc Jot, Olivier Warusfel, Spat~ : A Spatial Processor for Musicians and Sound Engineers, CIARM 95, Ferrara (Italy) 1995 [3] ISO/IEC JTC1/SC29/WG11 Information Technology - Coding of Audio-Visual objects. Part 1: Systems. [4] Creative Labs Developer Web Site, K.html [5] 3D Audio Rendering Guidelines Level 1, IA-SIG, June, 1998 [6] 3D Audio Rendering Guidelines Level 2, IA-SIG, September, 1999 [7] Steinberg Professional Systems, ASIO2 SDK Specification, o_sdk/. [8] Merging Technologies home page at [9] A. Simeonov, G. Zoia, R. Garcia, Rendering of Advanced 3D Room Models by Enhanced Application Programming Interfaces, AES-114, Amsterdam, March, 2003 [10] D. Rocchesso, J. O. Smith. Circulant and Elliptic Feedback Delay Networks for Artificial Reverberation, IEEE Transactions on Speech and Audio Processing, 5(1):51-63, January, [11] G. Zoia, C. Alberti. A Virtual DSP Architecture from a Complexity Analysis of MPEG-4 Structured Audio, IEEE Transactions on Multimedia, 5(3): , September [12] G. Zoia. Room Models and Object-Oriented Audio Coding: Advantages and Applications, Proceedings of the DAGA 03 Conference on Acoustics, Aachen, Germany, March [13] J. Plogsties, O. Baum, B. Grill. Conveying Spatial Sound Using MPEG-4, Proceedings of the AES Page 8 of 9

9 24 th International Conference on Multichannel Audio, Banff, Canada, June Page 9 of 9