Controlling Spatial Sound with Table-top Interface

Transcription

1 Controlling Spatial Sound with Table-top Interface Abstract Interactive table-top interfaces are multimedia devices which allow sharing information visually and aurally among several users. Table-top interfaces are utilized as groupware and are suitable for collaborative work, convenient for a group working on themes related to sound systems. Table-top interfaces for spatial sound environments are starting to be investigated in the field of the human interfaces. A representative table-top musical instrument is the reactable. In this paper, we present a way to control the position of multiple virtual sounds in a spatial sound environment via such a table-top interface. Sound localization is required to discriminate and clearly recognize sounds. We have been investigating musical table-top instruments which are capable of controlling multiple input and output channels in spatial sound environments. One of the main features of this newly developed system is that multiple users can independently control the spatialization of sounds in real-time. Yuya Sasamoto, Michael Cohen, & Julián Villegas Spatial Media Group, University of Aizu Aizu-Wakamatsu, Fukushima ; Japan {m , mcohen, julian}@u-aizu.ac.jp I. INTRODUCTION Table-top interfaces have been studied in the ubiquitous computing community [1][2][3]. Such devices are suitable for groupware as roomware in private spaces or office meetings [4]. Advances in table-top interfaces and input technologies have led to new systems to support co-located collaboration. Technology that supports consultation on a tabletop can take advantage of the rich experience people have with such natural styles of work and play. Table-top collaboration is being explored, including multitouch display with two dimensional gestures for interactions [5] and manipulation of virtual interface elements via affordances on table-top interfaces [6]. Most studies assume a meeting focused on visual information, including documents and images, and do not highlight the appropriateness of a table-top interface for adjusting sounds. Table-top interfaces can be deployed in collaborative activities featuring dynamically adjustable sound. For controlling stereophonic and surround sound, such interaction styles could be more natural to use such an interface than mouse or keyboard. Users can leverage their everyday experience, without depending on traditional computer-enforced idioms, for editing diffusable sound. As a basic idea of research, we focus on how to use a tabletop interface for meetings featuring spatialized sounds. We designed a prototype of table-top interface and arranged eight loudspeakers to experiment with it, as shown in Fig. 1. The system allows multiple users to control and steer independent audio channels. We use position data of tangible objects on table interface for configuration of sound spatialization. By using this table-top interface, users do not need to concentrate on the expression of words and are able to express sounds by moving tangible objects with spatial environment. Hearing spatial sounds in a meeting is promoted to improve mutual understanding. Fig. 1. Table-top interface and eight-loudspeaker array II. BACKGROUND 1) ReacTable: The reactable is an electronic musical instrument with a tangible user interface. It is a synthesizer which can make music from freely moving or rotating objects [7] on a table. Each control object has a designated role, such as looping sound sources, oscillators, and filters. Bringing objects close and virtually connecting them, sound sources can be mixed or effects can be added. Information such as fiducialid,position (x, y), andangle is acquired via markers [8] and tracked by original image processing software called. 2) Tuio: TUIO [9] is an open framework that defines a common protocol and API for tangible multitouch surfaces. As shown in Fig. 2, the protocol allows the transmission of an abstract description of interactive surfaces, including touch gestures and tangible object states. This protocol encodes control data from a tracker application (e.g., based on computer vision) and sends it to any client application capable of decoding the protocol. The TUIO protocol has been implemented using OpenSound Control [10] and is therefore usable on any platform supporting this protocol. There exists a growing number of TUIO-enabled tracker applications and TUIO client libraries for various programming environments, as well as applications that support the protocol. This combination of TUIO trackers, protocol, and client implementations allows rapid development of table-based tangible multitouch interfaces. TUIO has been designed mainly as an abstraction for interactive surfaces, but has also been used in many other related application areas.

2 multi-touch gestures tangible objects loudspeaker n virtual source loudspeaker m y Display Sensor TUIO protocol TUIO client application Fig. 2. TUIO (diagram adapted from Fig. 4. Stereophonic configuration formulated with vectors x 3) : The [11] is an open source, cross-platform, computer vision framework for fast, robust tracking of markers attached to physical objects, as well as for multitouch finger tracking. It was designed mainly as a toolkit for rapid development of table-based (TUI) and multitouch interactive surfaces. The application acquires a video stream from a camera, searches the stream frame by frame for predefined symbols, and sends data about identified symbols via a network socket to a listening application. The application was designed modularly, making it easy to add new image recognition and frame processing components. Fig. 3. Amoeba symbols 4) Fiducial Symbols: In virtual reality applications, fiducial symbols are often deliberately applied to objects in a scene so that the objects can be recognized in images. The symbol set called amoeba (Fig. 3) [8][12] was specifically developed for the reactable. In the fiducial engine the source image frame is first converted to a black and white image with an adaptive threshold algorithm. This image is then segmented into a region adjacency graph reflecting the containment structure of alternatingly nested black and white regions. This graph is searched for unique tree structures encoded into the fiducial symbols. Finally the identified trees are matched against a dictionary to retrieve unique marker ID numbers. 5) Vector Base Amplitude Panning (VBAP): Two of the most common techniques for spatial audio reproduction are VBAP [13] and Ambisonics [14]. They share the ability to place virtual sound sources anywhere on a surface outlined by a loudspeaker array. A major difference between them is the extent of the sweet spot, within which listeners can easily apprehend a projected soundscape. A large sweet spot can be supported in VBAP, whereas a narrow sweet spot is obtained with Ambisonics. In Ambisonics, the sound wave which comes out of coordinated loudspeakers (a minimum of four for planar fields) is composed by adjusting mutual phase, creating a desired acoustic field within the sweet spot but which is incoherent outside the sweet spot. For a spatial sound environment with table-top interface, VBAP is more suitable for the system because there might be several users around the table. For a two-dimensional VBAP method [15], the law of sines and the law of tangents model the directionalization: sinϕ sinϕ 0 = g n g m g n +g m (1) tanϕ tanϕ 0 = g n g m g n +g m (2) where 0 < ϕ 0 < 90 is the bearing of each speaker, ϕ 0 ϕ ϕ 0 is the bearing of a virtual source, and g n,g m [0,1] are gain factors. The law of sines is valid if the listener s head is pointing directly forward, but to support head rotation, as when a user turns to follow a virtual source, the law of tangents is more appropriate. As shown in Fig. 4, the unit vector p = [p n p m ] T, which points toward a virtual source, can be treated as a linear combination of loudspeaker vectors, p = g n l n +g m l m. (3) In Eqns. 1 3 g n and g m are gain factors, which can be treated as positive scalar variables. We may write the equation in matrix form, p T = gl nm (4) where g = [g n g m ] and L nm = [l n l m ] T. This equation can be solved if the inverse L 1 nm exists, III. A. Hardware Environment g = p T L 1 nm. (5) IMPLEMENTATION Mainly by controlling objects on the table-top interface from users playing music, input is generated, the ID and

3 Inputs Pure Data & Processing Outputs multiple audio channel source audio Sound fiducial ID fiducial ID, x, y Audio Interface Loudspeakers Tangible Ojbects Panning Sound Positions fiducial ID, x, y Projection Interaction Table Display Fig. 5. System diagram showing data flow and relationship among components localization data of each tangible object is organized as fiducialid [0,3], x & y [ 240,240] 2 in TUIO, then the data is passed to the sound and graphic systems. The basic system architecture consists of a Mac mini computer (OS X 10.8) connected to the audio interface, as well as to an Roland Edirol UA-101 audio interface to play musical sounds. The audio interface has ten input channels and ten output channels. It gets music sounds directly from the music player and distributes coordinated audio signals to eight loudspeakers (Yamaha MSP3) according to localization of each object on the table-top interface. A system diagram showing data flow and the relationship between components is presented in Fig. 5. A camera is mounted beneath the table to capture the location of the tabletop objects through the translucent screen. Dynamic graphics displayed through the surface follow each object on the table. An LCD video projector receives the computer video output (and loops it back to a monitor for programmer convenience). The system is built as a patch (like a procedure) in Pure Data (Pd), a visual programming data flow language specialized for interactive computer music and multimedia. The system processing is as follows: At first, multiple audio channel sources are received from the audio interface or generated in a Pd patch. Tangible objects are recognized their fiducial ID and localized to x & y by processing of the camera-captured scene. Object ID and position, based on the data from, are sent to sound controller, VBAP, and projection controller. Audio sources are selected by fiducial ID in sound controller. Panning is controlled in VBAP by information received from, and the gain of each speaker is calculated. The audio interface compiles the data comprising the location of sound sources, then outputs the diffused soundscape as modulated signals distributed to eight loudspeakers. The projection controller generates graphics using the fiducialid, x & y, which are projected to the table display. B. Software Environment A schematic of the software sound environment is shown in Fig. 6, including detail of the sound software environment in Fig. 5. The Pd system features four-channel input and eightchannel output, and consists of five kinds of objects: reactivision, multiple audio channel sources, panning controller, sound controller, and the signal objects. The loudspeaker setup is defined using define_loudspeakers, initialized by specifying the direction of loudspeakers. In our configuration, the loudspeakers are evenly spaced, separated at intervals of 45. To articulate control among the four input and eight output channels, there are four panning controllers. The runtime signal processing is shown as follows: Each gain factor of the corresponding loudspeakers is calculated at the same time in panning controllers to realize independent sound localization. Each panning controller is parameterized by x & y via fiducial ID of tangible objects, and azimuth is calculated. Users may optionally adjust spread control. VBAP objects are parameterized by define_loudspeakers,azimuth, andspread control. The gain factor is calculated in VBAP objects, as described above in II-5. In sound controller, audio sources are identified by fiducial ID.

4 Panning (x 4) x, y-axis azimuth spread control Sound fiducial ID VBAP Loudspeaker multiple audio channel sources audio Fig. 6. Schematic of software sound architecture The sound controller distributes audio sources along input channel number and fiducial ID to eight signal objects for modulation of the audio signals by the respective gain factors. The signal objects perform distribution of sound sources to eight loudspeakers. For simple example, a tangible object (fiducial ID = 0) sends location data (x = 100, y = 200) to the first panning controller. The panning controller calculates respective gain factors for eight loudspeakers. The sound controller sends audio source of each channel input to eight signal objects. Each signal object combines sound sources and respective gain factors, then distributes to all eight loudspeakers at once (not individually). fiducial makers affixed to the undersides. Users can determine a path for sound localization by handling and arranging the tangible objects. Fig. 8 shows a snapshot of a couple of users playing the table-top interface using four tangible objects. Our table-top interface is for groupware and assumed session play of up to four users. Multiple users can play collaboratively each other, including controlling sound localization with four tangible objects using their both of hands freely. Sound manipulation of users is a kind of cooperative group work. If they want to change a music or synthesizer, they can freely select a tune by using an audio device or synthesizer such as oscillator, phasor, sine wave, etc. C. User Manipulation Fig. 8. Snapshot of a couple of users playing the table-top interface Fig. 7. Four tangible objects and fiducial markers (amoeba design) All manipulations at the user side are performed using tangible objects, like those shown in Fig. 7. We use four square beverage coasters made of oak cork as tangible objects with IV. RESULT We have presented the implementation of a prototype tabletop interface with integrated spatial sound environment. We could realize three-dimensional spatial sound control through eight loudspeakers, adjusting multiple tangible objects in realtime. The wide sweet spot of the VBAP soundscape with eight

5 loudspeakers allows multiple users apprehension of spatial sound. No noticeable delay is observed regarding realtime calculation of the multisource soundscape, and users can freely express their spatially musical ideas. It became possible to express closely what users would like without a sense of incongruity. Because, contrary to our expectation, the frame rate in the reactivison is low (around 20 fps), recognition of the marker breaks off, and sounds may be disabled. The problem arises from the web camera s frame rate, which couldn t reach the maximum claimed in the product specification. V. CONCLUSION Table-top interfaces are utilized as groupware, suitable for collaborative activities. Such interfaces have potential for meetings with topic of sounds, especially surround music and spatial sound. We have presented the implementation of a prototype of table-top musical instrument with spatial sound environment. We have been investigating controlling position of multiple sounds in real-time with multitouch groupware. We succeeded in controlling position of four sounds in realtime by deploying four panning controllers, which calculate gain factors separately. VBAP features a large sweet spot; sound localization using VBAP with eight loudspeakers around a reactable was deemed adequate. When users control four independent sounds, they can play them without worrying about interference, and flexible, cooperative group activity is possible. However, sound localization performance is poor when tangible objects are moved quickly because the frame rate of the web camera which we used was insufficient for capturing the markers. For future work, we would like to extend the acoustic display to go beyond the current planar space, allow expression of fully three-dimensional sound, controlled by rotation of tangible objects. ReacTIVision can capture multitouch finger gesture on the table, so we will investigate the expression of spatial sound with such gestures. REFERENCES [1] G. D. Abowd and E. D. Mynatt, Charting past, present, and future research in ubiquitous computing, ACM Transactions on Computer- Human Interaction (TOCHI), vol. 7, no. 1, pp , [2] P. Isenberg, U. Hinrichs, M. Hancock, and S. Carpendale, Digital tables for collaborative information exploration, in Tabletops-Horizontal Interactive Displays. Springer, 2010, pp [3] W. G. Gardner, 3-D audio using loudspeakers. Kluwer Academic Pub, [4] N. A. Streitz, P. Tandler, C. Müller-Tomfelde, and S. Konomi, Roomware: Towards the next generation of human-computer: Interaction based on an integrated design of real and virtual worlds, Human- Computer Interaction in the New Millenium, Addison Wesley, pp , [5] W. Westerman, J. G. Elias, and A. Hedge, Multi-touch: A new tactile 2-d gesture interface for human-computer interaction, in Proc. of the Human Factors and Ergonomics Society Annual Meeting. SAGE Publications, 2001, pp [6] G. W. Fitzmaurice, H. Ishii, and W. A. Buxton, Bricks: laying the foundations for graspable user interfaces, in Proc. SIGCHI Conf. on Human Factors in Computing Systems. ACM Press/Addison-Wesley Publishing Co., 1995, pp [7] J. Patten, H. Ishii, J. Hines, and G. Pangaro, Sensetable: a wireless object tracking platform for tangible user interfaces, in Proc. SIGCHI Conf. on Human Factors in Computing Systems. ACM, 2001, pp [8] R. Bencina and M. Kaltenbrunner, The design and evolution of fiducials for the reactivision system, in Proc. Int. Conf. on Generative Systems in the Electronic Arts, [9] Kaltenbrunner, Martin and Bovermann, Till and Bencina, Ross and Costanza, Enrico, TUIO: A protocol for table-top tangible user interfaces, in Proc. Int. Workshop on Gesture in Human-Computer Interaction and Simulation, [10] M. Wright, A. Freed, and A. Momeni, Opensound control: state of the art 2003, in Proc. Conf. on New Interfaces for Musical Expression, 2003, pp [11] M. Kaltenbrunner and R. Bencina, : a computer-vision framework for table-based tangible interaction, in Proc. Int. Conf. on Tangible and Embedded Interaction. ACM, 2007, pp [12] R. Bencina, M. Kaltenbrunner, and S. Jorda, Improved topological fiducial tracking in the reactivision system, in Computer Vision and Pattern Recognition-Workshops, CVPR Workshops. IEEE Computer Society Conf. IEEE, 2005, pp [13] V. Pulkki, Generic panning tools for MAX/MSP, in Proc. Int. Computer Music Conf., 2000, pp [14] D. G. Malham and A. Myatt, 3-D sound spatialization using ambisonic techniques, Computer Music, vol. 19, no. 4, pp , [15] V. Pulkki, Virtual source positioning using vector base amplitude panning, JAES, vol. 45, no. 6, pp , 1997.