Motion Origami. Introduction. Daniel Bartoš

Size: px

Start display at page:

Download "Motion Origami. Introduction. Daniel Bartoš"

Clifford Douglas
5 years ago
Views:

Motion Origami Daniel Bartoš FAMU, Prague, Czech Republic daniel.bartos@gmail.com Abstract. The Motion Origami project explores live performance strategies focused on gesture based control of sound.

1 Motion Origami Daniel Bartoš FAMU, Prague, Czech Republic Abstract. The Motion Origami project explores live performance strategies focused on gesture based control of sound. The sound processing involves granular sound synthesis in a Max/MSP patch. Motion Origami is designed for a live performance scenario as an interactive music instrument. Motion Origami makes use of the live performance audio input or of the pre-recorded sound material. This live interface prototype explores a gesture-based music composition and music performance techniques. The sound transformations are driven by the hand gestures, while the use of motion tracking device lets the user build up a specific experience and virtuosity. Keywords: Leap Motion sensor, gesture recognition, motion tracking, music expression, granular synthesis, Max/MSP Introduction The name of the Motion Origami project is inspired by the Japanese art tradition of origami folding. The actual process of paper folding is reflected in a specific compositional strategy, which uses captured hand gestures. In other words the performing artist, the musician, is able to 'fold' sounds with his own hand gestures into new sound objects. The simple 'folds' therefore result into complex soundscapes during the performance. Figure. Motion Origami MAX/MSP patch with the Leap Motion Motion Origami; the project presentation can be accessed online at

2 This paper describes an original Max/MSP patch, which uses the Smooth Overlapping Granular Synthesis object sogs~ 2 as it's main sound transforming element. The novelty of this approach is not in the production of a new code or a particular specific design, but in making a creative use of the existing and advanced audio processing tools in the sound and music performance. The project shows how a live interface prototype can be turned into an original compositional tool. As such, the Motion Origami project represents a recipe on how to approach a design of an experimental music instrument and shows an approach similar to the rapid prototyping technique applied to the realm of the advanced Max/MSP audio processing domain. Leap Motion The Leap Motion 3 sensor was designed for touch-less tracking of hand gestures and their movements. The sensor in conjunction with the Leap Motion SDK becomes a sophisticated controller and delivers complex information monitoring the hand movements in real-time. Hand gestures are captured with a high precision accuracy along with the individual finger positions, rotations and finger tips accelerations. The Leap Motion sensor was introduced to the market in the mid 203 and swiftly found its place among other devices designed for body movements and gesture tracking as for example the Kinect 4 and the Wii controller 5. Such devices work usually with general body tracking and can be repurposed. The Leap Motion on the contrary is designed to capture the hand gestures and movements only. In fact the Leap Motion sensor could be thought of as a special interactive space 6 of predefined volume of air. Common music applications of the Leap Motion sensor are primarily based on the imitations of physical interfaces of existing musical instruments. This is true especially for the following projects: Air- Keyes, Crystal Piano, Air Harp, etc. (Han 204). The main reason for that is, that the virtual keyboard represents an ideal setup for testing of the system latency (Silva 203), as the low latency response is one of the most important elements for any real-time music performance application. The latency of the Leap Motion, as advertised by the manufacturer, is anywhere from 5ms to 20ms, but this particular figure obviously depends on the whole system configuration and components used. Another category in the existing applications of the Leap Motion sensor is represented by various hybrid instruments. A specific selection of such projects is described in the paper Lessons Learned in Exploring Leap Motion (Han, Gold 204). The Motion Origami Body & Sound Interaction The theme of physical interaction and sound processing is thoroughly investigated in the paper called Sound Design as Human Matter Interaction (Wei 203), whereas the most important keyword becomes the term material computation 7. In the extended sense we can think of the Leap Motion sensor as if it is constituting an interactive space on its own, where the calculations and interaction take place. The gesture recognition in conjunction with a music performance is 2 MAX/MSP object sogs~ (Smooth Overlap Granular Synthesis) by Norbert Schnell, IRCAM Centre Pompidou; more information can be accessed online at 3 Leap Motion, complete sensor and SDK specification can be accessed online at 4 Microsoft Kinect is a sensor using depth map to create 3D space representation. Developed by Microsoft for a game console Microsoft Xbox 360. Kinect specification can be accessed online at 5 Wii Remote is part of Nintendo Wii Console developed by Nintendo Company, Ltd. Wii specification can be accessed online at 6 Interactive space of 8 cubic feet, equivalent of 0,22 m 3 respectively, as stated by the manufacturer; more information can be accessed online at 7 Realtime, continuously responsive environments for the design of technical systems for human-computer interaction design. Ibid, p. 200.

3 8 also being explored by the IRCAM's ISMM research team. While the physical body & sound interaction concept is 9 present for example in the projects of Marco Donnarumma, who uses set of repurposed bio-sensors. Figure 2: Motion Origami the individual patch sections explained In the case of the Motion Origami Max/MSP patch, the performer's physical gesture interaction is the primary source of the resulting sound transformations. The performer creates new sound objects with the captured hand gestures. The hand gestures in the Motion Origami patch are recognised in the interactive space above the sensor and are turned into a control mechanism coded in the Max/MSP patch. A single recognised hand gesture initialises the audio buffer with the incoming audio. The buffer acts as a starting point for the granular synthesis soundscape building. The hand gestures control the granular synthesis engine parameters, along with the timing of the buffer initialization with a new audio material during the live performance as well. The hand gestures control the wet & dry ratio of the audio signal input and 0 also the multichannel audio distribution via the Ambisonics engine. Motion Origami Patch Implementation The Motion Origami patch is programmed in Max/MSP. Data from the Leap Motion sensor are captured by the 2 SwirlyLeap object. The updated version of the patch uses the current and well documented IRCAM's leapmotion 8 IRCAM ISMM team {SOUND MUSIC MOVEMENT} INTERACTION; more information can be accessed online at 9 Marco Donnarumma; project presentations can be accessed online at: 0 Ambisonics Tools from the Institute for Computer Music and Sound Technology ICST, Zurich University of the Arts can be accessed online at: Max/MSP visual programming environment by Cycling74, more information can be accessed online at

4 object 3. The Smooth Overlapping Granular Synthesis object sogs~ 4 was chosen because it offers a simple and creative control over the audio captured into the audio buffer and can be used for a specific navigation and exploration of the audio buffer. The sogs~ object also mimics the paper folding technique in the sense, that the original paper surface is substituted with a 2D plain made of two individual parameters: the grain position and the grain size. The data from the Leap Motion sensor are mapped to drive the sogs~ object with those two selected parameters: the performer than navigates the space of the audio buffer defined by the grain size and the grain position parameters respectively. Figure 3. Motion Origami the recognised gestures illustrated The wet & dry mix ratio, which is also mapped to hand gestures, offers detailed control over the sound merging and accents the actual 'sound folds'. These 'sound folds' can build up a certain level of complexity thanks to the fact, that the live audio source is coupled with the audio material stored in the buffer. Although the audio is modified in the granulation process, it shares the same spectral and tonal characteristics with the original sound source. This in turn creates elaborated sound transformations, which can be though of as the introduced 'sound folding' process. The patch recognises a specific gesture, which is required to start the initialisation of the audio buffer. In this way the audio buffer is filled with a new audio material. The buffer initialisation starts with a gesture of a closed hand. This gesture can be paraphrased as 'grab the sound' gesture. In this very moment, the buffer is filled with the live sound input and becomes available to sogs~ object. Subsequent hand gestures control various aspects of the granular synthesis engine: horizontal hand swipe controls grain position selection, vertical hand movement controls time length of a grain. Moreover the overall palm position above the sensor in the x - y plane defines the sound source position in the multichannel Ambisonics space and adds a multichannel spatialisation layer to the performance. The other Leap Motion recognised variables as yaw, pitch a roll are alternatively mapped to extended FX processing (reverb, distortion, etc.), depending on the performance scenario. 2 Swirly Leap Max/MSP access the Leap Motion API, written by Tom Ritchford from New Yorku. Project can be accessed online at 3 Well documented Max/MSP object leapmotion by IRCAM, more information can be accessed online at 4 Max/MSP object sogs~ (Smooth Overlap Granular Synthesis) by Norbert Schnell, IRCAM Centre Pompidou, more information can be accessed at

Figure 4: Motion Origami a detail of the subpatch controlling the audio buffer initialisation Conclusions & Performance 5 The most inviting application of the Motion Origami patch is a vocal

5 Figure 4: Motion Origami a detail of the subpatch controlling the audio buffer initialisation Conclusions & Performance 5 The most inviting application of the Motion Origami patch is a vocal performance or simply a music instrument, which leaves enough space for the interaction with the sensor itself. The beauty of the live performance approach lies the fact, that the performer can interact with his/her captured music material and add multiple layers of expression by solely using the hand gestures. New layer of improvisation can be introduced, while new themes and phrases emerge. The performer than interacts with a new music material, which is based on the sound qualities of the original music instrument or the vocal performance. The performer can control the following parameters in the Motion Origami live interface: time based selection of a phrase sampled into the buffer; grain size and it's position in the buffer; wet & dry mix ratio and Ambisonics sound source space position (if applicable). Using gestures in music composition and performance proves to be very intuitive. The sensor alone has to be 'learned' to be operated properly and this fact delivers a specific virtuosity over time. The Leap Motion sensor with the Motion Origami patch opens up a new exciting field of music composition and sound processing coupled with immediate gestural interaction. The biggest challenge in the gesture based performance is the recognition of the quantized gestures (Potter 203). While parametric control of the various patch elements doesn't present any technical problem trackingwise, the recognition of the quantized and unique gestures proved difficult through out the development phase of the patch. For example, while playing on a virtual keyboard, one can limit the key strokes to a specific scale and limit the mis-triggered notes this way. But when it comes to evoking a specific functionality (sampling initialization, sound bank toggle, etc.) the gestures have to be recognized with exceptionally high precision as those decisions create an integral part of the performance itself. This aspect of the live performance gives us a specific constrains and we have to consider them in the live performance scenarios, when using the Leap Motion sensor. The quality of the tracking depends also on the present light conditions and the overall sensor setup. For example, a direct light reaching the sensors's surface can introduce inconsistency in the tracking. 5 Motion Origami, the project presentation can be accessed online at

6 Overall, the Leap Motion is very suitable for various intuitive music composition and performance scenarios. Occasional errors in tracking can be overcome with a good patch design and the restricted use of the quantized gestures leaving out the quantized gestures to be services by traditional hardware controllers. Having said that, the Leap Motion sensor excels in the intuitive gesture interaction performance and the gesture based music composition strategies. Acknowledgements. The Motion Origami project was funded by a research grant scheme Sensors as music instruments at HAMU, Music Academy in Prague run by Michal Rataj. Special thanks go to Michal Rataj6 and Matthew Ostrowski7 for their support, help with the patch design and their ideas about performance and music composition. References Han, Jihyun; Gold, Nicolas Lessons Learned in Exploring the Leap Motion Sensor for Gesture-based Instrument Design, in Caramiaux, B and Tahiroğlu, K and Fiebrink, R and Tanaka, A (eds.). Proceedings of the International Conference on New Interfaces for Musical Expression (NIME'4) London 204, Goldsmiths University of London: Potter, Leight Elen; Araullo, Jake; Carter Lewis The Leap Motion controller: A view on sign language, Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration Adelaide, ACM, New York: Silva, Eduardo S; de Abreu, Jader Anderson O.; de Almeida, Janiel Henrique P.; Teichrieb, Veronica; Ramalho, Geber L A Preliminary Evaluation of the Leap Motion Sensor as Controller of New Digital Musical Instruments, Proceedings of the Brazilian Symposium on Computer Music Brasilia, SBCM: Schwarz, Diemo Concatenative sound synthesis: The early years, Journal of New Music Research 35(): 3 22 (March). Wei, Sha Xin; Freed, Adrian; Navab, Navid Sound Design As Human Matter Interaction, Proceedings of CHI '3 Extended Abstracts on Human Factors in Computing Systems in Paris, ACM, New York: Michal Rataj music composer and researcher at HAMU (Music Academy in Prague). More information can be accessed online at 7 Matthew Ostrowski expert on sensors and multimedia programming based at Harvestworks NYC, offered valuable insights into Leap Motion programming. More information can be accessed online at and

Spatial Interfaces and Interactive 3D Environments for Immersive Musical Performances

Spatial Interfaces and Interactive 3D Environments for Immersive Musical Performances Florent Berthaut and Martin Hachet Figure 1: A musician plays the Drile instrument while being immersed in front of