A Left Hand Gesture Caption System for Guitar Based on Capacitive Sensors!

Transcription

1 A Left Hand Gesture Caption System for Guitar Based on Capacitive Sensors! Enric Guaus, Josep Lluís Arcos, Tan Ozaslan, Eric Palacios! Artificial Intelligence Research Institute! Bellaterra, Barcelona, Spain!

2 Music computing challenges! Study expressivity by putting together main involved facets: instrument, performer, music genre.! Model expressivity by combining partial information from multiple sources (audio, gesture,...).! Design tools for realistic usage.! Machine learning challenges! Managing Multimodal input streams.! Dealing with multi-scale events.! Combining Knowledge Intensive and Data Intensive methods.! NIME

3 Understand articulations performed by the left hand! Most common fingering configurations! Time-dependent articulations! Gesture acquisition system for left hand! Non intrusive! Independent of lighting and position (on stage)! Able to detect from macro-scale to micro-scale gestures! Fingers do not press the fingerboard! Barcelona Laptop ESMUC! barcelonalaptoporchestra.blogspot.com! NIME

4 Gesture Acquisition! D. Young. The hyper bow controller: Real-time dynamics measurement of violin performance. In Proc. of NIME, 2002.! M. M. Wanderley, and P. Depalle. Gestural control of sound synthesis. Proc. IEEE, 92(4): , April 2004.! M. Laurson, C. Erkut, V. Välimäki, and M. Kuuskankare. Methods for modeling realistic playing in acoustic guitar synthesis. Comput. Music J., 25(3):38 49, 2001.! A. Burnsand, and M. Wanderley. Visual methods for the retrieval of guitarist fingering. Proc. NIME 2006, June2006.! H. Heijink and R. G. J. Meulenbroek. On the complexity of classical guitar playing: functional adaptations to task constraints. Journal of Motor Behavior, 34(4): , 2002.! J. Norton. Motion capture to build a foundation for a computer-controlled instrument by study of classical guitar performance. PhD thesis, Stanford University, September2008.! NIME

5 Right hand: Hexaphonic pickups! Steel & electric guitar! Nylon guitar! NIME

6 Capacitive sensing! Lev Termen invented the Theremin in 1919.! Musical interfaces and augmented instruments use capacitive sensors.! J. Paradiso, and N. Gershenfeld. Musical applications of electric field sensing. Computer Music Journal, 21(2):69 89, 1997.! S. Hughes, C. Cannon, and S. Modhráin. Epipe: A novel electronic wood wind controller. Proc. NIME, 2004.! The capacitance is the ratio of electric charge over a voltage: Q=C/V! The capacitance of an ideal capacitor built up with two conductive parallel plates is computed as: C=E*S/d! Our configuration follows the load mode (Miranda 2006):! the distance between the electrode and a single object (the performer's finger in our case) is measured through a change in capacitance of the electrode to ground.! NIME

7 Prototype! Arduino + CapSense! NIME

8 Prototype! Fret sensors! NIME

9 Noise, stability! R values, MEAS_TIME, LPF! Sampling rate! Depending on the measured capacitance! About 30[Hz]! Crosstalk! NIME

10 Finger bars! Chromatic scales! Grace notes! Diatonic scale! Basic arpeggios! Slurs! Vibrato! NIME

11 Finger bars! Chromatic scales! Grace notes! Diatonic scale! Basic arpeggios! Slurs! Vibrato! NIME

12 Finger bars! Chromatic scales! Grace notes! Diatonic scale! Basic arpeggios! Slurs! Vibrato! NIME

13 Finger bars! Chromatic scales! Grace notes! Diatonic scale! Basic arpeggios! Slurs! Vibrato! NIME

14 Finger bars! Chromatic scales! Grace notes! Diatonic scale! Basic arpeggios! Slurs! Vibrato! NIME

15 Finger bars! Chromatic scales! Grace notes! Diatonic scale! Basic arpeggios! Slurs! Vibrato! NIME

16 Finger bars! Chromatic scales! Grace notes! Diatonic scale! Basic arpeggios! Slurs! Vibrato! NIME

17 Finger bars! Chromatic scales! Grace notes! Diatonic scale! Basic arpeggios! Slurs! Vibrato! NIME

18 Capacitive sensors detect left hand articulations! Pros:! non intrusiveness! low cost! high dynamic range! low background noise! high fidelity to the finger and hand movements (even without contact!)! Cons! Crosstalk & electric interference! low and not constant sampling rate! no discrimination between strings.! NIME

19 Improving prototype! Arduino inside the body! Aluminum foils included in the fretboard! Coaxial cables! Individual strings! Expand the study including! Multiple fingering! Static position detection! Evolution in time! Musical context! Don t miss our paper at SMC2010!!! NIME

20 NIME

21 A Left Hand Gesture Caption System for Guitar Based on Capacitive Sensors! Thank you!!