INTRODUCTION. General Structure

Transcription

1 Transposed carrier and envelope reconstruction Haptic feature substitution Pitch and Envelope extraction EMD decomposition (mus. features) Spatial vibrotactile display Synth acoustic signal Auditory EMD Analysis V S A pitch extraction on/offset detection timbre analysis Beat analysis loudness estimation polyphonic analysis A V S acoustic signal flow vibrational signal flow symbolic information flow FIGURE : General model structure. INTRODUCTION Traditionally, specialized haptic displays have only played a minor role in music performance, although current haptic technology shows great potential for both normal-hearing and hearingimpaired listeners. In this paper, we report on the potential of haptic devices for telematic music performances with a strong focus on a sensory-substitution device for hearing impaired musicians that can be used within telepresence scenarios. To our experience, haptic signals have been proven useful for (i) the reproduction of floor vibrations of music performances [, 2], (ii) to use haptics as a side channel for additional communication within a music performance e.g., using the Vibrobyte, a hand-held wireless haptic display [3], and (iii) to use haptics to substitute for sound with hearing-impaired users. Sensory Substitution has been described in numerous publications [4, 5, 6, 7, 8], and can be defined as the act of presenting information from one modality using another, often to make this information directly accessible to sensory-impaired users. It is well known that vibrations can be extremely important for hearing impaired musicians/listeners. The world-renowned percussionist, Evelyn Glennie, for example, profoundly deaf since the age of 2, always performs barefoot to better sense the floor vibrations that result from the sounding musical instruments. Glennie is known for her preference to feel the music through vibration rather then relying on her residual hearing as it is processed through hearing aids [9]. While a few investigations have dealt with augmenting music through haptic interfaces [,, 2, 3] these papers did not focus on impaired listeners. SENSORY SUBSTITUTION SYSTEM General Structure One of our initial aims was to understand to what extent the haptic and auditory sensory systems are comparable from a perceptual viewpoint. Western traditional music theory is based on several fundamental parameters including melody, rhythm, harmony, texture and form. While

2 FIGURE 2: Haptic Sensory Substitution Prototype. there is little doubt that the temporal resolution of touch is fine enough (and much better than vision) to resolve rhythmic patterns and form, the frequency resolution is not high enough to directly extract pitch information from haptically-displayed audio signals. Numerous studies have investigated tactile frequency selectivity, mostly in the context of work safety or fundamental psychophysical research e.g., see [4, 5, 6]. Interestingly, to the best of our knowledge, none of these studies asked the question that is fundamental for haptic music perception: Can haptic frequency intervals have the same perceptual quality as acoustic frequency intervals, and thus do we perceive them as musical intervals? To our surprise, this was to some extent the case in the low-frequency range [7]. To investigate this normal hearing listeners wore ear muffs and in-ear headphones with pink noise to avoid stimulating the auditory system via bone conduction. Our developed prototype, see Fig. [8], acknowledges the substantial differences between the senses of touch and audition. Unlike he case of the auditory organ, we do not have mechanisms to segregate individual frequency components for touch. While the somatosensory system has different touch receptors, initial test showed that one cannot discriminate between the lower and higher frequencies the Meisner Corpuscles and the Pacinian Corpuscles are tuned to. Further, the most sensitive region is lower in frequency for touch compared to the auditory sense. Consequently, our sensory-substitution prototype transposes the digitized acoustic music signal into the low-frequency range after removing the overtone spectrum, which is distracting for haptic sensation. Further, the information can be spatially distributed using an 8-channel vibrotactile board (see Fig. 2). There are at least two ways of coding polyphonic pitch for a haptic display. Both procedures take advantage of the excellent spatial resolution of the haptic sensory system as well as its inability to segregate an incoming signal into different frequency bands. The first method is to capture monophonic musical instruments individually. This can be done using closely positioned microphones and then processing and presenting each instrument with a different actuator on the 8-channel device. In the second approach, a complex music signal is analyzed using a polyphonic pitch model and then each pitch channel is processed and sent to a different actuator. In both cases, the user can use different fingers for each actuator to perceive different pitches simultaneously. POLYPHONIC PITCH PERCEPTION MODEL Methods The pitch model builds on a functional model of the auditory periphery and previous pitch perception models [9, 2, 2, 22]. In the first step, to simulate the behavior of the basilar membrane, the signal is sent through a Gammatone filterbank with 28 bands to segregate the sound

3 into different auditory bands. Then, the signal frequency f n in each band n is estimated using auto-correlation, measuring the delay τ n between the main and the largest side peak: f n = τ n. () A novel aspect of the model is that not only the frequency is measured in each frequency band, but also the pitch strength by calculating the amplitude ratio a between the largest side peak and the main peak. Further, the deviation b between the estimated frequency f n and the center of the frequency band f c,n is calculated. Next, all results are grouped into four categories:. a >.9, b.3 octaves ( + symbols) 2. a >.9, b >.3 octaves ( symbols) 3. a.9, b.3 octaves ( symbols) 4. a.9, b >.3 octaves ( symbols) FIGURE 3: Pitch estimation for a 44-Hz sinusoidal signal (left graph) and a 44-Hz tone complex (right graph). Results The graphs on the left in Fig. 3 show the results of the pitch model for a 44-Hz sinusoid. The top graph shows the broadband autocorrelation function, the center graph the Fourier Transformation of the signal, and the bottom graph depicts the excitiation of the auditory bands (solid black curve). For the curve, the energy in each of the 28 bands was measured and plotted at the center frequency of the band. All values for the Group (a >.9, b.3, + symbols) are located at 44 Hz, the frequency of the sinusoid, as indicated by the grey curve. The height of the values represent the energy of the band in which the frequency was measured. The values for Group 2 (a >.9, b >.3 octaves, symbols), also point to 44 Hz. They were measured in the adjacent side bands. All other values (Groups 3 and 4) were measured from the noise spectrum at low energies and do not represent the frequency of the sinusoid. The right graphs of Fig. 3 depict the same context but this time for a tone complex with eight higher harmonics at integer multiples of the fundamental frequency: f = n f. The amplitude of the tone complex rolls off with /f. Again, all values for Groups and 2 ( + and symbols) point to the fundamental frequency of 44 Hz, even those that belong to the higher harmonics.

4 FIGURE 4: Same as Fig. 3, but for 22-Hz signals Figure 4 shows the results for a /f tone complex at a lower fundamental frequency of 22 Hz (left graphs). Again, the results in all harmonics point to the fundamental, with the exception of two values in the octave region (44 Hz). It is not clear why in this case the octave is recognized but not for the 44-Hz tone complex. This will be further investigated. For higher harmonics, more than one overtone falls into the same auditory band. The overtones interfere with each other, and based on this interference the autocorrelation method identifies the common fundamental f. For the same reason, the algorithm is able to detect a missing fundamental. The right graphs show the results for the same tone complex, but this time the fundamental of 22 Hz was removed. Still, most values point to 22 Hz. Clearly those values belong to Group 2 (a >.9, b >.3 octaves, symbols) since there is no energy around 22 Hz and the values were computed at higher frequency bands FIGURE 5: Same as Figs. 2 and 4, but for polyphonic tone clusters. Left: Sinusoids with frequencies of 22 Hz, 262 Hz and 33 Hz, right: tone complexes with frequencies of 22 Hz, 262 Hz, 33 Hz and 88 Hz. Finally, chord complexes were analyzed using the model as depicted in Fig. 5. The left graph shows a triad of sinusoids with frequencies of 22 Hz, 262 Hz and 33 Hz. The model correctly identifies all tones. The right graphs shows a cluster of / f tone complexes with the following fundamental frequencies: 22 Hz, 262 Hz, 33 Hz and 88 Hz. The model identifies all fundamental frequencies correctly, but also a number of octaves, for example at 56 Hz which is the octave of the 262-Hz tone.

5 ACKNOWLEDGMENT This material is based upon work supported by the National Science Foundation under Grant No REFERENCES [] C. Abercrombie and J. Braasch, A method for multimodal auralization of audio-tactile stimuli from acoustic and structural measurements, in Proc. of the Convention of the Audio Eng. Soc. New York, NY, volume 27 (29), paper Number [2] C. L. Abercrombie, Influence of vibration and stage construction on the perception of musical performance, Master s thesis, Rensselaer Polytechnic Institute (29). [3] K. McDonald, D. Kouttron, C. Bahn, J. Braasch, and P. Oliveros, The vibrobyte: A haptic interface for co-located performance, in Procceding of the 9th International conference on New Interfaces for Musical Expression (NIME) (Pittsburgh, PA) (29), paper ID 6. [4] K. Kaczmarek and J. Webster, Electrotactile and vibrotactile displays for sensory substitution systems, IEEE Transactions on Biomedical Engineering 38, 6 (99). [5] M. Massimino and T. Sheridan, Sensory substitution for force feedback in teleoperation, Presence 2, (994). [6] P. Bach-y-Rita and S. W. Kercel, Sensory substitution and the human machine interface, Trends in Cognitive Sciences 7, (23). [7] J. Zelek, S. Bromley, D. Asmar, and D. Thompson, A haptic glove as a tactile-vision sensory substitution for wayfinding, Journal of Visual Impairment & Blindness (JVIB) 97, (23). [8] P. Bach-y-Rita, Tactile sensory substitution studies, Annals of the New York Academy of Sciences 3, 83 9 (24). [9] H. Finch, Rain on my parade, please: Evelyn Glennie tells why she relishes getting a soaking but is angry about the future, The Times, July 9 (24). [] M. S. O Modhrain, Playing by feel: incorporating haptic feedback into computer-based musical instruments, Ph.D. thesis, Stanford, CA, USA (2), adviser-chafe, Chris. [] A. Chang, S. O Modhrain, R. Jacob, E. Gunther, and H. Ishii, ComTouch: design of a vibrotactile communication device, in Proceedings of the 4th conference on Designing interactive systems: processes, practices, methods, and techniques, (ACM) (22). [2] E. Gunther, G. Davenport, and S. O Modhrain, Cutaneous grooves: composing for the sense of touch, in NIME 2: Proceedings of the 22 conference on New interfaces for musical expression, 6 (National University of Singapore, Singapore, Singapore) (22). [3] S. O Modhrain and G. Essl, Pebblebox and crumblebag: tactile interfaces for granular synthesis, in NIME 4: Proceedings of the 24 conference on New interfaces for musical expression, (National University of Singapore, Singapore, Singapore) (24). [4] M. E. Altinsoy, Audiotaktile interaktion in virtuellen umgebungen, Ph.D. thesis, Ruhr- Universität Bochum (26).

6 [5] M. J. Zigmond, F. E. Bloom, S. Landis, J. L. Roberts, and L. R. Squire, Fundamental Neuroscience (Peninsula Publishing) (999). [6] E. Kandel, J. Schwartz, and T. Jessell, Principles of neural science (McGraw-Hill New York) (2). [7] D. Egloff, A vibrotactile music system based on sensory substitution, Master s thesis, Rensselaer Polytechnic Institute (2). [8] D. Egloff, J. Braasch, P. Robinson, D. Van Nort, and T. Krueger, A vibrotactile music system based on sensory substitution, The Journal of the Acoustical Society of America 29, 2582 (2). [9] A. de Cheveigné and H. Kawahara, Yin, a fundamental frequency estimator for speech and music, J. Acoust. Soc. Am., (22). [2] A. Cheveigné, Pitch perception models, in The Psychophysics of Pitch, edited by C. J. Plack and A. J. Oxenham, (Springer, Berlin, Heidelberg, New York) (25). [2] J. Licklider, A duplex theory of pitch perception, Cellular and Molecular Life Sciences 7, (95). [22] R. Meddis and L. O Mard, A unitary model of pitch perception, The Journal of the Acoustical Society of America 2, 8 (997).