Dimensional Design; Explorations of the Auditory and Haptic Correlate for the Mobile Device

Transcription

1 Dimensional Design; Explorations of the Auditory and Haptic Correlate for the Mobile Device Conor O Sullivan Motorola, Inc. 600 North U.S. Highway 45, DS-175, Libertyville, IL 60048, USA conor.o sullivan@motorola.com Angela Chang Motorola, Inc. 210 Broadway, 4 th Fl Cambridge, MA 02139, USA anjchang@motorola.com ABSTRACT Designing for the mobile device currently allows for the exploration of additional dimensions beyond that of simple graphic treatment. The relationship between two of these dimensions, audio and haptics is discussed here. A method for creation of content for the generation of audio-haptic feedback is discussed. A particular evaluation of such content and its application to design for the auditory and haptic correlate is offered. General observations on this work and the need for further exploration are described. 1. INTRODUCTION The space of content design for the mobile device has evolved rapidly in recent times. Today s designer has moved beyond the relatively straightforward needs of simple text and graphics into a domain that brings a stream of new technologies and with it, new challenges. In the sound space alone, the demanding consumer has ensured the move from the most basic electronic outputs to newer rich media. This has required designers to react accordingly and provide content that is compelling, exploratory and that takes full advantage of these new opportunities. Recent technological developments have also brought forth opportunities for design in the haptic space. The haptic capacity of the mobile device is moving from that of simple buzzing alerting to more extensive textural output that can engage the user more fully in their interactions. The one thing that hasn t increased however, at least in the general case, is screen size. This inevitably means that the difficulties posed by the increase in applications, user behaviors and overall information, will likely need to be addressed by the development of newer more relevant interaction techniques [1]. In this paper the authors suggest that one way of addressing this need is by taking fuller advantage of two areas of technology that increasingly appear in enhanced form on mobile devices, that of audio and haptics. More specifically the tandem pairing and crossover of these technologies, here referred to as the audio and haptic correlate, is explored. As these devices become increasingly complex, the potential for increase in overall user experience becomes more apparent and the authors present the preliminary results of an informal evaluation that would seem to suggest the same. The evaluation consisted of 42 participants comparing audio-based haptic user interface (audio-haptic UI) feedback, with audioonly feedback. Figure 1 A post-evaluation analysis of the qualities of haptics and audio For the designer of these new technologies, it is important to be able to create an experience or an element of the interface in a workspace that is not too far removed from their existing tools. Some methods of design are discussed here. The authors present a technical method for how to create content for this type of mobile device and also discuss the method of design in more general terms. The challenges of designing for this space, particularly haptics, are also addressed. The design can be thought of as dimensional design in that we are trying to give added dimension to an existing interface or domain. Of course, the dimensions here referred to are haptics and audio with particular attention on the crossover of the two. 2. DOMAIN OVERVIEW The authors have identified three of the most common methods for generation of haptic feedback in mobile devices today. These include the Rotary Mass Vibrator (RMV), the Multifunction Transducer (MFT) and Piezoelectric actuators. RMVs are also known as pager motors and allow for on-off vibration. The MFT was the chosen technology of the study described in this paper and will be presented in more detail later. Piezoelectric devices offer the promise of high resolution haptic generation but currently generally require high voltage to run Technology The MFT is a small audio speaker with an integrated oscillating mass that resonates at certain frequencies (here referred to as the haptic frequencies). The haptic frequencies occur in the lower end of the auditory frequency spectrum, generally focusing around a particular point between Hz. This frequency range is where the skin s sensitivity to vibration is most actively engaged. The MFTs are here employed for reasons of suitability to the mobile device, including power efficiency, lag time, size. Another key reason the MFT is useful in this context is its ICAD05-212

2 ability to act as an audio speaker and haptic texture generator in one device. This enables a smooth and synchronous output mechanism of audio and haptic information, thereby allowing us to explore the effect of correlation on a single device. It is hypothesized that an enhancement of effects such as realism, desirability or pleasureability is possible once synchronicity between haptic and audio is assured. In this case the hardware bundle allows the lines between audio and tactile feedback to merge successfully. A key advantage of the MFT is the ready availability of content that can be easily manipulated to appropriate useful haptic feedback. The techniques for haptic manipulation from an audio file, or haptic inheritance have been previously introduced, along with a synthesis and matching technique [2]. Both shall be explored further in context here. Basically this allows for an existing body of sound designers to write or generate content in their familiar environments or using existing tools Overview 3. DIMENSIONAL DESIGN The mobile device is one of a small set of unique media that is used and is most useful entirely in close proximity to the user; that is it is handheld and hand-interacted. In addition to this, as these devices become increasingly media-centric it becomes part of an even smaller set of devices where auditory feedback and musical expression are key features. For example, the tv, laptop, desktop, radio, stereo, mp3 player are everyday devices with limited windows of tactile engagement. That is not to say that these displays cannot be enhanced by haptic engagement they can, and most certainly have at various instances [5][6][7]. What is being suggested however is that the ubiquity of the mobile device offers perhaps the best existing engine for exploring means of the auditory-haptic correlate in an everyday context The Audio-Haptic Correlate in Design a) b) Figure 2 a) The Motorola E398 device and b) the Motorola V400 device used in the evaluation In phones with MFTs, such as the Motorola e380, e398 (See Figure 2a), e680, haptic effects can be felt when audio frequencies are between approximately Hz. Frequencies above 300Hz are heard as audio. By manipulating different parameters (e.g frequency, amplitude, and waveform), it is possible to design a variety of sensations that can range from a subtle buzz to discrete taps. Further information for content authoring for the devices from this evaluation can be found in the Media Guide sections of the respective devices, available at [3]. It is also true to say that the propagation of haptics on a mobile device should be expected. People are used to objects, such as speakers or musical instruments, making noise and feeling haptic information upon physical engagement [8]. In the case of the mobile, for such a small device to make such big noise and not react tactilely may be considered not entirely convincing. Of course, this can be explained by the higher frequency response of the speaker or even by successful engineering techniques but what might be considered unnecessary or obtrusive vibrations can be turned into useful and pleasant feedback that enhances the overall perception of quality and desirability. That is the goal of design that adds dimension, in this case along the auditory-haptic correlate. This dimensionality can be explored in several areas of the mobile device such as the provision of direct feedback, as alerting mechanisms, in information gathering, or as sensorial perception enhancement. For practical purposes, this applies to device applications, UI navigation paradigms, gaming and audio. In the case of the study and evaluation described here, the focus was primarily on providing direct haptic navigation feedback and a preliminary investigation into sensorial perception enhancement Challenges One of the greatest challenges facing the domain of audiohaptics and more particularly haptics today is the difficulty with which people find expression and move comfortably to a nomenclature [4]. The combination of temporal and spatial information provided by haptics to reinforce perception and communication is ubiquitous. For many years, vibration in mobile devices, such as pagers and cell phones, alerted people of calls or messages. The ubiquity of vibrating alerts in mobile devices has also misled many consumers to believe haptics is simply equivalent to vibration. This idea is explored further in the evaluation discussion, below. Figure 3 - Approximation of the important ranges of the frequency response of the MFT The technology of the MFT described above allows the design of audio-haptic content for the mobile device to be taken to new levels when compared with the design of pure audio or pure haptics. This is enabled because of the existing close ICAD05-213

3 (perhaps even somewhat synonymous) relationship between haptics or vibration feedback and audio or auditory information, especially as it pertains to this particular device. The graph in Figure 3 shows an approximation of the significant ranges of the frequency response of the MFT device used in the evaluation described here. Note the section of the graph at the lower end describes the resonance that will drive the haptic information, while the upper register displays the response of the device that will be heard primarily as audio. Given the value at which the device resonates, the vibration will of course be heard, but this can easily be masked by accompanying audio, or in the case of solo vibration, background noise. Voltage (V) Signal Acceleration MFT Signal Delay On Delay = 0ms Ramp Up = 40ms Time (s) Figure 6 - Approximate Ramp-up and On Delay of the Multi-Function Transducer Figure 4 An illustrative comparison of the Multifunction Transducer and the Rotary Mass Vibrator This can be compared to a response of the RMV, as shown in Figure 4. The RMV, or Pager Motor is shown as the narrow curve in the lower end of the frequency spectrum. These graphs are offered as approximations, to demonstrate the range of control and additional dimension the designer has when designing for the MFT. The MFT allows for an easy way of directly synching audio and haptic information on the same device. As described in the method, this can be useful in the design of such haptic information. This information can be directly extracted from the audio signal and applied as haptic information in conjunction with the audio signal, thus allowing the inherent haptic qualities of the sound, where applicable, to be felt. Note that the graph that represents the MFT response in Figures 3 and 4 are actually continuous in practice and here represented as discontinuous approximations to highlight the ability for the designer to separate treatment of the haptic and audio portions. Voltage (V) Rotary DC Motor Delay Signal Acceleration Linear 5 per. Mov. Avg. (Acceleration) On Delay = 50ms Ramp Up = 150ms Another key advantage to the use of the MFT in the mobile device is the performance when compared to a standard RMV. Figure 5 shows that there is a delay of about 50 ms before the RMV begins to ramp up. The ramp-up time, of about 150ms, is significant too. These factors come into play in the perception of tactile feedback and pose difficulties in audio synchronization. Figure 6 shows corresponding information for the MFT. Note the On Delay tends to 0, while the Ramp-Up time is significantly reduced. This allows for greater control by the designer of perception of haptic texture and synchronization with the audio stream Information Gathering Existing methods of both incorporating and describing tactile information have been examined for this evaluation. These include symbolic and alphabetic languages, tactile languages, speech reading with vibrotactile intonation information and force-feedback devices such as the Phantom [9]. Using the exploratory procedures (EPs) [10] as a basis for consideration of the crossover attributes of audio and haptics, Figure 7 shows some qualities of haptics as they apply to information gathering and how they apply to audition in the general case. To identify the areas that are relevant in designing for audio-haptics, it is useful to examine this table and determine the areas of crossover and conflict to exploit. Two areas that are relevant to design for the mobile device are the identification of temporal attributes and the limitations given by the illusory possibilities of tactile feedback generation. The auditory and haptic information that can be given over time is enhanced by pairing the audio and haptic information to potentially reinforce each component. The difficulty in generation of high-resolution haptic textures, in comparison with audio resolution, on the mobile device is a design limitation that can also be addressed somewhat by this pairing Time (s) Figure 5 Approximate Ramp-up and On Delay of a standard Rotary Mass Vibrator ICAD05-214

4 Information given relative to person Touch Useful for details smaller than a person Audition Useful for gauging distance relative to a person approach (as opposed to manipulative). This is the synthesis and matching technique. The method enables the designer to approximate a desired response by adding extra haptic frequencies, perhaps from a haptic library or by pure synthesis. Temporal aspects Active and Passive Identification Spatial aspects Illusory Possibilities Identification by parts. Information is gathered by sensing information over time.. Active and Passive methods for information gathering (e.g. applying pressure or echolocation) One-to-one only, a Can be close or close proximity remote, can be sensory broadcast experience Touch is hard to fake Easy to represent, replicate, reproduce digitally Figure 7 - Comparison of haptics attributes to audition In terms of the actual design of the haptic feedback information for the study, the navigation part was based on work initially developed for the E398 device [3]. This product features two MFTs and has haptic feedback incorporated. The device currently contains feedback as a feature both within the UI and as an enhancement to some of the audio (ringtone) content. The tactile information in the study was developed so that the feedback was more extensive in the UI for the participants while still carrying over the basic paradigms of design. There were also a number of enhanced-audio examples used. These were principally short musical sequences, such as enhanced ringtones and an enhanced musical sequence taken from the initial sound when the device is powered up. The goal for this part of the design was to enhance the perception of quality of the device output through the addition of a kind of bass boosting tactile feedback at appropriate points in the sequence. One of the tracks designed for the evaluation featured a rhythmic pulsing that extended into a drumbeat, out of which the music (featuring multi-dimensional audio) grew. 4. METHODS Two methods for processing sound to optimize the effect of haptic sensation, using actuators such as the MFT are here described. These methods are employed for generating audio with associated tactile information at lower frequencies. The proposed flow diagram for creating audio-haptics is depicted in Figure 8. Both methods require a preliminary analysis of the sound and its components to determine the appropriate path for generation of the haptic effect. The first method describes a technique the authors refer to as haptic inheritance. This method should be used after an appropriate analysis of the base sound to determine whether or not enough information around the haptic frequencies can be used. If there is, this method will provide an output that can be used as singular haptic feedback or in conjunction with some sound information in the form of an audio-haptic icon for example. The second method takes more of a generative Figure 8 Flowchart showing the method and path of design for the Inheritance and Synthesis/Matching techniques 4.1. Haptic Inheritance This method assumes some relevant degree of audio information around the resonant peak of the audio-response actuator. This can be determined after a simple analysis of the spectral information. The key part of the process using this method is to split the sound into its constituent parts, that of haptic and audio. Of course it is useful to point out that these parts both contain audio (and similarly haptic) information. From a practical standpoint this method is concerned with the separate treatment of the haptic and audio portions. The benefit of using this method is that the inherent haptic qualities of the audio can be reproduced on the mobile device, using the MFT. One such technique for arriving at these constituent bins is given here. Two filters can be applied to the original sound signal so that these bins are obtained and further processing can take place. The first, a band stop filter, is used to isolate the audio portion of the source signal. The cutoff frequency values are chosen relative to the resonant peak of the audio-response actuator. To obtain the second bin, a converse type of filter is used, here a band-pass filter. ICAD05-215

5 Once these constituent bins are obtained, it is then necessary to process them efficiently for the optimal level of haptic and audio re-combination output. In the first instance of this step of the method, the haptic bin is analyzed to determine the level of existing tactile effect. The best way to perform this analysis is of course through a physical determination of sensation, however an analysis and subsequent algorithmic adjustment of the signal is given here and has been found to be both useful and compelling. The amplification of this haptic bin is given by a normalization of up to -12dBVrms. The second part of this step involves additional filtering of the audio bin to further optimize for the audio speaker part of the output device. For a typical device used for this study, the frequency response of the output device begins to drop off sharply from 900Hz through 400Hz and below. The waveforms of the audio bins are thus ramped off accordingly. some differing haptic textures and playing of ringtones and other audio to compare the two devices. Then they were asked to fill out a questionnaire to asses whether people could distinguish the haptics, and also which device experience they preferred. There was also space for comments on their perception of the UI. The evaluations were performed on a commercially available Motorola e398 phone with dual MFTs, as shown in Figure 2a. This device was loaded with audio-haptic ringtones and UI-sounds for the menu key, end-of-list tone, power up key, and send and end keys, generated in the manner described above. The other phone, a Motorola v400 (in figure 2b) was clam shaped and was commercially available at the time of the test. It contained a RMV, and the standard audio files. During the conference, 42 volunteers (30 male, 12 female) tried out the phones for approximately 10 minutes each. The average age group of each user was between Synthesis and Matching The Synthesis/Matching Technique can be considered more subjective in nature. This allows for perhaps less natural correlation between the audio and the haptic component, but certainly, allows for greater freedom of design. This technique is best used when an analysis of the input signal reveals no useful audio information around the resonant Q peak of the audio-response actuator. It incorporates the generation of haptic textures using several possible methods and a means of matching the desired haptic response to appropriate or preferred points in the audio/time signal. Haptic icons are information signals that occur in the low frequency range between Hz, where the skin s sensitivity to vibration is engaged. These haptic icons are designed to serve as concise tactile representations of information to the user, similar to audio and visual icons, (i.e. messaging, alert, and confirmation). Some means to generate haptic texture icons include standard audio synthesis techniques such as modulation, mixing, filtering and enveloping. Alternatively, an approach suggested by the authors is to derive a haptic signal using the Haptic Inheritance method described above from one signal and incorporate this into the design of another. The techniques for synthesis and manipulation of audio signals are the same as can be found in any comprehensive digital audio textbook [11] and so need not be described here. There are a couple of points for the designer of haptic textures to note however; work should be performed in the frequency domain around the resonant Q peak of the MFT and also the same normalization of the haptic signal should apply. This normalization is necessary due to hardware drive limitations. 5. PARTICIPANT FINDINGS The DIS 2004 Conference [12] provided the forum for the evaluation described here. Two mobile devices were used for comparisons; one with haptic capabilities and one without. Given the tactile nature of the study, uses were very much encouraged to play around with the devices as they wished. They could perform tests on the devices in whatever order they chose, including side-by-side comparisons. The tasks included navigating the menu keys, UI navigation that would lead to 5.1. Observations The results clearly indicated a preference for the audio-haptics on the mobile device, over audio alone. This is given by the fact that 20 participants thought the haptic phone had better audio, while 22 felt the audio between the two were the same. One of the reasons why more people may not have been able to judge the audio well was that the environmental conditions were not controllable; hence the conference room was quite noisy. None of the participants reported that the non-haptic phone sounded better. Next, a Pearson correlation analysis was run between the sound and feel responses. On a scale of 1 to 5, with 5 being the most agreeable, the haptic sensations on the e398 phone received a mean rating of / out of 5. This above average rating is an important result, as it shows that most people said that they liked the haptic feedback phone. Of the comments and feedback received, here is a selection that indicates some of the key attitudes towards the audio-haptic feedback: [E398] sound is deeper, is felt vibration seemed to intensify sound the vibrating phone had better speakers could get annoying or desensitized if used for every click or selection the tactile one feels like it sounds better haptic feedback is only useful in certain situations (e.g. end of list) the tactile buzz is a bit startling but without it the sounds are tinny and chirpy It was found that there was a significant relation between sensation and the perception of sound quality. Phones that are said to sound better are significantly correlated with phones that garner high sensations ratings. This is a nice result, but not entirely surprising. This occurrence can be attributed to the idea that phones that sound good also rate high in sensation. The presence of haptics increases the perception of sound in the phone. It can be ICAD05-216

6 inferred therefore that there is some significant correlation between sound quality and haptic sensation Findings on Expression for Haptics The results of preliminary investigations into perception enhancements given by designing for the audio-haptic correlate suggest that it is possible to influence the perception of quality and desirability for the mobile device using these methods. A controlled study will likely help to validate these claims in more absolute terms, but the methods presented herein can already be used as a means of authoring content that is useful for these devices and, it is anticipated, especially meaningful to the user. 7. ACKNOWLEDGMENTS a) b) Figure 9 - Spectrograms can describe the difference between a) audio sound and b) vibration-enhanced audio. What was found was the people were only able to describe the vibrations very generally. There appears to be a large vocabulary gap between the scientific community and haptics understanding in the common usage. Quality in consumer products often relates directly to touch (weight, surface finish, and contours). However, there is little vocabulary to describe haptic sensations. When asked to describe haptic quality, a surprising number of people would relate haptic vibrations to audio [4]. Figure 9 displays the spectrograms of two audio files used for comparison. Many people simply said it feels different, but were unable to say why. Interestingly, this ease of expressiveness about haptics is surprisingly minimal in contrast with that of the sense of smell. Before the days of aromatherapy, there was little conscious awareness of the cause and effect of smell. In fact, until the force of marketing began to show itself, body odor was not considered a problem [13]. People did not typically describe smells, nor did they obsess about altering them. Nowadays, people can identify common scents in products (cinnamon, vanilla, bayberry, cologne scents). They can also understand that smells can be controlled and are able to distinguish differences between smells. 6. CONCLUSIONS The field of design for the mobile device has grown exponentially in recent times. As new methods of off-loading information from an ever-busy screen are increasingly required, the audio and haptic technologies are emerging as ideal solutions. The authors have presented some principles and methods for designing content for these technologies, with particular reference to the audio-haptic correlate as enabled by same-device solutions such as the MFT. The methods, of Inheritance and Synthesis/Matching, involve an analysis of the audio component to draw out and use any existing haptic information. One outcome of this study is that the authors have been alerted to the fact that people generally find difficulty in expressing or describing haptic sensations. It is hoped that as more consumer products make use of these technologies, the terms of description or nomenclature will find a common and comfortable footing amongst people. We thank the members of Motorola Consumer Experience Design for their generous support of this haptics work. In particular, we would like to thank the members of the audio team, the human factors team, and the MFT phone product teams. Finally, we would like to thank all the volunteers at DIS who served as test subjects for our preliminary evaluation. 8. REFERENCES [1] Leplatre, G. and Brewster, S.A. Designing Non-Speech Sounds to Support Navigation in Mobile Phone Menus, in Proceedings of ICAD2000 Atlanta, USA, 2000 pp [2] Chang, A. and O Sullivan, C. Audio-Haptic Feedback in Mobile Phones, to be published in the Extended Abstracts of CHI 2005, April 2005, ACM Press. [3] (Accessed Feb 01 st 2005) [4] Chang, A. and O Sullivan, C. (2005, April) Describing Haptic Phenomena to be presented at CHI 2005, April 2005 [5] Les Nelson, et.al. Quiet Calls: Talking Silently on Mobile Phones, Proc. CHI 2001, ACM Press, [6] Ishii, H. and Ullmer, B. Emerging frameworks for tangible user interfaces, IBM Systems Journal 39, (3&4): 915- (2000) [7] Crease, M.G. and Brewster, S.A. Making Progress With Sounds - The Design and Evaluation Of An Audio Progress Bar, in Proceedings of ICAD'98, Glasgow, UK, British Computer Society. [8] S. O'Modhrain. Playing by Feel: Incorporating Haptic Feedback into Computer-Based Musical Instruments. Ph.D. Dissertation. CCRMA, Stanford University 2000 [9] m.asp (Accessed Feb 01 st 2005) [10] Lederman, S. J. and Klatzky, R. L. Hand Movements: A Window Into Haptic Object Recognition, Journal of Cognitive Psychology; vol 19, no 3, pp , [11] Roads, C. The Computer Music Tutorial, MIT Press, Feb [12] Proceedings of the ACM Symposium on Designing Interactive Systems (DIS'04) Aug 2004 Cambridge, MA, USA [13] Twitchell, J.B. Twenty Ads that Shook the World. Crown Publishing Group, New York, NY, USA, 2001, pp.62, ICAD05-217