Haptic Techniques for Media Control

Transcription

1 In Proceedings of the 14th Annual ACM Syposiu on User Interface Software and Technology (UIST 2001), Orlando, Florida, Noveber 2001 Haptic Techniques for Media Control Scott S. Snibbe 541 Mississippi Street San Francisco, CA USA Karon E. MacLean Dept. of Coputer Science Univ. of British Colubia Vancouver, B.C. V6T 1Z4 Canada Rob Shaw Jayne Roderick Willia L. Verplank Mark Scheeff ABSTRACT We introduce a set of techniques for haptically anipulating digital edia such as video, audio, voiceail and coputer graphics, utilizing virtual ediating dynaic odels based on intuitive physical etaphors. For exaple, a video sequence can be odeled by linking its otion to a heavy spinning virtual wheel: the user browses by grasping a physical force-feedback knob and engaging the virtual wheel through a siulated clutch to spin or brake it, while feeling the passage of individual fraes. These systes were ipleented on a collection of single axis actuated displays (knobs and sliders), equipped with orthogonal force sensing to enhance their expressive potential. We deonstrate how continuous interaction through a haptically actuated device rather than discrete button and key presses can produce siple yet powerful tools that leverage physical intuition. KEYWORDS: Haptic force feedback, user interface design, interaction techniques, tangible interfaces, edia browsing, ultiedia control, video editing. 1 INTRODUCTION We use our haptic (touch) sense of the forces acting on our bodies to perfor any everyday tasks like walking, driving, operating achinery, cooking and writing. In doing so, we interact with physical dynaic systes, whose coponents oveent is deterined by physical laws. This paper describes several approaches to using virtual, haptically displayed dynaic systes to ediate a user s control of various sorts of edia. These dynaic systes are constructed as physical task etaphors, rather than as literal representations of the edia: e.g. instead of literally rendering the content of individual fraes, we use the haptically perceived spinning of a large heavy wheel to indicate a video strea s progress. Huans are accustoed to anipulating static visual edia with physical dynaic systes: pencil and paper, brush and canvas, fingers and clay, chisel and stone. Where these edia have igrated to the coputer, we engage with a generic ouse or keyboard and have lost distinctive physical sensations. Soe research seeks to duplicate the traditional feels with positive results [2, 15]. Others have built haptic dynaic systes for drawing and sculpting that have no direct physical analog [11, 19]. Siilarly, traditional physical tools offered now-lost benefits to the anipulation of dynaic sounds and iages. Fil sound designer Walter Murch observed that the physical properties of editing echaniss and the edia itself enabled a level of control lost in nonlinear digital editing systes [13]: the duration of otion picture fil and audiotape is related to physical length or bulk, and physical arks can be scratched and re-found. The spinning ass of a gang synchronization wheel (used to create fil audio tracks) allows sooth adjustent of review speed and rapid, accurate location of a critical frae. DJs cling to vinyl for direct access to audio tracks, control over play speed and zero-latency audio response. Naiark perfored early experients to re-introduce the feel of traditional fil equipent to digital video [14]. Past haptics research has pursued the anipulative use of force feedback priarily through siulation of edical and usical instruents [4, 10]; and its ore exploratory function in the refineent of techniques for haptic rendering (creating a sensation of touching an object that doesn t physically exist [16]). The research described here deploys both knowledge bases in rendering and anipulating a ediating dynaic syste, rather than the object of anipulation itself. Here, we describe an exploration in restoring physicality to nonlinear edia, introducing a series of etaphors and techniques for anipulating digital video, digital audio and coputer graphics using haptic force feedback. We introduce the general principles of 1 Copyright 2001 ACM

2 these etaphors and docuent the hardware prototypes developed to explore the, then describe the haptic etaphors and behaviors theselves. Finally, we offer inforal observations on their use, a note on coputational architecture and reference to the continuation of soe eleents of the work. 2 DESIGN PRINCIPLES The following design principles are a cobination of core interface intuition based on past activities (e.g. as videographer or sound designer) that we brought to our research, and insights that eerged over its course. Discrete vs. Continuous Control A anual controller can be a button that triggers a discrete action or a handle that continuously odulates a paraeter. Due to price pressures, arrays of buttons doinate conteporary edia control tasks, even for continuous properties such as volue or video scrub rate. A continuous controller like a knob can provide perceptually infinitesial control for anipulating paraeters or edia. Its greater size and cost ay require it to accoodate ore functions, but well-designed haptic feedback can keep it intuitive. Inforation Delivery through Touch Visual interfaces copete for attention with visual edia; controls on border toolbars and floating palettes shrink or obscure content. Siilarly, auditory interfaces copete with audio content. Maxiizing inforation delivery through a edia-independent channel (touch) and thus reducing perceptual noise can enhance both ease-of-use and accuracy of control. Dynaic Systes for Control One can eploy direct anipulation, a fundaental tenant of conteporary interface design [6, 18], by odeling a task or process with a dynaic physical etaphor and then haptically rendering this etaphor as the process controller. We thus place an abstract tool between the application task and the user s hand: it is the leverage that a paintbrush offers over one s fingers or a fil-editing table over a knife and tape. Modeless Operation Modes necessitate a sensable or ental record of a device s current state; the forer are often not provided by electronic and coputer interfaces, and the latter increases workload and errors. Modeless interaction is achieved, however, by a consistent and trustworthy physical behavior, like that of a car steering wheel. Haptic feedback facilitates sophisticated tools for which physical intuition can be developed and inforation subconsciously absorbed. Application and Interface Counication Conflicting needs often dictate coputational separation of haptic feedback control fro application content. To achieve the low-latency force feedback that provides a sense of task presence and control, we have found it advantageous to locate the dynaic odel with the haptic controller. This in turn requires high bandwidth inter-cpu counication, in conflict with the typical design goal of independence between application and interface (Section 6). 3 DEVICES Here we present the anual controllers used in these interaction experients, largely single degree-offreedo (df) devices designed for low cost and ease of integration into ebedded contexts. Engineering prototypes are actuated, sensed and coputer-interfaced with little attention to appearance, used to evaluate a particular technology or as general-purpose developent platfors. For studies were used to explore ergonoic design and act out task scenarios. They often have oving parts, but are not actuated or coputer-controlled. Functional prototypes are working devices with a for factor relevant to their specific application, and required the greatest effort. We have developed ost of our haptic behaviors using engineering prototypes, with for studies to develop specific physical application contexts. Orthogonal Force Sensing As described elsewhere [12, 16], shading force agnitude along one actuated axis can create an illusion of force supplied along a second, orthogonal axis (Figure 1). Conversely, user-applied deflection or pressure can be easured along an axis orthogonal to the actuation. Many of our behaviors are designed to supply an actuated response to such a deflection, enhancing the illusion of a second actuated df and adding a valuable, integrated control diension. Orthogonal force can be sensed in any ways, depending on the precision required. We have eployed at one end a high-perforance force-torque sensor (Figure 2a), and at the other a variety of cheap force sensing technologies including force-sensing resisters and optical and Hall-effect easureent of sall displaceents (Figure 2b). Figure 1: Orthogonal force illusion, created by projecting the slope of a two-diensional geoetric profile (botto) onto a single-axis force profile. Arrows indicate direction and agnitude of displayed force. A user perceives the two-diensional surface as a dip (shown) or a hill (arrows reversed). 2 Copyright 2001 ACM

3 Figure 2: Engineering Prototypes. Clockwise fro upper left: Big Wheel for high quality sensing (F/T sensor and optical encoder are located in series behind otor), large diaeter knobs and a variety of orientations (otor/sensors assebly terinates in a post which can be inserted at various angles); Cheap Force Sensing Wheel, which easures displaceent of a cantilevered ount with Hall-effect and optical sensors; Brake with an encoder and particle brake linked via a belt; and the Slider for experients in absolute positioning, with a pressure sensor on the handle to select engageent with the haptic odel. Big Wheel: Multi-axis Force Sensing For behavior developent we used a powerful otor to directly drive a large-diaeter wheel while sensing both knob rotational position and the forces exerted by the user on the handle. The knob axis ay be aligned at 0º, 45º and 90º relative to the device s ount, facilitating siulation of different kinds of hand/wheel interaction e.g. edge versus top surface (Figure 2a). The device has a 90W, 490 N Maxon DC brush otor and an HP 4000-line optical encoder. The F/T sensor (ATI Mini ) easures 6 axes of decoupled force (x, y, z at N saturation and 1/800 resolution) and torque (roll, pitch, yaw at 1 N and 1/32000 resolution). Handles of various shapes and diaeters were designed by specifying hand interaction and tangential forces e.g. 4 N at the ri for a 300- diaeter platen to siulate a phonograph record. Different force easureent schees are possible; e.g. of a specific axis or the axiu fro all axes. Observing activity on all axes also helps deterine how a 1-df sensor should be ounted: if the wheel is pressed radially at its ri, the force ay be best sensed in the x or y-axis (noral to the knob shaft). Behaviors engaged by touching the top knob surface utilize z-axis force (aligned with the knob shaft). Brake: Passive Haptic Display We perfored a series of sall experients with a brake (Figure 2c). Since brakes can only reove energy fro a syste, stability is guaranteed, aking the safer and ore predictable to inexperienced users. Lower cost and power needs for a given torque ake the attractive for consuer products. However, crafting a precise haptic experience can be ore challenging: e.g., a position error cannot be corrected with closedloop control. Instead, one can synchronize application to position to create an illusion of hitting precise targets (sticky channels, Section 4.2). Slider: Absolute Positioning We constructed a device with liited range of otion (Figure 2d), affording absolute rather than relative positioning. This constraint is an opportunity to exploit uscle eory, as we do when operating a radio dial without looking: specific destinations are stationary relative to the device s base. Haptic landarks such as bups or textures further anchor locations. In a high-end audio ixing board slider, a sall otor drives the slider open-loop via a toothed belt. To eliinate cogging and obtain position readout, we replaced the original actuator with a Microo 1524 otor/encoder, geared 6.3:1 and with post-quadrature position resolution of ~1.25 cnts/; and added a handle with a force-sensitive resistor to sense squeeze pressure. Despite belt copliance, this syste (with an 85- range of otion) worked well for the experients of Section 4.2. Tagged Handles: Discrete & Continuous Control The versatility of force feedback eans that a device can change behavior while retaining the sae appearance, coproising predictability. Further, a generic handle ight not be appropriate for a given task. We developed a concept where behavior is 3 Copyright 2001 ACM

4 Figure 3: Tagged Handles. Detachable physical knobs with distinctive shape can be connected to a single haptic device (left). A functional prototype with top-ounted textural tags aids in switching functions, by pressing different tags. deterined by the handle attached to it (Figure 3, left), by cobining force feedback and tagged objects [7]. Handles are electronically tagged; a reader near the otor shaft ascertains which handle is current. Thus, the discrete selection of application behavior is coupled with the continuous control of our haptic displays. Although described elsewhere [9] we ention this concept here in the context of our edia control experients. The functional prototype used in the applications of Section 4.3 has five textured buttons ounted over FSR pads (Figure 3, right). One PIC icrocontroller sends force easured fro each button to the base via a wireless transitter; another in the base counicates serially with the host coputer. A Maxon 20W brush otor drives the wheel. Rock-n-Scroll The Rock-n-Scroll, designed for gae-like interactions where a control is held continually, is a finger-sized actuated wheel ounted on a second passive, sprung axis that swings parallel to the thub joint (Figure 4). Thus the thub both pushes down on the wheel (rocks) with deflection sensed; and rotates the wheel at its edge (scrolls) with wheel rotation sensed. Microo 1524 otor/encoders geared 6.3:1 were used for both axes, Figure 4: Rock-n-Scroll. A for study (left) of Lego and rubber bands helped to tune ergonoics and act out scenarios. A functional prototype (right) has a left/right actuated scroll axis and a non-actuated in/out rock axis for orthogonal force sensing. although only the encoder was used for the passive axis. Design specifics proved subtle, involving optiizing of diensional constraints between echanis and grip. The ain paraeters of echanis iteration were spring stiffness, outer wheel diaeter and friction on the wheel s edge. The prototype shown proved sensitive to hand size, and its reliance on thub otion raises ergonoic concern not addressed here. 4 CONTROLLING MEDIA VIA HAPTIC INTERFACES We used our devices in three categories of experients for controlling digital edia, aiing to construct odeless dynaic systes with the iediacy of realworld physical controls. Techniques for haptic navigation and control aid in navigating any digital edia strea. Haptic annotation refers to physical arking of content, by anual or autoatic processes. Finally, ethods for functional integration point towards physical fors that can incorporate a variety of etaphors and techniques into a final real-world application. In practice, we have ipleented the etaphors across several different devices, but we show each with only one or two for clarity. 4.1 Haptic Navigation and Control Haptic Clutch We can feel the road through a steering wheel and control a piano s haer action with a key. In the sae way, we can perceive and anipulate a coplex virtual odel through a single-axis wheel, and thus increase a behavior s power and expressiveness [4]. Selective engageent with the virtual odel ay require an additional user input channel, such as force or position. We built several applications on the principle of a haptic clutch. Here we siulate the clutched engageent of a concentric pair of wheels (Figure 5): the outer wheel s otion corresponds to that of the physical wheel turned by the user. Pressing down on the physical wheel engages it with the virtual inner wheel; pressing is easured with a force sensor orthogonal to the outer wheel s rotation. Both virtual wheels are odeled as inertial eleents with bups on their facing surfaces, which correspond to features in the edia. The wheels couple when the bups esh, with the anipulated wheel driving the inner one. In one ipleentation of this virtual dynaic syste, video frae rate is coupled to otion of the virtual 4 Copyright 2001 ACM

5 Figure 5: Haptic Clutch. The user selectively engages the physical outer wheel with the virtual inner wheel by pressing down, and iparts oentu to the inner wheel by spinning and releasing. With a light pressure, the user can feel bups slip by. inner wheel. If the user pushes down firly, the two wheels engage as a single rigid body that the user can rotate in either direction to shuttle between fraes. If downward force is relaxed, the released inner wheel continues to spin with iparted oentu while the real outer wheel continues to ove with the user s hand. The video s speed is thus continuously variable fro slow advance to extree fast-forward. The user can push down and shove the inner wheel to increase the frae rate, or push steadily without shoving to brake the inner wheel, with a satisfying slip as bups fly by at decreasing speeds. The ediu s physicality is thus restored the user ust exert force and dissipate the flywheel's oentu in order to stop the video. This behavior can be produced in ultiple ways; one ipleentation is described by the dynaic syste of Eq. 1, shown in Cartesian coordinates. The kineatic state of the virtual outer wheel is atheatically equated to that of the real wheel, which in turn is directly controlled by the user (i). H H xo = x Fclutch = f hsin ( 2π n( xo xi) ) Mx i i + Bx i i = Fclutch () i ( ii) ( iii) [1] F = F + B x ( iv) act clutch o o Throughout this paper, x is the easured state of the physical wheel and is nuerically differentiated for velocity. x H and x H o i are the siulated positions of the outer and inner rings respectively, expressed in noralized units (1.0 = one revolution). F clutch is the force transitted between the two virtual rings (ii). f is derived fro the easured orthogonal applied force, and controls the degree of engageent with the virtual inner ring. h is a constant defining bup height and n is fraes per wheel revolution (typically 10-30). (iii) odels the inner ring s state: ass and daping. (iv) derives M i and F act B i are its virtual, the actuator force where B o is virtual daping applied to the outer ring to increase its stability. At each tie-step, we solve this syste using Euler s ethod to deterine the new state of the virtual inner wheel. Paraeter values are sensitive to hardware as well as the desired behavior. The reader ay observe that due to the atheatical syetry of (ii), this syste initially appears unable to transfer energy to the virtual inner wheel. If we assued a stationary controlling x and a nonzero initial x while i braking, F clutch would integrate to zero. However, in reality the user s control is not rigid or unifor. The copliant interaction between hand and echanical syste induces sall bup-linked oveents of x and f, which ipart asyetry to F clutch s periodicity and a corresponding net energy transfer. Browsing digital audio and voiceail with a wheel and an absolute slider were siilar to the video experience and also brought new challenges. Audio required pitch correcting for coprehensibility at arbitrary play rates [1]. We also applied the clutch in two diensions using a Phanto: for perusing an iage, we replaced the video s frae bups with height ap based on edges, brightness or color. Siilarly, a three-diensional grid can be selectively engaged by pressing a button or force sensor ounted on the surface of a stylus. We discovered that clutching is less copatible with directediting tasks like painting and sculpting, because the clutch forces interfere with the interaction forces generated by virtual aterial laydown and reoval [19]. However, the structure iposed by the clutch ight be well suited to CAD applications. Haptic Fisheye Here we anipulate an interediate virtual odel by continuously varying haptic resolution based on the user s orthogonal pressure. As with graphical fisheye views [17], this supplies iediate access to fine and coarse details of the anipulated odel as a non-odal and continuously varying process (Figure 6). In video browsing, a strong applied force either decreases or Figure 6: Haptic Fisheye. In this version, the user presses the knob to increase the resolution of browsed edia, rather than its speed. 5 Copyright 2001 ACM

6 increases the rate of fraes passing under the user s fingers, displayed as a fine texture; the choice of polarity depends on the user s default viewing ode. As pressure changes, the rate slows, until the fraes are felt as individual ticks. Thus the user can rapidly browse an entire video and still find individual fraes or scenes by altering the applied force. The resolution-proportional-to-pressure version of the haptic fisheye can be ipleented as Equation 2: x = x K(1 f ) () i s x = ( ) s xdt s ii [2] F = hsin 2 nx ( iii) act ( π ) x H s describes the edia strea s otion: we constrain x s to x, scaled in a non-newtonian anner by the oentary orthogonal force f, then nuerically integrate. F, the coputed actuator force, displays a act sinusoidal texture related to strea rate while h, bup height and n, the nuber of detents (or fraes) per revolution, are constant. Frictionless Shuttle The frictionless shuttle is one of the few odels where the physical wheel oves when not touched by the user. Under user control, it provides evenly spaced haptic detents that correspond to video fraes. One full revolution ight correspond to 30 fraes of video. If the user lets go of the wheel while it is oving, it will continue on its own at the sae rate. If this were ipleented with a non-actuated wheel, the rate would diinish due to friction; but here we can aintain a strict correspondence between the rotation of our wheel and the advancing fraes. Thus, the user can initiate any rate fro a single frae advance to rapid fastforward. While slightly dangerous, this behavior exeplifies the type of agical behavior possible with haptic feedback in this case reoving friction fro a echanical syste. While the wheel is touched, haptic feedback is a sinusoid of fixed spatial frequency: s F = hsin(2 π nx ) [3] act Untouched, velocity is aintained at the let-go rate with PID control on error between target velocity x t and easured velocity x at release: e = x x t release () i de F = K e+ K edt K act p i + d dt ( ii) [4] We tried several ethods of pinpointing wheel release. The ost successful copared orthogonal user force over a saple window; the signal is soother in noncontact. Handle capacitance can soeties be onitored [5], but this ight constrain both continuous rotation and handle aterial. If only a position signal is available, we found we could watch for a sooth slowdown, then search back to the last occurrence of hand jitter noise and use that x. 4.2 Haptic Annotation We used haptic annotation to ark, highlight or deliit significant segents of video and audio aterial, a function useful at varying coplexity and abstraction to professionals and casual browsers alike. Most siply, we literally represent the edia s for e.g. fraes of video or teporal audio intervals. The next level can be autoatically extracted scene breaks, activity, color, brightness, location or tie. The highest level requires huan intervention to indicate qualities such as actor, ood, genre, etc. Any iconic representation requires a apping fro paraeter to sensation [3]; hapticons require a relatively abstract correspondence and are challenging to ake both perceptible and eorable. Allowing users to design their own aps, as done here, is one laborious and individualized approach. Developing a ore universal language is an ongoing project. Foreshadowing In viewing, a visual ark typically appears at the oent of annotation and inevitably is overshot, particularly with unfailiar footage or annotations. We foreshadow arks haptically by gradually increasing the aplitude of a pre-annotation before reaching the ark fro either direction. The wheel is used as a conventional spring-centered video shuttle knob, where deflection sets frae rate. A texture whose intensity gradually increased in agnitude or frequency proved ost effective (Figure 7). E.g., an annotation texture is overlaid as a vibration on the spring force, with frequency increasing as the ark approaches. A user ay ake new arks while browsing by firly pressing down on the wheel, Figure 7: Haptic Foreshadowing. As the user approaches arks in the strea, a texture is overlaid on the wheel s spring restoring force; the texture gradually rises and falls around the point of interest, alerting the viewer to the upcoing event. Users can also add arks by pressing down. 6 Copyright 2001 ACM

7 A B C B Beach Boys Blondie Bartok Beatles Bowie C Figure 8: Alphabet Browser. Hierarchically arranged audio tracks are quickly indexed without a screen using haptic and audio feedback. At high rates of rotation (above) spoken letters accopany detents; at slower rates (below), individual artists and then titles are spoken. engaging a force sensor. We tried an active haptic nudge as an alternate arking ethod, but this interfered with the dynaic act of browsing nudging is ost effective when the user s hand is static and receptive. Modifying viscosity and friction siilarly interfered with navigation. Alphabet Browser Browsing through large edia collections like MP3 recordings is tedious. Screen interfaces are the nor, but portable device screens are sall, difficult to read and can unduly divert attention. The alphabet browser uses a haptic knob with an auditory display to browse audio collections eyes-free (Figure 8), and ight be ost useful in driving or portable contexts where visual attention is least available. Turning the knob activates a spoken and felt alphabetic index. When the knob is turned rapidly, one hears the first letter fro each entry - A, B, C ; full titles eerge at slower rates. Haptic detents enhance the audio feedback, aid navigation and indicate volue of aterial under each heading: alphabet letters get strong clicks, individual titles gentle clicks that fuse with rapid rotation. We ipleented an alphabet browser for an MP3 audio player on Bigwheel (artist naes are traversed alphabetically and selected by pressing); and for a voice ail collection using the slider. We observed users adjust scroll rate continuously to control the aount of artist or caller nae revealed as the search narrowed, suggesting an optial but probably nonlinear relation between scroll speed and list traversal rate which will depend on typical entry lengths. Audio feedback alone provided soe utility, but haptic annotation seeed to iprove user s speed, accuracy and confidence of navigation as well as their aesthetic appreciation. Other application possibilities include a haptic dial integrated with a cell phone for the fast retrieval of nubers, a car audio control for radio channels and audio tracks, and eail on a wireless PDA. Sticky Channels Conventional anual interfaces such as channel-change knobs have detents at channel boundaries, but with current television reotes, users generally ust reeber a nuerical association with a channel. With active force feedback we can custoize the feel of individual detents to reflect frequency of use, genre and other characteristics. Like wagon trails, sticky channels are ruts that get worn into the haptic landscape. i = floor ( d x ) () i [5] F = h sin(2 π d x ) ( ii) act i In Equation 5, the current channel nuber i is coputed fro hand position assuing a regular spacing as an index into an array of channel detent strengths h i. We applied this construct to switching television channels, digital audio tracks and voiceail recordings, on our wheels, the tagged handle, slider and brake. In our scenarios the annotation ight be ade a priori based on popularity, genre based on station ID or predicted user preference; or a user could set favorites explicitly. The haptic cues seeed to facilitate navigation and generated positive response fro heavy TV viewers chosen outside our group. The slider provided a redundant cue of absolute position, speeding navigation, but its display set was liited. Video Carousel We extended sticky channels to a three-diensional graphical ring of TV channels for the Brake and Rockn-Scroll (Figure 9). With the brake, a channel initially fills the entire video screen with dynaic content. With Figure 9: Video Carousel with Sticky Channels. Haptic detents ark channels on a graphically displayed ring (top, zooed view). With a otor, favorite channels have a stronger attractive force; with the brake, ore friction. However, when the brake doesn t stop exactly on a frae boundary (botto), the fraes ust be slewed into alignent. 7 Copyright 2001 ACM

8 handle rotation the visual channel slides to one side with a click, and the adjacent channel slides in. We found that users appreciated channel snapping, i.e. bringing the nearest channel to the screen center on a pause. With active feedback, we corrected the resulting isatch between visual and anual position by servoing the handle into place. With the brake, we had to slew fraes into their centered position as handle velocity decreased, anticipating a full stop. At higher velocities, the visual display gradually zoos to a view of a ring of available channels. The current channel is live, while stills updated at ulti-second intervals represent the others. With Rock-n-Scroll, we used the scroll axis to change channels and the rock axis for zoo. This decoupling of functions, separating tie and velocity dependence, proved the ost popular. Absolute Media Browsing We used the slider to tie absolute position to the current position in a edia strea. For digital ovies, the slider slowly advances with the playing ovie. However, the user can pull the slider forward or back, feeling a spring connecting the handle to their current play position and gently returning to that position when let go. For voiceail, this ethod is ipleented hierarchically at the top level, the slider browses individual essages, playing the brief introduction and haptically providing inforation about the tie and iportance of the call. Squeezing the slider pops the user down into an absolute traversal of an individual essage the slider is drawn to the far left and then advances linearly with the essage. As in our video exaple, the user can pull the slider forward or back to review, fast forward or listen slowly. Squeezing again pops the application back up into ulti-track browsing. This ipleentation proved overly odal for haptically inexperienced users, who were uncertain whether they were in list or individual essage ode. While state was evident haptically with a light touch, soe grasped the handle too tightly to notice this; haptically sophisticated users found it uch ore intuitive. A siilar schee for digital usic files garnered little enthusias: casual listeners wanted to listen straight through rather than jup within a track. Super-Sapling The differing resolutions of the haptic device and browsed aterial can coplicate ipleentations. A typical strea contains thousands to illions of eleents, whereas the haptic display s position resolution is at best thousands of counts per revolution. Media eleents ust be filtered or super-sapled over the haptic servo interval to produce a suitable output [20]. To retain control over individual eleents, we coupled the physical device s position to the exact edia position with a virtual spring [16]. With encodercount-sized jups, the spring pulls the probe along the edia. The probe s position is coputed using real nubers, and is thus not aliased at the encoder resolution and has the well-behaved derivatives crucial for ultirate display of digital audio. 4.3 Functional Integration With two final projects, we aied to prototype a coplete device concept bringing together diverse functionality in a sealess and odeless anner. Tagged Handles We designed a suite of five behaviors for the tagged handle wheel to prototype a broadband universal reote; our goal was to provide a consistent tactile interaction across disparate edia. Buttons on the wheel s base select digital edia target (e.g. audio library, TV or video-on-deand), while the functions applicable to those targets (e.g. sticky channel behavior or frictionless shuttle) are assigned to the textured pads on the wheel s face. When the edia target changes (fro TV channels to audio tracks), etaphor and haptic feedback do not. In general, users found this ethod of applying browse tools to different edia genres intuitive. However, the device itself was unsuccessful because it required physically or visually searching for a tag on the rotating knob face. Ipleenting the sae classes of behavior on a side-ounted wheel [9] sees proising. Preview Button We found the preview button to provide one of the ost intuitive ways to cobine discrete and continuous control. We installed pressure sensors on the surface of noral pushbuttons, allowing the user to preview the button s action before coitting (Figure 10). For exaple, when using a row of preview buttons to select radio stations, a light touch gently fades the sound up Figure 10: Preview Buttons. This engineering prototype shows buttons retrofit with force sensors, so that a light touch can provide a preview of a button s behavior for exaple fading up an audio track, or gradually enlarging a picture-in-picture. 8 Copyright 2001 ACM

9 while the previously selected channel continues if it passes uster, a fir push engages the track. The preview button can also be used for TV picture-inpicture so that a video inset grows based on the pressure, providing a preview of an alternate channel. The preview button can be outfit with haptic feedback, so that the preview is textural haptic annotation representing the genre of edia or a coarse representation of an audio signal. 5 OBSERVATIONS These prototypes have been used by ourselves and ~50 fellow researchers over the project s 12-onth life; frequent, inforal sessions with regulars, novices, enthusiasts and skeptics fored a crucial aspect of our iterative design approach. In-depth studies in ost cases would have been preature, but we nevertheless obtained critical insights that integrate the saller lessons interspersed in the previous descriptions. These dynaic syste etaphors proise to give users functional integration together with siplification. Integration worked well, for exaple, with tagged handles, and people adapted easily to a ultiplebehavior odel. It is abundantly clear that we need to know ore about haptic language: the perceptibility, salience and associability of coplex haptic signals. Hand-crafting and quality of haptic experience were essential to our techniques acceptance, with any users siply enjoying the feel of the tools theselves. Textures generally worked better than forces for ephasis and annotation. Varying copliance, viscosity or inertia was less salient than, e.g., noise frequency. With the clutch and fisheye, which use textural arks, users were able to rapidly locate individual fraes. Careful physical and visual affordance design is critical for these close-coupled applications. We had to explain where and how to interact with prototypes ipleented on the general-purpose platfors, and errors were coon. Physically custoized versions often eradicated these probles. Copliantly ounted haptic displays reduce the ipact of changes in texture and feature size. Soe of Rock-n-Scroll s applications suffered fro the rock axis absorbing subtle haptic signals and reducing its controllability. The type and aount of haptic feedback to include in a coplete syste reains an open question. Balancing its liitations, we did find that passive force feedback eliinated fear and surprise fro soe novice users. Certain etaphors worked better with the brake because its features are so solid. Stickiness seeed to register subconsciously for soe, who found theselves stopping on favorites without knowing why. 6 COMPUTATIONAL ARCHITECTURE Our applications ran on two-cpu systes of 1998 vintage, with one processor dedicated to haptic feedback while the second anaged the edia. The haptic server was a PC-based syste running QNX, a real-tie UNIX. The edia server was a PC with a digital disk recorder for video experients, an SGI O2 for three-diensional video, or an Apple running MAX for digital audio. Inter-CPU counication eployed a custo RS-232 serial protocol that proved siple, reliable and just fast enough. The haptics software used a custo architecture [8]. Requiring an environent where a non-prograer could rapidly prototype custo dynaic systes, we created a syste that could hide details of scheduling, counication, thread and device anageent but register a callback function (one line to half a page) to ipleent the low-level haptic odel at as low a level as desired. This approach is at odds with current trends in coercial haptics software architectures, where developers use a high-level toolkit of priitives such as springs and boundary-representation but cannot odify ipleentation. As harsh as the econoic constraints are, we believe that for haptics to becoe successful in the ass arket, such hand-crafting is necessary. 7 CONCLUSIONS AND FUTURE WORK With Interval Research s ipending deise, this work was curtailed as it approached a full ripening. In soe cases, such as the Tagged Handles and Rock-n-Scroll, we iterated partway to a solution and felt close to an optial for. In others, such as the Slider, we have only conceptual sketches showing a device in the side of a cell phone or reote control physical prototyping is essential to see if these for-factors would really be pleasing and practical. Soe concepts continue in altered contexts (e.g. driving controls and edia browsing) in current projects at UBC. Our own exaination of the cost and feasibility of ebedded ipleentations of our techniques cobined with evidence of other products shipping in 1999 deonstrate that ebedded haptic feedback will be in our future; power requireents are the greatest challenge, particularly for portable displays. We are investigating novel power schees specifically for haptic displays. We also believe that it is possible to copletely encapsulate techniques such as the fisheye or clutch into a general-purpose haptic device such as a ouse. In this case only high-level inforation need pass to the application and these techniques could becoe part of a coodity product requiring no special counication to the host. In designing haptic edia controllers, we want to axiize both the rate and vocabulary of inforation 9 Copyright 2001 ACM

10 transfer. What types of forces and agnitudes can be cobined without interference, capture or blocking? What do (or could) sensations ean to users? The developent of a haptic language and of a flexible, ultiodal realtie control platfor is a foci of interdisciplinary work at UBC. Finally, soe of these techniques ay ultiately be redesigned in a passive echanical for, eliinating the need for powered devices and opening the door to portable devices. ACKNOWLEDGMENTS This paper describes uch of the final 12 onths of work by the haptics tea at Interval Research in Tina Blaine, Cy de Groat, John & George Kebel, Caille Norent, Ti Perkis, Terry Winograd and Tricia Wright helped with ideas and early prototypes. Ki Johnson and Oliver Bayley ade early contributions to the etaphors for arking and browsing edia. Jesse Dorogusker, John Ananny, Brad Niven and Lee Felsenstein helped build ost devices. REFERENCES [1] M. Covell, M. Withgott, and M. Slaney, Mach1: Nonunifor Tie-Scale Modification of Speech, Proc. of the IEEE Int'l Conf. on Acoustics, Speech, and Signal Processing, [2] T. A. Galyean and J. F. Hughes, Sculpting: An Interactive Voluetric Modeling Technique, Proc. of SIGGRAPH, vol. 25(4), pp , [3] W. Gaver, Auditory Interfaces, in Handbook of Huan-Coputer Interaction, M. G. Helander, T. K. Landauer, and P. Prabhu, Eds., 2nd ed. The Netherlands: Elsevier Science, [4] B. Gillespie, The Virtual Piano Action: Design and Ipleentation, in Proc. of Int'l Coputer Music Conf., Auhus, Denark, pp , [5] K. Hinckley and M. Sinclair, Touch-Sensing Input Devices, in Proc. of ACM CHI'99 Conference on Huan Factors in Coputing Systes, [6] E. L. Hutchins, J. D. Hollan, and D. A. Noran, Direct anipulation interfaces, in User Centered Syste Design: New Perspectives on Huan- Coputer Interaction, D. A. Noran and S. W. Draper, Ed. Hillsdale, NJ: Lawrence Erlbau Associates, [7] H. Ishii and B. Uller, Tangible Bits: Towards Sealess Interfaces between People, Bits and Atos, in Proc. of CHI '97, Atlanta, GA, [8] K. E. MacLean and S. S. Snibbe, An Architecture for Haptic Control of Media, in Proc. of the 8th Ann. Syp. on Haptic Interfaces for Virtual Environent and Teleoperator Systes, ASME / IMECE, Nashville, TN, DSC-5B-3, [9] K. E. MacLean, S. S. Snibbe, and G. Levin, Tagged Handles: Merging Discrete and Continuous Control, in Proc. of CHI '2000, The Hague, Netherlands, [10] A. Madhani, G. Nieeyer, and J. K. Salisbury, The Black Falcon: A Teleoperated Surgical Instruent for Minially Invasive Surgery, in Proc. of IEEE/RSJ Int. Conf. on Intelligent Robots and Systes (IROS), Victoria, BC, Canada, [11] T. Miller and R. C. Zeleznik, The Design of 3D Haptic Widgets, ACM Syposiu on Interactive 3D Graphics, pp , [12] H. B. Morganbesser and M. A. Srinivasan, Force shading for shape perception in haptic virtual environents, in Proc. of the 5th Ann. Syp. on Haptic Interfaces for Virtual Environent and Teleoperator Systes, ASME/IMECE, Atlanta, GA, DSC:58, [13] W. Murch and F. F. Coppola, In the Blink of an Eye : A Perspective on Fil Editing. Los Angeles: Silan-Jaes Press, [14] M. Naiark, Moviap and a 3D Panoraa, SPIE, vol. 3012, pp , [15] W. Plesniak and R. Pappu, Coincident Display Using Haptics and Holographic Video, in Proc. of CHI'98, [16] D. D. Ruspini, K. Kolarov, and O. Khatib, The Haptic Display of Coplex Graphical Environents, in Proc. of SIGGRAPH '97, Los Angeles, CA, pp , [17] M. Sarkar, S. S. Snibbe, O. Tversky, and S. Reiss, Stretching the rubber sheet: A etaphor for viewing large layouts on sall screens, in Proc. of UIST, pp , [18] B. Schneideran, Direct anipulation: A step beyond prograing languages, IEEE Coputer, vol. 16, 8, pp , [19] S. S. Snibbe, S. Anderson, and W. L. Verplank, Springs and Constraints for 3D Drawing, in Proc. of the 3rd Phanto Users' Group, Dedha, MA, MIT AI Lab TR #1643, [20] J. Watkinson, The Engineer's Guide to Motion Copensation: Petersfield: Snell & Wilcox, Copyright 2001 ACM