Gestural Interaction With In-Vehicle Audio and Climate Controls

PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 54th ANNUAL MEETING - 2010 1406 Gestural Interaction With In-Vehicle Audio and Climate Controls Chongyoon Chung 1 and Esa Rantanen Rochester Institute of Technology, Rochester, NY 1 Now at Samsung Electronics, Seoul, South Korea Among the most distractive in-vehicle interactions are audio and climate controls. If these interactions were as easy and spontaneous as natural language, driving could be much safer. Through this research it was found that drivers preferred gestural language to voice language when the control was simple and repetitive; subsequently, gestural interactions with secondary in-vehicle tasks were investigated. Following the principle of eyes on the road and hands on the wheel, a steering wheel design with two touch pads on the wheel to recognize gestures was conceived. The physical design of the steering wheel incorporated good ergonomics and anthropometric data, while gesture stereotypes assigned to a number of in-vehicle controls were determined empirically by two experiments. The new steering wheel design does not have any buttons, which may contribute to driver distraction, yet it incorporates 19 functions through natural thumb-gestures. This compares favorably with most current steering wheel designs, which have more than 11 buttons and 13 functions on the average. Copyright 2010 by Human Factors and Ergonomics Society, Inc. All rights reserved. 10.1518/107118110X12829369836247 INTRODUCTION Modern automobiles have a myriad of manual controls for increasingly complex auxiliary systems that have the potential of distracting drivers from their primary task of driving. Car audio, climate control, and navigation systems have steadily increased in sophistication, and CD players and mp3 player interfaces are now common even in least expensive models. Manual control of all these systems requires drivers to take their eyes off the road and traffic environment, which has obvious safety implications. Pickering (2005) found that an average glance time to control a radio was 1.2 seconds; in that time a car travels over 50 ft at 30 mph. Summala, Lamble, and Laakso (1998) showed that ambient vision was not sufficient for hazard detection: response times increased significantly with increasing eccentric viewing by up to 2.9 seconds, suggesting that timely hazard detection required some degree of focal visual resources. Driver distraction has recently received long overdue attention as a major contributor in accidents. Adjusting radio, cassette, and CD players has been estimated to cause 11.4% of drivers distractions (Stutts, Reinfurt, Staplin, & Rodgman, 2001). In addition, it is best for safe driving to keep both hands on the steering wheel in case of sudden maneuvers are needed to avoid road hazards. Current automobile interfaces can be very confusing with too many functions per control. For example, the BMW 7 series driver-controlled systems have over 700 functions (Gilbert, 2004). Many controls are hard to find and even invisible. In an attempt to make in-vehicle controls better accessible to drivers and to allow them to keep their hands on the steering wheel most of the time, most recent car models have placed many controls on the steering wheel spokes as push-buttons or toggle switches. Unfortunately, steering wheels can become very crowded with nonstandard ways of grouping and assigning functionality to the buttons. A survey of eight car models (ranging from luxury to compact cars) showed that modern automobiles have on the average 11.62 buttons and 13.86 functions on their steering wheels. These values are undeniably high. To reduce driver distraction due to operation of in-vehicle systems, their control interfaces should be as intuitive and easy to use as possible. The most natural modes of communication and control are language and gestures. Speech recognition, such as Microsoft s Sync system in new Ford automobiles, has recently been introduced to consumers. Speech recognition has limitations, however, as it is vulnerable to noises, dialects, and individual voice differences. Moreover, current technology recognizes only specific words preprogrammed into the system. Gestural Control Interfaces Gesture control has yet to be implemented in automobiles but gesture interfaces have been very successful in many mobile communications and computing devices with touchscreen interfaces. Similar interfaces could also be designed and implemented for in-vehicle systems control in automobiles. Gesture recognition interaction addresses many of the problems associated with voice control and allows for reduction of both visual and cognitive load of drivers. Gestures can also be viewed as an integral part of natural language and therefore they would be easy for drivers to learn. There are two main techniques for gesture recognition, camera recognition and touch sensor recognition. Camerabased gesture recognition has serious spatial limitations, however. A camera requires certain distance to recognize a driver's gesture, but most interiors of automobiles are quite small, making it difficult to install a camera that would accurately detect and recognize gestures. Another problem is the positioning of the camera. As the majority of drivers are righthanded it seems natural that gestures could be performed by the right hand. The best location for a gesture recognition camera would therefore be to the right side of drivers. However, the camera could be confused by a passenger s gestures,

PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 54th ANNUAL MEETING - 2010 1407 as the right side of the driver could be shared with a front passenger. Touch sensor-based gesture recognition system is a good solution that circumvents the spatial limitations of camerabased systems. Touch-based gesture recognition requires no distance from drivers. There are two feasible locations for touch-sensitive surfaces that could recognize gestures, on the center console or on the steering wheel. If a touch pad is installed on the center console, however, drivers will need to take their hand off the steering wheel to operate the system, defeating the eyes on the road and hands on the wheel design principle. Therefore, the best place for touch pads is on the steering wheel. There are also two kinds of touch recognition systems, pressure and twist recognition system, and surface touch recognition system. For automobile applications, pressure points could be incorporated in any place along the steering wheel rim while a touch sensor area could be located in the hub or spokes of the steering wheel (Figure 1). This paper describes the design process for a gestural vocabulary interface for selected in-vehicle tasks. How people perceive and understand the meaning of certain most commonly used gestures was investigated. This gestural knowledge or gesture stereotypes were then applied to controls in an automobile. To simplify this analysis, only audio and ventilation controls were considered. Finally, a prototype steering wheel accommodating touch pads for gesture control was designed. Note that only the initial design process is described here. The steering wheel prototype is not functional and so no usability testing of how well this design would work in an actual driving environment has yet been conducted. PRACTICE INNOVATION There were two main considerations in the design. First, anthropometric principles and measures had to be considered in the physical placement and size and shape of the touch pads on the steering wheel. Second, the actual control gestures would have to be intuitive to the users. Anthropometric dimensions could be found in literature but gesture stereotypes had to be determined empirically. Anthropometric Considerations Recognizes pressure and twist Recognizes touch on the surface Because the steering wheel-mounted touch pad would necessarily be operated by thumbs, the anthropometric measures of hand and in particular thumbs were the take as a starting point of the design. The average length of a male thumb from crotch to end is 2.3 in, with a range from 2.1 to 2.9 in, and the average width 0.9 in, ranging from 0.55 to 1.25 in. For female thumbs, the average length is 2.1 in, with a range of 1.7 to 2.5 in. The average width of a female thumb is 0.75 in, ranging from 0.63 to 0.87 in. The lateral movement range of the thumb is from 80 degrees abduction to 45 degrees of adduction (Tilley, 1993). Figure 1. Possible locations of pressure points and touchsensitive surfaces on a steering wheel. Examples of how to use pressure and twist recognition system include a twist out to turn climate control on while two twists out would turn climate control off; a twist in could be air conditioner on and two twists in could be air conditioner off. Available mappings of gestures to pressure and twisting motions are very limited, however. A surface touch based recognition system would allow for a wider range of gestures to be used as well as have other benefits as it is incorporated into the steering wheel design. Two touch pads on the wheel would also overcome some of the problems with depth of the menu. For example, one side would be for climate controls and another would be for audio controls. In addition, the best hand position of drivers while driving is 10 2 position and therefore touch pads on the steering wheel should be located close to these locations. Purpose of the Research Experiment 1 Materials. To research gesture stereotypes of people for certain functions, a mock-up steering wheel with touch pads at the 10:2 positions was created (Figure 2). The touch pads were made out of thin white fabric, through which participants thumb gestures could be videotaped. The length of the sides of the pads was 2.5, or 95% women s thumb length. The angle between the two sides of the touch pad was 80, which is the radial abduction angle. The participants were also queried about their preferred gestures by a questionnaire. The questions concerned which side would be better for audio controls and asked the participants to draw a preferred gesture for common control actions. Procedure. Nineteen people volunteered for the experiment, 13 males and 6 females. Everyone had a driver s license. Most participants were young college students, with the exception of one person. The participants sat facing the experimenter and a video camera. The experimenter asked the participants to perform a control action on the steering wheel mock-up and they complied by making a gesture they felt was the most intuitive for the required control with their thumbs on the touch pads. The gestures could be videotaped through the fabric mimicking the touch pads. After this the participants drew gestures they made on the paper.

PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 54th ANNUAL MEETING - 2010 1408 Two problems with gestures were discovered in this experiment. One is that the participants had trouble imagining gestures in front of a camera and became nervous or rushed to figure them out. Another problem was that they used all imaginable good, easy, and spontaneous gestures to Turn the controls on and off. Hence, they ran out of ideas for good gestures for the remaining functions. last three functions, defrost, air from dashboard and air from underneath were a tap or a tap that has spatial meaning such as top, center or bottom of the touch pad. In the videotaped gestures, 10 participants gestured thumb up and down for volume up and down, and 6 of them did thumb right and left for change station to higher and lower station. However, 5 answers had directional meaning from right to left for changing radio station to higher station and left to right for changing station to lower station. Twelve people performed a tap for mute, 16 did a tap for pause Mp3 player and 14 a tap for play. Eleven people gestured thumb right from left for forward to next and 13 moved their thumbs from right to left for previous song. Ten participants did thumb up and down for temperature up and down, and 4 of them did thumb right and left for fan intensity up and down; 4 other gestured thumb up and down for fan intensity up and down. Nine participants tapped for defrost and 3 participants tapped for air form dashboard; 2 others used a tap that had spatial meaning like top and bottom. For example, for air from underneath, 3 participants used a tap of the bottom of a touch pad, one tapped the bottom left of the touch pad; and another just tapped once. Experiment 2 Figure 2. A mock-up steering wheel with touch pads for the study of gesture stereotypes. Positions of thumbs making the gestures were videotaped through the thin material in the mock touch pads. Consequently, the order of testing was changed to have participants first draw gestures on paper with time to imagine the gestures for the controls, and then perform the gestures for the camera. This arrangement helped most participants to make more gestures that were also more variable, but many still ran out of ideas for defrost and air from underneath, for example. Regardless, they were asked to just do something for each control action. Findings. The results between the drawings of gestures on paper and those performed for video were almost identical. Fourteen (out of 19) people chose right side as audio control. Over half of the participants offered same gestures for each function, with the exception of controls for fan intensity and airflows. In the drawing task, 11 out of 19 participants suggested thumb up and down for volume up and down, 9 did thumb right and left for change station to higher and lower station. Eleven participants offered a tap for mute, 9 drew a tap to pause Mp3 player, and 11 a tap for play. Even though these three functions shared the same gesture, it would work because radio and Mp3 player are different features and play and pause are opposite functions. Twelve people gestured thumb right and left for forward to next and previous song. One participant suggested using a long tap as turn on and off like on a cellular phone, which would be easy to learn and perform based on this experience. Ten participants offered thumb up and down for temperature up and down, and 6 of them did thumb right and left for fan intensity up and down. The most common answers of the Since it was possible that the relatively small touch pads used in Experiment 1 (see Figure 2) may have constrained the gestures performed by the participants and limited the variety of gestures, a second experiment was run with a new steering wheel mock-up that had larger touch pads and with a new group of participants. Materials. The length of the new mock-up touch pad was 2.7 in, longer than the average length of male thumb. The width of the touch pad now fully accommodated thumb movement of 45 adduction and 80 abduction angle. Because the participants in the first experiment were predominantly young college students, older participants were recruited for the second experiment. Fourteen people volunteered, 5 males and 9 females. Their ages ranged from twenties (one participant) to fifties. All had driver s licenses and drove daily. The paper-and-pencil questionnaire was also slightly modified for the second experiment: questions about preferences between gestural and voice command were added and the boxes for drawing gestures were eliminated not to constrain free drawing of gestures in any way. Finally, a rating scale (1-5) was added to gauge how good, easy, and spontaneous the most common gestures were. Procedure. With the exception of research material and questionnaires, the procedure was identical to that in Experiment 1. One participant only answered the questionnaire and was not videotaped. Findings. The large surfaces of the touch pads confused some participants who thought that a big surface had spatial meaning; they tried to push or touch a certain points as if pushing imaginary buttons in the pad instead of making gestures. This group as a whole could not carry out gestures as

PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 54th ANNUAL MEETING - 2010 1409 commands, having been accustomed to controlling devices with buttons for most of their lives. All 14 participants chose a conventional blinker over voice command; 8 preferred a conventional wiper control and 6 preferred voice control, suggesting a preference for gesture to voice as a control of a simple and repeated function. Eleven participants selected right side for audio controls, 2 people selected left side. Most of the gestures both drawn and performed by the second group of participants corresponded to those discovered in Experiment 1. On average, over a half of the participants preferred the same gestures as the first group. Gestures for air flow controls were most variable, as was the case in Experiment 1. Interaction needs to be consistent to the user. Switching play and stop or pause is a single tap on right side for audio control. Therefore commands of airflow from defrost to air from dashboard, from air from dashboard to air from underneath and from air from underneath to defrost is a circulating single tap. DISCUSSION: FINAL DESIGN Based on the anthropometric data, an ergonomically designed shape of a touch pad that looks similar to a piece of pie was conceived. The sides of the touch pad are is 2.3 in long, which corresponds to the average thumb length of males. The sides form an angle of 80, which is the abduction angle of thumb. Similar touch pads are placed on both sides of the steering wheel (Figure 3). FINDINGS There was much commonality in how people imagined gestures to perform common in-vehicle control tasks. However, the number of different gestures offered by the participants was limited, and some less used controls yielded diverse gesture suggestions. There were also some meaningful differences between the participants in the two experiments. Participants in the first experiment consisted of young college students in their twenties whereas participants in the second experiment were older adults. Younger people are accustomed to digital devices, especially touch screens and pads and were very good at making various gestures on touch pad. The gestures gathered from the second experiment were less variable; some of them tried push imaginary buttons in their minds. All participants seemed to run out of imagination for gestures for airflow control. In both experiments participants sometimes used their thumbs as if drawing a gesture on the touch pad. From these results most common gestures could nevertheless be identified and used for assigning controls to gestures on a touch pad. These gestures for the right side, or audio controls were: Figure 3. A technical drawing of the steering wheel design. The dimensions are in inches and the angle between the two sides of the touch pad in degrees. Tap and hold: Audio system on/off Up/down movement: Volume up or down Left/right movement: Radio station, previous or next A single tap, or index finger on backside: Radio to Mp3 player switch Left/right movement: Previous or next song A single tap: Pause/play For the left side, or climate controls the gestures were: Tap and hold: Climate system on/off Up/down movement: Temperature up or down Left/right movement: Fan slower or faster A single tap: Defrost A single tap: Air from dashboard A single tap: Air from underneath. Figure 4. Computer rendering of the final steering wheel design. The two touch pads on the wheel appeared to surround and wrap the wheel. This appearance was carried over as an overall design concept, which underwent several iterations. The first draft had two spokes on wheel, which started from the pads. To differentiate the design from others, a one-spoke

PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 54th ANNUAL MEETING - 2010 1410 design instead of the two-spoke design was adopted. This initial steering wheel design was full circular. However, to prevent the two touch pads, which were protruding into the inside of the wheel, from occluding the instrument cluster, the steering wheel was stretched laterally by one inch. The final design is depicted in Figure 4. On the steering wheel, most automobiles have average 11.6 buttons and 13.7 functions. In this design there are no buttons. Consequently, drivers do not need to look for grouped and small buttons. Audio and climate controls cause the most drivers distractions among the secondary in-vehicle tasks. If these control interactions were as easy as our daily language, it would be very easy for drivers to operate invehicle tasks. Eyes on the road and hands on the wheel is the maxim for safe driving. Automobile interactions must satisfy this for safe drive. This imposed the least visual and cognitive load when controlling of in-vehicle systems. The proposed gesture-based interaction satisfies all the requirements stated above, potentially allowing drivers to drive less distracted and more safely. Note that only the initial design process has been described in this paper, culminating in a non-functional prototype steering wheel. To carry the development of this product further, many engineering problems (e.g., materials of the touch pads, their sensitivity, and gesture-recognition algorithms and software) would have to be solved. An extensive usability study would also need to be conducted to investigate how well drivers could learn the gestures mapped to various control actions and perform them reliably while driving in actual traffic environments. Despite these limitations, however, this study revealed many new and potentially significant aspects about drivers interactions with ever-increasing invehicle technologies and functions. Our research also suggests that gestural control interfaces that are already ubiquitous in many mobile communication and computing devices (e.g., socalled smart phones, Apple s iphone, ipod, and ipad, and most laptop computers) can find wider applications in automobiles. ACKOWLEDGMENTS This paper is based on the first author s Master of Fine Arts thesis for the graduate Industrial Design program at the Rochester Institute of Technology. Many thanks are due to his chief adviser, Prof. David Morgan for his help and support throughout this program. Dr. Rantanen served as an associate adviser in the thesis committee. The helpful suggestions by a third thesis committee member, Dr. Michelle Harris, are also gratefully acknowledged. REFERENCES Tilley, A. R. (1993). The measure of man and woman. Human Factors in Design. New York: Henry Dreyfuss Associates. Pickering, C. A. (2005). Interacting with the car. IEE Computing & Engineering, 16(1), 26. Summala, H., Nieminen, T., & Punto, M. (1996). Maintaining lane position with peripheral vision during in-vehicle tasks. Human Factors, 38 (3), 442-451. Stutts, J.C., Reinfurt, D. W., Staplin, L., & Rodgman, E. A. (2001). The role of driver distraction in traffic crashes. AAA Foundation for Traffic Safety. Washington, D.C. http://www.aaafoundation.org/pdf/distraction.pdf Gilbert, R. K. (2004). BMW i-drive. INFSCI 250. Pittsburgh, PA: University of Pittsburgh