Haptic messaging Katariina Tiitinen 13.12.2012
Contents Introduction User expectations for haptic mobile communication Hapticons Example: CheekTouch
Introduction Multiple senses are used in face-to-face communication. Touch is also a big part, and it can be used to express emotions, intimacy between people, and things that cant be put into words. Current technologies mostly rely on audio or text, so users have learned other means for communicating their emotions (e.g. using emoticons). Haptic modality is seen as a possibility to create more richer and emotional communication between users over distance.
Introduction Mobile communication is often done while simultaniously performing other tasks (walking, driving, listening to other people). These cognitively challenging situations are competing for our limited attention, which makes it hard and even dangerous to use mobile devices that are heavily based on visual attention. Vibration in mobile phones is currently used mainly for alerting or informing purposes. If more information could be represented via haptics, cognitive resources could be saved for more important tasks.
User expectations for haptic mobile communication 1/9 When developing new interaction techniques and applications, it is important to understand the needs and expectations of the users. Heikkinen et al. [2009] have addressed this concern with a user study to research the needs and expectations of users for person-to-person haptic communication using a mobile device. The study was conducted by seven focus group sessions with heterogenous (no specific application area represented), and homogenous groups (an expected common need for communication technology). Before the focus group sessions participants answered a background questionnare measuring: the level of technical orientation of the participant, the use of touch in communication, and communication of feelings via mobile phone.
User expectations for haptic mobile communication 2/9 Participants were moderately technically oriented, touch was considered to be an important part of communication with others, and it was common to communicate feelings using a mobile phone. No prototype of a mobile haptic device was presented in the study. Possible use case scenarios were used as the base for discussion in the group sessions. Communication situations that have a high cognitive load, group communication, private hidden communication, communication of emotions, and private communication while gaming. Unimodal (only haptics) and multimodal (all possible modalities) situations.
User expectations for haptic mobile communication 3/9 An example scenario used in focus groups. [Heikkinen et al., 2009]
User expectations for haptic mobile communication 4/9 Haptic communication was mostly considered to be an interesting consept that could bring something new to communication over distance. Touch was considered to be a very natural way to communicate, but it was not expected to be used very often. The participants felt like touch is the most initmate way of communicating, and they would like to share it with only certain people. Users need to be able to control who can communicate with them via haptics.
User expectations for haptic mobile communication 5/9 Based on the different scenarios, the participants found binary information (yes/no answers) and emotional content to be most suitable for unimodal haptic communication. Group communication or more complex information weren't liked as much, because of the possible confusion and the need for special agreenments between the participants, or even learning a new language (Morce code). It was also seen as a possibility that a new kind of language or symbolism (e.g. hapticons) could evolve when haptics are embraced in everyday communication. By offering some preset haptic gestures, like easy to understand binary options, a lot of information could already be conveyed. For the haptic language to evolve, the system needs to support a rich way to create and edit haptic information, not just to rely on predefined templates.
User expectations for haptic mobile communication 6/9 Added richness and immersion were seen as great possibilities in haptic communication. It was emphasized that haptics could add spontaneity and quickness to communication, while also being easy, simple and fun. Beacuse it is difficult not to feel physical stimulation, immediate responses could be expected. It was also considered that there should be feedback whether the haptic message has been received or not. While it was said that haptics could be used to send emotive and private messages between loved ones, it was also brought up that it can't replace actual closeness and intimacy. Some participants were afraid that haptics could open up new ways for bullying and harrasment, or that it could even alienate people from face-toface communication.
User expectations for haptic mobile communication 7/9 Physical interaction between the device and user is a very big part of user experience. The participants thought that it would be best to mimick real-life communication situations: stroking, poking and shaking were seen as best ways to interact with the device, and vibration, touch-like pressure, and temperature changes the most apropriate feedback from the device. It was important that the gestures are created by using the same modality they are received, and that the device has to be able to sense and display the intensity of touch. Fluency of making haptic gestures and privacy were important to the participants. Large movements were felt inapropriate in public situations, and more subtle ways of creating messages was preferred.
User expectations for haptic mobile communication 8/9 When talking about the physichal appearance of the haptic device, a watch-like solution was mentioned most often. Ideal, because it can be worn all the time and both hands are left free. Placing actuators on the wrist has been studied before, and it's feasible. Hand held haptic devices were also mentioned, often as an added feature in future mobile phones. Not ideal, because it requires the device to be actively held. Could be easily implemented and accepted by the users.
User expectations for haptic mobile communication 9/9 Some preferred other wearable solutions, that could produce haptic feedback to the whole torso. A more extensive and immersive experience while leaving the hands free. Not very transferrable. Participants came up with the idea of tactile stickers, that could be placed freely anywhere in the body. Placement of these tactile feedback elements could bring new possibilities and variety for the communication. It would allow the user to choose the preferred place for the stimulation, and you could even use different actuators to represent different contacts. Head mounted solutions also came up a few times, but the response was quite mixed.
Hapticons 1/4 Rovers and van Essen [2004 1 ] present a framework for haptic emoticons, hapticons, that could contribute to overcome the loss of subtle non-verbal elements in communication. The Haptic Instant Messaging (HIM) framework combines communication of textual messages with haptic effects and hapticons. Hapticons are small programmed force patterns that can be used to communicate a basic notion in a similar manner as ordinary icons are used in graphical user interfaces. Hapticons consist of different vibration patterns that are specified by frequency, amplitude, duration etc. When these patterns are recognised and associated with a special meaning, a hapticon is created.
Hapticons 2/4 The predefined hapticons are based on the intuition of writers, and thus lack the research base for interpreting emotions. Rovers and van Essen [2004 2 ] have also created a tool for designing new hapticons based on an existing library. By giving the users a possibily to create their own hapticons could lead to more descriptive solutions, and even create a new language for using haptics. Some emoticons and proposed hapticons by Rovers and van Essen [2004].
Hapticons 3/4 One of the main problems with using hapticons with IM, is the timing when they should be played back to the receiver. Impossible to know when the user is looking at the message, or the particular words connected to the hapticon, without using other techniques like eye-tracking. Possible solutions presented by Rovers and van Essen [2004 2 ]: Playing just one background vibration that indicates the mood of the whole message. Using some sort of visual pointers that indicate the text that is being read. To use hapticons as attachments that need to be activated by the user. These solutions would still restrict the use of hapticons and would effect the spontaneity of using haptics.
Hapticons 4/4 Another approach to hapticons by Hunter Sebresos: http://concepthunter.com/touch/hapticons.html Using texture to convey mood of the message. Textures which correspond to certain emotions [Sebresos, 2011].
Example: CheekTouch Park et al. [2010] have presented a novel interaction technique that combines tactile feedback and multi-finger input while using a mobile phone. The tactile feedback is delivered through the cheek on the recipients phone, and the input is produced using a touch screen on the back of the phone. This allows tactile feedback to be sent spontaniously while talking on the phone. The cheek is a good medium for haptics, because it is naturally in contact with the mobile phone, and it has rich reseptors that are well suited for detecting haptic gestures. While holding the phone, the index and the middle finger are left free for making the gestures.
Example: CheekTouch 1/5 Touching someones cheek is a very intimate gesture and it can have different meanings depending on who does it (touch by a stranger can feel aggressive and obtrusive, while between lovers or close friends it can present more positive and playful affection). There are also cultural differences how a touch is experienced. Generally touch can be used to give encouragement, express tenderness and to give emotional support in non-verbal communication. For the prototype of CheekTouch, six type of touch behaviours were selected from Argyle's definition of 16 kinds of touch behaviours, that are considered to be most common among human communication.
Example: CheekTouch 2/5 Type of touch Patting Slapping Pinching Stroking Kissing Meanings in social interaction Comfort, Love, Farewell, Want for concentration Congratulate, Hard joke, Attention, Emphasize Tease, Playful, Intimacy Encouragement, Farewell, Love Love, Friendship, Appreciation, Farewell Tickling Tease, Intimacy, Love Selected touch behaviours and their meaning [Park et al., 2010] A different vibrotactile feedback and a touch gesture was designed to match these six touch behaviours.
Example: CheekTouch 3/5 Park et al. [2010] conducted a two part preliminary user study on CheekTouch to evaluate the usability of the presented touch gestures, and to examine if the mappings between the vibrotactile feedback patterns and touch gestures was appropriate. The twelve participants (6 male, 6 female) were first introduced to the gestures by showing a picture of the proposed method for touch input, and then asked to express each gesture three times while thinking about the scenario connected to the gestures. After, the participants were asked 13 questions concerned with: usefulness, ease of use, ease of learning, satisfaction and intention of future usage of the proposed gestures. In the second part of the study, the six touch behaviours were randomly expressed three times by the experimenter, and the participants task was to recognize them.
Example: CheekTouch 4/5 The usability results of the first part of the study were generally positive. Most participants found making of the gestures easy to learn, but three reported pinching behaviours to be quite difficult. Intention of future usage was clearly higher among the female participants, and in all females responded slightly more positively in all categories. Nine participants expressed the need for more touch gestures instead of the limited six options. The results of the second part of the study showed that kissing, tickling and slapping were most easily recognizable. The gestures for pinching and stroking found out to be hard to distinguish. It was reported that the combination of sound and vibration while using the tickle gesture felt like tickling with fingers.
Example: CheekTouch 5/5 CheekTouch has the possibility to add the needed intensity and enlivenment to the mobile communication reported by Heikkinen et al. [2009]. The downside of this solution is that the device/mobile phone has to be actively held, and in the presented example it needs to touch your cheek. There is also the possibility to place the phone, or separate actuators, somewhere else on the body, where vibrotactile stimulation can be easily felt. Prototype of CheekTouch [Park et al., 2010]
References [Heikkinen et al., 2009] Jani Heikkinen, Thomas Olsson, and Kaisa Väänänen-Vainio-Mattila. 2009. Expectations for user experience in haptic communication with mobile devices. In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '09). ACM, New York, NY, USA,, Article 28, 10 pages. [Park et al.. 2010] Young-Woo Park, Chang-Young Lim, and Tek-Jin Nam. 2010. CheekTouch: an affective interaction technique while speaking on the mobile phone. In CHI '10 Extended Abstracts on Human Factors in Computing Systems (CHI EA '10). ACM, New York, NY, USA, 3241-3246. [Rovers and van Essen, 2004 1 ] A.F. Rovers and H.A. van Essen. 2004. HIM: a framework for haptic instant messaging. In CHI '04 Extended Abstracts on Human Factors in Computing Systems (CHI EA '04). ACM, New York, NY, USA, 1313-1316. [Rovers and van Essen, 2004 2 ] A.F Rovers and H.A van Essen. Design and evaluation of Hapticons for enriched Instant Messaging. In Proceedings of EuroHaptics'04, 2004, Munich, Germany. [Sebresos, 2011] Hunter Sebresos. Mobile hapticons. 2011. Available at: http://concepthunter.com/touch/hapticons.html
Thank you! Questions?