Measuring emotions: New research facilities at NHTV Dr. Ondrej Mitas Senior lecturer, Tourism, NHTV
experiences are key central concept in tourism management one of three guiding research themes of NHTV
emotions make experiences valuable brief, intense reactions to specific things complex (mind, body, behavior) emotions are the value in tourism experiences fuel memory, (re)purchase decisions, loyalty, reputation, health
Emotion mangement Which emotions do you want which customers to feel at which touchpoint? Emotion measurement Which emotions do which customers actually feel when? Experience Marketing Using the right emotion to that triggers decisions Experience Design Intervening with people, interfaces, and setting to match the promised experience
Emotion mangement Do you plan long enough Which emotions do you want which into the future? customers to feel at which touchpoint? Emotion measurement Are you really open to Which emotions do which customers finding out? actually feel when? Experience Marketing Do you add value Using the right emotion beyond price? to that triggers decisions Experience Design Is there anything you Intervening with people, interfaces, can actually and setting change? to match the promised experience
Emotion mangement Which emotions do you want which customers to feel at which touchpoint? LEGO Serious Play workshops: Facilitation to inspire and formulate customer experience statements Contact: Xander Lub
CRADLE: Digital and blended experiences Storytelling: Creating emotion from your product DNA Experience Marketing Using the right emotion to that triggers decisions Experience Design Intervening with people, interfaces, and setting to match the promised experience
Emotion measurement Which emotions do which customers actually feel when?
Self-response questionnaires + inexpensive, flexible - validity problems Narrative interviews + precise, very flexible - validity problems Facial coding + time- and emotion-specific, highly accurate - clear view of the face and expression required Neuro and bio techniques + very time-specific, accurate, also unconscious/-expressed emotions - less emotion-specific, limited by context
A laboratory at the leading edge of emotion measurement A three-part laboratory in which partners can measure, design, and evaluate experiences Mobile emotion recording Measuring emotional reactions Virtual reality simulations
Part 1: Mobile emotion recording Measure GSR (sweating), heart rate, and facial expressions using wearables and cameras Record emotional experiences during leisure/tourism, as they happen Contact: Ondrej Mitas
Tourist Info Center as an experience A mobile emotion measurement study The Gent TIC facilitates quality experiences of this beautiful medieval city Employees recently completed soft selling training, further developing the experience of visiting the information center itself Thus, an interesting context for testing and refining experience measurement tools
Using the tools of the NHTV Experience Lab Empatica E4 wristbands on 7 student mystery visitors A variety of action and professional cameras pointed at students and TIC employee FaceReader facial expression analysis software students fitted with wristbands at beginning of day one student at a time sent on a mystery visit; face also recorded for realism, students then enjoyed day based on advice
Experience lab tools reveal the Gent TIC experience Facial expressions of visitors were neutral about half of the time, especially when looking at maps; positive about a quarter of the time, especially when starting and ending the conversation Facial expression of employee was most positive when talking with visitors Heart rate often showed a local peak during conversation, suggesting positive valence (Example participant below; orange line is heart rate) Enters office Begins conversation Ends conversation
Facial expression analyses show the importance of the beginning and end of the visit these are most positive 0.35 Valence (positivity) 0.3 0.25 0.2 0.15 0.1 0.05 0-0.05 first quarter second quarter third quarter fourth quarter participant 1 2 3 4
Higher recommending visitors are more positive at beginning (TIC and Gent in general) Facial expression positivity of more and less Gent (the city in general) recommending visitors 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 first quarter second quarter third quarter fourth quarter 0.35 0.3 0.25 0.2 0.15 0.1 Facial expression positivity of more and less TIC recommending visitors recommend==8 recommend==9 0.05 0 first quarter second quarter third quarter fourth quarter recommend==8 recommend==10
Part 2: Measuring emotional reactions Measure high-quality EEG (brain activity), GSR (sweating), and heart rate Accurate picture of physical aspects of emotional responses Neuroscience techniques for testing marketing Contact: Marcel Bastiaansen
What is Electroenceophalography (EEG)? Electrodes placed on the scalp in an elastic swim cap Amplifier strengthens the small changes in electrical potential due to brain activity Responses in this signal are averaged over many pictures and participants to show differences
When averaged over many pictures, differences due to emotion show up This difference shows valence (+/-) vs. emotionally neutral This difference shows strongly arousing vs. less arousing Time: 0 Time: 1 second We use facial coding or electrodes to determine which emotion Building AI algorithms at EUR to classify desire emotion Be careful with unsupported promises of commercial firms
User generated and commercial photos in Zeeland An in-lab emotion reaction measurement study Zeeland destination marketing organization traditional marketing with TV commercials (Land in Zee) and stock photos, as well as usergenerated photo and video competitions (Fan van Zeeland) Strategic decisions among these approaches? Are TV commercials still effective, and are user-generated or stock photos more powerful?
EEG EEG Using the tools of the NHTV Experience Lab Consumer neuroscience approach using EEG Experimental design with different combinations of commercials and pictures Group 1 TV commercial Zeeland UG pictures of Zeeland Stock pictures of Zeeland UG pictures of Zeeland Stock pictures of Zeeland Group 2 TV commercial Honda UG pictures of Zeeland Stock pictures of Zeeland
Experience lab tools reveal the emotional impact of TV commercials Before Zeeland commercial After Zeeland commercial
Component 3: Interaction with technology Use VR and eye tracking to design and evaluate virtual or simulated experiences Combine with EEG or wearables for accurate emotion measurement Contact: Marnix van Gisbergen
How we experience VR A virtual reality movie study (for PhD of Wim Strijbosch) For marketing and experience design in attractions such as amusement parks, a fundamental question is how people remember experiences Specifically, how do we pick experiences from our stream of consciousness to remember?
Using the tools of the NHTV Experience Lab Watching two versions (one suspenseful) of a VR film while EEG, skin, and heart rate signals are recorded Virtual reality stimulus Retelling the experience Self-report on retold experience 50 participants Virtual reality stimulus susp. Retelling the experience Self-report on retold experience
Experience lab tools reveal the VR experience
Experience lab tools reveal the VR experience Correlating self reported emotions to physiology to refine physical detection of emotions
Experience lab tools reveal the VR experience Comparing physical signals at key moments in film to determine which emotions mark beginning and end of experience
Manage customer experiences with us Excellent customer experiences require managing, measuring, and designing for emotions The future of emotion measurement is physical Our lab offers the most up-to-date emotion measurement and management process in leisure and tourism
To the demo Ondrej Mitas ondrejmitas@gmail.com Wilco Boode boode.w@nhtv.nl