Designing & Deploying Multimodal UIs in Autonomous Vehicles Bruce N. Walker, Ph.D. Professor of Psychology and of Interactive Computing Georgia Institute of Technology
Transition to Automation Acceptance We must provide drivers with information to make automation more transparent Vehicle decision making Vehicle situation awareness Acceptance and trust is required for us to benefit from the improved safety and traffic management of automated vehicles.
Automated Driving Displays Focus thus far has been presenting system performance information Reliability at levels 2-3. At Level 5, system performance information is not as helpful. What information is important?
Multimodal User Interfaces: Think Different(ly) Visual Auditory Tactile/haptic etc.
Sidebar: Auditory Interface Elements Speech Non-speech: Music Spindex Spearcons Alerts & warnings
Taxonomy of Sounds Alarms, Alerts, & Warnings Object, Item, & Status Indicators Auditory Menus Data Representation & Exploration 3D Audio Displays & Spatial Representations Soundscapes & Ambient Audio Arts & Entertainment
Think Differently Deeply: Auditory Menus Computers, phones, etc., have a menu of functions or options Simple or complex 1-Dimensional or hierarchical
Visual Menus are Sophisticated Menu Title Menu Item (grayed out) Divider Menu Item Shortcut Icon Submenu Dialog Box
Typical Auditory Menus = TTS Text-to-Speech (TTS) Renders text contents only Missed opportunity for rendering other details Slow Low ratings Allegra Seidner Print Preview
TTS Enhancements to Consider Change TTS to render available vs. unavailable items Normal vs. quieter Male vs. female Normal vs. whisper Performance & preference Whisper, same voice Print Preview Print Preview
Spearcons Brief sounds used to represent the menu item, before the TTS Generated from TTS Performance & preference Faster, more accurate Easier to learn Preferred over earcons, auditory icons, TTS-only Allegra Seidner (TTS) Allegra Seidner Deborah Joseph
Auditory Scrollbars Represent: Size of menu Location within it Performance & preference Faster navigation Better awareness of menu size Better situational awareness
Spindex Speech-based index Letter sounds ( A, B, etc.) Provides location in alphabetical list Like tabs on a reference book Performance & preference Especially in long lists Basic Type Attenuated Type Decreased Type Minimal Type
Case Study 1: Multimodal Situation Awareness Vision is occupied by driving or by not driving Audio can provide SA in either case Combine music, audio alert, lighting, HUD cues Think outside the box!
Vigilance Display
Case Study 2: Automation Reliability Displays Automation monitors its own status Convey reliability to driver? Or required driver engagement? Visual display or audio or multimodal?
Reliability or RDE? Multimodal?
Case Study 3: Personalized Displays in AD Defensive Driving profile I want to arrive as safely as possible! Transit profile Get me there as efficiently as possible. ETA, Time to arrival, fuel economy, route, impact Thrill-Seeking profile Driving as a hobby, fun, not a chore
Defensive Driving Provides insight into: Near navigation Recognized vehicles Following distance Speed
Transit Provides insight into: ETA Time to arrival Speed Fuel economy Route
Thrill-Seeking Provides insight into: Speed RPM Immediate lane change maneuvers Tire pressure
Summary of Multimodal Displays in AD Automated driving presents new tasks, requires new displays and controls Multimodal UIs can help with Situation Awareness and Trust Creative solutions and careful evaluation are crucial
Thank You! Bruce N. Walker Sonification Lab Georgia Tech Atlanta, USA http://sonify.psych.gatech.edu bruce.walker@gatech.edu