Visual and auditory feedback to improve touchscreen usability in turbulence

Transcription

1 Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting 89 Visual and auditory feedback to improve touchscreen usability in turbulence Yuzhi Wan, Julie C. Prinet, and Nadine Sarter University of Michigan Touchscreens are being introduced to various mobile environments that are, at times, affected by vibrations and turbulence, such as modern car cockpits or flight decks of commercial and military aircraft. To assess and enhance the usability of touchscreens in these domains, this experiment examined the performance effects of turbulence on two flight-related tasks and the effectiveness of visual and auditory feedback for supporting error detection, fast completion times and multitasking. Nineteen pilots performed a flight plan entry and a checklist task in calm and turbulent conditions during manual flight and on autopilot. Results show that unaided performance suffers greatly in turbulence, both in terms of the number of errors and completion time. However, visual and auditory feedback both helped reduce these performance costs by improving error detection and multitasking. Participants preferred auditory feedback for text entry during manual flight and in turbulence. The findings from this study can inform the design and evaluation of touch screens for mobile environments, such as the flight deck, ambulances and surveillance operations. Not subject to U.S. copyright restrictions. DOI / INTRODUCTION Over the past decade, there has been growing interest in introducing touchscreens to various mobile environments, such as modern aircraft operating in the Next Generation Air Transportation System (NextGen). While an abundance of devices and software is available on the market, a limited number of usability studies have been conducted to assess the benefits and potential problems of touch-based cockpit interfaces and to determine their most appropriate design (e.g., Bonelli et al., 2013; Kaminani, 2011). Such studies are needed to address concerns regarding possible risk factors associated with touchscreens, for instance, turbulence (e.g., Bauersfeld, 1992; Dodd et al., 2014). Turbulence, which is fairly common during flight operations, creates a relative movement between the touchscreen and the pilot, thus increasing the risk of erroneous and slower inputs. This problem is amplified by the lack of haptic feedback from a touchscreen. Unlike most physical input devices, such as buttons, knobs, switches, and mice, touchscreens provide no information on shape, texture, mechanical movement or kinesthetic feel. This lack of feedback has been demonstrated to result in slower and less accurate performance (Barrett & Krueger, 1994; Wilson & Liu, 1995). In order to improve touchscreen entry speed and accuracy, a number of visual, auditory, and tactile feedback designs have been suggested. Visual feedback can take various forms, such as highlighting the component being pressed or pop-up keys on a keyboard. These designs have been shown to facilitate touchscreen usage (Park & Han, 2011) and reduce error rates (Deron, 2000). Auditory feedback includes auditory icons and speech output. Auditory icons are computer-generated sounds that simulate either the sound of a physical device, or have other assigned meanings, such as the sound of pressing on a button. Speech feedback usually consists of a verbal read-out of the input, which has the advantage of providing semantic information but is limited by the time required for the presentation of spoken language (Poppinga, Magnusson, Pielot & Rassmus-Gröhn, 2011). In general, auditory touchscreen feedback supports multitasking better than visual feedback, since most operations on the flight deck require visual attention. Previous studies have shown also that this form of feedback reduces errors and increases entry speed (Bender, 1999). Finally, tactile feedback can be created by embedding in the screen devices that generate vibrations on the finger when an input is made. Electric stimulation can also be used to simulate different textures on the screen. Tactile feedback has been shown to improve entry speed and reduce error of touchscreen usage (Altinsoy & Merchel, 2012; Kaaresoja, Brown & Linjama, 2006). Although tactile feedback has been shown to be effective in static environments, it seems less promising than visual and auditory feedback in a turbulent environment, given the unpredictable relative movement between the finger and the touchscreen. To date, studies on touchscreen feedback have focused primarily on usage in physically stable environments. The effectiveness of various types of feedback in more challenging conditions, such as turbulence, is unknown. The objective of this research is to help fill this gap by comparing the usefulness of visual and auditory feedback for supporting the performance of two different tasks - flight plan entry and checklist completion - in calm conditions and during simulated moderate turbulence. These two tasks were chosen because they involve different types of input (keyboard and checkboxes) and thus require different levels of precision. Apparatus METHODS The study was run on a medium-fidelity flight simulator in The Human-automation Interaction and Cognition laboratory (THInC Lab) at the University of Michigan. Two 24 monitors were used to display flight deck instruments; two 34 monitors were used to display the outside view; a 7" Asus

2 Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting 90 Nexus 7 tablet was used as the touchscreen. Participants could control the airplane manually, using a yoke, rudder pedals, and throttles, as well as through automation systems/interfaces. In order to simulate moderate turbulence on the flight deck, we chose a maximum amplitude of 1 inch (25.4 mm), a frequency range of 1-5 Hz, and an average acceleration of 1g (based on AIM, 2015; Hourlier et al., 2015). Turbulence was created by attaching the tablet to a Vibration Test System VG Shaker that created a relative movement between the tablet and the pilot. The unit was installed under the central pedestal panel (see Figure 1). The vibration signal was generated using a Teensy USB Board, Version 3.2, and the signal was amplified using a Crown DC-300A II amplifier before sending it to the vibration unit. to be sufficient for reducing errors and task completion time according to a study in calm conditions (Dodd et al., 2014). The task was performed with either no feedback or with visual or auditory feedback. The visual feedback consisted of highlighting the key that was pressed on the keyboard in a "pop-up" fashion (see Figure 2). The auditory feedback consisted of a verbal read-out of the letters/keys that were pressed. Figure 2. Flight plan edit task with visual feedback Figure 1. Flight simulator with touchscreen in center pedestal Participants Nineteen instrument-rated pilots (18 males and 1 female) were recruited as participants for this study. They were recruited from the University of Michigan and through two Ann Arbor-based aviation clubs, Michigan Flyers and Solo Aviation. Three of the pilots held an airline transport pilot (ATP) rating. The mean age of the participants was years (SD = years). Their average flight hours were 3,410 (SD = 6,437 hours). All participants gave informed consent and received compensation for their time ($40 per participant). Checklist task. For the checklist task, participants were required to complete several different checklists. Each checklist consisted of 5 items (see Figure 3). Checklist items include modes and parameters on the panels, such as altimeter status, stall light status, hydraulic pressure, flaps, heading selection mode, autopilot connection, master warning/master caution, and thrust levels. Participants were asked to check the item on the instrument panel or central pedestal before checking it off the list on the tablet. They were told to complete the checklist items from top to bottom, without skipping any items or changing the order. This task was also associated with either no feedback or visual or auditory feedback. The visual feedback consisted of changing the background color of the selected checklist item to green, along with checking a box located next to the item (see Figure 3). The auditory feedback consisted of a speech readout of the selected checklist element. Tasks and feedback design During the experiment, participants manually flew the aircraft for half of the flight and used the autopilot during the other half. They were instructed to maintain the airspeed at 270 knots, the altitude at 35,000 feet and the heading at 140 degrees. The order of manual flight and automated flight was counterbalanced between participants. Concurrent with the flight task, they were required to use the touchscreen to edit the flight plan and complete checklists. Flight plan edit task. For the flight plan edit task, participants were instructed to add a new waypoint (a 5-letter identifier) to the existing flight plan displayed on the touchscreen, using a QWERTY keyboard (see Figure 2). The size of the keys was 1 cm on the diagonal, which was shown Figure 3. Checklist task with visual feedback Experiment procedure Upon arrival, participants took part in a training session (about 30 minutes) to learn how to fly the simulator and perform the two tasks on the tablet either with or without

3 Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting 91 turbulence. Following the training, the participants completed 24 task, which took approximately 45 minutes. The order in which the various tasks were presented was counterbalanced. Following the experiment, participants were asked to fill out a debriefing questionnaire to collect their opinions and experiences with the touchscreen in the various experimental conditions. vs. 2.8%, F(1, 34) = , p < 0.001) and the checklist entry task (9.2% vs. 2.%, F(1, 34) = , p < 0.001). Correction rate Experiment design This research employed a 2x2x3 within-subject full factorial design. The three factors were workload (manual vs. autopilot), turbulence (turbulent vs. stable), and feedback (visual, auditory, or none). The dependent measures for the flight task were the rootmean-square values of deviation from the target airspeed, altitude and heading. The dependent measures for the touchscreen usage were error rate, number of corrections, and task completion time. Error rate. An error was defined as an incorrect letter entry (for the flight plan editing task), or a missed/skipped item (for the checklist task). Error rate was defined as the number of errors divided by the total number of entries for each task. Correction rate. A correction was defined as first entering a wrong letter and then replacing it with the correct entry (for the flight plan editing task), or as checking a wrong item and later unchecking it (for the checklist task). Correction rate was defined as the number of corrections divided by the number of errors. It was calculated for each condition as a whole. Task completion time. Task completion time was defined as the total time to complete the checklist or edit the flight plan. Figure 5. Correction rate as a function of task, turbulence and automation For both tasks, there was a trend toward lower correction rates in the manual condition compared to the autopilot condition (flight plan: 82.1% vs. 97.1%; checklist: 83.2% vs. 100%). Correction rates were also lower in the stable condition compared to turbulence, especially for the checklist task (42.9% vs. 93.3%). The availability and type of feedback did not affect correction rates. Completion time RESULTS One of the pilots was not able to finish the experiment, and the data were thus excluded. The data of the remaining 18 participants were analyzed using repeated-measures ANOVAs. The significance level was set at Error rate Figure 4. Error rate as a function of task, turbulence and automation (*p<0.05) The error rate was significantly higher in turbulence than in the stable condition for both the flight plan entry task (11% Figure 6. Completion time as a function of task, turbulence and feedback type (*p<0.05) Completion time was significantly shorter in the stable condition compared to turbulence. This was the case for both the flight plan entry task (13.26s vs s, F(1, 34) = , p < 0.001) and the checklist task (24.35s vs s) (F(1, 34) = , p < 0.001). There was a significant effect of feedback type for the flight plan (but not the checklist) task, (13.27s without feedback, 15.00s with auditory feedback and 22.31s with visual feedback; F(2, 34) = 4.900, p < 0.01). No interaction effects were found. Also, for both tasks, all errors significantly increased completion time (from 13.85s to 30.58s for the flight plan edit task and from 25.02s to 45.28s for the checklist; both p < 0.001). Finally, completion time was significantly shorter in the autopilot condition compared to the manual condition. This was again true for both the flight plan entry task (15.05s vs s, F(1, 34) = 8.043, p < 0.005) and the checklist entry task (24.47s vs s; F(1, 34) = , p < 0.001).

4 Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting 92 Error numbers and types A total of 128 errors were made on the flight plan entry task (see Table 1). Most of these errors were substitution errors, defined here as entering a wrong character on the tablet; 12 of the errors were errors of duplication, defined as entering the correct character more than once; finally, there were 2 errors of omission, defined as skipping one character. The errors were either corrected right away, after a few other entries had been made, or they were not corrected at all. Table 1 Error types for the flight plan entry task Errors of Substitution Duplication Omission Total Corrected right away Corrected afterwards Not corrected Total A total of 72 errors were made on the checklist entry task. 66 of these errors were corrected before the participant proceeded to the next item on the checklist. Only 6 errors went unnoticed and were not corrected. Flight performance No significant difference in flight performance was found for any factors for performance of the checklist entry task. For the flight plan entry task, the RMS value of the altitude deviation was significantly lower in the turbulent condition compared to the stable condition (58.35 vs , F(1, 34) = , p = 0.001). Both airspeed and altitude deviations were significantly lower when either type of feedback was provided (2.80 vs. 3.73, F(2, 34) = 3.509, p = 0.034; vs , F(2, 88) = 5.831, p = 0.004). Table 2 Root-mean-square value of airspeed deviation (knots), altitude deviation (feet) and heading deviation (º) Airspeed Stable Turbulent Checklist Flight plan Altitude Stable Turbulent Checklist Flight plan Heading Stable Turbulent Checklist Flight plan Subjective Preference Following the experiment, participants were asked about their subjective preference for any of the feedback conditions (see Table 5). The majority of participants preferred to have some feedback (over no feedback at all), especially during manual flight and/or in turbulent conditions. Participants preferred visual feedback for the checklist tasks and when using the autopilot. Auditory feedback was preferred when they were editing the flight plan or flying manually. Auditory feedback was preferred by participants who commented that it reduced information access cost by eliminating the need for re-orienting visual attention; visual feedback, on the other hand, was preferred by other because it provided immediate feedback and had minimal distraction. Table 3 Subjective preference for different types of feedback (multiple answers allowed) Preference (%) No feedback Visual Auditory No preference Flight Plan Checklist Manual Autopilot Stable Turbulent DISCUSSION This experiment examined the performance effects of turbulence on the use of touchscreens for two flight-related tasks. Also evaluated was the effectiveness of visual and auditory feedback for supporting error detection, fast completion and multitasking. Confirming findings from previous studies, the results of this study showed that touchscreen usage suffers greatly in turbulence (Dodd et al., 2014). The task completion time increased by 38-70%, and the error rate was 4 to 5 times higher than in static conditions. These performance decrements were exacerbated during manual flight (especially entry speed). The latter finding can be explained by the scanning cost associated with switching between monitoring the primary flight display and/or outside view and the touchscreen in the center pedestal. Flight performance also suffered during multitasking in turbulence, especially for the flight plan edit task which requires more precise hand control given the smaller input fields. Participants were more likely to notice and correct errors in turbulence. This could be attributed to pilots increased expectation of making mistakes in this condition, and thus increased error checking. The improved error detection and subsequent error correction, in turn, explains the longer completion times in turbulence. With respect to the effectiveness of feedback for improving performance with touchscreens, most studies that were conducted in static environments found that multimodal

5 Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting 93 feedback reduced error rates. For example, Bender (1999) reported a reduction in error rate from 33.3% to 7.6% with auditory feedback for mm targets, and Deron (2000) reported a reduction from 40% to around 15% with visual feedback for a keypad entry task. These studies reported only the final error counts, after error corrections. In our experiment, error rates and correction rates were analyzed separately. Not surprisingly, neither visual nor auditory feedback had a significant effect on the number of errors committed. However, contrary to our expectations and earlier findings, feedback did not improve the error correction rate, either. This may be explained by a ceiling effect - the correction rate was very high even in the absence of feedback (around 90% in most conditions). The effect of visual and auditory feedback on completion time differed between the two tasks. While the checklist task was unaffected, longer completion times were observed for the flight plan edit task, especially with visual feedback. This confirms findings from previous studies where multimodal feedback either showed no effect or slowed down data entry (Bender, 1999; Yu et al., 2016). One likely explanation for the different effects on the two tasks is that, for the flight plan entry task, participants waited for feedback on each entry before proceeding to the next because of the larger variety of possible errors, unlike the checklist task where they either succeeded or failed to check off an item. Another possible explanation is that the visual feedback for the checklist task was somewhat more salient and easier to discern. While feedback did not improve error rates and corrections and had only a limited effect on task completion time, both visual and auditory feedback improved performance on the flying task, which suggests that attentional resources were freed up and could be devoted to controlling the aircraft a net performance gain. In conclusion, the usability of touchscreens is greatly affected by turbulence. Visual and auditory feedback do not reduce errors or improve error detection/correction and can lead to slightly slower entry speed. Still, the presence of feedback is beneficial as it supports multitasking, and it is preferred by touchscreen users. The above findings are likely not limited to aviation, but generalizable to other domains that involve mobile operations, such as touchscreen use in ambulances (Viviani & Calil, 2015) and, generally, modern car cockpits. ACKNOWLEDGMENTS This research was supported, in part, by a research grant from the Federal Aviation Administration (13-G-019; Technical monitors: Drs. Tom McCloy, Sheryl Chappell and Regina Bolinger). We would also like to thank John Knudson and Jacob Durrah for software development, and Kejia Xu for his help with data analysis. REFERENCES Altinsoy, M. E., & Merchel, S. (2012). Electrotactile feedback for handheld devices with touch screen and simulation of roughness. Haptics, IEEE Transactions on, 5(1), Barrett, J., & Krueger, H. (1994). Performance effects of reduced proprioceptive feedback on touch typists and casual users in a typing task. Behaviour & Information Technology, 13(6), Bauersfeld, K. G. (1992). Effects of turbulence and activation method on touchscreen performance in aviation environments (Master's Theses, San Jose State University). Bender, G. T. (1999). Touch screen performance as a function of the duration of auditory feedback and target size (Doctoral dissertation, Wichita State University). Bonelli, S., Napoletano, L., Bannon, L., Chiuhsiang, J. L., Chi, N. L., Chin, J. C.,... & Mack, R. L. (2013). The usability evaluation of a touch screen in the flight deck. Cases on Usability Engineering, 53, Deron, M. (2000). How important is visual feedback when using a touch screen?. Usability news, 2(1). Dodd, S., Lancaster, J., Miranda, A., Grothe, S., DeMers, B., & Rogers, B. (2014, September). Touch Screens on the Flight Deck The Impact of Touch Target Size, Spacing, Touch Technology and Turbulence on Pilot Performance. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 58, No. 1, pp. 6-10). SAGE Publications. Kaaresoja, T., Brown, L. M., & Linjama, J. (2006, July). Snap-Crackle-Pop: Tactile feedback for mobile touch screens. In Proceedings of Eurohaptics(Vol. 2006, pp ). Kaminani, S. (2011, October). Human computer interaction issues with touch screen interfaces in the flight deck. In Digital Avionics Systems Conference (DASC), 2011 IEEE/AIAA 30th (pp. 6B4-1). IEEE. Park, J., & Han, K. H. (2011). Effect of Target Size and Duration of Visual Feedback on Touch Screen. In HCI International 2011 Posters Extended Abstracts (pp ). Springer Berlin Heidelberg. Poppinga, B., Magnusson, C., Pielot, M., & Rassmus-Gröhn, K. (2011, August). TouchOver map: audio-tactile exploration of interactive maps. InProceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (pp ). ACM. Viviani, C. A. B., & Calil, S. J. (2015). Recommendation in the Use of Touchscreen Technology in Medical Devices. In VI Latin American Congress on Biomedical Engineering CLAIB 2014, Paraná, Argentina 29, 30 & 31 October 2014 (pp ). Springer International Publishing. Wilson, K. S., & Liu, M. I. S. (1995, October). A comparison of five user interface devices designed for point-of-sale in the retail industry. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 39, No. 4, pp ). Sage CA: Los Angeles, CA: SAGE Publications. Yu, C., Wen, H., Xiong, W., Bi, X., & Shi, Y. (2016, May). Investigating Effects of Post-Selection Feedback for Acquiring Ultra-Small Targets on Touchscreen. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp ). ACM.