1375 A USEABLE, ONLINE NASA-TLX TOOL David Sharek Psychology Department, North Carolina State University, Raleigh, NC 27695-7650 USA For over 20 years, the NASA Task Load index (NASA-TLX) (Hart & Staveland, 1988) has been successfully used as a self-report measure of cognitive workload, yet emphasis on the usability and accessibility of the NASA-TLX as a research tool has remained lacking. A freelyavailable, user-friendly, online version of NASA-TLX is introduced (Sharek, 2009). Key features and benefits of the online version for researchers and participants are discussed. Copyright 2011 by Human Factors and Ergonomics Society, Inc. All rights reserved DOI 10.1177/1071181311551286 INTRODUCTION The NASA-TLX is a subjective, multidimensional assessment tool that can be used to rate perceived workload on six subscales: Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration. It was developed by the Human Performance Group at NASA Ames Research Center over a three year development cycle that included more than 40 laboratory simulations (NASA, 1986). It has been cited in over 550 studies (Hart, 2006) and a recent search for NASA-TLX on Google Scholar revealed over 4,820 articles (Google, n.d.). These statistics highlight the large influence the NASATLX has had in Human Factors research. Some electronic versions of the NASA-TLX were designed many years ago (NASA, 2003; NCARAI, 2004) and have not been updated. One version is available as part of a software package that costs over 400 dollars (Inquisit, 2010); the Web version costs over 1000 dollars. Another version lacks instructional text for participants (Cao, Chintamani, Pandya, & Ellis, 2009) and doesn t provide definitions for the subscale items during the pairwise comparison section. All versions require a download and installation on Windows compatible computers since Apple or Linux machines are not supported. These limitations, coupled with often poorly designed user interfaces have been addressed in the form of a freely available, Rich Internet Application (RIA) version of the NASA-TLX (Online-TLX). The Online-TLX is not subject to any of the previously discussed limitations. Additionally, based on user feedback, new updates with enhanced features are released regularly. A login system allows researchers to store all their data in one location so they can concurrently run multiple instances of the software, for example, in computer labs, or when conducting completely Web-based experiments. ONLINE-TLX TOOL OVERVIEW The Online-TLX can be accessed at http://www.nasatlx.com. The URL for the tool was created to be both easy to remember and descriptive. The entire file size for the program is 210KB and typically loads in less than one second on a consumer-grade high speed Internet connection. Maintaining a small file size was important during the development process so that end-user download times would be kept to a minimum. A MySQL database using PHP as the server-side scripting language manages the data. The Online-TLX was developed in Adobe Flash CS4 using ActionScript 3. The minimum required Flash player is version 9.0. The decision to develop the tool in Flash was based on the 98.9% penetration rate for version 9 of the plug-in (Table 1) in mature markets (Adobe, 2011). Additionally, the decision to use Flash as the developmental platform was due to Flash s cross-platform compatibility. That is, the intrinsic nature of a Flash file (.SWF) allows it to accommodate a variety of operating systems that run on Windows, Apple, or
Linux machines. This allows for rapid-prototyping and development and reduces potential formatting and programming errors across platforms. Table 1. Flash Player Ubiquity (Adobe, 2011) Flash Player 9 Mature Markets US, Canada, UK, Germany, France, Japan, Australia, New Zealand 99.6% US/Canada 99.9% Europe UK, Germany, France 99.7% Japan 98.6% Australia/New Zealand 99.1% Emerging Markets China, S. Korea, Russia, India, Taiwan 98.8% 1376 a user may have already entered to remain, so that only the secondary password textbox needs to be filled out. Email addresses are used for user IDs. This reduces the amount of information required at sign up, and ensures the selection of a unique user ID. Additionally, if users forget their password, a new temporary password can be sent via their email address. Passwords are stored as secure 32-digit hexadecimal values using a MD5 cryptographic hash function. Figure 2. Online NASA-TLX (Sharek, 2009) Sign up Screen WALKTHROUGH Login Screen After the tool has loaded, the login screen (Figure 1) is displayed. This screen allows returning users to enter their user ID and password. Figure 1. Online NASA-TLX (Sharek, 2009) Login Screen Options After a researcher logs in, they are presented with an Options screen (Figure 3). From this screen, researchers can setup and begin a new TLX measurement session, or they can select to download all the data that have already been collected in previous sessions. New users can register by clicking on the Sign Up button at the top right of the screen. Clicking on this button simply loads an additional password text box and changes the instructional text (Figure 2). This allows any email or password information that
Figure 3. Online NASA-TLX Options Screen. In order to begin a new TLX measurement session, a Participant ID must be entered. This value should help the researcher to identify who was participating in the task. If a participant is taking part in multiple TLX studies, an optional Experiment ID value can be entered. The use of these values is left up to the researcher, but using a standard convention for identification is recommended. Researchers also can choose to include the 15 pairwise comparison questions by clicking on a radio-type toggle button. According to Hart (Hart & Staveland, 1988) if participants are required to use the TLX tool multiple times, they only need to answer the pairwise comparisons once per task type. If a participant s workload needs to be measured for intrinsically different tasks, then revisiting the pairwise comparisons may be required. However, according to one school of thought, removing the pairwise comparisons altogether may actually increase experimental validity and reduce experimental error (Bustamante & Spain, 2008). All TLX session data are stored in a database and are always available to be securely downloaded. This allows the TLX to be administered in computer lab settings across multiple workstations without the need to save data locally to each machine; this can become a laborious task when retrieving the data from each machine at the end of the experiment. Data are saved in a comma delineated.csv file. This file type can be natively opened by Microsoft Excel as well as traditional statistical software packages such as SPSS and SAS. Additionally, data is formatted so that it can be easily analyzed. That 1377 is, each row of data represents one participant, and each column represents a variable. A total of 23 variables are saved for each participant including: researcher ID, experiment ID, participant ID, overall TLX workload score, raw ratings for each subscale, individual weights based on the pairwise comparisons for each subscale, weighted workloads for each subscale, and timestamp. The primary benefit for storing as much data as possible is that researchers have the freedom to organize and conduct a variety of analyses. A secondary benefit for including this large amount of data is that trust in the tool may be increased through revealing all data used in the calculations. Each time a file is downloaded, it is saved using a standard naming convention that includes the user s ID and a versioning system which helps reduce the potential to accidentally overwrite a file. Currently, all TLX sessions are downloaded to the file, but future iterations will allow users to only download data that matches specific participant IDs or experiment IDs. Subscale Ratings Screen After researchers set up the tool and populate the Participant ID and the optional Experiment ID, they can click on the Next button to end the setup process and initiate the start of the TLX. An introductory screen (Figure 4) is displayed that includes instructions for participants to read before they begin. These instructions are editable and can be modified and saved by the researcher. After reading the Instructions, participants click on the start button to begin the ratings.
Figure 4. NASA-TLX Instruction Screen 1378 their rating, they can click on the new point along the continuum and the high-contrast indicator will automatically scroll to the new location. The use of animation here is to provide a visual confirmation that a change has been made to the scale. If all six ratings have not been made then a forcing function will prevent the participant from advancing, otherwise, clicking on the Submit button will allow them to continue. Pairwise Comparison Screen Figure 5 shows the subscale rating screen. In this screen, participants are asked to rate their experience with the task on all six subscales. The instructions and definitions remain on-screen so that participants won t have to memorize the operational definition for each subscale. As can be seen in Figure 5, participants also can retrieve an extended definition for each subscale by hovering over each definition. Figure 5. Online NASA-TLX Subscale Ratings Screen. A hover help is displayed (shown) when a user hovers over the shortened definition for each subscale. Participants can click anywhere along the continuum for each subscale. The rationale for providing a continuum was so that the scale would map as closely to the original paper and pencil version as possible. If a participant needs to adjust After completing the subscale ratings, and if the pairwise comparisons option was selected, the participant will be presented with a screen similar to the one in Figure 6. All six subscale definitions remain on the screen while a smaller window displays the individual comparisons. Clicking on the subscale that contributed the most to the task will highlight that subscale to indicate it has been selected. The participant then must click on the Submit button to advance to the next comparison. This two-step selection process allows the participant to change their mind. It also prevents a participant from quickly clicking on the same button in order to advance to the end of the measurement without thoughtfully participating. After a comparison has been submitted, the window slides off screen to the left while a new comparison window slides in from the right. This quick animation indicates to the participant that a new comparison is ready for their selection. After the final selection is made, the participant is taken to a debriefing page. The text for this page can be edited by the researcher. Additionally, the participant can be transferred to another Web page of the researcher s choosing.
Figure 6. Online NASA-TLX Pairwise Comparison Screen. http://hfs.sagepub.com/supplemental DESKTOP VERSION A free stand-alone version also is currently available at KWWS SUR VDJHSXE FRP VXSSOHPHQWDO for both Windows and Apple machines (Sharek, 2011). The stand-alone version was developed using Adobe Air s RIA environment. This development platform essentially allows for the same online version to be deployed to the desktop with only minor modifications thus reducing the development load when new updates or features are released., FUTURE DEVELOPMENT Mobile versions of the Online-TLX are currently under development. An iphone/ipod Touch version will be released first followed by a version for the Android mobile operating system. Releasing the Online-TLX to mobile platforms provides the potential to facilitate easier workload measurements in the field or in situations where use of a personal computer or laptop is not possible. A Feedback button is available on the Login screen, and all users are encouraged to submit feedback regarding new features. The further development of features for the Online-TLX will be influenced by user input and continued usability analysis of the tool. 1379 REFERENCES Adobe. (2011). Flash Player Version Penetration, from http://www.adobe.com/products/player_census/flashp layer/ Bustamante, E. A., & Spain, R. D. (2008). Measurement invariance of the NASA TLX. Human Factors and Ergonomics Society Annual Meeting Proceedings, 52, 1522-1526. Cao, A., Chintamani, K. K., Pandya, A. K., & Ellis, R. D. (2009). NASA TLX: Software for assessing subjective mental workload. Behavior Research Methods, 41(1), 113-117. doi: 10.3758/brm.41.1.113 Google. (n.d.), Retrieved June 15, 2011, from Google Scholar, http://scholar.google.com/scholar?&q=%22nasatlx%22 Hart, S. (2006). NASA -Task Load Index (Nasa-TLX); 20 Years Later. Human Factors and Ergonomics Society Annual Meeting Proceedings, 50, 904-908. Hart, S., & Staveland, L. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In P. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 139-183). Amsterdam: North Holland. Inquisit. (2010). NASA Task Load Index (NASATLX) (Version 3.0.5.0) [Computer Software]. Seattle, WA: Millisecond Software LLC. Retrieved from http://www.millisecond.com/download/samples/v3/n ASATLX/default.aspx NASA. (1986). Nasa Task Load Index (TLX) v. 1.0 Manual NASA. (2003). NASA Task Load Index (TLX): Computerized Version (Version 2.0) [Computer Software]. Moffett Field, CA: NASA-Ames Research Center, Aerospace Human Factors Research Division. Retrieved from http://humansystems.arc.nasa.gov/groups/tlx/comp uter.php NCARAI. (2004). NASA TLX for Windows (Version 1) [Computer Software]. Washington DC: Navy Center for Applied Research in Artificial Intelligence. Retrieved from http://www.nrl.navy.mil/aic/ide/nasatlx.php Sharek, D. (2009). NASA-TLX Online Tool (Version 0.6) [Internet Application]. Raleigh, NC. Retrieved from http://www.nasatlx.com Sharek, D. (2011). NASA-TLX Desktop Tool (Version 2.0.1) [Computer Software]. Raleigh, NC. Retrieved from http://www.playgraph.com/apps/nasa-tlx