Joseph Cafazzo, PhD PEng
Modern Times, United Artists 1936
Science Finds, Industry Applies, Man Conforms Slogan from the 1933 Chicago World s Fair
"People Propose, Science Studies, Technology Conforms" Don Norman s person-centered motto for the 21st century. "Things that make us smart." Addison-Wesley, 1993.
Poor Design Images - c prompt ballot friendly fire
We have a serious training problem that needs to be corrected We need to know how our equipment works: when the battery is changed, it defaults to its own location. We ve got to make sure our people understand this Senior Air Force Official
Recount, HBO 2008
Radiation Oncology and Patient Safety You are not alone
How do we prevent error? Hierarchy of Effectiveness 1. Forcing functions & constraints 2. Automation/computerization 3. Simplification/standardization 4. Reminders, checklists, double checks 5. Rules & policies 6. Training & education tech focus people focus
Human Factors in Healthcare IOM 1999: 44, 000 to 98, 000 US deaths annually due to medical error Baker, Norton 2004: 9, 000 to 23, 000 deaths annually due to preventable adverse events
User Centered Design Analyze user requirements Evaluate Design & Prototype
Design & Prototype
Evaluate Expert/Heuristic Analysis Visibility Consistency Efficiency Flexibility Autonomy
Evaluation Criteria Efficiency & Speed Task completion time Task accuracy & error frequency Number of requests for help Number of attempts to correct errors Impact & Severity of error (patient safety) Workload/Difficulty scale (1-7) User feedback & preferences
Evaluate Usability Testing Representative users Realistic scenarios Think-aloud protocol Record video & audio Qualitative & quantitative
Evaluation of a Functional Prototype
Improving Patient Safety during Radiation Therapy through Human Factors Alvita Chan Principle Investigators: Dr. Mohammad Islam, Dr. Joe Cafazzo Co-Investigators: Dr. Tony Easty, Dr. David Jaffray
Systematic Literature Review A systematic review on existing literature was conducted to understand sources of errors 7 distinctive studies were found 40-50% errors occur during treatment delivery Most frequent errors: Incorrect patient position (e.g. wrong site, wrong SSD, wrong couch height) Incorrect treatment accessories (i.e. Shielding blocks, bolus) Yeung TK et al. Radiotherapy and Oncology. 2005, 74:283-291 Marks LB et al. International Journal of Radiation Oncology, Biology and Physics. 2007, 69:1579-1586
40.4% Yeung TK et al. Radiotherapy and Oncology. 2005, 74:283-291
Work Environment Analysis Identified issues and developed recommendations regarding treatment/ control room setup Ambient Noise/Interruptions Furniture/computer features and placement Storage Lighting Temperature and humidity
Workflow Analysis Mapped out entire workflow of the treatment delivery process using the Unified Modeling Language (UML) 2.0 Activity Diagrams * From 24 observations, 6 for each treatment team
Heuristic Evaluation Systematic inspection of current user interfaces according to a list of guidelines/heuristics E.g. Visibility of system state, minimize memory load, informative feedback, flexibility and efficiency, prevent errors Identified 75 usability problems Number of Problems Identified 40 30 20 10 0 Usability Problems by Severity Ratings 37 37 18 20 High Medium Low Positive Severity Ratings Zhang et al. Journal of Biomedical Informatics. 2003, 36: 23-30
Workflow Analysis Identified areas that can be improved: Heavy reliance on policy Checking process during patient setup Least effective Hierarchy of Effectiveness 1. Forcing functions and constraints 2. Automation / Computerization 3. Simplification / Standardization 4. Checks, double checks, checklists 5. Policy and training Hierarchy of effectiveness in preventing errors Institute for Safe Medication Practices. ISMP Medication Safety Alert. 1999, 4 (11)
How can the process be improved? Focused on checking process during patient setup Redesigned components of the user interface: To lessen the reliance on policy and procedure To improve efficiency of the checking process To display information based on the users needs Design process: Expert input + 2 focus groups 12
DRR is displayed visibly to improve checking compliance Redesigned Interface Patient picture is displayed visibly for easy identification
Redesigned Interface Notes that required certain actions to be performed are highlighted New messages and outstanding items are highlighted Statuses of approval dates and images are clearly displayed DOB given by patient is checked by system Both SSD and depths are displayed Forcing function/checklist is used to improve checking compliance
Redesign of Existing System Current interface: 14 mouse clicks Redesigned interface: 4 mouse clicks Improved efficiency 17
Experimental Evaluation - Usability Testing 16 radiation therapy students from PMH were recruited to participate An actor played the role of a second therapist Performed patient setup tasks using both the current and redesigned interfaces 18
Experimental Design 2 interfaces x 4 scenarios 3 out of the 4 scenarios were designed with a high potential for common errors to occur: Overlooking an important note Shifting couch incorrectly Overlooking a change in approval dates 19
Rate of Errors P<0.04 P<0.01 20
Task Time P<0.02 21
User Satisfaction Post-test Questionnaire: 22
User Satisfaction Better at showing the information needed Attributes Current Redesigned Significance Better at drawing attention to important items Better for catching an important message, a change in approval date, or an unapproved image Enable treatment to be delivered more safely 0% 94% Yes (P<0.01) 0% 100% Yes (P<0.01) 0% 100% Yes (P<0.01) 13% 81% Yes (P<0.01) Easier to use 25% 56% No More efficient 19% 63% No Preferred system 25% 56% No 23
Limitations Students vs. experienced therapists Mannequins vs. real patients Redesigned interface was only a prototype, not all features were functional Only the first iteration of the design cycle 24
Acknowledgements Project team: - Dr. Joe Cafazzo - Dr. Mohammad Islam - Dr. Tony Easty - Dr. David Jaffray - Tara Rosewall Funding agencies: - NSERC, NPSF, AAPM, CPSI Other: - Catherine Dupuis - Mary Stewart-Hardy - Varuna Prakash - Mark Fan - Radiation therapists at PMH - Radiation therapist students 26
Summary Humans are fallible. We will err. We can reduce frequency and consequences of errors by designing systems within our limitations Solutions should aim towards the top of the Hierarchy of Effectiveness
healthcarehumanfactors.com Joseph Cafazzo, PhD PEng