ADVANCES IN THE USABILITY AND USE OF HEALTHCARE INFORMATION SYSTEMS: EXPERIENCES IN CANADA AND INTERNATIONALLY

Transcription

1 ADVANCES IN THE USABILITY AND USE OF HEALTHCARE INFORMATION SYSTEMS: EXPERIENCES IN CANADA AND INTERNATIONALLY Dr. Andre Kushniruk Director and Professor School of Health Information Science, University of Victoria, Victoria, Canada

2 OVERVIEW (AND QUESTIONS) Usability does it matter in healthcare IT? Is it a problem? What are the main issues with healthcare IT usability? What is the relation of usability to patient safety? What can be done about it - here in Canada? internationally?

3 HUMAN-COMPUTER INTERACTION IN HEALTH CARE Challenges of implementing information systems in health care continue: Issues with acceptance of systems Usability problems Inadvertent introduction of error User satisfaction issues Inadvertent impact on workflow Issues related to human-computer interaction may be single-most important barrier to successful implementation of systems in health care

4 THE SOLUTION Applying concepts from human-computer interaction (HCI) and usability engineering into system design, implementation, procurement and customization This talk will introduce A framework that can be applied to developing more useful and usable healthcare systems Examples of application of HCI approaches to the study of use of healthcare IT in Canada and internationally

5 USABILITY Usability measure of ease of use of a system in terms of (Preece et al., 1993): 1. Learning 2. Effectiveness 3. Efficiency 4. Enjoyability 5. Safety Usability Engineering - scientific approaches to designing and testing usable systems

6 USABILITY ENGINEERING METHODS Usability Testing Observe representative users doing representative tasks with system under study Think Aloud Protocols Video Recording Usability Inspection Usability inspector steps through system Cognitive Walkthrough and Heuristic Evaluation

7 WHEN CAN USABILITY STUDIES TAKE PLACE? 1. Planning (needs analysis) 2. Analysis (requirements) 3. Design 4. Implementation (programming) 5. Support (maintenance) -cognitive task analysis -analysis of decision making -analysis of early project concepts -throw-away prototyping (with usabilty testing) -cognitive task analysis -usability testing -usability inspection -design walkthroughs -usability testing -usability inspections -clinical simulation -clinical simulation and naturalistic analysis -summative usability testing -VuLab Figure 1. The systems development life cycle (SDLC) in relation to usability engineering methodologies. Kushniruk, A. W. (2002). Evaluation in the design of health information systems. Computers in Biology and Medicine, 32(3),

8 LOW-COST RAPID USABILITY TESTING (KUSHNIRUK & BORYCKI, HEALTHCARE QUARTERLY, 2006) Usability testing does not require an expensive fixed usability laboratory Needs to be done locally (central conformance testing not enough will not ensure usable or safe systems in context of local use) Can be used to predict and rectify errors and problems with system using qualitative/quantitative video coding Highest level of fidelity if conducted in real setting Can also be extended to analysis of uses of systems in REAL clinical situations Low-cost rapid usability testing Low-cost in-situ clinical simulations

9 LOW-COST RAPID USABILITY TESTING EQUIPMENT Video camera to record user physical actions Microphone to record user verbalizations Screen cam to record user facial expressions Recording of computer/pda screens to CD using screen capture software e.g. Hypercam From: Kushniruk & Borycki (2006), Low-cost rapid usability engineering, Healthcare Quarterly

10

11 A Multi-Level Approach to Analyzing HCI Aspects of System Use in Healthcare Level 3 Level 2 Level 1 Multiple Users Interacting with Each Other and the System to Carry Out Multiple Tasks as part of the Organization User(s) Interacting with the System and their Environment to Carry Out a Work Task User Interacting With the System in Isolation Clinical Simulations Naturalistic (organizational Impact) Clinical Simulation (for impact on workflow) Usability Testing (for surface usability Problems) Adapted from Borycki, Kushniruk, Open Medical Informatics J., 2010

12 LOW-COST RAPID USABILITY TESTING EXAMPLE: ANALYSIS OF EDUCATIONAL SOFTWARE LEVEL 1 How effective is the interface and content of a decision support/learning tool tool? (Kushniruk et al., 1995) Data Video recording of doctors and students interaction with system Audio recording of think aloud Screen captures

13 CVideo Log File - Subject #1 00:01:03 to 00:02:26 Introduction I ve already had to loosen my tie Shifts in seat and studies screen 00:02:27 Risk Analysis Guidelines/Recommendations Screen COMMENT: CRITIQUE-CONTENT My first comment is that this is a little old, I don t know if any of this has been updated since then 00:01:17 to 00:01:17 Checks out the sex factor ACT: 00:01:44 Raises cholesterol levels by 1 to 6.3 ACT: 00:03:00 Goes to help screen

14 ACT: 00:03:12 surveys the powerbar choices (3) and selects the Introduction GOAL: How do I fast forward? NAVIGATION PROBLEM COMMENT: There is no way out of this? EXP: You can kill this COMMENT: STATE OF SYSTEM I can kill this, Sorry you told me that Stops the explanation

15 00:02:14 to 00:02:14 Surveys the icons OK, so, Risk Analysis ACT: 00:03:37 Selects risk analysis (help menu) EVENT: 00:03:32 Crashes program COMMENT: CRITIQUE -CONTENT When I said it was old recommendations, my first impressions was that it was kind of old. When was this made? Things have changed a lot since then. At that point I would probably turn it off (laughs), I need something newer. Things have changed a lot and I think the recommendations have been cleaned up a lot

16 ACT: 00:04:49 Returns to risk analysis screen 00:05:29 Turns head sideways, and looks for options menu on powerbar COMMENT: CRITIQUE-INTERFACE Options menu on the help, I don t know which one of these is the options menu. Which one of these is the options. It didn t come with instructions GOAL: I want to change the units because height is in centimeters 00:05:44 Returns to Help and surveys the icons on the powerbar So now I have to get out the owner s manual!

17 Questionnaire Response (same subject) I found the program easy to use Agree somewhat It was fairly easy to learn Agree somewhat The content of the program is presented clearly Agree completely The interaction is properly paced Neutral The narration is easy to follow Agree somewhat Conclusion: User s perceptions do not agree with actions captured on video multiple methods required

18 COGNITIVE ASPECTS OF USABILITY Importance of evaluating effects on cognition Reasoning Individual decision making Group decision making and workflow Technology-induced error Critical to have knowledge of in complex domains such as health care Analysis of cognitive aspects can be integrated with analysis of usability

19 A Multi-Level Approach to Analyzing HCI Aspects of System Use in Healthcare Level 3 Level 2 Level 1 Multiple Users Interacting with Each Other and the System to Carry Out Multiple Tasks as part of the Organization User(s) Interacting with the System and their Environment to Carry Out a Work Task User Interacting With the System in Isolation Clinical Simulations Naturalistic (organizational Impact) Clinical Simulation (for impact on workflow) Usability Testing (for surface usability Problems) Adapted from Borycki, Kushniruk, Open Medical Informatics J., 2010

20 Studies of Impact on Decision making and Knowledge Organization (Kushniruk, et al., MD Computing 1996; Patel, Kushniruk, Yang, Yale, JAMIA 2001) Paper record to Electronic medical record (EMR), back to Paper record Comparison of matched records Learner styles Direct observation of process

21 Study 1: Experimental Study of Use Doctors asked to enter case data into the system think aloud Conduct patient interview using system simulated patient Video recordings of sessions novice system users - tested over 4 sessions, from baseline and training Analysis of contents of paper -> computer records

22

23

24 Study 2: Study of use in diabetes clinic over six month period naturalistic approach Interviews (pre and post) 16 clinic staff Usability testing with subset of subjects Training recorded as well Logging of all system use Study of contents of paper and computer records

25 Results: More irrelevant information in paper records Overall less information recorded in computer based records For corresponding records, EMR version contained 25% less information. Fewer diagnoses recorded in EMR for matched records However more detail about primary diagnosis entered Change in reasoning -- from hypothesis driven to screen driven

26 Diagnostic Reasoning Using Paper Record Patient Data Multiple Hypotheses Diagnostic Reasoning Using EMR Patient Data Hypotheses

27 Narrative of Doctor-Patient Interaction (Involving Experienced User) Time Episode 00:00-02:30 - Creates a Patient Visit 02:47-03:26 - Reviews Presenting Complaint - Starts to Select a Filter but stops 03:55-04:39 - Gathers info without EMR (paper) 04:39-08:48 - History of Present Illness - Hypothesis: hyperthyroidism 09:21-09:49 - Chooses clinical note template 09:54-13:25 - Collects personal history, in order of categories on the screen ( screen-driven behavior ) Experienced users become screen-driven - Affects reasoning and requests for information by physician - An unintended consequence

28

29 A Multi-Level Approach to Analyzing HCI Aspects of System Use in Healthcare Level 3 Level 2 Level 1 Multiple Users Interacting with Each Other and the System to Carry Out Multiple Tasks as part of the Organization User(s) Interacting with the System and their Environment to Carry Out a Work Task User Interacting With the System in Isolation Clinical Simulations Naturalistic (organizational Impact) Clinical Simulation (for impact on workflow) Usability Testing (for surface usability Problems) Adapted from Borycki, Kushniruk, Open Medical Informatics J., 2010

30 A Continuum of Studies and Settings - from lab to real-world and back again (Kushniruk, 2002, Computers in Biology and Medicine, Kushniruk, MIE 2012) LABORATORY - Fixed usability lab - Experimental tasks - think aloud - cognitive task analysis - Simulations - E.g. simulated doctor-patient interviews -Simulated clinical workflow NATURALISTIC - Study in the real setting (e.g. clinic) - Virtual usability lab

31 MOVING FROM THE LAB TO SIMULATIONS IN REAL ( IN-SITU ) CONTEXTS AND NATURALISTIC SETTINGS LEVEL 3 For error identification, predication and mitigation Assessment of impact of systems on workflow before they are widely released Need for low-cost portable approaches for conducting simulations in realistic or even real settings Emergency rooms ICUs Hospital wards

32 IN-SITU CLINICAL SIMULATION OF WORKFLOW IN MEDICATION ORDER ENTRY Study of medication administration systems Including the study of bar coding with the use of a medication administration system 16 subjects 5 nurses and 11 doctors Used dummy patient (mannequin) in simulations Conducted simulation and naturalistic study (in real setting) using real equipment Task: administer IV medications to a patient

33 NEW WORKFLOW

34 EXAMPLE CODED TRANSCRIPT FIRST ORDER: 00:14.3 SEARCH FOR PATIENT ON COMPUTER 00: 45.7 VIEW ORDER LIST 00:51:9 SELECT ORDER 00:55:.3 VERIFICATION SCREEN APPEARS MOVES OVER TO PATIENT 00:59:6 TALKING TO PATIENT S. Nice to meet you. Is your name Toridai, right? I will now give you an IV drip 01:09.5 SCAN PATIENT ID (FROM WRIST) MOVES BACK TO COMPUTER 01:25:2 VIEW EXECUTION INFORMATION MOVES OVER TO PATIENT AND SETS BAG SECOND ORDER: 02:24.6 SEARCH FOR PATIENT ON COMPUTER 02:40.2 VIEW ORDER LIST

35 EXAMPLE OF POST-TASK SEMI-STRUCTURED INTERVIEW E: Do you find any difficulty with handling the barcode reader? S: In today s operation there were no problems. But in the real situation, sometimes the scanner doesn t respond to the barcode. Also sometimes the cord of the scanner is too short to reach the patient. E: Do you find any difficulty with entering the data? S: In general, I want a more simplified system for the verification process. The more patients there are, the more difficult the verification would become. Sometimes in the emergency we have to skip this procedure due to its time-taking process and someone might need urgent help, but with this system I don t think I d be able to do that

36 FINDINGS - Fundamental changes in workflow for task under study (from parallel to serial) - Potential sources of medication error in medication administration and documentation - when many medications/iv bags, or during interruptions - when locked in sequence in emergency - This included implications not thought about by the designers or customizers - such testing of healthcare systems needs to be done prior to widespread release

37 Combining the Approaches: Recommendation for a Multi-Phased Approach to Testing Clinical Systems (for Usability and Safety) Phase 1 Phase 2 Phase 3 Laboratory Testing Clinical Simulation In-Situ Testing Artificial Real From: Kushniruk et al., IMIA Yearbook of Medical Informatics, 2013

38 OTHER APPLICATIONS WE HAVE WORKED ON Mobile apps Applications for the elderly Electronic health record systems Decision support systems Pervasive computing Home-based healthcare information systems Patient clinical information systems (e.g. PatCIS)

39 USABILITY AND ERROR -- WHAT ARE TECHNOLOGY-INDUCED ERRORS? Technology-induced errors are those sources of error that arise from the: (a) design and development of a technology (b) implementation and customization of a technology (c) interactions between the operation of a new technology and the new work processes that arise from a technology s use (Borycki & Kushniruk, 2008, p. 154) Differ from errors caught by traditional software testing Differ from unintended consequences broader concept includes positive and negative consequences (Ash et al., 2004; Borycki et al., 2010) Borycki & Kushniruk 2011

40 OUR WORK SINCE 2000 PREDICTIVE ANALYSIS TO PREVENT ERRORS AND IMPROVE USABILITY A range of usability testing and simulation studies Need to apply to make system usable and safer Feedback into : Improvement of software (Borycki et al, 2008) Improvement of process of development (Borycki et al, 2008) Customization of vendor products and improvement in training (Kushniruk et al,2009) Software testing new approaches (Kushniruk & Borycki, 2010) Selection and procurement of safer software IT (Kushniruk et al., 2011)

41 USABILITY AND ERROR EXAMPLES FROM USABILITY STUDIES Display visibility problem e.g. cannot see that need to scroll down a menu to get right dose (result -> wrong dose entered) (Kushniruk et al., 2004) Navigation problem e.g cannot backtrack to screen with critical patient information (result -> allergies not reviewed) (Kushniruk, 1996) Consistency problem - e.g. interface seems to indicate can enter data when cursor blinking, in other cases can t (result -> data not entered in some cases) (Kushniruk et al., 1996) Meaning of labels problem cannot access function needed due to not understanding label (result -> best practice advisory (BPA) ignored) (current work)

42 EMERGENCE OF ARTICLES ABOUT TECHNOLOGY-INDUCED ERRORS Borycki & Kushniruk 2011

43 CLINICAL PLUS COMPUTER BASED SIMULATION Extension of computer based simulations Clinical simulations are used as input parameters to a computer-based simulation model Borycki & Kushniruk 2011

44 USABILITY AND PROCUREMENT -- Continuum of Evidence for System-Organization Fit in Selecting Systems Low Fidelity Testing High Fidelity Testing Demo by Vendor no CLIPS Demo by Vendor CLIPS (predefined scenarios) Demo by Vendor CLIPS (not given beforehand) Usability Testing of Products (not by Vendor) Simulation Based Analyses of Usability and Workflow (not by vendor) Weak Evidence Strong Evidence Kushniruk et al. Healthcare Quarterly, 2011

45 Simulation Room in ITX Laboratory Used in Selection of EHRs for Copenhagen Region

46 USABILITY AND INTEROPERABILITY PatCIS first patient access to own records over the WWW Sundhed.dk National Portal Kanta Finnish Portal VA interoperability work in the US -- human factors of interfacing the VA s CPRS with local RHIO portals

47

48 NATIONAL AND INTERNATIONAL EFFORTS National usability questionnaire in Finland Results of second national questionnaire (4000 respondents) indicate ratings of usability have not improved (although clinicians wanted more input) Regulation on safety of healthcare IT European directives and also in North America International Medical Informatics Working Groups Human factors Patient safety People and organizational issues Sharing of information EHR usability rating sites

49

50 CONCLUSIONS To ensure system usability and safety, need for dissemination of low-cost in-situ methods For extending both usability testing and clinical simulations Laboratory-based simulations not enough as evidenced by systems that were tested in labs elsewhere, deemed usable and safe, but can be shown to lead to error in local organization!! BOTH local in-situ testing and clinical simulation is needed to ensure both safety and efficiency of systems Methods exist - need to take testing into context of system use, and need to apply in local, vendor and regional contexts

51 QUESTIONS Usability does it matter in healthcare IT? To what extent is it a problem? why are we still talking about usability issues? What are the main issues with healthcare IT usability? What is the relation of usability to patient safety? What can be done to improve the situation - here in BC and Canada? internationally?