Voice recognition versus transcriptionist

Transcription

1 Voice recognition versus transcriptionist Poster No.: C-0263 Congress: ECR 2010 Type: Audit/Professional Issues Topic: Audit/Professional Issues Authors: R. H. Strahan; Clayton South VIC/AU Keywords: Voice recognition dictation, Transcriptionist, Radiologist DOI: /ecr2010/C-0263 Any information contained in this pdf file is automatically generated from digital material submitted to EPOS by third parties in the form of scientific presentations. References to any names, marks, products, or services of third parties or hypertext links to thirdparty sites or information are provided solely as a convenience to you and do not in any way constitute or imply ECR's endorsement, sponsorship or recommendation of the third party, information, product or service. ECR is not responsible for the content of these pages and does not make any representations regarding the content or accuracy of material in this file. As per copyright regulations, any unauthorised use of the material or parts thereof as well as commercial reproduction or multiple distribution by any traditional or electronically based reproduction/publication method ist strictly prohibited. You agree to defend, indemnify, and hold ECR harmless from and against any and all claims, damages, costs, and expenses, including attorneys' fees, arising from or related to your use of these pages. Please note: Links to movies, ppt slideshows and any other multimedia files are not available in the pdf version of presentations. Page 1 of 17

2 Purpose Fig 1 on page 2 Fig 2 on page 3 A computerized voice recognition (VR) or digital dictation system has been available in the Diagnostic Imaging Department at our institution for more than four years. Proponents of VR argue its merits as marked reduction in report turnaround times (TAT) and cost savings, specifically from transcriptionist salaries (Reinus 2007, Boland et al 2008, Rana et al 2005) [1,2,3]. Opponents of VR argue that the task of transcriptionist is shifted to the radiologist and more time per report is needed in correcting VR errors. This is false economy as the extra time needed to report is associated with increased costs per report. [4] There is some evidence to show that VR can be associated with a significant incidence of typographical errors (Pezullo et al, 2008, Quint et al, 2008) [4,5]. Previous studies state error rates of 20% to 30% and interestingly, radiologists consistently underestimated their own error rates (Quint et al, 2008) [5]. Despite the frequent introduction of VR into radiology departments, little evidence still exists about its impact on workflow, error rates and costs. In November 2008, with increased workflow and pressure to provide timely MRI reports, a transcriptionist was hired at our department to type some MRI reports. This was a unique situation as some reports were generated by VR and, some by a transcriptionist. We therefore designed a study to compare typographical errors, TAT and productivity for VR generated reports versus transcriptionist generated reports in MRI. Images for this section: Page 2 of 17

3 Fig. 1: Title page. Page 3 of 17

4 Fig. 2: Purpose Page 4 of 17

5 Study/Project design Material and Methods: Our radiology department is part of a teaching hospital that is spread over three campuses, performing approximately 15,000 MRI examinations per year in total. Voice Recognition System The VR system at the time of the study was a 'runtime' version of Powerscribe 3.5, which is integrated into the GE RIS. All reports were dictated using Dictaphone handpieces. Transcriptionist generated reports were dictated using the same equipment but under 'transcriptionist edit', not 'self edit'. The transcribed reports were then placed into the radiologists' electronic approval queue in the Radiology Information System,(RIS). Two radiologists who had at least four years experience with the VR system, performed the reporting using both the voice recording (before Nov 2008), as well as a transcriptionist (after Nov 2008) to report routine MRI examinations referred during this time period. Case selection Fifty consecutive randomly selected finalized MRI reports generated by VR prior to November 2008 and fifty finalized MRI reports generated by the transcriptionist after November 2008 were sampled retrospectively from the PACS system by an independent investigator for each of the two radiologists. Hence, a total of 200 reports (100/radiologist) were scrutinized for typographical errors and the average TAT from dictate to final approval was calculated from the date and times of the verified reports. This information is printed at the end of each report and stored together with each report on the PACS system. Typographical errors Each report was scrutinized for typographical errors in six categories, wrong word substitution (A), nonsense phrases (B), missing word (C), extra word (D), punctuation (E), or other (F). The most significant error (B) included sentences which were meaningless or contained words completely irrelevant to the report. This analysis was carried out by a third, blinded radiologist who was not involved in the dictation and creation of the reports. Productivity Assessment Page 5 of 17

6 To assess productivity, the average MRI reports per hour for one of the radiologists was calculated using data from weekend reporting sessions, when there were minimal interruptions. The large variability in productivity during normal working hours, due in large part to the number of interruptions, precluded meaningful data from this source. Other sources of variability can include registrar training and scheduling of clinical meetings during the day. Data from at least three weekend reporting sessions using VR only, before the transcriptionist was employed, and three sessions using the transcriptionist only were also available. The only variable between the two groups was the type of report dictation (VR versus transcriptionist). The number of MRI reports, number of sessions and type of MRI examinations was not significantly different between the two groups. Statistical Analyses The proportion of error rates and types of errors between VR and transcriptionists were compared using Chi Square tests. Report turn-around-times between the two groups were compared using Mann Whitney U test. Significance was afforded when p< All analyses were carried out using SPSS (version 16.0, Chicago, USA). Images for this section: Page 6 of 17

7 Fig. 1: Methods Page 7 of 17

8 Results Results: The number of finalized reports between VR and transcriptionist generated reports containing errors and the type of errors are shown in Table 1 on page 8. Forty-two and 30% of the finalized VR reports for each of the two radiologists investigated contained errors. These were mainly wrong word substitution, extra or missing words. There was at least one report which contained a phrase which could have caused misinterpretation. In contrast to the VR reports, only six and eight % of the transcriptionist generated final reports contained errors, all of which were minor and would not change the meaning of the report or be a potential source of misinterpretation. (p<0.001 compared to VR reports for the respective radiologist). Fig 1 on page 8 The average TAT for VR, from reported to verified, was 0 hours. The average TAT for transcriptionist reports was 89 and 38.9 hours for the two radiologists respectively. The mean (± SDEV) time from dictated to transcribed report was six and 27.5 hours respectively. Fig 2. on page 9 Productivity for one of the radiologists was calculated at 8.6 MRI reports per hour using VR and 13.3 MRI reports per hour using the transcriptionist. This represented a 55% increase using the transcriptionist over VR. Fig 3. on page 10 When calculated in Australian dollars, assuming a 40 hour week, 40 weeks per year with an average Medicare MRI rebate of $300 per MRI, the costs for a radiologist (at $204 per hour) and transcriptionist (at $23 per hour), there is a potential gain in revenue to the hospital of $55,000 per week, a 58% increase in revenue or $2.2 Million per year if all the scans attracted a Medicare rebate. The cost per MRI, in terms of wages alone, for using VR and the radiologist as transcriptionist was $6.65. ($23.72 per MRI for VR and $17.07 per MRI for the transcriptionist). When the increased revenue from the rebates was also considered, there was $6.65 also in lost revenue per MRI. Revenue (rebate - cost) per MRI was $ for VR and $ for transcriptionist. Fig 3. on page 10 Images for this section: Page 8 of 17

9 Fig. 1: Results - Typographical errors Page 9 of 17

10 Fig. 2: Results - Turn-around-time Page 10 of 17

11 Fig. 3: Results - Productivity Page 11 of 17

12 Discussion Vendors of VR concentrate on the benefits of VR in radiology, however there is not much evidence available on the errors and productivity associated with VR generated reports. Claims of 99% accuracy rates with VR and other potential benefits are not being realized with anecdotal evidence that radiologists are frustrated and disillusioned by the technology. Voll et al [6] commented that the use of the radiologist as a transcriptionist is poor use of time and resources and our study would confirm this view. Pezzullo et al [4] assessed the use of VR and a transcriptionist for MRI reports in a private practice radiology department. They demonstrated that over 30% of reports contained errors following verification by the reporting radiologist. This was compared to transcribed reports which contained errors in 10% of cases. In another study, 22% of radiology reports were found to contain errors and there was no significant difference in that rate between senior or junior radiology staff, nor between English and non-english speakers (McGurk et al 2008) [7]. Similar results were demonstrated by Rana et al (2005) [3] who showed that VR resulted in significantly more errors than transcriptionists generated reports. However, in that study, there was a significant difference between experienced and inexperienced radiologist with in-experienced radiologist making more errors with the use of VR (Rana et al, 2005) [3]. Our study would confirm this data in a hospital teaching department. In contrast to these studies, Ramaswamy et al [2000)[8] reported on the impact of VR in MRI and demonstrated a decrease in the number of spelling mistakes per report using VR. In that study, the error rate was only about 7%, comparable to those achieved by transcriptionists. The reported TAT with VR appears to be shorter than those by transcriptionists (Rana et al 2008, Pezzulo et al 2008, Ramaswamy et al 2000, Thrall, 2005) [3,4,8,9]. However, the TAT investigated were dependent on the radiologists' experience, English as first language and the use of macros and headsets (Bhan et al 2008) [10]. It is also possible that studies conducted in private practices produce more favorable TAT. In our busy teaching hospital environment, TAT was significantly longer when using VR and timely verification of transcribed reports needs to be improved regardless of the radiologists' characteristics or the type of films reported. Economic evaluation also suggested that installing VR has beneficial effects on costs [8]. This however has been challenged by the more recent studies of Pezzullo [4] and by our present study. Our results have demonstrated that VR is not an effective method of generating reports for MRI. While it is likely that error rates and report times may decrease with newer versions Page 12 of 17

13 of VR, with the present equipment, radiologists are not good transcriptionists and costs per MRI are increased when using VR. The very real disadvantage of transcriptionist generated reports is the TAT in a hospital setting. Any cost saving using VR, in negative transcriptionist wages is well and truly lost in the extra time the radiologist spends correcting their reports. There is no real saving to the hospital when revenue is taken into account. It remains to be seen whether the TAT for the transcriptionist could be substantially reduced by expedient verification of approval queues in the system. It should be pointed out that reports for inpatient MRIs are available within the hospital computer system as an interim report before final verification. The transcriptionists are not physically in the department or hospital but works from home during hours convenient to them. Reports generated over the weekends may not be transcribed until after the weekend, which may explain some of the longer time for the reports of one radiologist to be transcribed. Images for this section: Fig. 1: Discussion Page 13 of 17

14 Conclusions Fig 1 on page 14 Radiologists are not good transcriptionists and VR is associated with increased report errors and costs. In the ideal world, we would have the report error rate and productivity of a transcriptionist and the turn-around-time of VR. With better VR systems we look forward to closing the gap. Images for this section: Page 14 of 17

15 Fig. 1: Conclusion Page 15 of 17

16 References 1. Reinus WR (2007). Economics of radiology report editing using voice recognition technology.j Am Coll Radiol 4(12): Boland GWL, Giumaraes AS, Mueller PR (2008). Radiology report turnaround: expectations and solutions. European Radiology 18: Rana DS, Hurst G, Shepstone L, Pilling J, Cockburn J, Crawford M (2005). Voice recognition for radiology reporting: Is it good enough? Clinical Radiology 60: Pezzullo JA, Tung GA, Jefferey MR, Lawrence MD, Jefferey MB, William WM (2008) Voice recognition dictation: radiologist vs transcriptionist. J Digit Imaging 21: Quint LE, Quint DJ, Myles JD. (2008). Frequency and spectrum of errors in final radiology reports generated with automatic speech recognition technology. Journal American College Radiology 5: Voll K, Atkins S, Forster B (2008) Improving the Utility of Speech Recognition Through Error Detection. J of Digit Imaging 21: McGurk S, Brauer K, Macfarlane TV, Duncan KA (2008) The effect of voice recognition software on comparative error rates in radiology reports. BJR 81: Ramaswamy MR, Chaljub G, Esch O, Fanning DD, vansonnenberg E (2000). Continous speech recognition in MR imaging reporting: advantages, disadvantages, and impact. American Journal of Roentgenology 174: Thrall JH (2005). Reinventing radiology in the digital age: Part 1. The all digital department. Radiology 236: Bhan SN, Boblentz CL, Norman GR, Ali SH (2008). Effect of voice recognition on radiologist reporting time. Canadian Association of Radiologists Journal 59: Page 16 of 17

17 Personal Information Strahan RH 1,* Schneider-Kolsky ME 2 1. Dept of Diagnostic Imaging, Monash Medical Centre, Southern Health, Victoria, Australia 2. Dept of Medical Imaging & Radiation Sciences, Faculty of Medicine, Nursing & Health Sciences, Monash University, Clayton, 3800 Victoria, Australia *Corresponding author: Dr R Strahan Department of Diagnostic Imaging Monash Medical Centre 246 Clayton Road Clayton, VIC Australia 3169 rodney.strahan@southernhealth.org.au Page 17 of 17