Survive Splitting an EHR into Private and Public Entities March 3, :30PM - 3:30PM PT

Transcription

1 Survive Splitting an EHR into Private and Public Entities March 3, :30PM - 3:30PM PT Marcus Hobgood, Chief Information Officer, University Health System

2 Conflict of Interest Marcus Hobgood Has no real or apparent conflicts of interest to report. Marcus Hobgood Chief Information Officer University Health System Marcus D. Hobgood, is University Health System Chief Information Officer and former Chief Information Officer of LSU Health Science Center Shreveport. As the state director of the LSU Health electronic health record project, Mr. Hobgood led LSU Health to became the first state run public health system to successfully implement a fully electronic health record system. Mr. Hobgood has over 15 years in healthcare information technology experience, is a member of the American College of HealthCare Executives, a HIMSS Analytics member, the Gartner CIO program, and a member of the Cyber Information Technology Advisory Board.

3 Agenda Introduction to University Health Challenge presented by privatization of the LSU System University Health s mission DRIVE Cost out of EHR platform (Initiative and Vision) Summary Post privitization

4 Learning Objectives Describe how to successfully manage an organization through a major birfurcation of two hospitals Discuss how to create operational efficiency and EHR responsiveness despite a major transformation Identify steps needed to ensure improved performance after a major transformation

5 Goals University Health System s goals were to: 1. realize savings 2. increase satisfaction 3. expand patient engagement

6 Mission: Impossible Dividing One Epic System Serving Five Hospitals into Two Epic Systems Serving Two Hospitals Each

7 University Health System University Health System is a northern Louisiana healthcare network, consisting of two hospitals, University Health Shreveport and University Health Conway in Monroe. Specialty Services and Centers of Excellence: Level 1 Trauma Center Regional Burn Center Feist-Weiller Cancer Center St. Jude Affiliate Clinic Children s Hospital Center for Excellence for Arthritis and Rheumatology Facts: Shreveport Licensed Beds Conway Inpatient Admissions 20,902 6,648 Births 1, Emergency Visits 60,425 35,618 Outpatient Visits 362, ,662 Medical Staff Residents 539

8 University Health History University Health was once a part of the LSU Health state hospital system that included 10 hospitals. From Shreveport to New Orleans, our charity hospitals have provided care for its citizens for over 200 years. Of the 10 hospitals, 5 joined together as the first state hospital system to implement Epic for EHR. The first go live was in November 2011 and the last in March 2013.

9 Mandatory Privatization Due to deep cuts in the state budget, our governor made the decision to privatize the Louisiana public hospital system. This made it possible for the hospitals to continue providing patient services.

10 Our Mission

11 Our Mission One EHR, 10 hospitals Started with the Shreveport hospital, Nov 2011, 5 th one done 2013 Change of plans. State wanted no hospitals, started looking for organizations interested in buying 2014 started plan to split EPIC. Children s bought NO, BRFF bought EAC/SHV, HPL closed and LAK still state Completed in May 2015.

12 Current Environment Strategy The plan was new compute, SAN and Virtualization One team was working on deciding hardware and getting ready to start GenIO testing on new SAN My cache dba team, EPIC TC(Sarah) and server TS(Jeff) and i were coming up with a plan on how we could do this split. EPIC doesn t have a lot of split project plans laying around so we did took a normal GoLive project plan and tried to pick out the items that we knew we would have to do. This was our existing setup that was supporting the 5 hospitals for the state(lsu). We also have REL,RELBLD,VLD for upgrade purposes. We had to take this and create two completely separate instances.

13 Technical Practice Migration On the left side, was our existing IBM compute and SAN plus UCS blades for NonProd On the right side, was our new Dell compute and SAN plus the UCS blades for Nonprod So to start, we created RedHat VMs for our new Prod and TST. Then we took backups and restored to create UH PRD and UH TST We decided that rather than build the new UH POC and have the analysts build there yet, we continued to have the analysts build in LSU POC, build for the current live EPIC would go on to PRD and be auto DC d to UH TST. New build was manually DC d to LSU TST then UH TST Created MST from LSU MST, Ace envs created later

14 Cutover / Go-Live For cutover Stop shadowing, finish build for UH PRD, Stop the DC from LSU TST TO UH TST

15 Timeline Started May 2014 with hardware discussions Create a new UH(University Health)domain Created new VM datacenter Created new Citrix Farm All UH users were create new domain IDs Their EMPs in live EPIC were flipped to these new IDs

16 Timeline - continued Created all NonProd EPIC servers, Xenapp/Hyperspace,BLOB,CareEverywhere Jan 13 th had EPIC NonProd build trip At this point we had TST

17 Timeline - continued Since our UH Prod env was a shadow of LSU env, we couldn t connect the Windows client servers except during a cutover practice. The shadow was brought down, env brought live and testing would happen. Feb 25 th had the production build trip which included Client, server and interface TS. Configurations on production windows servers were completed. In Mar tested document scanning, barcode scanning and printing April 13 our 3r and final cutover test. Went from taking a day ½ to run clinical scripts to remove LSU inpatients, down to a little over 4 hours After cutover, we changed our test interfaces to point to UH PRD and completed full integrated testing. Since our production interfaces were still live on the LSU instance, we couldn t test with them.

18 Interface Split Most interfaces were already divided between the LSU OL engine and the UH Corepoint engine but all connections to EPIC flowed thru the CP engine. Had to change Ips for any interfaces going to the new UH production Since we were a new company, we got new AIP number. So all our interfaces went from 747 prefix to 582. Plus we renamed them from LSU_ to UH_ Handful of interfaces that had to be created, Surescripts, MdStaff, GE RIS/Pacs(same interfaces but new RIS/PACs) For the Ris/Pacs there were new servers and thus new Ips. Interfaces had to be connected to the new servers and then there was full integrated testing.

19 Cutover Steps These are just the highlights of the steps performed during each cutover and then the final GoLive. Along with the EPIC split, we were also splitting off and bringing our own instance of GE RIS/PACS. This started at the same time. Stop shadowing Build UH PRD Disable UH employees access to the LSU EPIC, LSU employees can continue to work Sarah was doing some configuration of new env, plus licenses Started a script to discharge all LSU inpatients 4 hours

20 Simultaneous Migration Currently, our Production, Reporting, and Support instances are hosted on 2 Dell r920s running Red Hat 6.5 and VMWare ESXi v 5.5. The non-prod environments are living on Cisco UCS hosts. For storage we are using Dell Compellent SC8000 w/ all flash array drives. When we first implemented Epic in 2011, Intel was not supported. We went live with IBM Power 770/780 physical servers running AIX 6.1 w/ IBM DS K storage. It worked well for the most part but we did see some I/O performance issues. We were the first Epic customer who attempted a simultaneous migration of both servers and storage, going from physical to virtual, AIX to Red Hat Linux as well as splitting the patient database all at one time. Both Epic and Dell had a huge stake in the success of this undertaking and were there throughout the project, providing support on every level, both remotely and onsite.

21 Pre-Migration Infrastructure

22 Documentation This is the documentation we used most of which is available on Epic s UserWeb. Preliminary Hardware Configuration Guide (created for our organization) Cache on Linux on VMware Architecture Operational Database Storage Layout Recommendations Epic s Operating System Settings and Security Setup and Support Guide Cache Install and Upgrade Setup and Support Guide Dell server and storage migration checklists

23 Post-Migration Infrastructure This is a diagram of our current layout. We had an excellent support team from both Epic and Dell. The Dell team made several trips to our site, guiding us through the initial install, testing the servers and storage to make sure that they met the performance criteria specified by Epic, as well as providing cutover support. Jeff Karle, our Epic TS was also immensely helpful and both were available day and night to help us no matter how big or small the issue.

24 Post-Migration Flow The VM s were designed around the environment functionality. The Production VM lives on a dedicated Dell host. We are using VMWare clustering for high availability (? Ask Huey). Reporting and Support live on the 2 nd Dell host as separate VMs. We grouped the Development POC and TST - together on the same VM on a Cisco UCS host. The Training instances MST, REF, PREP, ACE, and PLY - are also grouped together on the same VM on a Cisco UCS host. Release Engineering - REL, RELBLD, and VLD - will be built as we get closer to our next major upgrade.

25 How Did We Get There? Created Gold Image VM Created VM from Gold Image for Gen IO Testing Determined Cache Migration method Created UH TST environment by copying LSU TST Created UH PRD environment by copying LSU PRD Cutover Testing

26 Gen IO Testing Before the new Dell server and storage hardware could be setup for the migration, we had to set up a VM on the new Dell host and test it. We followed Epic s recommendations in the Operational Database Storage Layout document. Epic's Gen-IO tool was used to identify configuration and hardware issues. The critical factors that had to be reviewed were: Average write latency < 1ms Average random read latency < 15ms 99% of reads < 60ms 99.9% < 200ms 99.99% < 600ms Write cycle needs to be less than 80s

27 Gen IO Test Results

28 Gen IO Test Results After the faulty host bus adapter was replaced, our gen i/o test was successful and we given the green light to proceed with building the new production server.

29 Cutover Testing

30 Cutover Data Migration

31 SUP Refresh

32 Physical I/O

33 Grefs/s

34 Spindles vs All-Flash Array

35 For once printers were not the issues Because we created our new AD accounts and changed our EPIC EMPs over to these prior to the split, all those issues were already taken care of by the time of the split. With the icons divided by locations prior to the split, we were able to block the UH users from access to the LSU EPIC and allow LSU to continue working. Make sure you Order, receive and install all hardware long before the split. We were still setting up secondary SAN the day before. Some workstations were still pointing to the LSU Citrix URL for icons, Physicians were still logging into desktop with the LSU ID but logging into EPIC with UH ID When you bring a shadow env live and interfaces were set to autostart on the original env, they will start up before you get connections straight. Live outgoing interfaces pointed to the production engine, can connect to the new environment. Thus sending orders and such to a non-production env. So what happens when you have a room full of geeks, hyped up on caffeine, who just pulled off a successful project? They start thinking again. Without testing, the tech team decided to do a Veeam backup of the live primary SAN for EPIC. Veeam uses VMware native snapshots and there is a problem when attempting a snapshot on large systems. The IO latency gets so high VMware disconnects the disks which caused Prod to go down again. Then a little later that day, they decided to swap the live data from Prod over to the secondary SAN. Unfortunately there was a step missing on this process and is caused the storage to disappear from Prod. Never test on a live production.

36 Response Times Of all the exciting things we did with the split, this is the best. With the new hardware, compute and SAN, we brought our Response Time Exceptions percentages down from Around 1.20% to under.25%. Our users should be happier.

37 Conflict of Interest Jason Jones Has no real or apparent conflicts of interest to report. Jason Jones National Director of Healthcare Strategy Dell Healthcare Enterprise Solutions Group Jason Jones is the National Director of Healthcare Strategy with Dell Healthcare and Life Sciences. He has implemented and managed an EHR in a clinical setting and currently works with leading clinical, academic, research and biotechnology organizations to integrate and implement healthcare IT (HIT) enterprise platforms. As the lead for Dell s enterprise healthcare strategy practice, he is working to address the cost and complexity of HIT and ongoing management and maintenance in the data center.

38 Delivering Innovation in EHR

39 A Vision for Healthcare Make It Personal! Life Sciences Healthcare IT & Medical Devices Consumer Health Diagnosis based on individual genome for a targeted treatment plan Seamlessly connect patients, clinicians and data for holistic proactive care across settings Engaging connections among people, their data and care community empower health ownership Powerful & Trusted Computing, Storage and Communications Advances from Intel Intel Health & Life Sciences Make it Personal

40 Intel in Health & Life Sciences Intelligent Systems Workstation/ Desktop/All in One Ruggedized 2 in 1/ Ultrabook* Tablet Phone & Handheld Wearable / Internet of Things One architecture across connected devices Enabling a Secure, High Performance, and Consistent User Experience Intel Health & Life Sciences Make it Personal *Ultrabook products are offered in multiple models. Some models may not be available in your market. Consult your Ultrabook manufacturer.

41 A summary of how benefits were realized for the value of Health IT Direct Savings: 40% lower TCO Satisfaction: Dropped response time from 1.20% to.25% Patient engagement: Expanded patients use of portal technology

42 Questions

43 Legal Disclaimers INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined." Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. Approved use: NDA only Do not post publicly In countries where restrictions on competitive comparison materials exists PRC, Russia, Belarus, Kazakhstan, Ukraine, Vietnam the material must be shown under NDA only with no leave behind. Intel, the Intel logo, Intel Atom, Intel Core, Intel. Experience What s Inside, the Intel. Experience What s Inside logo, Intel RealSense, Intel Unite, Intel vpro, Intel Xeon Phi, Ultrabook and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. Copyright 2015, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.