Contingency Planning. for the Year 2000

Transcription

1 Chris D. Carroll 1 Contingency Planning for the Year 2000 Chris D. Carroll Director, Contigency Planning

2 Chris D. Carroll 2 Canadian Pacific Railway Suite 400 Gulf Canada Square th Ave. SW Calgary, AB T2P 4Z4 A Certified Business Continuity Professional with the Disaster Recovery Institute International of Missouri and the Disaster Recovery Institute Canada. Member of the American Association of Railroad Superintendents.

3 Chris D. Carroll 3 Featured in countless articles, the focus of Companies, Government Industries, the general public, and the subject of much debate, the Year 2000 has changed the way we think about computers and technology forever. This paper reports on Canadian Pacific Railway s Contingency Planning for the millennium. Discussing risk analysis, risk mitigation through workaround solutions and business recovery or restoration, the paper highlights the three phases of this final stage of preparation, and how the company is working to ensure that regardless of circumstance, the Railway will continue to operate without Y2K impact. Keywords: Year 2000, Contingency Planning.

4 Chris D. Carroll days, 11 hours, 55 minutes... Catchy, isn't it? It is not my intention to speak to all that good work of repair, remediation, testing, replacement, certification, letters of comfort, nor to reflect upon the countless hours spent explaining oneself to auditors. This is not what Contingency Planning is about, nor at this point is it very topical. It's more to the point of what is required "just in case". It's a little like that encounter one had as a child at the front door every morning, "Did you bring your coat?" The Year 2000, or more specifically, the stroke of midnight December 31st, is shaping up to be the biggest "what if" scenario that business and industry has ever faced. No amount of preparedness seems to be considered as sufficient. Interestingly, it is also a unique situation from a business continuity perspective. All other types of situations one plans and prepares for are unknown events in time, but are known in circumstance. We have all practiced evacuating buildings in the event that the fire alarm sounds. Y2K is the exception. We all know exactly when it will happen, we just cannot be certain what, if anything, will happen. Even with all the work and planning done to date, no one organization or company has exhibited the intestinal fortitude to come forward and report that they will function without issue or exception. This may of course be driven more by a desire to minimize liability, than by a true reason for concern, but it also recognizes our strong dependency on others to be able to conduct business in our global economy. The bright side to this equation however, is because of the due diligence exhibited by many government organizations and companies large and small, the North American infrastructure has arguably never been in better shape. But again, back to the skepticism, so what?

5 Chris D. Carroll 5 There are three main tenants to Business Continuity or Contingency Planning. Risk Identification, Mitigation of Risk and Business Recovery or Restoration. In identifying the risks associated with Y2K, Canadian Pacific Railway determined that our greatest dependencies, and therefore our greatest vulnerabilities, lay without outside suppliers of electricity and communications. This is not to say that we had proven that these two key supplies were in jeopardy of failure, but more importantly, that they were areas where the greatest negative impact would be felt to our business if in fact they did fail. In addition, CPR recognized our increased dependency on IT systems. In the last two decades, we have gone from two ton vacuum tube Centralized Traffic Control machines, train orders, motor cars, trackside phones, coal oil switch lamps and the venerable teletype, to computerized CTC and track warrant control, multi-channel digital radios, highrail trucks, satellite phones, and laptops with more computing power than was used to run the Mercury space program. This technology is not however without cost, as it is also an area of great vulnerability. Once these three, power, telco and IT were identified as CPR's three greatest areas of risk, a window of outage was then attached, in order to put some size and scope to the problem. Through meetings, correspondence and examination of test and certification data provided by our suppliers, CPR developed what was termed the "most reasonably, likely, worse case year 2000 scenario" (That phrase had to have been written by a lawyer), namely; Electric Power and/or Telecommunications (voice or data) loss in multiple local sites for a duration of up to 24 hours, repeating over a period of 2 to 4 weeks after December 31, In addition, in analysis with our IT department, it was determined that because a Y2K related failure of one or more of our IT systems would be date specific, no more than seven days would be required to repair

6 Chris D. Carroll 6 and restore the affected system or systems. Finally, we needed to identify exactly where these risks would have the greatest impact. By categorizing all facilities and systems, then correlating one to the other, a list of 7 rail critical sites, 11 major yards, 19 intermodal terminals and 69 crew change locations was developed. Armed with these pieces of information, we then proceeded to Stage Two, the mitigation of risk. Railways, simply because of the nature of our business, are fairly self-sufficient. We construct our own right of way, repair and rebuild large pieces of equipment, house and feed legions of employees, and maintain and deploy huge inventories of material, all the while through some of the most inhospitable terrain this continent has to offer. And we didn't just start doing this yesterday. Power outages are something that we are accustomed to, and for which contingencies already exist. Battery banks, UPS equipment and stand by generators are all things that are part of our day to day operation, and their use rarely raises an eyebrow. CPR examined the list of critical sites and locations to ensure that existing contingencies were sufficient, and where they were found lacking we either improved or installed new. These improvements ranged from increasing fuel capacity on existing backup generators to conversion of SD40 locomotives for use as power plants in our major yards. Our dependency on telecommunications wasn't nearly as easy to mitigate, much less reduce. In September of 1997, CPR undertook a major capital project to install a trackside fibre network, somewhat of an irony, given that twenty years ago we had the equivalent, known simply as a pole line. This system, which would allow us to again be our own phone company, was not planned to be entirely completed for the millennium change. Thus, recognizing our continued reliance on outside supply of telecommunications, we re-examined the test and certification data provided and made the determination that the North American communications backbone would not be significantly impacted. In addition, we are confident that the combination of available suppliers affording path redundancy, coupled with the portions of fibre that CPR has operational, would hold us in good stead.

7 Chris D. Carroll 7 This left us with one final area of telecommunications risk as identified in our analysis, referred to in the industry as "last mile failure". At this point, we again fell to our existing contingencies designed to deal with this type of outage, use of cellular phones and applicable operating rules, and felt that this type of failure was manageable. One major area requiring focus remained, our IT systems. We examined each of our systems in turn, and by applying to them our seven day outage template, boiled the list down from several hundred to twenty two that were considered fatal to the Operation. Each one in turn was then examined to determine what manual workaround solution could be implemented in order to permit continued operation in the event that the system failed. As in the case of power outages, we referred back in our history, and pulled from the past old methods of manual Track Warrant Control, car checking, paper inventories, crew calling with phone lists, all methods previously employed for decades prior to implementation of technological upgrades. Now that the workaround or contingency plans were complete, one piece remained to tie it all together, a process CPR refers to as the day one Health Check. This in essence would be our Business Restoration or Recovery phase. For the past several years, CPR, as many other carriers, has curtailed operations over the Christmas and New Years statutory holidays, for two basic reasons. First, to improve the quality of life for our employees, and second, recognizing that most if not all of our customers are closed, delivery of product is not possible for the four days in question. We simply have no place to take the traffic. To that end, CPR has again scheduled our standard New Years shutdown period, from 1800 December 31 st until 1000 January 2 nd. However, sending our operating employees home at 1800 New Years eve is where the similarity to previous years ends. This year, a legion of supervisory and technical staff will remain on duty across the system, to execute a detailed shutdown and startup process. The first step, commencing at 1800 local time, is to prepare all the documentation that may be required to implement one or more of our manual workaround solutions.

8 Chris D. Carroll 8 A standing inventory of all yards and facilities, and the location of all locomotives and trains will be produced to give us an accurate snapshot of what our railroad looked like prior to the date change. At 2330 ET, all access to our mainframes, servers, and systems will be shut off by our IT Department, in preparation for the stroke of midnight. At 0301 ET, once the millennium wave has crossed in to the Pacific, the actual Health Check process will begin. IT infrastructure testing will commence, ensuring that all the main physical components of our IT network continue to function correctly. This will be done simultaneously in five locations. Concurrent to this phase, Signals & Communications will begin office testing of our rail traffic control systems. At approximately 0400 ET, provided our IT infrastructure is functioning, we will begin bringing each one of our critical systems on line, and perform testing to ensure all is well from an IT perspective. Subsequent to passing these hurdles, operations personnel will then commence functional testing of the systems. The final health check will then be conducted, actual movement testing. A representative set of subdivisions have been selected for the operation of special train consists, comprised of single units, several cars, and a caboose, for the transportation of signal maintainers and technicians. Function of all trackside equipment will be checked to ensure full connectivity between office and field. A post mortem will then be conducted, with final work around solutions implemented where required. Full operation of the railway will then commence at 1000 January 2 nd, within the scheduled shutdown window, and fully transparent to the customer. Will it be enough? Time, unfortunately will tell. Our ability as an industry to be innovative and resourceful in times of adversity, coupled with a level of preparedness unmatched in our collective histories, should make this New Year s as uneventful as those in the past. How confident are we? One final contingency has been put in place, a New Years party for all those involved, scheduled for January 2, 2000.