A REPORT TO THE BOARD OF COMMISSIONERS OF PUBLIC UTILITIES Electrical - NA Protection & Control - NA \\Q 30 JuLy Transmission & Distribution - NA Telecontrol System Planning - NA Replace MDR-4000 Microwave Radio (West) Various Locations June 2012 newfoundland labrador ) hydro a nalcor energy company
SUMMARY This proposal is for the replacement of the microwave radios on the major communications links connecting the Stoney Brook Terminal Station with the Bay d Espoir Hydro Plant along with the associated multiplexing equipment. This equipment has reached the end of its service life and can no longer be supported. The reliable transmission of communications over this equipment is necessary for the remote control and monitoring of a significant portion of the power grid. This includes teleprotection, SCADA (Supervisory Control and Data Acquisition), voice and administrative data. Newfoundland and Labrador Hydro i
TABLE OF CONTENTS SUMMARY... i 1 INTRODUCTION... 1 2 PROJECT DESCRIPTION... 2 3 JUSTIFICATION... 4 3.1 Existing System... 4 3.2 Operating Experience... 7 3.2.1 Reliability Performance... 7 3.2.2 Industry Experience... 7 3.2.3 Vendor Recommendations... 8 3.2.4 Maintenance or Support Arrangements... 8 3.2.5 Maintenance History... 8 3.2.6 Historical Information... 8 3.2.7 Anticipated Useful Life... 8 3.3 Development of Alternatives... 9 4 CONCLUSION... 10 4.1 Budget Estimate... 10 4.2 Project Schedule... 11 APPENDIX A... A1 Newfoundland and Labrador Hydro ii
1 INTRODUCTION Hydro currently maintains an Island wide communications system for the support of the power generation, transmission and distribution facilities. This project will replace the remaining high-capacity OC-03 MDR-4000 microwave radios that were installed in 1999 between the Stoney Brook Terminal Station and the Bay d Espoir Hydroelectric Generation Station including the upgrade or replacement of the Telnar 1603 Synchronous Optical Network (SONET) multiplexer to include Ethernet access. The multiplexers were also installed in 1999. This radio system provides teleprotection, supervisory control, departmental and administrative voice and administrative data between Hydro Place and the Bay d Espoir, Upper Salmon and Granite Canal hydroelectric generating stations. High speed and highly reliable telecommunication facilities are required to ensure the stability and reliability of the provincial power grid. Multiplexers are data conversion units used extensively in communication networks to multiplex (combine and/or break out) multiple lower speed circuits for different equipment and purposes, into higher speed data circuits. These higher speed circuits are then suitable for transmission by the microwave radio. The particular grade of multiplexer considered for replacement in this report is a SONET multiplexer which has 155 Mbps connections to the radio and primarily DS1 (1.544 Mbps) connections for the lower speed connections. This proposal is to replace radios in the west portion of the Hydro microwave system. A separate proposal to replace radios in the east portion of the Hydro microwave system is to be submitted in the future. Newfoundland and Labrador Hydro 1
2 PROJECT DESCRIPTION The radio links to be replaced are listed in Table 1. The 1603 multiplexers to be replaced are listed in Table 2. It is proposed to replace the existing Alcatel MDR-4000 microwave radios with Alcatel MDR-8000 radios with the same data capacity using the same frequencies. Table 1: MDR-4000 Radio to be Replaced Radio Location Toward 1 Stoney Brook Terminal Station Sandy Brook Hill Microwave Repeater Site 2 Sandy Brook Hill Microwave Repeater Site Stoney Brook Terminal Station 3 Sandy Brook Hill Microwave Repeater Site Gull Pond Hill Microwave Repeater Site 4 Gull Pond Hill Microwave Repeater Site Sandy Brook Hill Microwave Repeater Site 5 Gull Pond Hill Microwave Repeater Site Bay d Espoir Hill Microwave Repeater Site 6 Bay d Espoir Hill Microwave Repeater Site Gull Pond Hill Microwave Repeater Site Table 2: 1603 Multiplexers to be Replaced Multiplexer Location Toward Multiplexer(s) at 1 Stoney Brook Terminal Station Sandy Brook Hill Microwave Repeater Site 2 Sandy Brook Hill Microwave Repeater Site 3 Bay d Espoir Hill Microwave Site Stoney Brook Terminal Station and Bay d Espoir Hill Microwave Site Sandy Brook Hill Microwave Repeater Site and Bay d Espoir Hydro Plant 4 Bay d Espoir Hydro Plant Bay d Espoir Hill Microwave Site The Alcatel MDR-8000 microwave radio is currently utilized elsewhere in the Hydro microwave radio system. A map of the system is shown in Figure 1. This allows Hydro to utilize the existing spares, training and mock-ups currently available. Newfoundland and Labrador Hydro 2
Modules from the retired equipment will be used to prolong the service life of other links in the system. The choice to upgrade or replace the 1603 multiplexers will be made during the detailed design phase of the project based on functionality, reliability, cost and expected life cycles. Note that while the microwave radios and multiplexers are a critical component of the microwave facilities, they are part of the overall infrastructure which includes equipment shelters, large stationary batteries, backup generators, high towers, waveguide, antennas and many other support items. None of this equipment is proposed for replacement by this proposal. Newfoundland and Labrador Hydro 3
3 JUSTIFICATION The project is justified on the need to replace obsolete equipment. The Alcatel MDR- 4000 radio has been discontinued by the manufacturer since 2004. The manufacturer has also discontinued the guaranteed return and repair of failed modules since 2009 due to the obsolescence of the components within the modules. Failed modules have been returned as un-repairable in 2011 as a result. Hydro has obtained some components from the used market, however this supply is unreliable. Some components were also obtained from the microwave link decommissioned in 2010 between the Bay d Espoir Hill microwave site and the Bay d Espoir hydroelectric generating station to extend the life of the radios to this time. The replacement of the radio system is required to ensure a reliable and safe provincial power system. The spares freed up from this capital project will extend the life of the MDR- 4000 radios on the Avalon Peninsula until the scheduled replacement in 2016. 3.1 Existing System Hydro has a network of seven gigahertz (GHz) frequency microwave radio, by which corporate communications and system data are transmitted. The microwave radio system provides the backbone for corporate voice and data communications. Traffic carried over the microwave system includes: Teleprotection signals for the provincial transmission system; Data pertaining to Hydro s Supervisory Control and Data Acquisition (SCADA) system; Data pertaining to the corporate administrative system; and Operational and administrative voice systems. The existing microwave network is composed of the main backbone of a series of SONET radios from the Energy Control Center (ECC) in St. John s to the Sandy Brook Hill Microwave Newfoundland and Labrador Hydro 4
Repeater Site and from the Stoney Brook Terminal Station to the Bay d Espoir Hydro Plant. The SONET capacity is 155 Mbps (OC-3). Secondary, lower capacity microwave links and other communications methods are fed from the main microwave backbone as shown in Figure 1. The site locations and links affected are shown in Figure 1 with the specific equipment at each site detailed in Table 1 and Table 2. The string of microwave links affected are from the Stony Brook Terminal Station (STB) in the North central part of the Island to the Bay d Espoir Hydro Plant (BDE) in the South central part of the Island. There is a list of station abbreviations used in Figure 1 provided in Appendix A. Newfoundland and Labrador Hydro 5
Figure 1: NLH Telecommunications Network showing the Microwave Radios Newfoundland and Labrador Hydro 6
3.2 Operating Experience The original Alcatel MDR4000 radio and 1603 multiplexers were installed in 1999 and have proved to be a valuable asset in the remote control and monitoring of the provincial power system. However, the equipment has been discontinued by Alcatel. 3.2.1 Reliability Performance The communications equipment proposed for replacement is an integral part of the communications used for monitoring and control of the power grid requiring a very high reliability in order to provide the continuous support. The path design parameter used is 99.9999 % availability. Redundant hardware and multiple frequencies are used to achieve this design. 3.2.1.1 Outage Statistics The following are the communications outage events since 2005: Table 3: Outage Statistics STB - BDE Microwave Outage Statistics Time of Outage Outage Duration 2009 - Summer 2 hours 2008 - Spring 6 hours 2005 - Winter 30 minutes 2005 - Spring 2.5 hours 3.2.2 Industry Experience Power utilities throughout North America use dedicated microwave systems in support of its operations in the same way that Hydro uses its system and use the type of equipment listed for replacement in this proposal. All of the major power utilities in Canada have similar systems. Special permission and frequencies are allocated by Industry Canada for this requirement. It is a common practice for utilities to replace this equipment in a planned proactive method to avoid communications outages. Newfoundland and Labrador Hydro 7
3.2.3 Vendor Recommendations The vendor has discontinued the MDR-4000 microwave being replaced and has recommended the MDR-8000 microwave as a replacement, as proposed in this report. 3.2.4 Maintenance or Support Arrangements Corrective and preventative maintenance of the equipment is provided by internal forces. There is also an existing technical support agreement with Alcatel to provide detailed hardware and software support. The replacement equipment will be maintained using the same arrangement. 3.2.5 Maintenance History Detailed preventative and corrective maintenance costs have not been kept on this equipment. A total of $115,000 has been recorded since 1999. 3.2.6 Historical Information There has been one similar project completed in the last five years. The project was for the replacement of the microwave radio between the Bay d Espoir Hydroelectric Generating Station and the Bay d Espoir Hill microwave site. There were no multiplexer replacements. Year Capital Budget ($000) Table 4: Historical Information Actual Expenditures ($000) Units Cost per unit ($000) 2010 488.7 375.0 2 radios 187.0 Comments 3.2.7 Anticipated Useful Life Hydro microwave radio equipment is depreciated over 22 years. However, the useful service life can vary depending on the vendor support, and the availability of spares. Newfoundland and Labrador Hydro 8
3.3 Development of Alternatives There are no viable alternatives to this proposal at this time. The existing microwave infrastructure is the only available facilities for providing the required communications to this part of the island. Newfoundland and Labrador Hydro 9
4 CONCLUSION The replacement of this microwave equipment is necessary in order to prevent outages caused by the failure of older equipment that can no longer be supported for the high reliability required for critical, high speed, low latency communications. This replacement proposed by Hydro is based on operational experience and manufacturer s recommendations. Due to financial and operational risks associated with the failure of corporate microwave equipment, this project is required to minimize the likelihood of failure of these microwave radios. 4.1 Budget Estimate The budget estimate for this project is shown in Table 5. Table 5: Project Budget Estimate Project Cost:($ x1,000) 2013 2014 Beyond Total Material Supply 400.0 73.6 0.0 473.6 Labour 90.0 307.6 0.0 397.6 Consultant 0.0 0.0 0.0 0.0 Contract Work 0.0 0.0 0.0 0.0 Other Direct Costs 22.0 41.4 0.0 63.4 Interest and Escalation 27.0 97.4 0.0 124.4 Contingency 0.0 186.9 0.0 186.9 TOTAL 539.0 706.9 0.0 1,245.9 Newfoundland and Labrador Hydro 10
4.2 Project Schedule The anticipated project schedule is shown in Table 6. Table 6: Proposed Project Schedule Activity Start Date End Date Planning Complete a detailed schedule and budget with resource assignments. January 2013 March 2013 Design Complete site visits. Complete a detailed design with all items and outages identified. April 2013 November 2013 Procurement Complete all ordering and detailed schedules for delivery. October 2013 February 2014 Construction Complete all outage requests and install and test all equipment at site. May 2014 June 2014 Commissioning Cutover new equipment during staged communications outages. July 2014 August 2014 Closeout Complete all drawings and project management closeout procedures. November 2014 December 2014 Newfoundland and Labrador Hydro 11
Appendix A APPENDIX A Site Abbreviations Newfoundland and Labrador Hydro A1
Appendix A Abbreviation BAH BDE BDH BFI BGH BUC CAH CBC DLK DLP ECC FMH GCH GDH GPH HRP HWD MMH NDH OPD PHH SBH SHH SPH SSD STB USL WAP WAV Site Name Bull Arm Hill Microwave/Repeater Bay d Espoir Hydro Plant Bay d Espoir Hill Microwave/Repeater Bishop Falls Office Blue Grass Hill Microwave/Repeater Buchans Terminal Station Chapel Arm Hill Microwave/Repeater Come By Chance Terminal Station Deer Lake Terminal Station Deer Lake Passive Repeater Energy Control Center Four Mile Hill Microwave/Repeater Granite Canal Hill Microwave Godaleich Hill Microwave/Repeater Gull Pond Hill Microwave Holyrood Plant Hardwoods Terminal Station Mary March Hill Microwave Notre Dame Hill Oxen Pond Terminal Station Petty Harbour Hill Microwave/Repeater Sandy Brook Hill Microwave Shoal Harbour Hill Square Pond Hill Sunnyside Terminal Station Stony Brook Terminal Station Upper Salmon Plant Western Avalon Passive Repeater Western Avalon Terminal Station Newfoundland and Labrador Hydro A2