2016 Operations Stay Treat Improving System Reliability. A Case Study of Accelerator UPSs. Anthony Cuffe

Transcription

1 2016 Operatins Stay Treat Imprving System Reliability A Case Study f Acceleratr UPSs Anthny Cuffe

2 Summary A Case Study f Imprving UPS Reliability Typical UPS Usage Backstry Initial Status and Analysis Reliability Study Maximizing Availability, Reliability and Manageability Autmatin, Mnitring and Lgging Tls Results Key Pints Q & A

3 Typical UPS Usage in A.C.E. Line Pwer Line Pwer & Battery Pwer Prtectin frm Pwer Lss and Pr Line Quality fr: Critical Wrkstatins and Servers Netwrk Infrastructure RF, Vacuum and ther critical IOCs

4 Backstry What just happened? Histrically ur grup maintained arund rack-munted, enterprise grade UPSs. This was manageable withut much effrt. After a re-rg, ur grup became the prud wners f 100+ UPSs. And mst f them weren t shiny and new! After staging thrugh denial, anger, bargaining, and depressin, we accepted it. This is hw we gt ver mst ur grief.

5 Initial Status The Reality f Square One Respnsible fr 100+ UPSs (and grwing!) ~10 were utright failed ~10 were in a faulted state 40+ were fund t be 8-16 years ld (5 year lifetime) Cnfiguratin was manual, nerus and incnsistent Deplyment tk a very lng time (1++ hur) Mnitring was impssible Sheer vlume f errr messages (X,000+/per day) System age Our attitudes twards the system were awful Dread Apathy A few resumes were tuned up

6 Initial Analysis Is the system wrth imprving? Is the system s functinality still necessary? Servers, netwrk gear and critical IOCs need t survive pwer interruptins. UPSs prvide the nly lw-cst slutin t fulfill this need. Is the benefit wrth the cst and staff time? ~ $50K t make the verall system reliable and ~$10-20K/year t maintain The recvery csts and reduced beam availability assciated with frequent pwer-utages greatly exceeds the cst f maintaining UPSs fr critical systems. Is it pssible t imprve the cst/benefit rati? We felt we culd decrease the time spent maintaining these systems by applying practices cmmn t ur ther systems. The benefit culd be imprved with better reliability and selective applicatin. Csts culd be spread ver years with a well maintained system. D yu have management buy in? We had t cnvince management that the benefit wuld be wrth the time, effrt and prcurement dllars. It is impssible t imprve r maintain a system withut resurces! Cst Benefit Analysis

7 Reliability Study UPS Failure Rate Failure Rate Near Cnstant Failures Safe Zne FY17? Increasing Failures Where we think we are nw Where we were Burn-In Failures Rare Battery Failures Randm Suggested Lifespan 6-10 Years Time 1 st Weeks 4 Years 6 Years 10 Years 15 Years * Curve estimated based n experience, dependent n perating envirnment

8 Imprving Availability Availability Depends n Reliability and Manageability AVAILABILITY MTBF (MTBF + MTTR) Maximize Reliability Make MTBF as big as pssible Mitigate unreliability where MTBF cannt be reduced Maximize Manageability Make MTTR as small as pssible Practive replacement/maintenance where MTTR cannt be reduced MTTR - Mean Time t Repair. MTBF - Mean Time Between Failures. Availability, Reliability and Manageability

9 Maximize Reliability Increase MTBF Get the System t a Manageable State (stability pint). Inventry audit Yu have t knw what is ut there. Replace failed units. Repair/replace units in errr/faulted state. Reduce Cmplexity Get rid f unnecessary units. Remve unnecessary cmpnents (PDUs, Transfer Switches ) Reduce unwanted lads n the system. Implement Redundancy Design/Deply systems with redundant pwer supplies. Implement redundant systems s failures are actually tlerable. Mnitr fr Faults, Failures and Predicted Failures Alerts frm units themselves and active plling Alerts fr lw Runtimes, Faults and Failures Perfrmance Graphing fr histrical perspective and statistics Failure Predictin Centralized Lgging fr histry and analytics

10 Maximize Maintainability Decrease MTTR Crrective rather than Preventative maintenance Predicted failures are replaced befre prblems arise Actual failures are replaced quickly Batteries are replaced when there are faults nt n a schedule Cnfiguratin changes and updates are dne glbally On the shelf spares Implement redundancy wherever pssible MTBF and MTTR are zer if system is redundant! Critical systems always have dual pwer supplies and UPSs Autmate as much as pssible Research available tls (pen surce and vendr prvided!) Autmatin prduces cnsistent and cmplete cnfiguratin Custm scripts can autmate tasks and deplyment (~5 min) If yu d smething mre than nce, yu shuld write a script t d it!

11 Vendr Tls Autmatin and Mnitring APC UPS Netwrk Management Tls Mnitring, Cntrl and Remte Access Web brwser Cmmand line interface SNMP, SSH, FTP... Event and Data Lgging Scheduled Self-Tests Ntificatin Faults, Failures & Warnings Self-Test Results Cnfiguratin Changes

12 Custm Tls Autmatin Scripting Framewrk Glbal management tasks via custm scripts Scripts leverage vendr tls Simple text file database f UPSs Centralized cnfiguratin management Cnfiguratin cnsistency checks Infrmatin gathering tls Autmated reprting tls (daily and manual reprts)

13 Custm Tls Autmatin Rapid Deplyment System (~5 mins) Autmated deplyment with custm scripts Prcess leverages vendr prvided tls Easy and well dcumented prcedure Cnsistent and cmplete cnfiguratin Rapid and autmated recnfiguratin scripts]#./deply_new_ups.pl mccups13 Username [apc]: Passwrd [apc]: Cmmunity [public]: Ready t run n the fllwing UPSes. mccups13 Cntinue (y/n):

14 Mnitring Tls Graphing with Cacti Cacti Open Surce Plling and Graphing Framewrk Perfrmance Mnitring via SNMP Adaptable t Graph Virtually Anything Histrical Statistics Prblem Identificatin Failure Predictin Glbal Overview Free

15 Mnitring Tls Cacti Custm Glbal Overviews

16 Mnitring Tls Cacti Adaptable t Graph Virtually Anything even mis-behaving users

17 Mnitring Tls Status Plling with Nagis Nagis Open Surce Mnitring Framewrk System and Service Status Plling via SSH, SNMP, Runtime Warnings Fault and Failure Detectin Active and Negative Service Mnitring (want SSH but nt FTP running) Easily adapted t almst any need and fr multiple grups Already mnitring services fr thers grups like AHLA Grup based service/system ntificatin /SMS Alerts Web based H.U.D Mnitr fr Mnitring Free

18 Mnitring Tls Status Plling with Nagis Heads-Up-Display

19 Lgging Tls Centralized Lgging w/analytics Swatch (Syslg Watch) Filter and Summary Tl fr centralized lgging filter ut nrmal, nisy lgs Daily Reprting via Extensible Filtering easily add custm filters Open Surce Splunk Splunk is a fully featured, pwerful platfrm that cllects and indexes any machine data frm virtually any surce in real time. We feed it ur centralized lgs. Index any Text Data Interactive Search Results Mnitring and Alerting Reprting and Analysis Event Crrelatin Custm Dashbards Add-in apps Already extended t ther grups Nt Free but DOE cntract in wrks!

20 Lgging Tls Deep-Dive Yur Data Splunk

21 Results Higher Availability with Lwer TCO? Befre (10 15 units): Cnfiguratin was manual, nerus and incnsistent Deplyment tk a lng time (+1 hur) Attitudes twards system were apathetic r dreadful Mnitring was nnexistent r a deluge. Failures were cmmnplace and faults were ignred. After (100+ units): Cnfiguratin is autmatic, fast and cnsistent. Deplyment is very quick (~5 minutes). Replacement is nw simply a rutine task. Mnitring is practive, accurate and very predictive. Failures are rare and faults are dealt with quickly.

22 Key Pints t Take Away Buy in by management a must. Yu cannt maintain a system withut resurces!! Redundancy wherever pssible MTTR and MTBF are virtually zer with redundancy! Autmate as much as pssible Ensures standardizatin. Reduces staff time. Imprves the culture. Mnitr as much as pssible Failure ntificatin and predictin is critical! Fixing things befre they break means zer dwntime! What yu dn t knw usually hits yu hardest! Pretty graphs and real data translate int better budgets! Find r create tls t make things faster and easier Research vendr and pen surce slutins Simple slutins like shell scripts can be very pwerful Leverage the expertise f ther grups

23 Q & A