IOCs with Redundancy Support

Transcription

1 IOCs with Redundancy Support EPICS Collaboration Meeting, Spring 2012 Jörg Penning (DESY / MKS-2) 1

2 Overview Goals Principle of operation Design of the redundant IOC Experiences from operation Conclusion 2

3 Goals Survival of the IOC in case of failure Redundant power supply Redundant ethernet connection Redundant wiring of I/O components Enable replacement of parts Update software while maintaining operation 3

4 Goals: Embedding redundant IOCs Prerequisites for redundant design Analysis of: Power supply Ethernet cabling Ethernet devices Wiring of I/O components Servers 4

5 Principle of operation Typical parts for standby redundancy Client Voter Active System Standby System Our architecture Redundancy Monitor Task (RMT) decides EPICS parts are controlled by the RMT Clients rely on Channel Access reconnect functionality Client RMT RMT Active IOC Standby IOC 5

6 Principle of operation An IOC now is two IOCs in one rack Active? Slave? Master? Which IP? Rack PWR IOC Profi bus I/O PWR IOC Profi bus I/O private ethernet link 6

7 Principle of operation Active IOC Started like IOC without redundancy RMT is started too Standby IOC Started like IOC without redundancy but not put into operation (ie. Sequencer and Scanning are paused) RMT is started too 7

8 Design considerations RMT is a defensive add-on A system with RMT is more reliable than one without A failing RMT will not cause trouble We don t want RMT to constantly failover ( ping-pong ) 8

9 Design: RMT and clients A Primary Redundancy Resource (PRR) Is controlled by the RMT Implements the operations to start / stop the EPICS parts Synchronizes the state of the active IOC to its partner RMT PRR Basically 5 operations Start / stop as active Start / stop as standby State inspection Sequencer Control Scan Control Profibus Control 9

10 Design: PRRs Primary Redundancy Resources SNLEXEC Synchronizes the state of the sequence programs CCEEXEC Synchronizes the state of the database BaseCtrl Start / Stop of continuous control SeqCtrl Start / Stop of sequencer PBEXEC Synchronizes the state of the profibus driver Start / Stop of profibus operation rmtspy State inspection Error simulation 10

11 Design: RMT The state-machine of the Redundancy Monitor Task Init no partner or partner is standby partner is active Active partner died or is commanded to failover Standby active failed standby is ok cancel manual failover Commanded to failover partner is active active failed partner is not active Active failed partner is active Performing I/O-Test i/o is not ok i/o is ok 11

12 Design: Synchronizers PRRs are triggered by the RMT they manage the synchronization and the Run / Pause of EPICS parts become active Standby connected become standby Active connected gained connection lost connection lost connection gained connection Standby disconnected become standby Active disconnected become active 12

13 Operation: Software updates Requirements for software changes Build a simple model Communicate this to developers Eg.: Serializing in Java Adding a variable Deleting a variable Renaming a variable Changing a type Changing the format of the.ser file 13

14 Operation: Software updates EPICS update Not related to redundancy RMT update Keep the protocol to the partner Database Add / remove EPICS records SNL programs Add state sets Add states Change state implementation Ongoing work 14

15 Operation: What actually happened Nettask crash all ethernet connections are gone no more updates for the operators controls are still working Bugs SNL synchronizer Monitoring and Assignment timeout of field bus hardware Ping-Pong Transferring dynamic state Transferring for all state sets according to spec while having an error on the field bus 15

16 Conclusion Impact on the system Sequencer had to be enhanced Minor changes to EPICS Special firmware for the Profibus cards Long-term software maintenance problem Currently updating the sequencer Adding another piece of hardware Keep special hardware in stock Keep the on-call team informed 16

17 Conclusion System is up and running since October 2011 Pre-Version is up for more than 2 years We had to fix 2 major errors Currently 3 redundant IOCs are in production (will be 15) We already survived Several nettask crashes Intermittent profibus errors We did several software updates Databases and SNL programs 17

18 Conclusion What do you think? 18