AIDA Advanced European Infrastructures for Detectors at Accelerators. Conference Contribution

Transcription

1 AIDA-CONF AIDA Advanced European Infrastructures for Detectors at Accelerators Conference Contribution Evaluating Multi-Gigabit Transceivers (MGT) for Use in High Energy Physics Through Proton Irradiation. Wirthlin, M (INFN) et al 01 February 2015 The research leading to these results has received funding from the European Commission under the FP7 Research Infrastructures project AIDA, grant agreement no This work is part of AIDA Work Package 8: Improvement and equipment of irradiation and test beam lines. The electronic version of this AIDA Publication is available via the AIDA web site < or on the CERN Document Server at the following URL: < AIDA-CONF

2 Evaluating Multi-Gigabit Transceivers (MGT) for Use in High Energy Physics Through Proton Irradiation Matthew Cannon, Michael Wirthlin, Alessandra Camplani, Mauro Citterio and Chiara Meroni Abstract The paper summarizes the radiation test results of Xilinx 7-Series Multi-Gigabit Transceivers (MGT) operating in a 180 MeV proton beam to test the suitability in High Energy Physics experiments. Corresponding Author: Michael Wirthlin, Center for High-Performance Reconfigurable Computing (CHREC), Department of Electrical and Computer Engineering, Brigham Young University, Provo, UT 84602, USA, phone: , fax: , wirthlin@byu.edu Contributing Authors: Matthew Cannon, NSF Center for High Performance Reconfigurable Computing (CHREC), Brigham Young University, Provo, UT, USA Alessandra Camplani, INFN Milano, Via G. Celoria 16, Milano, Italy Mauro Citterio, INFN Milano, Via G. Celoria 16, Milano, Italy Chiara Meroni, INFN Milano, Via G. Celoria 16, Milano, Italy Session Preference: SEE: Devices and ICs Presentation Preference: Oral Acknowledgements: This work was supported by the I/UCRC Program of the National Science Foundation under Grant No The research leading to these results has received also funding from the European Commission under the FP7 Research Infrastructures project AIDA, grant agreement no , as a part of AIDA Work Package 8.4: qualification of components and common database. The authors would like to thank Alexander Prokofiev, Torbjörn Hartman, and Elke Passoth from the TSL accelerated radiation testing facility at Uppsala University for their assistance in the setup and operation of the experiment.

3 Evaluating Multi-Gigabit Transceivers (MGT) for Use in High Energy Physics Through Proton Irradiation Abstract The paper summarizes the radiation test results of Xilinx 7-Series Multi-Gigabit Transceivers (MGT) operating in a 180 MeV proton beam to test their suitability in High Energy Physics experiments. I. INTRODUCTION Serial transceivers embedded in the lastest generation of Field Programmable Gate Arrays (FPGA) are interesting for applications where high-speed data transmission is required. The GTX transceiver offered on Xilinx 7- Series FPGAs, for example, supports line rates from 500 Mb/s to 12.5 Gb/s [1]. The Xilinx Kintex 325T FPGA contains 16 such MGTs supporting up to 200 Gb/s aggregate bandwidth. This paper focuses on the radiation tolerance of these devices for application in high energy physics experiments, with particular regard to the radiation environment expected in the ATLAS experiment at the Large Hadron Collider (LHC) at the European Organisation of Nuclear Research (CERN) [2]. ATLAS is a general purpose detector built around one of the proton beam collision points of the LHC accelerator. ATLAS has various components such as inner tracking devices, a superconducting solenoid providing a 2 T magnetic field, a calorimeter system and a muon spectrometer. Each detector has its own readout electronics that will be upgraded in the coming years to study the new physics reach and to meet the challenges of the high-luminosity environment. The radiation produced during the experiment affects the electronic devices. The absorbed dose varies from 0.1 to 100 Mrad of ionizing radiation and up to neutrons/cm 2 depending on the electronics position in respect to the collision point. The replacement of existing Application-Specific Integrated Circuits (ASICs) with commercial devices, for detector regions with medium or low radiation doses, would be an important breakthrough in the redesign of experiment electronics. One of the commercial devices that we would like to introduce in the data acquisition system are FPGAs. The use of FPGAs will guarantee high computing speed by massive parallel computation, easy re-programmability and flexibility in the algorithms during the evolution of the design, as well as efficient data transmission with fast serializers. Our experiment evaluates the performance of the GTX transceiver of Xilinx Kintex 7 FPGAs, tested with 180 MeV protons, at The Svedberg Laboratory (TSL) in Uppsala, Sweden. Proton beam usage is useful to properly simulate the complex radiation environment expected at the LHC accelerator. Protons provide both ionization and displacement damage on the Device Under Test (DUT). Several previous efforts have investigated the effects of radiation on MGTs. MGTs within Xilinx Virtex-5 FPGAs were tested to understand the impact of heavyions on bit errors and lane connections [3]. This work suggests that the cross-section of the MGTs is small and most symptoms can be corrected using several simple mechanisms. The use of a simple connection protocol built on top of MGTs was tested to demonstrate the ability to address MGT symptoms with link-level protocol mechanisms [4]. In addition, the low-level cross section of the FPGA configuration memory and Block memory of Xilinx 7-Series FPGAs was tested with neutrons and protons to estimate the upset rate within high energy physics environments [5]. II. ORGANIZATION OF RADIATION EXPERIMENT This experiment was organized to achieve three specific goals: first, understand the failure mechanisms of the MGT in the presence of proton irradiation, second, collect sufficient MGT upset data to create estimates of MGT failure in a typical high energy physics experiment, and third identify any hard failures in the device or device problems when exposed to a large radiation dose. This section will describe how the test was organized to achieve these goals.

4 A. MGT Lane Configuration The approach used in this experiment for collecting error data is to create a bi-directional Gbps point-topoint link between two distinct FPGAs and continuously send/receive data between this point-to-point link during irradiation. One FPGA, the DUT, is exposed to radiation while operating and the other, called the service FPGA, operates outside of the radiation beam. The DUT side of the link is carefully monitored and any errors or anomalies are logged for data analysis. Each end of the point-to-point link is identical and includes both a transmitter and receiver as shown in Figure 1. Each end contains the FPGA MGT tile as well as circuitry created within the programmable fabric for generating data packets (i.e., Frame Generator) and for checking the integrity of received data packets (i.e., Frame Checker). All programmable logic used in conjunction with the link operates at MHz. In addition to the Frame Generator and Frame Checker, logic is provided for monitoring the status of the link and for transmitting errors within the logic for upstream processing. A LE Event suggests that the link has recovered and is transmitting normally again. B. Error Logging and Reporting The Error Manager as shown in Figure 2 is responsible for monitoring the Frame Checker error conditions on all lanes used in the design. When any error condition occurs, the Error Manager prepares an error message that contains the ID of the lane in error, the error condition, and a time stamp. This information is sent to the PicoBlaze (a small 8-bit soft-processor that is implemented in the FPGA fabric) which sends the error messages to the remote computer over the UART connection. The Error Manager, PicoBlaze processor, and UART in both the DUT and Service FPGA and are used to send event information to a remote computer connected to both FPGAs. Fig. 2. Mult-Lane Error Capture and Logging. Fig. 1. Test Architecture The Frame Generator creates a repeating 320-bit pseudo-random data sequence (16 words of 20-bits each) to transmit over the TX link of the MGT. This sequence was chosen to meet the minimum transition requirements of the MGT tile and avoid the use of the built-in 8B/10B encoder. The Frame Checker reports a variety of status signals, the most important being the Loss of Link (LOL) and Link Established (LE) events. The LOL Event suggests a large problem with the link and a complete loss in communication between the transmitter and the receiver. C. Test Hardware Setup This experiment was performed using the Xilinx 7- Series Kintex 325T FPGA. Two Kintex KC705 evaluation boards provided the FPGAs for the test. These boards contain the FPGA power supplies, a variety of I/O interfaces, and the physical MGT links. The arrangement of the two Kintex KC705 boards used in the experiment are shown in Figure 3. The Kintex 325T device contains 16 MGTs for a total aggregate bandwidth of 200 Gb/s (80 Gb/s when run at Gb/s/lane). All 16 of the MGTs are available on the KC705 board but only 13 of the 16 MGTs are used in the experiment. The 13 lanes used in this experiment include the following: FMC HPC Connector: 4 lanes SMA: 2 lanes SATA: 2 lanes SMA Connectors: 1 lane PCIe Interface: 8 lanes A custom physical interface was provided for these 13 lanes. A custom PCI crossover board was developed to provide a connection between the 8 PCI MGTs. As seen

5 TABLE I E RROR R ESULTS FOR M ULTI -L ANE E XPERIMENTS Error Type Cross Section [(errors/lane)/(p/cm2 )] Fig. 3. Experimental Test Setup using Two KC705 Evaluation boards and External Configuration Scrubbing Board. in Figure 3, the crossover board connects the DUT and the Service boards through the PCI interface requiring that the boards be placed in close proximity during testing. Two SATA crossover cables provide the physical connection for the two SATA lanes and 12 coaxial cables were provided to support the three SMA lanes. In addition to the two KC705 boards, this test involved the use of a power supply monitoring module, a configuration memory scrubbing board, and a custom fan for cooling. The heat sink and fan mounted over DUT FPGA were removed during irradiation and an external cooling fan was added to prevent the DUT chip from overheating during the test. A Texas Instruments (TI) PMBus USB Interface Adapter was used to monitor and log the voltage and current of all ten voltage supplies for the FPGA. A solid mounting board was designed to reliably attach all of the experimental hardware in front of the beam. A custom board was created to perform scrubbing of the FPGA configuration memory. During irradiation, it is expected that the configuration memory will experience Single Event Upsets (SEU). If not repaired, these upsets will cause increasing contention within the FPGA and lead to operational errors. This scrubbing system continuously monitors the configuration memory and repairs upsets within the memory in real-time. III. E XPERIMENTAL R ESULTS The experiment described above was tested at the PAULA proton beam facility at TSL during September of Configuration Error 3.81E-12 Lane Error 2.51E-12 DUT Error 6.12E-13 Composite Error 7.27E The DUT FPGA chip was exposed to protons/cm2 over a period of 24 hours (three days of eight hours each). The logs recorded over the UART from the experiment were analyzed to identify a variety of error events. These events were identified by analyzing and correlating both the DUT errors and corresponding Service errors. A variety of events of interest were identified during this process. Due to the high amount of data collected only some of the errors and results will be discussed here. Additional error signatures will be reported in the final paper. The most important errors are summarized below: Configuration Error This is an error due to the circuitry surrounding the MGT and is not a problem with the actual MGT. Lane Error This is any error that is caused by a MGT failure. DUT Error This error caused all 13 lanes to fail simultaneously. This suggests that some form of global failure or Single Event Functional Interrupt (SEFI) is occurring (i.e., in the global clocking or reset network). The error events identified during the test are summarized in Table I and are reported in terms of sensitive cross section. The results are normalized on a per lane basis to aid in estimation of error rates with system using a different number of lanes. In addition to the three error conditions, a composite error result is provided that combines the results from all three error types. These errors were calculated by dividing the fluence by the total number of errors we observed. A number of important observations are seen in these results. First, errors due to upsets in the configuration memory are the most prevalent error condition observed in the test. This result suggests that most errors that occurred during the test are unrelated to the MGT

6 failures and can possibly addressed by proper FPGA logic mitigation such as Triple Modular Redundancy (TMR). In previous results, the cross section for a single configuration bit within the same proton radiation beam was estimated at cm 2 /bit [5]. The MGT configuration error cross section is 381 times the sensitive cross-section of a single-bit suggesting that approximately 381 configuration bits are used for each lane to define the Frame Generator, Frame Checker, and error logging circuitry. It is important to note that no mitigation was provided for the Error Manager, PicoBlaze processor, or UART. These circuits were subject to the effects of configuration upsets and experienced failures unrelated to the MGTs during the radiation experiment. A variety of mitigation methods can be applied to these circuits to improve their tolerance of upsets within the configuration memory. During the test, the FPGA was exposed to a fluence of protons/cm 2. It is estimated that the FPGA accumulated a total ionizing dose of approximately 1 Mrad during the test. Other than the effects of SEUs, no permanent operational failures of the FPGA programmable logic were observed during the test 1. A modest increase in the current draw by the VINT core-logic power supply was measured at the end of the test but power consumption was well within the specifications of the FPGA. It is important also to evaluate the MGT errors in the context of high energy physics experiments like ATLAS. The calorimeter system for example has a total fluence similar to the one reached during this test. Based on the estimated results in this experiment, individual MGTs will experience a composite error rate of approximately 4 errors/(year lane). A full-scale calorimeter system will require approximately lanes (over 1600 FPGAs) resulting in a lane failure approximately every 6.5 minutes. Additional MGT mitigation techniques and error detection/correction methods will be needed to improve this estimated lane failure rate. Additional details on this estimated ATLAS error rate will be provided in the full paper. IV. CONCLUSION This work describes the radiation test and test result of the Xilinx 7-Series MGTs in a 180 MeV proton beam. 13 bi-directional MGT lanes with constant traffic patterns were exposed to radiation and carefully monitored for functional failures. A variety of failures were observed including configuration upset induced errors and individual MGT lane errors. An estimate of the lane error rate is provided along with an estimate of the error rate for a subdetector of the ATLAS experiment. This test also demonstrates that the 7-Series FPGA can withstand a relatively high total ionizing dose with no permanent functional failures. This work represents an ongoing effort to understand the risks and benefits of using FPGAs in large high energy physics experiments. Additional tests are being planned to integrate SEU mitigation methods and obtain more detail on MGT lane failures. Additional details on this experiment will be presented in the final paper. ACKNOWLEDGMENT This work was supported by the I/UCRC Program of the National Science Foundation under Grant No The research leading to these results has received also funding from the European Commission under the FP7 Research Infrastructures project AIDA, grant agreement no , as a part of AIDA Work Package 8.4: qualification of components and common database. The authors would like to thank Alexander Prokofiev, Torbjörn Hartman, and Elke Passoth from the TSL accelerated radiation testing facility at Uppsala University for their assistance in the setup and operation of the experiment. REFERENCES [1] 7 Series FPGAs GTX/GTH Transceivers: User Guide, Xilinx Corporation, February , UG476 (v1.10). [2] G. Aad et al., The ATLAS Experiment at the CERN Large Hadron Collider, Journal of Instrumentation (JINST), vol. 8, no. S08003, [3] R. Monreal, C. Carmichael, and G. Swift, Single-event characterization of Multi-Gigabit Transceivers (MGT) in space-grade Virtex-5QV field programmable gate arrays (FPGA), in Radiation Effects Data Workshop (REDW), 2011 IEEE, July 2011, pp [4] A. Harding, K. Ellsworth, B. Nelson, and M. Wirthlin, Characterization and mitigation of the MGT-based Aurora protocol in a radiation environment, in Radiation Effects Data Workshop (REDW), 2013 IEEE, July 2013, pp [5] M. J. Wirthlin, H. Takai, and A. Harding, Soft error rate estimations of the Kintex-7 FPGA within the ATLAS Liquid Argon (LAr) Calorimeter, Journal of Instrumentation (JINST), vol. 9, no. C01025, The XADC module within the FPGA failed sometime on the first day of the test. The XADC provides analog to digital conversion within the FPGA but was not used as part of this experiment.