Electronics, trigger and physics for LHC experiments 1
The Large hadron Collider 27 km length, 100 m underground, four interaction points (experiments) proton-proton collisions, 7 TeV + 7 TeV (14 TeV in CM) 2808 bunches per beam with 11245 rounds per second = 32 Millions collisions per second Nominal luminosity 1034 cm-2 s-1 2
The LHCb detector Forward spectrometer to study heavy mesons (b,c) physics: rare decays and CP violation Most of this heavy mesons are produced close to the beam axis (~ 40% in acceptance) Low pt and high rapidity kinematic region 3
Electronics and trigger for LHC The vertex locator of LHCb Silicon detector to track the charged particles close to the interaction region. In particular it is crucial to reconstruct the secondary vertecies 172K channels Strips in R and φ projection (~10 μm vertex resolution) Located 1cm from beam Blue/yellow layers correspond to R and φ sensors Analog readout (via twisted pair cables over 60m) ~1m Analog signal to DAQ beam from Si sensors 4
Electronics and trigger for LHC Digital optical link High speed: 1Ghz - 10GHz 40GHz Extensively used in telecommunications (expensive) and in computing ( cheap ) Encoding Reliability and error rates strongly depending on received optical power and timing jitter Multiple (16) serializers and deserializers directly available in modern chips (FPGA s). Transmission goodness by BER factor (bit error rate) 5
Electronics and trigger for LHC DAQ interfaces / readout boards Large Front-end data reception Receive optical links from multiple front-ends: 24 96 Located outside radiation Event checking Verify that data received is correct Verify correct synchronization of front-ends Extended digital signal processing to extract information of interest and minimize data volume Event merging/building Build consistent data structures from the individual data sources so it can be efficiently sent to DAQ CPU farm and processed efficiently without wasting time reformatting data on CPU. Requires significant data buffering High level of programmability needed Send data to CPU farm at a rate that can be correctly handled by farm 1 Gbits/s Ethernet (next is 10Gbits/s) In house link with PCI interface: S-link Requires a lot of fast digital processing and data buffering: FPGA s, DSP s, embedded CPU Use of ASIC s not justified Complicated modules that are only half made when the hardware is there: FPGA firmware (from HDL), DSP code, on-board CPU software, etc. 6
Electronics and trigger for LHC New problems Going from single sensors to building detector read-out of the circuits we have seen, brings up a host of new problems: Power, Cooling Crosstalk Radiation (LHC) Some can be tackled by (yet) more sophisticated technologies 7
Electronics and trigger for LHC Radiation effects In modern experiments large amounts of electronics are located inside the detector where there may be a high level of radiation. This is the case for 3 of the 4 LHC experiments (10 years running) Pixel detectors: 10-100 Mrad Trackers: ~10Mrad Calorimeters: 0.1 1Mrad Muon detectors: ~10krad Cavern: 1 10krad 1 Rad = 10 mgy 1 Gy = 100 Rad Normal commercial electronics will not survive within this environment. One of the reasons why all the on-detector electronics in the LHC experiment are custom made Special technologies and dedicated design approaches are needed to make electronics last in this unfriendly environment Radiation effects on electronics can be divided into three major effects Total dose Displacement damage Single event upsets 8
Electronics and trigger for LHC Total dose Generated charges from traversing particles gets trapped within the insulators of the active devices and changes their behavior For CMOS devices this happens in the thin gate oxide layer which have a major impact on the function of the MOS transistor Threshold shifts Leakage current In deep submicron technologies ( <0.25um) the trapped charges are removed by tunneling currents through the very thin gate oxide Only limited threshold shifts The leakage currents caused by end effects of the linear transistor (NMOS) can be cured by using enclosed transistors For CMOS technologies below the 130nm generation the use of enclosed NMOS devices does not seem necessary. But other effects may show up No major effect on high speed bipolar technologies 9
Electronics and trigger for LHC Displacement damage Traversing hadrons provokes displacements of atoms in the silicon lattice. Bipolar devices relies extensively on effects in the silicon lattice. Traps (band gap energy levels) Increased carrier recombination in base Results in decreased gain of bipolar devices with a dependency on the dose rate. No significant effect on MOS devices Also seriously affects Lasers and PIN diodes used for optical links. 10
Electronics and trigger for LHC Single event upsets (SEU) Deposition of sufficient charge can make a memory cell or a flip-flop change value As for SEL* (single event latchup), sufficient charge can only be deposited via a nuclear interaction for traversing hadrons The sensitivity to this is expressed as an efficient cross section for this to occur This problem can be solved at the circuit level or at the logic level Make memory element so large and slow that deposited charge not enough to flip bit * SEL: An abnormal high-current state in a device caused by the passage of a single energetic particle through sensitive regions of the device structure and Triple redundant (for registers) resulting in the loss of device functionality Hamming coding (for memories) In telecommunication, Hamming codes are a family of linear error-correcting codes 11
Electronics and trigger for LHC Powering Delivering power to the front-end electronics highly embedded in the detectors has been seen to be a major challenge (underestimated). The related cooling and power cabling infrastructure is a serious problem of the inner trackers as any additional material seriously degrades the physics performance of the whole experiment. A large majority of the material in these detectors in LHC relates to the electronics, cooling and power and not to the silicon detector them selves (which was the initial belief) How to improve Lower power consumption Improve power distribution Simulation of material budget 12
Electronics and trigger for LHC Electronic crates in DAQ Going from single sensors to thousand channels readout forces to use a dedicated electronic design Put many of these multi-port modules together in a common chassis or crate The modules need VME board plugged into backplane Mechanical support Power A standardized way to access their data (our measurement values) All this is provided by standards for (readout) electronics such as VME (IEEE 1014) 13
Electronics and trigger for LHC Communication in crate: buses A bus connects two or more devices and allows the to communicate The bus is shared between all devices on the bus arbitration is required Devices can be masters or slaves (some can be both) Devices can be uniquely identified ("addressed") on the bus Famous examples: PCI, USB, VME, SCSI older standards: CAMAC, ISA upcoming: ATCA many more: FireWire, I2C, Profibus, etc Buses can be local: PCI external peripherals: USB in crates: VME, compactpci, ATCA long distance: CAN, Profibus Theoretically ~ 16 MB/s can be achieved Better performance by using block-transfers Easy to add new device, boards with standard interface 14
DAQ and trigger at LHC Network and farm For such huge amount of data to digest buses are not enough subdetector often very far from each other. Number of devices and physical bus-length is limited (scalability!). Useful for systems < 1 GB/s Network technology solves the scalability issues of buses In a network devices are equal ("peers") In a network devices communicate directly with each other (no arbitration necessary and bandwidth guaranteed) data and control use the same path much fewer lines (e.g. in traditional Ethernet only two) At the signaling level buses tend to use parallel copper lines. Network technologies can be also optical, wire-less and are typically (differential) serial Examples: 1 The telephone network Ethernet (IEEE 802.3) ATM (the backbone for GSM cell-phones) Infiniband Network technologies are sometimes functionally grouped Cluster interconnect (Myrinet, Infiniband) 15 m Local area network (Ethernet), 100 m to 10 km Wide area network (ATM, SONET) > 50 km 2 While 2 can send data to 1 and 4, 3 can send at full speed to 5 4 2 can distribute the share the bandwidth between 1 and 4 as needed 3 5 15
DAQ and trigger at LHC A large experiment example: CMS 15 million detector channels @ 40 MHz ~15 * 1,000,000 * 40 * 1,000,000 bytes ~ 600 TB/sec (impossible to record) HEP experiments usually consist of many different sub-detectors: tracking, calorimetry, particle-id, muon-detectors We need: A selection mechanism ( trigger ) Electronic readout of the sensors of the detectors ( front-end electronics ) A system to keep all those things in sync ( clock ) A system to collect the selected data ( DAQ ) A Control System to configure, control and monitor the entire DAQ Time, money, students 16
Trigger at LHC A typical collision is boring Although we need also some of these boring data as cross-check, calibration tool and also some important low-energy physics Interesting physics is about 6 8 orders of magnitude rarer (EWK & Top) Exciting physics involving new particles/discoveries is 9 orders of magnitude below tot 100 GeV Higgs 0.1 Hz 600 GeV Higgs 0.01 Hz We just need to efficiently identify these rare processes from the overwhelming background before reading out & storing the whole event 17
Trigger at LHC Technical requirements No (affordable) DAQ system could read out O(107) channels at 40 MHz 400 TBytes/s to read out even assuming binary channels! What s worse: most of these millions of events per second are totally uninteresting: one Higgs event every 0.02 seconds A first level trigger (Level-1,L1) must somehow select the more interesting events and tell us which ones to deal with any further Millions of channels try to work as much as possible with local information Keeps number of interconnections low Must be fast: look for simple signatures Keep the good ones, kill the bad ones Robust, can be implemented in hardware (fast) Design principle: fast: to keep buffer sizes under control every 25 nanoseconds (ns) a new event: have to Decide within a few microseconds (μs): trigger latency 18
Trigger at LHC Physical requirements Requirements driven by the physics objectives of the experiments ATLAS and CMS (general-purpose, proton-proton, discovery physics) LHCb (B physics, proton-proton) ALICE (specialized for heavy-ion collisions) 19
Trigger for LHC ATLAS and CMS: requirements Triggers in the general-purpose proton proton experiments, Retain as many as possible of the events of interest for the diverse physics programs of these experiments Higgs searches (Standard Model and beyond): e.g. H ZZ leptons, H gg; also H tt, H bb SUSY searches, with and without R-parity conservation Searches for other new physics Using inclusive triggers that one hopes will be sensitive to any unpredicted new physics Precision physics studies: e.g. measurement of W mass B-physics studies (especially in the early phases of these experiments) N.b. selections often need to be made at analysis level to suppress backgrounds, so focus especially on events that will be retained 20
Trigger for LHC ATLAS and CMS: constraints L = 1034 cm-2s-1, σ = 100 mb (inelastic) 109 interaction rate W or Z decays is O(100 Hz) Total data flow = event rate events size = 109 Hz 1 MByte = 1000 TByte/s, absolutely impossible to record and also useless. Most of events are not interesting from the Physics point of view Mandatory to insert filters (intemediate processing units) in order to reduce (order of magnitude) the events to record Hardware filters with dedicated electronics Software filters with online analysis and discrimination on commercial CPU farm 21
trigger for LHC LHCb The LHCb experiment, which is dedicated to studying B-physics, faces similar challenges to ATLAS and CMS It operates at a comparatively low luminosity (~2 1032 cm-2s-1), giving an overall proton proton interaction rate of ~20 MHz Chosen to maximise the rate of single-interaction bunch-crossings The event size is comparatively small (~100 kbyte) Fewer detector channels Less occupancy due to lower luminosity However, there is a very high rate of beauty production Given σ ~ 500 μb, bb production rate ~100 khz The trigger must therefore search for specific B decay modes that are of interest for the physics analysis Event rate of only ~3 khz 22
trigger for LHC Alice The heavy-ion experiment ALICE is also very demanding, particularly from the DAQ point of view The total interaction rate will be much smaller than in the pp experiments L ~ 1027 cm-2s-1 R ~ 8 khz for Pb Pb collisions The trigger will select minimum-bias and central events (rates scaled down to total ~40 Hz), and events with dileptons (~1 khz with only part of the detector read out) However, the event size will be huge due to the high particle multiplicity in Pb Pb collisions at LHC energy Up to O(10,000) charged particles in the central region Event size up to ~ 40 MByte when the full detector is read out Even more than in the other experiments, the volume of data to be stored and subsequently processed offline will be massive 23 Data rate to storage ~1 GByte/s (limited by what is possible/affordable)
Trigger for LHC High level trigger rates High level trigger rate vs event size for several experiments It is clear the progress with time The four LHC experiments differ mong them: from the highest L1 rate of LHCb to the huge event size of the ALICE Rate*Size = bandwidth ~ constant for ATLAS, CMS and LHCb 24
trigger for LHC How to defeat minimum bias: transverse momentum pt p-p (inelastic) collisions produce mainly hadrons with transverse momentum pt ~ 1 GeV/c Interesting physics (old and new) has particles (leptons and hadrons) with large pt W eν, M(W) = 80 GeV/c2 and pt(e) ~ 40 GeV/c H(120 GeV/c2) γγ, pt(γ) ~ 50 GeV/c B μμ, pt(μ) ~ 3 GeV/c Impose high threshold on pt of the particles Implies distinguishing between difrent type of particles. This is possible for electrons, muons, jets p pt 25
trigger for LHC How to defeat minimum bias: transverse momentum pt 26
trigger for LHC Particle identification CMS experiment 4T TRACKER γ n μ+ 2T e- Silicon Microstrips Pixels π+ ECAL SUPERCONDUCTING COIL Scintillating PbWO4 crystals IRON YOKE HCAL Plastic scintillator/brass sandwich MUON CHAMBERS Drift Tube Chambers Resistive Plate Chambers Cathode Strip Chambers 27
trigger for LHC LHCb trigger First Level (L0): 40 MHz 1 MHz High-pT µ, e, γ, hadron candidates (ECAL, HCAL, Muon). Software level (High Level Trigger) Access all detector data. Farm with 15000 CPU cores on multi-processor commodity boxes. HLT1: Confirm L0 candidate with more complete info, add impact parameter and lifetime cuts: 1 MHz 30 khz. HLT2: global event reconstruction + selections: 30 khz 3 khz, where 1 khz being dedicated to charm. 3 khz 28
DAQ for LHC Event building 1) Event fragments are received from detector front-ends 2) Event fragments are read out over a network by an event builder system 3) Event builder assembles fragments into complete event 4) Complete events are sent to the high level trigger algorithm Push based: event fragments are sent without feedback with the event builder system Pull based: event builder system tells readout supervisor when and where (which event builder is ready) send the data 29 Readout supervisor
DAQ for LHC LHCb DAQ 30
LHC trigger/daq parameters Trigger levels Level 1,2 Rate (Hz) Event size (Bytes) Readout BW (GB/s) HLT out (MB/s) (Events/s) 4 500 (Pb-Pb) 103 (p-p) 5 107 2 106 25 1250 (100) 200 (100) 3 105 (LV1) 3 103 (LV2) 1.5 106 150 300 (200) 2 105 106 100 1000 (100) 2 106 3.5 104 35 70 (3000) 31