SBS VME DAQ SBS collaboration Meeting July 7 th 2014
Outline SBS DAQ overview GEM readout Fastbus readout HCAL readout Plan Manpower Conclusion 2
SBS DAQ Overview Calorimeter ECAL : Fastbus HCAL SBS GEM APV25 INFN MPD BigBite Scintillator Shower preshower Coordinate detector 3
SuperBigbite Spectrometer Focal Plane Polarimeter setup 4
HCAL Calorimeter Rates (CDR section 5.1.7) Most demanding Hadron rate estimate using SLAC & DESY data, Wiser code: w/4.5 GeV threshold: 1.5 MHz ECAL From Hall A Real Compton Scattering experiment NB: Good resolution 16%/ E Background rate vs. cut on deposited energy (MC studies in progress) Electron rate estimate w/2.5 GeV threshold (73% of E elas ): 200 khz 9 khz coincidence rate w/ 30 ns window 5
DAQ concept Hybrid Fastbus and pipelined electronics Level 1 100 ns latency by analog summing and discrimination Generated by electron arm ( 200 khz rate) Gate for Fastbus & non-pipelined VME for BigCal Level 2 : coincidence proton in HCAL and electron in ECAL Assume up to 1.8 μs latency ( L2 800 ns max + Fast Clear 1 μs) 9 KHz with 30 ns coincidence windows FPGA-based coincidence logic using geometrical constraints reduction by factor 5 2 khz physics DAQ rate Fast Clear FB & VME after L2 timeout 13% Electronics Dead Time 6
e -p Kinematic Correlation 11 x 22 HCAL blocks 20 x 76 ECAL blocks (CDR section F.3) Using geometric correlations from elastic kinematic one can reduce final rate by a factor of 5 and tracker data by at least a factor of 3 7
Data flow 100 Mb/s CODA GEM APV25 VME digitizer VME CPU GigE GigE Event builder Event recorder Summing L1 trigger TS SAS RAID BigCal Bigbite HCAL L1 trigger Fastbus Digital Sum VME CPU VME CPU Logic L2 trigger Compression + 10 GigE Data file Silo 8
GEM GEM APV25 APV25 BigCal Bigbite HCAL VME digitizer L1 trigger FEC SRS SRU VME CPU Summing Fastbus Digital Sum GigE 100 Mb/s GigE GigE L1 trigger VME CPU VME CPU 10 GigE PC Event builder 10 GigE Logic TS Event recorder L2 trigger SAS RAID Compression + 10 GigE CODA Data file Silo 9
GEM readout INFN MPD VME64X board New version with fast VME protocol Working with CAEN controller Adapting software package to Intel VME CPU to use with CODA SRS readout Using Mississippi State SRS Running with DATE, starting to look integration into CODA Standard UDP based protocol Request to use GEM for PREX and Tritium in addition to A1n Will have baseline performance number before November 10
2048 channel 1 MPD Front Tracker layout 2048 channel 1 MPD 11
1 MPD 2048 Front tracker Region of Interest from BigCal position Trackers layout 1 MPD 2048 Back tracker Region of Interest from HCal position Middle tracker Interpolated from both front and back information 12
Trackers layout Worst case configuration 13
Suppression schemes Several algorithm can be implemented in FPGA for further data reduction Thresholds Timing Fitting c2 slope 14
2048 FPP Tracker layout 2560 512 15
Tracker event size with 3 samples readout Detector Rate Channels Occupancy Time window Hits Geometr ical factor Data size Effective data size Data rate 3KHz Mb/s Front Tracker 400 49000 13.5% 75ns 6615 3 52Kb 14.3Kb 43 Second Tracker 130 13600 7.4% 50ns 1010 5 8Kb 1.6Kb 5 Third Tracker 64 13600 3.6% 50ns 490 5 4Kb 0.8Kb 2 Electron arm GEM 173 12000 2.4% 50ns 288 1 2.3Kb 2.3Kb 6 Calorimeters 125 0.5Kb 0.5Kb 1.5 Total 67.8Kb 19.5Kb 58.5 Mb/s 16
Data rates Detector Rate Channels Occupancy Hits Data size Bytes Data rate MB/s 5 KHz Geometrical factor Data size no sup Data rate MB/s Front Tracker 400.00 49000.00 0.50 6615.00 161929 809.65 3.00 269.88 134.94 Second Tracker 130.00 61440.00 0.50 1010.00 203040. 1015.20 5.00 203.04 101.52 Third Tracker 64.00 61440.00 0.50 490.00 203040 1015.20 5.00 203.04 101.52 Total 171880.00 8115.00 568009 2840.05 675.96 337.98 17
Fastbus readout Time : 1877S Amplitude 1881M or MQT+1877S Fastbus max transfer speed : 40 MB/s can use either Intel or Old vxworks VME CPU Test Lab : 4 sets of 3 crates, will be able to test performance 18
HCAL readout 288 channels 2 VXS crates, 18 s 1.5 MHz singles 16 block clusters 250 MHz 12 bit = 2 bytes 10 samples : 320 bytes Need HCAL occupancy 19
Tape price Experiment Days Rate (MB/s) seconds Total TB Double DLO5 DLO6 GEp 45 250 3888000 972 1944 97200 58320 GEn 50 250 4320000 1080 2160 108000 64800 GMn 25 250 2160000 540 1080 54000 32400 SIDIS 64 250 5529600 1382.4 2764.8 138240 82944 Total 397 K$ 239K$ 20
Man power A. Camsonne : General, MPD readout S. Abraham, M. Jones : Fastbus Students Jessica Campbell (SMU) : Fastbus Jessie Twigger (FIT SULI),?: SRS CODA readout HCAL trigger readout :? 21
Plan summary Setup on 3 Fastbus crate setup and test performance APV25 performance with SRS and MPD and integration with CODA Getting started with for HCAL Start write-up about DAQ 22
Conclusion Need good ways to reduce data Fastbus setup almost ready for testing Need to start thinking about SIDIS Start writeup 23
Backup
VETROC JLAB Electronics group 25
CPU CTP SD TI CPU GTP SD TI CPU SD SD TS L1 Trigger Diagram VXS Crate CTP VXS Crate 250 12 bit @ 250 MHz, 16 ch Sums amplitude from all channels Transfer total energy or hit pattern to CTP VXS Serial Link 16 bit @ 250 MHz: 4 Gbps VXS Crate Fiber Optics 64 bit @ 125 MHz Crate Trigger Processor Sums energies from s Transfer total energy or hit pattern to 26
CPU CTP SD TI CPU GTP SD TI CPU SD SD TS L1 Trigger Diagram VXS Crate VXS Crate Sub-System Processor Consolidates multiple crate subsystems Report total energy or hit pattern to GTP VXS Serial Link 32 bit @ 250 MHz: 8 Gbps VXS Crate Copper Ribbon Cable 32 bit @ 250 MHz: 8 Gbps Global Trigger Processor Collect L1 data from s Calculate trigger equations Transfer 32 bit trigger pattern to TS 27
Level-1 Trigger Electronics Custom Designed Boards at JLAB Detector Signals (16) (1) (1) (1) Fiber Optic Links Clock/Trigger (16bits @ 62.5MHz (12) (1) (1) ( ) Number in parentheses refer to number of modules Fiber Optic Link (~100 m) (64bits @ 125 MHz) fadc250 CTP Crate Trigger Processor SD Signal Distribution (8) (2) (1) Copper Ribbon Cable (~1.5 m) (32bits @ 250 MHz) TI Trigger Interface VXS Backplane Pipelined detector readout electronics: fadc Trigger Latency ~ 3 μs 28
CTP GTP CTP Pipelined Hall D DAQ Calorimeter Light Gas Cerenkov Heavy Gas Cerenkov C T P C T P 29
Pipelined Hall D DAQ Electron shower accidental accidental 3 us latency Above threshold 30
Read Out Controllers Raid Disk L3 Farm blocked event fragments partially recombined event fragments full events All nodes connected with 1GB/s links ROC ROC ROC ROC ROC ROC ROC ROC ROC EB1 Event Builder stage 1 EB1 Event Builder stage 1 EB1 Event Builder stage 1 EB2 Event Builder stage 2 EB2 Event Builder stage 2 node node node L3 Farm node node node node Switches connected with 10GB/s fiber optics ER Event Recorder Front-End Crates ~60 crates ~50MB/s out per crate Staged Event Building N x M array of nodes (exact number to be determined by available hardware at time of purchase) Level-3 Trigger and monitoring Event Recording 300MB/s in 300MB/s out 31
Energy BCAL (GeV) Energy BCAL (GeV) Energy BCAL (GeV) Hall D L1 Trigger-DAQ Rate Low luminosity (10 7 g/s in 8.4 < E g < 9.0 GeV) 20 khz L1 High luminosity (10 8 g/s in 8.4 < E g < 9.0 GeV) 200 khz L1 Reduced to 20 khz L3 by online farm Event size: 15 kb; Rate to disk: 3 GB/s SC Detectors which can be used in the Level-1 trigger: Forward Calorimeter (FCAL) Barrel Calorimeter (BCAL) Start Counter (SC) Time of Flight (TOF) Photon Tagger Basic Trigger Requirement: Energy Energy Hits Hits Hits Electromagnetic background Hadronic E g < 8 GeV Hadronic E g > 8 GeV E BCAL + 4 E FCAL > 2 GeV and a hit in Start Counter Energy FCAL (GeV) Energy FCAL (GeV) Energy FCAL (GeV) 32
Custom Electronics for JLab VME Switched Serial (VXS) backplate 10 Gbps to switch module (J 0 ) 320 MB/s VME-2eSST (J 1 /J 2 ) All payload modules are fully pipelined 125 (12 bit, 72 ch) 250 (12 bit, 16 ch) F1-C (60 ps, 32 ch or 115 ps, 48 ch) Trigger Related Modules Crate Trigger Processor (CTP) Sub-System Processor () Global Trigger Processor (GTP) Trigger Supervisor (TS) Trigger Interface/Distribution(TI/D) Signal Distribution (SD) 125 F1-C 33
CPU CTP SD TI CPU GTP SD TI CPU SD SD TS L1 Trigger Diagram VXS Crate VXS Crate Trigger Distribution Distribute trigger, clock and synchronize signals to TI in each Crate Fiber Optics 16 bit @ 62.5 MHz: 1 Gbps VXS Crate VXS Serial Link 16 bit @ 62.5 MHz: 1 Gbps Trigger Supervisor Calculate 8 bit trigger types from 32 bit trigger pattern Prescale triggers Transfer trigger and sync signal to (16 bit total) 34
CPU CTP SD TI CPU GTP SD TI CPU SD SD TS L1 Trigger Diagram VXS Crate VME Readout Controller Gigabit ethernet Signal Distribution Distribute common signals to all modules: busy, sync and trigger 1/2 TID VXS Crate VXS Crate Trigger Interface Receive trigger, clock and sync signals from Make crate trigger decision Pass signals to SD VXS Serial Link 4 bit @ 250 MHz: 1 Gbps 35
The GlueX Detector 2.2 Tesla Solenoid 2.2T superconducting solenoidal magnet Fixed target (LH 2 ) 10 8 tagged g/s (8.4-9.0GeV) hermetic TOF time of flight SC start counter Charged particle tracking Central drift chamber (straw tube) Forward drift chamber (cathode strip) Calorimetry Barrel Calorimeter (lead, fiber sandwich) Forward Calorimeter (lead-glass blocks) PID Time of Flight wall (scintillators) Start counter Barrel Calorimeter 36
LHC JLab BNL CHEP2007 talk Sylvain Chapelin private comm. GlueX Data Rate Front End DAQ Rate Event Size L1 Trigger Rate Bandwidth to mass Storage GlueX 3 GB/s 15 kb 200 khz 300 MB/s CLAS12 0.1 GB/s 20 kb 10 khz 100 MB/s ALICE 500 GB/s 2,500 kb 200 khz 200 MB/s ATLAS 113 GB/s 1,500 kb 75 khz 300 MB/s CMS 200 GB/s 1,000 kb 100 khz 100 MB/s LHCb 40 GB/s 40 kb 1000 khz 100 MB/s STAR 50 GB/s 1,000 kb 0.6 khz 450 MB/s PHENIX 0.9 GB/s ~60 kb ~ 15 khz 450 MB/s * ** * Jeff Landgraf Private Comm. 2/11/2010 ** CHEP2006 talk MartinL. Purschke 37
CODA3 What s different CODA 2.5 CODA 3 Run Control (X, Motif, C++) (rcserver, runcontrol) Communication/Database (msql, cdev, dptcl, CMLOG) Event I/O C-based simple API (open/close read/write) Event Builder / ET System / Event Recorder (single build stream) Front-End vxworks ROC (Interrupt driven event by event readout) Triggering: 32 ROC limit, (12 trigger bits -> 16 types) TS required for buffered mode Experiment Control AFECS (pure JAVA) (rcplatform, rcgui) cmsg CODA Publish/Subscribe messaging EVIO JAVA/C++/C APIs Tools for creating data objects, serializing, etc EMU (Event Management Unit) Parallel/Staged event building Linux ROC, Multithreaded (polling event blocking) 128 ROC limit, (32 trigger bits -> 256 types) TI supports TS functionality. Timestamping (4ns) 38
Encoding Example 39
GTP Trigger Bit Example 40