Computing Software and Analysis Challenge 2006

Transcription

1 Computing Software and Analysis Challenge 2006 N. De Filippis Department of Physics and INFN Bari On behalf of the CMS Collaboration IPRD06, Siena, Italy, 1st - 5th October 2006 Nicola De Filippis IPRD06, Siena, 1-5 October p. 1

2 What is CSA06? A 50 million event exercise to test the workflow and dataflow as defined in the CMS computing model A test at 25% of the capacity needed in 2008 Main components: Preparation of large MC simulated datasets (some with HLT-tags) Prompt reconstruction at Tier-0: Reconstruction at 40 Hz (over 150 Hz) using CMSSW Application of calibration constants from offline DB Generation of Reco, AOD, and AlCaReco datasets Splitting of an HLT-tagged sample into 10 streams Distribution of all AOD to all participating Tier-1s Distribution of some Reco+AlCaReco to all participating Tier-1s Calibration jobs on AlCaReco (&Reco) datasets at some Tier-1s and proto-caf (CERN) Skim jobs at some Tier-1s with data propagated to Tier-2s Physics jobs at Tier-2s and Tier-1s on AOD and Reco Nicola De Filippis IPRD06, Siena, 1-5 October p. 2

3 CSA06 computing metrics CMS needs to be at production scale services in 2008 CMS should be able to demonstrate 25% of the expected 2008 scale in this year and be able to reach 50% scale early in 2007 Nicola De Filippis IPRD06, Siena, 1-5 October p. 3

4 Resources Tier-0 (CERN): 1.4M SI2K (~700 boxes at CERN) 180 TB Tier-1 (7 sites): 2500 CPUs in total TB disk + tape at participating Tier1 Expected 70 TB to participate Tier-2 (25 sites): 2400 CPUs in total Average 10 TB disk at participating Tier-2 5 TB minimum to participate, but many with much more then that to run exercise at realistic scale Nicola De Filippis IPRD06, Siena, 1-5 October p. 4

5 Official timeline June 1 June 14: First Version of Detector and Physics Reconstruction software for CSA06 June 1: Computing systems ready for Service Challenge SC4 June 15: physics simulation validation complete July 1: start MC production Aug.15: Calibration, alignment, HLT (and first version L1 simulation), reconstruction, and analysis tools ready Aug.30: 50 Mevt produced, 5M with HLT pre-processing Sep. 1: Computing systems ready for CSA Sep 15: Start CSA06 Oct 1: Smooth operation for CSA06 Oct 30: End smooth operation for CSA06 Nov 15: Finish CSA06 Nicola De Filippis IPRD06, Siena, 1-5 October p. 5

6 CSA06 MC production Nicola De Filippis IPRD06, Siena, 1-5 October p. 6

7 CSA06 MC production: software and tools ProdAgent tool used to manage the MC dataset production of 50 Mevt: consists of many agents running in parallel: JobCreator, JobSubmitter, JobTracking, MergeSensor. ouput files are registered in Data bookkeping service (DBS) blocks of files are registered in Data Location System (DLS) which takes care of mapping of file blocks and storage elements where they exist able to handle thousands of jobs per day automatically produced dataset can be injected in the database for the subsequent transfer to CERN. Further details in the talk of D. Spiga CMS software (CMSSW) installed via grid tools or directly by site admins in remote sites. A local catalogue used to map LFNs to local PFNs Storage technologies deployed: CASTOR, dcache, DPM Nicola De Filippis IPRD06, Siena, 1-5 October p. 7

8 CSA06 MC production: organization 4 production teams active: 1 for OSG with contact person: -- Ajit Mohapatra Wisconsin (taking care of 7 OSG CMS Tier2) 3 for LCG: -- LCG(1) with contact person Jose Hernandez Madrid (Spain, France, Belgium, CERN) -- LCG(2) with contact person Carsten Hof Aachen (Germany, Estonia, Taiwan, Russia, Switzerland, FNAL) -- LCG(3) with contact person Nicola De Filippis Bari (Italy, UK, Hungary) Large partecipation of CMS T1s and T2s involved Nicola De Filippis IPRD06, Siena, 1-5 October p. 8

9 Monitoring of production via web interface (1) First prototype of monitoring was developed by Bari team: Nicola De Filippis IPRD06, Siena, 1-5 October p. 9

10 Monitoring of production via web interface (2) Nicola De Filippis IPRD06, Siena, 1-5 October p. 10

11 Dataset statistics 1. Minimum bias (40M) 2. Z µµ (2M) 3.T-Tbar (6M) All decays 4. W eν (4M) events selected in narrow range to illuminate 2 SMs 5. Electroweak soup (5M) W l nu + Drell-Yan (m>15 GeV) + WW +H WW 6. HLT soup (5M): 10 effective MC HLT triggers (no taus pass) W (leptons) + Drell-Yan (leptons) + t-tbar (all modes) + dijets 7. Jet calibration soup (1M) dijet + Z+jet, various pt-hat ranges 8. Soft Muon Soup (2M) Inclusive muons in minbias + J/Psi production 9. Exotics Soup (1M) LM1 SUSY, Z (700 GeV), and excited quark (2000 GeV) [all decays] Nicola De Filippis IPRD06, Siena, 1-5 October p. 11 Transfer of files to CERN is underway Total: ~ 66 M events Total FEVT: O(150) TB

12 Efficiency and problems Efficiency: Submission and Grid efficiency: 90% Aborted jobs: jobs not submitted because requirements not met (merge jobs) or jobs once submitted fail due to Grid infrastructure reason Job efficiency: 91% Fraction of jobs that started to run and did not crash Application and job wrapper failures Stage out efficiency: 98% Fraction of jobs which managed to stage out data successfully Overall efficiency: 88% Probability for a job to end successfully once it is submitted Problems: stage out is the main cause of job failures. More robust checking are under test, more attempts, a fallback strategy etc.. merging job needs particular care and baby sitting most of problems now fixed site validation: storage, software tag, software mount points, matching of CE consistency Nicola De Filippis between fileblock/files in DBS/DLSIPRD06, and the Siena, reality 1-5 at sites. October p. 12

13 CSA06 reconstruction, calibration/alignment and analysis exercises Nicola De Filippis IPRD06, Siena, 1-5 October p. 13

14 Readiness for CSA06 CMSSW_1_0_2 for CSA just released. Includes: All main reconstruction components (tracking, muons, jets, e/γ) Calibration and alignment tools for application of constants and preparation of specialized datasets (AlCaReco) Prompt reconstruction (PR) tests performed at Tier-0: 40 million events in 100 hours (a mix of all prepared samples) 135 Hz sustained on 380 CPUs ( 3s/evt/CPU! ) Caution: not final reconstruction used and one tracking algo disabled Started officially on Oct 2 a Generally planned to distribute RECO and AOD at Tier1; HepMC information will be added to AOD and RECO Average Event sizes: AOD: 73 kb RECO: 540 kb FEVT: 1.9 MB Nicola De Filippis IPRD06, Siena, 1-5 October p. 14

15 Prompt reconstruction Tier-0 Reconstruction ran at 50Hz. Smooth running, no problems. 4M Events were already reconstructed. Nicola De Filippis IPRD06, Siena, 1-5 October p. 15

16 Calibration/alignment Calibration/alignment tasks: Specialized tasks to align/calibrate subsystems using start-up miscalibrated samples, e.g. Align a portion of Tracker with HIP algorithm by using Z µµ sample on the central analysis facility (CAF) for prompt calibration/alignment Intercalibrate ECAL crystals by phi symmetry in minbias events, π 0 /η, or by isolated electrons from W/Z Specialized reduced RECO data format (AlCaReco) to be used for calibration/alignment stream from Tier-0 Mechanism to write constants back into offline DB to be used Re-reconstruction at Tier-1 required to test new constants Propose that miscalibration is applied at RECO Datasets proposed to be miscalibrated at Tier-0: Z µµ, W eν, Minbias, HLT Soup streams Nicola De Filippis IPRD06, Siena, 1-5 October p. 16

17 Skim jobs at T1s To fit data at T2, and to reduce primary datasets to manageable sizes for analyses, need to run skim jobs at T1s to select events Skim job details depend on the needs of subsequent analyses Computing operations will provide service to run officially released code for skimming with config files prepared by groups, to create secondary datasets that will registered into dataset bookkeeping Output must be of same type supported by primary datasets (AOD, FEVT) Of course at T2 (i.e. with CRAB), users can do any sort of skimming/selection Proposed dataset distribution at T1s: Nicola De Filippis IPRD06, Siena, 1-5 October p. 17

18 Analyses at T2s Nicola De Filippis IPRD06, Siena, 1-5 October p. 18

19 Analysis at T2 Bari The analysis exercise for csa06 will consists of running over the output file of the skim jobs the code for studying the performances of the track reconstruction : a) with the perfect tracker geometry; b) in the short term (and in the long term with smaller priority) misalignment scenario by reading misalignment position and errors via frontier/squid from the offline database ORCAOFF. This step requires to refit tracks with misaligned geometry but it can be done at the T2. c) by using the tracker module position and errors as obtained by the output of the alignment process that will be run at CERN T0 to verify the efficiency of the alignment procedure on the track reconstruction. Refit of tracks to be done in the T2. Global efficiency of track recostruction, track parameter resolution and fake rate will be compared in the a), b) and c) cases. The same analysis was performed in ORCA. Plots and documents at link: Data samples needed: Z µµ and TTbar (the second for computing the fake rate) Nicola De Filippis IPRD06, Siena, 1-5 October p. 19

20 Preliminary plots: CMSSW Using single muons events with PT=100 GeV/c Nicola De Filippis IPRD06, Siena, 1-5 October p. 20

21 Timetable for CSA exercise (Oct.) 2 Oct. CSA exercise officially launched Prompt reconstruction (PR) begins at T0 on minbias stream Output shipped to T1s 9 Oct. T1 Skim jobs started T1 skim jobs launched to transfer data to T2s Minbias shipped where needed, other samples when ready PR begins on calibration sample events: Z µµ, W eν, Jets Minbias production continues in parallel 16 Oct. Prompt Calibration jobs started, T2 analysis jobs started Prompt calibration jobs start at T0 to produce AlCaReco streams from RECO Including merging jobs (~100:1) T2 analysis jobs started PR begins on remaining physics samples: tt, EWK, SoftMuon 23 Oct. Calibration, Alignment jobs started Calibration/Alignment jobs started based on AlCaReco streams T2 user skim and analysis jobs fully ramped up PR begins on HLT streams Nicola De Filippis IPRD06, Siena, 1-5 October p. 21

22 Timetable for CSA exercise (Nov.) 30 Oct. T0 operations ramp down Finalize transfers to T1s Final calibration/alignment results in at proto-caf, T1, etc. 6 Nov. T1 Re-reconstruction Re-reconstruction initiated at (some) T1s based on new calibration constants 13 Nov. CSA ends Final results in from analysis exercises 4 Dec. CMS Week Show it off! Write up lessons learnt Nicola De Filippis IPRD06, Siena, 1-5 October p. 22

23 Conclusions CSA06 is a 50 million event exercise to test the workflow and dataflow associated with the data handling model of CMS The CSA06 MC production was really successful thanks also to the effort of many people at remote sites. 66 M events produced in about 1.5 months better than what expected! CMS is almost ready for the next steps of the challenge Prompt reconstruction at Tier-0 tested up to at 135 Hz and is on going officially Skim jobs ready for the needs of the different analyses Analyses are well defined and will be performed at Tier-2s. Nicola De Filippis IPRD06, Siena, 1-5 October p. 23

24 Backup slides Nicola De Filippis IPRD06, Siena, 1-5 October p. 24

25 CSA06 computing metrics For CSA06 goal is 40K jobs/day across all CMS sites 25% of foreseen load of jobs in 2008 For CSA06 target jobs success rate is 90% from middleware and 90% from CMS applications At each site demonstrate data serving capability from disk storage to CPU and 1MB/sec per execution slot or 200MB/sec goal for 2008 is 800MB/sec at Tier1 and 200MB~1GB at Tier2 Throughput metrics: data transfer rates Tier-0 Tier-1 (to tape): CSA06: 150 MB/sec from Tier0 to Tier1 (sum over all T1) that is 25% of 2008 goal Tier-1 Tier-2: CSA06 desire is to reach ~10MB/s for Tier-2s with nominal 1Gbit/sec connectivity (100MB/s to best connected Tier-2s ). I.e. 1TB/day at all Tier-2 centers; up to 10TB per day Latencies: For data to travel from CERN and be analyzed by users at T2 Efficiency of operation: Aim to demonstrate a system that can run 24x7 without continuous Nicola De attention; Filippis run daily operations, document IPRD06, down Siena, times, 1-5 understand October 2006 reasons - p. 25