Research Management Plan for the Heavy Ion Physics Program Using the Compact Muon Solenoid Detector at the Large Hadron Collider

Transcription

1 DRAFT: 22-Aug-2007 Research Management Plan for the Heavy Ion Physics Program Using the Compact Muon Solenoid Detector at the Large Hadron Collider Project # XXXX For the U.S. Department of Energy Office of Science Office of Nuclear Physics Physics Research Division (SC 26.1) Date approved: XX XX XX Revision 2 1

2 Research Management Plan for the Heavy Ion Physics Program using the Compact Muon Solenoid Detector at the Large Hadron Collider XXX XXXX CONCURRENCES: Boleslaw Wyslouch US CMS Heavy Ion Project Manager Richard Milner Director of the Laboratory for Nuclear Science MIT Date: Date: APPROVED: Gulshan Rai Program Manager Office of Nuclear Physics, Office of Science Date: 2

3 Table of Contents 1 INTRODUCTION MISSION STATEMENT SCHEDULE, MILESTONES, AND PERSONNEL LHC BEAM SCHEDULE HARDWARE AND DATA COLLECTION TASKS PHYSICS ANALYSIS INSTITUTIONAL SCHEDULES TECHNICAL SCOPE HIGH LEVEL TRIGGER FARM ZDC CALORIMETER HEAVY ION OFF-LINE COMPUTING Available Computing Resources Heavy Ion Offline Computing Requirements Heavy Ion Computing Purchase Plan Site Selection for the US CMS Heavy Ion Computing Center MANAGEMENT ORGANIZATION GROUP ORGANIZATION Service Tasks PROJECT MANAGEMENT RESPONSIBILITIES Department of Energy US CMS HI Project Manager US CMS Heavy Ion Institutional Board US CMS HI Deputy Project Manager Subsystem Managers Laboratory for Nuclear Science at MIT SCHEDULE AND COST SCOPE EQUIPMENT COST SCOPE OPERATIONS COST SCOPE CONTINGENCY CHANGE CONTROL ANALYSES, ASSESSMENTS, AND PLANS ENVIRONMENT, SAFETY AND HEALTH QUALITY ASSURANCE RISK MANAGEMENT PROJECT CONTROLS AND REPORTING SYSTEMS RELATED COSTS

4 1 INTRODUCTION This Research Management Plan (RMP) describes the coordinated efforts of the US collaborators working on the heavy ion (HI) physics program using the Compact Muon Solenoid (CMS) detector at the Large Hadron Collider (LHC) at CERN. This document describes the research goals of the US CMS HI group as well as the procedures to be used by group management to achieve those goals. The RPM defines the program scope and organizational framework, identifies roles and responsibilities of the participants, and presents the work plan and schedule. It also describes the process by which project cost, schedule, or scope may be revised in consultation with the Office of Nuclear Physics within the DOE Office of Science. Physics opportunities of the LHC heavy ion program The CMS heavy ion program is a continuation of the research effort designed to expand our understanding of Quantum Chromo-Dynamics (QCD), in particular it is aimed at studying the behavior of QCD matter at high temperature and energy density. Lattice gauge calculations suggest that at very low baryon chemical potential and at energy densities above 1 GeV/fm 3, QCD matter is a state of deconfined quarks and gluons in which chiral symmetry is restored. The existence and properties of this and other possible phases are currently the subject of very active research. Data from the Relativistic Heavy Ion Collider (RHIC) at BNL suggest that heavy ion collisions at s NN = 200 GeV form an equilibrated non-hadronic system. There is strong evidence that this dense medium is almost opaque to fast partons and is highly interactive, perhaps best described as a quark-gluon liquid. In addition, global characteristics of particle production have been found to exhibit many surprisingly simple scaling behaviors. Extrapolation of these results to LHC energies ( s NN = 5.5 TeV) suggests that the CERN heavy ion program has significant potential for major discoveries. The much higher densities will result in more rapid thermalization and an increased duration of the non-hadronic phase. In addition, the higher collision energy will result in a significantly enhanced yield of hard probes with high mass and/or transverse momentum. Studies of heavy ion collisions at the LHC will either confirm and extend the theoretical picture emerging from RHIC, or challenge and redirect our understanding of strongly interacting matter at extreme densities. The 2004 report of the NSAC Subcommittee on Heavy Ion Physics stated The LHC will open up a new regime... with significant opportunities for new discoveries and the recent Town Meeting preparing for the 2007 Long Range Plan recommended significant and timely participation of U.S. groups in the LHC heavy ion program. CMS as a heavy ion detector The RHIC results have not only transformed our picture of high temperature QCD matter, but have also redirected the experimental emphasis from a focus on soft physics to observables such as differential studies of elliptic flow, very high p T jets, and open and hidden heavy flavors. The study of hard probes requires detectors with large acceptance, high rate capability, and high resolution, leading to a convergence in experimental design between high energy heavy ion and particle physics. The CMS detector in particular excels in the following areas: 4

5 Rate capability: The CMS DAQ bandwidth will allow a detailed inspection of every heavy ion collision event in the High Level Trigger (HLT). Only a small fraction of the events can be written to tape but the HLT will allow an almost complete selection and recording of the most interesting events. To ensure adequate CPU resources to maximize the physics output of heavy ion running, the US heavy ion group will contribute to the HLT as described in this RMP. High resolution and granularity: The resolution and granularity of all CMS detector elements have been pushed to the extreme. Algorithms have been developed which exploit the capabilities of the silicon tracking system for use in heavy ion collisions. Large acceptance tracking and calorimetry: CMS includes high-resolution tracking, electromagnetic and hadronic calorimeters, and muon chambers over 2π in azimuth and over a uniquely large range in rapidity. The main hardware contribution of the US CMS HI group to the detector itself are the Zero Degree Calorimeters ( η >8.0) which allow measurement of neutral particles at very forward rapidities. Participation of US groups in the CMS heavy ion program Although studies of heavy ion collisions were part of the CMS physics program from the very beginning, the effort was pursued at a relatively low level. The primary concern was to ensure that detector design decisions did not conflict with the constraints of heavy ion running. In 2002, following expressions of interest from a number of US heavy ion groups, Bolek Wyslouch from MIT was asked to become the CMS Heavy Ion physics coordinator and has been leading the program since then. Members of the US groups have taken strong leadership and advocacy positions within the full CMS heavy ion group. These efforts have significantly enhanced the overall program and have also attracted new non-us collaborators to join the CMS heavy ion program, thus notably multiplying the return on the investment of US resources. The US heavy ion involvement is strengthened by the existence of CMS high energy physics groups at most of the participating institutions. There is considerable synergy of activities between the two groups, especially in the areas of computing, data acquisition, and triggering. The US groups have constructed Zero Degree Calorimeters that will be completed and installed during FY2007. Our future hardware contribution will consist of components of the HLT. Members of the US groups have made significant contributions to tracking software, simulations, HLT validation, and the software framework. The Office of Nuclear Physics organized a Technical, Cost and Schedule Review of the US CMS HI project at the DOE office in Germantown in October, The proposal to participate in CMS was reviewed. Per the review panel s recommendation, funding of the prototype HLT farm was approved. The funding of subsequent stages of the HLT was made contingent on demonstrating the physics sensitivity of the trigger algorithms. Schedule of LHC beams LHC running will begin with p+p collisions at a center of mass energy of 14 TeV in Starting with the 2009 LHC run, Pb+Pb runs are planned for one month at the end of each year s p+p running period. The US heavy ion groups are planning to fully participate in the data taking and analysis of the low-luminosity minimum bias p+p runs which will provide crucial reference data for the Pb+Pb analysis. Furthermore, full 5

6 participation in the 2008 p+p run is critical to ensure readiness of the HLT hardware as well as the trigger and physics analysis algorithms for the first Pb+Pb data. 2 MISSION STATEMENT The mission of the Nuclear Physics program within the DOE Office of Science is to foster fundamental research in nuclear physics that will provide new insights and advance our knowledge on the nature of matter and energy. The US CMS heavy ion group will support this mission by using ion+ion results and p+p reference data to elucidate the properties of QCD matter at the highest temperatures and energy densities ever achieved in the laboratory. Concurrently, the characteristics of particle production in nuclear interactions will be studied over a much broader range than previously possible. More specifically, the mission of the US group focuses on the following goals and tasks: Construct, install, commission, calibrate, and maintain two ZDC detectors. Design, commission, and maintain Level 1 triggers for heavy ion collisions and minimum bias p+p. Purchase, install, and help to maintain a subset of the compute servers in the HLT. Design, evaluate, commission, and maintain HLT algorithms for heavy ion physics. Design, implement, and maintain procedures for monitoring data quality for heavy ion running. Monitor local and remote storage of heavy ion physics data. Perform analysis of heavy ion physics results and p+p reference data in order to achieve the physics goals summarized above. 6

7 3 SCHEDULE, MILESTONES, AND PERSONNEL 3.1 LHC BEAM SCHEDULE The first LHC p+p collisions at the design energy of 14 TeV are expected in the summer of Currently, we expect that the first Pb+Pb run at s NN =5.5 TeV will occur in late 2009 (FY10 Q1), at the end of the p+p run scheduled to begin in the spring of that year. Future p+p runs are expected yearly starting in the spring, with Pb+Pb runs for one month at the end of the p+p running period. An approximate estimate of the data samples from the first few LHC runs is shown in Table 1. For p+p collisions, only the expected number of minimum bias events is listed since the more restrictive triggers focus to a large degree on observables of interest only to the high energy program. Reference p+p data at a center of mass energy of 5.5 TeV must be taken as part of the one month per year allocated to heavy ion running and, therefore, will not occur until later in the program, as shown in the table. The total number of events is limited by the tape writing speed and resources available to store data. As the LHC integrated luminosity grows and as the heavy ion physics topics become more focused, the triggering algorithms will be continually updated to make them more selective. Date Beam Million Events FY08 Q3/4 Physics p+p, low luminosity 100 FY09 Q3/4 Physics p+p, full luminosity 50 FY10 Q1 Physics Pb+Pb, 1/20 nominal luminosity 50 Q3/4 Physics p+p 50 FY11 Q1 Physics Pb+Pb, full luminosity 100 Q3/4 Physics p+p 50 FY12 Q1 Physics Pb+Pb 200 Q3/4 Physics p+p 50 FY13 Q1 Physics Pb+Pb 100 " p+p at Pb+Pb beam energy 250 Table 1: The projected run schedule at the LHC. Note that only the minimum bias events of interest to the heavy ion program are counted for the p+p runs. These estimates assume the LHC run schedule described above. The heavy ion groups are planning to fully participate in the data taking and analysis of the early low-luminosity, minimum bias p+p runs which will provide crucial reference data for the Pb+Pb analysis. Furthermore, full participation in these runs will help ensure readiness of the trigger hardware and analysis algorithms for the first Pb+Pb data. Due to the relatively short time allocated to Pb beams, it is critical to take advantage of any opportunities to begin evaluation and testing of the trigger and physics analysis tools well in advance. The significantly lower luminosities in the early p+p runs will allow a cleaner environment for determining global observables and other properties of interest to the heavy ion physics program. In addition, minimum bias triggers will only be allocated a small fraction of the DAQ bandwidth at higher luminosities. There is a small possibility of a short Pb+Pb commissioning run at s NN =5.5 TeV in late 2008 (FY09 Q1) at the end of the first LHC p+p run. Even a short run would provide an invaluable opportunity to further test the HLT and physics analysis algorithms which will 7

8 be initially evaluated using p+p data. A few days of usable collisions will provide enough data to make significant and fundamental global measurements such as charged multiplicity, elliptic flow, etc. Early data could also be used to tune the HLT algorithms that depend on the detailed properties of the underlying event and detector performance. 3.2 HARDWARE AND DATA COLLECTION TASKS The hardware and data collection tasks directly related to heavy ions fall into six main categories: the ZDC, the Level 1 trigger, the HLT, data quality monitoring, movement and storage of data, and offline computing facility(ies). Note that the ZDC construction and installation is funded through a separate project but the calibration, maintenance, and analysis of the ZDC will be part of the US CMS HI effort. The hardware and computing sub-tasks are listed in Table 2 and the major milestones for these aspects of the project are listed in Table 3. Beyond the first full heavy ion run, the tasks in this category will consist of maintaining all of the components and adapting them as necessary due to changes in hardware performance and/or physics focus. The HLT algorithms will clearly evolve over time to reflect the physics potential of the expected increase in luminosity. Hardware and Computing Task Date Complete ZDC testbeam FY07 Q3 Complete data handling procedures for min-bias p+p data FY08 Q2 Commission Level 1 min-bias p+p trigger Q3 Maintain and fine tune Level 1 min-bias p+p trigger Q3/4 Evaluate data monitoring, ZDC, and HLT using p+p data " Complete data handling procedures for heavy ion data Q4 Complete initial data monitoring for heavy ion collisions " Maintain Level 1 min-bias p+p trigger FY09 Q3/4 Fine tune data monitoring, ZDC, and HLT using p+p data " Complete final data handling procedures for heavy ion data Q4 Complete final data monitoring for heavy ion collisions " Fine tune Level 1 heavy ion trigger with Pb+Pb data FY10 Q1 Fine tune data monitoring, ZDC, and HLT with Pb+Pb data " Maintain Level 1 min-bias p+p trigger Q3/4 Update final HLT configuration for heavy ion collisions Q4 Fine tune HLT with higher statistics Pb+Pb data FY11 Q1 Table 2: Schedule of major hardware and computing project tasks assuming that the LHC run schedule proceeds as described in Table 1. The responsibilities for completing the tasks and milestones listed in Tables 2 and 3, respectively, are, to a large extent, concentrated within individual institutions. All ZDC tasks will be done by Kansas and Iowa. UIC will take charge of all of the items involving the Level 1 trigger as well as the centrality determination. Maryland will arrange and maintain the data handling procedures. To the extent that on-line trigger algorithms are similar to their off-line counterparts, input to the HLT software will come from many sources, including many of the US groups. For example, as their STAR commitments wind down, UC Davis will apply their experience to the development of a heavy flavor trigger that will be useful in the high luminosity runs. MIT has responsibility for the 8

9 tracking and jet-finding components and is responsible for the implementation and optimization of software within the HLT framework. LANL, UIC, and Iowa will play significant roles in the development and utilization of off-line computing. Vanderbilt and Colorado have significant remaining commitments at RHIC and will initially devote a smaller fraction of their group effort to CMS but will nonetheless make important contributions to the early preparations. The data-monitoring area will have the largest input from non-us groups. However, US institutions will contribute in their particular areas of expertise. For example, the Iowa group has extensive experience with and connections to the calorimetry groups within CMS. Hardware and Computing Milestone Date Select site for the US CMS HI computing center FY07 Q4 Complete ZDC construction " Complete design of Level 1 p+p min-bias trigger FY08 Q1 Install prototype HLT farm at CERN " Demonstrate HLT suitability for CMS heavy ion program " Install ZDC Q2 Install Stage 1 HLT (200 computers) in CMS " Begin installation of computers in the US CMS HI computing center Q3 Complete design of the Level 1 heavy ion trigger Q4 Complete initial HLT configuration for heavy ion collisions " Install Stage 2 HLT (200 computers) in CMS FY09 Q2 Install additional 100 offline analysis computers Q3 Complete final HLT configuration for heavy ion collisions Q4 Install Stage 3 HLT (200 computers) in CMS FY10 Q2 Table 3: Major hardware and computing milestones for the US CMS HI project. 3.3 PHYSICS ANALYSIS The US heavy ion group has mapped out an analysis plan focusing on areas where extrapolations from RHIC indicate exciting prospects, as well as highlighting the new possibilities at the LHC and the unique capabilities of CMS. The goal is to probe and characterize the medium formed in heavy ion collisions at the LHC using both parton energy loss and the suppression of heavy quarkonia. The timescale with which various physics topics can be addressed is determined by the evolution of the beam luminosity. The initial effort will concentrate on jet quenching, starting with individual jets and moving to correlations as statistics improve. The culmination of the initial phase of the plan is the extraction and detailed study of γ-jet correlations at high p T in Pb+Pb collisions. The γ-jet analysis will utilize the extensive calorimetric coverage of CMS coupled with the excellent capabilities of the silicon-tracker for charged particles. Using the γ as a non-interacting partner associated with a hadronic jet will provide a unique tool to study modifications and energy loss of high momentum partons, essentially performing tomography of the high density QCD matter. The quarkonia program will consist predominantly of detecting particles decaying to muons and will require the increased luminosity expected in the later runs. As more personnel make the transition from RHIC experiments to CMS and as the LHC heavy ion 9

10 luminosity ramps up, the study of quarkonia and other analyses using muons will take on an increasingly important role in the heavy ion program. Clearly, it is impossible to predict the most interesting physics to be found using heavy ion beams at the LHC. Therefore, it should be stressed that almost every step in the planned analysis chain has important physics content and a very real potential for significant discovery. Global analyses, which can begin to provide useful physics using relatively small data samples, can produce interesting and unexpected results while simultaneously detailing the background on top of which all rarer processes must sit. As just one example, elliptic flow data have produced some of the most exciting results at RHIC and also provided critical input to particle correlation studies. Building on the lessons learned at RHIC, the analysis framework should be flexible in order to adapt to surprises or changes in focus in response to earlier results. The elements of the analysis (in planned order of completion) are the following. Note that high energy physics analysis also requires many of these elements. The heavy ion groups will be working in close collaboration with their high energy colleagues to determine which elements can be treated identically and which require modified or new approaches. Vertex reconstruction Multiplicity determination (single/multiple layers in the Si-tracker) Centrality determination using initially multiplicity + ZDC, expanding to include Forward HCAL and other detectors for best results Physics: dn ch /dη, multiplicity vs. centrality Reaction plane determination Physics: Integrated elliptic flow vs. eta at midrapidity Tracking reconstruction and extraction of p T Track embedding to get corrections for the high p T correlations measurement. Physics: Elliptic Flow vs p T and centrality (for charged particles) The following physics topics can be addressed following the acquisition of additional data in the higher luminosity runs. It is possible that initial jet results could be attainable using data from the FY10 Q1 Pb+Pb run. Physics: High p T azimuthal correlations (charged particles) vs. centrality Track matching between Si-tracker and calorimetry Calorimetry calibrations HLT trigger on jets for statistically significant sample Physics: Direct jet measurements with calorimeters and jet-correlations Jet-tagging trigger improvement. Physics: Jet fragmentation functions and modification by the medium Physics: γ-jet correlations, followed by γ-jet correlations vs. centrality, reaction plane orientation, and other differential measurements as statistics improve. Figure 1 shows the sequence and connections between the required measurements in the analysis with highlighted boxes indicating the most interesting possibilities for standalone physics, in many cases for both pp and heavy ion collisions. 10

11 Figure 1: Major analysis steps leading to the study of γ-jet correlation functions. Shaded boxes indicate items with significant stand-alone discovery potential. Preliminary schedules of analysis tasks and physics milestones are listed in Tables 4 and 5, respectively. The dates in these tables depend strongly on the decision concerning an early pilot Pb+Pb run since many of these physics topics can be addressed at some level with very small data samples. Physics Analysis Task Date Complete initial min-bias p+p physics analysis code FY08 Q2 Perform and fine tune first min-bias p+p analysis during run Q3/4 Complete initial analysis code for heavy ion data Q4 Complete final min-bias p+p physics analysis code FY09 Q2 Perform min-bias p+p analysis during run Q3/4 Complete updated analysis code for heavy ion data Q4 Perform and fine tune first heavy ion analysis during run FY10 Q1 Table 4: Preliminary schedule for physics analysis tasks. Note that the dates are contingent on the delivery of the expected luminosity of heavy ion collisions. The goals of the entire CMS HI physics program must be carefully matched to the available personnel resources. Therefore, physics analysis by all US groups will be conducted in close collaboration with each other and also with non-us heavy ion groups within CMS. Where appropriate, there will also be significant overlap with high energy groups, especially in the development of some of the more basic algorithms. Although individual US groups have expressed particular interest in one or more of the topics discussed above, none have yet been given specific responsibility for completing any elements of the plan outlined above. Instead, the effort of the entire CMS heavy ion group will be coordinated and prioritized to focus on the most important physics questions at any given time. 11

12 Physics Milestone Date First min-bias p+p physics results (dn/dη, p T distributions) FY09 Q2 First global heavy ion physics (dn/dη, spectra, elliptic flow) FY10 Q3 Observation of jets, jet shape characterization Q4 Jet-jet correlations, particle correlations at high p T FY11 Q2 Observation of γ-jet signal Q3 Muon spectra, heavy quark production rate Q4 γ-jet correlations FY12 Q3 First results for quarkonia in heavy ion collisions Q4 Table 5: Major milestones for physics analysis tasks. Note that the dates are contingent on the delivery of the expected luminosity of heavy ion collisions. Initially, the highest priority will be achieving the goals described in this section. Assuming that early results do not indicate that a different approach might be more productive, the analysis will proceed as outlined. Based on the current interpretations of RHIC data, the planned measurements are sure to expand our understanding of this exciting QCD regime. However, experience at RHIC and elsewhere has taught us that whenever a new energy range is reached, we must be prepared to redirect our resources to respond to the unexpected. Several of the US CMS HI groups have expressed an interest in muon physics in general and quarkonia in particular. This topic will have very significant involvement of non-us groups. Because these analyses require the larger statistics resulting from high luminosity data, they cannot begin until later in the program. At that time, the physics efforts discussed above should be well developed. Also, as members of the US CMS HI groups fulfill their responsibilities at RHIC, we expect that available personnel resources will slowly expand. Therefore, we expect that the physics program will also gradually expand to encompass measurements beyond those in the core jet physics plan. In addition to the development of physics analysis infrastructure, significant resources will be applied to simulations used to optimize the trigger performance, evaluate detector capabilities, and study how background affects the various observables. To a greater or lesser extent, all three of these tasks have already started. This work will continue in parallel with preparation for collecting data and starting physics analysis. However, no individual group will be entirely responsible for performing the simulations. Instead, each group will determine the needs of their particular project(s) and the work to generate Monte Carlo data will be organized incorporating effort from all groups. The various efforts are being coordinated to avoid duplication of effort and set priorities. 12

13 4 INSTITUTIONAL SCHEDULES The US CMS HI group intends to fully exploit the physics potential of the CMS apparatus for heavy ion physics. This will include participation in the data taking, maintenance of hardware and software, and physics analysis. The leadership role that the US CMS HI group members play in the program requires significant and timely involvement. Our collective RHIC experience and excellent graduate and undergraduate students will allow us to make significant contributions. However, in order to fully exploit the physics potential of CMS, the US CMS HI group needs to reach critical mass and group members need to be engaged at a significant level well in advance of the arrival of heavy ion beams. The unique measurements that CMS is planning to do require multi-year preparations to ensure that the trigger, software, and physics analysis chain are all fully understood and operational. Institution FY2007 FY2008 FY2009 FY2010 FY2011 FY2012 Ph St A Ph St A Ph St A Ph St A Ph St A Ph St A Colorado Iowa Kansas LANL Maryland Minnesota MIT UC Davis* UIC Vanderbilt FY2007 FY2008 FY2009 FY2010 FY2011 FY2012 Cat-A (DOE) PhD Students All Table 6: Plan for participation of personnel at individual institutions in the US CMS HI project. The table contains the number of PhD physicists (Ph), graduate students (St), and total (A) for each institution as a function of fiscal year. * Note that UC Davis is funded by NSF. The individual groups participating in the US CMS HI project carefully considered their future RHIC and CMS obligations and prepared an estimate of the number of physicists and graduate students who would participate in each program over the next five years. The rate of increase of CMS participation is different at each institution. Due to their smaller RHIC commitments, the Iowa, Kansas, MIT, and UIC groups will contribute a large fraction of the personnel early in the program but vital early contributions will also be made by UC Davis, Colorado, LANL, Maryland, and Vanderbilt. The Minnesota group will provide theoretical guidance with the possible addition of an experimental 13

14 effort in later years. The fiscal year profile of participation by personnel at individual institutions is shown in Table 6. Institutions participating in the US CMS HI program will coordinate with the Project Manager to ensure that the appropriate CMS M&O payments required for their work on the program are paid. The PhD row near the bottom of Table 6 counts all physicists working on the program including the NSF-supported UC Davis personnel while the Cat-A row counts only DOE-funded personnel. Note that the estimated number of PhD physicists listed in the table for FY2007 differs significantly from the Category-A payments for FY2007. The number in the PhD row indicates the count of physicists participating in the program as of April, 2007 and continuing until the end of FY2007, whereas the Cat-A number was fixed in September, 2006 based on the status of the CMS personnel database at that time. For FY2008 and beyond, the number of DOE-supported PhD s and number of DOE Cat-A payments should be identical. 14

15 5 TECHNICAL SCOPE The physicists participating in the US CMS HI program, as members of the full CMS collaboration, are contributing to three areas of detector construction and maintenance: 1) High Level Trigger Farm 2) ZDC Calorimeter 3) Computing system 5.1 HIGH LEVEL TRIGGER FARM In CMS, the online inspection of events is divided into two steps. The Level 1 trigger uses local data from the calorimeter and muon systems to make electron/photon, jet and energy sum, and muon triggers. This decision is sent to the front-end detector electronics after a latency of 3 µs. All events fulfilling the requirements are then transferred to the High Level Trigger (HLT), where further selection takes place. The rejection rate at Level 1 needs to be sufficiently large to reduce the accepted event rate to about 100 khz for p+p in order to match the HLT input bandwidth. The lower luminosity, and hence the lower frequency, of heavy ion collisions allows the use of a very loose Level 1 trigger and consequently allows most if not all of the trigger decisions to be made at the HLT stage. This an extremely important advantage of the CMS detector since heavy ion events are expected to have a large background of soft particles, therefore requiring more complex algorithms to do careful selection of classes of interesting events. Raw data triggered at Level 1 is read via about 650 Front End Controllers and combined in a system of network switches to form complete events. These events are then transferred to Filter Units, dedicated computers that run the HLT selection algorithms. Each computer node is an individual PC with limited local resources (both memory and disk space) but with access to external, shared resources. It is part of a distributed computing environment including network, servers, and disk storage. The Filter Unit computers make the ultimate decision about which events will be written to permanent storage. The selection algorithms need to be fine-tuned to match the physics goals of the program and they have to be reliable, fast, and well understood. The HLT will eventually have over 1500 Filter Unit computers. Heavy ion physics has different requirements than those of the p+p program and, therefore, it is essential that the heavy ion group is directly involved in the installation, running, and maintenance of the HLT farm which determines the physics selectivity of the entire DAQ chain. The US CMS HI group plans to purchase about 200 computers in each of the three fiscal years FY08, FY09, and FY10 and install them in the CMS Counting House. The staged purchase will allow us to take advantage of the improvements in PC technology as well as to match the growing LHC luminosity. Members of the US CMS HI group will also be involved in the operation and maintenance of the HLT farm. 5.2 ZDC CALORIMETER The ZDC was designed and will be constructed, tested, and installed by Kansas and Iowa as part of a separate project with separate management. Once installed, calibration, maintenance, and analysis of the ZDC will become part of the US CMS HI effort and will be covered by this management plan. 15

16 5.3 HEAVY ION OFF-LINE COMPUTING In recent years, nuclear physics, along with other fields of experimental science, has experienced an explosion in the volume of digitally stored data. The level of available computing significantly impacts the scientific reach of many research programs, stressing the need to keep the growth of computing facilities in step with the data generation rate. It is of crucial importance for the success of the heavy ion physics program at CMS to have adequate resources for data processing, storage, and analysis. This section outlines the plan for a computing facility dedicated to the US CMS heavy ion program Available Computing Resources The heavy ion physics program is an integral part of the CMS experiment. For both p+p and Pb+Pb runs, the data acquisition, archiving and first pass reconstruction of the events are expected to be performed in real time at a large computer farm at CERN, the Tier-0 center. This center will also distribute the raw data to the next layer of Tier-1 centers located in different countries. As is the case for all other Tier-1 centers, the US facility at FNAL will be responsible for storing a second copy of the raw data. CPU resources for further reconstruction and calibration are provided as well, but only for p+p data. Batch processing of data analysis, as well as Monte Carlo simulations, will be performed at the many Tier-2 centers, each of which will provide significant quantities of CPU power and disk storage. It is our plan to take full advantage of the resources and infrastructure provided by CMS. We have designed the computing model for heavy ion data to be closely aligned with that for CMS in general. Our goal is to use the standard infrastructure for transferring, storing, and accessing the data, as well as the standard software framework for reconstruction and analysis. Members of US CMS HI groups have already started participating in software development, including the critical validation of all elements of the framework for heavy ion events which are typically larger in size than those for p+p data. In areas where the heavy ion data and analysis are sufficiently different from p+p, heavy ion specific algorithms and optimizations will be incorporated into the existing framework. At the present time, however, CMS off-line computing resources outside CERN, including both Tier-1 and Tier-2 have not been specifically allocated to heavy ion physics analysis Heavy Ion Offline Computing Requirements The goal of our computing plan is to build a facility that is sufficient for the specific needs of the heavy ion program. The required computing includes calibration, event reconstruction, Monte Carlo simulation, and physics analysis. In contrast to the p+p data analysis, some of the heavy ion event reconstruction can be carried out during periods outside the heavy ion running periods. As data become available, understanding of the input parameters for the reconstruction software, such as alignment and calibration, will improve and other enhancements to the reconstruction algorithms may be developed. Therefore, it is expected that the data will have to be reprocessed, possibly multiple times during the first year(s). Simulated event data, created using Monte Carlo techniques, play a crucial role in basic studies of the detector response, the development of reconstruction software, and the comparison to predictions of physics models. The heavy ion computing facility will need sufficient CPU capacity for all of these tasks. 16

17 Our estimates for the CPU requirements and data storage needs are based on the expected data volume which is constrained by the available tape writing speed at Tier-0, currently expected to be 225 Mbyte/sec. This translates into about 250 TByte of raw data, and approximately 430 TBytes of reconstructed data for each heavy ion run. Note that the Pb+Pb design luminosity produces a collision rate of roughly 8 khz while the tape writing speed for heavy ion events (which are larger than those for p+p) is estimated to be in the range of Hz. It is expected that data from the early runs will include a high fraction of minimum bias events while later runs will focus on rarer processes. Therefore, even if the luminosity is orders of magnitude lower than the design goal, the event rate will be sufficient to fill the available tape storage bandwidth, and all of these events can be used for productive physics analysis. Consequently, we assume that all heavy runs will produce data samples of roughly 250 TByte, independent of the luminosity or HLT settings. To efficiently complete the off-line data reconstruction and physics analysis, the US CMS HI group will need to store all raw heavy ion data locally. The necessary CPU resources are estimated based on early studies with prototype CMS analysis algorithms and the experience gained at RHIC Heavy Ion Computing Purchase Plan Based on a quote from the FNAL Tier-1 facility, the cost of tape storage of the heavy ion data, as well as Monte Carlo simulation results, is expected to be $75K annually starting in FY2008. It is clear that significant computing resources beyond those currently allocated in the CMS experiment are needed in order to keep the analysis commensurate with the running schedule. For the purchase plan, we estimate the future capability of the CPUs based on Moore's Law, projecting the recent growth per year at roughly 50% in CPU speed and disk size for constant cost. We plan facility growth at about 80 nodes a year at a cost of $325K annually. The anticipated funding profile for the facility is shown in Table 7. This growth would ultimately provide 100 SI2Ks of integrated CPU power. Based on our current estimates, such a facility will support the expected basic storage and CPU needs for heavy ion physics. Computing Facilities Equipment (k$) M&S (k$) Total (k$) Startup Initial production system Production expansion Production expansion CPU replacement CPU replacement Table 7: The anticipated annual funding profile for the heavy ion computing facility Site Selection for the US CMS Heavy Ion Computing Center The budget outlined above includes only tape storage and computing costs and does not cover infrastructure costs such as power and cooling or IT support. The funding for those additional aspects is being sought from other sources, and the outcome of these searches will strongly impact the choice of the hosting institution. Possible candidates include Iowa and Vanderbilt, both of which have presented bids for hosting high energy physics Tier-2 centers; MIT, where the high energy and heavy ion groups are already cooperating 17

18 in the installation and operation of a Tier-2 facility; and UIC, which is seeking state and institutional funding to establish a dedicated CMS heavy ion computer facility. In order to perform a comparative analysis of the capabilities of the various institutions, we will form a committee consisting of representatives from all US CMS HI institutions and external experts (from FNAL and other Tier-1 or Tier-2 sites) to review the existing infrastructure and institutional support commitments at interested sites. This committee will prepare a list of requirements and benchmarks, using the high energy Tier-2 selection procedure as a guideline. Their final recommendation is expected to be based largely on the quality of available infrastructure. Although the detailed list of requirements is not yet finalized, some obvious criteria include availability of adequate space, networking bandwidth, power, cooling, and support personnel as well as the contributions from local funding sources. The committee will conduct the review of proposals, possibly including site visits, and conclude its work with a recommendation to the US CMS HI Institutional Board. Following a decision by the Board, a proposal for a facility at the selected site will be prepared and submitted to DOE. 18

19 6 MANAGEMENT ORGANIZATION 6.1 GROUP ORGANIZATION This section describes the management organization of the US CMS HI group and its interaction with other groups within CMS. The group includes all US university groups, laboratories, and individuals interested in the CMS heavy ion program. The management structure is designed to facilitate development of the program, construction of hardware, and participation in physics analysis. For simplicity and consistency, all DOE funding for this project will be fiscally managed by the Laboratory for Nuclear Science (LNS) at MIT. The interaction of members of the US CMS HI group with other physicists working on the CMS experiment is organized at four levels: CMS Collaboration: The physicists and students from the US CMS HI group must be official members of the CMS collaboration in order to have access to the experiment and its facilities and to be able to analyze and publish the collected data. Collaboration members are expected to: Participate in preparations for runs, staff shifts during data taking for both p+p and heavy-ion runs, and perform other service tasks as described below. Assume responsibility for experimental service tasks as a part of maintenance and operation of the experimental hardware (Category-B) Contribute a yearly Maintenance and Operation fee (M&O, Category-A) Conduct physics analysis within the CMS physics program and publish results following CMS publication and authorship rules. Figure 2: Relationship of the US CMS HI Group to the US CMS Research Program. US CMS Collaboration and Research Program: CMS is an international collaboration and the management structure is based on the national origin of the collaborating members. The US CMS HI group is the part of the US CMS collaboration and a sub-task of the US CMS Research Program (US CMS RP) which manages M&O activities. All US CMS physicists, including those working in the heavy ion program, coordinate their 19

20 service activities and hardware contributions through the US CMS RP. The structure of the Research Program is shown in Figure 2. The US CMS Heavy Ion Project Manager or designate will be a member of the US CMS RP Technical Advisory board. He or she will coordinate the activities of the members of the US CMS HI group in the area of service contributions, shift taking, daily interactions with CERN, and computing operations. The elected Chair of the US CMS Heavy Ion Institutional board or designate will be a member of the US CMS Collaboration Institutional Board. She or he will coordinate the activities of the US CMS HI group members on national collaboration matters. US CMS Heavy Ion group: As described in the Research Plan, the members of this group share their interest in the heavy ion physics program of CMS and are funded by the DOE or NSF offices responsible for Nuclear Physics. For the purposes of planning and managing the US CMS HI physics program, institutions will sign Memoranda of Agreement (MOA) with MIT/LNS detailing their commitments to the project. In most cases, these MOA s will be a subset of the full CMS MOA for each institution as described below. US CMS Heavy Ion Institutions: The main governing body of CMS and its equivalent in US CMS, is the Collaboration Board. It is a representative body with individual institutional Team Leaders representing the collaboration members from their own institutions. The US CMS HI group members, with the exception of those at LANL, are at institutions where high energy groups are already part of CMS. Each institution will have a single joint representative on the Collaboration Board serving both heavy ion and high energy groups. Institutions sign a Memorandum of Agreement with CMS specifying the scope of their service tasks and responsibilities. Both heavy ion and high energy members from a given institution are included in these MOA s. Note that these documents do not represent a legally binding commitment on the part of the institution and therefore should not require approval beyond that of the group. New institutions wishing to join the collaboration must be individually accepted through a process described in the CMS constitution which includes a vote by the Collaboration Board Service Tasks The CMS experiment is a collaborative endeavor. Following the tradition in nuclear and high energy physics, collaboration members contribute to the experiment by providing manpower for service tasks that benefit the experiment as a whole but are not necessarily connected to the individual s physics interests. Examples include maintenance and construction of hardware, shift taking, online and offline software development and maintenance of computing facilities. It is expected that a CMS collaboration member will spend a significant fraction (25-50%) of his or her research time on such tasks, with a larger fraction during the first year of participation. In order to appear as an author on a physics paper one must be a full, contributing member of the CMS experiment for one calendar year before publication. In addition to taking shifts, members of the US CMS HI group plan to contribute in the areas related to our hardware activities, namely ZDC, HLT, and computing. These will be detailed in the CMS HI institutional MOA s which have been described above. 20

21 6.2 PROJECT MANAGEMENT RESPONSIBILITIES The management structure for US CMS HI activities is shown in Figure 3. The group s hardware projects and charges related to collaboration membership (M&O activities) are funded by the DOE Office of Nuclear Physics. The Laboratory for Nuclear Science at MIT (LNS) will provide administrative support and fiscal supervision for the project. Figure 3: Management organization chart for the US CMS HI group Department of Energy Within DOE s Office of Science, the Office of Nuclear Physics (NP) has responsibility for the US CMS HI research program. Gulshan Rai is the Program Manager in the Office of Nuclear Physics in charge of the funding of the US CMS HI group. Responsibilities The DOE Program Manager s responsibilities include: Providing programmatic direction for US CMS HI via the Project Manager. Functioning as DOE headquarters point of contact for the program. Overseeing the progress of the program and organizing reviews as necessary. Budgeting the funding to execute the program. Controlling changes to program baselines in accordance with the RMP US CMS HI Project Manager The US CMS HI Project Manager shall be appointed by the US CMS HI Institutional Board with the concurrence of DOE and LNS. The Project Manager will report directly to the appropriate Program Manager at DOE and to the Director of LNS. This position is currently held by Prof. Boleslaw Wyslouch. 21

22 Responsibilities The Project Manager be in charge of the overall management of the US CMS HI program and will appoint key program staff in consultation with the Institutional Board. The Project Manager also will have the following responsibilities: Ensuring the successful execution of the full scope of the US CMS HI program. Providing oversight of project documentation. Identifying and ensuring timely resolution of critical issues. Acting as the spokesperson for the project to the DOE, other participating institutions, and the scientific community. Appointing the Deputy Project Manager in consultation with the Institutional Board. Collaborating with the Deputy Project Manager and the group leaders of US CMS HI institutions to assemble the staff and resources needed to complete the project. Recommending major subcontracts to the Director of LNS, in consultation with the Deputy Project Manager and the Institutional Board. Reporting regularly to the Institutional Board on project progress and finances. Keeping the scientific community informed of the progress of the US CMS HI program. Coordinating the areas of collaboration and the relationship between the institutions in the US CMS HI group and CMS through Institutional Memoranda of Agreement. Providing reporting to DOE. Ensuring the work is performed safely and in compliance with appropriate rules. Producing necessary ES&H documentation US CMS Heavy Ion Institutional Board Composition The US CMS HI Institutional Board shall be composed of one representative from each of the participating institutions. There will be an elected Chair of the Board. The LNS Director or designate will serve ex-officio on the Board. Responsibilities Serving as an oversight committee to the project. Ensuring that the project tasks are proceeding on schedule and on budget. Reviewing progress of project. Appointing the Project Manager in consultation with DOE and the LNS Director US CMS HI Deputy Project Manager The US CMS HI Deputy Project Manager shall be appointed by the Project Manager, in consultation with the Institutional Board. The Deputy Project Manager will report to and work closely with the Project Manager. This position is currently held by Prof. David Hofman. Responsibilities Under the direction of the Project Manager executing the scope of US CMS HI project, and supplying deliverables on time and within budget. Collaborating with the Project Manager to assemble the staff and resources needed to complete hardware components of US CMS HI. Communicating the functional requirements to the subsystem managers. Ensuring the work is performed safely and in compliance with the appropriate rules. Acting as Quality Assurance Manager or assigning a designate as QAM. 22