Preserving and Expanding Access to Legacy HEP Data Sets

Transcription

1 Preserving and Expanding Access to Legacy HEP Data Sets Gregory Dubois-Felsmann, SLAC BaBar Computing Coordinator 2005-last week LSST Data Management system architect from 11/15 ICFA Seminar - 28 October 2008

2 Preserving and Expanding Access to Legacy HEP Data Sets General thoughts BaBar s work in this direction HEP-wide initiative(s) to this end Legacy HEP Data - Dubois-Felsmann 2

3 The challenge of legacy data sets A generation of HEP experiments across the field are concluding their data-taking and beginning to wind up their data analysis efforts Notably at HERA, the B-factories, and the Tevatron Their data sets still have much to tell us Things we know are interesting to study Each experiment will have a long tail of analyses being completed... and things we don t know about yet Measurements that we ll only think to make in the light of new physics we might find at the LHC or elsewhere - or just new theoretical ideas They may be able to tell us more when used in concert PDG-style averaging will not always yield optimal results Legacy HEP Data - Dubois-Felsmann 3

4 The challenge of preserving data HEP has little or no tradition of or existing art for the longterm preservation of data and the means to exploit it. The preservation of usable access to data is generally not part of experiments planning, not part of their software design, and neither the up-front nor the maintenance costs are funded. Why not? Probably an implicit assumption that the next experiment was always going to come soon enough that it didn t really matter It s not easy - technology is continually evolving out from under us Legacy HEP Data - Dubois-Felsmann 4

5 The challenge of expanding access HEP has little tradition of providing access to raw data beyond the collaboration that collected it Even after a proprietary period, as is common in astronomy and space science This is a undeniably hard problem Organizationally hard Issues of control, attribution, correctness, reputation, return on investment, free-rider problems Technically hard Virtually every experiment s data, and associated tools, are unique We often fail to adequately document our work even internally Costly Real expenditures of time and equipment needed to do this well Other fields budget for this up front; we have not Legacy HEP Data - Dubois-Felsmann 5

6 The opportunities Many benefits could flow from preserving legacy data and, on some time scale, expanding access to it Re-analyzing data in the light of future discoveries, e.g., to check whether new physics discovered at Super-LHC has a measurable effect in loop processes in the BaBar & Belle data Evaluating what if? questions concerning new theories Constructing global analyses for phenomena that could be visible in multiple data sets Educational and outreach activities The ability for students to replicate critical measurements of the past Valuable even if not all of the original physics reach can be preserved (but we should not give this up lightly) E.g., because a simplified common data format were used Legacy HEP Data - Dubois-Felsmann 6

7 A purely personal opinion It seems to me to rise to the level of a moral obligation to our peers and the fellow citizens who funded us that we at least make a serious attempt to address these issues and understand both the costs and benefits Legacy HEP Data - Dubois-Felsmann 7

8 BaBar work in this area

9 BaBar s data BaBar has a unique data sample Very competitive Y(4S) dataset with an excellent detector and tools By far the largest Y(2S) and Y(3S) datasets in the world Very large and clean sample of tau lepton decays, D (s) decays Many new-physics signals could be visible in this data The mass scales being investigated at the energy frontier are accessible in many channels at BaBar, both at tree and penguin levels, producing enhancements in rare decays and/or distortions in the pattern of CP violation and the unitarity triangle parameters We are already looking for those that seem most promising a priori However... Discoveries at the Tevatron or LHC (or beyond) could point in new directions, even many years from now Legacy HEP Data - Dubois-Felsmann 9

10 The walls that are ahead At some point we will no longer have sufficient BaBar-specific expertise to be able to continue porting the analysis and simulation environment (code and data) to new hardware or new, maintainable/secure operating systems At some other point it will become difficult to maintain a functioning collaboration that can provide mentorship, advice, review, and publication approval for new analyses Must we abandon the data set when these things happen? Legacy HEP Data - Dubois-Felsmann 10

11 Strategy Push the wall away as far as we can Do R&D on how to sustainably support analysis and simulation Think about what would be required to lower the cost of entry for new analysts Legacy HEP Data - Dubois-Felsmann 11

12 Pushing the computing wall away Make sure that our analysis and simulation environment is as easy to support as possible Port forward aggressively to buy time (SL5, gcc 4.x, ROOT 5.20+) Slice away things we don t need any more (e.g., online code and its external dependencies) Replace unique tools with variants on common ones Capture documentation while the experts are still around Prepare for loss of physical resources Simplify the computing infrastructure (e.g., servers) needed for analysis, and redesign for fewer simultaneous users Legacy HEP Data - Dubois-Felsmann 12

13 Technical alternatives What s beyond the walls? What s always happened before: SuperB or something similar is under way, and this doesn t matter Or: We archive a snapshot of everything (in what format?), along with some reasonable attempt at documentation; someone funds the occasional recopying of the snapshot (~2PB?) but nothing beyond that; and if/when a really compelling motivation to look at the data arises, somebody pays for the major investment in FTEs needed to revive data analysis and simulation (and whatever hardware is needed). Or: We try to devise a low-maintenance way to maintain access for the very long term Legacy HEP Data - Dubois-Felsmann 13

14 R&D for the longer term We are looking at two basic alternatives Preserve a streamlined version of the existing environment, with its full analytical power Most likely this means, at some point in the future, freezing the environment and virtualizing it Converts the problem... from BaBar-specific work needed to move the environment forward from generation to generation... to an industry-wide problem of how to run virtualized environments from 2012 on 2030 hardware and operating systems Adopt a new, simpler data model and something like a tabledriven parametric simulation, using new HEP-wide tools This would likely lose some of the analytical power of the data Legacy HEP Data - Dubois-Felsmann 14

15 Community-wide efforts 1) A survey 2) A workshop series being organized, chaired by Cristinel Diaconu of IN2P3 and H1, on Data Persistency and Long Term Analysis in HEP

16 PARSE.insight / CERN data preservation survey Legacy HEP Data - Dubois-Felsmann 16

17 Data Persistency and Long Term Analysis in HEP Several large HEP experiments have large/full data sets and head towards final analyses HERA: H1,ZEUS,HERMES (end of collisions July 2007) Tevatron: CDF,D0 (end of collisions in 1-2 years) PEP II: BaBar (end of collisions April 2008) KEKB: Belle ( upgrade) The complexity of data sets is similar, driven by technological choices (trigger) convoluted with the complexity of the measured final states The conservation of the data and possibility of long term analysis are important issues Legacy HEP Data - Dubois-Felsmann 17 (Slides from the project steering group)

18 An Inter-Experiment Study Group Common enterprise of experiments and associated computing centres Positive reply from H1/ZEUS/CDF/D0/Babar/Belle and computing centers from DESY/SLAC/KEK/FNAL More contacts ongoing (CERN-IT, LEP,LHC), more participation welcome (IHEP, Cornell,...) Contacts with DOE, UE-FP7(PARSE) Legacy HEP Data - Dubois-Felsmann 18 (Slides from the project steering group)

19 An Inter-Experiment Study Group Preparatory discussions started (first phone discussion end of September): First (quick) exchange on data status and typology Converged on a set of objectives (At least) two aspects are crucial for the subject: Technical and organizational matters Intellectual property and open access matters Legacy HEP Data - Dubois-Felsmann 19 (Slides from the project steering group)

20 The Objectives Review and document the physics objectives of the data persistency in HEP. Exchange information concerning the analysis model: abstraction, software, documentation etc. and identify coherence points Address the hardware and software persistency status Get some external (non-hep, non-research etc.) input Review possible funding programs and other related international initiatives Converge to a common set of specifications in a document (blue print) that will constitute the basis for future collaborations Legacy HEP Data - Dubois-Felsmann 20 (Slides from the project steering group)

21 Steps Towards Long Term Analysis in HEP 1. Experiment-wise preparation/organisation for proper conservation of the data/knowledge Proper planning and (new) projects required (hot topic!) 2. Common framework for similar experiments Similar experiments converge on data release policy/format Enable (further) combined analyses 3. Open access to expert community Require sufficient knowledge encapsulation, is a natural and necessary result of the previous steps. 4. Open access to a wider community: educational projects, outreach etc. Steps 2-4 imply a policy for open access to the HEP data (status?) Legacy HEP Data - Dubois-Felsmann 21 (Slides from the project steering group)

22 Workshops of the Study Group First Workshop organized in DESY (January 26-28, 2009) Preparatory remote discussion will be organized in November 2009, extra-participation envisaged, contacts ongoing Exploratory, but with clear agenda/objectives Initiate further work/working groups on technical and organizational matters Define the editorial procedure for the document (blue print) for data preservation and long term analysis in HEP. A second workshop mid-2009 (SLAC) Objective: prepare a status report/document for LP2009 A path for further collaborations/meetings is expected Legacy HEP Data - Dubois-Felsmann 22 (Slides from the project steering group)