LOFAR: Lessons Learnt Michiel van Haarlem van Weeren, Bonafede, Ferrari, Orrù, Pizzo, Shulevski, van der Tol, Macario Jason Hessels & Pulsar Team
LOFAR 40 stations in NL and 8 stations throughout Europe (DE(5), FR, SE, UK) Netherlands 24 core stations (2 HBA fields) and 16 remote stations (single HBA field) LBA: 10-80 MHz (96 antennas) HBA: 110-240 MHz (48 antennas) 48 (dual pol.) signal paths Bandwidth: 32 (48) MHz N beams > 8 Baselines up to 100 km in NL 2000 2010 International International Stations 96 LBA, 96 (4x4) HBA tiles and 96 dual pol. signal paths International LOFAR Telescope (ILT) will provide access to LOFAR Operations by ASTRON in collaboration with local staff @ international stations Key Science Projects (6 in total) to play large role in first years Open time 10 & 20% in first two years, t.b.d. after that PrepSKA WP2 Oxford - 28 March 2010 van Haarlem - LOFAR 2
Current Status (NL) Construction of stations started March 2009 Network + Phase 1 Central Processing Completed On-line CEP based on 3 IBM BlueGene/P racks 33 stations completed in 2009 and 2010 Phase 2 CEP complete by end of 2010 7 station to go in 2011 First LOFAR Station - CS 302 early May 2009 PrepSKA WP2 Oxford - 28 March 2010 van Haarlem - LOFAR 3
Developing the Core Area, Building Stations early July 2008 14 August 2009 late July 2008 24 August 2009 September 2008 October 2008 9 March 2010 PrepSKA WP2 Oxford - 28 March 2010 van Haarlem - LOFAR 4
23 May 2010 PrepSKA WP2 Oxford - 28 March 2010 van Haarlem - LOFAR 5
Chilbolton (UK) Potsdam (D) Tautenburg (D) Effelsberg (D) Nançay (F) LOFAR Stations Across Europe PrepSKA WP2 Oxford - 28 March 2010 van Haarlem - LOFAR 6 Unterweilenbach (D)
Software Pipelines Standard Imaging Pipeline Known Pulsar Pipeline Transient Detection Pipeline VHECR Pipeline PrepSKA WP2 Oxford - 28 March 2010 van Haarlem - LOFAR 7
Timeline of LOFAR Project First Plans System Requirements Review Preliminary Design Review Complete Initial Test Station Critical Design Reviews (subsystems) Construction CS-01 First Results from CS-01 Final System Critical Design Review Start Procurement Start Roll-Out 1998 2002 2003 2003 2005/6 2006 early 2007 April 2007 December 2007 Summer 2008 PrepSKA WP2 Oxford - 28 March 2010 van Haarlem - LOFAR 8
CS-01 (2006) 96 LBA antennas (30-80 MHz) 6 HBA tiles (120-240 MHz) Distributed over 4 station locations (48/16/16/16) Max. baselines: 500 m Prototype hardware Tested Procurement Process Technical/System Tests: data transport, synchronization, correlation, stability, RFI, station calibration, Platform for Software Pipelines PrepSKA WP2 Oxford - 28 March 2010 van Haarlem - LOFAR 9
Procurement Rounds European directives apply above threshold values (193k for public sector supply and service contracts, 4.845 M for public sector and utility works contracts) Construction of CS-01 (early 2006): Rehearsal Main Roll-Out: three Invitations to Tender: 1. Printed Circuit Boards, Cabinets and Station Hardware (HBA, cables) December 2007 2. Network Equipment Januari 2008 3. Central Processing, Storage Cluster and Local Control Computers April 2008 (selected two preferred suppliers) Lessons learnt CS-01 procurements allowed overall cost estimate within 2% Demands from funding sources sometimes contradict directives Public tenders complicate collaboration with industry smaller companies will find it difficult to invest Get professionals to run procurements and manage contracts Freeze design if uncertainties exist incorporate escapes Be careful in running procurements, errors can cause serious delays PrepSKA WP2 Oxford - 28 March 2010 van Haarlem - LOFAR 10
Supervising Mass Production Producibility before procurement otherwise costly initial production run Define test procedures early make them as short as possible. Watch out for changes in production large numbers, high cost Average 3% production loss PrepSKA WP2 Oxford - 28 March 2010 van Haarlem - LOFAR 11
Producing High Band Antenna Contract awarded to local company Set up dedicated assembly line in Exloo Production of electronics, cables, connectors not part of this contract Uncertainties in roll-out (weather) affected start of production Delays, extra transport and storage all cost money PrepSKA WP2 Oxford - 28 March 2010 van Haarlem - LOFAR 12
Station Construction Performed by contracted workforce under supervision of ASTRON roll-out team: Integration, testing and verification of electronics performed off-site Fully completed cabinet transported and installed Flattening and setting out antenna positions Cabling + placement of antennas Validation and initial testing Final testing, handover (including documentation) by ASTRON personnel PrepSKA WP2 Oxford - 28 March 2010 van Haarlem - LOFAR 13
Requirements Formalize requirements early Lessons (1) Tough to get clear requirements from users (boring, repetitive) Users prefer to comment on/criticize a design Manage changing requirements Long project, many temptations, lots of good ideas Engineering Balance between simulations and prototyping End-to-End simulation approaches complexity of the full system Prototyping is costly, but worthwhile (don t forget software: integration & test cycles) When to stop developing does it need to be perfect? have clear requirements, and how they are assessed Operations Get operations involved early on (operators, maintenance) PrepSKA WP2 Oxford - 28 March 2010 van Haarlem - LOFAR 14
Lessons (2) Software Software deserves a design process similar to all the hardware Get people working together in teams Write only the software you need (PVSS for monitoring and control) Use industry standards where possible Hire professional software developers (no PhDs and Postdocs) Roll Out Define standard transport sizes (truck, container, plane) for the equipment Release one software build for the roll out Planning, logistics and weather are unpredictable All parts should have unique barcode/version number configuration management, maintenance Commissioning Dedicated multi-disciplinary commissioning teams are a good idea (busy weeks PrepSKA WP2 Oxford - 28 March 2010 van Haarlem - LOFAR 15
Summary Roll-Out complete in 2011 40 stations in NL + 8 in DE/FR/SE/UK Hard work needed to complete software pipelines Eagerly awaiting first scientific results! Lots of valuable experience for the SKA McKean (ASTRON) Griessmeier & LOFAR Planet WG PrepSKA WP2 Oxford - 28 March 2010 van Haarlem - LOFAR 16