Netherlands Institute for Radio Astronomy LOFAR: From raw visibilities to calibrated data John McKean (ASTRON) [subbing in for Manu] ASTRON is part of the Netherlands Organisation for Scientific Research (NWO) 1
Overview AIM: Give a basic 30 min summary of the reduction processes used to calibrate LOFAR data. The LOFAR Cookbook Imaging Pipeline The raw data and data editing NDPPP Radio Frequency Interference (RFI) removal Calibration with Black Board Self-calibration (BBS & SageCal) 2
Overview The LOFAR Cookbook Imaging Pipeline The raw data and data editing NDPPP Radio Frequency Interference (RFI) removal Calibration with Black Board Self-calibration (BBS & SageCal) 3
LOFAR Imaging Cookbook Full details of the calibration process can be found in the LOFAR imaging cookbook. Please send any updates or suggestions to Roberto Pizzo. http://www.astron.nl/~mckean/lofar_cookbook.pdf 4
Overview The LOFAR Cookbook Imaging Pipeline The raw data and data editing NDPPP Radio Frequency Interference (RFI) removal Calibration with Black Board Self-calibration (BBS & SageCal) 5
Standard imaging pipeline 6
Overview The LOFAR Cookbook Imaging Pipeline The raw data and data editing NDPPP Radio Frequency Interference (RFI) removal Calibration with Black Board Self-calibration (BBS & SageCal) 7
Data volumes Like many new instruments, LOFAR will also investigate data handling management. Interferometric Data Data Vol = Ba * P * T * C * S * Be * (bytes/t + overhead) Ba = baselines = 2556 (for HBA Dual) or 1128 (for HBA Single). P = Polarisations = 4 (XX, YY, XY, YX). T = Time Samples = 21600 (for 6h observations and 1 s visibility averaging). C = Channels = 256 (or 64) S = Sub-bands = 244 Be = 1 bytes/sa + overhead = 8 + 0.2 Data Vol = 113 TBs 8
Data inspection General details 9
Inspection in CASA First have to correct for the Antenna-Mount type 10
Overview The LOFAR Cookbook Imaging Pipeline The raw data and data editing NDPPP Radio Frequency Interference (RFI) removal Calibration with Black Board Self-calibration (BBS & SageCal) 11
Default Pre-Processing Pipeline (NDPPP) NDPPP can be used to Flagging (automatic or manual) Averaging in time and/or frequency Phase shift to another phase centre Count flags and write counts into a table Combine (concat) sub-bands into a single measurement set De-mix and subtract the A-team sources Add stations to form a superstation (i.e. superterp) Filter out baselines and/or channels Is run by the Radio Observatory to carry out basic flagging, averaging and subtraction of bright A-team sources. Can be re-run by the user to carryout further tasks. 12
Basic operation and parsets A simple text parameter set (PARSET) file is used to operate NDPPP 13
De-mixing The de-mixing operation removes the contribution of the A-team to the visibilities (v. important for LBA). See van der Tol et al. (2007, IEEE TSP, 55, 4497) for the technical details. Before After 14
Overview The LOFAR Cookbook Imaging Pipeline The raw data and data editing NDPPP Radio Frequency Interference (RFI) removal Calibration with Black Board Self-calibration (BBS & SageCal) 15
Radio frequency interference Europe is a highly populated area - lots of radio frequency interference! LOFAR mitigates RFI by i) having a small time and frequency resolution (1s; 763 Hz). ii) having 40 db receiver units to stop saturation/spill over to other channels iii) having digital filters to remove signals at < 30 MHz, 80--110 MHz. 16
AOflagger Standard LOFAR RFI rejection using AOflagger (Offringa et al., 2010, MNRAS, 405, 155). Combination of statistics automatically fits a surface to the time-frequency plane to identify RFI, iteratively defines the threshold. Low level of false-positive RFI detections Struggles with very broad-band RFI. 17
Radio frequency interference (Offringa et al., 2013, A&A, 549, 10) RFI occupancy is low and day / night results are consistent. LBA: 1.8% HBA: 3.2% 18
Diagnostic plots 19
Overview The LOFAR Cookbook Imaging Pipeline The raw data and data editing NDPPP Radio Frequency Interference (RFI) removal Calibration with Black Board Self-calibration (BBS & SageCal) 20
Calibration: Basics Calibration challenges of wide field imaging: The visibility function is not dominated by a single source. A good model for the sky is needed to start (see MSSS). M51 field 21
Calibration: Basics Phase variations: The ionosphere can change over minutes and patches can extend over few degrees on the sky - need multiple solutions across the primary beam. Clock drifts will effect the phase (for non-core Stations only). Amplitude variations: The ionosphere will also change the gain. The beam is also changing, so the response to a source varies during the observation. -> Direction dependent gains must be determined 22
Calibration: Ionospheric Waves 23
Calibration: Faraday Rotation Differential Faraday Rotation: Ionospheric rotation measure (RM) varies from station to station and converts Stokes I visibilities to Stokes V. Need to carry out calibration using full stokes. 24
Calibration: Changing Gains Unlike typical dish-based interferometers (e.g. JVLA, WSRT) the gain (amplitude) of the visibilities are not (almost) constant. Due to the source moving through the beam, effectively the change in the projected area of the station. Need beam correction in the calibration. 25
Calibration: Changing Gains Unlike typical dish-based interferometers (e.g. JVLA, WSRT) the gain (amplitude) of the visibilities are not (almost) constant. Due to the source moving through the beam, effectively the change in the projected area of the station. Need beam correction in the calibration. 25
Calibration: The RIME RIME: The radio interferometer measurement equation is solved during the calibration process, Baseline based, non closing errors Gain amplitude and phase Errors due to elevation Opacity and path length variation Observed visibility for ant. i and j V obs ij = M ij B ij G ij D ij E ij P ij T ij V true ij true visibility for ant. i and j Bandpass response Instrumental polarization Change in paralactic angle Jones matrices only valid for solving in one direction, need multiple solutions to the RIME - Direction dependent effects. 26
BBS A software package has been developed for LOFAR - Black-Board Self Calibration (see Tutorials). Has the ability to: Carry out parallel processing of multiple sub-bands - global solve across frequency to increase the SNR of the solutions (not standalone). Can predict the observed visibility function for a given sky model [predict] Carry out direction dependent solutions (amplitudes / phases). [solve] Carry out a subtraction of model sources from the uv-data. [subtract] Carry out full Stokes calibration. [correct] 27
BBS Parset file Example, very simple BBS parset file that solves for the amplitude and gain solutions given a sky model. Single direction solution 50 28
Building the sky model Can provide a text file containing a list of sources/source components (delta function, Gaussians, Shaplets). Build using existing catalogues. Build using model images (e.g. at different frequency or exisiting data, i.e. selfcal) and Fourier transform into a model column of the MS Add to BBS parset file. 29
Model building tips Model resolution should be comparable to your observation resolution - think about your longest baseline. Include all the flux in the primary beam or bright sources far away - think about what sky each visibility is sampling. Simulate your model and image it. Make a large image of the uncalibrated data to spot far away, bright sources that need to be included in the model. 30
Inspect solutions Looking at your gain solutions and inspect the corrected data for obvious errors. 31
Inspect solutions Bad Amp. Spikes Good 32
Post calibration errors...ddes 33
Direction dependent gains: BBS There will always be direction dependent effects in LOFAR data due to the varying beam and ionosphere. BBS can be used given a beam model (determined from the combined anntenna/tile response). Step.solve.Model.DirectionalGain.Enable = T Step.solve.Model.Sources = [src1, src2, CygA] Step.solve.Model.Cache.Enable = T Step.solve.Model.Beam.Enable = T Step.solve.Correlation.Selection = CROSS Step.solve.Correlation.Type = [] Step.solve.Solve.Parms = ["DirectionalGain:*"] Computationally expensive, so can only use a few directions. For averaged data, bandwidth and time smearing effects are not corrected for. 34
Direction dependent gains: SAGECal Space Alternating Generalised Expectation Maximisation Calibration (Yatawatta et al. 2009, Kazemi et al. 2011). First run BBS to calculate Gains and Beam correction (on averaged data). Cluster sources in various directions. Solve for different directional gains with SAGECal (constraints v free parameters). Fast. 35
Imaging The aim of imaging is to determine an accurate surface brightness distribution (positions and flux-densities) of the sky. We need: i) w-projection because the 2-d approximation does not hold over wide fields of view ii) a-projection because the LOFAR beam is constantly changing. iii) Speeeeed! Limits the dynamic range of images, and allows for self-calibration. Simulations show flux-densities recovered at the 1% level. 36
Self-calibration and self-learning... Our calibration can be improved by carrying out self-calibration using the LOFAR data. Build up a better sky model. This needs an imager that correctly measures the sky brightness distribution so that the correct input model can be used... See next talk... Learning to calibrate LOFAR data will not come purely from the Talks presented here, or by reading the cookbook, but from experience... Have a go yourself!!! 37
Looking back at the pipeline... 38