NHSC/PACS Web Tutorials Running PACS photometer pipelines. PACS 201 (for Hipe 5.0) Level 0 to Level 1 processing: From raw to calibrated data cubes

Transcription

1 NHSC/PACS Web Tutorials Running PACS photometer pipelines PACS 201 (for Hipe 5.0) Level 0 to Level 1 processing: From raw to calibrated data cubes Prepared by Nicolas Billot and Roberta Paladini August page 1

2 IntroducCon NHSC PACS This tutorial presents the main steps of the standard pipeline stargng from the raw data cube (Level 0) to the calibrated data cube (Level 1). It also provides numerous break points to interacgvely check the intermediary results of the pipeline. DocumentaCon on PACS Photometry standard data processing: PACS Pipeline Reference Manual Chapter 3 Level 1 products from the archive can be used directly to build maps (see tutorial PACS 202). However it might be necessary to reprocess the raw data to Level 1 to use updated calibragon files or pipelines, and interacgvely improve the deglitching process for instance. Yet the Level 0 to 1 standard processing is already close to opgmum, and one should keep to a minimum modificagons to the pipeline. DefiniCon of data processing Levels: PACS Data ReducGon Guide SecGon page 2

3 Pre requisites: 1. HIPE is running (version 5.0), 2. You have completed the following tutorials: PACS 101 to 104: How to use these tutorials, load data and scripts in your HIPE session. 3. You know the idengficagon number of your observagon: OBSID In parallel to this tutorial, you can execute standard ipipe scripts available in HIPE (see tutorial PACS 102) or custom scripts available on the NHSC website hwps://nhscsci.ipac.caltech.edu/sc/index.php/pacs/dataprocessing (see Step 1 of this tutorial). - page 3

4 Level 0 to 1: Overview NHSC PACS Step 1 Load the script Step 2 Retrieve the Observa3onContext Step 3 Extract the frames and relevant informacon Step 4 From Level 0 to 0.5 Step 5 From Level 0.5 to 1 - page 4

5 Step 1 Load the relevant data reduccon script - page 5

6 What script should I use? ipipe scripts: ipipe (interacgve pipeline) scripts are part of HIPE. They are delivered and updated with new material with each release of HIPE. They are to be used as templates to reduce your data. You might want to SAVE AS and modify these scripts to suit the specifics of your own set of observagons (need for concatenagng scan/cross scan observagons, or modify deglitching parameters). Custom scripts: The NHSC provides examples of customized scripts based on ipipe scripts. They can be downloaded from our webpage hwps://nhscsci.ipac.caltech.edu/sc/index.php/pacs/dataprocessing We encourage you to write your own custom scripts stargng from the ipipe scripts and incorporagng processing steps specific to your own data set. Load your script following tutorials PACS page 6

7 Step 2 Retrieve the ObservaConContext - page 7

8 ObservaConContext: HIPE variable holding engre Herschel observagons. It contains the raw data cube as well as the data processed with the standard pipeline up to Level 0.5, 1 and 2. It also contains auxiliary informagon such as the telescope poingng, and all the calibragon files necessary to reprocess the data. DefiniCon of ObservaCon Content: Herschel Data Analysis Guide SecGon page 8

9 The ObservaConContext can be loaded in HIPE from the Herschel Science Archive (HSA) HIPE> obsid = # public observation of Delta Draconis HIPE> obs = getobservation(obsid, usehsa = True) or from a local pool on your hard drive disk assuming it was previously saved to a local store (see the usage of the getpacsdata.py script in tutorial PACS 103) HIPE> obs = getobservation(obsid, poollocation = /where/you/saved/it/ ) - page 9

10 Check # 1: Inspect the ObservationContext HIPE> print obs metadata PoinGng, House Keeping, etc. CalibraGon files are included in the ObservaGonContext Different levels of data processing generated by the pipeline and served by the archive - page 10

11 Check # 1: Inspect the ObservationContext Double click obs in the variables window to open the ObservaGon Viewer in the Editor window (creagon of a new tab) - page 11

12 Check # 1: Inspect the ObservationContext Click Meta Data and scroll to explore some informagon about your observagon - page 12

13 Check # 1: Inspect the ObservationContext Single Click obs/level2/ HPPPMAPB/0/image to view the image of the level2 product MAP Blue - page 13

14 Check # 1: Inspect the ObservationContext Click the zoom buwon to fit the image to the screen Double Click obs/level2/ HPPPMAPB/0/image to open the image with the Image Viewer (creagon of a new tab) - page 14

15 Step 3 Extract the frames and related informacon from the ObservaConContext and HIPE - page 15

16 Extract the Level 0 data cube (frames) from the ObservaConContext Level 0: raw data cube Syntax for Blue/Red array HIPE> frames = obs.level0.refs["hppavgb"].product.refs[0].product HPPAVGB/R: Herschel PACS Photometer AVGerage Blue/Red This is the signal downlinked from the spacecra^ a^er on board averaging HIPE> frames = obs.level0.refs["hppavgr"].product.refs[0].product - page 16

17 Check # 2: Inspect the frames HIPE> print frames.class herschel.pacs.signal.frames HIPE> print frames metadata Actual datasets stored in the frames: Signal, Status and Mask - page 17

18 Check # 3: Inspect the frames - meta data HIPE> print frames.meta Access individual meta data parameter values HIPE> print frames.meta["mapscannumlegs"].value 8 Number of scan legs - page 18

19 Check # 4: Inspect the frames - Signal HIPE> print frames.signal.class herschel.ia.numeric.double3d HIPE> print frames.signal.dimensions array([16, 32, 2890], int) frames.signal contains the raw data cube The cube dimensions depends on the array and the observagon duragon: 16x32 (32x64) pixels for the Red (Blue) array Third dimension: number of readouts - page 19

20 Check # 4: Inspect the frames - Signal 32 (16) pixels 64 (32) pixels Readout sequence 10 Hz PACS Prime 5 Hz Pacs PMode Most sources will not be visible on single frames due to intrinsic offset dispersion. Signal dispersion ~ ADU - page 20

21 Check # 5: Inspect the frames - Signal - Spatial Indicates that the third dimension is the readout sequence NHSC PACS HIPE> Display(frames.signal, depthaxis=2) Double Click on the image and select Edit cut levels Then click Median Filter to adapt the color scale - page 21

22 Check # 5: Inspect the frames - Signal - Spatial Indicates that the third dimension is the readout sequence NHSC PACS HIPE> Display(frames.signal, depthaxis=2) Complete documentacon on Data Display: Herschel Data Analysis Guide Chapter 2 Scroll to explore the data cube - page 22

23 Check # 6: Inspect the frames - Signal - Temporal HIPE> PlotXY(frames.signal[8,8,:]) Temporal evolugon of pixel (8,8) CalibraGon Block: prior to any observagons, 2 internal sources are observed with the internal chopper for calibragon purposes Science Block Complete documentacon on Data Plo]ng: Herschel Data Analysis Guide Chapter 3 - page 23

24 Check # 6: Inspect the frames - Signal - Temporal HIPE> PlotXY(frames.signal[8,8,:]) Temporal evolugon of pixel (8,8) Zoom onto the CalibraGon Block Chopping pawern on internal sources 1 & 2 Complete documentacon on Data Plo]ng: Herschel Data Analysis Guide Chapter 3 - page 24

25 Check # 6: Inspect the frames - Signal - Temporal HIPE> PlotXY(frames.signal[8,8,:]) Temporal evolugon of pixel (8,8) Zoom onto the Science Block The spike is likely a glitch Complete documentacon on Data Plo]ng: Herschel Data Analysis Guide Chapter 3 - page 25

26 Check # 6: Inspect the frames - Signal - Temporal HIPE> PlotXY(frames.signal[8,8,:]) Temporal evolugon of pixel (8,8) Zoom onto the spike The spike is asymmetric and affects only 2 readouts: It is a glitch Complete documentacon on Data Plo]ng: Herschel Data Analysis Guide Chapter 3 - page 26

27 Check # 7: Inspect the frames - Status & Mask NHSC PACS HIPE> print frames.status HIPE> print frames.mask - page 27

28 Extract the auxiliary data from the ObservaConContext HIPE> pp = obs.auxiliary.pointing HIPE> phothk = obs.level0.refs["hpphk"].product.refs[0].product["hpphks"] HIPE> oep = obs.auxiliary.orbitephemeris pp: poingng product phothk: photometer HouseKeeping oep: orbitephemeris product We recommend not to modify these variables, they contain crigcal informagon required in subsequent processing modules. - page 28

29 Extract the calibracon files (caltree) from the ObservaConContext The CalibraGon Tree caltree contains all the files necessary to process your data HIPE> caltree = getcaltree( time = frames.startdate ) Some calibragon files changed with Gme, e.g. the SIAM or poingng calibragon file, so it is necessary to provide the date of observagon for retrieving the appropriate calfiles Complete DocumentaCon on CalibraCon Files: h_p://herschel.esac.esa.int/twiki/bin/view/public/ PacsCalibraConWeb#PACS_calibraCon_file_versions - page 29

30 Check # 8: Inspect the caltree Double click caltree in the variables window Explore the CalibraGon Tree with the GUI This is the flakield in the blue filter - page 30

31 Check # 8: Inspect the caltree HIPE> print caltree 13 th version of the caltree Flight Model Product specific caltree - page 31

32 Check # 8: Inspect the caltree HIPE> print caltree.photometer Photometer specific calibragon products - page 32

33 Check # 8: Inspect the caltree Access informacon in the caltree from the command line HIPE> print caltree.photometer.flatfield.blue["flatfield"].data.dimensions array([32, 64], int) HIPE> trans = caltree.photometer.filtertransmission.blue["transmission"].data HIPE> wav = caltree.photometer.filtertransmission.blue["wavelength"].data HIPE> PlotXY(wav, trans, xtitle = "Wavelength [micron]", ytitle= Transmission") Zoom on the feature - page 33

34 Step 4 From Level 0 to Level page 34

35 IdenCfy Blocks in the observacon and remove the CalibraCon Blocks HIPE> frames = findblocks(frames, caltree=caltree) HIPE> frames = detectcalibrationblock(frames) HIPE> frames = removecalblocks(frames) Before After HIPE> PlotXY(frames.signal[8,8,:]) The chopping pawern from the CalibraGon Block is gone. - page 35

36 Flag Bad Pixels and populate frames.mask HIPE> frames = photflagbadpixels(frames, caltree=caltree) OpGonal addigon of rogue pixels to the BADPIXEL mask HIPE> blue_badpix = caltree.photometer.badpixelmask.blue # get the mask HIPE> blue_badpix[2,30] = 1 # This flips the value of pixel(2,30) from False to True HIPE> frames.setmask("badpixels", blue_badpix) - page 36

37 Check # 9: BADPIXEL Mask HIPE> print frames.mask The BADPIXEL mask is created in frames.mask - page 37

38 Flag saturated pixels and populate frames.mask HIPE> frames = photflagsaturation(frames, caltree=caltree, hkdata=phothk) CL and ADC saturation checking Provide the photometer HouseKeeping to derive so^ saturagon limits that depend on the bolometer bias semng The PACS bolometer arrays are subject to hard saturagon of the warm electronics (ADC saturagon), as well as so^ saturagon from the cold electronics (CL saturagon) - page 38

39 Check # 10: SATURATION Masks HIPE> print frames.mask SATURATION masks were created in frames.mask - page 39

40 Convert the signal from digital units (ADU) into physical units (Volts) HIPE> frames = photconvdigit2volts(frames, caltree=caltree) Convert the chopper angle from digital units (ADU) into physical units (degrees) HIPE> frames = convertchopper2angle(frames, caltree=caltree) - page 40

41 Add an inical escmate of the bolometers noise level to the frames HIPE> frames = photaddnoiseperpixel(frames, method = "median", caltree=caltree) The NOISE dataset is created in the frames object The noise esgmate is used later in the deglitching process - page 41

42 Step 5 From Level 0.5 to Level 1 - page 42

43 Deglitching NHSC PACS The spagal deglitching algorithm, or Second Level Deglitching, is now (since bulk reprocessing 6.1) the default deglitching algorithm in the pipeline. It relies on spagal redundancy to detect outliers. The Second Level Deglitching works on a mapindex variable. The mapindex is populated with the signal contribugons from all detector pixels for each individual map pixels - page 43

44 Deglitching NHSC PACS Syntax for running the second level deglitching on frames Complete documentacon on Second Level Deglitching: hwp:// - page 44

45 Deglitching NHSC PACS Check # 11: Creation of 2 nd level glitchmask HIPE> print frames.mask 2 nd level glitchmask was created in frames.mask - page 45

46 Deglitching NHSC PACS MapIndexViewer: an interacgve second level deglitching HIPE> from herschel.pacs.spg.phot.gui.mapindexview import MapIndexViewer HIPE> MapIndexViewer(mi, frames) Image and mapindex Sigma clipping opgons/parameters - page 46

47 Add the poincng informacon to the frames HIPE> frames = photaddinstantpointing(frames, pp, caltree=caltree, orbitephem=oep) Provide the telescope poingng product and the orbit ephemeris that were extracted from the ObservaGonContext This module adds the spacecra^ poingng to the frames.status as coordinates and addigonal poingng informagon - page 47

48 Check # 12: focal plane coordinates HIPE> print frames.status photaddinstantpoingng has created all these variables in frames.status HIPE> print frames.status["raarray"].data.dimensions array([2586], int) - page 48 RA of the center of the focal plane

49 Check # 12: focal plane coordinates This command allows to compile a funcgon at the command line or within a script HIPE> execfile("/path/to/the/script/plotscanpath.py") HIPE> plotscanpath(frames) plotscanpath.py exploits the poingng informagon added by the module photaddinstantpoingng, namely RaArray and DecArray, to plot the trajectory of the center of the focal plane. - page 49

50 Add the poincng informacon to the frames HIPE> frames = photassignradec(frames, caltree=caltree) This modules assigns coordinates to individual pixels using distorgon informagon from the caltree - page 50

51 Check # 13: RA, Dec HIPE> print frames photassignradec has created the datasets Ra and Dec in the frames HIPE> print frames.ra.dimensions array([16, 32, 2586], int) - page 51

52 Apply the flat field correccon and convert the signal from Volt/pixel into Jy/pixel HIPE> frames = photrespflatfieldcorrection(frames, caltree = caltree) DocumentaCon on the photrespflaeieldcorreccon module: PACS Data ReducGon Guide SecGon PACS Pipeline Reference Manual SecGon You have reached Level 1 The frames are now calibrated and ready for the map making process (see tutorial PACS 202 & 401) - page 52

53 Save the frames before further processing HIPE> save("/my/directory/my_frames.save","frames") The frames are saved in the save format of HIPE The saved file can only be read back into HIPE HIPE> restore("/my/directory/my_frames.save ) - page 53

54 Save the frames before further processing HIPE> FitsArchive().save("/my/directory/my_frames.fits", frames) The frames are saved in standard fits format The saved file can be read back into HIPE or IDL HIPE> frames = FitsArchive().load("/my/directory/my_frames.fits ) This is the extension of the fits file - page 54

55 DocumentaCon NHSC PACS h_p://herschel.esac.esa.int/hcss doc 5.0/ h_p:// Most relevant documentagon for this tutorial: PACS Data ReducGon Guide PACS User s Reference Manual HCSS User s Reference Manual General documentagon necessary to manipulate the data: HIPE Owner s Guide Herschel Data Analysis Guide ScripGng and Data Mining - page 55

56 NHSC/PACS Web Tutorials Running PACS photometer pipelines Please contact the NHSC Helpdesk if you are having difficulges with this tutorial - page 56