NHSC/PACS Web Tutorials Running PACS photometer pipelines PACS-201

Transcription

1 NHSC/PACS Web Tutorials Running PACS photometer pipelines PACS 201 Level 0 to Level 1 processing: From raw to calibrated data cubes Prepared by Nicolas Billot February page 1

2 Introduc=on NHSC PACS This tutorial presents the main steps of the standard pipeline starhng from the raw data cube (Level 0) to the calibrated data cube (Level 1). It also provides numerous break points to interachvely check the intermediary results of the pipeline. Documenta=on on PACS Photometry standard data processing: PACS Pipeline Reference Manual Chapter 3 Level 1 products from the archive can be used directly to build maps (see tutorial PACS 202). However it might be necessary to reprocess the raw data to Level 1 to use updated calibrahon files or pipelines, and interachvely improve the deglitching process for instance. Yet the Level 0 to 1 standard processing is already close to ophmum, and one should keep to a minimum modificahons to the pipeline. Defini=on of data processing Levels: PACS Data ReducHon Guide SecHon page 2

3 Pre requisites: 1. HIPE is running (version 5.1 or later), 2. You have completed the following tutorials: PACS 101 to 104: How to use these tutorials, load data and scripts in your HIPE session. 3. You know the idenhficahon number of your observahon: OBSID In parallel to this tutorial, you can execute standard ipipe scripts available in HIPE (see tutorial PACS 102) or custom scripts available on the NHSC website hxps://nhscsci.ipac.caltech.edu/sc/index.php/pacs/dataprocessing (see Step 1 of this tutorial). - page 3

4 Level 0 to 1: Overview NHSC PACS Step 1 Load the script Step 2 Retrieve the Observa3onContext Step 3 Extract the frames and relevant informa=on Step 4 From Level 0 to 0.5 Step 5 From Level 0.5 to 1 - page 4

5 Step 1 Load the relevant data reduc=on script - page 5

6 What script should I use? ipipe scripts: ipipe (interachve pipeline) scripts are part of HIPE. They are delivered and updated with new material with each release of HIPE. They are to be used as templates to reduce your data. You might want to SAVE AS and modify these scripts to suit the specifics of your own set of observahons (need for concatenahng scan/cross scan observahons, or modify deglitching parameters). Custom scripts: The NHSC provides examples of customized scripts based on ipipe scripts. They can be downloaded from our webpage hxps://nhscsci.ipac.caltech.edu/sc/index.php/pacs/dataprocessing We encourage you to write your own custom scripts starhng from the ipipe scripts and incorporahng processing steps specific to your own data set. Load your script following tutorials PACS page 6

7 Step 2 Retrieve the Observa=onContext - page 7

8 Observa=onContext: HIPE variable holding enhre Herschel observahons. It contains the raw data cube as well as the data processed with the standard pipeline up to Level 0.5, 1 and 2. It also contains auxiliary informahon such as the telescope poinhng, and all the calibrahon files necessary to reprocess the data. Defini=on of Observa=on Content: Herschel Data Analysis Guide SecHon page 8

9 The Observa=onContext can be loaded in HIPE from the Herschel Science Archive (HSA) HIPE> obsid = # public observation of Delta Draconis HIPE> obs = getobservation(obsid, usehsa = True) or from a local pool on your hard drive disk assuming it was previously saved to a local store (see the usage of the getpacsdata.py script in tutorial PACS 103) HIPE> obs = getobservation(obsid, poollocation = /where/you/saved/it/ ) - page 9

10 Check # 1: Inspect the ObservationContext HIPE> print obs metadata PoinHng, House Keeping, etc. CalibraHon files are included in the ObservaHonContext Different levels of data processing generated by the pipeline and served by the archive - page 10

11 Check # 1: Inspect the ObservationContext Double click obs in the variables window to open the ObservaHon Viewer in the Editor window (creahon of a new tab) - page 11

12 Check # 1: Inspect the ObservationContext Click Meta Data and scroll to explore some informahon about your observahon - page 12

13 Check # 1: Inspect the ObservationContext Single Click obs/level2/ HPPPMAPB/0/image to view the image of the level2 product MAP Blue - page 13

14 Check # 1: Inspect the ObservationContext Click the zoom buxon to fit the image to the screen Double Click obs/level2/ HPPPMAPB/0/image to open the image with the Image Viewer (creahon of a new tab) - page 14

15 Step 3 Extract the frames and related informa=on from the Observa=onContext and HIPE - page 15

16 Extract the Level 0 data cube (frames) from the Observa=onContext Level 0: raw data cube Syntax for Blue/Red array HIPE> frames = obs.level0.refs["hppavgb"].product.refs[0].product HPPAVGB/R: Herschel PACS Photometer AVGerage Blue/Red This is the signal downlinked from the spacecra_ a_er on board averaging HIPE> frames = obs.level0.refs["hppavgr"].product.refs[0].product - page 16

17 Check # 2: Inspect the frames HIPE> print frames.class herschel.pacs.signal.frames HIPE> print frames metadata Actual datasets stored in the frames: Signal, Status and Mask - page 17

18 Check # 3: Inspect the frames - meta data HIPE> print frames.meta Access individual meta data parameter values HIPE> print frames.meta["mapscannumlegs"].value 8 Number of scan legs - page 18

19 Check # 4: Inspect the frames - Signal HIPE> print frames.signal.class herschel.ia.numeric.double3d HIPE> print frames.signal.dimensions array([16, 32, 2890], int) frames.signal contains the raw data cube The cube dimensions depends on the array and the observahon durahon: 16x32 (32x64) pixels for the Red (Blue) array Third dimension: number of readouts - page 19

20 Check # 4: Inspect the frames - Signal 32 (16) pixels 64 (32) pixels Readout sequence 10 Hz PACS Prime 5 Hz Pacs PMode Most sources will not be visible on single frames due to intrinsic offset dispersion. Signal dispersion ~ ADU - page 20

21 Check # 5: Inspect the frames - Signal - Spatial Indicates that the third dimension is the readout sequence NHSC PACS HIPE> Display(frames.signal, depthaxis=2) Double Click on the image and select Edit cut levels Then click Median Filter to adapt the color scale - page 21

22 Check # 5: Inspect the frames - Signal - Spatial Indicates that the third dimension is the readout sequence NHSC PACS HIPE> Display(frames.signal, depthaxis=2) Complete documenta=on on Data Display: Herschel Data Analysis Guide Chapter 2 Scroll to explore the data cube - page 22

23 Check # 6: Inspect the frames - Signal - Temporal HIPE> PlotXY(frames.signal[8,8,:]) Temporal evoluhon of pixel (8,8) CalibraHon Block: prior to any observahons, 2 internal sources are observed with the internal chopper for calibrahon purposes Science Block Complete documenta=on on Data Plo]ng: Herschel Data Analysis Guide Chapter 3 - page 23

24 Check # 6: Inspect the frames - Signal - Temporal HIPE> PlotXY(frames.signal[8,8,:]) Temporal evoluhon of pixel (8,8) Zoom onto the CalibraHon Block Chopping paxern on internal sources 1 & 2 Complete documenta=on on Data Plo]ng: Herschel Data Analysis Guide Chapter 3 - page 24

25 Check # 6: Inspect the frames - Signal - Temporal HIPE> PlotXY(frames.signal[8,8,:]) Temporal evoluhon of pixel (8,8) Zoom onto the Science Block The spike is likely a glitch Complete documenta=on on Data Plo]ng: Herschel Data Analysis Guide Chapter 3 - page 25

26 Check # 6: Inspect the frames - Signal - Temporal HIPE> PlotXY(frames.signal[8,8,:]) Temporal evoluhon of pixel (8,8) Zoom onto the spike The spike is asymmetric and affects only 2 readouts: It is a glitch Complete documenta=on on Data Plo]ng: Herschel Data Analysis Guide Chapter 3 - page 26

27 Check # 7: Inspect the frames - Status & Mask NHSC PACS HIPE> print frames.status HIPE> print frames.mask - page 27

28 Extract the auxiliary data from the Observa=onContext HIPE> pp = obs.auxiliary.pointing HIPE> phothk = obs.level0.refs["hpphk"].product.refs[0].product["hpphks"] HIPE> oep = obs.auxiliary.orbitephemeris pp: poinhng product phothk: photometer HouseKeeping oep: orbitephemeris product We recommend not to modify these variables, they contain crihcal informahon required in subsequent processing modules. - page 28

29 Extract the calibra=on files (caltree) from the Observa=onContext The CalibraHon Tree caltree contains all the files necessary to process your data HIPE> caltree = getcaltree( time = frames.startdate ) Some calibrahon files changed with Hme, e.g. the SIAM or poinhng calibrahon file, so it is necessary to provide the date of observahon for retrieving the appropriate calfiles Complete documenta=on on the Calibra=on Framework: hxp:// The_PACS_CalibraHon_Framework_ _issue_0.10.pdf - page 29

30 Check # 8: Inspect the caltree Double click caltree in the variables window Explore the CalibraHon Tree with the GUI This is the flalield in the blue filter - page 30

31 Check # 8: Inspect the caltree HIPE> print caltree 13 th version of the caltree Flight Model Product specific caltree - page 31

32 Check # 8: Inspect the caltree HIPE> print caltree.photometer Photometer specific calibrahon products - page 32

33 Check # 8: Inspect the caltree Access informa=on in the caltree from the command line HIPE> print caltree.photometer.flatfield.blue["flatfield"].data.dimensions array([32, 64], int) HIPE> trans = caltree.photometer.filtertransmission.blue["transmission"].data HIPE> wav = caltree.photometer.filtertransmission.blue["wavelength"].data HIPE> PlotXY(wav, trans, xtitle = "Wavelength [micron]", ytitle= Transmission") Zoom on the feature - page 33

34 Step 4 From Level 0 to Level page 34

35 Iden=fy Blocks in the observa=on and remove the Calibra=on Blocks HIPE> frames = findblocks(frames, caltree=caltree) HIPE> frames = detectcalibrationblock(frames) HIPE> frames = removecalblocks(frames) Before After HIPE> PlotXY(frames.signal[8,8,:]) The chopping paxern from the CalibraHon Block is gone. - page 35

36 Flag Bad Pixels and populate frames.mask HIPE> frames = photflagbadpixels(frames, caltree=caltree) OpHonal addihon of rogue pixels to the BADPIXEL mask HIPE> blue_badpix = caltree.photometer.badpixelmask.blue # get the mask HIPE> blue_badpix[2,30] = 1 # This flips the value of pixel(2,30) from False to True HIPE> frames.setmask("badpixels", blue_badpix) - page 36

37 Check # 9: BADPIXEL Mask HIPE> print frames.mask The BADPIXEL mask is created in frames.mask - page 37

38 Flag saturated pixels and populate frames.mask HIPE> frames = photflagsaturation(frames, caltree=caltree, hkdata=phothk) CL and ADC saturation checking Provide the photometer HouseKeeping to derive so_ saturahon limits that depend on the bolometer bias senng The PACS bolometer arrays are subject to hard saturahon of the warm electronics (ADC saturahon), as well as so_ saturahon from the cold electronics (CL saturahon) - page 38

39 Check # 10: SATURATION Masks HIPE> print frames.mask SATURATION masks were created in frames.mask - page 39

40 Convert the signal from digital units (ADU) into physical units (Volts) HIPE> frames = photconvdigit2volts(frames, caltree=caltree) Convert the chopper angle from digital units (ADU) into physical units (degrees) HIPE> frames = convertchopper2angle(frames, caltree=caltree) - page 40

41 Add an ini=al es=mate of the bolometers noise level to the frames HIPE> frames = photaddnoiseperpixel(frames, method = "median", caltree=caltree) The NOISE dataset is created in the frames object The noise eshmate is used later in the deglitching process - page 41

42 Step 5 From Level 0.5 to Level 1 - page 42

43 Deglitch the data and create a glitch mask in frames.mask The standard pipeline uses a Hme domain deglitching algorithm: MulH resoluhon Median Transform of individual pixel Hmelines (Stark et al. 1998) HIPE> frames = photmmtdeglitching(frames, incr_fact=2, mmt_mode='multiply', scales=3, nsigma=5, imagenoisearray = frames.noise[:,:,0]) The set of parameters given here seem to work well with most observahons Bright sources might however be flagged as glitches by this module (see tutorial PACS 202 & 402 for advice on disabling the GlitchMask on source) Complete documenta=on on MMTDeglitching: PACS Pipeline Reference Manual SecHon PACS User s Reference Manual SecHon page 43

44 Check # 11: Creation of MMT_Glitchmask Mask HIPE> print frames.mask MMT_Glitchmask was created in frames.mask - page 44

45 Check # 12: Inspect the MMT_Glitchmask Mask HIPE> from herschel.pacs.signal import MaskViewer HIPE> MaskViewer(frames) Import the MaskViewer module See tutorial PACS 202 to learn how to inspect masks with the MaskViewer - page 45

46 Add the poin=ng informa=on to the frames HIPE> frames = photaddinstantpointing(frames, pp, caltree=caltree, orbitephem=oep) Provide the telescope poinhng product and the orbit ephemeris that were extracted from the ObservaHonContext This module adds the spacecra_ poinhng to the frames.status as coordinates and addihonal poinhng informahon - page 46

47 Check # 13: focal plane coordinates HIPE> print frames.status photaddinstantpoinhng has created all these variables in frames.status HIPE> print frames.status["raarray"].data.dimensions array([2586], int) - page 47 RA of the center of the focal plane

48 Check # 13: focal plane coordinates This command allows to compile a funchon at the command line or within a script HIPE> execfile("/path/to/the/script/plotscanpath.py") HIPE> plotscanpath(frames) plotscanpath.py exploits the poinhng informahon added by the module photaddinstantpoinhng, namely RaArray and DecArray, to plot the trajectory of the center of the focal plane. - page 48

49 Add the poin=ng informa=on to the frames HIPE> frames = photassignradec(frames, caltree=caltree) This modules assigns coordinates to individual pixels using distorhon informahon from the caltree - page 49

50 Check # 14: RA, Dec HIPE> print frames photassignradec has created the datasets Ra and Dec in the frames HIPE> print frames.ra.dimensions array([16, 32, 2586], int) - page 50

51 Apply the flat field correc=on and convert the signal from Volt/pixel into Jy/pixel HIPE> frames = photrespflatfieldcorrection(frames, caltree = caltree) Documenta=on on the photrespflacieldcorrec=on module: PACS Data ReducHon Guide SecHon PACS Pipeline Reference Manual SecHon You have reach Level 1 The frames are now calibrated and ready for the map making process (see tutorial PACS 202 & 401) - page 51

52 Save the frames before further processing HIPE> save("/my/directory/my_frames.save","frames") The frames are saved in the save format of HIPE The saved file can only be read back into HIPE HIPE> restore("/my/directory/my_frames.save ) - page 52

53 Save the frames before further processing HIPE> FitsArchive().save("/my/directory/my_frames.fits", frames) The frames are saved in standard fits format The saved file can be read back into HIPE or IDL HIPE> frames = FitsArchive().load("/my/directory/my_frames.fits ) This is the extension of the fits file - page 53

54 Documenta=on NHSC PACS hdp://herschel.esac.esa.int/hcss doc 5.0/ hdp:// Most relevant documentahon for this tutorial: PACS Data ReducHon Guide PACS User s Reference Manual HCSS User s Reference Manual General documentahon necessary to manipulate the data: HIPE Owner s Guide Herschel Data Analysis Guide ScripHng and Data Mining - page 54

55 NHSC/PACS Web Tutorials Running PACS photometer pipelines Please contact the NHSC Helpdesk if you are having difficulhes with this tutorial - page 55