MIT Kavl Insttute 1 All Broken Up Chandra X-Ray Center How to, & Why (or Why Not) Combne Data Davd Huenemoerder, John Davs, John Houck, & Mke Nowak May 31, 2011 The gratng programs carred out by Chandra typcally have long exposures, and gven operatonal constrants, they are usually dvded nto two or more dstnct exposures. Fewer than half of the targets have only a sngle observaton even after accountng for the calbraton montorng programs. For programs n whch long-term montorng of spectral varablty s not of nterest, t s desrable to be able to perform analyss on the data as f t were a sngle observaton. It may seem, at frst glance, that to do ths one should combne the observatonal data fles. Ths can certanly be convenent n reducng the number of fles to manage. However, t s not necessary to do such snce the same effects can be obtaned durng an analyss sesson. In some cases, there are advantages to preservng the dentty of the ndvdual observatons. Here we wll explore some of the ssues wth combnng data and some of the methods avalable for analyss of combned data. Gven that we have multple datasets whch conceptually make a sngle observaton, there are several requrements or preferences for an analyst: easy manpulaton of multple datasets, whether two or ten observatons, t should be easy to do analyss; robust analyss; methods and results should not depend, except statstcally, on the number of observatons; flexble analyss; ablty to easly work wth dfferent permutatons of combned data. Ths dscusson s relevant to X-ray spectral analyss n general; t s not partcular to hgh-resoluton gratng spectroscopy, but apples to low-resoluton (e.g, CCD non-dspersve) spectroscopy as well. 2 Defnton of Combne There are many dfferent data products generated for each Chandra observaton. By observaton, we wll refer to the data assocated wth a sngle Observaton Identfer (ObsID). 1 For a sngle observaton, we can express the spectral counts hstogram n terms of the source model and nstrumental response as follows: C m (h) = t s de S(E) A m (E) R m (E, h) + B sm (h) (1) where h refers to our output channel (a wavelength or energy bn), E the model wavelength or energy coordnate bn, and subscrpt m refers to a dffracton order n the gratng s data case, whch we wll assume mplctly henceforth and only consder a sngle order. The other terms are 1 Ths s usually a suffcent defnton. Sometmes there are further subdvsons, such as for Observaton Intervals, or the two frames of nterleaved mode. We mean the set of fles sorted by any such tme nterval or mode selecton. 1
C(h) s the spectrum, an hstogram of counts per coordnate bn ( PHA (Pulse Heght Analyzer) fle ). Ths counts spectrum ncludes whatever background or confusng source counts cannot be explctly removed (the B-term on the rght-hand-sde). t s s the exposure tme n [s] for the source (plus mplct background) spectrum. S(E) s the source spectrum, n unts of [photons/cm 2 /s/bn]; A(E) s the effectve area, or ARF (Auxlary Response Fle), whch has unts of [cm 2 counts/photon]; R(E, h) s the redstrbuton matrx, or RMF (Response Matrx Fle), whch encodes the nstrumental energy or wavelength resoluton t s untless, but not necessarly normalzed. For gratng data, ths encapsulates the Lne Spread Functon and aperture extracton effcency; B s (h) s the background contrbuton to the source spectrum s extracton regon. Ths term need not be a lteral background, but s whatever value s deemed background that s, unnterestng for the analyss, but a nusance term whch needs to be ncluded. It could be derved from the same observaton as C, a dfferent one, or from models. It s typcally obtaned from some other regon or exposure and then scaled to the source s regon and exposure. Whatever t s, t s ether not to be convolved wth the nstrumental response, or already has been. Ths contans all scalng factors (exposure tmes and geometrc terms). Ths wll be dscussed n more detal below. Equaton 1 s the famlar forward-foldng expresson; snce R s typcally of a non-dagonal form, ths ntegral cannot be nverted and one must rely on teratve technques. 2 2.1 Multple Datasets Now we can see that a reasonable hgh-level defnton of a dataset s those quanttes from Equaton 1, C(h), A(E), R(E, h), and B s (h). In general, these are partcular and unque to each observaton, and consderng dfferent dffracton orders or extracton regons, there could be several sets per observaton. To consder multple datasets approprate for combned analyss, we ntroduce the subscrpt,, to denote each nstance of a lke dataset to be consdered for combnaton (the order subscrpt, m, has been suppressed for clarty): C (h) = t s de S(E) A (E) R (E, h) + B s (h) (2) We can then defne our combned data as a summaton, C (h) = de S(E) [t s A (E)R (E, h)] + B s (h) (3) We here assume that all grds (h, E) are commensurate, otherwse the summatons are not vald. 3 (In general, we mght also consder a weghtng factor, w to be appled to each spectrum.) Durng model optmzaton, a statstc s defned n the usual way from the summed counts and the summed model counts, usng assocated uncertantes propagated durng the summatons of C and B s. When data have been combned, we wll have more counts per channel, h; ths s advantageous f we requre Gaussan statstcs and such s not acheved for the ndvdual datasets. We wll also have a sngle counts spectrum to vsualze, and so can potentally see more features appear above the nose vsualzaton of the combned data s especally mportant for gudng analyss and presentaton of results. 2 We are also not consderng any tme-dependent effects n ether the source, response, or background! The assumpton s that these quanttes pertan to an nterval for whch tme-nvarance s vald. 3 Grds can be defned when data are produced. Chandra standard processng and CIAO use standardzed spectral grds; analyss tools are avalable for regrddng or re-extracton of products. 2
2.2 Combned vs. Jont Analyss Note that analyss of combned data as defned above s dfferent from the common practce of fttng and modelng jontly. Jont analyss means that multple datasets are modeled smultaneously, but ther counts arrays are not summed, nor are the correspondng model counts data summed. Ths s approprate and necessary, for example, when fttng data from dfferent nstruments (low resoluton and hgh resoluton) or data from dfferent observatores (e.g., Chandra gratngs wth Suzaku low-resoluton spectra). Jont analyss can also be performed on observatons whch could be summed f each dataset s n the Gaussan regme or f the approprate statstc s used, then combned only for vsualzaton. 3 Practcal Methods Gven multple datasets of the same object wth the same nstrument, how can the data be combned? 3.1 Mergng Event Data Frst, one should never-ever consder mergng the low-level products ( Level 1, event fles and assocated fles). Mergng at ths level would produce a sngle set of fles for processng to a sngle set of products, but there are ambgutes and nconsstences whch can make such dffcult or error-prone operatonally, or just slently produce nvald results. Just don t do t. 3.2 Dynamc Method Use an analyss system whch supports dynamc combnaton of datasets, n other words, one whch mplements Equaton 3 n memory. Ths means that you do not need to produce any more fles than the standard data products for spectral analyss (PHA, ARF, and RMF). Nor do you need to produce fles n dfferent combnatons (e.g., to manage possble varablty, systematcs, or grddng ssues). The ndvdual datasets can be ncluded or excluded at wll n the analyss sesson, and each unque response, background, and all ancllary data (exposures, backscales quanttes typcally read from fle headers) are handled robustly and automatcally durng the optmzaton. 3.3 Fle-based Methods If dynamc data combnaton s not supported by your favorte analyss package, t s possble to sum PHA and ARF fles to produce new such fles. For Chandra gratng spectra, the RMF (whch encodes the calbraton of the lne spread functon) depends on gratng and order, but rarely on observaton (n fact, the gratng RMFs are only calbrated and produced for on-axs observatons). The only extracton-dependence s on the cross-dsperson regon wdth (the tg d coordnate specfed by tgextract). Hence, one can typcally use one set of RMFs (one per order and gratng type) and then sum the PHA and strongly observaton-dependent ARF fles. We can put Equaton 3 back nto the standard form of Equaton 1 f we defne a mean exposure tme and some tme-weghted quanttes: C(h) = T s de S(E) R(E, h) + B s (h) (4) where C(h) = C (h) s our summed source plus background counts spectrum, T s = 1 N N =1 t s s the mean source observatons exposure tme; R(E, h) = 1 T s t sa (E)R (E, h) s an exposure-weghted response, and 3
B s (h) = B s(h) s our weghted, summed, scaled background spectrum. Furthermore, f our redstrbuton matrx s a constant for the datasets to be summed (as s typcally the case for the postve and negatve order pars of each of HEG or MEG gratngs, or for LETGS correspondng orders), we can take R(E, h) out of the weghted mean and have C(h) = T s de S(E) A(E) R 0 (E, h) + B s (h) (5) wth A(E) = 1 T s t sa (E) beng an exposure-weghted ARF. What we ultmately requre are thus C and B s, our summed source and background spectra, and A, the exposure-weghted ARF. When these are made, care must be taken to nclude the above defntons of the mean exposure and the exposure-weghted backscale (see below) factors n the fles headers. Gven such, analyss s then the same as for any sngle dataset. 3.4 The Background Term Our generalzed background term can be treated n a smlar fashon as the source spectrum, though gnorng the nstrumental responses, and consderng only nstrumental space that s, the detector channel h, and not energy or wavelength. Here we wll only consder the nosy knd of background those unformly spatally dstrbuted, random, tme-nvarant counts from nternal (electronc) sources or from the sky. Such are typcally extracted n regons adjacent to the source spectrum from the same exposure, or from separate exposures. (They may, however, have a characterstc spectral shape.) Other knds of background are left as exercses for the reader (but they may be able to be expressed n the smplfed forms descrbed here). Consder extractng a background spectrum, B (h) from the th observaton wth exposure tme t b, from the regon Ω b (h), wth an underlyng (and constant) background rate of r b [counts area 1 s 1 bn 1 ]. The area, Ω(h), s a generalzed area; for a dspersed spectrum, t may be the angular wdth of the extracton regon, but for an magng spectrum, t could be a number of pxels or a sky area. We wll call t area, and t s related to the FITS BACKSCAL keyword quantty. We wll also consder that Ω may be a functon of h, whch s possble for a non-unform dspersed spectrum background regon. (Snce t eventually occurs n a rato wth another such term, the unts cancel, and possbly dependence on h.) Our background spectrum can thus be expressed as B (h) = t b r b Ω b (h) (6) As before, we can now defne some mean quanttes: a mean exposure tme, and an exposure-weghted area to wrte: B(h) = B (h) = r b Ω b (h) T b (7) where T b = 1 N N =1 t b s the mean background observatons exposure tme; Ω b (h) = 1 T b t bω b (h) s an exposure-weghted area. 4 Together Agan We can now fnsh our assembly of multple datasets from fles by defnng the background term n Equatons 4 and 5. The source regons also have assocated extracton areas, Ω s (h) and mean, Ω s (h). We can therefore express our background contrbuton to the source regon as B s (h) = B(h) T s Ω s (h) T b Ω b (h) (8) 4
We equate our famlar FITS keyword, BACKSCAL wth Ω (and t s a column nstead of a keyword f t or more strctly, the rato depends on h!). Ths means that we can treat summng of the source spectra and background spectra dentcally: sum the source counts spectra (C(h)); sum the background counts spectra (B(h)); compute the mean source exposure (T s ); compute the mean background exposure (T b ); compute the exposure-weghted mean backscale for the source regon (Ω s (h)); compute the exposure-weghted mean backscale for the background regon (Ω b (h)); compute the exposure-weghted response (ARF, or ARF*RMF) (A(E) or R(E, h)); Gven these fles, whch are of standard types wth standard keywords, subsequent analyss s the same as for a sngle dataset. 5 Implementatons 5.1 Mergng Event Data Whle there are CIAO programs for mergng Level-1 datasets (e.g., dmmerge), don t use them for the purpose of smplfyng analyss by producng fewer fles to process. Data wll be ambguous, nconsstent, or ncomplete. We re-repeat, just don t do t. 5.2 Dynamc Mergng The only current analyss system we know of whch explctly supports dynamc mergng (ncludng vector BACKSCAL values) s the CXC s ISIS 4 program. (The prmary functon s combne datasets; see ts documentaton for use and assocated functons). Hgh-level vsualzaton of combned data s not yet bultn to ISIS, but an ISIS package s avalable n fancy plots.sl, whch s ncluded n the TGCat analyss package dstrbuton. 5 (Ths s used for the TGCat on-lne nteractve plottng of combned data.) The CIAO nteractve package, Sherpa, s scrptable and extensble, so n prncple, a Python package could probably be wrtten to mplement dynamc data combnaton; any custom package would need to provde a user-defned statstc whch sums the datasets flagged for combnaton, and would need to provde the combned model evaluaton. 5.3 Fle-based Mergng The analyss wth packages such as XSPEC and Sherpa have no bult-n support for dynamc combnaton of data. Hence there are several optons for combnng multple data fles externally. dmarfadd s a CIAO program to sum ARFs. Ths s used to produce the standard gratng ARFs by summng the ARF made for each detector element, and so does the exposure weghtng, and averagng of the exposure tme. It can also be used for non-dspersve ARFs. addresp s a CIAO program for summng ARFs or RMFs. 4 ISIS, The Interactve Spectral Interpretaton System, http://space.mt.edu/cxc/ss/ 5 TGCat utlty ISIS scrpts are avalable from http://space.mt.edu/cxc/analyss/tgcat/ndex.html 5
combne spectra s a CIAO program for summng PHA fles, ARFs, and RMFs. (Note n the caveats that only constant BACKSCAL s supported (even f a scalar quantty for each observaton.) add pha s an ISIS scrpt for summng PHA fles 6 ; ths s ntended to be used n conjuncton wth dmarfadd. (Note that scalar BACKSCAL values are properly averaged, but vector BACKSCAL s not supported). ftools 7 nclude several programs for manpulaton of spectral fles: addspec for addng two or more PHA fles; addarf for addng two or more ARFs; addrmf for addng two or more RMFs; marfrmf for multplcaton of an ARF by an RMF. (But beware the ftools expresson parser, whch can make t very dffcult to handle fle names ncludng / or -, for example.) 6 Pros & Cons of the Methods 6.1 Dynamc Combnaton Pros Cons Unque, well defned responses for each dataset; models for ndvdual datasets are treated consstently for any ft kernel (e.g., pleup). Flexble dataset management can nclude/exclude any dataset. Other applcatons are supported by the nfrastructure, such as fttng spectra of coupled sources, wth overlappng regons (see the ISIS help for combne datasets for an explct example). Multple fles to manage durng analyss; Greater computer memory requrements for storage of ndvdual datasets; Addtonal software requred for vsualzaton of combned data. 6.2 Fle Combnaton Pros Cons Fewer fles to manage durng analyss; More memory effcent; No addtonal vsualzaton software s requred. Extra preparaton s requred to produce merged data fles, beyond the standard set; For each permutaton of combned data, a separate set of fles must be prepared; Need to merge counts and responses consstently, wthout ambguty. 6 For add pha source and documentaton, see http://space.mt.edu/cxc/analyss/add pha/. 7 A General Package of Software to Manpulate FITS Fles http://heasarc.gsfc.nasa.gov/docs/software/ftools/ftools menu.html 6
7 Caveats 7.1 Multple Sources vs. Multple Observatons Note that wth Chandra gratng data, there s a subtle dstncton between addng orders n one dataset (analogous to multple sources n one feld n magng observatons; e.g., to combne 1 and +1), and combnng the same order n dfferent observatons (lke the same source n multple observatons). In the frst case, you ncrease the effectve area (ARF) for constant exposure. In the latter, you ncrease the exposure for constant area. In the defntons gven n Equatons 4-5, we made no such dstncton n computng T or A. Ths s suffcent when computng counts snce they nvolve the product, T A. However, care should be taken f computng count-rates, snce the mean exposure and weghted area mght be strctly ncorrect. 7.2 ARF & RMF, vs. RSP Whle use of ARF and RMF matched pars (whether ndvdual or combned) s suffcent, t s probably safest to use the product, the RSP (the response), whch ncludes the approprate weghted sum of ARF RMF, snce n general any combnaton necessarly requres ther product. 7.3 Specal Cases In general, the ntegrand of Equaton 1 need not be a lnear functon of the source model and responses. In fact, wth CCD-photon pleup, the response s a non-lnear functon of the source model. In that case, t s easy to see that we cannot effect a physcal sum of response fles (especally the RMF) snce we do not know the responses untl we have a source model. Usually, there s no need to sum counts for analyss of pled spectra. For gratng spectra, however, t s possble to have pleup n some regon of a spectrum but stll wsh to combne counts n other spectral regons (gven the very hgh dynamc range of the nstrument); here dynamc mergng s the only vald opton. For magng data, the RMF s much more observaton-dependent than for gratng spectra, so an exposure-weghted response s requred. It s possble to have an varable szed extracton regon for gratng spectra, but have scalar BACKSCAL values. In fact, ths s the default for Chandra LETG/HRC-S spectra: the extracton regons are not of constant wdth, but they are of constant wdth rato. Hence, we only store the relatve values n the headers as scalars. In ths case, Ω s (h)/ω b (h) = a constant. 7