CCD Image Processig: Issues & Solutios Correctio of Raw Image with Bias, Dark, Flat Images Raw File r x, y [ ] Dark Frame d[ x, y] Flat Field Image f [ xy, ] r[ x, y] d[ x, y] Raw Dark f [ xy, ] bxy [, ] Raw Dark Flat Bias r[ x, y] d[ x, y] f [ xy, ] bxy [, ] Output Image Bias Image b x, y [ ] Flat Bias Correctio of Raw Image w/ Flat Image, w/o Dark Image Raw File r[ x, y] Bias Image b x, y [ ] Flat Field Image f xy, [ ] r[ x, y] b[ x, y] Raw Bias f [ xy, ] bxy [, ] Flat Bias Assumes Small Dark Curret (Cooled Camera) Raw Bias Flat Bias r[ x, y] b[ x, y] f [ xy, ] bxy [, ] Output Image CCDs: : Noise Sources Sky Backgroud Diffuse Light from Sky (Usually Variable) Dark Curret Sigal from Uexposed CCD Due to Electroic Amplifiers Photo Coutig Ucertaity i Number of Icomig Photos Read Noise Ucertaity i Number of Electros at a Pixel Problem with Sky Backgroud Ucertaity i Number of Photos from Source How much sigal is actually from the source object istead of from iterveig atmosphere? Solutio for Sky Backgroud Measure Sky Sigal from Images Take i (Approximately) Same Directio (Regio of Sky) at (Approximately) Same Time Use Off-Object Regio(s) of Source Image Subtract Brightess Values from Object Values 1
Problem: Dark Curret Sigal i Every Pixel Eve if NOT Exposed to Light Stregth Proportioal to Exposure Time Sigal Varies Over Pixels No-Determiistic Sigal = NOISE Solutio: Dark Curret Subtract Image(s) Obtaied Without Exposig CCD Leave Shutter Closed to Make a Dark Frame Same Exposure Time for Image ad Dark Frame Measure of Similar Noise as i Exposed Image Actually Average Measuremets from Multiple Images Decreases Ucertaity i Dark Curret Digressio o Noise What is Noise? Noise is a Nodetermiistic Sigal Radom Sigal Exact Form is ot Predictable Statistical Properties ARE (usually) Predictable Statistical Properties of Noise 1. Average Value = Mea µ 2. Variatio from Average = Deviatio σ Distributio of Likelihood of Noise Probability Distributio More Geeral Descriptio of Noise tha µ, σ Ofte Measured from Noise Itself Histogram Histogram of Uiform Distributio Values are Real Numbers (e.g., 0.0105) Noise Values Betwee 0 ad 1 Equally Likely Available i Computer Laguages Noise Sample Histogram Mea µ Histogram of Gaussia Distributio Values are Real Numbers NOT Equally Likely Describes May Physical Noise Pheomea Mea µ Mea µ Variatio Mea µ Variatio Mea µ = 0.5 Variatio Mea µ = 0 Values Close to µ More Likely Variatio 2
Histogram of Poisso Distributio Values are Itegers (e.g., 4, 76, ) Describes Distributio of Ifrequet Evets, e.g., Photo Arrivals Mea µ Histogram of Poisso Distributio Mea µ Mea µ Variatio Mea µ Variatio Mea µ = 4 Values Close to µ More Likely Variatio is NOT Symmetric Variatio Mea µ = 25 Variatio How to Describe Variatio :: 1 Measure of the Spread ( Deviatio ) of the Measured Values (say x ) from the Actual Value, which we ca call µ The Error ε of Oe Measuremet is: ( x ) ε = µ (which ca be positive or egative) Descriptio of Variatio :: 2 Sum of Errors over all Measuremets: ( x ) ε = µ Ca be Positive or Negative Sum of Errors Ca Be Small, Eve If Errors are Large (Errors ca Cacel ) Descriptio of Variatio :: 3 Use Square of Error Rather Tha Error Itself: ( x ) 2 2 ε µ = 0 Must be Positive Descriptio of Variatio :: 4 Sum of Squared Errors over all Measuremets: ( ε ) ( x µ ) 2 2 = 0 Average of Squared Errors 1 N ( ε ) 2 ( x µ ) 2 = 0 N 3
Descriptio of Variatio :: 5 Stadard Deviatio σ = Square Root of Average of Squared Errors Effect of Averagig o Deviatio σ Example: Average of 2 Readigs from Uiform Distributio σ ( x ) 2 µ 0 N Effect of Averagig of 2 Samples: Compare the Histograms Mea µ Mea µ Averagig Reduces σ Averagig Does Not Chage µ Shape of Histogram is Chaged! σ 0.289 More Cocetrated Near µ Averagig REDUCES Variatio σ σ 0.289 σ 0.205 σ is Reduced by Factor: 0.289 1. 41 0.205 Averages of 4 ad 9 Samples Averagig of Radom Noise REDUCES the Deviatio σ Samples Averaged Reductio i Deviatio σ N = 2 1.41 N = 4 2.01 N = 9 3.01 σ 0.144 0.289 0.144 σ 0.096 Reductio Factors 0.289 2.01 3. 0.096 01 Observatio: σ σ = Average of N Samples Oe Sample N 4
Why Does Deviatio Decrease if Images are Averaged? Bright Noise Pixel i Oe Image may be Dark i Secod Image Oly Occasioally Will Same Pixel be Brighter (or Darker ) tha the Average i Both Images Average Value is Closer to Mea Value tha Origial Values Averagig Over Time vs. Averagig Over Space Examples of Averagig Differet Noise Samples Collected at Differet Times Could Also Average Differet Noise Samples Over Space (i.e., Coordiate x) Spatial Averagig Compariso of Histograms After Spatial Averagig Uiform Distributio µ = 0.5 σ 0.289 Spatial Average of 4 Samples µ = 0.5 σ 0.144 Spatial Average of 9 Samples µ = 0.5 σ 0.096 Effect of Averagig o Dark Curret Dark Curret is NOT a Determiistic Number Each Measuremet of Dark Curret Should Be Differet Values Are Selected from Some Distributio of Likelihood (Probability) Example of Dark Curret Example of Dark Curret Readigs Oe-Dimesioal Examples (1-D Fuctios) Noise Measured as Fuctio of Oe Spatial Coordiate Readig of Dark Curret vs. Positio i Simulated Dark Image #1 Readig of Dark Curret vs. Positio i Simulated Dark Image #2 Variatio 5
Averages of Idepedet Dark Curret Readigs Average of 2 Readigs of Dark Curret vs. Positio Average of 9 Readigs of Dark Curret vs. Positio Ifrequet Photo Arrivals Differet Mechaism Number of Photos is a Iteger! Differet Distributio of Values Variatio Variatio i Average of 9 Images 1/ 9 = 1/3 of Variatio i 1 Image Problem: Photo Coutig Statistics Photos from Source Arrive Ifrequetly Few Photos Measuremet of Number of Source Photos (Also) is NOT Determiistic Radom Numbers Distributio of Radom Numbers of Rarely Occurrig Evets is Govered by Poisso Statistics Simplest Distributio of Itegers Oly Two Possible Outcomes: YES NO Oly Oe Parameter i Distributio Likelihood of Outcome YES Call it p Just like Coutig Coi Flips Examples with 1024 Flips of a Coi Example with p = 0.5 Secod Example with p = 0.5 Strig of Outcomes N = 1024 N heads = 511 p = 511/1024 < 0.5 Histogram Strig of Outcomes N = 1024 N heads = 522 µ = 522/1024 > 0.5 T H Histogram 6
What if Coi is Ufair? p 0.5 What Happes to Deviatio σ? For Oe Flip of 1024 Cois: p = 0.5 σ 0.5 p = 0? p = 1? Strig of Outcomes T H Histogram Deviatio is Largest if p = 0.5! The Possible Variatio is Largest if p is i the middle! Add More Tosses 2 Coi Tosses More Possibilities for Photo Arrivals Sum of Two Sets with p = 0.5 Sum of Two Sets with p = 0.25 Strig of Outcomes N = 1024 µ = 1.028 Histogram 3 Outcomes: 2 H 1H, 1T (most likely) 2T Strig of Outcomes N = 1024 Histogram 3 Outcomes: 2 H (least likely) 1H, 1T 2T (most likely) 7
Add More Flips with Ulikely Heads Add More Flips with Ulikely Heads (1600 with p = 0.25) Most Pixels Measure 25 Heads (100 0.25) Most Pixels Measure 400 Heads (1600 0.25) Examples of Poisso Noise Measured at 64 Pixels 1. Exposed CCD to Uiform Illumiatio 2. Pixels Record Differet Numbers of Photos Variatio of Measuremet Varies with Number of Photos For Poisso-Distributed Radom Number with Mea Value µ = N: Stadard Deviatio of Measuremet is: σ = N Average Value µ = 25 Average Values µ = 400 AND µ = 25 Histograms of Two Poisso Distributios µ = 25 (Note: Chage of Horizotal Scale!) µ=400 Quality of Measuremet of Number of Photos Sigal-to-Noise Ratio Ratio of Sigal to Noise (Ma, Like What Else?) SNR µ σ Variatio Average Value µ = 25 Variatio σ = 25 = 5 Variatio Average Value µ = 400 Variatio σ = 400 = 20 8
Sigal-to to-noise Ratio for Poisso Distributio Sigal-to-Noise Ratio of Poisso Distributio µ N SNR = = σ N More Photos Higher-Quality Measuremet N Solutio: Photo Coutig Statistics Collect as MANY Photos as POSSIBLE!! Largest Aperture (Telescope Collectig Area) Logest Exposure Time Maximizes Source Illumiatio o Detector Icreases Number of Photos Issue is More Importat for X Rays tha for Loger Wavelegths Fewer X-Ray Photos Problem: Read Noise Ucertaity i Number of Electros Couted Due to Statistical Errors, Just Like Photo Couts Detector Electroics Solutio: Read Noise Collect Sufficiet Number of Photos so that Read Noise is Less Importat Tha Photo Coutig Noise Some Electroic Sesors (CCD- like Devices) Ca Be Read Out Nodestructively Charge Ijectio Devices (CIDs) Used i Ifrared multiple reads of CID pixels reduces ucertaity CCDs: : artifacts ad defects 1. Bad Pixels dead, hot, flickerig 2. Pixel-to-Pixel Differeces i Quatum Efficiecy (QE) # of electros created Quatum Efficiecy = # of icidet photos 0 QE < 1 Each CCD pixel has its ow uique QE Differeces i QE Across Pixels Map of CCD Sesitivity Measured by Flat Field CCDs: : artifacts ad defects 3. Saturatio each pixel ca hold a limited quatity of electros (limited well depth of a pixel) 4. Loss of Charge durig pixel charge trasfer & readout Pixel s Value at Readout May Not Be What Was Measured Whe Light Was Collected 9
Bad Pixels Issue: Some Fractio of Pixels i a CCD are: Dead (measure o charge) Hot (always measure more charge tha collected) Solutios: Replace Value of Bad Pixel with Average of Pixel s Neighbors Dither the Telescope over a Series of Images Move Telescope Slightly Betwee Images to Esure that Source Fall o Good Pixels i Some of the Images Differet Images Must be Registered (Aliged) ad Appropriately Combied Pixel-to to-pixel Differeces i QE Issue: each pixel has its ow respose to light Solutio: obtai ad use a flat field image to correct for pixel-to-pixel ouiformities costruct flat field by exposig CCD to a uiform source of illumiatio image the sky or a white scree pasted o the dome divide source images by the flat field image for every pixel x,y, ew source itesity is ow S (x,y) = S(x,y)/F(x,y) where F(x,y) is the flat field pixel value; bright pixels are suppressed, dim pixels are emphasized Issue: Saturatio Issue: each pixel ca oly hold so may electros (limited well depth of the pixel), so image of bright source ofte saturates detector at saturatio, pixel stops detectig ew photos (like overexposure) saturated pixels ca bleed over to eighbors, causig streaks i image Solutio: put less light o detector i each image take shorter exposures ad add them together telescope poitig will drift; eed to re-register images read oise ca become a problem use eutral desity filter a filter that blocks some light at all wavelegths uiformly faiter sources lost Solutio to Saturatio Reduce Light o Detector i Each Image Take a Series of Shorter Exposures ad Add Them Together Telescope Usually Drifts Images Must be Re-Registered Read Noise Worses Use Neutral Desity Filter Blocks Same Percetage of Light at All Wavelegths Faiter Sources Lost Issue: Loss of Electro Charge No CCD Trasfers Charge Betwee Pixels with 100% Efficiecy Itroduces Ucertaity i Covertig to Light Itesity (of Optical Visible Light) or to Photo Eergy (for X Rays) Solutio to Loss of Electro Charge Build Better CCDs!!! Icrease Trasfer Efficiecy # of electros trasferred to ext pixel Trasfer Efficiecy = # of electros i pixel Moder CCDs have charge trasfer efficiecies 99.9999% some do ot: those sesitive to soft X Rays loger wavelegths tha short-wavelegth hard X Rays 10
Digital Processig of Astroomical Images Computer Processig of Digital Images Arithmetic Calculatios: Additio Subtractio Multiplicatio Divisio Digital Processig Images are Specified as Fuctios, e.g., r [x,y] meas the brightess r at positio [x,y] Brightess is measured i Number of Photos [x,y] Coordiates Measured i: Pixels Arc Measuremets (Degrees-ArcMiutes- ArcSecods) Sum of Two Images [, ] + [, ] = [, ] r x y r x y g x y 1 2 Summatio = Mathematical Itegratio To Average Noise Differece of Two Images [, ] [, ] = [, ] r x y r x y g x y 1 2 To Detect Chages i the Image, e.g., Due to Motio Multiplicatio of Two Images [, ] [, ] = [, ] r x y m x y g x y m[x,y] is a Mask Fuctio Divisio of Two Images [, ] [, ] r x y f x y = g xy [, ] Divide by Flat Field f[x,y] 11
Data Pipeliig Issue: ow that I ve collected all of these images, what do I do? Solutio: build a automated data processig pipelie Space observatories (e.g., HST) routiely process raw image data ad deliver oly the processed images to the observer groud-based observatories are slowly comig aroud to this operatioal model RIT s CIS is i the data pipelie busiess NASA s SOFIA South Pole facilities 12