Computational Optical Imaging - Optique Numerique. -- Noise, Dynamic Range and Color --

Transcription

1 Computational Optical Imaging - Optique Numerique -- Noise, Dynamic Range and Color -- Winter 2013 Ivo Ihrke

2 Organizational Issues I received your addresses Course announcements will be send via Course webpage at Teaching -> Computational Optical Imaging

3 Noise

4 Sources of noise Photon shot noise Dark current shot noise Fixed pattern noise Readout noise [Janesick97]

5 Noise Sources [Reibel2003] photon shot noise dark current noise read noise

6 Photon shot noise Variance in number of photons that are counted they arrive in a Poisson random process Standard deviation is square root of signal relative noise decreases with signal Fundamental limit on photodetector precision! Can be reduced by averaging multiple exposures.

7 Fixed pattern noise Caused by variations in component values Big problem for CMOS sensors An amp at every pixel, and one for every column Gain variation (proportional to signal PRNU) Bias variation (independent of signal dark current) Can be partially canceled by correlated double sampling (CDS) CCD s transfer all charge to a single output amplifier

8 Dark current Things besides photons can knock electrons loose in the silicon. These are collected, too. Highly temperature dependent doubles every 5-8 degrees C May be reduced by cooling the sensor. Proportional to exposure time Limits exposure durations eventually, the dark current fills your well capacity.

9 Dark Current Noise Dark current has fixed pattern noise. Dark current varies because of irregularities in the silicon. Dark current has shot noise, too! dominates in dark areas for long exposures Mean dark current may be subtracted but subtracting frames increases shot noise subtract the average dark current Dark current is why astronomers chill their image sensors.

10 Peltier Cooling of CMOS chip [Gary Honis]

11 Thermal Noise Generated by thermally induced motion of electrons in resistive regions (resistors, transistor channels in strong inversion ) What does it mean? Independent of the signal. Zero mean, white (flat, wide bandwidth) Another problem for CMOS, not CCD imagers Dominates at low signal levels Can limit dynamic range

12 Dark Current Noise Removal cooling the chip noise removal techniques to separate image data from noise e.g. median filtering uncooled cooled 25 s exposure time

13 Noise, noise, noise Reset (ktc) noise thermal noise when resetting the CMOS photodetector a big deal, actually. can be corrected with CDS Amplifier noise thermal spatially non-uniform 1/f noise non-linearities Quantization noise truncate analog value to N bits

14 Analog/Digital Conversion

15 Correlated Double Sampling reduce noise by comparing against a reference charge

16 Combined Noise Model [Reibel2003] 2 N TOT 2 FPN 2 R 2 DSN 2 PSN 2 PSN 2 PRNU C NL 2 FPN - fixed pattern noise 2 - readout noise R 2 - thermal dark current shot noise DSN 2 PSN - photon shot noise 2 - photo response non-uniformity PRNU - non-linear effects C NL

17 Combined Noise Model [Reibel2003] 2 N TOT 2 FPN 2 R 2 DSN 2 PSN 2 PSN 2 PRNU C NL 2 - fixed pattern noise (can be calibrated) FPN 2 - readout noise (CDS) R 2 - thermal dark current shot noise (cooling) DSN 2 PSN - photon shot noise (multiple exposures) 2 PRNU- photo response non-uniformity (per-pixel gain) C NL - non-linear effects (can also be calibrated for)

18 Noise Distribution [Reibel2003] ADU = Analog-Digital Unit, e.g. 1 ADU = 0.5 e-

19 Digital Images Images are now numbers (corrupted by noise)

20 Digital Images - Limitations Digital Sensor noise Dynamic Range Tone Curve Recording Medium Monochromatic Optical Distortions Aberrations

21 Dynamic Range

22 Dynamic Range dr = max output swing noise in the dark = Saturation level dark current Dark shot noise + readout noise noise in the dark is random noise sources that cannot be corrected with circuit tricks Photon shot noise and read noise

23 Dynamic Range of Standard Sensors 13.5 EVs or f-stops = contrast 11,000:1 = color Ivo Ihrke negative / Winter 2013

24 Dependency of Dynamic Range on ISO Unity gain is where 1 digital unit (ADU) equals 1 electron (e-) This happens at different ISO settings for different sensors Above that, the gain only increases the voltage before A/D conversion (possibly reducing the relative effect of some of the read noise) digital gain multiplies the digital values All gain settings beyond unity gain reduce dynamic range

25 What is High Dynamic Range (HDR)?

26 HDR Acquisition Exposure Brackets Radiance Map Tonemapped HDR Image Exposure Sequence [Debevec & Malik 97]

27 Ways to vary the exposure Shutter Speed F/stop (aperture) Neutral Density (ND) Filters Gain / ISO / Film Speed (DOF) (noise)

28 Combining the image radiance I ( x) l( x, t) dt scene constant over exposure time (or ND-filter) I ( x) t l( x, ) assumes linear response (radiometric calibration!) have several measurements with different t

29 Combining the image introduce a weighting function for the pixels: centered at the sensor mean value, e.g. Gaussian (image data in [0,1]) w( I ( x)) e ( I ( x) 0.5) compute final image as I final ( x) i w( I i i ( x)) I w( I i i ( x) / ( x)) t i

30 Pixel Values Radiometric Calibration Important for many vision and graphics algorithms g 1 : I Use a color chart with precisely known reflectances. E g? g 90% 59.1% 36.2% 19.8% 9.0% 3.1% Irradiance = const * Reflectance? Use more camera exposures to fill up the curve. Method assumes constant lighting on all patches Works best when source is far away (example sunlight). Inverse exists (g is monotonic and smooth for all cameras)

31 Response Curve - Practice Measurement: ColorCalibrationToolbox Example: 29 exposures of Gretag-Macbeth color checker (uses EXIF info - exiftool)

32 Color Calibration Toolbox Zoom-in

33 Color Calibration Toolbox Mark the patch rectangle

34 Color Calibration Toolbox Make sure the patches are properly extracted

35 Color Calibration Toolbox Verify response curve the example is for jpg on the Canon 5D mark II Make sure the samples are fit well Response curves (R,G,B) samples Inverse response curves

36 Color Calibration Toolbox Check HDR image

37 Color Calibration Toolbox How is the curve estimated? Variant of Mitsunaga and Nayar, Radiometric Self Calibration, CVPR 1999 Polynomial fit to data samples Variations: enforce monotonicity (derivative > 0) Prevents wiggling enforce passing of curve through (0,0) and (1,1) map range to range Perform a weighted fit accounts for sample non-uniformity

38 Response Curve Take Home Points Usually linear for RAW images Don t rely on it verify Usually non-linear for jpg or other compressed/processed formats Estimation from random images may be unstable Use well defined target (color checker) Prefer continuous-curve algorithms, especially for high bit depths

39 Applications - HDR Display 47 TFT LCD, LED backlight aspect ratio 16:9 resolution 1920 x 1080 contrast >1,000,000:1 brightness 4,000 cd/m 2 Images courtesy Dolby

40 Applications - Image Based Lighting Slides by Paul Debevec

41 EDR and HDR Cameras Extended range High dynamic range Spheron - Scanning - 26 f-stops Grass Valley Viper (10 bits log) Panavision Genesis (10 bits log)

42 Super CCD (Fuji) octagonal grid elements with different sensitivity extended DR better in low light Used in consumer products (Finepix)

43 HDRC Log Encoding CMOS pixel amplifier output is logarithmic U - logarithmic

44 Per-Pixel Exposure Time Control no pixim with pixim no pixim with pixim

45 Adaptive Dynamic Range Imaging [Nayar and Branzoi 03]

46 modulation signal modulated [Nayar & Branzoi 06] unmodulated Programmable Imaging

47 Textbook HDR image / video encoding capture, display, tone reproduction visible difference predictors image based lighting, etc.

48 Color

49 source: Kodak KAF-5101ce data sheet Sensing color Eye has 3 types of color receptors Therefore we need 3 different spectral sensitivities

50 Ways to sense color Field-sequential color simplest to implement only still scenes Proudkin-Gorskii, 1911 (Library of Congress exhibition)

51 Ways to sense color Field-sequential color simplest to implement only still scenes Proudkin-Gorskii, 1911 (Library of Congress exhibition)

52 Ways to sense color Field-sequential color simplest to implement only still scenes Proudkin-Gorskii, 1911 (Library of Congress exhibition)

53 Ways to sense color Field-sequential color simplest to implement only still scenes Proudkin-Gorskii, 1911 (Library of Congress exhibition)

54 Examples - Prokudin-Gorskij Self-portrait 1915

55 Examples - Prokudin-Gorskij Photograph 1910, Emir of Bukhara, Prokudin-Gorskii

56 Examples - Lew Tolstoy 1887 painting, Ilya Repin 1910 photograph, Sergey Prokudin-Gorskii

57 Color Wheel one color channel is captured at one shot 3 times the acquisition time static images only

58 Liquid Crystal Tunable Filter (LCTF) Computer controllable spectral filter VariSpec LCTF

59 Spectral Response of Lyot Stage Relative Transmissivity: T( ) 2 cos T max t ( ) Waveplate parameters (birefringence, thickness) Electrically Tunable Birefringence is implemented by liquid crystal in Lyot stage

60 Spectral Response of 7 Differently tuned Lyot Stages using several stages in sequence: product = result

61 VariSpec spectral curves

62 Ways to sense color 3-Chip Camera dichroic mirrors divide light into wavelength bands does not remove light: excellent quality but expensive interacts with lens design problem with polarization image: Theuwissen

63 Foveon Technology 3 layers capture RGB at the same location takes advantage of silicon s wavelength selectivity light decays at different rates for different wavelengths multilayer CMOS sensor gets 3 different spectral sensitivities don t get to choose the curves

64 Ways to sense color Color filter array paint each sensor with an individual filter requires just one chip but loses some spatial resolution demosaicing requires tricky image processing G R B G C M Y G primary secondary

65 Demosaicing bilinear interpolation sampling theory edge-directed/pattern-based interpolation correlation-based

66 Demosaicing Original image Bilinear interpolation Ron Kimmel,

67 Demosaicing Bilinear interpolation Edge-weighted interpolation Ron Kimmel,

68 Bilinear Interpolation G R B G = + + perform interpolation for each color channel separately

69 Bilinear Interpolation G R B G = + + R 23 R 12 R 14 4 R 32 R 34

70 Bilinear Interpolation G R B G = + + R 23 R 12 R 14 4 R 32 R 34 R 33 R 32 2 R 34

71 Bilinear Interpolation set all non-measured values to zero then convolve G R B G = / , B R F 4 / F G

72 Problem: Aliasing [Alleysson & Suesstrunk05]

73 Problem: Aliasing [Alleysson & Suesstrunk05]

74 Fourier Space /4

75 Excessive Blurring

76 Grid Effect

77 Bleak colors

78 Color Aliasing

79 [Alleysson & Suesstrunk05] optimize r1 and r2 to gain best separation Low-pass filter luminance High-pass filter chrominance (orthogonal filter) Demultiplex chrominance Interpolate opponent colors Add luminance and interpolated colors

80 Gradient-based (dcraw) [Chuang et al. 99] 1.Calculate gradients in 5x5 region 2.Select subset of gradients (below threshold) 3.Average color differences in the determined regions

81 Gradients Gradient S = G18 G8 / 3 R23 R13 B19 B9 / 2 B17 B7 / 2 G24 G14 / 3 G22 G12 / 3

82 Regions S selection: gradient < 1.5*Min+0.5*(Max-Min) e.g. {S,W,NE,SE} S: R = (R13+R23)/2, G = G18, B = (B17+B19)/2 NE: R = (R13+R5)/2, G = (G4+G8+G10+G14)/4, B = B9

83 Average Rsum = (Rs + Rw + Rne + Rse)/4 Gsum = (Gs + Gw + Gne + Gse)/4 Bsum = (Bs + Bw + Bne + Bse)/4 average of color differences G13 = R13 + (Gsum-Rsum); B13 = R13 + (Bsum-Rsum)

84 Demosaicing Take-home-points 2/3 of your image are just made up! color resolution is less than image resolution be careful with spiky BRDFs combining multiple video frames might help 98% of all demosaicing algorithms are ad-hoc smoothing based on constant hue assumption afterwards

85 White Balance capture the spectral characteristics of the light source to assure correct color reproduction tungsten daylight flourescent flash

86 White Balance Human perception adapts to illumination condition Practice: division of RGB values

87 White Balance Camera built-in function derive scale from white point sun infrared red green blue tungsten incandescent ultra violet wavelength

88 White Balance Camera built-in function derive scale from white point infrared red green blue ultra violet wavelength

89 White Balance Camera built-in function derive scale from white point infrared red green blue ultra violet wavelength

90 White Balance Human perception adapts to illumination condition Practice: division of RGB values Theory: achieve a neutral spectrum (only works for broad band sources and broad band reflectance) Conversion to RGB is an integral over the divided spectrum + linear transformation + gamma

91 Spectrum to Image do not have spectral display not a huge problem: humans have only three types of cones (color vision) and one type of rod (night vision) cones 6-7 million rods ~120 million rods more sensitive

92 Color Vision color vision by cones significant overlap of the response functions L = long M = mid S = short

93 Color Vision L ~63%, M ~31%, S ~6% of cones eye least sensitive to blue, most sensitive to yellowishgreen spectral region outside of support of the response functions cannot be perceived

94 Spectral response of human eye reproducing color is tricky color matching experiments use light source with known spectral distribution (i.e. assume uniform spectral distribution, can e.g. be achieved by normalization) filtered by a narrow band filter additionally, use monochromatic 700,546,435 nm let human observers adjust apparent brightness of one of the sources to match the other Color matching functions

95 Color Spaces RGB matching functions negative!

96 XYZ space The CIE (1931) standard observer

97 How to compute a tristimulus image from a spectral representation? We have to integrate with the spectrum with the appropriate color matching function I ( x) f ( )ˆ l ( x, ) d X X I ( x) f ( )ˆ l ( x, ) d Y Y I ( x) f ( )ˆ l ( x, ) d Z Z

98 Now to RGB convert XYZ to RGB

99 Horseshoe Diagram

100 White Point for Different Color Temperatures Planckian Locus: - convert black body temperature to XYZ and put intohorseshoe diagram L_\lambda = spectral radiance [W/m^2/m] lambda = wavelength [m] h = Planck s constant [J.s] k = Boltzmann constant [J/K] c = speed of light [m/s] T = temperature of black body [K]

101 Display Gamut white point

102 Bibliography Holst, G. CCD Arrays, Cameras, and Displays. SPIE Optical Engineering Press, Bellingham, Washington, Theuwissen, A. Solid-State Imaging with Charge- Coupled Devices. Kluwer Academic Publishers, Boston, Curless, CSE558 lecture notes (UW, Spring 01). El Gamal et al., EE392b lecture notes (Spring 01). Several Kodak Application Notes at pplicationnotes.jhtml Reibel et al., CCD or CMOS camera noise characterization, Eur. Phys. J. AP 21, 2003

103 Bibliography D. Alleysson, S. Suesstrunk: Linear Demosaicing inspired by the Human Visual System, IEEE Trans. on Image Processing, 14(4), B. K. Gunturk, Y. Altunbasak, R. M. Mersereau: Color Plane Interpolation Using Alternating Projections, IEEE Trans. on Image Processing, 11(9), E. Chang, S. Cheung, D.Y. Pan: Color filter array recovery using a threshold-based variable number of gradients. Proc. SPIE, vol. 3650, pp ,

104 Bibliography Y. Takahashi, K. Hiraki, H. Kikuchi, S. Muaramtsu: Color Demosaicing Using Asymmetric Directional Interpolation and Hue Vector Smoothing, IEICE 20 th Workshop on Circuits and Systems, R. Kimmel, Demosaicing: Image Reconstruction from Color CCD Samples, IEEE Trans. on Image Processing. Vol. 8, No. 9, Boris Ajdin, Matthias B. Hullin, Christian Fuchs, Hans- Peter Seidel, Hendrik P. A. Lensch: Demosaicing by Smoothing along 1D Features. Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008.

105 ICC Profiles (ICC international color consortium) color management system capture the properties of all devices camera and lighting monitor settings output properties display device (e.g. monitor) common interchange space srgb standard as a definition of RGB monitor profile input device (e.g. camera) input profile profile connection space output profile output device (e.g. printer)

106 ICC Profiles and HDR Image Generation profile connection spaces CIELAB (perceptual linear) linear CIEXYZ color space can be used to create an high dynamic range image in the profile connection space allows for a color calibrated workflow... input device (e.g. camera) input profile profile connection space output profile output device (e.g. printer)