Introduction to Image Processing and Computer Vision -- Noise, Dynamic Range and Color -- Winter 2013 Ivo Ihrke
Organizational Issues I received your email addresses Course announcements will be send via email Course webpage at http://giana.mmci.uni-saarland.de/ Teaching -> Computational Optical Imaging
Digital Images Images are now numbers (corrupted by noise) ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ
Digital Images - Limitations Digital Sensor noise Dynamic Range Tone Curve Recording Medium Monochromatic Optical Distortions Aberrations
Dynamic Range
Dynamic Range dr = max output swing noise in the dark = Saturation level dark current Dark shot noise + readout noise noise in the dark is random noise sources that cannot be corrected with circuit tricks Photon shot noise and read noise
www.dxomark.com Dynamic Range of Standard Sensors 13.5 EVs or f-stops = contrast 11,000:1 = color Ivo Ihrke negative / Winter 2013
Dependency of Dynamic Range on ISO Unity gain is where 1 digital unit (ADU) equals 1 electron (e-) This happens at different ISO settings for different sensors Above that, the gain only increases the voltage before A/D conversion (possibly reducing the relative effect of some of the read noise) digital gain multiplies the digital values All gain settings beyond unity gain reduce dynamic range
Dependency of Dynamic Range on ISO
Camera Response Curve
Radiometric Response Curve Raw sensor readings are usually linear (CCD), for CMOS it can be non-linear (depending on amplifier type) (linear) (log) On-chip processing may modify the relation #photons/digital number i.e. the mapping irradiance to digital value may become non-linear This mapping is called radiometric response curve
Pixel Value Log Encoding CMOS pixel amplifier output may be logarithmic Example: advertisement for IMSChips HDRC-MDC04 eye HDRC Log intensity
Programmable Response Curve Example: Photonfocus MDC-1024
Pixel Values Radiometric Calibration the process of determining the radiometric response curve g 1 : Use a color chart with precisely known reflectances. I E 255 g 1? g 90% 59.1% 36.2% 19.8% 9.0% 3.1% 0 0 1 Irradiance = const * Reflectance? Use more camera exposures to fill up the curve. Method assumes constant lighting on all patches Works best when source is far away (example sunlight). Inverse exists (g is monotonic and smooth for all cameras)
Response Curve - Practice Measurement: ColorCalibrationToolbox http://giana.mmci.uni-saarland.de/website-template/software.php Example: 29 exposures of Gretag-Macbeth color checker (uses EXIF info - exiftool)
Color Calibration Toolbox Zoom-in
Color Calibration Toolbox Mark the patch rectangle
Color Calibration Toolbox Make sure the patches are properly extracted
Color Calibration Toolbox Verify response curve the example is for jpg on the Canon 5D mark II Make sure the samples are fit well Response curves (R,G,B) samples Inverse response curves
Color Calibration Toolbox Check HDR image
Color Calibration Toolbox How is the curve estimated? Variant of Mitsunaga and Nayar, Radiometric Self Calibration, CVPR 1999 Polynomial fit to data samples Variations: enforce monotonicity (derivative > 0) Prevents wiggling enforce passing of curve through (0,0) and (1,1) map range to range Perform a weighted fit accounts for sample non-uniformity
Response Curve Take Home Points Usually linear for RAW images Don t rely on it verify Usually non-linear (gamma) for jpg or other compressed/processed formats Estimation from random images may be unstable Use well defined target (color checker) Prefer continuous-curve algorithms, especially for high bit depths
Gamma Mapping/Correction Account for properties of human vision Logarithmic, similar to hearing Approximated by Gamma curve
Gamma Mapping: Effect Emphasizes contrast on lower end of linear intensity range for γ<1 Emphasizes contrast on higher end of linear intensity range for γ>1
Textbook HDR image / video encoding capture, display, tone reproduction visible difference predictors image based lighting, etc.
Color
Spectrum to Image not a huge problem: humans have only three types of cones (color vision) and one type of rod (night vision) cones 6-7 million rods ~120 million rods more sensitive
Color Vision color vision by cones significant overlap of the response functions L = long M = mid S = short
Color Vision L ~63%, M ~31%, S ~6% of cones eye least sensitive to blue, most sensitive to yellowishgreen spectral region outside of support of the response functions cannot be perceived
Spectral response of human eye reproducing color is tricky color matching experiments use light source with known spectral distribution (i.e. assume uniform spectral distribution, can e.g. be achieved by normalization) filtered by a narrow band filter additionally, use monochromatic sources @ 700,546,435 nm let human observers adjust apparent brightness of one of the sources to match the other Color matching functions
Color Spaces RGB matching functions negative!
XYZ space The CIE (1931) standard observer
How to compute a tristimulus image from a spectral representation? We have to integrate with the spectrum with the appropriate color matching function I ( x) f ( )ˆ l ( x, ) d X X I ( x) f ( )ˆ l ( x, ) d Y Y I ( x) f ( )ˆ l ( x, ) d Z Z
Now to RGB convert XYZ to RGB Possibly to srgb (non-linear space, gamma) Where C = {R,G,B}
Other Color Spaces Many linearly related spaces exist have different separation properties example: YCbCr (JPEG) Cr Cb Y=0 Y=0.5 Y=1.0 [Wikipedia]
source: Kodak KAF-5101ce data sheet Sensing color Eye has 3 types of color receptors Therefore we need 3 different spectral sensitivities
Ways to sense color Field-sequential color simplest to implement only still scenes Proudkin-Gorskii, 1911 (Library of Congress exhibition)
Ways to sense color Field-sequential color simplest to implement only still scenes Proudkin-Gorskii, 1911 (Library of Congress exhibition)
Ways to sense color Field-sequential color simplest to implement only still scenes Proudkin-Gorskii, 1911 (Library of Congress exhibition)
Ways to sense color Field-sequential color simplest to implement only still scenes Proudkin-Gorskii, 1911 (Library of Congress exhibition)
Examples - Prokudin-Gorskij Self-portrait 1915
Examples - Prokudin-Gorskij Photograph 1910, Emir of Bukhara, Prokudin-Gorskii
Examples - Lew Tolstoy 1887 painting, Ilya Repin 1910 photograph, Sergey Prokudin-Gorskii
Color Wheel one color channel is captured at one shot 3 times the acquisition time static images only
Ways to sense color 3-Chip Camera dichroic mirrors divide light into wavelength bands does not remove light: excellent quality but expensive interacts with lens design problem with polarization image: Theuwissen
Foveon Technology 3 layers capture RGB at the same location takes advantage of silicon s wavelength selectivity light decays at different rates for different wavelengths multilayer CMOS sensor gets 3 different spectral sensitivities don t get to choose the curves
Ways to sense color Color filter array cover each sensor with an individual filter requires just one chip but loses some spatial resolution demosaicing requires tricky image processing G R B G primary
Multi-Shot (example Jenoptik C14) take four images, moving the sensor by one pixel (use fourth image for noise reduction) can be used for supersampling (move by ½, ¼ pixel)
Demosaicing bilinear interpolation sampling theory edge-directed/pattern-based interpolation correlation-based
Settings: f/8 1/25s ISO: 800, no noise reduction Example raw image (Canon 5D markii) Processed image Raw image
Example raw image (Canon 5D markii) Processed image Raw image
Example raw image (Canon 5D markii) Processed image Raw image
Colors assigned Raw image Processed image Example raw image (Canon 5D markii)
Demosaicing Original image Bilinear interpolation Ron Kimmel, http://www.cs.technion.ac.il/~ron/demosaic.html
Demosaicing Bilinear interpolation Edge-weighted interpolation Ron Kimmel, http://www.cs.technion.ac.il/~ron/demosaic.html
Bilinear Interpolation G R B G = + + perform interpolation for each color channel separately
Bilinear Interpolation G R B G = + + R 23 R 12 R 14 4 R 32 R 34
Bilinear Interpolation G R B G = + + R 23 R 12 R 14 4 R 32 R 34 R 33 R 32 2 R 34
Bilinear Interpolation set all non-measured values to zero then convolve G R B G = + + 4 / 1 2 1 2 4 2 1 2 1, B R F 4 / 0 1 0 1 4 1 0 1 0 F G
Color White Balancing
White Balance Colors appear different under different illumination conditions tungsten daylight flourescent flash
White Balance Why is there a constant appearance for human observers? Human perception adapts to illumination condition Practice: division of RGB values
White Balance Camera built-in function derive scale from white point sun infrared red green blue tungsten incandescent ultra violet wavelength
White Balance Camera built-in function derive scale from white point infrared red green blue ultra violet wavelength
White Balance Camera built-in function derive scale from white point infrared red green blue ultra violet wavelength
Horseshoe Diagram
White Point for Different Color Temperatures Planckian Locus: - convert black body temperature to XYZ and put intohorseshoe diagram L_\lambda = spectral radiance [W/m^2/m] lambda = wavelength [m] h = Planck s constant [J.s] k = Boltzmann constant [J/K] c = speed of light [m/s] T = temperature of black body [K]
White Balance Human perception adapts to illumination condition Practice: division of RGB values Theory: achieve a neutral spectrum (only works for broad band sources and broad band reflectance) Conversion to RGB is an integral over the divided spectrum + linear transformation + gamma
White Balance capture the spectral characteristics of the light source to assure correct color reproduction tungsten daylight flourescent flash
Textbook Physical principles color spaces, encoding chromatic adaption, perceptual issues Display technology, color management
Compression
Compression Lossless Entropy coding based (e.g. Huffmann coding) Popular example: zip/gzip (used in png format) reproduces exact copy of bit signals Lossy Takes advantage of the fact that information is an image (removes inperceptual data) Popular example: jpeg
Compression Example: Jpeg Selectable compression ratio Picture: gradually varying compression ratio from left to right [Wikipedia]
Compression JPEG: Convert image to YCbCr Subsample chroma channels (Cb,Cr) Split into 8x8 blocks Apply discrete cosine transform Remove small coefficients Color channels are treated independently 8x8 DCT basis
Bibliography Holst, G. CCD Arrays, Cameras, and Displays. SPIE Optical Engineering Press, Bellingham, Washington, 1998. Theuwissen, A. Solid-State Imaging with Charge- Coupled Devices. Kluwer Academic Publishers, Boston, 1995. Curless, CSE558 lecture notes (UW, Spring 01). El Gamal et al., EE392b lecture notes (Spring 01). Several Kodak Application Notes at http://www.kodak.com/global/en/digital/ccd/publications/a pplicationnotes.jhtml Reibel et al., CCD or CMOS camera noise characterization, Eur. Phys. J. AP 21, 2003
ICC Profiles (ICC international color consortium) color management system capture the properties of all devices camera and lighting monitor settings output properties display device (e.g. monitor) common interchange space srgb standard as a definition of RGB monitor profile input device (e.g. camera) input profile profile connection space output profile output device (e.g. printer)
ICC Profiles and HDR Image Generation profile connection spaces CIELAB (perceptual linear) linear CIEXYZ color space can be used to create an high dynamic range image in the profile connection space allows for a color calibrated workflow... input device (e.g. camera) input profile profile connection space output profile output device (e.g. printer)
Other HDR approaches Determine for each pixel when enough photons haven been collected. Logarithmic timings yields floating point representation (mantissa + exponent).
PMD measured distance in each pixel exploit interference emit light (modulated) at each pixel compare reflected light to reference light computation in a smart pixel
Next week Signal Processing
What is High Dynamic Range (HDR)? http://en.wikipedia.org/wiki/high-dynamic-range_imaging
HDR Acquisition Exposure Brackets Radiance Map Tonemapped HDR Image Exposure Sequence [Debevec & Malik 97]
Ways to vary the exposure Shutter Speed F/stop (aperture) Neutral Density (ND) Filters Gain / ISO / Film Speed (DOF) (noise)
Combining the image radiance I ( x) l( x, t) dt scene constant over exposure time (or ND-filter) I ( x) t l( x, ) assumes linear response (radiometric calibration!) have several measurements with different t
Combining the image introduce a weighting function for the pixels: centered at the sensor mean value, e.g. Gaussian (image data in [0,1]) w( I ( x)) e ( I ( x) 0.5) 2 0.2 2 compute final image as I final ( x) i w( I i i ( x)) I w( I i i ( x) ( x)) / t i