Introduction to Image Processing and Computer Vision -- Noise, Dynamic Range and Color --

Transcription

1 Introduction to Image Processing and Computer Vision -- Noise, Dynamic Range and Color -- Winter 2013 Ivo Ihrke

2 Organizational Issues I received your addresses Course announcements will be send via Course webpage at Teaching -> Computational Optical Imaging

3 Digital Images Images are now numbers (corrupted by noise) ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ ±σ

4 Digital Images - Limitations Digital Sensor noise Dynamic Range Tone Curve Recording Medium Monochromatic Optical Distortions Aberrations

5 Dynamic Range

6 Dynamic Range dr = max output swing noise in the dark = Saturation level dark current Dark shot noise + readout noise noise in the dark is random noise sources that cannot be corrected with circuit tricks Photon shot noise and read noise

7 Dynamic Range of Standard Sensors 13.5 EVs or f-stops = contrast 11,000:1 = color Ivo Ihrke negative / Winter 2013

8 Dependency of Dynamic Range on ISO Unity gain is where 1 digital unit (ADU) equals 1 electron (e-) This happens at different ISO settings for different sensors Above that, the gain only increases the voltage before A/D conversion (possibly reducing the relative effect of some of the read noise) digital gain multiplies the digital values All gain settings beyond unity gain reduce dynamic range

9 Dependency of Dynamic Range on ISO

10 Camera Response Curve

11 Radiometric Response Curve Raw sensor readings are usually linear (CCD), for CMOS it can be non-linear (depending on amplifier type) (linear) (log) On-chip processing may modify the relation #photons/digital number i.e. the mapping irradiance to digital value may become non-linear This mapping is called radiometric response curve

12 Pixel Value Log Encoding CMOS pixel amplifier output may be logarithmic Example: advertisement for IMSChips HDRC-MDC04 eye HDRC Log intensity

13 Programmable Response Curve Example: Photonfocus MDC-1024

14 Pixel Values Radiometric Calibration the process of determining the radiometric response curve g 1 : Use a color chart with precisely known reflectances. I E 255 g 1? g 90% 59.1% 36.2% 19.8% 9.0% 3.1% Irradiance = const * Reflectance? Use more camera exposures to fill up the curve. Method assumes constant lighting on all patches Works best when source is far away (example sunlight). Inverse exists (g is monotonic and smooth for all cameras)

15 Response Curve - Practice Measurement: ColorCalibrationToolbox Example: 29 exposures of Gretag-Macbeth color checker (uses EXIF info - exiftool)

16 Color Calibration Toolbox Zoom-in

17 Color Calibration Toolbox Mark the patch rectangle

18 Color Calibration Toolbox Make sure the patches are properly extracted

19 Color Calibration Toolbox Verify response curve the example is for jpg on the Canon 5D mark II Make sure the samples are fit well Response curves (R,G,B) samples Inverse response curves

20 Color Calibration Toolbox Check HDR image

21 Color Calibration Toolbox How is the curve estimated? Variant of Mitsunaga and Nayar, Radiometric Self Calibration, CVPR 1999 Polynomial fit to data samples Variations: enforce monotonicity (derivative > 0) Prevents wiggling enforce passing of curve through (0,0) and (1,1) map range to range Perform a weighted fit accounts for sample non-uniformity

22 Response Curve Take Home Points Usually linear for RAW images Don t rely on it verify Usually non-linear (gamma) for jpg or other compressed/processed formats Estimation from random images may be unstable Use well defined target (color checker) Prefer continuous-curve algorithms, especially for high bit depths

23 Gamma Mapping/Correction Account for properties of human vision Logarithmic, similar to hearing Approximated by Gamma curve

24 Gamma Mapping: Effect Emphasizes contrast on lower end of linear intensity range for γ<1 Emphasizes contrast on higher end of linear intensity range for γ>1

25 Textbook HDR image / video encoding capture, display, tone reproduction visible difference predictors image based lighting, etc.

26 Color

27 Spectrum to Image not a huge problem: humans have only three types of cones (color vision) and one type of rod (night vision) cones 6-7 million rods ~120 million rods more sensitive

28 Color Vision color vision by cones significant overlap of the response functions L = long M = mid S = short

29 Color Vision L ~63%, M ~31%, S ~6% of cones eye least sensitive to blue, most sensitive to yellowishgreen spectral region outside of support of the response functions cannot be perceived

30 Spectral response of human eye reproducing color is tricky color matching experiments use light source with known spectral distribution (i.e. assume uniform spectral distribution, can e.g. be achieved by normalization) filtered by a narrow band filter additionally, use monochromatic 700,546,435 nm let human observers adjust apparent brightness of one of the sources to match the other Color matching functions

31 Color Spaces RGB matching functions negative!

32 XYZ space The CIE (1931) standard observer

33 How to compute a tristimulus image from a spectral representation? We have to integrate with the spectrum with the appropriate color matching function I ( x) f ( )ˆ l ( x, ) d X X I ( x) f ( )ˆ l ( x, ) d Y Y I ( x) f ( )ˆ l ( x, ) d Z Z

34 Now to RGB convert XYZ to RGB Possibly to srgb (non-linear space, gamma) Where C = {R,G,B}

35 Other Color Spaces Many linearly related spaces exist have different separation properties example: YCbCr (JPEG) Cr Cb Y=0 Y=0.5 Y=1.0 [Wikipedia]

36 source: Kodak KAF-5101ce data sheet Sensing color Eye has 3 types of color receptors Therefore we need 3 different spectral sensitivities

37 Ways to sense color Field-sequential color simplest to implement only still scenes Proudkin-Gorskii, 1911 (Library of Congress exhibition)

38 Ways to sense color Field-sequential color simplest to implement only still scenes Proudkin-Gorskii, 1911 (Library of Congress exhibition)

39 Ways to sense color Field-sequential color simplest to implement only still scenes Proudkin-Gorskii, 1911 (Library of Congress exhibition)

40 Ways to sense color Field-sequential color simplest to implement only still scenes Proudkin-Gorskii, 1911 (Library of Congress exhibition)

41 Examples - Prokudin-Gorskij Self-portrait 1915

42 Examples - Prokudin-Gorskij Photograph 1910, Emir of Bukhara, Prokudin-Gorskii

43 Examples - Lew Tolstoy 1887 painting, Ilya Repin 1910 photograph, Sergey Prokudin-Gorskii

44 Color Wheel one color channel is captured at one shot 3 times the acquisition time static images only

45 Ways to sense color 3-Chip Camera dichroic mirrors divide light into wavelength bands does not remove light: excellent quality but expensive interacts with lens design problem with polarization image: Theuwissen

46 Foveon Technology 3 layers capture RGB at the same location takes advantage of silicon s wavelength selectivity light decays at different rates for different wavelengths multilayer CMOS sensor gets 3 different spectral sensitivities don t get to choose the curves

47 Ways to sense color Color filter array cover each sensor with an individual filter requires just one chip but loses some spatial resolution demosaicing requires tricky image processing G R B G primary

48 Multi-Shot (example Jenoptik C14) take four images, moving the sensor by one pixel (use fourth image for noise reduction) can be used for supersampling (move by ½, ¼ pixel)

49 Demosaicing bilinear interpolation sampling theory edge-directed/pattern-based interpolation correlation-based

50 Settings: f/8 1/25s ISO: 800, no noise reduction Example raw image (Canon 5D markii) Processed image Raw image

51 Example raw image (Canon 5D markii) Processed image Raw image

52 Example raw image (Canon 5D markii) Processed image Raw image

53 Colors assigned Raw image Processed image Example raw image (Canon 5D markii)

54 Demosaicing Original image Bilinear interpolation Ron Kimmel,

55 Demosaicing Bilinear interpolation Edge-weighted interpolation Ron Kimmel,

56 Bilinear Interpolation G R B G = + + perform interpolation for each color channel separately

57 Bilinear Interpolation G R B G = + + R 23 R 12 R 14 4 R 32 R 34

58 Bilinear Interpolation G R B G = + + R 23 R 12 R 14 4 R 32 R 34 R 33 R 32 2 R 34

59 Bilinear Interpolation set all non-measured values to zero then convolve G R B G = / , B R F 4 / F G

60 Color White Balancing

61 White Balance Colors appear different under different illumination conditions tungsten daylight flourescent flash

62 White Balance Why is there a constant appearance for human observers? Human perception adapts to illumination condition Practice: division of RGB values

63 White Balance Camera built-in function derive scale from white point sun infrared red green blue tungsten incandescent ultra violet wavelength

64 White Balance Camera built-in function derive scale from white point infrared red green blue ultra violet wavelength

65 White Balance Camera built-in function derive scale from white point infrared red green blue ultra violet wavelength

66 Horseshoe Diagram

67 White Point for Different Color Temperatures Planckian Locus: - convert black body temperature to XYZ and put intohorseshoe diagram L_\lambda = spectral radiance [W/m^2/m] lambda = wavelength [m] h = Planck s constant [J.s] k = Boltzmann constant [J/K] c = speed of light [m/s] T = temperature of black body [K]

68 White Balance Human perception adapts to illumination condition Practice: division of RGB values Theory: achieve a neutral spectrum (only works for broad band sources and broad band reflectance) Conversion to RGB is an integral over the divided spectrum + linear transformation + gamma

69 White Balance capture the spectral characteristics of the light source to assure correct color reproduction tungsten daylight flourescent flash

70 Textbook Physical principles color spaces, encoding chromatic adaption, perceptual issues Display technology, color management

71 Compression

72 Compression Lossless Entropy coding based (e.g. Huffmann coding) Popular example: zip/gzip (used in png format) reproduces exact copy of bit signals Lossy Takes advantage of the fact that information is an image (removes inperceptual data) Popular example: jpeg

73 Compression Example: Jpeg Selectable compression ratio Picture: gradually varying compression ratio from left to right [Wikipedia]

74 Compression JPEG: Convert image to YCbCr Subsample chroma channels (Cb,Cr) Split into 8x8 blocks Apply discrete cosine transform Remove small coefficients Color channels are treated independently 8x8 DCT basis

75 Bibliography Holst, G. CCD Arrays, Cameras, and Displays. SPIE Optical Engineering Press, Bellingham, Washington, Theuwissen, A. Solid-State Imaging with Charge- Coupled Devices. Kluwer Academic Publishers, Boston, Curless, CSE558 lecture notes (UW, Spring 01). El Gamal et al., EE392b lecture notes (Spring 01). Several Kodak Application Notes at pplicationnotes.jhtml Reibel et al., CCD or CMOS camera noise characterization, Eur. Phys. J. AP 21, 2003

76 ICC Profiles (ICC international color consortium) color management system capture the properties of all devices camera and lighting monitor settings output properties display device (e.g. monitor) common interchange space srgb standard as a definition of RGB monitor profile input device (e.g. camera) input profile profile connection space output profile output device (e.g. printer)

77 ICC Profiles and HDR Image Generation profile connection spaces CIELAB (perceptual linear) linear CIEXYZ color space can be used to create an high dynamic range image in the profile connection space allows for a color calibrated workflow... input device (e.g. camera) input profile profile connection space output profile output device (e.g. printer)

78 Other HDR approaches Determine for each pixel when enough photons haven been collected. Logarithmic timings yields floating point representation (mantissa + exponent).

79 PMD measured distance in each pixel exploit interference emit light (modulated) at each pixel compare reflected light to reference light computation in a smart pixel

80 Next week Signal Processing

81

82 What is High Dynamic Range (HDR)?

83 HDR Acquisition Exposure Brackets Radiance Map Tonemapped HDR Image Exposure Sequence [Debevec & Malik 97]

84 Ways to vary the exposure Shutter Speed F/stop (aperture) Neutral Density (ND) Filters Gain / ISO / Film Speed (DOF) (noise)

85 Combining the image radiance I ( x) l( x, t) dt scene constant over exposure time (or ND-filter) I ( x) t l( x, ) assumes linear response (radiometric calibration!) have several measurements with different t

86 Combining the image introduce a weighting function for the pixels: centered at the sensor mean value, e.g. Gaussian (image data in [0,1]) w( I ( x)) e ( I ( x) 0.5) compute final image as I final ( x) i w( I i i ( x)) I w( I i i ( x) ( x)) / t i