High Dynamic Range Imaging 1
2
Lecture Topic Discuss the limits of the dynamic range in current imaging and display technology Solutions 1. High Dynamic Range (HDR) Imaging Able to image a larger dynamic range of a scene using multiple photos 2. Tone Mapping or Tone Reproduction Addresses how to display an HDR image Actually doesn t overcome the display range, but produces compelling mappings of HDR that fit into the range of the display We call this process tone mapping 3
Papers We will discuss two papers in this lecture: 1. Paul Debevec and Jitendra Malik Recovering High Dynamic Range Radiance Maps from Photographs - Paul is now a famous Graphics guy - Prof. Malik has been a famous Computer vision guy for years and 2. Erik Reinhard and others Photographic Tone Reproduction for Digital Images - Erik s paper made Tone Mapping a hot topic again - Mainly because Paul made HDR realizable by photographs - Erik recently authored a book on Tone Mapping 4
HDR SIGGRAPH, 1997 Paul Debevec; wrote paper while a student at Berkley, then Research A/Prof and Assoc Director Graphics Lab U of Southern California now, leading Google Daydream Idea Problem with Film and Digital Cameras They have limited dynamic range; have non-linear response curves to exposure (scene radiance) Solution Use multiple photos to recover the radiance of the scene. Require us to compute the non-linear response curve of the imaging device Result: able to determine a high dynamic range of irradiance falling on the sensor s pixel... 5
Scene Radiance Amount of radiance in a 3D scene varies greatly Each point is a different radiance reading 6
Problem Film and digital cameras cannot record the full dynamic range* of the radiance in a scene Fundamental limitation of film Fundamental limitation of CCD sensor *recall: dynamic range is the range of min-radiance to max-radiance 7
Problem Film and digital cameras cannot record the full dynamic range* of the radiance in a scene Fundamental limitation of film Fundamental limitation of CCD sensor Photographer s must make a decision Set the exposure to capture a portion of the range of the scene Exposure too short: low radiance all map to 0 Exposure too long: high radiance all map to 255 (max intensity) *recall: dynamic range is the range of min-radiance to max-radiance 8
Example: Long exposure Real world radiance Picture intensity 10-6 dynamic range 10 6 10-6 10 6 Pixel value 0 to 255 We can see more things in the room, but the scene outside the window is too bright. Short Longer 9
Example: Short exposure Real world radiance Picture intensity Short 10 6 dynamic range 10 6 10 6 10 6 Pixel value 0 to 255 Here we can see outside the window, but the things in the room are too dark. 10
Question Paul addressed: How does radiance map to a pixel value z? 1) 2) 3) 4) Many steps from the scene to the final pixel value z. 5) 6) 7) 12 bits 8 bits 11
Many Steps 1) 2) 3) 4) 1. Scene generates radiance L 2. This can be attenuated through a lens, then hits the imaging devices sensor (now we call it irradiance, E) 3. E is exposed for Δt seconds The product (E Δt) is the exposure 4. Film has a response curve to E Δt This response is often not linear; The development process may also not be linear. 5. If we are using a digital camera, the CCD response is linear! 6. However, this response is quantized 7. And typically (almost always) 6 is remapped through a non-curve to behave like film, so even though the CCD is linear, we get back a non-linear response! 12 bits 8 bits 5) 6) 7) 12
Non-linear Response Curve Film/Digital cameras have non-linear responses in terms of exposure (E Δt) For a variety of reasons (see paper) The question is, can we find the response curve as follows: Z ij = f(e i Δt j ) Where Z ij is the final pixel value (from 0-255) at pixel i, E i is the irradiance at i, and Δt j is the shutter speed. Thus: E i Δt j is the exposure of light on pixel i 13
How to Change Exposure Remember: Ways to change exposure Shutter speed Aperture Natural density filters We will use shutter speed, but there are other options. Follow-on papers used different techniques (filters). 14
Use Shutter speed to control exposure Note: shutter times usually obey a power series each stop is a factor of 2 Camera settings say: ¼, 1/8, 1/15, 1/30, 1/60, 1/125, 1/250, 1/500, 1/1000 sec In reality is: ¼, 1/8, 1/16, 1/32, 1/64, 1/128, 1/256, 1/512, 1/1024 sec 15
Exposure and mapping There is a point where too much exposure saturates the CCD (or film) and we get a peak... (255 white) 16
Varying shutter speeds 17
Saturation, but other pixels OK Not saturated at lower exposures, but other pixels too dark 18
Idea of the paper Recall: E i is irradiance falling on pixel i Δt j is the shutter open time for a setting j Z ij is a pixel response at pixel location i, given exposure time j 1. E (irradiance) doesn t change, it is the same at pixel i in all photo taken of the same scene in the same position 2. Amount of light over time (exposure), E Δt, does change, based on Δt. But we know Δt, it s the shutter speed 3. We also know z, is the pixel value - this is the image we get 4. SO, we need to solve for f, actually, we solve for f -1 Once we have f, we can solve for E (I ve dropped the subscripts): E Δt = f -1 (z) -> E = f -1 (z)/δt 19
Math for recovering response curve This is a regularization term makes the solution smooth where 20
Idea behind the math Pixel x, +, o are 3 different pixels under going 5 exposure levels. The are map to different pixel values due to the E falling on each pixel, and the curve g. g should be a smooth curve. What is unknown? E. Lets adjust the E s so they make a smooth response curve g. 21
Idea behind the math Adjust for o Adjust for x Adjust for + Curve g Adjust E of each pixel so we get a smooth curve g. 22
Recovering response curve The solution can be only up to a scale, add a constraint Add a hat weighting function 23
Recovered response function Recover each R, G, B channel separately. 24
Constructing HDR radiance map combine pixels to reduce noise and obtain a more reliable estimation 25
Reconstructed radiance map This is the radiance map of the scene. Note the range is very detailed (a floating point image). Dynamic range is: 0.005 to 121.741 Assume that 0.005 is the minimum quantization size, Then we have a range of 1 to 24349 much higher than 0-255. 26
What is this for? Human perception Vision/graphics applications 27
A side note: Easier HDR reconstruction raw image = 12-bit CCD snapshot If we could get access to the RAW CCD output, it would be easy to construct an HDR. Just use multiple exposures. RAW CCD response is linear with exposure (more light, more voltage) 28
Easier HDR reconstruction Exposure (Y) This is the response for the RAW CCD output, which responses linearly to exposure. The question is how to get this information? Unfortunately commodity cameras don t allow you access to the direct output. Yij=E i * Δt j Δt 29
HDR Summary This work made HDR practical and popular Debevec s website has many useful links and software Idea is quite simple Use multiple exposure to capture dynamic range Need to overcome cameras non-linear response Mathematical solution provided (code available) Paper is very well done 30
Tone Mapping SIGGRAPH, 2002 Eric Reinhard (German), PhD at Bristol (Britain) This paper while a post-doc at Utah (US) Now back as a Lecturer at Bristol, after short time at U. of Central Florida Idea HDR is nice, but monitor is still limited How can we map the HDR back to finite range Idea considers how real photographers do this. Some parts of his algorithm are inspired from real photographic methods proposed by the famous photography Ansel Adams 31
Motivation Linear remapping of HDR for display. Erik s remapping of HDR for display. 32
Ansels Adams http://en.wikipedia.org/wiki/ansel_adams Famous American photographer (known for high-contrast outdoor scenes). Developed the zone system for photography. 33
Zone System Scene is divided into 11 Zones Each zone represents a level of dynamic range As a photographer, you d like to capture as many zones in your photograph as possible. Adams says you need at least 9 zones to capture the detail of a scene. More than 9 levels, you ll get saturation or a dark image. 34
Zone Approach Photographer selects region that is middle grey (for a given exposure). This is subjective: darker scenes middle grey will be darker than a lighter scene. To describe the scenes we use the term key. Measurement made with a photometer. Photography selects the brightest and darkest regions. Measuring these regions on a photometer gives an estimate of the dynamic range. 35
Controlling range and tricks Using these readings compute range If you have nine zones in your range, you can capture the detail The middle gray should be roughly 18% brightness level of the final output Photography can adjust the middle grey level This card is printed to be gray, the photography adjust the exposure until this gray becomes the desired middle grey. 36
If you can t fit all in Dodge and Burn If the dynamic range is beyond 9 levels We will have regions too dark or too bright in the image You can control the final result through dodging and burning For film photography During development you control the light through the negative, to make parts brighter (burn) or darker (dodging) 37
Terms Recap: Terms used in the paper: Zone: 11 print zones related logarithmically to scene luminance and sensor irradiance. Dynamic Range for Photographers: We can use the zones to calc the difference between highest and lowest scene zones (photographic dynamic range) Key: Subjective measure of light (high key) or dark (low key). Dodging and Burning: Print technique where more light is exposed to a region to dodge or withhold light from that area or burn (darken). 38
Erik s approach Algorithm: Use the log-average luminance to find the "key" of a scene Automatic dodging a burning (as in photography): all portions of the print receive difference exposure time 39
Tone Mapping Log Average: Scale Luminance's to a key: a is called the key value : a=.18 would similar to what Ansels would recommend. L w (x,y) is the world luminance i.e. HDR data 40
Tone Mapping 41
Other mappings Compress the high luminance: Burning high luminance in a controlled fashion: Where L 2 white is the desired max white level. 42
Controlling Max-White 43
Spatially Varying Operator Dodging and Burning Typically applied to regions bounded by large contrasts The size of a local region is estimated using a measure of local contrast; computed at multiple spatial scales At each spatial scale, a center-surround function is implemented by subtracting two Gaussian blurred images. Gaussian profiles are of the form: 44
Spatially Varying Operators Response function of image location, scale, and luminance distribution L: Center-surround function: a = key value, phi is the sharpening parameter Provides a local average of the luminance around (x,y) roughly in a disc of radius s. V 2 operates on a slightly larger area but same scale 45
Spatially Varying Operators 46
Spatially Varying Operators To choose the largest neighborhood around a pixel with fairly even luminance: (start from the lowest scale and stop when this is satisfied) The global operator is converted to a local operator by replacing L with V 1 47
Spatially Varying Operators 48
Example 49
More Results (see slide 6 to compare against Debevec s linear scaling) 50
Scene Radiance Amount of radiance in a 3D scene varies greatly Each point is a different radiance reading 51
Tone Mapping Summary Consider real photographic techniques in mapping radiance to finite range Allows people to think in photographic terms: key, zone, middle grey Introduce local operators to control burn and dodge Overcome effects of saturation or too dark Most previous tone-mapping approaches were for computer generated results not photos 52