High Dynamic Range Imaging

Transcription

1 High Dynamic Range Imaging IMAGE BASED RENDERING, PART 1 Mihai Aldén mihal915@student.liu.se Fredrik Salomonsson fresa516@student.liu.se Tuesday 7th September, 2010 Abstract This report describes the implementation of several HDR imaging techniques for capturing HDR images and displaying them on regular displays using different tone mapping methods. This project is done in the course TNM083 Image Based Rendering. The main purpose of this part of the project is to produce HDR panoramic images that can be used to produce photorealistic renderings of synthetic objects. This project resulted in a simple image processing program that given a set of differently exposed images and the camera response curve can produce HDR images, create panoramas and perform some basic image processing operations.

2 X Response Curve Good data Bad data E No data X Weight Function Figure 1: The response curve and weighting function. W(X) There are currently no commercial devices that can display HDR images since HDR imaging is a relatively new area and not many consumers have heard of or seen this new technique. Therefore it is necessary to prepare these images for display on regular devices using different tone mapping methods. 2 Method 2.1 Creating an HDR image from a set of LDR images 1 Introduction Today, one byte is used for each color channel to represent the color intensity of the captured or synthesized scene, which means that each channel can only have 256 different values and in most scenes this is inadequate, since these images can cover a dynamic range of about 1:100 while a common outdoor scene can have a dynamic range of 1: or more. Therefore to be able to represent the full dynamic range of a scene more than eight bits are needed. Images that can represent the entire dynamic range of a real world scene are called High Dyamic Range HDR images. One of the major issues with HDR images is that there are not so many capturing devices that can actually capture the full dynamic range of a real world scene. Therefore it is quiet common to use a series of differently exposed Low Dynamic Range LDR images, that together cover the full dynamic range, in order to produce the HDR image. These LDR images can be captured using a regular camera. Another considerable issue with HDR images is the storage, since these images contain much more information for each pixel compared to LDR images. The two major file formats that are used today are OpenEXR [2] and RGBE [3]. First step is to capture a set of differently exposed LDR images, keeping the camera and the scene stationary. Knowing the exposure settings of each image (shutter speed, aperture, etc) and the scene is the same in all the images, it is possible to recover the response curve for the camera from the image data Theory This response curve describes the mapping from photometric units E to pixel values X. For most cameras this is a highly non-linear function X = f (E). From the response curve, a range of good values can be selected by using a suitable weight function W(X). And by good values meaning values that are not completely black or white, which are large enough to have a reasonable precision and which are in the range where the slope of the response curve is reasonably large so they have enough accuracy. The weight function is constructed such that good values have a high weight (close to one), bad values have a lower weight (close to zero), completely black or white pixels has zero weight so that they do not influence the final value at all. The response curve and the weight functions are illustrated in figure 1. By mapping the pixel values through the inverse of he response function, it is possible to recover the photometric exposure: E = f 1 (x) (1) 1

3 Different shutter times and adaptive settings influence the exposure values in image i by a scaling factor s i, so the true intensity values can then be recovered by scaling the photometric exposure with the scaling factor: I i = s i E i (2) The final value for each pixel is a weighted sum of the linearized and exposure corrected input values: I = ( i W(Xi )s i f 1 (X i ) ) (3) i W(X i ) In order to calculate the response function we use the HDR Shop program developed by Paul Debevec 1. For more information on how to calculate the response function pleas see [1]. 2.2 Storing HDR images In order to store the HDR images we chose the RGBE file format for its simplicity. The RGBE file format is very compact and flexible because it can be stored in many standard 8-bit file formats by encoding the E channel in the alpha channel. In the RGBE format, 8-bits are used for the mantissa of each color channel and a common 8-bit exponent E is stored as an extra channel which gives a 32-bit per pixel size. 2.3 Tone mapping the HDR input to LDR output using some kind of tone mapping method. Tone mapping methods can be split in two categories: Global and Local tone mapping. Global methods use a monotonic function to map HDR value to LDR values. The properties of the function can depend on some local or global statistics of the HDR data, but the exact same function is applied to all pixels. For more information on local methods please see [4]. We chose to implement a global method commonly refereed to as the S-curve [4], because it is computationally inexpensive and mimics how the human visual system adapts to high dynamic ranges. I = I n I n + σ n (4) This function is shown in figure Panoramic images A common way to capture omnidirectional panoramic images is to photograph a reflective sphere. This process will capture the incident light at a desired point in the scene, where the sphere is placed. By capturing HDR images of the reflective sphere we are able to calculate the correct radiance and later use the information to relight synthetic objects. R N θ θ/2 logi Figure 2: The S-Curve. In order to display HDR images on an regular LDR display it is required to first map Figure 3: Reflected light rays on sphere surface. 1 debevec/ 2

4 Figure 4: Side view of the reflective sphere, image property of [5] Angular mapping A reflective sphere will reflect the entire environment in a single view on its surface. The reason for this can be explained with some basic trigonometry, see figure 3. Before the captured image can be used as a panorama we first need to remap the image through several steps. Pixel coordinates in the image (s, t) are first mapped to the unit circle using the following equtions: s u = 2( 0.5) I width t v = 2( 0.5) I height (5a) (5b) This will ensure that we only sample the reflected image on the sphere and not the background. We assume that the reflected image is centered and that the captured images are uniformly cropped around the sphere. Then the polar coordinate system can be used to find the azimuth angle φ: r = u 2 + v 2 φ = atan( v u ) (6a) (6b) See figure 5, and in order to describe reflected direction from the surface of the sphere into the Figure 5: Front view of the reflective sphere, image property of [5] environment we also need to find the elevation angle θ. In figure 4 the azimuth angle φ = 0, this gives: r = sin( θ 2 ) θ = 2asin(r) (7a) (7b) Unfortunately the sampling of the sphere is non-uniform. This means that the outermost pixels in the image of the reflective sphere will each cover a large solid angle and this makes it difficult to get good data with sharp features for those angles. Therefore it is necessary to remap the sphere image to have a uniform sampling. This is done by replacing θ with the radial distance from the center of the sphere r multiplied 3

5 by π, this will give a linear sampling for the backwards directions, as follows: θ = πr (8) Using these angles it is now possible to go to cartesian (world space) coordinates: x = sinφcosθ y = sinφsinθ z = cosφ (9a) (9b) (9c) and the inverse mapping from cartesian to angular is as follows: r = acos( z) 2π x 2 + y 2 u = 1 2 ry v = rx (10a) (10b) (10c) Equation 8 solves the uniformity problem by avoiding to undersample the edges but due to the spherical geometry very few rays are actually traced backwards and the light information on the edges of the sphere will be mixed together with many different reflections. This will result in a singularity that can be observed in the center of the image. The only way to remove this artifact is to photograph the sphere once more 90 apart and combine the two images together to cover the missing light samples. Therefore if the image is to be viewed directly in 2D it is preferable to shift the singularity to the edges by rotating the coordinate system 90. This can be achieved by modifying the φ to span [ π, π] instead of [0, 2π] Latitude-Longitude mapping The angular map is then mapped into a latitude-longitude map for a more convenient way of using it to relight a synthetic scene. Since the latitude-longitude map stores the angular map s azimuth angles along the horizontal axis and it s elevation on the vertical axis, and also flattens the spherical image into a rectangular area, therefore it is quiet simple to index the latitude-longitude map. The top edge of the map corresponds to the top pole of the sphere, and the bottom edge corresponds to the bottom pole. To map the (u, v) coordinates of the image to cartesian (world space) coordinates we use the following formula, first we remap them to angles that account for the previously stated rotation: φ = πu θ = π v (11a) (11b) then the angles are used to create the cartesian coordinates: x = sinφsinθ y = cosθ z = sinφcosθ (12a) (12b) (12c) these are just the spherical coordinates slightly modified to take inconsideration that the up vector of the reflective sphere is on the y-axis and that the spherical coordinates are defined inside a sphere while the reflective image is seen from the outside Implementation For each sample in the Lat-Long image we need to find the corresponding sample in the angular map. We do this by converting the Lat- Long (u, v) coordinates to cartesian coordinates using equations 13 and 12, then equation 10 is used to calculate angular (u, v) coordinates which are remapped to floating pixel indices. The final pixel value is calculated using bilinear interpolation in the angular map. If the sampling process is not done from the final image (Lat-Long) to the input image (angular map) there is a great risk that not all pixels will be set Cube map Cube maps are today a standard part of the OpenGL API and are typically used to to create reflections from an environment. Cube maps together with HDR data have become very 4

6 common in real-time rendering applications because it is possible to do very realistic lighting of objects at a very cheap cost on any modern GPU. We have chosen to produce cube maps by resampling the latitude-longitude map. To do this we firs create six images and map each image, in 3D space, to the side of the unit cube. Then for each pixel in each image the direction vector from the center of the coordinate system through that particular pixel is calculated and then used to sample the Lat-Long image, see figure 6. The sample positions in the Lat-Long vill almost never be at fixed pixel coordinates, it is therefore necessary to perform a bilinear interpolation to get the final pixel value. Figure 7: LDR image set. Figure 6: Constructing the cube map 3 Results All images are captured using a Canon EOS 20D camera with Canon EF 200mm f/2.8 L. The camera is mounted on a tripod and aimed towards a reflective sphere, in order to capture the surrounding environment. Figure 7 shows different images of the same scen captured with 1/2f-stop apart. Figure 8: Canon EOS 20D Response Curve 3.1 The response curve Figure 8 shows the response curve of the camera we used, calculated using HDR Shop. 5

7 3.2 Tone mapping Figure 9 demonstrates the S-Curve tone mapping method with different σ values and figure 10 demonstrates different exponential values. Figure 12: Motalastrom. Figure 9: Varying intensity. A: σ = 0.01, B: σ = 0.05 and C: σ = 0.2. Figure 10: Varying contrast. A: n = 0.9, B: n = 1.4 and C: n = Figure 13: T appan. Latitude longitude images Below are some latitude longitude images captured at different locations around the LiU Campus area in Norrkoping. Figure 11: K akenhus. 3.4 Cube map Figure 14 shows the six different views of the cube map. Figure 14: Cube map. 6

8 4 Discussion As the results demonstrate it is currently possible to capture high dynamic images using consumer level cameras. However because several images are captured within a small time interval the captured scene needs to be static during that time. This requirement excludes a vast majority of the everyday scenes. Because it is not always possible to control every aspect of an outdoor scene some artifacts will appear in the final HDR image. These artifacts can be noticed as blurred trails behind the clouds in our images. This limitation is one of the major obstacles in making HDR imagine mainstream. Some artifacts in the resulting images are due to irregularities and scratches on the surface of the reflective sphere. The S-Curve tone mapping method works quite well for most images, but it has its limits. References [1] Paul E. Debevec and Jitendra Malik. Recovering High Dynamic Range Radiance Maps from Photographs. University of California at Berkeley, [2] Rod Bogart Florian Kainz. OpenEXR, TechnicalIntroduction. Industrial Light And Magic, [3] Greg Ward Larson. Graphics gems II, Chapter 11.5: Real Pixels. Morgan Kaufmann, [4] Erik Reinhard, Greg Ward, Sumanta Pattanaik, and Paul Debevec. High Dynamic Range Imaging Acquisition, Display, and Image-Based Lighting. Morgan Kaufmann, [5] Jonas Unger, Stefan Gustavson, and Joakim Löw. Light Probes, Panoramas and Image Warping. Linköping University, TNM083 Image Based Rendering, If more time where given we would like to include support for several file formats such as the the Canon raw format CR2 and the HDR format OpenEXR. Add some post processing methods to compensate for small camera movements and moving objects (ghosting). We would also like to be able to overlap two images of the same sphere as described in [4] to remove the singularity effect and the reflection of the camera. 7