Simulating Film Response Curves from a Pair of LDR Images Asla Sa, Luiz Velho, Paulo Cezar Carvalho. Technical Report TR Relatório Técnico

Transcription

1 Laboratório VISGRAF Instituto de Matemática Pura e Aplicada Simulating Film Response Curves from a Pair of LDR Images Asla Sa, Luiz Velho, Paulo Cezar Carvalho Technical Report TR Relatório Técnico September Setembro The contents of this report are the sole responsibility of the authors. O conteúdo do presente relatório é de única responsabilidade dos autores.

2 Abstract HDR Radiance Maps have being widely studied in the last years. In this work we recover HDR Radiance Maps from photographs, using only two shots of the same scene exposed diferently. We also recover film response curves to visualize HDR color images as if they were taken with the photographic film. All the operations used to reconstruct the HDR Radiance Map work on image histograms, the presence of moving objects is allowed with a final effect similar to a motion blur. 1

3 Simulating film response curves from a pair of LDR images. Asla M. Sá, Luiz Velho, Paulo Cezar Carvalho September 28, Introduction In the last years, many fields related to Computer Graphics and Vision have been demanding the capture of real scenes that could be visualized as if they were in an ambient with illumination conditions different from the capture ambient. This problem requires precision in the acquisition of radiance properties of scenes. The available imaging systems have a limited range to record radiance intensities, what limits the precision of measurements in the scene. This limitation has led to the study of the recovery of High Dynamic Range(HDR) Radiance Maps - the dynamic range is the ratio of the highest to the lowest in a set of values. The main goal is to overcome sensor limitations that are not able to register the entire range of radiances present in a real scene. Real scenes can span ten orders of magnitude from star-lit scenes to sun-lit snow, typical sensors and displays spans at best two orders of absolute dynamic range, while human eye can distinguish four orders of magnitude. The research in building HDR images has three main branches: 1) for synthetic images the goal is to simulate the illumination and the light intensity of a virtual scene, commonlly done with renderers like RADIANCE [7]; 2) for real scenes we can recover HDR radiance maps from photographs captured with a common Low Dynamic Range (LDR) imaging systems, or 3) one can focus the attention on the research of sensors and aquisition hardware with augmented sensitivity to radiance values. Here we are interested in the recovery of HDR from LDR photographs, that is, to analize a set of images of a real scene taken with different exposures, with the goal of modeling the behaviour of the imaging system. We point out that, if cameras take multiple pictures in a rapid succession, we can use these algorithms as a way to augment sensor sensibility. The visualization of an HDR image in the low range of a display is a complementary subject of research. It is the problem of mapping scene radiances to display intensities. The challenge is to reduce the dynamic range of an HDR image by a Tone Mappig Operator (TMO) in order to fit in the low dynamic range of the display (that could also be a photographic paper) obtaining a good perceptual result. The film industry has been studying this problem for chemical emulsions since the begining of photography s history. Our goal in this work is to employ some recent results obtained by [6] to recover HDR from photographs using only two differently exposed pictures as input. We take a look at TMO research from the point of view of a photographer that 2

4 Figure 1: Differently exposed pictures from the same scene are processed to compose a HDR radiance Map of the input images. A TMO is then applied to visualize the HDR image. In this case the used TMO is a simulated film response curve. 3

5 wishes to obtain digital pictures with the same look as if a photographic film were used, to achieve that we simulate film response curves as shown in Figure 1. We are going to integrate the pipeline of recovery and visualization unifying and simplifying these complementary problems. We are interested in color images, but many techniques of black-and-white photography can be extended for the color context. The paper is organized as follows: Next session is a background on HDR recovery and TMO research. In section 3 classical photographic techniques for color images are presented. In section 4 we describe the technique proposed in [6] to recover the response curve of an imaging system, and we discuss the method for building an HDR image given an imaging system response curve. In section 5 we describe some TMOs that have been proposed in literature and discuss the acquisition of film response curves. In section 6 we present our pipeline and show our results. Conclusions and future work are in the last section. 2 Background In the literature, the terms used to refer to light measures are ambiguous and frequently unclear. The source of the problem is that two different comunities has been researching on the same subject to different purpouses: the photon based comunity and the eletromagnetic based comunity. The dual behaviour of light is reflected in its nomenclature! In Table 1 the correspondence of terms some is shown. We explain our use of some terms: We call brightness (B d ) the value registered by the imaging system for each pixel (one value per channel). Irradiance (B w) measures the actual light in the scene, the exposure value (E) is a function of irradiance, exposure time and optical properties of the imaging system, it is measured in Lux.sec. Pixels are under or overexposured if their actual value of irradiance in the world was mapped to extreme values of brightness in a LDR image. In computer graphics comunity, we call luminance the achromatic information of brightness given by the standard formula L = 0.299B r B g B b, this is adopted convention here. Geometric Radiometric Photopic Flux Power Luminous Flux W atts Lumens Flux Area Irradiance Illuminance Watts Lumens m 2 m = Lux 2 Flux SolidAngle (Radiant)Intensity (Luminous)Intensity Watts Lumens sr sr = Candela Flux Area.SolAng Radiance Luminance = Nit Watts m 2.sr Candela m 2 Table 1: Dual nomenclature of light measurement found in literature The HDR building problem have been studied by many researchers. The subject became popular after Debevec and Malik published [5] in SIGGRAPH Here we focus our attention on the solution proposed in 2003 by Grossberg and 4

6 Nayar [6]. In their paper, a good overview on recovery of camera response function from images is done and a theoretical discussion on the problem is carried on. They propose a solution to the problem based on image brightness histograms and prove the theoretical and practical advantages of their approach. Once we have the response function of the imaging system we are able to recover the irradiances of the real scene by combining brightness information of correspondent pixels. In using the information of a couple of images to compose a new one, a precise texture alignment has to be available between them. This problem, usually stated as an optimization problem, is known as the correlation problem and is hard to solve. Several file formats have been proposed to store HDR images, and there are at least three of them freely available; the reader is refered to [8]. We do not use any formal format, instead we consider each image channel as a floating point matrix. Finally, once we have a scene with its irradiance values, we need to visualize it on a display of limited range. As we mentioned before, this problem is known as the tone mapping problem. In the Computer Graphics comunity, Tone Mapping Operators (TMO) have been proposed to solve the visualization of synthetic images generated with the RADIANCE renderer, and many TMOs had already been proposed before the recovery of HDR from photographs became popular. This problem has also been intrinsically present in the research of chemical products to be used in film emulsions and photographic paper. 3 Classical Photography Since the begining of its discovery, the design of photographic materials has evolved to the goal of optimal response for human viewing under a variety viewing conditions, and is well known that contrast plays a huge role in achieving good images. All this knowledge can be used in digital context to enhance image quality. The photogaphic comunity is involved in a deep change from chemical to digital photography. In the coming decade HDR will be a reality to photographers and it is already now a field of hot discussion. In the classical photographic process, the film s photosensitive emulsion is exposed to light during a certain period of time (exposure time). The film is then processed to transform the emulsion s latent image into density values. The concept of density is central in photography and relates the incoming and outcoming light. For films it is a transmission ratio D T = log 101/T ; and for photo papers D R = log 101/R is the reflection ratio. Both T and R are in the interval [0, 1]. Films come in two flavors: negative films and chrome (positive) films. Negative films are used as filters of the enlarger light to produce copies in photographic papers. Thus, in a negative image, dark areas represent regions of high illuminance. Negative and chrome films, as well as photographic paper, have many different characteristics varying from sensibility to graininess, and users make their own choice of materials usually based on their experience. Technical information about photographic material is available for reference in data sheets provided by manufacturers. The standard information provided is storage, processing and reproduction information, as well as the technical curves: characteristic curves, spectral sensitivity curves, MTF curve and spectral dye density curve, as shown in Figure 2. 5

7 Figure 2: Film curves. Information from the data sheet provided by FujiFilm about Provia 400 reversal (positive) film. Color films have three layers of emulsion sensitive to distinct wavelength intervals, characterized by spectral sensitivity and spectral dye density curves. The MTF curve is well explained in [3]; it is the Fourier transform of the point spread function (PSF) that gives the scattering response to an infinitesimal line of light and is instrumental in determining the resolution of an emulsion. The characteristic curve of a film is the curve that relates exposure and density. It is in the core of image formation. In photography, dynamic range is given in terms of stops, which is a log 2 scale. Films produce a density range of about 7 stops (that is, 128:1, or two orders of magnitude in base 10). As photo paper has a much lower dynamic range, equivalent to 4 or 5 stops (aproximately 20:1). Several techniques are adopted in the printing process to overcome this limitations and they are a source of inspiration to create TMOs. In [2], for instance, automatic dodging-and-burning is proposed 6

8 and black-and-white development process is simulated in [3]. 4 Building an HDR Radiance Map from photographs In this section we are going to describe the method proposed in [6] and briefly compare it to previous approaches. 4.1 Recovering camera response function The brightness value registered by the sensor in each channel of an image is related to the irradiance in the world by a function f that also varies with the senor s exposure time, that is, if k is the index on exposure times, and E k = B wij t k is the exposure of the image at pixel ij, then: k B dij = f(b wij t k ) where k = 1..N Therefore f 1 k (1) (B dij ) = B wij t k and lnf 1 k (B dij ) = lnb wij + ln t k The function f is the imaging system response curve, which is reasonably assumed to be monotonically increasing; thus, its inverse f 1 is well defined. When we take a set of N pictures of a scene by varying the exposure, we get k a set of B dij brightness values. If we know the pixels correspondence between images we can recover the radiance value that generated those brightness values. If enough different radiance values are mapped to different brightness values - that is, not all the information was clamped - then we can recover f for many values. By imposing some restrictions on f, it can be recovered for all values. In [5] Debevec and Malik assume that t k is known for each image, and solve the problem of finding an smooth g = lnf 1 using an optimization process. To assemble the system to be solved, the images are assumed to be registered and a subset of pixels in the image are chosen to represent the values - if all pixels were taken, the formulated problem would be too large with a lot of redundant information. In [6] Grossberg and Nayar demonstrate a theorem that states that: Given the histogram h 1 of one image, the histogram h 2 of a second image is necessary and sufficient to determine the intensity mapping function τ. This function τ is given by the relation between two corresponding tones in a pair of images: Let 1 B dij = f(b wij t 1) 2 (2) B dij = f(b wij t 2) then B d 1 ij = f ( f 1 (B d 2 ij ) t 2 t 1 ) = f(γf 1 (B d 2 ij )) (3) where γ = t 1 t 2 that is 1 B dij = f(γf 1 2 (B dij )) = 2 τ(b dij ) (4) 7

9 The intensity mapping function τ : [0, 1] [0, 1] is the function that correlates the measured brightness values of two different exposures. It is completely defined by the accumulated histograms of the images τ(b) = H 1 2 (H1(B)), and expresses the concept that the m brighter pixels in the first image will be the m brighter pixels in the second image for all m. In [6] the inverse of the sensor response curve g = f 1 is recovered assuming that g is a sixth order polynomial, and imposing the additional restrictions that no response is observed if there is no light, that is, g(0) = 0. assuming also that f : [0, 1] [0, 1] (that is a convention) we fix g(1) = 1 which means that the maximum scene s irradiance is leds to the maximum brightness response. The system given by equation g(τ(b)) = γg(b), plus the additional restrictions, is then solved on the coefficients of the polynomial. As γ is constant for all pixels, it also can be recovered as shown in [6]. g(τ(b)) = γg(b) g(τ(b)) γg(b) = 0 Considering g a polynomial of nth degree, we have: (5) a n(τ(b i)) n a 1(τ(B i)) + a 0 = γ(a nb n i a 1B i + a 0) a n(τ(b i) γb n i ) n a 1(τ(B i) γb i) + a 0(γ 1) = 0 (6) To satisfy g(0) = 0 we have a 0 = 0; i span the tones present in the image, typically 1 to 256. Then, if we call g t = [a n,..., a 1] and A ij = τ(b i) j γ(b i) j, we have to solve the system Ag = 0 with the aditional restriction of g(1) = 1. Here we assume γ as a known constant value. In Figure 3 we plot the histograms, H 1, H 2, τ and f 1 related to the church images shown in Figure 8 (a) and (b). We explore the robustness of the proposed algorithm to obtain an estimate of g using as input only two images. The main assumption for the pair of images is that the histograms do not change much between them, only a shift in the well exposed part of the histogram is observed. The use of histograms has many advantages: they are less sensitive to noise, all the information present in the image is used instead of a subset of pixels, and the images need not to be registered since the spatial information is not present on histograms. 4.2 Recovering the Radiance Map In the reconstruction phase it is not possible to avoid the correspondence problem: if non correspondent pixels are combined, wrong radiance values will be recovered. In this paper we do not solve the correlation problem. Instead we assume that there is no camera movement between shots and in the presence of small movements in the scene an effect of motion blur is observed as shown in Figure 7 (c). To construct the HDR Radiance Map we apply the recovered f 1 to the values of brightness and obtain B wij for each pair of correspondent pixels. Actually B wij = f 1 k (B dij ) t k and different weights are given according to our confidence on k B dij. If the pixel is almost over or underexposure, a lower weight is given to it, augmenting the influence of the middle of the f curve, where sensors (and films) are well behaved. 8

10 Figure 3: Image histograms, accumulated histograms and the correspondent functions τ and f 1 for the green channel of the church original images - see Figure 8 (a) and (b) in the results Section 5 Tone Mapping Operators (TMO) In the tone mapping phase we adjust the range of image irradiances to display brightness, mantaining color chrominance values. Because dynamic range may vary between channels, if the TMO is applyed in each channel independently, chromatic distortion is likely to occur. To avoid this we can adopt two different approaches: the first would be to calculate the achromatic intensity (luminance) of each pixel in the image, and apply the TMO on the achromatic image; a second approach would be to consider a common range for the three channels, clamping the values that are off this range. This is exactly how LDR images are naturally formed by film emultions. In TMO research three main approaches are adopted, and we now discuss them. 5.1 Arithmetic compression Lots of functions and heuristics can be proposed to compress HDR irradiance s range, and the comparison between them is mainly perceptual, see [1]. We will refer to this engineering approach as arithmetic compression of radiance values. The most simple way to reduce the range of an HDR is by using linear tone mapping, obtained as follows: where R w = (B wmax B wmin ) and R d = (B dmax B dmin ). B d = (Bw Bw min ) R w R d + B dmin (7) 9

11 In [4], Larson et al. propose the histogram adjustment heuristic, that is inspired on the fact that in typical images luminance levels occurs in clusters rather than being uniformly distributed throughout the dynamic range. The algorithm proposed is: where B d = P(B w)r d + B dmin (8) P(B w) = = [ [ h(b i )] b i <Bw h(b i )] b i <Bwmax H(B w) H(B wmax ) The tone mapping problem can be thought of as a quantization problem in irradiance domain and we can interpret the TMOs available in literature from this point of view. The uniform quantization algorithm is equivalent to linear TMO, the populosity quantization is similar to histogram adjustment. The main difference between color quantization and tone mapping is that human eye is more sensitive to spatial changes in luminances than in color; this explains the focus of several TMOs in working on spatial domain, like in human eye perception simulation and in dodging-and-burning algorithm [2]. (9) (a) (b) Figure 4: The figure (a) is visualized using linear TMO and figure (b) using histogram adjustment. In Figure 4, after reconstruct the church HDR image (using the same input of Figure 8) linear TMO was applied to visualize (a) and histogram adjustment was used to visualize (b). We can observe that image (b) is more contrasted than (a). 5.2 Imaging system simulation Imaging systems characteristic curves are responsible for transforming real HDR scenes in an LDR image. The intention behind the simulation of an imaging system, is to change image look to seems like if it were acquired with the simulated system. 10

12 To recover imaging systems curves, one can photograph the desired scene with the chosen film, scan the chromes (or the negatives) and use the algorithm discribed in Section 4 to recover the curve. However, the recovered curve would be a composition of film and scanner curves, and we could not easily separate them. If a digital camera is being used we simply apply the algorithm to the input images to recover camera curve. As we have seen in Section 3, film s response curves are available in film data sheets provided by manufacturers. Typically, this information is given as a graph in bitmap format. In our implementation, we recover curve points with a graphical interface and linearly interpolate the points to aproximate the sampled curve (better interpolation could also be used here). The scale of points is also recovered, picking axes references. We recall that exposure and density are plotted in log scale Fujifilm Velvia 50 3 Kodak Ektachrome Density (D) Fujifilm Provia Kodak Ektachrome Exposure [log H (lux seconds)] Figure 5: Recovered chrome film response curves for the blue channel, with variable ISO. Without loss of generality, we focus our attention on chrome (positive) film curves in our tests. In Figure 5 we plot the recovered curves of the chrome films: Kodak Ektachrome 1600 ISO, Fujifilm Provia 400 ISO, Kodak Ektachrome 100 ISO and Fujifilm Velvia 50 ISO, the difference of their sensitivity to exposure is clear. 5.3 Human eye simulation The behaviour of human eye perception is not the same for all illumination conditions, especially in very dim scenes and when there are abrupt illumination changes in time varying scenes, the reality of visualized images is achieved only if human vision behaviour is modeled. In [4] the histogram adjustment algorithm is extended in many ways to model human vision. We do not simulate this effects since our scenes do not vary in time. 6 Our approach: film simulation We now discribe the pipeline used to process the images and obtain the final visualization choosing the desired TMO. Results are then presented and discussed. 11

13 6.1 The pipeline input: Two LDR images of the same scene with known exposures ratio γ; the camera is supposed to be fixed on a tripod while the scene is free to move (not too fast) between shots. output: An HDR image of the scene and an LDR image simulating the chosen imaging system. 1. Recover f r, f g, and f b response curves, solving the system proposed in sec Apply the recovered f 1 to compose the HDR image, as described in Section Choose and apply TMO to visualize HDR: Reescale linearly the radiance range, or compute the radiance histogram (in 3 channels or achromatic) to compute histogram adjustment, or choose the imaging system response curve and apply it to each pixel. 6.2 Results In our results we explore several ways to vary exposure between a pair of shots. In Fig. 7 we change illumination conditions of the room. In Fig. 8 the church image was downloaded from the web, it is an example of a scene photographed with film and scanned. In Fig. 9 and Fig. 10 we compare the results when two different digital cameras were used to acquire the same scene: a Canon EOS Rebel Digital and a FujiFilm Finepix 2400 zoom camera. In all results, the reported variable t used in the visualization of images do not correspond to real exposure times. This is due to the fact that the real exposure time is not included in our computations, only γ is required. In Fig. 6 we show the luminances of the reconstructed HDR (with input images of Fig. 10). We visualize it with histogram adjustment algotithm to show that in some regions, like in the sky, we do not have information to reconstruct correctly the irradiance value. This is expected since the sky is overexposed in both input images. The same problem can be observed when linear TMO is used, see Figs. 9 (c) and 10 (c). However, it is not a problem for visualization with curve films, if we do not extrapolate excessively the original exposures - the region will be naturally overexposed in the output image. In Fig. 7, the input images were taken with the Canon camera, the camera s set up was left unchanged between shots and the illumination condition was modified. In this case we do not have control of the ratio of exposures and γ is arbitrary. The assumption that the γ is constant for all the pixels is violated, since in shadow regions illumination doesn t change much. This is an inadequate use of the algorithm since we cannot guarantee that radiance values were recovered correctly, but it is interesting to see the obtained result. Motion blur can be observed in the human figure due to scene movement between shots. In Fig. 8, the church - a reference image used in several papers about recovering HDR from photographs - was originally photographed with a Fuji ASA 100 film, developed professionaly and scanned to produce the digital images used here 12

14 Figure 6: Luminances break down [5]. We reconstruct the HDR from the two input images (a) and (b) - out of 16 original ones - and visualize it simulating films. In images (c), (e) and (g), the film simulated was Fujifilm Provia 400, with relative exposure times t equals 1, 1/2, 1/4 respectively. Image (d) simulates Fujifilm Velvia 50 with t = 1. Image (f) uses Kodak Ektachrome 100 curve with t = 1 and, finally, image (h) simulates Kodak Ektachrome 1600 with t = 1/8. In Fig. 9 original images were taken with the Canon camera, we fix f = 1/30 and change the aperture to modulate exposure, the image was stored with 180 dpi, pixels. In Fig. 10 the Fuji camera was used and time was modulated, images were stored with 72 dpi, pixels. Both cameras where setted up for ISO 100 and images were saved in jpeg format. Choosen input images are distant 1f stop between shots. A noticeble noise can be observed in the Fuji images visualization. 7 Conclusion and future works We have shown that with only two input images it is possible to reconstruct irradiance information, and simulate the change of imaging system to visualize it. The histogram approach is fundamental since it uses all the information present in the images in a robust manner. The presence of moving objects between shots is admitted, and the final effect is similar to a motion blur. To complete the simulation of the photographic process we should simulate the behaviour of MTF film curves. All techniques related to enlarging pictures to paper prints could be implemented. The solution of the correspondence problem could free our camera to move between shots. Our goal was to obtain images with a classical photographic look. One can argue that using another set of processing tools the same look coud be achived, but for application in a set of images it could be a hard task, while working in HDR data is much more powerful, and the simulation of the desired system curve is 13

15 (a) (b) (c) Figure 7: Original pictures (a) and (b) were taken with the Canon EOS Digital Rebel, the camera setup was maintained fix and illumination condition was changed. Image (c) is visualized simulating Kodak Ektachrome 100. immediate. Although HDR is the future of image formats, there will be a transition period, where both LDR and HDR will coexist, and algorithms to compatibilize their look are welcome. References [1] F. Drago, W.L. Martens, K. Myszkowski and H.P. Seidel, Perceptual Evaluation of Tone Mapping Operators with Regard to Similarity and Preference, Tech. Repport MPI-I (2002). [2] E. Reinhard, M. Stark, P. Shirley and J. Ferwerda, Photographic Tone Reproduction for Digital Images, Proc. ACM SIGGRAPH (2002). [3] J. Geigel and F.K. Musgrave, A Model for Simulating the Photographic Development Process on Digital Images, Proc. ACM SIGGRAPH - pp (1997) [4] G.W. Larson, H. Rushmeier and C. Piatko. A Visibility Matching Tone Reproduction Operator for High Dynamic Range Scenes, IEEE Trans. on Vis. and Comp. Graph., 3(4): (1997) [5] P.E. Debevec and J. Malik, Recovering High Dynamic Range Radiance Maps from Photographs, Proc. ACM SIGGRAPH - pp (1997). 14

16 [6] M.D. Grossberg and S.K. Nayar, Determining the Camera Response from Images: What Is Knowable?. IEEE Trans.PAMI vol.25 no.11 (2003). [7] [8] gwlarson/pixformat/ 15

17 (a) (b) (c) (d) (e) (f) (g) (h) Figure 8: Original pictures (a) and (b) were used to construct the HDR image, and then visualized with different film curve in (c) to (h). The used curves are: (c)fujifilm Provia 400 (t=1), (d)fujifilm Velvia 50 (t=1), (e)fujifilm Provia 400 (t=1/2), (f)kodak Ektachrome 100 (t=1), (g)fujifilm Provia 400 (t=1/4), (h)kodak Ektachrome 1600 (t=1/8). 16

18 (a) (b) (c) (d) (e) (f) Figure 9: Original pictures (a) and (b) were taken with Canon EOS Digital Rebel and the results were visualized with (c) linear 17 TMO, (d) Fujifilm Velvia 50 (t=8), (e) Fujifilm Provia 400 (t=1) and (f) Kodak Ektachrome 1600 (t = 1/2).

19 (a) (b) (c) (d) (e) (f) Figure 10: Original pictures (a) and (b) were 18 taken with FujiFilm Finepix 400 zoom and the results were visualized with (c) linear TMO, (d) Fujifilm Velvia 50 (t=8), (e) Fujifilm Provia 400 (t=1) and (f) Kodak Ektachrome 1600 (t = 1/2).