High Dynamic Range Image Reconstruction with Spatial Resolution Enhancement

Transcription

1 The Computer Journal Advance Access published October 4, 2007 # The Author Published by Oxford University Press on behalf of The British Computer Society. All rights reserved. For Permissions, please journals.permissions@oxfordjournals.org doi: /comjnl/bxm080 High Dynamic Range Image Reconstruction with Spatial Resolution Enhancement JONGSEONG CHOI, MIN KYU PARK AND MOON GI KANG * Institute of TMS Information Technology, Yonsei University, Seoul, Korea *Corresponding author: mang@yonsei.ac.r For the last two decades, two related approaches have been studied independently in conjunction with limitations of image sensors. The one is to reconstruct a high-resolution (HR) image from multiple low-resolution (LR) observations suffering from various degradations such as blur, geometric deformation, aliasing, noise, spatial sampling and so on. The other one is to reconstruct a high dynamic range (HDR) image from differently exposed multiple low dynamic range (LDR) images. LDR is due to the limitation of the capacitance of analogue-to-digital converter and the nonlinearity of the imaging system s response function. In practical situations, since observations suffer from limitations of both spatial resolution and dynamic range, it is reasonable to address them in a unified context. Most super-resolution (SR) image reconstruction methods that enhance the spatial resolution assume that the dynamic ranges of observations are the same or the imaging system s response function is already nown. In this paper, the conventional approaches are overviewed and the SR image reconstruction, which simultaneously enhances spatial resolution and dynamic range, is proposed. The image degradation process including limited spatial resolution and limited dynamic range is modelled. With the observation model, the maximum a posteriori estimates of the response function of the imaging system as well as the single HR image and HDR image are obtained. Experimental results indicate that the proposed algorithm outperforms the conventional approaches that perform the HR and HDR reconstructions sequentially with respect to both objective and subjective criteria. Keywords: resolution, dynamic range, sampling, quantization, reconstruction Received 15 May 2006; revised 22 August INTRODUCTION Recently, digital imaging systems with CCD and CMOS imaging sensors have widely been used for various purposes such as medical tomography, industrial monitoring system, surveillance system, computer vision system, scientific research applications, broadcasting system, consumer appliances, etc. Although imaging sensors are widely used and suitable for most digital imaging applications, they have physical limitations such as limited spatial resolution and dynamic range. Figure 1 shows the various physical limitations of imaging sensors [1]. The spatial resolution that represents the number of pixels per unit area in an image is the principal factor to determine the quality of an image. However, because an imaging sensor is the composition of a number of photodiodes, spatial resolution of the sensor is limited. Besides, in the process of recording a digital image, there is a natural loss of spatial resolution caused by the optical distortions (out-of-focus, diffraction limit, etc.), motion blur due to limited shutter speed and noise, which occurs within the sensor or during transmission. Thus, the recorded image usually has the limited spatial resolution. Therefore, highresolution (HR) images should be considered not only to have higher spatial density of pixels but also to be visually sharper and clearer than low-resolution (LR) images. Since an HR image that has more details may be critical in various applications than LR images, the reconstruction of an HR image from multiple LR images is needed. Low dynamic range (LDR) is also one of the main problems when an image is acquired from imaging sensors. The light intensity of natural scenes has a very wide range, whereas the dynamic range of an imaging sensor is limited.

2 Page 2 of 12 J. CHOI et al. SR reconstruction algorithm that simultaneously enhances the spatial resolution and dynamic range is proposed. The image degradation process including limited spatial resolution and limited dynamic range is modelled and the maximum a posteriori (MAP) estimates of both the response function of the imaging system and a single image with HR and HDR are obtained. In the reconstruction process, the response function of the imaging system is updated in every iteration step. The rest of this paper is organized as follows. The previous wors to overcome the limitations mentioned above (i.e. the limited spatial resolution and limited dynamic range) are reviewed in Section 2 and a novel SR approach is proposed in Section 3. In Section 4, experimental results are presented and conclusions are drawn in Section 5. FIGURE 1. Physical limitations of imaging sensors. This results in the presence of no response with low exposure and saturation with high exposure. When a scene with a very wide range of light intensity that exceeds the dynamic range of the sensor is captured, it is inevitable that there is a loss of information in either the low-intensity areas, the high-intensity areas or both of them. The limitation of the analogue-to-digital converter (ADC) s capacitance is one reason for LDR. If the ADC of the imaging system has small capacitance, the quantization interval of the system response function becomes wider and it generates a loss of light intensity information. Therefore, the high dynamic range (HDR) imaging system is needed. Owing to the demand and need for HR and HDR images, many researches have been studied to overcome LR and LDR [2 32]. The researches to overcome the spatial resolution and dynamic range limitations are categorized into two groups. The first one is based on device physics and circuit technology, and the other one includes signal processingbased approaches. In device-based approaches, the most direct solution to increase spatial resolution is to increase the number of pixels per unit area by sensor manufacturing techniques. According to the decrease of the amount of light per pixel, shot noise that severely degrades the image quality is generated. There are several signal processing techniques to obtain an HR image from observed multiple LR images, and it is called super-resolution (SR) image reconstruction in the literature [2 17]. To enhance the dynamic range, the new architectures of image sensors have been proposed, and the algorithmic techniques to increase the dynamic range of an image using differently exposed multiple frames of the same scene have been researched [19 30]. The signal processing approaches to overcome the limitation of dynamic range are also called SR image reconstruction in wide sense. The SR image reconstruction is proven to be useful in many practical cases. In this paper, the conventional approaches to enhance the spatial resolution and dynamic range are reviewed, and an 2. PREVIOUS WORK ON SPATIAL RESOLUTION AND DYNAMIC RANGE ENHANCEMENT 2.1. Spatial resolution enhancement In most electronic imaging applications, images with HR are desired and often required [3]. HR means that pixel density within an image is high, and, therefore, an HR image can offer more details that may be critical in various applications. For example, HR medical images are very helpful for a doctor to mae a correct diagnosis. It may be easy to distinguish an object from similar ones using HR satellite images, and the performance of pattern recognition in computer vision can be improved an HR image is provided. Although imaging sensors are suitable for most imaging applications, the current resolution level and consumer price will not satisfy the future demand. For example, people want an inexpensive HR digital camera/camcorder, and scientists often need a very high-level resolution close to that of an analogue 35 mm, film which has no visible artefacts when an image is magnified. Thus, finding a way to increase the current resolution level is needed. The most direct solution is to increase the number of pixels per unit area by sensor manufacturing techniques. As the pixel size decreases, however, the amount of light available also decreases. It generates shot noise that degrades the image quality severely. Therefore, there is a limitation on the pixel size reduction. Another approach for enhancing the spatial resolution is to increase the chip size, which leads to an increase in capacitance. Since large capacitance maes it difficult to speed up a charge transfer rate, this approach is not considered effective. The high cost for high precision optics and image sensors is also an important concern in many commercial applications regarding HR imaging. Therefore, a new approach towards increasing spatial resolution is required to overcome these limitations of the sensors and optics manufacturing technology. One promising approach is to use signal processing techniques to obtain an HR image (or sequence) from the observed multiple LR images. Recently, such a

3 HDR IMAGE RECONSTRUCTION WITH SPATIAL RESOLUTION ENHANCEMENT Page 3 of 12 resolution enhancement approach called as SR image reconstruction has been one of the most active research areas. The basic premise for increasing the spatial resolution in SR techniques is the availability of multiple LR images captured from the same scene. In SR, LR images typically represent different loos at the same scene. That is, LR images are subsampled and aliased as well as shifted with subpixel precision from an HR image. If the LR images are shifted by integer units, each image contains the same information, and thus there is no new information that can be used to reconstruct an HR image. However, if the LR images have different subpixel shifts from each other and if aliasing is present, each image cannot be obtained from the others. In this case, the information contained in each LR image can be exploited to obtain an HR image. To obtain different loos at the same scene, some relative scene motions must exist from frame to frame via multiple frames or video sequences. Multiple frames can be obtained from one camera with several captures or from multiple cameras located in different positions. These scene motions can occur due to the controlled motions in imaging systems, e.g. images obtained from orbiting satellites, and the uncontrolled motions, e.g. movement of local objects or vibrating imaging systems. If the scene motions are nown or can be estimated within subpixel accuracy, the SR image reconstruction is possible as illustrated in Fig. 2. There are various approaches for SR reconstruction. Nonuniform interpolation approach [4 6] is the most intuitive method and the frequency domain approach [7 10] maes explicit use of the aliasing that exists in each LR image. Deterministic and stochastic regularization [11 13], projection onto convex sets approach [14 16] and ML-POCS approach [17] are other methods of SR reconstruction. An example illustrating the improvements of resolution through SR reconstruction appears in Fig. 3. Figure 3a c shows a frame of LR images (nearest interpolated), a conventional bicubic interpolated and an SR reconstructed image by MAP estimation based on deterministic regularization approach [13], FIGURE 2. Basic premise for SR. FIGURE 3. Simulation results of SR reconstruction. (a) One frame of LR images (nearest neighbor interpolated). (b) A bicubic interpolated image. (c) An SR reconstructed image. respectively. The SR image reconstruction shows significantly improved image details Dynamic range enhancement The human visual system exhibits an enormous optical dynamic range of 200 db from the scotopic threshold to the glare limit, as it can adapt to an extremely high level of light intensity [18]. This capability is also required in numerous imaging applications. Imaging sensors, however, have a narrow limited dynamic range comparing with the human visual system. When capturing a scene with a very wide intensity range of light that exceeds the dynamic range of an imaging sensor, it is inevitable to lose detail in either the dar areas, the bright areas or both. By controlling either exposure time, aperture size or both, we can choose the luminance level of the captured image. For example, with increased exposure time or aperture size, we can obtain a better representation of dar areas at the cost of losing information in bright areas. However, adjusting the exposure

4 Page 4 of 12 time or aperture size does not help to represent both low- and high-light areas of the scene with HDR. Details will be definitely lost, and varying the exposure time just allows some control over where the loss occurs. Therefore, the novel architectures of image sensors and the algorithmic techniques to increase the dynamic range of images have been proposed [19 30]. Fabrication of multiple sensor elements within a pixel is a solution of the new architectures. Each pixel includes two or more photo detectors of different size [19, 20]. When an image is captured, each measurement is made in each photo detector on the same pixel and the HDR pixel values are computed on chip. Another approach is a fully new architecture, which is thoroughly different from CCD or CMOS image sensors [21]. Each pixel on the device includes an additional computation element that measures the time to attain full potential well capacity. Since the full potential well capacity is the same, the time to attain full potential well capacity is proportional to the radiance value of the pixel. The signal processing-based approaches have a common feature of using multiple frames with different exposures. Figure 4 shows an example of multiple frames with different exposures. The simplest method to obtain the frames is sequentially capturing a scene with different exposures [22, 23]. The exposure for each frame can be controlled by varying the lens aperture, exposure time or both. Multiple image detector is proposed as another method for capturing multiple frames by Doi et al. [24] and Aggarwal and Ahuja [25]. They proposed a method to capture multiple images using only one imaging optics. This method generates multiple copies of the optical image using a mirror or a beam-splitter and captures them using multiple image detectors. The advantage of this method is that it can be applied for a scene with motion and produce the HDR images in real time. The J. CHOI et al. FIGURE 5. Spatially varying pixel exposures [26]. reconstruction method proposed by Aggarwal and Ahuja [25] produces the HDR image by transforming the recorded intensities into the actual luminance values. This is possible only when the response function of the sensor is nown. Nayar and Mitsunaga [26] proposed spatially varying pixel exposures as an alternative method. The ey feature of this method is the simultaneous sampling of the spatial domain and the exposure domain using a pattern masing as shown in Fig. 5. The brightness level associated with each pixel represents its sensitivity. The pixel e 3 is the most sensitive pixel, and the pixel e 0 is the most insensitive pixel. They proposed the HDR images reconstruction by aggregation and interpolation. However, most of the previous approaches using multiple frames have several limitations such as the assumption that the image capture devices have a linear response [23, 27] and there is no saturation even for highest exposure. Debevec and Mali [22] proposed an algorithm considering the nonlinearity of the response function and saturation of the pixel value for the highest exposure. Figure 6 shows the FIGURE 4. An example of multiple frames with different exposures. FIGURE 6. Experimental result of HDR reconstruction. (a) An under-exposed image. (b) A properly exposed image. (c) An overexposed image. (d) The reconstructed HDR image using multiple LDR images.

5 HDR IMAGE RECONSTRUCTION WITH SPATIAL RESOLUTION ENHANCEMENT Page 5 of 12 experimental results of dynamic range enhancement. Figure 6a c shows an under-exposed image, a properly exposed image and an over-exposed image, respectively. Figure 6d is a reconstructed HDR image with Fig. 6a c. We can see that both dar and bright regions are well represented Simultaneous enhancement of spatial resolution and dynamic range Since the image sensors were invented, the enhancements of spatial resolution and dynamic range have been researched independently. However, in many practical situations, since observations suffer from limitations of both spatial resolution and dynamic range, it is necessary to reconstruct the HR and HDR images at once. Recently, the approaches, which simultaneously enhance the spatial resolution and the dynamic range, have been proposed [31, 32]. On the basis of the spatially varying pixel exposures, the algorithm that enhances the resolution of multisampled images was proposed by Narasimhan and Nayar [31]. Multisampled images mean that a scene is sampled in multiple dimension such as space, time, spectrum and brightness. Guntur and Gevreci [32] proposed the MAP approach, which enhances the spatial resolution and the dynamic range with prior nowledge of imaging system s response function. In the following section, an SR reconstruction algorithm that simultaneously enhances the spatial resolution and dynamic range without any prior nowledge about the system response function is proposed. 3. SIMULTANEOUS RECONSTRUCTION ALGORITHM TO ENHANCE SPATIAL RESOLUTION AND DYNAMIC RANGE 3.1. Problem formulation The proposed reconstruction algorithm simultaneously reconstructs the response function of imaging system and the HR and HDR images without any prior nowledge about the system response function. For this purpose, the image degradation process including limited spatial resolution and limited dynamic range is modelled. Among various observation models to represent the limited dynamic range, we use the reciprocity, which means that the exposure is defined as the product of the luminance r at the surface of an imaging sensor and exposure time Dt [28, 29]. In addition, Hurter Driffield curve is used as the response function of an imaging system [22], which curve is a graph of the optical density of the imaging sensor against the logarithm of the exposure rdt. The typical shape of Hurter Driffield curve is shown in Fig. 7. Consider the desired HR and HDR images of size L 1 N 1 L 2 N 2 written in lexicographical notation as the vector x ¼ [x 1, x 2,..., x N ] T, where N ¼ L 1 N 1 L 2 N 2. Namely, x is an ideal undegraded image that is sampled at or above the FIGURE 7. Typical shape of Hurter Driffield curve. Nyquist rate from continuous scene which is assumed to be bandlimited. Now, let the parameters L 1 and L 2 represent the downsampling factors in the observation model for the horizontal and vertical directions, respectively. Thus, each observed LR image and LDR image is of size N 1 N 2. Let the th LR and LDR images of p frames be denoted in lexicographic notation as y ¼ [y,1, y,2,..., y,m ] T, for ¼ 1, 2,..., p and M ¼ N 1 N 2. Now, it is assumed that x remains constant during the acquisition of the multiple LR and LDR images, except for any motion and degradation allowed by the model. Then, the observed th LR and LDR image captured with the exposure time Dt results from warping, blurring, subsampling operators and system response function performed on the HR and HDR images x. Assuming that each LR image and LDR image is corrupted by additive noise and quantization error, we can then represent the observation model as y ¼ f ðdt W x þ n a Þþnq for ¼ 1;...; p; ð1þ where the matrix W of size (N 1 N 2 ) 2 L 1 N 1 L 2 N 2 represents, via subsampling, blurring and motion, the contribution of HR and HDR pixels in x to the LR and LDR pixels in y, n a is a lexicographically ordered noise vector and n q represents a lexicographically ordered quantization error vector. A bloc diagram for the observation model is illustrated in Fig. 8. FIGURE 8. The bloc diagram of the observation model.

6 Page 6 of 12 The nonlinear function f includes the characteristics of image capture devices and quantization process during the image acquisition. Generally, the function f for an 8-bit system has the form as 8 < 0 for z [ ½0; I 0 Š; f ðzþ ¼ m for z [ ði m 1 ; I m Š; : 255 for z [ ði 254 ; 1Þ; where I m21 is the lower bound and I m is the upper bound of the mth quantization interval. To reconstruct the HR and HDR images based on the proposed image formation model, the inverse function of f is required. Even if it is a reasonable assumption that the response function of an imaging system is monotonically increasing, the inverse function f 21 does not exist because f is a many-to-one mapping due to the quantization process. Therefore, the reverse mapping function g is defined as the function, which maps the pixel intensity to the mid-level of the corresponding quantization interval, and used instead of f 21. It is a reasonable assumption that g is monotonically increasing and differentiable. With the definition of the reverse mapping function g and Taylor series expansion, Equation (1) can be rewritten as ð2þ gðy Þ¼gð f ðdt W x þ n a Þþna Þ Dt W x þ n a þ g0 ðy n q Þnq ; ð3þ where g 0 is the first-order derivative. The approximate equality of the second line holds by ignoring the higher-order terms above the second order. Because, the quantization error n q is relatively small compared with y, it is reasonable that Equation (3) is approximated as gðy Þ Dt W x þ n a þ g0 ðy n q Þnq ; Dt W x þ n a þ g0 ðy Þn q ¼ Dt W x þ n t ; where n t ¼ n a þ g 0 (y ).n q is the total noise term including the additive noise and quantization error. Accurately characterizing the total noise n t would be extremely difficult, as it would require detailed nowledge of the specific image capture device being used. However, with the general assumption that the additive noise n a and the quantization error n q are the independent zero-mean Gaussian random vectors whose elements have common variances s 2 a and s 2 q, respectively, the total noise of the ith pixel in the th frame also has the zero-mean Gaussian distribution with the variance s 2,i ¼ s 2 a þ g 0 (y,i ) 2.s 2 t q. Therefore, the total noise n is modelled as zero-mean independent Gaussian random variables with covariance matrix C n, which is a diagonal matrix, and its diagonal elements are equal to s 2,i. ð4þ J. CHOI et al Reconstruction of the HR and HDR images and the reverse mapping function Because the reverse mapping function g is unnown as well as the HR and HDR images x, x and g should be simultaneously reconstructed. To solve this problem, Gauss Seidel relaxation is used to determine the solution for both x and g. Gauss Seidel relaxation minimizes an objective function with respect to a single variable, and then uses these values when minimizing with respect to subsequent variables. Using the total noise model and a Gaussian prior for x with mean vector m x and covariance matrix C x, the problem is formulated as finding MAP estimates of the reverse mapping function g and the HR and HDR images x that minimize the objective function given as EðxÞ ¼ Xp ¼1 ½gðy Þ Dt W xš T C 1 n ½gðy Þ Dt W xš þðx m x Þ T C 1 x ðx m xþ: ð5þ With the assumption that there is no spatial correlation among pixels in x, C x is a constant diagonal matrix of which diagonal elements are equal to the local variances of the x. First, the gradient descent method is used to solve for x minimizing Equation (5). The estimated x at the nth gradient descent iteration is x ðnþ ¼ x ðn 1Þ areðx ðn 1Þ Þ; where a is the step size, and re(x) is derived from Equation (5) as reðxþ ¼ Xp ¼1 Dt W T C 1 n ðgðy Þ Dt W xþ ð6þ þ C 1 x ðx m xþ: ð7þ The step size a of the gradient descent can be fixed or updated adaptively during the iterations. By Gauss Seidel relaxation, the reverse mapping function g is reconstructed with the solution x of the gradient descent minimization. Generally, reconstruction of g only requires recovering the finite number of values, since the values of the pixel brightness are finite. For example, the 256 values g(z) for z ¼ 0,..., 255 are needed to be recovered for 8-bit system. However, because Hurter Driffield curve is a simple S-curve and g is the reverse mapping of Hurter Driffield curve including the quantization, g can be reconstructed using the polynomial fitting method. Let us assume that g is the third-order polynomial function with coefficients a 0, a 1,

7 HDR IMAGE RECONSTRUCTION WITH SPATIAL RESOLUTION ENHANCEMENT Page 7 of 12 a 2 and a 3 defined as gðy Þ¼a 0 i þ a 1 y þ a 2 y ð2þ ; ð8þ where i is the M 1 vector, which elements are all 1, y (2) ¼ [y 2,1, y 2 2,2,..., y,m ] T, y (3) ¼ [y 3,1, y 3 3,2,..., y,m ] T, for ¼ 1, 2,..., p. There are two additional constraints to mae g maintain the shape of typical inverse Hurter Driffield curve. The one is that g(z) should be a monotonically increasing function within the interval 0 z 255. The other one is that g(128) should be zero. With Equation (8), Equation (5) can be rewritten as EðxÞ ¼ Xp ¼1 ½ða 0 i þ a 1 y þ a 2 y ð2þ Dt W xþ T C 1 n ða 0i þ a 1 y þ a 2 y ð2þ Dt W xþš þðx m x Þ T C 1 x ðx m xþ: ð9þ To minimize Equation (9) with respect to a 0, the partial derivative of Equation (9) with respect to a 0 is taen and set equal to 0 ¼ Xp ¼1 i T ða 0 i þ a 1 y þ a 2 y ð2þ Dt W xþ¼0: ð10þ Similarly, the minimizations with respect to a 1, a 2 and a 3 are also calculated 3 ¼ Xp ¼ Xp ¼1 y T ða 0i þ a 1 y þ a 2 y ð2þ Dt W xþ¼0; ¼1 ðy ð2þ ÞT ða 0 i þ a 1 y þ a 2 y ð2þ Dt W xþ¼0 ¼ Xp ¼1 ðy ð3þ ÞT ða 0 i þ a 1 y þ a 2 y ð2þ Dt W xþ¼0: ð11þ ð12þ ð13þ From Equations (10) (13), the polynomial coefficients are solved as 0 a 0 a 1 a 2 a P p ¼1 Dt 1 P i T W x p C A ¼ A 1 P ¼1 Dt y T W x B p ¼1 Dt P ðy 2 ÞT W x A ; p ¼1 Dt ðy 3 ÞT W x ð14þ where A is a 4 4 matrix of which the ijth element e ij is expressed as e ij ¼ Xp ¼1 ðy ðiþ ÞT y ð jþ for i; j ¼ 1;...; 4; ð15þ where y (0) ¼ i and y (1) ¼ y. The matrix A is calculated only once before the iteration process. With the calculated polynomial coefficients, g is reconstructed, and another gradient descent iteration is started with the reconstructed g by Gauss Seidel relaxation Algorithm for implementation In the reconstruction, the other parameters but the HR and HDR images x and the reverse mapping function g should be either nown or estimated in advance. The matrix W requires information about the subsampling factor, the point spread function of blurring and the motion information with subpixel precision. The exposure time Dt is another required parameter. In this paper, the subsampling factor, the point spread function, the motion information and the exposure times of each frames are assumed to be nown values. The algorithm starts with initial estimates of x and g. The algorithm for implementation is shown below. (i) Initialization of x (0) with a bilinearly interpolated LR image and LDR image. (ii) Calculation of the matrix A 21. (iii) Polynomial fitting as calculating coefficients a 0, a 1, a 2 and a 3 in g (n) with x (n21). (iv) Gradient descent iteration for the reconstruction of x (n) with g (n). (v) If the stopping criterion is satisfied, terminate iterations. If not, go to (iii). 4. SIMULATION RESULTS In order to demonstrate the performance of the algorithm, several experimental results are presented here. The simulations are performed on two approaches. The first experiment is the reconstructing an HR image and HDR image from multiple LR and LDR images which are shifted, blurred and downsampled from the LDR images with different exposure times. In order to measure the quantitative improvement in the reconstructed images, the other experiment is the

8 Page 8 of 12 J. CHOI et al. FIGURE 10. The reconstructed reverse mapping function using the conventional and the proposed method. smoother than the result of the conventional method, the proposed method reconstructs a more accurate HR image and HDR image. The reconstructed HR image and HDR memorial image is shown in Fig. 11. Since the reconstructed values are not integers but floating point numbers that exceed the dynamic range of the traditional image format, it cannot directly be displayed. In this experiment, the dynamic range of the reconstructed HR FIGURE 9. Four LDR memorial images. reconstructing an HR image and HDR image from multiple LR and LDR images which are shifted, blurred, downsampled and quantized into 6-bit grey level from one 8-bit gray level image. Since g converges very fast in comparison with the convergence of x, the criterion x nþ1 2 x n 2 /x n was used for terminating the iteration in both approaches. To acquire 16 LR and LDR images, each frame of four memorial images as shown in Fig. 9 are shifted with four different subpixel motion information, blurred with 3 3 Gaussian ernel whose variance is 1.0 and downsampled with a factor of 2 in each dimension. The reverse mapping function g is simultaneously estimated with the reconstruction of a single image with HR and HDR. Figure 10 presents the estimated g by the the conventional method and the proposed method from memorial images. Conventional method proposed by Robertson et al. [30] reconstructs the 256 values g(z) for z ¼ 0,..., 255 for an 8-bit system in advance. Compared with the result of conventional method, the estimated g by the proposed algorithm well maintains the typical form of the inverse Hurter Driffield curve. Since g taes the role of mapping pixel values to HDR radiance values, the fluctuation of g can cause the inversion of the local contrast of the reconstructed HR and HDR images. Since the reconstructed g of the proposed algorithm is FIGURE 11. The reconstructed HR and HDR images using the proposed method.

9 HDR IMAGE RECONSTRUCTION WITH SPATIAL RESOLUTION ENHANCEMENT Page 9 of 12 and HDR images was compressed with a simple log-scale mapping. The proposed algorithm is compared with two conventional algorithms. The first conventional method sequentially enhances the dynamic range and spatial resolution, and the other conventional method simultaneously enhances. First, the four LR and HDR images were reconstructed from four LR and LDR images with the same subpixel motion information. Then, HR image is reconstructed with a conventional SR image reconstruction, which uses MAP estimates based on deterministic regularization approach [13] (CM1). The other conventional method simultaneously enhances the spatial resolution and the dynamic range with reconstructed imaging system s response function in advance [32] (CM2). The partially magnified results are shown in Fig. 12. Figure 12a shows a frame of LR images (nearest neighbour interpolated). Figure 12b and c shows the results of CM1 and CM2, respectively. Figure 12d is the reconstructed HR and HDR images of which the spatial resolution and the dynamic range are simultaneously enhanced by the proposed methods. The proposed reconstruction method shows improved image details compared to the results of CM1 and CM2. The other experiment is the reconstruction from multiple LR and LDR images, which are generated with an original 8-bit grey level image. The 36 LR and LDR images were produced from an 8-bit grey level image. Figure 13a and b is the original images of des and sunrise, respectively. The original image is shifted with nine different subpixel motion FIGURE 13. The original HR and HDR images. (a) Des image. (b) Sunrise image. information, blurred with 3 3 Gaussian ernel, downsampled with a factor of 4 in each dimension, and re-quantized to 6-bit grey level with four different exposure times. Figure 14 shows the examples of the produced differently exposed LR des images. The reconstructed HR and HDR des and sunrise image are presented in Fig. 15 and the partially magnified results of des image are shown in Fig. 16. Figure 16a shows one nearest interpolated frame of LR images. Figures 16b and c shows the result of CM1 and CM2, respectively. Figure 16d is the simultaneously reconstructed HR and HDR images by the proposed methods without prior nowledge about the FIGURE 12. Partially magnified results of memorial using the conventional and the proposed methods. (a) One frame of LR images (nearest neighbour interpolated). (b) The result of CM1. (c) The result of CM2. (d) The result of the proposed method. FIGURE 14. Four examples of LR and LDR des images.

10 Page 10 of 12 J. CHOI et al. FIGURE 16. Partially magnified results of des using the conventional and the proposed methods. (a) One frame of LR images (nearest neighbour interpolated). (b) The result of CM1. (c) The result of CM2. (d) The result of the proposed method. TABLE 1. PSNR of the reconstructed images PSNR (db) CM1 CM2 Proposed Des Sunrise FIGURE 15. The reconstructed HR and HDR images using the proposed method. (a) Des image. (b) Sunrise image. imaging system. The proposed method shows improved image details compared to the CM1 and CM2. In order to measure the quantitative improvement in the estimated images, the pea signal-to-noise ratio (PSNR) values for results with des and sunrise image are measured. The PSNR is defined in decibels as PSNR ¼ 10 log N ^x x 2 ; ð16þ where N is the total number of the pixels in the image, x the original image and xˆ the reconstructed image. In Table 1, the PSNR values calculated between the reconstructed images and the original image are shown in order to compare the performance of the proposed algorithm to the conventional methods. The PSNR of the proposed algorithm improves 4 and 1.5 db over the conventional methods, respectively. 5. CONCLUSION HR and HDR images can be used in many fields. Since an HR image and HDR image has more detail and light intensity information than an LR and LDR image, it is very useful in the fields such as biometrics, medical imaging and astronomical imaging. Spatial resolution and dynamic range enhancement based on SR approach has the advantage that it can be applied to the existing imaging devices in relatively low price as compared with the methods to manufacture the new architecture of imaging sensors. We have proposed an SR-based method for the reconstruction of an HR image and HDR image from multiple LR and LDR frames with different exposures. The image degradation process including limited spatial resolution and limited dynamic range is modelled and the MAP estimates of both the system response function and a single HR image and HDR image are simultaneously obtained. The system response function is simultaneously estimated in every iteration step with the reconstruction of a single HR image and HDR image. With the simulation results, it is shown that the proposed algorithm outperforms the the conventional approaches

11 HDR IMAGE RECONSTRUCTION WITH SPATIAL RESOLUTION ENHANCEMENT Page 11 of 12 with respect to both objective and subjective criteria. Simulation results verify that the proposed algorithm estimates the reverse mapping function g without any prior nowledge about the system response function as well as an HR image and HDR image. ACKNOWLEDGEMENTS This wor was supported by the MIC (Ministry of Information and Communication), Korea, under the ITRC (Information Technology Research Center) support program supervised by the IITA (Institute of Information Technology Assessment). REFERENCES [1] Kang, M.G. (2003) SPIE Milestone Series Vol. MS177 Selected Papers on CCD and CMOS Imagers. SPIE Press, Washington. [2] Komatsu, T., Aizawa, K., Igarashi, T. and Saito, T. (1993) Signal-processing based method for acquiring very high resolution image with multiple cameras and its theoretical analysis. IEE Proc. Commun. Speech Vision., 140, [3] Par, S.C., Par, M.K. and Kang, M.G. (2003) Super-resolution image reconstruction: a technical overview. IEEE Signal Process. Mag., [4] Clar, J.J., Palmer, M.R. and Laurence, P.D. (1985) A transformation method for the reconstruction of functions from nonuniformity spaced samples. IEEE Trans. Acoust. Speech Signal Process., 33, [5] Kim, S.P. and Bose, N.K. (1990) Reconstruction of 2-D bandlimited discrete signals from nonuniform samples. IEE Proc. Radar Signal Process., 137, [6] van Viley, L.J. and Hendris, C.L.L. (1999) Improving spatial resolution in exchange of temporal resolution in aliased image sequences. Proc. 11th Scandinavian Conf. Image Analysis, pp Kangerlussnag, Greenland. [7] Tsai, R.Y. and Huang, T.S. (1984) Multipleframe image restoration and registration. Adv. Comput. Vision Image Process., 1, [8] Kim, S.P., Bose, N.K. and Valenzuela, H.M. (1990) Recursive reconstruction of high resolution image from noisy undersampled multiframes. IEEE Trans. Acoust., Speech Signal Process., 38, [9] Kim, S.P. and Su, W.Y. (1993) Recursive highresolution reconstruction of blurred multiframe images. IEEE Trans. Image Process., 2, [10] Rhe, S.H. and Kang, M.G. (1999) Discrete cosine transform based regularized high-resolution image reconstruction algorithm. Opt. Eng., 38, [11] Hong, M.C., Kang, M.G. and Katsaggelos, A.K. (1997) An iterative weighted regularized algorithm for improving the resolution of video sequences. Proc. Int. Conf. Image Processing, Vol. 2, Santa Barbara, USA. [12] Kang, M.G. (1998) Generalized multichannel image deconvolution approach and its applications. Opt. Eng., 37, [13] Hardie, R.C., Barnard, K.J., Bognar, J.G., Armstrong, E.E. and Watson, E.A. (1998) High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system. Opt. Eng., 37, [14] Star, H. and Osoui, P. (1989) High resolution image recovery from image-plane arrays, using convex projections. J. Opt. Soc. Am., 6, [15] Eren, P.E., Sezan, M.I. and Tealp, A.M. (1997) Robust, objectbased high-resolution image reconstruction from low-resolution video. IEEE Trans. Image Process., 6, [16] Patti, A.J. and Altunbasa, Y. (2001) Artifact reduction for set theoretic super resolution image reconstruction with edge adaptive constraints and higher-order interpolants. IEEE Trans. Image Process., 10, [17] Schulz, R.R. and Stevenson, R.L. (1996) Extraction of a high-resolution frames from video sequences. IEEE Trans. Image Process., 5, [18] Cornsweet, T.N. (1970) Visual Reception. Academic, New Yor, NY. [19] Handy, R.J. (1986) High dynamic range CCD detector/imager. U. S. Patent [20] Street, R.A. (1998) High dynamic range segmented pixel sensor array. U. S. Patent [21] Brajovic, V. and Kande, T. (1993) A sorting image sensor: an example of massively parallel intensity-to-time processing for low-latency computational sensors. Proc. Int. Conf. Robotics and Automation, 1996, September, Vol. 2, [22] Debevec, P.E. and Mali, J. (1997) Recovering high dynamic range radiance maps from photographs. SIGGRAPH 97 Conf. Proc., August 3 8, [23] Technical Report MS-CIS (1993) Extended Intensity Range Imaging. Grasp Laboratory, University of Pennsylvania. [24] Doi, H., Hara, Y., Kenbo, Y. and Shinba, M. (1986) Image sensor. Japnese Patent [25] Aggarwal, M. and Ahuja, N. (2001) Split aperture imaging for high dynamic range. Proc. IEEE Int. Conf. Computer Vision, (ICCV 2001), July, Vol. 2, pp Vancouver, Canada. [26] Nayar, S.K. and Mitsunaga, T. (2000) High dynamic range imaging: spatially varying pixel exposures. Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2000, June, Vol. 1, Hilton Head Island, USA. [27] Yamada, K., Naada, T. and Yamamoto, S. (1994) Wide Dynamic Range Vision Sensor for Vehicles. Proc. IEEE Conf. Vehicle Navigation and Information Systems Conference, 1994, August, pp Yoohama, Japan. [28] Tani, T. (1995) Photography Sensitivity: Theory and Mechanisms. Oxford University Press. [29] Kolb, C., Mitchell, D. and Hanrahan, P. (1995) A realistic camera model for computer graphics. Computer Graphics, Proc. SIGGRAPH 95, pp Los Angeles, USA.

12 Page 12 of 12 [30] Robertson, M.A., Borman, S. and Stevenson, R.L. (1999) Dynamic range improvement through multiple exposures. Proc. IEEE Int. Conf. Image Processing, (ICIP 99), October, Vol. 3, pp Kobe, Japan. [31] Narasimhan, S.G. and Nayar, S.K. (2005) Enhancing resolution along multiple imaging dimensions using J. CHOI et al. assorted pixels. IEEE Trans. Pattern Anal. Mach. Intell., 27, [32] Guntur, B.K. and Gevreci, M. (2006) High-resolution image reconstruction from multiple differently exposed images IEEE Signal. Process. Lett., 13,