Dynamic Range Enhancement Algorithms for CMOS Sensors With Non-Destructive Readout

Transcription

1 IEEE International Worksho on Imaging Systems and Techniques IST 2008 Chania, Greece, Setember 10 12, 2008 Dynamic Range Enhancement Algorithms for CMOS Sensors With Non-Destructive Readout Anton Kachatkou, Roelof van Silfhout School of Electrical and Electronic Engineering University of Manchester Manchester, United Kingdom Abstract Image sensors caable of a non-destructive readout (NDR) allow reading several frames during the integration time without affecting a hoto charge being collected in the ixel s well. We show that two ublished algorithms for rocessing the sequence of NDR frames into a high dynamic range (HDR) image suffer from "steed gradient" artifacts using data taken with an ''off-the-shelf" CMOS image sensor. We resent a novel algorithm that is caable of alleviating this image artifact that is due to subotimal saturated ixel treatment. Our rocedure works for both black and white and color images. In addition, we resent oint demosaicing and denoising for color HDR image reconstruction and its erformance is comared with conventional demosaicing algorithms. The overall erformance of the combined methods rovides a DR increase of u to 44 db comared to raw data. We demonstrate our algorithms using images taken with a black-and-white and color NDR caable camera system. Keywords-non-destructive readout; CMOS image sensor; dynamic range enhancement; saturated ixels extraolation; demosaicing; noise removal; embedded camera system I. INTRODUCTION Image sensors caable of non-destructive readout (NDR) allow for reading several frames during a single exosure without affecting the integrated signal levels. This aroach was originally develoed for high dynamic range (HDR) infrared detectors [1]. Advances in comlementary metal oxide semiconductor (CMOS) sensor design have enabled sensor designers to imlement NDR schemes in high-resolution visible light image sensors [2-4]. Comared to other dynamic range (DR) enhancement techniques [5, 6] the rocessing of multile NDR frames was shown [4-6] to have several advantages such as reservation of the signal to noise ratio (SNR) over the extended art of the dynamic range, a otential to enhance DR at both the lower and higher ends of the signal range and digital correlated double samling (DCDS) [7]. Several works describing a sensor architecture with on-chi (hardware) imlementation of NDR based DR enhancement have been resented [3, 6]. However, these aroaches substantially increase ixel comlexity and, therefore, reduce the hotosensitive area (lower fill factor). In addition, there are more sources of noise and a higher design and manufacturing cost. This work resents an alternative aroach of external NDR data rocessing using a state-of-the-art CMOS sensor with a reconfigurable logic read out system for both black-andwhite and color images. We also address the roblem associated with extraolation of saturated ixels and color reconstruction of HDR images under noisy conditions. II. DNAMIC RANGE ENHANCEMENT The NDR scheme rovides several oortunities for imrovement in the dynamic range of a catured image scene. Firstly, at low signal levels the DR is determined by signal-tonoise ratio and limited by read noise. Using an NDR caable sensor, read noise is reduced by removing its fixed comonents by DCDS. Furthermore, the information extracted from several NDR samles can be exloited to filter a random noise out. Secondly, the integrated intensity of each ixel grows linearly with time and, deending on the intensity of the incident light, can saturate before the full integration time is comlete. This saturation limits the DR on the higher end. Having several signal samles taken before saturation it is ossible to extraolate ixel intensity to the full integration time. To enhance the DR of the imaging system at the lower signal range we alied and tested two noise filtering routines described elsewhere: least-squared error signal sloe estimation (LSESE) [1] and weighted averaging (WA) [4]. Both assume a linear relation between the intensities of the same ixel taken from different NDR frames and try to estimate a value of a signal sloe that is equivalent to a signal to frame number ratio (for uniformly distributed samles in time). LSE estimation for a signal sloe [1, 2] is given by n n + 1 X i, i sloe i 1 2 = = 12, = 1... M, (1) n n 2 ( 1) where n is a number of the last frame before saturation (LSBS), is a ixel index, X,i is a raw value of the th ixel in the ith frame, M is a total number of ixels in the image. Here, a uniform distribution of samles over the full integration time is assumed. For n = 1 the sloe is estimated as /08/$ IEEE

2 X X sloe,1,0 = t1 t, (2) 0 where X,0 and t 0 are a ixel value and a readout time of CDS image, i.e. data collected immediately after the reset signal. It is easy to derive from (1) that using this aroach there is no need to erform a DCDS directly as the first n / 2 frames have negative coefficients and, accordingly, remove any fixed offset contribution from the final estimation. However, the CDS image still has to be ket in memory to make evaluation (2) ossible. The main advantage of this method is its comutational simlicity. Since all coefficients in (1) do not deend on actual ixel intensity they can be calculated for every ossible n and saved in memory before the acquisition rocess is started. The amount of memory that is necessary to kee these values is relatively small. According to the WA algorithm roosed in [4] signal sloe value is estimated as follows: = A, = 1... M, (3) sloe T sloe n, n, sloe where A n, is a vector of n averaging coefficients, n, is a column of n raw sloe values calculated after a modified DCDS [4] has been alied. The values of vector A n, are given by the solution of a minimization roblem Figure 1. Steed gradient artifact. Left art of Attenuator image taken with a short integration time, no ixel is affected by saturation.. Right same art of the image taken with the integration time long enough to cause saturation and then reconstructed using a basic LSBS aroach. The reconstruction result evidently suffers from unexected shar edges and sots that do not aear in the original image. =, (5) k, k, i, i, where is an index of the ixel from 3 3 neighborhood of the th ixel, k and i are numbers of NDR samles, k > i. Hence, in order to calculate the intensity of the th ixel in NDR samle k it is necessary to find in its 3 3 neighborhood all nonsaturated or already reconstructed saturated ixels and use the following equation to udate its value: sloe sloe ( ) 2 n, E A = min n, = 1... M, () i (4) An, = 1 i= 1 sloe where is a true signal sloe for the th ixel. This algorithm takes into account several tyes of noise that CMOS sensors are rone to: random read noise, shot noise, reset noise and offset FPN. However, it requires significant comuting ower that will limit its usage in embedded camera systems. A straightforward method to enhance DR beyond the saturation level uses LSBS to linearly extraolate a corresonding signal value. In both algorithms described above this is done by reserving the LSBS estimation for signal sloe as a final reconstruction result for a given ixel. This aroach may cause undesirable artifacts in image regions containing a well-ronounced gradient of the light intensity (Fig. 1). The intensity gradient makes neighboring ixels saturate in different NDR samles and, accordingly, reconstruction arameter n in (1) and (4) has different values for these ixels. This brings discontinuity into the signal estimation across an image and, due to resence of noise in the raw data, results in unexected edges and sots aearing in a roblem region. We call this henomenon a steed gradient. Addressing the issue of steed gradient we develoed a new extraolation routine to reconstruct a signal in saturated regions. It considers a correlation between intensities of adacent ixels as well as between the intensities of the same ixel taken from different NDR samles. In other words, it assumes that k, ˆ i, k, =, (6) N ˆ P P where P is a collection of ixels from the 3 3 neighborhood used for extraolation (Fig. 2), N is a number of elements in P. If P is an emty collection (current ixel is surrounded by other saturated ixels that are not udated yet) then extraolation for this ixel should be deferred until one or some of its neighbors are udated. The averaging over P is used in order to suress a negative effect of noise that can result in erroneous value of the ratio in the right art of (5). Figure 2. Same 3 3 neighborhood of th ixel croed from ith (left) and kth (right) NDR samles. The value of the th ixel from kth NDR samle is estimated using its values from ith samle and the values of four nonsaturated or already reconstructed ixels from the deicted neighborhood. Dashed ixels are saturated and not used for reconstruction. P i,

3 III. COLOR RECONSTRUCTION In case of color filtered sensors a demosaicing roblem [8] contributes into the overall comlexity of the task. Two aroaches are available. The first assumes a two-stage rocessing when every ixel is reconstructed searately using one of the routines described above to generate a HDR image and then any conventional demosaicing algorithm is alied to create a full-color image. In this case, similarly to black-andwhite sensors, the roblem of steed gradient artifacts should not be overlooked. Our exeriments showed that the extraolation method described above can be alied directly to the color filtered data. The second aroach is a oint rocessing that utilizes NDR samles to achieve two obects: to enhance DR and to reduce a number and intensity of color artifacts due to noise filtering. To evaluate its efficacy we develoed a oint demosaicing, denoising and DR enhancement algorithm similar to the one suggested in [9] for a oint demosaicing and suer-resolution roblem. It solves a maximum a osteriori (MAP) roblem 2 n i ˆ ˆ sloe = arg min ˆ ti + ˆ W k k i D U k i (7) k= R, G, B i= 1 = 1 2 with bilateral satial and Tikhonov color regularization terms [9, 10]. Here, k is a color channel index (R, G or B), D k is a samling matrix corresonding to the kth color, U is a shot noise value generated between th and ( 1)th readouts and estimated as a zero mean Gaussian distribution with variance 2 σ = D k t t, stands for a vector of measured ( 1 ) sd k k= R, G, B data and ˆ k for a column of estimated signal sloe values corresonding to the kth color. W i is a saturation ma for the ith NDR samle. Its elements are equal to zero if corresonding ixels in the ith NDR samle are saturated, otherwise they have a unit value. Symbol designates an element-wise matrix multilication. In (7) the limits of the summation over i can be altered to exclude several first NDR samles that are articularly noisy and can substantially degrade the quality of reconstruction results. IV. RESULTS The exeriments used an embedded camera system equied with an off-the-shelf 640x480 resolution imager in both black-and-white and color versions [11]. Although the camera includes a rogrammable comutational module (FPGA) under control of embedded rocessor, the maor art of calculations is currently imlemented in Matlab and run offline. The camera system was used to control the CMOS sensor and to read, store and transfer NDR samles to a host comuter. The ossibility for embedded imlementation of the DR enhancement algorithms is discussed in the next section. Firstly, the erformance of the black-and-white imager in conunction with noise filtering algorithms was tested. Camera s initial arameters in normal oerational mode were estimated using a PT technique. Then, keeing these data as a reference oint, noise and dynamic range erformance of NDR-based data rocessing algorithms was evaluated. Fig. 3 demonstrates a gradual imrovement in noise erformance while the comlexity of the data rocessing algorithm in use increases. It is evident that DCDS is an essential art of the image ost-rocessing erformed in the cameras based on CMOS sensors and NDR gives the most efficient way of doing this. The best noise erformance over the full DR is demonstrated by WA method. For low and mid-level signals LSESE demonstrates nearly identical SNR enhancement by on average 1-2 db; however, the exeriments suggest that it takes about five NDR frames to show the advantage over DCDS. WA method also maintains its erformance u to saturation whereas LSESE tends to degrade for higher signal values and shows the results similar to DCDS on the uer end of the DR. This can be exlained by the fact that its design does not take into account a relation between read and shot noise for different signal levels. In order to evaluate an effective number of frames that is sufficient for the otimal trade-off between the quality of the outut signal and the data rocessing overhead the integration time value was fixed and NDR algorithms were run for a Figure 3. SNR vs. signal mean Figure 4. DR vs. number of samles used for reconstruction

4 different number of frames. Fig. 4 shows that DR steadily increases when a number of NDR frames grows. However, the maor gain of more than 17 db for WA is achieved for about 16 images the use of a greater number of samles rovides with a further grow of only db for as many as 128 images. At the to range DR is limited by the shortest image readout time. For our exerimental setu the maximum DR enhancement beyond a saturation level is about 26 db for full resolution images read out at aroximately 230 frames er second. Consequently, the total DR enhancement of u to 44 db is obtained comared to non-ndr image acquisition. Secondly, black-and-white ictures of a real HDR scene were catured and two methods of saturated ixels extraolation were comared. Fig. 5 highlights the sueriority of the novel extraolation method over its LSBS counterart. The new algorithm fully eliminates undesired artifacts in stee gradient regions. Finally, the erformance of two-color reconstruction methods for Bayer mosaiced NDR sequence was comared. For the two-stage rocessing simulated NDR samles were firstly rocessed by WA to generate an intermediate Bayer image that then underwent a color reconstruction using one of conventional demosaicing algorithms. Bilinear interolation, high-frequency sensitive linear interolation [13] and oint demosaicing and denoising [12] algorithms were chosen. The former is the fastest and simlest method; the latter is a good reresentative of traditional demosaicing algorithms that address the correlation between different color channels as well as the sensitivity to high satial frequencies. The bilinear interolation takes into account the resence of signal deendent noise that characterizes most images catured by digital image sensors. Reconstruction results were comared with the original full color image using PSNR [14] and S- CIELAB [15] measures (Fig. 6). Simulated NDR samles were generated as follows: = t + U + R + C, i = 0... n, (8) i i i i k= R, G, BT = 1 where Ŷ is an original full color image, T = t n is a full integration time, R i reresents readout noise, C stands for reset noise and dark offset FPN, both R i and C were simulated as a Gaussian noise with zero mean and standard deviations σ r and σ c resectively. Fig. 6 shows the results obtained for the lighthouse image that is art of the Kodak test image set. It is clear that the oint reconstruction method roosed in this work maintains a high reconstruction quality over the bigger range of readout noise and demonstrates better quantitative results. Fig. 7 shows an examle of the HDR image catured by the NDR camera system. Image on the left is scaled linearly and gives a good imression how an image catured by the camera oerating in normal integration mode would look like. The necessity to kee a bright edge of the railway station roof in the left art of the image unsaturated makes features in the right dark art of the image almost indiscernible due to a limited DR of the sensor. However, using the NDR mode and a oint color reconstruction described above all the details in both bright and dark arts of the scene were successfully catured. Figure 5. Attenuator images obtained using WA with saturated ixels extraolated by conventional LSBS (uer row) and the roosed (lower row) algorithms. To demonstrate the actual extent of the DR the left column contains images with linearly scaled intensity whereas images in the right column have ixel intensity scaled logarithmically Figure 6. PSNR and S-CIELAB measures vs. readout noise value for four different color reconstruction routines: a two-stage aroach with resectively bilinear, Hirakawa [12] and Malvar [13] demosaicing exloited and a roosed oint demosaicing, denoising and DR enhancement aroach

5 Figure 7. HDR image taken by our NDR setu. Right image is a logarithmically scaled reresentation of the image on the left and highlights the large dynamic range V. CONCLUSION Our comarative analysis of LSESE and WA noise suression algorithms for NDR image DR enhancement shows that LSESE allows for a good trade-off between comutational overhead and reconstruction quality rovided a sufficient amount of NDR samles are available. Its main advantage is that most calculations can be done in advance and coefficients corresonding to each NDR samle and saturation time can be stored in memory and used later during acquisition. Desite slightly better results of WA its comutational requirements make it hard to use in embedded camera systems. HDR images generated from several NDR samles using reviously ublished algorithms are suscetible to a steed gradient artifact. To our knowledge, this roblem is reorted for the first time. A roosed extraolation method revents this artifact from aearing and can be used for both black-andwhite and color mosaiced sensor configurations. This aer also demonstrates our oint demosaicing and denoising for color HDR image reconstruction based on several NDR image samles. Couled with motion detection [4] and multi-frame suer resolution [9] algorithms this aroach can lead to a suerior color image quality in comarison with a traditional two- or multistage image rocessing. The current roblem of the oint rocessing is its relative comlexity that may require additional adatation for embedded imlementation. Concluding, the NDR caability of state-of-the-art CMOS imagers combined with dedicated external data rocessing rovides significant imrovements in dynamic range of the rocessed images. ACKNOWLEDGMENT We would like to thank P. Scott, The University of Manchester, for develoing embedded code to control a camera with NDR caable image sensor and Dr K. Hirakawa, Harvard University, who kindly gave us a Matlab imlementation of his oint demosaicing and denoising algorithm. REFERENCES [1] J. D. Garnett and W. J. Forrest, "Multily samled read-limited and background-limited noise erformance," in Infrared Detectors and Instrumentation, Orlando, FL, USA: SPIE, 1993, [2] B. Fowler, "Low Noise Wide Dynamic Range Image Sensor Readout using Multile Reads During Integration (MRDI)." Pixel Devices Intl. Inc. (PDI), [3] D. X. D. ang, A. E. Gamal, B. Fowler, and H. Tian, "A 640x512 CMOS image sensor with ultrawide dynamic range floating-oint ixellevel ADC," IEEE Solid-State Circuits, vol. 34, , Dec [4] X. Liu and A. El Gamal, "Synthesis of high dynamic range motion blur free image from multile catures," IEEE Trans. Circuits Syst. I, Fundam. Theory Al. [see also IEEE Trans. Circuits Syst. I, Reg. Paers], vol. 50, , Ar [5] D. X. D. ang and A. El Gamal, "Comarative analysis of SNR for image sensors with enhanced dynamic range," in Sensors, Cameras, and Systems for Scientific/Industrial Alications, San Jose, CA: SPIE, 1999, [6] S. Kavusi and A. El Gamal, "Quantitative study of high-dynamic-range image sensor architectures," in Sensors and Camera Systems for Scientific, Industrial, and Digital Photograhy Alications V, San Jose, CA: SPIE, 2004, [7] J. R. Janesick, Photon Transfer: DN->λ. Bellingham, Washington: SPIE Press, 2007,.186. [8] B. K. Gunturk, J. Glotzbach,. Altunbasak, R. W. Schafer, and R. M. Mersereau, "Demosaicking: color filter array interolation," IEEE Signal Processing Mag., vol. 22, , Jan [9] S. Farsiu, M. Elad, and P. Milanfar, "Multiframe demosaicing and suerresolution of color images," IEEE Trans. Image Process., vol. 15, , Jan [10] S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, "Fast and robust multiframe suer resolution," IEEE Trans. Image Process., vol. 13, , Oct [11] P. F. I. Scott and R. G. van Silfhout, "FireFlash embedded camera system," unublished. [12] K. Hirakawa and T. W. Parks, "Joint Demosaicing and Denoising," IEEE Trans. Image Process., vol. 15, , Aug [13] H. S. Malvar, L.-W. He, and R. Cutler, "High-quality linear interolation for demosaicing of Bayer-atterned color images," in Proc. Acoustics, Seech, and Signal Processing 2004 IEEE Int. Conf. (ICASSP '04). vol. 3, [14] W. Lu and.-p. Tan, "Color filter array demosaicking: new method and erformance measures," IEEE Trans. Image Process., vol. 12, , Oct

6 [15] X. Zhang and B. A. Wandell, "A satial extension of CIELAB for digital color image reroduction," in SID 96 Digest, 1996,