ANALYSIS OF THE EFFECT OF CALIBRATION ERROR ON LIGHT FIELD SUPER- RESOLUTION RENDERING

Transcription

1 04 IEEE International Conference on Acostic, Speech and Signal Processing (ICASSP) ANALYSIS OF THE EFFECT OF CALIBRATION ERROR ON LIGHT FIELD SUPER- RESOLUTION RENDERING Kang-Ts Shih, Chen-Y Hs, Cheng-Chieh Yang, and Homer H. Chen ABSTRACT National Taiwan University Light field photography, which has recently drawn considerable attention, provides novel fnctionalities sch as refocsing and depth estimation at the same time. However, the resoltion of the rendered refocs image is incomparably lower than the nmber of light ray samples in the light field. In this paper, we show that sper-resoltion can be performed to bridge the gap and that deconvoltion, which is missing in most previos methods, is essential to the sccess of light field sper-resoltion rendering. In addition, we investigate the effect of for different kinds of camera calibration error on the qality of rendered images. Given an expected level of image qality, the pperbond of the camera calibration error is analyzed. Index Terms Light field photography, camera calibration, sper-resoltion, error analysis.. INTRODUCTION Light field photography has drawn worldwide attention in recent years and has become an important field of research. The most attractive fnctionality enabled by light field photography is perhaps refocsing, which allows the ser to change the focal plane after the pictre is taken. In addition, light field photography can provide other sefl fnctionalities sch as scene geometry inference and extended-depth-of-field imaging. Despite that light field cameras are commercially available today, there is a serios image resoltion redction on the order of 0: with respect to the raw light field data. Sper-resoltion can be applied in light field rendering process to address the image resoltion redction problem by selecting a proper set of projection directions in the ray space. However, the sper-resolved image is often not as sharp as the image directly acqired by a high-resoltion camera, see Fig. (a). Therefore, there is a need to improve sper-resoltion light field rendering. Or paper has two main contribtions. First, we introdce a deconvoltion step at the end of the sperresoltion process and show the essentialness of the step for light field sper-resoltion rendering. Withot it, blrring of the rendered image is inevitable. Second, we investigate the effect of calibration error on the qality of the rendered (a) (b) (c) Fig.. Cropped refocs image rendered by (a) direct projection and (b) projection and deconvoltion. (c) The grond trth image. image and analyze the pper bond of the error for a targeted image qality. The rest of the paper is organized as follows. We introdce the related work in Section and describe the importance of the deconvoltion operation in Section 3. The effect of camera calibration error on light field rendering is discssed in Section 4, and the conclsion is drawn in Section 5.. RELATED WORK Light field photography dates back several decades [6], [7]. In this section, we briefly review some relevant work from the practical point of view. The first compact hand-held light field camera is developed by Ng et al. [9], which ses Forier slice theorem [0], [] to describe the image formation process in freqency domain. However, becase the spatial resoltion is traded off for the anglar resoltion, the resoltion of the rendered image is one order less than the nmber of captred light field samples. The prototype developed by Liang et al. [] incorporates a programmable apertre to captre the light field in mltiple exposres. Unlike Ng s camera, the resoltion of the rendered image is the same as the nmber of photo sensors in the camera. However, like Ng s camera, the resoltion of the rendered image is still one order less than the total nmber of recorded light ray samples. Several methods were proposed to address the resoltion redction problem. Pérez derived a set of proper projection directions in the ray space for light field sperresoltion rendering [3] [5]. However, the rendered image still appears blrry despite its large pixel cont. On the other hand, Bishop et al. [] modeled the image formation process for light field cameras sing microlens arrays and rendered the sper-resoltion image by energy minimization /4/$ IEEE 534

2 However, it is difficlt to extend the techniqe to other types of light field cameras sch as camera array. The design by Georgiev et al. [3], [4], which is a variant of Ng s camera, changes the distance between the main lens and the microlens array (or eqivalently the focal length of the main lens). Their method takes raw Bayer image as inpt to avoid any ndesired effect of color interpolation. Since it exploits the low fill-factor of the red, green, and ble pixels, the resoltion magnification ratio is limited to two. 3. LIGHT FIELD SUPER-RESOLUTION In this section, we first describe the light field sperresoltion rendering method in [4], [5], and then analyze the case of the blr problem. We later show that the problem can be solved by adding a deconvoltion step to the rendering process. We focs on the analysis of this method becase, as proved in [5], it is more general than the other methods [] [4]. Since for-dimensional (4D) light fields are difficlt to visalize, we consider two-dimensional (D) light fields captred with a one-dimensional (D) camera array in this section. The reslt can be easily generalized to 4D light field. We explain the sper-resoltion method sing the toy camera array shown in Fig. (a), which consists of 4 cameras. Sppose each camera has 7 pixels in the common field of view. Therefore, a total of 8 light rays are captred for the entire camera array. Assme the camera centers are collinear and the spacing between cameras is niform. Define -axis to be the line connecting camera centers and x-axis to be the line connecting the shared sample points of each camera. Conseqently, the points representing the sampled rays form a rectanglar grid pattern in ray space constrcted by x and, Fig. (b). It has been shown that refocsing is eqivalent to the projection of light ray samples in ray space, and different refocs planes correspond to different projection directions [], [0]. For example, the refocs planes and in Fig. (a) correspond to the projection directions indicated by slope m and m, respectively, in Fig. (b). Note that m = 4/ where x is the sampling pitch on the x-axis and is the distance between adjacent camera centers (Fig. (b)). This way, the rendered refocs image on plane has for times the nmber of samples than the individal image captred by any single camera. On the other hand, for refocs plane (the corresponding slope m is / the nmber of samples does not increase. From this example, we can see that if the refocs plane is selected properly, it is possible to render refocs images with the nmber of samples eqal to the nmber of light rays samples of the discrete light field. A discssion on the selection of proper refocs planes can be fond in [3] [5]. However, an image with more pixels does not necessarily imply a higher image resoltion. Althogh choosing a proper refocs plane increases the nmber of (a) (b) Fig.. (a) A one-dimensional camera array consisting of for cameras. The dashed lines represent the recorded light rays. (b) The distribtion of the light rays samples captred by the camera array in (a). Slope m = 4/x and m = /x. samples, the reslting image still appears blrry (Fig. (a)). The reason is as follows. Assme that each camera in the array has identical PSF. The recorded radiance vale r ij of the light ray parameterized as (x i, j ) is r ij A x A x j A/ xi x / j A/ xi x / j A/ j A/ l( ) K ( dxd l( ) K( o xxi d, where l denotes the continos light field, A denotes the apertre diameter, the symbol * denotes the convoltion operator ( is treated as a constant in the convoltion), K o denotes the optical PSF, and K denotes the overall PSF by convolving K o with the sensitivity fnction of the sensor, which is assmed to be a rectanglar fnction of width x. If the apertre size A is small enogh, r ij can be approximated by rij l( j ) K( xx i. () x If the object of interest has Lambertian srface and lies on the refocs plane, the light field l has the property l( x, ) l( 0). (3) m The variable m in (3) is the slope corresponding to the depth of the object. Combining () and (3), we have rij l( 0) K(. (4) j x xxi m For or toy example, if we project the samples in the ray space along the direction m, the reslting rendered D refocs image is the seqence of pixels r, r, r 3, r 4, r, r, r 3, r 4,, r 74. Sbstitte m=m =4/x into (4), we can see that the D image is the densely sampled version of the low-pass filtered signal l(0)*k( with sampling period x/4. The low pass kernel K( is the case of the blr in the image rendered by direct projection of the ray samples. Therefore, a deconvoltion step is essential for light field sper-resoltion rendering to cancel the blr reslting from K(. To show the effectiveness of adding a deconvoltion step, compare the blrred image by simple ray projection () 535

3 (a) (b) (c) (d) Fig.3. Cropped sper-resolved images with different p : (a) p =0 (ideal) (b) p=0.05 (c) p=0. (d) p=0. (nit: pixel). (a) (b) Fig. 4. The rendered image with camera position error (a) direct projection (b) sper-resolved image after deconvoltion. Fig. 5. The flowchart describing the procedre to generate the image and grond trth for each camera in the array. shown in Fig. (a) and the sper-resolved refocs image after deconvoltion shown in Fig. (b). Compare with the grond-trth image in Fig. (c), the sharp edges in the sper-resolved image are sccessflly reconstrcted. 4. ANALYSIS Althogh introdcing a deconvoltion step is capable of sharpening sper-resolved refocs image, we observe that the qality of the rendered image degrades greatly when there is camera calibration error. As shown in Figs. 3 and 4, if the position estimates of the cameras are not exact, the edges in the reslting sper-resolved image wold definitely appear blrry (Figs. 3(b) (d)) compared with the ideal case (Fig. 3(a)). In this section, we analyze the effect of camera calibration error on light field sper-resoltion rendering. We generate discrete light field sed in or analysis with the ray tracing tool POV-ray [5]. The light field is captred with a 5x5 virtal camera array. The test scene is a checker board placed at a fixed distance from the camera array. To control the PSF, we follow the procedre shown in Fig. 5 to generate the image for each camera. The reslting discrete 4D light field is of size 640x480x5x5. We se the grond trth image (generated by the procedre shown in Fig. 5) as reference and the peak signal-to-noise ratio (PSNR) as the objective qality measre. In addition, we se se the method described in Section 3 to generate sperresolved images. Specifically, the light ray samples are projected along the direction corresponding to the refocs Fig. 6. The effect of calibration error on the qality of rendered image. (a) Camera position error (b) Camera orientation error (c) PSF size error. For (a) and (b), the horizontal axis is p and r, respectively. plane (aligned with the checker board in the test scene), and then the deconvoltion algorithm developed by Shan [8] is applied to the reslting image. The camera calibration errors are classified into for categories: camera position error, camera orientation error, PSF size error, and PSF shape error. We analyze the effect of each error. 4.. Camera position error To analyze the effect of camera position error on light field sper-resoltion, a random vector (n x, n y, n z ) is added to the coordinates of the camera center (C x, C y, C z ) of each camera, where n x, n y, and n z are independent and identically distribted (i.i.d.) Gassian random variables with variance p. We plot the PSNR of the sper-resolved image as a fnction of p in Fig. 6(a). For sbjective evalation, we also show cropped reslts in Fig. 3. From Fig. 6(a) we can see that the camera position error greatly degrades the qality of the sper-resolved image. To keep the PSNR of the sper-resolved image above 4 db, the camera position error p mst be controlled within 0. pixel. As shown in Fig. 5(a), the error introdces zigzag artifact to the edges in the image constrcted by direct projection of the light ray samples. Althogh the prior term introdced to the deconvoltion algorithm alleviates the zigzag artifact, it reslts in blrred edge, Fig. 5(b). 4.. Camera orientation error In this experiment, we analyze the effect of camera orientation error. Let the estimated optical axis of each camera in the array be parallel to the vector (0, 0, L z ). We assme the tre optical axis is slightly tilted and is parallel to the vector (m x, m y, L z ), where m x and m y are i.i.d. Gassian random variables with variance r. In Fig. 6(b), the x-axis represents r and L z is 30 (nit: pixel) in this experiment. Similar to the effect cased by position error, tilting of 536

4 (a) (b) (c) (d) Fig. 7. Sper-resolved images generated with Gassian kernel of different variances: (a) Var = (b) Var = (c) Var = 3 (d) Var = 4 (nit: pixel). (a) (b) (c) (d) (e) Fig. 9. PSNR of sper-resolved images generated with PSFs with different shapes. (f) (g) (h) (i) (j) Fig. 8. PSFs sed in the experiment described in Sec. 4.4: (a) tre PSF, Gassian (variance= pixels) (b) box (c) large diamond (d) diamond (e) long horizontal rectangle (f) horizontal rectangle (g) long vertical rectangle (h) vertical rectangle (i) triangle (j) Gassian (variance=7 pixels). optical axis also reslts in zigzag edges in the image generated by direct projection of light rays PSF size error In Sec. 3, we assme that the PSF of each camera can be estimated beforehand, and hence the deconvoltion is applied in a non-blind manner. However, the estimation may not be exact in real world. Here, we focs on the estimation error of the size of the PSF spport. Specifically, we generate the inpt discrete light field sing a Gassian kernel with a variance of pixels in the filtering step in Fig. 5, while Gassian kernels of different variances are sed to generate the sper-resolved images in the deconvoltion step. The objective qality is plotted as a fnction of the variance of the Gassian kernels in Fig. 6(c), and the sperresolved reslts are shown in Fig. 7. We can see that the sper-resoltion qality degrades as the variance of the estimated PSF deviates from the tre vale ( pixels). We also observe that, compared with camera position and orientation error, the PSF size error degrades the qality less. The spport size (related to ct-off freqency) of the PSF affects the sharpness of the sper-resolved image, bt it does not prodce significant zigzag artifacts like camera position error and orientation error do PSF shape error Besides the PSF spport size, we analyze the effect of inexact estimation of PSF shape. In this experiment, the deconvoltion is applied sing PSFs of different shape and the discrete light field is generated sing a Gassian kernel with variance of pixels. We show deconvoltion kernels that are sed in this experiment in Fig. 8 and the PSNR of the sper-resolved images generated with these kernels in (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) Fig. 0. Cropped sper-resolved images with different shapes of gassian PSF: (a) ideal camera PSF, Gassian (variance= pixels) (b) box (c) large diamond (d) diamond (e) long horizontal rectangle (f) horizontal rectangle (g) long vertical rectangle (h) vertical rectangle (i) triangle (j) Gassian (variance=7 pixels). Fig. 9. For sbjective evalation, we also show a cropped portion of the sper-resolved reslts in Fig. 0. For those PSFs having a similar shape to the tre PSF, sch as the one in Fig. 0(c), the PSNR is higher. Fig. 0(d) can be regarded as an approximation of a Gassian kernel with small variance, and the reslting PSNR is consistent to the reslt in Fig. 6(c). For PSF shapes that are less similar to the tre PSF, sch as the ones in Fig. 0(e), 0(g), and 0(i), significant artifacts appear on the edge of the sper-resolved image. The qality of the sper-resolved image is less sensitive to PSF shape compared with camera position and rotational error. 5. CONCLUSION In this paper, we have shown that deconvoltion is an essential step in light field sper-resoltion rendering and provided a qantitative analysis of the effect of varios camera calibration errors on the qality of the sperresolved image. The reslts show that the calibration errors of camera position and orientation introdce more serios degradation to the rendering qality than the calibration error of the spport size and the estimation error of the PSF shape of the camera. This work provides valable insights into the fndamental limits of sper-resoltion light field rendering. 537

5 REFERENCES [] C.-K. Liang, T.-H. Lin, B.-Y. Wong, C. Li, and H. H. Chen, "Programmable Apertre Photography: Mltiplexed Light Field Acqisition," ACM Trans. on Graphics, Vol. 7. No. 3, pp. 55: 55:0, 008. [] C.-K. Liang, Y.-C. Shih, and H. H. Chen, "Light Field Analysis for Modeling Image Formation," IEEE Trans. on Image Processing, Vol. 0, No., pp , 0. [3] F. Pérez, "Sper-Resoltion in Plenoptic Cameras by the Integration of Depth from Focs and Stereo," Compter Commnications and Networks, Proc. of 9th International Conference on, pp. 6, 00. [4] F. Pérez and J. P. Lüke, "Simltaneos Estimation of Sper- Resolved Depth and All-in-Focs Images from A Plenoptic Camera," 3DTV Conference: The Tre Vision-Captre Transmission and Display of 3D Video, pp. 4, 009. [5] F. Pérez, A. Pérez, M. Rodrígez, and E. Magdaleno, "Forier Slice Sper-resoltion in Plenoptic Cameras," Comptational Photography, IEEE International Conf. on, pp., 0. [6] G. Wetzstein, I. Ihrke, D. Lanman, and W. Heidrich, "Comptational Plenoptic Imaging," Compter Graphics Form, Vol. 30, No. 8, pp , 0. [7] M. Levoy and P. Hanrahan, "Light Field Rendering," Proc. of the 3rd Annal Conf. on Compter Graphics and Interactive Techniqes, pp. 3 4, 996. [8] Q. Shan, J. Jia, and A. Agarwala, "High-Qality Motion Deblrring from A Single Image," ACM Trans. on Graphics, Vol. 7, No. 3, pp. 73: 73:0, 008. [9] R. Ng, M. Levoy, M. Bredif, G. Dval, M. Horowitz, and P. Hanrahan, "Light Field Photography with A Hand-Held Plenoptic Camera," Compter Science Technical Report CSTR, 005. [0] R. Ng, "Forier Slice Photography," ACM Trans. on Graphics, Vol. 4, No. 3, pp , 005. [] R. Ng, "Digital light field photography," PhD dissertation, Stanford University, 006. [] T. E. Bishop, S. Zanetti, and P. Favaro, "Light Field Sperresoltion," Comptational Photography, IEEE International Conf. on, pp. 9, 009. [3] T. Georgiev, G. Chnev, and A. Lmsdaine, "Sperresoltion with the Focsed Plenoptic Camera," Proc. SPIE Comptational Imaging, Vol. 7873, 0. [4] T. Georgiev, A. Lmsdaine, and G. Chnev, "Using Focsed Plenoptic Cameras for Rich Image Captre," Compter Graphics and Applications, pp. 6 73, 0. [5] 538