Light field panorama by a plenoptic camera

Transcription

1 Light field anorama by a lenotic camera Zhou Xue, Loic Baboulaz, Paolo Prandoni and Martin Vetterli École Polytechnique Fédérale de Lausanne, Switzerland ABSTRACT Consumer-grade lenotic camera Lytro draws a lot of interest from both academic and industrial world. However its low resolution in both satial and angular domain revents it from being used for fine and detailed light field acquisition. This aer rooses to use a lenotic camera as an image scanner and erform light field stitching to increase the size of the acquired light field data. We consider a simlified lenotic camera model comrising a inhole camera moving behind a thin lens. Based on this model, we describe how to erform light field acquisition and stitching under two different scenarios: by camera translation or by camera translation and rotation. In both cases, we assume the camera motion to be known. In the case of camera translation, we show how the acquired light fields should be resamled to increase the satial range and ultimately obtain a wider field of view. In the case of camera translation and rotation, the camera motion is calculated such that the light fields can be directly stitched and etended in the angular domain. Simulation results verify our aroach and demonstrate the otential of the motion model for further light field alications such as registration and suer-resolution. Keywords: Plenotic camera, non-classical image cature, light field stitching, light field anorama 1. INTRODUCTION Plenotic cameras, also known as light-field cameras, can cature both color and geometric information of a scene in a single shot. This is achieved by recording the intensity and direction of each light ray at the same time. It enables many new imaging alications including deth estimation, digital refocusing and ersective shift. Light-field acquisitions also rovide new aroaches for many of the standard roblems in image rocessing such as denoising, suer-resolution, image stitching, etc. The first consumer-grade lenotic camera, Lytro, has generated a lot of interest when aeared in It rovides a comact and ortable solution for the acquisition of the light field by adding a microlens array to a conventional camera 1. 2 Its major limitation however is the low resolution of the rendered images: the camera s sensor is indeed used to record both the satial and angular information of the scene resulting in a rendered image with a resolution equal to that of the microlens array. To benefit from the great otential of light field cameras, increasing the resolution in both satial and angular domain is a crucial ste for many alications. Since increasing the resolution of the microlens array can be very eensive, taking multile light field images and merging the data is an aealing way to tackle the roblem. Increasing the size of light field data can be achieved by aligning and stitching multile light fields with overlaing areas for wider field of view. Birklbauer, et al. 3 resent the first aroach to construct anorama light field image by using all-in-focus image for registration and focal-stack image for view synthesis. Our work is different from theirs in two asects: firstly, we give a comlete motion model of the light field catured by the lenotic camera, including both camera translation and rotation. Secondly, we directly erform stitching in the 4D light field whereas both all-in-focus and focal-stack images are rojections of the original light field data into low dimensions as mentioned in the work by Levin and Durand. 4 This aer describes a general lenotic camera model which is used for light fields stitching and show that classic image alignment and stitching is a sub-roblem of the more general stitching roblem of light fields. In the remainder of this aer, we first formalize the roblem of image stitching from the ersective of light fields. We then resent our acquisition and motion model of the lenotic camera. Finally we give two alications in light field stitching based on the roosed motion model and verify the model with simulation results. Further author information: (Send corresondence to Zhou Xue) Zhou Xue.: zhou.ue@efl.ch Comutational Imaging XII, edited by Charles A. Bouman, Ken D. Sauer, Proc. of SPIE-IS&T Electronic Imaging, SPIE Vol. 9020, 90200S 2014 SPIE-IS&T CCC code: X/14/$18 doi: / Proc. of SPIE-IS&T/ Vol S-1

2 2.1 Background on light fields 2. IMAGE STITCHING IN LIGHT FIELDS In this aer, we use a two lane arameterization to reresent the light field in which each light ray in the sace is described by using its intersection with two arallel lanes, the image lane (, q) and camera lane (,y) 5. 6 The distance between these two lanes is normalized to 1. Therefore, light field L(,y,,q) is actually the radiance intensity of the light ray intersecting with (,y) lane from the (,q) direction. Without loss of generality, we erform the analysis on a 2D light field L(,) by fiing y and q for the sake of simlicity. Since the light field is indeed an intensity function of light rays, the ray transfer matri analysis is used in this aer to describe the linear transformation of the light field. The two fundamental ray transformations, ray roagation in sace A and refraction A f by a lens are described resectively as follows: A = [ ] [, A f = 1 0 f 1 1 ] (1) where denotes the roagating distance and f is the lens focal length. A simle inhole camera model is used to demonstrate the relation between the light field and catured images. As shown in Figure 1, an eamle of 2D light field is generated by using a moving inhole camera to cature one horizontal line from an oil ainting. For the sake of simlicity, the oil ainting is modeled as a fronto-arallel lane Z meters away from the camera with Lambertian-reflectance roerties. The moving camera is on the lane and each image is defined on the lane relative to the camera osition. P: i Figure 1: An eamle of a 2D light field is generated from one horizontalline in a fronto-aralleloil ainting with Lambertian-reflectance roerties. In the 2D light field on the right, the red slice denotes an image catured by a inhole camera on the camera lane; the blue slice denotes a bundle of arallel light rays catured theoretically by a camera at infinity; the black slice denotes an image taken by a inhole camera laced on the oil ainting. The red slice in the 2D light field reresents an image taken by the moving inhole camera on the camera lane. All the light rays within the slice converge on the lane. The blue slice reresents a bundle of arallel light rays which can theoretically be catured by a inhole camera at infinity. Another secial slice is reresented by the black slice whose sloe is 1/Z and Z is the distance between the oil ainting and the camera lane. The light rays within the green slice all converge at Z meters in front of the camera lane and they are eactly from the same sot on the surface of the oil ainting. Since the oil ainting is assumed have Lambertian-reflectance roerties, the intensities within the black slice remain constant for all the light rays from different directions. Theoretically, the black slice is catured by a inhole camera ositioned on the oil ainting. In conclusion, slices in the light field with different sloes are images which can be virtually catured by inhole cameras at lanes away from the original camera lane. A slice in 2D light field with a sloe 1/z can Proc. of SPIE-IS&T/ Vol S-2

3 be seen as equivalent to an image catured by a inhole camera ositioned at z meters away from the original camera lane. A ositive z indicates that the osition is in front of the camera lane whereas a negative z indicates that the osition is behind the camera lane. Two etreme cases are vertical slices and horizontal slices which are images catured by camera on lane and at infinity resectively. Last but not the least, reresents the image coordinate normalized by the focal length of the inhole camera. The range of the light field slice in the dimension remains constant for the same camera. Slices with negative sloes are images with magnified teture of the oil ainting whereas the ones with ositive sloes are images with shrunken teture. 2.2 Understanding image stitching in light fields Aligning images and stitching them into wide-angle anoramas are among the most widely used algorithms in comuter vision. A ractical stitching algorithm requires robust solutions for image alignment, waring, artifacts removal, etc. In this aer, we only focus on the mathematical model relating iel coordinates in one image to iel coordinates in another (see 7 for an ecellent survey). A inhole camera model is also used for the sake of simlicity. Figure 2: Image stitching with camera translation or rotation on the left and their reresentation in 2D light field on the right. The red slices on the to row denote the camera rotation whereas the blue ones on the bottom denote the camera translation. The green areas are used to reresent overlaing areas among cature images. A stitched anorama is usually made by joining multile hotograhs with slightly overlaing fields of view as shown in Figure 2 on the left. These images could either be taken by camera rotations around camera s otical center or camera translations arallel to the image lane Creating anorama by camera rotations To describe the motion model of light fields, we consider a inhole camera caturing a 1D image in the 2D light field L(,). Camera rotations around the otical center in 3D are reduced to a tilt of the inhole camera around its otical center for 1D images. The image catured by a inhole camera can thus be reresented as follows: [ I() = L( 0,), S 2f, S ] (2) 2f where 0 reresents the location of the camera, S reresents the size of the camera sensor and f reresents the focal length of the inhole camera. The original image I() and the image I r () after camera rotation satisfy I r (tan(arctan() θ)) = I() where θ is the rotation angle. Proc. of SPIE-IS&T/ Vol S-3

4 2.2.2 Creating anorama by camera translations As for the images catured by camera translations, [ the 1D images ] are slices of the light field at different camera locations such as L( 0,) and L( 0,) where S 2f, S 2f. Under the assumtion that the catured scene is a fronto-arallel Lambertian surface, the relation between two images can be reresented as I() = I (+ ), = ( 0 0 )f Z (3) where Z is the deth of the scene. As shown on the right of Figure 2, image stitching by camera rotations and translations can both be described in the light field. By rotating the camera, acquired images are shifted in the dimension and a wider field of view is achieved by merging these images directly. By translating the camera, multile images in the dimension can be seen as equivalent to multile images in the dimension when the catured scene is a lanar surface with Lambertian-reflectance roerties. In conclusion, image stitching mainly focuses on merging multile slices in the light field for a wider field of view in the dimension. However, light field stitching must consider both and dimensions which makes it a more general roblem. 3. MOTION MODEL FOR LIGHT FIELD STITCHING 3.1 Acquisition model of lenotic camera We consider a simlified lenotic camera model comrising a inhole camera moving behind a thin lens of focal length f. The distance between the inhole camera lane and the main lens is defined as b. Then the focused lane of the main lens is a meters away which satisfies the thin lens equation 1/f = 1/a + 1/b. We denote by L[m,n] the discrete 2D light field data catured by a lenotic camera where m and n resectively stand for the osition of the inhole camera and the iel inde within a inhole camera image. The 1D image obtained by fiing m is commonly referred as the inhole camera image whereas the 1D image obtained by fiing n is referred as the sub-aerture image. Then the imaging rocess shown in Figure 3 can be described as follows: L[m,n] = L(,)φ(/T m,/t n)dd = L(,),φ(/T m,/t n) (4) where T and T reresent the samling eriods on, R, [m,n] Z reresent the iel inde of the acquired light field data and φ reresents the samling kernel in the light field. L() (-) T L[n] Figure 3: Samling rocess of a lenotic camera. We use to reresent the light ray vector [ ] T, n to reresent the inde vector [m n] T and T to reresent the samling eriods diag(t,t ) for the sake of simlicity. By assuming a erfect otical system, the oint sread function of the main lens becomes a Dirac function. Each iel of the inhole camera is assumed to be infinitely small and free of noise. Then we can rewrite the samling equation Equation (4) as follows: L[n] = L(,),δ(/T m,/t n) φ(,) = L(,),δ mt,nt (,) = L(),δ T n (). (5) As described in Equation (5), each iel of the inhole camera catures one light ray emitted from the scene. The discrete light field L[m,n] can be directly related to the continuous light field L(). Proc. of SPIE-IS&T/ Vol S-4

5 To get the eact values of the samling eriods, we introduce the light field L (,) defined on the lane of the moving inhole camera as shown in Figure 4. The samling eriods on L (,) are directly defined by the inhole camera s motion and its sensor resolution. The satial samling eriod T is the moving ste size of the inhole camera and the angular samling eriod T is the iel size normalized by inhole camera s focal length. b T L[m,n] L (,) L(,) A b A f Figure 4: An illustration of the simlified lenotic camera. A inhole camera moves behind a thin lenswith a ste size of T. The light field L(,) is defined on the main lens lane whereas L (,) is defined on the inhole camera lane. The transformation between L(,) and L (,) is a combination of the refraction A f and the roagation A b. By using Equation (1), the transformation between the light field L(,) and L (,) is a combination of a refraction A f by the main lens and a b-meter roagation A b as follows: A = A b A f = [ 1 b 0 1 ][ 1 0 f 1 1 ]. (6) Therefore, we have L () = L(A 1 ) from which we can estimate the samling eriods on L() as follows: L[n] = L (),δ T n () = L(A 1 ),δ T n () = L(),δ T n (A) = L(),δ A 1 T n(). (7) In Equation (7), we used T = diag(t,t ) to reresent the samling eriods on the lane of the moving inhole camera where T and T are defined by the inhole camera s moving ste size and normalized iel size resectively. By using Equation (6), the samling eriods on L() which is defined on the main lens lane can be calculated as follows: [ ][ ] A 1 1 b T 0 T = f 1 1 bf 1 0 T [ ] T = bt f 1 T ba 1. (8) T The columns of the matri A 1 T are the samling eriods inhole-camera-wise and iel-wise resectively. The first column [T f 1 T ] T determines the distance between iels within the same inhole camera and the second column [bt ba 1 T ] T determines the distance between the same iel under two consecutive inhole camerasin the light field. Therefore, the rectangularlattice onl (,) becomes a arallelogramlattice on L(,) as shown in Figure 5. The arallelogram-shaed samling attern is secified by the camera arameters. 3.2 Translation model of light fields We first describe the motion model by only considering camera translations, namely when the lenotic camera is shifted in a lane arallel to the main lens and inhole camera lane. The 2D light field data L 0 (,) and Proc. of SPIE-IS&T/ Vol S-5

6 L (, ) L(, ) f 1 a 1 1 T f T bt b a T T T Figure 5: Samling oerator of lenotic camera on L (,) and L(,). The iels from the same inhole camera are shown inside the dashed rectangles. L 1 (+,) areacquired bymoving the lenotic camerafor meters in worldcoordinates. Then the discrete translation ( m, n) between these two acquired light fields is as follows: L 0 [m,n] = L 1 [m+ m,n+ n]. (9) By using the samling eriod A 1 T from Equation (6), we can relate the camera translation to the iel shift in the discrete light field as follows: [ ] [ ] A 1 m T = n [ ] [ ] A 1 m+ m + T = n+ n which can be further simlified by removing the common terms to derive [ ] [ ] A 1 m T = n 0 (10) which also leads to the linear relation between m and n as follows: n = a T m. (11) f b T Therefore, the 2D camera translation results in a 4D shift in the discrete light field data with only 2 degrees of freedom. The ratio between the iel shift in angular and satial dimension is determined by the secific arameters of the lenotic camera as shown in Equation (11). 3.3 Rotation model of light fields In this section, we describe the light field motion model by only considering camera rotations around the main lens otical center. A 2D eamle is used to show how a lenotic camera is rotated around its otical center by an angle φ clockwise in Figure 6. The blue line with two arrows reresents the main lens of the original lenotic camera whereas the green one reresents the main lens after rotation. The dashed lines denote the same light ray defined on these two main lens lanes. The variable L r ( r, r ) is used for the light field after rotation to distinguish it from the original light field L(, ). Proc. of SPIE-IS&T/ Vol S-6

7 Lr( r, r ) r L(,) atan() Figure 6: The lenotic camera is rotated by φ around the otical center of its main lens. The dashed lines L(,) and L r ( r, r ) denote the same light ray catured by the lenotic camera before and after rotation. The geometric relations between ( r, r ) and (,) are shown on the left. By using geometric relations between ( r, r ) and (,), we conclude that = r tanφ 1+ r tanφ = tan(arctan( r) φ) (12) = r sinφ + r cosφ (13) In Equation (12), we observe that the light field transformation by camera rotation is not a linear oeration in the dimension. As forthe transformationin the dimension, the shift deends on both the camerarotationand the direction of the light ray. Only when the rotation angle φ is very small, then the light field transformation by camera rotation can be seen aroimately as a linear transformation. Therefore, etending the light field in dimension by camera rotation can only work for small rotation angles. As φ increases, the arallelogram shae of the samling attern after rotation cannot hold when aligning to the original light field. Therefore, we don t consider light field stitching urely by camera rotations. 4. EXPERIMENTS AND DISCUSSIONS Based on the roosed motion model, we use the lenotic camera as an image scanner and erform light field stitching to increase the size of the acquired light field data. We reort two alications in light field stitching by camera translations and a combination of camera translations and rotations. Our main goal here is to verify the motion model and demonstrate its otential for light field acquisition. 4.1 Light field stitching by camera translation By using Equation (7), we establish the relations between the camera motion and the acquired light field data defined by the inhole camera inde m and the iel inde n. Then we can simly cature multile light fields without overlaing areas as shown in Figure 7. A direct stitching aroach is demonstrated with the blue dashed arallelogram. Each inhole camera image is etended at the cost of reducing the size of the sub-aerture image (same iel inde in all inhole cameras). The reduction in satial range is actually due to the fact that the catured data has the shae of a arallelogram in the 2D light field defined on the main lens lane of the lenotic camera. To avoid reducing the size of the sub-aerture image, we roose to resamle the catured light field to increase the field of view. As mentioned in Section 2.1, slices with different sloes are actually images catured virtually by cameras at different lanes. By carefully choosing the slicing sloes in the light field, the sub-aerture image reresented with the blue dashed line obtains a wider field of view as shown in Figure 8. Based on the strategy in Figure 8, a virtual lenotic camera in Matlab is used to cature multile light fields at a close distance to a slanted lane ainted with ink circles. The teture is chosen to be a band-limited signal. Since the teture is ainted on a slanted lane, the light field is also a band-limited signal as shown in. 8 The Nyquist condition is satisfied by the samling eriods in the 4D light field. The dimension (n,n q,m,m y ) of each acquired light field data is A total number of light field acquisitions are used for light field stitching. One stitched sub-aerture image of the final anorama light field is shown in Figure 9 with a resolution of (31 21) (31 21). Proc. of SPIE-IS&T/ Vol S-7

8 Original light field Stitched light field Figure 7: The red arallelograms reresent the light field taken by lenotic cameras at 0, 1 and 2 and the blue arallelogram with dashed line reresents a direct stitched light field by etending the inhole camera image. Re-samled Image Original light field Figure 8: Re-samle multile light field samles to create a virtual light field image taken behind the camera lane. 4.2 Light Field Stitching with Camera Rotation and Translation We can also erform light field stitching by combining camera rotations and translations. Therefore, we can etend each inhole image directly without resamling. As shown in Figure 7, inhole camera images at the boundaries of the light field acquisition cannot be etended because of the arallelogram shae. By rotating the lenotic camera, acquired light fields are shifted in dimension in order to etend each inhole camera image as shown in Figure10. The simulation results of 2D light field stitching by both camera rotations and translations are given in Figure 10. Each lenotic camera with different translation is rotated around the otical center of its main lens. The secific camera arameters for the simulation is shown in Table 1. The arameters are similar to the ones of a Lytro. Then we first use Equation (6) to calculate the range of the each inhole camera image in dimension as Proc. of SPIE-IS&T/ Vol S-8

9 n n q m m y m m y n n q Figure 9: A anorama image created from light field images. One sub-aerture image of the final anorama light field is on the left and one acquired light field image with different dislay orders (n,n q,m,m y ) (an array of microlens image) and (m,m y,n,n q ) (an array of sub-aerture image) is on the right. Plenotic camera arameters for simulation Main Lens f a b 6.45 mm 0.3 m 6.9 mm Microlens Array itch f b m mm mm Sensor itch angular dimension satial dimension m Table 1: Secific camera aramters for light field stitching simulation. follows: = N T b (14) where N reresents the size of each inhole camera image, T reresents the samling eriod in for each iel and b reresents the distance between the main lens and the inhole camera lane. Then the camera is translated for to maimize the stitched range in. As shown in Figure 5, each inhole camera image is a slice in the light field with a sloe 1/a. To comensate for the arallelogram shae, each inhole camera image should be shifted by /a in the dimension. By using Equation (12), the lenotic camera is rotated around the main lens otical center by an angle Φ as follows: which is an aroimation by assuming the original to be zero. Φ = arctan( a ) (15) As shown in Figure 10, five light field images are generated with different translations ( 2,, 0, and 2 ). To comensatefor the shift in dimension, eachcamerais rotatedfor anangle calculatedby Equation (15). From the simulation results in Figure 10, we can observe that the required range of the light field to etend each inhole camera image is covered by the acquired data. Further treatment for interolation are also needed esecially when φ increases. In conclusion, light field stitching by camera rotations and translations effectively increases the size of each inhole camera image of the light field. However, camera rotations are usually more difficult to imlement. Due to non-linearity of the light field transformation by camera rotation, this stitching method only works effectively for when the rotation angle φ is small. Proc. of SPIE-IS&T/ Vol S-9

10 /meters atan(2 /a) atan( /a) Figure 10: Five lenotic cameras with different translations ( 2,, 0,, 2 ) and rotations are used to samle the light field. As φ increases, the distortion of the arallelogram shae becomes more obvious. 4.3 Conclusion and future work In this aer, we roose a motion model of a simlified lenotic camera and resent its alications in light field stitching. The lenotic camera is used as a scanner and multile images are merged into one light field with larger satial range or angular range. Firstly, light fields by camera translations are resamled to create light field anoramas. By slicing the acquired multile light fields with different sloes, each sub-aerture image of the new light field has a wider field of view. Secondly, by combining camera rotations and translations, the angular range of the light field can be increased directly. The motion model of camera rotation is described and verified by the simulation of a 2D light field which can be easily etended to 4D. But this tye of acquisition is limited to a certain angular range because of the non-linearity of the light field transformation by camera rotation. In the future, we will etend our work to light field registration and suer-resolution based on the roosed motion model. ACKNOWLEDGMENTS This research is being suorted by Google Focus Award-eFacsimile research, SNSF grant number and ERC Advanced Investigators Grant: Sarse Samling: Theory, Algorithms and Alications SPARSAM no REFERENCES [1] Ng, R., Fourier slice hotograhy, in [ACM SIGGRAPH 2005 Paers], SIGGRAPH 05, , ACM, New York, NY, USA (2005). [2] Lumsdaine, A. and Georgiev, T., The focused lenotic camera, in [In Proc. IEEE ICCP], 1 8 (2009). Proc. of SPIE-IS&T/ Vol S-10

11 [3] Birklbauer, C. and Bimber, O., Panorama light-field imaging., in [SIGGRAPH Posters], ACM (2012). [4] Levin, A. and Durand, F., Linear view synthesis using a dimensionality ga light field rior, in [In Proc. IEEE CVPR], 1 8 (2010). [5] Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F., The lumigrah, in [Proceedings of the 23rd Annual Conference on Comuter Grahics and Interactive Techniques], SIGGRAPH 96, 43 54, ACM, New York, NY, USA (1996). [6] Levoy, M. and Hanrahan, P., Light field rendering, in [Proceedings of the 23rd Annual Conference on Comuter Grahics and Interactive Techniques], SIGGRAPH 96, 31 42, ACM, New York, NY, USA (1996). [7] Szeliski, R., Image alignment and stitching: A tutorial, Found. Trends. Comut. Grah. Vis. 2, (Jan. 2006). [8] Do, M. N., Marchand-Maillet, D., and Vetterli, M., On the bandwidth of the lenotic function, Trans. Img. Proc. 21, (Feb. 2012). Proc. of SPIE-IS&T/ Vol S-11