Plenoptic Rendering With Interactive Performance Using GPUs

Transcription

1 Plenotic Renering With Interactive Performance Using GPUs Anrew Lumsaine a, Georgi Chunev a, an Toor Georgiev a Iniana University, Bloomington, IN, USA; Qualcomm, San Diego, CA, USA ABSTRACT Processing an renering of lenotic camera ata reuires significant comutational ower an memory anwith. At the same time, real-time renering erformance is highly esirale so that users can interactively exlore the infinite variety of images that can e renere from a single lenotic image. In this aer we escrie a GPU-ase aroach for lightfiel rocessing an renering, with which we are ale to achieve interactive erformance for focuse lenotic renering tasks such as refocusing an novel-view generation. We resent a rogression of renering aroaches for focuse lenotic camera ata an analyze their erformance on oular GPU-ase systems. Our analyses are valiate with exerimental results on commercially availale GPU harware. Even for comlicate renering algorithms, we are ale to rener 39Mixel lenotic ata to 2Mixel images with frame rates in excess of 500 frames er secon. Keywors: Focuse lenotic camera, real-time renering, GPU, GLSL 1. INTRODUCTION Integral hotograhy has more than 100 years of history, starting with Ives 1 an Limann. 2 Limann motivate his work in this area y oserving that even the most erfect hotograhic rint only shows one asect of reality; it reuces to a single image fixe on a lane, similar to a rawing or a han-rawn ainting. With integral hotograhy he attemte instea to rener infinitely more the full variety offere y the irect oservation of ojects. Unfortunately, ecause of inherent limitations of technologies availale at the time, the otential of integral hotograhy was not realize. Integral hotograhy re-emerge in the 1990s with the introuction of the lenotic camera, originally resente as a techniue for caturing 3D imagery an for solving comuter vision rolems. 3 It was esigne as a evice for caturing the istriution of light rays in sace, i.e., the 4D lenotic function. The lightfiel 4 an lumigrah, 5 introuce to the comuter grahics community, estalishe a theoretical framework for analyzing the lenotic function. A han-hel lenotic camera, along with new methos of rocessing, was introuce y Ng. 6 Making integral hotograhy ractical reuire two final stes. First, the images renere from a lenotic camera neee to e at a resolution accetale to moern hotograhers. The traitional lenotic camera renere 2D images that were uite small (e.g., 90k ixels 6 ). Recent work on the focuse lenotic camera greatly increase the availale satial resolution of lenotic cameras to a level comarale to that of non-lenotic igital cameras. 7 The final ste, which we resent in this aer, is to enale rocessing of lightfiel ata at interactive frame rates. Because of the infinite variety of images that can e rouce from a cature lightfiel, interactivity is an essential feature for the en user. We achieve this level of erformance y taking avantage of the ower an rogrammaility of moern Grahics Processing Units (GPUs). The massively-arallel high-erformance architectures of moern GPU latforms are well matche to lightfiel rocessing an renering. In aition, moern GPUs can e rogramme in a numer of ways, making this comutational ower irectly availale to en users. In this aer we escrie our GPU-ase aroach for lightfiel rocessing an renering, with which we are ale to achieve real-time erformance for well-known lightfiel tasks such as novel-view generation, refocusing, an stereo renering. The lan of the aer is as follows. Section 2 contains an overview of relate work. In Section 3 we rovie ackgroun information aout lightfiel rocessing an renering. Our aroach to imlementation of lightfiel rocessing renering tasks using the OenGL Shaing Language is given in Section 4. The results of erformance exeriments are given in Section 5 an we conclue in Section 6.

2 1 1 ( 1, 1 ) 2 2 ( 2, 2 ) Figure 1: Rays are reresente with coorinates an. The sace of all rays comrises the - hase sace. 2. RELATED WORK Interactivity has long een recognize as an imortant feature for lightfiel renering (e.g., see aers y Antunez et al., 8 Isaksen et al., 9 an Ng et al. 6 ). At the same time, the comutational reuirements of lightfiel renering were an imeiment to interactivity, articularly as lightfiel imagery grew into the gigaixel range. Particularly as CPU erformance has lateaue, there is interest in using GPUs in almost all areas of comuting where erformance is a concern. Comuter grahics an comuter vision are no excetion. 10 For examle, Tot et al. aly GPUs to the rolem of accelerating image-ase renering in comuter grahics. 11 In an unulishe technical reort, Meng et al. teste the erformance of several lenotic 1.0 renering algorithms on various harware: single core rocessors, multirocessors, an a GeForce 7 series GPU. 12 They ientifie the unerutilization of the GPU s resources, ue to the fixe ieline, as its rawack, an issue that has een remove entirely since the introuction of unifie shaer architecture GPUs y Nviia. 13 Luke et al. focus on eveloing ractical technological solution for 3D TV. 14 They suggest a art of the solution will involve lenotic (1.0) vieo cameras, which have the same rawacks as lenotic cameras, namely low-resolution images ue to satio-angular traeoffs. A GPU imlementation is escrie for the suer-resolution an istance extraction algorithm resente y Owens et al Raiance 3. BACKGROUND The lenotic function 3 (also calle the lightfiel 4 or raiance 15 ) is a ensity function escriing the light rays in a scene. Since the raiance is a ensity function over the ray sace, we escrie raiance transformations via transformations alie to elements of the unerlying ray sace. Rays in three imensional sace are reresente as four imensional vectors: Two coorinates are reuire to escrie osition an two are reuire to escrie irection. Following hysicsase conventions, 16 we enote the raiance at a given lane erenicular to the otical axis as r(, ), where escries the location of a ray in the lane an escries its irection. (These coorinates are also use in otics texts such as those y Gerrar 17 an Wolf. 18 ) For illustrative uroses, an without loss of generality, we aot the convention of a two-imensional - lane in this aer. A two-imensional image is create from the four-imensional raiance y integration over all rays incient at a given. Given a raiance r(, ) at the image lane of a camera, an image I() is renere for a given range of the availale values accoring to I() = r(, ). (1)

3 I() = r(, ) I() Figure 2: The sensor ixels in a traitional camera cature an image y hysically integrating the intensities of all of the rays iminging on them (a) Rays converging at a inhole will searate from each other as they travel ehin it an can e cature iniviually y a sensor ehin the inhole. () Reresentation of Figure 3a in hase sace. Iniviual inholes samle one osition in the - lane, while iniviual ixels samle ifferent irections. (c) A single image cature ehin the inhole thus samles a vertical strie in the - lane, while an array of inholes samles a gri. Figure 3: Raiance samling with inholes. 3.2 The Limann Sensor As shown in Figure 2, a traitional camera catures an image y hysically integrating the intensities of all of the rays iminging on each sensor ixel. A lenotic camera, on the other han, catures each ray searately. One aroach to searating, an then iniviually caturing, the rays in a scene is to ut a inhole where the sensor ixel woul e while lacing the sensor itself some istance ehin the inhole. In this case, the rays that converge at the inhole will iverge as they roagate ehin the inhole. The searate ixels in the sensor now cature searate rays. I.e., the intensity as a function of osition at the sensor reresents the raiance as a function of irection at the osition of the inhole. A single image cature ehin the inhole thus samles a vertical strie in the - lane, while an array of inholes samles a gri. This rocess is illustrate in Figure 3 Although the ieal inhole makes an ieal ray searator, in ractice microlenses are use instea to gather sufficient light an to avoi iffraction effects. Figure 4 shows a iagram of such a sensor, which we will refer to as the Limann sensor (after Gariel Limann who first roose it 2 ). In the iagram, is the istance from the sensor to the microlens lane an a is the istance from the microlens lane to the main lens image lane. The microlens focal length is f; a,, an f are assume to satisfy the lens euation 1/a + 1/ = 1/f. Sensor ixels have size δ an without loss of generality we take to e the microlens aerture an the sacing etween microlenses. A secific instance of the Limann sensor was roose y Ng, 6 in which the istance was taken to e eual to the microlens focal length f. In this case, as has een erive elsewhere, 6 we have the following exression for how the image

4 I () r(, ) r a (, ) δ = a I () = r a( a, 1 ) a Figure 4: Geometry of Limann sensor for a lenotic camera. The CCD (or CMOS) sensor is lace at a istance ehin an array of microlenses. In one notale limit, f an a. δ a δ a δ f 1 a a f a (a) Samling y the Limann sensor at a finite istance in front of the microlenses, where a,, an f satisfy the lens euation. Note that shearing changes the with of the satial region samle y a single ixel. () Samling y the Limann sensor at the microlens array when the istance etween the microlens array an the sensor is eual to the microlens focal length (i.e., when = f). Figure 5: Raiance samling y the Limann sensor in ifferent lenotic cameras. The geometry of a single ixel for each case is shown in the uer right. cature on the sensor (I f ) samles the raiance at the microlens lane (r): This samling is shown grahically in Figure 5. I f () = f r(0, 1 ). (2) f A ifferent realization of the Limann sensor was roose y Lumsaine an Georgiev, 7 in which the istance was chosen not to e eual to f in orer to form a relay system with the camera main lens. In this case, as erive y Lumsaine an Georgiev, 7 we have the following exression for how the image cature on the sensor (I ) samles the raiance at the microlens focal lane (r a ): This samling is shown grahically in Figure 5a. I () = r a( a, 1 ) (3)

5 f (a) The traitional lenotic camera focuses the main lens on the microlens array of the traitional Limann sensor. a () The focuse Limann sensor in the focuse lenotic camera acts as a relay system with the main lens. Figure 6: A comarison of the traitional an focuse lenotic cameras. As reorte y Lumsaine an Georgiev, 7 each microimage cature y the focuse Limann sensor reresents a slante strie of the raiance at istance a in front of the microlens array, allowing significantly higher satial resolution when renering Plenotic Cameras A lenotic camera can e constructe y sustituting a Limann sensor for the traitional sensor in the camera oy. The rincile esign issue then ecomes the main lens otics. The main lens of the camera will ma the light in the exterior worl into the light in the interior worl of the camera. This maing will resent a certain lenotic function insie the camera. In the case of Ng s han-hel lenotic camera, the main lens focal lane (the rimary lane of hotograhic interest) was mae to the lane of the microlens array. In the case of the focuse lenotic camera, the main lens focal lane was mae to a istance a in front of the microlens array. A comarison of the traitional lenotic camera with the focuse lenotic camera is shown in Figure Renering Euation (1) efines renering for the continuous case. In the iscrete case, renering ecomes a summation: I[k] = j r[k, j] (4) where the suare rackets inicate inexing into iscretize uantities. We note however, that in general, a lenotic camera oes not rovie us irectly with an array containing r[k, j]. For examle, the focuse lenotic camera catures an image reresenting the raiance accoring to Euation (3). In that case we cannot simly sum u along a single array axis to otain an outut ixel. One aroach to renering woul e to suitaly interolate the cature lenotic function to a gri so that Euation (4) coul e alie a rocess that is oth memory an comute intensive an one which can introuce noise an other unwante artifacts. An alternate aroach, an the one we use in this aer, is to rener irectly from the cature lenotic function. To accomlish this, we sustitute the iscrete form of Euation (3) into Euation (4) to otain the iscrete lenotic renering formula: I[k] = I [k a j, j]. (5) j In other wors, renering is accomlishe y summing u ixels along a slante line corresoning to the minification factor, as illustrate in Figure 7. We note that ue to linearity, shearing transformations of the raiance will correson to ifferent slants for summation in the cature lenotic function, an hase sace volume of each ixel is conserve.

6 j j I [k, j] r[k, j] k k a r[k, j] = I [k j, j] Figure 7: Summation of the raiance in the vertical (angular) irection at the main lens image is euivalent to summation at a slant of the raiance at the microlenses. Figure 8: Screenshot of our OenGL lenotic renering alication. The sliers in the right-han anel are use to interactively change the values of uniform variales share etween the control rogram an the shaers. 4. GPU IMPLEMENTATION OF RENDERING 4.1 Programming the GPU A numer of rogramming languages an tools have emerge for GPU rogramming: the OenGL Shaing Language (GLSL),19 Cg,20 CUDA,21 an OenCL.22 Whereas GLSL an Cg are aime at renering an relate tasks in min, CUDA an OenCL are aime irectly at general urose rogramming tasks. Because lightfiel renering is similar to traitional renering, an ecause a key art of interactively working with lightfiels is to islay them, GLSL is well-suite to our nees. Our GLSL imlementations of lightfiel renering are carrie out comletely in OenGL fragment shaers. Although our examles were imlemente in Python (via the PyOenGL lirary23 ), interfaces to OenGL are similar across languages. Other than roviing suort for creating an installing the shaer, the main functionality rovie y OenGL were the following: 1. Rea in the raiance ata (from a store image), 2. Serialize the raiance ata to a format suitale for OenGL, 3. Create a 2D OenGL texture oject for the ata,

7 Plane = M a Plenotic M Microimage a Sensor Plane (a) cature geometry. For otimal sensor coverage, the iameter of a microimage circle nees to e eual to the istance etween microlenses, the microlens itch. M Renere M = a () Renering a single view from the raiance cature y the focuse lenotic camera is the inverse of cature. A single view sans multile ixels in each cature microimage. Figure 9: cature an single view renering for the focuse lenotic camera. 4. Create an OenGL scene geometry of a single ua to which to rener the texture, an 5. Uate share ( uniform ) variales use y the shaer. Renering the lenotic image ata is then accomlishe y renering the installe texture, using our custom shaer. The OenGL suort coe is omitte here for sace reasons. A screenshot of our OenGL lenotic renering alication is shown in Figure Basic Focuse Plenotic Renering Our asic focuse lenotic renering algorithm reners only one angular samle er ixel, eening on a user secifie magnification factor /M (see Figure 9), which shows how the focuse lenotic camera catures the image lane on its sensor. Here is the size of each microlens image i.e., the iameter of the microimage circle. M is the size of a suare atch that we ick from it. In other wors, we use only a suare atch of M 2 ixels from each microimage, as oose to using all ixels in the microimage circle of iameter. We remark that one way of interreting this renering rocess is that we take regions of size M from each microlens image in the flat an then magnify (y a/, or euivalently /M) an fli them, then iece them together to make an outut image as if cature irectly at the main lens focal lane. This rocess, illustrate in Figure 9, is the inverse of the otical rocess in the microlenses uring lenotic image cature. This rocess is interrete in hase sace in Figure 10, where M is selecte to e such that there is no overla of ixels with same values from ifferent microlenses. For examle, all samles from one microlens are insie the grey ox, i.e. M = 4. We can ick any other atch of M 2 ixels insie the microlens, which woul correson to a ifferent grey ox move higher in an renering a ifferent view. 4.3 Renering Directly from the Plenotic Although the lenotic function is four imensional, a hysical lenotic camera must necessarily cature this 4D function as a 2D array of 2D microimages. For renering, one coul first convert this flat (the flattene lenotic function) into a 4D array. However, this conversion rocess may not e as straightforwar as simle inex remaing ecause the microimage size may not e an exact integer numer of ixels or the microimage array axes may not e recisely aligne with the CCD ixel axes. In this case, the lenotic ata woul nee to e interolate to the 4D array, a rocess that is comutationally exensive, an that can introuce noise into the ata. Moreover, as escrie elow, we access the lenotic ata as a texture on the GPU car, a rocess universally suorte in 2D, ut not suorte in 4D. However, the ata are the same,

8 Microimage Pixels View Pixels Renere Figure 10: Phase sace interretation of renering a single view from focuse lenotic ata. k v M Plenotic Microimage Renere x M = a Figure 11: Basic renering irectly from the lenotic image (without lening). Note the magnification a/. regarless of the imensionality of reresentation. Therefore, as riefly iscusse aove ( 3.4) an escrie more fully elow, our aroach using the GPU is to rener irectly from the flat. Let e the size of one microlens image, measure in ixels. For a given oint x in the outut image, we nee to fin the corresoning samling oint on the flat. To accomlish this rocess, we nee to erform two comutations. First, given x, we nee to etermine from which microlens we will rener x. Secon, we nee to etermine where in the region of size M the oint x lies. To comute which microlens corresons to x, we simly take the integer art of x ivie y. Let us enote the microlens numer y k, given y k = x. (6) The ixel location of the eginning of that microlens in the flat is then given y multilying the microlens numer y the size of one microlens image, i.e., k. Next, we nee to comute the offset within the region of size M corresoning to x. To o this, we comute the ifference etween x an the start of microlens (k). This will give the offset in the renere image, ut we nee the offset in the flat. Since we are scaling a region of size M in the flat to a region of size in the final image, the offset nees to e scale y M, which is the flat image ste size corresoning to a 1 ixel ste in the renere image. That is, the offset v is given y v = (x x ) M = ( x k ) M. (7) Finally, we nee to make one more ajustment. We want the center of the M M region of the flat to rener to the center of the corresoning region of the final image. The formulas aove will ma the left ege of the microlens image to

9 uniform samler2drect flat ; uniform float M, ; uniform vec2 offset ; // Plenotic image inut // Share variales asse y control rogram // is microlens size, M is atch size voi main() { vec2 x = gl TexCoor[0]. st /; // x/ (Note: gl TexCoor[0]. st is x) vec2 k = floor (x ); // k = x/ vec2 v = (x k) M; // (x/ k)m vec2 v = v ( M); // v = v + ( M)/2 vec2 fx = k + v + offset ; // f(x) = k + v gl FragColor = texture2drect( flat, fx ); } // Looku ixel value Figure 12: GLSL fragment shaer coe for asic focuse lenotic renering. the left ege of the corresoning region in the renere image. To accomlish this centering, we a an offset of M 2 to v: v = v + M ( x ) = 2 k M + M. (8) 2 Comining (6) an (8), the corresoning oint in the flat for a given oint x in the outut image is given y f(x) where f(x) = k + v. (9) The GLSL fragment shaer coe to carry out this algorithm is otaine in a straightforwar fashion from this formula an is shown in Figure 12. The lenotic image as well as the values for an M are rovie y the user rogram via uniform variales. The shaer rogram comutes v, v, an f(x) as v, v, an fx, resectively. Novel view generation is enale y aing a user-secifie offset vector (eual to (0, 0) y efault) is ae to the coorinates fx, resulting in a shift in the viewoint of the renere image. Finally, we look u the value of the ixel in the flat an assign that value to the reueste fragment color. 4.4 Renering with Blening To realize the integration over in euation (1), we must average together ( len ) the same satial oint across microlens images. For microlenses sace aart an a atch size of M, the ixels that nee to e average together to a given ixel in the renere image will e istance ( M) aart. That is, we average all ixels at osition f i (x) where where f i (x) = k i + v i (10) k i = x + i( M) (11) v i = v + M 2 im (12) for i =..., 2, 1, 0, 1, 2,... Since is constant this means that for images generate with a given samling itch M there is a fixe uer oun for the numer of microimages that can e samle to contriute to a given x. This uer oun, lim, is given y lim = ( M 1 ). (13) 2

10 Plenotic M M Microimage Pixels Microimage View Renere = M a (a) Renering with lening. Pixels Renere () Phase sace interretation of renering with lening. Figure 13: Otical an hase-sace interretations of renering with lening. uniform samler2drect flat, weightm; uniform float M, ; uniform int lim; // Plenotic image inut an weight mask // Share variales asse y control rogram // is microlens size, M is atch size // lim is numer of neighors to len together voi main() { vec2 x = gl TexCoor[0]. st /; // x/ (Note: gl TexCoor[0]. st is x) vec2 k = floor (x ); // k = x/ vec2 v = (x k) M; // (x/ k)m vec2 v = v ( M); // v = v + ( M)/2 vec4 colxy = vec4 (0.0); float total weight = 0.0; for ( int i = lim; i <= lim; ++i) { for ( int j = lim; j <= lim; ++j) { vec2 ij = vec2( float ( i ), float ( j )); vec2 v = v ij M; vec2 vposxy = (k + ij ) + v; float weight = texture2drect(weightm, v); colxy += texture2drect( flat, vposxy) weight; total weight += weight; } } gl FragColor = colxy / total weight ; } // For each microimage in // lim lim neighorhoo // Comute ixel location // Look u weight value // Look u ixel value an scale y weight Figure 14: GLSL fragment shaer coe for focuse lenotic renering with weight mask for lening.

11 Plane = M 1 a 1 Plane = M 2 a 2 a 1 a 2 Sensor Plane M 1 = a 1 Sensor Plane M 2 = a 2 Figure 15: The focuse lenotic focusing rincile. lanes at ifferent eths are renere through aroriate choice of M. In the focuse lenotic camera, it is likely that the ixels in the center of the microlens image are of higher uality than the ixels close to the eges. Moreover, since lim an M may e secifie y the user, it is ossile that we may samle out of ouns ixel values. Accoringly, we may want to weight the ixels ifferently in our summation rocess. To accomlish weighte lening within GLSL, we use another texture (single-comonent in this case) as a looku tale to secify the weight of each ixel in the microlens as a function of osition. If fragment coorinates fall outsie the weight mask then the texture wraing moe woul etermine what haens with the looku. This situation occurs when the weight mask is smaller than one microlens image, or when the chosen lim value is larger than the one otaine from euation (13). We suggest the use of a Gaussian mask with GL CLAMP set as the wraing moe. The GLSL imlementation is shown in Figure Focusing an Refocusing An imortant characteristic of the focuse lenotic camera the focuse lenotic focusing rincile can e seen in Figure 15. The focuse lenotic focusing rincile states that within the eth of fiel of the microcameras in the Limann sensor, the image lane that is in focus is etermine y the choice of M. To see this, consier the left an right illustrations in Figure 15, which show two ifferent image lanes focuse at the sensor. We assume that an are the same in oth cases an that the microlens aerture is small enough that the microimages in oth cases are in focus. Since is constant, the istance to the focal lane etermines the value of M that nees to e use to roerly rener that lane. The focuse lenotic focusing rincile irectly follows as the converse: The choice of M use to rener the outut image etermines the lane in the image that is in focus. s cature with the focuse lenotic camera can therefore refocus renere images y choosing ifferent values of M. Thus, the GLSL shaers shown in Figures 14 an 12 are alreay refocusale. All that is reuire is to vary the value of the uniform variale M, something that can e one interactively via the control rogram associate with shaer. Tyical (interesting) scenes are not at a uniform eth throughout the scene. Choosing a single value of M will rener in focus those arts of the scene that correson to eth a = M. Other arts of the scene will not e renere in focus. In the case of single-view renering, out of focus ortions of the scene will e renere with visile artifacts ue to misalignment of atch ounaries. 7 In the case of renering with lening, if there are a sufficient numer of views to len, lening the misaligne atch ounaries results in the execte focus lur (see Figure 17). If there are not sufficient numer of views to suress the artifacts, other techniues such as eth-ase renering must e use Exerimental Setu 5. EXPERIMENTAL RESULTS Renering tests were erforme on a DELL Precision T7400 with two Intel Xeon rocessors at 3.40GHz, 32.0GB RAM. The GPU was a Quaro FX 5600, with eight 16-core stream multi-rocessors clocke at 1.35GHz an connecte to a

12 6x64it 800MHz (effective 1600MHz) vieo memory us. The oerating system use was 64-it Vista. We use OenGL rivers from Nviia, release The shaer rograms were run using a simle wraer rogram written in Python (2.5). Plenotic image ata was acuire with a meium format camera with an 80-mm lens an a 39-megaixel P45 igital ack from Phase One. The lens is mounte on the camera with a 13-mm extension tue, which rovies the neee sacing a. The microlens array is custom mae y Leister Axetris. The microlenses have focal length of 1.5 mm so that they can e lace irectly on the cover glass of the sensor. We are ale to rovie aitional sacing of u to 0.5 mm to enale fine tuning of the microlens focus. The itch of the microlenses is 500 µm with recision 1 µm. The sensor ixels are 6.8 µm. The value of 1.6 mm was estimate with recision 0.1 mm from known sensor arameters an ineenently from the microlens images at ifferent F/numers. A cro from the lenotic is shown in Figure 16. Notice the use of suare main lens aerture (for efficiency), resulting in suare microimages. Figure 17a shows a cro from image renere using the asic renering algorithm (an the asic renering shaer). Notice the locky artifacts of areas that are not in focus, i.e. for which a/ is not right. Figure 17 shows a cro of the same region as Figure 17a. Notice the artifacts are now suresse an that the ortions of the image for which a/ is not right aear out-of-focus (lurre). The renerings are one interactively at 1600 an 500 frames er secon, resectively. Figure 16: A small cro of aout 5% from a 39 megaixel raw image (flat) cature with our Plenotic 2.0 camera. 5.2 Performance With our GPU ase aroach, we effect lenotic renering y renering a texture to a single OenGL ua. The size of that ua is secifie y the user, i.e., it is the size of renere image to e islaye to the screen (or save to a file). Conseuently, the amount of comutation to rener the final image is roortional to the size of the renere image, not the size of the lenotic ata. Moern GPU architectures are uite sohisticate an although our shaers are uite straightforwar, moeling their comutational erformance woul e uite comlicate. However, a simle lower oun on erformance can e estimate y assuming that renering will e rimarily memory-oun. The GPU car we use for our exeriments, an Nviia Quaro FX 5600, has eight 16-core stream multi-rocessors clocke at 1.35GHz an connecte to a 6x64it 800MHz(effective 1600MHz) vieo memory us. With these secifications, the execte vieo memory anwith is 6*(8 Bytes)*1600MHz, or 76.8GB/sec (71.5GiB/sec), which can eliver 19.2 illion IEEE 32FP color comonents to the stream rocessors. There is some aitional enefit ue to caching (which is not as significant a enefit in GPUs as it is for CPUs). We exect the gain from caching to e 1.35/0.8, which is the ratio etween the GPU memory clock an the

13 (a) renere using asic algorithm. Note the resence of artifacts at the mountains at infinity ue to non-matching atches at that eth. () renere using weighte lening algorithm. Note that the artifacts at infinity are now suresse an that the out-of-focus regions aear roerly lurre. Figure 17: Comarison of asic (a) an weighte lening () renering. (a) M = 1, lim = 1. () M = 10, lim = 1. Figure 18: Performance comarisons of the shaers resente in this aer showing time to rener an outut image from lenotic ata as a function of outut image size. Iealize erformance is given in grey, an aitionally marke with, x, an + for the asic, simle lening, an weighte lening shaers, resectively. 16Bytes shaer clock. In the case of uncache fetches, one 4-comonent (128it) ixel reuires 71.5GiB/s secons to comlete. Accoringly, the time to rener a single frame for a memory-oun shaer can e exresse as SP F = ((nf ncf ) + ncf (0.8/1.35)) 16 OR 71.5 (14) where, nf is the numer of fetches er shaer call; ncf is the numer of cache fetches er shaer call; an OR is the outut resolution, measure in gigaixels. For the asic shaer, we have nf = 1 an assume ncf = 0. For the simle an weighte lening shaers we have nf = (2lim + 1)2 an assume ncf = 3lim2 + 2lim. There are aitional memory (2lim + 1)2 looku costs for the weighte lening shaer ue to the weights, which woul all e cache. For our exerimental erformance measurements, we measure the time taken to rener outut images of varying sizes,

14 given the same lenotic image source, an comare the results to our iealize estimates. The lenotic image ata was a color image, an the assigne weight mask was a grayscale suare Gaussian. Our erformance results are shown Figure 18, which lot time er frame as a function of outut image size. The grahs show results for asic, simle lening, an weighte lening shaers, with lim = 1 an M = 1 on the left grah M = 10 on the right grah. Also lotte are the times reicte y our memory-oun moel. Our memory-oun moel slightly overestimates the erformance of the asic shaer an of the lening shaers for lim = 1, ecause we have not taken into account latencies ue to raster oerations an ecause we neglect comutational latencies, which contriute to the larger eviation from the moel at smaller image sizes in the asic an weighte lening shaers. 6. CONCLUSION In this aer we have resente an aroach to full-resolution lightfiel renering that uses commercially availale GPU harware to rener images at interactive rates. Although GPU harware rovies a significant amount of comuting ower, for renering lenotic image ata we foun that vieo memory anwith was also an imortant factor for achieving interactive erformance. Using commercially availale (an economical) harware to interactively rener lenotic image ata into final images with high resolution is another ste towars making lenotic cameras ractical. REFERENCES [1] Ives, F., Patent US 725,567, (1903). [2] Limann, G., Ereuves reversiles. Photograhies integrales., Acaemie es sciences, (March 1908). [3] Aelson, T. an Bergen, J., The lenotic function an the elements of early vision, in [Comutational moels of visual rocessing], MIT Press (1991). [4] Levoy, M. an Hanrahan, P., Light fiel renering, Proceeings of the 23r annual conference on Comuter Grahics an Interactive Techniues (Jan 1996). [5] Gortler, S. J., Grzeszczuk, R., Szeliski, R., an Cohen, M. F., The lumigrah, ACM Trans. Grah., (1996). [6] Ng, R., Levoy, M., Breif, M., Duval, G., Horowitz, M., et al., Light fiel hotograhy with a han-hel lenotic camera, Comuter Science Technical Reort CSTR (Jan 2005). [7] Lumsaine, A. an Georgiev, T., The focuse lenotic camera, in [International Conference on Comutational Photograhy], (Aril 2009). [8] Antunez, E., Barth, A., Aams, A., Horowitz, M., an Levoy, M., High erformance imaging using large camera arrays, International Conference on Comuter Grahics an Interactive Techniues (Jan 2005). [9] Isaksen, A., McMillan, L., an Gortler, S. J., Dynamically rearameterize light fiels, ACM Trans. Grah., (2000). [10] Owens, J., Lueke, D., Govinaraju, N., Harris, M., Kruger, J., Lefohn, A., an Purcell, T., A survey of generalurose comutation on grahics harware, Comuter Grahics Forum 26(1), (2007). [11] Tot, S., Rezk-Salama, C., Kol, A., an Kuhnert, K.-D., GPU-ase sherical light fiel renering with erfragment eth correction, Comuter Grahics Forum 27, (Dec 2008). [12] Meng, J., Weikle, D. A. B., Humhreys, G., an Skaron, K., An aroach on harware esign for comutational hotograhy alications ase on light fiel refocusing algorithm, Tech. Re. CS , University of Virginia (2007). [13] Nviia, NVIDIA GeForce 800 GPU architecture overview, Tech. Re. TB v (2006). [14] Luke, J. P., Nava, F. P., Hernanez, J. G. M., Ramos, J. M. R.,, an Rosa, F., Near real time estimation of eth suer-resolve an all-in-focus images from a lenotic camera using grahic rocessing units (GPU), International Journal of Digital Multimeia Broacasting (2009). [15] Nicoemus, F. E., e., [Self-stuy manual on otical raiation measurements], National Bureau of Stanars (1978). [16] Guillemin, V. an Sternerg, S., Symlectic techniues in hysics, (1985). [17] Gerrar, A. an Burch, J. M., [Introuction to Matrix Methos in Otics], Dover Pulications (1994). [18] Wolf, K. B., [Geometric Otics on Phase Sace], Sringer (2004). [19] Rost, R. J., [OenGL(R) Shaing Language (2n Eition)], Aison-Wesley Professional (January 2006).

15 [20] Mark, W. R., Steven, R., Kurt, G., Mark, A., an Kilgar, J., Cg: A system for rogramming grahics harware in a C-like language, ACM Transactions on Grahics 22, (2003). [21] Nickolls, J., Buck, I., Garlan, M., an Skaron, K., Scalale arallel rogramming with cua, Queue 6(2), (2008). [22] Khronos Grou, OenCL - The oen stanar for arallel rogramming of heterogeneous systems. htt:// [23] PyOenGL. htt://yoengl.sourceforge.net/. [24] Georgiev, T. an Lumsaine, A., Reucing lenotic camera artifacts, Comut. Grah. Forum 29(6), (2010).