Camera Calibration and Performance Evaluation of Depth From Defocus (DFD)

Transcription

1 Please verfy that () all pages are present, () all fgures are acceptable, (3) all fonts and specal characters are correct, and (4) all text and fgures ft wthn the margn lnes shown on ths revew document. Return to your MySPIE ToDo lst and approve or dsapprove ths submsson. Camera Calbraton and Performance Evaluaton of Depth From Defocus (DFD) Tao Xan, Mural Subbarao Dept. of Electrcal & Computer Engneerng* State Unv. of ew York at Stony Brook, Stony Brook, Y, USA ABSTRACT Real-tme and accurate autofocusng of statonary and movng objects s an mportant problem n modern dgtal cameras. Depth From Defocus (DFD) s a technque for autofocusng that needs only two or three mages recorded wth dfferent camera parameters. In practce, there exst many factors that affect the performance of DFD algorthms, such as nonlnear sensor response, lens vgnettng, and magnfcaton varaton. In ths paper, we present calbraton methods and algorthms for these three factors. Ther correctness and effects on the performance of DFD have been nvestgated wth experments. Keywords: Autofocusng, Depth From Defocus (DFD), camera calbraton, nonlnear sensor response, lens vgnettng, magnfcaton varaton. ITRODUCTIO Depth From Defocus (DFD) s a technque for autofocusng that needs only two or three mages recorded wth dfferent camera parameters. It recovers depth nformaton by estmatng the degree of blur. Due to the nherent advantage of beng local n nature, the spatal doman approach for DFD s more sutable for real-tme autofocusng applcatons. In practce, there exst many factors that affect the performance of DFD algorthms. In partcular, nonlnear sensor response, lens vgnettng, and magnfcaton varaton affect the accuracy of DFD. In order to mplement DFD on offthe-shelf commercal dgtal cameras, these factors need to be calbrated and corrected. In ths paper, we present new calbraton methods for these three factors. Ther correctness and effects on performance of DFD have been evaluated wth experments. Most dgtal cameras utlze the nonlnear sensor response to extend the dynamc gray-level range through a log-lke or gamma transform. DFD theory requres nverse mappng of ths non-lnear response to lnear response through calbraton. The ntensty measured by the mage sensor depends on llumnaton, exposure perod, and reflectance. A method s proposed and tested for correctng ths non-lnear sensor response. Optcal vgnettng s the phenomenon where the effectve lght energy transmtted by the optcal system decreases wth ncreasng nclnaton of lght rays wth respect to the optcal axs. A vgnettng calbraton method s mplemented and tested for ts effects on DFD performance. In DFD based autofocusng where the lens poston s moved, the magnfcaton of an object wll change when two mages are recorded wth dfferent camera parameters. A magnfcaton calbraton method s mplemented and the estmaton error has been evaluated. The calbraton methods for nonlnear response and vgnettng correcton are drect methods based on llumnaton measurement usng a dgtal lux tester. They do not need expensve and strctly controlled laboratory envronment and * E-mal: {txan, mural}@ece.sunysb.edu; Tel: ; WWW: 6- V. (p. of ) / Color: o / Format: Letter / Date: 9/4/5 5:4:9 PM

2 Please verfy that () all pages are present, () all fgures are acceptable, (3) all fonts and specal characters are correct, and (4) all text and fgures ft wthn the margn lnes shown on ths revew document. Return to your MySPIE ToDo lst and approve or dsapprove ths submsson. can be used for off-the-shelf cameras. Therefore, these calbraton methods should be of general value to other mage based algorthms.. STM-DFD ALGORITHM The prncple of Spatal-doman Convoluton/Deconvoluton Transform (S Transform) based Depth from Defocus algorthm s proposed by Subbarao and Surya [], the performance evaluaton for dfferent STM based technques s analyszed n Xan and Subbarao []. Here a bref summary of STM s ncluded for further dscusson. Accordng to S Transform, the focused mage f ( can be obtaned from ts correspondng blur verson g ( by local decomposton n the spatal doman. σ h f ( = g( g( () 4 where σ s a spread parameter of the pont spread functon (PSF) h h (, And = + denotes Laplacan x y operator. For smplcty, the focused mage f ( and defocused mages g ( are denoted by f and g n the followng dscusson. If two mages g and g are recorded wth two dfferent parameter settngs e = s, f, ) ( D e = ( s, f, D ), from Eqn. () we have g g = G g () 4 where 4( g g ) G = σ σ = (3) g Under the local thrd order polynomal model of mage brghtness, g = g. Therefore g and g can be replaced by: g = g + g ) / ( If the lens poston s changed durng the acquston of the two mages g and g, the sgma can be calculated from: β σ = G (4) β where β s a system parameter. For a specfc magng system, β s fxed and can be determned from optcal [, ] confguraton. The focus lens step can be calculated from a Sgma-Lens Step lookup table. A new bnary mask s ntroduced to mprove the robustness of STM algorthm n []. The bnary mask s formed by thresholdng Laplacan values, whch removes unrelable ponts wth low Sgnal-to-ose Rato (SR). In the Bnary Mask based STM verson of Wthout Square Wthout Integraton (BM_OSOI), the average of G s calculated based on the bnary mask: [ g x y g x y ] G = 4 (, ) (, ) M ( (5) U ( W g( where U = M ( y s the total weght of the bnary mask. ) ( W In practce, there exst many factors such as the nonlnear sensor response, lens vgnettng, and magnfcaton varaton, whch may affect the performance of the DFD algorthm. To further understand ther effects, the followng calbraton methods are presented. and 6- V. (p. of ) / Color: o / Format: Letter / Date: 9/4/5 5:4:9 PM

3 Please verfy that () all pages are present, () all fgures are acceptable, (3) all fonts and specal characters are correct, and (4) all text and fgures ft wthn the margn lnes shown on ths revew document. Return to your MySPIE ToDo lst and approve or dsapprove ths submsson. 3. OLIEAR SESOR RESPOSE COMPESATIO The formaton of a dgtal mage on the mage sensor of a camera can be descrbed by: τ + g( = q ( y, λ, t) s( λ) dλdt s where g ( s the photo-quantty of the specfc sensor element ( x, ; q s ( y, λ, t) s the actual lght energy fallng on the mage sensor ( x, ; s (λ) s the spectral senstvty of an element of the sensor. τ denotes the ntegraton perod, whch s controlled by exposure tme of the camera. From ths equaton, photo-quantty g ( s nether radometrc nor photometrc unt, snce t also related to the sensor spectral senstvty s (λ). For a specfc camera system, the photo-qualty depends on the lght energy fallng on the sensor cell per unt tme, and camera exposure tme. Once the parameters of DFD ( s, s, f, D for STM) are fxed, the measurement from DFD algorthms should only be related to the object dstance, and should not be affected by other changes such as llumnaton and camera exposure. However, most dgtal cameras utlze the nonlnear sensor response to extend the dynamc gray-level range through transforms (e.g. log(z)). That means more graylevels are assgned to the photo-qualty range wth hgher probabltes whle less graylevels are assgned to the photo-qualty range wth lower probabltes. 3.. Error analyss of non-lnear sensor response The nonlnear sensor response s a pont-wse mappng, whch can be formulated by a functon K: g '( = K[ g( ] (7) where g '( s the dstorted ntensty after pont-wse sensor response mappng, and g ( s the orgnal photoquantty formed as n Eqn. (6). If dgtal mages are quantzed to n bts, the pont-wse sensor mappng can be expressed by the transform vector k wthout sacrfcng generalty: K( ) = k I + b (8) where b s the dark offset, I s the orgnal photo-quantty vector, and k s the th coeffcent to map from level n the orgnal photo-quantty g to dstorted ntensty g '. n [ ] t [ k k k ] I = Λ (9) k = Λ () n For a lnear mappng, the components n the coeffcent vector for each level should be the same,.e. k = k j = k ; whle for a nonlnear mappng, k k s vald for some level, j. j (6) Due to nonlnear sensor response, the sgma σ ' n Eqn. (4) can be calculated from: ( g' g') β 4( ksg ksg) β σ ' = = β g' ( ks+ ks) β g where g ' and g ' correspond to the dstorted blur mages acqured at dfferent lens step. () The sgma error due to sensor response ε s expressed by the dfference between sgma calculated from dstorted mage pars and the ones wthout dstorton. ks ks ( g+ g) ε = σ' σ = () ks + ks β g From Eqn. (), the sgma error ε s not only related to the system parameter β, but also depends on camera nonlnear mappng coeffcent and the mage tself. The sgma error ε s transferred to the step error through the sgma-step mappng SS ( σ, ε ). A dependence on llumnaton and/or exposure s hence ntroduced. Ths s not desrable n autofocusng applcatons. 6- V. (p.3 of ) / Color: o / Format: Letter / Date: 9/4/5 5:4:9 PM

4 Please verfy that () all pages are present, () all fgures are acceptable, (3) all fonts and specal characters are correct, and (4) all text and fgures ft wthn the margn lnes shown on ths revew document. Return to your MySPIE ToDo lst and approve or dsapprove ths submsson. Ths sgma error ε can be elmnated only f k s = k n Eqn. (). Snce the value of s g and g are arbtrary n ntenstes n the range of [, ], the followng equaton should be vald for any ntensty to compensate for ε : k s = k s = k (3) Eqn. (3) demonstrates ε can be elmnated when there exsts a lnear relaton between observed ntensty and photoquantty. The lnearzaton can be obtaned by nverse mappng of sensor response K. g = K ( g' ) = K [ K ( g)] = g (4) 3.. Drect calbraton Sensor response calbraton s needed to compensate for ts non-lnearty. There are several approaches to measure the sensor response, through statstcal averagng on the whole mage for arbtrary scenes, such as [3] and [4]. However averagng on the mage plane ncludes the effect of vgnettng, whch wll be dscussed n Secton 4. The resultng response s a weghted average of on-axs ponts and off-axs ponts, and the weght depends on scene content. A drect measurement method s presented here to calbrate the nonlnear response for off-the-shelf cameras. The setup of the nonlnear sensor response calbraton s brefly outlned n Fg.. A dffusve whte screen WS s llumnated by multple lght sources from L to L. The lght sources are controlled from the lnear lamp controller module 4 LC to create varable/adjustable llumnaton. The ntensty at the central area of the whte screen s measured by a Dgtal Lux Tester YF-65. The mage of the whte screen s acqured by a dgtal camera to be calbrated. A lookup table s establshed by changng the llumnaton ncrementally whle recordng the mage at each llumnaton step. The relatonshp between camera gray-levels and normalzed llumnaton gray-levels s shown n Fg.. Mean brghtness n a * mage regon s used for reducng nose n the central area of the whte screen. To evaluate the effect of nonlnear sensor response, a seres of DFD experments were conducted under dfferent photoquantty condtons. The ambent llumnaton s 53 lux measured at the center of an object plane, and the dstance from the front surface of lens to the object plane s 54 mm. Accordng to Eqn. (6), the photo-quantty can be changed by ether llumnaton level or camera shutter speed. We control the shutter speed to obtan a wder range on photoquantty. The shutter speed changes from 5.65 ms to 5 ms, whch correspond to a change factor of 3 n photoquantty, as dsplayed n Fg. 3. In Fg.3, Images range from under-exposed as n (a), (b), to over-exposed as n (e), (f). The photo-quantty s doubled at each stage from (a) to (f), however the gray level of observed mage does not ncrease correspondngly due to camera range compresson. About 384 DFD experments were conducted at 8 dfferent random postons and 6 dfferent exposure levels. The 8 randomly selected postons are 35., 383., 474., 538.6, 63., 784., 58.4 and mm respectvely, whch are measured from the front surface of the lens. The correspondng lens steps are obtaned from Depth From Focus (DFF) experments, and they are 5, 38, 58, 74,,, 3 and 45 lens step respectvely. As shown n Fg. 4, STM wthout sensor response compensaton has a mean error of up to 4 lens steps (Dstance 63. mm, Shutter Speed 5.65 ms), whle the correspondng RMS lens step error s.354 step, the mean error reflects a systematc bas whch s predcted by Eqn. (). After nonlnear sensor response compensaton, at the same photoquantty, the mean lens step error s reduced to.667 step, and all 8 DFD measurements get exactly the same step number, snce RMS lens step error s. The detaled results of DFD wthout/wth nonlnear sensor response compensaton are summarzed n Table and. Table shows the mean and RMS lens step error before nonlnear sensor response compensaton, whle Table shows errors after sensor response calbraton. When the photo-quantty contnues to ncrease from (e) to (f), step shfts n far feld for both DFF and DFD can be observed. In the mage (f), the sensor s already saturated, and the observed mage s no longer a correct measure of photo-quantty. In ths extreme condton, the error of DFD wth sensor compensaton ( 4.43±. 36 step) s stll better than the one wthout compensaton ( ±. 535 step). 6- V. (p.4 of ) / Color: o / Format: Letter / Date: 9/4/5 5:4:9 PM

5 Please verfy that () all pages are present, () all fgures are acceptable, (3) all fonts and specal characters are correct, and (4) all text and fgures ft wthn the margn lnes shown on ths revew document. Return to your MySPIE ToDo lst and approve or dsapprove ths submsson. 5 Camera Graylevel 5 5 Fgure. Setup for nonlnear sensor response 5 5 ormalzed Illumnaton Level Fgure. onlnear sensor response calbraton (a) x (b) x (c) 4x (d) 8x (e) 6x (f) 3x Fgure 3. Images obtaned at dfferent photo-quantty From (a) to (f), mages are captured wth exposure tme of 5.65, 3.5, 6.5, 5, 5 and 5 ms. Assume the photo-quantty n (a) as a unt measure, the photo-quantty doubles n each stage from (a) to (f) 6- V. (p.5 of ) / Color: o / Format: Letter / Date: 9/4/5 5:4:9 PM 5

6 Please verfy that () all pages are present, () all fgures are acceptable, (3) all fonts and specal characters are correct, and (4) all text and fgures ft wthn the margn lnes shown on ths revew document. Return to your MySPIE ToDo lst and approve or dsapprove ths submsson. 5 5 wol wl DFF Steps Exposure Level [n x] Fgure 4. DFD Results wthout/wth nonlnear sensor response calbraton wol: DFD STM wthout on-lnear sensor response compensaton wl: DFD STM wth on-lnear sensor response compensaton DFF: Depth From Focus Pos Pos Pos 3 Pos 4 Pos 5 Pos 6 Pos 7 Pos 8 Exposure Exposure Exposure 3 Exposure 4 Exposure 5 Exposure 6 Mean Std Mean Std Mean Std Mean Std Mean Std Mean Std Table. DFD lens step error by mean and rms wthout nonlnear sensor compensaton Exposure Exposure Exposure 3 Exposure 4 Exposure 5 Exposure 6 Mean Std Mean Std Mean Std Mean Std Mean Std Mean Std Pos Pos Pos Pos Pos Pos Pos Pos Table. DFD lens step error by mean and rms wth nonlnear sensor compensaton 6- V. (p.6 of ) / Color: o / Format: Letter / Date: 9/4/5 5:4:9 PM

7 Please verfy that () all pages are present, () all fgures are acceptable, (3) all fonts and specal characters are correct, and (4) all text and fgures ft wthn the margn lnes shown on ths revew document. Return to your MySPIE ToDo lst and approve or dsapprove ths submsson. 4 LES VIGETTIG COMPESATIO Optcal vgnettng s the phenomenon wheren the effectve lght energy transmtted by the optcal system decreases wth ncreasng nclnaton of lght rays wth respect to the optcal axs. The consequence of optcal vgnettng for a focused scene s merely a reduced brghtness towards the mage corners. However, optcal vgnettng can also have a pronounced effect on out-of-focus parts of the mage. Because the shape of an Out-Of-Focus Hghlght (OOFH) mmcs the shape of the clear aperture, ths leads to the so-called cat's eye effect. Wth an ncreasng dstance from the optcal axs the shape of the OOFH progressvely narrows and starts to resemble a cat's eye. The larger the dstance from the mage center, the narrower the cat's eye becomes. A vgnettng calbraton method s used to evaluate the effect of vgnettng on DFD measurement. If a unform llumnaton s avalable, the vgnettng coeffcent could be smply calculated from a sngle mage of a dffusve whte screen. However t s dffcult to obtan a unform llumnaton that s accurate, although not mpossble. An alternatve way s used n our calbraton. The setup for vgnettng calbraton was smlar to that n Fg.. A 5*5 grd pattern s used as a calbraton pattern (see Fg. 3(a) ). In each grd, llumnaton s measured by the Dgtal Lux Tester YF-65 at the center of grds, and the mage of the grd pattern s captured by the camera. The gray level obtaned s a transformed value of real photo-quantty due to the nonlnearty of sensor response. A lookup table for the reverse mappng dscussed n Secton 3 s used. The vgnettng coeffcent s calculated by the rato of llumnaton ntensty at pxel ( x, to the ntensty at the center of the mage. Due to the rotatonal symmetry property, the relaton between vgnettng coeffcent and pxel dstance n polar coordnate s obtaned from a thrd-order polynomal fttng: V ( ρ) = * * ρ * * ρ * * ρ + (5).95.9 Vgnettng Factor (a) Fgure 5. Vgnettng calbraton (a) Pattern (b) Result Dstance [Pxel] (b) The result of vgnettng factor vs. pxel dstance s shown n Fg. 5(b). From Fg. 5(b), f the DFD AF wndow s n the center area, the dstorton of vgnettng can be gnored. (for a 96*96 focusng wndow, the ntensty attenuaton s.4%). When the focusng wndow s near a corner of the vew, there could be a.% dfference n the dagonal drecton. In ths case, vgnettng should be compensated by multplyng the recprocal of the correspondng vgnettng coeffcent. 6- V. (p.7 of ) / Color: o / Format: Letter / Date: 9/4/5 5:4:9 PM

8 Please verfy that () all pages are present, () all fgures are acceptable, (3) all fonts and specal characters are correct, and (4) all text and fgures ft wthn the margn lnes shown on ths revew document. Return to your MySPIE ToDo lst and approve or dsapprove ths submsson. 5. MAGIFICATIO CALIBRATIO In STM, when the object to be focused s fxed, there s a magnfcaton change between defocused mages acqured at two dfferent lens steps. A magnfcaton calbraton method s developed to determne ths varaton. A chessboard pattern whose grd sze s 5 mm by 5 mm, s captured by the camera at focus steps 35 and 98, and the corner ponts are detected as shown n Fg. 6. The dstance between the camera and the chessboard pattern s 5 mm, whch corresponds to the focused mage approxmately at step 6. For convenence, we defne the defocused mage at step 35 as mage, and the one at step 98 as mage. The transformaton between mage and mage can be calculated through a projecton matrx. The corners on mage and mage are detected and sorted row by row nto two corner arrays respectvely. Corners on the same poston of array make a correspondng corner par. There are *8 corner pars as demonstrated n Fg. 6(a) and (b). For each correspondng corner par, the coordnates of the corners can be expressed as: V = MU, (6) where U and V are coordnates of corner pars n mage and mage respectvely. These coordnates are expressed t t n projectve space,.e. U = [ y, ], and V = [ y, ]. s the total number of correspondng pars. M s a 3*3 transformaton matrx. Snce we have no pror knowledge about the transformaton, there are 9 unknown elements n M. For pars of corner pars, a least-square matrx can be obtaned: AM ' = b (7) where A s a 3*9 matr and b s a 3 column vector that s made from lappng over V, =, Λ,. y, y, x, y, (8) A = Μ Μ Μ Μ Μ Μ Μ Μ Μ x, y, y, y, b = [ x y x ] t,, Λ, y, (9) and M ' s reorganzed from M : M '= [ m m m m m m m m m ] t () Then the transform matrx M can be calculated by: t t M ' = ( A A) A b () The transformaton matrx M between mage and mage s calculated and reformed from Eqn. (): M =.8.5 () Usng ths matrx M to project mage to mage, the projecton error for each corner s plotted n Fg. 7, the projecton error can be expressed by mean and RMS pxel error (-.83e- 3±.6, -.389e - 4 ±.5), and the maxm error s less than. pxel both n x and y drecton. The new mage can be generated by a bcubc nterpolaton of mage. 6- V. (p.8 of ) / Color: o / Format: Letter / Date: 9/4/5 5:4:9 PM

9 Please verfy that () all pages are present, () all fgures are acceptable, (3) all fonts and specal characters are correct, and (4) all text and fgures ft wthn the margn lnes shown on ths revew document. Return to your MySPIE ToDo lst and approve or dsapprove ths submsson. Extracted corners Extracted corners 5 O dx dy 5 O dx dy Yc (n cam era fram e) Yc (n cam era fram e) Xc (n camera frame) (a) Xc (n camera frame) (b) Fgure 6. Magnfcaton calbraton usng pattern captured at dfferent steps (a) chessboard pattern captured at step 35 (b) chessboard pattern captured at step 98. Projecton Error.5..5 Y dr [pxel] X dr [pxel] Fgure 7. Project error from estmated transformaton matrx M 6. COCLUSIO In ths paper, calbraton methods and procedures for nonlnear sensor response, optcal vgnettng, and magnfcaton varaton are presented. The correctness and effects on the performance of DFD have been evaluated wth experments. These calbratons do not need expensve and strctly controlled laboratory envronment. They can be used for off-theshelf cameras. Therefore, these calbraton methods should be of general value to other mage based algorthms. 6- V. (p.9 of ) / Color: o / Format: Letter / Date: 9/4/5 5:4:9 PM

10 Please verfy that () all pages are present, () all fgures are acceptable, (3) all fonts and specal characters are correct, and (4) all text and fgures ft wthn the margn lnes shown on ths revew document. Return to your MySPIE ToDo lst and approve or dsapprove ths submsson. REFERECES. M. Subbarao and G. Surya, "Depth from Defocus: A Spatal Doman Approach", Internatonal Journal of Computer Vson, Vol. 3, o. 3, p. 7-94,994. T. Xan, and M. Subbarao, Performance Evaluaton of Dfferent Depth-From-Defocus (DFD) Technques, Proc. of SPIE, Boston, Oct S. Mann, Comparametrc Equatons wth Practcal Applcatons, IEEE Trans. Image Processng, Vol. 9, o. 8, P389-46, 4. S.Y. Park, and M. Subbarao, Automatc Focusng of a Dgtal Stll Camera usng a Depth-from Defocus Technque: An Approach for Compensatng on-lnear Camera Response Functon, Internal Techncal Report, Computer Vson Lab, ECE, State Unv. of ew York at Stony Brook, 4 6- V. (p. of ) / Color: o / Format: Letter / Date: 9/4/5 5:4:9 PM