Dynamics-based motion deblurring for a biologically-inspired camera positioning mechanism

Transcription

1 013 IEEE/RSJ Iteratioal Coferece o Itelliget Robots ad Systems (IROS) November 3-7, 013. Tokyo, Japa Dyamics-based motio deblurrig for a biologically-ispired camera positioig mechaism Michael D. Kim, Studet Member, IEEE ad Ju Ueda, Member, IEEE Abstract This paper presets a ew dyamics-based method for image processig i coordiatio with rapid ocular movemet. A camera positioig mechaism with piezoelectric cellular actuators will be employed to demostrate the effectiveess of this approach. There are a umber of mechaisms for automatic camera positioig, but few have much i commo with the huma ocular positioig system. Whe a rapid poit-to-poit motio is created i a camera positioer, like huma saccadic motio, the image sesor receives blurry images. Existig image techiques rely solely o obtaied images, or use exteral sesors to estimate a blur kerel, or poit spread fuctio (PSF), for motio deblurrig. Ispired by recet oculomotor studies, this paper proposes a method for estimatig PSFs directly from the system dyamics of a camera positioig mechaism without motio sesors. The proposed method has bee evaluated by usig a sigle degree-of-freedom camera orietatio system. Results are compared with covetioal methods i terms of speed ad accuracy. I. INTRODUCTION The huma eye is a remarkable orga that detects light, allowig the brai to recogize the eviromet. Due to its limited field of view (FOV), huma eyes eed to chage the orietatio to sca the eviromet or to chage the regio of iterest (ROI). Huma eye movemets are created by six surroudig extraocular muscles [1]. These six extraocular muscles cosistig of four recti muscles ad two oblique muscles create various eye motios i three degrees of freedom (DOF). Huma eyes are kow to create two represetative movemets: smooth pursuit ad saccade. It is kow that movemet time is withi 50ms for 10 degree saccade ad the maximum saccade velocity is 50 degree per secod []. Saccade is some of the fastest ad most accurate movemet made by huma [3]. Saccade occurs much more rapidly tha proprioceptive, vestibular, or visual feedback that ca be retured to the brai [4]. This fact suggests that the saccade is completed i a ope-loop maer i the cotrol poit of view [4]. A variety of camera positioig devices, ispired by the huma ocular system, have bee desiged. La et al. have developed a two DOF camera-orietatio mechaism by usig rigid liks ad servo motors [5]. Lesmaa et al. have developed a bio-ispired ope-loop cotroller for fast eye movemets by usig DC servo motors [8]. Sog et al. have developed a This research was supported i part by NSF grat ECCS M. D. Kim ad J. Ueda are with Bio-Robotics Huma Modelig Lab, George W. Woodruff School of Mechaical Egieerig, Georgia Istitute of Techology, Atlata, GA , USA. ( mkim44@gatech.edu & ju.ueda@me.gatech.edu) active biocular itegrated system actuated by DC motors [9]. While these mechaisms successfully geerated a rapid motio, heavy idustrial servo motors have little i commo with biological muscle actuatio. I cotrast, the huma eye is positioed by meas of atagoistic pairs of six extraocular muscles. There are several otable desigs that oriet the camera by meas of atagoistic pairs of alterative actuators. Villgratter et al. have developed a compact high-dyamic two DOF camera-orietatio system drive by pairs of piezoelectric actuators [6]. Lez et al. have desiged a robot eye mechaism drive by peumatic actuators [7]. Eve though these desigs have uique solutios to camera positioig, there is a discrepacy with muscle actuators. For example, the rigidity is a artifact of usig traditioal servo motors while the extraocular muscles are cotractile ad compliat. I additio, these actuators are cotrolled i closed-loop for a rapid motio while physiology study idicates that saccade is completed i a ope-loop maer. Schultz et al. have developed a camera positioer drive by atagoistic pairs of compliat cellular actuators that are cotrolled by a ope-loop, switchig cotroller [10-11]. All aforemetioed biologically ispired camera positioig robots are oly focused o desig ad cotrol of the mechaism but ot o visio or image process. Although there was o visio processig for the previous studies, most of them employ a camera with a high frame rate over 100 frames per secod (fps). Cotrary to a high-speed camera that was employed to the existig systems, the frame rate of the huma eye is kow to be aroud 4 fps. Therefore, durig saccade or right after saccade is completed, the huma eye will perceive blurred images. Physiology studies idicate that the brai predicts cosequeces of the eye movemet so that motio smear ca be reduced by a eural compesatio mechaism usig iformatio of eye movemets [13-14]. There are various approaches proposed i the area of computer visio for restorig degraded blurry images caused by motio. A poit spread fuctio (PSF) must first eed to be estimated. The PSF is a blur kerel that describes the camera motio durig exposure. Give the kerel, a blurry image ca be recostructed by usig a stadard decovolutio algorithm. Fergus et al. have attempted to estimate a PSF from a sigle blurred image ad restored images usig statics of a obtaied image [15]. Multiple images have bee used to remove blur i the images [16-17]. However, these approaches require a large amout of computatio time for aalyzig the blurry image /13/$ IEEE 689

2 Figure. Sigle degree-of-freedom tiltig motio of the camera orietatio system Figure 1. Camera orietatio system There are studies usig differet approaches by usig additio sesors. Joshi et al. have used a gyro sesor ad accelerometer to track a camera shake ad to estimate a PSF for image deblurrig [18]. Be-Ezra et al. have used a extra high frame camera with a low resolutio to track the camera motio ad to estimate a PSF [19]. While these approaches were able to remove blur i images by usig motio iformatio, they also require a large amout of computatioal effort. Be-Ezra et al. s approach requires post image processes of multiple images obtaied from a secodary high frame camera. Joshi et al. s approach ivolves computatios estimatig the path of a camera shake from oisy gyro ad accelerometer sesors. This paper presets a biologically ispired estimatio of the PSF by visual-motor coordiatio for image deblurrig. The huma brai uses iformatio of eye movemets to reduce motio smear [13-14]. This paper presets a method ispired by this observatio. Uderstadig both mechaical properties ad cotrol architecture, PSFs ca be estimated i a ope-loop maer without the aalysis of iheret image characteristics. This method directly estimates the PSF from the cotrol system without ay sesors resultig i less computatioal effort for image restoratio. Although the ultimate goal of camera orietatio system is a three degree-of-freedom system, the proposed method is tested ad verified usig sigle degree-of-freedom system as show i Fig. 1. II. DESIGN AND CONTROL OF THE CAMERA-ORIENTATION SYSTEM A. Mechaical Desig The camera orietatio system used i this study is show i Fig. 1. A Logitech C70 HD webcam is used for image acquisitio. The movig plate is coected to a rod that trasmits the force from atagoistic pairs of cellular actuators located o both sides. The cellular actuator cosists of sixtee Lead Zircoate Titaate (PZT) stackactuators with deformable amplificatio mechaisms [1], [0]. The multi-layered strai amplificatio structure amplifies the displacemet of PZT from its extremely small strai. This mechaism exhibits zero backlash ad oise-less operatio sice o gears or slidig mechaisms are used i the structures while still extremely fast movemet is achieved. I additio, the ested compliat mechaism has bee reported to resemble characteristics of huma extraocular muscles [7]. The axis of the rod is assembled perpedicular but ot orthogoal to the pivot axis. Therefore, whe the rod is pushed or pulled by the atagoistic pairs of the cellular actuators, a momet is applied to the movig plate resultig i a tiltig motio of the camera positioig system as show i Fig.. B. Mechaical Cotrol A total of 3 APA50XS (Cedrat corporatio) PZT actuators are used to positio the camera. The maximum displacemet of the PZT stack actuator is 80 m ad it ca be cotrolled cotiuously by adjustig the iput voltage. The PZT actuator has hysteresis respose requirig closed-loop cotrol with additioal sesig devices. I this study, however, the PZT is give oly maximum iput voltage of 150V or miimum iput voltage of 0V i order to operate i a o-off maer so that the hysteresis respose issue ca be avoided. Sice the system has a umber of PZT stacks i a amplificatio mechaism to the maximum free displacemet of 1.9mm, a discretized desired distace ca be obtaied by defiig the umbers of o ad off PZT actuators. I this approach, the cellular actuators share the priciples i commo with huma muscles where a motio of the huma muscles is a summatio of motor uits [10].The camera positioig mechaism exhibits a sigle domiat frequecy at 14.4 Hz with a dampig ratio of 0.1. Therefore, the system ca be represeted as a liear secod-order system give as: G( s) s K ]Z sz where K is the residue, Z is the atural frequecy, ad ] is the dampig coefficiet. A saccade-like motio is a ope-loop cotrolled poit-to-poit motio with 50ms settlig time at the maximum agular velocity of 50deg/sec. I order for camera positioig mechaism to geerate the saccade-like motio, quatized (1) 690

3 Figure 3. (a) Time respose of a rapid poit-to-poit motio of the camera orietatio system ragig i various desired agles (b) Discrete switchig commads to suppress vibratio Figure 5. (a) Exposure Widow I. The image is obtaied durig partially statioary ad i motio. The image is partially blurry. (b) Exposure Widow II. The image is obtaied i full motio (c) Exposure Widow III. The image is obtaied durig partially statioary ad i motio but fully blurred. (d) Exposure Widows of image acquisitio by the bio-ispired camera orietatio device Figure 4. Velocity profiles of a rapid poit-to-poit motio ragig i various desired agles give the discrete switchig commads iput commads must be give at appropriate times to the PZT stacks with successful vibratio suppressio. This ca be achieved by uderstadig the mechaical properties of the system such as the atural frequecy ad the dampig ratio. By applyig a phase-vector aalysis of mechaism cotrol [11], discrete switchig commads ca be give to the system as show i Fig. 3(b). Figure 3(a) shows represetative time resposes of a poit-to-poit movemet of the camera orietatio system i various desired agles give iteger values of amplitudes at appropriate times. The results show that the vibratio has bee suppressed ad the settlig time of each motio is withi 60ms. If a step commad is give to the system, however, the respose shows cosiderable oscillatio due to the lightly damped system. Figure 4 shows that the maximum peak velocity of the camera orietatio system give the discrete switchig commads is approximately 350 deg/sec. As a result, the speed of respose of the camera positioig mechaism is comparable to that of the huma ocular motio. III. IMAGE ACQUISITION AND DEBLURRING A. Exposure Widows Images ca be obtaied by the camera i various situatios whe a motio is ivolved as show i Fig. 5(d). Exposure Widow I is that the camera first starts takig a image at a stable state, but soo the motio takes place durig the image acquisitio. This results i acquisitio of a blurred image i partial regio because data is ot loaded to a image sesor simultaeously for all pixels. Exposure Widow II is that a image is obtaied i motio for full exposure widow. Exposure Widow III is similar to Exposure Widow I where a image is obtaied partially statioary ad i motio, but the whole image is blurred. The period of image acquisitio icludes writig, readig ad erasig of data. For Exposure Widow III, the data is writte durig motio, but is read ad erased whe the motio is relatively steady-state. Exposure Widow IV is a fully steady-state. The image acquisitio i Exposure Widow I is very differet to the visual process i humas. For Exposure Widow II, images are blocked i visual process of the huma which is called saccadic maskig []. Saccadic maskig is the pheomeo i the brai where the brai blocks images with substatial motio blur i the course of the eye motio. Therefore, i this paper, images oly i Exposure Widow III are cosidered for deblurrig. B. Dyamics-based Kerel Estimatio A spatially-ivariat blurred image B ca be represeted by a covolutio betwee a shift-ivariat blur kerel K ad a latet sharp image I plus N: B mu B, I, N ƒ, where is a covolutio operator. K I N () K lul ƒ 691

4 A blur kerel, or a PSF, represets a trajectory of the motio with eergy levels. I order to perform deblurrig, it is importat to estimate a blur kerel or PSF accurately. To perform deblurrig, it is importat to estimate a blur kerel or PSF accurately. I the literature, PSFs are obtaied by the aalysis of a sigle image, requirig a large calculatio time [15], [17]. Aother approach is to use a additioal high-speed camera. However, the hardware teds to be bulky [18-19]. This paper proposes the estimatio of PSFs from the system dyamics of the camera orietatio system. Excludig the case where the system has large exteral perturbatios, the motio of a robotic camera positioig system is predictable sice the dyamics of the systems ad actuatio commads are kow ad give. This predictability of the system allows for sesorless estimatio of PSFs. This approach is ispired by the observatio that the brai seems to use eye movemets to reduce motio smear [13-14]. Sice the camera positioig mechaism ca be modeled as a liear, time-ivariat secod order system, the time respose of the sigle degree-of-freedom camera orietatio system ca be give as: T Z ( tti ) ( t) Ai e si( Z 1] i 1 1] ]Z ( t t )) (3) where A i is the amplitude of the i th impulsive iput, t i is the time of give i th impulsive iput, is the umber of amplitudes give to the system, dampig coefficiet. A PSF ca be give as Z is the atural frequecy, ad ] is the k (4) ( x, h( x, i without the use of sesors. This results i fast estimatio of the PSF because it is estimated simultaeously with the image acquisitio. O the cotrary, image-based methods require a large amout of computatio time for estimatio ad sesor-based methods also require a large amout of computatioal effort to covert data to the path of a camera shake. C. Decovolutio After estimatig the PSF, image deblurrig ca be achieved by meas of a decovolutio process. I this paper, a widely-used Richardso-Lucy decovolutio method is chose. This method is kow to be robust agaist oise. It produces deblurrig images with high quality i presece of high oise levels. It recovers the latet image iteratively with a estimated PSF [3]. For the decovolutio process, MATLAB s decovlucy.m fuctio ca be used ˆ I I 1 K I K ¹ where B is the obtaied image, K is the blur kerel, Kˆ is the flipped blur kerel, ad I is the recovered image at th iteratio. B (8) where h is a eergy level at the locatio (x, of a pixel. The pixel of the kerel (x, is tb ( x, L³ T ( t) dtxˆ (5) where L is a coversio factor depedet o the size of the blur kerel. ad the eergy fuctio is ta tb ta h( x, (6) t where t acq is the total time of sigle image acquisitio. Thus, the eergy is proportioal to the time remaiig at the locatio (x,. The PSF is a eergy distributio fuctio for which the eergy coservatio costrait must hold: acq ³³ ( x, dxdy 1 k (7) Give the mechaical properties ad cotrol commads of the system, therefore, a discretized PSF ca be estimated Figure 4. Visual-motor coordiatio for a biologically ispired estimatio of the PSF IV. EXPERIMENTAL SETUP The PSF ca be estimated i a embedded cotroller, NI crio-904, i real-time as show i Fig. 6. First a image is obtaied from the Logitech C70 ad desired agle is defied from the iverse kiematics. Secod, ope-loop commads are geerated i the embedded cotroller ad give to the camera orietatio system. Lastly, the system produces a rapid poit-to-poit motio show i Figs. 3 ad 4. The PSF for image deblurrig is estimated i the embedded cotroller i a ope-loop maer. A image obtaied i phase III, discussed i the previous sectio, is used for decovolutio. The images are obtaied 180x960 RGB 4 format i DirectShow bus at maximum 30 fps via USB.0 commuicatio. V. RESULTS AND DISCUSSION Figure 7 ad 8 show the results of the proposed image deblurrig method. The results are compared with other 69

5 well-kow algorithms: methods proposed by Xu et al. [4], Goldstei et al. [5], Sha et al. [6], ad Fergus et al. [15]. The top row shows full images ad bottom row shows cropped local images to clearly show the results. The blurred images are obtaied i Exposure Widow III. The images i Fig.9 are take at 5 fps similar to a typical frame speed of the huma eyes [8]. The images i Fig. 10 are take at 15 fps due to low ambiet light. The results show a improvemet over the iput blurry images. It ca be see that the results are better tha Xu et al. s method, Sha et al. s method ad Fergus et al. s method. The Sha et al. ad Fergus et al. results show large rigig artifacts. The Xu et al. sharpes the origial image but there are still residual rigig artifacts. The results of Goldstei et al. method are comparable. However, the output images from all methods are still locally blurry. The reaso is that the image blur is itroduced due to a rapid motio of the camera with short exposure widow while the previous studies have focused o the image blur caused by log exposures due to low light. Also, a spatial-ivariat blur kerel is used for image restoratio. TABLE I. CONDITIONS FOR COMPARISON OF COMPUTATOIN TIME Kerel Size Image Size Software Xu et al. [4] 19x19 800x600 Complied Executable* Goldstei et al. [5] 1x1 800x600 MATLAB Sha et al. [6] 7x7 800x600 Compiled Executable* Fergus et al. [15] 1x1 800x600 MATLAB Proposed Method 1x1 800x600 MATLAB (*: Complied executable files are distributed by the Authors) The output images show i Fig. 8 are deblurred, but slightly oisy due to a existece of dark curret oise i the captured images. It ca be see that the methods of Xu et al., Sha et al., ad Fergus et al. are ot robust to dark curret oise. Figure 9 shows a compariso of computatio times. The Figure 5. Deblurrig results ad comparisos. Images are take at 5fps. Top row: full image. Bottom row: cropped local image. The cropped regio is specified i the full blurry image (a) The origial blurred iput image (b) Xu et al. [4] (c) Goldstei et al. [5] (d) Sha et al. [6] (e) Fergus et al. [15] (f) Proposed method Figure 6. Deblurrig results ad comparisos. Images are take at 15fps. Top row: full image. Bottom row: cropped local image. The cropped regio is specified i the full blurry image. Images are take uder low ambiet light. (a) The origial blurred iput image (b) Xu et al. [4] (c) Goldstei et al. [5] (d) Sha et al. [6] (e) Fergus et al. [15] (f) Proposed method. The output images of some methods are degraded 693

6 Figure 7. Compariso of computatio times computatio times iclude estimatio of a PSF ad decovolutio. A total of 10 trials are executed for each algorithm. The results show that the proposed method is the fastest sice the PSF is estimated from the embedded motio cotroller i a parallel process while those of others are estimated by the aalysis of images. The computatio time of the proposed method could be decreased by usig a o-iterative decovolutio method. VI. CONCLUSION AND FUTURE WORK This paper has preseted a biologically ispired estimatio of the spatial-ivariat PSF. The PSF is estimated i a ope-loop maer based o the dyamics of the system with o motio sesors. Blurry images were obtaied i Exposure Widow III. The blurry image was restored by the decovolutio process i kowledge of the estimated PSF. The results show that the proposed method is effective for image deblurrig caused by fast motio. I additio, the computatio time is faster tha other approaches. Future work will ivolve a estimatio of the spatial-variat PSF from mechaical properties ad cotrol architecture, fast geeratio of paoramic images, ad visual servoig by usig a multiple DOF camera positioer. Oce the PSF is estimated, the cotroller should be able to determie a ideal profile to positio the camera ad exposure times to take images i order to re-use the estimated PSF. ACKNOWLEDGMENT The authors wish to express their thaks to Dr. Joshua Schultz for the valuable work o the camera positioig hardware ad cotrol. REFERENCES [1] R. H. S. Carpeter, Movemets of the eyes, d ed. Lodo: Pio, [] C. M. Harris ad D. M. Wolpert, Sigal-depedet oise determies motor plaig, Nature, vol. 394, pp , [3] R. J. Leigh ad C. Keard. Usig saccades as a research tool i the cliical euroscieces, Brai, vol. 17, o. 3, pp , 004. [4] D. A. Robiso, The mechaics of huma saccadic eye movemet, J. of Physiology, vol. 174, pp , [5] Y.-C. Lee, C.-C. La, C.-Y. Chu, C.-M. Lai ad Y.-J. Che, A Pa-Tilt Orietig Mechaism with Parallel Axes of Flexural Actuatio, IEEE/ASME Tras. Mech., vol. 18, o.3, pp , Jue 013. [6] T. Villgratter ad H. Ulbrich, Desig ad cotrol of a compact high dyamic camera orietatio system, IEEE/ASME Tras. Mech., vol. 16, o., pp. 1-31, April 011. [7] A. Lez, S. R. Aderso, A. G. Pipe, C. Melhuish, P. Dea ad J. Porrill, Cerebellar-Ispired Adaptive Cotrol of a Robot Eye Actuated by Peumatic Artificial Muscles, IEEE Tras. Sys., Ma ad Cyberetics: Part B, vol. 39, o. 6, pp , Dec [8] M. Lesmaa ad D. K. Pai, A Biologically Ispired Cotroller for Fast Eye Movemets, i Proc. IEEE It. Cof. Robotics ad Automatio, 011, vol., o., pp [9] Y. Sog ad X. Zhag, A Active Biocular Itegrated System for Itelliget Robot Visio, i Proc. IEEE It. Cof. Itelligece ad Security Ifo., 01, vol., o., pp [10] J. Schultz ad J. A. Ueda, A camera positioer drive by muscle-like actuatio, i Proc. IEEE It. Cof. Biomed. Robot. Biomechatro., 01, pp [11] J. Schultz ad J. Ueda, Experimetal verificatio of discrete switchig vibratio suppressio, IEEE/ASME Tras. Mech., vol. 17, o., p , 01. [1] A. Ehud ad A. Amos, Seeig via miiature eye movemets: a dyamic hypothesis for visio, Frotiers i Computatioal Neurosciece, vol. 6, o. 89, pp. 1-7, 01. [13] J. Tog, S.B. Steveso, ad H.E. Bedell, Sigals of eye-muscle proprioceptio modulate perceived motio smear, Joural of Visio, vol. 8, o. 14, pp , Oct [14] M. Rolfs, D. Joikaitis, H. Deubel ad P. Cavaagh, Predictive remappig of attetio across eye movemets, Nature Neurosciece, vol. 14, pp.5-56, 010. [15] R. Fergus, B. Sigh, A. Hertzma, S. T. Roweis, ad W. T. Freema, Removig camera shake from a sigle photograph, ACM Tras. Graph., vol. 5, o. 3, pp , 006. [16] M. Shi ad J. Y. Zheg, A slit scaig depth of route paorama from statioary blur, i Proc. IEEE Comp. Soc. Cof. Computer Visio ad Patter Recogitio, 005, vol. 1, pp [17] L. Bar, B. Berkels, M. Rumpf ad G. Sapiro, A variatioal framework for simultaeous motio estimatio ad restoratio of motio-blurred video, i Proc. IEEE 11th It. Cof. Computer Visio, 007, pp [18] N. Joshi, S. B. Kag, C. L. Zitick, ad R. Szeliski, Image deblurrig usig iertial measuremet sesors, ACM Tras. Graph., vol. 9, o. 4, pp. 1 9, 010. [19] M. Be-Ezra ad S. Nayar, Motio-based Motio Deblurrig, IEEE Tras. Patter Aaly. ad Mach. Itell., vol. 6, o. 6, pp , 004. [0] J. Ueda, T. Secord, ad H. Asada., Large effective-strai piezoelectric actuators usig ested cellular architecture with expoetial strai amplificatio mechaisms, IEEE/ASME Tras. Mech., vol. 15, o. 5, pp , Oct [1] J. W. Miski ad D. J. C. MacKay, Esemble learig for blid image separatio ad decovolutio, i Advaces i Idepedet Compoet Aalysis, M. Girolami, Ed. New York: Spriger-Verlag, July 000. [] Mati, Ethel. "Saccadic suppressio: a review ad a aalysis." Psychological bulleti 81.1 (1974): 899. [3] D. Fish, A. Briicombe, E. Pike, ad J. Walker, Blid Decovolutio by Meas of the Richardso-Lucy algorithm, J. Optical Soc. Am., vol. 1, [4] L. Xu ad J. Jia, Two-phase kerel estimatio for robust motio deblurrig, i Proc. ECCV, 010, pp [5] A. Goldstei ad R. Fattal, Blur-Kerel estimatio from spectral irregularities, i Proc. ECCV, 01, pp [6] Q. Sha, J. Jia, ad A. Agarwala, High-Quality Motio Deblurrig from a Sigle Image, ACM Tras. Graphics, vol. 7, o. 3, 008. [7] Ju Ueda, Piezoelectrically Actuated Robotic Ed-effector with Strai Amplificatio Mechaisms, Advaced Mechatroics ad MEMS Device, Microsystems, vol. 3, pp. 5-5, Edited by Da Zhag, Spriger, ISBN , 01. [8] F. Kuester ad T. C. Hutchiso, A virtualized laboratory for earthquake egieerig educatio, Comput. Appl. Eg. Edu, vol.15, pp. 15-9,