An Integrated Architecture for Adaptive Image Stabilization in Zooming Operation

Transcription

1 An Inegraed Archiecure for Adapive Image Sabilizaion in Zooming Operaion Angelos Amanaiadis, Suden Member IEEE, and Ioannis Andreadis Absrac This paper presens a novel archiecure for inegraing digial sabilizer wih he camera zooming process. The proposed archiecure scheme allows inegraion for boh opical and digial zooming operaion scenarios. The sabilizer adjuss is operaional parameers from he curren opical zooming saus for a refined local moion esimaion. Subsequenly, he global moion compensaion vecors, produced by he digial sabilizer, are sen o he digial zooming sysem, where he image compensaion is merged wihin he inerpolaion process. Experimenal resuls indicae ha he proposed archiecure can improve no only he quaniaive performance of digial sabilizaion bu also he compuaional efficiency when zooming is employed 1 Index Terms Camera zooming, digial video sabilizaion, moion esimaion, image inerpolaion. I. ITRODUCTIO Two feaures ha are considered as a de faco in recen consumer imaging devices are he zooming abiliy and he video sabilizaion. Boh characerisics can be found in a single embedded sysem of a digial sill camera, video recorder or even in a 3G mobile phone. Camera zooming helps he user o modify he size of he objec projeced on he sensor surface, while he purpose of he video sabilizaion is o remove he undesirable posiion flucuaions of he capured video sequence caused by he hand shaking or plaform movemen. These wo feaures can be enabled a he same ime in a single imaging device, producing a zoomed and sabilized sequence. However, in hese operaion condiions, he user can noice in he oupu video a laency and an undesired shaking, which aggregaes while he user increases he magnificaion facor. Image sabilizaion sysems can be classified ino hree major caegories. The elecronic image sabilizer compensaes he image sequence by employing moion sensors o deec he camera movemen for compensaion [1], [2]. The opical image sabilizer employs a prism assembly ha moves opposie o he shaking of camera for sabilizaion [3], [4]. Boh elecronic and opical image sabilizers are hardware dependen and require buil-in devices such as inerial sensors and servo moors, making he imaging device unsuiable in erms of volume and cos. The digial image sabilizer (DIS) does no need any mechanical or opical devices since he image compensaion is made hrough image processing 1 The auhors are wih he Laboraory of Elecronics, Deparmen of Elecrical and Compuer Engineering, Democrius Universiy of Thrace, 12 Vas. Sofias Sr, 67100, Xanhi, Greece. ({aamana, algorihms. This aribue makes i suiable for consumer elecronics [5] - [8]. The DIS comprises wo processing unis: he moion esimaion uni and he moion compensaion uni. The purpose of moion esimaion uni is o esimae reliably he global camera movemen hrough he local moion esimaion vecors of he acquired image sequence. The effeciveness of a DIS is closely ied wih he accuracy of deecing he local moion vecors, in order o produce he righ global moion vecor. Following moion esimaion, he moion compensaion uni generaes he compensaing moion vecor and shifs he curren picking window according o he compensaing moion vecor o obain a smooher image sequence. Moion compensaion can ake up 10% of he oal compuaion of a digial sabilizer [9]. Various algorihms have been developed o esimae he local moion vecors, such as represenaive poin maching [6], edge paern maching [10], block maching [7] and bi plane maching [5]. All he previous algorihms, despie heir high accuracy and reliabiliy, are sricly consrained o regular condiions exhibiing high sensiiviy o various irregular condiions such as moving objecs, inenional panning, sequences acquired in low signalo-noise raio or zooming. For moving objecs, soluions have been proposed such as in [11]. For sequences acquired in low signal-o-noise raio, a blind DIS wih he use of geneic algorihms is proposed in [12]. Inenional panning has also been grealy improved using background esimaion [13]. For zooming, CMOS digial image sabilizaion schemes have been proposed in [14] and [15]. However, due o he fac ha hey are based on he disorion effec caused by he rolling shuering mechanism of he CMOS sensor, hey are effecive only in CMOS sensors. Furhermore, he effeciveness is observed only during he change of he focal lengh. As far as he lenses have reached he new focal lengh, here is no enhancemen in he DIS performance. Zooming sysems can be classified in wo main caegories. The opical zooming (OZ) sysems exend physically he lenses, wih he use of moors, in order o change he focal lengh affecing he size of he objec projeced on he sensor surface [16], [17]. Digial zooming (DZ) is achieved by inerpolaing he discree inpu image o a higher resoluion oupu image [18], [19]. Thus, digial image scaling does no require any mechanical servoing. The DZ sysems can be found in all consumer imaging devices while OZ sysems will be soon migraed even o camera mobile phones. Mos cameras ha have zooming lenses, apply he digial zoom auomaically once heir longes focal lengh has been reached. Since users compromise only wih high qualiy DZ, he implemenaion of a sophisicaed zooming algorihm in an

2 embedded sysem has proved a necessiy. While DZ enails complex image inerpolaion algorihms, he undesirable effec of increased laency is observed in he oupu sequence, especially in case of a concurren use of image sabilizaion. In his paper we propose a new archiecure for adapive digial sabilizaion in zooming operaion. The archiecure is efficien for any possible opical or digial zooming operaion scenario. In he firs scenario, he image sabilizaion is performed when only opical zooming is enabled. Inaccurae esimaion of he local moion vecors is he major issue in his scheme. In he proposed archiecure, he sabilizer adaps is operaional parameers from he curren opical zooming saus for a refined local moion esimaion, improving he performance of he DIS in erms of accuracy. In he second scenario he DIS operaes while only he digial zooming is enabled. The main shorcoming of his scenario is he noiceable laency of he oupu video. However, in he proposed archiecure, he global moion compensaion vecors, produced by he DIS, are fed direcly o he digial zooming sysem, where he image compensaion is inegraed wihin he inerpolaion process. Here, he performance of he DIS is improved in erms of compuaional efficiency, since he image compensaion and digial zooming are combined ogeher in one process. In he hird scenario boh opical and digial zooming are enabled wih concurren use of he DIS [20]. The resuls of he proposed archiecure in his scenario are beer performance of he DIS in boh erms of accuracy and compuaion efficiency compared wih convenional DIS archiecures. The res of he paper is organized as follows. In Secion II, we describe he opical and digial zooming process. In Secion III, we invesigae he archiecure of a convenional DIS. In Secion IV, we focus on he impac of zooming on image sabilizaion. In Secion V, we describe he proposed inegraed archiecure for he hree operaion scenarios. In Secion VI, we presen he experimenal resuls and he performance evaluaion of he proposed inegraed archiecure. We provide concluding remarks in secion VII. II. CAMERA ZOOMIG We begin his secion by giving an overview of he opical zooming process, followed by a descripion of he basic principals used by he digial zooming algorihms. A. Opical Zooming Designing an opical sysem wih significan magnificaion around he limied volume of recen imaging devices is a challenge. Furhermore, zoom racking [21], which involves he auomaic adjusmen of he focus moor in response o he zoom moor movemens for keeping an objec of ineres in focus, is also a complex process. Curren opics offer grea zooming wihou deerioraing he image qualiy from possible effecs of diffracion [22]. The focal lengh of a lens is defined as he disance in mm from he opical cener of he lens o he focal poin, which is locaed on he solid-sae sensor. A change in focal lengh allows he user o come closer o he objec or o move away from i. The opical zooming facor is calculaed by Opical zooming facor = f f max min where f max represens he maximum focal lengh and f min he minimum focal lengh. From (1), i is clear ha he volume of he lens sysem mus be increased in order o increase he magnificaion facor suppored by he device. Smaller focal lenghs of he zoom lens correspond o relaively wide angles of view generaing lower magnificaions, referred o as wide angle zoom. On he oher hand, larger focal lenghs of he zoom lens correspond o relaively narrow angles of view generaing higher magnificaions referred o as ele angle zoom. B. Digial Zooming A digial zooming echnique performs spaial inerpolaion since i enlarges he spaial resoluion of he inpu image and complees he missing daa from he spaially adjacen daa using some ype of inerpolaion operaion. Furhermore, i ries o preserve he specral conens of he inpu image. Convenional mehods such as he neares neighbor or bilinear inerpolaion [23] exhibi high compuaional simpliciy bu face serious blurring problems paricularly in edge regions. Linear approaches are he ones mos frequenly used since non-linear mehods, such as bicubic inerpolaion [24] and spline inerpolaion [25], [26] enail a larger compuaional burden and involve blurring, despie producing beer resuls. Edge-direced inerpolaions have been aemped o overcome such shorcomings by applying a variey of operaors according o he edge direcions [27]. However, only cerain angles of edges are idenifiable. Vecor quanizaion zooming has been also repored [28] wih good resuls bu wih he very resricive scaling facor of wo. eural neworks have also been used for image inerpolaion [29]. These, on he oher hand, require a large amoun of cells in he neworks, rendering hem compuaion-wise, oo inensive. Fuzzy inerpolaion approaches have been proposed [30], [31] for wo dimensional signal resampling, wih opimal visual resuls bu lack in simpliciy and require addiional processing for edge idenificaion. Area-based inerpolaion [18], compues each inerpolaed pixel by proporional area coverage of an applied filering window. The Michell and Lanczos [32], [33] inerpolaion mehods are no widely used and are unsuiable in erms of hardware resources required. Fracal zooming [34], [35] has been recenly repored wih very good visual resuls bu he compuaional burden for is coding remains is main disadvanage. Discree cosine scaling echniques for compressed mulimedia have been also proposed in [36] and [37]. These echniques ranscode a compressed frame from one spaial resoluion o a differen spaial resoluion bu hey are no common in consumer imaging devices. (1)

3 Fig. 1. Typical archiecure of a digial image sabilizaion echnique. Many imaging devices ha have an opical zooming sysem, apply digial zoom auomaically once is longes focal lengh has been reached. Despie he fac ha in digial cameras he digially zoomed image shown o he user is only a par of he image capured by he sensor, he whole procedure is idenical o he algorihms used in many deskop image processing applicaions. Therefore, a digial zooming echnique can be described in hree basic seps. 1) Mapping oupu pixel ono he inpu image Le S(i,j) be he inpu image capured from he solid-sae sensor wih m,n dimensions and s he magnificaion facor. In he digial zooming process, he sysem consrucs an oupu image T(k,l), where k=1,2,,s m and l=1,2,,s n. In case of s m and s n are no inegers, hey are rounded o he neares ineger. The inensiy values of all pixels in T are lef undefined. All seps mus be execued for every pixel of T. The firs sep is o map he oupu pixel of T ono he inpu image. The process can be consider equivalen o a coninuous filering, calculaing each ime he cener of he mask used. 2) Inerpolaion This sep describes he main ask of he digial zooming process and depending on he mehod used we can characerize he algorihm iself. The algorihms generally sar by deermining a coninuous funcion from a se of discree signal samples and hen resample his funcion according o some feaures. The basic feaure is he number of he pixels ha will conribue for he oupu resul. The masks used range from zero-order o hree or muli-polynomial masks. There are also non-polynomial-based inerpolaion masks such as he bell-shaped Gaussian mask and he Mexican ha-shaped sinc funcion. 3) Se he inensiy of he oupu pixel The las sep ses he inensiy calculaed by he previous sep o he corresponding oupu pixel. Seps 1 and 3 are common in all zooming echniques. III. DIGITAL IMAGE STABILIZATIO A ypical archiecure of a DIS is shown in Fig. 1. The moion esimaion uni compues he moion beween wo consecuive frames. This is achieved firsly by he local moion esimaion subuni, which esimaes he local moion vecors wihin frame regions. Secondly, he global moion esimaion uni deermines he global moion vecors by processing he previously esimaed local moion vecors. Following he moion esimaion uni, he moion compensaion uni firsly, generaes he compensaing moion vecor and secondly, shifs he curren picking window according o he compensaing moion vecor o obain a free of high frequency image sequence bu sill keep he global ego-moion of he sequence. Fig. 2. Frame o frame moion compensaion. A. Moion Esimaion Many algorihms have been developed in order o esimae he local moion vecors such as represenaive poin maching, edge paern maching, opic flow, bi plane maching, and ohers. While he approach of a full search block maching is infeasible in real-ime demanding consumer elecronics, all he above algorihms are rying o have he bes accuracy and reliabiliy wih he leas possible compuaional demands. In a convenional background moion esimaion uni [13], he cener par of he image is divided ino a number of search regions. The cenral pixel of each region is seleced as he represenaive poin X r,y r o represen he paern of his subregion. The correlaion of represenaive poin maching (RPM) is calculaed using SAD by R ( p, q) = I( 1 ), X, Y ) I(, X, Y ) (2) i r 1 r r r+ p r+ p where is he number of represenaive poins in one region, I(-1,X r,y r ) is he inensiy of he represenaive poin (X r,y r ) a frame -1 and R i (p,q) is he correlaion measure for a shif (p,q) beween he represenaive poins in region i a frame -1 and he relaive shifing poins a frame. Assuming R imin is he minimum correlaion value in region i, i.e, R = min( Ri ( p, q)) imin (3) p, q The shif vecor v i ha produces he minimum correlaion value for he region i represens he local moion vecor of his region, i.e v i =(p,q), for R i (p,q)=r imin. For Bi-plane maching algorihms he correlaion measure is based on he same procedure. Each local moion vecor of a region in he curren bi plane image is deermined by evaluaing he bi plane mach over subimages in he previous bi plane image and selecs he subimage which yields he closes maching. The correlaion measure is given by M 1 C j m n = (, ) g k ( x, y) g k ( x + m, y + n) M (4) x= 0 y= 0 p m, n p where g k (x,y) and g -1 k (x+m,y+n), respecively, are he curren and previous k-h order bi planes, and p is he maximum displacemen in he search window. Like RPM, he smalles C j (m,n) yields he bes maching for each region and

4 hus, he moion vecor of his region is C j =(m,n), for C j (m,n)=c jmin. Fig. 3. Typical archiecure of a digial image sabilizaion echnique. B. Moion Compensaion The objecive of moion compensaion is o keep some kind of hisory of he moion esimaes in order o creae a sabilized sequence wihou removing he ego-moion of he camera. The compensaion can be performed in wo possible implemenaions: The one ha always use wo consecuive frames from he inpu image sequence o esimae he moion parameers, which is referred o as frame-o-frame algorihm, and he second one ha keeps a reference image and uses i o esimae he moion beween he reference and he curren inpu image, which is referred o as he frame-o-reference algorihm. The mos widely used algorihm is he frame o frame algorihms while in he frame-o-reference algorihms he reference frame can become obsolee quie fas, even when a mosaic is buil and used as reference. This is mainly because of forward ranslaions bu mainly because of zooming operaion, since he reference becomes obsolee as soon as he curren frame does no share enough overlap area wih he reference. The compensaion vecor esimaion is compued by [10] CMV ( ) = k( CMV( 1) + ( acmv ( ) + ( 1 a) GMV( 1) (5) where represens he frame number, 0<k<1 and 0 a 1. The increase in k causes he decrease in unwaned shaking effec bu increase in he value of CMV, which means he effecive area of images is reduced in order o mainain he consisen image size for he whole image sequence. Fig.2, shows he frame o frame moion compensaion. The calculaed compensaion vecor shifs he curren frame according o he x and y values in order o minimize he shaking effec. IV. IMPACT OF ZOOMIG O IMAGE STABILIZATIO A ypical procedure for he zooming and sabilizaion procedure is depiced in Fig. 3. Firs, he focal lengh is changed by he user in order o enable he opical zooming feaure. Servos move mechanically he lens unil he desired opical zoom is achieved. The objec is now projeced on he sensor in a scale equal o he opical magnificaion facor. The capured from he sensor image sequence is hen processed in he moion esimaion uni for he global moion vecor calculaion. The nex cascading sep is he moion compensaion which evenually compensaes he image according o he moion compensaing vecors from he Fig. 4. Jier effec in differen opical magnificaion facors and par of he 9x9 search window. previous frame reference. In case he user has enabled also he digial zooming feaure, he sabilized image subsequenly, is mapped and inerpolaed resuling o a sabilized and zoomed image sequence. However, during his procedure wo shorcomings are easily observed. The firs one is an undesirable flucuaion of he oupu sequence which deerioraes while he user increases he opical zooming and he second one is a noiceable laency in he oupu sequence which was no presen during normal operaion. A. Inaccurae esimaion of he local moion vecors As described in deail in Secion III(a), he sabilizaion algorihm applies cerain search areas in he image regions o find he correlaion pixels hrough wo consecuive frames. These search areas, which are usually a 5x5 pixel window in convenional sysems, are pre-defined by he algorihm used and canno be changed. However, during he opical zooming, he images capured from he sensor are scaled leading o an inaccurae esimaion of he local moion vecors from sabilizaion algorihm. Fig. 4, illusraes he same pixel elemens represening he same paern in wo consecuive capured frames. The gray ones represen he paern in -1 frame and he red ones represen he same paern in frame. Here, we examine he effec of a jier in he sabilizaion process in hree differen opical zooming scenarios. The jier is equivalen o a ranslaion of wo and one pixel in horizonal and verical axis respecively for he 1 magnificaion. Therefore, he local moion esimaion uni is able o find he equivalen pixel in he nex frame because i is included in he applied search region. In case of a 2 magnificaion, he same movemen from he user equals o a four and wo pixel movemen, capured from he sensor. Here, he pixel ranslaion is also inside he acive range of he search window, hus he local moion vecors can be esimaed efficienly. However, for high magnificaion facors like 4, he wo and one iniial pixel ranslaion equals o an eigh and four pixel ranslaion, respecively. This means ha he RPM or bi-plane maching algorihms canno esimae reliably

5 Fig. 5. Block diagram of he proposed zooming sabilizaion archiecure. he moion vecors, while he represenaive pixels are ouside he range of he search window. This lack of effeciveness produces unwaned effecs like grea insabiliy of he oupu sequence or in a lack of keeping he ego moion of he camera, as shown in he performance evaluaion secion. In he CMOS digial image sabilizaion schemes in [14] and [15], he zooming and ranslaional effecs are aken also ino accoun. However, due o he fac ha hey are based on he disorion effec caused by he rolling shuering mechanism of he CMOS sensor, hey are effecive only in CMOS sensors. Furhermore, in case of zooming, he effeciveness is observed only during he change of he focal lengh. As far as he lenses have reached he new focal lengh, here is no enhancemen in he DIS performance. B. Increased Laency The recen digial image zooming echniques exhibi in performance bu lack in compuaional complexiy. Since users compromise only wih high qualiy in digial zooming, he implemenaion of a sophisicaed zooming algorihm in an embedded sysem have proved a challenge. While digial zooming enails complex image inerpolaion algorihms, he undesirable effec of increased laency is observed in he oupu sequence, especially in case of a concurren use of image sabilizaion. Approaches for reducing he compuaional burden have been proposed in [38] and [9], where he sabilizer is inegraed in video codec. The inegraion is performed using he moion vecors already calculaed wihin he sabilizer or he encoder. These schemes presen very good resuls in performance and coding efficiency. However, as described in [38], due o boundary effecs, he oupu can only be a sub-image of he original encoded version. In order o avoid his rade-off a subsequen zooming procedure is required o provide he original picure size. This means ha in addiion o he user's digial zooming an addiional zooming procedure mus be carried ou. Thus, a possible inegraion of he digial zooming would decrease even more he overall laency. The cascading archiecure in Fig. 3, found in mos implemenaions [9], [14], shows ha he sabilizaion and zooming are performed serial in he image signal processing module and he video encoder module, respecively. The daa in he laency column of Table I, show ha he oal laency of he zoomed and sabilized sequence is higher han ha of he only sabilized or only digially zoomed sequence. The variaion in he magnificaion facor does no affec he overall laency due o he fac ha in digial cameras, unlike imaging sofware applicaions, he number of pixels o be inerpolaed is fixed and equals o he sensor resoluion. TABLE I OBSERVED LATECY I THE OUTPUT SEQUECE FOR DIFFERET OPERATIG SCHEMES Image Sabilizaion Digial Zooming Laency (msec) OFF OFF 85 O OFF 160 OFF O 105 O O 195 V. ITEGRATIO OF IMAGE STABILIZATIO WITH ZOOMIG OPERATIO The purpose of he proposed inegraion is o increase he accuracy of he sabilizer and reduce he compuaional complexiy of he overall process. I mus be noiced, ha in case of digial or opical zooming operaion, each magnificaion facor is considered as a priori inpu o he embedded sysem. Three possible operaion scenarios are considered which can be found in an imaging device wih zooming and sabilizaion capabiliies. In he firs scenario, boh opical zooming and sabilizer are enabled. The inegraion leads o a refined local moion esimaion which improves he image sabilizaion performance. In he second scenario, he image sabilizaion is performed wih he digial zoom enabled. Here, he inegraion reduces he overall compuaional complexiy presening a reduced laency in he oupu sequence. In he hird scenario he image sabilizaion is performed wih boh opical and digial zooming enabled. In his scenario, he overall proposed inegraed archiecure is improved in boh erms of accuracy and compuaion efficiency compared o he convenional and sricly sequenial archiecure of Fig. 3. A. Scheme 1 When boh opical zooming and sabilizer are enabled, he effec of inaccurae esimaion of local moion vecors is presen, as described in deails in Secion IV(a). A soluion for his shorcoming would be he applicaion of a larger search region in order o include he corresponding pixels inside he new search region. However, his makes he moion esimaion algorihm even more compuaionally complex, making he uni no appropriae for real-ime applicaions. We propose an inegraed archiecure ha uses he opical scaling facor as an addiional inpu in he local moion esimaion uni, hus making he esimaion adapive o any zooming case. The moion esimaion uni, faciliaes he opical magnificaion facor as an exra inpu ha adjuss he search region, as shown in Fig. 5. In case of represenaive poin maching he (2) is

6 changed o R ( p, q) = I( 1 ), X, Y ) I(, X, Y ) (6) i r 1 r r r+ p os r+ p os where os is he opical magnificaion facor, inroduced by he curren opical zooming saus. The dynamic search region has he same amoun of represenaive pixels and can be adaped for any opical zooming facor. Wih his approach he accuracy is increased, since he correc represenaive pixels are locaed a he scaled search region. However, a correlaion value hreshold is required in he case of a non moving frame. For Bi-plane maching algorihms he inegraed correlaion measure also uses he opical magnificaion facor. Here, each local moion vecor of a region in he curren bi plane image is deermined by evaluaing he bi plane mach over adaped subimages in he previous bi plane image and selecs he subimage which yields he closes maching. The inegraed correlaion measure of (4) is given by M 1 Cj m n = 1 1 (, ) gk( x, y) gk M x= 0 y= 0 p m, n p 1 ( x+ m os, y+ n os) where g k (x,y) and g -1 k (x+m,y+n), respecively, are he curren and previous k-h order bi planes, and p is he maximum displacemen in he search window and os is he opical magnificaion facor. Like RPM, he amoun of bipixels o be evaluaed remains he same, wihou increasing he compuaional burden. B. Scheme 2 When boh digial zooming and sabilizer are enabled, he effec of an increased laency in he oupu sequence appears, as described in deails in Secion IV(b). We propose an inegraion of he image compensaion uni wih he mapping in digial zooming uni as shown in Fig. 5. Since, image compensaion is a frame ranslaion operaion, i can be inegraed ino he digial zooming uni. While his process is carried ou in he firs sep of he scaling process, which is common in all algorihms, he proposed inegraion is universal and can be implemened in all widely used inerpolaions. As a resul, he image compensaion process, inside he sabilizer can be omied, reducing he compuaional complexiy of he overall sysem. The proposed inegraion is depiced in Fig. 6. For a digial scaling facor ds and a inpu image P(i,j) le he oupu image be (i ds,j ds). For each pixel of he oupu image, new inensiy values should be calculaed according o he seleced inerpolaion mehod. Consequenly, he firs sep, requires he projecion of he arge pixel ono he source image as described in sep 1 of Secion II(b). The addiional inpu inroduced o he DZ, is he compensaion vecor CMV()=(g,h), where g, h are he pixel ranslaions in x and y axis, respecively, calculaed in he previous compensaion moion vecor esimaion uni. The addiional inpu is uilized in a way ha i will finally reallocae he inpu pixels o be inerpolaed. (7) Fig. 6. The four seps of he inegraed digial zooming archiecure. The x and y, Caresian coordinaes of he cener of he mapped oupu pixel, ono he inpu image are calculaed using he following equaions in order o inegrae he ranslaion operaion for he frame sabilizaion ino he zooming inerpolaion as shown in Fig. 6. l k x = + g, y = + h (8) ds ds where k=1,2 d,s m and l=1,2,,ds n, assuming ha each pixel has a heigh and widh equal o one. C. Scheme 3 In his scheme he image sabilizaion is performed wih boh opical and digial zooming enabled. The whole sysem flow is shown in Fig. 5, comprised by he wo previously discussed scenarios. Firsly, he opical zoomed sequence capured from he sensor, is processed in he moion esimaion uni, as explained in Scheme 1. The exra inpu of he opical magnificaion facor, makes he local moion esimaion procedure dynamic, wihou having he permanen seup of convenional DIS archiecures. Secondly, as he digial zooming is also enabled by he user, he image compensaion uni is excluded, and merged in he mapping phase of he digial zooming uni, as described previous in Scheme 2. As a resul of he full inegraion, he DIS presens beer performance in addiion o a reduced compuaional complexiy. VI. PERFORMACE EVALUATIO In order o evaluae and confirm he validiy of he proposed archiecure, exensive simulaions were conduced and compared o convenional exising sabilizaion archiecures found in cameras. The performance of he inegraion scheme is evaluaed in wo aspecs: performance accuracy and compuaional burden. Evaluaion of he performance of a digial sabilizaion archiecure in differen zooming condiions is a difficul ask since we mus have he same image sequence capured in several zooming magnificaion facors. Furhermore, he ground ruh of a hand shake is unavailable. Evaluaion procedures for image sabilizaion algorihms and archiecures were presened in [38]-[40].

7 Fig. 7. Video camera plaform wih inerial sensor, four servos wih encoders and four servo conrollers. A. Performance Accuracy The esbed designed for he performance evaluaion is depiced in Fig. 7. The seup consiss of mechanical plaform wih four d.o.f, equipped wih high precision servo moors wih encoders. A digial commercial camera wih opical zooming funcion is mouned on he plaform for capuring video in differen opical magnificaion facors. An inerial sensor module is mouned on he video camera o deec he camera movemen inroduced by he mechanical plaform. The same moion profiles are applied, hrough he servo conrollers, for each differen zooming facor for comparison purposes. The moion profiles generae jiggling moions in he verical direcion, simulaing he handshaking movemens. Performance is evaluaed on he basis of he roo mean square error (RMSE) beween he desired pixel displacemen and he pixel displacemen of he archiecure used. The RMSE is given by 1 e rms = + ( xn xˆ n ) ( yn yˆ n ) (9) n= 1 where (x n,y n ) is he compensaing moion vecor, afer he opical zooming process, calculaed from he evaluaed archiecure and x ˆ, y ˆ ) is he desired moion vecor ( n n convered from he angular velociy deermined by he inerial sensor. Sabilizaion resuls using he non-inegraed archiecure and he proposed inegraed archiecure are presened in Fig. 8. The es sequence Lab is shown in Fig. 9, which is capured and hen sabilized, in hree differen opical magnificaion facors. According o he experimens, he proposed inegraed archiecure is more robus compared o he sricly sequenial one, when opical zooming is used. The non-dynamic sabilizaion produces unreliable moion vecors, and i canno deec efficienly he grea pixel displacemens due o he zooming effec. When operaing wihou zoom, in Fig. 8(a), he experimenal resuls are equivalen, since here is no opical magnificaion facor used and all he represenaive pixels are inside he range of he search window. However, in (a) (b) (c) Fig. 8. Performance comparison of differen sabilizaion archiecures for he Lab sequence, in respec o differen opical zooming saus: (a) 1; (b) 5; (c) 10. The red line denoes he raw sequence, he gray line denoes he sequence sabilized by he non-inegraed archiecure and he blue line denoes he sequence sabilized by he proposed inegraed archiecure.

8 (a) (b) (c) (d) (e) (f) (g) (h) (i) Fig. 9. Sequence Lab: (a) reference frame in 1 opical zoom; (b) reference frame in 5 opical zoom; (c) reference frame in 10 opical zoom; (d) sabilized by non-inegraed archiecure in 1 opical zoom; (e) sabilized by non-inegraed archiecure in 5 opical zoom; (f) sabilized by noninegraed archiecure in 10 opical zoom; (g) sabilized by proposed inegraed archiecure in 1 opical zoom; (e) sabilized by proposed inegraed archiecure in 5 opical zoom; (f) sabilized by proposed inegraed archiecure in 10 opical zoom. TABLE II RMSE COMPARISOS OF THE ITEGRATED AD O-ITEGRATED STABILIZATIO ARCHITECTURES WITH RESPECT TO DIFFERET OPTICAL ZOOMIG FACTORS. Archiecure 1 Opical zooming facor 5 10 on-inegraed Inegraed Fig. 8(b), for 5 magnificaion facor, he non inegraed archiecure canno esimae he righ local moion vecors in some secions where he pixel displacemens are ouside he search window. In Fig. 8(c) he effec of inaccurae esimaion is even more noiceable, in conras o he proposed archiecure which is able o reduce he high frequency moions and keep he ego moion of he camera. The RMSE for he wo differen archiecures is shown in Table II. B. Compuaional Cos Fas moion esimaion has been exensively sudied. The inegraion framework presened here, does no propose a new algorihm for fas moion esimaion. However, he coding performance is affeced by he exclusion of he frame ranslaion operaion, inside he sabilizer. Therefore, he compuaional burden is reduced by a cerain number of operaions, equivalen o a frame ranslaion operaion. For a M-pixel image resoluion and a frame rae per second f he compuaional burden is reduced by 2 M f addiions, in a second. I is clear ha he compuaional gain in he inegraed archiecure is even more noiceable in high frame raes and large pixel image resoluions. VII. COCLUSIO In his paper we have proposed a new archiecure for adapive digial sabilizaion in zooming operaion. The archiecure is efficien for any possible opical or digial zooming operaion scenario. The sabilizer adaps is operaional parameers from he curren opical zooming saus for a refined local moion esimaion. Furhermore, he global moion compensaion vecors produced by he digial sabilizer are send o he digial zooming sysem, where he image compensaion is merged wihin he inerpolaion process, in order o reduce he compuaional complexiy. Experimenal resuls have showed ha he proposed archiecure exhibi beer quaniaive performance compared wih he exising convenional archiecure, when zooming process is enabled. Furhermore, he proposed sysem has less compuaional complexiy minimizing he observed laency of he convenional archiecures, when digial zooming is enabled. The observed improvemens make he archiecure appropriae for real-ime imaging applicaions.

9 REFERECES [1] K. M. Oshima, T. Hayashi, S. Fujioka, T. Inaji, H. Miani, J. Kajino, K. Ikeda, and K. Komoda, VHS camcorder wih elecronic image sabilizer, IEEE Trans. Consum. Elecron., vol. 35, no. 4, pp , [2] T. Kinugasa,. Yamamoo, H. Komasu, S. Takase, and T. Imaide, Elecronic image sabilizer for video camera use, IEEE Trans. Consum. Elecron., vol. 36, no. 3, pp , [3] B. Cardani, Opical image sabilizaion for digial cameras, IEEE Conrol Sys. Mag., vol. 26, no. 2, pp , [4] J. Tokyo, Conrol echniques for opical image sabilizing sysem, IEEE Trans. Consum. Elecron., vol. 39, no. 3, pp , [5] S. Ko, S. Lee, S. Jeon, and E. Kang, Fas digial image sabilizer based on gray-coded bi-plane maching, IEEE Trans. Consum. Elecron., vol. 45, no. 3, pp , [6] F. Vella, A. Casorina, M. Mancuso, and G. Messina, Digial image sabilizaion by adapive block moion vecors filering, IEEE Trans. Consum. Elecron., vol. 48, no. 3, pp , [7] L. Xu and X. Lin, Digial image sabilizaion based on circular block maching, IEEE Trans. Consum. Elecron., vol. 52, no. 2, pp , [8] Z. Pan and C. go, Selecive objec sabilizaion for home video consumers, IEEE Trans. Consum. Elecron., vol. 51, no. 4, pp , [9] H. Chen, C. Liang, Y. Peng, and H. Chang, Inegraion of digial sabilizer wih video codec for digial video cameras, IEEE Trans. Circuis Sys. Video Technol., vol. 17, no. 7, pp , [10] J. Paik, Y. Park, D. Kim, and S. Co, An adapive moion decision sysem for digial image sabilizer based on edge paern maching, IEEE Trans. Consum. Elecron., vol. 38, no. 3, pp , [11] S. Erurk, Digial image sabilizaion wih sub-image phase correlaion based global moion esimaion, IEEE Trans. Consum. Elecron., vol. 49, no. 4, pp , [12] A. ai-ali, Geneic algorihms for blind digial image sabilizaion under very low SR, IEEE Trans. Consum. Elecron., vol. 53, no. 3, pp , [13] S. Hsu, S. Liang, and C. Lin, A robus digial image sabilizaion echnique based on inverse riangle mehod and background deecion, IEEE Trans. Consum. Elecron., vol. 51, no. 2, pp , [14] W. Cho, D. Kim, and K. Hong, CMOS digial image sabilizaion, IEEE Trans. Consum. Elecron., vol. 53, no. 3, pp , [15] W. Cho and K. Hong, Affine moion based CMOS disorion analysis and CMOS digial image sabilizaion, IEEE Trans. Consum. Elecron., vol. 53, no. 3, pp , [16] D. Zhang,. Jusis, and Y. Lo, Inegraed fluidic adapive zoom lens, Opics Leers, vol. 29, no. 24, pp , [17] S. Park and R. Shannon, Zoom lens design using lens modules, Opical Engineering, vol. 35, no. 6, pp , [18] I. Andreadis and A. Amanaiadis, Digial image scaling, in Proc. of he IEEE Insr. Meas. Technol. Conf., vol. 3, 2005, pp [19] A. Amanaiadis, I. Andreadis, and K. Konsaninidis, Fuzzy area-based image scaling, in Proc. of he IEEE Insr. Meas. Technol. Conf., 2007, pp [20] A. Amanaiadis and I. Andreadis, An inegraed dynamic image sabilizer applied o zooming sysems, in Proc. of he IEEE Insr. Meas. Technol. Conf., 2007, pp [21] V. Peddigari and. Keharnavaz, A relaional approach o zoom racking for digial sill cameras, IEEE Trans. Consum. Elecron., vol. 51, no. 4, pp , [22] W. Smih, Modern Lens Design. McGraw-Hill, [23] A. Jain, Fundamenals of Digial Image Processing. Prenice-Hall, Inc. Upper Saddle River, J, USA, [24] R. Keys, Cubic convoluion inerpolaion for digial image processing, IEEE Trans. Acous., Speech, Signal Process., vol. 29, no. 6, pp , [25] J. Hwang and H. Lee, Adapive image inerpolaion based on local gradien feaures, IEEE Signal Process. Le., vol. 11, no. 3, pp , [26] M. Unser, A. Aldroubi, and M. Eden, Fas B-spline ransforms for coninuous image represenaion and inerpolaion, IEEE Trans. Paern Anal. Mach. Inell., vol. 13, no. 3, pp , [27] S. Baiao, G. Gallo, and F. Sanco, A locally adapive zooming algorihm for digial images, Image and Vision Compuing, vol. 20, no. 11, pp , [28] C. Chang, Y. Chou, Y. Yu, and K. Shih, An image zooming echnique based on vecor quanizaion approximaion, Image and Vision Compuing, vol. 23, pp , [29]. Plaziac, E. Sys, and. Woodbury, Image inerpolaion using neural neworks, IEEE Trans. Image Process., vol. 8, no. 11, pp , [30] J. Chen, J. Chang, and K. Shieh, 2-D discree signal inerpolaion and is image resampling applicaion using fuzzy rule-based inference, Fuzzy Ses and Sysems, vol. 114, no. 2, pp , [31] H. Ting and H. Hang, Edge preserving inerpolaion of digial images using fuzzy inference, Journal of Visual Communicaion and Image Represenaion, vol. 8, no. 4, pp , [32] D. Michell and A. eravali, Reconsrucion filers in compuergraphics, in Proc. of he 15h annual conference on compuer graphics and ineracive echniques, 1988, pp [33] G. Grevera and J. Udupa, Shape-based inerpolaion of mulidimensional grey-level images, IEEE Trans. Med. Imag., vol. 15, no. 6, pp , [34] D. Giuso, M. Murroni, and G. Soro, Slow moion replay of video sequences using fracal zooming, IEEE Trans. Consum. Elecron., vol. 51, no. 1, pp , [35] D. Monro and P. Wakefield, Zooming wih implici fracals, in Proc. of he In. Conf. on Image Processing, vol. 1, 1997, pp [36] R. Dugad and. Ahuja, A fas scheme for image size change in he compressed domain, IEEE Trans. Circuis Sys. Video Technol., vol. 11, no. 4, pp , [37] C. Wang, H. Yu, and M. Zheng, A fas scheme for arbirarily resizing of digial image in he compressed domain, IEEE Trans. Consum. Elecron., vol. 49, no. 2, pp , [38] A. Engelsberg and G. Schmid, A comparaive review of digial image sabilising algorihms for mobile video communicaions, IEEE Trans. Consum. Elecron., vol. 45, no. 3, pp , [39] C. Morimoo and R. Chellappa, Evaluaion of image sabilizaion algorihms, in Proc. In. Conf. Acous., Speech, Signal Process., vol. 5, 1998, pp [40] S. Balakirsky and R. Chellappa, Performance characerizaion of image sabilizaion algorihms, in Proc. In. Conf. Image Process., vol. 1, 1996, pp Angelos Amanaiadis (S 00) received he Diploma Degree from he Deparmen of Elecrical & Compuer Engineering, Democrius Universiy of Thrace, Greece, in He is currenly pursuing his Ph.D. degree. His areas of ineress include image processing, compuer vision, roboics and real-ime FPGA-based sysems. He is member of he Insiue of Elecrical and Elecronics Engineers (IEEE) and member of he Technical Chamber of Greece (TEE). He is involved in several naional (Greek) and European research projecs in he field of real-ime machine vision sysems. Ioannis Andreadis received he Diploma Degree from he Deparmen of Elecrical & Compuer Engineering, Democrius Universiy of Thrace (DUTH), Greece, in 1983 and he MSc and PhD Degrees from he Universiy of Mancheser Insiue of Science & Technology, UK, in 1985 and 1989, respecively. His research ineress are mainly in elecronic sysems design, inelligen sysems and machine vision. In hese areas he has published more han 150 referred publicaions in book chapers, inernaional journals and conferences. He joined he Deparmen of Elecrical & Compuer Engineering, DUTH in Professor Andreadis is a Fellow of he Insiuion of Engineering and Technology, (IET - Formerly IEE), an Associae Edior of he Paern Recogniion Journal and a member of he Technical Chamber of Greece.