Combined Independent Component Analysis and Kalman Filter Based Real-Time Digital Video Stabilization

Transcription

1 Internatonal Journal of Sgnal Processng Systems Vol. 1, No. 2 December 2013 Combned Independent Component Analyss and Kalman Flter Based Real-Tme Dgtal Vdeo Stablzaton Hassaan S. Quresh, Syed A. Jabr, Syeda H. Taqdees, and Khawar Khurshd Department of Electrcal Engneerng, School of Electrcal Engneerng and Computer Scences Natonal Unversty of Scences & Technology (NUST) Islamabad, Pastan Emal: {11mseehsaadat, 11mseeaabr, 11mseestaqdees, hawar.hurshd}@seecs.edu.p In the frst stage, we estmate the moton of each frame wth respect to the prevous frame. Ths s called moton estmaton. At ths stage, the moton vector s the mxture of the ntentonal moton and the unwanted hgh frequency moton. Step 2 nvolves the separaton the ntentonal moton from the unwanted moton and fnally n Step 3, we apply the nverse of the unwanted moton to the vdeo sequence, whle eepng the ntentonal moton. A lot of research has been done on each of the three steps. In ths paper, we focus on the step of moton taxonomy whch s the heart of Dgtal Vdeo Stablzaton. Recently, Amanatads and Andreads [1] have proposed a novel technque for moton separaton based on Independent Component Analyss. The authors assume that both ntentonal and unntentonal motons are ndependent of each other and hence they can be separated by ICA. However, they have mentoned that ther technque s not suted for onlne stablzaton as the methodology requres all moton vectors pror to the applcaton of ICA. In ths paper, we propose a modfcaton to the proposed algorthm n [1] to mae t sutable for real tme mplementaton. We combne Kalman flter [2] wth the proposed approach of ICA to acheve ths target. Smulaton results show that the output ntentonal motons vector are less ttery than both Kalman based stablzaton output and ICA based stablzaton output when they are used separately. The organzaton of ths paper s as follows. In Secton II, we present the exstng approaches for dgtal vdeo stablzaton. A bref ntroducton to ICA and Kalman flterng s gven n Secton III. The methodology proposed n [1] and our proposed modfcaton s presented n Secton IV. In Secton V we present the smulaton results and fnally, we conclude the paper n Secton VI wth some dscusson of the future wor. Abstract We present an onlne approach for dgtal vdeo stablzaton. Ths paper bulds upon the wor presented n where the authors have proposed the use of Independent Component Analyss (ICA) for vdeo stablzaton. Ther scheme however s ntended for offlne systems and cannot be used for onlne systems. We propose a wndow based approach to combne the ICA based technque wth Kalman flter, n order to mae the method sutable for onlne systems. The proposed approach s tested usng smulatons and the mean square error between the outcome and the ground truth s used as performance measure. The performances of the proposed method, separate ICA and separate Kalman flter applcaton are compared. The performance of the proposed method turns out to be better than separate applcaton of Kalman flterng or ICA alone. Index Terms vdeo stablzaton, ICA, Kalman flter I. INTRODUCTION Over the recent years, the doman of vdeo stablzaton has been actve area of research. The technques for vdeo stablzaton are becomng popular wth the growng advancement n dgtal magng devces. Most of the tmes, the base of the camera may not be perfectly fxed. The camera mounted on a traffc sgnal, on a movng vehcle or hand held may be subected to varous vbratons and swngs caused by wnd, engne vbratons or hand unsteadness respectvely. The effect of these unntentonal camera moton results n annoyng hgh frequency motons n the captured mage sequence. Furthermore, a part of the moton s ntentonal when we ntentonally move the camera or the camera carryng car turns. Thus the process of vdeo stablzaton s to remove the unntentonal moton whle eepng the ntentonal moton. These unntentonal hgh frequency vbratons are not only unpleasng for humans but also reduce the performance of other computer vson and compresson algorthms le obect tracng and segmentaton etc. The process of vdeo stablzaton can be dvded nto three man steps, 1) Moton estmaton, 2) Moton taxonomy or Moton separaton and 3) Moton compensaton. II. In ths secton, we present dfferent procedures that have already been proposed n lterature by dfferent authors. Manuscrpt receved September 13, 2013; revsed November 19, do: /sps RELATED WORK 164

2 Internatonal Journal of Sgnal Processng Systems Vol. 1, No. 2 December 2013 In one desgn proposed by Rataonda K. [3], a computatonal effcent technque s developed to estmate the local and global moton vectors, assumng that small varatons can be handled as transtonal model usng proecton matchng. However, ths paper does not handle the movng obect problem. Pva S. et al. [4] propose the real tme vdeo stablzaton for the camera mounted on a locomotve of a hgh-speed tran. They have appled translatonal model to fnd the rotatonal parameters by assumng large perpheral of camera doesn t change maxmum correlaton value. The authors n [5] apply the smlarty transformaton to mages obtaned by the camcorder mounted on a movng vehcle by estmatng moton based on global feature extracton such as road lanes. Zhu J. et al. [6] estmated the local and global moton vectors by dvdng the mage nto four sub-mages and fndng the local moton parameters for each one separately. These local vectors are weghted adaptvely n order to fnd the global moton parameters. The authors n [7] proposed a desgn to estmate camcorder motons by calculatng the hstograms of local motons. The poston of the hghest pea n each local moton hstogram s ndependent to estmaton errors, thus provdng a robust soluton to the nterference of movng obects. Yang J. et al. [8] proposed the affne model transformaton for the moton estmaton obtaned by the hand held camera. They facltated the use of affne model by usng the feature pont detecton and partcle flters to estmate the global camera moton between successve frames. The authors n [9] propose stablzaton technque usng both the smlarty and the affne transforms. The approach s to extract the robust weghted feature traectores from the nput vdeo and fnd a set of transformatons to smooth out these traectores and fnally stablze the vdeo. Battato et al. [10] proposed a method to perform bloc matchng, pre-flterng, memory flterng, robust estmaton and then computaton of error vector. Battato et al. [11] have also proposed another approach n whch the global moton model for the vdeo frame caters translatons n x and y axs, a rotaton and zoom factor. Least square method s used to avod moton estmaton errors. In [12], Hu et al. use affne model for global moton model and SIFT s appled to estmate camera moton. Shay moton and ntentonal moton are separated out by Gaussan flterng. III. THEORETICAL BACKGROUND In ths secton we present some basc ntroducton to the Kalman flter and the ICA. A. Kalman Flter In most smple words, Kalman flterng can be smply descrbed as nosy data n, hopefully less nosy data out [13]. The Kalman flterng conssts of two sets of computatons, the predcton part, and the correcton part. In the predcton part, at tme ndex -1, the Kalman flter frst predcts the process state vector for tme ndex. Ths s called as apror estmate. In the correcton stage, the Kalman flter correct ths apror estmate after obtanng the measurement at tme ndex. Ths s the posteror estmate, calculated by gvng an optmal weghtage to the measurement and the predcton usng the Kalman gan. xˆ Axˆ Bu 1 P AP A Q T 1 Above equatons are nown as pror estmates or predctor equatons. xˆ and P are the state and covarance estmates at ndex 1. A s the state transton matrx and Q s the process nose covarance. K P H ( HP H R) T T 1 xˆ xˆ K ( z Hxˆ ) P ( I K H) P These equatons are nown as posteror estmate or corrector equatons. K s the Kalman gan, z s the observed output whch s used to estmate state at ndex. R s the measurement nose covarance. In ths paper, we have used the constant speed model as t s suffcent for most of the cases. B. Independent Component Analyss Independent component analyss s a technque to recover the unobservable ndependent sgnals from ther observable lnear combnatons by maxmzng ther nongaussanty. It s the extenson of Prncple Component Analyss (PCA) requrng the addtonal constrant of Independence of sources along wth ther uncorrelatedness [14]. ICA s descrbed usng the statstcal latent varables model of [15]. Suppose that we have observed n random varables x1, x2,..., xn whch are the lnear combnatons of n ndependent sources s1, s2,..., s n such that n n (1) (2) x a s a s a s (3) Usng matrx notaton, observaton of n x random varables can be expressed as x As (4) where x and s are the vectors contanng x and respectvely. A s called the mxng matrx whch conssts of the rows of mxng coeffcents for each of the observed lnear combnatons. More precsely, the columns of A depct the contrbuton of each source s to the lnear combnatons x.e. n 1 s x a s (5) In ICA, sources s must be non-gaussan as well as ndependent. It analyzes the relatonshp between 165

3 Internatonal Journal of Sgnal Processng Systems Vol. 1, No. 2 December 2013 ndependent sources and ther lnear combnatons by estmatng de-mxng matrx W (the nverse of the unnown matrx A ) by maxmzng the non-gaussanty of the pdfs of x. The ey dea s that the sum of non- Gaussan random varables s are much more Gaussan then the orgnal ones based upon the central lmt theorem of statstcal random varables. So for estmatng the latent varables s, we try to maxmze the nongaussanty and ndependence of x by multplyng t wth a rotatonal matrx W such that s Wx (6) Non-gaussanty of random varables are measured by calculatng urtoss.e. fourth-order cumulate. For a random varable wth zero mean, the normalzed urtoss s calculated through the followng formula. 4 { u } urt( u) (7) 2 2 ( { u }) The basc property of the normalzed urtoss s that, for Gaussan random varables, urtoss s zero. For most non-gaussan random varables, urtoss s nonzero. There are some lmtatons assocated wth ICA. As both the A and s n (4) are unnown, we can freely change the order of the terms n the sum n (5), so we cannot determne the exact order of the ndependent components. Thus the order of the output sgnal s uncertan. Ths ambguty s called as permutaton ambguty. Another lmtaton s that the energes of the resultng ndependent sgnals may be a scaled verson of energy of the actual sgnals. Ths s agan because both A and s are unnown n (4) and any scalar multplcaton wth s can be balanced wth the correspondng scalar dvson wth A. Ths s called ampltude ambguty. IV. PROPOSED METHOD AND MODIFICATIONS In ths secton, we frst descrbe the approach presented by Amanatads and Andreads [1] n detal, then n the next subsecton, we present our proposed approach to mae the technque sutable for onlne vdeo stablzaton. A. Vdeo Stablzaton Usng ICA As mentoned earler, the authors n [1] have proposed a novel approach of separatng the wanted and unwanted moton from the estmated moton vectors. They assume the two motons are ndependent of each other, and hence they can be separated by Independent Component Analyss. ICA s bascally a technque used for separaton of ndvdual sgnal from a mxture of ndependent sgnals mostly popular for the coctal party problem [16]. The authors proposed that the problem of vdeo stablzaton can be made analogous to the coctal party problem. The two recordng mcrophones correspond to dfferent regons of the frame and the recorded sound corresponds to the estmated moton vectors for each regon. The regons whch nclude the foreground obects should not be selected for moton vector estmaton. The authors also refer to the dfferent technques to detect whether a regon contans foreground obects or not. Thus the wanted and unwanted moton can be separated by the applcaton of ICA. As mentoned earler, the ICA has two maor lmtatons, the permutaton ambguty and the ampltude ambguty. The authors have also suggested a soluton to these problems. The permutaton ambguty can be resolved by assumng that the wanted moton has lower frequency that the unntentonal moton and dong the frequency analyss. After ths ambguty has been resolved, we can easly resolve the magntude ambguty by proectng the sgnal bac to the observatons space. As a result we get the contrbuton of each consttuent sgnal, the ntentonal and unntentonal moton, n each of the observaton. The sgnal representng the ntentonal and unntentonal moton can then be obtaned by averagng ther respectve contrbutons n the observaton sgnals. However, the authors descrbe that ths algorthm cannot be used for onlne stablzaton as ths technque requres moton vectors for all vdeo frames pror to the ICA applcaton. In the followng subsecton, we present our approach as the soluton to ths problem where we use Kalman flter n combnaton wth ICA to modfy the algorthm n order to mae t sutable for onlne mplementaton. B. Proposed Modfcatons As descrbed n Secton III, the Kalman flter frst predcts the state of the process at tme ndex, and then corrects the predcton after obtanng the measurement at tme ndex. The orgnal Kalman flterng for the moton separaton stage [17] proposes the applcaton of Kalman flter on the observed moton vector. The moton vector s taen as the nput nosy data and the smooth output moton s taen as the separated ntentonal moton. In our method, we use the dea of sldng wndow and propose to compute ICA for each wndow separately. If the wndow sze s W, then at each frame ndex, we compute the ICA for (-W)th frame to th frame. The value of resultant ICA-separated ntentonal moton at s gven as th measurement to the Kalman flter whch then computes the actual value of the curve at that pont. Effectvely, ths means that the Kalman flter s now operatng on less nosy data, whch wll result n smoother output. For the startng frames, when <W, we assume that the moton vector data s our measurement and s passed drectly to Kalman flter. Hence the performance wll be relatvely worse for the frame when <W, but after ths, the moton separaton performance of ths system provdes better results than the cases when ICA or Kalman flter s appled separately for the moton estmaton stage. V. RESULTS 166

4 Internatonal Journal of Sgnal Processng Systems Vol. 1, No. 2 December 2013 The proposed algorthm of combnng ICA wth Kalman flter has been tested on smulated data and the results are shown and dscussed below. A. Testng Setup The wanted moton curve s modeled wth a sgnal that s sum of snes and cosnes (Fg. 1). The unntentonal moton s smulated by zero mean unform random process (Fg. 2). Fgure 4. Observaton 2. Fgure 1. Desred moton curve. Fgure 5. Mean of observaton 1 & 2. B. Dscusson Fgure 2. Unntentonal moton. The two sgnals are mxed usng a mxng matrx to get two observatons whch are then passed to our system (Fg. 3 and Fg. 4). To apply Kalman flter, we need only a sngle measurement, so we tae average of the two observatons (Fg. 5). Ths step s ustfed as the ICA based algorthm also returns the ntentonal moton as the average of ts contrbutons n each observed sgnal. The bloc sze W s set to 50 for the experments. Fgure 6. Output of Kalman flter only. Fgure 3. Observaton 1. Fgure 7. Output of ICA alone. 167

5 Internatonal Journal of Sgnal Processng Systems Vol. 1, No. 2 December 2013 n the wor of [1]. Instead of computng ICA for the complete Moton Vector data, we proposed a wndow based approach, where ICA s computed for the bloc of data contanng the value of moton vector at the present frame and at W-1 prevous frames. The output of ths system s passed as measurements to the Kalman flter. The results show that the smoothng performance of ths system s better than the case ICA or Kalman flterng s appled separately. At present we have only tested the approach on smulated data. In future, we plan to test the method on real vdeo. Another thng that can be tested s that we used pure sldng wndow where each wndow s shfted by sngle ndex at a tme. The ump of the wndow can be ncreased to save the computatonal requrements. Fgure 8. Output for combned Kalman flter and ICA (CICAK). Fg. 6 Fg. 8 shows the output for the Kalman flter alone, the ICA alone and the proposed combned ICAKalman (CICAK). Clearly, t can be observed the result for the CICAK s the best among the three as far as smoothness s concerned. The output for the CICAK s ntally same as the out of Kalman because at those nstances, the observed moton vector s gven as measurements to the Kalman flter. After W samples, the ICA-separated ntentonal moton s gven as measurements to the Kalman flter. Snce ths data s less nosy, thus the output of the Kalman flter becomes smoother. We have used frst dervatve as a smoothness measure. Fg. 9 shows the smoothness curve for Kalman alone, ICA alone, CICAK and GT (ground truth). We can see that after W samples CICAK follows GT n a smoother way compared to ICA and Kalman alone. The mean square error between dervatve of GT and CICAK s REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] Fgure 9. Smoothness curve. VI. [13] CONCLUSION AND FUTURE WORK [14] In ths wor, we have presented an onlne vdeo stablzaton method based on ICA and Kalman flter. The wor s ntended to solve the lmtaton of non-causalty 168 A. A. Amanatads and I. Andreads, Dgtal mage stablzaton by ndependent component analyss, IEEE Transactons on Instrumentaton and Measurement, vol. 59, no. 7, pp , R. E. Kalman, A new approach to lnear flterng and predcton problems, Journal of Basc Engneerng, no. 82, Seres D, pp. 3545, K. Rataonda, Real-tme dgtal vdeo stablzaton for multmeda applcatons, n Proc. IEEE Internatonal Symposum on Crcuts and Systems, 1998, pp S. Pva, M. Zara, G. Gera, and C. Regazzon, Color-based vdeo stablzaton for real-tme on-board obect detecton on hgh-speed trans, n Proc. IEEE Conference on Advanced Vdeo and Sgnal Based Survellance, 2003, pp Y. Lang, H. Tyan, S. Chang, H. Lao, and S. Chen, Vdeo stablzaton for a camcorder mounted on a movng vehcle, IEEE Transactons on Vehcular Technology, vol. 53, no. 6, pp , J. Zhu and B. Guo, Vdeo stablzaton wth sub-mage phase correlaton, Chnese Optcs Letters, vol. 4, no. 9, pp , J. Ca and R. Waler, Robust moton estmaton for camcorders mounted n moble platforms, presented at Dgtal Image Computng: Technques and Applcaton 2008, Canberra, Australa, 1-3 December J. Yang, D. Schonfeld, and M. Mohamed, Robust vdeo stablzaton based on partcle flter tracng of proected camera moton, IEEE Transactons on Crcuts and Systems for Vdeo Technology, vol. 19, no. 7, pp , K. Lee, Y. Yu, C. Yu, and C. Ouhyoung, Vdeo stablzaton usng robust feature traectores, n Proc. IEEE Internatonal Conf. on Computer Vson, Japan, September 2009, pp S. Battato and G. Pugls, A robust vdeo stablzaton system by adaptve moton vectors flterng, presented at 2008 IEEE Internatonal Conference on Multmeda and Expo, Hannover, Germany, June 23-26, S. Battato, G. Gallo, and S. Scellato, SIFT features tracng for vdeo stablzaton, n Proc. 14th Internatonal Conference on Image Analyss and Processng, 2007, pp R. Hu, R. Sh, and I. Shen, Vdeo stablzaton usng scalenvarant features, n Proc. 11th Internatonal Conf. on Informaton Vsualzaton, 2007, pp G. Bshop and G. Welch. An Introducton to the Kalman Flter. [Onlne]. Avalable: 8&rep=rep1&type=pdf J. Karhunen, E. Oa, L. Wang, R. Vgaro, and J. Joutsensalo, A class of neural networs for ndependent component analyss, IEEE Transactons on Neural Networs, vol. 8, no. 3, pp , 1997.

6 Internatonal Journal of Sgnal Processng Systems Vol. 1, No. 2 December 2013 [15] A. Hyvärnen and E. Oa, Independent component analyss: Algorthms and applcatons, Neural Networs, vol. 13, pp , May-June, [16] Y. Anhoren, S. Engelberg, and S. Fredman, The coctal party problem, IEEE Instrum. Meas. Mag., vol. 11, no. 3, pp , June [17] A. Ltvn, J. Konrad, and W. C. Karl, Probablstc vdeo stablzaton usng Kalman flterng and mosacng, n Proc. SPIE Image and Vdeo Communcatons and Proc., 2003, pp Hassaan S. Quresh receved hs B.E. degree n electrcal engneerng from Natonal Unversty of Scences & Technology (NUST), Pastan. He s currently pursung M.S degree n electrcal engneerng from School of Electrcal Engneerng & Computer Scence (SEECS), NUST. He s a research assstant at Vson Image and Sgnal Processng (Vspro) Lab and research nterests nclude mage processng, embedded systems and HDL based dgtal desgn. Ms. Hra Taqdees obtaned her Bachelors of electrcal engneerng degree from College of Electrcal and Mechancal Engneerng (CE&ME), NUST, Pastan. and s currently pursung her Masters n electrcal engneerng from School of Electrcal Engneerng and Computer Scence (SEECS), NUST. Her feld of specalzaton s dgtal mage processng and formal verfcaton of analog crcuts. Current and prevous research nterests ends the paragraph. Syed A. Jabr receved hs B.E. and M.S degrees n computer engneerng and electrcal engneerng respectvely from Natonal Unversty of Scences & Technology (NUST), Pastan. He s currently a lecturer at Center for Advanced Studes n Engneerng (CASE). Hs areas of research nterest nclude sgnal processng, mage processng embedded systems and FPGA based hardware-software co-desgn. Dr. Khurshd obtaned hs doctorate degree from Mchgan State Unversty n Bomedcal Imagng Systems and currently he s headng the Insttute of Appled Electroncs and Computers at Natonal Unversty of Scences and Technology (NUST), Pastan. He s also the drector of Cypress-NUST Research Center focusng on solutons development on PSoC. Dr. Khurshd specalzes n Medcal Imagng, Computer Vson, Pattern Recognton, Image and Sgnal Processng wth research nterests n the areas of mage segmentaton, regstraton, vdeo encodng, multmeda streamng and 3D vdeo dsplay systems. 169