CONTROLLING A VIRTUAL MARIONETTE USING A WEB CAMERA

Transcription

1 CONTROLLING A VIRTUAL MARIONETTE USING A WEB CAMERA b Ale Siroa, 9688, ale@elbrus.com Dov Sheinker, 8977, dov.sheinker@inel.com Oren Yossef, 89, oossef@qualcomm.com Projec in inelligen ssems, 675 Compuer Science Deparmen Technion Israel Insiue of Technolog Under supervision of: Professor Alfred Brucksein, Adi Bar-Lev Sepember

2 TABLE OF CONTENTS. INTRODUCTION.... SOLUTION AND ALGORITHMS..... GENERAL IMAGE SEGMENTATION AND LOCATING THE OBJE CT... ORIENTATION AND TRANSLATION RECONSTRUCTION...5. SOFTWARE ARCHITECTURE GENERAL SOFTWARE LIBRARIES USED Microsof's VisionSDK Carnegie Mellon Universi Vision Librar (CMVision) SOFTWARE MODULES Class CWacher Class CRecon MarioLibDemo Program..... FLOW.... EXPERIMENTS GENERAL... VISION..... RECONSTRUCTION RESULTS AND CONCLUSIONS RESULTS PROBLEMS AND ENHANCEMENTS REFERENCES APPENDIX A: THE HSV COLOR MODEL APPENDIX B: MATLAB IMPLEMENTATION OF THE RECONSTRUCTION.... APPENDIX C: OLD ALGORITHM FOR ORIENTATIO N AND TRANSLATION RECONSTRUCTION...5

3 . INTRODUCTION The problem we were ring o solve is o manipulae a virual D objec like a marionee using a real world D objec and a camera. The real objec image, as received from he camera is analzed, he objec is locaed in he image and is orienaion and posiion are deermined. The orienaion and posiion of he real objec deermine he orienaion and posiion of he virual objec marionee. The problem as saed has wo main pars firs, given a color image, effecivel locae he objec in i. Then, given he D projecion of he objec on he image plane, deermine i s D orienaion and posiion.

4 . SOLUTION AND ALGORITHMS.. General One of he requiremens of he projec was ha he camera be a simple web camera. We used Veo s Veloci Connec web camera, which is a pre sandard web camera. The real objec we used is a planar cross, having colored balls a is edges. The colors of he balls are (in a clockwise order): blue, red, green, red. The dimensions of he cross are approimael 8cm 8cm. The cross iself is whie, and he boh he background and he hand holding he cross should be whie. A whie wall and a whie glove work well... Image segmenaion and locaing he objec We locae he objec in he image using he four colored balls a is edges. To effecivel segmen he image using he color informaion, we conver he inpu RGB image o HSV forma (see appendi). This forma is much more suiable for color analsis, because we have a separae channel (hue) which denoes he color, wheher in RGB he color informaion is a D space. HSV also separaes he color informaion (hue) from is puri (sauraion) and brighness, allowing he segmenaion process o be more invarian o lighing condiions and hus more robus. Once we have an HSV image, we perform a conneced componens analsis on i. Each ball color (red, green, blue) is given i s own hreshold range for he Hue channel corresponding o he hue range of he color. We wan colorful piels, so he hreshold range for he Sauraion channel is he upper

5 par of he range (i.e. 8-55). Finall, we consider piels of all Brighness values. We go over all he image piels and classif hem as one of he balls (if he fall inside he hreshold) or as background. This gives us a map an image having possible values. On ha image we run a conneced componens algorihm, which for each ball gives us a lis of all found conneced componens sored b heir size in a descending order and having heir cenroid info aached. The green/blue balls are locaed a he cenroid of he larges green/blue componen. The wo red balls are locaed a he cenroids of he wo larges red componens. We now have four coordinaes, bu we sill have o disambiguae he wo red balls. We consruc hree vecors: Green-Blue, Green-Red, Green-Red and perform a cross produc beween he firs vecor and he oher wo. The signs of he cross producs allow us o disambiguae he firs red ball from he second. Thus, we have obained he four ball coordinaes in he image... Orienaion and ranslaion reconsrucion The ask here is o recover he roaion mari and he ranslaion vecor given he correspondence poins beween he image plane and he objec plane (noe ha he objec is necessaril planar, so he coordinaes in he objec space are also D). See [Brucksein 99] for more deails. We used a weak perspecive projecion model, which assumes ha he objec is viewed from sufficienl large disance. In his case, he perspecive equaions:

6 X f Y ' ' ' i ' i i = ' i = ' Zi Zi f become more simple: = α ' ' i X i = α ' ' i Y i for some posiive consan α ( α = f /, where is he displacemen of he objec). These simplified equaions allow a closed form algebraic soluion presened in [Brucksein 99], secion.. One big advanage of his mehod is ha he recover of roaion when he daa is nois is he same as in noise-free case. This is a consequence of having enough informaion o deermine he unknowns. The recovered roaion mari will alwas be orhogonal. The equaions give wo soluions for he equaion, as is generall he case wih orhographic and weak perspecive projecion (duali). We solve he duali problem using he following procedure. The objec we are observing is a planar cross. The inersecion of is wo aes in image space gives us he cener poin of he cross. Under perspecive projecion, he observed cener of he ais is no he real cener. Having he knowledge of boh real cener and he observed one can help us o disambiguae cross pose. 5

7 Observed cener Real cener Real cener Observed cener As can be seen in he diagram, in he firs case, he observed cener is closer o he blue ball han he real cener, which means ha he blue ball is closer o he viewer. In he second case, he opposie siuaion is shown. We calculae he disance beween he real and he observed ceners for boh aes. If his difference eceeds he hreshold, we use he ais having he bigger difference for disambiguaion. 6

8 . SOFTWARE ARCHITECTURE.. General This paragraph deals wih he ssem implemened, is specific modules and daa flows beween hem... Sofware Libraries Used Several sofware libraries were used:... Microsof's VisionSDK This sofware package is used o inerac wih he web camera or AVI file for acquiring images.... Carnegie Mellon Universi Vision Librar (CMVision) Carnegie Mellon Universi Auhor: James R. Bruce This sofware package was heavil revised b us o suppor he HSV color model. I is used for segmenaion and conneced componen deecion in he process of locaing objec in he image... Sofware Modules Main modules of our program... Class CWacher This class is responsible for analzing he image from he web camera/avi file o produce he four image coordinaes, which are inpu for he reconsrucion process. The heoreical grounds for his class' operaion are laid in secion.. 7

9 The class procedure is as follows:... A given inpu image in he RGB color model is convered o he HSV color model. There are hree modes for he conversion:... - Real ime conversion (CPU consuming)... - Full lookup conversion (memor consuming)... - Quanized lookup conversion (CPU and memor conserving, some of he precision is los when quanizing 56 levels ino 6)... CMVision librar (see secion..) is used wih proper hreshold values o remove noise and isolae desired hues (balls' hues).... CMVision librar is used o find conneced componens on image afer hreshold (segmenaion).... Four coordinaes are eraced from he conneced componens (one for each hue). These coordinaes are he coordinaes used as inpu for he reconsrucion process... Class CRecon This class is responsible for reconsrucing a D ransformaion from four image coordinaes. This class uses MTL (see secion..) for mari algebraic. Class' iniializaion is done according o a proper focal lengh value (web camera's parameer) and four coordinaes which describe he planar objec. Afer each image acquisiion and analsis, he four image coordinaes, which are eraced using CWacher (see secion..) are sen o his class for 8

10 reconsrucion. As a resul a orienaion mari is reurned as oupu of his process, his mari is he ranslaion and roaion of he D objec according o he given image coordinaes in relaion o he four coordinaes received in he iniializaion which describe he planar objec. The heoreical grounds for his class' operaion are laid in secion..... MarioLibDemo Program A clien program for MarioLib.DLL. This program uses MarioLib.DLL o rerieve he orienaion mari and displas a D planar cross wih he orienaion mari applied. For displa he program uses Microsof's DirecX 9.. 9

11 .. Flow : VisionSDK : CWacher : CMVision : CRecon : MTL : MarioLibDemo AVI file/webcam feed RGB o HSV Hue segmenaion Conneced componens Four image coordinaes Solve linear equaions orienaion mari

12 . EXPERIMENTS.. General This projec had undergone man changes ill he las version has been finalized. We have ried several algorihms in order o srive for he soluion, and read man aricles ring o find he righ wa o handle he problem. Our eperimens can be divided ino wo major opics. A firs, we ried o ge good resuls for he vision par of he projec. On he second par, we concenraed on ring o find he righ wa o reconsruc he D orienaion from he D image... Vision A he beginning of he projec we hough of using a simple RGB analsis. The idea was o use balls colored in red, green, or blue, each placed a a differen edge of a whie cross, wih a whie background. I urned ou ha he RGB forma is no so simple in erms of color space and is ver sensiive o lighing condiions. Therefore we go differen resuls ever ime we esed i. Looking for a more sable forma, we ried o use he YUV color forma. The image vision par of he code was implemened using CMVision librar. We go much beer resuls bu sill i was no sufficien. I seemed ha since he web camera was of ver low quali and lighing condiions weren sable enough, he analsis should have been ver clever o idenif he colors (a imes, even we could no idenif he colors when looking a he camera s image).

13 In order o be oall insensiive o ligh changes, we ried o use LEDs insead of he balls. We pu LEDs on he edges of he cross, each wih a unique color. We creaed complee darkness in he room and hen acivaed he web camera. The image analsis was sill using YUV. The idea was ha we can also creae complee darkness in an oher room. This wa, he colors ha he camera should idenif will be equal in ever room and i will be simpler o analze hem. There were wo problems wih his idea. The firs was ha he LEDs ligh was differen in is inensi and sensiive o elecrici condiions (baer power, connecions resisance, ec.). The second problem was ha here was a blur in he image aken b he web camera when he cross was in moion. We could overcome he firs problem, bu since we did no find an wa o overcome second problem, he idea was dropped. We urned back o he balls and ried o make i more sable. We replaced he cross and he balls o more vivid and less shining colors. Then we used he HSV image forma insead of YUV because i is less sensiive o ligh changes. When using he HSV we deermined he hreshold according o colors of he balls raher han he inensi. In order o use he HSV forma we rewroe he CMVision librar o work wih HSV space. We used Phooshop o ge he hue ranges of each color and made some ess using Malab o es he new vision algorihm. When we saw ha he resuls were pre good we implemened i in C++ and add some more minor color unings o ge he bes resuls... Reconsrucion A he beginning of he projec we hough ha i would no be oo comple o solve he problem of reconsruc a D objec from a D image using he fac he we know he objec dimensions. We ried o solve he problem using naïve mehods wih our informaion abou he cross objec. We know ha we

14 need o find D poins and he daa we have is: D poins from he camera, he fac ha all D poins are on he same plane, we know he real disance beween an poins, and we know ha he poins creae orhogonal lines. When we used all his informaion we go some comple non-linear equaions ha we could no solve even wih he help of some professional programs (MATLAB) Then we ried o define he Euler angles as he unknowns. We had problems of non-uniqueness of he decomposiion o he hree angles, singulariies ec.. Reading a lo of maerial on he subjec we came o an undersanding ha Euler angles are good for specifing small incremenal roaion, bu are ver problemaic specifing orienaion. We also considered a soluion using arificial inelligence, which would r o guess he movemen of he cross according o is former locaion. B calculaing a ree of all he possibiliies, he algorihm would pick he closes sae, which is similar o in is D dimensions o he image we ve go from he web camera. Afer some shor aemps we did no hink i was pracical. We also hough abou a soluion based on equaions using Quaernion represenaions. The advanage of he quaernion represenaion is ha i direcl displas he inrinsic geomeric properies of he roaion--is ais and angle--and moreover has all he algebraic informaion we need o compue anhing we need o compue abou he roaion. We read man aricles ring o see if anone had alread deal wih a problem like ours and o figure ou wha was wrong wih our former soluions. Evenuall we found [Horn ] aricle, which solved a ver similar problem. We ook he equaions from his aricle and in order o es heir correcness for he mari reconsrucion of he roaion, we creaed a projec in malab ha ess i mahemaicall and visuall.

15 A descripion of he malab projec is described in [APPENDIX B: MATLAB implemenaion of he reconsrucion]. When we saw ha he code solves he problem, we implemened i in C++. The descripion is in [APPENDIX C: Old algorihm for Orienaion and ranslaion reconsrucion]. Afer implemening he code in C++ and esing i wih our applicaion we noiced ha he reconsrucion isn working as epeced in some cases. Afer invesigaion we discovered ha he roaion mari ha is reconsruced isn orhogonal, which gives some unusual effecs when applied o he objec. We began looking ino algorihms for making he obained roaion mari orhogonal. There are man mehods of doing his, man of which involve ieraive process of aking he iniial guess and refining i unil a orhogonal mari is obained. While choosing a appropriae algorihm, we ried he simplified weak perspecive model described in [Brucksein ] and found ou ha i solves he problem nicel in our case. We esed he proposed algorihms in MATLAB as earlier and hen implemened i in C++.

16 5. RESULTS AND CONCLUSIONS 5.. Resuls The objec recogniion is sable given ha he camera seings are se o accommodae he lighing condiions (gain, eposure ec.). Oherwise, srong noise is inroduced and he recogniion becomes less sable. The following are some screenshos of he applicaion we implemened. In his applicaion we can see on he righ window he camera inpu (or AVI inpu), and he balls recogniion. On he lef window we can see he reconsrucion of he cross. 5

17 6

18 5.. Problems and enhancemens If a low quali WebCam is used, here can be some minor problems in he recogniion of he balls. Because of he low quali of he web camera, srong green noise is presen and someimes i confuses he vision algorihm and herefore he recogniion of he green ball is no sable. Changing he camera gain or using a beer Web camera usuall solves his problem. We, a firs, used a Logiech s QuickCam web camera, and eperienced srong green noise, which made he vision algorihm difficulies recognizing he green ball, laer we moved o a beer web camera: Veo s Veloci Connec, and he recogniion improved dramaicall. 7

19 6. REFERENCES. Alfred Brucksein, Rober Hol, Thomas Huang, Arun Neravali Opimum Fiducials under Weak Perspecive Projecion Inernaional Journal of Compuer Vision, 999. B.K.P. Horn, Projecive Geomer Considered Harmful, 999. J.D.Fole, A.Van-Dam, S.K.Feiner and J.F.Hughes, Compuer Graphics - Principles and Pracice. B. K. P. Horn, Robo Vision, A. K. Jain, Fundamenals of Digial Image Processing, James Bruce, Tucker Balch, Manuela Veloso, Fas and Inepensive Color Image Segmenaion for Ineracive Robos, School of Compuer Science Carnegie Mellon Universi 7. James Bruce, Realime Machine Vision Percepion and Predicion, 8

20 8. APPENDIX A: THE HSV COLOR MODEL The HSV (Hue, Sauraion, Value) color model, a cone, is shown in he figure. This is one of he percepual color spaces and was designed o mimic he wa humans perceive color. The HSV color cone defines a color b hue, sauraion, and value (brighness). The value or brighness of he color varies from zero o one along he ais, and he sauraion of he color varies as he radial disance from he cener ais. The hue is represened as an angle, wih H = degrees being red. 9

21 9. APPENDIX B: MATLAB IMPLEMENTATION OF THE RECONSTRUCTION The main funcion (msim_ep) ges he parameers of he ransformaion which are:. Name - an sring. Ro angle Z. Ro angle X. Ro angle Y 5. Displacemen X 6. Displacemen Y 7. Displacemen Z The procedure: We creaed a D cross o illusrae our real cross. This cross iniial locaion look:

22 . From properies above we build he X mari ha defines he ransformaion.. We le malab calculae he new posiion of he corners of he cross (using he original poins and he X mari).. We ake he resul ( new poins of he new D image) and reconsruc he movemen wih our calculaions (as described earlier in he documen). We build anoher X mari wih our algorihm and compare i wih he original one.. To check our resuls visuall, we le malab calculae he new posiion of he cross wih our new mari and we see if he wo images (of wo ransformaions, real and reconsruced) seem similar. Resuls: We ried he simulaion on several roaions and displacemens and he reconsruced mari was ver similar o he original one and in some cases he marices were equal. The following are some eamples of he ransformed cross. The firs image is he real ransformed cross and he second is he reconsruced ransformaion.

23 msim_ep('firs_es', 5, 5,,,, );

24 msim_ep('firs_es', 9,, 5,,, );

25 . APPENDIX C: OLD ALGORITHM FOR ORIENTATION AND TRANSLATION RECONSTRUCTION The ask here is o recover he roaion mari and he ranslaion vecor given he correspondence poins beween he image plane and he objec plane (noe ha he objec is necessaril planar, so he coordinaes in he objec space are also D). See [Horn ] for more deails. In order o achieve his, we firs have o find he mari T, which represens a homogeneous ransformaion from he objec plane o he image plane. T muliplied b a -vecor (,, ) T represening posiion in he objec plane ields a -vecor (ku, kv, k) T ha represens he corresponding posiion in he image plane boh in homogeneous coordinaes: = k kv ku Given he scale facor ambigui, we can arbiraril pick = and choose he oher eigh elemens of T independenl. In order o recover T we solve a linear ssem of 8 equaions: = v v v v u u u u v v v v v v v v u u u u u u u u

26 where (,) are he coordinaes in he objec plane and (u,v) are he coordinaes in he image plane. To recover he roaion and ranslaion info from T, we use he concep of vanishing poins. Recover of roaion: he vanishing poin for he -ais is jus (,, ) T in he objec coordinae ssem. Mulipling he mari T b his vecor ields he homogeneous image coordinae (,, ) T. Similarl, we ge (,, ) T from (,, ) T for he -ais. These wo correspond o image coordinaes (/, /) T and (/, /) T respecivel. If we connec he cener of projecion o hese poins in he image plane we obain direcion vecors parallel o: = (,, f ) T = (,, f ) T We can divide hese wo vecors b heir magniude o obain uni vecors ˆ and ŷ in he direcion of he - and -aes of he objec plane (epressed in he camera coordinae ssem). Since he z-ais perpendicular o he objec plane has o be a righ angles o an line in he objec plane, we can?nd is direcion simpl b aking he cross produc of he direcions of he - and -aes found above. A roaion mari relaing (D) objec coordinaes o (D) camera coordinaes can now be consruced b adjoining he hree uni column vecors in he direcions of he coordinae aes: R = ( ˆ, ˆ, zˆ) where above. ˆ, ˆ, zˆ are uni column vecors consruced from T, as described Recover of ranslaion: he homogeneous coordinaes of he origin in he objec plane are obviousl jus (,, ) T. Mulipling T b his vecor ields 5

27 (,, ) T. The image of he origin of he objec coordinae ssem hen is a (/, /) T. Connecing he origin o his poin in he image plane (z = f ), ields a vecor parallel o = (,, f ) So we found direcion of he ranslaional vecor o he objec origin direcl from he las column of T.We can?nd he disance o he objec origin from he cener of projecion if we can deermine he magni?caion of a line parallel o he image plane a ha disance (ha is, he raio of he lengh of he line in he image o he lengh of he line on he objec). If he magni?caion is M (picall less han one), hen he z-componen of he ranslaion vecor mus be f/m. We can use his value o scale he direcion vecor found above. The linear magnificaion facor M can be compued direcl b aking he square roo of he deerminan of mari T. Once we know he magni?caion M we can deermine he ranslaional offse of he objec origin from he camera origin b mulipling = (,, f ) T b M/(f ). We have recovered he roaion mari and he ranslaion vecor corresponding o he objec posiion and orienaion in D space, and can now combine he wo o obain he mari (using homogenous coordinaes) encapsulaing boh he roaion and he ranslaion. 6