MESURING COMPLETE GROUND-TRUTH DT ND ERROR ESTIMTES FOR REL VIDEO SEQUENCES, FOR PERFORMNCE EVLUTION OF TRCKING, CMER POSE ND MOTION ESTIMTION LGORITHMS R Solkin a *, Greig b, J Gilby c a Cener for Mariime Sysems, Sevens Insiue of Technology, Hoboken, NJ 73 US RSolkin@sevensedu b Dep of Mechanical Engineering, Universiy College London, WCE 6T UK a_greig@menguclacuk c Sira Ld, Souh Hill, Ken R7 5EH, UK johngilby@siracouk KEY WORDS: Vision, roboics, racking, navigaion, regisraion, calibraion, performance, accuracy STRCT: Fundamenal asks in compuer vision include deermining he posiion, orienaion and rajecory of a moving camera relaive o an observed objec or scene Many such visual racking algorihms have been proposed in he compuer vision, arificial inelligence and roboics lieraure over he pas 3 years Predominanly, hese remain un-validaed since he ground-ruh camera posiions and orienaions a each frame in a video sequence are no available for comparison wih he oupus of he proposed vision sysems mehod is presened for generaing real visual es daa wih complee underlying ground-ruh The mehod enables he producion of long video sequences, filmed along complicaed six degree of freedom rajecories, feauring a variey of objecs, in a variey of differen visibiliy condiions, for which complee ground-ruh daa is known including he camera posiion and orienaion a every image frame, inrinsic camera calibraion daa, a lens disorion model and models of he viewed objecs We also presen a means of esimaing he errors in he ground-ruh daa and plo hese errors for various experimens wih synheic daa Real video sequences and associaed ground-ruh daa will be made available o he public as par of a web based library of daa ses INTRODUCTION n imporan and prolific area of compuer vision research is he developmen of visual racking and pose esimaion algorihms Typically hese fi a model o feaures exraced from an observed image of an objec o recover camera pose, rack he posiion and orienaion of a moving camera relaive o an observed objec or rack he rajecory of a moving objec relaive o a camera Clearly, proper validaion of such algorihms necessiaes es images and video sequences wih known ground-ruh daa, including camera posiions and orienaions relaive o he observed scene a each frame, which can be compared o he oupus of proposed algorihms in order o compue errors Surprisingly, very few such daa ses or mehodologies for creaing hem are discussed in he lieraure, wih repored vision sysems ofen validaed in ad hoc ways Many papers aemp o demonsrae he accuracy of racking algorihms by superimposing, over he observed image, a projecion of he racked objec based on he posiions and orienaions oupu by he algorihm In fac i can be shown (Solkin 4) ha even very close D visual maches of his kind can resul from significanly erroneous 3D racked posiions One reason for his is ha cerain combinaions of small roaions and ranslaions, eiher of cameras or observed objecs in 3D space, ofen make lile difference o he resuling D images This is especially rue for objecs wih limied feaures and simple geomery Such errors can only be properly idenified and quanified by means of es images wih accompanying complee 3D ground-ruh I is relaively simple o consruc arificial image sequences, wih pre-programmed ground-ruh, using commonly available graphics sofware (eg POV-Ray for windows) and his is also common in he lieraure However, alhough esing compuer vision algorihms on synheic scenes allows comparison of performance, i gives only a limied idea of how he algorihms will perform on real scenes Real cameras and real visibiliy condiions resul in many kinds of noise and image degradaion, far more complicaed han Gaussian noise or sal and pepper speckling and i is no rivial or obvious how o realisically synhesise real world noise in an arificial image (Rokia, 997; Kaneda, 99) This becomes even more difficul when he scene is no viewed hrough clear air bu hrough mis, smoke or urbid waer rificial scenes do no compleely reproduce he deailed variaion of objecs, he muliude of complex lighing condiions and modes of image degradaion encounered in he real world Vision and image processing algorihms ofen seem o perform much beer on arificial (or arificially degraded) images han on real images The only rue es of compuer vision algorihms remains heir performance on real daa To his end, several researchers have aemped o combine real image daa wih some knowledge of ground-ruh Oe, 994, describes he use of a robo arm o ranslae a camera a known speeds, generaing real image sequences for he assessmen of opical flow algorihms The measured ground-ruh daa is limied o known opic flow fields raher han explici camera posiions and he camera is only ranslaed Roaional camera moion is no addressed McCane,, also describes image sequences wih known ground-ruh moion fields The work is limied o simple D scenes conaining planar polyhedral objecs agains a fla background The echnique involves laborious hand-labelling of feaures in each image and so only very shor sequences are usable Wunsch, 996, uses a robo arm o posiion a camera in known poses relaive o an observed objec Similarly, Sim, 999, generaes individual images from known camera posiions using a camera mouned on a ganry
robo In he work of boh Wunsch and Sim, ground-ruh posiions are only measured for individual sill images as opposed o video sequences oh auhors appear o obain camera posiions from he robo conroller I is no clear if or how he posiions of he camera (opical cenre) were measured relaive o he robo end-effecor gapio,, generaes ground-ruh image sequences using heir Yorick sereo head/eye plaform The work is limied o providing roaional moion wih only wo degrees of freedom lhough daa for angles of elevaion and pan can be exraced from he moor encoders of he plaform, hese are no in relaionship o a paricular observed objec The ranslaional posiion of he camera remains unknown Maimone, 996, discusses various approaches for quanifying he performance of sereo vision algorihms, including he use of boh synheic images and real images wih various kinds of known ground-ruh Maimone does menion he use of an image of a calibraion arge o derive ground-ruh for a corresponding image of a visually ineresing scene, filmed from an idenical camera posiion However, he echniques are limied o he acquisiion of individual, sill images from fixed camera posiions The addiional problems, of generaing ground-ruh for exended video sequences, filmed from a moving camera, are no addressed In conras, our mehod enables he producion of long video sequences, filmed along a six degree of freedom rajecory, feauring a variey of objecs, in a variey of differen visibiliy condiions, for which complee ground-ruh daa is known including he camera posiion and orienaion a every image frame, inrinsic camera calibraion daa, a lens disorion model and models of he viewed objecs METHOD pparaus and procedure n indusrial robo arm (six degree of freedom Unimaion PUM 56) is used o move a digial cam-corder (JVC GR- DV) along a highly repeaable rajecory Tes sequences, (feauring various objecs of ineres in various differen visibiliy and lighing condiions), and calibraion sequences (feauring planar calibraion arges in good visibiliy) are filmed along idenical rajecories (figures, ) Figure Tes sequence -camera views a model oil-rig objec in poor visibiliy Figure Calibraion sequence -camera views calibraion arges in good visibiliy complee camera model, lens disorion model, and camera posiion and orienaion can be exraced from he calibraion sequence for every frame, by making use of he relaionship beween known world co-ordinaes and measured image coordinaes of calibraion feaures This informaion is used o provide ground-ruh for chronologically corresponding frames in he visually ineresing es sequences Objecs o be observed are measured, modeled and locaed precisely in he co-ordinae sysem of one of he calibraion arges For hose researchers ineresed in vision in poor visibiliy condiions (eg Solkin ) dry ice fog can be used during he es sequences (figure ) in addiion o various lighing condiions (eg fixed lighing or spo-lighs mouned on and moving wih he camera) Noe, i is no feasible o exrac camera posiions from he robo conrol sysem since he posiion of he camera relaive o he erminal link of he robo remains unknown; indusrial robos, while highly repeaable, are no accurae; chronologically maching a series of robo posiions o a series of images may be problemaic Synchronisaion The calibraion and es sequences are synchronised by beginning each camera moion wih a view of an exra synchronisaion spo feaure (a whie circular spo on black background) frame from each sequence is found such ha he synchronisaion spo maches well when he wo frames are superimposed Thus he n h frame from he maching frame in he es sequence is aken o have he same camera posiion as ha measured for he n h frame from he maching frame in he calibraion sequence The wo sequences can only be synchronised o he neares image frame (ie a wors case error of ± seconds a 5 frames per second) There are wo ways of minimizing his error Firsly, he camera is moved slowly so ha emporal errors resul in very small spaial errors Secondly, many examples of each sequence are filmed, increasing he probabiliy of finding a pair of sequences ha mach well (correc o he neares pixel) If en examples of each sequence are filmed, hen he expeced error is reduced by a facor of
3 Feaure exracion and labelling The calibraion arges are black planes conaining square grids of whie circular spos The planes are arranged so ha a leas one is always in view and so ha hey are no co-planar The posiions of spos in images are deermined by deecing he spos as blobs and hen compuing he blob cenroid small number (a leas 4) of spos in each of a few images scaered hrough he video sequence are hen hand-labeled wih heir corresponding arge plane co-ordinaes The remaining spos in all images are labeled by an auomaed process The iniial four labels are used o esimae he homography mapping beween he arge plane and he image plane This homography is hen used o projec all possible arge spos ino he image plane ny deeced spos in he image are hen assigned he labels of he closes maching projeced spos Spos in chronologically adjacen images are now labeled by assigning hem he labels of he neares spos from he previous (already labeled) image These wo processes, of projecion and propagaion, are ieraed backwards and forwards over he enire image sequence unil no new spo labels are found 4 Camera calibraion and posiion measuremen Our calibraion mehod is adaped from ha of Zhang, 998, which describes how o calibrae a camera using a few images of a planar calibraion arge Relaed calibraion work includes Tsai, 987 The following is a condensed summary of our implemenaion of hese ideas 4 Homography beween an image and a calibraion arge: Since he calibraion arges are planar, he mapping beween he (homogeneous) arge co-ordinaes of calibraion feaures, [ Y ] T, and heir corresponding x i u v form a homography, expressible as a 3 3 marix: (homogeneous) image co-ordinaes, [ ] T [ h h h ] xi H 3, mus () Thus each calibraion feaure, whose posiion in an image is known and whose corresponding arge co-ordinaes have been idenified, provides wo consrains on he homography large number of such feaure correspondences provides a large number of simulaneous equaions: wu wv w w u w v w wnun n w nvn H Y Y Y () n w n leas squares fi homography is hen found using singular value decomposiion 4 Consrains on he camera calibraion parameers: The mapping beween he arge and image planes mus also be defined by he inrinsic and exrinsic camera calibraion parameers of he camera: x H CE i where C is he inrinsic or calibraion marix : fk C u fk v u v (3) (f is focal lengh, k u and k v are pixels per uni lengh in he u and v direcions, (u, v ) are he co-ordinaes of he principal poin, pixel array assumed o be square) and E is he exrinsics marix defining he posiion and orienaion of he camera (relaive o he arge co-ordinae sysem), ie r r T [ ] E, where r and T denoe roaion and ranslaion vecors Noe ha only wo roaion vecors (no hree) are needed since he calibraion arge plane is defined o lie a Z in he arge co-ordinae sysem Hence: [ h h h ] C[ r r T] H 3 Since he column vecors of a roaion marix are always muually orhonormal, we have: r T r (5) r Since n r T T r r r C h h n hese become: T T C C h T T T T and h C C h h C C h Thus one homography provides wo consrains on he inrinsic parameers Ideally, many homographies (from muliple images of calibraion arges) are used and a leas squares fi soluion for he inrinsic parameers is found using singular value decomposiion Once he inrinsic parameers have been found using a few differen views of a calibraion arge, he exrinsic parameers can be exraced from any oher single homography, ie he camera posiion and orienaion can be exraced for any single image frame provided ha i feaures several spos from a leas one arge 43 Locaing arges relaive o each oher: We use muliple calibraion arges o ensure ha a leas one arge is always in view during complicaed (six degree-of-freedom) camera rajecories Provided ha a leas one arge is visible o he camera a each frame, he posiion of he camera can be compued by choosing one arge o hold he world co-ordinae sysem and knowing he ransformaions which relae his arge o he ohers The relaionship beween any wo arges is deermined from images which feaure boh arges ogeher, by deermining he homography which maps beween he coordinae sysems of each arge For wo arges, and : i (4) (6) (7) (8) x H H (9) where and are he posiions of a single poin in he respecive co-ordinae sysem of each arge Thus: ( H ) xi ( H ) H () 44 Modeling lens disorion: Lens disorion is modelled as a radial shif of he undisored pixel locaion (u, v) o he disored pixel locaion ( uˆ, vˆ ), such ha: 4 u ˆ u + u u k r + k r () ( )( ) 4 and v ˆ v + ( v v )( k r + k r ) () where ( ) ( ) r u u + v v
45 Refining parameer measuremens wih non-linear opimizaion: In pracice, all imporan parameer measuremens (camera inrinsics, lens disorion, arge o arge ransformaions, camera posiions), which are iniially exraced using he geomerical and analyical principles oulined above, can be furher improved using non-linear opimisaion n error funcion is minimised, consising of he sum of he squared disances (in pixels) beween he observed image locaions of calibraion feaures and he locaions prediced given he curren esimae of he parameers being refined This resuls in a maximum likelihood esimae for all parameers Firsly a small se (abou ) of images are used o compue camera inrinsic parameers, lens disorion parameers, camera posiion and orienaion for each image (of he small se) and he ransformaions beween he co-ordinae sysems of each arge These parameers are hen muually refined over all views of all arges presen in all images of he se, by minimising he following error funcion: n m images arge spo s ( ) C, k, k, R, T x xˆ (3) images, arges image s Where, for m poins (spo cenres) exraced from n arge views, x is he observed image in pixelaed camera coordinaes of he world co-ordinae arge poin arge s, and xˆ is he expeced image of ha poin given he curren image s esimaes of he camera parameers ( C, k, k, R, T ) Noe ha he values of he co-ordinaes of are also dependen arge s on he curren esimaes of arge-o-arge ransformaions and hese ransformaions are also being ieraively refined Secondly, using he refined values for inrinsics, lens disorion parameers and arge-o-arge ransformaions, he camera posiion and orienaion is compued for a single image aken from he middle of he calibraion sequence, again using analyical and geomerical principles Keeping all oher parameers consan, he six-degrees of freedom of his camera locaion are now non-linearly opimized, minimizing he error beween he observed calibraion feaure locaions and hose prediced given he curren esimae of he camera locaion and he fixed values (previously refined) of all oher parameers Lasly, he camera posiion for he above single image is used as an iniial esimae for he camera posiions in chronologically adjacen images (previous and subsequen images) in he video sequence These posiions are hen hemselves opimized, he refined camera posiions hen being propagaed as iniial esimaes for successive frames, and so on hroughou he enire video sequence, resuling in opimized camera posiions for every image frame along he enire camera rajecory 3 Consruced daa ses 3 RESULTS We have filmed video sequences of around frames (a 5 frames per second) along a complicaed six degree-of-freedom camera rajecory Figure 3 shows he camera posiion a each frame, as calculaed from he calibraion sequence The rajecory is illusraed in relaion o he spos of he hree calibraion arges (3mm spacing beween spos) Figure 3 The compued rajecory for a six-degree of freedom of moion video sequence The sequences feaure various differen known (measured and modelled) objecs (figure 4) in various differen visibiliy and lighing condiions as well as a corresponding calibraion sequence nalysis of he calibraion sequence has yielded a complee camera model, lens disorion model and a camera posiion and orienaion for every frame in each of hese sequences Figure 4 Two of he objecs filmed in he video sequences, block and model oil-rig 3 Smoohness of rajecory One indicaor of accuracy is he smoohness of he measured rajecory Figure 3 is a useful visual represenaion of he rajecory and figures 5 and 6 are plos of he ranslaional and roaional camera co-ordinaes a each frame Poins,, C, D are corresponding way mark poins beween figures 3, 5 and 6 For abou he firs 4 frames, he camera is saionary a poin I will be noiced ha small secions of he rajecory appear somewha broken and erraic, approximaely frames 4 6 and 88 9 These ranges correspond o he beginning and end of he rajecory during which he camera is moved from (and back owards) a posiion fixaed on he synchronizaion spo (see secion ) a poin During hese periods, comparaively few calibraion feaures are in he field of view These secions of he video sequence do no correspond o visually ineresing porions of he image sequence and are no used for esing vision algorihms They are included only for synchronizaion The remainder of he measured rajecory is exremely smooh, implying a high degree of precision The robo is old, and is dynamic
performance less han perfec, so he disurbance jus afer moion is iniiaed (shorly afer poin ) is probably due o he ineria of he sysem Second and hird peaks of decaying magniude a exacly and 4 frames laer sugges ha hey have a mechanical origin 4 prediced image has been superimposed over he real image This helps illusrae he errors involved (in his case ± 3 pixels discrepancy in block edges) This dispariy in error magniude (compared o 6 pixels above) may be due o over-fiing of he camera model o feaures in he calibraion arge planes and under-fiing o poins ouside hose planes 3 3 6 9 3 6 9 Figure 7 n image from a sequence feauring a block objec The superimposed wire frame image corresponds o he prediced image given he measured camera co-ordinaes - Figures 5 & 6 Top graph shows ranslaional componens of camera moion along x, y and z axes Verical scale in mm oom graph shows roaional componens of camera moion abou x, y and z axes Verical scale in radians For boh graphs, he horizonal scale is image frame number 33 Robo repeaabiliy In order o assess repeaabiliy, he robo was moved along a varied, six-degree of freedom moion ha included pauses a hree differen posiions during he moion Several video sequences were filmed from he robo-mouned camera while moving in his fashion Images from differen sequences, filmed from he same pause posiions, were compared Superimposing he images reveals an error of beer han ± one pixel This implies ha errors in image repeaabiliy due o robo error approach he scale of he noise associaed wih he camera iself Our robo is approximaely weny years old Modern machines should produce even smaller errors 34 ccuracy of scene reconsrucion In order o assess accuracy, he image posiions of calibraion feaures were reconsruced by projecing heir known world coordinae posiions hrough he measured camera model placed a he measured camera posiions Comparing hese prediced image feaure posiions wih hose observed in he real calibraion sequence yielded an rms error of 6 pixels per calibraion feaure (spo) When some of he observed objecs have been reconsruced in he same way, he errors are worse Figure 7 shows an image from a sequence feauring a whie block objec The measured camera posiion for he image frame has been used o projec a prediced image (shown as a wire frame model) and his 35 ccuracy of camera pose measuremen In order o esimae he poenial overall accuracy of measured camera posiions, we have used synheic calibraion daa lhough, in general, synheic images do no reproduce he noise inheren in real images, calibraion sequences are filmed in highly conrolled condiions which are more reasonably approximaed by synheic images Graphics sofware (POV- Ray for windows) was used o generae compuer models of calibraion arges series of synheic images were hen rendered which would correspond o hose generaed by a camera viewing he arges from various posiions These images were fed ino he calibraion scheme Ground-ruh as measured by our calibraion scheme was hen compared wih he pre-programmed synheic ground-ruh in order o quanify accuracy For simpliciy, we have used a synheic camera array of by pixels-somewha beer han curren ypical real digial video resoluion bu far worse han ypical real single image resoluion Over a se of 6 images filmed from several differen ranges, bu all feauring views of hree approximaely orhogonal calibraion arges (see second paragraph of secion 4), he error in measured principal poin posiion was 76 pixels and he error in measured focal lengh was 6% The average error in measured camera posiion was 38mm and 4 degrees Translaional error / mm 5 5 5 6 4 8 Range / m Figure 8 Variaion in ranslaional camera posiion error wih range from calibraion arges
Roaional error / degrees 3 6 4 8 Range / m Figures 9 Variaion in camera orienaion error wih range from calibraion arges Figures 8 and 9 plo he variaion of error wih disance of he camera from he calibraion arge origin 4 SUGGESTED IMPROVEMENTS The problem, oulined in secion 34, of over-fiing he camera model o poins lying in he calibraion arge planes should be avoided in fuure work by using calibraion images filmed a a variey of differen ranges from he calibraion arges lhough i should be possible o deermine he posiion of a calibraed camera given a view of a single calibraion arge (Zhang, 998), in pracice various small coupled ranslaions and roaions of he camera can resul in very similar views, causing measuremen uncerainy These errors can be consrained by ensuring ha, hroughou he moion of he camera, all hree arges, posiioned approximaely orhogonally o each oher, are always in view In our original experimens wih real video sequences, only one or wo arges were viewed in mos images and so our camera posiion accuracies are worse han can be achieved Fuure researchers should ensure ha he camera can always view hree, approximaely orhogonal, calibraion arges in every image I is possible o furher auomae he labeling of calibraion spos y making a specific poin, or poins, on each arge a differen colour, i may be possible o eliminae he need o hand-label a small number of spos in each video sequence Viewing he synchronizaion spo afer he cam-era has already sared moving would eliminae he mechanical vibraion problems of he sep response noed a he sar of he robo s moion The synchronisaion problem (see secion ), ha wo sequences can only be synchronised o he neares image frame (ie wors case error of ± seconds a 5 frames per second), migh be eliminaed by riggering he camera exernally wih a signal from he robo conroller such ha video sequences sared a a specific locaion in he rajecory Noe ha es sequences can be filmed which feaure virually any kind of objec Even deforming or moving objecs could conceivably be used alhough measuring ground-ruh for he shapes and posiions of such objecs would pose addiional challenges Specifically, he use of objecs wih known exures migh benefi researchers wih an ineres in surface reconsrucion or opic flow Wih appropriae equipmen, i should also be possible o creae real underwaer sequences using our echnique 5 CONCLUSION The field of compuer vision sees he frequen publicaion of many novel algorihms, wih comparaively lile emphasis placed on heir validaion and comparison If vision researchers are o conform o he rigorous sandards of measuremen, aken for graned in oher scienific disciplines, i is imporan ha our communiy evolve mehods by which he performance of our echniques can be sysemaically evaluaed using real daa Our mehod provides an imporan ool which enables he accuracy of many proposed vision algorihms, for regisraion, racking and navigaion, o be explicily quanified REFERENCES gapio, L, Hayman, E, Reid, I, Self-Calibraion of Roaing and Zooming Cameras Inernaional Journal of Compuer Vision Vol 45(), pages 7-7 Kaneda, K, Okamoo, T, Nakamae, E, Nishia, T, 99 Phoorealisic image synhesis for oudoor scenery The Visual Compuer Vol 7, pages 47-58 Maimone, M, Shafer, S, 996 axonomy for sereo compuer vision experimens ECCV workshop on performance characerisics of vision algorihms Pages 59-79 McCane,, Novins, K, Crannich, D, Galvin,, On enchmarking Opical Flow Compuer Vision and Image Undersanding Vol 84, pages 6-43 Oe, M, Nagel, H, 994 Opical Flow esimaion: dvances and Comparisons Proc 3 rd European Conference on Compuer Vision Pages 5-6 POV-Ray for windows, hp://wwwpovrayorg Rokia, P, 997Simulaing Poor Visibiliy Condiions Using Image Processing Real-Time Imaging, 3, pages 75-8 Sim, R, Dudek, G, 999 Learning and Evaluaing Visual Feaures for Pose Esimaion Inernaional Conference on Compuer Vision Vol Solkin, R, 4 Combining observed and prediced daa for robo vision in poor visibiliy PhD hesis, Deparmen of Mechanical Engineering, Universiy College London Solkin, R, Hodges, M, Greig,, n EM/E-MRF Sraegy for Underwaer Navigaion Proc h riish Machine Vision Conference Tsai, R versaile camera calibraion echnique for highaccuracy 3D machine vision merology using off-he-shelf v camera and lenses IEEE Journal of Roboics and uomaion Vol 3(4), pages 33-344 987 Wunsch, P, Hirzinger, G, 996 Regisraion of CD-Models o Images by Ieraive Inverse Perspecive Maching Proceedings of he 3 h Inernaional Conference on Paern Recogniion Pages 77-83 Zhang, Z, 998 Flexible New Technique for Camera Calibraion Microsof Research Technical Repor, MSR-TR- 98-7