Robust Visual-Inertial State Estimation with Multiple Odometries and Efficient Mapping on an MAV with Ultra-Wide FOV Stereo Vision

Transcription

1 M. G. Müller F. Seidle M. J. Schuser P. Luz M. Maier S. Soneman T. Tomic W. Sürzl Robus Visual-Inerial Sae Esimaion wih Muliple Odomeries and Efficien Mapping on an MAV wih Ulra-Wide FOV Sereo Vision 8 IEEE/RSJ Inernaional Conference on Inelligen Robos and Sysems (IROS 8) c 8 IEEE hp://ieeexplore.ieee.org/search/searchresul.jsp?querytexrobus Visual- Inerial Sae Esimaion Muliple Odomeries Efficien Mapping MAV Ulra-Wide FOV Sereo Vision 8 Robus Visual-Inerial Sae Esimaion wih Muliple Odomeries and Efficien Mapping on an MAV wih Ulra-Wide FOV Sereo Vision M. G. Müller F. Seidle M. J. Schuser P. Luz M. Maier S. Soneman T. Tomic W. Sürzl Absrac The here presened flying sysem uses wo pairs of wide-angle sereo cameras and maps a large area of ineres in a shor amoun of ime. We presen a mulicoper sysem equipped wih wo pairs of wide-angle sereo cameras and an inerial measuremen uni (IMU) for robus visual-inerial navigaion and ime-efficien omni-direcional D mapping. The four cameras cover a 4 degree sereo field of view (FOV) verically which makes he sysem also suiable for cramped and confined environmens like caves. In our approach we synhesize eigh virual pinhole cameras from four wide-angle cameras. Each of he resuling four synhesized pinhole sereo sysems provides inpu o an independen visual odomery (VO). Subsequenly he four individual moion esimaes are fused wih daa from an IMU based on heir consisency wih he sae esimaion. We describe he configuraion and image processing of he vision sysem as well as he sensor fusion and mapping pipeline on board he MAV. We demonsrae he robusness of our muli-vo approach for visual-inerial navigaion and presen resuls of a D-mapping experimen. I. INTRODUCTION Micro Aerial Vehicles (MAVs) have been used in a vas variey of applicaions in recen years. Their abiliy o quickly reach poins of ineres or o obain perspecives which were previously difficul or impossible o reach no only makes hem ineresing for asks like exploraion inspecion and search and rescue bu also for general consumer applicaions e.g. generaing virual realiy conen by mapping environmens. Since mos MAVs do no have a very long fligh ime due o baery limiaions mapping large scenes has o be done in a very efficien manner. Therefore i is helpful if he MAV s mapping sensors have a wide field of view (FOV). A wide FOV is also beneficial for obsacle deecion and pah planning. Furhermore in semi-dynamic environmens or siuaions where pars of he scene are exure-less i is more likely o find enough reliable feaures for ego-moion esimaion wih a wide FOV han wih a narrow one. As cameras are ligh-weigh and provide a huge amoun of informaion hey are ideally suied for mobile robos ha have limied payload. Sereo cameras have been employed successfully on MAVs o obain images as well as deph informaion in boh indoor and oudoor environmens [] [] [] [4]. These facs moivaed us o build an MAV Ardea ha suppors wide-angle sereo vision shown in Fig.. Ardea is equipped wih four wide-angle cameras arranged in wo sereo configuraions. The oal verical FOV of Insiue of Roboics and Mecharonics German Aerospace Cener (DLR) Germany Corresponding auhor: marcus.mueller@dlr.de Curren affiliaion: Skydio Inc. Unied Saes Fig. : In-house buil Y6-hexacoper Ardea on M. Ena. The field of view is illusraed by he blue overlay. (size: 68 cm 68 cm cm weigh:.65 kg including baery). 4 enables Ardea o efficienly and effecively map he environmen. A fully spherical view of a scene including deph informaion can easily be obained by roaion abou he verical axis (yaw-roaion). Each wide-angle camera image is remapped o wo images wih pinhole projecion resuling in four synhesized pinhole sereo sysems ha provide inpu o independen visual odomeries (VOs). In [5] he visual feaures from muliple cameras are racked in a join opimizaion leading o a igh coupling. In conras we process our sereo pairs decoupled form each oher (similar o [6]) and fuse heir visual odomery resuls in a real-ime capable filer for local sae esimaion. Thereby he complexiy of our global graph-based esimaion is no affeced by he number of high-frequency sensors allowing for fas online opimizaion seps [7]. The approach described in [8] uses wo visual odomeries where only one is seleced o be fused wih an inerial measuremen uni (IMU). In our work all four VO pose esimaes are fused wih he daa of an IMU wih a single filer. Running four independen visual pose esimaions provides addiional redundancy o he sysem which can be criical in he case of complex scenes where i is likely ha one of he VOs will reurn a poor resul or even enirely fail. Our main conribuions are herefore: sensor fusion of muliple VOs wih independen keyframe selecion for improved navigaion wih respec o accuracy robusness and redundancy descripion of a wide-angle muli-camera seup for efficien environmen mapping on an MAV deailed descripion of a muli-fisheye camera calibraion and pinhole remapping for compuaional efficiency

2 The paper is organized as follows: We begin by briefly inroducing our MAV and is hardware in he nex secion. Then in secion III we presen he camera seup and image processing of he MAV vision sysem in deail. In secion IV we describe our muli-vo approach and he sensor fusion wih he IMU. Afer summarizing our mapping framework in secion V we presen resuls demonsraing he effeciveness of our approach (secion VI). II. G ENERAL H ARDWARE S ETUP We chose a rigonal Y6 frame consrucion for our MAV o be able o place he cameras in a way ha hey have a high coverage wihou having any pars from he MAV in heir FOV. This seup also enables he MAV o fly in more confined areas such as indoors or in caves. The MAV is propelled by hree coaxial roor pairs providing a maximum hrus of.6 kg. I consiss of wo main separable pars: Frame: includes all engines speed-conrollers and he ouer carbon frame ubes. Sack: consiss of all navigaion sensors and on-board compuers. The sack can also be operaed by iself which makes i easier o develop and es new hardware and sofware before incorporaing i wih he frame. The on-board compuing hardware consiss of an Inel NUC wih a dual core i7 ( GHz) an FPGA (Xilinx Sparan6) for SGMbased sereo processing [9] [] and a BeagleBoneBlack embedded compuer which runs an aiude and posiion conroller a 5 Hz. A wide-angle muli-camera sysem described in deail in secion III and an IMU (Analog Devices ADIS647) are used as on-board sensors. below and above he horizon longer exposure and higher gains can be used for he lower FOV o cope wih he differen inensiies. The camera sysem consiss of four synchronized cameras each providing 8 86 px. For achieving a large FOV and reasonable resoluion in he verical axis he smaller image dimension (86 px) of he camera sensors are horizonally aligned. The cameras are arranged in wo sereo sysems wih he opical axes of he lower cameras a a 6 angle wih respec o he horizon and hose of he upper cameras a +6 (see Fig. ). The FOV of each camera is abou 8 horizonally and abou 5 verically. A. Calibraion of Wide-Angle Cameras wih Modified Kannala-Brand Model For cameras wih very large FOVs he sandard pinhole camera model is no suiable since for insance he z coordinae of an objec poin can approach zero or can even become negaive for FOV > 8. Therefore we used a modified version of he Kannala-Brand model [] wih parameers (k k k k4 p p mu uc mv vc ). The radial projecion of he lens is modeled by a polynomial θ f (θ) θ + k θ + k θ5 + k θ7 + k4 θ9 () where θ arccos( z ) x +y +z opical axis and a D poin wih coordinaes (x y z) in he camera frame. The mapping ono he camera plane (u(x y z) v(x y z)) is given by x u θ cos φ θ p () x + y y v θ sin φ θ p () x + y u u + p u v + p (u + v ) (4) v v + p v u + p (v + u ) u mu u + uc 4 Fig. : Top and side view illusraion of he Ardea mulicoper. Bluish areas indicae he horizonal and verical FOV of he muli-camera sysem. III. M ULTI -C AMERA S ETUP In he following we describe he wide-angle muli-sereo camera sysem of our mulicoper Ardea ha provides approx. 4 verical field of view as illusraed in he righ side of Fig.. In addiion o he large FOV he arrangemen of cameras is well suied for he high dynamic range siuaion in oudoor scenes wih ofen much higher brighness above he horizon han below. As separae cameras cover he FOV is he angle beween he v mv v + vc. (5) (6) In conras o he pinhole model he Kannala-Brand model does no discriminae beween radial projecion and (symmeric) radial disorion. Eqs. (4) and (5) describe he angenial disorion model due o lens misalignmen (decenering disorion) moivaed by he angenial disorion model of pinhole lenses. As in he pinhole projecion model he angenial disorion is a funcion of he undisored projecion coordinaes u v and depends on jus addiional parameers p and p. In conras he generic disorion model proposed in [] inroduces 4 addiional parameers. B. Epipolar Geomery of he Muli-Camera Sysem As he four wide-angle cameras are arranged in wo sereo configuraions he epipolar geomery of each sereo pair has o be compued []. From exrinsic camera calibraion we obain he ransformaion i T from he reference camera o all oher camera frames i 4 i.e. i i R x i T x i R x + i. (7) is he roaion of he reference frame wih respec o camera frame i and i is he vecor o he origin of in

3 coordinaes of frame i. All epipolar planes inersec he line connecing he nodal poins of boh cameras. If as i is he case for he lower sereo sysem on our mulicoper reference camera is one of he cameras of he sereo sysems hen he vecor poining from he reference camera o he oher camera (i on our mulicoper) of he sereo pair is simply R >. vcaml vcaml cam (8) defines he firs axis (he x-axis) of he sereo coordinae sysem in he coordinae sysem of reference camera ex k k R > ez mz k mz k k. (9) ( e> x mz ) ex > ( ex mz ) ex k () where mz ez + ez ez + R > ez is he sum of he z-axis vecors of boh camera frames in reference frame i.e. ez ez ( )>. The sereo y-axis can be calculaed using he cross produc of ex and ez ey ez ex. () The roaion marix s R ( ex ey ez ) describes he roaion of he sereo sysem wih respec o he reference camera frame. For oher sereo sysems (ha do no conain he reference camera ) we firs have o compue he ransformaion beween cameras. In our seup he second sereo sysem consiss of camera and 4. The ransformaion from 4 o camera he reference camera of his sereo sysem is he ransformaion from 4 o followed by ransformaion from o x 4 4 T x T (4 T ) 4 x. () As for he firs sereo sysem (wih camera and ) he direcion of vecor 4 defines he x-axis of he second sereo sysem (wih camera and 4) in coordinaes of camera ex 4 k 4 k and he lengh of vecor Similarily using () 4 gives he sereo baseline. mz ( e> x mz ) ex k mz ( e> x mz ) ex k ez (4) where mz ez + ez4 ez + 4 R 4 ez4 is he sum of he z-axis vecors in boh camera frames i.e. ez 4 ez4 ( )> and ey ez ex vcaml The lengh of his vecor is he sereo baseline bs k k. As sereo z-axis i.e. as direcion of he opic axes we use he vecor cam (5) we can define he roaion of he second sereo sysem wih respec o camera s R ( ex ey ez ). vcaml Fig. : Lef: Laeral view of he lef side of he muli-camera sysem. From each physical wide-angle camera ( cam cam ) wo virual pinhole cameras are synhesized ha are roaed by ± around he sereo axes ( vcaml and vcaml belong o cam vcaml and vcaml belong o cam ). On he mirror symmeric righ side he physical cameras cam and cam4 are locaed and remapped o vcamr / vcamr and vcamr / vcamr respecively. Each virual pinhole camera has a FOV of 8 65 ; he oal verical FOV of he sereo camera sysem is approx. 4. Righ: Phoo of he cameras. C. Remapping o Pinhole Cameras Alhough alernaive projecion models like he KannalaBrand model presened in he previous secion are well esablished many compuer vision algorihms sill require images projeced according o he pinhole camera model. Depending on he applicaion he pinhole projecion model can also simplify image processing. For insance planar paches viewed fronally do no change heir apparen size for ranslaions perpedicular o he camera axis. Also maching in moion sereo is easier as corresponding poins lie on lines and no on more complex curves []. Alhough i is possible in principle o remap any camera image wih FOV < 8 o a single pinhole image i is no recommended. Since he pinhole projecion is usually a poor descripion of he acual mapping of wide-angle cameras i would eiher arificially magnify he angular resoluion in he ouer par of he image and lead o image blur or if mached o he resoluion in he ouer par of he camera image would reduce resoluion in he cener significanly. A reasonable workaround is he remapping o several pinhole images which allows closer approximaion of he original image resoluion.in he following we describe he remapping of wide-angle images o muliple pinhole images which allowed us o run several insances of our efficien sereo odomery and inegrae heir esimaions ino our filer framework as described in secion IV. As depiced in Fig. each wide-angle camera is spli ino wo virual pinhole cameras. The virual cameras share he same viewpoin (he nodal poin /cener of projecion of he wide-angle lens) bu are roaed by ± around he sereo axis o which heir horizonal axes (he u-axes ) are aligned. For Wide-angle or fisheye cameras are ofen close o having a consan angular resoluion radially i.e. camera angle θ ρ where ρ p (u uc ) + (v vc ) is he disance from projecion cener. In he pinhole projecion wih ρ an θ however he resoluion increases wih camera angle according o ρ cos θ wih limθ 9 ρ. θ θ

4 Fig. 4: Raw and pinhole camera images. From lef o righ: raw images of camera (boom) and camera (op); raw images of camera (boom) and camera 4 (op); remapped pinhole images of virual cameras LLLL (from boom o op); remapped pinhole images of virual cameras RRRR. Noe ha sraigh lines are sraigh afer remapping o pinhole images bu appear ben in he original camera images. each of he 4 virual cameras (i D L R) of he firs fisheye sereo sysem wih pixel coordinaes ( id u id v) he non-normalized direcion vecor (in coordinaes of sereo sysem ) is s v( id u id v) s id ˆR ( id u id u c id v id v c f i ) s il ˆR ( id u id u c id v id v c f i ) (6) where we used he fac ha s ir ˆR s il ˆR i.e he roaions for he lef and righ virual sereo camera are he same since heir coordinae sysems have he same orienaion wih respec o sereo frame s. Similarly for he 4 virual cameras (i D R L) of he second sereo sysem s v( id u id v) s il ˆR ( id u id u c id v id v c f i ). (7) Using Eqs. (6) and (7) he roaion marices ˆR ˆR s s derived in he previous sub-secion he ransformaion beween camera frames esimaed by exrinsic calibraion as well as he insrinsic camera calibraion parameer he remapping ables for each virual camera can be calculaed. Noe ha x-axes of sereo sysems are defined by lines connecing he nodal poins of he cameras. Therefore while he x-axes of he virual sereo cameras ha belong o he same physical cameras are perfecly aligned his is no rue for physically differen sereo cameras sysems. For our muli-camera sysem his means ha he roaion of virual sereo frames L wih respec o L and of L wih respec o L is a roaion of 6 around he x-axis. However for insance he roaion of L wih respec o L is only approximaely described by a roaion of 6 around he x- axis of L. As shown in Fig. 4 each of he four wideangle images of size 86 8 px are debayered and hen remapped o wo RGB pinhole images of size px using bilinear inerpolaion. For FPGA sereo processing he lef and righ pinhole image are scaled by facor.5 and Fig. 5: FPGA Sereo processing: lef inpu image consising of he pinhole images of all lef virual cameras i.e. LLLL arranged in a grid. The resuling deph map is shown on he righ. arranged in a layou see Fig. 5. Currenly he deph resoluion is limied by he maximum image heigh of 58 px in he FPGA based sereo implemenaion. An overview of he image processing pipeline is shown in Fig. 6. IV. VISUAL ODOMETRY AND FUSION The following wo paragraphs describe he visual odomery and he sae esimaion used on he MAV. A. Visual Odomery The ask of a visual odomery is o give an esimae of he camera moion based on he perceived images. Our visual odomery esimaes he relaive ransformaion from one camera frame o anoher aken a differen imesamps. The algorihm is based on [4] [5] where he reader is referred o for more deails. We assume ha he scene is mainly saic and ha he camera moion can be arbirary. AGAST feaures [6] are deeced in each of he lef remapped virual camera images. For each corner feaure he D informaion is provided by he resuling deph maps from dense sereo maching. Therefore we obain hree-dimensional feaures which can be used for moion esimaion. Feaures which have no valid deph value because of occlusions or oher reasons are discarded. Also feaures ha have oo large deph values are ignored as uncerainy increases quadraically wih disance in sereo vision. In heory jus hree non-colinear D feaure poins are sufficien o calculae Fig. 6: Basic camera and visual odomery seup. Fisheye images are capured. The fisheye images are remapped ino eigh pinhole images. All lef and righ images are grouped ino one combined lef and one combined righ image. 4 Lef and righ images are sen o FPGA for sereo processing resuling in a deph map. 5 Each VO insance receives a pinhole image and he corresponding deph map.

5 ranslaion and roaion of he relaive movemen. Neverheless i is advanageous o have more feaure poins o reduce he effec of noise which increases accuracy and o improve rejecion of ouliers. Afer feaure exracion we search for correspondences from he previous image and he curren image. Before a feaure is used for moion esimaion i has o pass wo addiional oulier rejecion seps. Since we assume a mainly saic scene we can expec ha he relaive disance d P Q of wo poins P and Q does no change from frame i o curren frame i. Therefore he disance d i P Q and disance di P Q in Eq. (8) should equal up o a measuremen error. d i P Q Pi Q i d i P Q P i Q i (8) The resul of his oulier rejecion sep is a se of consisen correspondences which mainain a relaive disance beween each oher. The nex oulier rejecion sep is based on an upper limi for he roaion angle of he camera. Afer he oulier rejecion he remaining corresponding feaures P i k and P i k k... n can be used o esimae he camera moion. In principle i is done by minimizing E( ˆR ) k σ k (P i k ( ˆR P i k + )). (9) In Eq. (9) a spherical error model is used as a rough approximaion of an image-based error model. The advanage of his error model is ha he ransformaion can be calculaed in a closed form soluion. Afer ˆR and are esimaed Chauvene s crierion [7] is applied o remove he likely false correspondences. Finally he values for he ranslaion and roaion wih he iniial guess from he previous spherical error model and he reduced se of consisen correspondences are opimized using an ellipsoid error model as described by Mahies and Shafer []. VO- VO- VO- VO- Fig. 7: Keyframe handling of muliple visual odomeries. Red dos illusrae keyframes arcs indicae relaive o which reference frame he camera poses are esimaed. Samples wihou arc indicae frames for which a pose esimaion was no possible. The four VOs selec differen keyframes. Similar o [8] [9] [] [] we are using keyframes for esimaing no only he pose of he curren camera frame relaive o he previous one bu also relaive o a se of seleced camera frames in he pas o increase he accuracy of he overall pose esimaion. Every ime an image is aken is relaive pose o all previous keyframes is calculaed. The keyframe wih he highes residual error is replaced by he new image. In our curren seup we use 5 keyframes. A key feaure of running independen visual odomeries wih differen FOV is he abiliy o selec differen keyframes for each VO as shown in Fig. 7. Each displayed poin illusraes one keyframe and each arc from which o which frame he moion is esimaed. The four VOs use differen keyframes. For insance while flying close o ground feaures in he lower field of view used by VO- are usually visible for much shorer inervals han feaures closer o he horizon herefore VO- and VO- will selec differen keyframes which will be demonsraed in secion VI. B. Sae Esimaion wih Delayed Muli-VO Measuremens We obain robus and accurae sae esimaes by combining inerial measuermens from he IMU and he oupu of four independen visual odomeries. The acceleraion and angular rae readings from he inerial measuremen uni are used o updae he sysem sae a high frequency. They are fused wih lower frequen measuremens from muliple visual odomeries in an error sae space Kalman filer as described in [] and []. The direc form of he esimaed sae also called main sae is defined by x ( n b p n b v n b q b b a b b ω ) () where n b p R is he posiion of he body frame (bframe) relaive o an earh-fixed inerial frame (n-frame) n b v R is he velociy n b q he orienaion represened as a quaernion and b b a and b b ω are he acceleraion and angular rae biases of he IMU. To fuse he VO measuremens i is imporan o ake ime delays due o sensor processing ino accoun. Therefore hardware riggers are used o define he ime samp of an image wih high accuracy. Each ime an image is riggered a sub-sae is added o he main sae. The componens of he main sae which have o be augmened are defined by he measuremen equaion. In case of VO measuremens which provide esimaes of pose differences and heir covariances he equaion is in he form of h h( n b p n b q n b p n b q ). () The imes and refer o he sar and end ime of he visual odomery measuremen i.e. is he ime samp of he keyframe relaive o which he moion a ime has been esimaed. Measuremens from differen VOs wih idenical end imes can have differen sar imes. Therefore each ime a hardware rigger arrives he main sae has o be augmened by he curren pose n b p n b q and he covariance marix by he sub-marix represening he uncerainy of he curren pose. In addiion he equaion for sysem propagaion has o be adaped. The resul from a visual odomery is available o he filer wih some delay. However due o he augmenaion of he main sae a he ime of image capure he measuremen can refer o he relevan sysem sub-sae when he measuremen finally arrives and correc he curren sae including he augmenaions. To rejec ouliers in he

6 measuremens from a visual odomery he Mahalanobis disance [4] is calculaed. I compares he acual measuremen from he visual odomery and he prediced measuremen based on he filer esimae and rejecs measuremens above a hreshold depending on he fusion esimaion uncerainy. V. MAPPING In his secion we give a quick overview of our global localizaion and mapping framework which is based on he archiecure presened in [5] [7]. We perform an online global D mapping of he environmen based on our filer esimaes and dense deph daa from our fisheye camera sysem. We hereby aggregae he merged deph daa from he four virual pinhole sereo cameras along he rajecory esimaed by he filer. As he filer esimaes are locally sable bu globally subjec o drif we spli he aggregaed daa ino so-called submaps of limied uncerainy and size. Our navigaion filer is a local reference filer [] we hus can always swich is frame of reference ino he origin of he curren submap. This allows us o mainain long-erm consisency and numerical sabiliy wihin he filer as well as a more accurae inegraion of he filer s esimae ino he overlying SLAM sysem [6]. We add he submap origins as nodes o a SLAM graph and connec hem via he filer esimaes as edges weighed by heir Gaussian uncerainy. Loop closure consrains from landmark deecions or map maches can easily be inegraed ino he graph for online global pose and map opimizaion as described in [7]. We consruc he SLAM graph a a high level of absracion i.e. on op of he local reference filer esimaes and can hereby keep is size small and incremenal online opimizaion seps fas. This is in paricular beneficial in a seup wih muliple high-frequency daa sources and filer-inernal saes like our seup on Ardea wih four key frame-based visual odomeries. As he informaion of all visual odomeries is fused in he local reference filer he SLAM graph does neiher increase in size nor in complexiy by adding more high-frequency measuremens or esimaes like in his case he addiional visual odomeries. VO- VO- Fig. 8: Keyframes seleced by VO- and VO- during a forward fligh. The upper row of dos in each sub-figure illusraes he keyframes. The lower row of dos represens he curren image. Lines indicae which keyframe was picked o perform he pose esimaion. Noe ha boh VOs choose quie differen keyframes. VI. EXPERIMENTS AND RESULTS To show he benefis of our sysem we presen resuls of four experimens. The firs experimen demonsraes he independen selecion of keyframes by each visual odomery. In he second experimen daa from an MAV fligh was used o simulae a camera failure. The hird experimen shows a robus fligh over a poorly-exured scene. The final experimen illusraes he mapping capabiliies of our MAV. A. Independen Keyframe Selecion The moion esimaes of he differen visual odomeries are based on differen regions of a scene depending on he field of views of he corresponding cameras. Depending on he curren movemen and srucure of he scene each VO will experience differen opic flow. Therefore depending on he field of view he opimal se of keyframes will be differen. Our sysem is aking his ino accoun since each visual odomery is choosing heir own individual keyframes as described in secion IV. This is an advanage over a sysem running jus a single VO in a join-opimizaion over he enire field of view. Here he sysem is jus picking a single se of keyframes which is likely o be sub-opimal. In our case each VO chooses a se of keyframes based on is individual FOV which will resul in a beer selecion. To demonsrae ha behavior we moved he MAV in a sraigh line and recorded he keyframes seleced for each VO which will be used for pose esimaion. Figure 8 illusraes he keyframes of VO- and VO-. The VO- is looking sraigh down on he floor whereas he VO- is looking in an angle (see Fig. ). One can see ha he upper VO is referencing o images much furher back in ime han he lower one. This is an expeced behavior since he flow field is sronger in he lower image han he upper one. This is because he movemen is perpendicular o he z-axis of he lower virual camera whereas he z-axis of he upper camera is closer o he direcion of ranslaion. Therefore feaures canno be racked for a very long ime in he lower image since hey do no occur for many frames. z (m) - y (m) ref p... S occ D posiion E -4 - x (m) ref y (m) z (m) x (m) 4 S occ E occ S E x (m) Fig. 9: Lef: Esimaed and reference rajecory of a fligh saring a poin S and ending a poin E. Top-righ: Top-down view of he rajecories. Boomrigh: Laeral view of he rajecories. B. Robus Sae Esimaion despie Camera Failures During his experimen he MAV was flying along a rajecory as shown in Fig. 9. I ook off a locaion S and landed a locaion E. Beween is saring and landing locaion i flew wo rounds a differen heighs. During

7 he fligh all four visual odomeries were used for sae esimaion unil ime occ. A ime occ he lower lef camera ha provides inpu o V- and VO- was swiched off in order o show he robusness of he approach o camera or visual odomery failures. In Fig. he ranslaional error p... beween ground ruh provided by a Vicon racking sysem and he esimaed posiion is shown. The subscrip... indicaes ha all four visual odomeries were used if available. In addiion o he error p... he posiional errors p i for sae esimaion using a single odomery i p p p p are shown. They resul from several replays of he filer wih only a single visual odomery acivaed a a ime. Due o muliple unrecognized ouliers and poor feaures caused by direc illuminaion from lighs suspended from poorly lighed ceiling which are used for he visual odomery VO- he error p increases very fas. As he visual odomeries VO- and VO- are swiched off a ime occ he corresponding errors increase aferwards. Neverheless he error p... says small afer occ and is below he smalles error based on a single odomery p. posiion error (m).5.5 occ p... p p p p ime (s) Fig. : Posiion error beween esimaed and reference rajecory for five differen configuraions wih a failure of wo visual odomeries (VO- VO- ) beginning a ime occ (indicaed by dashed verical line). Fig. : Esimaed (solid lines) and reference (dashed lines) posiion during fligh over wo blue floor mas. The colored areas indicae periods where VO- failed. floors walls and ceilings is no uncommon. If he visual odomery of a robo perceives mainly such an area i canno rack enough feaures o calculae a reliable pose esimaion. As a resul he filer begins o drif and in cases where he area is large he MAV migh even crash due o he accumulaion of drif errors. To sudy such a scenario and o evaluae he performance of our approach we conduced an experimen in which maresses were laid ou in he laboraory as shown in figure. In his experimen Ardea flew from posiion o and back again. The rajecory leads over wo maresses which have almos no visual exure. 4 VOs are fused online and on-board wih he IMU o conrol he posiion of he MAV. Figure shows he resuls of he experimen. The illusraion displays he esimaed rajecory wih he acual flown rajecory obained by a visual racking sysem. When Ardea flew over he maresses VO- failed several imes o perform a reliable pose esimaion. This occasions are illusraed in he figure wih shaded red regions. Since he fusion filer is no jus fusing he oupu of one VO he failing of his paricular VO can be compensaed and a poenial crash of he MAV could be avoided. Therefore his experimen shows he srengh of fusing muliple visual odomeries o ge a more robus pose esimaion. Fig. : Fligh of Ardea from he saring and landing area over he wo blue floor mas o poin. While flying over he poorly-exured floor mas no all four visual odomeries generae valid oupu. C. Robus Fligh over Texure-Less Areas In pracice a fligh over or nex o areas wih poor or indiscernible exure like small bodies of waer exure-less Fig. : Single sho poincloud compued from he eigh pinhole images.

8 D. Mapping of Scene wih MAV Fig. shows he poin cloud creaed by he eigh virual pinhole cameras from a single ime sample. The coordinae frame indicaes he posiion of Ardea. The ulra wide field of view of he poin clouds provides valuable informaion above in fron of and below he MAV. Fig. 4 shows he fligh rajecory of Ardea and he resuling accumulaed D poin cloud map compued by our SLAM sysem described in secion V. I consiss of a series of nine submaps. To keep he accumulaed error in he local reference filer low new submaps were sared whenever he esimaed posiional or roaional covariance reached a hreshold of. m or 5 respecively. In his experimen he MAV was manually conrolled o wo waypoins in our laboraory. A hese poins i roaed around is yaw-axis. Due o Ardea s large verical field of view he floor and ceiling can be mapped simulaneously resuling in a dense D poin cloud. Fig. 4: Top: Resuling map of an MAV fligh. The covariance ellipsoids (in magena) indicae he esimaed posiional uncerainy wih respec o he sar frame. Boom: Phoo of he lab for comparison. VII. CONCLUSION In his paper an MAV equipped wih four wide-angle cameras was inroduced. The large verical sereo FOV of 4 enables he MAV o perceive objecs below above and in fron of he MAV which is relevan for obsacle avoidance pah planning and efficien mapping. Sae esimaion also benefis from he large FOV due o robus moion esimaion provided by four sereo odomeries wih independen keyframes which was shown in experimens. REFERENCES [] T. Tomic K. Schmid P. Luz A. Dömel M. Kassecker E. Mair I. L. Grixa F. Ruess M. Suppa and D. Burschka Toward a fully auonomous UAV IEEE Roboics & Auomaion Magazine. [] L. Mahies R. Brockers Y. Kuwaa and S. Weiss Sereo visionbased obsacle avoidance for micro air vehicles using dispariy space. in ICRA 4. [] P. Gohl D. Honegger S. Omari M. Achelik and R. Siegwar Omnidirecional visual obsacle deecion using embedded FPGA. in IROS 5. [4] A. J. Barry and R. Tedrake Pushbroom sereo for high-speed navigaion in cluered environmens. in ICRA 5. [5] S. Houben J. Quenzel N. Krombach and S. Behnke Efficien mulicamera visual-inerial SLAM for micro aerial vehicles in IROS 6. [6] M. Beul N. Krombach Y. Zhong D. Droeschel M. Nieuwenhuisen and S. Behnke A high-performance MAV for auonomous navigaion in complex D environmens in ICUAS 5. [7] M. J. Schuser K. Schmid C. Brand and M. Beez Disribued Sereo Vision-Based 6D Localizaion and Mapping for Muli-Robo Teams Journal of Field Roboics 8. [8] T. Oskiper Z. Zhu S. Samarasekera and R. Kumar Visual odomery sysem using muliple sereo cameras and inerial measuremen uni. in CVPR 7. [9] H. Hirschmüller Sereo processing by semiglobal maching and muual informaion IEEE Trans. Paern Anal. Mach. Inell. 8. [] K. Schmid and H. Hirschmüller Sereo vision and IMU based realime ego-moion and deph image compuaion on a handheld device. in ICRA. [] J. Kannala and S. S. Brand A generic camera model and calibraion mehod for convenional wide-angle and fish-eye lenses IEEE Trans. Paern Anal. Mach. Inell. 6. [] S. Abrahama and W. Försner Fish-eye-sereo calibraion and epipolar recificaion ISPRS Journal of Phoogrammery and Remoe Sensing 5. [] D. Caruso J. Engel and D. Cremers Large-scale direc SLAM for omnidirecional cameras in IROS 5. [4] H. Hirschmüller P. R. Innocen and J. M. Garibaldi Fas unconsrained camera moion esimaion from sereo wihou racking and robus saisics in ICARCV. [5] A. Selzer H. Hirschmüller and M. Görner Sereo-vision-based navigaion of a six-legged walking robo in unknown rough errain The Inernaional Journal of Roboics Research. [6] E. Mair G. D. Hager D. Burschka M. Suppa and G. Hirzinger Adapive and generic corner deecion based on he acceleraed segmen es in ECCV. [7] J. R. Taylor An Inroducion o Error Analysis. Universiy Science Books 98. [8] C. Forser Z. Zhang M. Gassner M. Werlberger and D. Scaramuzza SVO: Semidirec visual odomery for monocular and mulicamera sysems IEEE Transacions on Roboics 7. [9] R. Mur-Aral and J. D. Tardós ORB-SLAM: an open-source SLAM sysem for monocular sereo and RGB-D cameras IEEE Transacions on Roboics 7. [] R. Wang M. Schwörer and D. Cremers Sereo DSO: Large-scale direc sparse visual odomery wih sereo cameras in ICCV 7. [] J. Engel V. Kolun and D. Cremers Direc sparse odomery IEEE Trans. Paern Anal. Mach. Inell. 8. [] K. Schmid F. Ruess M. Suppa and D. Burschka Sae esimaion for highly dynamic flying sysems using key frame odomery wih varying ime delays in IROS. [] K. Schmid F. Ruess and D. Burschka Local reference filer for life-long vision aided inerial navigaion in FUSION 4. [4] H. Wu S. Chen B. Yang and K. Chen Feedback robus cubaure Kalman filer for arge racking using an angle sensor Sensors 6. [5] M. J. Schuser e al. Towards auonomous planeary exploraion Journal of Inelligen & Roboic Sysems 7. [6] M. J. Schuser C. Brand H. Hirschmüller M. Suppa and M. Beez Muli-robo 6D graph SLAM connecing decoupled local reference filers in IROS 5.