Experiments in Vision-Laser Fusion using the Bayesian Occupancy Filter

Transcription

1 Experimens in Vision-Laser Fusion using he Bayesian Occupancy Filer John-David Yoder, Mahias Perrollaz, Igor Paromchik, Yong Mao, Chrisian Laugier To cie his version: John-David Yoder, Mahias Perrollaz, Igor Paromchik, Yong Mao, Chrisian Laugier. Experimens in Vision-Laser Fusion using he Bayesian Occupancy Filer. Inernaional Symposium on Experimenal Roboics, Dec 2010, Delhi, India <inria > HAL Id: inria hps://hal.inria.fr/inria Submied on 23 Dec 2010 HAL is a muli-disciplinary open access archive for he deposi and disseminaion of scienific research documens, wheher hey are published or no. The documens may come from eaching and research insiuions in France or abroad, or from public or privae research ceners. L archive ouvere pluridisciplinaire HAL, es desinée au dépô e à la diffusion de documens scienifiques de niveau recherche, publiés ou non, émanan des éablissemens d enseignemen e de recherche français ou érangers, des laboraoires publics ou privés.

2 Experimens in Vision-Laser Fusion using he Bayesian Occupancy Filer John-David Yoder 1, Mahias Perrollaz 2, Igor E. Paromchik 2, Yong Mao 2 and Chrisian Laugier Ohio Norhern Universiy, Ada, OH, USA. INRIA Grenoble Rhône-Alpes, Sain Ismier, France. Firsname.lasname@inrialpes.fr Absrac Occupancy Grids have been used o represen he environmen for some ime. More recenly, he Bayesian Occupancy Filer (BOF), which provides boh an esimae of likelihood of occupancy of each cell, AND a probabilisic esimae of he velociy of each cell in he grid, has been inroduced and paened. This work presens he firs experimens in he use of he BOF o fuse daa obained from sereo vision and muliple laser sensors, on an inelligen vehicle plaform. The paper describes he experimenal plaform, he approach o sensor fusion, and shows resuls from daa capured in real raffic siuaions. 1 Inroducion The primary moivaion for his work is moving oward reducing he number of road accidens which are currenly responsible for over a million deahs annually [1]. While auonomous cars such as he enries in he DARPA Urban Challenge [2][3] may increase safey in he very-long-erm, in he shor and medium-erm, echnology can hope o reduce he number of accidens hrough a variey of sraegies. As par of he ArosDyn projec [4], we are developing an embedded sysem o provide real-ime esimaion and predicion of he curren level of collision risk. This advance warning of high-risk siuaions could creae enough addiional reacion ime o le he driver avoid, or reduce he severiy of, an acciden. The work presened here is a par of ha projec, focusing on he experimenal challenge of real-ime sensor fusion. This work is based on occupancy grids, which have been used o represen he environmen [5] and fuse sensor daa [6] for quie some ime. Sereo vision and laser sensors are boh regularly used o creae occupancy grids, bu unlike oher works, his informaion is fused using he Bayesian Occupancy Filer (BOF) [7]. The BOF, an adapaion of Bayesian filering o he occupancy grid framework, offers several advanages. Because he BOF is highly parallelizable, i has been shown o run in real-ime on a Graphics

3 2 Yoder, Perrollaz, Paromchik, Mao, Laugier Processing Uni (GPU). In addiion, he BOF, because i mainains velociy esimaes, provides he abiliy o predic he fuure moion of dynamic areas in he environmen even if hey are emporarily obscured. Finally, he oupu of he BOF is used o provide he esimaed posiion and velociy of objecs in he ego-vehicle s environmen. Having velociy esimaes, raher han jus occupancy, allows for more robus clusering [8]. I should be noed ha hese velociy esimaes are relaive o he ego-vehicle raher han absolue. 2 Technical Approach While he overall goal of his work is o improve raffic safey, his paper focuses on an embedded sysem which can updae he driver on he curren sae of risk in real ime. This requires a srong emphasis on efficien algorihms, and on approaches which are highly parallelizable in order o offer good performance on a GPU, and evenually on an embedded parallel processor. The BOF has been specifically developed wih he inenion of implemening i on hardware for highperformance applicaions. The BOF also provides he significan advanage of performing sensor fusion a a very low semanic level, he cell level. This reduces he daa associaion problem o is simples form here is no need o associae objecs or sensor readings direcly, only o relae hem o a cell in he occupancy grid. Figure 1 gives an overview of he process. The inpus o he sysem are a dispariy image from he sereo camera sysem and he laser range daa. For each sensor, a probabilisic sensor model convers his inpu ino an occupancy grid. The laser is modeled in a manner similar o ha presened in [9]. The sereovision sensor uses a novel approach of calculaing he occupancy grid firs in he u-dispariy image [10]. One of he srenghs of our u-dispariy approach is ha i is highly parallelizable, and herefore benefis grealy from implemenaion on a GPU. I also allows for he deecion of parially occluded obsacles, which gives he vision sensor he abiliy o find objecs ha he lidar canno. This is in conras o oher approaches which use vision o consruc occupancy grids, such as [11]. In ha work, he auhors rea he visual sensor very much like a laser, saying ha he space before he firs objec is free, and he space behind he firs objec is unknown. While his approach allows for simplified processing, i is sensiive o false posiives, and does no allow for he deecion of parially-occluded objecs. In our applicaion, hen, his model would remove any advanages of he visual sensor. In essence, here would be no need o fuse sensors, since he visual sensor would no be adding any informaion compared o he laser.

4 Experimens in Vision-Laser Fusion using he Bayesian Occupancy Filer 3 LIDAR Occupancy grid Sereo Camera Occupancy grid BOF Predicion P(O c A c ) Z 1 Z s Esimaion P(O c A c Z 1 Z S ) Probabiliy esimaes of occupancy and velociy for each cell Figure 1: Overview of he sensor fusion process These occupancy grids (from he laser and from sereo camera) are inpu ino he BOF. As shown in Figure 1, he BOF consiss of a predicion phase and an esimae phase. A a given ime sep, he predicion phase uses he curren probabiliy of occupancy (O) and probabiliy disribuions of velociy (A) of each cell o obain he predicion P(O c A c ), where c denoes he cell and he curren ime sep. This predicion is based on he consan-velociy assumpion. In he esimaion sep, P(O c A c ) is updaed based on Z i, he observaion of sensor i a ime. This provides he a poseriori sae esimae P(O c A c Z 1 Z S ) where S is he number of sensors. Noe ha in our case he occupancy grid is he same size as he BOF grid, meaning ha his occurs for all cells c. This predicion-esimaion paradigm provides wo major advanages. Firs, should a sensor reading be unavailable, he BOF coninues o produce predicions abou he sae a each ime sep. Secondly, i preserves emporal consisency, so suddenly-occluded areas can coninue o be racked for several ime seps. Noe ha he exac number of ime seps would depend on he parameers of he BOF and he probabiliy disribuions. The oupu of he fusion process, hen, is he nex ieraion of he BOF. In our applicaion, he nex sage of he process is o model saic and dynamic objecs in he environmen. This is accomplished via he previously published Fas Clusering and Tracking Algorihm [8]. This algorihm provides esimaes of he posiion and velociy of objecs in he proximiy of he vehicle, along wih he associaed uncerainies. These values hen become inpu o a risk-assessmen model [12].

5 4 Yoder, Perrollaz, Paromchik, Mao, Laugier 3 Experimenal Plaform The experimens are conduced using a Lexus LS600 as he base for he experimenal plaform. I was desired o insrumen he vehicle for experimens wih as lile modificaion o he vehicle as possible. The wo laser sensors, IBEO Lux Lidars, are placed in he fron bumper. A TYZX sereo camera sysem is mouned jus forward of he rear-view mirror. These modificaions can be seen in Figure 2, and are he only modificaions visible from he exerior. Figure 2: Sensors mouned in he Lexus, made available o INRIA by Toyoa Europe as par of a long-erm ongoing collaboraion. In addiion, he vehicle has been equipped wih a Dell compuer wih an NVidia GPU. An Xsens IMU wih GPS is placed a he cener of he rear axle. Hugr middleware [13] is used o record, synchronize, and replay daa sequences. Algorihm parameers (for sensor models, he BOF, ec) are adjusable hrough a Qbased user inerface. Specifics for he hardware are as follows. The sereo camera baseline is 22 cm, wih a field of view of 62. Camera resoluion is 512x320 pixels wih a focal lengh of 410 pixels. Each lidar provides four layers of up o 200 impacs wih a sampling period of 20 ms. Maximum deecion range is abou 200 m, he angular range is 100, and he angular resoluion is 0.5. Using he wo lidars, he observed region is 40 m long by 40 m wide, wih a maximum heigh of 2 m. Cell size for he occupancy grids is 0.2x0.2 m. I should be noed ha each layer of laser measuremens is reaed as a differen sensor. This means ha he BOF is really fusing 8 laser measuremens wih he sereo vision measuremen. The on-board compuer is equipped wih 8GB of RAM and an Inel Xeon 3.4GHz processor. The GPU is an NVIDIA GeForce GTX 480. The daases described in his paper were obained on French roadways. Furher processing, including sensor fusion, was done off-line. Deerminaion of he exrinsic parameers of he sensors was done manually for his work.

6 Experimens in Vision-Laser Fusion using he Bayesian Occupancy Filer 5 4 Resuls Figure 3 shows he advanage of sensor fusion, he abiliy o find a parially occluded objec (a pedesrian in a parking lo). Figure 3a shows he view from boh cameras, wih impacs from he lidar represened by colored dos in he images. Red and green dos denoe his from he lef and righ lidars, respecively. Figure 3b shows he occupancy grid creaed by a BOF based on only he sereo camera. Figure 3c shows he occupancy grid creaed by a BOF based on only he lidars. Finally, Figure 3d shows he occupancy grid obained from he fused BOF. In Figure 3b, we see he occupancy grid creaed by he sereo vision sensor. Green cells correspond o low likelihood of occupancy (close o 0), while brigh red cells correspond o high probabiliies of occupancy (close o 1). The background yellow color represens he case of unknown occupancy (value of 0.5). The sereo sensor has deeced he road in he foreground (unoccupied cells), oba b c d Figure 3: Example of Sensor Fusion: (a) image of he scene, (b) occupancy grid creaed by he sereo vision, c) occupancy grid from he lidar, and (d) fused grid wih he BOF.

7 6 Yoder, Perrollaz, Paromchik, Mao, Laugier sacles along he perimeer (parked cars), and he box and person in he cener of he image. This frame shows he abiliy of he visual sensor o deec he pariallyoccluded pedesrian. Figure 3c shows he occupancy grid from lidar. As is ypical of laser sensor models, he area in fron of he obsacles has a low likelihood of occupancy, whereas areas behind he obsacles are unknown. As such, he box in he cener of he image is found, bu he pedesrian is no. The fused image in Figure 3d mainains he advanage of boh sensors, finding he pedesrian, and mainaining he more defined obsacles around he perimeer. The wo calculaion-inensive porions of he algorihm are he visual processing and he updaing of he BOF. As described earlier, one of he cenral objecives in he creaion of he BOF was ha is highly parallel srucure could allow real-ime processing on a GPU, and evenually on an embedded processor. Similarly, he u-dispariy approach for occupancy grid creaion from he sereo cameras is highly parallelizable. To give some sense for his, he following processor imes are provided. We are using a grid size of 40m x 40m wih a cell size of 0.2m x0.2m. For he compuer vision algorihms on he CPU, he compuaion akes 159 ms, or abou 6 Hz, oo slow for real-ime operaion. However, when implemened in CUDA on he GPU, his same algorihm runs in 8 ms. On he GPU, he BOF updaes require 5 ms, including he sensor fusion. Thus he sereo processing and he BOF updaes can readily be accomplished in real ime. Figures 4-6 show example siuaions based on daa ses obained urban raffic. In each case, here are five images: (a) shows he scene as viewed by he lef camera, (b) shows he occupancy esimaes from he BOF, (c) and (d) are he occupancy grids creaed by he lidar for inpu o he BOF, and (e) is he occupancy grid creaed by sereo vision for inpu o he BOF. Noe ha he color scheme for he BOF oupu (b) is as described above. For he inpu occupancy grids, black represens open areas (close o 0), whie represens occupied areas (close o 1), and he background color of gray represens he case of unknown occupancy (value of 0.5). Firs, consider he case of Figure 4. The scene in (a) is quie cluered wih a variey of saic and dynamic obsacles (pedesrians, cars, a bus and buildings in he background, ec.) The wo pedesrians in he foreground can be seen as wo small red circles a he boom of he BOF oupu in (b). The car, he buildings a he corners of he inersecions, ec., all appear in he BOF, as does he pedesrian crossing on he oher side of he inersecion. When inspecing he occupancy grids (c-e), we can see ha he sereo vision produces a less cerain occupancy grid (here are more shades of gray). This is due o he novel mehod of creaing his grid. As such, in areas where he lidars are confiden, he vision adds lile informaion. In he background, where he lidar has low cerainy, he informaion from vision can fill in he grid. Figure 5 presens us wih a good example of his. This scene is shorly afer he scene from Figure 4, he main difference being ha he pedesrians have moved ou of he scene and he bus has moved closer. Inspecion of he BOF in Figure 5b shows ha, wih a fair degree of cerainy, he bus is being observed (he long red

8 Experimens in Vision-Laser Fusion using he Bayesian Occupancy Filer 7 a b c d e Figure 4: Example Scene. (a) view from lef camera. (b) occupancy based on BOF. (c,d) occupancy from lef and righ lidar. (e) occupancy from sereo a b c d e Figure 5: Example Scene. (a) view from lef camera. (b) occupancy based on BOF. (c,d) occupancy from lef and righ lidar. (e) occupancy from sereo

9 8 Yoder, Perrollaz, Paromchik, Mao, Laugier a b c d e Figure 6: Example Scene. (a) view from lef camera. (b) occupancy based on BOF. (c,d) occupancy from lef and righ lidar. (e) occupancy from sereo area jus o he lef of cener). However, looking a he grids from lidar (c,d), he bus is no observed he car in he inersecion is blocking he view of he lidar. The sereo vision, however, can see over he car o deec he bus. As such, ha occupancy grid shows boh he car (owards he boom) and he bus (a bi furher up, hough i blends in wih he building). So in his case, he vision has allowed he BOF o find a parially occluded objec, in much he way i found he person if Figure 3. Finally, Figure 6 shows anoher advanage of he BOF. In his case, he wo cars in (a) crossing he inersecion are occluding par of he scene, and a pedesrian is occluding par of he car o he righ. However, because of he predicionesimaion framework of he BOF, cells are sill esimaed o be occupied behind he cars (noably he building on he righ) despie he fac ha hey are no in any of he occupancy grids a his insan.

10 Experimens in Vision-Laser Fusion using he Bayesian Occupancy Filer 9 5 Conclusion The paper describes iniial experimens wih vision-laser fusion using he BOF in an inelligen vehicle applicaion. The BOF akes as inpu occupancy grids based on probabilisic sensor models of he laser and sereo-camera sensors. The sereo-camera model has been developed for his applicaion, specifically o allow he deecion of parially occluded obsacles. The BOF and he sereo-camera model have been developed o be highly parallelizable, and have been shown o run on a GPU a speeds allowing real-ime operaion. I is also imporan o noe ha since he BOF includes velociy esimaes as well as occupancy informaion, objecs can be racked even while hey are emporarily hidden. The paper has described he sensor-equipped Lexus which serves as he experimenal plaform for his work. This vehicle has recenly been driven in ciy raffic siuaions o begin gahering road daa. Our experimens have shown ha he BOF resuls in efficien and robus fusion of sensor daa from sereo vision and lidars. On a GPU, his can be run in real ime. Work is ongoing o es his approach wih addiional urban driving daa, and o evaluae he sensiiviy o various BOF parameers. The oupu of he BOF is being used o idenify saic and dynamic objecs, which in urn will be used o provide real-ime assessmen of he collision risk as par of he ArosDyn projec a INRIA. Addiionally, work is ongoing o auomaically calibrae he exrinsic parameers of he sereo camera and laser sensors. Acknowledgemens The auhors wish o hank Toyoa Moors Europe for ongoing suppor of his experimenal work on he Lexus car. ProBayes is a parner wih INRIA Grenoble Rhône-Alpes in he developmen and paening of he BOF. Furher hanks o Amaury Nègre, Nicolas Turro and Jean-Francois Cunibero (INRIA) for heir echnical assisance in seing up our experimenal plaform.

11 10 Yoder, Perrollaz, Paromchik, Mao, Laugier References [1] World Healh Organizaion, Global Saus Repor on Road Safey: Time for Acion. hp://whqlibdoc.who.in/publicaions/2009/ _eng.pdf [2] C. Urmson e al. Auonomous driving in urban environmens: Boss and he urban challenge. J. of Field Roboics, vol. 25(8), [3] M. Monemerlo e al. Junior: The Sanford enry in he Urban Challenge. J. of Field Roboics, vol. 25 (9), [4] I.E. Parmochik, C. Laugier, M. Perrollaz, M. Yong, A. Nègre, C. Tay, The ArosDyn Projec: Robus Analysis of Dynamic Scenes. Proc. of he 11 h In. Conf. on Conrol, Auomaion, Roboics, and Vision, Singapore, [5] H.P. Moravec, A.E. Elfes. High Resoluion Maps from Wide Angle Sonar. Proc. of he 1985 IEEE In. Conference on Roboics and Auomaion, pp , March [6] H.P. Moravec, Sensor Fusion in Cerainy Grids for Mobile Robos. AI Magazine, 9(2), [7] C. Coue, C. Pradalier, C. Laugier, T. Fraichard, P. Bessiere. Bayesian Occupancy Filering for Muliarge Tracking: An Auomoive Applicaion. In. J. Roboics Research, No. 1, [8] K. Mekhnacha, Y. Mao, D. Raulo, C. Laugier. Bayesian Occupancy Filer based Fas Clusering-Tracking Algorihm, IEEE/RSJ In. Conf. on Inelligen Robos and Sysems, Nice, [9] S. Thrun, W. Burgard, D. Fox, Probabilisic Roboics, MIT Press, [10] M. Perrollaz, J.-D. Yoder, C. Laugier, Using Obsacle and Road Pixels in he Dispariy Space Compuaion of Sereo-vision based Occupancy Grids, Proc. of he IEEE In. Conf. on Inelligen Transporaion Sysems, Madeira, Porugal, [11] D. Murray, J. Lile, Using real-ime sereo vision for mobile robo navigaion, Auonomous Robos, vol. 8, January [12] C. Tay. Analysis of Dynamics Scenes: Applicaion o Driving Assisance. PhD Thesis, INRIA, France, [13] CyCab Toolki, hp://cycabk.gforge.inria.fr/