High Accuracy Measurement with 360 and Giga-pixel Panorama Images

Transcription

1 High Accuracy Measuremen wih 36 and Giga-pixel Panorama Images Jafar Amiri Parian FODIS, Haupsr. 14, CH-851 Lusdorf, Swizerland Absrac Panoramic cameras wih Linear Array CCD have been originally buil for purely imaging purposes, bu hey also have a high poenial for use in high accuracy measuremen applicaions. They have he advanage of high informaion conen wih giga-pixel forma size and 36 field of view. Viewing 36 images is a naural way for he human being o perceive an environmen. The abiliy o process daa direcly from 36 high resoluion panorama RGB images makes he workflow exremely inuiive, naural and realisic. FODIS Measure3D is a combinaion of a fas panorama camera and innovaive sofware packages for he exracion of 3D informaion from panoramic imagery. We presen he sensor model for FODIS panorama camera and he resuls of selfcalibraion, which indicae a subpixel accuracy level. We demonsrae he sysems accuracy of 1mm in 1 meers in 3D poin posiioning, using a 3D esfield wih a free nework adjusmen. Wih FODIS Measure3D we do have addiional powerful sensors for image recording and efficien 3D objec modeling. 1 Inroducion The word panorama is a combinaion of Greek erms, namely he suffix pan (παν), meaning all, and horama (οραµα), meaning sigh. In more echnical erms, a panorama is defined as a picure or a series of picures of a landscape, a hisorical even, ec. represening a coninuous scene, enclosing he specaor and providing an unlimied view in all direcions (synonymously he erm omnidirecional is used). In boh cases he meaning of a very wide field of view is conveyed. The echniques of panorama producion can be divided in wo differen groups (Fig. 1): caadiopric and diopric sysems. Dioprics is he science of refracing elemens (lenses) whereas caoprics is he science of he reflecing surfaces (mirrors). The combinaion of refracing and reflecing elemens is called caadioprics. Caadiopric sysems (Baker and Nayar, 1998; Svoboda e al., 1997; Ollis e al., 1999; Boul, 1998) use one or more mirrors wih one or more cameras o capure a wide view angle of he scene. The images hen need mosaicking and in special cases, seamless panoramas can be creaed. Planar and curved mirrors are used in hese sysems. Fig. 1. Panoramic echniques. Subdivision ino caadiopric and diopric sysems. A diopric sysem relaes o is refracive elemens (lenses). Mirrors may be included in hese sysems, bu hen he aim is o fold he opical sysem assembly and no increase he field of view. Diopric sysems are divided ino four groups: camera cluser, fisheye lens, siching and direc scanning. The firs group uses several cameras, mouned on a surface and looking ouwards ono he scene, which enables he capuring of he wide view (Nielsen, 5). The second group consiss of a camera wih a fisheye lens which usually has more han 18 field of view (Herber, 1987). They have been used in measuremen applicaions (Colcord, 1989; Rich, 199; Schwalbe, 5; Heuvel e al., 6). The hird group produces a panoramic image by mosaicking or siching he images (Shum and Szeliski, 1997). This echnique has also been used in measuremen applicaions 1

2 (Pöninen, 1999; Pesa e al, 1; Luhmann and Tecklenburg,, 4; Kukko, 4; Heikkinen, 5). The fourh group is relaed o a camera sysem wih a roaing camera or roaing lens, which produces a seamless image wihou any need of siching (Anipov and Kivaev, 1984; Harley, 1993). A radiional erresrial panoramic camera which operaes by his mehod exposes a small porion of a film coninuously a each specific ime. By using digial echnology, digial roaing panoramic cameras were developed. Such camera sysem consiss of a Linear Array which is mouned parallel o he roaion axis on a high precision urnable. By roaion of he urnable, he Linear Array sensor capures he scene as a coninuous se of verical scan lines. FODIS Measure3D is a combinaion of a fas high resoluion 36 panorama camera (Fig. ) wih innovaive sofware packages, allowing direc 3D measuremen in RGB images rapidly and accuraely. The FODIS panorama camera has hree pars: a camera head, an opical par, and a high precision moor. The camera head, which consiss of a linear array of CCD, allows rapid image acquisiion in Time Delay Inegraion (TDI) mode ypically in few seconds for a 36 image capure. Camera conrol and image ransfer o a able PC are performed via gigabi Eherne. The opical par of he sysem allows he use of differen lenses. The camera head is mouned on a high precision roaing moor. By roaion of he moor, he linear array sensor capures he scenery as a coninuous se of verical scan lines. Roaion speed and scanning angle are pre-selecable and correspond o he shuer speed, image size and he focal lengh of he lens. FODIS Measure3D sofware is a unique close range phoogrammery sofware capable of processing frame array and panoramic imagery. I includes advanced mehods of camera calibraion, image processing and opimizaion o assure high accuracy and reliabiliy. Fig.. FODIS panorama camera ogeher wih a able PC for camera conrol and image sorage. In his paper we presen he mahemaical sensor model for panoramic cameras. We show he resuls of selfcalibraion and accuracy esing. We also presen he resuls of some applicaions. Sensor Model for he Ideal Panoramic Camera The sensor model as a mapping funcion is based on a perspecive projecion in he form of bundle equaions, which maps he 3D objec space poins ono he Linear Array coordinae sysem. For he derivaion of he equaions four coordinae sysems are defined: pixel coordinae sysem, Linear Array coordinae sysem, urnable coordinae sysem, 3D objec coordinae sysem. Fig. 3 shows he coordinae sysems: pixel (i, j), Linear Array (, y, z), urnable (X', Y', Z') and objec space (X, Y, Z) coordinae sysems. To define he urnable coordinae sysem, an ideal panoramic camera is assumed. The origin of he urnable coordinae sysem (O) coincides wih he projecion cener. The roaion axis passes hrough he projecion cener and coincides wih Z'. X' passes hrough he sar posiion of he Linear Array before roaion and Y' is defined o ge a righ-handed coordinae sysem. Eq. (1), which is a 3D conformal

3 ransformaion wih 6 parameers, shows he relaion beween he objec space coordinae sysem and he urnable coordinae sysem: X ' X X Y ' = M ω, ϕ, k. Y Y (1) Z' Z Z wih (X, Y, Z )... locaion of he origin of he urnable coordinae sysem in he objec space coordinae sysem (X, Y, Z)... objec space coordinaes (X', Y', Z').. objec poin coordinaes in he urnable coordinae sysem M ω,ϕ,k... roaion marix wih elemens defined in Eq. (). cosϕ cos k cosϕ sin k sinϕ M = = ω, ϕ, κ R cosω sin k + sin ω sinϕ cos k cosω cos k sin ω sinϕ sin k sin ω cosϕ () sin ω sin k cosω sinϕ cos k sin ω cos k + cosω sinϕ sin k cosω cosϕ ω, ϕ and κ are roaion angles of he urnable coordinaes axes (X', Y', Z') wih respec o he objec where space coordinaes axes (X, Y, Z). The 6 parameers ( ω, ϕ, κ, X, Y, Z ) of Eq. (1) define he exerior orienaion parameers of a panoramic camera. Eq. (3) shows he relaion of an image poin in he Linear Array coordinae sysem wih respec o an objec poin in he urnable coordinae sysem. (a) (b) Fig. 3. Objec coordinae (X, Y, Z), urnable coordinae (X', Y', Z') and Linear Array (, y, z) coordinae sysems. wih X ' Y ' = R Z' Z ( θ ) λ T y c N 1 y = ( i ) p y, θ = j p, x T = 1 (4) 1 (3) Where p x. p y. c.. N.. angular pixel size of he urnable (roaion angle beween wo successive Linear Array image acquisiions) pixel size of he Linear Array camera consan number of pixels in he Linear Array 3

4 R Z T.. (,y,-c) (i, j).. λ. 3D roaion marix around Z' axis ransfer marix from he Linear Array o he urnable coordinae sysem image poin coordinaes in he Linear Array coordinae sysem image poin coordinaes in he pixel coordinae sysem scale facor. Finally he model which relaes he image poin coordinaes (i, j) o he objec poin coordinaes (X, Y, Z) becomes for an ideal sensor: wih ( θ, y) as shown in Eq. (4). X X c M w, ϕ, k Y Y = RZ ( θ ) λ (5) Z Z y 3 Sysemaic Errors and Addiional Parameers of Panoramic Cameras In pracice a real panoramic camera deviaes from he ideal one. These deviaions, which are denoed as sysemaic errors, are modeled by addiional parameers. The sources of sysemaic errors are: he lens, he configuraion of he Linear Array wih respec o he opical axis and he urnable roaion axis, and he urnable iself. The sources of some of he sysemaic errors of panoramic cameras are similar o he frame array CCD cameras. Therefore, he same addiional parameers are used for he modeling. The oher sysemaic errors, which can only be observed for panoramic cameras, are explored hrough he geomerical analysis of he sensor mechanical design and physical measuremens of he errors. In addiion, image poin residuals are analyzed for a beer undersanding of he behavior of he sysemaic errors. Wih respec o he sabiliy of a panoramic camera, sysemaic errors can be divided ino wo differen classes: saionary and non-saionary sysemaic errors. However, some of he parameers of one class migh shif o he oher class depending on he sabiliy of he camera sysem. The aim of his classificaion is o enable he undersanding of he sensor behavior and sensor modeling. The sysemaic errors ha are no varying wih ime are called saionary sysemaic errors. These errors remain consan in a leas one epoch of daa acquisiion and over relaively long ime (e.g. beween wo calibraions). The saionary sysemaic errors for panoramic cameras are divided ino wo groups as follow: 1. Errors only for panoramic cameras are differen scale facors for he wo image axes, induced by errors of he angular pixel size il and inclinaion of he Linear Array wih respec o he roaion axis of he urnable eccenriciies of he projecion cener of he lens wih respec o he roaion axis of he urnable and he origin of he urnable coordinae sysem.. Errors common beween frame array CCD and panoramic cameras which are lens disorions shif of principal poin size of camera consan. The second group of sysemaic errors is modeled by a sub-se of Brown s addiional parameers (Brown, 1976). A new se of addiional parameers is developed for he firs group. Eq. (6) shows he inegraion of he new addiional parameers ino Eq. (5): wih M X X c ex Y Y = RZ ( θ dθ ) RY ( ly) RX ( lx) + ey Z Z y ez w, ϕ, k λ (6) 4

5 where R Y, R X... ex, ey, ez.. lx, ly. dp x... dθ = j dp x (7) 3D roaion marices around Y' and X' axes of he urnable coordinae sysem eccenriciies of he projecion cener from he origin of he urnable coordinae axis inclinaion and il of he Linear Array wih respec o he urnable coordinae axis correcion o he angular pixel size. Fig. 4 shows he eccenriciies of he projecion cener, il and inclinaion of he Linear Array wih respec o he roaion axes. In pracice, he heigh of he projecion cener (PC), ez, from he plane of he urnable is considered o be known and used as a consan parameer, because oherwise i would have a high correlaion wih Z (exerior orienaion parameer). A sub-se of Brown s addiional parameers (Brown, 1976) is used wih some modificaion o model he radial symmerical lens disorion, a correcion o he camera consan and a shif of he principal poin: wih y = y y, where y 3 dy = dy + dc + y ( k1 + k y ) (8) c dy. correcion o he shif of he principal poin along Linear Array k 1,k... parameers of radial symmerical lens disorion dc... correcion o he camera consan. Eq. (9) shows he inegraion of he Eq. (8) ino Eq. (6). M X X c ex Y Y = RZ ( θ dθ ) RY ( ly) RX ( lx) + ey Z Z y dy ez w, ϕ, k λ (9) For non-saionary sysemaic errors he reader is referred o Amiri Parian 7 and Amiri Parian & Gruen 1. (a) (b) (c) Fig. 4. Addiional parameers for he configuraion of he Linear Array wih respec o he urnable coordinae sysem. (a) Eccenriciies, (b) inclinaion of he Linear Array, (c) il of he Linear Array wih respec o he roaion axis. 5

6 4 A Sensor Model wih Addiional Parameers for Panoramic Cameras A sensor model wih addiional parameers is based on an observaion equaions model in which each single observaion is expressed as an explici funcion of unknown parameers. In he firs sep, for he sake of simpliciy, Eq. (3) is considered. From his equaion, wih he aim of consrucing he funcional model, Eq. (1) is consruced (Amiri Parian, 7) as: 1 Y θ = an ( ) X c Z y = X + Y (1) The observaion equaions model of Eq. (9) is obained similarly o Eq. (1). Eq. (11) shows his model afer simplificaions. and wih ( θ, y) of Eq. (4): Y ~ 1 θ = an ( ~ ) θ X ~ c Z y = ~ ~ X + Y (11) and ~ X ~ Y = M ~ Z X X ex, ϕ, k Y Y RZ ( θ ey (1) Z Z ez w ) dθ dy = dy dc dc θ dy θdp θlx θ x ly θlens θex θey dy dpx lx ly lens ex ey (13) Noe ha ez is a consan value and is equal o he heigh of he projecion cener from he urnable. The erms of Eq. (13) are expanded in Eqs. (14) and (15) according o and dθ dc dy ly dθ dp = j dp x x 1 y dθlx = an ( sin( lx)) c lens dydp x = dyξ = dyex = dyey = y dy dc = dc c dy dy = y c cos( lx) dy lx = y (1 ) c + y sin( lx) sin( ly) ( y + c ) dy ly = c cos( ly) + y sin( ly) 3 = y (k +k y ) dy lens 1 ex ey = (14) (15) 6

7 5 Sensor Self-Calibraion and Accuracy Tess The camera calibraion was performed hrough self-calibraion using he mahemaical model developed in he previous secion. The self-calibraion was performed wih block riangulaion. This secion repors he resul of FODIS panoramic camera calibraion hrough self-calibraion. 5.1 Panoramic Tesfield of Conrol Poins FODIS panoramic esfield of conrol poins wih a 36 horizonal field of view were used for self-calibraion and accuracy esing. The conrol poins are he coded arges. The bases of arges are whie and he conrass are black. FODIS panoramic esfield consiss of more han 5 conrol poins which have been measured wih an independen measuremen. The poin posiioning accuracy of he esfield is in average.5mm. 5. Resuls of Camera calibraion and Accuracy Tess The goal is o do he calibraion and accuracy esing of he enire FODIS sysem. Several panorama images were acquired from his esfield. Fig. 5 shows he ypical configuraion of he camera saions for camera calibraion. Image and phoogrammeric processing were performed wih FODIS Measure3D sofware. Phoogrammeric compuaions were performed wih he free nework adjusmen. The free nework adjusmen compues he overall geomery of he nework by processing only he measuremens. I needs a leas one known disance beween wo poins for scale he nework of measuremen. In hese cases, scale was recovered from he size of coded arges. Fig. 5. A ypical camera saions for panoramic camera calibraion. As he resuls he RMS of he image poin residuals is.5 pixel and.19 pixel along column and row axes. The RMS error of posiioning a check poins is.7 mm projecing i a disance of 1 meers i would be equal o 1 mm accuracy. 6 Applicaions In he field of surveying and mapping, accuracy and reliabiliy of measuremen ools are key o success. A new sysem needs validaion and esing under real environmenal condiions. To his end, he FODIS sysem was esed in he field, comparing is workflow, produciviy and accuracy o esablished measuremen sysems such as oal saions and laser scanners. Under he leadership of he Japanese Associaion of Surveyors, he new FODIS Measure3D sysem was benchmarked wih a oal saion and a single-frame camera. One es was performed oudoor for accuracy and wo ess indoor for produciviy and sysem performance. The FODIS Measure3D sysem proved o be a viable and accurae alernaive o oal saions, especially in siuaions where he objec o be measured can be viewed from differen perspecives. Wih regards o produciviy (ime spen in he field and in he office) and in auomaic daa acquisiion/sorage, FODIS Measure3D has clear benefis (Amiri Parian, 1). In anoher pracical projec he FODIS sysem measured he dimensions of glass domes a Zurich s Boanical Gardens. This indoor assignmen proved o be challenging for oher mapping insrumens because of many reflecions, caused by he glass and meal srucure, and occlusions due o he confined inerior. To deal wih 7

8 his complex environmen more effecively, FODIS coded arges were aached o he dome srucure, hus allowing auomaed image orienaion and maximum accuracy for he final 3D measuremens. Here again, he FODIS Measure3D soluion se a new benchmark in produciviy, as he 3D projec was finished in less han a day (Fig. 6). For more informaion please see Fig. 6. 3D measuremen and documenaion wih FODIS Measure3D of he Boanical Garden of Zurich. 7 Conclusions FODIS Measure3D is a srong alernaive especially for indoor measuremens where rapid, accurae and selecive poin measuremens are required. Furhermore, because he FODIS sysem provides a panoramic view and rich informaion conen in resoluion, i deals wih complexiy more efficienly han oher measuremen ools. A he same ime, he FODIS sysem has he advanage of he image wih embedded measuremens, which allows easy sharing and visualizaion of resuls on new media such as inrane and he web. I is an accurae ool wih 1mm in 1meers accuracy. References Amiri Parian, J., 7. Sensor Modeling, Camera Calibraion and Poin Posiioning wih Terresrial Panoramic Cameras. PhD disseraion, ETH No Amiri Parian, J., Gruen, A., 1. Sensor Modeling, Self-Calibraion and Accuracy Tesing of Panoramic Cameras and Laser Scanners. ISPRS Journal of Phoogrammery and Remoe Sensing, 65(1), January 1, pp Amiri Parian, J., 1. Panorama Phoogrammery: Accurae 3D Documenaion wih High Resoluion 36 Panoramic Images, GIM Inernaional Magazine, February 1. Anipov, I. T., Kivaev A. I., Panoramic Phoographs in Close Range Phoogrammery. Inernaional Archives of Phoogrammery and Remoe Sensing, XXV(A5), Comission V, pp Baker, S., Nayar, S. K., A Theory of Caadiopric Image Formaion. IEEE Proceedings of ICCV, pp Boul, T Remoe realiy via omnidirecional imaging. Proceedings of DARPA Image Undersanding Workshop. Brown, D.C., The bundle adjusmen progress and prospecs. Inernaional Archive of Phoogrammery, Helsinki, Finland, 1(B3), 33 p. Colcord, J.E., Using Fish-Eye Lens for GPS Sie Reconnaissance. Journal of Surveying Engineering, 115(3),

9 Harley, R., Phoogrammeric Techniques for Panoramic Camera. SPIE Proceedings, Vol. 1944, Inegraing Phoogram- meric Techniques wih Scene Analysis and Machine Vision. Orlando, USA, pp Heikkinen, J., 5. The Circular Imaging Block in Close-Range Phoogrammery. Insiue of Phoogrammery and Remoe Sensing, Helsinki Universiy of Technology, Docoral Disseraion. Herber, T.J., Area Projecions of Fisheye Phoographic Lenses. Agriculural and Fores Merology (39), Heuvel, F.A. van den, Verwaal, R. and Beers, B., 6. Calibraion of Fisheye Camera Sysems and he Reducion of Chromaic Aberraion. Inernaional Archives of Phoogrammery and Remoe Sensing, 36(5), Dresden, Germany, 6 p. (on CD ROM). Kukko, A., 4. A New Mehod for Perspecive Cenre Alignmen for Spherical Panoramic Imaging, The Phoogrammeric Journal of Finland, 19(1), Luhmann, T., Tecklenburg, W.,. Bundle Orienaion and 3-D Objec Reconsrucion from Muliple-Saion Panoramic Imagery. Inernaional Archives of Phoogrammery and Remoe Sensing, 34(5), Corfu, Greece, pp Luhmann, T., Tecklenburg, W., 4. 3-D Objec Reconsrucion from Muliple-Saion Panorama Imagery. Inernaional Archives of Phoogrammery and Remoe Sensing, 34(5/W16), 8 p. (on CDROM). Nielsen, F., 5. Surround video: a mulihead camera approach. The visual compuer, 1(1-), 9-13 Ollis, M., Herman, H. and Singh, S., Analysis and Design of Panoramic Sereo Vision Using Equi-Angular Pixel Cameras. The Roboics Insiue Carnegie Mellon Universiy, CMU-RI-TR-99-4, Pisburgh, PA 1513, 43 p. Pesa, E., Kouroupis, S. and Karras, G.E., 1. Insering he pas in video sequences. Inernaional Archives of Phoogrammery and Remoe Sensing, 34(5C7), pp Pöninen, P., On he Creaion of Panoramic Images from Image Sequences, Phoogrammeric Journal of Finland, 16(), Rich, P.M., 199. Characerizing Plan Canopies wih hemispherical Phoographs. Remoe Sensing Reviews, 5(1), Schwalbe, E., 5. Geomeric modeling and calibraion of fisheye lens camera sysems. Inernaional Archives of Phoogrammery and Remoe Sensing, 36(5/W8), 6 p. (on CDROM). Shum, H., Szeliski, R., Panoramic Image Mosaics. Microsof Research, Technical Repor, MSR-TR-97-3, 5 p. Svoboda, T., Pajdla, T. and Hlavac, V., Cenral Panoramic Cameras: Geomery and Design. Research repor K335/97/147, Czech Technical Universiy, Faculy of Elecrical Engineering, Cener for Machine Percepion, 16 p. 9