Geometrical Design Concept for Panoramic 3D Video Acquisition

Transcription

1 Gomtrical Dsign Concpt for Panoramic 3D Vido Acquisition O. Schrr, P. Kauff, P. Eisrt, C. Wissig, J.-C. Rosnthal, Fraunhofr Hinrich-Hrtz-Institut, Brlin, Grmany ABSTRACT papr prsnts a nw gomtrical concpt of an omnidirctional and omni-stroscopic multi-camra systm. With such a systm, 3D vido panoramas can b capturd and displayd at cylindrical 3D projction systms. prsntd concpt can b considrd as an approximat solution of wll-known concntric mosaics that ar dfind as a sub-st of plnoptic function. papr discusss gomtrical rlationship to idal cas, whil taking practical constraints for a ral-working acquisition st-up into account. Indx Trms Panoramic vido, stro, acquisition 1. INTRODUCTION immrsiv snsation by panoramic imaging lasts back to rnaissanc paintrs. first xprimnts with moving panoramic imags hav bn prsntd bginning of last cntury. For instanc, Cinrama systm, an immrsiv 360 projction has alrady bn prsntd at lgndary Millnium World Exposition, 1900 in Paris [1]. Sinc n, a larg varity of or systms hav bn dvlopd targting ntrtainmnt markt as wll as training cntrs or vnt and xhibition tchnology []. In contrast to that, provision of panoramic vido supporting stroscopic 3D is much mor challnging and a stadily difficult task. Whil panoramic 3D projction is possibl using today s projction tchniqus from 3D cinma, acquisition of 3D vido panoramas is a widly unsolvd problm and is rfor invstigatd in this papr. Capturing a still panoramic D imag is possibl by simply rotating a camra, warping and n stitching imags togr [3]. Evn digital consumr camras hav built-in this fatur to crat own panoramas. Mor difficult is acquisition of panoramic D vido, which rquirs a spcial camra arrangmnt. A common solution is to us multipl camras, whr individual singl camras look into diffrnt dirctions such that rsulting imags can b stitchd samlssly to larg panoramic viws. First systms applying multipl camras and s to achiv full surround captur with high imag rsolution hav alrady bn usd in 60s by Ub Iwrks for Disny m park productions [4]. Sinc n, many -basd systm approachs hav bn proposd (.g. [5]). Or approachs plac a hypr- or parabolic in front of a singl camra to captur panoramic viws [6] with disadvantag of having a much lowr rsolution and plnty of distortions. Today, advancs and ongoing miniaturization of digital vido camras nabls mor compact systms and svral commrcial companis offr omni-dirctional camras for a wid rang of applications [7]. Good ovrviw about diffrnt approachs on panoramic vido acquisition is givn in [8]. If now panoramic 3D acquisition coms into play, situation is mor difficult. acquisition of static omnistro panoramas has alrady bn invstigatd sinc mor than 15 yars. A nic ovrviw on major principls can b found in [9]. basic ida is to mount camras on a rotating bar. From litratur, this concpt is also known as concntric mosaics [3], a spcial vrsion of plnoptic function [10]. In this papr, concpt of concntric mosaics is xtndd towards vido acquisition. It prsnts a 3D panoramic vido camra that optimizs a trad-off btwn contradicting rquirmnts on an adquat stro imprssion, sufficintly ovrlapping viws and corrct positioning of focal points for accurat stitching. In nxt sction, major principls and gomtrical aspcts of omni-dirctional and omni-stroscopic systms for static scn ar discussd. Sction 3 n prsnts a dsign of an omni-stroscopic vido acquisition systm that can b considrd as an approximat solution to ory of concntric mosaics. Sction 4 shows first rsults of proof of concpt. A conclusion nds papr.. THEORETICAL BACKGROUND Sinc many yars, acquisition of panoramic imags is a wll-known and alrady solvd problm of computr vision. As known from projctiv gomtry, an rror-fr captur of panoramic D imags rquirs that focal points of multipl camra viws coincid in a common point and look in diffrnt dirctions. Usually, this condition is achivd by rotating a singl camra at a tripod with a rvolving camra had. In this idal cas, singl imags can n b stitchd to a panoramic D imag without parallax rrors for arbitrary scns covring ntir dpth rang from zro to infinity. xtnsion of this D cas towards 3D is calld omni-stroscopic imagry. common approach is to

2 mount on or mor camras, looking ir outwards or in tangntial dirction, on a rotating bar (s Fig. 1) [9]. For s approachs it is sufficint to us so-calld slit- camras, i.. camras that captur on column only. In a r- binning procss, columns from slit camras ar usd to gnrat multipl prspctivs. column of on slit camra at ach angular incrmnt contributs to on panoramic viw such thatt at last two slit camras ar ndd to provid stro panoramas. For practical scnarios, a prspctiv camra is oftn usd, again mountd at nd of rotating bar looking ir in tangntial or normal dirction. For cration of a stro panorama two rquird columns can b takn from imag snsor whr distanc btwn two slctd columns again dfins baslin. two approachs ar only suitabl for capturing static scns, but distinctionn btwn two catgoris swing imaging with radial camra orintation and concntric mosaics with tangntial orintation hlps to motivat undrlying concpt of an omni-stroscopic vido systm as xplaind in nxt sction. C 0 of adjacnt camras (s paramtr S in Fig. 3 (lft) is much smallr than stro baslin B. Howvr, this can only b achivd by xtrmly small camras. Not that B is usually in a rang of 6 cm. Hnc, vn if ratio B/S can b rducd to 6, width of camras must b in rang of about 1cm. Clarly, r is nir a HD camra nor a high-quality lns of this small siz availabl. In contrast, if B is qual to or vn smallr than S, systm runs in a fundamntal conflict that can bst b xplaind by a star-lik arrangmnt as in Fig. 3 (right). Fig.. lft) Optimal camra arrangmnt; right) star-lik approach. B B S C 0 swing imag SI 1 S 1/ B 1 Fig. 1. Camra mounting and rbinning for swing imag. For vido, prvious approachs on omni-dirctional or omni-stroscopic imagry ar impractical du to nd of multipl camras for simultanous capturing on on hand and physical dimnsions of ach camra on or hand. For instanc, rquirmnt of omni-dirctional imagry, that all optical cntrs hav to coincid in on common point, is difficult to rspct in vido (s Fig., lft). For illustration rasons, optical cntrs ar drawn in front of lns in Fig. (lft), but y ar considrd bing insid lns. only solution to mt this rquirmnt prfctly is to us rigs by which virtual points of optical cntrs coincid bhind. Anor but lss prfct solution is to captur vido panoramas with star-lik approach from Fig. (right). In this cas, focal points of all camras ar locatd on a common circl, whil optical axs ar prpndicular to arc. Howvr, xistnc of a non-zro parallax angl dos not allow samlss stitching in cas of clos objcts in ovrlap ara. xtnsion of omni-stroscopic imagry towards vido is vn mor complicatd. This spcially holds for swing imag approach from Fig. 1 with a radial camra orintation. This approach supposs that distanc l ni swing imag SI n Fig. 3. lft) arrangmnt with small micro HD camras; right) star- lik arrangmnt for a panoramic 3D camra stup. As S is largr than B, parallax rror is largr than stro parallax, if sam nar and far objcts ar also prsnt in ovrlap ara. Hnc, if visibl parallax rrors wantd to b avoidd, stro ffct is lost. abov considrations also hold for concntric mosaics with a tangntial orintation. Howvr, in contrast to radial orintation of swing imags, it can b implmntd by using a rig to rduc ratio B/S to a rasonabl valu. rlatd concpt will b discussd in nxt sction. 3. DESIGN OF AN OMNI-STEREO VIDEO CAMERA In this sction, basic ida from concntric mosaicss is furr laboratd towards a practical and ralistic acquisition systm for omni-stro that allows us to bring optical cntrs of all camras clos to ach or (s drawing in Fig. 4, vido. dsign is basd on a -rig lft). In this xampl, thr stro camras ar placd in front of thr s in a way that virtual camra pairs cross at radial cntr bhind s. gomtrical arrangmnt of virtual focal points bhind is shown in Fig. 4 (right). It dpicts a sctional drawing at horizontal plan that cuts pyramid at points, whr optical axs of ral camras intrsct surfacs. In Fig. 4, virtual focal

3 points of sam stro pair ar connctd by solid bold lins (baslin ). Black dots rlat to lft camras L 1, L and L 3, whras gry dots rfr to right camras R 1, R and R 3. As shown by dashd lins, fild of viw of ach camra is framd by opning angl α of rlatd sgmnt. stro camras ar tod-in such that y convrg at r surfac (i.. ach camra L i and R i of sam stro pair looks through a window givn by surfac of sgmnt i). In givn xampl, opning angl is α=60 such that all thr camra pairs covr 180 in total. According to this drawing, two major problms can b idntifid, so-calld stro-gap and stitching-gap. rasoning for both problms and approachs to minimiz m ar prsntd in following sctions. Fig. 4. Stro arrangmnt using a rig. (lft); sctional drawing showing virtual stro pairs bhind s (right) stro-gap stro-gap is shown in Fig. 4 (right) at r dg btwn sgmnt and 3 in dark gry. In this viwing ara, stro information dos not com from rgular stro pairs lik L /R or L 3 /R 3 in adjacnt sgmnts and 3, rspctivly, but from a mixd stro pair L /R 3 from nighboring sgmnts and 3. Hnc, a basic constraint for samlss stro ovr ntir panoramic viw is that baslin B m of such mixd stro pairs (s dashd bold lins in Fig. 4 (right) must b qual to baslin of rgular stro pairs. Howvr, a simpl calculation using Euclidian distanc btwn focal points of L i and R i+1 shows that this constraint is not rspctd by gomtrical rlations in Fig. 4 (right). Hnc, w obtain following rlation btwn B m and : B m ( 1 cos( ))/ stitching gaps sgmnt 1 α baslin Br of rgular stro cos( ) sgmnt Eq. (1) dnots that B m and ar only idntical, if α is qual to zro or at last sufficintly small. This limit cas rfrs to ory of concntric mosaics. In practical implmntations with xisting high quality vido camras, howvr, two baslin trms may diffr considrably. nxt sction will show that this systmatical rror can b compnsatd by an off-cntr shift of stro pairs. L L 3 Z R 1 R stro gap sgmnt 3 L1 R 3 baslin Bm of mixd stro radial cntr surfac X (1) 3. radial off-cntr shift systmatical rror of stro gap can b compnsatd by shifting virtual stro pairs in radial dirction out of cntr. In practic, it can b achivd by moving ral stro camras towards, rsulting in an off-cntr shift of rlatd virtual stro systm as shown in Fig. 5 (right). corrsponding arrangmnt of virtual focal points bhind s is dpictd in Fig. 5 (lft). sgmnt 1 baslin Br of rgular stro pair Fig. 5. Virtual stro pairs with off-cntr shift (lft), sid viw of off-cntr shift by moving ral camra towards (right). B m Forcing abov mntiond constraint B m =, rsults in following xprssion for off-cntr shift : Tabl 1 shows valus of, in dpndnc of opning angl α of sgmnts. Not that for α=0, limit cas of concntric mosaics rsults in =0. L r L3 Z 1 cos tan 4 Tabl 1. Off-cntr shifts forcing quality of and B m B B 0.0 B B B B B 0.07 B 3.3 stitching-gap L 1 radial cntr R3 baslin B m of mixd stro pair 4 R 1 R Stitching-gaps ar shown in Fig. 4 (right) at dg btwn sgmnts 1 and in light gry. In this camra arrangmnt, optical rays from virtual focal points L 1 and L (or R 1 and R, rspctivly) throughh dg divrg outsid s and, as a consqunc, stitching is not possibl du to missing ovrlap btwn two rlatd camra imags. This situation changs in cas of off-cntrwhr optical rays divrg or convrg outsid arrangmnt in Fig. 5 (lft). Hr, fact Z Y Y off-cntdistanc dg C D tangntial plan sgmnt 3 1 cos sin () Euclidian distanc can b calculatd by considring mixd stro pair L /R 3, but now by taking into account off-cntfollowing rlation for B m and shift. In analogy to Eq. (1) w obtain : X virtual optical cntr (3)

4 rig dpnds on distanc of surfacs from radial cntr. This can asily b shown by using coordinat systm X Y Z in Fig. 5 (lft). Compard to original coordinat systm XYZ, it has bn rotatd clockwis by α/ around radial symmtry axis and, thus, axis Z passss through dg btwn sgmnts and 3. Hnc, abov dg is locatd at C D=(0,0,D) whr D dnots distanc of dg from radial cntr. To achiv ovrlap at stitching bordrs, D should b largr than D min. A typical valu is D=D min. minimal distanc of s rlats n to baslin as follows: D D min with This rlation is an important sid-condition for dsigning practicabl rigs bcaus siz of rig also incrass with D min. Tabl shows som possibl valus for D min dpnding on sgmnt angl α. calculations tak into account that baslin of a stro systm of =7cm is worst-cas situation for dsigning distanc. Not that distanc bcoms largr than on mtr for angls lowr than 16. Tabl. Minimal distanc D min rlatd to sgmnt angl α 3 D min 535cm 3.4 stitching rror Using positions of virtual focal points L and L 3 in coordinat systm X Y Z, following xprssions can b dfind for intr-focal distanc btwn L and L 3 : X B tan r 4 sin Br ( 1 cos( )), Y 0, Z sin (4) It is obvious that s intr-focal distancs caus parallax rrors whil stitching two imags from rlatd virtual camras. parallax rrors can b calculatd by following xprssions using pixl mtric of L as rfrnc: d u F X Z min 1 variabls u L and v L dnot horizontal and vrtical componnts of a cntrd coordinat systm in imag plan of L, whras F rprsntss focal lngth of usd camras. u L can b calculatd by following rlation: u F L D / B D min co cm 133cm u L Z Z min Z Z min d, 1 tan( / 4) ot( / ) cm v v 66cm Z L Z min Z 1 Z min horizontal parallax rror Δd u in Eq. (5) consists of two trms, on is drivn by horizontal intr-focal distanc ΔX and or by Z-diffrnc ΔZ of cm 43cm (3) 45 34cm (5) (6) two virtual focal points. s two trms can compnsat ach or. A full compnsation occurs, if D=D min holds. In this hard-cut situation, points L, L 3 and C D li at on common optical ray and, thus, horizontal parallax rror disappars. Tabl 3 shows rsiduals for D=DD min dpnding on sgmnt angl α. A usual valu =6 cm has bn slctd for stro baslin in this calculation. distanc of nar objct has bn st to Z min =m. Hnc, a dpth rang from m to infinity is allowd for stitching ara. Furrmor, assuming that horizontal fild of viw of usd camras quals to sgmnt angl α, focal lngth F is givn by F= =540pl/tan(α/ /). This assumption rfrs to a rig from Fig. 4 (lft) that uss HD camras in portrait format to allow small sgmnt angls α. Hnc, u L3 rangs from -540 pl to +540 pl, and, accordingly, v L3 from -960 pl to +960 pl. Tabl 3. Horizontal stitching rror vrsus sgmnt angl α Δd u [pl] 4. Proof of concpt To proof concpt, systm in Fig. 5 has bn invstigatd by computr simulations as wll as by a prototyp rig from Fig. 4 (lft). In this contxt, Fig. 6 (lft) shows st-up for CGI xprimnts with arrangmnt of virtual focal points according to Fig. 5 (lft). simulations wr basd on six r sgmnts with α=30 covring a panoramaa of 180 in total. six crossing off-cntrd stro systms ar shown in middl of drawing. corrsponding six stro viws hav bn capturd for diffrnt CGI scns. curvd panoramic scrn indicats whr six stro viws ar finally r-rndrd. Fig. 7 shows final rsult for an xampl scn. clos-up viws show dtails of a critical rgion in stitching ara. A first prototyp implmntation of a rlatd rig is shown in Fig. 6 (right). A sgmnt angl of 4 has bn slctd for this implmntation. This rprsnts bst trad-off btwn siz, rduction of stitching rrors and good stro quality. To achiv a prcis calibration camras ar mountd on mchanical slidrs, which can b adjustd with micro-mtr scrws in any dirction and orintation. Fig. 6. St-up for simulations on CGI basis (lft), prototyp of an omni-stro rig (right).

5 Fig. 8. Stro contnt capturd by prototyp: lft and right viw from rgular stro (lft) and from mixd stro pair (right). Fig. 7. Rsult of simulatd stro panoramaa with lft and right viw on top, clos-up of critical rgion in stitching ara (bottom) usd HD camras ar mountd in portrait format and spatial rsolution of a rsulting panorama is 7000 by 000 pixls for 180. Fig. 8 shows imags from rgular and mixd stro viws (compar to Fig. 5) that hav bn capturd during tst shoots. shard rctangls with solid whit lins show ffctiv imag bordrs prund by sgmnts. sharing is givn by fact that camras ar not positiond in cntr of, but ar movd horizontally by half a baslin to lft or right, rspctivly, and ar additionally tod-in to compnsat shift. Furrmor, lft picturs in Fig. 8 show that, in contrast to standard stro applications, ovrlap btwn two viws of a rgular stro systm is considrably limitd. rmaining stro information has to b takn from mixd stro pairs. By using an anaglyph ovrlay rprsntation, Fig. 9 shows how rgions from rgular and mixd stro pairs ar assmbld to an ntir stro panorama. 5. Conclusion A dsign of an omni-stro vido acquisition systm has bn prsntd that offrs an approximat solution to ory of concntric mosaics. Svral important challngs hav bn discussd such as stro-gap and stitching-gap. By introducing a radial off cntr shift of stro systms mountd in a rig, it is possibl to achiv a ralizabl solution for such a novl panoramic stro camra, which offrs a minimal stitching rror whil kping a significant amount of parallax. A first prototyp of such a camra has provn corrctnss of ortical drivations. Rfrncs Fig. 9. Composition of 3D vido panorama. [1] Australian Cntr for Moving Imag, Advnturs in Cybrsound; Cyclorama, Cinorama, Maroramaa and Myrioama, ACMI, 011, [] HPC Markt Watch, Sattl Cinrama Grand Ropning, 011, [3] H.-Y. Shum, L.-W. H, Rndring with Concntric Mosaics, Proc. SIGGRAPH 99, ACM, Los Angls, hpcwir/nws/ rad?guid= &channlid=3197. [4] U. Iwrks, Panoramic Motion Pictur Camra Arrangmnt, Canadian Patnt Publication, no. CA , [5] Majumdr, M. Gopi, B. Sals, H. Fuchs, Immrsiv tlconfrncing: A nw algorithm to gnrat samlss panoramic vido imagry, Proc. of 7th ACM Intrnational Confrnc on Multimdia, pp , [6] S. Bakr, S. Nayar, A ory of singl-viwpoint catadioptric imag formation Int. Journal of Computr Vision, 35: , [7] Full Viw, FC-1005 Camra & FC-110 Camra, [8] K. A. Tan, H. Hua, N. Ahuja, Multiviw Panoramic Camras Using Mirror Pyramids, Trans. on Pattrn Analysis and Machin Intllignc, Vol. 6, no7, pp , 004. [9] S. Plg, M. Bn-Ezra, Y. Pritch, "Omnistro: panoramic stro imaging", IEEE Trans. on Pattrn Analysis and Machin Intllignc, Vol.3, No.3, pp.79-90, March 001. [10] E. H. Adlson, J. R Brgn, Plnoptic Function and Elmnts of Early Vision, Computational Modls of Visual Procssing (pp. 3-0). Cambridg, MA: MIT Prss, 1991.