IEEE Trasatio o Cosumer Eletrois, vol. 56, o., 010IEEE Trasatio o Cosumer Eletrois, vol. 56, o., 010 Fast Paorama Stithig for High-Quality Paorami Images o Mobile Phoes Yige Xiog ad Kari Pulli, Member, IEEE Abstrat This paper addresses the problem of reatig high-resolutio ad high-quality paorami images from log image sequees with very differet olors ad lumiae i soure images. A fast stithig approah is proposed for ombiig a set of soure images ito a paorami image usig little memory, ad implemeted o mobile phoes. I this approah, olor orretio redues olor differees of soure images ad balaes olors ad lumiae i the whole image sequee, dyami programmig fids optimal seams i overlappig areas betwee adjaet images ad merges them together, ad image bledig further smoothes olor trasitios ad hides visible seams ad stithig artifats. A sequetial paorama stithig proedure ostruts paorami images. The advatages ilude fast proessig speed usig dyami programmig for optimal seam fidig, reduig memory eeds by usig the sequetial paorama stithig, ad improved quality of image labelig ad bledig due to the use of olor orretio. The approah has bee tested with differet image sequees ad it works well o both idoor ad outdoor sees1. Idex Terms Mobile paorama, image stithig, fast labelig, image bledig. I. INTRODUCTION A paorami image has a wide field of view, muh wider tha is available o ormal ameras suh as those i mobile phoes. By stithig together a sequee of overlappig ormal images, we a reate a paorami image. Image stithig is a very importat step i reatig paoramas. A simple pastig of overlappig images ito the fial paorama produes visible seams due to hages of see illumiatio ad amera resposes, or spatial aligmet errors. The task of image stithig is to fid optimal seams i overlappig areas of soure images, merge them alog the seams, ad miimize mergig artifats. I this paper, we are reatig high-resolutio ad high-quality paorami images o mobile phoes, so that a user a apture a image sequee of a wide rage of sees with a amera phoe ad see a paorami image reated immediately o the phoe. A. Bakgroud Mobile phoes are ot oly effiiet ommuiatio tools, but also apable omputatioal devies equipped with highresolutio digital ameras, high-quality olor displays, ad GPU hardware. Appliatios suh as mobile augmeted 1 Yige Xiog ad Kari Pulli are with Nokia Researh Ceter, Palo Alto, CA 94304, USA (e-mail: yige.xiog@okia.om; kari.pulli@okia.om). reality, mobile loal searh, ad mobile image mathig ad reogitio used to oly work o desktop omputers, but a ow ru o mobile phoes. Here we are buildig paorama appliatios o these devies. A paorama ostrutio proess requires a lot of omputatio ad memory. Mobile phoes oly have limited resoures. It is eessary to develop effiiet stithig methods to fit mobile appliatios. B. Related Work There are two mai ategories of urret image stithig approahes: trasitio smoothig ad optimal seam fidig. Trasitio smoothig approahes redue olor differees betwee soure images to make seams ivisible ad remove stithig artifats. Alpha bledig [1] is a widely used simple ad fast trasitio smoothig approah, but it aot avoid ghostig problems aused by objet motio ad small spatial aligmet errors. Reetly, gradiet domai image bledig approahes [5]-[8] have bee applied to image stithig. These algorithms a redue olor differees ad smooth olor trasitios usig gradiet domai operatios, produig high-quality omposite images. Optimal seam fidig approahes [4], [9]-[1] searh for seams i overlappig areas alog paths where differees betwee soure images are miimal. The seams a be used to label eah output image pixel with the iput image that should otribute to it label whih iput image otributes to eah output pixel. The ombiatio of optimal seam fidig ad trasitio smoothig for image stithig has also bee used i paorama appliatios [4], [13], ad [15]. Soure images are ombied by ompositig alog optimal seams. If the seams ad stithig artifats are visible, trasitio smoothig is applied to redue olor differees to hide the artifats. Curret paorama stithig approahes ruig o amera phoes a be foud i [13], [], ad [3]. I [13], graph ut is used for fidig optimal seams to merge the soure images together ad Poisso bledig is used for smoothig olor trasitios. High-quality paorami images a be obtaied. However, omputatioal ad memory osts are high. I [] ad [3], soure images are stithed together with a proedure iludig olor orretio, seam fidig, ad simple badliear bledig. The stithig proess is simple. However, the quality of paorami images is ot high. There are several problems i this approah. Pixels are easy saturated i olor orretio. It does ot work well for soure images i very differet olors ad lumiae. The simple bad-liear bledig is ot suffiiet whe olor orretio a ot
remove olor differees effiietly, whih results i lowquality paorami images. Like other liear bledig, movig objets o the overlappig areas will ause ghostig artifats. All these problems are solved i our proposed approah. We have reated a fast image stithig approah that uses relatively little memory. It iludes olor orretio, image labelig, ad image bledig operatios. We perform olor orretio for all soure images to redue olor differees ad smoothe remaiig olor trasitios betwee adjaet images. Sie the RGB pixel values of iput images are gamma-orreted ad therefore o-liear, we alulate the olor averages used to fid olor orretio oeffiiets usig liearized RGB values. A global adjustmet proess is applied to redue magitude of average olor orretio to lower the hae of saturatig pixel values durig olor orretio. I the image labelig operatio, a error surfae is ostruted with squared differees betwee overlappig images. A lowost path is foud through the error surfae by dyami programmig ad used as a optimal seam to reate labelig. The overlappig images are merged together alog the optimal seam. Compared to the ommoly used graph ut method, the labelig proess is muh faster ad memory osumptio is muh lower. I order to further smoothe olor trasitios betwee adjaet soure images, we perform image bledig after the soure images are merged usig image labels. A simple liear bledig is used whe soure images are similar i olor ad lumiae. Whe the olors remai too differet, Poisso bledig hides visible seams. The use of olor orretio for the soure images a improve qualities of image labelig ad image bledig. It a also speed up the bledig proess. A sequetial paorama stithig proedure is reated with the fast image stithig approah. I this way, we a produe high-resolutio paorami images from large soure images with low omputatioal ad memory osts. C. Cotributios We (i) propose a fast paorama stithig approah with olor orretio, fast labelig, ad image bledig for reatig high-resolutio ad high-quality paorami images o mobile phoes; (ii) improve qualities of optimal seam fidig ad trasitio smoothig by ombiig olor orretio with image labelig ad image bledig; (iii) redue omputatio of the Poisso bledig proess with pre-smoothig olor differees of soure images; (iv) reate a sequetial image stithig proedure for mobile appliatios to quikly ostrut high-resolutio paoramas with log image sequees usig little memory; (v) preset various examples ad ompare performae with other approahes to demostrate advatages of the proposed approah; (vi) implemet it o mobile phoes. II. SUMMARY OF OUR APPROACH Fig. 1 shows the workflow of the fast paorama stithig proedure. We start with settig the stithig order (S 0, S 1,, S ) of the soure images by sortig their offsets with respet to the fial paorama. We alulate olor orretio oeffiiets for eah eighborig image pair i the liearized RGB olor spae for all soure images, ad the ompute a global adjustmet fator that redues umulative olor orretio ad the risk of saturatig olors. Next, we fid a image with more realisti olors i the image sequee, ad adjust the first image usig a hai of relative olor orretios, modified with the global orretio fator, so that the olors of the best image remai after orretio as they were. After alloatig memory for the fial paorami image I ad iitializig it with the first image S 0, we start to stith other soure images to the paorami image sequetially. Start Set a stithig order (S 0, S 1,, S ) for the soure images by sortig their offsets. Calulate olor orretio for eah soure image pair ad obtai olor oeffiiets for all soure images. Calulate global adjustmet fator for the olor orretio oeffiiets. Fid a image with best olors i the image sequee ad use it to orret the olors of the first image. Perform olor orretio for the first image S 0 relative to the best image, set it as the base image, ad put it ito the paorami image I. Load the ext soure image as the urret image S. Perform olor ad lumiae ompesatio for the urret image S with the ompesatio oeffiiets ad the global adjustmet fator. Determie the overlap betwee the urret paorami image I ad the urret image S. Compute a error surfae i the overlappig area. Fid a miimal ost path through the error surfae with dyami programmig ad use it as a optimal seam for labelig. Cut the overlappig images alog the optimal seam ad merge them. o Perform image bledig ad update the urret paorami image I with the bledig result. Are all soure images doe? yes Obtai the fial paorami image I. Stop Fig. 1. Workflow of the fast paorama stithig approah. We load the ext soure image as the urret proessig image S ad perform olor orretio with the olor orretio oeffiiets ad the global adjustmet fator. I order to merge the urret image with the urret paorama, we extrat the overlappig area betwee these two images ad ompute a squared differee betwee the overlappig images as a error surfae. We fid a miimal ost path through the error surfae with dyami programmig. That path is used as a optimal seam to ut the overlappig images ad merge them together. We perform image bledig to further redue olor differees ad smooth olor trasitios betwee the
urret soure image S ad the paorami image I. With the bledig result, we a update the paorami image I. The proess is repeated for all soure images, util we obtai the fial paorami image. Ulike the global image stithig i [13], we do ot eed to keep all soure images i memory due to the sequetial stithig. The use of dyami programmig for optimal seam fidig allows image labelig muh faster tha usig graph ut. The ombiatio of olor orretio ad image bledig allows us to ostrut highquality paorami images. Although we desribe the approah usig the 1D stithig ase, i.e. ameras move horizotally or vertially, it has bee already exteded to 1D stithig, i.e. ameras move i ay arbitrary diretio ad soure images a be stithed together i ay arbitrary order. (a) (b) Fig.. Image stithig without ad with olor orretio III. COLOR AND LUMINANCE COMPENSATION We apture the images usig automated settigs for fous, exposure, ad white balae. As illumiatio hages aross the see, differet images have differet values for exposure ad white balae, leadig sometimes to large differees i olors i eighborig images. If o further olor proessig is doe, visible artifats may be reated i paorama stithig. Fig. (a) shows a example, where the upper row shows three soure images with differet olors. The bottom row shows a stithig result. I this ase, we a learly see olor differees ad seams betwee the soure images. It is eessary to perform olor orretio for the soure images to redue the differees ad improve the stithig quality. I order to better math the olors, we ompute light averages i the overlap area usig liearized RGB values istead of the default gamma-orreted RGB. I a image sequee S 0, S 1, S i, S, suppose S i-1 ad S i are adjaet images, ad o Si1 ad Sio are where image overlap. We ompute olor orretio oeffiiets for image S i by liearizig the gamma-orreted RGB values as ( P, i1( p)) p, i { R, G, B} ( i 1,,3,, ), (1) ( P ( p)) p, i where P,i-1 (p) is the olor value of pixel p i image S o i1 ; P,i (p) o is the olor value of pixel p i image S ; ad γ is a gamma i oeffiiet. Usually we set γ to.. For the first image S 0, we set α,0 to 1. To avoid saturatig olor values, we perform a global adjustmet for olor i the whole image sequee. We alulate a global adjustmet fator g for eah olor hael so that the overall adjustmets g α,i approximate 1 by solvig the least-squares equatio mi ( g, 1) { R, G, B}. () g 0 i i Equatio () is a quadrati futio i adjustmet g whih a be solved i losed form by settig the derivative to 0, g i0 i0, i, i { R, G, B} ( i 0,1,, ). (3) With the orretio oeffiiets α,i ad the global adjustmet fator g, we perform olor orretio for the whole image S i, 1/ P, i( p) ( g, i) P, i( p), { R, G, B} ( i 0,1,, ), (4) where P,i (p) is the olor value of pixel p i image S i i olor hael {R, G, B}. Sie the iput ad output values are gamma-orreted, we also gamma-orret adjustmets g α,i. As desribed before, we use a best image foud i the image sequee to orret olors for the first image. It is diffiult to automatially determie the image with the best olors, sie that is also partially a estheti judgmet all, ad ideally the user should selet the image whose olors she likes the best. As a heuristi, we selet the image with most similar meas i the R, G, ad B haels, usig the gray world assumptio ofte used i white balaig. Fig. (b) shows the results of olor orretio ad image stithig for the soure images show i Fig. (a). From the results we a see that olor orretio redues the olor differees so that hardly ay seam remais visible. There are two mai advatages i the way we do the olor orretio. Liearizig the light while alulatig the orretio fators mathes the olors better tha if the averages were alulated i gamma-orreted RGB, ad the global adjustmet for olor orretio oeffiiets atteuates the orretios, reduig aumulatio of orretios that may lead to olor saturatio. (a) (b) Fig. 3. Ghostig artifats aused by objet motio ad deghostig. IV. OPTIMAL SEAM FINDING AND IMAGE LABELING Objet motio ad spatial aligmet errors may ause ghostig artifats durig image stithig. Fig. 3 (a) shows a example where a perso was movig while the image sequee was aptured. From the stithig result we a see the ghostig problem aused by the motio. The objetive of optimal seam fidig is to fid seams i
the overlappig areas of soure images, reate labelig for all pixels i the omposite image, ad merge soure images alog the optimal seams. Sie eah pixel i the omposite image omes from oly oe soure image, the ghostig problems a be avoided. I mobile settigs, we wat a method that fids optimal seams quikly with usig little memory, so that it a be applied for reatig high-resolutio paorami images o mobile phoes. Like [11] ad [], we also use dyami programmig to fid optimal seams. We wat to merge the images o plaes where they differ the least. Suppose that abd is the overlappig area betwee the urret omposite image I ad the urret soure image S. I i o ad S i o are the overlappig images i the area abd of I ad S respetively. We ompute squared differees e betwee I i o ad S i o as a error surfae, o o e ( I S ). (5) We apply dyami programmig to fid a miimal ost path through this surfae. We sa the error surfae row by row ad ompute a umulative miimum squared differee E E( h, w) e( h, w) mi( E( h 1, w 1), (6) E( h 1, w), E( h 1, w 1)), where h =,, r ad w =,, are the idies of the rows ad olums of the error surfae, respetively. The optimal path m a be obtaied by traig bak the paths with a miimal ost from bottom to top. O the last row, the pixel with the miimum value is at the ed (h 0, w 0 ) of the optimal path. O the previous row, the miimum E(h 0 1,w), w{w 0 1,w 0,w 0 +1} deotes the positio (h 0 1,w) of the optimal path at this row. Similarly, we a follow the path up oe row at a time. Fig. 4 shows the proess of optimal seam fidig with dyami programmig. Fig. 4 (a) ad (b) are the overlappig areas of I ad S, respetively. The error surfae e show i Fig. 4 () is omputed as the squared differees betwee images Fig. 4 (a) ad (b). Usig that, the umulative miimum squared differee E is omputed ad is show i Fig. 4 (d). Fig. 4 (e) shows all possible paths. After traig bak with dyami programmig, we obtai the optimal path show i Fig. 4 (f), alog whih the two images i (a) ad (b) math best. We use that path as a optimal seam to reate labelig. We update the urret omposite image I by mergig the urret image S with the labelig iformatio ad otiue the labelig proess with the ext soure image. After all soure images are proessed, we obtai the fial omposite image. Sie optimal seams are used i the image stithig proess, the ghostig problems a be avoided. Fig. 3 (b) shows the result obtaied by image stithig with optimal seam fidig for the soure images show i Fig. 3 (a). From the result we a see that ghostig artifats i the overlappig area have bee removed by the optimal seam fidig proess. I this ase the paths resulted i a opy of the movig perso, other hoies for the path might result i the left, right, or either versio of the perso. I ay ase we avoided trasparet opies, or paths splittig the perso i two. Color orretio a improve quality of optimal seam fidig ad image labelig. We wat to fid a path where the images agree. This is more diffiult to do if the olors of the two images disagree as muh as i the sequee of Fig. 5. There is a movig objet (ar) i the see, ad we would like to fid a path that does ot iterset the ar, as the other images do ot otai it. However, o top left, where the images have ot bee olor-orreted, the miimum differee path goes through the ar. I top right, whe the olors have bee orreted before the path searh, the path avoids the ar, ad a osistet paorama ould be reated. Fig. 4. Proess of optimal seam fidig with dyami programmig. Fig. 5. Color-orretio improves the seam quality. Fig. 6. Simple liear bledig. V. TRANSITION SMOOTHING WITH IMAGE BLENDING Color orretio redues the differees betwee the images, whih makes bledig easier ad faster. I our fast paorama stithig approah, we have two image bledig proesses that a be used. A. Simple Liear Bledig For the soure images that are similar i olor ad lumiae after olor orretio, we perform a simple image bledig o a bad that is δ pixels wide o both sides of the seam, as show i Fig. 6. The ew olor value of pixel p i the overlappig area a be alulated by a weighted ombiatio of the orrespodig pixels P I (a) (b) () (d) (e) (f) I ( p) d 1 PI ( p) d PS p ( ) d1 d, (7), ew a m d p d 1 where d 1 ad d are distaes from pixel p to boudaries; P ( p) is the ew olor of pixel p; I, ew P I ( p) is the olor of pixel p i image I ; P S ( p) is the olor of pixel p i image S ; S
differet values of result i differet olor trasitios. Liear bledig is simple ad omputatioal ad memory osts are low. However, movig objets i the bledig bad areas will ause ghostig artifats. Furthermore whe soure images differ, liear bledig is ot eough to get rid of seams ad stithig artifats; more itesive bledig is required. B. Poisso Bledig Poisso bledig is a itesive image bledig approah that performs image bledig i the gradiet domai. I Poisso bledig, we reate a gradiet vetor field (G x, G y ) with gradiets of soure images usig the labelig obtaied usig optimal seams. I the sequetial image stithig proedure, the gradiet vetor field is opied from the urret soure image S, up util the seam betwee it ad the urret paorami image I (i Fig. 6 all the pixels of S to the right of the alulated seam). A divergee field div(g) is the omputed from the gradiet vetor field, G G x y div( G). (8) x y We use the divergee field as a guidae to ostrut a Poisso equatio I ( div( G), (9) where is the Laplaia operator I( I( I(. (10) x y I pratial implemetatio, we eed to use the disrete form of Equatio (9) I( x 1, I( x 1, I( y 1) I( y 1) 4 f ( Gx ( Gx( x 1, Gy ( (11) G y ( y 1) Equatio (11) is a liear partial differetial equatio, whih we solve by fixig the olors at the seam ad solvig ew olors I( over the gradiet field. We a solve the equatio usig a iterative ojugate gradiets solver. Fig. 7 shows a ompariso betwee the results reated by simple liear bledig ad Poisso bledig for the olororreted soure images show i Fig.. The upper figure shows the result usig simple liear bledig. A fait seam a still be see betwee the two soure images. However, o visible seam a be observed i the result reated by the Poisso bledig show o the bottom. By ompariso, the liear bledig is simple ad fast, but bledig quality is low. The Poisso bledig has higher quality; however it eeds more omputatio ad memory. While olor orretio was ruial for good quality i liear image bledig, it a also help to speed up the Poisso solver. Fig. 8 shows three soure images with very differet olors ad lumiae. The top row shows the results after 0 iteratios of Poisso solver, o the left startig from the origial iputs, o the right startig from olor-orreted iputs. Hudreds of further iteratios would be eeded to obtai omparable results without olor orretio. With loger sequees differees beome eve more prooued. Fig. 7. Results of liear bledig ad Poisso bledig. Fig. 8. Improved bledig quality ad speed with olor orretio. VI. IMPLEMENTATION A sequetial paorama stithig proedure is reated with the fast image stithig approah. We have two implemetatios for the proedure: keep the full resolutio paorami image i memory; reate a low-resolutio paorami image i memory for display ad save the full resolutio oe to disk blok by blok while it is reated. By ompariso, the previous oe is faster ad more oveiet for the viewig proess. It a keep stithig with frames as log as there is eough memory for the full resolutio paorami image, the urret soure image, ad some work arrays. The latter oe has o limitatio for the umber of frames as log as there is eough memory for the low-resolutio paorami image, the urret soure image, ad some work arrays. It eeds to re-load the full resolutio paorami image for viewig. By omparig with the global image stithig [13], both implemetatios use muh less memory. A omparig result is give i Setio VII.B. VII. EXAMPLES AND RESULT ANALYSIS We have implemeted the fast paorama stithig approah o mobile phoes for reatig high-resolutio ad highquality paorami images. We have tested it o both idoor ad outdoor sees ad obtaied good results. We preset examples for various sees, iludig log image sequees with soure images with very differet olors ad lumiae, ad ompare performae with other approahes to demostrate advatages of our paorama stithig i proessig speed ad memory osumptio. I this paper, the example appliatios ad results are obtaied o a mobile phoe with a 33 MHz proessor ad 18 MB RAM. It a also be ru o other mobile devies. I these appliatios, the size of soure images is 104 768. We have also applied it to larger soure images, with good results.
(a) Soure image sequee. (b) Optimal seam fidig. () Liear bledig alog with the seams. (d) Poisso bledig with 5 iteratios. (e) Poisso bledig with 150 iteratios. (f) Soure images after olor orretio. (g) Liear bledig for soure images after olor orretio. (h) Poisso bledig with 5 iteratios for soure images after olor orretio. Fig. 9. Appliatio to a log image sequee with very differet olors ad lumiae i soure images. A. Log Image Sequees with very Differet Colors Fig. 9 shows a example of a log image sequee with images that have very differet olors ad itesities. With the results of this example, we a also demostrate the performae of eah proess i the approah. Fig. 9 (a) shows the origial soure images i the image sequee. There are 13 soure images with very differet olors ad lumiae i the image sequee. While it is aptured, some objets move i the see. The differet olors ad lumiae betwee the soure images are aused by the use of automated settigs of the amera. Fig. 9 (b) shows the omposite image obtaied by optimal seam fidig. From the result we a see that the optimal seam fidig proess i our paorama stithig a fid the best way to label images ad merge them to a omposite image. Although there are movig objets i overlappig areas of soure images, there is o ghostig or blurrig problems i the omposite image. The use of dyami programmig for optimal seam fidig is oe of the mai reasos why the proposed paorama stithig works fast. Fig. 9 () shows the result reated by the simple liear bledig for the omposite image obtaied by optimal seam fidig. From the result we a see that the olor differees aross the optimal seams are redued to some extet. This meas that the simple liear bledig proess a smooth olor trasitios aross the seams. The proessig speed of the bledig is very fast. However, sie the soure images are very differet i olors ad lumiae, the olor differees i the whole omposite image still a be see. Other proessig is eeded to further redue the differees.
Fig. 10. Paorami image reated by the approah i []. Fig. 11. Compariso of olor orretio approahes i [] ad i the proposed fast paorama stithig approah. Fig. 9 (d) shows a result produed by Poisso bledig i whih the liear solver uses 5 iteratios. From the result we a see that the effet of the bledig is almost the same as the simple liear bledig show i Fig. 9 (). It is still far away from a satisfyig result. The olor differees i the omposite image a still be see learly. Fig. 9 (e) shows a result obtaied by Poisso bledig after 150 iteratios. The result is improved muh ompared to the result show i Fig. 9 (d). This meas that muh more omputatio is eeded to obtai a better result. However, the result is still ot satisfyig. We a still see olor disotiuity i the omposite image, espeially o the right side. Fig. 9 (f) shows the soure images after olor orretio. From the result we a see that the olor orretio proess a redue differees i olors ad lumiae betwee two images ad adjust olors globally i the whole image sequee. Although the origial soure images are very differet, the differees are smoothed after olor orretio. Also, there are o pixel saturatio artifats after the olor orretio proess. We a see that the performae of the olor orretio proess is very satisfyig. Fig. 9 (g) shows a result reated by the simple liear bledig proess after olor orretio. As we a see, ombiatio of olor orretio with the simple liear bledig a produe very satisfyig paorami images. Both proesses of olor orretio ad liear bledig are simple ad use little memory. The ombiatio is suitable for mobile implemetatio ad appliatios. Fig. 9 (h) shows a result produed by Poisso bledig with the soure images after olor orretio. We a see that there are o visible artifats. I this ase, Poisso bledig still uses 5 solver iteratios, however, the result is muh better tha i Fig. 9 (e), whih uses 150 iteratios for the soure images that are ot proessed by olor orretio. Agai, the olusio is that olor orretio a improve Poisso bledig quality ad speed up the proessig speed. The ombiatio makes Poisso bledig muh more suitable for mobile devies. From the evaluatio of this example we a see that eah proess of the proposed paorama stithig approah futios well. The approah a produe high-quality ad high-resolutio paorami images o mobile phoes. It a hadle soure images i log image sequees with very differet olors ad lumiae. B. Compariso with Other Approahes Fig. 10 shows a paorami image reated by the approah proposed i [] with soure images show i Fig. 9 (a). From the result we a see that olor differees ad seams betwee soure images a be see learly. Sie the soure images are very differet i olors ad lumiae, the olor orretio approah a ot remove the differees ompletely ad the simple bad-liear bledig a ot smooth the olor trasitios, so that a low-quality paorami image is obtaied. Atually, this is oe of the mai disadvatages of the paorama stithig approah i []. It a ot hadle log image sequees with soure images i very differet olors ad lumiae. O the other had, for same image sequee, the proposed fast paorama stithig approah a produe high-quality paorami images show i Fig. 9 (g) ad (h) due to better olor orretio ad image bledig proedures. I geeral, the proposed approah a hadle this kid of image sequees very effetively. Fig. 11 shows a ompariso of olor orretio results betwee the approah i [] ad the proposed approah. I this ase, there are 14 soure images with very differet olors ad lumiae show i Fig. 11 (bottom). Fig. 11 (top) shows the paorami image reated with the olor orretio i []. From the result we a see that the olor orretio approah does ot work well. The olor differees ould ot be removed. There is a mai problem i this result that a large part of the pixels are saturated after olor orretio. Most details suh as i the sky ad road i this result are lost. Fig. 11 (middle) shows the paorami image reated with the olor orretio i the proposed approah. From the result we a see that all details are kept ad pixels are ot saturated after olor orretio. Colors i the whole paorami image are very atural. Color trasitios are smoothed. The good olor orretio promises to obtai high-quality paorami images. Furthermore, Poisso bledig a further improve quality of the fial result.
Fig. 1. Paorami image produed by the fast paorama stithig with 7 soure images i a idoor see with movig objets. Fig. 13. Paorami image produed by the fast paorama stithig with 8 soure images i a outdoor see with movig objets. Fig. 14. A paorami image reated by our fast paorama stithig with 17 104 768 soure images o mobile phoes. TABLE I stithig is show o the top of COMPARISON OF MEMORY CONSUMPTION OF IMAGE STITCHING IN [13] AND THE PROPOSED APPROACH A B C 11.4 10.1 3 13.3 10.8 4 15.1 11.4 5 16.9 1. 6 19 13 7 0.5 13.6 8 1.4 13.7 9 3.4 14.7 10 4. 15.0 We have ompared memory osumptios betwee the proposed sequetial paorama stithig with the global paorama stithig i [13] whih eeds to keep all soure images i memory for global optimizatio durig image stithig. The result is show i Table I. I the table, row A meas the umber of soure images used, B shows the memory osumptio usig global paorama stithig, ad C shows the memory osumptio of sequetial paorama stithig. The uit of memory osumptio is MB. From the results we a see that the more soure images i paorama stithig, the more memory the sequetial stithig saves. I this ompariso, both implemetatios keep full paorami images i memory durig paorama stithig. C. Image Stithig of a Idoor See Fig. 1 shows a example of a idoor see with 7 soure images. The result reated by the proposed fast paorama the figure. The stithig proess takes 19 seods ad the graph ut approah [13] takes 67 seods, about 35 times loger. We a also otie some other aspets. Although the people i the see are movig durig the apture of the sequee, the stithig proess fids good seams ad avoids ghostig ad blurrig problems aused by these movig objets. Although there are some differees of the soure images i olors ad lumiae, they are removed after olor orretio ad image bledig i the resultig paorami image. The olor trasitios are smoothed i the fial results. D. Image Stithig of a Outdoor See The outdoor image sequee i Fig. 13 (bottom) iludes eight soure images that are stithed together to reate a paorami image. Fig. 13 (top) shows the result reated by the fast paorama stithig approah. The stithig takes 3 seods ad the graph ut [13] takes 756 seods, about 3 times loger. Also here we fid good seams, ad seletig sigle iput image per output pixel helps to avoid ghostig problems due to objet motio.
E. Creatig 360 Paoramas with very Log Image Sequees Fig. 14 shows a 360 o paorami image. The top shows the reated paorami image ad the bottom shows the 17 soure images. From the image sequee we a see that the soure images are very differet i olors ad lumiae ad there are some movig objets i the see while the image sequee is aptured. However, the approah still produes a high-quality paorami image. The paorama stithig proess takes 34 seods ad agai is muh faster tha the ommoly used graph ut approah. Aordig to our tests, the loger the image sequees, the greater the speed advatage of the fast paorama stithig is. Sie the fast paorama stithig is a sequetial image stithig proedure, it oly eeds to keep the paorami image ad the urret soure image i memory. As log as there is eough memory for the fial paorama ad the urret soure image, the approah does ot are how may soure images are proessed. Fast proessig speed ad low memory osumptio are the mai advatages of the proposed approah, both very importat i a mobile implemetatio. Our approah has bee tested with may image sequees with differet ases o differet types of mobile phoes ad it performs well. VIII. DISCUSSION AND CONCLUSIONS A fast paorama stithig approah that uses little memory is developed ad implemeted o mobile phoes for reatig high-resolutio ad high-quality paorami images. It has bee tested with differet image sequees aptured uder differet lightig oditios. It is muh faster tha the graph ut approah. The fast stithig approah a be applied to reate high-resolutio paorami images with large soure images as log as the system has eough memory for the fial paorama ad the urret proessig soure image. The fast speed of the proposed approah is maily due to the fast labelig approah reated with dyami programmig. It is very simple to implemet. After the overlap betwee two images is loated, a error surfae is reated by omputig the squared differees of olors i the overlappig area. A lowost path where the image values agree is foud by dyami programmig. The path is used as the optimal seam to reate labelig, ad the two images a be ut alog the seam ad merged together. Labelig allows us also to avoid ghostig whe objets move as the images are aptured. Two image bledig proesses a be used i this fast paorama stithig approah. Whe soure images are suffiietly similar i olors after olor orretio, a simple ad fast liear bledig suffies. Whe soure images are too differet for the simple liear bledig, a Poisso bledig removes visible seams. Applyig olor orretio helps also Poisso solver to fid a good solutio faster. A sequetial paorama stithig proedure is reated ad itegrated with olor orretio, fast labelig, ad image bledig to reate paorami images. The itegratio allows us to reate high-resolutio paorami images from several large soure images quikly usig little memory. Future work iludes speedig up the Poisso bledig proess ad reduig its memory osumptio. REFERENCES [1] Y. Xiog ad K. Pulli, Mask based image bledig approah ad its appliatios o mobile devies, i SPIE Multispetral Image Proessig ad Patter Reogitio (MIPPR), 009. [] S. Ha, H. Koo, S. Lee, N. Cho, ad S. Kim, Paorama mosai optimizatio for mobile amera systems, IEEE Trasatios o, Cosumer Eletrois, vol. 53, o. 4, pp. 117 15, Nov. 007. [3] S. Ha, S. Lee, N. Cho, S. Kim, B. So, "Embedded paorami mosai system usig auto-shot iterfae," IEEE Trasatios o Cosumer Eletrois, Vol. 54, No. 1, pp.16-4, Feb. 008. [4] A. Agarwala, M. Dotheva, M. Agrawala, S. Druker, A. Colbur, B. Curless, D. Salesi, ad M. Cohe, Iterative digital photomotage, ACM Tras. Graph, vol. 3, pp. 94 30, 004. [5] A. Levi, A. Zomet, S. Peleg, ad Y. Weiss, Seamless image stithig i the gradiet domai, i Europea Coferee o Computer Visio (ECCV), 004, pp. 377 389. [6] P. Pérez, M. Gaget, ad A. Blake, Poisso image editig, ACM Tras. Graph., vol., o. 3, pp. 313 318, 003. [7] J. Jia, J. Su, C.-K. Tag, ad H.-Y. Shum, Drag-ad-drop pastig, i ACM SIGGRAPH, 006, pp. 631 637. [8] Z. Farbma, G. Hoffer, Y. Lipma, D. Cohe-Or, ad D. Lishiski, Coordiates for istat image loig, ACM Tras. Graph., vol. 8, o. 3, pp. 1 9, 009. [9] N. Graias, M. Mahoor, S. Negahdaripour, ad A. Gleaso, Fast image bledig usig watersheds ad graph uts, Image Visio Comput., vol. 7, o. 5, pp. 597 607, 009. [10] D. L. Milgram, Computer methods for reatig photomosais, IEEE Tras. Comput., vol. 4, o. 11, 1975, pp. 1113 1119. [11] A. A. Efros ad W. T. Freema, Image quiltig for texture sythesis ad trasfer, i ACM SIGGRAPH, 001, pp. 341 346. [1] J. Davis, Mosais of sees with movig objets, i IEEE Coferee o CVPR, 1998, pp. 354 360. [13] Y. Xiog ad K. Pulli, Gradiet domai image bledig ad implemetatio o mobile devies, i Iteratioal Coferee o Mobile Computig, Appliatios, ad Servies (MobiCase), 009. [14] V. Kolmogorov ad R. Zabih, What eergy futios a be miimized via graph uts, IEEE Trasatios o Patter Aalysis ad Mahie Itelligee, vol. 6, pp. 65 81, 004. [15] Y. Xiog ad K. Pulli, Sequetial image stithig for mobile paorama, i IEEE Iteratioal Coferee o Iformatio, Commuiatios ad Sigal Proessig (ICICS), 009. BIOGRAPHIES Yige Xiog works at Nokia Researh Ceter. His researh iterest areas ilude omputer visio, patter reogitio, ad omputatioal photography. Previously he was a researh professor i Virgiia Polytehi Istitute ad State Uiversity ad Wright State Uiversity. He reeived PhD degree from Najig Uiversity of Aeroautis ad Astroautis. Kari Pulli is a researh fellow at Nokia Researh Ceter. He has bee a ative otributor to several mobile graphis stadards ad reetly wrote a book about mobile 3D graphis. Pulli reeived a PhD i omputer siee from Uiversity of Washigto ad a MBA from Uiversity of Oulu. Cotat him at kari.pulli@okia.om.