Motion Layer Extraction in the Presence of Occlusion using Graph Cut

Transcription

1 Oral preenaion a he IEEE Conference on Compuer Viion and Paern Recogniion, CVPR Moion Layer Exracion in he Preence of Occluion uing Graph Cu Jiangjian Xiao Mubarak Shah Compuer Viion Lab, School of Compuer Science Univeriy of Cenral Florida, Orlando, Florida 32816, USA {jxiao, hah}@c.ucf.edu Abrac Exracing layer from video i very imporan for video repreenaion, analyi, compreion, and recogniion. Auming ha a cene can be approximaely decribed by muliple planar region, hi paper decribe a robu novel approach o auomaically exrac a e of affine ranformaion induced by hee region, and accuraely egmen he cene ino everal moion layer. Fir, a number of eed region are deermined by uing wo frame correpondence. Then he eed region are expanded and refined uing he level e repreenaion and employing graph cu mehod. Nex, hee iniial region are merged ino everal iniial layer according o he moion imilariy. Third, afer exploiing he occluion order conrain on muliple frame he robu layer exracion i obained by graph cu algorihm, and he occluion beween he overlapping layer are explicily deermined. Several example are demonraed in he experimen o how ha our approach i effecive and robu. 1 Inroducion Auomaic exracion of layer from video equence ha broad applicaion, uch a video compreion and coding, recogniion, ec. Once, a moion egmenaion i achieved, a video equence can be efficienly repreened by differen layer. The major ep of moion egmenaion coni of: (1 deermining he layer decripion, which include he number of layer and he moion parameer for each layer; (2 aigning each pixel in he image equence o he correponding layer and idenifying he occluded pixel. In an earlier work, Wang and Adelon propoed he ue of opical flow o eimae he moion layer, where each layer correpond o a mooh moion field [17]. Ayer and Sawhney combined MDL and MLE in an EM framework o eimae he number of layer and he moion model parameer for each layer [1]. Several oher approache ued MAP or MLE for eimaion of model parameer auming differen conrain and moion model [18, 22, 7, 16, 12]. Anoher cla of moion egmenaion approache i o group he pixel in a region by uing linear ubpace conrain. In [6, 5], Ke and Kanade fir expanded he eed region ino he iniial layer by uing k-conneced componen. Afer enforcing a low dimenional linear affine ubpace conrain on muliple frame, hey cluered hee iniial layer ino everal group, and hen aigned he image pixel o hee layer. Zelnik-Manor and Irani alo ued he homography ubpace for planar cene o exrac a pecific layer and o regier he image baed on hi layer [21]. Recenly, graph cu [2, 8, 9, 3] wa propoed o uccefully minimize energy funcion for variou compuer viion problem, uch a ereo, image egmenaion, image reoraion, exure ynhei [10], ec. Afer formulaing hee differen problem ino graph cu framework, an opimal oluion of energy minimizaion problem can be obained in a polynomial ime. In he moion egmenaion area, Shi and Malik fir ued he normalized graph cu o exrac layer from a video equence [14]. However, ince hey grouped pixel baed on he affiniy of moion profile, a local meauremen, heir mehod ignored he global conrain and appeared unable for he noiy image equence. Will e al propoed he ue of graph cu o exrac layer beween wo wide baeline image [19]. Afer employing he RANSAC echnique, hey cluered he correpondence ino everal iniial layer, hen performed he dene pixel aignmen via graph cu for hee wo image. Afer exenively reviewing he previou work on moion egmenaion, we did no find any work in he lieraure, which deal wih he explici deecion of he occluded pixel ha do no belong o any layer. However, he occluion problem ha been widely udied for a long ime in ereo algorihm [8, 9, 4]. The accurae deecion of he occluded area i very imporan o improve he dene dipariy map and he qualiy of 3D reconrucion. Similarly, in moion egmenaion area, hi occluion problem i eenial o deec he diconinuiie beween he overlapping layer and improve he qualiy of he layer boundarie. In hi paper, we propoe a novel approach o exrac accurae layer repreenaion from a video equence and ex-

2 plicily deermine occluion beween he overlapping layer. Our algorihm i implemened in wo age. In he fir age, we deermine eed correpondence over a hor video clip (3-5 frame. Then, we gradually expand each eed region from an iniial recangular pach of fixed dimenion ino an enlarged uppor region of an arbirary hape. Thi i achieved uing a graph cu approach inegraed wih he level e repreenaion. Afer ha, we employ wo-ep merging proce o obain a layer decripion of he video clip. In he econd age, according he obervaion of ha he occluion area i increaing wih he emporal order, we inroduce he occluion order conrain over muliple frame egmenaion. Afer applying hi conrain on he graph cu algorihm, we obain a able and accurae video egmenaion in erm of layer and heir 2D moion parameer. A he ame ime, he occluded pixel beween overlapping layer are correcly idenified, which grealy improve he qualiy of he layer boundarie. The paper i organized a follow. Secion 2 addree how o exrac layer decripion from hor video clip. Secion 3 deal wih he ue of he occluion order conrain and a muli-frame graph cu algorihm for obaining precie layer egmenaion in he preence of occluion. In Secion 4, we demonrae everal reul obained by our mehod. 2 Layer decripion exracion In our approach, he fir age i o exrac he layer decripion from video equence, which include he number of layer and he moion parameer for each layer. In hi age, we fir deec he robu eed correpondence over a hor video clip. Nex, uing he previou hape prior of he eed region, he region fron i gradually propagaed along he normal direcion uing bi-pariioning graph-cu algorihm inegraed wih he level e repreenaion. Third, we ue a wo-ep merging proce o merge he eed region ino everal group, uch ha each group belong o a ingle moion field. 2.1 Deermining robu eed correpondence In order o correcly exrac he layer decripion, we conider a hor video clip L inead of only wo conecuive frame. The reaon i ha if he moion beween wo conecuive frame i oo mall, he moion parameer of differen layer are no diinc o exrac. Therefore, we ue an average pixel flow ν of he eed correpondence a a meauremen o decide he number of frame in he video clip L. If ν beween I n and I 1 i greaer han ome hrehold (i.e. 3 pixel, he number of frame L i e o n. In our approach, fir we deec he Harri corner in he fir frame, hen we ue KLT racking algorihm [15] or our maching algorihm [20] o rack he corner over hi hor period uing a window uppor. Since he Harri corner are locaed in he exured area, we can obain reliable affine ranformaion for he eed region, and kip he non-exured area, where he moion parameer eimaion i unreliable. 2.2 Expanding eed region Once he eed correpondence are deermined beween frame I 1 and I n, we conider a window pach around each eed corner a an iniial layer, which correpond o a planar pach in he cene. Thi way, we ge a number of iniial layer, and each layer i uppored by a mall pach wih he correponding affine ranformaion. Neverhele, he affine moion parameer eimaed uing he mall pache may over-fi he pixel inide he region, and may no correcly repreen he global moion of a larger region. Paricularly, when he corner i locaed a he boundary of wo rue layer, he over-fiing may inroduce a eriou diorion on hi pach afer applying he affine ranformaion. One raighforward oluion i o imply exend he region by including neighboring pixel which are conien wih he affine ranformaion. Such pixel can be deermined by applying hrehold o he SSD (Sum of Squared Difference compued beween he original and warped window. However, hi cheme ha wo problem: Fir, he reuling expanded region may no be compac and mooh. Second, he new pach may include he pixel from muliple layer, and may no be conien wih a ingle planar pach in he cene. Fig. 1.b how one ample reul obained by uing hi imple cheme. The eed region i originaed from he eed on he roaing ball (Fig. 1.a. Afer expanding he boundary and pariioning by applying a imple hrehold, he region i no ha mooh, and i alo include he pixel from he oher layer. In order o deal wih hee problem, we propoe a novel approach o gradually expand he eed region by idenifying he correc upporing pixel by uing he bi-pariioning graph cu mehod and employing he level e repreenaion. We inroduce a moohne energy erm, which can mainain he pariion piecewiely mooh and naurally olve he fir problem. Then, uing level e repreenaion, he conour of he eed region i gradually evolved by propagaing he region fron along i normal direcion. A weighed graph G = V, E i defined by a e of node V (image pixel and a e of direced edge E which connec hee node a hown in Fig. 2. In hi graph, here are wo diinc node and, called he ource and ink repecively. The edge conneced o he ource or ink are called

3 (a (b (c (d (e (f (g (h (i (j Figure 1: Region expanion proce. Top: A procedure for expanion of an iniial eed region (a o a large uppor region. (b The imple expanion and pariioning. (c bi-pariioning wihou he level e repreenaion. (d and (f repecively are he expanion of he eed region during he fir and fourh ieraion uing he level e repreenaion. (e and (g are he reul obained afer he graph cu pariioning. Boom: Several reul from he mobile-calendar (h and flower-garden equence (i, where he new region can have an arbirary compac conour. The la hree image (j how hree poor eed region deeced by employing he coverage hrehold. Noe: The red box i he iniial eed region. The green conour are obained afer uing bi-pariioning algorihm. -link, uch a (, p and (p,. The edge conneced o wo neighboring pixel node are called n-link, which have wo direcion, uch a (p, q and (q, p. The problem of expanding he eed region can be eaily formulaed ino he graph cu framework a a bi-pariioning problem of a verex e. In hi framework [2], we eek he labelling funcion f by minimizing he energy E = E mooh (f+e daa (f = V (p, q+ D p(f p, (1 (p,q N p P where E mooh i a piecewie moohne erm, he E daa i a daa error erm, P i he e of pixel in he image, N i a 4-neighbor yem, f p i he label of a pixel p, D p (f p i daa penaly funcion, and V (p, q i mooh penaly funcion 1. In hi bi-pariioning problem, he label f p of he pixel p i aigned eiher 0 or 1. If f p = 0, he pixel p i upporing hi eed region, oherwie, hi pixel i no upporing he region. In graph G, afer eing he weigh of -link o D p (0 on he ource ide and D p (1 on he ink ide, and he weigh of n-link o V (p, q, we can compue he minimum cu C uing he andard graph cu algorihm and obain piecewie mooh pariion of he upporing region. However, he pariioning uing graph cu canno guaranee he gradual expanion or hrinking of a region along he normal direcion a hown in Fig. 1.c, where ome pixel no belonging o hi region are alo included. Since he conour informaion of he iniial eed region i no inegraed in he funcion given in equaion 1, he graph cu algorihm canno correcly evolve he region conour along he normal direcion. In order o olve hi problem, we 1 Thi i a impler verion of Eq. 4. p (,p (p, (p,q (q,p (,q (q, q r o cu C edge (,p (p, (p,q (q,p weigh D p (0 D p (1 V(p,q Figure 2: An example of a graph G for 1D image. Node p, q, r, and o are he pixel in he image. Afer compuing minimum cu C, he node are pariioned ino upporing pixel (ource and un-upporing pixel (ink. The weigh of he link are lied in he able on he righ. ue he conour of he eed region a a prior o compue he level e repreenaion of hi region. Then, afer adjuing he weigh of he -link for pixel on he region boundary in graph G, we ue he graph cu algorihm o gradually expand he eed region. The deailed proce i decribed a follow: Sep 1: Conruc a mak β of he original eed region, which ha a value in [0, 1], where he inide pixel of he region are marked by 1 and he oher are marked by 0. Then, compue a level e v by imply convolving he region mak β wih a Gauian kernel a: v = G β, where G i he Gauian kernel. For each pixel i, he v i inide of he eed region ha a high conan value, and he v i ouide of he region fall down along he conour normal direcion unil v i =0. Therefore, we obain an implici urface for hi conour evoluion, which can be repreened by

4 level e [11, 13]. Here we propoe anoher approach o evolve he region conour by inegraing he level e repreenaion ino graph cu mehod a he nex wo ep. Sep 2: Warp he econd image uing he correponding affine ranformaion, and compue SSD beween he warped image and he fir frame. Conruc a graph G for he pixel wih v i > 0. Compue daa penaly D p according o he compued SSD and moohne penaly V (p, q for each link in hi graph. Sep 3: Ue he level e v a a weigh o change - link of each pixel a he ink ide, hen compue he minimum cu C. The weigh of he pixel inide he region are almo no changed, while he weigh (p, will decreae when he pixel p i away from he boundary. A a reul, he minimum cu C i mo likely o cu he ouide pixel, and label hem a he un-upporing pixel for hi region. Thi way, he eed region will gradually propagae from he cener o ouide. Sep 4: Ue he new compued region a he eed region o compue a new affine ranformaion by minimizing he image reidue inide he region, hen goo Sep 1 o do he nex ieraion. If he new region i hrinking and canno cover 75% area of he original eed region (coverage hrehold, i i dicarded a a poor iniial layer. Afer a few ieraion of he above ep, he fron of he eed region will eiher expand or hrink along he normal direcion of he conour. Fig. 1 how a deailed proce for eed region expanion. Fig. 1.d how he level e repreenaion obained from he iniial eed region (Fig. 1.a. Fig. 1.e and 1.g are he pariioning reul afer he fir and fourh ieraion. In he econd row of Fig. 1, we how everal reul for eed region expanion of he mobile-calendar and flower-garden equence. The la hree image (Fig. 1.j how ha we can idenify he poor eed region uing he coverage hrehold. Mo of hee poor eed region are locaed a he boundary of muliple layer. 2.3 Two-ep region merging proce Afer expanding he region, each good eed region become an iniial layer. Mo of hee layer may hare he ame affine ranformaion. Therefore, we deign a woep merging algorihm o merge hee layer o obain he layer decripion. In he fir ep, we only merge he layer which overlap wih each oher. Given wo region R 1 and R 2, we e if he number of overlapping pixel are more han half of pixel in he maller region. If hi i rue, we compue he Figure 3: Four layer of he mobile-calendar equence, which are correponding o he calendar, rain, ball, and wall repecively. The green conour i he region boundary, and non-upporing pixel are marked by red. Noe: The non-exured area may belong o everal layer due o heir ambiguiie, uch a he whie paper a he lower par of he calendar in he mobile-calendar equence. SSD by warping he fir region, R 1, uing he ranformaion H 2 of he econd region R 2. Uing hi SSD a he meaure, we can deec how many pixel uppor H 2 uing he graph cu algorihm. If he majoriy (ay 80% of pixel of R 1 uppor R 2, we merge hee wo region and recompue he moion parameer uing he merged pixel. Then, uing bi-pariioning graph cu algorihm, we prune he un-upporing pixel from he new region. If only a few pixel of R 1 uppor H 2, we repea he proce by warping R 2 uing he ranformaion H 1 of R 1. In order o achieve large merged region, we ierae he whole proce a few ime (ypically 3 o 4 o make ure he merging proce converge. A a reul, only a few large region remain, and ome of non-overlapping region may ill hare a ingle moion ranformaion. In he econd merging ep, we merge hee non-overlapping region uing a imilar proce. Fig. 3 how he reul for he mobile-calendar equence. 3 Muli-frame layer egmenaion in preence of occluion via graph cu In hi ecion, we will addre he following problem: Given he exraced layer decripion, how o compue an accurae layer egmenaion in preence of occluion uing muliple frame of hi hor video clip. In hi paper, we propoe an approach o explicily idenify he occluded pixel, where every occluded pixel i aigned a new occluion label, f oc. Fir, we will ae he occluion order conrain.

5 frame 1 and 2 frame 1 and 3 frame 1 and 4 frame 1 and (p 3,0,p 2,0 (p 2,0,p 3,0 b a occluion beween 1 and 2 occluion beween 1 and 3 occluion beween 1 and 4 occluion beween 1 and 5 p 1,0 q 1,0 p 2,0 p 2,1 p 3,0 (p 3,1,p 2,1 (p 2,1,p 3,1 p 3,1 p 4,0 p 4,1 Figure 4: The occluion order in a hor video clip conaining five conecuive frame (he fir image i he reference image. The op row i he five-frame equence, where a olid circle i moving along he lef-op direcion. The boom image how ha he occluion (color area beween he fir frame and he oher frame are increaing wih ime. Then, we embed hi conrain ino he labelling funcion, which can be minimized uing he muli-frame graph cu algorihm. 3.1 Occluion order conrain Wih he inenion of compuing an accurae moion layer egmenaion of a video clip, fir we analyze he occluion proce over a emporal domain. Fig. 4 how he occluion ha a emporal order for a linearly moving objec. I i obviou ha occluion area i increaing wih he emporal order. During a hor period (3-5 frame, hi obervaion i no violaed if he objec i no oo hin or no moving oo fa. Therefore, baed on hi aumpion, we ae he occluion order conrain a follow: Rule 1: During a hor period, if one pixel i occluded beween frame 1 and j, hi pixel will alo be occluded beween frame 1 and (j +1. Rule 2: If he pixel p i aigned a label f p beween frame 1 and j, hen pixel p hould be aigned eiher f p or f oc beween frame 1 and k, where k>j; and pixel p hould be aigned f p beween frame 1 and k, where k<j. According o hi occluion order conrain, only pixel a he ame image coordinae in wo conecuive frame pair can influence each oher, uch a he frame pair (1,2 and (1,3. Now, hi muli-frame moion egmenaion problem can be formulaed a an energy minimizaion problem of he following funcion: n 1 n 2 E = (E mooh (f+e daa (f+e occ(f + E order (f, j=1 j=1 where j i frame number, and n i he oal number of frame. Compared o Eq. 1, here are wo addiional erm q 1,1 Figure 5: Thi graph i conruced by five conecuive frame, which have four image pair relaed o he reference image. The red line eparae each pair of image ino one block. The blue n-link are inroduced o mainain he occluion order conrain. Noe: Only par of he node and link are drawn. in hi equaion. The fir one i E occ (f, which i ued o impoe he occluion penalie for he occluded pixel beween frame 1 and (j +1. The econd one i E order (f, which i ued o impoe occluion order penalie for mainaining he occluion order conrain on each conecuive pair of image pair. 3.2 Muli-frame moion egmenaion In order o minimize hi energy funcion, we employ he graph cu mehod and conruc a graph G = V, E involving muliple conecuive frame a hown in Fig. 5. To illurae hi, we ack four pair of image node ogeher in hi graph, noe ha each image pair involve he fir frame (he reference frame and one of he oher frame, which i conien wih Fig. 4. In Fig. 5, each image pair i eparaed by he red doed line. In each image pair (1,j+1, j>1, and every pixel p, here i a pair of node p j,0 and p j,1. All of hee node from he image pixel form a new ube A = V {, }. According o he occluion order conrain, a e of order n-link (blue edge, uch a (p 3,0,p 2,0 and (p 2,0,p 3,0, are added in he graph G o inerac wih he node a he ame image coordinae. To implify he graph G, we only how wo node from one paricular pixel p for each image pair o illurae hee order n-link. The deailed ub-graph G 1,2 for he fir image pair i redrawn in Fig. 6. Before we decribe how o minimize he energy E for he whole graph G, we fir dicu he ineracion of he node in ub-graph G 1,2, and hen dicu how o aign he weigh o hee link. To reduce he complexiy, we only how hree pixel graph p, q, and r in graph G 1,2, where he

6 p 1,0 (,p 1,0 q 1,0 r 1,0 p 1,0 (,p 1,0 (, cu (,p 1,0 p 1,0 p 1,0 cu (,p 1,0, (q 1,1, (,q 1,1 q 1,1 (r 1,1,p 1,0,r 1,1 r 1,1,, (, cu, (a (b (c Figure 6: A graph G 1,2, where hree baic pixel graph are hown correponding o pixel p, q, are r repecively. The n-link beween neighboring pixel i o enforce he moohne penalie, uch a (,q 1,1 and (q 1,1,. The new blue n-link are inroduced o enforce he ymmeric propery of he occluion, uch a,r 1,1 and (r 1,1,p 1,0. pixel graph correpond o one pixel in he reference image, and i he baic elemen o conruc hi whole graph. In each pixel graph, here are wo node, uch a p 1,0 and, and one pair of occluion n-link, uch a, and (,p 1,0. If he minimum cu C cu he link,, he pixel p i occluded. Afer aigning weigh D oc,, D p (f p, and D p (α on link,, (,p 1,0, (,, and (, p 1,0 repecively, Fig. 7 how hree cae of C afer applying he andard α-expanion echnique [2, 8] on one independen pixel graph. Le f p be he original label of a pixel p in he reference image. The pixel, p, will be aigned a new label fp C a follow: α if (, p 1,0 C, (, C(Fig.7.a, fp C = f p f oc if, C, (, C(Fig.7.b, if, C, (, C,, C(Fig.7.c, where f oc i he occluion label. In he occluion cae, eiher D p (α or D p (f p of pixel p i greaer han he occluion penaly D oc. I mean ha i i no uiable o aign eiher he original label f p or he new label α o hi pixel. Therefore, hi pixel i an occluded pixel and i aigned f oc. In graph G 1,2, he moohne energy erm E mooh (f i implemened by he moohne n-link, which connec each pair of neighboring pixel graph uch a (q 1,1, and (,q 1,1. In order o compue he moohne penaly erm V (p 1,i,q 1,i of a link (p 1,i,q 1,i, we warp he econd image I 2 o obain he warped image I H f i 2 by applying he moion ranformaion H fi, correponding o label f i, for each label in he layer decripion. Here f i = (2 { α if i =0 f p if i =1. (3 Figure 7: Three poible cae afer one α-expanion of an independen pixel graph p. (a p i aigned o he new label α. (b p will keep he original label f p. (c p i occluded and aigned o he label f oc. Therefore, he moohne penaly erm V (p 1,i,q 1,i can be compued a 4λ if max( I 1(p I 1(q, I H f i (j+1 (p IH f i (j+1(q < 4, V (p j,i,q j,i = 2λ if 4 max( I 1(p I 1(q, (4 I H f i (j+1 (p IH f i (j+1(q < 8, λ oherwie, where λ i an empirical conan, I 1 i he fir frame, I H f i (j+1 i he warped verion of I (j+1 obained by applying ranformaion H fi. To deal wih he ymmeric properie of he occluion, a e of new ymmeric occluion n-link are added o connec he relaed node. In Fig. 6, a pair of ymmeric n-link are added o connec hee wo node, uch a he blue doed link (r 1,1,p 1,0 and,r 1,1. Wih he help of hee ymmeric n-link, he occluion penalie from frame 2 o frame 1 are alo pecified. Afer aigning weigh 0,,, and 0 o order n- link (p (i+1,0,p i,0, (p i,0,p (i+1,0, (p (i+1,1,p i,1, and (p i,1,p (i+1,1 repecively, he occluion order conrain i fully aified, which can be eaily verified by auming he minimum cu poiion. Fig. 8 compare he egmenaion reul obained uing five frame wih hoe obained uing only wo frame. Due o he ue of muliple frame, he arifac are removed and he egmenaion reul are conien a hown in Fig. 8bd. Moreover, i i obviou ha he occluded area beween he overlapping layer increae wih he ime. 4 Experimen We have eed our approach on wo andard moion equence, mobile-calendar and flower-garden (Fig. 9, and one addiional car-map (Fig. 10 equence.

7 (a 1 2 (b 1 3 (c 1 4 (d 1 5 Figure 8: Segmenaion reul of frame 1 in he mobile-calendar equence. Top: The egmenaion reul obained by uing only wo frame. Boom: The muli-frame egmenaion reul obained by uing five frame (1-5. (a d Segmenaion reul beween frame 1 and 2-5 repecively. The red pixel in egmened image are he occluded pixel. Fig. 9 how he egmenaion reul for he mobilecalendar and flower-garden equence. We ued five frame o exrac he layer for he mobile-calendar equence, and ued hree frame o exrac he layer for he flower-garden equence. We alo compared our reul wih he he oher mehod [6, 5, 17, 1] for hee wo andard equence. Since he ground ruh of hee daa are no available, we have o limi our analyi o he qualiaive comparion. Our approach can explicily deermine he occluded pixel, and in our reul, he boundarie beween overlapping layer are more precie and finer han he previou mehod. We alo applied our mehod o our own equence wih a large occluion, car-map (Fig. 10, where he car i moving behind he map and he cale of he car i apparenly changed. The equence i aken by a hand-held moving video camera. During ome frame, mo par of he car are occluded by he map. Once he car move behind he map, i i very difficul o compue he correc moion parameer for car layer baed on a mall region of he car due o he over-fiing problem. Therefore, we ue a common racking echnique o predic he moion parameer baed on he previou frame. If he region hrink by ome amoun (ay 20% and he prediced moion parameer are much differen han he new eimaed parameer, we keep he prediced parameer o perform he egmenaion. The reul are hown in Fig. 10. In all of our experimen, once he layer decripion are exraced, he average compuaional ime for one frame egmenaion i le han 30 econd. Noe: All of our reul are alo available a our web ie [23]. 5 Concluion In hi paper, we preened an effecive mehod o exrac robu layer decripion and o perform an accurae layer egmenaion for image equence conaining 2-D moion. Our conribuion coni of: (1 Iniial layer decripion by inegraing he level e repreenaion ino he graph cu mehod o obain gradually expanding eed region. (2 Uing he occluion order conrain, we uccefully combine muliple frame o compue accurae layer egmenaion and explicily deec he occluded pixel, which have no been done before. In he fuure, we will inveigae he relaionhip beween he level e and graph cu mehod, and unify hee wo approache ino one framework for differen applicaion. 6 Acknowledgemen We would like o hank Khurram Shafique, Yunjun Zhang, Alper Yilmaz, and Yaer Ajmal for helpful dicuion. Reference [1] S. Ayer, H. Sawhney, Layered repreenaion of moion video uing robu maximum-likelihood eimaion of mixure model and mdl encoding, ICCV, [2] Y. Boykov, O. Vekler, R. Zabih, Fa Approximae Energy Minimizaion via Graph Cu, PAMI, 23(11, [3] Y. Boykov, V. Kolmogorov, Compuing Geodeic and Minimal Surface via Graph Cu, ICCV, [4] S. Kang, R. Szeliki, J. Chai, Handing Occluion in Dene Muli-view Sereo, CVPR, 2001.

8 Figure 9: Top: The egmenaion reul for he mobile-calendar equence. Boom: The egmenaion reul for he flowergarden equence. The red pixel in egmened image are he occluded pixel. Figure 10: Segmenaion reul for he car-map equence. Top: Several frame from he equence. Boom: The egmenaion reul, where he layer are accuraely exraced even hough he mo par of he moving car are occluded in ome frame. [5] Q. Ke, T. Kanade, A Subpace Approach o Layer Exracion, CVPR, [6] Q. Ke, T. Kanade, A Robu Subpace Approach o Layer Exracion, IEEE Workhop on Moion and Video Compuing, [7] S. Khan, M. Shah, Objec Baed Segmenaion of Video Uing Color, Moion and Spaial, ICCV [8] V. Kolmogorov and R. Zabih, Viual Correpondence wih Occluion uing Graph Cu, ICCV, [9] V. Kolmogorov and R. Zabih, Muli-camera Scene Reconrucion via Graph Cu, ECCV, [10] V. Kwara, I. Ea, A. Schdl, G. Turk, A. Bobick, Graphcu Texure: Image and Video Synhei Uing Graph Cu, SIGGRAPH, [11] S. Oher, R. Fedkiw, Level Se Mehod and Dynamic Implici Surface. The Springer-Verlag Pre, [12] I. Para, E. hendirk, R. Lagendijk, Video Segmenaion by MAP Labeling of Waerhed Segmen. PAMI, 23 (3, [13] J. Sehian. Level Se Mehod and Fa Marching Mehod. Cambridge Univeriy Pre, [14] J. Shi, J. Malik, Moion Segmenaion and Tracking Uing Normalized Cu, ICCV, [15] J. Shi, C. Tomai, Good Feaure o Track, CVPR, [16] H. Tao, H. Sawhney, R. Kumar, Objec racking wih bayeian eimaion of dynamic layer repreenaion, PAMI, 24(1, [17] J. Wang, E. Adelon, Repreening moving image wih layer, IEEE Tran. Image Proceing, 3 (5, [18] Y. Wei, Smoohne in Layer: Moion Segmenaion uing Nonparameric on Homographic, CVPR, [19] J. Will, S. Agarwal, S. Belongie, Wha Wen Where, CVPR, [20] J. Xiao, and M. Shah, Two-Frame Wide Baeline Maching, ICCV [21] L. Zelnik-Manor, M. Irani, Muli View Subpace Conrain on Homographie, ICCV, [22] Y. Zhou, H, Tao. A Background Layer Model for Objec Tracking hrough Occluion, ICCV, [23] hp:// viion/projec/moion layer exracion/.