Virtual View Synthesis for Free Viewpoint Video and Multiview Video Compression using Gaussian Mixture Modelling

Transcription

1 Virual View Synhesis for Free Viewpoin Video and Muliview Video Compression using Gaussian Mixure Modelling D M Moiur Rahaman, Member, IEEE and Manoranjan Paul, Senior Member, IEEE Absra High qualiy virual views need o be synhesized from adjaen available views for free viewpoin video (FVV) and muliview video oding (MVC) o provide users wih a more realisi 3D viewing experiene of a sene. View synhesis ehniques have poor rendering qualiy due o he olusion and rounding ineger error hrough warping by reaing holes. To remove he holes in he virual view he exising ehniques use spaial and emporal orrelaion in inra/ iner-view images and deph maps. However, hey sill suffer qualiy degradaion in he boundary region of foreground and bakground areas due o he low spaial orrelaion in exure images and low orrespondene in iner-view deph maps. In he proposed ehnique, we use he number of models in Gaussian mixure modelling (GMM) o separae bakground and foreground pixels o overome he limiaions of he above menioned ehniques. Then, he missing pixels are reovered from he adapive weighed average of he pixel inensiies from he orresponding model(s) of he GMM and he warped image o overome he error inrodued in he warping proess. The weighs vary wih he ime o aommodae he hanges due o dynami bakground and moions of he moving objes for view synhesis. We also inrodue an adapive sraegy o rese GMM modelling if he onribuions of he pixel inensiies from he models drop signifianly. Our experimenal resuls indiae ha he proposed approah provides 5.40~ 6.60dB PSNR improvemen ompared wih relevan mehods. To verify he effeiveness of he proposed view synhesis ehnique, we use i as an exra referene frame in he moion esimaion for MVC. The experimenal resuls onfirm ha he proposed view synhesis is able o improve PSNR by 3.15~5.13dB ompared o he onvenional hree referene frames. Index Terms View synhesis, free viewpoin video, deph image based rendering, muliview video ompression, moving piure expers group (MPEG), inernaional eleommuniaion union (ITU) F I. INTRODUCTION VV has araed onsiderable aenion in reen years as i provides freedom o he user o observe a sene from differen angles or viewpoins [1][2][3]. A large number of views wih a small baseline are required o failiae his luxury, Manusrip reeived 21 November, 2016; revised 31 May, 14 Augus and 4 Oober, D M Moiur Rahaman is wih he Shool of Compuing and Mahemais, Charles Sur Universiy, Bahurs, NSW-2795, Ausralia (phone: ; drahaman@su.edu.au. Assoiae Professor Manoranjan Paul is wih he Shool of Compuing and Mahemais, Charles Sur Universiy, Bahurs, NSW-2795, Ausralia (phone: ; Fax: ; mpaul@su.edu.au. whih inreases ransmission bandwidh and sorage daa signifianly. Deph image based rendering (DIBR) is a praial way o redue sorage and ransmission bandwidh for muliview videos from olor exures and heir orresponding deph maps [1][2]. However, in he DIBR ehnique, porions of regions are no visible in he virual posiion due o he fron objes ermed as olusion, whih reae some holes in he video synhesis [3]-[6]. Moreover, warping proess from differen views ause anoher soure of error due o rounding ineger. Generally, here are wo ypes of mehods o fill missing pixels or holes. One is o exploi spaial orrelaion of he video o fill he missing pixels. In he spaial domain, view blending approahes an redue he number of holes as wo adjaen ameras an over a relaively wider viewing angle [6]. In his ehnique, adjaen warped views are ombined ino a single view, whih an redue he holes. However, only a small number of views are ransmied due o he bandwidh onsrain. Therefore, he rendered view would miss some pixel informaion [7]-[10]. Inpaining ehnique is normally popular o reover hese missing pixels wihou inroduing signifian blur arifas. Inpaining ehnique in [10][11], afer ompuing he prioriy of holes boundary pixels, he mos relevan pah is opied from he soure pah by exploiing spaial orrelaion. However, his proess an deeriorae he qualiy of he view synhesis by being unable o differeniae foreground and bakground pixels properly. This is due o he low spaial orrelaion in he perimeer beween foreground and bakground pixels [2][5][6][10][13]. In [14], bloks wih missing pixels in erms of dereasing diffiuly for inpaining were sored ou. In his ehnique, explii insruions alled auxiliary informaion (AI) of he mos diffiul bloks is ransmied o guide he deoder in he reonsruion proess. The deoder an independenly fill up missing pixels in he bloks ha are easy o inpain via a emplae-mahing algorihm. In [15], deph informaion was used o he prioriy ompuaion and pah disane alulaion of he algorihm in [11]. In his ehnique, he pah whose deph variane is low gives higher prioriy. However, his may produe disored synhesized resuls around he foreground obje boundaries in ase he boundaries of objes in he deph map are mismahed wih ha of he olor image. Inverse mapping is anoher popular ehnique for hole-filling. This ehnique re-maps he missing

2 pixel loaions in he original view based on he olumn-shifs of he neighbourhood. In his way, holes an be mapped bakward o one of he original views o idenify he missing pixel values [7][12]. As his ehnique also explois spaial orrelaion for he olumn par, i also suffers he hole filling problem in he foreground-bakground boundary areas. The oher mehods use emporal orrelaion o fill missing pixels of he view synhesis. They are popularly known as bakground updae ehniques and hey are based on he assumpion ha an oluded bakground in he one frame may beome visible in he oher frames when he foreground objes move away. The ehniques in [13][16] generaed a sai bakground frame by exploiing emporal orrelaion and hen removing any foreground obje wih onvenional inpaining and lusering ehniques depending on he deph map. The experimenal resuls reveal ha hese ehniques improve he qualiy of he view synhesis signifianly ompared o oher ehniques inluding inpaining ehniques [2]. However, his ehnique suffers qualiy degradaion, due o he dependeny on inpaining, warping of a bakground image and lusering mehods. Inauray of any hese seps deerioraes he qualiy of he view synhesis. Moreover, an imperfe deph map may lead o some arefas of he foreground. In addiion, if he bakground frame is generaed from he model sored by raio of weigh and sandard deviaion, i does no represen he reen hanges of he pixel, and i auses a poor bakground frame [2][6][17]. Unlike he exising GMM-based ehniques, he proposed ehnique uses he number of models in he GMM o separae bakground and foreground pixels and modify pixel inensiies aordingly. We use an adapive weighed average o generae a pixel inensiy o overome he error inrodued in he warping proess. We also use an adapive rese mehanism o keep he relevany of he modelling sysem. View synhesis ehniques are reognized as a promising ool for rendering new views from muliview video plus deph (MVD) for supporing advaned 3D video oding [1][18]. Reenly, inernaional organizaion for sandardizaion (ISO), moving piure expers group (MPEG) and inernaional eleommuniaion union (ITU) video oding expers group have joinly developed effiien oding ools suh as 3D-HEVC [1][3] [19][20]. The main fous of he ehnique in [21] is o inegrae a synhesized or dispariy-adjused view ino he blok-based rae-disorion (RD) opimizaion framework o improve prediion in MVD. For his, hey generae a virual view and inrodue new skip and dire modes using he synhesized view. However, hey did no inlude any explii hole filling ehnique o improve he qualiy of he synhesized view o address olusion and error due o rounding ineger problems. Therefore, he view synhesis prediion in [21] does no provide signifian ompression raio improvemen ompared o 3D-HEVC. 3D-HEVC provides he bes ompression raio for MVD daa by exploiing he view synhesis opimizaion (VSO) oding ool [3]. A VSO sheme for he exa view synhesis disorion alulaion was proposed by employing a measure alled synhesis view disorion hange (SVDC). The view rendering was performed ieraively in he enoding proess o ompue he RD os. This ehnique ahieves high ompression effiieny, bu i inflis heavy ompuaion burden o he enoder. Laer, view synhesis disorion and deph disorion models wihou performing view rendering were proposed o redue ompuaional omplexiy. However, he auray may no be high [22]. In [23], view synhesis disorion esimaion for AVC- and HEVC-ompaible 3-DV oding ehnique is proposed wih a soluion for eah of he problems in whih he view synhesis disorion funion onsisenly ahieves posiive oding gains. To enable auo sereosopy addiional views, he reeiver side generaes he urren frames from already enoded adjaen frames and he previous frame of he urren view [24]. Moreover, DIBR ehniques provide an exra referene by exploiing dispariy among adjaen views. Due o he high similariy of he proposed view synhesis wih he urren view, his ehnique provides beer prediion ompared o he onvenional hree referenes (i.e. wo frames from adjaen views and he previous frame of he urren view) sysem. To verify he effeiveness of he proposed view synhesis, we use i as an addiional referene frame in he moion esimaion for MVC. The experimenal resuls onfirm ha he proposed view synhesis is able o improve PSNR signifianly ompared o he onvenional hree referene frames. As we do no need any moion esimaion for he virual view, he ompuaional ime of using four referene frames is omparable o he hree referene frames. We also use wo referene frames using he proposed view synhesis view and he previous frame. The resuls show ha we an improve he PSNR ompared o he hree referene sheme in muliview ompression. This proposed sheme also redues ompuaional ime signifianly. The preliminary onep is published in [2]. The new onribuions in his paper are (i) adapive weighing, (ii) adapive rese sraegy for modelling, (iii) a new way o generae pixel inensiy of he virual view, (iv) view synhesis using synhesized images, (v) inroduing four and wo referenes ehnique insead of sandard hree referenes MVC. The res of his paper is organized as follows: Seion II desribes he proposed view synhesis approah wih adapive weighing hole filling ehnique, Seion III fouses on view synhesis for MVC, while Seion IV presens experimenal resuls. The onlusions are given in Seion V. II. PROPOSED VIEW SYNTHESIS TECHNIQUE In a number of oasions, GMM ehnique is used for view synhesis using bakground frame. However, in he proposed ehnique, he number of models in he GMM is used o separae bakground and foreground pixels and modify pixel inensiies using he orresponding model-pixel inensiies no only for he bakground model bu also oher models available in he GMM. The missing pixels of he bakground are reovered using he adapive weighed average of he pixel inensiies from he model(s) and he warped image o overome he error inrodued in he warping proess. In his ehnique, he inheren haraerisis of Gaussian mahemaial models are apialized o reover oluded areas.

3 RAHAMAN e al.: VIRTUAL VIEW SYNTHESIS FOR FVV AND MVC COMPRESSION USING GMM View 1 View 1 n-h Texure n-h Deph Warping View 1 Warped exure View 3 Warped exure Warping View 3 View 3 n-h exure n-h deph DIBR Blending Blended exure View 2 (n-1)- h exure View 2 (n-2)- h exure View 2 (n-i+1)- h exure Fig. 1: Proposed view synhesis ehnique. View 2 n-h exure I is rue ha GMM ehnique is more effeive for sai bakground senarios, however, i is also useful o address pixel inensiy problem for he even of olusion. Moreover, i an handle dynami (wih minor hanges) bakground senario. To handle more dynami bakground and foreground senario, we have used an adapive rese mehanism in he proposed mehod when he urren models lose heir relevany. Moreover, unlike in general senarios, sai ameras are normally used in he free viewpoin and muli-view video senarios. In he GMM ehnique, if a pixel posiion experienes similar inensiies over he period, here should be only one model, whih indiaes ha he pixel is a bakground. On he onrary, if a pixel posiion experienes differen pixel inensiies and i is needed o represen wih muliple Gaussian models. This indiaes ha he pixel has boh bakground and foreground in differen imes. Therefore, he hypohesis is ha he number of GMM models would be a good indiaor o idenify bakground/foreground pixels. In his ehnique, he GMM applies on he inerpolaed view insead of he adjaen view assuming ha synhesized previous images of he inerpolaed view are already available. This ehnique provides a beer pixel orrespondene, whih leads o beer qualiy ompared o boh inpaining and bakground updae mehods. However, if a pixel posiion experienes one foreground ogeher wih bakground in oher momens, i onsidered foreground hroughou he ehnique in [2] afer experiening he foreground. Even so, afer experiening foreground pixel inensiies, i an experiene bakground pixel inensiies again. Based on his hypohesis, we find appropriae bakground and foreground pixels for filling missing pixel inensiies of he virual view. Swihing Bakground modelling MFIS Bakground model(s) Hole-filling Moreover, he seing weigh is ruial as he PSNR of he view synhesis may vary up o 1.0~ 6.0dB by using differen weighs o balane he onribuions beween warping image and he learned foreground model. In his paper, we propose an adaping weighing ehnique o fill up foreground pixels of he view synhesis. In he experimen, we have observed ha if a video has more moving regions, he endeny is ha i has more pixels whih use wo or more Gaussian models. In his siuaion, he relaively large onribuion from warped image provides a beer qualiy of he view synhesis. I is due o he less relevany of he learned foreground wih he view synhesis for he rapid hanges of foreground wihin a shor period of ime. In his paper, we firs esablished a relaionship beween he weigh and he perenage of muliple Gaussian models using a number of videos. Then, we apply he relaionship in he view generaion. The experimenal resuls show ha he proposed ehnique does no sarifie any signifian qualiy degradaion ompared o he maximum ahievable qualiy hrough seing he weighs. In he proposed ehnique, n-h exure images from wo adjaen views are warped ino a virual posiion by using heir orresponding deph maps and amera parameers o generae n-h image of he inermediae view. Bu warped images onain holes due o he olusion and rounding ineger error. Two warped images are blended o redue hese missing pixels o make a warped image. This proedure redues he number of holes bu does no help o reover all missing pixel inensiies speially oluded regions. To reover hese missing pixels, we use GMM ehnique o model eah pixel wih available previous frames of he virual view as shown in Fig. 1. In our experimen, we assume ha we have already 1 o (n-1)-h frames for he GMM when we generae n-h frame of a virual view. In his ehnique, parameer i is used o rese he modelling afer a erain inerval, where i=2, 3, 4 n. The reseing of he modelling depends on weighing faor (deails see in Seion III (C)). Iniially, we use original frame for GMM. Moreover, we also use synhesized frame for GMM. Then, based on he number of GMM models, eah pixel is lassified as a foreground or bakground pixel. Afer ha, he missing pixel inensiies of he bakground and foreground areas are filled from he adapive weighed inensiies beween he blended image and he learned bakground and foreground model(s) of he GMM. The subsequen seion desribes inerpolaing virual view, GMM ehnique, adapive hole filling ehnique, and hoosing he value of weighing faor. A. Inerpolaing Virual View In our experimen, we assume ha he sender usually ransmis wo exure images and heir orrespondene deph maps of a same sene apured by wo ameras a he same insan. Generally, deph maps represen he disane of objes from he amera whih is quanized ino 256 differen values where 0 and 255 represen he farhes and neares disane respeively. The rue deph values Z are onvered from he enoded deph map by using (1) [2][7]: Z ZnearZ far. (1) Z Z Z 255 near far near

4 where Z far and near sene respeively. Z are he farhes and neares deph in a Then he dispariy (d) beween he referene view (i.e. adjaen views) and he virual view is deermined from he amera parameers suh as amera foal lengh (f) and baseline disanes (l) by using (2) fl (2) d. Z Afer ha, exure images are aligned in he virual posiion based on he alulaed dispariy values [7][25][25]. However, his aligned exure onains many holes due o rounding ineger error and he olusion problem. Warped images ( and ' 1 ' 3 ) are blended based on four ondiions o minimize he holes problems as follows: Case I: When here are no holes ino he warped exure ' 1 and warped exure 3 ', we are aking he average of he orresponding pixels. Case II: If here are no holes ino he warped exure ' 1, bu holes ino he warped exure ' 3, we are aking he pixel inensiy from he warped exure ' 1. Case III: If here are holes ino he warped exure, bu no holes ino he warped exure ' 3, we are aking he pixel inensiy from he warped exure ' 3. Case IV: If here are holes ino boh warped exures, we are onsidering pixel inensiy is equal o zero. This proedure redues he number of holes bu does no help o reover all missing pixel inensiies. To reover missing pixels, we model eah pixel by using he GMM ehnique using he previously generaed images in he virual view. B. GMM Tehnique The GMM ehnique is usually used for separaing bakground and foreground pixels a pixel level from dynami environmen, where eah pixel is modeled independenly by a mixure of K-h Gaussian disribuions (usual seing K=3) [27][28]. In our proposed ehnique, le us assume a ime, he value of k-h Gaussian inensiy = k,, mean = k,, variane = k 2,, and weigh in he mixure= k,, so ha k, 1 K k 1 ' 1. In our proposed ehnique, we se he iniial parameers from lieraure as follows [27][29]: sandard deviaion ( k ) = 2.5, weigh ( k ) =0.001 and learning rae, =0.1. A learning parameer 0<α<1 is used for balaning he onribuion beween presen and previous values of aforemenioned parameers. Afer iniializaion, he urren pixels are used o mah wih k-h Gaussian for every new observaion if he ondiion X k, 2. 5 k, is saisfied agains exising models, where X is he new pixel inensiy a ime. If a model mahes, he Gaussian model will be updaed as follows: k, (1 ) k, 1 X ; (3) 2 2 T k, ( 1 ) k, 1 ( X k, ) ( X k, k, ( 1 ) k, 1, and he weighs of oher Gaussians models are updaed as k, ( 1 ) k, 1. (6) Then, he value of weighs is normalized among all models in K suh a way ha 1. Conversely, if a model fails o k 1 k, mah, hen a new model is inrodued wih iniial parameer values. If i is already rossed he maximum allowable number of models, based on he value of weigh/sandard deviaion, he new model subsiues an exising model. If a pixel inensiy of a olor () saisfies a model (k), we sore he pixel inensiy as he reen value k, of he orresponding model and olor. Afer ha, we use his value o reover missing pixel values. C. Hole Filling If a pixel experienes only one model over he ime in differen frames, i represens sai bakground pixels, onversely, if a pixel experienes more han one models, i represens foreground and bakground pixels, where he highes value of weigh/sandard deviaion represens he mos sable bakground [2]. As he GMM has inheren apaiy o apure bakground and foreground pixel inensiies by exploiing emporal orrelaion, missing pixel inensiies of an oluded area are suessfully reovered. In he proposed ehnique, if a pixel has only one model, he pixel inensiy of he synhesized final image is aken from he reen value i.e. k, of he model and warped image. However, a video wih larger moving objes and high moions hanges he onen frequenly, as a resul, he models lose relevany wih he pas frames. In his senario, learned foreground using GMM does no provide adequae pixel inensiy for a virual view. Therefore, we need o rese he models afer a erain inerval. Oherwise, error propagaes hrough he whole sysems. On he oher hand, he pixel inensiy of he synhesized final image is aken as a weighed average from he blended image and he reen value of he model, whih provides he lowes value in erms of weigh/sandard deviaion. The deail of he inerpolaed image reovering ehnique using GMM is desribed below: Case 1: If a pixel experienes only one model over he whole duraion for a given olour, we sore he reen value k, of he olour for he final image synhesis by using ) ; ( ) ( ) k,. (7) (4) (5)

5 RAHAMAN e al.: VIRTUAL VIEW SYNTHESIS FOR FVV AND MVC COMPRESSION USING GMM where ξ is weighing faor (see deails alulaion of onsans used in (7) in he subseion D of seion II) and is he ouome of inverse mapping. Case 2: If a pixel experienes more han one model for a given olour over he duraion, wheher i would be he foreground or bakground pixels, iniially, we use he inverse mapping ehnique [7] o fill he holes. Then, we find he smalles disanes beween pixel inensiies of and he reen values 1,, 2, and 3, as follows: Δ 1 = Φ B 1, Δ 2 = Φ B 2, Δ 3 = Φ B 3, Δ = min (Δ 1, Δ 2, Δ 3 ) (8) If Δ = Δ 1, he pixel represens he bakground a ha momen for a given olour and we sore he reen value 1, of he olour for he final synhesis image by using (7) If Δ = Δ 2 or Δ = Δ 3 he pixel represens foreground a ha momen, herefore, we hoose a weigh faor (ξ) for seleing he fraion of he and he reen value of he seond or hird model ( 2, or 3, ) as follows: ( 1 ) k,. where he value of k is eiher 2 or 3. D. Adapive Weighing Faor In he proposed ehnique, we have observed ha differen values of weighing faor ξ provide differen qualiies of virual view (see Fig. 6). Thus, i is essenial o deermine he value of ξ in differen frames and videos. To deermine he value of ξ, we have ried o learn he faor whih influenes he value o provide a beer virual view. Our heory is ha if a video has larger foreground areas wih high moion, he video should have larger number of pixels wih muliple models in GMM. In his ase, he video should show a endeny o ake more pixel inensiies from he warped image ompared o he bakground image as he bakground image loses is relevane more Fig. 2: Trend of weighing faor (ξ). (9) frequenly over he ime. Thus, he value of ξ would be proporionae wih he number of muliple models in GMM. Through experimens, we have observed ha here is a posiive relaionship of he value of ξ wih he number of pixels wih muliple number of models of a frame. In his senario, learned foreground using GMM does no provide adequae pixel inensiy for a virual view, hus he value of ξ should be higher for hose ases as he warped image has more onribuion ompared o he learned foreground. We derive a relaionship beween muliple models and a weighing faor (ξ) for a number of videos and hen we use he relaion in eah frame of a video o adapively se he value of he weighing faor. The weighing faor is formulaed as a raio of he pixel number wih muliple models and he oal number of pixels of a frame in GMM, whih is given below: ξ=f(a) =(A 2+A 3)/(A 1+A 2+A3) *100 =A 2,3 (10) where A 1, A 2, A 3 and A 2,3 are he number of pixels wih one model, wo models, hree models and wo/hree models respeively. From (10) we fi a hird order polynomial (0.0004(A 2,3) (A 2,3) A 2, ) o derive he relaionship as shown in Fig. 2. Noe ha we have alulaed he weighing faor in (n-1) h frame and used i o generae n h virual frame. The main idea of he adapive weighing faor deerminaion is ha if a video has larger moving objes, he large onribuion omes from warped images ensure beer view synhesis. Moreover, i helps o rese modelling afer erain inerval depending on number of muliple models. Wih inreasing muliple models he onribuion of learned bakground/foreground redues o ensure beer view synhesis. In our experimen, when he value of ξ is 0.9 or higher we rese he modelling. The figure shows ha he value of ξ is very lose o 1 when he number of muliple models is lose o 13% or more i.e., here is no or lile onribuion of learned foreground o form a virual view if a frame has a higher moving obje. However, he learned bakground of GMM sill has a grea onribuion o form he virual frame in sai and unovered bakground areas. When we ompare he adapive weighing faor o generae virual view, we do no sarifie any signifian qualiy degradaion ompared o he maximum ahievable qualiy by seing he weighs from 0 o 1 (see Fig. 7). III. VIEW SYNTHESIS FOR MVC Adjaen views of muliview video sequenes are apured by muliple ameras wih slighly differen angles. Therefore, here are dispariies among he differen views. Moreover, o-loaed pixels/bloks a differen insanes of he same views are predied by he moion esimaion ehnique. However, finding o-loaed pixels/bloks on differen frames by using moion esimaion and dispariy esimaion is ime onsuming [30][31]. Therefore, a reduion of ompuaion for searhing moion parameers suh as moion veor is an imporan aspe of urren researh [32]-[34]. Thus, he bes poliy is reduing he

6 (n-1)-h Frame of View 2 Dynami Bakground Modelling Referene Blok (Xr3, Yr3) Moion Esimaion Synhesized Curren Frame Curren Blok (Xr4, Yr4) Synhesizing Referene Frame 1 Referene Frame 4 Referene Frame 3 Synhesizing Referene Frame 2 n-h Frame of View 1 Referene Blok (Xr1, Yr1) Dispariy Curren Blok (X, Y) Dispariy Referene Blok (Xr2, Yr2) n-h Frame of View 3 n-h Frame of View 2 / Curren Frame Fig. 3: Proposed MVC oding ehnique by using four referenes suh as (n-1) h frame of view 2, n h frame of view 1, n h frame of view 3 and he virual frame generaed by he proposed sheme. number of referene views. Tradiionally, hree referenes suh as already enoded frames of adjaen views (referene frames 1 and 2 in Fig. 3) and he previous frame of he urren view (referene frame 3 in Fig. 3) are used o enode eah frame of dependen views [3][24]. In his ehnique, a dispariy d is used o find a urren blok (X, Y ) on adjaen referene views ((X r1,y r1 ) and (X r2, Y r2 )) where X r1 = X d and X r2 = X ± d. This mehod only onsiders he horizonal omponen as muliview video sequenes are reified [3]. Furhermore, moion veors are predied o find a urren blok on he previous frame of he urren view i.e. (X r3, Y r3 ) [29] [35]. Insead of ypial approahes, we use he proposed view synhesis ehnique o generae a synhesized urren frame, whih is used as he fourh referene frame. This synhesized frame is almos similar in erms of obje posiion and is moion o he expeed urren frame. Therefore, we have four andidaes for hoosing eah blok o enode he urren frame of he middle view i.e. view 2 as shown in Fig. 3. As he fourh referene frame has more similar onen wih he urren frame ompared o he oher hree referene frames, i is expeed ha enoding he urren frame using four referene frames provides beer qualiy. Moreover, using four referene frames does no require signifian exra ompuaional ime ompared o hree referene frames as i does no require any dispariy or moion esimaion. To see he effeiveness of he proposed virual frame, we also onsider wo referenes suh as referene hree and referene four as shown in Fig. 3, i also provides beer prediion ompared o he radiional approahes. Obviously, wo referene sheme provides beer ompuaional ime ompared o hree referene frame sheme. IV. EXPERIMENTAL RESULTS In our experimen, PSNR is used o measure he squared inensiy differenes of synhesized and original image pixels. Then based on average PSNR performane, we ompare he ouome of he proposed mehod wih he sae-of-he-ar mehods namely View Synhesis Referene Sofware (VSRS) [36], inpaining [11], and he bakground updae ehnique [16]. Four sandard muliview video sequenes are seleed for esing he performane of he proposed ehnique. The inpu referene viewpoins, he virual viewpoin and he baseline are lised in Table1. Fig. 4 demonsraes he resuls. We use he same warping and blending ehniques for all ehniques for 100 frames using adjaen views, hen we apply an inpaining, bakground updae and he proposed mehod for refining he blended image. The figure shows ha he proposed ehnique provides beer performane ompared o he exising hole filling ehniques for all video sequenes. The improvemen range varies from 7.85dB o 11.69dB wih average improvemen 9.72dB for VSRS, 7.32dB o 8.85dB wih average improvemen 8.25dB for inpaining and 5.40dB o 7.65dB wih average improvemen 6.51dB for he bakground updae ehnique respeively. Furhermore, we ompare he ouome of our preliminary paper [2] wih he proposed ehnique. Fig. 5 shows ha he proposed ehnique ouperforms he preliminary ehnique in [2] for all video sequenes. In [2], only a single view was aken for warping, whereas he proposed ehnique uses wo adjaen views for warping. Moreover, he proposed mehod idenifies boh foreground and bakground pixel inensiies o refine a virual view when muliple models are used o model a pixel inensiies. The model whih provides he minimum differene in pixel inensiies beween he blended image and he differen models in GMM for a given momen, represens foreground or bakground. However, in he previous mehod, i was onsidered as a foreground pixel inensiy hroughou. Tha s why he PSNR of he virual view using he proposed mehod inreases wih number of frames ompared o he ehnique in [2] (see Fig. 5). Table 1: Tes sequenes, synhesized viewpoins and baseline Sequenes Inpu Targe Baseline Referene Viewpoins Viewpoin Newspaper 6, Lovebird1 8, Poznan Sree 5, Book Arrival 10, Fig. 4: Average PSNR omparison for 100 frames.

7 RAHAMAN e al.: VIRTUAL VIEW SYNTHESIS FOR FVV AND MVC COMPRESSION USING GMM Fig. 5: PSNR Comparison of he ehnique in [2] agains he proposed ehnique for Newspaper (NP), Lovebird1 (LB), Poznan Sree (PS) and Book Arrival (BA) video sequenes. We analyzed PSNR agains differen values of ξ i.e. 0 o 1. The onribuion of he GMM models and he blended images in he proposed approah o reonsru he final synhesized image is shown in Fig. 6 for 30 h frame of eah video sequene. The figure reveals ha he learned foreground of GMM has some onribuion o generae a final image for eah video. Noe ha if we ge he maximum PSNR value of a given image where he value of ξ is 0.6 (Lovebird1), i means ha he pixel inensiies of he 60% and 40% foreground are aken from blended image and learned foreground respeively. A fixed hreshold may work for some frames, however, we observed ha he hreshold is varied for oher frames. Thus, i is ruial o use adapive hreshold raher han a fixed hreshold. We also ompared he maximum PSNR and adapive PSNR agains predied frames for Newspaper, Lovebird1, Poznan Sree (up o 100 frames) and Book Arrival (full sequenes), whih is shown in Fig. 7. More speifially, Newspaper, Lovebird1, Poznan Sree and Book Arrival video sequenes sarifie 0.10dB, 0.08dB, 0.16dB and 0.23dB PSNR on average. The slope of his urve is onrollable by hanging he values of ξ. If he value of ξ is greaer han or equal o 0.9, we rese he modelling in his experimen as he onribuions from he GMM models redue (i.e. 0.1 or less). In our experimen, he modelling is rese 10, 5, 9 and 14 imes for 100 frames for Newspaper, Lovebird1, Poznan Sree and Book Arrival video sequenes respeively. Normally, due o he onen of he video sequene, oasionally he PSNR of he view synhesis Fig. 6: Weighing faor (ξ) vs PSNR (db). Fig. 7: Maximum PSNR (db) vs adapive PSNR (db) for 100 frames for four video sequenes by learning original frames. drops or rises over he ime. The proposed rese sraegy an handle his rend of hanging PSNR as he onribuions of he pixel inensiies from he model(s) redues over he ime due o he dynami bakground and moions of he moving objes. Fig. 8 illusraes he subjeive qualiy for Newspaper video sequene. Fig. 8 (a) shows he original images, i.e. 10 h original frame of he virual view and he green reangular boxes are used o mark he ropped and zoomed porion whih is shown in Fig. 8 (b) and (). Similarly, Fig. 8 (d), (g), (j) and (m) shows he view synhesis by VSRS, inpaining, bakground updae and he proposed ehnique and Fig. 8 (e), (f), (h), (i), (k), (l), (n) and (o) shows orresponding ropped and zoomed images. These figures demonsrae ha he proposed ehnique is able o generae a beer view synhesis ompared o he hree sandard mehods. To see he srengh of he proposed mehod, we have applied he proposed sheme in is urren form o he synhesized images for modelling and generaed view synhesis. We have used synhesized images generaed by he inverse mapping ehnique for GMM. Fig. 9 shows ha he proposed ehnique improves he qualiy of he synhesized view. This ehnique improves 0.57dB, 0.15dB, 0.27dB and 0.32dB PSNR on average for Newspaper, Lovebird1, Poznan Sree and Book Arrival video sequenes ompared o he inverse mapping ehnique respeively. If we ge good qualiy images for learning GMM i should provide beer synhesized image. As he parameers of he proposed ehnique are no opimized for he synhesized views, i gives us moderae improvemen. Fig. 10 demonsraes he subjeive qualiy of he proposed ehnique when we learned he oupu of he inverse mapping ehnique [7]. I shows ha he proposed ehnique provide beer synhesis images. To undersand he effeiveness of he proposed view synhesized ehnique in he moving bakground sequenes, we have ondued experimens using Balloons, Kendo, Poznan Hall2 and Undo Daner video sequenes wih 50 frames. Fig. 11 shows he performane of he proposed mehod ompared o oher reen and relevan mehod [7]. The figure reveals ha he proposed mehod performs beer for Ballons and Kendo videos sequenes bu no for oher wo sequenes as he firs wo sequenes have relaively less moving bakground ompared o oher wo. Thus, he proposed ehniques sill provide beer resuls if he moving bakground is no oo srong.

8 (a) Original frame (b) Crop and zoom image of original frame () Crop and zoom of original frame (d) VSRS Renderer [36] (e) VSRS Renderer [36] (f) VSRS Renderer [36] (g) Inpaining [11] (h) Crop and zoom image of inpaining ehnique [11] (i) Crop and zoom image of inpaining ehnique [11] (j) Bakground Updae [16] (k) Crop and zoom image of bakground updae ehnique [16] (l) Crop and zoom image of bakground updae ehnique [16] (m) Proposed (n) Crop and zoom image of proposed ehnique (o) Crop and zoom image of proposed ehnique Fig. 8: Original image (a), synhesis images (d, g, j and m), rop and zoom images (b,, e, f, h, i, k, l, n and o) for Newspaper video sequene by he proposed mehod hree sandard mehods.

9 RAHAMAN e al.: VIRTUAL VIEW SYNTHESIS FOR FVV AND MVC COMPRESSION USING GMM Fig. 9: Inverse mapping vs adapive PSNR (db) for four video sequenes when learning he oupu of inverse mapping for 50 frames. To enode differen resoluions and a wide range of video onen for differen views in 3D-HEVC, eah frame is divided ino a number of bloks wih various sizes suh as 8 8, 16 16, and [1] pixels and he searh lengh beome 8, 16, 32, 64 and 128 pixels. In our experimen, we have onsidered and pixel blok sizes and 64 pixel searh lenghs o demonsrae he performane of he proposed four and wo referene shemes ompared o he exising hree referene sheme. Due o he beer prediion of he synhesized view, he proposed ehnique provides beer PSNR ompared o he onvenional approahes, whih are shown in Fig. 12(a) for and Fig. 12(b) for I reveals ha he PSNR improvemens for wo referenes and four referenes are varied from 3.18 o 4.95dB, when blok sizes are pixels. Similarly, when he blok sizes are pixels, he PSNR improvemens for wo referenes and four referenes are varied from 3.15dB o 5.14dB. The four referene sheme provides beer PSNR ompared o he wo referene and hree referene shemes. Moreover, for he performane jusifiaion of he proposed wo referenes and four referenes mehods for eigh video sequenes, we have generaed he RD performane urve using differen QPs i.e. 22, 27, 32 and 37 in he senario of MVC as shown in Fig. 13. We have used 3D-HEVC sruure where he firs view and hird view are enoded using HEVC oding framework. The middle view is enoded using he wo adjaen iner-view images, immediae previous inra-view image, and synhesized image. All referene frames are generaed from he reonsrued (i.e. deoded) referene frames, so ha, boh enoder and deoder have he same referene frames. Fig. 13 shows he resuls for he middle view. We ompared he Fig. 11: Average PSNR omparison for moving bakground video sequenes for 50 frames using he adapive weighed view synhesized ehnique. srengh of he proposed mehod in erms of generaing beer synhesized view whih is used as one of referene frames for oding purpose. The experimenal resuls illusrae ha he proposed ehniques improve RD performane signifianly in all video sequenes by improving he qualiy of he synhesized views. Furhermore, he performane of he four referenes and wo referenes ehniques agains he hree referenes ehnique are evaluaed based on he Bjønegaard-Dela Bi Rae (BD-BR) and Bjønegaard-Dela PSNR (BD-PSNR) [37] in Table 2, where + and - sign indiae he inremen and deremen respeively. Over eigh differen video sequenes, he four referenes and wo referenes ehniques provide gains 1.07dB and 0.88dB BD-PSNR, while dereasing 29.68% BD-BR and 26.06% BD-BR on average respeively ompared o he (a) pixels blok size moion esimaion (a) (b) () Fig. 10: (a) Original image, (b) oupu images of inverse mapping ehnique and () proposed image afer learning inverse mapping oupu. (b) (64 64) pixels blok size Fig. 12: PSNR omparison for he proposed MVC sheme.

10 Table 2: The performane of he proposed four referenes and wo referenes mehods agains he exising hree referenes ehnique using BD- bi rae and BD-PSNR. Sequenes Four Referenes Two Referene BD-PSNR (db) BD-BR (%) BD-PSNR (db) BD-BR (%) Newspaper Lovebird Poznan Sree Book Arrival Balloons Kendo Poznan Hall Undo Daner Average onvenional hree referenes ehnique. The proposed ehnique ouperforms for all video sequenes in erms of boh improving he BD-PSNR and reduing he BD-BR. MVC leads o high ompuaional omplexiy, whih limis is appliaion on low power onsumpion eleroni devies suh as smar phones [29]. Toal enoding ime heavily depends on moion and dispariy esimaion. Researh shows ha here are no signifian ime differenes for esimaing moion and dispariy [30][38]. Moreover, MVC exhausively heks a number of iner/inra modes for a oding uni o sele a bes mode o enode he oding uni. This proedure inreases omplexiy muliple imes ompared o he uni-mode ehnique [39]. Therefore, any ehnique whih skips dispariy esimaion and/or moion esimaion should redue ime omplexiy. The proposed wo referene ehnique skips dispariy esimaion and improves PSNR ompared o he hree referene ehnique. Alhough he proposed ehnique needs exra ompuaional ime for synhesis virual view, i redues overall ime omplexiy for MVC by 28.95% whereas he four referene sheme requires an exra 18.42% ime on average ompared o he exising hree referene sheme (see Fig. 14). In boh ases, he proposed ehnique ouperforms he hree referene sheme in erms of image qualiy for a given bi rae. All experimens are ondued by a dediaed deskop mahine DELL OPTIPLEX 9020 (wih Inel ore i GHz, 8 GB RAM and 250 GB HDD) running 64-bi Windows 7 operaing sysem. Aording o he rae-disorion performane (see Table 2 and Fig. 13), he proposed four-referene ehnique ouperforms he proposed wo-referene ehnique for all video sequenes. The signifian performane gains are observed for he video sequenes wih amera moions e.g. Undo Daner, Kendo, Balloons e. However, he proposed four-referene ehnique requires around 47% exra ompuaional ime ompared o he wo-referene ehnique. Thus, our reommendaion is o use four-referene ehnique for he senarios wih video sequenes wih amera moions and no onern of he ompuaional ime requiremens, oherwise, use wo-referene ehnique.

11 RAHAMAN e al.: VIRTUAL VIEW SYNTHESIS FOR FVV AND MVC COMPRESSION USING GMM frames as an exra referene frame for MVC, whih improves he qualiy of he enoded frame on average 0.73dB ompared o he sandard ehnique. Anoher version of he proposed ehnique provides 0.68dB image qualiy improvemen wih redued ompuaional ime ompared o he exising MVC ehnique. Fig. 13: Rae-disorion performane relaionship using hree referenes, wo referenes and four referenes ehnique for four video sequenes where reonsrued deoded frames are used as referene frames. V. CONCLUSION In his paper, we presen a new view synhesis ehnique ha explois emporal orrelaion for hole filling. For our proposed ehnique, views are inerpolaed from adjaen exure images and heir orresponding deph maps. Inerpolaed images onain many holes due o he olusion and rounding ineger problem. Usually, spaial and/or emporal orrelaion-based ehniques e.g. inpaining and bakground updae are used o address hese issues. However, hese ehniques suffer qualiy degradaion due o he low spaial orrelaion in he boundary areas of he foreground and bakground. To address hese issues, in he proposed ehnique, we use a number of models in GMM o separae bakground and foreground pixels. The missing pixels are reovered from he adapive weighed average of he pixel inensiies from he orresponding model(s) of he GMM and he warped image. Experimenal resuls reveal ha he proposed ehnique provides 9.72dB, 8.25dB and 6.51dB PSNR improvemen on average ompared wih he VSRS, inpaining and bakground updae ehniques respeively. To evaluae he effeiveness of he proposed view synhesis ehnique, we used view synhesis from deoded Fig. 14: Enoding ime speedup he proposed four and wo referene ehniques ompared o he radiional hree referene ehnique for MVC. REFERENCES [1] G. Teh, Y. Chen, K. Muller, J.-R. Ohm, A. Vero, and Y.-K. Wang, Overview of he muli-view and 3D exensions of high-effiieny video oding, IEEE Transaions on Ciruis and Sysems for Video Tehnology, vol. 26, no. 1, pp , Jan [2] D M M. Rahaman, and M. Paul, Hole-filling for single-view plus-deph based rendering wih emporal exure synhesis, IEEE Inernaional Workshop on Ho Topis in 3D - Ho 3D (in onjunion wih ICME), pp.1-6, DOI: /ICMEW [3] K. Muller, H. Shwarz, D. Marpe, C. Barnik, S. Bosse, H. Brus, T. Hinz, H. Lakshman, P. Merkle, F. H. Rhee, G. Teh, M. Winken, and T. Wiegand, 3D high effiieny video oding for muli-view video and deph daa, IEEE Transaions on Image Proessing, vol. 20, no. 9, pp , [4] F. Zou, D. Tian, A. Vero, H. Sun, O. C. Au, and S. Shimizu, View Synhesis Prediion in he 3-D Video Coding Exensions of AVC and HEVC, IEEE Transaions on Ciruis and Sysems for Video Tehnology, vol. 24, no. 10, pp , [5] G. Luo, Y. Zhu, Z. Li, and L. Zhang, A hole filling approah based on bakground reonsruion for view synhesis in 3D video, IEEE Conferene on Compuer Vision and Paern Reogniion, pp , [6] D M M. Rahaman, and M. Paul, Free view-poin video synhesis using Gaussian mixure modelling, IEEE onferene on Image and Vision Compuing New Zealand, [7] S. Farid, M. Luenefore, and M. Grangeo, Deph image based rendering wih inverse mapping, IEEE Inernaional Workshop on Mulimedia Signal Proessing, pp , [8] L. Zhan-Wei, A. Ping, L. Su-xing, and Z. Zhao-yang, Arbirary view generaion based on DIBR, Inernaional Symposium on Inelligen Signal Proessing and Communiaion Sysems, pp , [9] C. M. Cheng, S. J. Lin, S.-H. Lai, and J. C. Yang, Improved novel view synhesis from deph image wih large baseline, Inernaional Conferene on Paern Reogniion, pp. 1 4, [10] A. Oliveira, G. Fikel, M. Waler, and C. Jung, Seleive hole-filling for deph-image based rendering, IEEE Inernaional Conferene on Aousis, Speeh and Signal Proessing, pp , April [11] A. Criminisi, P. Pérez, and K. Toyama, Region filling and obje removal by exemplar-based image inpaining, IEEE Transaions on Image Proessing, vol. 13, no. 9, pp , [12] D. H. Li, H. M. Hanh, and Y. L. Liu, Virual view synhesis using bakward deph warping algorihm, Piure Coding Symposium, pp , [13] C. Yao, Y. Zhao, and H. Bai, View synhesis based on bakground updae wih gaussian mixure model, Paifi-Rim Conferene on Mulimedia, pp , [14] Y. Gao, G. Cheung, T. Maugey,P. Frossard, and J. Liang, Enoderdriven inpaining sraegy in muliview video ompression, IEEE Transaions on Image Proessing, vol. 25, no. 1, pp , [15] I. Daribo and B. Pesque-Popesu, Deph-aided image inpaining for novel view synhesis, IEEE Inernaional Workshop on Mulimedia Signal Proessing, pp , [16] C. Yao, T. Tillo, Y. Zhao, J. Xiao, H. Bai, and C. Lin, Deph Map Driven Hole Filling Algorihm Exploiing Temporal Correlaion Informaion, IEEE Transaions on Broadasing, vol. 60, no. 2, pp , [17] M. Paul, W. Lin, C. T. Lau, and B. S. Lee, Explore and model beer I- frames for video oding, IEEE Transaions on Ciruis and Sysems for Video Tehnology, vol. 21, no. 9, pp , [18] C. Zhu, and S. Li, Deph Image Based View Synhesis: New Insighs and Perspeives on Hole Generaion and Filling, IEEE Transaions on Broadasing, vol. 62, no. 1, pp , [19] P. Pandi, A. Vero, Y. Chen, Join Muliview Video Model (JMVM) 7 Referene Sofware, N9579, MPEG of ISO/IEC JTC1/SC29/WG11, Analya, Jan

12 [20] M. Talebpourazad, 3D-TV onen generaion and muli-view video oding, Ph.D. hesis, [21] S. Yea, and A. Vero, View synhesis prediion for muliview video oding, Signal Proessing: Image Communiaion, 24, pp , [22] S. Ma, S. Wang and W. Gao, Low omplexiy adapive view synhesis opimizaion in HEVC based 3D video oding, IEEE Transaions on Mulimedia, vol. 16, no. pp , [23] B. T. Oh, and K. J. Oh, View synhesis disorion esimaion for AVCand HEVC-ompaible 3-D video oding, IEEE Transaions on Ciruis and Sysems for Video Tehnology, vol. 24, no. 6, pp , [24] A. I. Puria, E. G. Mora, B. P. Popesu, M. Cagnazzo, and B.Ionesu, Muliview plus deph video oding wih emporal prediion view synhesis, IEEE Transaions on Ciruis and Sysems for Video Tehnology, DOI: /TCSVT [25] H. Yuan, Y. Chang, J. Huo, F. Yang, and Z. Lu, Model-based join bi alloaion beween exure videos and deph map for 3-D video oding, IEEE Transaions on Ciruis and Sysems for Video Tehnology, vol. 21, no.4, pp , [26] P. Gao, and W. Xiang, Rae-disorion opimized mode swihing for error resilien muli-view video plus deph based 3-D video oding, IEEE Transaions on Mulimedia, vol. 16, no. 7, pp , [27] M. Haque, M. Murshed, and M. Paul, Improve Gaussian mixures for robus obje deeion by adapive muli-bakground generaion, IEEE Inernaional Conferene on Paern Reogniion, pp. 1-4, [28] S. Lee, Effeive Gaussian mixure learning for video bakground subraion, IEEE Transaions on Paern Analysis and Mahine Inelligene, vol. 27, no. 5, pp , [29] M. Paul, Effiien muli-view video oding using 3D moion esimaion and virual frame, Neuroompuing, pp , [30] X. Xu, and Y. He, Fas dispariy moion esimaion in MVC based on range prediion, IEEE Inernaional Conferene on Image Proessing, pp , [31] J. Seo, and K. Sohn, Early dispariy esimaion skipping for muli-view video oding, EURASIP Journal on Wireless Communiaions and Neworking, pp. 1-12, [32] X. Guo, Y. Lu, F.Wu, and W. Gao, Iner-view dire mode for muliview video oding, IEEE Transaions on Ciruis and Sysems for Video Tehnology, vol. 16, no. 12, pp , [33] J. Koniezny, and M. Domanski, Deph-based iner-view prediion of moion veors for improved muliview video oding, 3DTV- Conferene: The True Vision-Capure, Transmission and Display of 3D Video, pp. 1 4, [34] H.-S. Koo, Y.-J. Jeon, and B.-M. Jeon, Moion skip mode for MVC, ITU-T and ISO/IEC JTC1, Teh. Rep. JVT-U091, [35] F. Shao, G. Jiang, M. Yu, K. Chen, and Y. S. Ho, Asymmeri oding of muli-view video plus deph based 3-D video for view rendering, IEEE Transaions on Mulimedia, vol. 14, no. 1, pp , [36] View synhesis referene sofware 3.5, ISO/IEC JTC1/SC29/WG11 (MPEG). [37] G. Bjonegaard, Calulaion of Average PSNR Differenes Beween RD urves, ITU-T SC16/Q6, VCEG-M33, Ausin, USA, [38] J. Seo, and K. Sohn, Early dispariy esimaion skipping for muli-view video oding, EURASIP Journal on Wireless Communiaions and Neworking, DOI: / [39] P. K. Podder, M. Paul1, and M. Murshed, A Novel Moion Classifiaion Based Inermode Seleion Sraegy for HEVC Performane Improvemen, Neuroompuing, pp , Manoranjan Paul (M 03 SM 13) reeived Ph.D. from Monash Universiy in He has worked as a Researh Fellow a he Universiy of New Souh Wales, Monash Universiy and Nanyang Tehnologial Universiy during 2005 o He is urrenly working as an Assoiae Professor in he Shool of Compuing and Mahemais, Charles Sur Universiy. His major researh ineress are in he fields of image/video oding/ompression, EEG signal analysis, and ompuer vision. He has published more han 150 refereed papers in inernaional journals and onferenes. A/Professor Paul regularly published journal ariles in he IEEE Transaions, whih are onsidered as he op ranked journals in he relevan fields. He was a keynoe speaker a IEEE DICTA 2017, WOWMOM Workshop 2014, DICTA 2013, and ICCIT A/Professor Paul is a Senior Member of he IEEE and ACS. He has served as a Gues Edior of he Journal of Mulimedia and Journal of Compuers for five speial issues. Currenly A/Professor Paul is an Assoiae Edior of he EURASIP Journal on Advanes in Signal Proessing. He is seleed as an ICT Researher of he Year 2017 (Finalis) by Ausralian Compuer Soiey. M Moiur Rahaman has ompleed his B.S. degree from he Deparmen of Elerial and Eleroni Engineering, Rajshahi Universiy of Engineering & Tehnology, Rajshahi, Bangladesh. Afer aomplishing his degree, he joined as a leurer in he Elerial, Eleronis and Teleommuniaion Deparmen, Dhaka Inernaional Universiy, Dhaka, Bangladesh. He is a suden member of he IEEE. Currenly, he is working as a Ph.D. suden a Charles Sur Universiy, Ausralia. He has published several papers in he area of image proessing and video oding.