Cross-Based Local Multipoint Filtering

Transcription

1 Coss-Based Local Multioint Filteing Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Cente, 2 Sun Yat-Sen Univesity, 3 Univ. of Illinois at Ubana-Chamaign Abstact This ae esents a coss-based famewo of efoming local multioint filteing efficiently. We fomulate the filteing ocess as a local multioint egession oblem, consisting of two main stes: 1) multioint estimation, calculating the estimates fo a set of oints within a shae-adative local suot, and 2) aggegation, fusing a numbe of multioint estimates available fo each oint. Comaed with the guided filte that alies the linea egession to all ixels coveed by a fixed-sized squae window non-adatively, the oosed filteing famewo is a moe genealized fom. Two secific filteing methods ae instantiated fom this famewo, based on iecewise constant and iecewise linea modeling, esectively. Leveaging a coss-based local suot eesentation and integation technique, the oosed filteing methods achieve theoetically stong esults in an efficient manne, with the two main stes comlexity indeendent of the filteing enel size. We demonstate the stength of the oosed filtes in vaious alications including steeo matching, deth ma enhancement, edge-eseving smoothing, colo image denoising, detail enhancement, and flash/no-flash denoising. 1. Intoduction Edge-eseving o stuctue-eseving smoothing filteing is a desied oety and a ey comonent fo many comute vision and gahics alications. Basically, the goal of edge-eseving smoothing filteing is to seaate the main signal/image stuctues fom the measuement noise o fine details, wheeby the stuctue/edge should be well eseved and small fluctuations ae smoothed out. Fom a geneal esective, the signal to be filteed can tae diffeent foms such as an inut colo image with additive noise [8], o a cost slice/volume tyically constucted in discete labeling tass (e.g., steeo [18]). Liewise, the stuctue used to guide the filteing ocess can also be defined in a boad way. Fo instance, the stuctue encoded in the inut colo image itself, o that in anothe guidance signal of high signal-to-noise atio (SNR) aligned to the filte inut. The latte case is also nown as joint o coss filteing [12]. This wo was done when Keyang Shi was an inten at ADSC. The bilateal filte (BF) [15] is aguably the most oula edge-eseving smoothing filte that is widely adoted in a vaiety of alications [11]. As a local, non-iteative filte, BF is intuitively simle. Using a ange aamete σ and a satial aamete σ s, BF decides an adative weight fo each oint within a local squae window. The final filteing outut fo the cente ixel is simly comuted as a weighted aveage of these neighboing ixels. Based on the taxonomy by Katovni et al. [7], BF is a local ointwise estimato in that it gives the estimate fo a single oint only i.e., the cente ixel. BF uses a iecewise constant modeling whee a zeo-ode local olynomial aoximation is alied. Recently, the guided filte (GF) [6] was oosed. Being comutationally much faste than BF, GF has also demonstated its unique advantage ove BF in some alications such as detail enhancement and HDR comession. Moeove, when alied to cost volume filteing (e.g., steeo matching), GF has achieved state-of-the-at esults among local steeo methods [14]. Though the authos did not mae it exlicit in [6], GF is essentially a local multioint estimato accoding to [7]. In such a case, the estimates ae calculated fo all obsevation oints used by the estimato. Since tyically a numbe of such estimates ae available fo each oint, they ae aggegated (fused) togethe to comute the final estimate. This sot of edundant aoximations with multile estimates fo each oint is found to be dastically bette than any of the windowed estimates [7]. Unlie BF, GF efoms a fist-ode local linea modeling. Insied by the stong theoetical develoment in the image denoising field [7], we cast geneal-uose edgeeseving smoothing filteing unde a novel and boad famewo of local multioint filteing. Poviding new theoetical undestandings and extensions, the oosed famewo comises two majo stes: 1) multioint estimation, calculating the estimates fo a set of oints within a shaeadative local suot, and 2) aggegation, fusing a numbe of multioint estimates available fo each oint. Using satial adativity to define local suot egions and weighted aveaging to fuse multile estimates ae two ey genealizations in the oosed famewo. Theefoe, GF is a secial instance of this famewo in that fixed-sized squae windows and simle aveaging fo multi-estimate fu /12/$ IEEE 430

2 Gound-tuth 10% AWGN 2 h q 2 0 h h W H(q) H() 1 h q 3 h V() Ω 0 h q Ω s ' Ω Ω ' Ω Figue 2. Coss-based local multioint filteing. (a) Multioint estimation fo the local egion Ω anchoed at oint. (b) Aggegation of multile estimates contibuted by each Ω. (c) The oosed multi-estimate fusion aoach deals with the concave stuctue bette than using the single estimate fom alone [19]. Ω GF: RMSE=17.73,SSIM=0.69 Ous: RMSE = 9.47,SSIM=0.81 Figue 1. GF [6] does not emove the noise nea the edges well, while the oosed filteing method does a much bette job. All the aametes in both algoithms have been faily set. sion ae emloyed. Futhe, we oose two secific edgeeseving filtes as novel instantiations fom this geneic famewo, using a iecewise constant and iecewise linea aoximation esectively to model the signal locally. They demonstate bette functional efomance than GF and BF in a ange of comute vision and gahics tass. By leveaging and genealizing the coss-based local suot decision and integation technique [19], the oosed filtes efom the zeo-ode o fist-ode local olynomial modeling ove ointwise-adative suot egions efficiently. The two majo stes comlexity is indeendent of the filteing enel size, based on the integal image technique [4]. Also accommodating GF, ou famewo is vesatile and can be flexibly configued to give the best vaiant. 2. Related Wo and Motivation Inteested eades ae efeed to [11, 7, 6, 14] fo a detailed eview of edge-eseving smoothing filtes and local multioint filteing, coveing both theoy and alications. Hee we focus on thee most elevant filteing techniques. Due to the edge-eseving smoothing oety as well as its simlicity, BF [15] has been effectively emloyed in many alications [11]. Howeve, because of the iecewise constant modeling used, BF geneates the staicase effects in image smoothing oeations [3]. Anothe nown issue is the gadient evesal atifacts, caused by insufficient local suot aound tansitional edges [6]. Comutational efficiency is yet anothe challenge. As ointed out in [6], quantization-based fast imlementations [13, 17] achieve satisfactoy seed at the cost of quality degadation. Coss-based local suot decision and fast cost aggegation method wee oosed in the context of steeo matching [19]. Based on a comact, ixel-wise vaying local coss eesentation, the matching costs can be aggegated ove a shae adative full suot egion using two othogonal integation stes in O(1) time [4]. This method achieves the comaable disaity accuacy with the adative-weight method [18], but uns dozens of times faste. Late, this efficient and effective cost aggegation method has also been adoted as a fundamental building bloc in the to-aning steeo method [9]. Howeve, othe than steeo, its alications to othe oblems have not been well exloed yet. Also, the ointwise estimate based on had weighting may not be fine enough fo seveal gahics alications. As a secial case of moe geneal local multioint estimatos [7], GF [6] has shown its quality and seed advantages ove BF. It has also been successfully alied to fast cost volume filteing [14]. Howeve, GF is not without oblems. Fist, the local linea modeling becomes vey ineffective, when moe than one models ae esent in most of the local windows coveing the oint to be filteed, as GF does not classify the local samles disciminatively into a few classes. Also, when the tue signal is actually chaacteized by sha tansitions (e.g., deth discontinuities), GF esults in undesied fuzzy object bounday in alications lie deth ma enhancement. Fom a esective of statistics, it is clea that GF simly aveages multile estimates fo the cente ixel equally, without using any notion of linea egession quality o quality-of-fit fo weighted fusion. Without satial adativity, GF cannot be easily adated to a iecewise constant model fo the seed o quality uoses. Fig. 1 shows the deficiency of GF in denoising a colo image, and we will discuss the eason and insights in Sect Coss-Based Local Multioint Filteing 3.1. Definition and Algoithm Oveview Befoe esenting the oosed filtes, we fist define the following notations. The signal/image to be filteed is denoted as Z, the guidance image as I, and the filte outut image as Y. Note that I and Z can be identical, if the guidance image is the filte inut itself. Let be the ixel index of the estimation oint, and be an obsevation oint o suot ixel used by the estimato. As shown in Fig. 2(a), Ω W, whee Ω delineates an abitaily-shaed, connected local suot egion fo ancho oint, based on some citeia (e.g. [19]). W is a squae window of a adius 431

3 , defining the maximum satial extent as in GF [6]. The oosed coss-based local multioint filteing (CLMF) famewo consists of a few stages. Fist, fo each ixel, a set of fou vaying suot am lengths ae decided on the guidance image I, i.e., {h 0,h 1,h 2,h 3 }, the so-called coss seleton in [19]. We will esent an imoved stategy fo adative scale selection in Sect Once such a ixelwise coss seleton is decided, a shae-adative full suot egion Ω is eadily available as an aea integal of multile hoizontal segments H(q) sanned by ixel q [19]. Secifically, Ω = q V () H(q), whee q is a suot ixel located on the vetical segment V () defined fo ixel, as shown in Fig. 2(a). Afte this eocessing stage, CLMF efoms two main filteing stes sequentially: 1) multioint estimation, calculating the estimates Ys fo a set of oints s Ω within a locally adative egion anchoed on oint, and 2) aggegation, fusing all the estimates {Y } deived fom each -centeed local egion ( Ω ) to comute the final estimate (filte outut) Y fo each oint Genealization of Local Multioint Filteing Assuming a ixel-wise shae-adative suot egion Ω fo each oint is given, we fist esent a genealized famewo of local multioint filteing hee. Extending the local linea model adoted in GF [6], a local olynomial of ode m is fit between the guidance I s (indeendent vaiable) and the obsevations Z s with s Ω, fo each ancho ixel. We denote the fitted coefficient vecto with a m = [a0,a1,...,am ]. In this ae, we only conside the cases of m 1, so the zeo-ode (m = 0) o fist-ode olynomial model (m = 1) fo each s Ω is given as { Ys = a m I m a 0 m = 0 s = a 0 + a1 I (1) s m = 1. whee I m s = [1,I s,...,is m ] T. Simila with GF, we use the method of least squaes to fit the data, while enfocing that the model should be biased towad low-ode olynomials to avoid ove-fitting and gadient incease. Secifically, we minimize the following quadatic function: E(a m ) = ((a m I m s Z s ) 2 + ǫ (a i ) 2 ). (2) s Ω i [1,m] ǫ is a egulaization aamete to discouage the choices of lage a i (i 1). When m=0, the solution to (2) is given by a 0 a 0 = Z = 1 Z s, (3) Ω s Ω whee Ω is the numbe of ixels in Ω. When m = 1, the linea coefficients a 1 = [a0,a1 ] ae given as follows, 1 a 1 Ω = s Ω I s Z s µ Z σ 2 + ǫ, (4) a 0 = Z a 1 µ, (5) whee µ and σ 2 ae the mean and vaiance of I in Ω. Z is the mean of Z in Ω as given in (3). To genealize (4,5) to the case of a colo guidance image, the 3 3 covaiance matix Σ and colo vectos (e.g., I s ) ae used as in GF [6]. In contast to a ointwise estimato (e.g., BF) that gives the estimate fo the cente ixel only, the multioint estimato hee is to calculate an estimate Ys fo all obsevation oints used, i.e., s Ω. Fo a given ixel, as it is geneally coveed by multile ovelaing egions, it is involved in thei esective linea egessions. It hence has a numbe of multioint estimates {Y Ω }. Diffeent fom GF [6] whee these multioint estimates ae simly aveaged togethe to comute the final estimate Y, the oosed CLMF famewo taes into account of the fitting quality o the confidence of each multioint estimate in the aggegation ocess. Secifically, the final estimate fo each ixel is given as a weighted aveage of these multioint estimates: : Ω Y = w Y, (6) : Ω w whee w is the elative weight associated with each multioint estimate Y. As Ω is intended to involve only the data oints (inlies) that follow the simila signal stuctue o distibution in I [19], the outlies in the local squae window W ae ejected effectively. Theefoe, the fitting eo (esidue) at evey oint s Ω can be consideed as i.i.d. Gaussian noise, i.e., Z s =Y s +η s, η s N(0,γ 2 ). Nomally, moe data oints (inlies) lead to a statistically moe stable estimate of the tue signal, as the andom estimation eo deceases. We hence set the value of w in (6) ootional to the numbe of samle oints Ω, though moe sohisticated methods to decide w exist [7]. As a esult, the aggegation of all multioint estimates fo ixel is given by : Ω Y = Ω Y. (7) : Ω Ω Fo moe egula comutation and data access attens, it is advantageous to tansfom the summations in (7) into the following -ixel centic summations aoximately: Ω Y Ω Y. (8) Ω Ω Note that the above tansfomation may not lead to exactly the same final estimate as in (7). The eason is that fo a moe geneal way of defining Ω /, the mutual membeshi may not always hold, though most of time it does. Moe ecisely, thee could exist that Ω, but / Ω, o sometimes Ω, but / Ω (e.g., in Fig. 2(b)). As discussed late, with this tansfomation, the multioint estimate fusion ocess can be acceleated in the same way as the linea egession ocess fo each suot egion [6]. Fo GF, as the mutual membeshi is decided symmetically by equiing, (8) gives the same esult as (7). 432

4 2 h 1 =1 a 1 0 Y = a I + a h ' 1 =1 a Y = a I + a 1 ~ h Figue 3. 1-D illustation fo the eason and solution fo gadient evesal atifacts. (a) Unbalanced am lengths, maing the linea egession biased to the geen oints. (b) Symmetic am lengths Adative Scale Selection The goal of adative scale selection is to decide fo each diection an aoiate am length, so they jointly delineate a shae-adative local suot egion [7]. Unlie the oiginal coss-based method [19] whee I is used fimly as the efeence value fo colo similaity evaluation, we udate it by the unning aveage of the intensity of all the ixels coveed by the cuent san h. This change maes the scale decision moe obust against the measuement noise and the scales moe extensible. Conside the ight am h 0 fo ixel =(x,y ). The oosed udating function is given as Î (h) ~ h 0 = (1 α)î(h 1) + αi +(h,0), (9) whee Î(0) = I. α is a aamete to contol the udating ate. With an initial value set to 0, the otimal ight am length is decided as the lagest ight san h [1,] that satisfies the following equiements: j [1,h ],δ(i +(j,0) ;Î(j 1),τ) = 1, and δ(i +(h +1,0);Î(h ),τ) = 0. (10) is the eset maximum scale (window adius). As in [19], δ(i s ;I t,τ) measues the colo similaity based on all colo bands. If max c {R, G, B} I c s I c t τ, then δ(i s ;I t,τ) = 1, othewise 0. Instead of deciding the scale fo each colo channel seaately, this method decides the scale by using thee colo channels jointly. This addesses the issue when edges ae not disciminable in any single colo channel. Finally, we set the ight am length h 0 = max(h,1), enfocing a minimum suot scale fo eliable egession. When m = 0 o in othe wods, the zeo-ode olynomial model is used (named CLMF-0 heeinafte), we set α = 1/(h + 1) in (9). This is consistent with the iecewise constant assumtion made fo the egession in (3). Diffeent fom [19] that fimly taes I as the efeence value in (10), ou method consides I only as a noisy measuement of the ideal unnown signal, which emains to be estimated. When m = 1, the fist-ode olynomial model is used (named CLMF-1 heeinafte). We set α to a fixed value (e.g., 0.5) to allow fast satial udating. This leads to a desied oety of being able to extend the scale in gadient egions, so moe ixels ae involved fo eliable egession Ω ' Ω Figue 4. 1-D ste edge lining two (a) iecewise constant egions o (b) iecewise linea egions. To filte the oint, CLMF-0 and CLMF-1 only use the multioint estimates fom oints Ω. Howeve, if the CLMF-1 filte is diectly alied to detail enhancement, we still noticed the gadient evesal atifacts as the BF usually has [6]. The oigin of these atifacts can be exlained by a 1-D signal that contains a am edge in Fig. 3. Given a ixel (ed) on the tansitional edge, its left am length is decided as h 2 (i.e., coveing all the geen oints), while its ight am length is h 0 (i.e., coveing all the blue oints). It is clea that the numbes of the data oints on the two sides of ixel ae vey unbalanced, whee thee ae much moe oints such that I I than the numbe of oints satisfying I I. If the linea egession in (4,5) is comuted based on all these data oints, the egession esults will be moe biased to the left side, as shown in the inset of Fig. 3(a). Thus, the final estimate Y at ixel is much smalle than I, so boosting this diffeence ends u causing an uwad hum in the enhanced signal at. Similaly, a valley is ceated fo the ixel (oange). This is nown as the gadient evesal atifacts [6]. We also find that the above analysis is quite simila to the geometical inteetation of the staicase effect by Buades et al. [3]. To addess this issue caused by the asymmetic am lengths while still eeing the advantage of the scale adatation, this ae ooses to set the hoizontal and vetical am lengths symmetically to the lowe length, i.e., h 0 = h 2 = min(h 0,h 2 ), and h 1 = h 3 = min(h 1,h 3 ). The symmetic am lengths use moe balanced numbes of oints fom both sides of fo the linea egession, which tends to eseve the initial intensity I much bette. This method effectively avoids the gadient evesal atifacts as shown late in Fig Edge- and Gadient-Peseving Filteing CLMF-0 and CLMF-1 have the edge-eseving smoothing oety. This can be exlained by an examle of a 1-D ste edge in Fig. 4. As esented in Sect. 3.1, fo evey ixel, a shae-adative local egion Ω, without staddling ste edges, is fist constucted. Local multioint filteing is then efomed based on Ω. As a esult, the ixels (e.g., ) on the othe side of the ste edge ae not (o aely) used in the linea egession (o smoothing) ocess. In CLMF-0, τ contols the smoothing stength by deciding the neighboing ixels involved in (3). It has a simila effect as the ange vaiance σ 2 in BF [15]. Fo CLMF-1, τ is set to classify the neighboing ixels and only the oints within Ω ae used ' 433

5 (a) (b) (c) (d) (e) (f) Figue 5. Edge-eseving smoothing esults by CLMF-0 (to ow, =8) and CLMF-1 (bottom ow). (a) Inut gay-scale image. (b) τ =25/255. (c) τ =50/255. (d) Inut colo image. (e) =9, τ =50/255, ǫ= (f) Edges on (e) fo a stylized effect [16]. fo the linea egession. As in GF, ǫ decides the smoothing level fo these selected ixels. It is tyically set based on the exected noise/detail level. The edge-eseving smoothing esults of CLMF-0 and CLMF-1 ae shown in Fig. 5. Not only an edge-eseving smoothing filte, CLMF-1 also has the desied oety of eseving image gadients. When m = 1, (8) can be ewitten as: Y = ā 1 I + ā 0, whee ā 1 = P Ω Ω a 1 P Ω Ω, and ā 0 = P Ω Ω a 0 P Ω Ω. Thans to the symmetic am length constaint enfoced in Sect. 3.3, the deived linea coefficients [a 1,a0 ] become much moe eliable and less biased. As the low-ass aveaging outut of these eliable coefficients, ā 1 should have much smalle gadients than that of I nea stong edges. This means Y ā 1 I, and the gadients in I is bette eseved Limitations of Guided Filte Unlie the oosed CLMF-1, GF maes a stong assumtion that a single linea model is sufficient to model a local atch W centeed at ixel [6]. Howeve, conside the case that I = Z, if the guidance image I has a high local vaiance in W, then the linea coefficients [a 1,a 0 ] become vey close to [1,0]. This essentially means that Z just ees its oiginal measuement value without smoothing. Theefoe, fo the oints close to a high-contast edge, they do not undego sufficient smoothing due to the single model assumtion (ecall Fig. 1). Let s conside such an examle in Fig. 6(a). GF achieves stong smoothing fo ixel, only when it is contained in a (shifted) window without involving ixels fom the object A. The ue-left cones of these good windows ae denoted in oange, while thei coesonding window centes ae maed in yellow (i.e., egion U ). In the end, GF simly aggegates all the multioint estimates fo using unweighted aveaging. This lowass filteing uses fa moe un-smoothed values Z fom the windows centeed in the egion of W \U than Y fom those good windows with U. Hence, this esults in U W U ~ 2 h Figue 6. (a) GF does not smooth egions nea high-contast edges o high-vaiance egions. (b) CLMF-1. See the text fo details. insufficient smoothing o denoising fo. In contast, as shown in Fig. 6(b), CLMF-1 avoids the high-contast edge thans to its suot scale adatation, so the linea egession efomance is not affected by the ielevant oints belonging to the object A. Futhemoe, we use the weighted aveage to aggegate all the multioint estimates fom the windows centeed at, with Ω. This futhe ensues that moe confident estimates (defined based on Ω in this ae) have a highe influence on the final filte outut fo. While eseving high-contast edges, CLMF-1 also smooths out noise o details in the egions neaby. Anothe issue with GF is that it may geneate fuzzy object boundaies in the esulting filte outut Y, if the tue signal undelying the inut signal Z actually has a sha tansition hee. Such an atifact can be most visible in deth ma enhancement (see Fig. 9), and it is nown that the accuacy along deth discontinuities is vey imotant fo seveal alications. This atifact is due to the eason that GF efoms the iecewise linea egession that involves all the ixels coveed by a local window. So, it is aticulaly oblematic in eseving sha deth edges, if the colo contast acoss a deth edge is not high. Instead, using a low-ode fitting in shae-adative suot egions, CLMF-0 can significantly imove the ecovey quality fo sha ste edges in the inut signal Z, without causing bluy boundaies O(1) Time Linea Regession and Aggegation As detailed in [19], the integation of aw steeo matching cost o intensity ove a shae-adative local egion can be efomed efficiently in O(1) time. This means that the time comlexity is indeendent of the window adius. This is actually thans to the connectivity constaint made when constucting ixel-wise coss suot seletons. As a seaable filte, the integation ove a 2-D iegulalyshaed egion (3) can be exactly and also efficiently comuted by the integal image technique [4]. To nomalize the integation esult by the vaying samle numbe coveed by Ω in (3), one 2-D integation is also needed to comute Ω in O(1) time. As CLMF-0 augments [19] with a multioint aggegation ste as in (8), this ste equies only two additional 2-D integations in the same way. In total, only fou O(1)-time 2-D integations ove coss-defined suot egions ae needed in CLMF-0, fo both gay-scale Ω ~ 3 h ~ 1 h ~ 0 h 434

6 Table 1. Summay of the comaison between CLMF-0, CLMF-1 and othe filtes Filte Local modeling Multioint est. Soft enels Edge/Gad. eseving Exlicit suot Seed BF [15] Piecewise constant Pointwise Y Only edge Without Slow CLF [19] Piecewise constant Pointwise N Only edge With Vey fast CLMF-0 Piecewise constant Multioint Y Only edge With Vey fast GF [6] Piecewise linea Multioint Y Edge & gadient Without Fast CLMF-1 Piecewise linea Multioint Y Edge & gadient With Fast Table 2. Middlebuy steeo evaluation esults (as of Dec. 2011) Method Ran Avg. eo % Avg. time CLMF sec CLMF sec CostFilte (GF) [14] sec P-LineaS (GF) [5] sec AdatWeight (BF) [18] sec Va.Coss (CLF) [19] sec (a) (b) (c) (d) (e) Figue 7. Filte enels fo (a) diffeent image atches comuted by (b) BF (σ s = 9, σ = 0.1), (c) CLMF-0 (τ = 30/255), (d) GF (ǫ = ), (e) CLMF-1(τ = 30/255, ǫ = ), all with = 9. and colo guidance images. In comaison, fo a gay-scale guidance image, both GF and CLMF-1 need to comute the second-ode moments, thus equiing seven and eight O(1)- time 2-D integations, esectively. Fo RGB guidance images, fa moe numbes of O(1)-time 2-D integations ae needed in addition to intensive vecto aithmetic involved fo each ixel, so they un noticeably slowe than CLMF Connection with Othe Filtes Table 1 summaizes the comaison between the oosed CLMF filtes with othe oula filtes. Based on the fundamental assumtion made about local atches, these filtes can be categoized into two main classes: iecewise constant local modeling and iecewise linea local modeling. Fo the filtes using iecewise constant modeling, they usually cannot eseve the image gadient infomation well. Without an aggegation ste that adatively fuses multile estimates as adoted in CLMF-0 and CLMF- 1, the oiginal coss-based local filteing (CLF) [19] does not achieve a soft filte enel of satially vaying suot weights. This is due to the had decision of suot scales and absence of a soft ange enel. One advantage associated with all coss-based filtes is that an exlicit suot egion Ω mostly involving confident inlies is defined fo evey ixel. As shown late, this egion can be effectively eused in some alications such as efining steeo matching esults. Reesenting such a locally adative suot egion is also memoy efficient, as only fou bytes e ixel ae needed to stoe the fou am lengths [19]. Comaed with GF, the oosed CLMF famewo is a moe genealized fom. It efoms eithe the linea egession o local aveaging ove a shae-adative suot egion, athe than within a fixed-sized squae window. Besides the functional stength, CLMF-0 and CLMF-1 gain the comutational efficiency similaly fom the integal images technique as in GF [6]. Howeve, GF does not suot iecewise constant local modeling natively. Fig. 7 shows the filte enels comuted fo diffeent atches all extacted fom eal images. CLMF-1 defines the weights quite simila to GF. Even just using a had scale decision and had weighting in the fist ste, CLMF-0 achieves soft weights afte the aggegation ste, which ae visually bette than BF at cetain locations. 4. Alications and Exeimental Results This section esents vaious comute vision and gahics alications of the oosed CLMF-0 and CLMF Steeo Matching The accuacy of local steeo methods is highly deendent on the cost aggegation schemes used. As io methods that emloy BF [18] and GF [14] fo cost aggegation, CLMF can also be alied to filte the cost volume while eseving edges. Fist, the ixel-wise aw matching cost is defined by the sum of an absolute colo diffeence and the Hamming distance of two census tansfoms [9]. To allow the coss ams to extend in egions with vey simila colo attens, we also adot a lage window adius R than to include moe ixels fo textueless egions. Simila to [9], a much sticte colo similaity theshold τ s is enfoced in (10), when the cuent am length h>. Then, the cost volume is filteed by CLMF-0/1 using both inut images symmetically, followed by a simle Winne-Taes-All stategy and occlusion detection and filling as esented in [14]. Unlie [14] using a weighted median filte fo ostocessing, whee the weights ae given by a costly BF, ou method 435

7 Figue 8. Disaity mas fo Venus and Cones using CLMF-0. Table 3. Evaluation of diffeent cost aggegation methods Method Avg. eo % a. Single ointwise est., m= 0[19] 8.29 b. Simle avg. of multioint est., m= c. Weighted avg. of multioint est., m= d. Weighted avg. of multioint est., m= e. GF-based agg. w/o ostocessing [14] 8.85 Table 4. Quantitative evaluation fo deth ma usamling in disc. and all egions (All the aametes have been faily configued.) Eo ate % BF GF [10] CLMF-1 CLMF-0 Tsu. Venus Teddy Cones Disc All Disc All Disc All Disc All euses the exlicit local suot egion Ω (ceated aleady fo each oint ) fo efficient mode filteing as in [19]. We evaluated the oosed method based on the Middlebuy steeo benchma [1]. The aametes ae set constant acoss all datasets: = 5, τ = 20, R = 13, τ s = 6. Table 2 lists the esults of the most elevant local steeo methods. It shows that ou CLMF methods ae the best-efoming local steeo methods, even bette than [14] that used to be the best. Moeove, CLMF-0 achieves an aveage an of 5.3 and ans 3 d out of ove 110 steeo methods fo the challenging Cones scene. Fig. 8 shows the disaity mas estimated by CLMF-0, which ae iecewise smooth with deth discontinuities well eseved. Tuning off all the ostocessing in ou steeo method and that in CostFilte [14] (based on its ublic Matlab code), we have comaed the efomance of diffeent cost aggegation schemes in Table 3. It justifies the stength of each algoithmic imovement in the CLMF famewo as well as the advantage ove the GF-based aggegation. Table 2 also eots the oveall CPU untime of CLMF methods and the cometitos 1 fo Tsuuba. CLMF-0 stands out as the most cost-effective one Deth Ma Enhancement Given a low-esolution and/o noisy deth ma lus a egisteed high-esolution, noise-fee colo image, the esolution o quality of the deth ma can be enhanced by joint filteing with the colo image as the guidance. Table 4 comaes five diffeent filtes when alied to usamle a lowesolution deth ma by a scaling facto of 8. Comaed with BF and GF, CLMF-0 significantly imoves the deth ma accuacy, aticulaly fo challenging deth discontinuities. This is due to the eason evealed in Sect Fo the same eason, CLMF-1 does not efom so well lie CLMF-0, though it is bette than GF. In addition, CLMF- 1 We measued ou caeful C++ imlementation of CostFilte [14] on the same PC, but excluding its costly weighted median filteing untime. GF, deth disc. eo ate = 39.5% CLMF-0, deth disc. eo ate = 28.5% Gound-tuth 10% AWGN GF, RMSE=5.21 CLMF-0, RMSE=3.16 Figue 9. Deth usamling esults (to ow, 8 uscaling in Table 4) and deth denoising esults (bottom ow) by GF (ǫ= ) and CLMF-0 (τ =10/255), both with =9 and the best settings. 0 achieves the esults close to those of a state-of-the-at method [10], but it uns much faste. Fig. 9 shows the visual esults of deth usamling and deth denoising. CLMF-0 yields iecewise smooth deth mas with much cleane and shae deth edges than GF visually and quantitatively Single Colo Image Denoising Lie BF used fo noise eduction [8], GF and CLMF-1 can also be alied to this tas. Comaed with moe sohisticated methods e.g., non-local means [2], local filtes do not achieve the best denoising esults. Howeve, they tyically have advantages of fast seed and easy imlementation [11]. Fig. 10 comaes the colo denoising esults of CLMF-1 and GF, whee the best aamete settings have been used. Comaed to GF, CLMF-1 emoves the additive noise adequately, achieving much bette signal ecovey fo high-vaiance egions. Fig. 11 studies the effects of the aametes when vaied to denoise the same image. Inceasing σ o ǫ afte the best setting, both BF and GF ovely smooth the image (inceased RMSE), but CLMF-1 efoms consistently bette thans to τ used to select the data oints fo egession. We have also fixed ǫ to and vaied τ settings fo CLMF-1. When τ is set between 20/255 and 50/255, quite simila denoising esults ae obtained. If τ is set even lage, CLMF-1 behaves inceasingly close to GF. 436

8 10% AWGN GF esult, RMSE = Eo ma of GF Gound-tuth CLMF-1 esult, RMSE = 8.30 Eo ma of CLMF-1 Figue 10. Colo image denoising by GF (ǫ=0.2 2 ) and CLMF-1 (τ =30/255, ǫ=0.2 2 ) with =9. Intensity scaled by 3 fo insets. Fo 20% AWGN, RMSE is fo GF and fo CLMF-1. RMSE BF GF CLMF 1 (τ = 30/255) ε 1/2 (σ ) RMSE CLMF 1 (ε = ) τ (*255) Figue 11. Colo denoising efomance evaluation. (a) Vaying ǫ in GF and CLMF-1, o σ in BF. (b) Vaying τ in CLMF Gahics Alications Besides the alication to image/video abstaction [16] in Fig. 5, CLMF-1 can also be used fo seveal gahics alications. Next, we esent the esults fo detail enhancement and flash/no-flash denoising in Fig. 12. As a gadienteseving smooth filte, CLMF-1 achieves visually simila esults as GF [6]. Both of them do not have unwanted gadient evesal atifacts in the esulting images that BF has. 5. Discussion and Futue Wo This ae oosed a geneic famewo of efoming coss-based local multioint filteing efficiently. CLMF-0 and CLMF-1 find vey cometitive alications into many comute vision and gahics tass. On the theoetical side, the cuent famewo can be futhe extended to accommodate nonlocal algoithms [7]. Exloing othe aoaches to decide adative scales diectionally is inteesting. Ou ecent study has led to a O(1)-time coss constuction method that geatly educes the comutational ovehead. This will be eoted elsewhee. Zhang et al. [20] showed that the coss-based technique is vey fiendly fo GPUs, so we also lan to ma ou filtes onto GPUs fo significant seedu. Refeences [1] htt://vision.middlebuy.edu/steeo/. 7 [2] A. Buades, B. Coll, and J.-M. Moel. A non-local algoithm fo image denoising. In Poc. of CVPR, Figue 12. Detail enhancement ( = 16, τ = 60/255, ǫ = ) and flash/no-flash denoising esults (=8, τ =40/255, ǫ= ) by CLMF-1. See [6] fo details and comaison with BF and GF. [3] A. Buades, B. Coll, and J.-M. Moel. The staicasing effect in neighbohood filtes and its solution. TIP, , 4 [4] F. Cow. Summed-aea tables fo textue maing. In SIG- GRAGH, , 5 [5] L. De-Maeztu, S. Mattoccia, A. Villanueva, and R. Cabeza. Linea steeo matching. In Poc. of ICCV, [6] K. He, J. Sun, and X. Tang. Guided image filteing. In Poc. of ECCV, , 2, 3, 4, 5, 6, 8 [7] V. Katovni, A. Foi, K. Egiazaian, and J. Astola. Fom local enel to nonlocal multile-model image denoising. Int. Jounal of Comute Vision, 86(1):1 32, , 2, 3, 4, 8 [8] C. Liu, W. T. Feeman, R. Szelisi, and S. B. Kang. Noise estimation fom a single image. In CVPR, , 7 [9] X. Mei, X. Sun, M. Zhou, S. Jiao, H. Wang, and X. Zhang. On building an accuate steeo matching system on gahics hadwae. In Poc. of GPUCV, , 6 [10] D. Min, J. Lu, and M. N. Do. Deth video enhancement based on weighted mode filteing. IEEE TIP, Ma [11] S. Pais, P. Konobst, J. Tumblin, and F. Duand. Bilateal filteing: Theoy and alications. Foundations and Tends in Com. Gahics and Vision, 4(1):1 73, , 2, 7 [12] G. Petschnigg, M. Agawala, H. Hoe, R. Szelisi, M. Cohen, and K. Toyama. Digital hotogahy with flash and no-flash image ais. In SIGGRAGH, [13] F. Poili. Constant time O(1) bilateal filteing. In Poc. of CVPR, [14] C. Rhemann, A. Hosni, M. Bleye, C. Rothe, and M. Gelautz. Fast cost-volume filteing fo visual coesondence and beyond. In Poc. of CVPR, , 2, 6, 7 [15] C. Tomasi and R. Manduchi. Bilateal filteing fo gay and colo images. In Poc. of ICCV, , 2, 4, 6 [16] H. Winnemolle, S. C. Olsen, and B. Gooch. Real-time video abstaction. In ACM SIGGRAGH, , 8 [17] Q. Yang, K. H. Tan, and N. Ahuja. Real-time O(1) bilateal filteing. In Poc. of CVPR, [18] K. Yoon and I. Kweon. Adative suot-weight aoach fo coesondence seach. IEEE PAMI, , 2, 6 [19] K. Zhang, J. Lu, and G. Lafuit. Coss-based local steeo matching using othogonal integal images. IEEE Tans. CSVT, 19(7): , July , 3, 4, 5, 6, 7 [20] K. Zhang, J. Lu, Q. Yang, G. Lafuit, R. Lauweeins, and L. V. Gool. Real-time and accuate steeo: A scalable aoach with bitwise fast voting on CUDA. IEEE Tans. CSVT, 21(7): , July