A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters

Transcription

1

2 A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin EECS, Northwestern University

3 Advanced Uses of Bilateral Filters

4 Advanced Uses for Bilateral A few clever, exemplary applications Improved Stereo Depth Estimators (Ansar Flash/No Flash Image Merge Retinex Tone Management (Bae Exposure Correction (Bennett2006) Feature Fusion Image Merging Ansar 2004,5) (Petschnigg2004, Eisenman2004) (Elad 2006) Bae 2006) (Bennett2006) (Bennett 2007, Wang2008) Many more, many new ones Broad interest SIGGRAPH,EG,CVPR,ICIP, etc.

5 Enhanced Real-Time Stereo (Adnan 2004, ) Silhouettes Strong depth edges Corresp.. Errors Noisy depth textures Bilateral: preserve edges, remove noise:

6 Enhanced Real-Time Stereo (Adnan 2004, ) Silhouettes Strong depth edges Corresp.. Errors Noisy depth textures Bilateral: preserve edges, remove noise:

7 Enhanced Real-Time Stereo (Adnan 2004, ) Silhouettes Strong depth edges Corresp.. Errors Noisy depth values Bilateral: preserve edges, remove noise:

8 Spatial-Depth Super Resolution for Range Images (Yang (Yang et al. 2007) Edges from 2 registered high-res photos Depth from low-res, sparse, noisy Iterative bilateral refinement

9 Spatial-Depth Super Resolution for Range Images (Yang (Yang et al. 2007) Edges from 2 registered high-res photos Depth from low-res, sparse, noisy Iterative bilateral refinement

10 Spatial-Depth Super Resolution for Range Images (Yang (Yang et al. 2007) Edges from 2 registered high-res photos Depth from low-res, sparse, noisy Iterative bilateral refinement

11 Spatial-Depth Super Resolution for Range Images (Yang (Yang et al. 2007) Edges from 2 registered high-res photos Depth from low-res, sparse, noisy Iterative bilateral refinement RESULTS Exceptionally accurate on entire Middlebury Data set: Subpixel accuracy, 100X resol.

12 Spatial-Depth Super Resolution for Range Images (Yang (Yang et al. 2007) Edges from 2 registered high-res photos Depth from low-res, sparse, noisy Iterative bilateral refinement

13 Retinex from 2 Bilateral Filters [Elad05] M. Elad, "Retinex by Two Bilateral Filters", Scale-Space 2005, Hofgeismar, Germany, 7-10 April 2005 Retinex Theory (Edwin Land, 1972): Eyes discount the illuminant.. Computable? Color: set by spectral AND spatial relationships Done in retina? In visual cortex? Retinex

14 Retinex from 2 Bilateral Filters [Elad05] M. Elad, "Retinex by Two Bilateral Filters", Scale-Space 2005, Hofgeismar, Germany, 7-10 April 2005 Estimate Illumination & Reflectance Bilaterally Smooth between object edges Illum.. Sets image upper bounds (0 < Refl. < 1) Tailored Bilateral Filter Further Justifies [Durand&Dorsey02] speedup approx. Good Retinex Summary:

15 Flash / No-Flash Photo Improvement (Eisemann04) (Petschnigg04) Merge best features: warm, cozy candle light (no-flash) low-noise, detailed flash image

16 Joint Bilateral or Cross Bilateral (2004) Bilateral two kinds of weights, so Cross Bilateral Filter (CBF): get them from two kinds of images. Spatial smoothing of pixels in image A,, with WEIGHTED by intensity similarities in image B:

17 Recall: Cross or Joint Bilateral Idea Noisy but Strong Range filter preserves signal Noisy and Weak Use stronger signal s s range within weaker signal s s noise

18 Overview Basic approach of both flash/noflash papers Remove noise + details from image A, Keep as image A Lighting No-flash Obtain noise-free details from image B, Discard Image B Lighting Result

19 Petschnigg: Flash: + Strong, sharp edges - Stark, ugly light / shadow

20 Petschnigg: No Flash: - Weak, noisy edges + Warm, cozy light / shadow

21 Petschnigg: Result + Strong, sharp edges + Warm, cozy light / shadow

22 Approaches - Main Idea

23 Joint or Cross Bilateral Filter (CBF) Enhanced ability to find weak details in noise (B s s weights preserve similar edges in A) Useful Residues for Detail Transfer CBF(A,B A,B) ) to remove A s A s noisy details CBF(B,A B,A) ) to remove B s B s less-noisy details; add to CBF(A,B) for clean, detailed, sharp image (See the papers for details)

24 Joint or Cross Bilateral Filter (CBF) Enhanced ability to find weak details in noise (B s s weights preserve similar edges in A)

25 Petschnigg: : Detail Transfer Results Lamp made of hay: No Flash Flash Detail Transfer

26 Petschnigg04, Eisemann04 Features Eisemann 2004: --included image registration, --used lower-noise flash image for color, and --compensates for flash shadows Petschnigg 2004: --included explicit color-balance & red-eye eye --interpolated continuously variable flash, --Compensates for flash specularities

27 Tonal Management (Bae et al., SIGGRAPH 2006) Cross bilateral, residues visually compelling image decompositions. Explore: adjust each component s s contrast, find visually pleasing transfer functions,etc. Stylize: finds transfer functions that match histograms of preferred artists, Textureness ; local measure of textural richness; to guide local mods,, to match artist s

28 Tone Mgmt. Examples: Original

29 Tone Mgmt. Examples: Bright and Sharp

30 Tone Mgmt. Examples: Gray and detailed

31 Tone Mgmt. Examples: Smooth and grainy

32 Source Tone Management Examples

33 Tone Management (Bae06) Textured-ness Metric: (shows highest Contrast- adjusted texture)

34 Model: Ansel Adams Reference Model

35 Input with auto-levels Results

36 Direct Histogram Transfer (dull) Results

37 Best Results

38 Multi-Light Detail Transfer SIGG2007 Fattal et al., Multiscale Shape and Detail Enhancement from Multi-light Image Collections Different light Different visible details Extract, Control/Enhance, Merge details

39 Multi-Light Detail Transfer SIGG2007 Fattal et al., Multiscale Shape and Detail Enhancement from Multi-light Image Collections Different light Different visible details Extract, Control/Enhance, Merge details Light 1

40 Multi-Light Detail Transfer SIGG2007 Fattal et al., Multiscale Shape and Detail Enhancement from Multi-light Image Collections Different light Different visible details Extract, Control/Enhance, Merge details Light 2

41 Multi-Light Detail Transfer SIGG2007 Fattal et al., Multiscale Shape and Detail Enhancement from Multi-light Image Collections Different light Different visible details Extract, Control/Enhance, Merge details Light 3

42 Multi-Light Detail Transfer SIGG2007 Fattal et al., Multiscale Shape and Detail Enhancement from Multi-light Image Collections Different light Different visible details Extract, Control/Enhance, Merge details Bilateral filters User-set set weights Adjust to suit flat, detailed or with shadows

43 Multi-Light Detail Transfer SIGG2007 Fattal et al., Multiscale Shape and Detail Enhancement from Multi-light Image Collections Different light Different visible details Extract, Control/Enhance, Merge details Bilateral filters User-set set weights Adjust to suit flat, detailed or with shadows

44 Video Enhancement Using Per Pixel Exposures (Bennett, 06) From this video: ASTA: Adaptive Spatio- Temporal Accumulation Filter

45 VIDEO

46 The Process for One Frame Raw Video Frame: (from FIFO center) Histogram stretching; (estimate gain for each pixel) Mostly Temporal Bilateral Filter: Average recent similar values, Reject outliers (avoids ghosting ), spatial avg as needed Tone Mapping

47 The Process for One Frame Raw Video Frame: (from FIFO center) Histogram stretching; (estimate gain for each pixel) Mostly Temporal Bilateral Filter: Average recent similar values, Reject outliers (avoids ghosting ), spatial avg as needed Tone Mapping

48 The Process for One Frame Raw Video Frame: (from FIFO center) Histogram stretching; (estimate gain for each pixel) Mostly Temporal 3D Bilateral Filter: Average recent similar values, Reject outliers (avoids ghosting ), spatial avg as needed Tone Mapping (color: # avg pixels)

49 The Process for One Frame Raw Video Frame: (from FIFO center) Histogram stretching; (estimate gain for each pixel) Mostly Temporal 3D Bilateral Filter: Average recent similar values, Reject outliers (avoids ghosting ), spatial avg as needed Tone Mapping

50 Bilateral Filter Variant: Mostly Temporal FIFO for Histogram-stretched stretched video Carry gain estimate for each pixel; Use future as well as previous values; Expanded Bilateral Filter Methods: Static scene? Temporal-only only avg. works well Motion? Bilateral rejects outliers: no ghosts! Generalize: Dissimilarity (not just I p I q 2 ) Voting: spatial filter de-noises motion

51 Bennett2007: Multispectral Video Fusion Dual-Bilateral filter: fuses best of visible + IR

52 Video Relighting from IR illumination. EG2008, Wang,Davis et al. Video Relighting Using Infrared Illumination

53 Video Relighting from IR Illumination Switched IR illuminators, 8 photos per frame Ratio Images Hue Corrections

54 Conclusions Bilateral Filter easily adapted, customized to broad class of problems One tool among many for complex problems Useful in for any task that needs Robust, reliable smoothing with outlier rejection

55

56 Applications