Fusion and Reconstruction Dr. Yossi Rubner yossi@rubner.co.il Some slides stolen from: Jack Tumblin 1 Agenda We ve seen Panorama (from different FOV) Super-resolution (from low-res) HDR (from different dynamic range) Today Best focus (from different focuses) / no flash (from 2 images) Depth edge detection (from different flashes) Motion blur (from different sub-exposures) Multispectral (from visible light and IR) Clear day (from bad weather) 2 Single photo: forces narrow tradeoffs: Focus, Exposure, aperture, time, sensitivity, noise, Usual result: Incomplete visual appearance. Multiple photos, assorted settings for Optics, Sensor, Lighting, Processing Fusion: Merge the best parts or Reconstruction: Detect changes photo changes, compute scene invariants NEAR
FAR
Source images Elmar Eisemann and Frédo Durand, Photography Enhancement via Intrinsic Relighting Georg Petschnigg, Maneesh Agrawala, Hugues Hoppe, Richard Szeliski, Michael Cohen, Kentaro Toyama. Digital Photography with and No- Image Pairs FUSION Graph Cuts Solution (using global maximum contrast image objective) Denoise no-flash image using flash image No-flash Result
No Available light: + nice lighting - noise/blurriness - color : + details + color - flat/artificial Cross-Bilateral Filter based Approach Transfer detail from flash image to no-flash image No-flash + original lighting + details/sharpness + color Result
Registration Compensate for camera motion (image translation) Difficult because lighting changes Edge detection Cross Bilateral Filter Similar to joint bilateral filter by Petschnigg et al. When no-flash image is too noisy Borrow similarity from flash image edge stopping from flash image No-flash Bilateral Cross Bilateral Shadow Correction Why? Shadow Detection No flash - = No-flash Ι Straightforward recombination
Shadow Detection Difference Ι = light from the flash? Results Goal: Find a threshold for Ι Automatic Threshold Detection (see paper) No-flash Ι Results Non-photorealistic Camera: Depth Edge Detection and Stylized Rendering using Multi- Imaging No-flash Result Ramesh Raskar, Karhan Tan, Rogerio Feris, Jingyi Yu, Matthew Turk Mitsubishi Electric Research Labs (MERL), Cambridge, MA U of California at Santa Barbara U of North Carolina at Chapel Hill
ENGF343.JPG What are the problems with real photo in conveying information? Why do we hire artists to draw what can be photographed?
Depth Discontinuities Internal and external Shape boundaries, Occluding contour, Silhouettes Canny Our Method
Photo Result Our Method Canny Intensity Edge Detection Fluttered Shutter Camera Raskar, Agrawal, Tumblin Siggraph2006
Figure 2 results Input Image Rectified Image to make motion lines parallel to scan lines. Approximate cutout of the blurred image containing the taxi (vignetting on left edge). Exact alignment of cutout with taxi extent is not required. Image Deblurred by solving a linear system. No post-processing
Comparison Inverse Filter stable Inverse Filter Unstable
Solving: Single Line of Matlab Code Short Exposure Long Exposure Coded Exposure Our result Matlab Lucy Ground Truth No Post Processing! Bennett2007: Multispectral Video Fusion Clear Day from Foggy Days Two Different Foggy Conditions (Shree Nayar, Srinivasa Narasimhan 00) Dual-Bilateral filter: fuses best of visible + IR Clear Day Image Time: 3 PM Deweathering Time: 5:30 PM