IMAGE RESTORATION WITH NEURAL NETWORKS Orazio Gallo Work with Hang Zhao, Iuri Frosio, Jan Kautz
MOTIVATION The long path of images Bad Pixel Correction Black Level AF/AE Demosaic Denoise Lens Correction Metering Image Enhancing Tone Mapping Image Signal Processor (ISP)
Image credit: Wikipedia Image credit: Marc Levoy DEMOSAICING colors by interpolation
DENOISING Several types of noise involved in the image formation: Photon shot noise Dark current (AKA thermal noise) Photo-response non-uniformity Vignetting Readout noise: Reset noise (charge-to-voltage transfer) White noise (during voltage amplification amplification) Quantization noise (ADC)
DENOISING
MOTIVATION Bad Pixel Correction Black Level AF/AE Demosaic Denoise Lens Correction Metering Image Enhancing Tone Mapping Demosaicing before denoising changes the statistics of the noise. And the best de-noising algorithms require to know what the noise looks like.
MOTIVATION Bad Pixel Correction Black Level AF/AE Denoise Demosaic Lens Correction Metering Image Enhancing Tone Mapping Denoising first can change the color reproduction accuracy as the three channels may be denoised differently.
FLEXISP 1 A Flexible Camera Image Processing Framework PSF CFA Noise [1] Heide et al., ACM SIGGRAPH Asia 2012 (ToG)
CAN WE DO IT WITH A NEURAL NETWORK? Can we do it with a neural network, which moves the heavy lifting to the training stage and inference is very quick?
JOINT DEMOSAICING AND DENOISING
bilinear interpolation convolution convolution convolution JOINT DEMOSAICING AND DENOISING Network architecture
MEASURING IMAGE QUALITY Original Image adapted from https://ece.uwaterloo.ca/~z70wang/research/ssim/
MEASURING IMAGE QUALITY Higher sensitivity to errors in texture-less regions! Wang, et al. "Image quality assessment: from error visibility to structural similarity." IEEE TIP (2004)
MEASURING IMAGE QUALITY Original 0.988 0.662 Image adapted from https://ece.uwaterloo.ca/~z70wang/research/ssim/
MEASURING IMAGE QUALITY Higher sensitivity to errors in texture-less regions!
bilinear interpolation convolution convolution convolution JOINT DEMOSAICING AND DENOISING Network architecture
JOINT DEMOSAICING AND DENOISING Network training Training data 31 x 31 patches from 700, 999x666 RGB images (MIT-Adobe FiveK dataset) Input - noisy image (realistic noise model) - bilinear interpolation Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM
Ground truth
Noisy
RESULTS Visual comparison (+ unsharp masking) Noisy BM3D (state of the art) Ground truth
Noisy
RESULTS Visual comparison (+ unsharp masking) Noisy BM3D (state of the art) Ground truth
JOINT DEMOSAICING AND DENOISING: RESULTS Average image quality metrics on the testing dataset
DOES IT GENERALIZE? JPEG ARTIFACT REMOVAL & SUPER-RESOLUTION
JPEG ARTIFACT REMOVAL Network training Training data Input Training cost function 31 x 31 patches from 700, 999x666 RGB images (MIT-Adobe FiveK dataset) JPEG compressed image, 25% quality L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM
JPEG ARTIFACT REMOVAL: RESULTS Visual comparison (+ unsharp masking) JPEG L2 L1 + MS-SSIM Ground truth
JPEG ARTIFACT REMOVAL: RESULTS Numerical comparison Average image quality metrics on the testing dataset
SUPER-RESOLUTION Network training Training data Input Training cost function 31 x 31 patches from 700, 999x666 RGB images (MIT-Adobe FiveK dataset) 2x downsampled image + upsampled with bilinear interpolation L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM
SUPER-RESOLUTION: RESULTS Visual comparison (+ unsharp masking) Low rez L2 L1 + MS-SSIM
SUPERRESOLUTION: RESULTS Numerical comparison and literature
LEARNINGS?
LEARNINGS A closer look at the different losses
LEARNINGS and
LEARNINGS and 0.3939 0.3896 seems to have more convergence issues. converges faster and speeds up the convergence or other losses, too.
LEARNINGS A closer look at the different losses
LEARNINGS SSIM and MS-SSIM Higher sensitivity to errors in texture-less regions! Multi-scale is helpful when dealing with transition regions.
LEARNINGS A closer look at the different losses
RESULTS Why mixing MS-SSIM and?
CONCLUSIONS What have we learnt? Even a shallow network can produce state-of-the-art results if you train it carefully. Perceptually-motivated loss functions can help! But you have to be aware of their limitations!
Thanks! Zhao, Gallo, Frosio, and Kautz, Loss Functions for Image Restoration with Neural Networks, IEEE Trans. on Comp. Imaging, 2017