IMAGE RESTORATION WITH NEURAL NETWORKS. Orazio Gallo Work with Hang Zhao, Iuri Frosio, Jan Kautz

Transcription

1 IMAGE RESTORATION WITH NEURAL NETWORKS Orazio Gallo Work with Hang Zhao, Iuri Frosio, Jan Kautz

2

3

4 MOTIVATION The long path of images Bad Pixel Correction Black Level AF/AE Demosaic Denoise Lens Correction Metering Image Enhancing Tone Mapping Image Signal Processor (ISP)

5 Image credit: Wikipedia Image credit: Marc Levoy DEMOSAICING colors by interpolation

6 DENOISING Several types of noise involved in the image formation: Photon shot noise Dark current (AKA thermal noise) Photo-response non-uniformity Vignetting Readout noise: Reset noise (charge-to-voltage transfer) White noise (during voltage amplification amplification) Quantization noise (ADC)

7 DENOISING

8 MOTIVATION Bad Pixel Correction Black Level AF/AE Demosaic Denoise Lens Correction Metering Image Enhancing Tone Mapping Demosaicing before denoising changes the statistics of the noise. And the best de-noising algorithms require to know what the noise looks like.

9 MOTIVATION Bad Pixel Correction Black Level AF/AE Denoise Demosaic Lens Correction Metering Image Enhancing Tone Mapping Denoising first can change the color reproduction accuracy as the three channels may be denoised differently.

10 FLEXISP 1 A Flexible Camera Image Processing Framework PSF CFA Noise [1] Heide et al., ACM SIGGRAPH Asia 2012 (ToG)

11 CAN WE DO IT WITH A NEURAL NETWORK? Can we do it with a neural network, which moves the heavy lifting to the training stage and inference is very quick?

12 JOINT DEMOSAICING AND DENOISING

13 bilinear interpolation convolution convolution convolution JOINT DEMOSAICING AND DENOISING Network architecture

14 MEASURING IMAGE QUALITY Original Image adapted from

15 MEASURING IMAGE QUALITY Higher sensitivity to errors in texture-less regions! Wang, et al. "Image quality assessment: from error visibility to structural similarity." IEEE TIP (2004)

16 MEASURING IMAGE QUALITY Original Image adapted from

17 MEASURING IMAGE QUALITY Higher sensitivity to errors in texture-less regions!

18 bilinear interpolation convolution convolution convolution JOINT DEMOSAICING AND DENOISING Network architecture

19 JOINT DEMOSAICING AND DENOISING Network training Training data 31 x 31 patches from 700, 999x666 RGB images (MIT-Adobe FiveK dataset) Input - noisy image (realistic noise model) - bilinear interpolation Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM

20 Ground truth

21 Noisy

22

23

24 RESULTS Visual comparison (+ unsharp masking) Noisy BM3D (state of the art) Ground truth

25 Noisy

26

27

28 RESULTS Visual comparison (+ unsharp masking) Noisy BM3D (state of the art) Ground truth

29 JOINT DEMOSAICING AND DENOISING: RESULTS Average image quality metrics on the testing dataset

30 DOES IT GENERALIZE? JPEG ARTIFACT REMOVAL & SUPER-RESOLUTION

31 JPEG ARTIFACT REMOVAL Network training Training data Input Training cost function 31 x 31 patches from 700, 999x666 RGB images (MIT-Adobe FiveK dataset) JPEG compressed image, 25% quality L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM

32 JPEG ARTIFACT REMOVAL: RESULTS Visual comparison (+ unsharp masking) JPEG L2 L1 + MS-SSIM Ground truth

33 JPEG ARTIFACT REMOVAL: RESULTS Numerical comparison Average image quality metrics on the testing dataset

34 SUPER-RESOLUTION Network training Training data Input Training cost function 31 x 31 patches from 700, 999x666 RGB images (MIT-Adobe FiveK dataset) 2x downsampled image + upsampled with bilinear interpolation L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM

35 SUPER-RESOLUTION: RESULTS Visual comparison (+ unsharp masking) Low rez L2 L1 + MS-SSIM

36 SUPERRESOLUTION: RESULTS Numerical comparison and literature

37 LEARNINGS?

38 LEARNINGS A closer look at the different losses

39 LEARNINGS and

40 LEARNINGS and seems to have more convergence issues. converges faster and speeds up the convergence or other losses, too.

41 LEARNINGS A closer look at the different losses

42 LEARNINGS SSIM and MS-SSIM Higher sensitivity to errors in texture-less regions! Multi-scale is helpful when dealing with transition regions.

43 LEARNINGS A closer look at the different losses

44 RESULTS Why mixing MS-SSIM and?

45 CONCLUSIONS What have we learnt? Even a shallow network can produce state-of-the-art results if you train it carefully. Perceptually-motivated loss functions can help! But you have to be aware of their limitations!

46 Thanks! Zhao, Gallo, Frosio, and Kautz, Loss Functions for Image Restoration with Neural Networks, IEEE Trans. on Comp. Imaging, 2017