Upscaling Beyond Super Resolution Using a Novel Deep Learning System

Transcription

1 Upscaling Beyond Super Resolution Using a Novel Deep Learning System Pablo Navarrete Michelini pnavarre@boe.com.cn Hanwen Liu lhw@boe.com.cn BOE Technology Group Co., Ltd.

2 BOE Technology Group Co., Ltd.

3 BOE Ultra HD Panels

4 Chapter I : The Layer and the System

5 Standard Upscaling For example, a simple linear interpolation can be done with F = [ 1/4 1/2 1/4 1/2 1 1/2 1/4 1/2 1/4 ].

6 Standard Upscaling Efficient implementation avoids multiplying zeros. Break F into many filters W i.

7 Classic Upscalers Classic Upscalers: Nearest Neighbor, Linear, Bicubic, Lanczos,... Advanced Upscalers: Directional filters (NEDI), wavelets,... (a) Original (b) Nearest Neighbor (c) Bicubic Figure: Classic Upscalers

8 GTC 2016 MuxOut

9 GTC 2016 MuxOut

10 GTC 2016 MuxOut

11 GTC 2016 MuxOut

12 GTC 2016 MuxOut

13 GTC 2016 MuxOut

14 Similar Approaches SRCNN Dong C., et.al., Learning a Deep Convolutional Network for Image Super Resolution. Sept BOE MuxOut Navarrete P., et.al., Upscaling with Deep Convolutional Networks and Muxout Layers. May Google RAISR Romano Y., RAISR: Rapid and Accurate Image Super Resolution. Jun Twitter ESPCN Shi W., et.al, Real Time Single Image and Video Super Resolution Using an Efficient Sub Pixel Convolutional Neural Network. Sept Twitter GAN Ledig C., et.al., Photo-Realistic Single Image Super Resolution Using a Generative Adversarial Network. Sept Twitter GAN Sønderby C.K., et.al, Amortised MAP Inference for Image Super-resolution. Oct 2016.

15 Similar Approaches Twitter ESPCN: Sub pixel convolution layer = MuxOut r r. Differences: MuxOut considers several groups of r 2 features. MuxOut is design to factorize r and use as several layers within the network. Figure from: Shi W., et.al, Real Time Single Image and Video Super Resolution Using an Efficient Sub Pixel Convolutional Neural Network. Sept 2016.

16 Similar Approaches Google RAISR: Uses a ML approach to learn adaptive filters. Not based on convolutional networks. Similarity: We will show how to interpret the convolutional network approach as an adaptive filter. Figure from: Romano Y., RAISR: Rapid and Accurate Image Super Resolution. Jun 2016.

17 MuxOut Layer Old Version Problems of MuxOut: Reduces processing features Works very well only with easy content (e.g. text). Why? Filter parameters W ahve 2 tasks: Downsampling: Which combination of a b c d works better? Filter: Which values work better for interpolation?

18 MuxOut Layer New Version New Version: Consider all (or most) possible combinations of features Can keep the same number of processing features. Filter parameters can focus on interpolation. SGD algorithms converge fast and stable.

19 MuxOut Usage A convolutional layer ( block typically means: ) conv(x c in) = σ cin xc in W c in,c out +b cout We will consider convolution and activation independently: conv(x c in ) = c in x c in W c in,c out activ(x c ) = σ(x c +b c ) And we use MuxOut like:

20 First Approach Problem: Large upscaling factors color might be misaligned with lumminance.

21 Full Color Configuration Idea: RGB Input + RGB Output Problem: MuxOut mixes color channels. Need to process separately:

22 Chroma Sub sampling Configuration Note: HVS is less sensitive to the position and motion of color than luminance.

23 MSE vs SSIM Traditional approach: Loss(X,Y) = MSE(X,Y) = 1 H W Problem: Not well correlated with HVS Why not PSNR? PSNR is unbounded. H,W i,j=0 (X i,j Y i,j ) 2 Loss(X,Y) = SSIM(X,Y) = (2µ Xµ Y +C 1 )(2σ XY +C 2 ) (µ 2 X +µ2 Y +C 1)(σ 2 X +σ2 Y +C 2) Well correlated with HVS. Differentiable. Behaves well with SGD.

24 Results (a) Standard (PSNR db SSIM ) (b) Ours (PSNR db SSIM )

25 Chapter II : The Analysis

26 Analysis Linear Systems: Interpolation filter given by impulse response. CN: Is not Linear because of ReLU. (c) Activity Recorder (d) Mask Layer Use an input image and record activity. Replace all activations (ReLU) by a Mask layer. The system becomes linear! Check impulse response.

27 Analysis

28 Analysis

29 Analysis

30 Analysis

31 Analysis

32 Analysis

33 Analysis

34 Analysis

35 Analysis

36 Analysis

37 Analysis

38 Analysis

39 Analysis

40 Chapter III : Hyper Resolution

41 Hyper Resolution We say that x and y are aliases if Downscale(x) = Downscale(y) Many realistic images are aliased. MSE, SSIM, etc aim for only one alias. MSE, SSIM traget removes the innovation process! (e.g. linear regression). Give up the original content. We just want it to look real.

42 Hyper Resolution Generator (Upscaler) Discriminator

43 Generative Adversarian Networks (GAN) Increasing attention and significant progress in the last year. We will refer to the following important references: WGAN: Arjovsky M., et.al., Wasserstein GAN. Jan Improved WGAN: Gulrajani I., et.al., Improved Training of Wasserstein GANs. March Losses: L D = E[D(x fake )] E[D(x real )]+λ gp E [( ˆx D(ˆx) 2 1) 2] L G = E[D(x fake )]+λ LR (Downscale(G(x LR )),x LR )

44 Generative Adversarian Networks (GAN) We do not want to reveal the high resolution content during the Upscaler s training. We do not want to generate artificial images with no reference to the input. We ask the upscaler to be able to recover the low resolution input with a standard downscaler (e.g. area). ( Downscale(G(x LR )), x LR ) with: or (x,y) = MSE(x,y) (x,y) = 1 SSIM(x,y)

45 Results (e) Standard (PSNR db) (f) Original (PSNR ) (g) Ours (PSNR db)

46 Results (h) Standard (PSNR db) (i) Original (PSNR ) (j) Ours (PSNR db)

47 Results (k) Standard (PSNR db) (l) Original (PSNR ) (m) Ours (PSNR db)

48 Conclusions Overview: System: Proposed improved MuxOut. Analysis: Novel approach to visualize CN as adaptive filter. Super Resolution: Proposed SSIM loss and process color input/output. Hyper Resolution: Hallucinating details using GAN can produce results comparable to original content. Next Steps: Larger upscaling factors. Use analysis to improve design and test for other problems. Improve generalization of GAN approach.

49 Questions & Answers Thank you! LinkedIn: ResearchGate: