Supplementary Material: Deep Photo Enhancer: Unpaired Learning for Image Enhancement from Photographs with GANs

Transcription

1 Supplementary Material: Deep Photo Enhancer: Unpaired Learning for Image Enhancement from Photographs with GANs Yu-Sheng Chen Yu-Ching Wang Man-Hsin Kao Yung-Yu Chuang National Taiwan University 1 More comparisons on the MIT-Adobe 5K dataset In the paper, we proposed three models for enhancing images, SL (supervised learning using the proposed generator trained on the MIT-Adobe 5K dataset), UL (the proposed 2-way GAN trained on the MIT-Adobe 5K dataset) and HDR (the proposed 2-way GAN trained on the collected HDR dataset). We compare the proposed models with five state-of-the-art methods, including Cycle- GAN [10], DPED [5], CLHE [8], NPEA [9] and FLLF [1]. Note that our model can be taken as an enhanced CycleGAN with three proposed improvements, a better generator, a better WGAN model with the adaptive weighting scheme and a better 2-way GAN model with individual batch normalization layers. The CycleGAN was trained on the collected HDR dataset. The DPED models are tailored with di erent mobile phones. Here, we show results of the DPED models for iphone6, iphone7 and Nexus 5x. This section compares these methods on some images from the MIT-Adobe- 5K [2] testing dataset. In general, we have the following observations. Our models trained with the photographer s labels approximate the labels reasonably well. DPED models vary a lot with di erent phone models. Though trained on the same HDR dataset, CycleGAN cannot capture the characteristics of the dataset as well as our HDR model. It shows that the proposed improvements are e ective and important, at least for this application. CLHE, NPEA and FLLF are not robust and could generate unnatural enhanced images at times. Our HDR model captures the characteristics of the collected HDR dataset well and generates the most natural enhanced images.

2 2 Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang Input Label Our (SL) Our (UL) Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CycleGAN (HDR) CLHE NPEA FLLF Fig. 1: Comparisons of di erent methods on a MIT-adobe-5K testing image, a3552.

3 Supplementary Material: Deep Photo Enhancer 3 Input Label Our (SL) Our (UL) Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CycleGAN (HDR) CLHE NPEA FLLF Fig. 2: Comparisons of di erent methods on a MIT-adobe-5K testing image, a0212

4 4 Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang Input Label Our (SL) Our (UL) Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CycleGAN (HDR) CLHE NPEA FLLF Fig. 3: Comparisons of di erent methods on a MIT-adobe-5K testing image, a0481

5 Supplementary Material: Deep Photo Enhancer 5 Input Label Our (SL) Our (UL) Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CycleGAN (HDR) CLHE NPEA FLLF Fig. 4: Comparisons of di erent methods on a MIT-adobe-5K testing image, a3203

6 6 Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang Input Label Our (HDR) Our (SL) Our (UL) CycleGAN (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CLHE NPEA FLLF Fig. 5: Comparisons of di erent methods on a MIT-adobe-5K testing image, a0535

7 Supplementary Material: Deep Photo Enhancer 7 Input Label Our (HDR) Our (SL) Our (UL) CycleGAN (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CLHE NPEA FLLF Fig. 6: Comparisons of di erent methods on a MIT-adobe-5K testing image, a1305

8 8 Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang Input Label Our (HDR) Our (SL) Our (UL) CycleGAN (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CLHE NPEA FLLF Fig. 7: Comparisons of di erent methods on a MIT-adobe-5K testing image, a4963

9 Supplementary Material: Deep Photo Enhancer 9 2 More comparisons on images from the Internet We show results of di erent methods on enhancing images collected from the Internet. We compare our HDR model with DPED and CLHE. For DPED, we show the results using all three phone models. Although with good enhancement in general, the results of CLHE sometimes look unnatural, particularly on colors. The results of DPED vary among phone models, showing its dependence to phone models. In general, our results give natural results with enhanced color, contrast and details. Noe that the model was trained on HDR images. Thus, the results are HDR-like. Sometime, it could look too prominent. It is however possible to train a modest model with a set of images with that style. We also provide an accompanying video showing the results of our HDR model on a video. Each frame is processed independently. However, there is no obvious temporal flick. It shows that our model is quite stable. The video also demonstrates that our model can be used for a wide variety of images.

10 10 Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang Input Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CLHE Fig. 8: Comparisons of di erent methods on an Internet image.

11 Supplementary Material: Deep Photo Enhancer 11 Input Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CLHE Fig. 9: Comparisons of di erent methods on an Internet image.

12 12 Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang Input Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CLHE Fig. 10: Comparisons of di erent methods on an Internet image.

13 Supplementary Material: Deep Photo Enhancer 13 Input Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CLHE Fig. 11: Comparisons of di erent methods on an Internet image.

14 14 Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang Input Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CLHE Fig. 12: Comparisons of di erent methods on an Internet image.

15 Supplementary Material: Deep Photo Enhancer 15 Input Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CLHE Fig. 13: Comparisons of di erent methods on an Internet image.

16 16 Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang Input Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CLHE Fig. 14: Comparisons of di erent methods on an Internet image.

17 Supplementary Material: Deep Photo Enhancer 17 Input Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CLHE Fig. 15: Comparisons of di erent methods on an Internet image.

18 18 Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang Input Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CLHE Fig. 16: Comparisons of di erent methods on an Internet image.

19 Supplementary Material: Deep Photo Enhancer 19 Input Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CLHE Fig. 17: Comparisons of di erent methods on an Internet image.

20 20 Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang Input Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CLHE Fig. 18: Comparisons of di erent methods on an Internet image.

21 Supplementary Material: Deep Photo Enhancer 21 Input Our (HDR) DPED (iphone6) DPED (iphone7) DPED (Nexus 5x) CLHE Fig. 19: Comparisons of di erent methods on an Internet image.

22 22 Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang 3 The collected HDR dataset We show sample images of the collected HDR dataset below. Fig. 20: Sample images from the collected HDR dataset

23 Supplementary Material: Deep Photo Enhancer 23 4 Training This section shows some figures about the training process. Figure 21 shows the training progress of di erent GAN models, the proposed A-WGAN, WGAN- GP, DRAGAN, LSGAN (local D), LSGAN and GAN. It can be seen that the proposed A-WGAN generates the best results while some GAN models could totally collapse. Figure 22 shows the discriminator loss along the training process for di erent one-way GAN models trained on the MIT-Adobe 5K dataset. For the proposed model, the discriminator loss can be used as a good indicator for convergence. Figure 23 shows the discriminator loss for the proposed two-way GAN model with and without the individual batch normalization layers on the MIT-Adobe 5K and the collected HDR datasets. Although training on the MIT- Adobe 5K dataset is e ective without individual batch normalization, individual batch normalization layers play an important role on the training with the HDR dataset. PSNR Ours (A-WGAN) WGAN-GP DRAGAN LSGAN(Local D) LSGAN GAN epoch Fig. 21: PNSR values of testing on the MIT-Adobe-5K daatset with di erent oneway GAN architectures which use di erent GAN formulas, GAN [3], LSGAN [7], DRAGAN [6] and WGAN-GP [4]. (Local D: using the local discriminator proposed by CycleGAN [10].)

24 24 Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang Discriminator loss Ours (A-WGAN) WGAN-GP DRAGAN LSGAN (Local D) LSGAN GAN epoch Fig. 22: Discriminator loss of training on the MIT-Adobe-5K dataset for several one-way GAN architectures, GAN [3], LSGAN [7], DRAGAN [6] and WGAN- GP [4]. The value can be used as a good indicator of convergence for our model. Discriminator loss two-way trained on HDR (without ibn) two-way trained on HDR (with ibn) two-way trained on MIT-Adobe-5K (without ibn) two-way trained on MIT-Adobe-5K (with ibn) epoch Fig. 23: Discriminator loss of training on the MIT-Adobe-5K dataset and our HDR dataset for the proposed two-way GAN architecture with and without individual BN. It shows that individual BN is crucial for training on the HDR dataset.

25 Supplementary Material: Deep Photo Enhancer 25 References 1. Aubry, M., Paris, S., Hasino, S.W., Kautz, J., Durand, F.: Fast local laplacian filters: Theory and applications. ACM Transactions on Graphics (TOG) 33(5), 167 (2014) 2. Bychkovsky, V., Paris, S., Chan, E., Durand, F.: Learning photographic global tonal adjustment with a database of input/output image pairs. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2011) 3. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems (NIPS). pp (2014) 4. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.: Improved training of Wasserstein GANs. In: Advances in neural information processing systems (NIPS) (2017) 5. Ignatov, A., Kobyshev, N., Vanhoey, K., Timofte, R., Van Gool, L.: DSLR-quality photos on mobile devices with deep convolutional networks. In: IEEE International Conference on Computer Vision (ICCV) (Oct 2017) 6. Kodali, N., Abernethy, J., Hays, J., Kira, Z.: On convergence and stability of GANs. In: arxiv preprint arxiv: (2017) 7. Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: The IEEE International Conference on Computer Vision (ICCV) (Oct 2017) 8. Wang, S., Cho, W., Jang, J., Abidi, M.A., Paik, J.: Contrast-dependent saturation adjustment for outdoor image enhancement. JOSA A 34(1), (2017) 9. Wang, S., Zheng, J., Hu, H.M., Li, B.: Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE Transactions on Image Processing (TIP) 22(9), (2013) 10. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: The IEEE International Conference on Computer Vision (ICCV) (2017)