Miscellaneous Topics Part 1

Transcription

1 Computational Photography: Miscellaneous Topics Part 1 Brown 1

2 This lecture s topic We will discuss the following: Seam Carving for Image Resizing An interesting new way to consider resizing images This paper made a wave at SIGGRAPH 07 Color Harmonization An automatic approach to harmonize colors With an overview of HSV colorspce Image Inpainting Seminal paper by Bertalmio et al on image restoration 2

3 Three Papers Discussed Three papers in this lecture: 1. Shai Avidan and Ariel Shamir SIGGRAPH 07 Seam Carving for Content-Aware Image Resizing 2. Daniel Cohen-Or et al, SIGGRAPH06 Color Harmonization (With a pre-discussion on color) 3. Bertalmio et al SIGGRAPH 00 Image Inpainting 3

4 Seam Carving SIGGRAPH 07 Shai Avidan and Ariel Shamir Shai was at MERL when paper was published, left for Adobe, then totel-aviv University Ariel is a professor at the Efi Arazi School of Computer Science (Israel) Both got their PhD s from Hebrew University in 1999 Idea Shai Typical image operation: scale-up or down and image - Don t necessarily want the image to be scaled pixel by pixel - Instead want the content to fit into something smaller (call this re-targeting) - How can we do this in a clever way? Ariel 4

5 Idea Resizing We want to reduce the size of this image, to make it smaller. Here is an image and a plot of each pixels importance (blue=less important, yellow=more important) Let s say we want to reduce the width of this image by K columns. 5

6 Cropping Cropping Current solution crop the image: Several automatic techniques exist to crop important parts of the image (and remove the less important). Crop a region such that a total of k-columns are remove from the left and right. 6

7 Column Removal Column Removal (note artifacts) Another option is to remove whole columns, where columns with less energy are removed first. So, remove the K columns with minimum energy. This results in strange artifacts. 7

8 Pixel Removal Pixel Removal We can remove the K less important pixels per row. This produces very ugly results. 8

9 Seam Carving Seam Carving We can carve K seams (where a seam runs from top to bottom) with the least energy. 9

10 Comparison 10

11 Seam Carving orizontal seam A connected line from left to right that can move at most 1-pixel per row Vertical seam (connected line from top to bottom, that can move at most 1-pixel per column) 11

12 Seams and Energy If we want to reduce by either width or height, remove the seam with the least energy Optimal horizontal seam (right to left), where e is some energy function * Define (vertical seam) bottom-to-top similarly A pixel (i,j) can only be part of one seam. So the goal then is to final the optimal seams, s*, such that: 12

13 What Seam Energy to Use? The authors discuss several choices, but these are the two they found the most useful: Simply gradient energy (not this is not magnitude, but an approximation) Where, HoG is the histogram of oriented gradients. Using an 11x11 window about pixel (x,y), compute gradients orientation and then build a histogram of the orientation (in this case an 8-bin histogram). Take the max value of the histogram. A value with a large max means here is a strong edge in the 11x11 window. This energy attracts seams to edges, but not to cross the edge (because of the numerator). 13

14 e 1 and e HoG example 14

15 Image Expanding What if we want to expand the image? Picking the minimum energy seam and duplicating it, gives a strange effect! 15

16 Image Expanding Say we want to expand by K pixels Pick the K minimum seam Duplicate these seams by linear-interpolation 16

17 A little help from the user A failure case, seam carving is not aware of the content s meaning, only energy. 17

18 A little help from the user No problem, have the user assign maximum energy to regions. 18

19 Object Removal Assign max energy Assign 0 energy 19

20 No hope cases Structure of content is not suitable for seam-carving. OK, go back to regular resizing! 20

21 Seam Carving Summary Very, very cool idea So obvious after you have seen it that you wish you had this idea! You d be rich, Adobe would hire you and stuff your pockets with cash This paper is part of a move to re-think image editing Content-aware image editing Resizing while considering the content User assistant for hard cases 21

22 Color Harmonization SIGGRAPH 07 Daniel Cohen-Or et al Daniel is a professor at the Tel Aviv University in Israel He has many SIGGRAPH papers every year. Daniel Idea Harmonization is the result of choosing colors that are pleasing to humans - Can we provide a way to do this for images? - Retarget colors to be harmonized? 22

23 Recall Our discussion on RGB color CIE XYZ perceptual space And srgb color space Here, we will discuss HSV/HSI space 23

24 x = X / (X+Y+Z) y = Y / (X+Y+Z) z = Z / (X+Y+Z) CIE XYZ to CIE xy Chromaticity [X, Y, Z] (X=Y=Z) CIE XYZ CIE xy

25 Standard RGB (srgb) G In 1996, Microsoft and HP defined the standard RGB primaries. R

26 The HSV/HSI Colospace This is a different way at looking at the RGB color cube. Let s first consider the Hue, Saturation, and Value (HSV) color space which is a variation on the HLS. For HSV, we modify the RGB cube such that the greyscale line is the vertical axis. We now modify the cube to be a cone. A color in this cone is expressed by three values: 1) its position along the vertical axis (Value), 2) an angle (Hue) about that axis (with reference point Hue=0 o =red); 3) and the distance to the edge of the cone (Saturation). We sometimes refer to hue as the Color Wheel. RGB Transformation Visualization between RGB and HSV Example Applet Showing RBG and HSV 26

27 HVS and the related HLS Adobe uses the HLS space, which is similar to HSV. Instead of Value, however, HLS uses the term Lightness. In, HSV, a value=1 does not result in a pixel that is completely white, it would only be completely white if the saturation=0. However, for HLS, as the lightness value increases, the range of colors decreases, thus everything becomes white with a lightness=1. 27

28 Color Enhancement via HSV Manipulation We can manipulate each component of hue, saturation and intensity for all pixels simultaneously Normally for image-editing applications: 1. Convert colors in RGB representation to HSI 2. Manipulate HSI components, typically by Hue transformation involves adding a user-specified constant to the hues of all pixels (equivalent to rotating chromaticity plane about intensity axis) Saturation and intensity transformation involves scaling these values by a constant factor 3. Convert colors in HSI representation back to RGB 28

29 HSI Manipulation Examples Hue Saturation Intensity 29

30 Another example Hue modification original Saturation modifications Lightness modifications 30

31 Back to Color Harmony Paper 31

32 What is color harmony? Harmonic colors are pleasing to the eye. They engage the human observer and give a sense of order and balance in the visual experience. [slides from Cohen-Or s SIGGRAPH talk] 32

33 Formal definition of color harmony? Mathematical formulation has been developing together with color theory Newton, Goethe, Young, Maxwell Itten [1960]: harmony means relationships on the hue wheel: 2-color harmony: complementary colors 3-color harmony: equilateral triangle N-color harmony: equilateral N-gon 33

34 Formal definition of color harmony? Matsuda [1995]: extensive empirical studies, derived 8 hue templates Tokumaru et al. [2002] developed a fuzzy system to evaluate the harmony of color schemes i type V type L type I type T type Y type X type N type 34

35 Harmonic scheme The templates can be arbitrarily rotated Harmonic scheme is template type T m + specific orientation α i type V type L type I type T type Y type X type N type Type N is not considered in this paper, this is for grayscale images. 35

36 Harmony score To evaluate the harmony of an input image X we analyze its hue histogram: Every pixel p contributes its saturation S(p) to the bin of the hue H(p) 36

37 Harmony function The harmony of image X w. r. t. harmonic scheme (T m, α) : F X,( T, ) m H( p) ETm ( )( p) S( p) p X H(p) E Tm(α) (p) This term is the closest edge of the template (oriented at angle alpha) 37

38 Best template We compute α that minimizes F(X, (T m, α)) for each template T m using Brent s algorithm The best-fitting harmonic scheme: ( T,α ) arg min F X,( T,α) m 0 0 ( m,α) m So, given an image, they compute the best fit template from the different types (see 49) at the best orientation alpha. 38

39 Harmonization Given (T m, α) we shift the hues so that the hue histogram is contained in (T m, α) 39

40 Color shifting The hue of pixel p is shifted to its associated sector E Tm(α) (p) The amount of squeezing is controlled by a Gaussian fall-off function 40

41 Color Coherency Problem The problem, no way to force neighboring pixels to similar colors. Here, similar colors (blue) move to two different regions (green, purple). 41

42 Another example Color coherency Problem! This would be better! 42

43 Graph-cut optimization To make the coloring more coherent we assign E Tm(α) (p) by optimizing the labeling V E( V) E ( V) E ( V ) 1 2 Favors short distance to the template sector Favors coherent labeling of neighboring pixels [Back to MRF each pixel now has a data-cost E1 and a neighbor cost E2. This is similar to the lazy-snapping MRF formulation. ] 43

44 Graph-cut optimization To make the coloring more coherent we assign E Tm(α) (p) by optimizing the labeling V E( V) E ( V) E ( V ) 1 2 E ( ) ( ) ( ( )) ( ) 1 V H p H V p S p p 2 max { p, q} N E ( V ) V ( p), V ( q) H( p) H( q) S ( p, q) 1 44

45 Results 45

46 Overcoming segmentation problems The graph-cut may fail when an object in the image has several connected components 46

47 Overcoming segmentation problems User-assisted fix: scribbling on the erroneously labeled area Re-compute the labeling 47

48 Results choosing colors 48

49 Results cut and paste The background is harmonized according to the bestfitting harmonic template of the pasted foreground original harmonized harmonized 49

50 Text over a poster Results 50

51 Text over a poster Results 51

52 Results Images harmonized to different flags colors. Find the flags template, force the image to this. 52

53 Discussion Nature is already harmonic original best-fitting template poorly-fitting template 53

54 Discussion Cannot improve good artwork! Wassily Kandinsky, Composition VII,

55 Discussion Grayish colors will remain such 55

56 Harmony Summary Provides a method to enhances the harmony of colors in a given image Operates by fitting the image hues into a given harmonic distribution Several different harmonic chooses are predefined (based on color theory) Especially useful for artificial colors, cutand-paste settings and collages that combine imagery from different sources 56

57 Image Inpainting SIGGRAPH 00 Marcelo Bertalmio et al While Marcelo was a PhD student at U. Of Minnesota Now a professor at Universitat Pompeu Fabra (Spain) Idea Find a good way to fill in missing image information - The idea comes from how artists fix paintings when the paint chips away: they in paint Marcelo 57

58 Real Inpainting 58

59 Image Inpainting Assume you have an image that has been corrupted. Above you want to fill in the white pixels with the surrounding content. Or, another scenario, you the user draw the region you want to correct. 59

60 The idea Ω is a hole you want to repair Ω Ω Ω is a the border of the hole with pixel intensities Image I Idea: propagate information from Ω inside to Ω. But, do this in a clever manner. 60

61 The problem Hole to be filled Incorrect Problem: Need to propagate the information along image gradients! Otherwise the result will be incorrect. 61

62 Solution (start) (after many iterations) (more iterations) (and yet more iterations) The paper write up is very complicated, but can be followed if you read it slowly. Basic idea is that at each step you try to propagate the boundary Ω pixels in image gradient direction. The boundary will slowly shrink. This slowly fills in the hole (and propagates gradient) thus maintaining the direction. 62

63 Examples Inpainting result. The key here is that image gradient was incorporated. 63

64 Limitations Not perfect, because it can t reproduce texture 64

65 Inpainting Summary This idea spawned a great deal of further work Google Scholar has 1902 citations to this paper The idea is simple Fill in the hole by propagating information The approach is clever But the problem is good The user helps by giving the region to correct Again, we are seeing user-assistance in the procedure The algorithm itself is automated Similar ideas: Bayesian matting/possion Matting/PIE/and so on 65