Combining Sketch and Tone for Pencil Drawing Production. Cewu Lu, Li Xu, Jiaya Jia, The Chinese University of Hong Kong

Transcription

1 Combining Sketch and Tone for Pencil Drawing Production Cewu Lu, Li Xu, Jiaya Jia, The Chinese University of Hong Kong

2 Fundamental Pictorial Language Popular Artistic Forms Pencil Sketch High in real-work

3 Episode of <The Pacific>

4

5

6 Outline Previous work Observation (Two stages) Framework (Pencil drawing prior included) Discussion Comparison and Results Conclusion

7 Outline Previous Work Observation (Two stages) Framework (Pencil drawing prior including) Discussion Comparison and Results Conclusion

8 Previous Work (Model-based) Accurate Line and Contour - Pencil Line Drawing [Sousa and Buchanan 1999] [Lake et al. 2000] [Lee et al. 2006]

9 Previous Work (Model-based) Surface Normal Lighting Information Geometry Information Automatic Hatching Generation

10 Previous Work (Image-based)

11 Previous Work (Image-based)

12 Previous Work (Image-based) Li and Huang 2003 Sun et al Gao et al. 2010

13 Previous Work (Image-based) Li and Huang 2003 Sun et al Gao et al. 2010

14 Previous Work (Image-based) Li and Huang Sun et al Gao et al. 2010

15 Previous Work (Image-based) Li and Huang Sun et al Gao et al. 2010

16

17

18

19

20 Outline Previous work Observation (Two stages) Framework (Pencil drawing prior including) Discussion Comparison and Results Conclusion

21 Observation First Stage Second Stage Human Drawing Procedure

22 Observation Structure Tone

23 Our Framework Stroke Structure Generation Tone Rendering

24 Our Framework Gradient Line Drawing Tone Mapping Texture Rendering

25 Our Framework (Line Drawing) Gradient Line Drawing Tone Mapping Texture Rendering

26 Our Framework (Line Drawing)

27 Our Framework (Line Drawing)

28 Our Framework (Line Drawing) How to simulate it?

29 Our Framework (Line Drawing) Sketching cannot be achieved by detecting direction of each pixel and extending edges along their direction

30 Our Framework (Line Drawing) A naive method Pixel Direction yi atan xi

31 Our Framework (Line Drawing) A naive method Extent along their Direction

32 Our Framework (Line Drawing) Our Robust Edges Direction Classification C 1 C 2

33 Our Framework (Line Drawing) Our Robust Edges Direction Classification C 1 red blue arg max{ C, C } arg max{ C, C } C 2

34 Our Framework (Line Drawing) Why it works?

35 Our Framework (Line Drawing) Input

36 Our Framework (Line Drawing) Gradient

37 Our Framework (Line Drawing) G C C 1 2 C4 C8 Line in direction i 8 C i p if argmax { } G p i i G p i 0 otherwise Pixel direction classification (8 directions )

38 Our Framework (Line Drawing) G C C 1 2 C4 C8 Summing S 8 i 1 C i i

39 Our Framework (Line Drawing)

40 Our Framework (Line Drawing)

41 Our Framework (Line Drawing) Why it works? Robust pixel direction classification Convolution line shaping

42 Our Framework (Line Drawing) For Texture Gradient Ours

43 Our Framework (Line Drawing)

44 Our Framework (Line Drawing) Gradient

45 Our Framework (Line Drawing) Ours

46 Our Framework (Tone Mapping) Gradient Line Drawing Tone Mapping Texture Rendering

47 Our Framework (Tone Mapping) Natural Image Pencil Drawing

48 Our Framework (Tone Mapping) Ernest W. Watson The art of pencil drawing, 1968

49 Our Framework (Tone Mapping) Why is it? The underling mechanism is complex

50 Our Framework (Tone Mapping) Three-layer structure

51 Our Framework (Tone Mapping) Bright Layer: artists do not draw anything

52 Our Framework (Tone Mapping) 1 P v 1 v 1 b e if v 1 b 0 otherwise Bright Layer: artists do not draw anything

53 Our Framework (Tone Mapping) Mid-tone Layer: different pressure

54 Our Framework (Tone Mapping) 2 P v 1 if ua v u ub ua 0 otherwise b Mid-tone Layer: different pressure

55 Our Framework (Tone Mapping) Dark Layer: visually most salient

56 Our Framework (Tone Mapping) 3 P v 1 2 d e v 2 2 d 2 d Dark Layer: visually most salient

57 Our Framework (Tone Mapping) 1 P v 1P1 v 2P2 v 3P3 v Z Unknown Parameters Bright layer b Mild-tone layer ua ub Dark layer d d

58 Our Framework (Tone Mapping) 1 P v 1P1 v 2P2 v 3P3 v Z Unknown Parameters Bright layer b Mild-tone layer ua ub d Dark layer d Maximum Likelihood Estimation

59 Our Framework (Tone Mapping) Tone Mapping 1 P v 1P1 v 2P2 v 3P3 v Z 1 : 2 : 3 =29 : 29 : 42

60 Our Framework (Texture Rendering) Gradient Line Drawing Tone Mapping Texture Rendering

61 Our Framework (Texture Rendering)

62 Our Framework (Texture Rendering) 21 Degree 85 Degree 63 Degree 44 Degree 137 Degree

63 Our Framework (Texture Rendering)? Input Guess Hatch Direction

64 Our Framework (Texture Rendering) Input Guess Hatch Direction

65 Our Framework (Texture Rendering) No Guess! So it is robust to all cases! Tonal texture case Drawn by Katja, 2008

66 Our Framework (Texture Rendering) Tonal texture Pencil Tone Pencil Texture Transfer

67 Our Framework (Texture Rendering) Pencil Texture Transfer

68 Our Framework (Texture Rendering) P Multiplication Model

69 Our Framework (Texture Rendering) In multiplication model Repeated drawing Tone Mapping result? x P x J x Tonal Texture

70 Our Framework (Texture Rendering) In multiplication model, we require: x log P x log J x

71 Our Framework (Texture Rendering) In multiplication model, we require: P J 2 2 min log log

72 Our Framework (Texture Rendering) In multiplication model, we require: P J 2 2 min log log We also require: Local Smoothness min 2 2

73 Our Framework (Texture Rendering) Objective function * 2 2 arg min log P log J 2 2

74 Our Framework (Texture Rendering) Rendering Result S P *

75 Our Framework (Tone Mapping) Gradient Line Drawing Tone Mapping Texture Rendering

76 Our Framework (Combination) T T S S

77 Our Framework (Combination) T

78 Our Framework (Combination) S

79 T S Our Framework (Combination)

80 Outline Previous work Observation (Two stages) Framework (Pencil drawing prior including) Discussion Comparison and Results Conclusion

81

82 Sketchy only

83 Tone only

84 Combination

85 Outline Previous work Observation (Two stages) Framework (Pencil drawing prior including) Discussion Comparison and Results Conclusion

86

87 Sun et al, Efficient Region-Based Pencil Drawing,2007

88 Ours

89 Input Sun et al, 2007 Ours

90 Input Gao et al, 2010 Ours Gao et al, Automatic Generation of Pencil Sketch for 2D Images, 2010

91 Comparison with Software

92 Input PhototoSketch v3.5

93 Input FotoSketcher

94 Input Instant

95 Input Photoshop CS5

96 Input Ours

97 More Results

98 Input

99 Output

100 Input

101 Output

102 Input

103 Output

104 Input

105 Output

106 Input

107 Output

108 Input

109 Output

110 Input

111 Output

112 Input

113 Output

114 Color Pencil Drawing Y Channel U Channel V Channel

115 Color Pencil Drawing Pencil Drawing YUV->RGB U Channel V Channel

116 Input

117 Output

118 Output (color)

119 Input

120 Output

121 Output (color)

122 Input

123 Output

124 Output (color)

125 Input

126 Output

127 Output (color)

128 Input

129 Output

130 Output (color)

131 Input

132 Output

133 Output (color)

134 Input

135 Output

136 Output (color)

137 Input

138 Output

139 Output (color)

140 Conclusion An effective and robust image-based pencil drawing system Pencil drawing priors: crossing junction, pencil tone transfer, tonal pencil texture rending

141 Thank you Annecy, France