Virtual Restoration of old photographic prints. Prof. Filippo Stanco

Transcription

1 Virtual Restoration of old photographic prints Prof. Filippo Stanco

2 Many photographic prints of commercial / historical value are being converted into digital form. This allows: Easy ubiquitous fruition: an old print can be digitalized and shared in a database. Now, everybody can have a virtual copy. Reversible manipulation: a digital copy can be modified without retouching the original. Simpler data management: fast database retrieval, less physical space...

3 The image, after digitalization, can be virtually restored. Virtual restoration: Enhances the visual quality; Decreases noise-related high frequency components; Decreases the entropy; Decreases the number of features to store in the database. Advantages: Ubiquitous fruition is made easier The performance of retrieval and browsing tools are improved;

4 User-guided Restoration Automatic Reduced entropy = stronger compression Less high frequency content = efficient coding

5 User-guided restoration: Defects are not automatically traced; All the corrections must be suggested by the user; complex; expensive. Automatic restoration: quick; simple

6 Defects Caused by: Careless conservation; Prolonged exposure to light. Classification: Mechanical damage Chemical damage Deposited matter

7 Mechanical damages: cracks

8 Mechanical damages: cracks

9 Mechanical damages: scratches

10 Mechanical damages: cracks

11 Mechanical damages: Torn paper

12 Mechanical damages: Heavy human modifications

13 Chemical damages: water blotches or water blotches

14 Chemical damages: blotches

15 Chemical damages: foxing

16 Deposited matter

17 Photoshop

18

19

20 Water blotches or Gore d acqua

21 Water blotches Chemical damages; Careless conservation; They add noise to the original information; No regular structure The contour is darker than the blotch

22 Examples

23 Detection The user manually selects just one point in the damaged area, The remaining region of the damage is automatically detected.

24 Detection Graylevel image; Contrast improvement; Canny s edge detector; Click inside the blotch; Extended detection; More detailed edge detection; Further clicks and improved detection; Morphological closing;

25 Restoration Additive-multiplicative model Interpolation across the border Interpolation unchanged pixels

26 Restoration step 1 Additive-multiplicative model Interpolation across the border Interpolation unchanged pixels

27 Restoration step 1 Additive-multiplicative model over each RGB color plane. Used model: J ( Ω ) = α * I( Ω) + β I(x,y) uncorrupted image J(x,y) actual data image Ω area to restore α, β parameters to be determined

28 Restoration step 1 We denote with Var[.] and E[.] respectively the variance and the mean in [.]. 2 [ J ( Ω) ] = α var[ I( Ω) ] [ J ( Ω) ] = αe[ I( Ω) ] + β var E The variance and the mean of the uncorrupted image I are unknown.

29 Restoration step 1 Ω is an area around the blotch Ω W S

30 Restoration Ω and Ω are located in areas with significant details (border, edges); Ω and Ω are divided in n different blocks. We set min=min(i(ω )), max=max(i(ω )), and p=(max-min)/n p Ω V 0 =min V 1 V 2 V 3 V n-1 V n =max

31 Restoration Ω C 1 C 2 C 3 C n V 0 =min V 1 V 2 V 3 V n-1 V n =max Each pixel p in Ω is compared to C i and is assigned to the closest set.

32 Restoration Ω Ω median(ω 1 ) Ω 1 Ω 1 median(ω 1 ) median(ω 1 ) Ω 2 Ω 2 median(ω 2 ) median(ω 1 ) median(ω 1 ) Ω 1 Ω 3.. Ω 1 Ω 3.. median(ω 3 ) median(ω 4 ) median(ω 1 ) Ω n Ω n median(ω n )

33 Restoration We compute the differences diff j = median(ω 1 ) - median(ω j ) with j=1,...,n. The minimum value diff j determines the area Ω j to couple with Ω 1 This operation is repeated for each Ω i with i=1,...,n.

34 Restoration median(ω 1 ) median(ω 2 ) median(ω 3 ) median(ω 4 ).. median(ω n ) median(ω 1 ) median(ω 2 ) median(ω 3 ) median(ω 4 ).. median(ω n )

35 Restoration n = 2

36 Restoration step 1 We approximate I(Ω) with I(Ω ) ~ α = ~ β = E var[ J ( Ω) ] var[ I( Ω') ] [ J ( Ω) ] α * E[ I( Ω') ] Using the estimated parameters and, it is possible to compute the restored value for each pixel of the blotch: α ~ ~ ~ I ( Ω) = ( J ( Ω) β ) ~ α β ~

37 Results step 1

38 Restoration step 2a Additive-multiplicative model Interpolation across the border Interpolation unchanged pixels

39 Restoration step 2a Still apparent borders of the blotch We use linear interpolation across the contour

40 Restoration step 2a Luminance gradient is calculated for each pixel p in the contour;

41 Restoration step 2a We consider an array od 2L+1 pixels centred on p; This array contains L pixels inside the blotch and L outside.

42 Restoration step 2a P start is the first pixel of the array and P end the last one P end P start

43 Restoration step 2a If d(p i ) is the normalized distance of each pixel in the array from P start : d( P) i = P P i start The new intensity values are: P end P end I ~ ( P) i = d( P)* I ~ ( P i start ) + (1 d( P))* I ~ ( P i end )

44 - Original gray level in row 255 Restoration step 2a

45 Restoration step 2a -Original gray level in row restored line after step 1

46 Restoration step 2a -Original gray level in row restored line after step 1 - restored line after interpolation step 2a

47 Restoration step 2b Additive-multiplicative model Interpolation across the border Interpolation unchanged pixels

48 Restoration step 2b Some pixels are interpolated several times, while other ones are left unchanged; Each unchanged pixel is assigned a gray level corresponding to the average of its interpolated neighbors in a 3x3 area.

49 Experimental results:

50 Experimental results:

51 Experimental results:

52 Experimental results:

53 Foxing

54 Foxing It depends on joined fungal activity and metal induced degradation; Reddish-brown posts; Dark-brown center and an area where the color is smoothed; The area around the center can include residual original information.

55 Examples

56 Detection We change the color space RGB into YCbCr; In case of foxing, the histogram of Cr has a tail on the right (formed by a set of small bins), and the peak is in the left portion of the histogram.

57 Detection

58 Detection

59 Inpainting these points are darker than the others one, and they cover definitively the original information; We propagate inside the foxing the values in the border outside the stain.

60 Inpainting

61 Enlarged detection we find the pixels where the original information is anly partially affected by foxing; we start from the previous selection in Cr

62 Enlarged detection

63 Restoration We use the Additive-Multiplicative model: J ( x, y x, y Ω ) = α * I( Ω ) + β

64 Results

65 Results

66 Fragmented Glass Plate Photographs

67 Glass Plate Photographs In the second half of the nineteenth century photographs were still taken on a glass plate coated with a wet collodion emulsion with silver halide. Properties: Glass is a chemically more stable support than celluloid; Glass is more fragile than other supports.

68 Fragmented Glass Plate Photographs Careless manipulation Break Digital acquisition Restoration Reproduction

69 Examples

70 Examples

71 Detection The areas in the gap are much lighter than the rest of image. Performing a thresholding operation we label as: white all the pixels in the gap; black otherwise.

72 Detection

73 Detection We label as gray all the pixels that belong to the border. For each block of size 2x2 in the image: if (#white_pixels>0 and #white_pixels<4) Black_pixels become Gray_pixels end end

74 Detection

75 Detection Which fragment the gray pixels belong to? We automatically find one point inside a fragment and another one inside the other piece. Or The user clicks over two points in the image.

76 Detection. A 1. A 2

77 Detection. A 1. A 2

78 Detection. A 1. A 2

79 Detection BF 1 BF 2

80 Preprocessing False colours exist on the borders of the image fragments

81 Preprocessing M is the area to be adjusted For each pixel P in M end P = nearest pixel P : P-P < Th 2

82 Restoration: rotation For each angle δ from 0 to 359 End BF 1 δ is the image BF 1 rotated by δ degrees The cross correlation of the two images BF 1 δ and BF 2 is evaluated M(δ) = is the maximum value of the cross correlation matrix D(δ) = contains the values (dx,dy) related to the maximum The desired rotation angle α is equal to the position of the maximum value of M.

83 Restoration: rotation We rotate the fragment Fr 1 by α degrees in order to obtain New_Fr 1 Fr 1 New_Fr 1

84 Restoration: translation We shift the fragment New_Fr 1 by dx in the horizontal direction and by dy in the vertical direction. (dx,dy) are the values in D(α). We merge in the same image the fragments Shift_New_Fr 1 and Fr 2.

85 Restoration: translation

86 Considerations To reduce the computational cost: we use a suitable subset of quantized angles from 0 to 359 the decimated images BF 1 δ d and BF 2 d are considered instead of BF 1 δ and BF 2 The maximum of M determines the angle β that is a first estimate of the correct angle α. We then repeat the procedure using the original images, within an interval of angles centered in β.

87 Final interpolation To eliminate possible pixels that are still in the gap: For each pixel P in the gap P = average of pixels in the neighborhood of P but not in the gap end

88 Experimental results the images utilized in these experiments are acquired from real negatives; hence, an equivalent undamaged image does not exist; it is not possible to evaluate the performances quantitatively (e.g., using MSE, PSNR).

89 Experimental results

90 Experimental results

91 Without preprocessing

92 Without preprocessing

93 Descreening

94

95 Why? Civic museum of Pordenone Candiani collection

96 Halftoning or screening A grayscale photograph has hundreds of shades of gray, while black-and-white display devices requires only binary images. When an image is reproduced, the continuous tone image is converted in a binary image. This converting process, called screening or halftoning, breaks an image into a series of dots with different sizes. Each size approximates a shade of color: a group of large dots placed closely together appears black; a group of smaller dots with larger spaces between them produces a weaker gray shade; while a group of even smaller dots spaced widely apart appears almost white

97

98 Halftoning techniques Error Filter Floyd-Steinberg Error Filter Jarvis Clustered-dot dither Dispersed-dot dither

99 Which technique?

100 Inverse halftoning or descreening Usually, the image processing operators work better if they are applied over a continuous tone image. Gray images need to be reconstructed from the halftones through inverse halftoning or descreening. The screening operators lost some information, and there is no way to reconstruct a perfect gray image from the given halftoned image. Many efficient inverse halftoning algorithms have been developed in the past several years to improve the quality of the reconstructed image. Often, the performance of these methods is related to the knowledge of the used halftoning algorithm.

101 Descreening: our algorithm It is based over the idea that the original screen pattern is easy to detect in Fourier domain. They are localized in the peaks out of the central region around the DC component. If they are removed the image appears like the original continuous tone and any regular pattern is detectable.

102 Fuorier Transform

103

104

105 Butterworth We use a band reject filter where n and W are the degree and the width of the filter, respectively; and D(i, j) is the Euclidian distance between the value with the coordinate (i, j) and DC. Due to the Fourier symmetry there are four peaks at distance r from the center; the bandreject Butterworth filter remove all of them.

106 This filter is applied K times A typical plot for K = 3 and W k = 30

107

108 The radii r k are automatically determined using a simple but effective heuristic. Since the DC component is in the center of the Fourier transform, we use as r 1 the position of the peak of maximum value far from DC. The r 2 is the position of the second maximum far from DC and different from r 1. Finally, r 3 is the next higher value not in the center of the frequency domain and with r 3 r 1 and r 3 r 2.

109

110 This approach do not depend by the halftoning algorithm used. The method parameters are related to the image resolution and they not change for images with the same resolution. As proof that our algorithm remove only the screen pattern, we have inverted the fourier domain and, hence, we have reconstructed only the removed frequency.

111 Moreover, if this pattern is subtracted to the input image, the continuous tone image is obtained.

112 Experimental results

113 Experimental results

114 Experimental results

115 Experimental results

116 Algorithm generality To prove that the algorithm remove a regular pattern, we apply it over a different class of problem. The millepunti technique is a printing style where the regular pattern is desired. Our descreening technique remove the millepunti pattern.

117

118 Antique books

119

120 Experimental results

121 Experimental Results