Virtual Restoration of old photographic prints. Prof. Filippo Stanco

Virtual Restoration of old photographic prints Prof. Filippo Stanco

Many photographic prints of commercial / historical value are being converted into digital form. This allows: Easy ubiquitous fruition: an old print can be digitalized and shared in a database. Now, everybody can have a virtual copy. Reversible manipulation: a digital copy can be modified without retouching the original. Simpler data management: fast database retrieval, less physical space...

The image, after digitalization, can be virtually restored. Virtual restoration: Enhances the visual quality; Decreases noise-related high frequency components; Decreases the entropy; Decreases the number of features to store in the database. Advantages: Ubiquitous fruition is made easier The performance of retrieval and browsing tools are improved;

User-guided Restoration Automatic Reduced entropy = stronger compression Less high frequency content = efficient coding

User-guided restoration: Defects are not automatically traced; All the corrections must be suggested by the user; complex; expensive. Automatic restoration: quick; simple

Defects Caused by: Careless conservation; Prolonged exposure to light. Classification: Mechanical damage Chemical damage Deposited matter

Mechanical damages: cracks

Mechanical damages: scratches

Mechanical damages: cracks

Mechanical damages: Torn paper

Mechanical damages: Heavy human modifications

Chemical damages: water blotches or water blotches

Chemical damages: blotches

Chemical damages: foxing

Deposited matter

Photoshop

Water blotches or Gore d acqua

Water blotches Chemical damages; Careless conservation; They add noise to the original information; No regular structure The contour is darker than the blotch

Examples

Detection The user manually selects just one point in the damaged area, The remaining region of the damage is automatically detected.

Detection Graylevel image; Contrast improvement; Canny s edge detector; Click inside the blotch; Extended detection; More detailed edge detection; Further clicks and improved detection; Morphological closing;

Restoration Additive-multiplicative model Interpolation across the border Interpolation unchanged pixels

Restoration step 1 Additive-multiplicative model Interpolation across the border Interpolation unchanged pixels

Restoration step 1 Additive-multiplicative model over each RGB color plane. Used model: J ( Ω ) = α * I( Ω) + β I(x,y) uncorrupted image J(x,y) actual data image Ω area to restore α, β parameters to be determined

Restoration step 1 We denote with Var[.] and E[.] respectively the variance and the mean in [.]. 2 [ J ( Ω) ] = α var[ I( Ω) ] [ J ( Ω) ] = αe[ I( Ω) ] + β var E The variance and the mean of the uncorrupted image I are unknown.

Restoration step 1 Ω is an area around the blotch Ω W S

Restoration Ω and Ω are located in areas with significant details (border, edges); Ω and Ω are divided in n different blocks. We set min=min(i(ω )), max=max(i(ω )), and p=(max-min)/n p Ω V 0 =min V 1 V 2 V 3 V n-1 V n =max

Restoration Ω C 1 C 2 C 3 C n V 0 =min V 1 V 2 V 3 V n-1 V n =max Each pixel p in Ω is compared to C i and is assigned to the closest set.

Restoration Ω Ω median(ω 1 ) Ω 1 Ω 1 median(ω 1 ) median(ω 1 ) Ω 2 Ω 2 median(ω 2 ) median(ω 1 ) median(ω 1 ) Ω 1 Ω 3.. Ω 1 Ω 3.. median(ω 3 ) median(ω 4 ) median(ω 1 ) Ω n Ω n median(ω n )

Restoration We compute the differences diff j = median(ω 1 ) - median(ω j ) with j=1,...,n. The minimum value diff j determines the area Ω j to couple with Ω 1 This operation is repeated for each Ω i with i=1,...,n.

Restoration median(ω 1 ) median(ω 2 ) median(ω 3 ) median(ω 4 ).. median(ω n ) median(ω 1 ) median(ω 2 ) median(ω 3 ) median(ω 4 ).. median(ω n )

Restoration n = 2

Restoration step 1 We approximate I(Ω) with I(Ω ) ~ α = ~ β = E var[ J ( Ω) ] var[ I( Ω') ] [ J ( Ω) ] α * E[ I( Ω') ] Using the estimated parameters and, it is possible to compute the restored value for each pixel of the blotch: α ~ ~ ~ I ( Ω) = ( J ( Ω) β ) ~ α β ~

Results step 1

Restoration step 2a Additive-multiplicative model Interpolation across the border Interpolation unchanged pixels

Restoration step 2a Still apparent borders of the blotch We use linear interpolation across the contour

Restoration step 2a Luminance gradient is calculated for each pixel p in the contour;

Restoration step 2a We consider an array od 2L+1 pixels centred on p; This array contains L pixels inside the blotch and L outside.

Restoration step 2a P start is the first pixel of the array and P end the last one P end P start

Restoration step 2a If d(p i ) is the normalized distance of each pixel in the array from P start : d( P) i = P P i start The new intensity values are: P end P end I ~ ( P) i = d( P)* I ~ ( P i start ) + (1 d( P))* I ~ ( P i end )

- Original gray level in row 255 Restoration step 2a

Restoration step 2a -Original gray level in row 255 - restored line after step 1

Restoration step 2a -Original gray level in row 255 - restored line after step 1 - restored line after interpolation step 2a

Restoration step 2b Additive-multiplicative model Interpolation across the border Interpolation unchanged pixels

Restoration step 2b Some pixels are interpolated several times, while other ones are left unchanged; Each unchanged pixel is assigned a gray level corresponding to the average of its interpolated neighbors in a 3x3 area.

Experimental results:

Foxing

Foxing It depends on joined fungal activity and metal induced degradation; Reddish-brown posts; Dark-brown center and an area where the color is smoothed; The area around the center can include residual original information.

Examples

Detection We change the color space RGB into YCbCr; In case of foxing, the histogram of Cr has a tail on the right (formed by a set of small bins), and the peak is in the left portion of the histogram.

Detection

Inpainting these points are darker than the others one, and they cover definitively the original information; We propagate inside the foxing the values in the border outside the stain.

Inpainting

Enlarged detection we find the pixels where the original information is anly partially affected by foxing; we start from the previous selection in Cr

Enlarged detection

Restoration We use the Additive-Multiplicative model: J ( x, y x, y Ω ) = α * I( Ω ) + β

Results

Fragmented Glass Plate Photographs

Glass Plate Photographs In the second half of the nineteenth century photographs were still taken on a glass plate coated with a wet collodion emulsion with silver halide. Properties: Glass is a chemically more stable support than celluloid; Glass is more fragile than other supports.

Fragmented Glass Plate Photographs Careless manipulation Break Digital acquisition Restoration Reproduction

Examples

Detection The areas in the gap are much lighter than the rest of image. Performing a thresholding operation we label as: white all the pixels in the gap; black otherwise.

Detection

Detection We label as gray all the pixels that belong to the border. For each block of size 2x2 in the image: if (#white_pixels>0 and #white_pixels<4) Black_pixels become Gray_pixels end end

Detection

Detection Which fragment the gray pixels belong to? We automatically find one point inside a fragment and another one inside the other piece. Or The user clicks over two points in the image.

Detection. A 1. A 2

Detection BF 1 BF 2

Preprocessing False colours exist on the borders of the image fragments

Preprocessing M is the area to be adjusted For each pixel P in M end P = nearest pixel P : P-P < Th 2

Restoration: rotation For each angle δ from 0 to 359 End BF 1 δ is the image BF 1 rotated by δ degrees The cross correlation of the two images BF 1 δ and BF 2 is evaluated M(δ) = is the maximum value of the cross correlation matrix D(δ) = contains the values (dx,dy) related to the maximum The desired rotation angle α is equal to the position of the maximum value of M.

Restoration: rotation We rotate the fragment Fr 1 by α degrees in order to obtain New_Fr 1 Fr 1 New_Fr 1

Restoration: translation We shift the fragment New_Fr 1 by dx in the horizontal direction and by dy in the vertical direction. (dx,dy) are the values in D(α). We merge in the same image the fragments Shift_New_Fr 1 and Fr 2.

Restoration: translation

Considerations To reduce the computational cost: we use a suitable subset of quantized angles from 0 to 359 the decimated images BF 1 δ d and BF 2 d are considered instead of BF 1 δ and BF 2 The maximum of M determines the angle β that is a first estimate of the correct angle α. We then repeat the procedure using the original images, within an interval of angles centered in β.

Final interpolation To eliminate possible pixels that are still in the gap: For each pixel P in the gap P = average of pixels in the neighborhood of P but not in the gap end

Experimental results the images utilized in these experiments are acquired from real negatives; hence, an equivalent undamaged image does not exist; it is not possible to evaluate the performances quantitatively (e.g., using MSE, PSNR).

Experimental results

Without preprocessing

Descreening

Why? Civic museum of Pordenone Candiani collection

Halftoning or screening A grayscale photograph has hundreds of shades of gray, while black-and-white display devices requires only binary images. When an image is reproduced, the continuous tone image is converted in a binary image. This converting process, called screening or halftoning, breaks an image into a series of dots with different sizes. Each size approximates a shade of color: a group of large dots placed closely together appears black; a group of smaller dots with larger spaces between them produces a weaker gray shade; while a group of even smaller dots spaced widely apart appears almost white

Halftoning techniques Error Filter Floyd-Steinberg Error Filter Jarvis Clustered-dot dither Dispersed-dot dither

Which technique?

Inverse halftoning or descreening Usually, the image processing operators work better if they are applied over a continuous tone image. Gray images need to be reconstructed from the halftones through inverse halftoning or descreening. The screening operators lost some information, and there is no way to reconstruct a perfect gray image from the given halftoned image. Many efficient inverse halftoning algorithms have been developed in the past several years to improve the quality of the reconstructed image. Often, the performance of these methods is related to the knowledge of the used halftoning algorithm.

Descreening: our algorithm It is based over the idea that the original screen pattern is easy to detect in Fourier domain. They are localized in the peaks out of the central region around the DC component. If they are removed the image appears like the original continuous tone and any regular pattern is detectable.

Fuorier Transform

Butterworth We use a band reject filter where n and W are the degree and the width of the filter, respectively; and D(i, j) is the Euclidian distance between the value with the coordinate (i, j) and DC. Due to the Fourier symmetry there are four peaks at distance r from the center; the bandreject Butterworth filter remove all of them.

This filter is applied K times A typical plot for K = 3 and W k = 30

The radii r k are automatically determined using a simple but effective heuristic. Since the DC component is in the center of the Fourier transform, we use as r 1 the position of the peak of maximum value far from DC. The r 2 is the position of the second maximum far from DC and different from r 1. Finally, r 3 is the next higher value not in the center of the frequency domain and with r 3 r 1 and r 3 r 2.

This approach do not depend by the halftoning algorithm used. The method parameters are related to the image resolution and they not change for images with the same resolution. As proof that our algorithm remove only the screen pattern, we have inverted the fourier domain and, hence, we have reconstructed only the removed frequency.

Moreover, if this pattern is subtracted to the input image, the continuous tone image is obtained.

Experimental results

Algorithm generality To prove that the algorithm remove a regular pattern, we apply it over a different class of problem. The millepunti technique is a printing style where the regular pattern is desired. Our descreening technique remove the millepunti pattern.

Antique books

Experimental results

Experimental Results