Perceptual approach for unsupervised digital color restoration of cinematographic archives

Transcription

1 Pereptual approah for unsupervised digital olor restoration of inematographi arhives Majed Chambah ~, Alessandro Rizzi #, Carlo Gatta #, Bernard Besserer ~, Daniele Marini * ~ L3I, Université de La Rohelle, Frane mhambah@univ-lr.fr, bbessere@univ-lr.fr # Dept. of Information Tehnology - University of Milano/Italy rizzi@dti.unimi.it, gatta@dti.unimi.it *Dept. of Information Siene - University of Milano/Italy daniele.marini@unimi.it ABSTRACT The inematographi arhives represent an important part of our olletive memory. We present in this paper some advanes in automating the olor fading restoration proess, espeially with regard to the automati olor orretion tehnique. The proposed olor orretion method is based on the ACE model, an unsupervised olor equalization algorithm based on a pereptual approah and inspired by some adaptation mehanisms of the human visual system, in partiular lightness onstany and olor onstany. There are some advantages in a pereptual approah: mainly its robustness and its loal filtering properties, that lead to more effetive results. The resulting tehnique, is not just an appliation of ACE on movie images, but an enhanement of ACE priniples to meet the requirements in the digital film restoration field. The presented preliminary results are satisfying and promising. Keywords: Digital film restoration, olor fading, automati olor orretion, olor onstany, ACE. 1. INTRODUCTION Sine the 1950s, monopak olor film beame the standard on whih millions of inematographi works were reorded. A ouple of deades later, it turned out that this proess was hemially unstable, ausing the fading of whole film stoks with time. Usually, a bleahed olor release print is the only available reord of a film. Sine the bleahing phenomenon is irreversible, photohemial restoration of faded prints is not possible, hene the inontestability of digital olor restoration. The tehnique proposed here for the olor digital restoration of faded movies is based on a pereptual approah, inspired by some adaptation mehanisms of the human visual system (HVS), in partiular lightness onstany and olor onstany. The lightness onstany adaptation makes us stably pereive the sene regardless hanges in mean luminane intensity and the olor onstany adaptation makes us stably pereive the sene regardless hanges in olor of the illuminant. The work that preeded this artile foused, on one hand, on the apabilities of olor onstany-like methods to restore faded films [3][5] and the steps of digital olor film restoration [1][2][4]. On the other hand, an algorithm for digital images unsupervised enhanement, alled ACE for Automati Color Equalization was also presented [6][7][23]. It provided experimental evidene for orreting automatially the olor balane of an image using a pereptual approah. 138 Color Imaging VIII: Proessing, Hardopy, and Appliations, Reiner Eshbah, Gabriel G. Maru, Editors, Proeedings of SPIE IS&T Eletroni Imaging, SPIE Vol (2003) 2003 SPIE IS&T X/03/$15.00

2 We present in this paper the first results of a new tehnique for digital olor film restoration based on ACE. This work is the first result of the ollaboration between the L3I lab of Université de La Rohelle and the Department of Information Tehnology of University of Milano. After a detailed presentation of ACE algorithm, we present the new version of ACE for olor digital movie restoration. We report and disuss the results of our experiments and we give some future prospets of this work. 2. ACE: AUTOMATIC COLOR EQUALIZATION ACE, for Automati Color Equalization, is an algorithm for digital images unsupervised enhanement. It is based on a new omputational approah that merges the "Gray World" and "White Path" equalization mehanisms, while taking into aount the spatial distribution of olor information. Inspired by some adaptation mehanisms of the human visual system, ACE is able to adapt to widely varying lighting onditions, and to extrat visual information from the environment effiaiously. ACE implementation follows the sheme shown in figure 1: a first stage aounts for a hromati spatial adaptation (responsible for olor onstany) and a seond stage, dynami tone reprodution saling, onfigures the output range to implement an aurate tone mapping. The first stage merges the gray world and white path approahes and performs a sort of ontrast enhanement, weighted by pixel distane. The result is a loal-global filtering. The seond stage maximizes the image dynami, normalizing the global lightness. No user supervision, no statistis and no data preparation are required to run the algorithm. Fig 1: ACE basi shema In figure 1, I is the input image, R is an intermediate result and O is the output image; subsript denotes the hromati hannel. 2.1 Chromati / Spatial adaptation The first stage, the Chromati/Spatial adaptation, produes an output image R in whih every pixel is reomputed aording to the image ontent, approximating the visual appearane of the image. Eah pixel p of the output image R is omputed separately for eah hromati hannel as shown in equation (1). R r ( I ( p) I ( j)) j Im, j p d ( p, j) ( p) = (1) rmax d ( p, j) j Im, j p I( p) I ( j) aounts for the basi pixel ontrast interation mehanism, d(p, j) is a distane funtion whih weights the amount of loal or global ontribution, r( ) is the funtion, disussed below, that aounts for the relative lightness appearane of the pixel. Pro. of SPIE Vol

3 The lower part of the fration has been introdued to balane the filtering effet of pixels near the border, avoiding a vignetting effet [23]. A sort of lateral inhibition mehanism is realized by omputing the differene between eah pixel value and all other pixels of the seleted image subset. This differene mehanism enhanes the ontrast, tuned by r( ), the hromati adaptation funtion. The distane d( ) weights the global and loal filtering effet. It is well known that both global and loal adaptations are present in the HVS. Global models, in fat, are not able to simulate several loal hromati adaptation effets, e.g. simultaneous ontrast or Mah bands [23]. Different d( ) funtions have been tested. Some funtions seem to ahieve better results than others, but a best funtion has not yet been found. For the tests in this paper we have hosen the Eulidean distane r. For eah pixel of the image, r( ), together with d( ), ontrol the ontrast interation, aounting for the spatial hannel lightness adaptation. It omputes all the single ontributions of the image ontent (weighted by d( ) ) to eah final pixel value in the output image. To perform a gray world mehanism, r( ) has to be an odd funtion, while the white path mehanism is obtained by a non-linear enhanement of small differenes between neighbor pixels. We have tested different r( )funtions, trying to implement an effetive white path mehanism. Figure 2 displays the tested funtions, the odes mean: LI=Linear, SG=Signum, SA=Saturation. Fig. 2: r( ) funtions Linear and Signum funtions an be seen as limit ases of Saturation funtion with unitary or infinite slope respetively. For the test of this paper we have hosen Saturation with a slope value Dynami tone reprodution saling This seond stage maps the intermediate pixels array R into the final output image O. In this stage not only a simple dynami maximization an be made (linear saling), also different referene values an be hosen in the output range to map into gray levels the relative lightness appearane values of eah hannel. Aording to the hosen referene point an additional global balane between gray world and white path is added. The following two saling methods an be used to obtain a standard 24 bit output image from the signed floating point array R. The first simple method sales linearly the range of values in R independently on eah hannel to the range [0,255] by the formula: O p) = round[ s ( R ( p) m )] ( where s is the slope of the segment [(m,0),(m,255)], with M = max[ R ( p)] and m = min[ R ( p )]. p p In this ase the linear mapping fills exatly the available dynami range without further adaptation. 140 Pro. of SPIE Vol. 5008

4 The seond alternative method, yields better results, by saling linearly the values in R with the following formula O ( p) = round[127.5 s R ( p)] + using M as white referene and the zero value in R as an estimate for the medium gray referene point to ompute the slope s. For this reason, the available dynami ould not be used entirely, or tones around the very dark values ould be lost. Alternatively, some values in O an result negative. In this ase the values lower than zero are set to zero. The seond method adds a global gray world adaptation in the final saling, thus the dynami of the final image is always entered around the medium gray. 3. ACE FOR COLOR DIGITAL MOVIE RESTORATION The prinipal harateristi of ACE [6][7][23] is its loal data driven olor orretion, inspired by some adaptation mehanisms of the human visual system. ACE is able to adapt to unknown hromati dominants, to solve the olor onstany problem and to perform an image dynami data driven strething. Moreover, ACE algorithm is unsupervised and needs little involvement from the user. These properties make it suitable for film restoration, a problem in whih there is no referene olor to ompare the results of the filtering. The only reliable riterion is the pleasantness and naturalness of the final image. Faded movie images are drab and have a poor saturation and an overall olor ast. This is due to the bleahing of one or two hromati layers of the film. Sine we deal with lost hromati information, restoring the olor of faded movies is more intriate than balaning the olors of an image with a olor ast due to an illuminant shift. Figure 5 shows the original faded image of figure 4 proessed by ACE. Even though the ast was removed, the olors of the image are still drab. This is due to the fat that bleahed movie images are more damaged than images with a ast due to an illuminant shift. Vividness (saturation) of the image olors is a major issue in digital olor movie restoration. The naturalness of the image and its histogram shape is also important. The new unsupervised tehnique, that we present, is not just an appliation of ACE on movie images, but an enhanement of ACE priniples to meet the requirements in the digital film restoration field. To restore the vividness of the image olors, we enhane the saturation of the real olors of the image before removing the ast and balaning the olors with ACE (see figure 3). To avoid inreasing the olor ast, we use a non-uniform saturation enhanement tehnique presented in [3]. It onsists in strething the bounding ellipsoid of the points aording to the prinipal axes in CIELAB olor spae. Unlike uniform saturation inrease methods, this olor enhanement tehnique avoids inreasing the olor ast all over the image and enhanes the real olors of the image. Figure 6 shows the image of figure 4 after non-uniform saturation enhanement. Saturation enhanement Color balane ACE Original images Restored images Fig. 3: Steps of olor restoration One the olors of the image have been revived, the next step onsists in removing the olor ast, balaning the olors and orreting the ontrast of the image using ACE. Inner parameters of the ACE algorithm have to be properly tuned and new funtions have to be added to meet the requirements of image and histogram shape naturalness. Figure 7 shows Pro. of SPIE Vol

5 the result of ACE on saturation enhaned image of figure 6. The reovered olors are vivid, but the image is noisy and has an unnatural look due to the equalized histogram shape as shown on figure 9.Trying to avoid these unwanted behaviors we followed a onservative approah hoosing 0.2 for saturation parameter as a trade off between a good Color Constany behavior and a solution that doesn t amplify exessively the original low ontrast bakground noise. This preliminary hoie that needs further investigation derives from previous tests [6]. To preserve the natural histogram shape and to obtain natural images two funtions have been added in the ACE algorithm. A new saling funtion alled WP+GW (blak) and a new keep original gray (fade to blak) feature. In partiular the WP+GW (blak) saling is the same as WP+GW but with a modified mapping that fills exatly the available dynami solving the problem of losing dark tones using WP+GW. The keep original gray feature has been devised to relax the GW mehanism in the seond stage: instead of entering the hromati hannels around the medium gray, keep original gray preserve the original mean values; this results in histograms more similar in shape with original ones. Relaxing GW mehanism in the seond stage does not affet the Color Constany property of ACE sine it derives from WP/GW mehanism in the first stage and from global WP approah in the seond one. 4. EXPERIMENTAL RESULTS We tested different parameters to find the best onfiguration for faded images restoration. We run the experiments on several images oming from different films of different epohs and with different bleahing behaviors and olor asts. The riteria used to evaluate the quality of the restoration are visual judgment [8] aording to the naturalness of the aspet of the image (olors and ontrast) and the naturalness of the histogram shape (smooth, regular and strethed shape not an equalized one). To judge the fidelity of the olors of the restored image we fous on memory olors [9] [10] like the blue of the sky, the green of the foliage, skin tones, ahromati zones, et... We varied different parameters like the saling funtion, the shape funtion, and some features like keep original gray. To get a restored image with balaned vivid natural olors without noise, and having a natural histogram shape, the best shape funtion is saturation. Figure 18 shows the result of ACE on the image of figure 6 with linear shape. The image has still a strong ast. Using a saturation shape guarantees the removal of the olor ast. The most suitable parameter of saturation shape funtion is 0,2. Under this value the image beomes noisy, the ontrast is not ideal (see figure 16) and the histogram shape is unnatural as shown on figure 17. The WP+GW saling funtion (see Set. 2.2) does not give an image with a good ontrast espeially in the shadows, as shown in figure 10, sine the shadows are shifted as shown in figure 11. A linear saling gives a more strethed histogram (figure 13), but the result is still not satisfying (figure 12). The best result is given by WP+GW blak saling funtion ombined with the keep original gray feature, sine this onfiguration gives a good ontrast all over the image, whether in shadows, mid-tones or highlights. Figure 14 shows a well restored image using this onfiguration ombined with a saturation shape with a parameter of 0,2. The olors of the image are natural and vivid. The ontrast and the histogram shape are natural as well as illustrated by figure 15. Figure 23 shows another well restored image by the same onfiguration. The polar hue histograms of figures 21 and 25 show the hromati diversity of the images restored by this method whih onfirms the suitability of this tehnique to digital film olor restoration. 5. DISCUSSION The experimental results gave evidene for the suitability of the developed tehnique to restore olor faded movies. This tehnique has several advantages: It uses a pereptual approah with global and loal effets, is unsupervised, needs little involvement from the user and gives good results. The major drawbak of this method is its slowness. The algorithm an run on a loal LUT instead of a full image but the results have not the same reliability. To avoid running the algorithm on all the full images of the sequene, whih takes a huge amount of time (see table 1), it is preferable to run the algorithm on a redued image (at CIF resolution for instane), then do a linear olor mapping [4] between the original faded image and the restored redued image. The linear olor mapping will provide us with the matrix that will restore the whole sequene at full definition. The linear proessing gives a well restored image (figure 19) lose to the target (the image proessed by ACE), but it is not exatly the same sine ACE is a non linear and loal proessing. Alternative aelerating methods are under test. 142 Pro. of SPIE Vol. 5008

6 In the meanwhile, to aelerate the ACE and meet the eonomial requirements in the digital restoration field, we rewrote the algorithm using Intel s SSE instrutions [22]. SSE instrutions are SIMD (Single Instrution Multiple Data) instrutions permitting to proess a set of pixels with a single instrution. The first results are very promising as shown by table 1, the proessing time was divided by nearly 5 (a gain fator of 80%) on a PC equipped with an AMD Athlon XP proessor and 512 of DDRAM. Table 1: Computing time of the original and the SSE optimized ACE algorithm Image size Algorithm Proessing time t t ACE / t ACE SSE Gain fator 925 x 672 ACE 19 hours 5,43 81% pixels ACE w/ SSE 3 hours 30 min 310 x 293 ACE 22 minutes 4,88 80% pixels ACE w/ SSE 4 min 30 se 6. CONCLUSION We presented in this paper a new tehnique for digital restoration of olor faded movies using a new unsupervised olor equalization algorithm, based on a pereptual approah. This work results from the ollaboration between the L3I lab of Université de La Rohelle and the Department of Information Tehnology of University of Milano. The developed tehnique onsists in enhaning the saturation of the images, sine after fading the images beome too drab, then balaning the olors of the image using a new version of ACE algorithm (Automati Color Equalization). To meet the requirements in the digital restoration field the inner parameters of the new version were properly tuned, some new funtions were added and the whole algorithm optimized. The first obtained results are satisfying and promising. The future prospets of this ollaboration onsist in inluding the saturation enhanement proessing inside the ACE algorithm, optimizing and aelerating the algorithm. REFERENCES 1. M. Chambah, B. Besserer, Digital olor restoration of faded motion pitures. Computer graphis and image proessing onferene CGIP2000, Saint-Etienne, Frane, 2000, pp M. Chambah, B. Besserer, Digital restoration of faded olor movies: a four-step method, 8th Color imaging onferene CIC8, IS&T/SID, Sottsdale, AZ, USA, 2000, pp M. Chambah, B. Besserer, P. Courtellemont, Reent Progress in Automati Digital Restoration of Color Motion Pitures, SPIE Eletroni Imaging 2002, San Jose, CA, USA, janvier 2002, vol. 4663, pp M. Chambah, B. Besserer, P. Courtellemont, Approah to Automate Digital Restoration of Faded Color Films, IS&T Conferene on Graphis, Imaging, and Vision CGIV 2002, Poitiers, Frane, avril 2002, pp M. Chambah, B. Besserer, P. Courtellemont, Latest Results in Digital Color Film Restoration, Mahine Graphis and Vision (MG&V) Journal, (in press ) 6. A. Rizzi, C. Gatta, D. Marini, Color Corretion between Gray World and White Path, Eletroni Imaging 2002, 20-25/01/02, San Jose, California (USA). 7. C. Gatta, A. Rizzi, D. Marini, ACE: an Automati Color Equalization algorithm CGIV02 the First European Conferene on Color in Graphis Image and Vision, 2-5/4/2002, Poitiers (Frane). 8. M. Stokes, T. White, Color fidelity test methods, IS&T/SID s olor imaging onferene proeedings, 1998, pp K. Töpfer, R. Cookingham, The quantitative aspets of olor rendering for memory olors, IS&T s PICS onferene, C. Tuijn, W. Cliquet, Today s image apturing needs: going beyond olor management, IS&T/SID s 5th olor imaging onferene proeedings, 1997, pp S.N. Pattanaik, J.A. Ferwerda, M.D. Fairhild, D.P. Greenberg, A Multisale Model of Adaptation and Spatial Vision for Realisti Image Display, Pro. of SIGGRAPH98, pp , Orlando, Florida (USA), July A. C. Hurlbert, Formal onnetions between lightness algorithms, J. Opt. So. A, 1986, 3, M. Ramasubramanian, S.N. Pattanaik, D.P. Greenberg, A Pereptually Based Physial Error Metri for Realisti Image Synthesis Pro. of SIGGRAPH99, pp.73-82, Los Angeles, California (USA), Pro. of SPIE Vol

7 14. J. MCann, Lessons Learned from Mondrians Applied to Real Images and Color Gamuts, IS&T Reporter, Vol. 14, No. 6, November/Deember G.W. Larson, H. Rushmeier, C. Piatko, A Visibility Mathing Tone Reprodution Operator for High Dynami Range Senes, IEEE Transations on Visualization and Computer Graphis, Vol. 3, No. 4, Otober/deember J. Tumblin, H. Rushmeier, Tone Reprodution for Realisti Images, IEEE Computer Graphis and Appliation, November 1993, pp J. Tumblin, J.K. Hodgins, B.K. Guenter, Two Methods for Display of High Contrast Images, ACM Transations on Graphis, Vol. 18, No. 1, January 1999, pp J. MCann, Lessons Learned from Mondrians Applied to Real Images and Color Gamuts, IS&T Reporter, Vol. 14, No. 6, November/Deember D. Marini, A. Rizzi, "A omputational approah to olor illusions", in: A. Del Bimbo Ed., Image Analysis and Proessing, Springer Verlag, Berlin, pp.62-69, Marini D. Rizzi A, Rossi M., Color Constany Measurements for Syntheti Image Generation, Journal of Eletroni Imaging, 8, 9, (1999) pp Marini D., Rizzi A., "A Computationl Approah to Color Adaptation Effets", Image and Vision Computing, 18, 13 (2000) pp Intel Corporation, SIMD extensions in Pentium III, A. Rizzi, C. Gatta, D. Marini, A New Algorithm for Unsupervised Global and Loal Color Corretion, in press on Pattern Reognition Letters. 144 Pro. of SPIE Vol. 5008

8 Fig. 4: Original faded image to restore Fig 5: Image of fig 4 proessed by ACE. Even though the ast was removed, the olors are still drab. This is due to the fat that bleahed movie images are more damaged than images with a ast due to an illuminant shift. Fig 6: Image of fig 4 proessed by our saturation enhanement algorithm. The real olors of the image are revived without inreasing the ast. Pro. of SPIE Vol

9 Fig 7: The result of ACE on saturation enhaned image of fig 6. The reovered olors are vivid, but the image is noisy, has unnatural look due to the equalized histogram shape as shown on fig 9. Fig 8: Histogram of original faded image of fig4. The histogram is not strethed sine the olors of the image are damaged. Fig 9: Histogram of the image of fig 7 proessed by ACE. The histogram is unnatural and equalized. Fig 10: Result of ACE on image of fig 6 with saturation shape (parameter 0,2) and WP+GW saling. The olors are vivid and natural but the ontrast of the image is not good espeially at shadows. Fig 11: Histogram of fig 10. The highlights are equalized and the shadows are far from the edge, that is why the shadows are light. 146 Pro. of SPIE Vol. 5008

10 Fig 12: Result of ACE on image of fig 6 with saturation shape (parameter 0,2) and linear saling. The ontrast of the image is better than the ontrast of the image of fig10. Fig 13: Histogram of image of fig 12. The histogram is more strethed than the histogram on fig 11. That is why the ontrast of image of fig 12 is better than of image of fig 10. Fig 14: Result of ACE on image of fig 6 with saturation (0,2), WP+GW (blak) saling and with keeping original gray. This onfiguration gives one of the best results in terms of vividness of olors, ontrast and naturalness of the histogram. Fig 15: Histogram of fig 14. It is well strethed and has a natural shape whih explains the naturalness of the image of fig 14. Fig 16: Result of ACE on fig 6 with saturation (0,07), WP+GW (blak) and with keeping original gray. When the saturation parameter is too small, the image is noisy, the ontrast is not ideal and the histogram shape is unnatural (fig 17). Fig 17: Histogram of fig 16. It has an unnatural shape whih explains the noise on fig 16. Pro. of SPIE Vol

11 Fig 18: Result of ACE on image of fig 6 with linear shape, WP+GW (blak) saling and with keeping original gray. The image has still a strong ast. This shows the usefulness of a saturation shape. Fig 19: The result obtained by a linear olor mapping between fig 4 and fig 14. The olors and the ontrast of the image are very satisfying but not exatly the same as of fig 14, this is logial sine we approah two non-linear treatments with a linear one. The linear olor mapping permits to find the matrix that an restore the olors of the image. This matrix an be applied to the whole sequene, gaining hene a lot of proessing time. Fig 20: Hue polar histogram of the original faded image of fig4, showing its poor hromati diversity. Fig 21: Hue polar histogram of the image restored of fig 14, showing its good hromati diversity. 148 Pro. of SPIE Vol. 5008

12 Fig 22: Original faded image to restore. Fig 23: Image of fig 22 after saturation enhanement and ACE. Fig 24: Hue polar histogram of the image of fig. 22, showing its poor hromati diversity. Fig 25: Hue polar histogram of the image of fig. 23, showing its good hromati diversity. Pro. of SPIE Vol