Reference Free Image Quality Evaluation for Photos and Digital Film Restoration Majed CHAMBAH Université de Reims Champagne-Ardenne, France 1
Overview Introduction Defects affecting films and Digital film restoration Image quality evaluation Some reference free quality measures Experimental results and discussion Conclusion 2
Introduction The cinematographic archives contain cultural and historical recordings that are assets for the future All kinds of films are subject to damages Often, a degraded release print is the only available record of a film Digital film restoration is a significant hope : Does not affect the original Tackles defects out of reach of photochemical restoration 3
Introduction Progress made in digital restoration algorithms Quality assessment is still subjective Free reference quality measures (dye fading restoration) 4
Defects affecting films Film cycle & degradations Mechanical degradations Chemical degradations affecting base Chemical degradations affecting emulsion Optical reproduction degradations 5
Examples Jitter 6
Examples Dirt dusts 7
Examples Scratches 8
Chemical degradations of the emulsion Contrast saturation: Causes : chemical changes by slow continuation of the photochemical process in improperly processed black and white films or by oxidation under various influences. Effects : strong saturation of the dark and bright areas and severe loss of the middle tones. Color dye fading: Causes : chemical changes induced in the complex chemical composite of the emulsion in color film. It is almost always caused by oxidation under the influence of temperature and humidity changes. The layers of the emulsion are affected in proportion with their proximity with the surface. Effects : distinctive dominant color. 9
Examples Color dye fading 10
Examples Color dye fading 11
Issues in restoration evaluation In the cinema field evaluation is subjective (expert judgment) Often no reference is available Difficulty to characterize precisely the impairments affecting films High definition of images makes defects very visible Spatiotemporal nature of the images to restore Lack of correlation between the metrics and the perceived quality Degradation or artistic distortion? 12
Image Quality Assessment In many fields quality is judged visually (Image enhancement, digital film restoration, ) because : Often, objective quality metrics do not necessarily correlate well with perceived quality Some measures assume that there exists a reference in the form of an original to compare are not applicable on generic images Subjective evaluation is the most used and most efficient approach up to now 13
Image Quality Assessment Subjective evaluation : Expensive, Time consuming Does not respond to the economic requirements (DFR) Reliable automatic methods for visual quality assessment are needed Ideally, a quality assessment system would perceive and measure image or video impairments just like a human being. 14
Objective Quality Evaluation Use of metrics to assess quality Automated, less costing, no user interaction Categories : Full reference, No reference, Reduced reference Full-reference (FR) metrics Direct comparison between the image or video under test and a reference Require the entire reference content to be available MSE, E 15
Objective Quality Evaluation No-reference (NR) or Free reference metrics Look only at the image or video under test No need for reference information Measure the quality of any visual content Distinction between distortions and regular content (humans are able to make from experience and context) Reduced-reference (RR) metrics Midway between FR and NR Extract a number of features from the reference image or video Comparison with the image/video under test is then based only on those features 16
Some reference free metrics For color dye fading restoration evaluation Objective tools : Hue Polar Histogram for judging chromatic diversity Metrics derived from HSV and RGB 17
Indices from Hue Characterizing Hue with mean and standard deviation is inappropriate A correct mathematical framework is circular statistics π 2 Frequency 0 1 2 3 4 5 mean circ. mean π 0 0 1 2 3 4 5 6 data 3π 2 18
Indices from Hue High concentration parameter K always indicates dominating color π Circular mean is then relevant 2 π + 0 K = 130.036470083 µ = -0.002744742 (red) 3π 2 19
Indices from Hue Also works for natural images π 2 π + 0 K = 9.303678 µ = 1.850359 (green) What if no dominating color? 3π 2 20
Indices from Hue Low concentration parameter K may indicate no dominating color π 2 π + 0 K = 2,23626508906874 µ= 0,77068897 3π 2 21
Indices from Hue Low concentration parameter K may indicate no dominating color or several ones π 2 π + 0 K = 1.0446473 3π µ = 0.3930013 2 Need more sophisticated methods to identify several modes 22 (e.g. EM)
Indices from Hue Multi color dominant image Several modes: π j the mixing proportions, (µ j,κ j ) parameters of the j-th mode π Parameter estimate EM algorithm 2 π + 0 3π Π1 = 0,75, Π2 = 0,25, 2 µ1 = 27 deg (orange), µ2 = 216 deg (blue), K1 = 1,82, K2 = 1,84 23
Indices from Hue Multi color dominant image π 2 π + 0 3π 2 Π1 = 0,89, Π2 = 0,11, µ1 = 3 deg (red), µ2 = 202 deg (blue), K1 = 1,66, K2 = 1,46 24
Indices from RGB Principal Component Analysis of an RGB image gives: Three uncorrelated new variables (C1,C2,C3) maximizing projected variance Sum of eigenvalues l1+l2+l3 is the inertia I I = 0,05584405 K= 268,340131 Low inertia is often related to high K => slightly colorful or low saturated image 25
Indices from RGB High I and low K means colorful image I= 0,2940, K =0,6793 I= 0,2726, K is N.A. B&W images have l2=l3=0 26
Indices from RGB Extremely colorful images have high inertia I and low (l1+l2)/i (CEV2) I= 0,2940, CEV2=0,7504 I= 0,2726, CEV2=1 27
Indices from RGB Images well color balanced have their point set around the luminance axis (R=G=B) First eigenvector U1 should have the same property The angle between U1 and (1,1,1) = test the assumption angle = 0,0277 (1.59 ) angle = 0,4126 (23.64 ) 28
Contrast Evaluation Michelson s contrast : Relationship between the darkest and the brightest element in the image Global measure can not account for simultaneous contrast. Same min, max and same mean for background but different perception 29
Contrast Evaluation Local contrast measure : Take into account the neighborhood of each pixel for simultaneous contrast like phenomena sum of the differences between the grayscale of each pixel with its neighbors weighted by the distance of these neighbors. CL( c) = i, j I i j c I c ( i) I c ( d( i, j) Nb( I) j) 30
Contrast Evaluation Uniformity of the image histogram measure measures the difference between a flat histogram and the histogram of the judged image uses Bhattacharyya coefficient (measures the cosine angle between the histogram of judged image and a uniform histogram 1/256 for all grayscales) CB = 1 1 16 i:0..255 p( i) where p( i) = h( i) N 31
Contrast Evaluation Metrics Fig 1.a Fig 1.b Fig 1.c Fig 1.d Fig 1.e CM 0,98 0,98 0,98 0,97 0,97 Mean 140 140 140 112 112 CB 0,46 0,46 0,90 0,92 0,92 CL 1,00 2,26 1,05 7,03 21,08 32
Restoration Evaluation Pi1= 0,90, Pi2=0, Pi3=0,10 µ1= 340 (red), µ3=242 (blue) Angle= 10,87 I (inertia)= 0,10 (low) K = 2,02 CM=0,88 CL=6,45 Pi1= 0,40, Pi2=0,21, Pi3=0,39 µ1= 10 (red), µ2= 154 (green) µ3=201 (blue) Angle= 0,32 (more balanced) I (inertia)= 0,29 (increase) K = 0,17 (decrease) CM=0,94 CL=12,19 33
Restoration Evaluation Pi1= 1, Pi2=0, Pi3=0 µ1= 359 (red), Angle= 2,44 I (inertia)= 0,008 (low) K = 78 (high!) CM=0,73 CL=7,39 Pi1= 0,87, Pi2=0,13, Pi3=0 µ1= 14 (red), µ2= 140 (green) Angle= 2,78 I (inertia)= 0,29 (increase) K = 1,40 (decrease) CM=0,85 CL=10,27 34
Restoration Evaluation Pi1= 0,86, Pi2=0, Pi3=0,14 µ1= 310 (magenta), µ3= 290 (purple) Angle= 3,31 I (inertia)= 0,18 (low) K = 5,27 CM=0,90 CL=9,50 Pi1= 0,58, Pi2=0,01, Pi3=0,40 µ1= 12 (red), µ2= 154 (green), µ3=198 (cyan) Angle= 1 I (inertia)= 0,22 (increase) K = 0,29 (decrease) CM=0,95 CL=10,74 35
DAF Metric Take a model of human visual perception Assume that visual content extraction and visual quality are related From these two assumptions it follows that filtering with a model of HVS goes in the direction of visual quality (reference free) This is confirmed by other experiments 36
Automatic Color Equalization Algorithm for digital images unsupervised enhancement Like our vision system ACE is able to adapt to widely varying lighting conditions Able to extract visual information from the environment efficaciously ACE output is an estimate of our visual perception of a scene ACE enhances images in the way our vision system will perceive them, increases their overall perceived quality ACE output can differ from the input more or less according to the visual quality of the input image DAF metric : Use the difference between the output and input image as a non reference metric 37
Proposed Metric Original Image ACE filtered Image Proposed metric (DAF) f(difference) Original image is NOT modified 38
ACE Overview Chromatic/Spatial Adaptation Dynamic Tone Reproduction Scaling I c R c O c s( ) function local/global balancing subset selection GW WP scaling reference ACE basic scheme I: input image, R : intermediate result O : output image; c : chromatic channel 39
ACE : Chromatic / Spatial Adjustment R c ( p ) = j Im, j p j Im, r ( I j p ( p ) d ( p, r d ( max p, I ( j) j) j)) (I(p) - I(j)) basic pixel contrast interaction mechanism, d(p,j) distance function weights the amount of local or global contribution r() contrast tuning and relative lightness appearance of the pixel 40
ACE : Dynamic Tone Reproduction Scaling Map the intermediate pixels R into the final output image O simple dynamic maximization (linear scaling) reference values can be specified (mean, white) Global balance between gray world and white patch R Histogram O Histogram medium gray point R max 0 127 255 41
ACE Overview An important property of ACE is its quasi-idempotence if we apply ACE again on its own output it does not produce considerable effect the first filtering is responsible for almost all the visual normalization and the model converge to a stable output 42
DAF & Color cast Images having the worse quality have the worst rank according to DAF Images having the best quality are highly ranked according to DAF Rank correlation between DAF rank and visual rank : 0,63 43
DAF & Exposure The smaller distances DAF (differential ACE filtering) belong to the images that are correctly exposed DAF estimates correctly photo exposure Photos with less color cast has the least value of DAF 44
DAF & other metrics Correlation between DAF and some reference free measures: DAF is slightly correlated with these measures (less than 0,25) Computed among all the reference free measures developed and DAF which set of predictors are the best to estimate the visual judgment ratings Procedure of stepwise regression : model based on three variables to predict the visual rating : DAF and two other reference free metrics DAF slightly correlated to developed metrics BUT complementary to these metrics DAF permits to enhance the prediction of the visual quality rating 45
Discussion Reference free metrics very promising Restoration quality evaluation more objective Reliable (to some extent) Speed up evaluation process Metrics can characterize an image sequence before its processing to automatically fine tune the parameters of the restoration techniques But Human validation still needed Still a lot to do 46
Discussion 47
48
Thank you chambah@univ-reims.fr 49