Evaluation and improvement of the workflow of digital imaging of fine art reproductions in museums

Transcription

1 Evaluation and improvement of the workflow of digital imaging of fine art reproductions in museums Thesis Proposal Jun Jiang 01/25/2012 Advisor: Jinwei Gu and Franziska Frey Munsell Color Science Laboratory, Rochester Institute of Technology, Rochester, U.S. ABSTRACT Fine art reproductions refer to the reproductions of fine art paintings in the forms of catalogs, postcards and books, for example, by museums, libraries, archives, and so on. While traditionally reproductions are hard copy prints, such as books and postcards, more and more artworks are digitized in museums to be viewed on the display. For example, artwork collections are made available through museum websites and Google Art Project for art lovers to view on their own displays. Given that the media of reproductions change from hard copy prints (self-reflecting materials) to self-luminous displays, how fine art reproductions are evaluated and perceived may not remain the same. Therefore, the need to ensure the image quality of fine art reproductions viewed as soft copy on the display becomes obvious. Besides, how image quality can be assured when viewed on the display is also of great interest in the printing industry. In the printing industry, soft-proofing allows matching the images shown on the display with the hard copy originals, cutting cost by eliminating the need to make physical hard copies and to deliver them around the world. To ensure the image quality of the fine art reproductions, workflows have been developed and followed to digitize artworks in institutions. Berns and Frey found out that one of the most important reasons to digitize artworks in museums is to make collections accessible over the Internet. 1 Currently the workflows to digitize artworks rely heavily on visual editing and retouching (global and local color adjustments on the digital reproductions) to match the reproduction with the original. Given that the camera spectral sensitivity is not the same as human color sensitivity, the captured artwork off the camera usually do not match well in color with the original perceptually. Visual editing and retouching can be both timeconsuming and subjective in nature (depending on experts own experience and understanding of the artwork), lowering the efficiency of artwork digitization seriously. In addition, the widely varying workflows used by different museums further complicates the problem, making the image quality of the reproductions far from satisfaction. Further author information: Jun Jiang: jxj1770@rit.edu

2 Besides visual editing and retouching, another limitation of current digital workflows is that a metameric match between the original and the reproduction under an intended lighting can be achieved at best. However, the perceptual match in color may not hold once the lighting condition changes. The introduction of spectral imaging of artworks promises matching with the original under almost any light sources. Experiments have been conducted to learn the image quality of reproductions based on different spectral acquisition techniques. 2 However, little research has been focused on comparing the perceived image quality of fine art reproductions based on current digital workflow and those of spectral reproductions. My research is intended (1) to evaluate the perceived image quality of fine art reproductions viewed on the display and on the Internet, (2) to improve the workflow in museums by reducing visual editing and retouching, (3) to recover the spectral reflectance under more commonly available lighting conditions (such as daylight), and (4) to compare the perceived image quality of spectral reproductions with those by current three-channel (RGB) workflows. 1. INTRODUCTION The goals of fine art reproduction are usually to create backups, and to provide easy access to those who usually do not have exposure to the originals. While the requirement on accurate capture of the originals depends on the reproduction purpose (the accuracy in reproducing a calendar may not be as critical as that in making a catalog for auctions, for example), it is generally required that the fine art reproductions should be as close to the original as possible in order to faithfully deliver the artistic and aesthetic feeling. Painting Painting Image Painting Image Capture Visual edit & retouch Display Figure 1. A general workflow to reproduce fine art paintings in museums Print Given the advance in digital imaging, more and more artworks are now being digitized in museums. 90% of imaging was performed digitally in 2004 by the interviewed institutions. 1 A picture of the general components of current digital fine art reproduction is shown in Fig. 1. A painting is first captured by a digital camera under a certain lighting geometry in Fig. 1. The camera is usually set to be perpendicular to the artwork, and the lighting can be diffuse, directional, or a combination of both. The lighting setup may also depend on the artwork. For example, if the painting has a lot of texture, a diffuse lighting is likely to soften the reproduction by losing details on the edges in the painting. The captured picture

3 off the camera is then shown on the display for visual editing and retouching by experts in the institutions. Visual editing and retouching refer to the global and local color adjustments on the digitized artwork shown on the display to match more closely with the original. While not absolutely necessary, visual editing and retouching are usually conducted by a lot of museums to improve the image quality, as the camera spectral sensitivity do not match closely with our color sensitivity. Once visual editing is complete, the adjusted images are either accessed by visitors on the display or printed to make hard copies. Evaluation of perceived image quality Controlled environment Web-based Fine art reproductions Visual editing and retouching Evaluate current CAT models Derive a perceptually-based CAT model Spectral imaging Recover scene reflectance under commonly available light sources Evaluate the performance of the spectral reproductions Figure 2. Main components of the proposed research While most museums follow these general steps in Fig. 1 when digitizing artworks, the workflows employed by museums may differ from each other in one way or another. Berns and Franziska 1 found out that the current workflows to reproduce the fine art paintings vary widely, resulting in the perceived image quality by different museums far from

4 satisfaction. For example, the use of diffuse or directional lighting to capture the painting is likely to give the reproduction a different look. There are three main problems in the current workflow of fine art reproductions. (1) The color information in the digital reproduction is contaminated by the lighting, under which the picture is taken. As a result, a metameric match between the reproduction and the original artwork can be achieved at best. However, the perceptual match is likely to suffer once the lighting changes. (2) While used widely in the workflow of fine art reproductions to improve the image quality, visual editing and retouching are labor-intensive and time-consuming, as they have to be conducted for each painting individually. (3) The current workflow is to reproduce the fine art paintings as accurately as possible. However, when digitized artworks are appreciated by observers on their own displays through the Internet without the presence of the original, it becomes questionable whether reproduction accuracy is still the top concern. In our research, we try to understand the problem and propose our method to improve the workflow of fine art reproductions. The main components of this research are shown in Fig. 2 and summarized as follows, Evaluation of the perceived image quality of fine art reproductions We evaluated the perceived image quality of fine art reproductions based on current workflows in institutions under controlled/ uncontrolled environment with and without the presence of the original. We found color accuracy significant to the perceived image quality only when the reproductions are evaluated with the original present. On the other hand, when the original is absent, the perceived image quality is invariant to the viewing conditions. Therefore, the requirement on test conditions may be loosened when image quality is determined based on preference (without the original). Evaluation of CATs by predicting adjustments by observers We evaluated the performance of the state-of-the-art chromatic adaptation transforms (CATs) by predicting adjustments by observers on fine art reproductions. The CAT that provides the closest match with the adjustments by observers can be used as a closer starting point before any adjustments are made by observers. After analyzing the image difference between the predictions by CATs and adjusted images by observers, no CAT is found to outperform the rest for all test images. Based on the experimental data, a perceptually-based CAT model can be derived to serve a closer starting point by matching the prediction of the model more closely with the adjustments by observers. Recovery of the spectral reflectance under commonly available lighting conditions We proposed a spectral imaging method to recover the scene reflectance under commonly available light sources, and applied the method to recover the spectral reflectance of the painting. The wide variety of light sources that are found in everyday life allows recovery of the scene reflectance without extra filters or controlled illuminations. In addition, we are able to tell (1) which lighting condition overall yields more accurate spectral recovery, and (2) given a certain lighting, which reflectance is more likely to be estimated well through the noise analysis. Evaluations of the performance of spectral fine art reproductions We are going to examine the performance of the spectral reproductions of fine art paintings by comparing them with the reproductions based on current digital

5 workflows. The spectral reproduction has the advantage of closely matching with the original under almost any light sources. However, a closer spectral match does not always ensure a closer perceptual match in color, as our visual system is not equally sensitive to the whole visible wavelength range. A mismatch in the wavelength region that our eyes are more sensitive to may introduce noticeable color shifts despite of close spectral match at other wavelength bands. The results of the experiment can be indicative of whether workflows in reproducing fine art paintings should be spectra-oriented instead of solely relying on the red, green and blue channels. 2. RESEARCH BACKGROUND Visual editing & retouching Image quality assessment Fine art reproduction Spectral Imaging Chromatic adaptation Figure 3. Related areas of fine art reproduction My thesis work is related to image quality assessment, chromatic adaptation, spectral imaging, and visual editing and retouching as shown in Fig. 3. Image quality is usually assessed by human observers in psychophysics experiments. The assessment results, on the other hand, can be used to derive models to better correlate with our perceptual experience. While being time-consuming and labor-intensive, visual editing and retouching are widely used in current digital workflows of fine art reproductions to match the reproduction with the original more closely in color. A more perceptually-based chromatic adaptation model is derived to reduce the amount of visual editing and retouching. In addition, spectral imaging on artworks is likely to overcome the limitation of a metameric match between the original and the reproduction under a single lighting condition by recovering the spectra of the artwork. Evaluation of the image quality of fine art reproductions on the display: Image quality assessment plays an important role in various image processing applications. 3 Berns and Frey 1 investigated the direct digital capture of cultural heritage

6 in museums to reproduce fine art paintings. Susan et al. 4 conducted experiments to evaluate the image quality of fine art reproductions, and suggested frameworks to improve the art image interchange cycles. Perceived image quality are usually evaluated with the presence of the original (if the original is available). Color fidelity, among others, is usually highly correlated with the image quality. However, it is generally the situation that people experience fine art reproductions without the original artwork available for direct comparisons. 5 Assessment of image quality with no reference is usually made on distorted, blurred, compressed images, or images with different levels of noise. 6 8 The fine art reproductions are neither distorted nor compressed by any means, despite of being made by widely varying workflows by institutions. Furthermore, most art lovers view the reproductions on their own display through the Internet. Whether the perceived image quality of fine art reproductions viewed in uncontrolled viewing conditions can be assured has not been investigated throughly. Moroney 9 conducted a web-based color naming experiment, in which observers were asked to name the colors on the website through the Internet. As the displays used by participants of the experiment might vary widely, the color to be named became dependent on the color reproduction ability of the display. Evaluation and optimization of chromatic adaptation transforms (CATs) to predict adjustments by observers: Visual editing and retouching, defined as the global and local color adjustments on the reproduction to match with the original, are widely used to improve the image quality of fine art reproductions in museums. Berns and Frey found out that most museums included some visual editing and other forms of image processing in their workflow. 1 Visual editing and retouching are usually performed on each individual painting, significantly lowering the efficiency and automation of the reproduction workflow. Berns, et al 10 used the spectral imaging technique to reproduce paintings, achieving the level of accuracy exceeding that achieved by museums and libraries, even following global and local image editing. However, such approach requires capturing the artwork using the spectral imaging workflow instead of the current digital workflows employed by museums. The replacement of workflows may be out of reach for institutions short of an investment on the multi-spectral camera system. On the other hand, given that the default white point (D65) of the ICC profile (AdobeRGB1998, for example) embedded in the digitized artwork differs from that of the display white point (D50) used for color adjustments and proofing in museums, a chromatic adaptation transform (CAT) is usually needed to predict the color appearance under the second viewing condition. Chromatic adaptation refers to the largely independent sensitivity regulation of the mechanisms of color vision, and it is often considered to be independent changes in responsivity of the three types of cone photoreceptors. 11 Susstrunk and Finlayson 12 evaluated and derived CAT models on corresponding color dataset. Braun and Fairchild 13 tested five color appearance models using complex images. However, one limitation of the previous work is that the CATs were usually tested or derived from simple color patches. Therefore, CATs may not work well with complex images, as the complex and simple images are not perceived and processed the same way by our visual system. Secondly, even when complex images were used to evaluate or derive CATs, they were usually photographs rather than artworks. The colors in artworks can be very different from what are observed in natural scenes. A CAT model dedicated to fine art reproduction may better correlate with our perceptual experience when viewing artworks.

7 Recovery of the spectral reflectance: Spectral recovery refers to the recovery of the spectral reflectance of the scene. Marimont and Wandell 14 proposed a linear model of surface and illuminant spectra. Maloney and Wandell, 15 and Ohta and Hayashi 16 described a method to recover both the scene reflectance and illuminant spectrum simultaneously under the daylight, but the camera spectral sensitivity needed to be known. Recent works 17, 18 also use assorted pixel image sensors that trade spatial resolution for spectral recovery. Spectral imaging with filters Many previous work on spectral imaging are implemented with multiple filters, either mounted in front of the camera lens or use Liquid Crystal Tunable Filter (LCTF). While accurate, such systems are usually costly to build, as customizations have to be made to install filters in the camera. In addition, the change in filters mechanically may even introduce pixel shifts, requiring image registration later on. Spectral imaging with controlled illumination A multispectral imaging system is built to recover the spectral reflectance by using optimal multiplexed LEDs as lightings. 19 In another system, the color wheel in the DLP projector is used to produce distinct lights, simulating multiple light sources. 20 In both systems, the lightings need to be carefully designed and controlled to function in an indoor environment without strong ambient light. Besides, in, 19, 20 the spectral sensitivity of the camera and the spectral power distribution (SPD) of the light sources need to be either measured or estimated in advance. The measurement of the camera spectral sensitivity can be rather time-consuming, and the measurement instruments are expensive and rarely found outside universities or research centers. In addition, Ohta and Hayashi 16 proposed to estimate both the surface reflectance and light source spectrum simultaneously, but it is limited to the daylight, and the camera spectral sensitivity needs to be known in advance. Evaluation of the spectral fine art reproductions: Spectral reproduction of fine art paintings has the advantage of matching with the original under almost any lighting conditions, while the color matching between the original and the digitized artwork by current workflows tend to break once the lighting changes. Day 21 investigated the spatial image quality and color accuracy of reproductions by different spectral recovery methods. Sun 22 evaluated the image quality of spectral imaging on human portraits. While the spectral reproduction is likely to match with the original under different lighting conditions, its accuracy is limited by the noise in the imaging system, the surface property of the object, and the reflectance model, for example. Recent interest in matching the spectra of artworks in museums 10, has raised question on whether a spectral reproduction of fine art paintings is more likely to be appreciated than that based on current workflows in museums, as minimizing spectral error does not guarantee most accurate color reproduction, except when the spectral error approaches zero. 22

8 3. OUR APPROACH The goal of the thesis is to evaluate and improve the workflow in digital imaging of fine art reproductions in museums. To achieve this goal, the thesis work adopts an approach that includes the following components: (1) evaluation of the perceived image quality of fine art reproductions under different conditions, (2) evaluation and optimization of CAT models to reduce visual editing by finding a closer starting point, (3) recovery of the spectral reflectance of the paintings under commonly available lighting conditions, and (4) evaluation of the spectral reproductions by comparing with reproductions by current workflows. Evaluation of the perceived image quality of fine art reproductions in different viewing conditions: Psychophysics experiments were conducted to evaluate the perceived image quality of fine art reproductions shown on the display by observers under different conditions: (1) with the originals in a controlled environment, (2) without the originals in a controlled environment, and (3) without the originals through the Internet (an uncontrolled environment). An abridged paired-comparison method was used to rank the reproductions of fine art paintings collected from a number of institutions. The ranking data under these conditions were compared and contrasted to learn the correlation. In addition, reflectance at selected areas were measured on both the reproductions and originals to investigate the impact of the reproduction accuracy on the perceived image quality. Evaluation and optimization of CATs to predict adjustments by observers on fine art reproductions: Visual editing and retouching of digitized artworks are one of the most time-consuming procedures in the workflow of fine art reproductions. Berns and Frey 1 found out that visual editing and retouching of images off the camera significantly add to the total time required to archive the digital master. Attempts to improve the workflow by reducing visual editing and retouching of the reproduction is needed. Our method includes three steps: We developed a software that allowed global and local adjustments of fine art reproductions by observers to match with the originals in the light booth We predicted the adjusted images by observers by output from three state-of-art CATs We will derive a perceptually-based CAT model, whose output will match more closely with the adjusted images by observers Spectral imaging of fine art reproductions: The spectral reproductions of fine art paintings allow color matching with the originals closely under almost any light sources. We proposed a method to recover the scene reflectance under commonly available light sources (daylight, for example). Based on our method, scene reflectance can be recovered without modifying cameras by installing extra filters or designing lighting. Our method includes three main components: We recovered the spectral reflectance of the scene under commonly available light sources

9 We analyzed the noise in the system to understand (1) which lighting is optimal, and (2) given a lighting, which reflectance is likely to be estimated well or not We applied the method to recover the spectra of fine art reproductions Evaluation of spectral fine art reproductions: The goal of the project is to fully evaluate the performance of the spectral reproductions by comparing with reproductions based on current digital three-channel (RGB) workflows. Our method will include three components mainly: We will capture the paintings under different light sources and used as ground truth We will conduct psychophysics experiments, in which spectral reproductions will be compared with those made by current workflows in museums. The experiment will be run in both controlled environment and through the Internet We will compare and contrast the ranking of spectral reproduction with that based on current workflows to investigate any significant difference 4. EXISTING RESULTS 4.1 Evaluation of the perceived image quality of fine art reproductions A project to evaluate the perceived image quality of fine art reproductions was conducted, and the result is published here. 27 The goal was to understand how people appreciate fine art reproductions shown on the display (1) with the original, (2) without the original, and (3) through the Internet. When the original was present in the light booth, observers could always look back and forth when determining the image quality of the reproductions. On the other hand, observers usually do not have the originals at their disposal when viewing the artwork reproductions. It was of interest to understand what might contribute to their appreciation of the reproductions without the presence of the original. Furthermore, given that more and more artwork images are available online by museums, how reproductions are perceived in an uncontrolled environment was investigated. An abridged paired-comparison experiment was conducted to rank the reproductions collected from sixteen museums. During the experiment, test images were shown on a characterized display in Munsell Color Science Laboratory (MCSL) to ensure accurate color reproduction of the display. On the other hand, to understand how images were perceived in an uncontrolled environment, a web-based ranking experiment was designed as shown in Fig. 4 to evaluate the image quality. Observers could participate the web-based experiment from almost anywhere with little constraints on test conditions, as long as reasonable Internet speed and a web browser were available. Based on the experimental data, we found out that (1) color accuracy in fine art reproductions is highly correlated with the perceived image quality when the originals are present, (2) little correlation is observed between color accuracy and the perceived image quality when the originals are not available, (3) the ranking result from the web-based experiment is

10 (a) welcome Interface (b) paired-comparison Interface Figure 4. The Web-based ranking experiment (a) The welcome interface. (b) The paired-comparison interface. z score w/ Color difference Figure 5. The correlation between the perceived image quality of aquatint and the color fidelity of the reproductions in the controlled experiment w/ the presence of the original. The x-axis is the color difference between the original and the reproductions, and the y-axis is the z-score converted from the ranking by observers. The greater the color difference, the worse the image was ranked. highly correlated with the results from the experiment without the original, and (4) the areas on the painting that determine the image quality are mostly object-oriented. When a human figure is in the painting, the skin tone is of great importance to be reproduced correctly. An example is shown in Fig. 5 to show the high correlation between color accuracy in fine art reproductions and the perceived image quality when the originals were present. In Fig. 5, the greater the color difference between the original and the reproductions of aquatint, the worse the image was ranked (with the original). Similar relationship was found for other paintings. On the other hand, little correlation was found between color accuracy and the image quality assessed when the originals were not available in the experiment, indicating a shift in the criterion employed by observers to evaluate

11 the perceived image quality from color accuracy to preference once the original is not available. Besides, the ranking result from the web-based experiment was highly correlated with the results from the experiment without the original, indicating that the preference judgments of perceived image quality were invariant to viewing conditions. An example ( firelight ) is shown in Fig. 6. In Fig. 6, a high correlation was observed between the ranking results z score web z score web z score w/ (a) Firelight (web vs. w/) z score w/o (b) Firelight (web vs. w/o) Figure 6. The correlation between the perceived image quality of firelight in the web-based experiment and that in the controlled experiment w/ and w/o the original. The ranking result in the web-based experiment was highly correlated with that in the controlled experiment w/o the original. However, when image quality is evaluated w/ the original, the ranking was irrelevant to that in the web-based experiment. in the web-based experiment and that in the controlled experiment w/o the original. However, when image quality is evaluated w/ the original, such correlation did not exist. Besides, given that the originals were absent in both experiments (the experiment w/o the original, and the online experiment), a similar criterion (preference rather than color accuracy) was used by observers to evaluate the reproductions. In addition, the areas that were considered most important by observers to evaluate the reproductions in the online experiment were identified by observers clicks on the preferred image. By understanding the part of the paintings to which more attention was drawn, information regarding the image saliency could be learned. In Fig. 7 (a) and (b), user clicks were overlaid on the painting, indicating the areas that help observers determine the perceived image quality. The clicks are too scattered to be random. Besides, in Fig. 7 (b), a high density of clicks can be found on the lady s cheek, indicating the strong impact of skin tone reproduction on the perceived image quality when a human figure is in the painting. 4.2 Evaluation of CATs by predicting adjustments by observers The goal of the project is to see how people working in museums, libraries, and archives make color adjustments to artworks presented to them on the display, and to find a closer starting point before any adjustments are made by comparing adjusted images by observers with the output of different chromatic adaptation transforms (CATs). We published our result here. 28

12 (a) Bridge (b) Firelight Figure 7. The areas that were clicked by observers to indicate what contributed most to their ranking decisions in the web-based ranking experiment. (a) and (b) The user clicks overlaid on bridge and firelight. When investigating the workflow to reproduce fine art paintings in museums, visual editing and retouching are among the most labor-intensive and time-consuming part of the process. It is therefore of interest to reduce the visual editing or retouching in the reproduction workflow. Given that the display white point in museums (CIE D50) is different from the default white point in the embedded ICC profiles (CIE D65) in the captured digital reproduction, a CAT is needed to simulate the image appearance under D50. In the experiment, the CAT was left undone intentionally, and observers were asked to adjust the pictures on the display to match with the hard copy original in a D50 light booth. In Fig. 8, the adjustment operations by observers are shown on the left side of the seperator, and the predictions by the CAT models are on the right. Three CATs, Bradford, 29, 30 Fairchild92 11 and CAT02 31 were selected to predict adjustments by observers. Bradford transformation is essentially a von Kries transformation with an additional exponential nonlinearity on the blue channel. 11 In the experiment, the linearized Bradford transformation is included, given that it is the default chromatic adaptation in the latest ICC profile specification (ICC Version ). Fairchild92 and CAT02 were more advanced CATs by accounting for incomplete chromatic adaptation across different media. An example is given in Fig.9. The initial image shown on the display before adjustments is in Fig. 9 (a). It appears bluish, because no chromatic adaption transform was made from the default white point of the embedded ICC profile of the image, D65, to the display white point, D50. The adjusted image by one observer is shown in Fig. 9 (b) as an example, and it matches the color appearance of the predictions by the three models more closely than the initial starting image in Fig. 9 (a). The predictions by the Bradford, Fairchild92, and CAT02 models are shown in Fig. 9 (c), (d) and (e) respectively. A software was developed to allow visual editing (color adjustments on a global scale) and retouching (color adjustments on a local scale) as shown in Fig. 10. The experimental setup is shown in Fig. 10 (a), and observers were instructed to adjust the image on the display to match with the original in the D50 light booth. The overall hue of the image can be

13 Image Preparation Source image (srgb, AdobeRGB) Bradford Display Characterization XYZ Fairchild92 Display RGB CAT02 CAT Color adjustments by observers XYZ XYZ (target) LAB LCH XYZ (adjusted) Image Difference Figure 8. The workflow of the experiment to predict adjustments by observers by output from CAT models. adjusted in Fig. 10 (b). The eight surrounding images are of different hue to the image in the center in Fig. 10 (b), and the increase in chroma from the center image can be set by the slider on the user interface as well. The images of different hue are generated by Eq. 1 to 3. a in = (a + 128)/256 (1) a = (a in) 1/γ (2) a out = a (3) where a is the redness-greenness of a pixel ranging from [ 128, 128]. The shift of a by Eq. 1 is to ensure that the gamma correction by Eq. 2 takes non-negative values. γ is determined by the increase in chroma from the center image. Eq. 1 to 3 also apply to b as well. By transforming the images from XYZ to LAB color space and making adjustments on a and b, images of different hue can be created as shown in Fig. 10 (b). The brightness and chroma of the image can be adjusted in Fig. 10 (c) using gamma correction similar to that in Eq. 2.

14 (a) initial image (b) adjusted image by one observer (c) Bradford (d) Fairchild92 (e) CAT02 Figure 9. The prediction of adjustments by observers on daisy by different CAT models. (a) The initial image shown on the display before adjustments. (b) The adjusted image by observers as an example. (c) The prediction by Bradford transform. (d) The prediction by Fairchild92 model. (e) The prediction by CAT02 model. Sigmoid function is used to adjust image contrast. Image sharpness can also be adjusted in Fig. 10 (c) to blur or sharpen the images, and it is implemented using Unsharp Mask (USM) by Eq. 4 and 5. mask = img original img blur (4) img sharpened = img original + α mask (5) A blurred version of the image (img blur ) is generated first to create the mask by Eq. 4, and α in Eq. 5 is set to be 1 in the experiment. Local adjustments can be performed on certain colors without affecting other colors in Fig. 10 (d). The changes that can be made locally to the colors include brightness, chroma, contrast and hue. The color of interest is first selected by a mouse click by observers, and the selected colors are adjusted similarly using the equations above except that the changes in color only apply to the selected colors. To evaluate the performance of the CAT models, image difference is calculated between the predicted and adjusted images by observers. Howeer, directly calculating the color difference is likely to artificially enlarge the perceptual image difference, because some details in the image cannot be differentiated by our eyes at a certain viewing distance. Therefore, a spatial extension to the CIELAB system, S-CIELAB, 32 was used to remove the details that cannot be detected by our eyes due to spatial frequency reasons. An input image was initially converted into one luminance and two chrominance

15 (a) Lab setup (b) Hue adjustments (c) Global adjustments (d) Local adjustments Figure 10. The experimental setup and the software interfaces (a) The experimental setup. (b) The hue adjustments interface. (c) The global adjustment interface. (d) The local adjustment interface. color components. Each component image was then passed through a spatial filter that was selected according to the spatial sensitivity of the human eye for that color component. The final filtered images were transformed into XYZ format so that the color difference equation could be applied. 33 To analyze the image difference, analysis of variance (ANOVA) 34 was used in Minitab R. The result is shown in Fig. 11. Based on the image difference (CIEDE ) between the predicted images by the CAT models and the adjusted images by observers filtered by the S-CIELAB model, no model was found to be a clear winner for all six test images. Overall Fairchild92 model outperformed the Brandford and CAT02 model for all the test images except those with neutral appearance ( photo and aquatint in Fig. 11 (a) and (f)). 4.3 Spectral imaging of fine art reproductions Recovering the spectral reflectance of a painting is important in faithful reproduction of artworks under different lighting conditions. While current digital workflow is able to match the fine art reproduction with the original under an intended lighting, the color match is vulnerable to changes in the lighting conditions. On the other hand, spectral reproduction of the painting has the benefit of a close perceptual match under almost any lighting conditions. Previous approaches in spectral recovery use either specialized filters or controlled illumination where the extra hardware prevents many practical applications. We propose a method that accurately recovers spectral reflectance from two images taken with conventional consumer cameras under commonly available lighting conditions, such as daylight at different times over a day, camera flash and ambient light, and fluorescent and tungsten light. The result can be found here. 36 Our approach does not require camera spectral sensitivities or the spectra of the illumination, which makes it easy to implement for a variety of practical applications. Based on noise analysis, we also derive theoretical predictors that

16 (a) Group 1 (b) Group 2 Figure 11. Evaluation of the performance of CAT models by predicting adjusted images by observers. Two groups of observers were asked to adjust images on the display to match with the originals in the light booth. (a) and (b) The image difference between the prediction by CAT models and the adjusted images by observers. In both groups, no model is found to match with the adjustments by observers for all test images. answer: (1) which two lighting conditions lead to the most accurate spectral recovery overall, and (2) for two given lighting conditions, which spectral reflectance is more likely to be estimated accurately. We implement the method on a variety of cameras from high-end DSLRs to cellphone cameras. The spectral reproduction of a painting daisy by Canon 60D under the fluorescent light and tungsten light is shown in Fig. 12 as an example. In Fig. 12, a close spectral match could be achieved overall between our results and the ground truth at selected areas on the painting. Besides, the colorimetric error under both CIE D65 and IllA were calculated, and they were reasonably small. The rendering of the painting was made under CIE D65 and IllA in Fig. 12 (c) and (d). 5. PROPOSED RESEARCH Based on our existing results, we plan to focus on two projects to achieve the goal of the thesis. First, a CAT model will be optimized to serve as a closer starting point before any visual editing and retouching. The optimized CAT model should be able to better predict the adjustments by observers in the experiment. Second, psychophysics experiments will be conducted to evaluate the performance of the spectral reproductions by comparing with reproductions based on digital three-channel (RGB) cameras. 5.1 Toward a perceptually-based computational color constancy CAT model Given that the predictions of the state-of-art CAT models did not match with adjustments by observers very well, 28 it is of interest to see whether a more percpetually-based model can be derived based on the experimental data to match more closely with the adjusted images by observers. We propose a two-stage computational model as shown in Fig. 13. The first part of the model is a global operation to predict the change in color (hue, lightness and chroma) made to the image overall by optimizing a CAT based on the experimental data. The second part is to make fine adjustments locally to

17 captured images rendered images P Q R (a) fluorescent light (b) tungsten light (c) CIE D65 (d) CIE IllA reflectance ground truth our result (e) P (f) Q (g) R Figure 12. The spectral recovery and rendering of the oil painting daisy. (a) and (b) The captured images under the fluorescent light and the tungsten light. (c) and (d) The rendered image under CIE D65 and IllA. (e), (f) and (g) The reflectance estimated and measured at selected areas (P, Q and R) on the painting. Global Ajustment CAT Local Adjustment Spatial (x,y) Color (XYZ, LCH,...) Predicted Image by the model Computational model Adjusted Image by observers Image Difference Figure 13. The components of a two-stage computational model certain colors or areas on the image. The reason to include both is that global operation alone may not be sufficient to take into account all the adjustments made by observers to match with the original hard copy in the light booth. The output of the model will be compared with the adjusted images by observers in the previous experiment to validate the performance of the model. A smaller image difference between the adjusted and predicted images are expected.

18 Modern CAT models are generally based on the Von Kries form by Eq. 6, X d Y d Z d = M 1 A M M A X s Y s Z s (6) where M A is used to convert from XYZ color space to a some other spaces, relative cone responses (LMS) space, for example, and M is a diagonal matrix and it can be expressed as L w,d L w,s 0 0 M M = 0 w,d M w,s ] T ] T ] T [ ] T, where [L w,d M w,d S w,d = MA [X w,d Y w,d Z w,d and [L w,s M w,s S w,s = MA X w,s Y w,s Z w,s ] T [ ] T and [X w,d Y w,d Z w,d and X w,s Y w,s Z w,s are the target and source tristimulus value of the white point. Equation 6 and 7 are a mathematical representation of von Kries statement that each is fatigued or adapted exclusively according to its own function. 11 A closer match between the predictions by the model and adjustments by observers can be achieved by looking for a better M. A nonlinear optimization procedure can be used to minimize the image difference between the adjusted images by observers and the predictions by the optimized CAT model using M. In order not to overfit the data, cross validation will be used. Some but not all the test images are used to fit the model, and the rest will be used to test its performance. In addition to predicting global color adjustments by optimizing a M matrix, we propose to model the retouching (local adjustments made to the reproductions by experts) operation by making changes specific to certain colors or spatial locations in the image. To model the spatially local adjustments, a matrix M s can be derived based on the experimental data: where M s performed by Eq. 9, X d Y d Z d S w,d S w,s = M 1 A M s M A = f(x, y), a function of pixel spatial location in the image. Local adjustments on certain colors can be X d Y d Z d = M 1 A M c M A where M c = f(x, Y, Z), a function of color, [X, Y, Z]. Image difference will be calculated between the predicted images by our model with the adjusted images by observers to evaluate the model performance. S-CIELAB model will be used to filter out details that cannot be perceived in the image at a certain viewing distance before color difference (CIEDE00) is calculated. X s Y s Z s X s Y s Z s (7) (8) (9)

19 5.2 Evaluation of the spectral imaging workflow for fine art reproductions While current fine art reproduction workflows in museums can achieve a metameric match between the original and the reproduction under an intended lighting at best, spectral imaging of paintings is likely to match with the original under almost any lighting condition. Psychophysics experiments are to be conducted to compare the perceived image quality of the spectral reproductions of fine art paintings with reproductions by current workflows in museums. The experiments will be conducted (1) in a controlled environment with the presence of the original and (2) in an uncontrolled environment (through the Internet). When running the experiment in a controlled environment, a characterized display and a controlled viewing condition will be used to ensure accurate color reproduction of images shown on the display. Paired-comparison will be used and the software interface is shown in Fig.14. In Fig. 14, the spectral reproduction of the painting will be shown along with Figure 14. Experimental Setup. reproductions based on current workflows collected from a number of museums. Observers will be instructed to choose the reproduction that is a better representation of the original in the light booth. Test images will be selected to vary widely in both color and content as shown in Fig. 15. A 30-inch Apple Cinema Display (in Fig. 14) will be used for showing softcopy reproductions, and an LMT 1210 colorimeter will be used to characterize the display as shown in Fig. 16. The display characterization model proposed and detailed by Day, Taplin and Berns37 is used to ensure accurate mappings between LCD digital counts and XYZ tristimulus values. Display white point and luminance are adjusted to match with those of the light booth by using a Halon perfect reflecting diffuser (PRD). Additionally, the luminance and chromaticity of the background of the light booth are measured using a PhotoResearch-650 spectroradiometer. The background of the software interface will be adjusted to match these settings. On the other hand, when reproductions are evaluated through the Internet, participants are allowed to access the images on a website with little constraints on test conditions. As observers evaluate the image quality from their own displays and

20 (a) aquatint (e) mountain (b) bridge (c) firelight (d) daisy (f) photo (g) orchid (h) night sky Figure 15. Test images. Figure 16. Display characterization. viewing conditions, the rendering of the images becomes dependent on the color reproduction capability of the display as well. In addition, in the web-based experiment, the spectral reproduction will be rendered under different light sources, and compared with the reproduction by current workflows after chromatic adaptation transform. The flowchart is shown in Fig. 17 for one light source as an example, and the main components are identified: The artwork is captured under the light source and used as reference

21 The reproductions based on current workflows by N institutions are processed through the CAT, given a change in white point The spectral reproduction is rendered under the light source The image quality of the spectral reproduction and reproductions by current workflows by N institutions is assessed Artwork Reproduction by institution #1... Reproduction by institution #N Spectral reproduction capture CAT CAT Render light source reference current #1 current #N spectral Image quality assessment Figure 17. workflow to evaluate the spectral reproduction The user interface for the online experiment is shown in Fig. 18 with the reference image in the middle. The other two pictures in Fig. 18 are the spectral reproduction and that based on the current workflow by one of the institutions. The order of the two pictures is randomized so that observers will have little idea of which image is the spectral reproduction. Experts in the area of fine art reproductions, such as curators, photographers, and archivist will be invited to participate the experiment because of their expertise in the art and art history, and their sensitivity to subtle changes in color. On the other hand, the final judge of the reproductions is not limited to expert viewers. Therefore, it is also of great importance to understand whether the improvement of the spectral reproduction, if there is any, is noticeable to a much wider audience. A second group of observers composed of non-experts, students and professors in Rochester Institute of Technology (RIT), will attend the psychophysics experiment. The experimental data will be analyzed to understand the perceived image quality of reproductions by spectral and current workflows. The results by the two groups of observers will be compared and contrasted to learn the impact of expertise in the art and artwork overall on the perceived image quality.

22 Figure 18. web interface to evaluate the spectral reproduction. 6. TIME TABLE Part of my thesis work has been completed so far. There are a few projects I hope to continue in my future work: Toward a perceptually-based computational color constancy CAT model A perceptually-based CAT model is to be optimized to have the output match with the adjusted images by observers closely. Two components are included in the model to take care of the global and local adjustments by observers. Cross validation can be used to optimize the model using some test images, and the rest will be used to evaluate the model performance. Evaluation of the spectral imaging workflow for fine art reproductions Psychophysics experiments are to be conducted to evaluate the spectral reproductions of fine art paintings by comparing with reproductions by current workflows in museums. The experiment will be run in a controlled environment w/ the original, and through the Internet. Table 1 shows my plan for completion of the research. Thus, I plan to defend my thesis in July, 2013.

23 Table 1. Plan for completion of my research REFERENCES 1. M. R. R. Roy S. Berns, Franziska S. Frey, E. P. Smoyer, and L. A. Taplin, Direct digital capture of cultural heritage benchmarking american museum practices and defining future needs, tech. rep., Rochester Institute of Technology, E. A. Day, R. S. Berns, L. A. Taplin, and F. H. Imai, A psychophysical experiment evaluating the color and spatial image quality of several multispectral image capture techniques, Journal of Imaging Science And Technology, vol. 48, no. 2, pp , Z. Wang, A. C. Bovik, and L. Lu, Why is image quality assessment so difficult?, in Proc. IEEE Int. Conf. Acoust., Speech, and Signal Processing, pp , S. Farnand, F. Frey, and E. Anderson, Benchmarking art image interchange cycles: Image quality experimentation, tech. rep., Rochester Institute of Technology, S. Farnand, J. Jiang, and F. Frey, Comparing hardcopy and softcopy results in the study of the impact of the workflow on perceived reproduction quality of fine art images, in Image Quality and System Performance VIII, SPIE-IS&T, vol. 7867, Z. Wang, H. Sheikh, and A. Bovik, No-reference perceptual quality assessment of JPEG compressed images, in Proc. International Conference on mage Processing, vol. 1, pp. I 477, IEEE, 2002.