Color Contrast Enhancement for Visually Impaired people

Transcription

1 Color Contrast Enhancement for Visually Impaired people Anustup Choudhury and Gérard Medioni University of Southern California Department of Computer Science Los Angeles, California USA Abstract We propose an automatic color contrast enhancement algorithm that improves the visual quality of static images for both normally sighted people and for low-vision patients. Existing methods have been shown to work on people either with normal vision or with low vision. Due to the framework of this approach, an enhanced visual experience can be simultaneously provided to normally sighted people and low vision patients. This method is inspired from human color perception and separates the image into illumination and reflectance components. It then enhances only the illumination component while trying to achieve color constancy thus resulting in color contrast enhancement. We have simulated low-vision patients by asking normally sighted people to wear simulation AMD glasses. Experiments with static images revealed that both normally sighted people and low vision patients preferred enhanced images over original images. Comparison of the enhanced images with the original images showed a statistically significant improvement in the perceived image quality due to enhancement, for subjects with normal vision and for subjects with low vision. 1. Introduction As the population ages, the instances of people suffering from eye diseases and thus from vision impairment is increasing. According to [1], age-related macular degeneration (AMD) affects more than 1.75 million individuals in the United States with the prediction increasing to almost 3 million by These visual impairments affect the dayto-day life of many older people. As can be seen in Figure 1, patients suffering from AMD generally experience blurring or darkness in the center of their vision field resulting in a central field loss (CFL). This interferes with daily activities including reading, mobility, face recognition etc. (a) Normal perception (b) AMD perception Figure 1. (a) is the image that can be seen by a person with normal vision and (b) is the same image that is viewed by a patient with AMD. The images are from NIH website 1 Existing techniques to improve the visual condition of low vision patients cater to only the needs of such people. To the best of our knowledge, we are not aware of any work that enhances the visual experience for both normally sighted people and people suffering from AMD. That is why we propose a color contrast enhancement technique that improves the visual experience of both normally sighted people and patients suffering from AMD. In this paper, we modify the color contrast enhancement technique proposed by Choudhury and Medioni [5]. The method tries to enhance images by separating the illumination from the reflectance of a scene and enhances the illumination while achieving color constancy. The illumination image can be assumed to be a smooth image and in order to smooth the image, Non-local means filter[4] is used. We then try to achieve color constancy by using the White- Patch algorithm [11] to estimate the color of the illumination and then remove the color of the illumination from the scene. Once the color cast has been removed from the image, we process the image depending on the distribution of the intensity pixels, using logarithmic functions to estimate the enhanced illumination automatically and then multiply it back to the scene to get the enhanced image. Once we have obtained the enhanced images we conduct subjective evaluation experiments on visually impaired and 1 facts.asp /10/$ IEEE 33

2 normally sighted people. Since we do not have access yet to low vision patients, we simulate the vision of AMD patients with simulation glasses. The rest of the paper is organized as follows: In Section 2, we review previous work done in image enhancement techniques, including techniques used for low vision patients. Section 3 describes the details of the enhancement algorithm. Section 4 shows the enhancement results and experiments we conducted on normal vision and simulated AMD patients and discuss them. Finally, we conclude our paper in Section Previous Work There has been a lot of work in the area of color contrast enhancement. Some of the techniques include hisogram equalization [7], variations of the Retinex theory - by Kimmel et al. [10] and Jobson et al. [9], using non-linear mapping functions (inverse sigmoid functions) proposed by Tao et al. [15] and many more. Though contrast is improved in these methods, these techniques suffer from certain drawbacks - histogram equalization suffers from color shifts and color artifacts. The method by Kimmel et al. requires manual supervision to change the value of the enhancement parameter and the method by Jobson et al. suffer from halo effects. However, all these methods are used to improve the contrast of images for people with normal vision. Some techniques have been proposed specifically to improve the visual quality of images for subjects with low vision. The technique proposed by Peli et al. [13] called wideband enhancement super-imposes high contrast outlines over images. This method detects visually relevant features in an image (edges) and then marks the edge features with a bright line on the bright side of the edge and a dark line on the dark side of the edge. These features are then either added to the original image pixels or they replace the original pixel values. However, this technique resulted in an improvement in the perceived image quality for only 22% of the patients. Further, when viewed under normal viewing conditions, these features introduce a halo effect that is not desirable. Tang et al. [14] have also developed an enhancement method for JPEG images in the discrete cosine transform (DCT) domain by introducing a scale factor for the quantization table in the decoder. This method increases the contrast by applying a uniform enhancement factor at all frequencies. Though experiments with low vision patients have shown that this results in better perception of the enhanced images, the enhanced images have a lot of artifacts that may not be desirable for people with normal vision. Peli [12] used an adaptive enhancement algorithm to improve the visual quality of images for low vision subjects. In this method, a tuned range of frequencies is enhanced. In order to limit saturation, some low frequencies are reduced. This was tested on low vision patients and was found to be beneficial. However, the enhanced images appear to have some edge artifacts and may be undesirable for subjects with normal vision. 3. The Method Our method for image enhancement is described in the flowchart shown in Figure 2. It consists of 4 key steps:(1) Image Segmentation (2) Illumination Estimation (3) Color cast removal (4) Automatic illumination enhancement Figure 2. Flowchart of our method 3.1. Image Segmentation In order to remove the halo effect that occurs across strong edges, we pre-segment the image using Mean-shift segmentation algorithm proposed by Comaniciu and Meer [6]. The boundaries of the segmented image are used as a preliminary information for the smoothing process Illumination Estimation The illumination component of the image, L(x, y) can be considered to be a smooth image. In order to estimate it, a denoising technique called Non-local(NL) means technique [4] is used. We make the assumption that the denoised image can be considered to be the illumination image. The edge information from the pre-segmented image is used as a prior to reduce the smoothing when a neighborhood for a pixel is considered while estimating the illumination. This removes the halo effect from the image. Since an image can be considered to be a product of illumination and reflectance, once the illumination component has been estimated, the reflectance component of the image R(x, y) can be easily calculated. 34

3 3.3. Color cast removal We try to mimic the human ability of color constancy by removing color cast from the image. Once the illumination image has been computed, the color of the illumination is estimated by using the white-patch algorithm [11] and then removed, that is equivalent to adding white light to the scene Automatic illumination enhancement In order to preserve the color properties of the image, we modify only the luminance component of the illumination image, L ccv (x, y). Our method can be used for images that are underexposed, overexposed or a combination of both. The enhancement method deals with those cases separately and divide the intensity of the illumination map into 2 different regions as shown in Figure 3. Figure 3. The mapping function for enhancement and the effects of changing the value of ρ The division of the region is automatically determined by the positioning of the controlp t that is computed as follows: L ccv (x,y) 0.5 controlp t = 1 L ccv (x,y) 1 1. (1) controlp t [0.5, 1] for underexposed images whereas controlp t [0, 0.5] for overexposed images. In underexposed regions, Choose blue curve dark = Choose green curve Similarly, in overexposed regions, Choose green curve bright = Choose blue curve Lcc (x,y) V Lcc (x,y) 1 1 > T 1 V Otherwise (2) Lcc (x,y) V Lcc (x,y) 1 1 > T 2 V Otherwise (3) The thresholds T 1 and T 2 are determined experimentally. The curves are represented as a logarithmic function. The blue curve can be represented as - L enhv (x, y) = v K log((l cc V (x, y) v 1 )ρ+1)(v 2 v 1 ), and the green curve can be represented as - L enhv (x, y) = v 2 1 K log((v 2 L ccv (x, y))ρ+1)(v 2 v 1 ). (5) where K = log(ρ(v 2 v 1 ) + 1 is a constant and L ccv (x, y) [v 1, v 2 ]. For underexposed regions, L ccv (x, y) [0, controlp t] and for overexposed regions, L ccv (x, y) (controlp t, 1]. ρ represents the curvature of the curve. In an underexposed region, higher value of dark implies that intensity has to be increased more and so a higher value of ρ is used for the first curve. Similarly in an overexposed region, due to higher value of bright, we decrease the intensity of those regions by a higher value and therefore, a higher value of ρ is used for the second curve. This trend is shown in Figure 3. L enhv (x, y) is the enhanced luminance. This is combined with the original chrominance to obtain the enhanced illumination - L enh (x, y) that is multiplied with the reflectance component R(x, y) that was obtained in Section 3.2 to produce the enhanced image I enh. The entire process is automatic and the enhancement occurs depending on the distribution of the pixels in the image. 4. Results and Discussion In this section, we see the results of applying the enhancement algorithm described in Section 3 on a variety of images. The most obvious way to measure the effectiveness of contrast enhancement is to ask a subject (either visually impaired or normally sighted) to indicate their preference in a side-by-side comparison of the original and the enhanced images. The preferences can then be quantified to know the effectiveness of the method. However, large number of comparisons should be made and this should be repeated over multiple subjects. We conduct experiments to that effect and present results from those experiments and discuss them. We also discuss the computational costs of the enhancement algorithm Enhancement Results In order to show the results of our enhancement algorithm, we consider four different images as shown in Figure 4. Image 4(a) that is obtained from the database provided by Barnard et al. [3] is taken under strong blue illumination. Image 4(b) is obtained courtesy of P. Greenspun 2 and is underexposed. Image 4(c) is obtained from NASA 2 (4) 35

4 (a) Removal of color cast (b) Improved contrast (c) Improved contrast (d) Maintain good contrast Figure 4. Enhancement according to our algorithm. Top row: Original images. Bottom row: Enhanced images website3 and is also underexposed. Image 4(d) is obtained from Flickr and has good contrast. As shown in Figure 4, our enhancement technique improves the contrast of images that are overexposed or/and underexposed. It also removes the color cast from a scene thus improving the quality of the image. This technique also does not degrade the quality of images that already have a good contrast. Existing methods introduce color shifts (Figure 5(a)) or do not enhance contrast well enough (Figure 5(b)). of good contrast images and therefore, we want to study perception in those conditions The experiments were conducted independently on the subjects. Each subject was seated at a distance of roughly 20 from the computer monitor. The subjects were given the freedom to move closer to the screen or farther away from the screen according to their convenience. The screen is around 24 (diagonal) and the subjects were seated perpendicular to the center of the screen. SUBJECTS. All 12 subjects had normal vision (The subjects were asked to wear their prescribed corrective lens during the entire course of experimentation). All subjects were able to understand the instructions given to them in English. The subjects viewed the images on the screen with both eyes. (a) Histogram Equalization (b) Retinex by Kimmel Figure 5. Enhancement of original image from Figure 4(a) using existing methods 4.2. Experimental Setup We created a database of 40 images consisting of a variety of images under different lighting conditions (colored/white illumination) and different exposure conditions(over-exposure/under-exposure/good contrast images) taken either indoor or outdoor. The reason why we have included good contrast images is that, normal visual activity such as watching movies includes perception SIMULATION GLASSES. In order to simulate agerelated macular degeneration (AMD), we purchased simulation glasses from Low Vision simulators website4. These glasses have a blind spot that is white and opaque whereas the visual periphery is fogged (blurred). For our experiments, we have used simulation glasses that simulate visual acuity 20/400 (6/120) and have a large central scotoma. While conducting experiments, the subjects were asked to wear the glasses on top of their corrective lenses. PROCEDURE. All the images were processed off-line to produce the enhanced images. For every image, the experimental procedure has 2 steps 1. The original and the enhanced images were shown simultaneously on the screen. The placement of the original and the enhanced versions were random (either

5 the left or the right side of the screen) to remove any bias that may exist while selecting an image from only one side of the screen. Then the subject was asked to rate the image on the right side as Better, Same or Worse relative to the image on left side of the screen and their responses were recorded by the experimenter. Scores were thus assigned as 1 for Better, 0 for Same and -1 for Worse. These responses were used to evaluate the preference for enhanced image. 2. The original image from Step 1 was then presented on the screen and the subject was asked to rate the quality of the image as Poor, Average or Good that was duly recorded by the experimenter as 1, 2 or 3 respectively. The subjects were unaware of the fact that the image that is being presented to them in Step 2 is the original image from Step 1 to remove any bias that may occur due to knowledge of the original image. Rating preference prior to quality is important to remove the bias of prior information regarding the original image. This was repeated for all 40 images in the database. Finally, one image that had the most perceptible difference after enhancement was presented to the subject but with the order flipped from the previous display and the response was noted to check if the subjects were consistent with their responses. This entire procedure was repeated twice - firstly, while wearing the simulation glasses and secondly, without them. Since all the subjects have normal vision (with/without corrective lenses), conducting the experiment first while wearing the simulation glasses was important. This is because these glasses significantly degrade the visual quality and therefore, the subjects have no prior information regarding preference for a particular image which need not be the case if the experiments were conducted first without the glasses Data Analysis In our experiments, we found that subjects were not biased towards selecting any one side of the screen. The responses of all the subjects were also consistent when the preference ratings for the last comparison was compared with that of its earlier ratings. Figure 6 shows the original image quality ratings for all the subjects with normal vision. Figure 7 shows the original image quality ratings for all the subjects with simulated AMD. A better idea of how the visual perception of people with normal vision is different from that of people with simulated AMD can be indicated by the histogram in Figure 8. We can see that around 72.5% of images were rated above Average (quality >= 2) by people with normal vision. On the other hand, 50% of images were rated below Average (quality <= 2) by people with simulated AMD and most of the images were considered to be very close to Average. This gives us a relative idea on how poorly patients with AMD perceive the environment. Figure 6. Image quality ratings for subjects with normal vision Figure 7. Image quality ratings for subjects with simulated AMD Figure 8. Histogram of mean image quality ratings for all subjects with simulated AMD and normal vision Similarly, Figure 9 shows the preference ratings for the enhancement method for all the subjects with normal vision. Figure 10 shows the preference ratings for the enhancement method across all the subjects with simulated AMD. The histogram in Figure 11 gives a better visualization of how the preference for enhancement varies between subjects with normal vision and for subjects with simulated AMD. We can see that both normally sighted subjects and simulated AMD subjects prefer our enhanced images and the preference is larger for subjects with simulated AMD than for subjects with normal vision. We use the Wilcoxon signed-rank test [16] to check for statistical significance. In both cases, for subjects with nor- 37

6 Figure 9. Preference for enhancement ratings for subjects with normal vision this case. The raw data consists of the preference of subjects regarding perceived quality of the enhanced images with respect to the original images. In this case, the axes of the ROC plots have a different interpretation. The Y- axis, that originally corresponds to the true positive rate can be considered to be equivalent to the proportion of the enhanced image with higher perceived image quality. The X- axis, that originally corresponds to the false positive rate can be considered to be equivalent to the proportion of original image with higher perceived image quality. In order to generate the ROC plots, the program jrocfit [2] was used. The ROC plot for subjects with simulated AMD is shown in Figure 12 whereas the ROC plot for subjects with normal vision is shown in Figure 13. With a larger and a more continuous ratings scale, more operating points can be generated resulting in a smoother ROC curve. Figure 10. Preference for enhancement ratings for subjects with simulated AMD Figure 12. ROC plot for subjects with simulated AMD. The data being degenerate does not produce operating points off the axes Figure 11. Histogram of mean preference ratings for all subjects with simulated AMD and normal vision mal vision and for subjects with simulated AMD, we infer that the preference for enhanced images are statistically significantly different from that for the original images (The null hypothesis of zero median of the difference is rejected at 5% level) and by comparing the ranks we conclude that the enhanced images have a higher rank than the original images. This implies that subjects (both visually impaired and normally sighted) have a significant preference for the enhancement method. The evaluation of the preference of the enhancement method can be analyzed using a variation of the ROC approach as described by Peli et al. [13] to give an impression of perceived image quality with respect to the original images. Unlike in regular ROC analysis, where ground-truth information is provided, no such information is present in Figure 13. ROC plot for all subjects with normal vision. The data being degenerate does not produce operating points off the axes We consider the area under the ROC curve [8] (A z ) as a measure of the perceived quality of the enhancements. For the original images, the value of A z = 0.5. If A z > 0.5 then the perceived quality of the enhanced images can be considered to be better than that of the original images. On the other hand, if the perceived quality of the original images is better than that of the enhanced images, the value of A z < 0.5. The jrocfit program computes the empirical value of A z and those values for the different subjects are shown in Figure 14. For all the subjects, on an average, both with normal vision (A z = 0.67 ± 0.058) and with simulated AMD (A z = ± ), the enhanced images 38

7 are deemed to have better quality than the original images. Using the Wilcoxon signed-rank test [16], for both instances (simulated AMD and normal vision), our color contrast enhanced images was reported as having significantly better quality than original images. An interesting case is that of image label 29. The original image has a very high rating (quality > 2.5) and the subjects with normal vision almost always (10/12 subjects) prefer the original images(preference < 0). This is because the original image is that of a flower as shown in Figure 15(a) and after increasing the contrast, the enhanced flower no longer looks natural as shown in Figure 15(b). This causes the subjects to prefer the original image. However, when viewed with the simulation AMD glasses, the subjects preferred the enhanced image(preference > 0). Figure 14. A z s for subjects with normal vision and simulated AMD 4.4. Discussion As can be seen in Figure 11, subjects with simulated AMD show a preference for the enhanced images. We have also shown that the preference for the enhanced images is statistically significant. However, for certain images as can be seen in Figure 10, the subjects prefer the original images (preference < 0). For those images, we found that certain regions of the image were over-exposed. The enhancement method reduces the intensity of those regions of the image. As the simulation glasses degrades the visual quality, the subjects prefer the original images as the brighter regions of the image are better visible. The subjects with simulated AMD showed no preference (preference = 0) for 5 out of 40 enhanced images when compared with the original images. When we checked the quality of those original images (Figure 7), we found that 4 of those images were rated as above Average (quality > 2) implying similar rating for the enhanced images. For all the images that were present in the lower quarter of the quality ratings (quality [1, 1.5]), the subjects consistently showed preference for the enhanced images (preference > 0). For people with normal vision, as can be seen in Figure 11, on an average, there is a preference for the enhanced images (preference > 0). However as can be seen in Figure 9, for 5 images the subjects prefer the original images (preference < 0), though the preference is not significantly lower. If we check the image quality of those original images (Figure 6), we can see that 3 of those images were rated high (quality > 2.5). Also for all the 4 cases where enhanced images were considered to be the same as the original images (preference = 0), the quality of the original images was rated was rated high (quality > 2.5). For all the cases where the original images were rated Poor (quality = 1), the subjects consistently showed preference for the enhanced images (preference > 0). (a) Figure 15. (a) is the original image and (b) is the enhanced image The parameters of the approach can be modified to account for such instances thus improving the results of the method. We believe that modifying those parameters will not result in a significant change in the overall preference for the images. We also notice that, in general, the subjects with simulated AMD show more preference for the enhanced images than the subjects with normal vision. This is because subjects with normal vision have a better perception of the original quality of images. A lot of images (22/40) are good contrast images (quality > 2.5) thus reducing the scope for preference for enhancement for those images amongst people with normal vision. However, when viewed through simulation AMD glasses, due to degradation of original image quality, the scope for enhancement increases thus increasing the preference for enhanced images. The enhanced images improve the visual quality for subjects with normal vision and for subjects with simulated AMD. However, trade-offs can be done to obtain better enhancements for a particular category of subjects at the expense of others. For the experiments that are conducted here, no training database has been used to set the parameters of the algorithm. There is a higher chance of getting better results for either category of subjects by tuning the parameters of the enhancement method based on the results that were obtained from this approach Quantitative Evaluation A good way to assess the effectiveness of an enhancement algorithm is to measure the change in image quality in terms of brightness and contrast [15]. To measure the (b) 39

8 brightness of the image, the mean brightness of the image is considered and to measure the contrast of the image, local standard deviation of the image(standard deviation of image blocks) is considered. As shown in Figure 16, a plot is made of the mean brightness of the image against the mean of the local standard deviation of the image. For underexposed images, enhancement will result in an increase of the mean brightness while the reverse is true for overexposed images. Similarly, an increase in contrast corresponds to an increase in the mean local standard deviation. enhancement method with existing methods with regards to the preferences of low vision subjects.the task mentioned in this paper is predominantly to find the preference of the patients. However, daily activities involve recognition tasks (face, objects etc.,) and it would be useful to see how our algorithm can help low-vision patients with such activities. It will also be interesting to note the preferential effects of people for different levels of enhancements. 6. Acknowledgment This research was supported by the National Institutes of Health Grant EY References Figure 16. Image quality before and after enhancement 4.6. Computational Costs We have implemented the algorithm in MATLAB in Windows XP environment on a PC with Xeon processor. For an image with size 360 X 240 pixels, the enhancement takes around 45 sec. The speed can be improved to close to real-time by implementing the code in C++ on a GPU and optimizing it. 5. Conclusion and Future Work We have developed a color contrast enhancement method that is beneficial for normally sighted people as well as for people with visual impairment such as AMD due to CFL. All the subjects (both normally sighted and visually impaired) showed a preference for the enhanced images and we showed that the improvement in perceived image quality was significant. Though the tests were performed on static images, the framework can be easily extended to video sequences. One of the advantages of our method is that the parameters of the algorithm were estimated automatically. Better performances can be achieved by asking the users to manually modify the parameters according to their preferences. We are aware that the simulation glasses do not exactly replicate the real disorder as the eyes can shift to avoid the artificial scotoma. Since we are trying to simulate low vision, we believe that using these glasses is a good initial step to test our results. In future work, we will test our algorithm on real low-vision patients. We will compare our [1] Archives of Ophthalmology, 122: , [2] E. J. R. analysis:web-based calculator for ROC curves. Baltimore:Johns Hopkins University, [3] K. Barnard, L. Martin, B. Funt, and A. Coath. A data set for color research. Color Research and Application, 27(3): , [4] A. Buades, B. Coll, and J.-M. Morel. Nonlocal image and movie denoising. IJCV, 76(2): , [5] A. Choudhury and G. Medioni. Perceptually motivated automatic color contrast enhancement. In ICCV CRICV Workshop, [6] D. Comaniciu and P. Meer. Mean shift: a robust approach toward feature space analysis. PAMI, 24(5): , [7] R. C. Gonzalez and R. E. Woods. Digital Image Processing (3rd Edition). Prentice-Hall, Inc., NJ, USA, [8] J. A. Hanley and M. B. J. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143:29 36, [9] D. Jobson, Z. Rahman, and G. Woodell. A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE TIP, 6(7): , [10] R. Kimmel, M. Elad, D. Shaked, R. Keshet, and I. Sobel. A variational framework for retinex. IJCV, 52(1):7 23, [11] E. H. Land. The retinex theory of color vision. Scientific American, 237(6): , December [12] E. Peli. Recognition performance and perceived quality of video enhanced for the visually impaired. Opthal. Physiol. Opt., 25: , [13] E. Peli, J. Kim, Y. Yitzhaky, R. B. Goldstein, and R. L. Woods. Wideband enhancement of television images for people with visual impairments. Journal of the Optical Society of America A, 21: , June [14] J. Tang, J. Kim, and E. Peli. Image enhancement in the jpeg domain for people with vision impairment. IEEE Transactions on Biomedical Engineering, 51: , [15] L. Tao, R. Tompkins, and V. Asari. An illuminancereflectance model for nonlinear enhancement of color images. In CVPR Workshops, [16] F. Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80 83,