Fusion of Colour and Monochromatic Images with Chromacity Preservation

Transcription

1 Fusion of Colour and Monochromatic Images with Chromacity Preservation Rade Pavlović Faculty of Technical Sciences Trg Dositeja Obradovica 6 11 Novi Sad, Serbia rade_pav@yahoo.com Vladimir Petrović Imaging Sciencе University of Manchester Oxford Road, Manchester, M13 9PT, UK v.petrovic@manchester.ac.uk Boban Bondžulić Military academy University of Defence in Belgrade Gen. Pavla Jurišića Šturma 33, Belgrade, Serbia bondzulici@yahoo.com Abstract - We propose a novel method to fuse natural colour images with monochromatic non-visible range images that seeks to efficiently encode important structural information from monochromatic images but also preserve natural appearance of the available true chromacity information. We utilise the β colour opponency channel of the lαβ colour as the domain to fuse information from the monochromatic input into the colour input by the way of robust grayscale fusion. This is combined by effective intensity information fusion that enhances the visibility of monochromatic information in the final colour fused image. Images fused using this method preserve their natural appearance and chromacity better than conventional methods while at the same time clearly encode structural information from the monochorme input. This is demonstrated on a number of well known true colour fusion examples and confirmed by results of subjective trials on data from several colour fusion scenarios. Keywords - Colour image fusion, Chromacity preservation, Colour space. I. INTRODUCTION Multisensor systems observe a scene with multiple sensor modalities to provide a more complete and reliable representation than could be achieved with a single type of sensor. In order to fully exploit this additional information however, considerable processing effort is required. Moreover, when the intended user is a human, displaying multiple modalities simultaneously leads to confusion and overload, while integrating information across a group of observers is almost impossible [3]. Image fusion deals with this problem by presenting the information content of multiple input images in a single fused image [1,2,3,4]. The goal of image fusion can be broadly defined as: the representation of visual information contained in a number of input images, in a single fused image without distortion or loss of information. In practice however, representation of all available information from multiple inputs in a single image is almost impossible and fusion is generally a data reduction task. More so when one of the sensors provides a true colour image that by definition has all of its data dimensions already populated by the spatial and chromatic information. Fusing such images with information from monochromatic inputs in a conventional manner can severely affect natural appearance of the resulting fused image [9,1]. Colour fusion is a difficult problem and partly the reason it has thus far received only a fraction of the research attention afforded to better behaved grayscale fusion even long after colour sensors become widespread. Much work has been done in the related field of encoding colour channels of a "pseudo-coloured" fused image with information from monochrome inputs [5,6,7,8]. These images visualise scene structure well but generally exhibit unnatural appearance.. The quality of the fused colour image depends greatly on choice of colour system used to encode different monochromatic channels as well as the choice of a reference colour image for methods that aim to transfer natural appearance from a related true colour image. Fusion of a true colour image with a corresponding IR view was proposed by Xue and Blum [1] for a concealed weapons detection application. Fused images maintain their high resolution but the method is computationally expensive. Colour fusion where a monochrome input is fused with the intensity channel of a colour input was investigated in [13] for the HSV and IHS colour spaces. In the following we describe a method for fusion of true colour images with monochromatic inputs that both preserves available true chromacity information and provides advanced visualisation of important structural information from monochromatic images. Section 2 describes the basic fusion method while section 3 introduces the additional intensity encoding step. Section 4 demonstrates the method on some well known examples and provides results of a set of subjective trials run to evaluate robustness and naturalness of the proposed method. We conclude and consider directions of further research in section

2 Figure 1. A true colour image a) and corresponding IR image b), courtesy OCTEC Corp. II. COLOUR FUSION METHOD Fig. 1 shows a true colour and a monochromatic image (in this case IR) of the same scene. The IR image clearly shows a human figure but not the general structure of the scene while the colour image depicts the landscape but not objects hidden behind the smoke screen. Conventional methods [9,1] generally transform the colour image from its original RGB space into one that decorrelates chromacity and intensity information, e.g. HSV/IHS, LMS, LAB, and then fuse the monochromatic input with one of the now monochromatic channels of the colour image, most commonly the intensity channel. Success of this process depends on the choice of colour system and suitability of the chosen grayscale fusion [2,3,4]. Fig. 2 shows a fused image using HSV colour space and Laplacian pyramid fusion [3] of the V (intensity) channel with the IR image. Even though Laplacian fusion robustly transfers important structures from the IR image, the fused image has lower contrast and a significantly less natural appearance than the colour input. The aim of our fusion method is to make objects visible in the IR image clearly visible in the colour fused image that would also preserve all the contextual (landscape) information as well as the natural appearance of the colour input. Humans tend to see colours as contrasts between opponent colours [18,19,2] and an improvement in visibility of structures from the monochrome can be achieved when they are used to encode a single HVS colour dimension consistently. The lαβ colour system [11] effectively decorrelates the colour opponency and intensity channels and manipulating one causes no visible changes in the others. Colour fusion can be achieved by fusing one of the colour opponency channels with the monochrome image. We use Laplacian pyramid fusion [2,3] known to be one of the most robust monochrome fusion methods available [14]. The Laplacian, also known as the DOLP (difference of lowpass) pyramid is a reversible multiresolution representation that expresses the image through a series of sub-band images of decreasing resolution, increasing scale, whose coefficients broadly express fine detail contrast at that location and scale. A simple fusion strategy creates a new fused pyramid by copying the largest absolute input coefficient at each location. In order to avoid large scale discolouration we employ a modified Laplacian pyramid fusion described in more detail below. A. lαβ colour system In contrast to generally correlated colour spaces (RGB, HSI, LMS,...), Ruderman et al. [12] presented a colour system, the lαβ, whose coordinates are only weakly correlated and enable one of the channels to be manipulated with minimal impact on others [11]. The lαβ colour representation is derived from the LMS space obtained in turn directly from the RGB using a simple matrix transformation:,3811,5783, 42,1967,7244,782 (1),241,1288,8444 The LMS colour values are further compressed logarithmically [12]: ] (2) Even after compression the LMS values remain correlated so in [12] a solution is suggested using a further transformation that de-correlates colour information into a new lαβ system that best models human vision: Figure 2. Colour fusion of inputs in Figure 1 using HSV space and Laplacian pyramid fusion (3)

3 In the lαβ colour system the l component loosely represents intensity while α and β represents yellow-blue and green-red opponents, derived from L for red, M green and S blue. An example of the lαβ colour representation of image in Fig. 1a is on Fig. 3. An RGB frame representation is recovered from the lαβ via the LMS again: (4) 1 2 Conversely to the applied compression LMS components need to be stretched exponentially and then converted linearly to the RGB system: 4,4679 3,5873,1193 1,2186 2,389,1624 (5),497,2469 1,245 B. Colour Image Fusion The β channel of the lαβ space represents the red-green opponency and we base our fusion on encoding this channel of the colour input with the monochrome image. This causes warmer objects (lighter in IR) to appear redder in the fused image which is a perhaps the only innate way of perceiving thermal radiation. The fusion proceeds in several steps. Initially we transform the colour input RGB image into the lαβ image using (1-3). As in general the statistics of the β channel and monochrome images are different we need to normalise them prior to fusion in order to avoid large scale discolorations. We normalise the monochrome image to have the same mean and standard deviation as the colour β channel in order to preserve the natural appearance of the colour input: (6) where I m is the monochrome pixel, µ β, µ m and σ β, σ m are the mean and standard deviation values of the β channel and monochrome image respectively. Monochrome fusion is then performed by decomposing the β image and the normalised monochrome into their Laplacian pyramid representations [2,3]. We use above mentioned selectabsolute-max strategy to construct the fused pyramid but only apply it to a small number (up to 4) of high resolution pyramid sub-bands.. Larger scale features in lower resolution sub-band images that constitute the natural context of the scene are sourced entirely from the colour image (β). This ensures that well defined smaller objects from the IR image are transferred robustly into the fused but broad scene context remains from the colour input. Reconstructing the fused pyramid produces the fused β image which is combined with original l and α channels of the colour input in (4) and (5) to produce the fused RGB colour image. An example of this approach on the input images in Fig. 1 is shown on Fig. 4. III. AUGMENTED INTENSITY COLOUR FUSION Fused image obtained using the method described in 2.B represents clearly information from the monochrome image while preserving the natural appearance of the colour image. However, even though good results are obtained in the majority of tested cases fused images can also suffer somewhat from a lack of contrast when intensity contrast of the colour input is very low such as during night hours. This is shown on Fig. 5 where images obtained in daylight and night-time conditions are fused using the proposed method. In the low visibility example β channel fusion merely colours low contrast features and is not capable of improving their visibility. In good visibility conditions this is not a problem as the true colour image contains all landscape information and good contrast between objects such as people and their background. Information from the monochrome encoded in the fused is then clearly visible in its red channel while the natural appearance of other colours is well preserved. Figure 3. lαβ representation of image in Fig. 1a) intensity channel l a), blueyellow channel α b) and red-green channel β c) Figure 4. Fused image obtained using β channel encoding 1965

4 Figure 6. Fused image using augmented intensity colour fusion Figure 5. Input and fused images from the Dublin series onbtained in different visibility conditions This sensitivity to low contrast colour inputs can be solved by augmenting proposed β channel fusion with intensity fusion where the intensity of the resulting fused colour image is also encoded with the information from the monochrome input. This would improve visibility of low contrast details while preserving the benefits of colour encoding. Intensity encoding can be achieved by first fusing the monochrome input with the intensity of the colour input using again the trusted the Laplacian pyramid approach and a HSV representation of the true colour. The fused V channel is then mapped back with the true colour H and S channels into an intensity fused colour RGB image. This image is then used as a direct input into the lαβ fusion based on the β channel encoding as described in Section 2.B. In contrast to approach in 2.B however, Laplacian pyramid fusion of the monochrome input and the I channel use pyramid coefficient selection maps created during monochrome fusion rather than direct selectabs-max between these pyradmids, Fig. 7. This ensures that information fused from the monochrome image remains adequately encoded in the red-green channel. A fused image obtained using this augmented intensity encoding of the inputs shown on Fig. 5 is shown on Fig. 6. It is clear that the visibility of hot objects has been significantly improved even though they have almost no contrast in the true colour input while they still remain adequately coloured to indicate their origin (thermal image). IV. RESULTS An example fused image obtained with the proposed β channel encoding method described in 2.B on the input pair from Fig. 1 using pyramid depth of 7 and 3 lowest resolution levels sourced from the β image of the colour input are shown in Figure 4. The image exhibits a clear visualisation of objects and structures introduced from the monochrome input (1b) such as the human behind the smoke screen. At the same time the scene context (buildings, background, ground ) and natural appearance are preserved. Compare that to the appearance of the HSV + Laplacian fusion in Fig. 2. A further example from the OCTEC sequence shown on Fig. 8 illustrates the advantage of the proposed β channel method over conventional HSV [13] colour fusion in terms of target detection. In this scene another person is present in the scene, but hardly visible even in the IR input. In the HSV fused image 8c, even the larger figure is hard to detect while both humans and the smoke screen are clearly coloured and visible by the proposed method in 8d. Fig.9 illustrates proposed fusion on the challenging EDEN dataset [17] including objects of a wider range of scales and very different statistics. The inputs 9a and 9b are spatially registered but a time delay creates a degree of miss-registration. The IR input clearly shows a person in the scene not immediately detectable in the true colour image. Fusion using the proposed β channel method produces images with natural context where the human is clearly detectable, even including detailed structure of the face and hairline brought into focus from the colour input. An example of intensity augmented colour fusion is shown on Fig. 1. Inputs 1a and 1b are imaged in low visibility conditions and contrast in the colour input is modest. β channel fusion 1c produces relatively low visibility of monchrome information but intensity augmented fused image produces a much greater visibility of details. To show that you can never quite have it both ways the intensity augmented image exhibits a slight loss of naturalness compared to the true colour input. 1966

5 Figure 7. The complete colour fusion method A. Subjective Evaluation Results Proposed colour fusion algorithms were also evaluated with respect to perceived fusion success and naturalness of the fused images in subjective trials. During the trials, the subjects were shown series of 5 images including a true colour and a monochrome (mostly IR) input images and three different colour fused versions of the input. For each image set, observers were asked to choose the image they considered to be a best representation of available input information (fusion success) and the image that appeared the most natural (naturalness). Colour fusion methods evaluated in the trial were: HSV fusion where the monochrome image is fused with the intensity (V) channel of the colour input mapped to the HSV colour space [13] and two variants of the β fusion methods of which the first one is the one proposed in section 2.B. HSV fusion included in the trial used the same monochrome fusion approach (Laplacian pyramid fusion) so any difference in the recorded scores should be attributed to the novelties in the proposed approach. In all 23 image sets were shown covering five different scenarios (sequences) including both urban and natural scenes in a range of visibility conditions (majority with medium to good visibility). 15 semi-expert observers performed the test in standard room conditions (a realistic environment for users of similar systems) on a 2 inch monitor with an average focus to screen distance of around 8cm. Subjective preferences for the two metrics comparing HSV intensity fusion and the proposed β channel method of section 2.B are shown in Fig. 11. The results clearly show that the proposed β channel fusion outperforms the conventional in terms of fusion success (more than 3:1) and naturalness (2,5:1). The intensity augmented fusion was not a part of the trial. Figure 8. Colour fusion example: a) and b) true colour and IR inputs, c) HSV transformation fused image and d) fusion using the proposed β channel method 1967

6 Figure 9. Colour fusion of EDEN dataset: a) and b) true colour and IR inputs c) and d) fusion with HSV and proposed β channel method Figure 1. Colour fusion of inputs a) and b) using proposed method without and with intensity augmentation c) and d) 1968

7 Fusion Successs Naturanless β fusion HSV fusion β fusion HSV fusion Figure 11. Subjective trial average preferences for 23 datasets and 15 viewers V. CONCLUSION A new β fusion colour image fusion method is presented that both successfully visualises important structure information from the monochrome input and preserves the natural appearance of the true colour input. Colour fusion is performed in the lαβ colour space known to de-correlate colour information seen by the human visual system. β channel representing the red-green opponency is encoded with structural information from the monochrome image using a modified Laplacian pyramid fusion approach. Further to this method an intensity augmentation colour fusion is also proposed to deal with cases when intensity contrast of the true colour input is low (poor visibility conditions). Proposed fusion methods produce colour fused images with significantly better visualisation of important information from the monochrome input while almost entirely preserving the natural appearance of the true colour input. This was demonstrated on a number of well known colour fusion examples and measured using subjective trials on data from multiple surveillance scenarios. Further work in this direction will investigate scale dependency of the structure visualisation enhancement procedure as well as inclusion of background extraction methods in the fusion process to more effectively deal with large scale discolouration effects. More robust mechanisms for determination of (non) exclusivity of visual information in the input images and dealing with competing information from the inputs would also be pursued. Further subjective evaluation will also include intensity augmented fusion. REFERENCES [1] V Petrovic, T Cootes, Obejtively adaptive image fusion, Information fusion, Volume 8, Issue 2, April 27, pp [2] P Burt, E Adelson, The Laplacian pyramid as a compact image code, IEEE Transactions on Communication, COM-31,1983, pp [3] A Toet, Hierarchical image fusion, Mach. Vision Appl. 3, 3 11, 199 [4] V Petrović, C Xydeas, Gradient Based Multiresolution Image Fusion, IEEE Transactions on Image Processing, IEEE, Vol. 13(2), February 24, pp [5] A Toet, Natural colour mapping for multiband nightvision imagery, Information fusion, 23. Vol.4, No.3, pp [6] M A Hogervorst, A Toet, Fast natural color mapping for night-time imagery. Information Fusion, 21, Vol.11, No.2, pp [7] J Yhang, Y Han, B Chan, Y Yuan, Y Qian, Y Qiu, Real-time Color Image Fusion for Infrared and Low-light-level Cameras, International Synposium on Photoelectronic Detection and Imaging 29, Proc. Of SPIE Vol.7383 [8] L Guangxin, X Shuyan, An Efficient Color Transfer Method for Fusion of Multiband Nightvision Images, Information Engineering and Computer Science, 29. ICIECS 29. International Conference on. [9] A Toet, Color Image Fusion for Concealed Weapon Detection, Sensors, and Command, Control, Communications, and Intelligence (C3I) Technologies for Homeland Defense and Law Enforcement II, Vol. 571, pp [1] X Zhiyun, R Blum, Concealed Weapon Detection Using Color Image Fusion, Information Fusion, 23. Proceedings of the Sixth International Conference of 23, pp [11] E Reinhard, M Ashikhmin, B Gooch, P Shirley, Color transfer between images, IEEE Computer Graphics and Applications, 21, Vol.21, No.5, pp [12] A Ruderman, O R Joubert, M Fabre-Thorpe, Statistics of cone responses to natural images: implications for visual coding, Journal of the Optical Society of America A, 1998, Vol.15, No.8, pp [13] M Huang, J Leng, C Xiang, "A Study on IHS+WT and HSV+WT Methods of Image Fusion," International Symposium on Information Science and Engieering 28, vol. 1, pp [14] V Petrović, C Xydeas, Objective Image Fusion Performance Characterisation, Intl. Conf. Computer Vision ICCV25, Beijing, October 25 [15] The Online Resource for Research in Image Fusion, [16] D Marr, Vision, W.H.Freeman, San Francisco, 1982 [17] T Dixon, S Nikolov, J Lewis, J Li, E Canga, J Noyes, T Troscianko, D Bull, C Canagarajah, Scanpath analysis of fused multi-sensor images with luminance change: A pilot study. Proceedings of Fusion26, Italy, 26 [18] A M Waxman, M Aguilar, R A Baxter, D A Fay, D B Ireland, J P Racamato, W D Ross, Opponent-color fusion of multi-sensor imagery: visible, IR and SAR, Proceedings of IRIS Passive Sensors, vol.1, pp , [19] A M Waxman, at al., Solid-state color night vision: fusion of low-light visible and thermal infrared imagery, Lincoln Laboratory Journal, vol. 11, no. 1, pp. 41-6, 1998 [2] M Aguilar, D A Fay, W D Ross, A M Waxman, D B Ireland, J P Racamato, Real-time fusion of low-light CCD and uncooled IR imagery for color night vision, Proceedings of the SPIE, vol.3364, pp ,