Multispectral Imaging Development at ENST

Size: px

Start display at page:

Download "Multispectral Imaging Development at ENST"

Abel Perkins
5 years ago
Views:

1 Multispectral Imaging Development at ENST Francis Schmitt, Hans Brettel, Jon Yngve Hardeberg Signal and Image Processing Department, CNRS URA 82 École Nationale Supérieure des Télécommunications 46 rue Barrault, F Paris Cedex 3, France Abstract We present the development at ENST of a multispectral imaging system. Various methods have been implemented and tested for the characterisation of spectral camera sensitivity, the optimal choice of filters and the reconstruction of the spectral reflectance of the imaged surfaces. We first used a 4k-linear array camera with a set of large band filters. We are now experimenting with a cooled k x k single chip camera coupled with a liquid crystal tunable filter. Keywords: multispectral acquisition system, sensitivity characterization, filter selection, spectral reflectance reconstruction Introduction Multispectral imaging systems are developping rapidly because of their strong potential in many domains of application, such as remote sensing, astronomy, physics, museum, cosmetics, medicine, high-accuracy colour printing, computer graphics... They provide information about a number of spectral bands, from three components per pixel as for colour images up to one thousand bands for hyperspectral images. Hyperspectral image acquisition systems remains complex, expensive and difficult to manage due to the huge amount of data to be stored and processed. For high-end multimedia applications we have considered a more affordable approach based on digital imaging techniques in which a reduced set of chromatic filters are used with a CCD camera. This was a natural extension of the high quality digital color image system that we developed for the analysis of paintings, based on a Kodak Eikonix 42 line-scan CCD camera (4k-linear array) with a set of large band filters [, 2, 3]. The multispectral image acquisition system we are aiming at is inherently device independent; we seek to record data representing the spectral reflectance of the surface imaged in each pixel of the scene, independently of the spectral characteristics of the acquisition system and of the illuminant. To obtain device independent measurements of high quality, it is necessary to specify the spectral characteristics of the components involved in the image acquisition process. The first step is to process to a spectral characterisation of the camera sensitivity. Then we have to choose optimally a reduced set of colour filters which will provide the best spectral reflectance reconstruction according to a given criterion. The set of filters being fixed, the last step consists

of estimating from the camera responses the spectral reflectances of the actual surfaces which are imaged. These three steps are successively presented in the next sections.

2 of estimating from the camera responses the spectral reflectances of the actual surfaces which are imaged. These three steps are successively presented in the next sections. They remain valid with the new and more flexible acquisition system which we are now developing and which is based on a cooled k x k single chip camera coupled with a liquid crystal tunable filter. This system is describd in the last section. 2 Spectral characterisation of the acquisition system The main components involved in the image acquisition process are depicted in Figure. We denote the spectral radiance of the illuminant by l R (), the spectral reflectance of the object surface imaged in a pixel by r(), the spectral transmittance of the optical systems in front of the detector array by o(), the spectral transmittance of an optical colour filter by k (), and the spectral sensitivity of the CCD array by a(). In this section we first consider an unfiltered monochrome camera and will omit the spectral transmittance k () in the calculations. Illumination l ( λ) R o( λ) φ ( λ) k a( λ) r( λ) Observed object Sensor Optical path Colour filter Figure : Schematic view of the image acquisition process. Supposing a linear optoelectronic transfer function of the acquisition system, the camera response c to an image pixel is then equal to c = Z max min l R ()r()o()a() d = Z max min r()!() d () where!() = l R ()o()a() denotes the system unknowns. The assumption of system linearity is based on the fact that the CCD sensor is inherently a linear device. However, for real acquisition systems this assumption may not hold, due for example to electronic amplification non-linearities or stray light in the camera. Then, appropriate nonlinear corrections may be necessary [4]. By uniformly sampling the spectra at N wavelength intervals, we can rewrite Equation () as a scalar product in matrix notation, c = r t!; where! = [!( )!( 2 ) :::!( N )] t and r =[r( ) r( 2 ) :::r( N )] t.

3 Direct spectrophotometric measurements with monochromatic light require expensive equipment. We opted for the popular indirect approach where the vector! describing the system unknowns is estimated from the camera responses c p to a set of P samples with known reflectances r p [4, 5]. Denoting the sampled spectral reflectances of all the patches as the matrix R =[r r 2 :::r P ], the camera response c P =[c c 2 :::c P ] t to these P samples is then given by c P = R t!: (2) Several methods have been proposed for the estimation ~! of the camera characteristics!. The simple system inversion by: ~! =(RR t ), Rc P =(R t ), c P ; (3) where (R t ), denotes the Moore-Penrose pseudo-inverse of R t yields very large errors in the presence of noise. Instead, the Principal Eigenvector method should be used, the noise sensitivity of the system inversion being reduced by only taking into account the singular vectors corresponding to the most significant singular values. A Singular Value Decomposition (SVD) is then applied to the (P N ) matrix R t of the spectral reflectances of the observed patches: R t = UWV t ; where U and V are (P P ) and (N N ) unitary matrices repectively, W is a (P N ) matrix with general diagonal entry the singular values w i, i = :::R, and zeros elsewhere, R being the rank of R t. U and V being unitary matrices, it can easily be verified that (R t ), = VW, U t,wherew, has a general diagonal entry equal to w, i, i =:::R,and zeros elsewhere. Munsell patches Camera spectral sensitivity functions.9.8 Sensitivities under illuminant A Evaluation Reflectance r(λ) reflectances from the finnish Munsell database 2.5R 7/6 2.5G 6/2 2.5B 6/8 Camera model : : : Methods: PI, PE Sensitivities under illuminant A Wavelength, λ [nm] Munsell spectral reflectances Acquisition simulation Camera responses Spectral characterisation Estimated spectral sensitivity functions Figure 2: Simulation of the acquisition system to evaluate spectral characterisation methods. It is well known that the singular values of a matrix of spectral reflectances such as R t are strongly decreasing, and by consequence that reflectance spectra can be described accurately by a quite small number of parameters. By taking into account only the first r<rsingular values in the system inversion, the spectral sensitivity may thus be estimated by : ~! = VW (r), U t c P ; (4)

4 where W (r), has now a general diagonal entry equal to w, i, i =:::r, (r <R). We have performed a simulation to evaluate the spectral characterisation methods for three RGB channels and to determine the optimal number r of principal eigenvectors that should be retained, depending on the level of noise [6]. The scheme of the simulation is shown in Figure 2. It is based on the Eikonix camera with its built-in filter barrel. We have considered the spectral reflectances of 269 matte Munsell colour chips for R, and the effects of the quantization noise for the spectral characterisation of the camera with three RGB filters and CIE illuminant A. The results are shown in Figure 3. We see on the left that, for a given amount of noise (8 bit quantisation), too few and too many principal eigenvectors (PE) cause high estimation errors. The optimal number of PE s is defined by the curve minimum. As the amount of noise increases, fewer principal eigenvectors should be used for the spectral sensitivity estimation (center) and if the noise is too high, good results cannot be obtained because excessive RMS errors (right). Examples of the estimated camera sensitivity (with RGB filters) are shown in Fig. 4 by using r =and r =2PE s and 8 bits quantisation noise. We observe that the estimation results are severely deteriorated when using too many PE s. RMS estimation error PE, quantising using 8 bits Optimal number of principal eigenvectors Optimal RMS estimation error Number of principal eigenvectors Number of bits for quantisation Number of bits for quantisation Figure 3: (left) RMS sensitivity estimation error for different number of PE s with 8 bit quantisation noise ;(center) optimal number of PE s for different levels of acquisition noise and (right) resulting RMS estimation error..2 8 bits, PE s.2 8 bits, 2 PE s Figure 4: 3 channel sensitivity estimation with (left) and 2 (right) PE s, 8 bit quantisation. The use of 269 reflectance spectra from the matte Munsell album is possible for simulation but is a severe limitation for practical applications. However it is well known that the effective

5 .2 bits, PE s.2 bits, 2 optimally chosen reflectances, PE().2 bits, 2 MacBeth reflectances, PE() Figure 5: Camera sensitivity estimation with PE s and bit quantisation noise on (left) 269 Munsell reflectances, (center) 2 optimally chosen reflectances, (right) 2 Macbeth reflectances. dimension of reflectance spectra is much lower, e.g. 8 principal components suffice to represent 99 % of the Munsell atlas in cumulative energy. In order to reduce drastically the number P of target patches, we have proposed the following iterative method for the selection of a representative set of p samples r s ; r s2 ;::: ;r p [5]. First r s is chosen as the white of maximum norm. Next, we select r s2 as maximizing the ratio of the smallest to the largest singular value of the matrix [r s r s2 ]. Further sample spectra are added according to the same rule: at the pth iteration the sample r sp, which maximizes the ratio of the smallest to the largest singular value of [r s r s2 :::r sp,r p ], is chosen. This optimized constructive approach provides satisfactory results for the spectral characterisation as shown in Figure 5, where we can compare the camera sensitivity estimation with a bit quantization noise for the full set of 269 Munsell color patches (left) and a reduced set of 2 targets chosen with the above method (center). The relative increase of the RMS estimation error is only 2 %. On the right we present the result obtained with the Macbeth ColorChecker, this colour chart being extensively used for colour calibration tasks. We notice now a slight degradation in the estimation which warns us of the importance of the color target selection. 3 Choice of the filters The quality of the acquisition system depends heavily on the number and on the choice of the color filters which will be used. We have proposed a solution where they are chosen from a set of readily available filters [4]. This choice is optimized by maximizing the orthogonality of the camera channels when applied to reflectances which is highly representative of the statistical spectral properties of the objects that are to be imaged in a particular application. The filters are selected sequentially to maximize their degree of orthogonality after projection into the vector space spanned by the most significant reflectance eigenvectors. Although this approach remains suboptimal, it avoids the heavy computation cost required with an exhaustive search. We have performed a simulation to evaluate the complete multispectral image acquisition system. We used D65 as the scanning illuminant, the Eikonix CCD camera spectral characteristics, filters chosen from a set of 37 Wratten, Hoffman, and Schott filters, and the spectral reflectances of a colour chart of 64 pure pigments used in oil painting and provided to us by the National Gallery in London [4]. The results for the case of a selection of seven filters are shown in Figure 6.

6 Transmittance, φ(λ) Spectral transmittance of the 7 filters, θ(λ) Spectral sensitivity of the 7 channels Reflectance Spectral reconstruction using 7 filters Emerald green Ultramarine ochre Mercuric Iodide Reconstructions Wavelength, λ [nm] Figure 6: Optimized selection of seven filters : (left) Spectral transmittance; (center) Spectral channel sensitivity, the numbers denote the sequence in which the filters have been chosen; (right) Reconstruction of four spectral reflectances from the camera responses using the reconstruction operator Q. 4 Spectral reflectance estimation from camera responses The spectral characteristics!() of the unfiltered camera being determined, we may now model the camera responses c k, k =:::K to an unknown reflectance R r(), using a set of K chromatic filters with known spectral transmittances k (): c k = max min r() k ()!()d: The vector c K =[c c 2 :::c K ] t representing the response to all K filters may be described using matrix notation as c K = t r; (5) where is the known N-line, K-column matrix of filter transmittances multiplied by the camera characteristics, that is =[ kn ]=[ k ( n )!( n )]. The problem of the estimation of a spectral reflectance ~r from the camera responses c K is often formulated as finding a matrix Q that reconstructs the spectrum from the measurement vector (see Figure 7): ~r = Qc K : (6).9.9 Reflectance r(λ) Emerald green Ultramarine ochre Mercuric Iodide Digital CCD camera Q Reflectance r(λ) Emerald green Ultramarine ochre Mercuric Iodide Wavelength, λ [nm] Observed spectral reflectances Multi-channel image Spectral reconstruction operator Wavelength, λ [nm] Reconstructed spectral reflectances Figure 7: Spectral reflectance reconstruction with a multi-channel image acquisition system. As other authors, we can take advantage of apriori knowledges on the spectral reflectances that are to be imaged, by assuming that the reflectance r in each pixel is a linear combination of a known set of P smooth reflectance functions: r = Ra; with R =[r r 2 :::r P ] the matrix of the P known reflectances and a =[a a 2 :::a P ] t a vector of coefficients. We have defined in

7 [7] another reconstruction operator Q that minimizes the Euclidian distance d E (r;~r) between the original spectrum r and the reconstructed spectrum ~r = Q c K : Q = RR t ( t RR t ), : (7) To evaluate the quality of spectral reflectance reconstruction we use the mean and maximum RMS errors, that is, the Euclidian distance between original and reconstructed spectral reflectances. In Table, the RMS spectral reconstruction errors using different number of filters are reported. As expected, we see that the mean reconstruction error decreases when an increasing number of filters are used. In Figure 6-(right) we show some examples of spectral reflectance reconstruction by using the seven filter selection. Number of filters Mean RMS error () Max RMS error () Table : Comparison of the RMS spectral reconstruction error for varying number of filters using the reconstruction operator Q. 5 New development and conclusion In our experiments done with the Eikonix line-scan CCD camera, the multispectral acquisition process was prohibitively slow and the noise was a constant problem. We decided to develope a new experimental multispectral camera by assembling a PCO SensiCam monochrome CCD camera ( and a CRI VariSpec Liquid Crystal Tunable Filter (LCTF) ( The camera has a single chip and a dynamic range of 2 bit. It is cooled to,2 C. Its exposure times is from ms to s, and it operates at a 2.5 MHz readout frequency. Its spectral sensitivity function, as given by the manufacturer, is reported in Figure 8-right. The camera is controlled from a PC via a PCI-board. The peak wavelength of the tunable LCTF filter can be controlled electronically from an external controller unit, or from a computer via a RS-232 interface, in the range [4 nm, 72 nm]. Figure 8-left shows the measured transmittances for 9 peak wavelength positions. The transmittances have more or less a Gaussian-like shape, except for the wide-band setups at peak wavelengths > 65 nm. There is an additional infra-red filter present, the spectral transmittances being cut at the red end of the spectrum. For peak wavelengths 44 nm, the filters have an unwanted secondary peak at long wavelengths. The average Full-Width-at-Half- Maximum (FWHM) bandwidth is not constant and varies from 5 to 8 nm. These artifacts have to be carefully compensated for in a device-independant multispectral acquisition system. The LCTF technology offers a reasonably wide field of view (7 from the normal axis) but nevertheless, the limitation in field-of-view is a parameter that has also to be treated with care for imaging applications. The tunable filter and the CCD camera are both controlled by a PC under Win-NT. This SpectraCam computer is linked via Ethernet to the campus-wide Local Area Network (LAN) and the Internet. A modular client-server software architecture has been developed. The client parts are implemented in Java and allow for interactive multispectral acquisition across the Web. In conclusion the development of a multispectral image acquisition system requires a precise control of its spectral characteristics. The characterisation of camera sensitivity, the selection of the filters and the reconstruction of the spectral reflectances from the camera responses have

8 Transmittance CRI VariSpec LCTF Wavelength [nm] CCD spectral sensitivity Sony ICX85AL CCD Image Sensor Wavelength [nm] Figure 8: (left) Spectral transmittances of the CRI VariSpec Model VIS2 LCTF filter; (right) Spectral sensitivity of the PCO SensiCam Model 37 KL camera (from been carefully studied, mostly by simulation based on a previous system. We are currently extending these approaches with the new system under development. We expect from it much easier experiments with more precise measurements and more robust results. References [] F. Schmitt,H. Maître, and Y. Wu, First progress report: tasks 2.4 (Development / procurement of basic software routines) and 3.3 (Spectrophotometric characterization of paintings) Vasari project., Tech. Rep. 2649, CEE ESPRIT II, Jan. 99. [2] Laboratoire de Recherche des musées de France, LRMF, Corot , 85 œuvres du Musée du Louvre - Analyse scientifique. CD-ROM AV 54, Collection Art & Science, 996. [3] F. Schmitt, High quality digital color images, in Proceedings of the 5th International Conference on High Technology: Imaging Science and Technology, Evolution and Promise, (Chiba, Japan), pp , Sept [4] H. Maître, F. Schmitt, J.-P. Crettez, Y. Wu, and J. Y. Hardeberg, Spectrophotometric image analysis of fine art paintings, in Proceedings of IS&T and SID s 4th Color Imaging Conference: Color Science, Systems and Applications, (Scottsdale, Arizona), pp. 5 53, Nov [5] J. Y. Hardeberg, H. Brettel, and F. Schmitt, Spectral characterisation of electronic cameras, in Electronic Imaging: Processing, Printing, and Publishing in Color, vol. 349 of SPIE Proceedings, (Zürich, Switzerland), pp. 9, May 998. [6] J. Y. Hardeberg, Acquisition and reproduction of colour images: colorimetric and multispectral approaches. Ph.D dissertation, enst, École Nationale Supérieure des Télécommunications, ENST, Paris, France, 999. [7] J. Y. Hardeberg, F. Schmitt, H. Brettel, J.-P. Crettez, and H. Maître, Multispectral image acquisition and simulation of illuminant changes, in Colour Imaging: Vision and Technology (L. W. MacDonald and R. Luo, eds.), Wiley, 999.

Multispectral image capture using a tunable filter

Multispectral image capture using a tunable filter Jon Y. Hardeberg, Francis Schmitt, and Hans Brettel Ecole Nationale Supérieure des Télécommunications, Paris, France ABSTRACT In this article we describe