Combining multispectral reflectance and fluorescence imaging for identifying bruises and stem-end/calyx regions on Golden Delicious apples

Transcription

1 Sens. & Instrumen. Food Qual. (2007) 1: DOI /s ORIGINAL PAPER Combining multispectral reflectance and fluorescence imaging for identifying bruises and stem-end/calyx regions on Golden Delicious apples Juan Xing Æ Romdhane Karoui Æ Josse De Baerdemaeker Received: 7 November 2006 / Accepted: 18 May 2007 / Published online: 23 June 2007 Ó Springer Science+Business Media, LLC 2007 Abstract As a part of the project for detecting bruises on Golden Delicious apple using vision system, the present paper shows a method that could separate the stem-end/ calyx regions from the true bruises by combining the information of hyperspectral reflectance and fluorescence images. The images were scanned between 400 nm and 1,000 nm with a hyperspectral imaging system. Different light sources were constructed for capturing the reflectance and fluorescent images. Compared to the reflectance signal, the fluorescence signatures are much less intense, so that only the fluorescence of chlorophyll waveband was further examined (i.e., 685 nm). The analysis showed that the Principal Components scores images, which were based on the reflectance images, can be used for separating the bruised areas as well as the stem-end/calyx regions from the sound apple tissues; whereas only the stem-end/calyx was able to be recognized from the fluorescence images. For the samples investigated in this study, no stem-end/ calyx regions were misrecognized as bruises; however, about 12% of bruised surfaces were misclassified as stem-end or calyx regions. All of the healthy tissues were correctly recognized as non-stem-end/calyx regions. The classification results indicated that combining multispectral J. Xing J. De Baerdemaeker Division of Mechatronics, Biostatistics and Sensors, Department of Biosystems, Catholic University of Leuven, Kasteelpark Arenberg 30, Leuven 3001, Belgium J. Xing (&) Department of Biosystems and Agricultural Engineering, Michigan State University, East Lansing, MI 48824, USA juan.xing@gmail.com R. Karoui Unité de Recherche «Typicité des Produits Alimentaires», Enita Clermont, Site de Marmilhat, BP 35, Lempdes 63370, France reflectance and fluorescence imaging may help to identify the stem-end/calyx regions from the true bruised tissue and therefore to improve the accuracy for bruise detection on Golden Delicious apples. Keywords Apple Bruise Hyperspectral imaging Reflectance Fluorescence Introduction Appearance gives the consumers the first impression about the quality of fruit. A good sorting system could separate the damaged fruits from the sound ones and consequently improve the industry s ability to meet consumer demands for fruit quality. Apart from increasing the marketability of the products, when the bad commodities are separated from the good ones in time, it may also decrease the potential for microbial infestation and enhance and/or maintains the shelf life of the fruits. Bruise is one of the most common injuries on apple and appears as brown colour on the apple skin. In some cases, simple visual inspection may suffice for the non-destructive automatic grading system in the fruit industries. Therefore, the machine vision approach has been favoured for dozens of years. Various imaging processing and feature extraction techniques have been developed to enhance the contrast between the sound and bruised surfaces to improve the classification accuracy [1 3]. Many efforts have been also made for the identification of the stem-end or calyx regions from the true surface defects [4 7]. Since both of the bruised and stemend/calyx regions have lower intensity comparing to the sound tissue in an image, they can be very likely misrecognized for a conventional vision system. Thus, developing

2 106 Sens. & Instrumen. Food Qual. (2007) 1: a cost effective and efficient automatic inspection system is still challenging. A conventional imaging system can only provide limited waveband information, which restrains its ability for detecting some surface defects. It has been reported that spectra contain huge information allowing a good determination of the quality of foods [8 10]. However, a spectrophotometer measures only a relatively small area at one time, which means that multiple measurements are required in order to obtain the spectral information of an entire sample. It leads to a time-consuming procedure, which is not practical for detecting surface defects as such. To integrate spectral and spatial information, a hyperspectral imaging system is needed. For the same reason as using a spectrophotometer, the slow inspection speed makes the hyperspectral imaging impractical to be implemented in an online system. However, it is a very helpful laboratory-based setup for selecting the efficient wavebands for a machine vision system, or the so-called multispectral imaging system, which can work as fast as required in practice. In the food quality assessment, the hyperspectral and multispectral imaging technique has been used for the inspection of poultry carcasses [11 13], defects detection or quality determination on apples and tomatoes [14 18]. In most of the hyperspectral imaging attempts for defects detection on apples, the samples were oriented manually to provide the most contrast between normal and abnormal areas. Therefore, in most of the cases, the stem-end/calyx regions are not in the field of view of a camera. Recently, with the improvement of image sensors sensitivity for plant-specific fluorescence signals, fluorescence imaging has been used for the detection of various types of strain and stress as well as for plant pathology [19, 20]. This technique has also been used as a nondestructive method to screen different development stages, growth conditions, senescence of leaves and maturity of fruits [19, 21]. Kim et al. [22] applied multispectral fluorescence images to detect faecal contamination on apple surface. Recently, Ariana et al. [23] have demonstrated that integrating the reflectance and fluorescence multispectral imaging modes is a useful tool for apple disorder classification. Thus, fluorescence imaging could be a suitable technique quality control by automated inspection and sorting as well as remote sensing in agriculture, horticulture, forestry and environmental analysis [24]. The objective of the present investigation was to assess the potential of using multiple spectral reflectance and fluorescence images to detect bruises on Golden Delicious apples and more efforts were made for identifying stemend/calyx regions from true bruises. Materials and methods Sample preparation Golden Delicious apples (n = 47) used in this study were harvested at the Centre for Fruit Cultivation in Rillaar (Belgium). The apples were free from bruises by visual inspection and carefully packed into trays to avoid bruising during the transportation from orchard to the laboratory. After arrival at the laboratory (approximately 1 day), the apples were impacted with a pendulum device with energy of about 0.11 J in the midway between the calyx and stem end in order to get a controlled bruise. The pendulum consisted of a m long arm with an aluminium impactor of spherical shape (radius of curvature of 25 mm) at its tip. After the impact, the apples were stored at room temperature (22 C) for 24 h before being measured. Hyperspectral imaging system The developed hyperspectral imaging system is shown schematically in Fig. 1. The imaging optics includes an ImSpector V10 imaging spectrograph (Spectral Imaging Ltd, Oulu, Finland) coupled with a standard C-mount zoom lens (Cosmicar H6Z810), and a 10 bits Hitachi KP- F120 monochrome camera. The ImSpector spectrograph consists of prism-grating-prism (PGP) construction, which disperses the incoming line of light into the spectral and spatial matrices and then projects them onto the charge coupled device (CCD). The spectral region of the optic system ranges from 400 nm to 1,000 nm. The resolution of the image acquisition system was 800 1,040 pixels by 10 bits, which corresponds to a spatial resolution of 0.15 mm and a spectral resolution of 0.7 nm. To avoid low signal-noise ratio, only wavelengths between 500 nm Fig 1 Schematic of the hyper-spectral imaging system

3 Sens. & Instrumen. Food Qual. (2007) 1: and 950 nm were used in this investigation. The sample translation plate was utilised to move the apples passing through the field view of the optics. For the reflectance measurement, the samples were illuminated with two linear halogen lamps (150 W). Regarding fluorescence imaging, four UV-A lamps (BLB-T5, 365 nm, Sylvania) were used to excite the fluorescence of apples. Short-pass filters (UG1, Schott Glass Technologies, Germany) were placed in front of the UV lamps housing to avoid the pseudo-fluorescence. This laboratory-based system was operated in a closed dark chamber to minimise interference from ambient light. The camera and spectrograph were used to scan the apples line-by-line as the translation plate moved the apples through the field of view of the optical system. The apples were placed on the translation plate manually with the interested region (i.e., cheek, stem-end or calyx) facing to the camera. After finishing the scans on one entire apple, the spatial-by-spectral matrices were combined to construct a three dimensional (3D) spatial and spectral data space. The scanning time for one apple depends on the integration time used for the camera and the size of the apples. The integration time was fixed to 160 ms and 5 s for the reflectance hyperspectral imaging data acquisition and fluorescence measurements, respectively. Data processing and analysis The image capture program was developed in Labview v7.1 (National Instrument Corporation, USA); while the other processing program was developed in Matlab (The MathWorks Inc., USA). To reduce the noise and amount of data for calculation, the spatial-spectral data from each scan were averaged by 10 and 5 neighbouring pixels in the spectral and lateral spatial dimension, respectively. More details about the date pre-processing can be found in the study of Xing et al. [25]. Reflectance images The standard reference of 99% reflectance (Spectralon, Labsphere Inc.) was used for reflectance calibration. The dark current was measured by closing the chamber, turning off all light sources and covering the lens with a black cap. The reflectance calibration was performed with the following equation: R ¼ R im R dark R ref R dark ð1þ where R im is the intensity of an image; R ref is the intensity of the standard reference spectralon 99%; and R dark is the intensity of the dark image. Fluorescence images After finishing the reflectance scanning entirely, the apple was transported back to the origin position by the translation plate and prepared for the fluorescence scanning. Since fluorescence signatures are much less intense than reflectance signatures, longer integration time and larger slit width were used for the fluorescence image acquisition. Due to the absence of a proper fluorescence reference, the intensity values of the fluorescence images were used for the later analysis and processing. When the fluorescence image was taken, the UV lamps were turned on and the halogen lamps were turned off. Principal Component Analysis Principal Component Analysis (PCA) was applied to the reflectance spectra in order to investigate differences in the spectra of different apples. The purpose of this technique is to obtain an overview of all the information in the data set. In PCA, orthogonal directions in variable space describing the variation are found. In this way, a new set of fewer coordinate axes called principal components (PCs) is generated. Score is the estimated value for a principal component (PC). Each spectrum has a score along each PC. This statistical treatment makes it possible to draw similarity maps of the samples and to get loadings. Results and discussion The reflectance spectra of a sound and a bruise apple (Fig. 2a) exhibited two maxima located around 570 and 730 nm, varying slightly from apple to apple. As a result of the small differences and the complexity of the spectra, univariate analysis was not really appropriate to statistically analyse the data sets. Multivariate statistical techniques such as PCA make it possible to extract information related to the origin of the differences between spectra. The differences between the fluorescence spectra of two different apples following excitation at 365 nm were extremely small throughout the whole wavelength region (Fig. 2b). The spectra in the range between 400 nm and 700 nm were studied, because the signatures above 700 nm could be due to the harmonics of the light source. As shown in Fig. 2b, the fluorescence emission spectrum of green apple was characterized by three bands with maxima at 685 nm (red) and two shoulders around 440 nm (blue) and 520 nm (green) [24]. The red fluorescence is emitted by the chlorophylls, whereas the blue and green fluorescence is emitted by cinnamic acids (with ferulic acid as major substance) and other phenolics covalently bound to the cell walls [24]. As the fluorescence signal is relative weak, only

4 108 Sens. & Instrumen. Food Qual. (2007) 1: (a) nm 678 nm 728 nm 850 nm R e flectance (%) Bruised Sound Lo adings PC1 PC2 PC3 (b) Fl uo rescence I ntensity Wavelength (nm) nm 520 nm Bruised Sound 685 nm Wavelength (nm) Fig 2 Representative reflectance spectra (a) and fluorescence spectra following excitation at 365 nm (b) recorded on the investigated Golden Delicious apples the image captured at 685 nm, corresponding to the maximum emission wavelength of chlorophyll, was investigated in this study. Principal Components Analysis on the reflectance images Principal Component Analysis was applied to the reflectance image recorded on apples, on the one hand, to help visualising the hyperspectral data and, on the other hand, to determine the optimal wavebands for a multispectral imaging system. At the first step, the PCA was performed on the full wavelength region for the reflectance images. Loadings associated with the PCs provide the characteristic wavebands that may be used to discriminate between samples. As shown in Fig. 3, the loading 1 associated with PC1, presents a positive peak at 728 nm, while the loading 2 exhibits a positive peak at 728 nm and a Wavelength (nm) Fig 3 Loading plots corresponding to principal component 1 (PC1), principal component 2 (PC2) and principal component 3 (PC3) of reflectance image spectra recorded on investigated apples negative one at 678 nm. The loading 3 shows a positive peak at 678 nm, two negative peaks at 558 and 728 nm and a shoulder at 850 nm. As explained in the introduction, multiple spectral imaging is more preferred to the hyperspectral imaging in practice. Therefore, the later work was focused on few effective wavebands, i.e., 558, 678, 728 and 850 nm. The PCA procedure was thereafter repeated only with these selected four significant wavebands. Indeed, by replacing the reflectance value of each pixel on an apple with the scores value of one certain principal component, a new apple image could be created. The new image is then called PCA scores image in order to distinguish it from the normal reflectance or intensity image. If the first PC scores were used, the resultant image was then called PC1 scores image. If the second PC scores were taken, it was called PC2 scores image; and so on. The advantage of using PCs scores image might be found in displaying the variation information from multiple wavebands. As demonstrated in Fig. 4, PC1 scores image mainly displays the grey value information of the apple. The PC2 or PC3 scores image appears to provide the best discrimination between the sound and bruised tissue which confirms the results obtained from the loading plots. Figure 3 showed that the loading 1 exhibited similar trend as the spectrum of the sample and did not show any clear characteristic feature. However, the loadings of the second and third components presented maxima or minima at some characteristic wavebands (e.g., in the brown wavelength region and the chlorophyll maxima). Bruises can be seen clearly either in the PC2 scores image as illustrated in Fig. 4a; or in the PC3 scores image (Fig. 4b). For the sound apple (Fig. 4c) there is no obvious suspect region(s) observed in the PC2 or PC3 scores image. Though PC2 or

5 Sens. & Instrumen. Food Qual. (2007) 1: Fig 4 Principal Components scores image of a sound and two bruised apples; PCA was performed on the reflectance images; the arrows are pointing at the bruised areas on apples. (a) Bruise can be detected on the PC2 scores image. (b) Bruise can be detected on the PC3 scores image. (c) Scores images of a sound apple PC3 could be considered powerful tool for detecting the bruises, it is difficult to determine which of them (PC2 or PC3) should be used on the automatic sorting line. Therefore, a combination algorithm was developed to take into account the information contained in PC2 and PC3 scores images. Since the PC2 and PC3 score values have different scales, the images were first normalised before carrying out the combination procedure. The normalisation was calculated by using the following equation: I i;norm ¼ I i minði i Þ maxði i Þ minði i Þ ð2þ where I i is the original of PC i scores image; min(i i ) is the minimum value of PC i scores image; max(i i ) is the maximum value of PC i scores image; and I i,norm is the normalised PC i scores image. The combination was done by the following equation: I 23 ¼ I 3;norm I 2;norm I 3;norm þ I 2;norm ð3þ where: I 23 is the combination of PC2 and PC3 normalised scores image; I 2,norm is the normalised PC2 scores image and I 3,norm is the normalised PC3 scores image. The example of the combination images are shown in the right column of Fig. 5. It can be seen that most of the important features of the original scores images are preserved in the resultant combination images. For simplicity, the combination of the normalised PC2 and PC3 scores image will be referred as processed reflectance images in the text later. Fluorescence images As explained herein above, only the fluorescence image around 685 nm was investigated in this paper. Normally, in bruised tissue, some cellular structures are damaged, such as plasmallema and chloroplast, leading to the leakage of the chlorophyll. Thus, it was expected to observe some differences between the sound and bruised tissue by monitoring the fluorescence of chlorophyll. By integrating the reflectance and chlorophyll fluorescence information, Ariana et al. [23] obtained a good classification accuracy (>95%) for recognizing the disorders of apples (stem-end/calyx regions were not considered). However, the contrast between the sound and bruised tissue in the fluorescence images captured with this setup is not very clear. An explanation could arise from the low signal intensity, inducing some misclassification between the damaged and intact area. The unclear contrast between the sound and bruised tissue could be also explained by the small difference in the amount of chlorophyll between the sound and bruised tissue since they were studied after 24 h. A high difference can be observed between the cheek surface and the stem-end or the calyx region. Since the stem-end/calyx is often confused with the surface defects in the reflectance images, the fluorescence images could be used with the reflectance

6 110 Sens. & Instrumen. Food Qual. (2007) 1: images together for separating stem-end/calyx from the true surface defects. Comparison of the thresholding images from the reflectance and fluorescence imaging Fig 5 Example of the normalized combination PC2 and PC3 scores images recorded on four apples Since the slit width varied with the reflectance and fluorescence measurements, the images had different resolution in the translation plate moving direction. In order to make the both images (reflectance and fluorescence) comparable, the images were first cropped along the top, bottom, left and right edge of the apple and then resized to a same resolution by using bicubic interpolation method in Matlab software. All the image processing was done on the resized images. Figure 6 depicted how the information of the processed reflectance images and the fluorescence images were integrated. The first two rows correspond to the images taken on the apple cheeks and the two rows below were taken with the presence of the stem-end or calyx regions. Figure 6a present the results obtained from the processed reflectance images and Fig. 6b illustrate the corresponding binary images after applying Otsu s automatic thresholding in Matlab. Apparently, the bruised region and the Fig 6 Comparison of the thresholding images from the reflectance and fluorescence imaging recorded on investigated apples. (a) and (c) shows the reflectance and fluorescence images of the same 16 apples, respectively. (a) Combination of PC2 and PC3 scores images. (b) Thresholding images for the combination images. (c) Fluorescence images at 685 nm. (d) Thresholding images of the fluorescence images

7 Sens. & Instrumen. Food Qual. (2007) 1: Table 1 Classification results for identification of stem-end/calyx from cheek tissues stem-end/calyx region exhibited similar properties in the reflectance images, which allow remaining them in the thresholding images. Figure 6c showed the image obtained using fluorescence of the same investigated apples. The corresponding thresholding images were presented in Fig. 6d. As it can be seen, only the stem-end/calyx regions had zero values (black), while all the cheek flesh, no matter bruised or sound, presented one value (white) in Fig. 6d. Therefore, only the regions which had one value (white) in both of the images will be considered as true bruises. By comparing the thresholding images in Fig. 6b and d, the stem-end/calyx can be identified from the true bruise defects on the apple surface. In terms of identification of stem-end/calyx regions from the cheek tissue, the classification results are shown in Table 1. No healthy cheek was misclassified as stem-end/ calyx regions. All of the stem-end/calyx regions are recognised correctly. About 88% of the bruised apples were separated from the stem-end/calyx regions. Conclusion Classified as Stem-end/calyx Not stem-end/calyx Healthy (n a = 30) 0 30 (100%) Bruised (n = 17) 2 15 (88.2%) Calyx b (n = 28) 28 (100%) 0 Stem-end b (n = 24) 24 (100%) 0 a n is the number of samples b The apples used for calyx and stem-end images taken were picked up randomly from the investigated Healthy and Bruised apples A hyperspectral imaging technique was developed to detect bruises on Golden Delicious apples. Four wavebands centred at 558, 678, 728 and 850 nm could potentially be implemented in multispectral reflectance imaging systems for detecting bruises on Golden Delicious apples. Principal Components scores images (based on selected four wavebands), which were made from the reflectance images, could be used for separating both the bruised areas and the stem-end/calyx from the sound apple tissues; whereas only the stem-end/calyx was able to be recognized from the fluorescence images. By comparing the threshold image of processed reflectance images and fluorescence images, the stem-end/calyx regions was separated from the true bruises. For the samples used in this study, no stem-end/calyx regions were misrecognized as bruises; however, about 12% bruised surfaces were misclassified as stem-end or calyx regions. All of the healthy tissues were correctly recognized as non-stem-end/calyx regions. The classification results indicated that the fluorescence images could be used as a complementary method with the reflectance imaging system to identify the stem-end/calyx from the true bruised areas. A better camera which may sense the fluorescence signal better is needed in future work to improve the performance of the system. This study only considered bruises of 24 h old. The accuracy for separating the true bruises from the stem-end/calyx region may change with the age of bruises presented on apples. Further investigations are necessary to see the fluorescence contrast between the stem-end/calyx and cheek tissue (whether healthy or bruised) at different age of apple samples. Acknowledgements The authors gratefully acknowledge the financial support of the Research Council of Catholic University of Leuven. References 1. F. Geoola, F. Geoola, U.M. Peiper, J. Agric. Eng. Res. 58, (1994) 2. V. Leemans, H. Magein, M.-F. Destain, Comput. Electron. Agric. 20, (1998) 3. M.A. Shahin, E.W. Tollner, R.W. McClendon, H.R. Arabnia, Trans. ASAE 45, (2002) 4. R.R. Wolfe, W.E. Sandlerm, Trans. ASAE 28, (1985) 5. Q. Yang, Comput. Electron. Agric. 8, (1993) 6. X. Cheng, Y. Tao, Y.R. Chen, Y. Luo, Trans. ASAE 46, (2003) 7. Y. Ying, H. Jing, Y. Tao, Zhang N., Trans. ASAE, 46, (2003) 8. J. Kim, A. Mowat, P. Poole, N. Kasabov, Chemometrics Intell. Lab. Syst. 51, (2000) 9. S. Sivakesava, J. Irudayaraj, Appl. Eng. Agric. 16(5), (2000) 10. D. Wang, S. Ram, F.E. Dowell, Trans. ASAE 45(6), (2002) 11. Y. Tao, J. Shao, K. Skeeles, Y.R. Chen, Trans. ASAE 43(2), (2000) 12. K. Chao, Y.R. Chen, W.R. Hruschka, B. Park, Trans. ASAE 17(1), (2001) 13. C. Hsieh, Y.R. Chen, B.P. Dey, D.E. Chan, Trans. ASAE 45(2), (2002) 14. P.M. Mehl, K. Chao, M. Kim, Y.R. Chen, Trans. ASAE 18(2), (2002) 15. A. Peirs, N. Scheerlinck, J. De Baerdemaeker, B.M. Nicolaï, in Proceedings of EurAgEng 2002, Budapest, Hungary, Paper number: 02-PH-028 (2002) 16. G. Polder, G.W.A.M. Van der Heijden, I.T. Young, Trans. ASAE 45(4), (2002) 17. M.S. Kim, A.M. Lefcourt, K. Chao, Y.R. Chen, I. Kim, D.E. Chan, Trans. ASAE 45(6), (2002) 18. I. Kavdir, D.E. Guyer, Trans. ASAE 45(6), (2002) 19. W.B. Herppich, in The Proceedings of Fruit, nut and Vegetable Production Engineering, pp (2001) 20. J. Soukupová, S. Smatanová, A. Jegorov, L. Nedbal, M. Trtílek, in The Proceedings of Fruit, nut and Vegetable Production Engineering, pp (2001)

8 112 Sens. & Instrumen. Food Qual. (2007) 1: A.M. Nazir, P. Rufino, M.B. Randolph, Chlorophyll fluorescence and whole fruit senescene in Golden Delicious apple. [accessed 15 October 2003] 22. M.S. Kim, A.M. Lefcourt, Y.R. Chen, I. Kim, D.E. Chan, K. Chao, Trans. ASAE 45(6), (2002) 23. D.P. Ariana, B.P. Shrestha, D.E. Guyer, An ASAE Meeting Paper NO , Nevada, USA (2003) 24. C. Buschmann, G. Langsdorf, H.K. Lichtenthaler, Photosynthetica 38(4), (2000) 25. J. Xing, C. Bravo, P. Jancsok, H. Ramon, J. De Baerdemaeker, Biosyst. Eng. 90(1), (2005)