A Color Model for Recognition of Apples by a Robotic Harvesting System* Duke M. BULANON*l, Takashi KATAOKA*2, Yoshinobu OTA*3,

Transcription

1 Technical Paper Journal of JSAM 64(5) : , 2002 A Color Model for Recognition of Apples by a Robotic Harvesting System* Duke M. BULANON*l, Takashi KATAOKA*2, Yoshinobu OTA*3, Tatsuo HIROMA*3 Abstract The first major task of a fruit harvesting robot is recognition of fruit. This paper presents a color model suitable for recognition of Fuji apples during harvest using a machine vision system under variable conditions. Three color models ; RGB model, rg-chromaticity method, and LRCD (Luminance and Red Color Difference) method, were tested to determine thresholds for segmentation of apple fruit from images. The decision-oretic approach was applied to three models to determine thresholds. Results showed that rg-chromaticity method was hardly influenced by different conditions and had highest recognition rate and lowest noise rate, specifically under back condition. Therefore, it was concluded that rg-chromaticity method was suitable as one of recognition methods of Fuji apples in a robotic harvesting operation. [Keywords] agricultural robot, apple, color model, fruit harvesting, image processing, machine vision T Introduction 1. Robotic fruit harvesting The apple is one of major agricultural products in Japan. About one million tons of apples are produced annually'. At present, apples are harvested manually, resulting in a high demand for labor during harvest season. However declining size of available work force and increasing labour costs have become harvesting problems. The automation of apple harvesting with a robotic system2~ that emulates human picker is a reasonable alternative considering rapid development of computer and sensor technologies and application of artificial intelligence. Harvesting robots designed to harvest apples3~, oranges4~, tomatoes5~ and or fruit have been studied in recent years. A robotic system to harvest Fuji apples6~ on orchard has also been studied. The concept of this * Presented at 59th Annual Meeting of Japanese Society of Agricultural Machinery (Niigata University), in April 2000 * 1 JSAM Student Member, United Graduate School of Agricultural Sciences, Iwate University, Morioka, Japan Tel * 2 JSAM Member, Faculty of Agriculture, Iwate University, Morioka, Japan (At present) Graduate School of Agriculture, Hokkaido University, Sapporo, Japan Tel * 3 JSAM Member, Faculty of Agriculture, Iwate University, Morioka, Japan Tel apple harvesting robot system is illustrated in figure 1. This system consists of vision system for recognition of apple fruit, robotic manipulator, gripping hand, and autonomous travelling system. The hand system has already been developed, but or components are still under studying. The proposed visual guidance system would acquire an image from an apple tree, process image to recognize and locate each fruit. The image processing would recognize fruit object, determining fruit boundary, shape, and size. The three dimensional location of fruit will be estimated based on calculated shape and size. Using estimated location of fruit, robot picks fruit using nearest distance principle. Recognizing fruit on tree is first major task of a fruit harvesting robot. While humans can recognize apple fruit so easily, it is very difficult to develop a visual system similar to human visual system. Neverless, some researchers have reported machine vision sensor applications for robotic guidance. Parrish et al.7~ used a monochrome camera, to acquire apple images, and pattern recognition method to guide robot. On or hand, Slaughter et al.8~ tried a color camera to exploit fruit trees with high contrasting colors, such as orange. A color image processing algorithm to differentiate orange within image was im-

2 124 Journal of Japanese Society of Agricultural Machinery Vol. 64, No. 5 (2002) Fig. 1 Concept of apple harvesting robot plemented. The basic approach that has been used by se researchers for recognizing fruit on a tree was to examine spectral reflectance properties of fruit and used se in an image processing technique called segmentation to differentiate fruit from or portions of tree. 2. Segmentation under outdoor environment Although some researchers have demonstrated feasibility of machine vision system for robotic guidance, special difficulties must be considered if machine vision system would be used in outdoor fields such as orchard. Most orchard operations, including harvesting, use sunlight as light source. Solar light properties are variable and cause variable reflectance of targets. Changes in condition may result in poor quality images, causing incomplete and improper object segmentation, and affecting subsequent image processing steps. One solution to this difficulty was use of an artificial. Lee et al. used artificial for a machine vision system to guide an automatic weeder9~. The system and investigated row crop were enclosed in a chamber, thus providing a constant condition. apple tree in orchard, However, in case of it is not realistic to enclose tree and machine vision system in a chamber to provide a constant. When robot moves around apple tree and harvests apple fruits, condition of apple fruit is changeable from four directions (east, west, south and north). This means that fruit portion, leaf portion, and branch portion are not easily differentiated in orchard using difference of ir brightness. However, a solution for multiple conditions should be needed to recognize apple fruit portions. Ambient conditions, such as those found in apple orchards, are technical challenges for image processing systems10). The target apple is red color Fuji apple in this study. This is most popular variety in Japan. Considering machine vision system and computer should be used to recognize apple fruit, it is assumed that image processing technique based on RGB features would be proper for recognition. Three color models were tested to differentiate apple fruit from background in this study. One method of quantifying color is by RGB model. Anor method is by transformed color model based on red and green-chromaticity11) The third method represents color as luminance and color difference. This method focused on red color difference because targeted apple has a red color. For three color models, minimum distance classifier12) was used to determine coefficient matrix for decision-oretic functions. The minimum distance classifier works when distance between means is larger compared to spread of each class with respect to its mean. Although simultaneous occurrence of large mean separations and small class spread seldom occurs, Han13) showed that proper selection of class parameters would result in desired classification. Han developed an image processing algorithm to differentiate crop canopy from soil. Furrmore, minimum distance classifier is optimum in Bayes sense if pattern classes are Gaussian and all classes are equally likely to occur. 3. Objectives The goal of this study was to develop apple recognition system for robotic apple harvesting. Since this system will be used in an outdoor environment, recognition algorithm must be able to adapt to changing conditions and directions ; front, shade and back in sunny, and cloudy. Even if images are taken from same viewpoint, color values differ within images depending on conditions and direction. This means that optimum color model may also be different depending on image, and conditions. The objectives of this paper were 1) to develop image processing algorithms for recognition of red apples using three different color models, RGB model, red and green-chromaticity which

3 BULANON, KATAOKA, OTA, HIROMA : A Color Model for Recognition of Apples by a Robotic Harvesting System 125 was named as rg-chromaticity method in this paper, and luminance and color difference which was also named as LRCD (Luminance and Red Color Difference) method in this paper, and 2) to evaluate performance of se three color models in recognizing apples under different conditions. U Color model and decision function for segmentation 1. Color models for segmentation (1) RGB model Three color models were used to analyse sampled pixels. The first model was RGB model. In this model, R, G and B values generated by camera were used directly with intensity level ranging from o to 255. (2) rg-chromaticity method The second model was rg-chromaticity method. This method was based on chromaticity of color because chromaticity depends on hue and saturation taken toger. The chromaticity diagram was used to specify colors of three tree portions. This diagram showed color composition as a function of trichromatic coefficients12), r and g. Basically, a color can be specified by three trichromatic coefficients defined as, (1) (2) method. Since apple fruit was red, only red color difference, CR, was considered in this study. 2. Decision function for segmentation (1) Decision using linear discriminant function For three color models, decision-oretic approach14~ was applied to determine thresholds for segmentation. The decision-oretic approach is a statistical pattern recognition technique, where a discriminant function is defined that will give a certain response with one class and a different response with anor. The concept is illustrated in figure 2. As an example, two clusters were analysed, cluster A and cluster B. Two parameters, M and N, described two clusters. A parameter may be a color, size, shape, or any parameter that characterizes a cluster. The parameters of each cluster were plotted as points in a two-dimensional graph. The parameter M was x axis and parameter N was y axis. A decision function could be defined that separates two-parameter clusters and classifies clusters. A linear decision function, D, could be expressed such that, (8) where a, b and c were constants determined from graph. Graphically, as illustrated in figure 2, D (M, N) was perpendicular bisector of line connecting mean values of clusters and separated two clusters by equation (9). (9) where R, G and B represented (3) red, green and blue color intensity respectively. Only trichromatic coefficients diagram. r and g were used for rg-chromaticity (3) LRCD method The third model converted RGB data into luminance and color difference signals14). The following equations converted RGB data to luminance, Y, and red color difference, CR, blue color difference, CB and green color difference, CG. In this study, luminance was compared with red color difference and this method was named LRCD (Luminance and Red Color Difference) (4) (5) (6) (7) In this study, following three decision functions were defined ; DI : separate fruit portion and leaf portion, D2 : separate fruit portion and branch portion, D3 : separate leaf portion and branch portion. The clusters were apple fruit portion, leaf portion and branch portion, and parameters were R, G and B values on RGB model, trichromatic coefficients of r and g on rgchromaticity method and luminance and red color difference on LRCD method. The apple image was divided into fruit portion and background portion. The leaf and branch portions were both considered as background portion, thus D3 was not used in segmentation process. Figure 3 shows flowchart of this procedure. The decision functions for RGB model were

4 126 Journal of Japanese Society of Agricultural Machinery Vol. 64, No. 5 (2002) expressed by equation (10). (10) Where a, b, c, and d were constants calculated from RGB model of trained images. The decision function of RGB model was a plane because it is a three dimensional color model. The values of constants were dependent on conditions. For rg-chromaticity method, decision functions were expressed by equation (11). (11) Fig. 2 Concept of decision oretic approach Unlike decision functions of RGB model, decision function of rg-chromaticity method was a line. It was a function of two parameters, trichromatic coefficients r and g. For LRCD method, decision functions were expressed by equation (12). (12) The decision function of LRCD method was also a line, which was a function of luminance and red color difference. The minimum distance classifier12) was used to determine coefficient matrix for decisionoretic functions for three color models. The classification algorithm also used decisionoretic approach with linear discrimination method and nearest neighbor method to calculate decision functions. (2) Noise reduction using filtering technique A 3x3 median filtering technique was applied to every binary image after segmentation, and noise filtered images were used for evaluation of segmentation performance. 3. Average decision function For stable recognition of fruit portion under changeable conditions, an average matrix, M; which was average of each coefficient in four matrixes of different conditions was obtained as shown in equation (13). Fig. 3 Flowchart for segmentation of fruit portion where j was condition of (front, back, shade and cloudy ; j= 1, 2, 3 and 4). This was named average matrix. Experimental V materials and method 1. Materials and image acquisition system Color images of Fuji apples in orchard located at experimental farm of Iwate University were (13) acquired using a digital color CCD camera (Panasonic, NV-C7). The color signals from camera were transferred as a 24-bit RGB color image data (320 x 240 pixels in each color band) to IEEE

5 BULANON, KATAOKA, OTA, HIROMA : A Color Model for Recognition of Apples by a Robotic Harvesting System video capture board. More than one hundred images were acquired under different conditions shown in next section. 2. Lighting conditions of image The fruit used for imaging were randomly selected from apple orchard of Iwate University and images were taken of fruit hanging on tree under natural daylight conditions. Four conditions were investigated ; a) front, b) back, c) fruit in shade, and d) cloudy. Conditions a), b) and c) were taken under a sunny wear. In condition a), direct intensity of sunlight was at back of camera. In condition b), camera was in front direct sunlight. 3. Number of tested images The tested images were randomly selected from acquired images. The decision functions under different conditions were determined using five different apple tree images under each condition. One hundred pixels from fruit, leaf and branch portions of tree in acquired apple tree image were sampled manually based on operator recognition of parts of tree. Anor ten apple images under each condition were tested to evaluate segmentation performance by three color models and average matrix. Forty images were used in each procedure, and a total of one hundred sixty different images were obtained. 4. Evaluation of segmentation performance To clarify segmentation performance, following two equations were defined. One was rate of correctly segmented area, AR, and or was noise rate, NR, which was defined as area in image misclassified as apple portion. (14) (15) Where AS was correctly segmented area of fruit portion and A o was original area of fruit portion from image, and AN was misclassified area and Al was image area. These four area parameters were determined using Scion Image software. This software can be used to measure area, mean, centroid, perimeter of user defined regions of interest. The region of interest was selected using selection tool and n measure command computed area of selected region. W Results and discussions 1. Decision functions Figure 4 shows distributions of plotted data of fruit, leaf and branch portions, and decision functions with front conditions by RGB model, rg-chromaticity method and LRCD method, respectively.the black and gray lines represented decision functions, D1 and D2, respectively, in each figure. In RGB model, pixels of fruit portion were perfectly separated from two or portions. In rg-chromaticity method and LRCD method, some of plotted fruit portion data were located beyond decision function lines, D1 and D2. The errors in rg-chromaticity method and LRCD method were 11.4% and 11.6%, respectively, in front condition. The fruit, leaf and branch portions were well separated in all color models among tested images. This indicated that decision-oretic approach had a possibility to separate portions. Tables 1 to 3 show matrixes of decision functions in three color models, RGB model, rg-chromaticity method, and LRCD method, respectively, under different conditions. For RGB model, calculated constant matrixes had different values among four conditions. Taking constant a1 in equation (10) as an example, this constant was R intensity coefficient for discriminating between fruit and leaf. The front matrix had highest value, followed by shade, cloudy, and back. This meant that decision function must be adjusted to condition. This also indicated that a specific matrix was required for a specific condition. This trend was also observed in LRCD method. In Table 3 of LRCD method, bl, coefficient of red color difference between fruit and leaf, also changed considerably under different conditions. For RGB model and LRCD method, order of magnitude was front, cloudy, shade and back. The reason for this is that both methods were influenced by luminance. Luminance of image is dependent on source light and reflective property of object12). Therefore, if intensity of source light decreases, RGB values of image also decrease. However, in

6 128 Journal of Japanese Society of Agricultural Machinery (b) RGB model (a) Sample color image (c) rg-chromaticity rg-chromaticity difference between different This method, meant that by age values of constants 2. different Figure tion no condition large for in Table functions could analysis were and less that be calculated 2. aver- 5 shows In figure color of attributes 5(a), average condi- of fruit por- R, G and B values under front represented of with brightest. had highest R, G and was extracted, cloudy, fruit, misclassified difference of fruit G and B values trichromatic among values indicated slightly tions. had same as shown coefficients four This explained in figure why re matrixes by back trend from chromaticity conditions with R, in Table values in figure was no large changing as background, with noise. 5 (c). However, rg-chromaticity influenced and 5 (b). difference in 2. This also method was only condi- and image with a noise rate of 3.5%. A part of and red color r and g, had similar of that shade sky created by thresholding with RGB front matrix of RGB model. 74.0% of fruit portion it. The luminance branches, Figure 6 (b) was binary The front leaves, It has a ground as background. The leaves showed green and yellow colors. The middle part of fruit was ditions. and con- fruit differences followed In segmented white color. had large condition. images, apple portion was represented by black color and background portion was fruit B values between results Figure 6 shows sample images of Fuji apple single conditions. influence program Figure 6 (a) shows source color image. unaffect- of fruit on average tion. decision Recognition was as observed Image of matrixes conditions influenced ed by re values (d) LRCD method method Fig. 4 Vol. 64, No. 5 (2002) Figure was calculated 6(c) was binary front matrix. image It had highest success rate of 98.0% of apple area. The noise rate was 1.5%. The binary image by LRCD method is shown in figure 6(d). The success rate on this image was 87.0% and noise rate was 1.8%. Figure 7 shows result of back dition. condition It is easily is poorest assumed con- that back to recognize target in

7 BULANON, KATAOKA, OTA, HIROMA : A Color Model for Recognition of Apples by a Robotic Harvesting System 129 Table 1 Decision function matrixes of RGB model under different conditions Table 2 Decision function matrixes of rg-chromaticity method under different conditions Table 3 Decision function matrixes of LRCD method under different conditions image among four condition. Figure 7 (a) is source color image. There were three apples in foreground of this image, plus a number of or fruit that appear smaller in this image due to ir position in background. Since se background apples were smaller and re was a possibility that those segmented images could not be discriminated from noise, y were considered to be part of background. The binary image of RGB model is shown in figure 7 (b). The success rate for recognition of three fruit was 88.0%. However, noise rate was 24.0%. Part of leaves, branches and ground were classified as fruit pixels. This noise rate was higher compared to or noise rates and this would degrade subsequent image processing steps to estimate location of fruit. Figure 7 (c) shows binary image of rgchromaticity method. The success rate was 96.1% and noise rate was 42%. Although or fruit that appeared smaller were considered as part of background, se were correctly segmented, while se may not be good targets for robot, fact that y can be recognized as potential harvestable fruit could be useful for robot, particularly if it were to take images from multiple positions. This gives potential for stereovision that could give a graph showing three dimensional position of apples on tree. Figure 7 (d) is binary image from LRCD method. It had a success rate of 90.6% and it had a noise rate of 25.2%. Similar to binary image of RGB model under back condition, this noise rate was also high and this would affect estimation of fruit locations. 3. Evaluation of segmentation performance Table 4 summarizes averages of success and noise rates of segmentation by RGB model, rg-chromaticity method, and LRCD method. The values represented averages among tested ten images in each color model. It shows success rate, indicated by upper values, and noise rate, indicated by lower values, of three color models under four conditions. The three color models showed success rates higher than 73%. The rg-chromaticity method had highest success rate among three models under different conditions. On noise rate for RGB model and LRCD method, cloudy condition had least noise, followed by shade, front and back. The back condition for RGB model and LRCD method had highest noise rates of about 29%, but noise rates for or conditions were below 4%. It was considered that

8 130 Journal of Japanese Society of Agricultural Machinery Vol. 64, No. 5 (2002) (a) RGB model (b) rg-chromaticity method had least noise followed by cloudy, front and back, All of conditions had a noise rate below 3.3%. This was because coefficient values of r and g on rg-chromaticity method were constant in figure 5. It was summarized that results of this table indicated that rg-chromaticity method had better fruit recognition performance under changing conditions compared to RGB model and LRCD method. 4. Average decision function of rg-chromaticity method Focusing on coefficients of matrixes in table 1 to 3, variances of each coefficient among four conditions in rg-chromaticity method were evidently smaller compared to or two methods. For example, regarding with coefficients a1 in four conditions, average was and standard deviation was among four conditions in rgchromaticity method. However, averages were and , and standard deviations were and in RGB model and LRCD method, respectively. This meant that decision function matrix in rg-chromaticity method was uniform independent on conditions. The equation (16) defines average matrix on rgchromaticity method. (16) Table 5 shows performance of segmentation using average decision functions. When compared to segmentation performance by (c) LRCD method Fig. 5 Influence of condition on color attributes of fruit reason why RGB model and LRCD method had high noise rates under back condition was because intensity of back condition of se methods was lower compared to or conditions in figure 5. On noise rate for rg-chromaticity method, shade rg-chromaticity method in table 4, success rates of front condition was almost same, but rates decreased from 86.9 to 79.3%, from 90.2 to 84.0% and from 95.4 to 89.5% under or three conditions, respectively. On or hand, noise rates were also a little bit improved by this averaged decision functions involving decrease of success rates. The segmentation performance was considered acceptable because success rate was over 80% even under poorest condition ; back, and it was 90% under advantageous conditions ; front and shade s compared with decision functions of rgchromaticity method. The average matrix of rg-chromaticity method was successfully applied with four con-

9 BULANON, A Color KATAOKA, Model OTA, for Recognition HIROMA of Apples : by a Robotic Fig. 6 Segmentation (d) Binary image by LRCD method of front (a) Sample color image (c) Binary image by rg-chromaticity method Fig. 7 Segmentation System (b) Binary image by RGB model (a) Sample color image (c) Binary image by rg-chromaticity method Harvesting lighted image (b) Binary image by RGB model (d) Binary image by LRCD method of back lighted image 131

10 132 Journal of Japanese Society of Agricultural Machinery Vol. 64, No. 5 (2002) Table 4 Success and noise rates of color models Note : Upper values (%) indicate success rate, lower values (%) indicate noise rate Table 5 Success and noise rates by average matrix on rg-chromaticity method ditions tested in this study, as well as decision functions in each conditions on rgchromaticity method. Therefore, it was concluded that rg-chromaticity method was best method of three color models and its average matrix could be applied for recognition of Fuji apples in an outdoor environment. X Conclusion Three color models were evaluated for ir utility in thresholding color images for recognition of Fuji apples hanging on tree. Four different ambient conditions, front, back, shade and cloudy, were included in evaluation. Thresholding was used to segment fruit portion from or portions of tree, leaves and branches, as well as or background objects. Thresholding was done with decision functions derived from three color models ; RGB model, rg-chromaticity method, and LRCD method, using decision-oretic approach. The three models were all successful in extracting fruit portion in image. However, rgchromaticity method was most useful color model among three models because of its high recognition rate and low noise rate, specifically in back condition. The three models had acceptable performances in front, cloudy and shade. However, it was observed that RGB model and LRCD method were affected by different condition. Different decision functions were necessary for different conditions. This is because both RGB model and LRCD method are affected by luminance, and luminance is dependent on intensity of light source and reflective properties of objects12~. In case of rg-chromaticity method, luminance is decoupled from color information of image. Therefore, it deals only with color information (hue and saturation) of objects. That is why ; we could compute an average decision function that could be used under different conditions. An average decision function was used for rg-chromaticity method under four conditions because its decision functions were only slightly influenced by changing conditions. This situation was unlike RGB model and LRCD method, where a specific decision function was needed for each specific condition. Therefore, it was concluded that thresholding function used to obtain binary image for location of red Fuji apples should be based on rg-chromaticity method. Acknowledgement The authors are grateful to Professor Stephen W. Searcy of Texas A&M University, USA, for many suggestions about this study. References 1) Ministry of Agriculture, Forest, and Fisheries : Pocket Norm, Suisan Tokei,1998 2) Sang, Y.: Robotics of Fruit Harvesting, Journal of Agricultural Engineering Research, 54, , ) Grand d'esnon, A., G. Rabatel, R. Pellenc, A. Journeau, and M.J. Aldon : A Self Propelled Robot to Pick Apples, ASAE paper , ) Slaughter, D., Harrel, R.: Color Vision in Robotic Fruit Harvesting, Transactions of ASAE. 30 (4), ,1987 5) Subrata, D., Fujiura, T., Yamada, H., Hida, M., Yukawa, T., Nakao, S.: Cherry Tomato Harvesting Robot Using 3-D Vision Sensor (Part 1), Journal of JSAM, 58(4), 45-52,1996 6) Kataoka, T., Ishikawa, Y., Hiroma, T., Ota, Y., Motobayashi, K., Yaji, Y.: Hand Mechanism for Apple Harvesting Robot, Journal of JSAM, 61(1), ,1999 7) Parrish, E., Goksel, A.: Pictorial Pattern Recognition Applied to Fruit Harvesting, Transactions of ASAE, 20, , ) Slaughter, D., Barrel, R.: Discriminating Fruit for Robotic Harvest using Color in Natural Outdoor Scenes, Transactions of ASAE, 32 (2), , ) Lee, W., Slaughter, D., Giles, D.: Robotic Weed Control System for Tomatoes, Precision Agriculture, 1(1), , ) Tian, L., Slaughter, D.: Environmentally adaptive segmentation algorithm for outdoor image segmentation, Computers and Electronics in Agriculture, 21 (1998), , ) Searcy, S., Reid, J.: Machine See Red... And So Much More, Agricultural Engineering, 70 (7), 10, ) Gonzalez, R., Woods, R.: Digital Image Processing, Addison-Wiley Publishing Company, 1992

11 BULANON, KATAOKA, OTA, HIROMA : A Color Model for Recognition of Apples by a Robotic Harvesting System ) Han, Y., Hayes, J.: Soil Cover Determination Using Color Image Analysis, Transactions of ASAE, 33 (4), , ) Awcock, G., Thomas, R.: Applied Image Processing, McGraw-Hill. Inc., 1996 (Received : 22. November Question time limit : 30. November. 2002) 15) D.M. Bulanon, T. Kataoka, Y. Ota, T. Hiroma Estimating of Apple Fruit Location Using Machine Vision System for Apple Harvesting Robot, Proceeding of The XIV Memorial CIGR World Congress 2000 (CD-ROM), P , Tsukuba (Japan), 2000