Modern Physics Letters B Vol. 31, Nos. 19 21 (2017) 1740039 (7 pages) c World Scientific Publishing Company DOI: 10.1142/S0217984917400395 Method to acquire regions of fruit, branch and leaf from image of red apple in orchard Jidong Lv School of Information Science and Engineering, Changzhou University, Gehu Road, Changzhou 213164, China vveaglevv@163.com Liming Xu Department of Equipment Engineering, Jiangsu Urban and Rural Construction College, Heyu Road, Changzhou 213147, China lxccdx@163.com Received 7 September 2016 Published 12 April 2017 This work proposed a method to acquire regions of fruit, branch and leaf from red apple image in orchard. To acquire fruit image, R-G image was extracted from the RGB image for corrosive working, hole filling, subregion removal, expansive working and opening operation in order. Finally, fruit image was acquired by threshold segmentation. To acquire leaf image, fruit image was subtracted from RGB image before extracting 2G-R-B image. Then, leaf image was acquired by subregion removal and threshold segmentation. To acquire branch image, dynamic threshold segmentation was conducted in the R-G image. Then, the segmented image was added to fruit image to acquire adding fruit image which was subtracted from RGB image with leaf image. Finally, branch image was acquired by opening operation, subregion removal and threshold segmentation after extracting the R-G image from the subtracting image. Compared with previous methods, more complete image of fruit, leaf and branch can be acquired from red apple image with this method. Keywords: Apple; image segmentation; image processing; threshold segmentation. 1. Introduction Harvesting robot based on machine vision has primary task of acquiring regional image of fruit, leaf and branch. Various scholars 1 3 developed researches on the identification of fruit object. However, these methods do not touch the detection of Corresponding author. 1740039-1
J. Lv & L. Xu branch, leaf and other obstacles. Moreover, the incomplete information about fruit region may affect the precision in subsequent identification of fruit. When acquiring branch region, Ji et al. proposed a method of iterative threshold segmentation for branch based on adaptive histogram equalization with limited contrast ratio to realize the stripping of branch region. 4 According to aberration components of R-B and G-B in RGB color system, Cai et al. removed background by adaptive threshold segmentation with maximum between cluster variance. In addition, they segmented branch region from citrus image by removing bright region and dark region with gray threshold segmentation. 5 The above methods mainly involve the extraction of branch region. In order to acquire multi-regional information about fruit and leaf, Jia et al. identified apple, leaf and branch by optimized radial basis function (RBF) with neural network based on K-means cluster segmentation, genetic algorithm (GA) and least mean square (LMS). 6 This method has high training success rate and accuracy, but the accuracy improvement is time consuming. This work established a process flow with effective combination of basic image processing (with low complexity) and different characteristics of target regions based on above research. Consequently, complete region of fruit, branch and leaf can be acquired. 2. Materials and Methods The color difference between fruit and background is significant with R-G factors as color characteristics in RGB color space. After removing fruit in image, the color difference between leaf and background is significant with 2G-R-B factors as color characteristics. After removing fruit and leaf in image, the color difference between branch and background is also significant with R-G factors as color characteristics. These color characteristics can be used as segmentation vectors to acquire target region. 2.1. Acquisition method Erosion, dilation and opening operations In this work, the region of fruit, branch and leaf is acquired with aberration image, the non-target image in which is fragmented. At this time, the erosion operation is always effective to weaken or eliminate the most fragments of non-target image. Erosion can be described with Eq. (1). In order to maximize the integrity of target image, converse dilation working is always conducted after erosion. Relative to erosion, dilation operation is to expand boundary points of object outward. Dilation can be described with Eq. (2). Assumed that A and B are sets of Euclidean space. Figures 1 and 2 demonstrate the process that structural element B either erodes or expands binary image A. Obviously, the erosion of A by B makes C smaller in Fig. 1, and the dilation of A by B makes C larger in Fig. 2. Opening can remove isolated points, burr and pores and smooth the boundary of large object without obvious change of area. Opening can be described with Eq. (3). Opening forms after eroding and expanding the image. 1740039-2
Method to acquire regions of fruit, branch and leaf from image of red apple in orchard A B C Fig. 1. Erosion operation. A Fig. 2. B Dilation operation. A B = {x E N x + b A}, b B, (1) A B = {c E N c = a + b}, a A, b B, (2) Subregion removal and hole filling S = A B = (A B) B. (3) Some non-target regions in aberration image should be removed. These regions are always smaller than target image, so we marked the connected regions by 8- neighborhood tracer method. Then, small regions with pixel amount smaller than a threshold according to statistics were removed to remain target region. Apples are not in homogeneous color, so holes may occur in fruit region. However, we can directly fill these holes by flood fill algorithm which is suitable for the fill of internal defined region as a regional fill method. C Threshold segmentation Fixed threshold segmentation has good effects when segmenting image with obvious gray thresholds in target region and background region. However, for image with complex background, dynamic threshold segmentation has better effects compared with fixed threshold segmentation. As dynamic threshold segmentation with good performance, Otsu method can seek dynamic threshold by calculating interclass variance between target and background in image. Specifically, let the gray range of image I be {0, 1,..., m 1}, P i means that the probability of gray level is i. Let T be the threshold segmenting prospect and background, the image can be divided into target class A 0 = {0, 1,..., T } and background class A 1 = {T +1, T +2,..., m 1}. In the image, target class accounts for w 0 with average gray level of u 0, while background class for w 1 with average gray level of u 1. The total average gray level 1740039-3
J. Lv & L. Xu of image is u T. Therefore, w 0 = w 1 = T P i, (4) i=0 m 1 i=t +1 P i = 1 w 0, (5) u 0 = u 1 = T i=0 m 1 i=t +1 ip i w 0, (6) ip i w 1, (7) u T = w 0 u 0 + w 1 u 1. (8) When T traversed from the smallest to the largest gray level makes Eq. (9) for variance the largest, T is the optimal threshold in segmentation. 2.2. Execution flow σ 2 = w 0 w 1 (u 0 u 1 ) 2. (9) Figure 3 shows the process of acquiring fruit and leaf region from the image of red apple in orchard: (1) The image collected by visual sensor was used to extract regional information about target object. (2) In RGB color space, red apple has significant color difference with the background with R-G color factors as color characteristics. Then, R-G calculation was conducted to acquire aberration image according to RGB three-channel image collected in (1). Extraction of R-G image Original mage Segmentation by dynamic threshold Extraction of fruit image Subtraction Addition Extraction of leaf image Subtraction Extraction of branch image Fig. 3. General flowchart of the presented method. 1740039-4
Method to acquire regions of fruit, branch and leaf from image of red apple in orchard (3) The steps to acquire fruit image based on R-G aberration image are as follows. (a) In order to eliminate redundant branches in aberration image, erosive working was conducted to R-G aberration image based on the disc structural element with radius of 5. (b) There is a hole between the calyx at the end of apple and the fruit body because of large color difference. Without processing, fruit region may be deficient and incomplete. Therefore, floodfill algorithm was used to fill the hole in the eroded R-G aberration image. (c) After hole filling, there are still noisy points or blocks besides fruit. In order to eliminate these noisy points or blocks, all the pixels with gray level smaller than 20 in R-G aberration images were set as 0. Then, connected regions in image were marked and counted by 8-neighborhood trace method to remove subregions with total pixels less than 2000. (d) Reverse dilation working was conducted against the erosive working in Step (b). Therefore, the integrity of fruit image can be maximized after removing subregions based on disc structural elements with radius of 5. (e) There are still non-target objects on the edge of target region in expanded R-G aberration image. Therefore, opening was conducted based on disc structural elements with radius of 10 to weaken these non-target objects. (f) Fruit image was acquired from R-G aberration image by segmenting RGB original image with gray level of 0 as threshold after opening. (4) Fruit image was subtracted from RGB original image to acquire subtracted image. (5) The steps to acquire leaf image based on the subtracted image are as follows. (a) With 2G-R-B color factors as color characteristics, green leaf is significantly different from the background. Therefore, 2G-R-B operation was conducted to subtracted image to acquire aberration image. (b) In order to eliminate abundant noisy points or blocks in aberration image, connected regions in 2G-R-B aberration image were marked and counted by 8-neighborhood trace method. Then, subregions, with pixel smaller than 500, were removed. (c) After subregion removal, gray level of 0 in 2G-R-B image was taken as threshold. Then, RGB original image was segmented to acquire leaf image. (6) Small fruits in distance were not included in fruit image acquired by Step (3) (fruits far away from visual sensor look small, and harvesting robot based on machine vision can only pick large fruits near the sensor), while all the fruit image should be subtracted from RGB original image. Therefore, fruit image segmented with dynamic threshold was acquired from R-G aberration image according to Step (2). (7) In the fruit image acquired by dynamic threshold segmentation, fruit region may be missing. In order to supplement the missing fruit region, the image was added to the fruit image acquired in Step (3) to acquire addition image. 1740039-5
J. Lv & L. Xu (8) Addition image and leaf image were subtracted from RGB original image to acquire subtracted image. (9) The steps to acquire branch image based on subtracted image are as follows. (a) With R-G color factors as color characteristics, branch is significantly different from the background. R-G operation was conducted to subtracted image again to acquire aberration image. (b) In order to eliminate redundant noisy points or blocks in R-G aberration image, opening was conducted to R-G aberration image based on disc structural element with radius of 2. (c) Connected regions in R-G aberration image were marked and counted by 8-neighborhood trace method. Then, subregions with pixel smaller than threshold were removed. (d) After subregion removal, gray level of 0 in 2G-R-B image was taken as threshold. Then, RGB original image was segmented to acquire leaf image. 3. Experimental Results Above flows were applied in the test of fruit, branch and leaf region acquisition to verify validity (Apple variety was Fuji). Figure 4 shows the test effects. In order to further verify the effectiveness of the presented method, we compared the results with those by K-means based on a color channel in Lab color space and OTSU based on R-G image. This work used the evaluation index, relatively ultimate measurement accuracy (RUMA). The smaller the average value of RUMA is, the better the effect of this method. RUMA = Real area of target Area by segmentation Real area of target 100%. (10) Table 1 shows average RUMA results (120 apple images) by different segmentation methods. It can be seen that the average RUMA of fruit, branch and leaf are 5.2%, 10% and 13.9%, respectively. Compared with those by K-means, the average RUMA decreases by 21.93%, 34.34% and 16.6%, respectively. Compared with (a) Original image (Frontlighting,Backlighting, Complex background) (b) Fruit image (c) Branch image (d) Leaf image Fig. 4. Effect picture. 1740039-6
Method to acquire regions of fruit, branch and leaf from image of red apple in orchard Table 1. RUMA of different methods. RUMA (%) Methods Fruit Branch Leaf Otsu 17.10 30 14.9 K-means 27.13 44.34 30.5 The presented method 5.2 10 13.9 those by OTSU, the average RUMA decreases by 11.9%, 20% and 1%. Therefore, the presented method has better effects with superior RUMA. 4. Conclusions A method was proposed to acquire fruit, branch and leaf region from the image of red apple in orchard. The experiment shows that the effects of method to acquire fruit, branch and leaf in this work are much better. The average RUMA of fruit, branch and leaf are 5.2%, 10% and 13.9%, respectively. Acknowledgments This work was partly supported by Natural Science Foundation of Jiangsu Province with Grant No. BK20140266; Natural Science Research Program for Higher Education in Jiangsu Province with Grant No. 14KJB210001and NSFC with Grant No. 61501060. References 1. H. Ejvan et al., Biosyst. Eng. 86 (2003) 2. 2. D. M. Bulanon, T. F. Burks and V. Alchanatis, Trans. ASABE 52 (2008) 1. 3. Y. Song et al., Biosyst. Eng. 118 (2014) 1. 4. W. Ji et al., Trans. Chinese Soc. Agric. Mach. 45 (2014) 4. 5. J. R. Cai et al., Trans. Chinese Soc. Agric. Mach. 40 (2009) 11. 6. W. K. Jia et al., Trans. Chinese Soc. Agric. Eng. 31 (2015) 18. 1740039-7