Measurement and Evaluation of Ripening Process of Immature Tomato with Correlation Image Sensor and Ringview Optical System

Transcription

1 Proceedings of the SICE Annual Conference 2018 September 11-14, 2018, Nara, Japan Measurement and Evaluation of Ripening Process of Immature Tomato with Correlation Image Sensor and Ringview Optical System Mitsuru Baba 1 and Ikusaburo Kurimoto 1 1 National Institute of Technology, Kisarazu College Advanced Courses and Dept. of Control and Information Course Kiyomidai Higashi Kisarazu, Chiba, Japan (Tel: ; sdj17b11@kisarazu.kosen-ac.jp) Abstract: In the plant factory, there is a problem that agricultural workers harvest at ill-harvesting time of tomatoes because they evaluate maturity by experience and intuition. In order to solve this problem, from technical standpoint, the maturity of fruits is evaluated using an imaging device capable of measuring the spectral reflectance of an object. However, there are problems such as being affected by sunlight and object shape. In this study, we considered whether maturity of tomato can be predicted by using reflectance of tomato which is obtained by irradiating with light having wavelengths of 430 nm, 570 nm, and 680 nm with an imaging system which comprises the Correlation Image Sensor and the Ringview Optical System. Keywords: image processing, maturity, plant factory, reflection spectrum, tomato 1. INTRODUCTION Plant factories are attracting attention because they perform advanced environmental control using airconditioning equipment and sensors which is very efficient in performing annual planning from sowing of plants to shipping adjustment[1]. Those systems are efficient in the automation related to environmental control and harvesting. However, there are problems that growth status and quality evaluation as they are still dependent on the experience and intuition of agricultural workers. In order to solve this problem, estimation of the internal structure of plants, types of crops color, minor constituents, and moisture condition are carried out by measuring the spectral reflection, transmission and absorption characteristics by using the spectral characteristics of plants. Gopal et al. reported that the absorption at the wavelength of nm of immature tomatoes was high with handheld visible and near-infrared instrument( nm)[2]. Kaveh et al. reported that full-ripe tomatoes have high reflectance at the wavelength of nm by measuring the spectral reflection of tomatoes of maturity of the third and the fourth shown in the OECD color chart[3]. However, these imaging devices to measure a reflectance are affected by an object shape and a measurement environment such as presence of sunlight. In addition, since the imaging targets are not equidistant for each pixel, obtained reflectance distribution is affected by an imaging distance and an orientation of the object surface. In this study, we considered that it is possible to evaluate the number of ripe days using imaging system which comprises the Correlation Image Sensor and the Ringview Optical System, and also the captured reflectance that is not affected by measurement environment and object shape. Mitsuru Baba is the presenter of this paper. 2. PROPOSED METHODS 2.1. Correlation Image Sensor The Correlation Image Sensor is a semiconductor device which operates the procedure called correlation detection and lock-in detection in two dimensional parallel using light with a fluctuation of light and dark[4]. Normal image sensors cannot be read out because time fluctuation of a frequency higher than the scanning period. In addition, a three-phase sinusoidal wave as a reference signal is supplied by the Ringview Optical System, and the Correlation Image Sensor outputs correlation image which is taken from the time correlation between a light irradiated by amplitude-modulated illuminations and a rejected light of irradiated on the object. The Correlation Image Sensor outputs the time correlation value between the light intensity incident on each pixel I i,j (t) and the reference signal v k (t)(k = 1, 2, 3) which is input as an analog voltage signal in common to all the pixels. The operation is expressed by the following equation (2). g k i,j = v(t) = 1 3 t t T 3 v k (t) (1) k=1 I i,j (τ) v k(τ) v(τ) dτ + 1 t I i,j (τ)dτ v o 3 t T (2) T is 1 frame time, and v 0 is the equivalent thermal potential[4]. The Correlation Image Sensor is Brookman- BT130AM03/04 high-speed CMOS image sensor (Richo Elemex Corporation), effective pixels are (1 pixel: 5.6 µm 5.6 µm), standard C mount, correlation frame frequency is 128 fps (standard: 30fps) and a conventional CCD image frame frequency is max 2000 fps (standard: 100fps). The Correlation Image PR0001/ SICE 1245

2 Sensor is connected to a PC(OS: Windows 7, CPU: Intel (R) Core (TM) i CPU@3.20 GHz, RAM: 16.0 GHz) via USB3.0, and perform image processing using an exclusive software developed by Richo Elemex Corporation. Also, in connection with the Ringview Optical System, signal synchronization and reference signal transmission are carried out using a LAN cable Ringview Optical System The Ringview Optical System is a device which irradiates a target object with the light obtained by sine wave intensity modulation of the phase of each light source in synchronization with the signal of the Correlation Image Sensor[5]. This optical system arranges amplitude-modulated illuminations which irradiate an arbitrary wavelength with the light source of radius R on a point symmetry. The combination of the Ringview Optical System and the Correlation Image Sensor can detect a feature of objects which have the zenith angle and the azimuth angle of the object surface. In addition, the normal vector of the object surface from which the influence of the imaging distance H and the direction of the object has been removed can be calculated from the mathematical equation, and the reflectance distribution peculiar to the object can be measured. The reflectance ρ i,j of each pixel in row i and row j is obtained from the following equation, assuming that the average intensity of the correlation image is E i,j, the amplitude is F i,j, the zenith angle is Θ i,j, and the average incident light amount is A 0, ψ = tan 1 ( R H ). ρ i,j = E i,j 3A 0 cos Θ i,j cos ψ (3) Fig.1 shows the result of shooting a sphere which is made of Styrofoam by illuminating the wavelength 680 nm with the imaging system. The Ringview Optical System used in this study is Fig. 1 Results of shooting and calculating sphere object of Styrofoam. an amplitude-modulated illumination in which eight substrates with 8 8 LEDs are arranged in a ring shape so that the light source of radius R = 0.25 m illumination is used to illuminate each phase with π 4 (Fig.2). A reference signal is an output from the Correlation Image Sensor to the FPGA(Altera Corporation, DE0-Nano Development Board) which controls pulse-width modulation(pwm) of a LED driver, and the intensity modulated sinusoidal wave signal is supplied to the amplitude modulation illumination. Fig. 2 Geometry of imaging device with combination of Correlation Image Sensor and Ringview Optical System Samples of immature tomato In this study, twelve immature tomatoes(tomato variety: Red-Ole) were used as samples which were judged as having the same maturity level by visual confirmation harvested in the plant factory, Center of Environment, Health and Field Sciences, Chiba University on February 4th, Samples of tomato were stored in the nolighting space, temperature of 21 C, humidity of 71% from February 4th to February 14th, Spectral reflectance measurement of tomato In order to evaluate the maturity, a reflectance of the side face of tomato was measured using an imaging system combination of the Correlation Image Sensor and the Ringview Optical System once in a day at 9:00 AM JST from February 5th to February 14th, The sample of tomatoes was assigned numbers from 1 to 12 as identification numbers(label), and irradiated with different wavelengths of light on the side face of tomato. At this time, the order of tomato aligned in ascending order from the left, and four tomatoes were used in one measurement. In addition, the white balance target(12 inch) was measured to normalize the reflectance. The three wavelengths irradiated are 430 nm, 570 nm, 680 nm, distance between tomato and imaging system is H = 0.60 m, and the measurement surface of the side face of tomato was always the same every measurement. The measured information is the intensity image of the side face of the tomato, the phase and amplitude information of each pixel, and does not include the area not measured such as the back side of the tomato 1246

3 S430 is the sum of the reflectance ρi,j of each pixel of the tomato image taken by illuminating the wavelength 430 nm, is defined by the following equation. ρi,j (7) S430 = side face. As a background, we use a blackout that absorbs three wavelengths and near infrared, and also measures the wavelength 760 nm irradiated with near infrared light(fig.3,4). Therefore, the blackout absorb near infrared light and tomatoes reflect near infrared regardless of maturity to extract tomatoes from the measurement images. i j Also, S570, S680 is obtained in the same way as Eq. (7). The range of values of coloration degrees Co430, Co570, Co680 is 0 Co 1.0, and the value closer to 1.0 indicates that the coloration degree at that wavelength is higher. For extraction of the side face of tomato from the measurement image, a threshold was determined from the difference in the near infrared reflectance of the blackout and tomato, and the area below the threshold was removed to obtain a tomato area(fig.5). Fig. 3 An outline figure of measuring tomatoes which are assigned labels 1 to 4. Fig. 5 Extraction of tomato area by near infrared measurement. Fig. 4 Photograph of the imaging system which is taking spectral reflectance of tomatoes using wavelength of 680 nm. 3. RESULTS Fig.6 shows the tomato taken from a conventional CCD image sensor on February 5th and 9th, Tomatoes on February 5th, 2018 (elapsed days: 1 day) are almost immature, and tomatoes on February 9th, 2018 (elapsed days: 5 days) have matured from immature to semi-ripe. Representative measurement results(label 1, label 5, label 11) of the coloration degree of tomato are shown in Figs.7 to 9. In the graphs of Figs.7 to 9, the y-axis is the degree of coloration and the x-axis is the number of days elapsed since the harvest date. In label 1 and 11 of tomatoes, the values of coloration degrees Co570 and Co680 are intersected with each other after 3 days are elapsed, and the value of Co570 decreases, and the value of Co680 increase as the days goes. In label 5 of tomato(fig.9), the values of coloration degrees Co570 and Co680 intersected with each other after 6 days are elapsed, indicating that the change in coloration degree is slow compared to labels 1 and 11. On the other labels, the elapsed days, which the values of Co570 and Co680 intersected is, labels 10 and 12 are 2 days, labels 8 and 2.5. Measurement and Evaluation of Ripening Process of Immature tomato For the evaluation of maturity, the coloration degree defined below is used as an evaluation index. The calculated coloration degree of tomato is the ratio of reflectance of three wavelengths on the side face of tomato removing the calyx at each wavelength. The coloration degrees Co430, Co570, Co680 of the tomato of each wavelength are defined by the following Eqs. 4 to 6. Co430 = S430 (4) Co570 = S570 (5) Co680 = S680 (6) 1247

4 3 are 5 days, label 5 is 6 days, and the other labels are 3 days same as label 1 and 11. The value of Co 430 decreased as elapsed days arise from Figs.7 to 9, but it did not change like Co 570 and Co 680 massively. Fig.10 shows the approximate line is obtained by the least squares method for the relationship between all tomatoes and the coloration degree. The correlation coefficient of the approximate straight line of coloration degree Co 570 and Co 680 is strongly correlated with r = and r = 0.935, and Co 430 is correlated with r = Fig. 9 Transition of elapsed days and coloration degree of label11 of tomato. Fig. 6 Maturity change of tomatoes on February 5th and February 9th, 2018 (The numbers are labelled of tomato). Fig. 10 Approximate line between coloration degree and elapsed days are obtained using result of all tomatoes by the least squares method. Fig. 7 Transition of elapsed days and coloration degree of label1 of tomato. Fig. 8 Transition of elapsed days and coloration degree of label5 of tomato. 4. DISCUSSION In immature tomatoes, the reflectance of 570 nm is higher than the reflectance of 680 nm, the reflectance of 570 nm became lower than 680 nm due to changing from immature to semi-ripe. The reason is that immature tomatoes have matured, and tomatoes changing from green to red can also be confirmed by visual confirmation. In addition, tomatoes that were selected by visual confirmation, elapsed days, which the values of Co 570 and Co 680 intersected were 2-6 days and individual differences were observed. The reason is that humans evaluate the coloration degree of tomatoes by averaging the overall appearance, and it was considered that they do not consider the influence of chlorophylls. From the above, imaging system combination of the Correlation Image Sensor and the Ringview Optical System can obtain an approximate line of coloration degree and elapsed days from the maturation process of tomato measured beforehand, And by using the result, it is considered that it is possible to predict the number of ripe days from immature to full-ripe. 5. CONCLUSIONS In this study, maturation process was measured by measuring the reflectance of the side face of tomatoes, which were ripened under an environment of 70% humidity and temperature 21 C by using an imaging system combination of the Correlation Image Sensor and the Ringview Optical System. The coloration degrees Co 430, Co 570, and Co 680 obtained by measuring the reflectance 1248

5 of tomatoes suggested that the value of Co 570 becomes lower and the value of Co 680 becomes higher when tomatoes change from immature to full-ripe, And, the value of Co 430 suggested little change in the maturation process of tomato. In further studies, we plan to increase the number of samples which are measured the maturation process from immature to full-ripe to predict the number of ripe days of pre-harvest tomatoes, and to consider to measure the maturation process by changing the wavelength 430 nm to another wavelength such as in the near-infrared region and others. 6. ACKNOWLEDGEMENTS We would like to thank Prof.Maruo, Dr.Johokan, Mr.Moriuchi, Mr.Yoshimura and Dr.Ando for valuable discussions. This research was supported by R & D project of Ministry of Agriculture, Forestry and Fisheries. REFERENCES [1] Kouichi WATANABE, Yosuke ASANO, Ikusaburo KURIMOTO, Tsunaki NUKAYA, Atsushi KANO, Toru MARUO, Development of Temperature and Vapor Pressure Deficit Control System, Transactions of the Society of Instrument and Control Engineers, vol.52, No.5, [2] Gopal Tiwari, David C. Slaughter, and Marita Cantwell, Nondestructive maturity determination in green tomatoes using a handheld visible and near infrared instrument., Postharvest Biology and Technology, Vol. 86, 221/229(2013). [3] Kaveh Mollazade, Mahmoud Omid, Fardin Akhlaghian Tab, SS Mohtasebi, M Zude, Spatial Mapping of Moisture Content in Tomato Fruits using Hyperspectral Imaging and Artificial Neural Networks, Proceedings of the CIGR-Ageng2012: IV International workshop on Computer Image Analysis in Agriculture, Valencia, Spain, 8-12, [4] Shigeru Ando, Akira Kimachi, Correlation Image Sensor: Two-Dimensional Matched Detection of Amplitude-Modulated Light, IEEE TRANSAC- TIONS ON ELECTRON DEVICES, Vol. 50, No. 10, [5] Toru KURIHARA, Nobuta ONO, Shigeru Ando, Single-frame Surface Orientation Imager Using Triangular Three-phase Amplitude-modulated, Transactions of the Society of Instrument and Control Engineers, Vol. 48, No. 8,