AFITA/WCCA2012(Draft) Monitoring System with Flexibility and Movability Functions for Collecting Target Images in Detail Tokihiro Fukatsu Agroinformatics Division, Agricultural Research Center National Agriculture and Food Research Organization fukatsu@affrc.go.jp Masayuki Hirafuji Memuro Upland Farming Research Station, Hokkaido Agricultural Research Center National Agriculture and Food Research Organization hirafuji@affrc.go.jp Takuji Kiura Agroinformatics Division, Agricultural Research Center National Agriculture and Food Research Organization kiura@affrc.go.jp ABSTRACT In modern agriculture, it has recently become popular to monitor crop status in open fields with sensor networks such as Field Servers. A key technology for use in this monitoring is image data that can be easily collected at low cost as a result of the development of information technology. High resolution image data has the potential to provide useful information including information regarding changes in growth stage, the occurrence of pests, the degree of maturity, and so on. However, it is difficult to deploy monitoring equipment close to the target crops to collect image data in detail because deploying equipment within crop fields disturbs some farming operations and the growth of crops. Moreover, image data captured in close proximity to the targets can be unclear due to the effects of wind and the interference of growing leaves, and the target area also changes depending on the crop stage and condition. To solve these problems regarding the collecting of target images in detail, we have proposed an advanced monitoring system with flexibility and movability and developed a proto type system of the robotic Field Server. The robotic Field Server consists of a Field Server with image sensors, a locomotion unit so that it can move itself, and a handling actuator for monitoring targets closely in detail. The features of the proposed system are intended to obtain wide and dense monitoring coverage, to measure many targets with one device, to approach targets from different directions, and to monitor targets clearly by taking unnecessary objects out of the way. We conducted an experiment to collect crop images as the proto type Field Server was moved around the field and to clarify the potential and effectiveness of the locomotive monitoring system. Keywords: image data, Field Server, mobile robot, crop monitoring 1 INTRODUCTION In modern agriculture, it has recently become popular to monitor crop status in open fields using sensor networks [1, 2]. We have developed a Field Server system [3, 4] that functions as a sensor network for agricultural use. A Field Server, the sensor node for the system, has a Wi-Fi/3G module for networking, a main CPU board with a Web server for monitoring, various sensors that can be chosen according to the users preferences, and network cameras (Fig. 1). One of the key technologies for field monitoring is image data. It can be easily collected by Field Servers and can provide useful information including information regarding changes of growth stage, the occurrence of pests, the degree of maturity, and so on. We performed some field monitoring applications with Field Servers, and evaluated the potential and effectiveness of the system [5, 6]. However, there are 1
Figure 1: Field Server system still some issues that must be resolved in order to achieve a more practical and useful system (Fig. 2). For example, users want to deploy many Field Servers in their fields, and so reducing the cost of the Field Servers and the effort to maintain them is a basic problem. In order to monitor targets in detail, it is necessary to collect frequent sensor data, but the ability to do so depends on the capacity of the solar panels and batteries. In particular, image monitoring with cameras uses a lot of electricity. When a commercial power supply is used for the Field Servers, the locations where the Field Servers can be installed are limited in return for the resolution of the above problem. On the other hand, it is important and desirable, especially for image monitoring, to measure the target object closely or from a particular position. However, Field Servers installed within crop fields will become disturbances during farm operations and may affect the growth of crops. Figure 2: Trade-off problems of sensor network for agriculture To address these issues, with a special focus on the problems related to image monitoring, we have proposed robotic Field Servers for an advanced monitoring system that is movable and can be manipulated. Of course, robot monitoring in agricultural fields still requires a great deal of further development. However, the proposed system has sufficient potential to solve the problems and to expand the applications of Field Servers. In this paper, we represent the features of the system, clarify the architecture and action items, develop a proto type Field Server, conduct an experiment with the proto type, re-design the system on the basis of the result, and show the potential and effectiveness of the locomotive monitoring system. 2 ROBOTIC FIELD SERVER 2.1 Concept and features _ 2
The main issues for the sensor network in agricultural fields include how to obtain wide and dense monitoring coverage, how to perform smart measurement with sensitive handling and advanced sensors, and how to approach target crops closely and from the right direction for detailed measurement. In general, for sensor network systems, one way to perform spatial measurement is to deploy a large number of Field Servers. However, this is not a practical method because of the cost and effort involved. A remote sensing system such as a satellite, airplane or helicopter can collect spatial data effectively, but the monitoring cost is high and the resolution is low. It is also difficult to collect data at any time and to measure targets flexibly in response to users requests. To solve these problems effectively, we proposed a robotic Field Server (Fig. 3). The robotic Field Server consists of a conventional Field Server with smart sensors including cameras, a locomotion unit for moving them in the target fields, and a handling actuator unit for flexibly measuring the targets. Via the movability function with which it is equipped, a single Field Server can perform monitoring at multiple locations. Thus, it is possible to measure many targets with one sensing device such as an expensive multi-spectrum camera. The movability function also enables us to monitor targets within a crop field only when needed. Therefore, it is possible to monitor targets closely from the correct position without interrupting the farming operations, in contrast with a fixed-type Field Server that must be removed during every farm operation. A handling actuator unit enables advanced monitoring by moving the targets and other objects. It is usually difficult to monitor targets in their entirety and clearly for long periods due to the effects of wind, the interference of growing leaves, and so on. The manipulating function enables us to take unnecessary objects out of the way when monitoring. Users sometimes want to change the monitoring targets or monitoring point in order to check other parts of the targets depending on the crop stage and crop condition. The function also enables us to monitor targets from different directions and to set new targets in front of the cameras. By mounting sensor devices on the end of the handling actuator unit, we can also monitor in difficult locations, such as within crowded crop areas, when needed. The proposed system has many useful features as mentioned above, and it can be used in various applications. 2.2 Architecture Figure 3: robotic Field Server Here we describe the architecture of the proposed monitoring system with a robotic Field Server. The system is constructed on the basis of a conventional Field Server system. The robotic Field Server is treated as one of the other conventional Field Servers in the same manner by the control program of the system. Therefore, the robotic Field Server has a wireless LAN for communicating with other Field Servers and a main board with a Web server for operating the locomotion unit and the handling actuator unit via the Internet by the control program. In general, it is important and difficult to develop the hardware modules for the units, and the mechanism should be designed according to the targets and operation sites. On the other hand, 3
information technology and robotic technology have progressed rapidly in the 21 st century. Therefore, in this paper, we do not discuss the issues that further development of the hardware modules will be expected to solve. When the robotic Field Server is operated in a field site for a long time, we prepare a main dock for use as its home position. The robotic Field Server usually stays in the dock, which protects it from rain and other difficult conditions. By operating the robotic Field Server from the dock only when weather conditions are good, we can reduce the risk of experiencing problems with the Field Server and with the sensor devices. The battery of the robotic Field Server is charged by a charging cradle placed in the dock. The changes from the conventional system are the methods of operating the hardware and handling the data for the robotic Field Server. Controlling the hardware modules automatically requires a large amount of effort because the field environment and target conditions change often. We must also work to ensure the safety of people, crops and the Field Server, because the movability and manipulability of the system have the risk of leading to dangerous, unexpected actions. Therefore, we decided to control them manually from a remote site. In the manual control system, it is important for users to know the status of the robotic Field Server including its position and angles. Some statuses are directly collected by sensors and others are estimated by operating history data and sensor data including image data. Therefore, functions by which the status of the robotic Field Server can be perceived and provided to users for manual control are also present in the proposed system. These functions are also used to realize fail safe functioning of the system. The robotic Field Server is controlled via the wireless network provided by the other Field Servers, so it sometimes becomes non-operational if it exits the Wi-Fi range accidentally. Therefore, the status data is utilized to limit the unwanted motion of the robotic Field Server by estimating the range of its movement. Of course, it is also important to equip the robotic Field Server with the function of returning to its previous position automatically as a fail safe. In order to monitor the data from the robotic Field Server, we must modify the data format and the data correction of the traditional Field Server system. One robotic Field Server collects data under various conditions, so each piece of data requires additional information such as position, direction, measurement conditions, and so on. The robotic Field Server tries to monitor the targets from the same location every time, but the position is slightly different for each collection of data. Therefore, a function to normalize the collected data with an affine transform for the image data and an interpolation algorithm for the other data is required. 3 EXPERIMENT To test our proposed monitoring system, we developed a proto type robotic Field Server (Fig. 4). The robotic Field Server includes a standard conventional Field Server, a driving bogie, and a Figure 4: proto type robotic Field Server 4
robot arm. Each module is connected to the main board with digital I/O and RS232C so that the driving bogie and the robot arm can be controlled under the Field Server system. It runs for several hours on a full charged battery and is operated manually via the Internet in this experiment. This proto type robot does not have a charging function for its long-term operations, and its payload is not very heavy, so we conducted a fundamental experiment to collect serial field images with the system as it moves around a field site. With a few practices, an operator can conduct the robotic Field Server to collect the desired image data, and detailed target image data can be obtained by moving a non-target leaf with the controlled robot arm. Some slanted and shaken image data were captured under unstable conditions, so we needed to filter or correct these image data with image processing. In this experiment, the wheels of the driving bogie sometimes spun free, and a great deal of skill was required to control it, especially to position it accurately. 4 DISCUSSION AND FUTURE WORK We have proposed an advanced monitoring system with a robotic Field Server that has a locomotion unit and a handling actuator, and described the features and architecture of the system. Using a proto type robotic Field Server, we performed a fundamental experiment. Using the results, we are now developing the next robotic Field Server, which can operate various monitoring experiments at a high level, to evaluate the possibility and effectiveness of the system (Fig. 5). The next robotic Field Server has a pair of three-legged modules as a walking platform which provides enough payload space for measuring devices and a flat loading space even when moving, and consumes less energy in the standing position. The robotic Field Server also has a 3 D.O.F. manipulator that provides a light-weight mechanism and a wide work space with a weight compensation mechanism, not with a counter-weight but a non-circular pulley and a spring. These features of the robotic Field Server facilitate to expansion of the proposed system to a variety of applications. Figure 5: next robotic Field Server At the same time we seek solutions to the problems with the proposed system, we must also consider suitable applications that would justify the cost and effort involved in the robotic Field Server. We are planning to monitor plant growth and fruit maturity at multiple points with a laserinduced fluorescence device and a near-infrared spectrum device. These sensing devices are expensive and are sufficiently sensitive for outdoor use, so they would be effective and suitable for use with the proposed system. By attaching sensing devices on the end of a handling actuator, we can monitor a plant s condition closely in detail. Chlorophyll monitoring of rice leaves with a SPAD meter is one of the applications for the system, although a SPAD meter is not a camera. 5
A semi-robotic Field Server that has only moving arms and no locomotion unit has also been considered as one of the proposed systems. For example, monitoring crop images from directly overhead is important to determine the vegetation rate, but the device with which such monitoring can be accomplished truly disturbs the farm operation. Therefore, we also propose applications of the system that involve monitoring such images with a moving arm equipped with a camera from the sides of crop fields. The next robotic Field Server mentioned above performs well but is expensive. Therefore, we are also studying a monitoring system that uses a robotic Field Server equipped with a cheap locomotive unit that provides unstable image data as described in our experiment. For this system, we are developing a new image correction algorithm. This type of robotic Field Server does not have expensive sensors such as differential GPS sensors, so it is difficult to identify its position with satisfactory accuracy at field sites. In this case, we can obtain accurate data using a combination of information extracted from image data for one or a few targets and markers, and anteroposterior data that provides important information. We will also develop some support applications that help to control the robotic Field Server that is not a high performance server by providing motion planning data and operation guidance data. The conventional Field Server system achieved ubiquitous monitoring at field sites, and it solved the Last Mile Problem. Now, we have a Last Feet Problem which means we cannot monitor target crops closely in crop fields over a long term. By realizing our proposed system, we will solve it. 5 ACKNOWLEDGEMENTS This research has been supported in part by Ministry of Education, Culture, Sports, Science and Technology (MEXT) through Grant-in-Aid for Young Scientists (B): 24780252 and by Ministry of Internal Affairs and Communications (MIC) through Strategic Information and Communications R&D Promotion Programme (SCOPE): 122304001. REFERENCES [1] J. M. Kahn, R. H. Katz, K. S. J. Pister, (1999), Next Century Challenges: Mobile Networking for Smart Dust, In Proceedings of the 5th Annual ACM/IEEE International Conference on Mobile Computing and Networking, Seattle, WA, USA, pp. 271-278. [2] I. F. Akyildiz, W. Su, Y. Sankarasubramaniam, E. Cayirci, (2002), Wireless Sensor Networks: a Survey, Computer Networks, 38(4), 393-422. [3] T. Fukatsu, M. Hirafuji, (2005), Field Monitoring Using Sensor-Nodes with a Web Server, Journal of Robotics and Mechatronics, 17(2), 164-172. [4] T. Fukatsu, M. Hirafuji, T. Kiura, (2006), An agent system for operating Web-based sensor nodes via the Internet. Journal of Robotics and Mechatronics, 18(2), 186-194. [5] T. Fukatsu, T. Watanabe, H. Hu, H. Yoichi, M. Hirafuji, (2012), Field Monitoring Support System for the Occurrence of Leptocorisa chinensis Dallas (Hemiptera: Alydidae) using Synthetic Attractants, Field Servers and Image Analysis, Computer and Electronics in Agriculture, 80, 8-16. [6] T. Fukatsu, Y. Saito, T. Suzuki, K. Kobayashi, M. Hirafuji, (2008), A Long-Term Field Monitoring System with Field Servers at a Grape Farm, Proceedings of Application of Precision Agriculture for Fruits and Vegetables, Orland, FL, USA, pp.183-190. 6