APPLICATION OF FUZZY BEHAVIOR COORDINATION AND Q LEARNING IN ROBOT NAVIGATION

Transcription

1 APPLICATION OF FUZZY BEHAVIOR COORDINATION AND Q LEARNING IN ROBOT NAVIGATION Handy Wicaksono 1, Prihastono 2, Khairul Anam 3, Rusdhianto Effendi 4, Indra Adji Sulistijono 5, Son Kuswadi 6, Achmad Jazidie 7, Mitsuji Sampei 8 1,2,7 Department of Electrical Engineering, Sepuluh Nopember Institute of Surabaya, Indonesia 1,4 Department of Electrical Engineering, Petra Christian University, Surabaya, Indonesia (Tel : ) 2 Department of Electrical Engineering, University of Bhayangkara, Surabaya, Indonesia 3,4 Department of Electrical Engineering, University of Jember, Jember, Indonesia 5,6 Department of Mechatronics, Electronics Engineering Polytechnic Institute of Surabaya, Surabaya, Indonesia 8 Department of Mechanical and Environmental Informatics, Tokyo Institute of Technology, Tokyo, Japan handy@petra.ac.id ABSTRACT Behavior based architecture is widely used in mobile robot because it makes the robot response faster. If robot only works to achieve simple task, it can use some primitive behaviors. However, when the task is getting more complex, the behavior coordination is needed. In order to construct this coordinator, fuzzy logic can be applied as Fuzzy Behavior Coordinator (FBC). By using FBC, it can be seen from simulation that robot has smoother movement and shorter time to reach target in its navigation. When the robot is in new and uncertain field, it needs to learn. Q learning can be used to give intelligence behavior to robot. Robot can construct its own behavior by learning from its environment. By applying Q learning, the shortest path to reach target will be obtained after some episodes of robot training. Keywords: behavior-based robotics, fuzzy behavior coordinator, Q learning 1 INTRODUCTION Robot architecture that had been developed in early days is deliberative architecture. The robot senses its environment, thinks by creating a world model, and after that robot will act. This approach has these disadvantages: planning process (world modeling) takes a long time, no direct relationship between sensor and actuator, and a very sophisticated controller is needed [4]. To overcome them, Brooks [7] suggest reactive architecture (sometimes called behavior-based architecture). In this approach, we do not need a world model mathematically. The real environment is the only model which needed by robot. Another advantage is all behaviors run in parallel, simultaneous, and asynchronous way. Therefore, the sophisticated controller is not need anymore, and it can be replaced with a small-low cost microcontroller [4]. In behavior-based architecture, the problem will arise when there are many behaviors in a robot. Therefore, it has to decide which behavior will be executed at one time. Because of that, robot must have behavior coordinator (sometimes called arbiter). First approach that is also suggested by Brooks [7] is Subsumption Architecture that can be classified as competitive method. In this method, at one time, there is only one behavior that applied in robot. It is very simple method and it gives the fast performance result, but it has disadvantage of nonsmooth response and inaccuracy in robot movement. To overcome competitive method weakness, Arkin [2, 3] suggest Potential Fields Method that can be classified as cooperative method. In this method, at one time, there can be more than one behavior that applied in robot, so every behavior has contribution in robot s action. It results in a smoother response and more accurate robot movement. In the other hand, this method is more complex than competitive method. The complete list of behavior coordination method can be found in [12]. Another cooperative method is by using Fuzzy Logic as tool for coordinate the behaviors. There are many approaches to apply Fuzzy Behavior Based Control. Abreu [1] uses fuzzy behavior arbitration that combines some behaviors in its defuzzyfication process to control autonomous vehicle. Thongchai [15] use fuzzy to process each behavior, but use priority based arbitration to combine its behaviors. Saffiotti [13] suggest, context dependent blending to improve ordinary Fuzzy Logic weakness because of behaviors conflict. 41

2 42 The 5 th International Conference on Information & Communication Technology and Systems Beside the correct architecture and behavior coordination method, in order to overcome the unplanned thing in uncertain world, the robot will need a right learning mechanism. In supervised learning mechanism, teacher is needed, while in unsupervised learning the agent has to learn by itself. Therefore, the latter is more suitable for robotic application. Reinforcement learning is an unsupervised learning method which can learn from reward that given by environment directly (online) [8]. There are various methods to solve reinforcement-learning problem. Q learning algorithm is the famous one. The advantage of Q learning is its off policy characteristic, simple algorithm, and it is convergent to optimal policy [16]. 2 MODEL, ANALYSIS, DESIGN, AND IMPLEMENTATION 2.1 Behavior Coordination Methods In behavior based robotics approach, methods of behavior coordination are significant. The designer needs to know how robot coordinate its behaviors and take the action in the real world. There are two approaches: competitive and cooperative. In competitive method, at one time, there is only one behavior that applied in robot. The first suggestion in this class is Subsumption Architecture that suggested by Brooks [7]. This method divides behaviors to many levels, where the higher-level behavior have higher priorities too. Therefore, it can subsume the lower level ones. The layered control system figure is given below. Sensors Level 3 Level 2 Level 1 Level 0 Figure 1. Layered control system [7] Actuators The example architecture of behavior coordination that using Subsumption can be seen in Figure 2. That method above is very simple and fast, but it still has disadvantage of non-smooth response and inaccuracy in robot movement. The cooperative method overcomes this weakness. In this method, at one time, there can be more than one behavior that applied in robot, so every behavior has contribution in robot s action. Arkin [2] suggest the potential fields method, which every object will be described as vector that has magnitude and direction. The result behavior is mixing between each behavior. The motor scheme for this method appears in the Figure 3. Light Sensors Distance Sensors Reach the Target Obstacle Avoidance Wandering S I Motor Figure 2. Subsumption architecture example for navigation Environmental Sensors E N ES1 V I R O N ES2 M E N T ES2 Motor Schemas PSS1 PS1 PS2 PS3 PSS2 MS1 MS2 Vector S Robot Figure 3. Motor schema for potential field method [3] Motors This method results in a smoother response and more accurate robot movement. In the other hand, this method is more complex. It also can result in slow movement, it needs long time to reach the target, and it may fail to avoid trap. Another cooperative method is by using Fuzzy Logic as tool to combine the behaviors. After input from sensor is fuzzified, then it will be processed in rule base with other fuzzified input. Rule base represent the behavior. Defuzzification process will combine the behaviors [1]. The scheme of fuzzy behavior coordination is shown in Figure 4.

3 C08- Application Of Fuzzy Behavior Coordination And Q Learning In Robot Navigation - Handy Wicaksono Robot Learning Reinforcement Learning Mataric [10] says that an excellent behavior selection is important in behavior-based control, because it decides which behavior that will be applied in certain time. This problem can be formulated in reinforcement learning (RL) framework by deciding which policy that will map the state into the behavior, which will maximize the received reward in agent s lifetime. RL is classified as unsupervised learning, so its agent does not need any supervising to accomplish its task. Sensor 1 Sensor 2 Sensor 3 Sensor 4 Fuzzyfication Fuzzyfication Fuzzyfication Fuzzyfication Rule Base 1 (Behavior 1) Rule Base 2 (Behavior 2) Defuzzyfication Figure 4. Scheme of fuzzy behavior coordination Actuator Glorennec [8] also states that in supervised learning, which is mentioned too as learning from teacher, the learning system know the system error every time for each related input and output vector. This error can be used to modify the learning parameters. While in RL, also can be mentioned as learning from critic, the accepted signal is reward (positive, negative, or neutral) from behavior. This signal indicates what the robot has to do without any information about how to do it. Therefore, it can be said that RL is online and it can be applied directly to the robot. The general process scheme can be shown below. Figure 5. General process scheme reinforcement learning [11] Obelix, robot that push the box to the certain location [9], starts implementation of RL in robot application. RL also applied to another kid of robot, such as an underwater robot [11], a group of mobile robots, which are used to do foraging [10] Q Learning In many learning applications, the goal of RL is constructing a control policy, which will map the discrete input area to discrete output area so the maximal cumulative reward can be achieved. The most simple and well-known learning algorithm in robotic application is Q learning [16]. Q learning has some advantages. It is off policy (robot can follow any policy to produce optimal solution), it has simple algorithm, and it will give convergent result in an optimal policy. Here is the simplified algorithm of Q learning [14]. Initialize Q(s,a) arbitralily Repeat (for each episode) : Initialize s Repeat (for each step of episode): Choose a from s using policy derived from Q (e.g., - greedy) Take action a, observe r, s Apply [ r + γ max Q( s', a') Q( s, )] Q( s, a) Q( s, a) + α a' a where : action) s s ; until s is terminal Q(s,a) : component of Q table (state, s : state s : next state a : action a : next action r : reward α : learning rate γ : discount factor 2.3 Behaviors Description Behaviors in Subsumption Architecture There are some behaviors that robot should have in order to succeed in autonomous navigation. They are: Obstacle avoidance Search the target Wandering The scheme of these architectures can be seen on Figure 2. Wandering behavior is the lowest level behavior (in Subsumption Architecture) that will move the robot all around the arena in order to find the target. If the robot sees the obstacles, then it will avoid them because of presence of obstacle avoidance behavior. Then when the robot find the target (a light source), it

4 44 The 5 th International Conference on Information & Communication Technology and Systems will move closer to the robot. After the distance is close enough, then the robot will stop. will combine the MF of two sensors. The rule bases are not the same, but one is mirror of the other. The rule bases for them are shown in Table 3 and Fuzzy Behaviors Behaviors that use in Fuzzy Behavior (FB) and Modified Fuzzy Behavior (MFB) are almost same. Here are the descriptions of them Obstacle Avoidance Behavior Robot uses two distance sensors (which is put in left and right). Here are the triangular membership function (MF) for the sensor (their configuration are same). They can be divided as small, medium, and big. 1 Left & Right Distance Sensors Figure 6. Membership function of distance sensors A rule base of left & right motor will combine MF of two sensors. The rule bases are not the same, but one is mirror of the other. Here are the rule bases for them. Table 1. Rule base 1 for left motor Right D.S S M B Left D. S S P P N M Z Z N B P P N Table 2. Rule base 2 for right motor Right L.S. S M B Left L.S. S P Z P M P P P B N N N Search Target Behavior Left & Right Light Sensors Figure 7. Membership function of light sensors Table 3. Rule base 3 for left motor Right L.S. S M B Left L.S. S Z Z P M Z Z P B N Z P Table 4. Rule base 4 for right motor Right L.S. S M B Left L.S. S Z Z N M Z Z Z B P P P Q Learning Behavior In this paper, Q learning will be applied only in Search Target behavior. In the beginning, the robot will explore its environment and learn from it. In this exploration phase, the robot will move randomly and (in the same time) update its Q table. After some times, the value of Q table s component will be optimal and steady. Now it is time to switch to exploitation phase, when the robot only use the optimal value of Q table to navigate to the target. 3 RESULT Robot used here has two distance sensors and two light sensors. It only uses two motors. The complete parts of robot can be shown below. Robot uses two light sensors (which is put in left and right). the triangular membership function (MF) for the sensor (their configuration are same) is shown in Figure 7. They can be divided as small, medium, and big. A rule base of left & right motor

5 C08- Application Of Fuzzy Behavior Coordination And Q Learning In Robot Navigation - Handy Wicaksono 45 Right Distance Sensor Right Light Sensor Right Motor Body of robot Figure 8. Robot used in Simulation Left Distance Sensor Left Light Sensor Left Motor Webbots software from Cyberbotics has been used to simulate and test the performance of robot. 3.1 Behavior Coordination Simulation Simulations that have been done will observe the performance of these behavior coordination methods: 1. Subsumption Architecture (SA) 2. Fuzzy Behavior Coordinator (FBC) The simulation will measure how many time that robot used to reach the target, the performance of robot movement that can be seen by the robot s trajectory, and how well robot in avoid obstacle Target Reaching Simulation 2.03) 1:09 m 0:24 m 2 (0.42, 0, 0.41, 0.93) Reach in 1:05 m Reach in 0:13 m 3 (-0.41, 0,-0.42, 4.13) Reach in 1:45 m Reach in 1:21 m From the table we can see that SA needs most time (1:09 m, 1:05 m, and 1:45 m) to reach target from each position. It is happened because SA is a competitive coordinator type, so there is a chance to ignore the search target behavior when other behavior is active. The rough response from the robot can increase the time to reach target. FBC gives better result because every behavior gives contribution in robot s decision-making. However, it tends to give slow result, because every behavior will processed in fuzzy inference engine. MFBC come with a fastest result, because the search target behavior is not a fuzzy behavior, and it considers as more important behavior than other does Movement Performance Simulation In this simulation, robot s trajectory will be observed in order to understand robot s movement. Here are the simulation figures of robot with SA. This Simulation will be done to measure how many times that needed by robot to travel from the start position until the target is found. There are three start positions in the arena, which can be seen in figure below. Position 2 Target Position 1 Figure 9. Arena for the simulations Position 3 Each simulation has been done 5 times, and the best result is shown in table below. Table 5. Target reaching test result Pos Start position of Robot Time to reach target (minutes) (x,y,z,teta) SA FBC 1 (-0.41, 0, 0.43, Reach in Reach in Figure 10. Movement performance simulation result for robot uses SA From the figure, we can see that SA gives a sharp turn of robot. Look at the circle with the dash-line. This sharp turn happened because of the immediate change in robot s behavior. It has to switch from one behavior to another directly. So it make the robot s movement are little bit rough, although it gives faster movement than another approach. While the simulation figure of robot with FBC can be seen below.

6 46 The 5 th International Conference on Information & Communication Technology and Systems actions: forward, turn somewhat left, turn left, turn somewhat right, and turn right Simulation with three actions Figure 11. Movement performance simulation result for robot uses FBC From the figure above, we can see that FBC gives smoother response, slighter turn. Once again, it is marked by the circle with the dash-line. This is happened because fuzzy process that mix all robot s behavior. Therefore, there is no immediate change between behaviors. In overall, robot movement is very smooth but it gives the slowest motion. The robot will start in coordinate (x, y, z, and theta): (0.4, 0, 0, and 1.8). The simulation result in exploration stage is shown in Figure 13. After 5 times simulation, the Q table components values are already optimal. Here is the final Q table. Q = [ ] After that, robot enters the exploitation stage and the simulation result is shown in Figure Trap Simulation The last simulation that will be held here is called trap simulation. The robot will be put into the trap to see how well their performance to escape from the trap. Here is the result figure. SA FBC Figure 13. Exploration Simulation Result 1 Figure. 12 Trap performance simulation result From the figure, we can see that SA give better performance in escaping from the trap. SA robot succeeds to escape, but FBC robot are failed during the simulation. Both of them are trapped. 3.2 Q learning behavior simulation In this simulation, Q learning will be applied just in Search Target behavior. In the beginning, the robot only uses three actions: forward, turn left and turn right. After that, the robot will apply five Figure 14. Exploitation Simulation Result 1 From the figure above, it can be seen that after some training form environment, the robot can achieve its target accurately in short time Simulation with five actions

7 C08- Application Of Fuzzy Behavior Coordination And Q Learning In Robot Navigation - Handy Wicaksono 47 The robot will start in coordinate (x, y, z, and theta): (0.4, 0, 0, and 1.8). The simulation result in exploration stage is shown in Figure 15. After 3 times simulation, the Q table components values are already optimal. Here is the final Q table. Q = [ ] After that, robot enters the exploitation stage and the simulation result is shown in Figure 16. target than the Subsumption Architecture. Nevertheless, the FBC still has weaknesses in avoiding the trap. It also can be seen from the simulation result that by adding learning capability (using Q learning) in behavior that using five action, the robot can achieve the target accurately in short time, with less time in exploration. 5 ACKNOWLEDGEMENT This work is being supported by Japan International Cooperation Agency (JICA) through Technical Cooperation Project for Research and Education Development on Information and Communication Technology in Sepuluh Nopember Institute of Technology (PREDICT - ITS). Figure 15. Exploration Simulation Result 2 Figure 16. Exploitation Simulation Result 2 From this simulation it can be seen that the robot achieve its optimal value in less time simulation (three compare to five), and it is fast to reach the target accurately. 4 CONCLUSION AND DISCUSSION This paper described the application of fuzzy logic and Q learning in mobile robot navigation system. From the simulation result, it can be seen that Fuzzy Behavior Coordinator gives the smoother movement and shorter time to reach the REFERENCES [1] Abreu A. (1999) Fuzzy Behaviors and Behavior Arbitration in Autonomous Vehicles. Proceedings of the Portuguese Meeting in Artificial Intelligence EPIA99, volume 1695 of LNC. [2] Arkin R.C. (1987) Motor Schema Based Navigation for a Mobile Robot : an Approach to Programming by Behavior. IEEE Int. Conf. on Robotics and Automation, pp [3] Arkin R.C. (1998) Behavior-Based Robotics,. England : Bradford Books. [4] Asadpour M., and Siegwart R. (2004) Compact Q-Learning for Micro-robots with Processing Constraints. Journal of Robotics and Autonomous Systems, Vol. 48, No. 1, pp [6] Bekey G. A. (2005) Autonomous Robot : From Biological Inspiration to Implementation and Control.Massachusetts : MIT Press. [7] Brooks R. (1986) A Robust Layered Control System For a Mobile Robot. IEEE Journal of Robotics and Automation, vol. 2, no. 1, pp [8] Glorennec, P. Y. (2000) Reinforcement Learning : An Overview. Proceedings of European Symposium on Intelligent Techniques. Aachen, Germany. [9]Mahadevan, S. dan Connell, J. (1991) Automatic Programming of Behavior Based using Reinforcement Learning. Proceeding of the Eighth International Workshop on Machine Learning, ppl [10] Mataric, M.J. (1996) Reinforcement Learning in Multirobot Domain. Autonomous Robots, vol. 4, no. 1, hal [11] Perez M.C. (2003) A Proposal of Behavior Based Control Architecture with Reinforcement

8 48 The 5 th International Conference on Information & Communication Technology and Systems Learning for an Autonomous Underwater Robot, Tesis Ph.D., University of Girona, Girona. [12] Pirjanian P. (1999) Behavior coordination mechanisms : State-of-the-art, Technical Report IRIS , Univ. Southern California. [13] Saffiotti A. (1997) Fuzzy Logic in Autonomous Robotics : Behavior Coordination. Proceedings of the 6 th IEEE Int. Conf. on Fuzzy Systems. pp , Barcelona, Spain. [14] Sutton, R.S., dan Barto,A.G. (1998), Reinforcement Learning, an introduction. Massachusets : MIT Press. [15] Tongchai S, Suksakulchai S., Wilkes D.M., and Sarkar N.(2000) Sonar Behavior-Based Fuzzy Control for a Mobile Robot. Proc. of IEEE International Conference on Systems, Man and Cybernetics. Nashville, Tennesse. [16] Watkins, C., Dayan, P.(1992) Q- learning,thechnical Note. Machine Learning, Vol 8, pp