SENIOR PROJECT FINAL REPORT TEAM MARRS

Transcription

1 Prepared by: SENIOR PROJECT FINAL REPORT Farah Abdel Meguid Mai Khater Mennat Allah Ali Merihan El Hefnawi TEAM MARRS

2 Table of Contents 1. Introduction System Overview System Structure Usage Constraints Requirements Revisited Functional Requirements Modifications Base Station Guide Robot Ground Robots Non functional Requirement Modifications Performance Requirements Reliability Fault Tolerance Expandability Design Modifications Hardware Design Modifications The Lifting Mechanism The Wheels Software Design Modifications Human Detection Mission Control Testing and Evaluation Challenges Lack of Robot Components Availability Lack of Workshop Facilities Human detection Different Kinds of Floors Conclusion and Future Work More Mobile Guide Robot More Ground Robots Using 3D Cameras/Deep Sensors Stronger Motors:... 16

3 1. Introduction Out of the many different fields of computer science, we have chosen to explore the field of embedded systems in our thesis project. This field continues to witness massive advancement and research. It is utilized in all different disciplines such as medical surgeries, space explorations, arts, robotics and others. The robotics field is one that has greatly advanced over the past decade reaching great successes. Robots are now used to assist human beings in their different daily tasks such as cleaning and navigation. They are also used in industry such as in manufacturing cars and control. Other robotics projects have been developed to entertain humans especially the children and the seniors; they are equipped with the ability to listen and talk, and surprisingly, a new robot is being developed now that will be called the ethical robot. It will be taught how to think and take decisions concerning ethics and values, a thing that we thought only human beings could achieve. Despite the importance of all the fields previously mentioned, we have chosen to explore robotic systems in rescue missions. Catastrophes and accidents kill thousands of people every year, some of them are victims of the accidents themselves and others die while rescuing others. Therefore, interest has risen in developing robotic systems to rescue instead of human beings. This area has successfully been implemented and expanded to include rescuing in many of different environments. For instance, there are robots that rescue in case of fire, others under water surfaces, demining purposes, explosions and in case of gas leakage. Out of those environments, we have chosen to develop our project to work in an environment where gas has leaked and the site is inaccessible by human rescuers. The spill or leakage of hazardous materials renders sites inaccessible by human rescuers. This is because, if, for example, toxic gas has leaked in a building, certain measures have to be taken before rescuers can get inside. For example, they should attempt to know what type of gas has leaked to know how to handle it, they have to make sure that there isn t a potential threat of explosion and they have to ensure that they are perfectly insulated, all of which causes delay. This lack of timely intervention leads to an increased number of casualties. This loss can be prevented/decreased if the rescue mission is carried out faster. Also, many human rescuers suffocate or get poisoned while working in such environments. Even if instant harm was not detectable on rescuers, getting frequently subjected to such environments negatively affects their health on the long run. This is another loss that can be prevented by using rescue robots. Due to the problems mentioned above, we believe that robots should carry out rescue missions in hazardous environments. This would allow for more accurate

4 and faster detection and retrieval of survivors and would protect the lives and health of rescuers. For all that, in our thesis project, we have built a Multi Agent Rescue Robotic System (MARRS) that will perform rescuing in case of a gas leakage. Our target was building an inexpensive system that would perform both discovery and rescue. Meaning, even if our system didn t find any survivors inside, it would report back information about the environment such as the kind of gas and temperature inside using certain sensors that will be added to it. 2. System Overview 2.1 System Structure Below is a list of our system s main members and the functionality of each. a. Base Station: Launches the mission. Terminates the mission. Makes sure the system is performing correctly. Keeps track of the number of robots that are connected. Performs complex neural networks algorithms. Locates survivors. Receives data about the environment from Guide. b. Guide Robot: Identifies survivors. Identifies which gas has been leaked among the family of liquefied petroleum gas. Sends survivor locations to Ground Robots. Sends environment data to Base Station. c. Ground Robots: Rescue the survivors. Communicate all its findings to the Guide Robot. 2.2 Usage Constraints The system operates in an environment where a harmful liquefied petroleum gas (LPG) has been leaked. Potential survivors are expected to be found lying on the ground. The environment will also have obstacles that the Ground Robots are expected to overcome in

5 order to rescue the survivors. Some of the LPG gases pose an explosive risk that is why the Guide Robot will have a gas-detecting sensor and the Ground Robots will detect the environment temperature. 3. Requirements Revisited 3.1 Functional Requirements Modifications Base Station The Base Station will be represented in our system with a normal PC. A PC is good enough to meet the requirements and responsibilities of the Base Station. It has enough memory to store data temporarily and permanently. It has a WiFi module to communicate with the ground robots. It has a suitable processor to be able to run classifiers on frames fast enough Guide Robot Due to the tasks described in the system structure, the Guide Robot must be able to capture frames as fast as possible and process them. Suitable battery life of the Guide Robot is crucial for the mission because if the Guide fails, the mission shall be aborted. Also, the Guide must support stable and reliable communication with the other system features because it is the main coordinator between the different components Ground Robots Ground Robots requirements are very critical as they work in the field. They: 1. Are able to lift a human body through a lifting mechanism. 2. Can communicate together and with the Guide Robot too. 3. Can move in all four directions 3.2 Non functional Requirement Modifications Performance Requirements Performance is a very important aspect of our system. We need our system to have very high performance since it is all about saving lives. Therefore the classifiers are done on the Base Station, which will always make sure that the system is performing well and will also be responsible of detecting humans.

6 3.2.2 Reliability We need our system to be reliable while executing the search and rescue mission. Hence, we have installed different layers in detecting the human, so that the system does not detect something else as a human and rescue it instead. This is one of the reasons why we have chosen to use a neural network to detect a human; to make the results as accurate as possible Fault Tolerance The system needs to tolerate some faults if they were to happen. That is why the system is intended to have many inexpensive Ground Robots so that if one is destroyed or lost in the process of search or rescue, another one would be able to replace it. Hence, backup robots should be ready for that purpose Expandability We want our system to be applicable in other environments, not just gas leakage. This can be done with changing some of the sensors. For example, the LPG sensor that we are intending to use in the product can be changed to detect nuclear radiation leakages, and the search and rescue mechanism can be done the same way. 4. Design Modifications 4.1 Hardware Design Modifications Since the beginning of the project, we went through some modifications regarding the hardware. In this section, we will go through the hardware modification decisions we have made. First, let us go over the initial design. The ideal system would consist of 6 robots cooperating to lift a human. However, our prototype consists of 2 ground robots that will lift a mannequin when exploration has been done and the guide robot has detected a human. We have done a simulation of the robot s movement before implementing it in real life using a Solid Works, which is a mechanical simulation tool. Figure 1 below shows the result after simulation The robot body (which is the rectangular box shown in figure 1) is made of aluminum bars that are joined together via 90-degree joining plates. The rectangular box s dimensions are 40cmx30cmx20cm. The forklift is made of 3 mm aluminum sheet of dimensions 30cmx30cmx7cm. It is mounted on top of linear bearings. Those linear

7 bearings slide up and down 2 aluminum shafts, which are fixed on the aluminum bars via shaft end supports. Figure 1 Ground Robot Initial Design The Lifting Mechanism The initial design of the lifting mechanism was supposed to be a pulley system. As can be seen in the design in figure 1 above, the pulley system would have been used to lift the forklift up and down the aluminum shafts, which are supported by shaft end supports. The pulley system would have required the following components: 2 Metal Chains o Those would have been the chains that connect the motors to the forklift to make it move up and down the shafts 2 worm gear motors o The pulleys would have been fixed onto 2 worm gear motors. That way, when the motors work, the pulley system would move up or down depending on the rotation direction of the motor. 2 Idler Pulley Kits o Those would be placed on the robot from the top as can be seen in the design in figure 1. The metal chains would have been fixed to it. This would facilitate the vertical movement of the forklift.

8 However, all of these components would have been quite expensive. Also, if the components were not strong enough to handle the weight carried by the forklift, the chain might disconnect from the pulley idlers or the motors, which would have caused the forklift to drop to the ground. We solved this problem by replacing the pulley system in our design by a linear actuator motor. The motor could be seen in figure 2 below. Figure 2 Linear Actuator Motor This type of motors creates a vertical motion instead of a circular motion in conventional motors. When the motor rotates, it causes a rod to move up and down. All what we had to do was connect the linear actuator motor to the forklift. We did so by attaching a rectangular aluminum sheet between the forklift and the linear actuator. This can be seen in figure 3 below The Wheels Figure 3 Forklift and Linear Actuator Motor Connection The wheels we used were omnidirectional wheels. The reason why we chose omnidirectional wheels was because we want our system to move in all direction as is without changing the body s orientation as in the figure 4 below. That is because we want the system to be flexible to move in narrow spaces such as doors.

9 Figure 4 System Movements In the simulation, the wheels that we intended to use were a specific type of omnidirectional wheels called macnum wheels. They are fixed the same way normal wheels are fixed on cars (placed on each corner). The problem with those wheels is they are not available in Egypt for direct sale. They would have been customized specially for us. Hence, they are very highly expensive. We solved this problem by altering the design of our robot to use normal omnidirectional wheels. Those wheels are fixed in the middle of every side in the robot as shown in figure 5 below. Figure 5 Omnidirectional Wheels on the Ground Robot 4.2 Software Design Modifications The Base Station launches the mission by sending a signal to the Guide Robot. After that, there are three processes going on in parallel. First of all, the system in the base station starts to monitor the robots and their status. Second, the temperature sensor, which is mounted on one of the Ground Robots, is active and sends back readings to the base station. And finally, the Guide Robot starts capturing frames of the sight of the leakage and sends those images back to the base station to determine the possibility of the existence of human beings in those frames. Human identification in this project relies on a pre-trained classifier called Caffe. As a result, classifying could not be done

10 on a board because it would require a lot of processing power that can only be provided by a regular desktop computer. Hence, the frames captured have to be sent to the base station for processing. The base station then responds with a filtered image that segments the human body, if any exists. This image is used by the Guide Robot to guide the Ground Robots towards the human body. When they reach the body the rescue routine is launched and they start lifting the body Human Detection Human Detection is a main function of the Base Station. After the Guide Robot is launched into the site, it captures frames and sends them to the Base Station. The Base station detects whether there is a human in the frame or not. This is done through Caffe, which is further described below. Caffe is a framework for neural networks that was developed in Berkeley Vision and Learning Center (BVLC) and contains a lot of neural network models that have been tested for different purposes. Model Zoo in Caffe contains different deep learning architectures. We decided to use the Fully Convolutional semantic segmentation model. The reason behind choosing a fully convolutional neural network is the fast testing time and the reasonable accuracy that we can obtain from running images through such a network. A convolutional network contains several layers of convolution. Basically, a convolutional layer contains neurons each with a kernel that has a fixed window size. This kernel is convolved with the corresponding pixels in the image. Convolution, in the first layer, extracts features from the images and then the convolution layer extracts features from features and so on. There are also some pooling layers that reduced dimensionality of the image as we move from one layer to the other. The following figure 6 explains the system.

11 Figure 6 Neural Network Used The model used was trained to segment 60 different classes from any given frame. These 60 classes contain class Human as number (56). There are three resolutions for CFNs. We chose to use the highest resolution (8s) because of the better accuracy. Resolution here refers to the stride, which is the step between two consecutive kernels in the convolutional layer. The following figure 7 contains all the layers used. Figure 7 Neural Network Layers After passing captured frames to function Caffe_function(), which is a C++ wrapper for a python code that passes the image as an input to a pre-trained CFN, we obtain the results. A sample of the result is shown in figure 8 below.

12 Figure 8 Caffe Results Mission Control The control approach that is used in the system is vision based. This means that the system captures images from the site frequently and processes them and based on this the system makes new decisions regarding the robots next step. The red blob in the frame received from the Base station represents the human body in the frame. Pixels of the red blob are extracted and merged with another frame that has the robots detected. The blue and white blobs represent our two Ground Robot. The following figure 9 shows the Robot Detection before merging in the human frame (left) and after merging the human frame (right). Figure 9 The Final Detection Results Through the communication channels between the Guide and the Ground Robots, instructions of the Robot movement are sent. After each movement, new frames are taken and processed so the Guide Robot can make an informed decision about the next movement of each robot. When the system reaches the point that each Ground Robot is surrounding the human body from both sides, they receive a signal from the Guide robot to start the rescue routine during which they move a little bit forward and start lifting their forklifts.

13 5. Testing and Evaluation 5.1 Testing Methodology We tested the components of our system before integrating and making our final test. The following were the components that we tested on their own: Network Modules After implementing our network modules on a desktop computer, we tested the module on two computers and made sure text is being sent correctly. After we succeeded, we implemented sending images. Then, we implemented our module on the UDOO board (which is on the Guide Robot) and made sure it is sending and receiving correctly to and from a desktop computer. Lastly, we implemented the module on the two Raspberry Pis (which are on the two Ground Robots) and made sure every component is working well. Human Detection We tested the neural network based code obtained from Caffe by getting our own data set, which consists of different picture of humans we got in the lab. We tested to make sure it is working correctly. Also, we wanted to know the capabilities of the system and how it performs on different sizes, color skins, and shapes. Robot Detection We implemented a computer vision code to extract the location of our two Ground Robots based on pictures taken by the Guide Robot. Based on that, the Guide Robot tells the Ground Robots how to move. We tested the computer vision code thoroughly before integrating it with the robot movement codes. This leads us to the next point. Robot Movement The robot movement codes were first implemented on an Arduino board and tested on the wheels separately. The reason behind that was to minimize the cost of losing a Raspberry Pi in case the movement caused problems. After the codes were successful, we mounted them on the robot and on the Raspberry Pi. We made a separate executable to test all the movements of the robots before integrating them to the whole system. Integrating and Testing After making sure all the components work correctly, we integrated them together to test our finalized system. 5.2 Testing Results

14 Testing the entire system was not as hectic as we thought it would be since every component was tested on its own. The only part that needed fine-tuning after integration was the movement of the robot relative to its location. We needed to make sure it does not move more or less than it needs to. Other than that, all the other problems that appeared were minor problems that were fixed on the spot. 5.3 Discussion MARRS was definitely not the easiest project to implement and it needed a lot of hard work and determination to be built and functioning well. However, with a lot of research, work, and willpower, we were able to deliver the system we promised we would deliver. As can be seen, we have done intensive testing on the system to be able to reach this point. 6. Challenges We have met many challenges through our journey of building and programming the robots. This section will list these challenges and describe them in details. 6.1 Lack of Robot Components Availability a) Omni-directional wheels Using normal wheels allows the system to move in only two directions, which puts our system at a disadvantage. Our system is supposed to operate in narrow spaces and rescue humans to get them to safety. Hence, we wanted to provide our ground robots with maximum flexibility when it comes to motion, and so, we decided to use omnidirectional wheels to provide our system with this flexibility. The challenge we faced, however, was the availability of those wheels. In order to acquire them, we had them especially manufactured for us. There were a lot of delays and issues with the design and procurement of the wheels, which created a major setback for us. The alternative was importing the wheels, which was not a feasible option due to time constraints and high cost. b) Motors While testing the system, one of the motors broke down. And to obtain another motor with the exact dimensions and specifications was impossible. This was the toughest challenge among all because it was only a week before our final presentation. As a result, we had to resort to using another motor and fixing it on the body of the robot using a different approach so that all four wheels can move again. 6.2 Lack of Workshop Facilities Our prototype is built from scratch. Consequently, we needed heavy support when it came to mechanical parts and workshop facilities. Unfortunately, these facilities were not available at the mechanical workshops at our university. Therefore, we had to

15 outsource all our operations to workshops outside university, which consumed a lot of time and added complexity to the development process. 6.3 Human detection Before reaching the decision of using neural networks as a method of detecting humans, we were faced with many confusing choices. We had different implementations using Pattern Analysis algorithms, Computer Vision algorithms, or Neural Networks algorithms. We explored the option of Computer Vision and Pattern Analysis but found put the Neural Networks algorithms such as the module we are using gives off the highest accuracy percentage. 6.4 Different Kinds of Floors The material used to manufacture the wheels is very sensitive to the different kinds of floors. We realized this when we tested it in multiple locations. In our laboratory, where most experiments took place, the environment was very convenient. However, during rehearsals, we noticed that the floor in the presentation hall was very slippery causing the robots to drift and move at a different speed that they are supposed to. This created an overhead because we had to adjust the parameters of the robot motion whenever we changed testing locations. 7. Conclusion and Future Work In order to turn our prototype into an actual product, we need to make a few adjustments. Our product works in an environment where a poisonous gas been leaked and the site has been rendered inaccessible by human rescuers. We want our product to be able to explore a new environment successfully, locate and identify survivors, and even return useful information about the sight, in case it did not find any victims. In other words, we are employing rescue robotic systems to help reduce the number of casualties in the victims and rescuers side and provide faster intervention. Some of the upgrades we need to make to our prototype are: 7.1 More Mobile Guide Robot We propose changing our static guide robot guide into a drone. This was involved in our initial design but due to the fact that the import of drones has been banned in Egypt and that, locally, we could not find a convenient drone, we used a static guide robot instead. A more flexible guide robot allows us to collect more information about the accident site, locate more victims and cover a wider search area than a static guide. 7.2 More Ground Robots

16 Our prototype works for only a mannequin with maximum weight 20 kg. As we have shown in our initial design scheme, in order to rescue an actual human being, we need the weight of the human to be distributed among at least six ground robots so that they can get the human to safety. 7.3 Using 3D Cameras/Deep Sensors As per our initial design, we wanted the ground robots to be able to detect obstacles and avoid them while they move about the scene. Therefore, we propose employing 3D cameras or deep sensors on the ground robots to detect those obstacles. These sensors will yield more accurate results than the approach used in our system, which only looks for obstacles from the guide robot s point of view. 7.4 Stronger Motors: Since we will build more robots to support heavier weights, we will also need stronger motors with better specifications and more power so that they can lift heavier weights and allow the robots to move with an appropriate speed while carrying those weights. In conclusion, we have successfully built and programmed from scratch a prototype for a Multi Agent Rescue Robotic System to save human survivors from an area where gas has leaked. We hope that this system could save human lives and save the lives of rescuers who go inside the hazardous environments and risk their lives to save other humans.