Computer Vision Navigation for Robotic Campus Guide

Transcription

1 FURI Proposal for Fall 2018 Education, Sustainability Computer Vision Navigation for Robotic Campus Guide Zakk Giacometti, Computer Systems Engineering Advisor: Dr. Armando A. Rodriguez, Professor of Electrical Engineering Background: Robotic systems are steadily integrating into our society, working not only for humans, but alongside them. Consequently, this makes research into autonomous systems in human environments important. General robotic tasks such as perception, cognition, and action become much more difficult when the presence and actions of humans must be considered. Navigation is one such specific task that must be conquered for robots to operate outside of more structured environments such as factories [1]. Robotic guides (e.g. tour guides) are one application where navigation strategies considering humans are central to their operation. Robotic tour guides have previously been deployed in museums, fairs, and, expos allowing research to focus on making such systems more intelligent and improving integration with the environment [2]. Research in this area will contribute to improving the intelligence of robots in less structured human environments and help to pave the way for more sophisticated interactions between humans and robotic systems. The proposed research fits well in the FURI themes of sustainability and education, helping to incorporate intelligent robots into the college campus environment. Additionally, the proposed work will be thoroughly documented. As such, each of the algorithms and hardware/software used will be readily incorporated in relevant classes on robotics, planning, systems, programming, sensing and control. These include introduction to engineering as well as senior design. Moreover, the developed vehicle will be used for outreach purposes (e.g. visiting community colleges, high schools, middle and elementary schools, visits to ASU). As such, we expect the contributions of the project in education to be significant. Main Objective: The proposed research will improve on the wheeled guide robot platform developed during the Spring 2018 semester (FURI 1) to guide a person from building to building on the ASU campus. This will involve further refining the algorithms developed, fusing measured data from additional sensors. Focus will be placed on higher level behaviors; e.g. determining optimal routes and continuously updating those routes in response to obstacles observed. The algorithms and their integration are the main contribution of the research. The application of these algorithms require them to maximize efficiency to allow responsiveness in human environments (e.g. at typical human walking speed). The research will culminate in a final demo, detailed below. Overview of Work Accomplished (FURI 1): A strong software and hardware foundation has been laid for the design of robot guide navigation and control algorithms. The hardware provides two-wheeled differential drive control, motors with precision encoders, front and rear vision capability using two cameras, and LIDAR mapping capability for obstacle detection similar to that designed by Dr. Rodriguez s students [8]-[14]. An Arduino Mega provides motor-level control logic. A Raspberry Pi 3 provides high-level logic; e.g. for simple image processing. The software is a modern Robot Operating System (ROS) software stack with visual marker (AprilTag) recognition capability and closed loop software simulation with Gazebo. Other tools such as rqt, rviz, and OpenCV are leveraged from the ROS ecosystem where possible. It is capable of identifying a series of markers placed on the ground, determining their position and orientation relative to the robot base, and following the path outlined by them as well as identifying a marker in its rear vision to track the following person. The rear facing camera is used to identify a marker designating the person to be guided. This allows for those following the robot to be monitored without more complex and expensive mapping systems such as those described in [5]. The spacing control ideas within [9]-[10], [14] will be used. The robot will have two modes of operation, switchable on command from a human operator: (1) A manual control mode that enables a human operator to wirelessly steer the robot and (2) an autonomous guiding mode that requires no human input. Continuing work will lay the foundation for plotting routes between high-level landmarks (e.g. buildings) and advanced obstacle detection and avoidance algorithms. Details of Proposed Improvements (FURI 2): The NVIDIA Jetson TX2 (tested during FURI 1) will replace the Raspberry Pi as the main computing platform. This brings hardware accelerated image processing capability and increased computational resources to improve responses at typical human walking speed (and faster) without greatly sacrificing other constraints such as weight and power consumption. Navigation between markers will be further refined, incorporating ideas from [7]. Data from visual, LIDAR, GPS, and ultrasonic will be further

2 integrated into the path planning algorithm to continuously update the path to be traversed. Errors in these sensor data will be accounted for through techniques such as Kalman filtering [ZZ]. Algorithms to navigate shortest paths between buildings will also be implemented. This requires information about the relationship between buildings (e.g. a graph) to be available [ZZ]. Critical Questions to be Answered: Critical questions to be addressed are as follows: 1. What course of actions are best taken by a robot in a guide role when needing to, for example, follow a designated path to a destination while avoiding uncertain obstacles? [ZZ], [ZZ], [ZZ], [ZZ]. 2. How can we best manage the computationally heavy tasks described given limited computing resources? [ZZ], [ZZ], [ZZ], [ZZ]. Specific Technical Questions: Specific technical questions to be addressed include: 1. How do we accurately and swiftly correct our determined path in response to obstacles (e.g. person crossing path) while maintaining the guided person at a pre-specified range (e.g. 5-8 feet)? [ZZ], [ZZ]. 2. How can the robot achieve the objective of getting from one building to another with minimal or no initial knowledge of the global environment? This may require forming and storing an internal map and then using it to set way points and then moving accurately and swiftly from waypoint to waypoint while avoiding obstacles along the path [ZZ], [ZZ]. This requires full use of GPS, cameras, LIDAR, and ultrasonic sensors. 3. When will remote intervention be necessary (e.g. kill switch, tentative human remote control)? 4. For each of these, how do we systematically trade off computational speed and accuracy? Eventually, we can consider exploiting distributed computational resources (e.g. desktops situated in buildings along the path) to augment the very limited on-board computational resources. Each of the above questions are (strictly speaking) open design questions. Developing precise (qualitative and quantitative) answers to each of the above will permit the pursuit of more advanced work in the area; i.e. improving the robot so that it is nearly fully autonomous. The following key algorithms define the autonomous behaviors of the system: 1. Determine the shortest sequence of major landmarks (and their corresponding visual markers) to reach the desired destination. 2. Identify the visual marker designating the next waypoint along the path using the front facing camera and advance toward it (at typical walking speed), matching its designated heading [7], [8], [10], [12]. 3. Detect obstructions using data from ultrasonic, LIDAR, and the front facing camera. Stop to avoid a collision or apply course corrections if possible [4], [10], [14]. 4. Identify the visual marker designating the follower using the rear facing camera. Slow to a stop if the marker is not detected or leaves visual range [8]-[10]. YOU HAD # 5 WITH NOTHING???? Test Assumptions: Continuing on the previous FURI 1 work, more complex testing conditions will be introduced. Among them will be non-ideal terrain such as slopes and sharp turns, occluded markers, and nonideal lighting conditions. The original assumptions that paths are continuous and as designated by markers will hold, simplifying path planning. Obstacle crowding will be kept to a minimum, allowing simple obstacle avoidance algorithms to be developed and integrated into path planning [4], [10], [14]. Final Demo and Future Work: A final working demo will be conducted with the robot guiding a person between buildings along Tyler Mall on the ASU Tempe campus. Visual markers will be placed at regular intervals to mark the path and designated buildings. GPS will also be used. Full autonomous mode, manual control override, and obstacle avoidance will be demonstrated. This proposed work and demonstration will lay a foundation for future improvements to the system that can handle larger groups of guided persons and more complex navigation that may cover an entire campus. Additionally, visual markers on humans can be seamlessly replaced with heavier vision based solutions (machine learning?? algorithms) such as that described in [6]. Final Documentation: All project results will be documented in a final comprehensive report and on the required final poster. The work will also be submitted for publication within the proceedings of the American Control Conference (ACC), the Frontiers in Education (FIE) and ASEE Conferences. Career-Relevance: The proposed research will serve to further build on previous research conducted as well as serve as a platform for future advanced research and fellowships on my path to obtain a direct PhD. To facilitate the latter, I intend to apply for an NSF Fellowship this coming Fall in order to begin my PhD during Fall 2019.

3 References [1] Kruse, Pandey, Alami, & Kirsch. (2013). Human-aware robot navigation: A survey. Robotics and Autonomous Systems, 61(12), [2] López, Joaquín. (2013). GuideBot. A Tour Guide System Based on Mobile Robots. International Journal of Advanced Robotic Systems, 10(11), International Journal of Advanced Robotic Systems, 2013, Vol.10 (11). [3] Gomez, C., Hernandez, A., Crespo, J., & Barber, R. (2016). A topological navigation system for indoor environments based on perception events. International Journal of Advanced Robotic Systems, 14(1), International Journal of Advanced Robotic Systems, 2016, Vol.14(1). [4] Cherubini, A., Spindler, F., & Chaumette, F. (2014). Autonomous Visual Navigation and Laser- Based Moving Obstacle Avoidance. Intelligent Transportation Systems, IEEE Transactions on, 15(5), [5] Kanda, Arai, Suzuki, Kobayashi, & Kuno. (2014). Recognizing groups of visitors for a robot museum guide tour. Human System Interactions (HSI), th International Conference on, [6] Gupta, M., Kumar, S., Behera, L., & Subramanian, V. (2017). A Novel Vision-Based Tracking Algorithm for a Human-Following Mobile Robot. Systems, Man, and Cybernetics: Systems, IEEE Transactions on, 47(7), [7] Gomez, C., Hernandez, A., Crespo, J., & Barber, R. (2016). A topological navigation system for indoor environments based on perception events. International Journal of Advanced Robotic Systems, 14(1), International Journal of Advanced Robotic Systems, 2016, Vol.14(1). [8] A. A. Rodriguez, K. Puttannaiah, et. al. (2017). Modeling, Design and Control of Low-Cost Differential-Drive Robotic Ground Vehicles: Part I - Multiple Vehicle Study, IEEE, Conference on Control Technology and Applications. [9] Z. Li. (2013). Modeling and Control of a Longitudinal Platoon of Ground Robotic Vehicles," Master Thesis, ECEE, ASU. Committee: A. A. Rodriguez, A. Panagiotis, S. Berman. [10] Z. Lin. (2015). Modeling, Design and Control of Multiple Low Cost Robotic Vehicles, Master Thesis, ECEE, ASU. Committee: A. A. Rodriguez, J. Si, S. Berman. [11] X. Lu. (2016). Modeling and Control for Vision based Rear Wheel Drive Robot and Solving Indoor SLAM problem using LIDAR, Master Thesis, ECEE, ASU. Committee: A. A. Rodriguez, A. Panagiotis, S. Berman. [12] J. Lopez. (2016). Image Processing Based control of Mobile Robots, Master Thesis, ECEE, ASU. Committee: A. A. Rodriguez, A. Panagiotis, S. Berman. [13] I. Anvari. (2013). Non-Holonomic Differential Drive Mobile Robot Control and Design: Critical Dynamics and Coupling Constraints, Master Thesis, ECEE, ASU. Committee: A. A. Rodriguez, K Tsakalis, J Si. [14] D. Chopra. (2013). Feedback Control and Obstacle Avoidance for Non- Holonomic Differential Drive Robots, Master Thesis, ECEE, ASU. Committee: A. A. Rodriguez, K Tsakalis, J Si

4 Timeline For Fall 2018 Task Description Duration Start Finish 1 Sensor Integration 2 Obstacle Avoidance 3 Path Planning 4 Refinement 5 Reflection Integration additional sensors into ROS workflow. Further develop strategies for obstacle avoidance. Integrate obstacle avoidance strategies to continuously update path. Focused refinement of path planning algorithms. Assess performance metrics of the system and prepare for Symposium 7 Days 08/16/18 08/24/18 25 Days 08/27/18 09/28/18 20 Days 10/01/18 10/26/18 10 Days 10/29/18 11/9/18 10 Days 11/5/18 11/16/18 6 Wrap Up Finalize reports. 10 Days 11/19/18 11/30/18

5 Personal Statement My major motivation for pursing this FURI is to catalyze further research and projects, which will prepare me for my senior design project, graduate school, and future career. Immediately after finishing my bachelors degree, I plan to pursue a graduate degree in Computer Engineering focusing in robotics. My ultimate goal is to work in a career related to this field, particularly relating to applications in aerospace. This may take the form of ground robots, space and satellite systems, or planetary exploration vehicles. I believe this area lies at the cutting edge of technology and what can be gained from exploring it is vitally important to solving large-scale issues on Earth. This area requires extensive knowledge and study, which strongly motivates me to pursue a PhD following graduate school. Toward that purpose, I wish to gain valuable experience now. The associated stipend will allow me to focus on gaining this experience and help to relieve financial burdens on my family and myself. I intend to further translate the experience gained from this FURI into a National Science Foundation Graduate Fellowship this coming October. This will further expand on the guide robot concept in applications such as emergency situations and search and rescue. These applications are stepping stones that will prepare me for even more advanced research as described above. I have been deeply influenced by the opportunities afforded to me by Mesa Community College (MCC). I was able to engage with motivated peers, and seek close guidance from my professors. Through the MCC Engineering Club I was able to explore the different disciplines of engineering from outside the classroom environment, engaging with those in the industry to find where my interests aligned. I also had the opportunity to be selected for the NASA Community College Aerospace Scholars program (Fall 2017), a summer long program ending with a four day workshop working on a robotics project with fellow participants from around the country. I was inspired by the NASA engineers who came to speak and work with us. I was equally inspired by those community college students who were there alongside me. An internship at Marshall Space Flight Center (May-August 2017) deeply immersed me in an engineering environment. I worked extensively with a robotics software framework (ROS) on a system that was to be used by engineers in prototyping controls systems. I was introduced to an entirely new space of engineering and experienced a completely new perspective on where I fit in this area. Working closely with true experts in their field reinforced my motivation to continue learning and pushing the limits of my understanding. This summer, I will be participating in an internship at Ball Aerospace in Boulder, CO in their Software Engineering division. I targeted Ball for their expertise with satellite systems, which would contribute more experience working with embedded systems and sensor data and insight to my goals of working with space robotics. My coursework and FURI this spring has solidified my goals and desire to pursue them. Working closely with my mentor and other students, through both FURI and the Academic Success and Professional Development program has helped shape my vision for what lies ahead in terms of the research I can conduct and the education required for it. The work I am doing now with ROS for FURI and with embedded systems in CSE325 (Embedded Microprocessor Systems) has reinforced that this is not only something that I am passionate about, but that has many possibilities and applications that others may benefit from. Pursuing the proposed project is of the highest importance to me. It will allow me to apply what I have learned in the classroom to something that offers the potential to benefit society at large. To reiterate, it will prepare me for future projects, graduate work, and my future career, and provide an avenue to present my research and contribute to this field of study.