Wednesday, October 29, 2014 02:00-04:00pm EB: 3546D TELEOPERATION OF MOBILE MANIPULATORS By Yunyi Jia Advisor: Prof. Ning Xi ABSTRACT Mobile manipulators provide larger working spaces and more flexibility than standard manipulators by introducing mobility. Through teleoperation, they can be applied to a variety of areas such as hazardous material handling, outer space exploration, explosive ordnance disposal, etc. Inspired from application requirements, there are four major challenges in the system including modeling and control of mobile manipulators, teleoperation of multiple mobile manipulators, modeling human teleoperator in teleoperation system and communications between human teleoperator and mobile manipulator. Therefore, this study aims to address these challenges. For modeling and control of mobile manipulators, the motion control accuracy of the endeffector is a problem for the existing methods due to the system dynamics differences. This is an important impact on the efficiency of teleoperation. To address this issue, we introduce an online motion distribution method and an online coordinated control method which could significantly improve the end-effector accuracy and teleoperation efficiency. For teleoperation of multiple mobile manipulators, the system stability under random communication delays and unexpected events is a major problem for the existing methods. To address this issue, we propose a non-time based teleoperation and coordination method. A nontime perceptive reference is designed as a new reference to replace the time in the system
modeling and control. Through this design, the system stability under random communication delays and unexpected events could be ensured. For modeling human teleoperator in teleoperation system, there are no existing models for the human teleoperator and the teleoperation efficiency and safety are always subject to the operation status of the teleoperator in teleoperation. To address this issue, we propose a concept named quality of teleoperator (QoT) to represent the teleoperator and incorporate it into the modeling and control of the teleoperation system. Through this design, the teleoperation efficiency and safety could be improved under various operation status of the teleoperator. For communications between human teleoperator and mobile manipulator, the existing methods of using joysticks are neither efficient nor intuitive. Therefore, we introduce the natural language as a new communication manner. However, the existing natural language control methods could not online handle unexpected events in the environment and robotic system. To address this issue, a new natural language control method and a new robot planning and control method are designed to online handle such unexpected events. The proposed methods are all implemented on our developed MSU mobile manipulators. The experimental results demonstrate and validate their effectiveness and advantages. Publications Published and Accepted Publications Y. Jia, N. Xi, S. Liu, Y. Wang, X. Li and S. Bi, "Quality of Teleoperator (QoT) Adaptive Control for Telerobotic Operations," International Journal of Robotics Research, 2014. (In press) J. Zhao, B. Song, N. Xi, L. Sun, H. Chen and Y. Jia, Non-vector Space Approach for Nanoscale Motion Control, Automatica, vol. 50, no. 7, pp. 1835-1842, 2014 Y. Jia, N. Xi, Y. Cheng and S. Liang, "Coordinated Motion Control of A Nonholonomic Mobile Manipulator for Accurate Motion Tracking," IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2014, pp. 1635-1640. L. She, Y. Cheng, J. Chai, Y. Jia, S. yang and N. Xi, "Teaching Robots New Actions through Natural Language Instructions," IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2014. Y. Cheng, Y. Jia, R. Fang, L. She, N. Xi and J. Chai, "Modelling and Analysis of Natural Language Controlled Robotic Systems," World Congress of the International Federation of Automatic Control (IFAC), 2014, pp. 11767-11772.
Y. Jia, N. Xi, J. Chai, Y. Cheng, R. Fang and L. She, "Perceptive Feedback for Natural Language Control of Robotic Operators," IEEE Conference on Robotics and Automation (ICRA), 2014, pp. 6673-6678. Y. Jia, N. Xi and E. Nieves, "Coordination of a Nonholonomic Mobile Platform and an On-board Manipulator," IEEE Conference on Robotics and Automation (ICRA), 2014, pp. 4356-4361. L. She, S. yang, Y. Cheng, Y. Jia, J. Chai and N. Xi, Back to the Blocks World: Learning New Actions through Situated Human-Robot Dialogue, 15th Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL), 2014, pp. 89-97. H. Zhang, Y. Jia, Y. Guo, K. Qian, A. Song and N. Xi, "Online Sensor Information and Redundancy Resolution Based Obstacle Avoidance for High DOF Mobile Manipulator Teleoperation," International Journal of Advanced Robotic Systems, vol. 10, pp. 1-9, 2013. J. Liu, Y. Jia and N. Xi, "Identification of Road Surface Conditions Based on Laser Scanning," IEEE Conference on Robotics and Biomimetics (ROBIO), 2013, pp. 2476-2481. X. Wang, Y. Jia and N. Xi, "Mobile Robot Pose Estimation Using Laser Scan Matching based on Fourier Transform," IEEE Conference on Robotics and Biomimetics (ROBIO), 2013, pp. 474-479. E. Nieves, N. Xi, Y. Jia, C. Martinez and G. Zhang, Development of a position sensitive device and control method for automated robot calibration, IEEE/RAS International Conference on Automation Science and Engineering (CASE), 2013. pp. 1133-1138. D. Gong, Y. Jia, Y. Cheng and N. Xi, "On-line and Simultaneous Calibration of A Mounted Force/Torque Sensor and Tool Forces/Torques for Manipulation," IEEE Conference on Robotics and Biomimetics (ROBIO), 2013, pp. 619-624. F. Chen, Y. Jia and N. Xi, "Non-invasive EEG based Mental State Identification Using Nonlinear Combination," IEEE Conference on Robotics and Biomimetics (ROBIO), 2013, pp. 2160-2165. Y. Liu, Y. Jia and N. Xi, "Model and Control for 4-Powered Caster Vehicle: A Probability-based Approach," Transaction of the Institute of Measurement and Control, vol. 35, no. 7, pp., 875-882 2012. Y. Jia, N. Xi, Y. Wang and X. Li, "Online Identification of Quality of Teleoperator (QoT) for Performance Improvement of Telerobotic Operations," IEEE Conference on Robotics and Automation (ICRA), 2012, pp. 451-456. C. Zhang, J. Xu, N. Xi, Y. Jia and W. Li, "Development of an omni-directional 3D camera for robot navigation," IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), 2012, pp. 262-267. J. Zhao, Y. Jia, N. Xi, W. Li, B. Song and L. Sun, "Visual Servoing Using Non-vector Space Control Theory," IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), 2012, pp. 87-92. Y. Jia, N. Xi, S. Liu, H. Zhang and S. Bi, "Multi-Objective Optimization for Telerobotic Operations via the Internet," IEEE/RSJ Conference on Intelligent Robots and Systems (IROS), 2012, pp. 5197-5202. H. Zhang, Y. Jia and N. Xi, "Sensor-based redundancy resolution for a nonholonomic mobile manipulator," IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012, pp. 5327-5332. W. Li, C. Zhang, N. Xi and Y. Jia, "Hyperspectrum Image Fusion for Sensor Guided Mobile Manipulations," IEEE Conference on Robotics and Biomimetics (ROBIO), 2012, pp. 2288 2293.
E. Nieves, N. Xi, B. Du and Y. Jia, "A reflected laser line approach for industrial robot calibration," IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), 2012, pp. 610-615. H. Zhang, Y. Jia, N. Xi and A. Song, "Obstacle Avoidance for Mobile Manipulation by Real-Time Sensor-Based Redundancy Resolution," IEEE Conference on Robotics and Biomimetics (ROBIO), 2012, pp. 2369 2374. S. Liu, N. Xi and Y. Jia, "On-line Operator Skill Assessment for Telerobot Operation Using Electroencephalo-graph (EEG)," IEEE Conference on Robotics and Biomimetics (ROBIO), 2012, pp. 1598 1603. I. Elhajj, N. Boudargham, N. Xi and Y. Jia, "Real-Time Adaptive Content-Based Synchronization of Multimedia Streams," Advances in Multimedia, vol. 2011, pp. 1-13, 2011. Y. Jia and N. Xi, "Coordinated Formation Control for Multi-Robot Systems with Communication Constraints," IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), 2011, pp. 158-163. X. Li, Y. Jia, Y. Wang and N. Xi, "Image Based Approach to Obstacle Avoidance in Mobile Manipulators," IEEE Conference on Robotics and Biomimetics (ROBIO), 2011, pp. 1658 1663. Y. Jia, X. Li, Y. Jiang and N. Xi, "Sensor-based redundancy resolution for a mobile robotic manipulator," Unmanned Systems Technology XIII Conference, SPIE Defense, Security, and Sensing, 2011, pp. 101-105. Y. Jia, N. Xi, F. Wang, X. Li and Y. Wang, "Telerobotic Systems Adaptive to the Quality of its Operators," 3rd Int. Joint Topical Meeting on Emergency Preparedness and Response and Robotics and Remote Systems (EPRRSD) and 13th Robotics and Remote Systems for Hazardous Environments, Knoxville, TN, August, 2011, pp. 619-629. Y. Wang, Y. Jia, X. Li and N. Xi, "Dynamics Modeling of A Mobile Manipulator for Wheel Slip Avoidance," IEEE Conference on Robotics and Biomimetics (ROBIO), 2011, pp. 1621 1626. Y. Jiang, N. Xi, Q. Zhang and Y. Jia, "Target Object Identification and Localization in Mobile Manipulations," IEEE Conference on Robotics and Biomimetics (ROBIO), 2011, pp. 144 149. Y. Jia, N. Xi, F. Wang, Y. Wang and X. Li, "Controlling telerobotic operations adaptive to quality of teleoperator and task dexterity," IEEE/RSJ Conference on Intelligent Robots and Systems (IROS), 2011, pp. 184-189. Y. Jia, N. Xi and J. Buether, "Design of single-operator-multi-robot teleoperation systems with random communication delay," IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2011, pp. 171-176. H. Wang, Y. Jia, N. Xi, and J. Buether, "An online motion plan algorithm for a 7-DOF redundant manipulator," IEEE Conference on Robotics and Biomimetics (ROBIO), 2010, pp. 1057-1062. Y. Jia, H. Wang, P. Stürmer, and N. Xi, "Human/robot interaction for human support system by using a mobile manipulator," IEEE Conference on Robotics and Biomimetics (ROBIO), 2010, pp. 190-195. Y. Liu, Y. Jia, and N. Xi, "Dynamic model and adaptive tracking controller for 4-powered caster vehicle," IEEE Conference on Robotics and Automation (ICRA), 2010, pp. 4940-4945. P. Stürmer, H. Wang, Y. Jia, N. Xi and Y. Liu, "Foster-Bot: Multi-Sensor Enhanced Robot for Assisting the Handicapped in Remote Areas", IEEE International Hands-on Competition via Internet onintelligent Mechatronics and Automation (HCIMA), 2009, pp. 1-5.
Y. Liu, N. Xi, J. Zhao, E. Nieves, Y. Jia, B. Gao and J. Lu, "Development and Sensitivity Analysis of A Portable Calibration System for Joint Offset of Industrial Robot," IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2009, pp. 3838-3843. Submitted Publications Y. Jia, L. She, Y. Cheng, N. Xi and J. Chai, "Handling Unexpected Events in Natural Language Controlled Robotic Systems," IEEE Conference on Robotics and Automation (ICRA), 2015. (Submitted) Y. Cheng, C. Su, Y. Jia and N. Xi, "Correlation based Slippage Detection in Robotic Manipulators Using Tactile Matrix Sensor," IEEE Conference on Robotics and Automation (ICRA), 2015. (Submitted)