Robot Remote Control Using Bluetooth and a Smartphone Augmented System Gaowei Chen, Scott A. King, Michael Scherger School of Engineering and Computing Sciences, Texas A&M University Corpus Christi, Corpus Christi, TX, 78412, USA gchen@islander.tamucc.edu, scott.king@tamucc.edu, michael.scherger@tamucc.edu Abstract. In this paper, we present an innovative robotic system with augmented smartphones. We propose a smartphone accelerometer controlled differential steering algorithm and implement it on a two-motor driven robotic vehicle with real-time video rendering. In the system design, we create two selfvalidating protocols that build upon our Bluetooth communication network. The overall performance is evaluated by two projects, MOCOVE and RIZZY. The proposed system can be used in robotic remote control, autonomy, robotic localization and human-robot interaction. Keywords: mobile robot, smartphone, remote control, human-robot interaction 1. Introduction Research on robotics has gone on for several decades. Despite dissimilar appearances, robots all share a common mechanical and moveable structure under three distinct phases of control: perception, processing, and action. A robot uses sensors to perceive the environment. Consumer robots tend to have controllers with limited computational power and small amounts of memory. Sensors for consumer robots are also quite costly. For instance, a HiTechnic NXT Acceleration/Tilt sensor is sold for $55 at its official website, and a Lego Mindstorms TM sound sensor sells for $30. In addition, it takes microcontroller resources to manage these mounted sensors. With the advancement of contemporary smartphone technology, many common sensors are already embedded into a smartphone. Given the limited processing and high cost of individually packaged sensors, we chose to augment a consumer robot with a smartphone gaining the advantage of more computational power combined with numerous sensors. The greatest advantage of using a mobile phone to remotely control a robot is location independence [12]. There are a number of existing articles studying the development of communication models between a cellular phone and a robot. Smartphones tend to be powerful platforms for robotic autonomy and remote control. A contemporary smartphone possesses many auxiliary features (e.g. digital compass, accelerometer, audio, microphone and camera) and many convenient communication interfaces (e.g. Wi-Fi, Bluetooth, etc.). There is no doubt that smartphone processors have higher computational power and more memory than common microcontrollers.
This motivates us to integrate a smartphone into robotic systems in order to tackle complex tasks and improve the overall performance. In this research, we implemented a differential steering system with smartphone motion control. In addition, we develop two self-validating protocols for Bluetooth communication. The overall performance of the proposed system is evaluated by two projects we designed, MOCOVE (Motion Control Vehicle) and RIZZY. Our system can be used in remote control, robotic autonomy, robotic localization and human-robot interaction. Section 2 discusses the related research. Section 3 explicitly depicts our system design and implementation. Section 4 evaluates our work and Section 5 concludes. MOCOVE Project screenshots 2. Related Research Previous articles focus on remotely controlling a robot using cellular phones. Some communication models were proposed, especially through wireless connection. There are mainly two service platforms utilized for building the wireless communication between a cellular phone and a robot: to use it as a mobile service platform [6] [7][8] [9] [11] [12] or to communicate through the networks [3] [4] [10]. One way to achieve remote control is through telephony service. Dual tone multi frequency signaling (DTMF) is used for telecommunication and telephone signaling over the line in the voice-frequency band between phones and the switching center. [12] DTMF assigns each key on the dial pad a specific dual tones frequency. Based on DTMF, a microcontroller can easily identify the input key (i.e. pressing '5' results in sending a tone made by adding 1336 Hz and 770 Hz) [9]. Although it becomes relatively simple for signaling, it would be difficult to develop a complicated protocol using DTMF. A mobile phone is not only limited to voice calls. For instance, Simple Message Service (SMS) can be utilized for building a communication network. SMS is a mobile service transmitting a short message to a service-activated mobile phone. One communication model illustrates the process of how a SMS message is used to remotely control a robot [12]. The message is first sent and analyzed on an external server, which connects to a local IP network. The content of the SMS message contains a special protocol. After validation and verification of the message by the server, the exact control command extracted from the protocol will be sent to the
robot over the IP network. Using mobile services, low cost, large service location coverage and wide support by most service providers are the prominent advantages. However, because of the nature of the discrete SMS messages sending process, it is impossible to develop a continuous control flow on the robot. Wireless application protocol (WAP) is an open and international standard commonly used for mobile phones. WAP mobile phones can be used for a location independent remote robot control system. However, WAP is not really desirable due to its comparatively high cost, low data rate, high latency and uncomfortable data input [3]. Additionally, in order to develop a voice-controlled robot, the function of speech recognition (and speech understanding) is necessary [8]. As well as voice control, a robot can detect and even recognize human faces with facial detection functionality. 3. System Design and Implementation 3.1 Challenges and Problems On one hand, smartphone hardware and its operating system exert computational influence on problem solving. On the other hand, the communication latency between the smartphone and the microcontroller sets up a performance gap. Real-time remote control and synchronized data surveillance is a problem we encountered in this research. In addition, our proposal of having the smartphone augmented robot is relatively new. Until now, there are few mature solutions for the integration of a robot and smartphone. One solution is Cellbots [2], an open source project launched by Google Cloud Robotics, where robotic hobbyists and researchers are promoting a similar concept. 3.2 System Design and Implementation This innovative integration builds upon contemporary smartphone technology. To conquer the challenges, our methodology is to split problems into individual pieces, and solve them separately. This section describes the design and implementation of our system. The microcontroller is programmed with triggering logic under a thread-parallel structure. We implemented a motion control differential steering system based on the smartphone s accelerometer and achieved real-time remote control. Additionally, we adopt Bluetooth technology in our system design and implementation. Rather than SMS, DTMF, WAP, Bluetooth is more competent and fairly compatible. There are some existing projects using wired connection through USB interface. However, this structure requires assembling an extra break-out board. In contrast, Bluetooth does not require any tangible modification of hardware. In addition, the advantages such as robust connection, fast and continuous data flow propagation, low power consumption, and cheap cost are highly appreciated. Moreover, self-validating protocols are designed for building a reliable communication with as little overhead
as possible. Reliability implies a system with a robust continuous control flow. Security is one concern. By nature, Bluetooth is broadcasting. If a malicious user hacks in the system the robot can be kidnapped. Security is beyond the scope of this project, but we have active research projects in our department for securing Bluetooth communication that could be utilized if security was an issue. We built two projects to evaluate the proposed system. MOCOVE is a two-motor-driven robot vehicle that is remote controlled by a smartphone. MOCOVE has real-time video rendering on the controlling device. The motion control functionality is implemented so as to be similar to steering a car wheel. It uses an accelerometer sensor on a smartphone and is controlled by a human operator. Another smartphone mounted on MOCOVE transfers back real-time video from its camera. RIZZY is a university mascot robot that interacts with humans. The brained humanoid robot detects a human face and is able to make a simple scripted conversation with the person. 3.3 Object Oriented Concepts in Robotics The microcontroller is designed in a thread-parallel structure where multiple threads are working concurrently. Each thread is defined to conquer one task. This structure allows threads to be added for any incremented tasks. The main program is running in a loop that is ended either when the user presses the exit button or exits from the smartphone interface. Threads will start when the program is instructed to run. Separate threads for Bluetooth communication, locomotion control, and sensor reading are used in MOCOVE. BluetoothThread listens for incoming connections and continuously receives data from the established socket. LocomotionThread implements differential steering algorithm to perform movement operations. SensorReadingThread monitors sensors that are used in the project. All common data is exchanged through an object named CommData among threads. This is implemented on a customized firmware, lejos [6]. The following pseudocode describes this structure. BluetoothThread.start();//start a thread BluetoothThread.setDataEntity(CommData);//DATA exchange LocomotionThread.start(); SensorReadingThread.start(); //add threads here that performs other tasks While(CommData.getExit() or Button.pressExit()) Thread.yield(); 3.4 Smartphone Accelerometer Controlled Differential Steering Algorithm Our robot MOCOVE has a simple and reliable two-wheel-driven propulsion system. It is a typical example of a differentially steered drive system. In Lucas s paper [5], a
kinematics approach is used. The mathematical equations from his paper are especially useful for predicting two-wheel-robot s trajectory. Fig. 1. On left, a simplified two-wheel-robot with different velocities, figure retrieved from [7]. On right, a differential steering system with the input motor speeds from smartphone accelerometer is presented. Where, is the change in orientation V R is the right wheel speed V L is the left wheel speed Differential steering algorithm is intuitive. If V R is equal to V L, the robot moves in a straight line. The Robot moves along a curved path when V R and V L are different. Occasionally, the robot pivots if V R is equal to the opposite of V L. According to differential steering algorithm, the control of a robot is achieved by varying the rotational speed of the two driving motors. We propose an accelerometer controlled differential steering system, where the wheels speeds are obtained and scaled from the smartphone s accelerometer data. To simplify possible scenarios, we developed the equation below for path explanation Where, D is the robot s actual moving direction S is a positive constant number used to scale turning speeds x is the integer that represents left motor speed y is the integer that represents right motor speed As displayed in the right hand of figure 1, robot locomotion is conducted by motor speeds x and y. Seven scenarios could happen after directly changing the motor speeds. The initial scenario is when both x and y are 0 (case 0). The constant S scales the turning sharpness and partially determines the final moving speed. When x and y are the same positives, it is reflected as moving forward in differential drive algorithm (case 1). When x is less than y and both are positives, it s making a left forward turn (1)
(case 2). If both x and y are negative, and x is greater than y, the robot would be reversing towards the left (case 5). We use S to scale each motor s speed to be within the power range. In the actual implementation, the maximum power for a motor speed is up to 900 (8 volts). Each Android device has a coordinate system defined relative to the device screen [1] and accelerometer data is returned in this 3D space. By specifying a delivery rate to the sensor manager, we can acquire accelerometer data and filter improper data out from sharp flipping. In MOCOVE, only y and z are used and scaled properly to be the left and right motor speed. By sending speed integers continuously from smartphone to the microcontroller, a real-time remote control is accomplished. 3.5 Bluetooth Communication and Self-Validating Protocols The Bluetooth protocol is based on a master-slave structure where data is transferred as packets. At low-level transport protocol, RFCOMM (Radio Frequency Communication) creates a virtual serial stream that allows binary data to be transported. Each service is recognized with a unique UUID (Universally Unique Identifier) in a form of 16 bits. Thus, each socket connection is guaranteed unique. Mindstorms TM robotic kits include a powerful but unreliable Bluetooth protocol, which is explained in its BDK (Bluetooth Development Kit). Three channels can be used, whereas at one time a robot can only talk to one Bluetooth device. Nevertheless, we use lejos to build a customized Bluetooth Client-Server structure. Processing large packets by the microcontroller is not recommended. The protocol should have small overhead but be able to validate itself. Therefore, we assume no packet is larger than 256 bits, which is the length of the buffer used to receive data from the socket. We propose the following protocol, Protocol A between the smartphone and the robot, and it is illustrated in table 1. One byte is sacrificed for packet validation. When either the microcontroller or the smartphone receives this protocol, it calculates the whole packet length and converts an unsigned integer value from the head 1 byte. If both values are the same, the received packet is assumed correct. Otherwise, it indicates the packet is not legal in our network and the packet will be ignored. In a common case, data is 32 bits that represent left and right motor speeds. Therefore, a common packet is 48 bits. Table 1. Protocol A used between microcontroller and smartphone Name Packet length Task ID Actual data Data type unsigned int8 unsigned int8 Data length 8 bits 8 bits 0 ~ 240 bits Value range 0 ~ 255 0 ~ 255 Table 2. Protocol B used between smartphones Name Packet length Fragment sequence Fragment total Task ID Actual data
Data type unsigned int8 unsigned int8 unsigned int8 unsigned int8 Data length 2 * 8 bits 8 bits 8 bits 8 bits 0 ~ 472 bits Value range 0 ~ 511 0 ~ 255 0 ~ 255 0 ~ 255 In table 2, we introduce the protocol B used between smartphones. In this case, larger data packets need to be transmitted. And large datasets are split into fragments at the sender side. All fragments are put into a queue and recovered at the receiver side. For instance, images taken by MOCOVE s mounted smartphone are sent to the user end smartphone. 4. Evaluation of Proposed Work 4.1 MOCOVE This section discusses the MOCOVE project, a two-wheel-driven robot controlled by HTC 2.1 controller through Bluetooth. A HiTechnic compass sensor and a Kyocera Echo brain are assembled on the robot for real-time geographical positioning and surveillance vision acquisition. Control and sensor data between smartphone and robot is transmitted using the protocol A. Visual data and voice commands are sent via the protocol B. Video vision from Kyocera is rendered asynchronously on the controller. MOCOVE also sends back its GPS location to the controller, which draws a graph of its path trace. The smartphone accelerometer events are delivered to the sensor listener in nearly real-time, depending on the delivery rate. 4.2 RIZZY RIZZY is a project that demonstrates the proposed system in human-robot interaction. The talking and walking humanoid robot is assembled a Motorola TM XOOM head that hears voice commands from users. The robot will respond to the human talker when hearing special keywords. The first circumstance is the robot detects a human face. When realizing the talker is a human, it reacts. For example, when it hears go ahead, it makes the corresponding action. This feature is accomplished using Google TM Face Detection, by processing a face image. In addition, the feature of text to speech is also accomplished. RIZZY talks under certain circumstances. For instance, when it hears a command it cannot understand, it will say Sorry, I do not understand. The proposed system and multithreading structure enable robot to perform biped walking, data receiving, face detection, speech recognition and speech synchronization.
5. Conclusions and Future Work In this paper, we propose an innovative robotic system with augmented smartphones. In future work, we consider increasing a robot s intelligence by using a smartphone processing heavy image- or algorithms-based problems such as visual gesture control and robotic localization. We foresee the trend that smartphones are growing to be a popular platform for the IT industry. Developing a visual programming language for smartphone is also promising for this project. Instead of having protocols, robot program codes generated by the smartphone using a visual language would give users a complete flexibility to do everything. In future, this could be used to program a home service robot to do a customized routine task such as cleaning a kitchen. Acknowledgements This work was partially funded by NSF Grant #CNS-0708573. References 1. Android SDK, http://developer.android.com/ 2. Cellbots: Using Cellphones as Robotic Control Platforms, http://www.cellbots.com/ 3. d'angelo, P.; Corke, P.;, "Using a WAP phone as robot interface," Robotics and Automation, 2002. Proceedings. ICRA '02. IEEE International Conference on, vol.2, no., pp.1173-1178, 2002 4. Hebah, N.; Amjad, A.;, Smartphone control robots through Bluetooth, IJRRAS 4 (4), Sep. 2010. 5. Lucas G. W.; A tutorial and elementary trajectory model for the differential steering system of robot wheel actuators, The Rossum Project, 2001. http://rossum.sourceforge.net/papers/diffsteer/diffsteer.html. 6. Ji-Dong, Y.; Sungkuk, C.; Keechul, J.; Shaw, C.D.;, "Development of Communication Model for Social Robots Based on Mobile Service," Social Computing (SocialCom), 2010 IEEE Second International Conference on, vol., no., pp.57-64, 20-22 Aug. 2010 7. Kubik, T.; Sugisaka, M.;, "Use of a cellular phone in mobile robot voice control," SICE 2001. Proceedings of the 40th SICE Annual Conference. International Session Papers, vol., no., pp.106-111, 2001 8. Manikandan, D.; Pareek, P.; Ramesh, P.;, "Cell phone operated robot," Emerging Trends in Robotics and Communication Technologies (INTERACT), 2010 International Conference on, vol., no., pp.183-184, 3-5 Dec. 2010 9. Miyazaki, F.; Matsubayashi, S.; Yoshimi, T.; Arimoto, S.;, "A new control methodology toward advanced teleoperation of master-slave robot systems," Robotics and Automation. Proceedings. 1986 IEEE International Conference on, vol.3, no., pp. 997-1002, Apr 1986 10.Rui, Z.; Dongying, J.;, "Based on Mobile Device Multi-Biped Robot Monitor and Coordinated Control," Innovative Computing, Information and Control, 2007. ICICIC '07. Second International Conference on, vol., no., pp.25, 5-7 Sept. 2007 11. Sekmen, A.; Koku, A.B.; Zein-Sabatto, S.;, "Human robot interaction via cellular phones," Systems, Man and Cybernetics, 2003. IEEE International Conference on, vol.4, no., pp. 3937-3942 vol.4, 5-8 Oct. 2003 12. Yun Chan, C.; Jae Wook, J.;, "Remote robot control system based on DTMF of mobile phone," Industrial Informatics, 2008. INDIN 2008. 6th IEEE International Conference on, vol., no., pp.1441-1446, 13-16 July 2008