HAPTIC BASED ROBOTIC CONTROL SYSTEM ENHANCED WITH EMBEDDED IMAGE PROCESSING K.Gopal, Dr.N.Suthanthira Vanitha, M.Jagadeeshraja, and L.Manivannan, Knowledge Institute of Technology Abstract: - The advancement of technology in the field of robotics leads to the intelligent and isolated operation of robots. But it limits the ability for manipulation and interaction with human. Even though robots are autonomous and intelligent it needs human interaction in applications like diffusing bomb, handling hazardous material and Tele-surgery etc. The proposed Haptic based robotic control system enhanced with embedded image processing gives a novel approach for controlling the robots through haptic sensor input, contours and color recognition inputs. In this project the robotic arm is designed and it s controlled by haptic which increases the ability and efficiency of robot. The ultimate purpose of this project is to provide a new way of controlling technique for robots by human machine interaction using ARM. Index Terms ARM processor, Haptic, Human machine interaction, Image processing and robots. I. INTRODUCTION Robots of current generation are self-directed but controlling it is also needed, where human machine interaction plays a vital role. This paper focuses on design and implementation of a robotic arm and control it using a human arm by means of haptic technology. In the human arm the flex sensors are attached for giving the haptic input and the image processing, enhanced the control of the robot by recognizing the contours present in the human arm which is drawn with colors. The ARM microcontroller is working as the heart of the system and image processing is done by OPEN CV python software. The serial data transferred from the PC after image processing is received by the microcontroller which finally directs the servomotors of robotic arm. Hence it is possible to control the robot by new way, by means of haptic and image processing techniques. II.LITERATURE SURVEY Vipul J. Gohil and Dr. S D. Bhagwat focuses on the project Robotics arm Control using Haptic Technology designed the robotic arm which having two degrees of freedom and it was controlled by the flex sensors and wirelessly[1].taylor Jones and Kurt Graf, approached in the project Helping Hand 7 DOF Haptic Robotic Arm Project (2013) designed the robotic arm controlled by Velcro strap mounted motion and force sensors on a human operator's arm which controls the motion tracking robotic arm's proportional motion.[2] Pholchai Chotiprayanakul and Dalong Wang focuses on project A haptic base human robot interaction approach for robotic grit blasting proposed the remote operation method for robot arm in a complex environment by using the Virtual Force (VF) Based approach. A virtual robot arm was manipulated by a steering force, at the endeffecter, which was generated according to the movement of a feedback haptic. [3] It is analyzed from the different project works, diverse approaches for controlling robots. The most of related works got success in the area of using sensors for controlling the robot. In case of image processing few flaws are present in finding the hand gesture. Especially in the hand detection part it is important to look for skin color region in the image. But skin color classification is difficult because it can also detect other body parts as a hand in the image. By means the convexity hull and defects in the hand makes failure for hand gesture detection because of the environment in which the image is captured. III. OVERVIEW OF PROPOSED SYSTEM IJIRT 101809 INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH IN TECHNOLOGY 893
The proposed system combine both sensor and vision based approach. With support of markers attached to users hand glove can provide reliable detection, hence there is a need for red color strip on gloves. The interface uses camera to track the movement of hand & recognizes gesture. It uses multiple colors and contour detection algorithms for improving accuracy of the recognition results. Further the system gives the integrated control by means of sensors and image processing for controlling the robots. The high speed ARM processor and servo mechanism provides excellent solution in addition with better accuracy. resistance of the flex sensor varies which gives the input to the ARM. The analog to the digital converter, its converts the signal to the digital which is compared with the programming and the various PWM signals are generated for operating and controlling the servo motors. The analog to the digital converter, its converts the signal to the digital which is compared with the programming and the various PWM signals are generated for operating and controlling the servo motors. Similar to the haptic sensor input, image processing inputs can give to the robotic arm. The image processing is done with the help of computer and the data is transferred to the microcontroller by using the serial port. The proposed system is consist of the following three sections. Hand glove side The hand glove is attached with the flex sensor and for vision input it s stickered with the color strips. The hand glove with flex sensor is shown in below figure. IV.PROPOSED ARCHITECTURE Fig. 1. Fig.2. Robotic Glove with Flex Sensors Robotic arm side The robotic arm is the mechanical arm which having many degrees of freedom. In the proposed with the robotic arm with five degrees of freedom will be designed. Block diagram of the proposed system. This figure shows how the system functions. Fig.3. Robotic Arm In the proposed system the robotic arm is controlled by using the different technologies which provides a new way of controlling technique. Flex sensors array gives the signals to the each fingers of the robotic arm. According to the motion of the hand glove the IJIRT 101809 INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH IN TECHNOLOGY 894
Step1: Capture the video. Step2: Flip the video window. Step3: Resize the video window for fast processing. Step4: convert color BGR to HSV. Step5: Apply Gaussian filter for smoothing. Step6: Create a mask for finding color. Step7: Short list the colored object. Image processing input side Fig.4. Image processing Input System In the image processing side the image acquisition is the first process which is done by using the camera of the personal computers or web camera. Once the image acquisition done the next process will be image processing. The algorithm of image processing for contour detection and color detection is follows. Since the process concern with video capturing the basic operations like capturing the video, flipping and resizing the video window are taken place. The open CV image processing has function for those processes. Next to the basic operations image smoothing is following. Gaussian blur removal is smoothing the image. The color and contour detection process are the final steps of hand gesture identification which is done easily in open CV image processing. Step8: Find contours in the image. Step9: Find hand gesture using contours. Step 10: Send hand gesture signal to serial port. C. Flowchart for Hand gesture recognition A. Contours Contours is the curve which joining all the continuous points (along the boundary), having same color or intensity. The contours are a useful tool for shape analysis and object detection and recognition. For better accuracy, use binary images. So before finding contours, apply threshold or canny edge detection. In OpenCV, finding contours is like finding white object from black background. The function for finding contour is cv2.findcontours ( ) and for drawing the contours cv2.drawcontours ( ). B. Algorithm The algorithm of the hand gesture identification is following. Fig.5. Flowchart for Hand gesture recognition C.Psudeocode IJIRT 101809 INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH IN TECHNOLOGY 895
self.capture = cv2.videocapture(0) f, orig_img = self.capture.read() orig_img = cv2.flip(orig_img, 1) img=cv2.gaussianblur(orig_img,(5,5),0) img = cv2.cvtcolor(orig_img,cv2.color_bgr2hsv) orange_lower = np.array([0, 127, 125],np.uint8)orange_upper = np.array([25, 255, 255],np.uint8) Fig.6. Fabricated Flex sensor B. Servo Motor orange_binary= cv2.inrange(img,orange_lower,orange_upper) cv2.findcontours if contour idx==0: ##print "input 1" ser.write("input1") cv2.imshow('frame',frame) cv2.imwrite('frame.jpg',frame) A. Flux Sensors It is an analog resistor, which work as variable analog voltage divider. Inside the flex sensor are carbon resistive elements with thin flexible substrate. [8] More carbon means less resistance. When the substrate is bent the sensor produces resistance output relative to the bend radius. The flex sensor achieves great form-factor on a thin flexible substrate. When the substrate is bent, the sensor produces a resistance output correlated to the bend radius [6]. Servo refers to an error sensing feedback control which is used to correct the performance of a system. Fig.7. Servo PWM Servo or RC Servo Motors are DC motors equipped with a servo mechanism for precise control of angular position. The RC servo motors usually have a rotation limit from 90 to 180. But servos do not rotate continually. Their rotation is restricted in between the fixed angles. The Servos are used for precision positioning. They are used in robotic arms and legs, sensor scanners and in RC toys like RC helicopter, airplanes and cars. [7], [4]. 4.3 Flow Chart of the Proposed System IJIRT 101809 INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH IN TECHNOLOGY 896
Fig.7. Color and Contour detection B. PWM generation for controlling servo motor The Pulse width modulation is generated by the programming the ARM LPC 2138 controller. The PWM is generated according to the need for controlling the servo motor. When the sensor input or the Image Processing input is received by the processor it will generate the output for running the servo motor. The following figures shows the PWM generation and the servo motor excitation. Fig.6. Flow chart of the system V. SIMULATION AND RESULTS Haptic technology applied in the system is carrying the PWM generation for controlling the servo, contour and color recognition operations as an input to the system. Hence each part of the system is simulated by various tools. A.Color and Contour detection Fig.8. PWM signal generation for controlling servo The Hand Gesture detection is done by two processes in our project, which are finding colors and detecting contours. The simulation is done by using open cv python programming and the following results shows the Contours detection. IJIRT 101809 INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH IN TECHNOLOGY 897
The simulation of hardware is done with the help of the Proteus ISIS professional software. Fig.9. Servomotor excitation V. CONCLUSION In the proposed system a new method of controlling the robot is developed by using the haptic technology. The sensor gives analog output, if there is a change in their resistance value. This analog output is manipulated in such a way that they give proportional pulses of different duty cycles to the servo motors which give movement to the robotic arm. The image processing and the flex sensors input to the robotic arm enables it to function properly. And the servo motors ensure the precision of each and every degrees of freedom. VII. FUTURE SCOPE The proposed system can be implemented as robotic path planning system. Many methods like optical character recognition, symbol recognition and complex gesture recognition can be given as the input for controlling the robot. The human machine communication can be improved using the haptic also possible. ISSN (Online):2278-5299 Volume 2, Issue 2: Page No.98-102, March - April (2013) [2]. Taylor Jones and Kurt Graf, Helping Hand 7 DOF Haptic Robotic Arm Project (2013) Spring 2013 University of Central Florida, department of electrical engineering and computer Science, Orlando, Florida [3].Takahiro Endo, Haruhisa Kawasaki, Tetsuya Mouri,,Yasuhiko Ishigure,Hisayuki Shimomura,Masato Matsumura,andKazumi Koketsu, Five-Fingered Haptic Interface Robot:HIRO III, IEEE transactions on haptics, vol. 4, no. 1, januarymarch 2011. [4]. Hyun Soo Woo and Doo Yong Lee, Exploitation of the Impedance and Characteristics of the Human Arm in the Design of Haptic Interfaces IEEE transactions on industrial electronics, vol. 58, no. 8, august 2011. [5]. J. E. Colgate and G. Schenkel, Passivity of a class of sampled-data systems: Application to haptic interfaces, J. Robot. Syst., vol. 14, no. 1, pp. 37 47, 1997. [4]. G. C. Burdea, Force and Touch Feedback for Virtual Reality. New York: Wiley, 1996. [6]. D.K.Boman, Internationalsurvey:Virtualenvironment research, Computer, vol. 28, no. 6, pp. 57 65, Jun. 1995. [7]. S.D. Lay and A.M.Day, Recent developments and applications of haptic devices, Comp. Graph. Forum, vol. 22, no. 2, pp. 117 132, 2003. [8]. Ninja P. Oess,,Wanek, J. and van Hedel, H.J.A Enhancement of bend sensor properties as applied in a glove for use in neurorehabilitation settings, Engineering in Medicine and Biology Society (EMBC), 2010 Annual International Conference of the IEEE, Aug. 31 2010-Sept. 4 2010, [9]. J. E. Colgate and J. M. Brown, Factors affecting the Z-width of a haptic display, in Proc. IEEE Int. Conf. Robot. Autom., San Diego, CA, 1994, pp. 3205 3210. [10]. www.keil.com [11]. www.mbed.org REFERENCE [1]. Vipul J. Gohil and Dr. S D. Bhagwat Robotics Arm Control Using Haptic Technology International Journal of Latest Research in Science and Technology IJIRT 101809 INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH IN TECHNOLOGY 898