Developing a Gesture Based Remote Human-Robot Interaction System Using Kinect

Transcription

1 Developng a Gesture Based Remote Human-Robot Interacton System Usng Knect Kun Qan 1, Je Nu 2 and Hong Yang 1 1 School of Automaton, Southeast Unversty, Nanjng, Chna 2 School of Electrcal and Electronc Engneerng, Changzhou College of Informaton Technology, Changzhou, Chna kqan@seu.edu.cn Abstract Gesture based natural human-robot nterface s an mportant functon of robot teleoperaton system. Such nterface not only enables users to manpulate a remote robot by demonstraton, but also ensures user-frendly nteracton and software reusablty n developng a networked robot system. In ths paper, an applcaton of gesture-based remote human-robot nteracton s proposed usng a Knect sensor. The gesture recognton method combnes depth nformaton wth tradtonal Camshft trackng algorthm by usng Knect and employs HMM n dynamc gesture classfcaton. A Clent/Server structured robot teleoperaton applcaton system s developed, whch provdes a favorable functon of remotely controllng a dual-arm robot by gestural commands. Experment results valdate the practcablty and effectveness of the applcaton system. Keywords: Knect, Gesture, Robot, Teleoperaton, Human-robot Interacton 1. Introducton One key long-term goal of developng remote robot control system va local network and Internet requres a human-frendly nterface that transfers nformaton of gudance or other types or commands. In the feld of gesture recognton and robotcs, many studes have been carred out on adaptng gesture as an deal communcaton nterface n the Human-Robot Interacton (HRI) context. In the last decade, several methods and potental applcatons n the advanced gesture nterfaces for HRI or HCI have been suggested. Yoon [1] develops a hand gesture system n whch combnaton of locaton, angle and velocty s used for the recognton. Lu [2] develops a system to recognze 26 alphabets by usng dfferent HMM topologes. Computatonal models such as Neural Network [3], Hdden Markov Model (HMM) [4] and Fuzzy Systems have been popular for gesture classfcaton. Besdes from tradtonal monocular or stereo vson system [5] for buldng gesture nterface, low-cost Knect sensor [6, 7] has recently become an attractve alternatve to expensve laser scanners n robotc applcatons. The Knect s depth sensor conssts of an nfrared projector that emts a constant pattern and an nfrared camera that measures the dsparty between the observed pattern and a prerecorded mage at a known constant depth. The output conssts of an mage of scaled dsparty values n Knect dsparty unts. In ths paper, a remote robot control system s descrbed that utlzes Knect based gesture recognton as human-robot nterface. The system acheves synchronzed robot arm control that generates human-mmc actons, whch s a fundamental functon to buld a teleoperaton applcaton. 203

2 2. Gesture Recognton as Human-robot Interface 2.1. Camshft based Hand Trackng In ths paper, we defne four basc patterns of gestures whch are desgned as robot control commands. These gestures are WaveLeft, WaveRght, RseUp, and PutDown. An nteractve hand trackng strategy combng color data and depth data s desgned. Frstly, a person waves hs/her hand n front of a Knect sensor. Then, a moton detector n color camera between successve frames s appled to locate the hand, whch s taken as the ntal trackng target. After the ntalzaton, Camshft trackng algorthm s employed for real-tme hand trackng. Snce t s well known that Camshft-based trackng algorthm usng only color nformaton may produce trackng error when there s smlar color n background. We ntroduce depth nformaton to facltate hand trackng accordng to the fact that hand s usually separated from the background object n depth, and has fxed movng range. Hence threshold segmentaton n depth map can accurately dstngush the player from the background. The trackng algorthm extracts trajectory of a sngle hand movement, denoted as X = { X 1,, X n }, n whch X t = ( xt, yt ) s the two dmensonal spatal coordnate at tme t. Example of trackng results are shown n Fgure 1. Fgure 1. Trackng Trajectores of Three Types of Gestures 2.2. Hand Moton Trajectory Recognton Hdden Markov models are used to model the underlyng processes of gestures and determne the underlyng processes behnd the ndvdual components of a gesture. Frstly, the obtaned gesture trajectores are converted nto 16 drecton codes representng drecton vectors. Each angle s converted nto one of the sxteen drecton codes. The angle ranges of the drecton codes have dfferent wdths. The feature of 2D moton trajectory of hand gesture s comprsed of a seres of dscrete movement ponts, and t s represented by a seres of dscrete movement drecton value. For the 2D moton plane, we dvde the drectons nto eght dscrete values. Therefore, the trajectory of dynamc gesture ban be descrbed by the sequence of dscrete drecton value d1, d2,..., d 16. Ths transforms the orgnal state sequence X = { X 1,, X n } nto an encoded observaton sequence O = { O 1,, O n }. Then HMM s employed to model and classfy the gesture played by the user. We use a Left-Rght structure HMM and take 3 to 4 dfferent states to model the abovementoned gestures. As an example, Fgure 2 shows the basc structure of the HMM model for representng the gestures used n our work. 204

3 a 11 a 22 a 33 WaveLeft & a 12 a 23 WaveRght a 11 a 22 a 33 a 44 Wave a 12 a 23 a 34 Fgure 2. HMM Structure for Gesture Trajectory Recognton Baum-Welch algorthm s employed to tran λ ( A, B, π ), the computatonal the Hdden Markov model for each gesture pattern. Regular HMM based recognton algorthm computes the smlarty metrc P ( O λ ) = 1,, N of observed gesture to each gesture model of λ 1, = 1,..., N. The pattern wth hghest smlarty s consdered as the classfcaton result. Based on ths, we defne a probablstc confdence measure. Denote that the confdence measure of gesture pattern s C, whch s computed as follow: We also defne and adaptve threshold C = k log( P( O λk )) log( P( O λ )) = arg max{ C } k k Ck for each. Only n case that > C k k k C can gesture pattern be consdered as the gesture recognton result. For four canddate gesture patterns G = { WaveLeft, WaveRght,RaseUp, PutDown}, an example of beng classfed as WaveRght gesture s computed as follow: P( O λwr ), f (( CWR > Ck ) ( CWR = max{ Ck})) k Pr( G = WR) = k 0, otherwse 3. Development of Remote Robot Control System wth Gestural Interface The b-handed Cyton 14D-2G (as shown n Fgure 3) ncludes two arms wth 14 degrees of freedom n jont moton plus a par of actuated two-fnger grppers. The b-handed humanod manpulators have the property of redundancy and bfurcaton whch enables placement of multple hands at a desred poston and orentaton wthn ts workspace n an unlmted number of ways. To develop Clent/Server structured remote human-robot nteracton control software, a lghtweght toolkt ActnSE s employed. The software of robot control server contans a control-based calculaton API performng nverse-knematcs calculatons and other assocated commands, and a hardware API to drectly control the Cyton arm va codes. Communcaton nterface usng TCP/IP protocol s also mplemented n the robot controller. Meanwhle, clent software contans a gesture recognton engne, whch captures human gestures and produces gesture classfcaton results, and smlar communcaton nterface, whch sends the sgnals of control commands to the server. The system archtecture s shown n Fgure

4 Fgure 3. Robot Arm Knematc Model 4. Experments Fgure 4. System Archtecture In the experment, a gesturer stood 0.3m~0.5m n front of a Knect sensor connected to a clent computer. He made four of the command gestures such as WaveLeft, WaveRght, RseUp, and PutDown to drect robot. Clent and server computers are both wth Intel Core 2 Duo E GHz CPUs, and the software on both clent and server computers was developed usng Mcrosoft Vsual Studo To test the accuracy of gesture recognton, a set of 50 mage sequences contanng four of the above and other undefned gestures were collected. These sample sequences of mages were performed by dfferent gesturers and backgrounds under normal lghtng condtons. Fgure 5 shows an example of the confdence estmaton of a WaveRght gesture. In addton, Fgure 6 shows the result that corresponds to the stuaton that the user was makng a sequence of WaveLeft, WaveRght and RseUp gestures. In general, recognton accuracy of nearly 85% was acheved. 206

5 WaveRght WaveRght Threshold RaseUp WaveLeft Fgure 5. Gesture Confdence Fgure 6. Gesture Classfcaton Estmaton Results The system s developed for a gesture-based remote robot arm control applcaton. The clent software made use of OpenNI sute to develop fundamental Knect sensor data capture and processng modules. TCP/IP protocol based network communcaton nterface was mplemented on both clent and server sdes. Consequently, the applcaton system enabled relable remote robot arm control. We have mplemented four command gestures to gude the robot arm to execute several types of actons, such as rsng of puttng down one of ts arms, as shown n Fgure 7. Robot arm rased 5. Concluson Fgure 7. Software GUI Robot arm put down In ths paper a natural nteracton framework s proposed that ntegrates an ntutve tool for teleoperatng a moble robot through gestures. It provdes a comfortable way of nteractng wth remote robots and offers an alternatve nput for encouragng non-experts to nteract wth robots. Applcaton system s developed combng low-cost Knect devce and network technology. For our future work, the gesture recognton wll be mproved to support moble robot navgaton scenaros. We beleve that better gesture recognton rato wll allow the 207

6 usage of more ergonomc poses and gestures for the user, thus reducng the fatgue assocated wth the current gesture recognton. Acknowledgements Ths work s supported by the Natonal Natural Scence Foundaton of Chna (Grant No ). References [1] H. S. Yoon, J. Soh, Y. J. Bae and H. S. Yang, Hand gesture recognton usng combned features of locaton, angle and velocty, Pattern Recognton, vol. 34, (2001), pp [2] N. Lu, B. Lovel and P. Kootsookos, Evaluaton of hmm tranng algorthms for letter hand gesture recognton, Proc. IEEE Int l Symp. On Sgnal Processng and Informaton Technology, Darmstadt, Germany, (2003), pp [3] X. Deyou, A Network Approach for Hand Gesture Recognton n Vrtual Realty Drvng Tranng System of SPG, ICPR Conference, Hong Kong, Chna, (2006), pp [4] M. Elmezan, A. Al-Hamad and B. Mchaels, Real-Tme Capable System for Hand Gesture Recognton Usng Hdden Markov Models n Stereo Color Image Sequences, WSCG Journal, vol. 16, no. 1, (2008), pp [5] Y. Yu, Y. Ikush and S. Katsuhko, Arm-pontng Gesture Interface Usng Surrounded Stereo Cameras System, Proceedng of the 17th Internatonal Conference on Pattern Recognton, Cambrdge, England, UK, (2004) Aug 26-26, pp [6] X. Lu, R. Xu and Z. Hu, GPU Based Fast 3D-Object Modelng wth Knect, Acta Automatca Snca, vol. 38, no. 8, (2012), pp [7] W. Xu and E. J. Lee, Contnuous gesture recognton system usng mproved HMM algorthm based on 2D and 3D space, Internatonal Journal of Multmeda and Ubqutous Engneerng, vol. 7, no. 2, (2012), pp Authors Dr. Kun Qan receved the Ph.D. n control theory and control engneerng from Southeast Unversty n He s currently workng as a lecturer at School of Automaton, Southeast Unversty. Hs research nterests are ntellgent robotc system and robot control. Je Nu receved hs M.Eng. n control theory and control engneerng from Southeast Unversty n He s currently workng as a lecturer at School of Electrcal and Electronc Engneerng, Changzhou College of Informaton Technology. Hs research nterests are robot vson. Hong Yang receved hs B.Eng. from Shandong Unversty n He s now a master canddate majored n control theory and control engneerng at School of Automaton, Southeast Unversty. 208