Development of a Gesture Detection System for the Control of a Robotic Arm

Transcription

1 Ameican Jounal of Science and Technology 2016; 3(1): Published online Febuay 2, 2016 ( ISSN: Development of a Gestue Detection System fo the Contol of a Robotic Am Osaho Uche, L. O. Kehinde Depatment of Electonic and Electical Engineeing, Obafemi Awolowo Univesity, Ile-Ife, Nigeia addess ucheosaho@gmail.com (O. Uche) Keywods Gestue Detection, Kinect Senso, Gaphical Inteface and Robotic Am Received: Decembe 11, 2015 Revised: Decembe 26, 2015 Accepted: Decembe 28, 2015 Citation Osaho Uche, L. O. Kehinde. Development of a Gestue Detection System fo the Contol of a Robotic Am. Ameican Jounal of Science and Technology. Vol. 3, No. 1, 2016, pp Abstact This study developed a gestue detection system fo the contol of a obotic am. This was achieved with the view of developing a simple and use fiendly appoach fo obotic am manipulation. A Micosoft Xbox Kinect senso was used as the input device fo etieving eal-time joint infomation fom the use. Kinematic algoithms wee developed by applying Denavit-Hatenbeg paametes and geometic equations. The acquied gestues fom the human subject wee used to cay out vaious kinematic outines as deemed fit by the use. A gaphical inteface was also developed to povide eal time feedback to the use. The obot was able to execute basic gestues as instucted by the use ated at about 80% success ate. Most gestues developed wee effective enough fo the scope of this poject. Howeve, minimal gestue detection eos wee ecoded aveaging below 20%. A standad deviation eo of 3.8 at a vaiance eo of appoximately 14.1 was ecoded. The study implemented an effective gestue system that was capable of executing the basic outines equied fo obotic am manipulation in eal time. 1. Intoduction The ability to inteact with technology has become one of the geatest achievements in ecent times. A myiad of techniques have been developed by enginees and scientists to manipulate, instuct and communicate with electical/electomechanical devices. This fom of inteaction is mostly established between human and machine o machine to machine (M2M). Most devices that pocess infomation equie some fom of human inteaction to manipulate the content of thei memoy. Vaious intefaces have been developed to act as a link between man and machine [1]. The basic means of inteaction include keyboads, touch sceens, joysticks, touch pads etc. In ecent times, Human machine inteaction (HMI) has expeienced significant development ove the yeas and has integated to all aeas of life fom compute gaming to moe sensitive oles like obotic sugey, unmanned aeial vehicle maneuveing and Automobile manufactuing [2]. The poposed poject is aimed at adapting one of the latest technologies known as gestue contol. A senso known as the Kinect, developed by Micosoft was used to acquie gestue commands fom the use. The Kinect is a senso that compises of an RGB (ed geen blue) camea, 3-D depth camea and a multi aay micophone. The 3-D depth camea is designed to monito the gestues fom the human skeletal system by pocessing the data fom the depth camea. Hence, the gestues acquied fom the human is pocessed into instuctional code fo vaious applications.

2 18 Osaho Uche and L. O. Kehinde: Development of a Gestue Detection System fo the Contol of a Robotic Am 2. Review The typical appoach that goven obotic am manipulation apply pinciples which include muscula contol, angula displacement, embedded contol, mind contol, gestue contol etc. A bief eview of pevious woks that ae elated to this wok ae descibed in this section. Steven [3] developed a Natual Use Inteface fo Robot Task Assignment. The obotic system they employed pefomed object detection, gasp and motion planning, while the human opeato handled high-level tasks. Thei method tacked a human hand fom a specified vitual eality wokstation using a Kinect. The data acquied by the use inteface is used to manipulate vitual objects in the obots wokspace as detected by the peception system via a second Kinect senso. Howeve, this appoach was computationally intensive and complex to implement and quite expensive fo mass deployment. Shobhitha [4] made use of an electomyogam to acquie electical signals by mounting suface electodes on the uses foeam. The ecoded muscula activity was pocessed and used to contol a obotic am. They acquied esults fom a use by executing flexion and extension motions within a duation of 10 seconds fo 10 consecutive times. The signals ae then pocessed in LabVIEW. They also incopoated haptic technology that tansmitted stimulus signals fom the human am. A CAD model of the obotic am was developed using Solid woks softwae (Solid-Woks, 2015). Othe 3. The Gestue Detection System components that make up the system include an Aduino UNO and Aduino Mega fo tansmitting wieless signals to the obotic am Olawale [5] adapted the most widely used means fo the contol of obotic ams. Thei wok intefaced a obotic am (PUMA 560) with a pe-pogammed 8051 micocontolle. Most of the algoithms equied to execute the outines wee pogammed into the pocesso. The configuation of thei design incopoated an MCU 8051 that coodinates the opeation of the obotic am by collecting infomation fom othe components of the design, which include a LATCH 74LS373, an 8255 PIO and an EPROM The knowledge gained fom the afoementioned techniques summaize the key appoaches in obot contol with egads to human-machine inteaction. Howeve, a close insight into the vaious concepts eveal loop holes that wee addessed in this eseach. A key aea that was impoved upon was the gaphical inteface. A moe use fiendly inteface poviding eal time view of the thee dimensional data of the obotic am as well as the values geneated fom the invese and fowad kinematic algoithms was implemented. The entie wok was designed to ensue that the use has full contol of the obotic am without any pevious pogamming knowledge. Finally, a gestue ecognition algoithm was developed that filteed unwanted gestue commands, so as to pevent bottlenecks that might aise as a esult of false gestue commands Figue 1. A gaphical use inteface. A Micosoft Kinect 3-D depth camea acquies skeletal images of the human skeletal fame, the images wee captued at 15fps fo this expeiment. The data acquied contain key infomation that epesent the movements of tacked joints geneated fom the am gestues. This gestues epesent am movements that ae pepogammed to activate commands if successfully executed. An invese and fowad kinematics algoithm was developed by applying the Denavit-Hatenbeg (D-H) paametes and geometic equations that epesent the joints of a five degee of feedom (DOF) obotic am. The gestues fom the human subject wee mapped to the

3 Ameican Jounal of Science and Technology 2016; 3(1): equivalent joint of the obotic am and conveted to distance and degees by adapting vaious mathematical pinciples that epesent espective joints of the obotic am. The acquied infomation fom the human subject was used to execute contol commands. These commands wee adapted into outines that imitate eal life scenaios. A gaphical use inteface was also developed to povide a use fiendly inteface fo the use as shown in Figue 1. The gestue detection system compises of fou majo sections as illustated in the configuation schematic of Figue 2. The sections include: i. Gestue Input. ii. Image Acquisition Device - Kinect. iii. Compute System. iv. Robotic Am Gestue Input This fom of communication depends on the discussion context (the type of discussion detemines the hand gestue maneuves) of the individual. Evey convesation has a specific set of gestues that ae dependent on the use s discetion. The gestue inputs fom the humans wee captued by the Kinect at a set speed of 15 fames pe second fo both colo and depth images and the fame speeds can be changed to suit the design specifications. The time fame of 15fps was sufficient to acquie elevant inputs fom the am movements [6]. The two main popeties that ae acquied fom the human joints ae the speed and displacement of each joint with espect to thei distance fom the oot joint (Hip Cente). Refeence joints wee set as makes to ensue that gestues don t conflict each othe. Figue 2. The gestue detection system Skeletal Image Acquisition Device The skeletal images wee acquied with the Micosoft Kinect senso vesion The sensos wee designed to tack skeletal data at a maximum vetical angle of 47 degees and a hoizontal angle of 57 degees [7] Howeve in ode to suit the design configuation, the angles fom the human am wee escaled to lie within a wokable ange of the Kinect sensos maximum angles [8] Compute System The compute system plays a key ole in the gestue detection system. It should be noted that due to the fame speed (30 fps) the Kinect senso acquies the images, a lot of system esouces was demanded fom the compute. Fo this wok, the majo system specifications ae listed below.

4 20 Osaho Uche and L. O. Kehinde: Development of a Gestue Detection System fo the Contol of a Robotic Am i. Intel Dual coe pocesso GHz Pentium CPU. ii. RAM 4.0 GB. iii. Opeating System - Widows 7 Ultimate 32 bit. All essential system dives such as the Video Gaphics Aay (VGA), Univesal Seial Bus USB, Bus contolle etc. wee in use in the couse of the poject. Fo the softwae components, A Micosoft Visual Studio 2012 was installed. A vesion 1.8 Softwae Development Kit (SDK) was also installed as pat of the libaies fo the poject Robotic Am An RA-01 obotic am with 5 degees of feedom was connected to an SMC-05 contolle boad. The contolle boad was connected to the compute via an RS232 seial pot and the micocontolle handled. The pulse width modulation fo contolling the five motos of the obotic am. The micopocesso eceives instuctions in the following fomat: [7, #, #, #, #, #] whee 7 is the initialization/stat code that infoms the pocesso that the next set of 5 bytes epesent the desied angula displacements fo the obotic am joints. The fist # specifies the desied angula position fo the base moto, the second fo the shoulde moto, the thid fo the elbow moto, the fouth fo the wist moto and fifth fo the gippe moto. A baud ate of 9600 was pescibed fo the seial communication of the obotic am by the manufactues. Howeve seveal tials confimed that a baud ate 0f would povide bette esults [8]. A baud ate of implies that 19200/54bits = commands ae tansmitted pe second whee 54 bits epesent the six command bytes plus a stop byte [(6+1) x 8 = 54]. This means that one command is tansmitted evey (1/355.56ms) = 28milliseconds. Howeve, the manufactues specified that commands should be sent to the motos evey 18 milliseconds. If this is followed, it connotes that at a baud ate of 19200, a command is sent at fo 18milliseconds while no command is sent fo about 10 milliseconds i.e. fo only 2/3 the equied time. Although this configuation was not pefect as pe the specification equiements, it was sufficient fo tansmissions. A highe baud ate of did not poduce any impovement in outcome. 4. Fowad Kinematics The fowad kinematics of the end-effecto with espect to the base fame was detemined by multiplying all the T Matices, whee _ T Altenatively, it can be epesented as _ T T T T (1) p p p (2) Whee s epesent the otational elements of tansfomation matix (k and j = 1, 2 and 3). p, p, p denote the elements of the position vecto. Fo the joint axes of RA- 01 whee s epesent the otational elements of tansfomation matix (k and j = 1, 2 and 3). p, p, p denote the elements of the position vecto. Fo the joint axes of RA- 01 obotic am (Figue 3). The tansfomation fom fame to the base fame is given as: cos θ ( sin θ 0 0 sin θ T cos θ 0 0 (3) Figue 3. Robotic Am Joint Axes. Equation 3.5 shows the tansfomation matix of the shoulde to the joint. T cos+90 - θ. ( sin θ (1 0 sin θ. cos+90 - θ. 0 0 ( sin θ cos θ (1 0 (4) cos θ ( sin θ 0 0 Equation 3.6 shows the tansfomation matix of the elbow to shoulde joint cos θ ( sin θ 0 L sin θ T cos θ 0 0 (5) Equation 3.7 shows the tansfomation matix of the base to elbow joint

5 Ameican Jounal of Science and Technology 2016; 3(1): T cos θ 0. ( sin θ 0. 0 L 0 0 (1 0 sin θ 0. cos θ ( cos θ 0 sin θ 0 0 L 0 0 (1 0 (6) (sin θ 0 ( cos θ Equation 3.7 shows the tansfomation matix of the gippe to wist joint. ( ( T (7) Hence, the fowad Kinematics of the obotic am fom gippe to the base fame is given as: This implies that: 0 2T T T T 0T 2T (8) x = cos θ [L 0 cos+θ - θ 0. L 0 sin+θ - θ. ( L 0 sin θ 3 (9) y = sin θ [L 0 cos+θ - θ. - L sin+θ - θ. ( L sin θ 3 (10) z = sin+θ - θ 0. L cos+θ - θ. - L cos θ 3 (11) Whee x, y, z ae the Catesian coodinates of the gippe. 5. Invese Kinematics The invese kinematics of a obotic am detemines the joint angles and joint displacements of the obotic am when the end effectos coodinates ae known (Figue 4). The invese kinematics is computationally expansive and geneally takes a vey long time in the eal time contol of manipulatos. The invese kinematic poblem can be esolved by two majo appoaches, namely: i. The Geometic Solution Appoach. ii. The Algebaic Solution Appoach. The Geometic solution appoach was adapted in this wok by beaking down the spatial geomety of the manipulato into seveal plane geomety equations. The joint otations ae fo base (θ ), shoulde (θ ), elbow (θ ), and wist (θ 4 ). To obtain the +x, y, z. coodinate at the top of the object, it implies that: dist = Usin θ + Lsin θ + G (12) z B - U cos θ ( Lsin θ (13) Whee θ 180 ( +θ - θ. and U, L, and G ae the lengths of the uppe am, lowe am and gippe links espectively, θ 4 is the wist angle. < = < = θ = tan (14) θ = ABC 0 D (15) Figue 4. Robotic Am Geomety. whee b 2kL L 0, a 4L L - 4kL L 0 c 4+k ( 4L L. k +x - y - z. ( (L - L - L 0 ) (16) And θ = ABC 0L D (17) whee f k - k, e 2zk g 4+ z ( k.+ k - k.

6 22 Osaho Uche and L. O. Kehinde: Development of a Gestue Detection System fo the Contol of a Robotic Am k L 0 sin θ - L cos θ - L k L 0 cos θ ( L sin θ (18) 6. Softwae Design Concept The softwae fo the obotic am was designed to pesent an efficient appoach to the ecognition of gestues fo the contol of a obotic am. Ealie woks elated to obotic am gestue contol lacked basic design blocks and this pevented thei design fom being obust and adaptable by othe softwae pogammes. Thee main sections summaize the design appoach adapted in this wok, namely: i. Gestue Recognition Engine. ii. Am Contolle Engine and iii. Gaphical Use Inteface. the elevant command which is tansmitted to the obotic am via seial communication Am Contolle Engine The am contolle engine is esponsible fo contolling the obotic am. All the algoithms that define the fowad and invese kinematics of the obotic am ae stoed in the am contolle libay. The seial communication potocol of the obotic am is also pat of the am contolle engine Gestue Recognition Engine The gestue ecognition engine was designed to pocess the gestue inputs fom the use. The gestue ecognition is actually a libay witten in C# code. When a gestue is being executed, the location of the joints with espect to time and distance is of key impotance to the ecognition engine Gestue Tacking Algoithm The gestue tacking algoithm is at the heat of this poject, the algoithm is esponsible fo tacking the equied skeletal joint data of the human of inteest. To confim that a gestue has being coectly executed, a set of defined ules must be established to confim that the gestues ae valid and this is shown in Figue 5. Thee phases have being incopoated in the algoithm design to ensue that tacked gestues ae executed accoding to peset standads. Due to the complexity of human gestues pefomed in a discussion context, it is impeative that all the incopoated detecting phases ae linked to each othe. This is to ensue that inte phase communication is not beached [9]. Stage one of the algoithm detemines if a gestue stat mode has being initiated. To establish the initialization, it checks if the joints location falls within set limits. The limits ae dependent on joint makes such as the spine, shoulde joint etc. If successfully executed, the flow of command swings to stage two. Stage two monitos the joints to ensue that the gestues still lie within the discussion context of the specified gestue command (e.g. Swipe Left, Swipe Right etc.). In an event the gestue fall out of place the command etuns to stage one to await the next gestue command. Stage thee checks to confim that the gestue has not violated any of the constaints set by the pevious stages and it also confims that the gestue command was executed within 15 fames. At this final stage, an Event Handle is fied to tigge the Am Contolle Engine into action. The Am contolle Engine veifies the gestue type and executes Figue 5. Flow Chat of Gestue Detection System. The gestues that ae pocessed by the gestue ecognition engine geneates set of commands that ae passed to the am contolle. The vaious diections include: i. Swipe To Right ii. Swipe To Left iii. Am Fowad iv. Am Backwad v. Shoulde Up vi. Shoulde Down vii. Am Tigge 7. Hadwae Communication In ode to achieve a smooth opeation of the entie expeiment, the hadwae inteaction between devices must be taken into consideation. Figue 6 depicts fou stages that summaies the data flow of the setup. Human skeletal positional data was acquied at a set value of 15fps by a

7 Ameican Jounal of Science and Technology 2016; 3(1): Kinect XBOX360 senso (Model no: 1414), poweed extenally. The skeletal data acquied is conveted to binay values and tansmitted via a -5v to +5v poweed USB cable to the compute system. The acquied data steam fom the Kinect senso matches the USB pot of the compute system poweed by the Kinect USB plug n play peinstalled dive softwae. The connection between the compute and the obotic am is via a USB-Seial convete cable (with installed polific USB-Seial dive). The data tansmission is achieved by tansmitting signals geneated fom the compute system via the USB pot to the seial pot of the contolle boad of the obotic am. The seial pot data is usually sent as a packet at 20ms intevals with 8 bit wod, stat, stop, paity bits. The Stat bit is usually between +5v to +7v followed by data bits and finished by stop bit between -5v and -7v [8]. 8. Results Table 1 shows a set of data that depicts the espective values of the obotic am fo fouteen (14) tested outines. The table is divided into five (5) majo sections; the xyz coodinates, expected joint angles, actual joint angles, the joint angle eos and eo calculations. The xyz coodinates of the tacked uses am was imputed into the invese kinematics equation so as to povide the expected joint location of the espective joints (shoulde, base and elbow). The acquied values wee ecoded and compaed with the xyz coodinates of the am acquied in eal time fo the same xyz coodinates. Slight changes wee ecoded in the eal time computed values as compaed to the computed values. This inconsistences wee expected because the xyz value acquied in eal time ae subject to vaious factos such as light visibility, line of sight, poo gestue event handling, moto otational speed, seial communication and the multi-theaded esponse of the obotic am contolle engine [10]. A set of gaphical diagams (Figues 7a, b, c) show expected (dotted lines) and actual (solid lines) fo each monitoed joint (base, shoulde and elbow joints). Figue 6. Hadwae Inteaction. Table 1. Eo Test fo Catesian Position of the Robotic Base fom 90 t0 180 degees. Hand Catesian Position (mm) Expected Joint Angles (Degees) Actual Joint Angles (Degees) Eo in Joint Angles (Degees) X Y Z Base Shoulde Elbow Base Shoulde Elbow Base Shoulde Elbow Mean Eo (mm) Oveall Mean Eo(mm) = 3.9 Vaiance of Eo(mm) = 14.1 Standad Dev. of Eo(mm) = 3.8

8 24 Osaho Uche and L. O. Kehinde: Development of a Gestue Detection System fo the Contol of a Robotic Am leaning algoithms to expand the flexibility of the gestue detection system. Refeences [1] Cannan, J. & Huosheng Ho, Human-Machine Inteaction (HMI): A Suvey, Essex: School of Compute Science & Electonic Engineeing. [Online] 508%20HMI-Suvey.pdf 9. Conclusion Figue 7a. Elbow Joint gestue esponse. Figue 7b. Shoulde Joint gestue esponse. Figue 7c. Base Joint gestue esponse. A few glitches wee expeienced with the gestue ecognition engine, and this was caused by an ovelapping of detected gestues and the lighting of the oom in which the contol outines was tested. The system showcased the impotance of gestue contol in human machine inteaction. Howeve, futhe wok needs to be done to apply machine [2] Johannsen, G., (2007). Human Machine Inteaction. Contol Systems,Robotics and Automation, Volume XXI, p. 1. [3] Steven J, L., Shawn, S. & Neal, C., (2014). Natual Use Inteface fo Robot Task Assignment [Online] levine2014 natual.pdf [4] Shobhitha, A. J., Jegan, R. & Melwin, A., (2013). OWI-535 EDGE Robotic Am Contol Using ElectoMyoGam (EMG) Signals. Intenational Jounal of Innovative Technology and Exploing Engineeing (IJITEE), II(6), pp [5] Olawale, J., Oludele, A., Ayodele, A. & Miko, N., (2007). Development of a Micocontolle Based Robotic Am. Ameican Jounalof Engineeing Reseach (AJER), IV(2), pp. 8. [6] Jana, A., (2012). Kinect fo Window SDK pogamming guide. 1 st ed. Bimingham: Pack Publishing Pess. [Online] [7] Clement, G. and Massimo, F., (2013). Kinect in Motion - Audio and Visual Tacking by Example. 1 st ed. Bimingham: Packt Publishing Pess. [Online] %E2%80%93-audio-and-visual-tacking-example [8] Akinwale, O. B., L. O. Kehinde, K. P. Ayodele, A. M. Jubil, O. P. Jonah, O. Iloi, and X. Chen, " (2009). A LabVIEWbased on-line obotic am fo students laboatoy", 2009 ASEE Annual Confeence & Exposition, Austin, Texas, ASEE, pp. Pape [9] Catuhe, D., (2012). Pogamming with the Kinect fo Windows. Redmond,Washington: Micosoft Pess. [Online] [10] _Kinect_fo_Windows.html?id=66kgXfDKdYwC [11] Micosoft, (2015). Chapte 18. Theading and Synchonization. [Online] msdn.micosoft.com/enus/libay/ff652496(d=default,l=enus,v =om.10).aspx[accessed 9 July 2015].