30 Int'l Conf. IP, Comp. Vision, and Pattern Recognition IPCV'15

Transcription

1 30 Int'l Conf IP, Comp Vision, and Pattern Recognition IPCV'15 Spectral Collaborative Representation Based Classification by Circulants and its Application to Hand Gesture and Posture Recognition from Electromyography Signals Ali BOYALI, Naohisa HASHIMOTO, and Osamu MATSUMOTO National Institute of Advanced Industrial Science and Technology Robot Innovation Research Center - Smart Mobility Research Team / Tsukuba-Japan Abstract In this study we introduce and demystify a novel signal pattern recognition method, Spectral Collaborative Representation based Classification (SCRC) and demonstrate its application for recognition of hand gestures and postures using Electromyography sensors A recently released Thalmic Labs MYO armband is used to gather muscle electromyography signals Along with the new signal pattern classification algorithm, we also introduce a training approach which implicitly embeds the gesture boundaries in a training dictionary that allows continous gesture and posture recognition The worst recognition accuracy we obtained for a set of experiments is over 97% which is the highest recognition results in the literature where bio-signals are used Keywords: EMG gesture, continous gesture recognition, spectral representation, gesture training matrix, myo armband 1 Introduction Recognition of the patterns in bio-signal applications has been a challenging engineering problem due to the stochastic nature of the biological processes The bio-signal classification applications have been utilized to diagnose the metabolic anomalies and diseases, for rehabilitation and monitoring in medicine The proliferation of the mobile computing platforms such as smart phones and tablets made the bio-signal pattern recognition an appealing tool for creating intuitive Human Computer Interface (HCI) applications via gesture and posture recognition and detecting the physiological conditions such as heart rate on the human body by wearable gadgets In line with the development of mobile computing platforms, the advanced sensors have been introduced to the market that can communicate with these platforms Recently Thalmic Labs MYO armband which can measure and recognize the hand gestures using Electromyography (EMG) signals has been released to the developers The Software Development Kit (SDK) of the device allows the developers to capture raw EMG signals from the eight EMG sensors which is wearable on the arm in form of an armband In this study, we detail a novel method in signal pattern classification which yields far better gesture recognition accuracy then the armband reports The method gives over 97% accuracy which is the worst result for our experiment set but the best among the EMG and other biosignal classification studies in the related literature The methods we propose for the training phase allow the user to obtain a gesture dictionary on the spot The classification method proposed in this paper is a spectral variant of the Collaborative Representation based Classification (CRC) which has been successfully used in face [1] and signal pattern recognition applications [2] with an overwhelming classification accuracy A subspace clustering method is used to build a training dictionary which enables the system to recognize signal patterns in a streaming manner This study brings about the following contributions in the bio-signal pattern classification literature The spectral features of the observed signals are obtained using circulant and diagonalization matrices This approach remarkably reduces the computational complexity and computation time The subspace methods which are used to build a training matrix implicitly embed the start and end position of the hand gestures and postures, thus, they lead to continuous posture and gesture recognition eliminating spotting the patterns in the signals The training phase is easy to implement, accordingly the end users can build a training dictionary on the spot with regards to their requirements This flexibility of the training phase and the high recognition accuracy open the way for the use of proposed method in many research areas such as for manipulation of the robotic prosthesis The dynamic hand gestures and static postures are captured during the training phase, hence in the real time application of the study, 10 hand gestures are recognized and mapped to the five hand postures with an overwhelming accuracy and low computation times The number of gestures is the highest than the numbers

2 Int'l Conf IP, Comp Vision, and Pattern Recognition IPCV'15 31 reported in the related literature The rest of the paper is organized as follows In Section 2, we give the brief review of the CRC method and elaborate the SCRC The sensor MYO armband information and the training procedures are given in Section 3 The simulation results and discussions are given in Section 4 The paper concludes with Section 5 2 Collaborative Representation based Classification in Spectral Domain The CRC method was first introduced in the literature to compare the classification mechanisms of the Sparse Representation based Classification (SRC) [3] and CRC [1, 4] The authors prove that the SRC which makes use of l 1 norm in the objective function is a special variant of the CRC methods in which l 1 and l 2 norms are utilized depending of the requirements in the problem such as noise on the measurements or occlusion on a face image Both of the methods rely on representing the observed signal by a the linear combination of the representatives that are stacked into a training dictionary as a column vectors The SRC methods yield high accuracies using an over-complete dictionary or training matrix, on the contrary, the recognition accuracy does not depend on an over-complete dictionary matrix in the CRC methods in which the coefficients of the linear combination are computed by collaboratively making use of the other class representative samples In the Regularized Least Square version of the CRC method (CRC_RLS), given the training matrix A = [A 1,A 2,,A n ] R mxn and the observed signal y, the solution vector x which contains the linear representation coefficients for the systems of equations y = Ax are obtained using the ridge regression In this problem setup, the optimum coefficients for the objective function given in Eq(1) is calculated as ˆx = Py where P =(A T A+σI) 1 A T and σ is the regularization parameter min ˆx = y Ax 2 + σ x 2 (1) x Once the solution vector ˆx is obtained, the observed signal is labeled evaluating the minimum representation residuals r i given in Eq (2) where δ i : R n R n is the selection operator that selects the coefficients of i th class while keeping other coefficients zero in the solution vector ˆx min r i (y) = y Aδ i (ˆx) 2 (2) i The l 2 solution gives efficient results when there is data fidelity [1] However, biological signals do not exhibit data fidelity due to the stochastic nature of biochemical processes For example, the measured EMG signals are the superposition of Motor Unit Action Potentials (MUAP) of the individual muscle fibers and magnitude and shape of the signals which are random only depends on the duration of the muscle contraction or stretch and the force that the muscles produce [5] In this case, the ordinary CRC_RLS method yields poor accuracy results We overcome this difficulty and dramatically improve the recognition accuracy by using the spectral features of the observed signal The eigenvalues and eigenvectors of a signal reveal important information about the characteristics of the system In order to find the eigenvalues of a linear system of equations a square matrix is needed The EMG signal is a continuous time series and we observe the signals by a sliding time window as 1D data The conventional methods cannot be utilized directly to capture the spectrum of an 1D vector Therefore, we employ the circulant matrix approach to obtain the spectrum by creating a circular matrix from the observed 1D signal A circulant matrix is a square matrix with the circularly shifted columns In this study, the resultant matrix is a Hankel matrix which consists of skew constant diagonals, however other diagonal direction can be used as a circulant Formally expressing, let s assume the observed signal vector is y =[y 1,y 2,,y n ], then the trajectory matrix C becomes a square matrix with the skew diagonal entries given in Eq (3), the first row of which is the observed signal itself y 1 y 2 y n y 2 y 3 y 1 C y = y n y 1 y n 1 A unitary diagonalization matrix F can be used to obtain the eigenvalues of the composed circulant matrix C by relatively lesser number of matrix operation than that of the conventional eigenvalue decomposition A Fourier matrix can be used for spectral decomposition of any circulant matrix Assuming a Fourier matrix F, the eigenvalues and eigenvectors are computed by the following equations [6] W W 2 W N 1 F = 1 W 2 W 4 W 2 N 2 1 W N 1 W 2(N 2) W (N 1)(N 1) (4) with the entries of W = e 2πi N which are the n th roots of unity The eigenvalues are computed by the following matrix operations; (3) C = U ΩU (5) where Ω is a diagonal matrix, the entries of which are the eigenvalues of the circulant matrix and U = 1 N Fis the matrix of column eigenvectors The matrix operator () H represents the conjugate or Hermitian transpose operation

3 32 Int'l Conf IP, Comp Vision, and Pattern Recognition IPCV'15 which is important in the CRC method as the resultant eigenvalues consist of complex conjugate eigenvalue pairs The spectral CRC method operates on the complex eigenvalues which are taken as the features of the observed and training samples The regression operator is given in Eq (6) on the complex plane gestures are the Fist, Hand Relax (Free), Finger Spread, Wave In, Wave Out and Double Tap hand gestures Fig 2 P =(A H A + σi) 1 A H (6) It is important to note that, the eigenvectors of circulants captured by each time window then transformed are the same, therefore, the eigenvalues of the circulant matrix are the only features that increase the discriminative power of the method 3 Sensor Description and Implementation 31 MYO Armband The Thalmic Labs MYO armband is a fairly new technology which is equipped with eight EMG sensors Fig 1 The armband also reports the linear and angular acceleration of the device as well as the orientation angles The affordable armband enables the researchers and developers to access the raw surface EMG signals of an arm which was previously possible with the expensive EMG sensors or laboratory equipments The EMG sensor array of the armband reports the raw EMG measurements at a frequency of 200 Hz The sample codes are provided by the official SDK [7] In order to have access to the raw sensor measurements in a computation platform, we developed a Matlab library; MatMYO which is available online [8] Fig 1: MYO Armband Kit with Bluetooth Dongle The MYO armband has a built-in gesture recognition software which can recognize six different gestures These Fig 2: MYO Hand Gestures, 1- Fist, 2- Hand Relax, 3- Finger Spread, 4- Wave In, 5- Wave Out, 6- Double Tap (Thumb and middle finger tap each other two times) We used the same gesture and hand posture set for comparison the accuracy 32 Training The recognition accuracy of the classification algorithms the SRC and CRC highly depends on the reliability of the representative samples in the training dictionary Unlike the face recognition applications using these state of art classification methods, in the gesture recognition, obtaining the representative samples for the dynamical movements requires spotting the gestures, in other words, the start and end position of the gestures must be known in advance Spotting gesture boundaries can be performed either by analyzing the signal if the boundaries are distinguishable or employing a switch to mark the boundaries while collecting gesture data In this study and our previous studies [2], we employ a subspace clustering method to cluster the dictionary matrix its respective classes The recent subspace clustering methods which are based on the self-representation property are capable of clustering the representative columns with a high clustering accuracy At least two successive hand gestures are necessary for clustering the data For this reason, we collect training data by performing two successive gestures repeatedly in the training phase such as in the case for Wave In dictionary The hand performs Wave In and Hand Relax gestures repeatedly for a short time Fig 3 In our experiments, we perform two gestures that takes one second 05 seconds for each of the hand states Each gesture pair are repeated for only 10 seconds, but we use the first five seconds data to build a training dictionary A sliding window with a length of 100 data samples is used to capture the raw EMG sensor signal from the eight sensors and put in the dictionary matrix which is to be clustered as an 1D column vector This procedures result

4 Int'l Conf IP, Comp Vision, and Pattern Recognition IPCV'15 33 clusters Fig 4: Clustered Double Tap - Hand Relax Gestures on EMG data Fig 3: Wave In -> Hand Relax -> Wave In Repetition in a block circulant Hankel matrix which is the input of the subspace clustering method We use the Ordered Subspace Clustering (OSC) method [9] which meets the requirements of the continuous gesture recognition The OSC method is based on the Sparse Subspace Clustering (SSC) [10] approach with an additional penalty term in its objective function for the sequential data The additional penalty term in the objective function enforce the neighboring columns to be similar or close vectors In our training matrices, the neighboring columns are time shifted and close to each other The training phase is easy to implement and take a short time to obtain a training dictionary for each of the gesture class These classes are extracted from a gesture pair In fact, when two gestures are clustered, two posture and two gesture sets are implicitly clustered into two class dictionaries The hand switching between two gestures visits four hand states In the case of the Wave In and Hand Relax training pair, the hand stands still at the relaxed position for a relatively short time, then performs Wave In gesture, stays at the Wave In position for a relatively short time then returns to the beginning and the whole cycle starts again The representatives of the two postures are included in each dynamic gesture sets The continuous gesture recognition algorithm recognizes 10 dynamic hand gestures However returning the original position from each of the hand gestures given in the Fig (2) are mapped to hand relax posture The results of the subspace clustering phase for a single gesture pair Double Tap and Hand Relax is given in Fig (4) which demonstrates a successful clustering of the pair into the its respective EMG 4 Simulation Results We collect five experimental data sets for each gesture pairs and use one set to build the class training dictionary The remaining ones are used for testing In addition to the gesture pairs, we collected data for a hand gesture sequences in which the hand visits all hand gesture and posture states arbitrarily The results are given in the Figs (5) - (10) from for each gesture pairs including one of the arbitrary hand gesture sequence experiment The recognition accuracy for the Fist and Hand Relax gesture states is 100% Fig (5) for 1200 labelling computations The algorithm runs at every sampling time in the experiments There is no single error for the Fist and Hand Relax gestures in the Spectral CRC results Fig 5: Recognition Results for Fist and Hand Relax Experiment, Spectral CRC and MYO The worst recognition accuracy result for the Wave Out - Hand Relax experiment is 9834% out of 1274 classification computation We give the results of this experiment without assigning the return gestures to the Hand Relax position from the Wave Out gesture in Fig (6)

5 34 Int'l Conf IP, Comp Vision, and Pattern Recognition IPCV'15 Fig 6: Recognition Results for Wave Out and Hand Relax, Spectral CRC and MYO Fig 8: Recognition Results for Double Tap and Hand Relax, Spectral CRC and MYO Fig (6) shows that, the algorithm can classify the return gestures from Wave Out gestures to Hand Relax position In addition, the misclassifications occur on the change borders where hand switches between the gesture pairs The method gives the Hand Spread labels at these change points where the hand involuntarily might perform Fingers Spread in which only two gestures are performed repeatedly, the hand performs each gesture arbitrarily Fig 7: Recognition Results for Wave In and Hand Relax, Spectral CRC and MYO The results of a Wave In - Hand Relax experiments are given in Fig (7) The recognition accuracy for this experiment is 9934% while the recognition accuracy is 100% for Double Tap experiments Fig (8) The recognition accuracy for the Hand Spread-Hand Relax experiment is 973% As shown in Fig (9), the Spectral CRC method mis-classify the gesture as Wave Out while it is performing hand spread This is due to the overlapping states of the Wave Out and Hand Spread gestures on which the same muscle groups activated and Wave Out hand gesture trajectory encompasses the Hand Spread wrist trajectory We give the result of an arbitrary hand gesture sequence experiment in Fig (10) The recognition accuracy is 9847% for this experiment In these experiments, unlike the others Fig 9: Recognition Results for Hand Spread and Hand Relax, Spectral CRC and MYO 5 Conclusion In this study we introduced a new signal pattern classification algorithm and methodologies to recognize a set of hand gestures and postures continuously on a multi-channel streaming signal using the raw EMG data The initial results are promising in yielding high accuracy for a fairly rich hand gesture sets It is worth to note that no signal pre-processing has been used in the study and the simulations are completed using only the raw data The methods and experiments explained in this paper is a small portion of our main project [11, 12] by which we have been developing multi-modal HCIs for elderly people to enable them to command a robotic wheelchair with their available resources In our previous studies such

6 Int'l Conf IP, Comp Vision, and Pattern Recognition IPCV'15 35 Fig 10: Recognition Results for Random Hand Gesture Sequence, Spectral CRC and MYO, 1- Hand Relax, 2- Fist, 4- Wave In, 6- Wave Out, 8- Hand Spread as the braking state classification of a mobility robot [2], Segway using a tablet PC and inertial sensors, we showed that, the CRC method is capable of labeling braking states of the robot with a high accuracy real-time and fast We also verified the training approach and the CRC method for continuous hand gesture and posture recognition using the Leap Motion sensor which tracks the hand with optical cameras at sub-millimeter levels and achieved very high recognition accuracies more then reported in this paper with the same gesture sets We will finalize the project with the developed gesture and posture recognition methods devising experiments to which elderly people will participate In addition, with the developed interfaces, we will prepare a virtual reality training and rehabilitation environment for the elderly and the individuals with severe disabilities who are prescribed power wheelchairs This is due to the fact that most people who are prescribed a power wheelchair experience difficulty to use them and it takes time to adapt to the new technologies 6 Acknowledgments The study is supported by the Japan Society for the Promotion of Science (JSPS) fellowship program and the KAKENHI Grant (Grant Number 15F13739) mobility devices, Robotics and Autonomous Systems, 2015 [3] J Wright, A Y Yang, A Ganesh, S S Sastry, and Y Ma, Robust face recognition via sparse representation, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol 31, no 2, pp , 2009 [4] D Zhang, M Yang, and X Feng, Sparse representation or collaborative representation: Which helps face recognition? in Computer Vision (ICCV), 2011 IEEE International Conference on IEEE, 2011, pp [5] S Shahid, J Walker, G M Lyons, C A Byrne, and A V Nene, Application of higher order statistics techniques to emg signals to characterize the motor unit action potential, Biomedical Engineering, IEEE Transactions on, vol 52, no 7, pp , 2005 [6] D S G Pollock, Circulant matrices and time-series analysis, International Journal of Mathematical Education in Science and Technology, vol 33, no 2, pp , 2002 [7] Thalmic Labs (2015) Myo sdk [Online] Available: platform/the-sdkhtml [8] A Boyali (2015) Matmyo [Online] Available: [9] S Tierney, J Gao, and Y Guo, Subspace clustering for sequential data, in Proc Computer Vision and Pattern Recognition (CVPR) IEEE, 2014, pp [10] E Elhamifar and R Vidal, Sparse subspace clustering: Algorithm, theory, and applications, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol 35, no 11, pp , 2013 [11] N Hashimoto, Y Takinami, and O Matsumoto, An experimental study on vehicle behavior to wheel chairs and standing-type vehicles at intersection, in ITS Telecommunications (ITST), th International Conference on IEEE, 2013, pp [12] A Boyali and N Hashimoto, Block-sparse representation classification based gesture recognition approach for a robotic wheelchair, in Intelligent Vehicles Symposium Proceedings, 2014 IEEE IEEE, 2014, pp References [1] L Zhang, M Yang, X Feng, Y Ma, and D Zhang, Collaborative representation based classification for face recognition, arxiv preprint arxiv: , 2012 [2] A Boyali, N Hashimoto, and O Matsumoto, A signal pattern recognition approach for mobile devices and its application to braking state classification on robotic