Mu-Rhythm Template Matching classifier for One-Dimensional cursor control

Transcription

1 Mu-Rhythm Template Matching classifier for One-Dimensional cursor control Thesis submitted in partial fulfillment of the requirements for the degree of Master of Science by Research in Computer Science and Engineering by Divya Kodali International Institute of Information Technology Hyderabad , India February, 2015

2 Copyright Divya Kodali, 2015 All Rights Reserved

3 International Institute of Information Technology Hyderabad, India CE R TIFIC A TE It is certified that the work contained in this thesis, titled Mu-Rhythm Template Matching Classifier for One- Dimensional Cursor Control by Divya Kodali, has been carried out under my supervision and is not submitted elsewhere for a degree. Date: 14 February 2015 Adviser: Prof. Bipin Indurkhya

4 To My Parents K V Choudary, and Lakshmi Kantamma My Husband UmaMahesh My Little Niece - Arya

5 A c k n o w led g m e n ts Foremost, I would like to express my sincere gratitude to my advisor Dr. Bipin Indurkhya for the continuous support of my research for his patience, motivation, enthusiasm, and immense knowledge and for allowing to grow as a researcher. I decided to take up research in the Cognitive Science Lab after taking the Introduction to Cognitive Science course. Dr. Bipin gave a warm welcome and complete indepence to choose among various research topics. I was completely surprised and fascinated to learn about various kinds of research going on in this field. While the class room lectures gave a subtle idea of how this field would be, Mr. Amitash Ojha helped in every way to help me learn the topics in detail. He also supported me emotionally with his kind words and encouraging suggestions with the help of which I could publish my first international research paper. The literature study I have done has proved to be very helpful to learn about the varied research going on and my experimental studies helped me to learn how to design, conduct, collect and analyze various results. I duly thank Dr.Bipin for permitting me to explore whatever I was interested in. With the knowledge I have gained, I am sure I will be able to continue my research in this field more comfortably in a focused manner. Finally, Thanks to my fris Anupama, Harshita, Gopal, and Bharat who had encouraged me and backed me up when I was in need. I am very much thankful to Amitash Ojha for his valuable sugges-tions and guidance. I would not forget the long discussion I and Amitash had regarding analysis of some results. It is probably one of the longest and best professional discussions I had with him. More than anyone else, I would like to thank Dr. Bapiraju from the University of Hyderabad and Padmasri Dr. B V Raju Institute of Technology for providing the necessary infrastructure (EEG machine) and immense guidance in giving my research idea a proper shape and taking it along the correct way. A special thanks to my family. Words cannot express how grateful I am to my father, mother, husband, sister, and brother-in-law for all the encouragement and sacrifices that they have made on my behalf. Your unrelenting support was what sustained me thus far. I am forever indebted to you people. Thank you everyone for everything!

6 A b s tr a c t Recent advances in computer hardware and signal processing have made possible the use of EEG signals or brain waves for communication between humans and computers. Locked-in patients have now a way to communicate with the outside world, but even with the last modern techniques, such systems still suffer communication rates on the order of 2-3 tasks/minute. In addition, existing systems are not likely to be designed with flexibility in mind, leading to slow systems that are difficult to improve. A brain-computer interface (BCI) is a system that provides an alternate non muscular communication/control channel for individuals with severe neuromuscular disabilities. With proper training, individuals can learn to modulate the amplitude of specific electroencephalographic (EEG) components (e.g., the 8 12 Hz µ rhythm and Hz β rhythm) over the sensorimotor cortex and use them to control a cursor on a computer screen. Conventional spectral techniques for monitoring the continuous amplitude fluctuations fail to capture essential amplitude/phase relationships of the µ and β rhythms in a compact Fashion and, therefore, are suboptimal. By extracting the characteristic rhythm for a user, the exact morphology can be characterized and exploited as a matched filter. A simple, parameterized model for the characteristic rhythm is proposed and its effectiveness as a matched filter is examined offline for a onedimensional cursor control. Although the main objective of this research idea is to build an assistive device for ALS patients, rather than working with ALS patients directly the work done here limits itself to running experiments on healthy subjects and building a template based on one these test subjects and comparing the template with other subjects.

7 I ND E X CHAPTER 1: Preliminary Motivation Objectives Scope of the Research Work CHAPTER 2: Principles of electroencephalography The Nature of the EEG signals EEG wave groups CHAPTER 3: Brain Computer Interface Technology System Overview Neuropsychological signals used in BCI applications BCI research: existing systems CHAPTER 4: Artifact Rejection Removing EEG artifacts by ICA blind source separation Artifact rejection based on peak elimination Blinking artifact recognition using artificial neural network Artifact rejection based in band pass FIR filters CHAPTER 5: EEG Signal Pre - Processing Methodology and Experiment Doing Cursor control experiment Imagination of cursor control experiment PsychToolbox Data collection Artifact removal and Band pass filtering MU Rhythm Epochs Averaging CHAPTER 6: Template Generation for online classification STEPWISE LEAST SQUARES ESTIMATION FOR AR MODELS Fast Fourier transform Cross correlation CHAPTER 7: Feature extraction and Classification Feature extraction... 59

8 7.1.1 Doing the one dimensional cursor control experiment Imaging the cursor control experiment Classification using cross correlation Doing the cursor control experiment Imagining the cursor control experiment CHAPTER 8: SUMMARY AND FUTURE WORK 8.1 Summary Bibliography Annexure Code for Auto regression Code for cursor control Doing in PsychToolbox Code for cursor control imagination in PsychToolbox... 85

9 LIST OF FIGURES Figure 1. A segment of a multichannel EEG Figure 2. The System of Electrode Placement Figure 3. EEG signal recording... 4 Figure 4. Alpha (left) and Beta (right) waves Figure 5. Theta wave Figure 6. Delta wave... 6 Figure 7. Mu (left) and alpha (right) waves Figure 8. Cerebral hemispheres showing the motor areas... 8 Figure 9. BCI common structure Figure 10. P3 evoked potential Figure 11. Artifact free EEG waveform recorded by a forehead electrode, and its spectrum Figure 12. Eye blink artifact corrupted EEG waveform recorded by a forehead electrode, and its spectrum Figure 13. Electrooculogram electrode placement. Two EOG channels, related to vertical and horizontal eye movements (EOGV and EOGH), are recorded Figure 14. Scheme of the proposed system Figure 15. One-dimensional task trial structure. (1) The target and cursor are present on the screen for 1 s. (2) The cursor moves steadily across the screen for 16 s with its vertical movement controlled by the user. (3) If the user hits the target, the target flashes for 0.5 s. (4) The screen goes blank for a 1min after every 40 trials. (5) The next trial begins Figure 16. Timing Diagram of cursor control experiment Figure 17. Timing Diagram of cursor control imagination experiment Figure 18. EEG hardware setup Figure 19. Electrode locations and names by the International system Figure 20. Raw data Figure 21. Moving averaged data Figure 22. Band passed signal Figure 23. Mu rhythm signal Figure 24. Epochs Figure 25. Averaged top and bottom targets Figure 26. Extracting AR features and concatenating Figure 27. Template for top targets Figure 28. Template for bottom targets Figure 29. Features of a single subject Figure 30. Cross correlation waves Figure 31. Features obtained from the imaginary data Figure 32. Epochs obtained from the imagination data Figure 33. Cross correlation waveforms... 68

10 Figure 34. Accuracy obtained from subject Figure 35. Accuracy obtained from the subject Figure 36. Accuracy obtained from the imagination data Figure 37. Comparing accuracy of the all subjects Doing and Imagination LIST OF TABLES Table 1. Common neuropsychological signals used in BCIs Table 2. Comparison between existing BCIs. The Speed is presented in average number of items or movements per minute

11 C H A P TE R 1 : P r e lim in a r y 1.1. M o tiv a tio n. People who are paralyzed or have other severe movement disorders need alternative methods for communication and control. Currently available augmentative communication methods require some muscle control. Whether they use one muscle group to supply the function normally provided by another (e.g., use extra ocular muscles to drive a speech synthesizer) or detour around interruptions in normal pathways (e.g., use shoulder muscles to control activation of hand and forearm muscles [1]), they all require a measure of voluntary muscle function. Thus, they may not be useful for those who are totally paralyzed (e.g., by amyotrophic lateral sclerosis (ALS) or brainstem stroke) or have other severe motor disabilities. These individuals need an alternative communication channel that does not dep on muscle control. This interface is named as BCI or Brain Computer Interface O b ject iv e s. The use of EEG signals as a vector of communication between men and machines represents one of the current challenges in signal theory research. The principal element of such a communication system, more known as Brain Computer Interface, is the interpretation of the EEG signals related to the characteristic parameters of brain electrical activity. The role of signal processing is crucial in the development of a real-time Brain Computer Interface. Until recently, several improvements have been made in this area, but none of them have been successful enough to use them in a real system. The goal of creating more effective classification algorithms, have focused numerous investigations in the search of new techniques of feature extraction. The main objective of this project is the establishment of a Time Frequency method, which allows EEG signal classification between two given tasks, as well as the familiarization with the state of the art in time-frequency and Brain Computer Interface. 0

12 1.3. Scope of the Research Work The main aim of this work is to, ultimately, build an assistive device for ALS patients who are completely paralyzed; to differentiate between two particular thoughts that they will be, later, trained to condition. As a starting step, we limit the scope of this work to healthy subjects without involving any ALS patients. Here, we deal with four healthy subjects in this work who are all right-handed. 0

13 C H A P T E R 2 : P r in ci p les o f e le c tr o e nceph a log r a p h y 2.1. T h e N a tu r e o f th e EEG s ig n a ls. The electrical nature of the human nervous system has been recognized for more than a century. It is well known that the variation of the surface potential distribution on the scalp reflects functional activities emerging from the underlying brain [2]. This surface potential variation can be recorded by affixing an array of electrodes to the scalp, and measuring the voltage between pairs of these electrodes, which are then filtered, amplified, and recorded. The resulting data is called the EEG. Fig. 1 shows waveforms of a 10 second EEG segment containing six recording channels, while the recording sites are illustrated in Fig. 2. In our experiments, we have used the System of Electrode Placement, which is based on the relationship between the location of an electrode and the underlying area of cerebral cortex (the "10" and "20" refer to the 10% or 20% inter-electrode distance) [8]. F ig u r e 1.A s e g m e n t of a m u lticha nn e l EE G. 2

14 F ig u r e 2.T h e S ys te m of E le ct r o d e P la c e m e n t. Each site has a letter (to identify the lobe) and a number or another letter to identify the hemisphere location. The letters F, T, C, P, and O stand for Frontal, Temporal, Central, Parietal and Occipital. (Note that there is no "central lobe", but this is just used for identification purposes.) Even numbers (2, 4, 6, 8) refer to the right hemisphere and odd numbers (1, 3, 5, 7) refer to the left hemisphere. The z refers to an electrode placed on the midline. N a s io n : point between the forehead and nose. I n io n : Bump at back of skull The EEG is thought to be the synchronized sub threshold dritic potentials produced by the synaptic activity of many neurons summed [3]. In its formation not all types of brain activity have identical impact. The depth, orientation and intrinsic symmetry of connections in the cortex are significant in it. As it is exposed in previous works [3] [4], pyramidal cells are thought to cause the strongest part of the EEG signal. Nowadays, modern techniques for EEG acquisition collect these underlying electrical patterns from the scalp, and digitalize them for computer storage. Electrodes conduct voltage potentials as microvolt level signals, and carry them into amplifiers that magnify the signals approximately ten thousand times. The use of this technology deps strongly on the electrodes positioning and the electrodes contact. For this reason, 3

15 electrodes are usually constructed from conductive materials, such us gold or silver chloride, with an approximate diameter of 1 cm, and subjects must also use a conductive gel on the scalp to maintain an acceptable signal to noise ratio. This method of EEG signal recording is shown in Fig. 3. F ig u r e 3.EE G s ign al r e c o r d in g 4

16 2.2. EEG w a v e g r o u p s. The analysis of continuous EEG signals or brain waves is complex, due to the large amount of information received from every electrode. As a science in itself, it has to be completed with its own set of perplexing nomenclature. Different waves, like so many radio stations, are categorized by the frequency of their emanations and, in some cases, by the shape of their waveforms. Although none of these waves is ever emitted alone, the state of consciousness of the individuals may make one frequency range more pronounced than others. Five types are particularly important: B ET A: The rate of change lies between 13 and 30 Hz, and usually has a low voltage between 5-30 μv (Fig. 2-4). Beta is the brain wave usually associated with active thinking, active attention, focus on the outside world or solving concrete problems. It can reach frequencies near 50 hertz during intense mental activity. F ig u r e 4. A lph a (left ) a n d B et a (r ig h t) w a v e s. A L P HA: The rate of change lies between 8 and 13 Hz, with μv amplitude (Fig. 4). Alpha waves have been thought to indicate both a relaxed awareness and also inattention. They are strongest over the occipital (back of the head) cortex and also over frontal cortex. Alpha is the most prominent wave in the whole realm of brain activity and possibly covers a greater range than has been previously thought of. It is frequent to see a peak in the beta range as high as 20 Hz, which has the characteristics of an alpha state rather than a beta, and the setting in which such a response appears also leads to the same 5

17 conclusion. Alpha alone seems to indicate an empty mind rather than a relaxed one, a mindless state rather than a passive one, and can be reduced or eliminated by opening the eyes, by hearing unfamiliar sounds, or by anxiety or mental concentration. T H E T A : Theta waves lie within the range of 4 to 7 Hz, with an amplitude usually greater than 20 μ V. Theta arises from emotional stress, especially frustration or disappointment. Theta has been also associated with access to unconscious material, creative inspiration and deep meditation. The large dominant peak of the theta waves is around 7 Hz. F ig u r e 5. T he ta w a v e. D ELT A: Delta waves lie within the range of 0.5 to 4 Hz, with variable amplitude. Delta waves are primarily associated with deep sleep, and in the waking state, were thought to indicate physical defects in the brain. It is very easy to confuse artifact signals caused by the large muscles of the neck and jaw with the genuine delta responses. This is because the muscles are near the surface of the skin and produce large signals whereas the signal which is of interest originates deep in the brain and is severely attenuated in passing through the skull. Nevertheless, with an instant analysis EEG, it is easy to see when the response is caused by excessive movement. F ig u r e 6. D e lta w a v e. G A M M A : Gamma waves lie within the range of 35Hz and up. It is thought that this band reflects the mechanism of consciousness - the binding together of distinct modular brain 6

18 functions into coherent percepts capable of behaving in a re-entrant fashion (feeding back on themselves over time to create a sense of stream-of-consciousness). M U : It is an 8-12 Hz spontaneous EEG wave associated with motor activities and maximally recorded over motor cortex (Fig. 7). They diminish with movement or the intention to move. Mu wave is in the same frequency band as in the alpha wave (Fig. 7), but this last one is recorded over occipital cortex. F ig u r e 7. M u (le ft) a n d a lph a (r ig h t) w av e s. Most attempts to control a computer with continuous EEG measurements work by monitoring alpha or mu waves, because people can learn to change the amplitude of these two waves by making the appropriate mental effort. A person might accomplish this result, for instance, by recalling some strongly stimulating image or by raising his or her level of attention. 7

19 F ig u r e 8. C e r e b r a l he m is ph e r e s s h o w in g th e m o tor a r e as 8

20 C H A P T E R 3 : B r a in C o m p u ter I n te r fa c e T e c h n o lo g y 3.1. S y s tem O ve r v ie w A Brain-Computer Interface (BCI) is a system that acquires and analyzes neural signals with the goal of creating a communication channel directly between the brain and the computer. Such a channel potentially has multiple uses. For example: Bioengineering applications: assist devices for disabled people. Human subject monitoring: sleep disorders, neurological diseases, attention monitoring, and/or overall "mental state". Neuroscience research: real-time methods for correlating observable behavior with recorded neural signals. Man Machine Interaction: Interface devices between human and computers, machine. For many years, people have speculated that electroencephalographic (EEG) activity or other measures of brain function might provide this new channel. Over the past decade, productive BCI research programs have begun. Facilitated and encouraged by the new understanding of brain functions and by the low-cost computer equipments, these programs have concentrated mainly in developing new communication and control technologies for people with severe neuromuscular disorders. The immediate goal is to provide communication capabilities so that any subject can control the external world without using the brain's normal output pathways of peripheral nerves and muscles. Nowadays, such activities drive their efforts in: Br ain (n e u r al) s ig n al a c q u is itio n : development of both invasive and noninvasive techniques for high quality signal acquisition. A lg o r ith ms a n d p r o ce s s in g : advanced machine learning and signal processing algorithms, which take advantage of cheap/fast computing power (i.e. Moore's Law) to enable online real-time processing. 9

21 U nd e r ly in g n e u r o s c ien c e : a better understanding of the neural code, the functional neuro-anatomy, the physiology and how these are related to perception and cognition, enabling signals to be interpreted in the context of the Neurobiology. Present BCI s use EEG activity recorded at the scalp to control cursor movement, select letters or icons, or operate a neuro-prosthesis. The central element in each BCI is a translation algorithm that converts electrophysiological input from the user into output that controls external devices. BCI operation deps on effective interaction between two adaptive controllers: the user who encodes his or her commands in the electrophysiological input provided to the BCI, and the computer which recognizes the command contained in the input and expresses them in the device control. Current BCI s have maximum information transfer rates of 5-25 bits/min. Achievement of greater speed and accuracy deps on improvements in: S ig n al ac qu is itio n : methods for increasing signal-to-noise ratio (SNR), signal- tointerference ratio (S/I)) as well as optimally combining spatial and temporal information. S in g le tr ial a n a ly s is : overcoming noise and interference in order to avoid averaging and maximize bit rate. Co -le a r nin g : jointly optimizing combined man-machine system and taking advantage of feedback. E x p e r im e n ta l p a r a d ig m s fo r in te r p r et a b le r e a d a b le s ig n a ls : mapping the task to the brain state of the user (or vice versa). U nd er s ta n d in g a lg o r ith m s a n d m o d e ls w ith in th e c o n te x t of th e n e u r o b io lo g y : building predictive models having neurophysiologically meaningful parameters and incorporating physically and biologically meaningful priors. The common structure of a Brain Computer Interface is the following (Fig. 9): 10

22 S ig n al Ac qu is itio n : the EEG signals are obtained from the brain through invasive or non-invasive methods (for example, electrodes). After, the signal is amplified and sampled. S ig n al P r e -P r o c e ss in g : once the signals are acquired, it is necessary to clean them. S ig n al C la s si fica tio n : once the signals are cleaned, they will be processed and classified to find out which kind of mental task the subject is performing. Co m p u te r I n te r a c tio n : once the signals are classified, they will be used by an appropriate algorithm for the development of a certain application. F ig u r e 9. B CI c o m m on s tr uc tu r e N e u r o p sy c h o lo g ic a l s ig n a ls u s e d in B C I a p p lic a tio n s Interfaces based on brain signals require on-line detection of mental states from spontaneous activity: different cortical areas are activated while thinking different things (i.e. a mathematical computation, an imagined arm movement, a music composition, etc...). The information of these "mental states" can be recorded with different methods. Neuropsychological signals can be generated by one or more of the following three: implanted methods, evoked potentials (also known as event related potentials), and operant conditioning. Both evoked potential and operant conditioning methods are normally externally-based BCIs as the electrodes are located on the scalp. Table 1 describes the different signals in common use. It may be noted that some of the described signals fit into multiple categories. As an example, single neural recordings may use operant conditioning in order to train neurons for control or may accept the natural 11

23 occurring signals for control. Where this occurs, the signal is described under the category that best distinguishes it. Implanted methods use signals from single or small groups of neurons in order to control a BCI. In most cases, the most suitable option for placing the electrodes is the motor cortex region, because of its direct relevance to motor tasks, its relative accessibility compared to motor areas deeper in the brain, and the relative ease of recording from its large pyramidal cells. These methods have the benefit of a much higher signal-to-noise ratio at the cost of being invasive. They require no remaining motor control and may provide either discrete or continuous control. While most systems are still in the experimental stage, Kennedy s group has forged ahead to provide control for locked-in patient JR [14] [15]. Kennedy s approach involves encouraging the growth of neural tissue into the hollow tip of a two-wire electrode known as a neurotropic electrode. The tip contains growth factors that spur brain tissue to grow through it. Through an amplifier and antennas positioned between the skull and the scalp, the neural signals are transmitted to a computer, which can then use the signals to drive a mouse cursor. This technique has provided stable long term recording and patient JR has learned to produce synthetic speech with the BCI over a period of more than 426 days. It is unknown how well this technique would work on multiple individuals, but it has worked on both patients (JR and MH) who have been implanted E vo k e d po te n tia ls (EPs) are brain potentials that are evoked by the occurrence of a sensory stimulus. They are usually obtained by averaging a number of brief EEG segments time-registered to a stimulus in a simple task. In a BCI, EPs may provide control when the BCI application produces the appropriate stimuli. This paradigm has the benefit of requiring little to no training to use the BCI at the cost of having to make users wait for the relevant stimulus presentation. EPs offer discrete control for almost all users, as EPs are an inherent response. 12

24 Exogenous components, or those components influenced primarily by physical stimulus properties, generally take place within the first 200 milliseconds after stimulus onset. These components include a Negative waveform around 100 ms (N1) and a Positive waveform around 200 ms after stimulus onset (P2). Visual evoked potentials (VEPs) fall into this category. Sutter uses short visual stimuli in order to determine what command an individual is looking at and therefore wants to pick [16]. He also shows that implanting electrodes improves performance of an externally-based BCI. In a different approach, McMillan and colleagues have trained volunteers to control the amplitude of their steadystate VEPs to florescent tubes flashing at Hz [17][18][19]. Using VEPs has the benefit of a quicker response than longer latency components. The VEP requires subject to have good visual control in order to look at the appropriate stimulus and allows for discrete control. As the VEP is an exogenous component, it should be relatively stable over time. Endogenous components, or those components influenced by cognitive factors, take place following the exogenous components. Around 1964, Chapman and Bragdon [20] as well as Sutton et al. [21] indepently discovered a wave peaking at around 300 ms after taskrelevant stimuli. This component is known as the P3 and is shown in Fig 10. While the P3 is evoked by many types of paradigms, the most common factors that influence it are stimulus frequency (less frequent stimuli produce a larger response) and task relevance. The P3 has been shown to be fairly stable in locked-in patients, re-appearing even after severe brain stem injuries [22]. Farwell and Donchin (University of Illinois) first showed that this signal may be successfully used in a BCI [23]. Using a broad cognitive signal like the P3 has the benefit of enabling control through a variety of modalities, as the P3 enables discrete control in response to both auditory and visual stimuli. As it is a cognitive component, the P3 has been known to change in response to subject s fatigue. In one study, a reduction in the P3 was attributed to fatigue after subjects performed the task for several hours [24]. 13

25 F ig u r e 10. P 3 e v o k e d p o te n tia l. S o lid lin e : the general form of the P3 component of the evoked potential (EP). The P3 is a cognitive EP that appears approximately 300 ms after a task relevant stimulus. In this image is represented by the biggest negative peak. D o tte d lin e : the general form of a non-task related response. Operant conditioning is a method for modifying the behavior (an operant), which utilizes contingencies between a discriminative stimulus, an operant response, and a reinforcement to change the probability of a response occurring again in a given situation. In the BCI framework, it is used to train the patients to control their EEG. As it is presented in Table 1, several methods use operant conditioning on spontaneous EEG signals for BCI control. The main feature of this kind of signals is that it enables continuous rather than discrete control. This feature may also serve as a drawback: continuous control is fatiguing for subjects and fatigue may cause changes in performance since control is learned. 14

26 Signal Name Description Mu and Alpha Wave Operant Conditioning [9][32] The mu wave is a 8-12 Hz spontaneous EEG wave associated with motor activities and maximally recorded over sensorimotor cortex. The alpha wave is in the same frequency band, but is recorded over occipital cortex. The amplitudes of these waves may be altered Event-Related Synchronization/ Desynchronization (ERS/ERD) Operant Conditioning [10][28][29][30][31] Slow Cortical Potential Operant Conditioning [12] P3 Component of the Evoked Potential [23] Short-Latency Visual Evoked Potentials [16] Individual Neuron Recordings [15][33] Steady-State Visual Evoked Potential (SSVER) [17][18][19] through bio-feedback training. Movement-related increases and decreases in specific frequency bands maximally located over brain s motor cortex. Individuals may be trained through biofeedback to alter the amplitude of signals in the appropriate frequency bands. These signals exist even when the individual imagines moving as the movement-related signals are preparatory rather than actual. Large negative or positive shifts in the EEG signals lasting from 300ms up to several minutes. Individuals may be trained through biofeedback to produce these shifts. A positive shift in the EEG signal approximately ms after a task relevant stimulus. Maximally located over the central parietal region, this is an inherent response and no training is necessary To produce the component, a response to the presentation of a short visual stimulus is necessary. Maximally located over the occipital region, this is an inherent response and no training is necessary Individuals receive implanted electrodes that may obtain responses from local neurons or even encourage neural tissue to grow into the implant. Operant conditioning may be used to achieve control or the natural response of a cell or cells may be used A response to a visual stimulus modulated at a specific frequency. The SSVER is characterized by an increase in EEG activity at the stimulus frequency. Typically, the visual stimulus is generated using white fluorescent tubes modulated at around Hz or by another kind of strobe light. A system may be constructed by conditioning individuals to modulate the amplitude of their response or by using multiple SSVERs for different system decisions T a b le 1. C o mm on n e u r o p s y c h o logical s ig n a ls u s e d in B C I s. 15

27 Wolpaw and his colleagues train individuals to control their mu wave amplitude (Table 2) for cursor control. Mu wave control does not require subjects to have any remaining motor control [9]. For the cursor control task, normal subjects are trained on the order of sessions to learn to move the cursor up/down. In the several papers examined, it appears that not all subjects obtain control, although most seem to during this time frame. In related work, the Graz brain-computer interface trains people to control the amplitude of their ERS/ERD patterns. Subjects are trained over a few sessions in order to learn a cursor control task. As in the mu wave control, not all subjects learn to control the cursor accurately. Obtaining two out of six subjects who are not able to perform the cursor control task has been reported [10]. Part of the charm of this system is that it gives biofeedback to the user in the form of a moving cursor after training. Slow cortical potentials serve as the signal in the Thought Translation Device, a communication device created by Biurbaumer s group in Germany [12]. While the signals discussed are used currently, other signals may be possible. Several papers have been written on recognizing EEG signal differences during different mental calculations. These papers suggest that different parts of the brain are active during different types of mental calculation, and if these different tasks may be accurately recognized, they could be used in a BCI. Lin et al [13]. Describe a study where five tasks were compared: multiplication problem solving, geometric figure rotation, mental letter composing, visual counting, and a baseline task where the subject was instructed to think about nothing in particular. Results from this experiment suggest that the easiest tasks to identify are multiplication problem solving and geometric figure rotation, but even these tasks are not easily identified. Other papers have concentrated on mental tasks, but none have found easily recognizable d i f f e r e n c e s between different tasks [25][26].. 16

28 3.3. B C I re s e a r c h : e x is tin g s y s te m s Different research groups work on communication channels between the brain and the computer. The leading groups are presented in alphabetical order in Table 2. These experimental interfaces include the hardware used in the BCI, the underlying BCI back software, and the user application. In assessing current systems, several factors must be considered, including five mentioned by Ben Schneiderman [27] : What is the time to learn the system? What is the speed of performance? How many and what kinds of errors do users make? How well do users maintain their knowledge after an hour, a day, or a week? What is their retention? How much did users like using various aspects of the system? What is their subjective satisfaction? 17

29 SYSTEM TRAINING Brain Response Interface [16] SSVEP training [17][18][19] P3 Character Recognition [23] Mu Wave Training [9][32] ERS/ERD [10][28][29][30][31] Though Translation Device [12] Implanted Device [15][33] Flexible Brain Computer Interface [34] TIME NUMBER OF CHOICE S SPEED ERRORS RETENTION SUBJECTIVE SATISFACTION min % Excellent Considered 6 hrs N/A N/A < 20% Not Mentioned Not discussed Minutes % Excellent Not discussed sessions % Not mentioned Not discussed hrs 2 Not available <11% Not mentioned Not discussed Months % Not Good Indirectly Discussed Months N/A 2 Not reported Not mentioned Excellent Considered % Not mentioned Not mentioned T a b le 2. C o m p a r is on be tw ee n e x is tin g B C Is : T h e Sp ee d is p r e s e n te d in av e r age nu m be r o f ite m s o r m o v e m e n ts p e r m in u te. 18

30 1) T h e B r ain R e s p o n se I n te r fa c e (S m ith -K et tle w e ll I n s titu te of Vi s u al S c ienc e s in S an F r an c is c o ). Sutter's Brain Response Interface (BRI) [16] uses visually evoked potentials (VEP's) produced in response to brief visual stimuli. These EP's are then used to give a discrete command to pick a certain part of a computer screen. Word processing output approaches words/min. and accuracy approaches 90% with the use of epidural electrodes. This is the only system mentioned that uses implanted electrodes to obtain a stronger, less contaminated signal. A BRI user watches a computer screen with a grid of 64 symbols (some of which lead to other pages of symbols) and concentrates a given symbol. A specific subgroup of these symbols undergoes an equi-luminant red/green fine check or plain color pattern alteration in a simultaneous stimulator scheme at the monitor vertical refresh rate (40-70 frames/s). Sutter considered the usability of the system over time and since color alteration between red and green was almost as effective as having the monitor flicker, he chose to use the color alteration because it was shown to be much less fatiguing for users. This system is basically the EEG version of an eye movement recognition system and contains similar problems because it assumes that the subject is always looking at a command on the computer screen. 2) P 3 Cha r ac te r R ec o g n ition (U n iv e r s ity of I llin oi s, USA ). In a related approach, Farwell and Donchin use the P3 evoked potential [23]. A 6x6 grid containing letters from the alphabet is displayed on the computer monitor and users are asked to select the letters in a word by counting the number of times that a row or column containing the letter flashes. Flashes occur at about 10 Hz and the desired letter flashes twice in every set of twelve flashes. The average response to each row and column is computed and the P3 amplitude is measured. Response amplitude is reliably larger for the row and column containing the desired letter. After two training sessions, users are able to communicate at a rate of 2.3 characters/min, with accuracy rates of 95%. This system is currently only used in a research setting. 19

31 A positive aspect of using a longer latency component such as the P3 is that it enables differentiating between when the user is looking at the computer screen or looking someplace else (as the P3 only occurs in certain stimulus conditions). Unfortunately, this system is also slow, because of the need to wait for the appropriate stimulus presentation and because the stimuli are averaged over trials, and can cause epilepsy in some subjects. 3) E R S /E RD Cur s o r C o n tr o l (U n ive r s ity of T e c hn o logy G r a z, A u s tr ia ) Pfurtscheller and his colleagues take a different approach [10] [28] [29] [30][31]. Using multiple electrodes placed over sensorimotor cortex they monitor eventrelated synchronization/desynchronization (ERS/ERD). In all sessions, epochs with eye and muscle artifact are automatically rejected. This rejection can slow down subject performance. As this is a research system, the user application is a simple screen that allows control of a cursor in either the left or right direction. In another experiment, for a single trial the screen first appears blank, then a target box is shown on one side of the screen. A cross hair appears to let the user know that he/she must begin trying to move the cursor towards the box. Feedback may be delayed or immediate and different experiments have slightly different displays and protocols. After two training sessions, three out of five student subjects were able to move a cursor right or left with accuracy rates from %. Unfortunately, the other two students performed at 60% and 51%. When a third category was added for classification, performance dropped to a low of 60% in the best case [31]. 4) A S te a d y S ta te Vis u al E vo k e d P ot e n tial B CI (W r ig h t-p at te r s on A ir F o r c e B a s e, T h e Air F o r c e R e s e a r c h L a b o r a to r y, U S A ). Middorf and colleagues use operant conditioning methods in order to train volunteers to control the amplitude of the steady-state visual evoked potential (SSVEP) to florescent tubes flashing at Hz [17][18][19]. This method of control may be considered as continuous as the amplitude may change in a continuous fashion. Either a horizontal light bar or audio feedback is provided when electrodes located over the occipital cortex measure changes in signal amplitude. If the VEP amplitude is below or above a specified 20

32 threshold for a specific time period, discrete control outputs are generated. After around 6 hours of training, users may have an accuracy rate of greater than 80% in commanding a flight simulator to roll left or right. Recognizing that the SSVEP may also be used as a natural response, Middorf and his colleagues have recently concentrated on experiments involving the natural SSVEP. When the SSVEP is used as a natural response, virtually no training is needed in order to use the system. The experimental task for testing this method of control has been to have subjects select virtual buttons on a computer screen. From the 8 subjects participating in the experiment, the average percent correct was 92% with an average selection time of 2.1 seconds. 5) M u Wave C u r s o r C o n tr o l (Wa d s w o r th C e n te r, A lb a n y, US A ). Wolpaw and his colleagues free their subjects from being tied to a flashing florescent tube by training subjects to modify their mu wave [9] [32]. This method of control is continuous as the mu wave may be altered in a continuous manner. It can be attenuated by movement and tactile stimulation as well as by imagined movement. A subject's main task is to move a cursor up or down on a computer screen. While not all subjects are able to learn this type of biofeedback control, the subjects that do, perform with accuracy greater than or equal to 90%. These experiments have also been exted to two dimensional cursor movements, but the accuracy of this is reported as having not reached this level of accuracy when compared to the one-dimensional control [19]. 6) T h e T h o u g h t T r a n sl a tion D e vice (U n iv er s ity of T ü b in g e n, G e rm a n y) As another application used with severely handicapped individuals, the Thought Translation Device, was developed by Birbaumer's lab [12]. Out of six patients with ALS, 3 were able to use the Thought Translation Device. Of the other three, one lost motivation and later died and another used discontinuously the Thought Translation Device and was unable to regain control later. The training program may use either auditory or visual feedback. The slow 21

33 cortical potential is extracted from the regular EEG on-line, filtered, corrected for eye movement artifacts, and fed back to the patient. When using visual feedback, the target positivity/negativity is represented by a high and low box on the screen. A ball-shaped light moves toward or away from the target box deping on subject s performance. The subject is reinforced for good performance with the appearance of a happy face or a melodic sound sequence. When a subject performs at least 75% correct, he/she is switched to the language support program. At level one, the alphabet is split into two halves (letter-banks) which are presented successively at the bottom of the screen for several seconds. If the subject selects the letter-bank being shown by generating a slow cortical potential shift, that side of the alphabet is split into two halves and so on, until a single letter is chosen. 7) An I m p la n te d B CI (G e o r gia S ta te U n iv e r s ity, USA ). The implanted brain-computer interface system devised by Kennedy and colleagues has been implanted into two patients [15][33]. These patients are trained to control a cursor with their implant and the velocity of the cursor is determined by the rate of neural firing. The neural wave-shapes are converted to pulses and three pulses are an input to the computer mouse. The first and second pulses control X and Y position of the cursor and a third pulse as a mouse click or enter signal. The patients are trained using software that contains a row of icons representing common phrases (Talk Assist developed at Georgia Tech). There are two paradigms using this software program and a third one using the visual keyboard. In the first paradigm, the cursor moves across the screen using one group of neural signals and down the screen using another group of larger amplitude signals. Starting in the top left corner, the patient enters the leftmost icon. He remains over the icon for two seconds so that the speech synthesizer is activated and phrases are produced. In the second paradigm, the patient is expected to move the cursor across the screen from one icon to the other. The patient is encouraged to be as accurate as possible, and then to speed up the cursor 22

34 movement while attempting to remain accurate. In the third paradigm, a visual keyboard is shown and the patient is encouraged to spell his name as accurately and quickly as possible and then to spell anything else he wishes. Unfortunately, the maximum communication rate with this BCI has been around 3 characters per minute. 8) T h e F le x ib le B r ain C o m p u te r I n te r fa c e (U n iv er s ity of Ro c h e s te r, USA ). Bayliss and colleagues [34] have performed an environmental control application in a virtual apartment that enables a subject to turn on/off a light, television set, and radio or say Hi/Bye to a virtual person. This system uses the P3 evoked potential in an immersive and dynamic Virtual Reality world. The main drawback of P3-based BCI's is their slowness. Single trial analysis may speed up recognition, but often at the cost of accuracy. A single trial accuracy average of 85% was obtained in an environment of virtual driving. Subjects were instructed to drive in a virtual town and stop at red stop lights while ignoring both green and yellow lights. The subjects used a virtual reality helmet, and a go cart with brake, accelerator, and steering output to control the virtual car. While this choice could have caused more artifacts in the signal collection (due to turning the steering wheel and braking), most of the artifact discovered and preprocessed was due to eye movement. 23

35 C H A P TE R 4 : A r tifa c t R e je c tio n One of the main problems in the automated EEG analysis is the detection of the different kinds of interference waveforms (artifacts) added to the EEG signal during the recording sessions. These interference waveforms, the artifacts, are any recorded electrical potentials not originated in brain. There are four main sources of artifacts emission: 1. EEG equipment. 2. Electrical interference external to the subject and recording system. 3. The leads and the electrodes. 4. The subject her/himself: normal electrical activity from the heart, eye blinking, eyes movement, and muscles in general. In case of visual inspections, the artifacts can be quite easily detected by EEG experts. However, during the automated analysis these signal patterns often cause serious misclassifications thus reducing the clinical usability of the automated analyzing systems. Recognition and elimination of the artifacts in real time EEG recordings is a complex task, but essential to the development of practical systems. Previous works have shown that the most severe of the artifacts are due to eye blinks and eyeball movements. A movement of the eyeball and the eyelids causes a change in the potential field because of the existing potential difference of about 100mV between the cornea and the retina [35]. This change affects mainly the signals from the most frontal electrodes (Fp1 and Fp2 and also other frontal electrodes: F3, F4, F7 and F8), and induces in them many high and low frequencies, deping upon its duration and amplitude. This can be explained by the figures shown in the next page. Fig. 11 shows the artifact free bipolar EEG as recorded from standard forehead locations (Fp1 or Fp2), and its corresponding spectrum. Fig. 12 shows the eye blink corrupted forehead EEG waveform and its spectrum. Fig. 13 shows an EEG eye blink corrupted signal in different electrodes. It is easy to note that the eye blinks introduce 24

36 significant amount of interference in the EEG spectrum (Fig. 11), and also that the artifact is more visible in the two most frontal electrodes Fp1 and Fp2 (Fig. 13). F ig u r e 11. A r tifact fr e e EE s p ec tr u m. G w av ef o r m r ec o rd e d b y a fo r e h e ad e le c tr o d e, a n d its F ig u r e 12. E ye b lin k a r tifa c t c o r r up te d E E G w av ef o r m rec o r d e d b y a fo r e h e ad e lec tr o d e, a n d its s p ec tr u m. In a clinical situation, such artifacts are High amplitude of delta wave rejected by visual examination of recording. There are simple criteria artifact recognition, such as those presented in [37], which can help in the search of an appropriate online cleaning technique. Some simple criteria, for a corrupted EEG signal, are the following: (0.5-4 Hz) in channels Fp1 and Fp2. Similarity of signals in channels Fp1 and Fp2. Rapid decline of delta wave posteriorly (the amplitude of delta wave in Fp1 and Fp2 is much higher than in other channels). 25

37 F ig u r e 4-3. Artifact in an EEG signal Classical methods for removing eyeblink artifacts can be classified into rejection methods and subtraction methods [40]: R eject ion m et h o d s consist of discarding contaminated EEG, based on either automatic or visual detection. Their success crucially deps on the quality of the detection, and its use deps also on the specific application for which it is used. Thus, although for epileptic applications, it can lead to an unacceptable loss of data, for others, like a Brain Computer interface, its use can be adequate. Sub tr a c tion m et h o d s are based on the assumption that the measured EEG is a linear combination of an original EEG and a signal caused by eye movement, called EOG (electro-oculogram). The EOG is a potential produced by movement 26

38 of the eye or eyelid (Fig. 14). The original EEG is hence recovered by subtracting separately recorded EOG from the measured EEG, using appropriate weights (rejecting the influence of the EOG on particular EEG channels). F ig u r e 13. E lec tr o -o c u lo g r am e lec tr o d e p la c e m e n t. T w o E OG c h a nn e ls, r e la te d to v e r tic a l a n d h o r iz o n tal e ye m ov e m e n ts (E O G V a n d E O G H ), a r e r e c o r d e d. More recently, new methods, based on the concept of blind source separation (BSS), have been proposed in order to separate neural activity from muscle and blink artifacts in spontaneous EEG data. In the following, four methods are presented. The first uses a BSS technique called indepent component analysis (ICA). The second is a classical rejection method. The third shows an artifact recognition technique through neural networks. The last is a rejection method based on band pass FIR filters R e mo v ing EEG a r tifa c ts b y I C A b lind s o urce s e p a r a tion Indepent component analysis (ICA) is a relatively recent method for blind source separation (BSS), which has shown to outperform the classical principal component analysis (PCA) in many applications. In particular, it has been applied for the extraction of ocular artifacts from the EEG, where principal PCA could not separate eye artifacts from brain signals, especially when they have comparable amplitudes. 27

39 ICA assumes the existence of n signals that are linear mixtures of m unknown indepent source signals. At time instant I the observed n-dimensional data vector x (i)= [x1(i)...xn(i)]t is given by the model [41] : where both the indepent source signals s(i)=[s1(i)...sm(i)] and the mixing matrix A = [akj] are unknown. Other conditions for the existence of a solution are (1) n =m (there are at least as many mixtures as the number of indepent sources), and (2) up to one source may be Gaussian. Under these assumptions, the ICA seeks a solution of the form: where B is called the separating matrix. Recent experiments, as those made by Jung and colleagues [39], have developed new methods for removing a wide variety of artifacts based on ICA. Over EEG data collected from normal, autistic and brain lesion subjects, ICA could detect, separate, and remove contamination from a wide variety of artifactual sources in EEG records with results comparing favorably to those obtained using regression and PCA methods [42][43]. This method presents some advantages compared to other rejection methods, such as: 1. ICA separates EEG signals including artifacts into indepent components based on the characteristics of the data, without relying on the availability of one 28

40 or more clean reference channels for each type of artifact. This avoids the problem of mutual contamination between regressing and regressed channels. 2. ICA-based artifact removal can preserve all of the recorded trials, a crucial advantage over rejection-based methods when limited data are available, or when blinks and muscle movements occur too frequently, as in some subject groups. 3. Unlike regression methods, ICA-based artifact removal can preserve data at all scalp channels, including frontal and periocular sites. Nevertheless, it is important to keep in mind that it also has some inherent limitations, such as: 1. ICA can decompose at most N sources from N scalp electrodes. Usually, the effective number of temporally-indepent signals contributing to the scalp EEG is unknown, and it is likely that observed brain activity arises from more physically separable effective sources than the available number of EEG electrodes. 2. The assumption of temporal indepence used by ICA cannot be satisfied when the training data set is too small, or when separate topographically distinguishable phenomena always occur concurrently in the data. In the latter case, simulations show that ICA may derive a component accounting for their joint occurrence, plus separate components accounting for their periods of solo activation. Such confounds imply that converging behavioral or other evidence must be obtained before concluding that spatio-temporally overlapping ICA components measure neuro-physiologically or functionally distinct activities. 3. ICA assumes that the physical sources of artifactual and neural activity y contributing to EEG signals are spatially stationary through time. In general, there is no reason to believe that cerebral and artifactual sources in the spontaneous EEG necessarily remain stationary over time or occurrences. 4. The fact that this method needs more computations compared to a rejection approach, together with the inherently real-time nature of the EEG Brain computer Interface, makes its use a more difficult alternative. 29

41 4. 2. A r tifa ct r e je c tio n b a s ed o n p e a k e lim in a tion As previous works have shown, the presence of artifacts in EEG signals produces a rapid increase of energy in forehead locations Fp1 and Fp2. The method developed here consists of the analysis of these two channels by small overlapping windows, in order to check if the energy of the signals surpasses an established blink threshold. In case it does it, the samples coming from the corrupted signal are rejected from all the EEG signals. Despite the simplicity of this method, the results obtained have been satisfactory enough to consider it as an initial option for a real-time Brain Computer Interface. It is easy enough to be implemented on a low complexity signal processing platform. Nevertheless, this method has the inconvenience of rejecting some non-corrupted data in other scalp channels, as well as in the frontal channels. With the purpose of improving the data preservation, we have developed a similar system based on a quadratic Time Frequency Representation of the signal. This analysis takes advantage of the high resolution of this technique in time and frequency, for establishing, after an appropriate training, the differences between a corrupted EEG signal and a non-corrupted EEG signal, in order to be able of distinguish and reject the eyeblink artifact. This technique, which is based on energy distributions, is an alternative to the classical artifact rejection method presented here, and it should be considered in a future work B lin k in g a r tifa c t rec o g n itio n u s ing a rt ific ia l n e u r a l n e tw o r k The method proposed by Bogacz and colleagues, used a neural based approach to find artifacts in EEG signals [38]. The input to the neural network was not a raw sampled signal, but different coefficients computed for a window of one second of the signal, expressing some characteristic properties of blinking artifacts. 41 coefficients were designed. Some of them were designed by the authors and were based on their knowledge about the artifact recognition, and a total of 14 were chosen by terms of sensitivity and correlation. A large training set including coefficients for over windows was used, containing different kinds of blinking artifacts, pathological and proper waves, and 30

42 artifacts caused by other sources (e.g. jaw, muscle). Afterwards, three classification algorithms were tested and compared: k-neighbors, RBF networks and back propagation networks. The lowest classification error (1.40%) was obtained for the back propagation network, with a classification time of the test set (6227 windows) of 2 seconds [38]. This method achieves high classification accuracy thanks to two factors: Large training set containing different kinds of EEG waves. The coefficients delivered to the network s inputs, express the characteristic features of artifacts, since they encode large amount of domain expert s knowledge. Unfortunately, the first factor can be problematic in the use of a Brain computer Interface A r tifa ct r e jec tio n b a s ed in b a n d p a s s F I R filters The method proposed by Gupta and colleagues used a fixed band pass FIR filter, followed by a subject specific eye blink threshold, in order to remove the eyeblink and eyeball movement artifacts. These techniques, whose block diagram can be seen in Fig.15, consist of [35]: 1. Pass the raw EEG samples obtained from analog-to-digital converter through a digital band pass filter (BPF) to remove slow baseline drift. 2. Determine the blink threshold (Vt) for specific subject in brief training session. 3. Compare the absolute sample value with Vt. 4. If the value is exceeded then remove N samples from the vicinity of zero crossing (N/2 on either side of treshold crossing). 5. Shift the following N samples to fill up the gap created by blink removal. These gaps will, otherwise, grossly distort the spectrum. 31

43 F ig u r e 14. S c he m e of th e p r o p o s e d s ys te m. The experiments carried out through this scheme, for Fp1 and Fp2 electrodes location, provided interesting results in eyeblink artifact rejection. This method presents the advantage of working even under baseline drift artifacts conditions, and also is easy enough to be implemented on a low-cost digital signal processor, on a real time system. Nevertheless, it fails if the blink rate is unnaturally high, and the training session for each individual is quite long: 30 sec (6 eyeblinks on a average for a normal subject). 32

44 C H A P TE R 5 : E E G S ig n a l P re - Pr o c e ss in g 5.1 M et h o d o log y a n d E x pe r im e nt In one- dimensional cursor control studies, trained users are able to effectively modulate 8 12 Hz (µ band) spectral components over the sensorimotor cortex to move a cursor toward a randomly positioned target on a monitor. In order to investigate the nature of the rhythm for a particular user, the characteristics of the µ-rhythm were examined using data. Users had exhibited strong -band activity during an initial screening and were subsequently trained on a simple two-target, one-dimensional cursor control task Doing Cursor control experiment The one-dimensional sensorimotor rhythm cursor control task is shown in Fig 15. For the task, the users were presented with a target randomly positioned at the top or bottom of the right edge of the monitor. The trial began with the cursor at the center of the left edge of the monitor. It moved at a constant rate toward the right, reaching the right side of the monitor after 16sec. The user s goal was to move the cursor upward or downward to the height of the target so that it hit the target when it reached the right side of the monitor. The trials continued in 11-min runs, with a 1-min break given between runs. A single 11-min run consisted of 40 trails and 4 runs (160 times in total) constituted a single session. We collected the data from 3 healthy persons doing the experiment, the data was collected at 128Hz sampling rate. F ig u r e 15. O n e -d im e n s io n al ta s k tr ial s tr u c tu r e : (1) T h e ta r g e t a n d c u r s o r a r e p r e s e n t on th e s c r ee n fo r 1 s. (2) T h e c u r s or m o v e s s te a d ily a c r o s s th e s c r e e n fo r 16 s w ith its v e r tic al m ov e m e n t c o n tr ol le d b y th e u s e r. (3) I f th e u s e r h its th e ta r g e t, th e ta r g e t fla s h e s fo r 0. 5 s. (4 ) T h e s cree n go e s b la n k fo r a 1 m in a ft e r e v e r y 40 tr ial s. (5) T h e n e x t tr ia l b e g in s. 33

45 F ig u r e 16. Timing Diagram of cursor control experiment Imagination of cursor control experiment In this experiment the cursor moves automatically for every 1sec towards the target the user has to think about the voluntary action that he is pressing the respective key to move the cursor towards target. The experiment consists of 160 trails, the cursor moves towards the target in 9 sec. The task for the user is he has to think that he is pressing the respective key to move the cursor towards target. The experiment has a break after 80 trails and when the subject gets relaxed and feels he is comfortable he can continue the experiment again. F ig u r e 17. Timing Diagram of cursor control imagination experiment These experiments are designed using PsychToolbox. The attraction of using computer displays for visual psychophysics is that they allow software specification of the stimulus. 34

46 5.1.3 PsychToolbox Programs to run experiments are often written in a low-level language to achieve full control of the hardware for precise stimulus display. Although these low-level languages provide power and flexibility, they are not conducive to rapid program development. Interpreted languages are abstracted from hardware details and provide frilier development environments, but don't provide the hardware control needed for precise stimulus display. The Psychophysics Toolbox is a software package that adds this capability to the Matlab. Matlab is a high-level interpreted language with extensive support for numerical calculations. The Psychophysics Toolbox interfaces between Matlab and the computer hardware. The Psych toolbox's core routines provide access to the display frame buffer and color lookup table, allow synchronization with the vertical retrace, support millisecond timing, allow access to OpenGL commands, and facilitate the collection of observer responses. Ancillary routines support common needs like color space transformations and the QUEST threshold seeking algorithm. The Matlab & Psychtoolbox environment is flexible yet relatively easy to learn. Canned experimental programs fail because they usually can't do a really new experiment. For that you need the expressive scope of a full-fledged computer language, such as C or Matlab. Matlab is a particularly good language for running laboratory experiments. Even for experienced programmers, three features of Matlab greatly speed the development cycle over other languages. Matlab has a rich library of high-level functions available to do math and plotting. It operates on arrays and images as named variables. And it is interactive, so that one can type 1+1 and immediately see the answer 2 which is invaluable when developing laboratory software to run experiments. The Psychtoolbox doesn't limit the user if the experiment can be run on the hardware, it can be run with the Psychtoolbox. The Matlab-Psychtoolbox combination has four winning features that are recomm for any experiment-design environment: A general purpose language (Matlab) allows you to do new things. 35

47 a) For programs that use hardware intensely (e.g. display, keyboard), an interpreted environment (e.g. Matlab) speeds up software development greatly because simple tests can be performed immediately. b) The key Psychtoolbox routines are C code, callable as functions from Matlab that encapsulates the hardware, presenting a simple software interface to the user that provides full control. (In particular, the PsychtoolboxScreen.mex function provides a consistent high-performance user interface to the display, overcoming differences in synchronization behavior among graphics drivers from many manufacturers, within and between Mac and Win platforms.) c) The Psychtoolbox Rush function allows you to run an arbitrary bit of code with little or no interruption. We call this "hogging the machine", blocking interrupts for the few seconds of a critical stimulus presentation. The Psychtoolbox also provides interfaces for timing, sound, keyboard, and the serial port. And it includes many useful Matlab routines, such as color space transformations 5.2 Da ta c o lle c tion The details of the data collection and analysis are as follows: Using EEG software the EEG activity was collected from 24 channels at standard locations [5.6] distributed over the scalp. All 24 channels were referenced to the Cz, band pass filtered ( Hz) and digitized at 128 Hz, with sensitivity of 100µv/cm. The EEG equipment setup is shown below. Type-D EEG hardware mainly consists of master controller and type-d EEG amplifier. ERP stimulator can also be connected to the system. 1. M a s te r Co n tr ol le r : This is the control station of the system, coordinating the inputs from the ERP stimulator and the EEG amplifier. The following connections are present in master controller a) USB Port (HDMI Interface): EEG data collected is sent to this controller before we transmitted to the computer through this channel. 36

48 b) DB15 Stimulator Interface: This is used to connect with the ERP stimulator to do coordinated stimulation for research and clinical studies. c) Fibre optic Cable Ports: Connect the fibre optic cables between the Master Controller and the EEG amplifier to transmit EEG data. d) Power Supply Cable: Power supply to the controller, charging the DC 24V/2A battery within. 2. T y p e -D E E G A m p lifie r : The type-d EEG amplifier can be used for EEG and PSG studies. It also consists of a built in impedance tester for output noise control. a) When the amplifier is switched on the indicator light of hardware operation turns green. b) When the impedance testing is switched on, the testing function becomes available when the 5K indicator turns to green. The threshold for impedance testing is 5K. A cautionary red light will appear if the impedance is more than 5K and the impedance test fails. c) The amplifier consists of 16 EEG channel connection codes. d) The PSG section in the amplifier consists of 11 ports. 3. E E G a m p lifie r P o w e r C h a r g e r : amplifier charger connects to the power source, which is useful for recharging the battery and running the amplifier with AC power. 4. E E G c a p : During the procedure, electrodes consisting of small metal discs with thin wires are pasted on the scalp with a conductive gel or paste, usually after preparing the scalp area by light abrasion to reduce impedance due to dead skin cells. These electrodes detect tiny electrical charges that result from the activity of the brain cells. Difference in push or pull voltages between any two electrodes can be measured by a voltmeter. Recording these voltages over time gives us the EEG. Some systems use caps or nets into which electrodes are embedded. 37

49 EEG Amplifier Power Charger Single Electrode cable for type - D Model D EEG Amplifier Integrated monitoring EEG cap Fiber Optic Transmission Cable USB Cable (HDMI) Controller of EEG PC F ig u r e 18. EEG hardware setup 38

50 Electrode locations and names are specified by the International system for most clinical and research applications (except when high-density arrays are used). This system ensures that the naming of electrodes is consistent across laboratories, so that a subject's studies could be compared over time and subjects could be compared to each other. This system is based on the relationship between the location of a electrode and the underlying area of cerebral cortex. The "10" and "20" refer to the fact that the actual distances between adjacent electrodes are either 10% or 20% of the total front back or right left distance of the skull. Nasion Inion F ig u r e 19. E le c tr o d e lo c a tio n s a n d n a m e s b y th e I n te r n a tio n al sy s te m 39

51 Each site has a letter to identify the lobe and a number to identify the hemisphere location. The letters F, T, C, P and O stand for frontal, temporal, central, parietal, and occipital lobes, respectively. Note that there exists no central lobe; the "C" letter is used only for identification purposes. A "z" (zero) refers to an electrode placed on the midline. Even numbers (2, 4, 6, 8) refer to electrode positions on the right hemisphere, whereas odd numbers (1, 3, 5, 7) refer to those on the left hemisphere. In addition, the letter codes A, Pg and Fp identifies the earlobes, nasopharyngeal and frontal polar sites respectively. Two anatomical landmarks are used for the essential positioning of the EEG electrodes: first, the nasion which is the distinctly depressed area between the eyes, just above the bridge of the nose; second, the inion, which is the lowest point of the skull from the back of the head and is normally indicated by a prominent bump. 5. Cab le s : Three types of cables are used in this EEG hardware. They are a) S in g le E le c tr o d e c a b le fo r ty p e D: Each electrode on the EEG cap is connected to a specific location on a Model-D amplifier with a single cable. So every electrode is connected to the amplifier with individual cables. b ) F ib r e O p tic T r a n s m is s ion Cab le : The inputs of Model-D EEG amplifier is given to the master controller of EEG using this cable, by sing pulses of light through an optical fibre. Due to much lower attenuation and interference, this particular type of cable is used. USB Cab le (H D M I ): A USB cable is used for connecting the master controller to a USB port in a computer. This cable helps to easily transfer data from the controller to the computer. 40

52 C3 electrodes C4 electrodes F ig u r e 20. Raw d a ta 41

53 5.3 A r tifa ct re m o v a l a n d B a nd p a s s filte r in g The moving average is the most common filter in DSP, mainly because it is the easiest digital filter to understand and use. A moving average is commonly used with time series data to smooth out short-term fluctuations and highlight longer-term trs or cycles. The threshold between short-term and long-term deps on the application, and the parameters of the moving average will be set accordingly. The moving averages (MA) algorithms are the classical forecasting algorithms that have been used for decades in the forecasting field. They are basically simple algorithms that can even be computed by hand (and this is why they have been so popular in the past when computers were not available to the general public). All the Moving Averages algorithms share some common trait: They smooth the curve. They introduce some lag in the calculation so that the MA lags the original signal. They cut the highest frequencies (i.e. numerically they are low pass filters). The data collected from EEG machine at C3 and C4 electrode is passed through moving average filter. Here we use a window of size 12. Given a series of numbers and a fixed subset size, the first element of the moving average is obtained by taking the average of the initial fixed subset of the number series. Then the subset is modified by "shifting forward"; that is, excluding the first number of the series and including the next number following the original subset in the series. This creates a new subset of numbers, which is averaged. This process is repeated over the entire data series. The plot line connecting all the (fixed) averages is the moving average. A moving average is a set of numbers, each of which is the average of the corresponding subset of a larger set of datum points. Matlab code for the moving averages is shown in the appix. The plot of the data of a single subject after passing through moving average filter is shown in figure below. 42

54 The moving average code is written in Matlab, the code is as follows function movavg =movingavg(data,channel) a=importdata(data); chlen=length(channel); for i=1:chlen b(:,i)=a.data(:,channel(i)); sum=0; movavg.avg= zeros(length(b),chlen); k=1; window_size =12; for m= 1:chlen for j =1: length(b) if k < length(b) - window_size for i = k:k+window_size sum = sum + b(i,m); nd; movavg.avg(j,m)= sum/window_size; sum =0; k=k+1; k=1; 43

55 F ig u r e 21. M ovi n g av e r ag e d d a ta. The moving averaged data is them filtered using a band pass filter in the frequencies 0.1Hz to 25Hz. The code for band pass filter is written in the Matlab. Here a denotes frequency of lower band, b denotes frequency of higher band and Fs denotes sampling frequency. 44

56 function Hd = eegbandfilter(a,b,fs) if(nargin<3) Fs = 250; % Sampling Frequency Fstop1 = a-0.5; % First Stopband Frequency Fpass1 = a; % First Passband Frequency Fpass2 = b; % Second Passband Frequency Fstop2 = b+0.5; % Second Stopband Frequency Dstop1 = 0.001; % First Stopband Attenuation Dpass = ; % Passband Ripple Dstop2 = ; % Second Stopband Attenuation dens = 20; % Density Factor % Calculate the order from the parameters using FIRPMORD. [N, Fo, Ao, W] = firpmord([fstop1 Fpass1 Fpass2 Fstop2]/(Fs/2), [ ], [Dstop1 Dpass Dstop2]); % Calculate the coefficients using the FIRPM function. b = firpm(n, Fo, Ao, W, {dens}); Hd = dfilt.dffir(b); F ig u r e 22. B a n d p a ss e d s ig n al 45

57 5.4 M U R h y thm M u w av e s, also known as m u r h y th m s, comb or wicket rhythms, arciform rhythms, or sensorimotor rhythms, are synchronized patterns of electrical activity involving large numbers of neurons, probably of the pyramidal type, in the part of the brain that controls voluntary movement. These patterns as measured by electroencephalography (EEG), magnetoencephalography (MEG), or electrocorticography (ECoG) repeat at a frequency of 8 12 Hz. The band passed data is passé through a filter with frequencies 8 and 12Hz. Hd1 = eegbandfilter(8,12,128); for i=1:chlen movavg.dalpha(:,i) = filter(hd1,movavg.avg(:,i)); F ig u r e 23. M u r h y th m sig n al 46

58 5.5 E p o c h s The user has to move the cursor in vertical direction so as to hit the target present on the right side of the screen using z or m keys to move the cursor down and up respectively. These data are to be considered as a single trial so they are to be epoched for the further analysis. But all the epochs should be of same length with same time stamps. So finding the maximum time taken by cursor to hit the target, each epoch length is fixed to that duration. Keeping all these things the code for epochs is written in Matlab. function Epochs= extractepoch2(data,channel,csv) chlen= length(channel); d=data; k=1; - nk=1; for i=1:chlen c(:,i)=d((128*60):length(d),i); o=csvread(csv); p=max(o(:,15)); Epochs.epochC3=zeros(ceil(p*128),160); Epochs.epochC4=zeros(ceil(p*128),160); for i=1:160 Epochs.epochC3((1:ceil(p*128)),i)= c((1*k):((floor(p*128))+k),1); k=k+ceil(p*128); if (rem(i,40)==0) k=k+60*128; 47

59 Epochs.epochC4((1:ceil(p*128)),i)= c((1*nk):((floor(p*128))+nk),2); nk=nk+ceil(p*128); if (rem(i,40)==0) nk=nk+60*128; Here Data refers to the band passed signal data, channel refers to the electrode to which we are concerned and csv refers to the comma separated file generated by PsychToolbox during the experiment. F ig u r e 24. E p o c h s 48

60 5.6 A v e r a g in g To get the ERP curve, we have to average all the trials in which cursor hits top target and bottom target separately. Plotting these epochs we get ERP curves. To know whether the epoch is of top target or bottom target we take help of the CSV file which has the time stamps of the experiment. function zmepoch=zm_epoch(data1,data2,csv) timestamps=csvread(csv); a=1; b=1; for i=1:(length(timestamps(:,1))) if timestamps(i,5)==1 zmepoch.m(:,a)=data1(:,a); a=a+1; elseif timestamps(i,5)==2 zmepoch.z(:,b)=data2(:,b); b=b+1; l=length(zmepoch.m(1,:)); k=length(zmepoch.z(1,:)); avgm=zeros(length(zmepoch.m(:,1)),1); avgz=zeros(length(zmepoch.z(:,1)),1); for i=1:l avgm=avgm+ zmepoch.m(:,i); ezmepoch.avgm=avgm/l ; for j=1:k avgz=avgz+ zmepoch.z(:,j); zmepoch.avgz=avgz/k; 49

61 subplot(2,1,1) plot(zmepoch.avgm) ylabel('amlitude in micro volts') xlabel('time in seconds') title('averaged Epoch of all top targets'); subplot(2,1,2) plot(zmepoch.avgz) ylabel('amlitude in micro volts') xlabel('time in seconds') title('averaged Epoch of all bottom targets'); Here data refers to the Epoched data and csv refers to the comma separated file generated by PsychToolbox during the experiment. F ig u r e 25. Av e r a g e d to p a n d b o tto m ta r g e ts In this averaged top and bottom targets the first 1sec serve as a template for the offline classification. 50

62 C H A P TE R 6 : T e m p la te G en e r a tio n fo r o n line c la s s ific a tio n The first step toward extracting the characteristic µ rhythm wave shape for a particular user was the determination of the spectral component over the sensorimotor cortex that demonstrated the highest correlation with the target location. This was accomplished using features generated by a optimum order AR model. This spectral component was assumed to be the fundamental frequency of the user s rhythm. Next, data from each trial for the target location that corresponded to µ-rhythm synchronization was cross-correlated with a 1-s sinusoid template at the fundamental frequency. The 1-s segments having the maximum correlation for each trial were collected over two entire sessions of data. These phase aligned data segments were then averaged to expose the prevailing characteristics of the control signal. The averaging revealed distinctive characteristic µ-rhythm are non-sinusoidal and periodic, and that several of the user s characteristic waveforms resemble the classical arch shape. Although the rhythm typically ts to appear in bursts lasting anywhere from less than 1 s to several seconds, the averaged waveforms maintain a relatively constant amplitude envelope during the 1-s interval. Because of this constant envelope and because no significant temporal variations were evident, other than that attributed to noise, it is feasible to derive a parameterized model of the rhythm that could be easily applied as a matched filter template. 6.1 STE P W I SE LE A ST SQ UAR ES E S T I M A T I ON F OR A R M OD EL S To analyze the eigenmodes of an AR (p) model fitted to a time series of observations of a complex system, the unknown model order p and the unknown model parameters A1; : : : ;Ap, w, and C must first be estimated. The model order is commonly estimated as the optimizer of what is called an order selection criterion, a function that deps on the noise covariance matrix ^ C of an estimated AR (p) model and that penalizes the over parameterization of a model An m-variate AR (p) model for a stationary time series of state vectors observed at equally spaced instants v, is defined by 51

63 Where the m-dimensional vectors vectors with mean zero and covariance matrix are uncorrelated random and the matrices are the coefficient matrices of the AR model. To determine the model order popt that optimizes the order selection criterion, the noise covariance matrices C are estimated and the order selection criterion is evaluated for AR(p) models of successive orders pmin _ p _ pmax. If the parameters A1; : : : ;Ap, and w are not estimated along with the noise covariance matrix C, they are then estimated for a model of the optimum order popt. Both asymptotic theory and simulations indicate that, if the coefficient matrices A1; : : : ;Ap and the intercept vector w of an AR model are estimated with the method of least squares, the residual covariance matrix ^ C of the estimated model is a fairly reliable estimator of the noise covariance matrix C and hence can be used in order selection criteria. The least squares estimates of AR parameters are obtained by casting an AR model in the form of an ordinary regression model and estimating the parameters of the regression model with the method of least squares. Numerically, the least squares problem for the ordinary regression model can be solved with standard methods that involve the factorization of a data matrix. In what follows, we will present a stepwise least squares algorithm with which, in a computationally efficient and stable manner, the parameters of an AR model can be estimated and an order selection criterion can be evaluated for AR(p) models of successive orders pmin _ p _ pmax. Starting from a review of how the least squares estimates for an AR(p) model of fixed order p can be computed via a QR factorization of a data matrix, we will show how, from the same QR factorization, approximate least squares estimates for models of lower order p0 < p can be obtained. 52

64 F ig u r e 26. E x tr a c tin g A R fe a tu re s a n d c o n c a ten a tin g AR coefficients for C3 53

65 AR coefficients for C4 The coefficients obtained for the AR model for both the top target and bottom target are concatenated row wise. Concatenated matrix AR coefficients 54

66 6.2 F a s t F o u r ie r tr a n s fo rm To find the frequency of the coefficients we have to convert to frequency domain. To convert to frequency domain we use FFT in built MATLAB function. After applying FFT and converting into frequency domain we need to convert the complex terms to magnitude. After finding magnitude we have to find the maximum value from it. And the frequency is given by f_max = i_max * Fs / N; where f_max is frequency, i_max is magnitude, Fs is sampling frequency and the N is the size of FFT. The code for FFT and its magnitude is as follows function Ftrans= F_Transform(Data) b=data; [n,m] = size(b); for i=1:m c=b(:,i); Ftrans.d(:,i)=fft(c,160); for i= 1:m Ftrans.c(:,i)=abs(Ftrans.d(:,i)); 55

67 We get a maximum value of using this we can find frequency as we already know Fs which is 128 and N is 160 point FFT. Frequency = ( *128)/160 = Hz =9Hz 6.3 C r o s s c o rr ela tion The averaged top target epoch and bottom target epoch is cross correlated with the 1sec sinusoidal signal of the frequency obtained from the AR model to get the matched filter templates. function cross_cor=onesec_cross_cor(data1,data2,csv) t=0:0.08:1; f=9; a=sin(2*pi*f*t); o=csvread(csv); start1=0; count1=0; count2=0; start2=0; for i= 1:160 if (o(i,5)==1) start1=start1+o(i,17); count1=count1+1; 56

68 start1=start1/count1; cross_cor.m(:,1)=xcorr(a,data1(start1*128:start1* ,1)); for i= 1:160 if (o(i,5)==2) start2=start2+o(i,17); count2=count2+1; start2=start2/count1; cross_cor.z(:,1)=xcorr(a,data2(start2*128:start2* ,1)); Here data1 refers to the averaged top target and data2 refers to bottom target epoch and csv refers to the comma separated file generated by PsychToolbox during the experiment. F ig u r e 27. T e m p la te fo r top ta r g et s 57

69 F ig u r e 28. T e m p la te fo r b o tto m ta r g e ts From the above template we can observe that the user is synchronizing Mu rhythm for top targets, and desynchronizing Mu rhythm for bottom targets. These templates are used to extract features for other data. This is because the users are Right handed persons, as they frequently use right hand there is a synchronization in Mu rhythm for top targets as the users is controlling the cursor to hit top target using right hand and vice versa. These templates are used in classification of online cursor control. 58

70 C H A P T E R 7 : F e a tu r e e x tr a c tio n a nd C la s s ific a tio n 7.1 F e a tu re e xt r a c tion We know that cross correlation of two similar signals will give a larger peak compared to the cross correlation of two dissimilar signals. This concept is used for the extracting features and classifying them. So here we cross correlate every epoch with the two standard templates i.e. top and bottom targets templates when we have a larger peak in top target correlated signal we classify it as top target or else bottom target Doing the one dimensional cursor control experiment The data recorded from the other subjects used for classification, these data need to be preprocessed as mentioned above first i.e. we have to apply moving average to raw data. To the moving averaged data we have to band pass it to a filter of frequency 0.1 to 25Hz, after band passing we have to extract Mu rhythm( 8 to 12hz) signal from the data. After preprocessing we have to find features for the data. These features are obtained from the cross correlated data of each epoch and the standard templates. The code for feature extraction is written in Matlab. In this we are cross correlating each epoch with templates obtained after averaging and finding the maximum values of the correlated signals. These maximum values stand as the features for classification. function features=featureprocess(data,fs,channel,csv, paramsfile1, paramsfile2) if(nargin < 2) fs = 250; o=csvread(csv); dur=max(o(:,15)); 59

71 Epochs= extractepoch2(data,channel,csv); for i=1:160 start=o(i,17); features.cor_m(:,i) = xcorr(epochs.epochc3(start*128:start* ,i),paramsfile1); features.cor_z(:,i) = xcorr(epochs.epochc4(start*128:start* ,i),paramsfile2); feat=zeros(160,2); for i=1:length(feat) feat(i,1) = max(features.cor_m(:,i).^2); feat(i,2) = max(features.cor_z(:,i).^2); features.feat=feat; Here data refers to the preprocessed data, Fs refers to the sampling frequency, CSV refers to the comma separated file generated by PsychToolbox during the experiment, Params file1 refers to the offline template for top targets and the params file2 refer to the offline template of bottom target. 60

72 The feature obtained from a single subject can be seen in the above table as feat of size 160 by2. This feat indicates that there are 2 columns of maximum values. These maximum values are found for each trial. F ig u r e 29. F e a tu r e s of a s in g le s u b jec t 61

73 F ig u r e 30. Cross correlation waves 62

74 7.1.2 Imaging the cursor control experiment In this experiment the cursor moves automatically for every 1sec towards the target the user has to think about the voluntary action that he is pressing the respective key to move the cursor towards target. The experiment consist of 160 trails, the cursor moves towards the target in 9 sec the user has to think that he is pressing the key to move the cursor towards target. The experiment has a break after 80 trails and when the subject gets relaxed and feels he is comfortable he can continue the experiment again. The data is collected using the EEG cap. The data recorded is used for classification, this data need to be preprocessed as mentioned above first i.e. we have to apply moving average to raw data. To the moving averaged data we have to band pass it to a filter of frequency 0.1 to 25Hz, after band passing we have to extract Mu rhythm ( 8 to 12hz) signal from the data. After preprocessing we have to find features for the data. These features are obtained from the cross correlated data of each epoch and the offline templates. function features=featureprocessimag(data,fs,channel,csv,matfile, paramsfile1, paramsfile2) if(nargin < 2) fs = 250; o=csvread(csv); dur=max(o(:,23)); Epochs= extractepochimag(data,channel,csv,matfile); for i=1:160 start=o(i,25); 63

75 features.cor_m(:,i) = xcorr(epochs.epochc3(start*128:start* ,i),paramsfile1); features.cor_z(:,i) = xcorr(epochs.epochc4(start*128:start* ,i),paramsfile2); feat=zeros(160,2); for i=1:length(feat) feat(i,1) = max(features.cor_m(:,i).^2); feat(i,2) = max(features.cor_z(:,i).^2); features.feat=feat; Here data refers to the preprocessed data, Fs refers to the sampling frequency, CSV refers to the comma separated file generated by PsychToolbox during the experiment, matfile refers to the mat format file saved during the experiment, Params file1 refers to the offline template for top targets and the params file2 refer to the offline template of bottom target. 64

76 F ig u r e 31. F e a tu r e s o b tai n e d fr o m th e im ag in a r y d a ta As the experiment is slightly is modified the code for extracting the epochs will also varied because in previous case there are 4 breaks for every 40 trials but in imagination of the cursor movement there is only a single break after 80 trials and also each trail is of only 9sec in the latter and it is of 16sec in former experiment. So the code for extracting the epochs is as follows 65

77 function Epochs= extractepochimag(data,channel,csv,matfile) chlen= length(channel); d=data; k=1; nk=1; mat=importdata(matfile); for i=1:chlen c(:,i)=d((128*60):length(d),i); o=csvread(csv); p=max(o(:,23)); Epochs.epochC3=zeros(ceil(p*128),160); Epochs.epochC4=zeros(ceil(p*128),160); for i=1:160 Epochs.epochC3((1:ceil(p*128)),i)= c((1*k):((floor(p*128))+k),1); k=k+ceil(p*128); if (rem(i,80)==0) && (rem(1,160)~=0) k=k+mat.break*128; Epochs.epochC4((1:ceil(p*128)),i)= c((1*nk):((floor(p*128))+nk),2); nk=nk+ceil(p*128); if (rem(i,80)==0) && (rem(1,160)~=0) nk=nk+mat.break*128; 66

78 Here data refers to the preprocessing data, csv refers to the comma separated file obtained from the Psychtoolbox experiment, channel refers to the electrodes which we are concerned and the mat file refers to the matlab file obtained from the matlab, it has break time. F ig u r e 32. E p o c h s o b ta in e d fr om th e im ag in a tion d a ta 67

79 F ig u r e 33. C r o s s c o rre la tion w av ef o r m s 68

80 7.2 Cla s si fic a tio n u s ing c r o s s c o rre la tion The extracted features are to be classified that whether the trail is of top target are bottom target. So in order to compare the classified results we have to know the whether the trail is of top or bottom target before hand, we know this from the CSV file. Using this CSV file we compare the classified results and find the accuracy of the classifier Doing the cursor control experiment The features obtained are present in two columns. Each row indicate one trail, comparing the two columns of each trail and finding the maximum of it gives us the classified results. After classifying all the trails we have to verify the classified results with the known results and calculate the accuracy. function classify=classify_zm(features,csv) timestamps=csvread(csv); n = length(features); % Number of samples classify.labels1 = timestamps(:,5);% Labels -1 and 1 for i=1:length(timestamps(:,5)) if timestamps(i,5)==2 classify.labels1(i,1)=-1; r=[]; for i=1:n if features(i,1)>features(i,2) r=[r;1]; elseif features(i,1)<features(i,2) r=[r;-1]; 69

81 else r=[r;0]; classify.m1=r; k=0; count=0; for i=1:length(classify.m1) if classify.m1(i,1)==classify.labels1(i,1) k=k+1; count = k; classify.accu=((count)/length(classify.m1))*100; Here the features indicate the features obtained in the feature extraction and the CSV refers to the comma separated file generated by PsychToolbox during the experiment. F ig u r e 34. A cc u r a c y o b tai n e d fr o m s ub jec t 2 70

82 F ig u r e 35. A cc u r a c y o b tai n e d fr o m th e s u b jec t Imagining the cursor control experiment The features obtained are present in two columns. Each row indicate one trail, comparing the two columns of each trail and finding the maximum of it gives us the classified results. After classifying all the trails we have to verify the classified results with the known results and calculate the accuracy. function classify=classify_zm_imag(features,csv) timestamps=csvread(csv); n = length(features); % Number of samples classify.labels1 = timestamps(:,3);% Labels -1 and 1 for i=1:length(timestamps(:,3)) if timestamps(i,3)==2 classify.labels1(i,1)=-1; r=[]; 71

83 for i=1:n if features(i,1)>features(i,2) r=[r;1]; elseif features(i,1)<features(i,2) r=[r;-1]; else r=[r;0]; classify.m1=r; k=0; count=0; for i=1:length(classify.m1) if classify.m1(i,1)==classify.labels1(i,1) k=k+1; count = k; classify.accu=((count)/length(classify.m1))*100; As the experiment I different the timestamps in the CSV file stored are in different manner so we use this code for classifying the imagination data. Here the features indicate the features obtained in the feature extraction and the CSV refers to the comma separated file generated by PsychToolbox during the experiment. 72

84 F ig u r e 36. A cc u r a c y o b tai n e d fr o m th e im ag in a tion d a ta Comparing the accuracies of the classifiers for all the subjects by plotting them Accuracy of classifier Accuracy subject2 doing subject3 doing subject4 imagination F ig u r e 37. Comparing accuracy of the all subjects Doing and Imagination 73