GROUND VEHICLE NAVIGATION USING WIRELESS EEG. by Dilara Semerci

Transcription

1 GROUND VEHICLE NAVIGATION USING WIRELESS EEG by Dilara Semerci Submitted to the Department of Computer Engineering in partial fulfillment of the requirements for the degree of Bachelor of Science in Computer Engineering Boğaziçi University 2012

2 i ACKNOWLEDGEMENTS I would like to thank my advisor, H. Levent Akın for assigning me to this beautiful project and for his invaluable guidance and support. This project taught me a lot of things that I was unfamiliar with. I also appreciate Albert Ali Salah for supplying this incredible headset. It was great to work with it. Moreover, I would like to thank F.Serhan Daniş for helping me throughout the project; he helped me with the car and shared his brain signals with me during my experiments along with everyone in the AILAB for sharing their comments and arguments about the project. Finally, I would like to thank my family for supporting me during this long education life and giving all the opportunities to make this final possible in Bogazici.

3 ii ABSTRACT GROUND VEHICLE NAVIGATION USING WIRELESS EEG Brain Computer Interfaces are the communication devices between the brain and the applications where these applications can be robotic hand/leg control, wheel chair control, mouse emulator, game control etc. Their primary advantage is supplying a more livable environment to the handicapped people. As mentioned above, they do not serve only to the biomedical science but also started to be used for entertainment purposes, too. Emotiv has recently released a headset and an interface that communicates with that headser which can be used to generate any input as the user wishes to have in his/her application. Writing s or playing computer games became possible by only thinking about the commands or by mimicking. The enhancements in this area are leading our world to be a soon after mind-controlled. In this thesis, we examine the Emotiv EPOC and its built-in filtering and classification system with an application that is used for navigating a toy car. We worked on the classification of cognitive actions which is followed by facial-expressive actions.

4 iii TABLE OF CONTENTS ACKNOWLEDGEMENTS i ABSTRACT ii LIST OF FIGURES v LIST OF TABLES vii LIST OF SYMBOLS viii LIST OF ACRONYMS/ABBREVIATIONS ix 1. INTRODUCTION UNDERSTANDING THE BRAIN Structure of the Brain The Hemispheres The Lobes SCANNING THE BRAIN Introduction to Brain-Computer Interfaces Invasive BCIs Non-Invasive BCIs Electroencephalography (EEG) Magnetoencephalography (MEG) Functional Magnetic Resonance Imaging (fmri) Near Infrared Spectroscopy (NIRS) EEG Introduction to EEG Signal Types Delta Waves Theta Waves Alpha Waves Beta Waves Gamma Waves Mu Rhythm Signal Acquisition

5 iv Recording Techniques Sensor Layout Artifacts Feature Extraction Classification Output Application and Feedback PRELIMINARY WORK EMOTIV EPOC Control Panel EmoKey TestBench APPLICATION Toy Vehicle Program Using Emotiv Control Panel Introduction to Emotiv API and Emotiv EmoEngine Program written in C Racing Car Trial with EmoKey CONCLUSION REFERENCES

6 v LIST OF FIGURES Figure 2.1. Left Brain vs. Right Brain [1] Figure 2.2. Human Brain [2] Figure 3.1. BCI Design Pattern Figure 4.1. Delta Waves [3] Figure 4.2. Theta Waves [3] Figure 4.3. Alpha Waves [3] Figure 4.4. Beta Waves [3] Figure 4.5. Gamma Waves [3] Figure 4.6. Waves of the Brain [4] Figure 4.7. The international system seen from (A) left and (B) above the head. (C) Location and nomenclature of the intermediate 10% electrodes, as standardized by the American Electroencephalographic Society [5] Figure 5.1. Decomposition of DWT; A3 level filter bank [6] Figure 5.2. Two-layer, Feed-forward, Back propagation Neutral Network Figure 6.1. Emotiv EPOC Sensor Locations

7 vi Figure 6.2. Connecting the Headset. Green, yellow and orange imply good, fair, poor signal respectively Figure 7.1. Application Interaction with Emotiv EPOC and EmoEngine Figure 7.2. Flowchart of the System Figure 7.3. Training with unplugged Joystick Figure 7.4. Expressions with relevant commands. Raise eyebrow for GO, Furrow eyebrow for STOP, Smirk left for TURN LEFT and Smirk right for TURN RIGHT

8 vii LIST OF TABLES Table 4.1. EEG Signals and Frequency Intervals Table 5.1. Signal Decomposition Table 6.1. Emotiv Properties [7]

9 viii LIST OF SYMBOLS α Alpha Rhythm β Beta Rhythm γ Gamma Rhythm θ Theta Rhythm δ Delta Rhythm µ Mu Rhythm µv Microvolts Hz Hertz

10 ix LIST OF ACRONYMS/ABBREVIATIONS A3 ANN BCI BMI CNS D1 D2 D3 db10 DC ECoG EEG EKG ERP FFT fmri IP MEG MMI NIRS SDK TCP Discrete Wavelet Transform Level 3 Approximation Artificial Neural Network Brain Computer Interface Brain Machine Interface Central Nervous System Discrete Wavelet Transform Level 1 Detail Discrete Wavelet Transform Level 2 Detail Discrete Wavelet Transform Level 3 Detail Daubechies 10th Order Wavelet Direct Current Electrocorticography Electroencephalography Electrocardiography Event-Related Potentials Fast Fourier Transform Functional Magnetic Resonance Imaging Internet Protocol Magnetoencephalography Mind Machine Interface Near Infrared Spectroscopy Software Development Kit Transmission Control Protocol

11 1 1. INTRODUCTION We are living in an almost fully-computerized world where the limits of the technological advancements cannot be foreseen. Things that we have read from the sciencefiction books or watched as illusions on TV shows are becoming the reality itself. Maybe the most important enhancements are achieved in the field of biomedical science. With the help of signal acquisition and processing, it became easier to detect any disease and interfere to that before it gets serious. Beside helping to detect the diseases, these technological advancements also help the parallelized people while interacting with the society. Brain Computer Interface (BCI) systems are the communication tools between the brain and external devices. They mostly help communicating with a robotic arm, a wheel-chair, a mouse emulator etc. In order to get the relevant signals from the brain, several recording techniques can be used such as Electroencephalography (EEG), Magnetoencephalography (MEG) or Functional Magnetic Resonance Imaging (fmri). In this thesis, an application which helps the user to drive a toy car by processing the signals obtained from EEG reading is introduced. There are two experiments held. The Emotiv EPOC s SDK is used for giving the commands to the car by cognitive thoughts and then by facial expressions. Before going into the details of BCIs, Emotiv EPOC and the application let me briefly give information about the brain itself.

12 2 2. UNDERSTANDING THE BRAIN The nervous system or the nervous network of an animal is the organ system which allows the animals perceive the inner and outer environment through senses, gets information from outside world and processes them, transmits the signals to different parts of body with the help of cell network and also regulates the organ and muscle activities. All the multi-celled animals except the sponges do have this system. This system is basically divided into two: central nervous system (CNS) and the peripheral nervous system (PNS) [8] The central nervous system plays the role of the manager and the supervisor. It consists of the brain and the spinal cord or in more general words it consists of all the neural organs within the scalp and the spinal cord whereas the peripheral nervous system, which supplies the neural transmission between the CNS and the body organs, consists of all other nerves and neurons that do not lie within the central nervous system Structure of the Brain Brain is the biggest neural organ in the scalp and it is the center of the CNS. The cerebral hemispheres situated above the other brain structures form the largest part of the human brain. This part is also known as the large brain, fore-brain or the cerebrum. There lies the brain-stem underneath the cerebrum. It is the connection part between the large brain and the spinal cord and it is responsible for the automatic systems such as respiration, blood circulation, evacuation and digestion system. At the rear of the brain, behind the brain-stem and beneath the cerebrum, there

13 3 is the cerebellum which plays an important role in motor control. Cerebellum controls the harmonic movement between the arms and legs and the stability of the posture during active movements in addition to some cognitive functions such as attention and language as well. Besides, it also deals with regulating fear and pleasure responses [9] The Hemispheres The cerebral cortex is almost symmetrical as the left and right hemispheres are approximately mirror images of each other [10] This division is done by one of the major folds; Medial Longitudinal Fissure. Each hemisphere interacts with one half of the body in a reversed manner (i.e. right side of the brain interacts with the left side of the body and vice versa). The both hemispheres play important role in higher cognitive actions. The left brain is said to be the logical, mathematical, analytic reasoning, language and verbal materials center whereas the right brain is functioning in analysing the scale, volume and shape of the objects, processing the information with creativity; it is more related to artistic behaviours as shown in Figure The Lobes The human brain has four lobes; frontal lobe, parietal lobe, occipital lobe and temporal lobe that are named after the bones of the skull that overlie them; the frontal bone, the parietal bone, the occipital bone and the temporal bone. The visual processing center of the brain is located in the occipital lobe whereas the auditory perception center is located in the temporal lobe. Determining the location and the shape of the objects in space is done in parietal lobe with processing the emotions. The most important role in planned movements is located in the frontal lobe which is the primary motor cortex. The frontal lobe is also said to be the origin of complex cognitive behaviours and decision making.

14 4 Figure 2.1. Left Brain vs. Right Brain [1] Figure 2.2. Human Brain [2]

15 5 3. SCANNING THE BRAIN 3.1. Introduction to Brain-Computer Interfaces The expression Brain-computer Interface (BCI) was firstly appeared in the papers published after the research done in University of California Los Angeles in the 1970s. They called these direct communication pathway between the brain and external devices as Brain Computer Interfaces. BCIs are also named as mind-machine interfaces (MMIs), brain-machine interfaces (BMIs) and direct neural interfaces in following papers. The goal of BCI research is to create an open loop system that responds to users thoughts and a closed loop system that gives feedback to the users. The BCI systems can be decomposed into 4 main parts: signal acquisition, feature extraction, pattern classification and output application and feedback. Figure 3.1. BCI Design Pattern Invasive BCIs Invasive BCIs measure the neural activity of the brain either on the cortical surface or intra-cortically from within the cortex. The research in this kind of BCIs

16 6 has mainly focused on repairing the damaged sight and providing new functionality for paralysed people. Neurosurgeons implant them directly into the grey matter of the brain. Being that close to the brain and having direct communication give advantage to invasive BCIs over the non-invasive BCIs as they give the brain activity map more accurately. On the other hand, as it involves placing the electrodes on the brain through a neurosurgery operation, it has a huge risk to damage the brain tissue and therefore is more problematic. There are partially invasive BCI devices which are implanted inside the skull but outside the brain rather than within the grey matter. They again produce better resolution signals than non-invasive BCIs and they are safer than fully-invasive BCIs as they have a lower risk of forming scar-tissue in the brain. As an example to this kind of BCIs, Electrocorticography (ECoG) gets the signals from electrodes placed on the surface of the brain. It measures the electrical activity of the cerebral cortex with a high spatial resolution of (approximately 1cm) and a wide frequency range (0-200Hz). It requires less training sessions when compared to EEG. As indicated above, it also has lower clinical risk than fully-invasive BCIs Non-Invasive BCIs Non-invasive BCIs measure the neural activity of the brain over the scalp without any direct contact to the brain. As mentioned above, it is safer than invasive BCIs with a disadvantage of providing worse results Electroencephalography (EEG). EEG records the electrical activity of the neurons. It has a bandwidth of 0-40Hz with a worse spatial resolution than ECoG. Being cheap makes them preferable for individual use.

17 Magnetoencephalography (MEG). MEG records the magnetic field produced by the electrical activity of the brain. It is pretty expensive to build a MEG scan Functional Magnetic Resonance Imaging (fmri). fmri is a specialized MRI scan that measures the blood flow changes related to the brain activity. It is again very expensive Near Infrared Spectroscopy (NIRS). NIRS measures the oxygenated to deoxygenated hemoglobin ratio in the brain. The active parts of the brain are expected to have an increase in this ration. They are cheap and portable, which make them useful in home usage.

18 8 4. EEG The idea of using EEG waves as input to BCIs has existed since the initial conception of BCIs. However, the works with BCIs having the EEG input have recently begun. [11]. Most of these EEG-BCI systems have more or less the same paradigm of reading and analyzing the EEG data, translating the obtained data into device output and meanwhile give some feedback to the user. The main difficulty in creating this kind of system is that the feature extraction and classification of the EEG data is really hard in real-time [12] Introduction to EEG Electroencephalography (EEG) is the electrical activity recording along the scalp and it basically reflects the brain activity. The principle behind the EEG measurement is to calculate the potential difference between two electrodes. EEG waves are found to be created by the firing neurons in the brain. In 1875, a British scientist Richard Caton used a galvanometer and placed two electrodes over the scalp of a human and recorded the brain activity in the form of electrical signals for the first time. Over the years, the concepts of electro-, encephalo- and graphy were combined so that the term EEG was used to denote electrical neural activity of the brain where electro- refers to the registration of brain electrical activities; encephalo- refers to emitting the signals from the head and graphy refers to drawing/writing [13]. The discovery that the human cerebral can be electrically stimulated was done by two scientists Fritsch ( ) and Hitzig ( ). In 1877, Danilevsky investigated the brain activity following electrical stimulation in addition to spontaneous electrical activity in the brain of animals. Beside other names who have worked in this area, German physiologist and psy-

19 9 chiatrist Hans Berger ( ) is said to be the discoverer of the existence of human EEG signals in He began his studies in 1920 with expanding the previously work done by Caton and others and he became the inventor of the EEG which was described as one of the most surprising, remarkable and momentous developments in the history of clinical neurology [14]. Thanks to these scientists, the acquired EEG signals from a human or from other animals that have similar nervous systems as humans may be used for investigation of the following clinical problems: [13] monitoring alertness, coma and brain death monitoring cognitive engagement (alpha rhythm) locating areas of damage following head injury testing afferent pathways (by evoked potentials) monitoring brain development testing drugs for convulsive effects testing epilepsy drug effects assisting in experimental cortical excision of epileptic focus investigating sleep disorders and physiology investigating mental disorders providing hybrid data recording system together with other imaging modalities producing biofeedback controlling anaesthesia depth (servo anaesthesia) investigating epilepsy and locating seizure origin 4.2. Signal Types Our brains produce different kinds of waves depending on our mood, relaxation state or mental activities. In healthy adults, the amplitudes and frequencies of such signals change from one state of a human to another, such as wakefulness and sleep. The characteristic of the waves also change with age, meaning that the EEG reading from a new-born baby is different from the adult EEG.

20 10 The EEG signal itself is rather a complex signal, which can be decomposed into 5 different bands. These bands can be distinguished by their frequency ranges. The corresponding frequency intervals are given in Table 4.1. The alpha and beta waves were introduced by Berger in Gamma was used by Jasper and Andrews in Walter introduced delta in 1936 and the notion theta wave was also introduced by him with Dovey in Table 4.1. EEG Signals and Frequency Intervals Signal Delta Theta Alpha Beta Gamma Frequency Range 0-4 Hz 4-8 Hz 8-13 Hz Hz > 30 Hz Delta Waves These waves are in the frequency range up to 4Hz and they have the highest amplitude among all the wave patterns. They are also the slowest brainwaves that are usually associated with the deepest stages of sleep, as known as the slow-wave sleep. They may also be present in waking state. Because of the fact that the muscles are near to the surface of the skin and produce large signals, it is very easy to confuse artifact signals with the genuine delta response. Delta rhythm is decreased with age. Hence, it is not normal to observe them in awake-healthy people Theta Waves Theta is the frequency range from 4Hz to 8Hz. These signals are normally seen in young children therefore excess theta for age represents abnormal activity. These signals are associated with access to unconscious materials, creative thinking, relaxation

21 11 Figure 4.1. Delta Waves [3] and deep mediation. Figure 4.2. Theta Waves [3] Alpha Waves Alpha is the frequency range from 8Hz to 13Hz. They appear on the posterior part of the head and found over the occipital area of the brain. They are best observed when the eyes are closed and the individual is in mentally relaxed state. These waves are useful to trace mental effort because of their high amplitude. Figure 4.3. Alpha Waves [3] Beta Waves Beta is the frequency range from 13Hz to 30Hz. Beta activity is closely linked to motor behavior and is generally attenuated during active movements. They can be

22 12 measured from the frontal and central regions of the brain. A beta wave is the usual waking rhythm of the brain that is found in normal adults. It is associated with active thinking, active attention, focus on the outside world or solving concrete problems. Figure 4.4. Beta Waves [3] Gamma Waves Gamma is the frequency range approximately Hz with very low amplitude. These waves are sometimes called fast beta rhythm. They are rarely observable and used to detect high cognitive activities or motor functions. Figure 4.5. Gamma Waves [3] Mu Rhythm Mu rhythm is the frequency range between 8Hz and 13Hz; it partly overlaps with other frequencies. This rhythm is strongly connected to motor activities. Being very anti-symmetric, it is easy to detect. C3 and C4 electrodes in International system are used to detect this rhythm.

23 Figure 4.6. Waves of the Brain [4] 13

24 Signal Acquisition Recording Techniques An EEG voltage signal represents the difference between the voltages at two electrodes. EEG measurements are mostly done by Bipolar Montage or Referential Montage where the word montage refers to the EEG channel representation. In Bipolar Montage, each channel represents the difference between two adjacent electrodes whereas in Referential Montage, each channel represents the difference between a certain electrode and a designated reference electrode. Their placement is important in order not to be affected by the brain or any other muscular activity that causes electrical impulses Sensor Layout During the spontaneous EEG recording, the internationally standardized system is usually employed. In this system, there are 21 electrodes located on the surface of the scalp as shown in Figure 4.7. The position determination is as follows: Reference points are nasion and inion, which are the delve at the top of the nose, level with the eyes and the bony lump at the base of the skull on the midline at the back of the head respectively. Taking this points, the perimeters of the skull are measured in the transverse and median planes. The locations of the electrodes are determined by dividing these perimeters into 10% and 20% intervals. Three other electrodes are placed on each side equidistant from the neighbouring points as shown in Figure 4.7. [5, 15]. The special letters A, F, Fp, T, C, P, Pg and O in front of these numbers stand for Ear, Frontal, Frontal Polar, Temporal, Central, Parietal, Nasopharyngeal and Occipital lobes respectively. The part C of Figure 4.7 shows the intermediate 10% electrode positions that are used in addition to the 21 electrodes of the international system. The American Electroencephalographic Society standardized the names and the locations of the

25 15 electrodes. Figure 4.7. The international system seen from (A) left and (B) above the head. (C) Location and nomenclature of the intermediate 10% electrodes, as standardized by the American Electroencephalographic Society [5] Artifacts The signal that is read from the scalp has to be filtered in order to get rid of the artifacts. These artifacts can be caused by several reasons such as the impedance of the system, the ground loop or biological artifacts such as Electromyography (EMG) eye movements and blinks- and Electrocardiography (EKG) signals. Low pass and high pass filters are used in order to remove the EMG and EKG signals. Typical settings for the low-pass filter and high-pass filter are 35-70Hz and 0.5-1Hz respectively. Moreover, an additional notch filter (60Hz for USA and 50Hz for other countries) is used to remove the artifact caused by the electrical power lines [16, 17].

26 Feature Extraction Raw EEG signals are full of noise and therefore they need to be preprocessed in order to eliminate or improve bad quality signals. After getting filtered by the artifacts, the signal is reduced to a small subset (i.e. the feature set) that represents the vital information. Feature extraction is separating useful EEG data from noise and simplifying that data in order to prepare it for the classification process. As mentioned in previous sections, brainwaves can be decomposed into 5 subbands according to their frequencies. Very low and very high frequency modulations usually contain a lot of noise and need to be filtered out. (Emotiv EPOC filters out the less relevant frequencies and leaves the range of Hz). The possible feature extraction from EEG data options are wavelet transformations, Fourier transformations, common average reference, principle component analysis, independent component analysis etc Classification In order to classify the EEG data, there are several techniques can be used. It can be done on frequency or time domain. Neural networks, support vector machines, discriminant analysis, hidden Markov models and other pattern recognition algorithms can be used for classification. The algorithm to be used depends on the need Output Application and Feedback Typical output applications of BCI systems are cursor control, robotic arm control and wheelchair control.

27 17 5. PRELIMINARY WORK Before getting our hands into Emotiv EPOC, we examined the EEG signal processing with pre-recorded data supplied by BCI Competition II. [18] We used the dataset III which is for the motor imagery task. The dataset is recorded from a 25-year-old, female subject. She was asked to relax and imagine left or right hand movements according to the cue shown on the monitor. The data are gathered from the electrodes C3, C4 and Cz in order to detect the imagery task (see Mu rhythm section). This data is saved in Matlab format. It supplies the train data of 140 trials which are 9 seconds each with relevant output labeling. The cue showing the direction is presented from t=3s to 9s. Therefore only the period between the 4th second to 9th second is considered. Each one of these trials contains 3 electrodes information. The output labeling is 1 for left and 2 for right. We are also supplied the test data which contains another 140 trials that is to be labeled with the train input & output. The first thing to do is extracting the relevant features. In this work, we use db10 level 4 discrete wavelet transform regarding to a previous master thesis [19] for feature extraction. The Discrete Wavelet Transform provides an efficient wavelet representation that can be implemented with a simple recursive filter scheme and the original signal can be obtained by an inverse filter. [20] Figure 5.1 shows us the procedure of multi-resolution decomposition of a signal x[n] where h[n] is the high pass filter and g[n] is the low pass filter. By using this transform, the EEG signal is decomposed into the frequency subbands. Matlab helps us with calculating this by wavefun2 function. According to the

28 18 Figure 5.1. Decomposition of DWT; A3 level filter bank [6] motor imagery EEG signals, db10 level 4 discrete wavelet transformation is selected in order to extract β and µ rhythms. When the sampling rate is 128Hz, the levels of this decomposition look like in Table 5.1. Table 5.1. Signal Decomposition Decomposed Signal Frequency Range Level D D (β) 2 D (µ) 3 A We work with the statistical data of the sub-bands D2 and D3. These statistical features are selected to be the standard deviation of coefficients in the sub-band mean of absolute values of the coefficients in the sub-band maximum value of the coefficients in the sub-band the mean absolute deviation of the coefficients in the sub-band. Therefore each of channel C3 and C4 has 2 levels * 4 features = 8 statistical features. The classifier is fed by these inputs.

29 19 After the feature extraction, we use two-layer, feed-forward, back propagation neutral network for the classification of the data. The 16 statistical features obtained in the previous step for each trial were used as inputs to the neural network. The network is trained with 100 trials of the test data and tested for the rest 40 trials. 20 neurons are used in the hidden layer with tansig function and the output function is logsig. With the help of Matlab, we simulate our system as in Figure 5.2. NEURAL NETWORK OPERATIONS: net = newff(minmax(input ), [20,1], tansig, logsig ); net = init(net); net.trainparam.epochs = 1000; net.trainparam.goal =.1; net = train(net,input, y 100 ); SIMULATION: Y = sim(net,test input ); X = sim(net,input ); Figure 5.2. Two-layer, Feed-forward, Back Propagation Neutral Network. With this configuration, we obtain the true classification percentage between 65% and 78%. Playing with the numbers would lead us to better results.

30 20 For low-cost research and consumer markets, several companies have introduced EEG devices to the market. ModularEEG has been released in 2004 by OpenEEG as open source hardware that is compatible with open source software with a balancing ball game [21]. The first affordable consumer-based EEG has been released by NeuroSky in 2007 with the game NeuroBoy. It was also the first large scale EEG device that uses dry sensor technology. They also released an EEG device MindWave, a Guiness Book of World Record award winner for being the Heaviest machine moved using a BCI for educational purposes in 2011 [22 24]. The Final Fantasy, Mattel, Uncle Milton Industries have separately partnered with NeuroSky in order to create EEG-based games Judecca, Mindflex, Star Wars Force Trainer in 2008, 2009 and 2009 respectively [22]. Emotiv has released a 14 channel EEG device EPOC, which is the first commercial BCI that does not use dry sensor technology but rather requires saline solution to be applied to the headset sponges [25]. During this project, we used Emotiv EPOC.

31 21 6. EMOTIV EPOC Emotiv EPOC is an EEG headset that is actually designed for gaming on Windows PCs. It is based on EEG and BCI principles. The producers claim that this headset will make the users control the games with their minds and the keyboards/joysticks will soon after leave their places to this technology. The EPOC has 14 electrodes and two-axis gyro sensors for measuring head rotation. The electrodes are saline-based. They are placed according to the American Electroencephalographic Society s System that is presented in EEG chapter. In this system, odd numbers indicate the left hemisphere, even numbers indicate the right hemisphere and the letter Z indicates the placement on the center. Emotiv s sensor layout is shown in Figure 6.1. As it can be observed, the sensors are mostly located on the frontal cortex. Figure 6.1. Emotiv EPOC Sensor Locations The Emotiv communicates with PC via an ultra-low power Bluetooth interface. It sends a stream of encrypted data wirelessly to a Windows-based machine. The wireless chip is unique to the headset and operates in the frequency range 2,4GHz; meaning that the receiver is not a standard Bluetooth device, we cannot simply use a regular Bluetooth device but rather use the provided Emotiv dongle for the data

32 22 Table 6.1. Emotiv Properties [7] Number of channels 14 (plus CMS/DRL references, P3/P4 locations) Channel Names AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, Sampling method Sequential sampling. Single ADC Sampling rate 128 SPS (2048Hz internal) Resolution 16 bits (14 bits effective) 1 LSB = 1,95µV Bandwidth Hz digital notch filters at 50Hz and 60Hz Filtering Built in digital 5th order Sinc filter Dynamic range (input referred) 256mVpp Coupling mode AC coupled Connectivity Proprietary wireless, 2.4GHz band Power LiPoly Battery life (typical) 12 hours Impedance Measurement Contact quality using patented system

33 23 transmission. [26] Using the wireless technology, having long life batteries and being relatively cheap make this product desirable. As it can be seen in Table 6.1, the internal sampling rate of the EPOC headset is 2048Hz. This is filtered to remove artifacts with a 5th order Sinc filter to notch 50Hz and 60Hz and then down sampled to 128 sec/channel. This means that the headset samples all channels at 128Hz, each sample being a 14-bit value corresponding to the voltage of a single electrode. Overall effective bandwidth becomes Hz. [27] Emotiv offers six different software development kits (SDKs) and depending on the SDK, things that you can do differ. We use the Education Edition during this project which comes with detection libraries and Control Panel, EmoComposer (an emulator for simulating EEG signals), EmoKey (a tool for mapping various events detected by the headset into keystrokes, TestBench and raw EEG data API Control Panel Control Panel is a graphical user interface which allows us to observe and control the communication with the headset. It establishes the connection with the headset, pre-processes and classifies the input data signals gathered from it, gives feedback to the user, allows users to create their own profiles and train their thoughts & actions. Control Panel has 4 suits; Expressiv, Affectiv, Cognitiv and Mouse Emulator. The Mouse Emulator uses the gyroscope information and allows the user to control the mouse with head movements. Affectiv suite is designed to monitor the user s emotional states: engagement/boredom, frustration, meditation, instantaneous and long term excitement. Expressiv suite is designed to measure facial expressions based on EEG and EMG reading. It can recognize 12 actions: blink, left-right wink, left-right look, raise brow,

34 24 Figure 6.2. Connecting the Headset. Green, yellow and orange imply good, fair, poor signal respectively furrow brow, smile, laugh, clench, smirk left-right. Cognitiv suite is designed to monitor the user s conscious thoughts. These thoughts can be one of six directional movements: push, pull, lift, drop, move left-right or one of six rotations: rotate left-right-clockwise-counterclockwise-forward-backward or disappear. These are the built-in prototype action thoughts. Emotiv has been trained with prior tests and the data is classified by a neural network behind the Emotiv software. In order to be user-specific, the action thoughts must be trained via the user interface. The feedback is given as a virtual 3D floating cube. Cognitiv suite allows the user to choose up to four actions that can be recognized at any given time EmoKey EmoKey is the tool for mapping the output of the Control Panel to keyboard inputs. It can be used in various applications such as instant messengers, game controls etc. All you need to do is assigning the related emotion/expression/cognitive

35 25 action by their magnitude information into keys to be entered during the application. One obvious example usage is typing in instant messenger when you smile with a magnitude equal to/greater than a value that is normalized between 0-1. We use this property of Emotiv during the car racing game which will be given in Application section TestBench TestBench program provides a real-time display of the Emotiv headset data stream. It allows recording, saving and replaying the data in European Data Format (.edf) and this can be convertible to Comma Separated Value format (.csv). This converted data can be opened via EEGLAB toolbox of Matlab for later use. Moreover, the program can display a FFT of any incoming channel and can also display the five sub-bands (delta, theta, alpha, beta and gamma) as well as a user-defined custom band.

36 26 7. APPLICATION Figure 7.1. Application Interaction with Emotiv EPOC and EmoEngine 7.1. Toy Vehicle The toy vehicle is a 1/10 scaled Monster Truck, named as Tamiya Blackfoot Extreme. [28] It has two motors; one DC motor and one steering servo motor. Two batteries are used; one for the motors and one for the other electronic equipment. Both motors are driven by the AVR Atmel ATMEGA16 microcontroller. FitPC, an onboard computer, controls this microcontroller. This computer is also capable of receiving commands from an off-the-board client via TCP/IP network connection that is set up over Bluetooth and driving the motors of the vehicle. The car can receive commands from any controlling device such as keyboard, Joystick, mouse etc. This vehicle is implemented by F. Serhan Daniş for the MSc. project Development of a Multi-Sensored Autonomous Ground Vehicle under the supervision of H. Levent Akın. [29] For this project, we slightly modified the client side in order to use the Emotiv EPOC wireless EEG as the controller.

37 Program Using Emotiv Control Panel Introduction to Emotiv API and Emotiv EmoEngine The Emotiv API is exposed as an ANSI C interface that is declared in 3 header files (edk.h, EmoStateDLL.h, edkerrorcode.h) and implemented in 2 Windows DLLs (edk.ḋll and edk utils.ḋll). In order to use the Emotiv API in programs implemented in C or C++, the developer should simply include edk.h and link with edk.ḋll. [7] The Emotiv EmoEngine refers to the logical abstraction of the functionality provided in edk.dll. It manages user-specific or application specific settings, post-processes the data and translates the detection results into EmoStates which contains the current state of the Emotiv detections, which reflects the user s facial, emotional and cognitive state beside communicating with the headset and receiving pre-processed EEG and gyroscope data. [7] There are three main categories of EmoEngine events that the applications should handle; hardware-related events such as user connection or disconnection with Emotiv input devices to the computer (i.e: EE UserAdded), new EmoState events such as the changes in user s facial/cognitive/emotional state (i.e: EE EmoStateUpdated) and suite-specific events such as the training and the configuring Cognitiv and Expressiv detection suites (i.e: EE CognitivEvent). The complete list of EmoEngine events can be found in the user manual. [7] Program written in C++ As it can be seen in Figue 7.2, in the beginning of the program, we try to connect to Emotiv. If we can connect without any problem, a socket connection is established with the ToyCar with static IP number and port number The first thing to be sent to the car is Mx01 in order to activate the manual control when the user is added to the system. The idea is as follows: whenever a pull/push/left/right cognitive activity occurs, the program sends forward, backward, turn left and turn right commands to

38 28 the ToyCar via socket connection respectively. Figure 7.2. Flowchart of the System As the program requires the user s data for cognitive events, we have to train the system beforehand. However, this procedure is not as easy as one might think. The Emotiv has its own classification algorithm behind the program, which is said to be an ANN. In order to train for the basic cognitive actions that Emotiv provides, namely pull, push, left, right, you have to think moving the floating cube with relevant thoughts with a clear mind. It can get affected easily as mentioned in previous sections. While training the system, you can keep the track of the progress via the Control Panel. Due to the fact that one cannot use the recorded training of the morning session in the evening experiment, it takes a huge time and effort to obtain the training just before the experiment. In order to increase the accuracy of the data, we tried to train

39 29 the system as if we are moving a joystick as shown in Figure 7.3. The joystick is not connected to the computer; it is just used for triggering the motor activity. Figure 7.3. Training with unplugged Joystick However, this trick did not affect the system as we thought it would do. The system still required a lot of effort and time in order to be trained. On the other hand, the Expressiv Suit of Emotiv can be more successfully trained and detect facial expressions when compared to Cognitiv Suit. Therefore, we also worked on sending commands through mimics. Among the expressions given in the previous section, we used raise eyebrow for triggering forward movement with furrow eyebrow for disabling the forward movement, smirk left and smirk right for left turn and right turn respectively as shown in Figure 7.4. By controlling the strength of the expression, we gave more accurate commands to the ToyCar. Again the system must be well-trained but it is less painful than training the cognitive module. The videos related to this application is supplied within the DVD and also uploaded to youtube channel [30].

40 30 Figure 7.4. Expressions with relevant commands. Raise eyebrow for GO, Furrow eyebrow for STOP, Smirk left for TURN LEFT and Smirk right for TURN RIGHT Racing Car Trial with EmoKey For setting the correct power limit for the facial expressions while converting them into commands, we worked with EmoKey tool of Emotiv. As mentioned in the previous section, EmoKey converts the classification output of the Control Panel to keystrokes. By converting the Raise Eyebrow to W, Furrow Eyebrow to S, Smirk Left to A and Smirk Right to D, we can control any first-person-game such as a Racing Car game. In order to test our desired mimic strength, we used the game that is built in Unity game engine. [31] By working on this, we obtained the best power values of the mimics to be set to avoid fluctuations in turning left/right commands.

41 31 8. CONCLUSION In this project, we implemented applications using Emotiv EPOC headset to process and classify the brain signals. Emotiv s built-in pre-processors and classifiers are used in this purpose. We first worked on offline data that is supplied by the BCI Competition II organizators to warm up. We used Wavelet Transformation for feature extraction and Artificial Neural Network for classifying the left and right motor imagery hand movements. The features to be selected can be in time or frequency domain. The classification algorithm depends on the application. Unfortunately, we cannot state a generic feature extraction and classification algorithm as the EEG signals are complex and their patterns are all different for each individual. After starting working with Emotiv, we discovered that the signals are too noisy and the training results for the cognitive actions are not very accurate. In order to obtain better results, we used the Expressiv suit of the Emotiv API. By using this, the training sessions became shorter and the classification became better when compared to Cognitiv suit. The real-time applications need to have signal [acquisition + feature extraction + classification] be in the same cycle rate as the application in order not to miss any important data. This is one of the main problems in real-time applications. For the future work, the raw data that is obtained directly from the headset can be processed, different feature extraction and classification methods can be applied on the data to get better results.

42 32 REFERENCES 1. Kerr, B., Why left brain/ right brain Theories won t Go Away,, 2012, billkerr2.blogspot.com/2012/03/why-left-brain-right-brain-theories. html [Online; accessed 01-June-2012]. 3. Wikipedia, Electroencephalography,, 2005, Electroencephalography, [Online; accessed 01-June-2012]. 4. How to Make a Brain Machine, the-brain-machine/, [Online; accessed 01-June-2012]. 5. Malmivuo, J. and R. Plonsey, Bioelectromagnetism - Principles and Applications of Bioelectric and Biomagnetic Fields, chap. 13, Oxford University Press, 1995, 6. Wikipedia, Discrete Wavelet Transform,, 2012, wiki/discrete_wavelet_transform, [Online; accessed 01-June-2012]. 7. Emotiv EPOC, Emotiv Software Development Kit User Manual, edn., Nervous System, system.htm, [Online; accessed 01-June-2012]. 9. Wolf, U., M. J. Rapoport and T. A. Schweizer, Evaluating the affective component of the cerebellar cognitive affective syndrome, The Journal of Neuropsychiatry and Clinical Neurosciences, Vol. 21, pp , Wikipedia, Human Brain,, 2012,

43 33 brain, [Online; accessed 01-June-2012]. 11. Vidal, J. J., Toward Direct Brain-Computer Communication, Brain Research Institute - UCLA, Szafir, D. J., Non-Invasive BCI through EEG, Tech. rep., Boston College CS Department, Sanei, S. and J. Chambers, EEG Signal Processing, John Wiley & Sons, Inc., NY, USA, Annual Meeting of ISHN, 7th, The Origins of EEG, Malmivuo, J. and R. Plonsey, Bioelectromagnetism - Principles and Applications of Bioelectric and Biomagnetic Fields, chap. 13, Oxford University Press, 1995, Niedermeyer, E. and F. da Silva, Electroencephalography: Basic Principles, Clinical Applications, and Related Fields, Lippincot Williams & Wilkins, Drogdelen, W. V., Signal Processing for Neuroscientists, Acedemic Press, BCI, BCI Competition II, [Online; accessed 01-June-2012]. 19. Gunaydin, O., Design of a Brain Computer Interface System Based on Electroencephalogram, Master s Thesis, Bogazici University, Xu, B. and A. Song, Pattern Recognition of Motor Imagery EEG Using Wavelet Transform, Biomedical Science and Engineering, Vol. 1, The ModularEEG, html, [Online; accessed 01-June-2012].

44 NeuroSky Products, Products, [Online; accessed 01-June-2012]. 23. NeuroSky Store, the-adventures-of-neuroboy-bci-technology-demo, [Online; accessed 01-June-2012]. 24. Mindwave, [Online; accessed 01-June-2012]. 25. Emotiv EPOC, [Online; accessed 01-June-2012]. 26. NeuroPhone: Brain-Mobile Phone Interface using Wireless EEG Headset, Moro, G., BCI for Alternative Input,, 2011, bci.html#x INC., T., Blackfoot Xtreme,, 2003, [Online; accessed 01-June-2012]. 29. Daniş, F. S., Development of a Multi-Sensored Autonomous Ground Vehicle, Master s Thesis, Boğaziçi University, UNITY, Unity Car Tutorial, tutorials/car-tutorial, [Online; accessed 01-June-2012]. 32. Varol, O., Raw EEG Data Classification And Applications Using SVM, Tech. rep., Istanbul Technical University, Anderson, C. W., S. V. Devulapalli and E. A. Stolz, Determining Mental State from EEG Signals Using Parallel Implementations of Neural Networks, Scientific Programming, Vol. 4, pp , 1995.

45 Rodrigues, S. M. S., Brain-Controlled Robots: Context-Dependant Brainactivity Interpretation., Master s Thesis, University of Luxembourg, Wolpaw, J. R., N.Birbaumer, D. J. McFarland, G. Pfurtscheller and T. M. Vaughan, Brain Computer Interfaces for Communication and Control, Clinical Neurophysiology, Vol. 113, pp , 2002.