Master Thesis Proposal: Chess Brain-Computer Interface Design and Optimization for Low-Bandwidth and Errors

Transcription

1 Master Thesis Proposal: Chess Brain-Computer Interface Design and Optimization for Low-Bandwidth and Errors Samuel A. Inverso Computer Science Department College of Computing and Information Sciences Rochester Institute of Technology Rochester, NY Chair: Jessica D. Bayliss, Ph.D. Reader: Mike Van Wie, Ph.D. Observer: Roger Gaborski, Ph.D. March 6, Summary Researchers at the Institute of Medical Psychology and Behavioral Neurobiology (IMPBN) at the University of Tübingen in Tübingen, Germany study cortical and behavioral plasticity in the central nervous system pertaining to the neuronal and psychological basis of learning. In addition, through their research, behavioral treatments for disabilities are developed and verified in clinical trials [6]. IMPBN developed a brain computer interface (BCI) to enable completely paralyzed, or locked-in, patients the ability to communicate using slow cortical potentials. Currently this system is used for communication between patients care-givers and researchers, however, one patient has requested an interface to play chess, which he currently plays with his nurses using eyebrow movements. The technical resources at IMPBN are not available to create this application. Jessica Bayliss learned of the patient s request and began communication with the IMPBN researchers to determine their interest in having her students create the chess application. The project portion of this thesis includes the BCI chess application for the patient. In addition, this thesis will experiment with bandwidth optimization between the patient and the chess application. Current BCIs are slow, allowing communication of 10 to 25 bits/min [16], however, bandwidth can be increased through choice organization and intelligent 1

2 choice prediction. As part of bandwidth optimization, ideas from the Human Computer Interface field pertaining to Artificial Intelligence in task automation will be investigated. The degree to which the system is automated versus manual input will also be addressed. 2 Background Amyotrophic lateral sclerosis (ALS), multiple sclerosis, cerebral palsy, and spinal cord injury are a few neuromuscular disorders which can completely paralyze individuals but not affect their brains. If restorative treatments are ineffective, the afflicted may still live many years with life support systems and twenty-four hour care, but without the ability to communicate or control their environment through normal means. These individuals are locked-in to their bodies[17]. Depending upon the individuals afflictions, there are three options available. First, they may use remaining voluntary muscle control for communication. For example, some paralyzed people retain control of eye movements which they can use to answer simple questions or control word processing programs with eye tracking. Second, neural pathways may be reconnected around the break to control healthy muscle. Electromyographic (EMG) signals from muscles unaffected by spinal cord injury can control paralyzed muscles. Third, if muscular movement is nonexistent, the locked-in patient may employ a direct brain communication mechanism to communicate and control the environment using a computer [16]. 2.1 Brain Interface Methods In 1973 Jacques Vidal experimented with visual evoked potentials (VEP), recorded across the scalp, to track subjects eye movements. The eye movements were translated into computer cursor movements. In his research, Vidal used the term brain-computer interface (BCI) to describe this and other computer systems that record functional brain information [14]. Over the past three decades this definition has narrowed to a communication system that does not depend on the brain s normal output pathways of peripheral nerves and muscles [17]. There are a variety of devices that can be used for BCI. These devices include electroencephalography (EEG), magnetoencephalography (MEG), positron emission tomography (PET), and functional magnetic resonance (fmri) described in Table 1. 2

3 Table 1: Brain interface methods [4]. Method Electroencephalography (EEG) Magnetoencephalography (MEG) Positron Emission Tomography (PET) Single-Photon Emission Tomography Functional Magnetic Resonance Imaging (fmri) Description Maps general brain activity using scalp electrodes. Measures magnetic fields generated by electrical currents at cell level. The subject ingests radioactive tagged glucose. After the glucose enters the blood stream the PET machine measures the concentrations of glucose which corresponds to the brain s active areas. Similar to PET, but with poorer spatial resolution because it only measures a single photon. Based on Magnetic Resonance Imaging (MRI) technology. fmri can detect oxygen levels in blood to show variations in the subject s neural activity without ingesting radioactive markers. Figure 1 depicts spatial resolutions for different brain imaging techniques. As shown, Computer Tomography (CT) scans are the most fined grained, approximately 0.3 mm to 1 mm. Unfortunately, CT uses X-rays to scan the brain which can damage cells. In addition, it does not provide functional information about the brain s activity. EEG provides a good functional map of the overall brain activity but has poor spatial resolution, ranging from 26.6 mm to 35.3 mm. A neuron s width is between 4 microns (granule cell) to 100 microns (motor neuron) [13]. Figure 2 compares the methods spatial resolution to temporal resolution. MEG provides the best spatial and temporal resolution, however, it requires a roomful of expensive monitoring equipment and the subject cannot move his head during the recording. While EEG spatial resolution is poor, it has very good temporal resolution, which is important when translating rapidly changing brain activity to users desires. EEG signals are widely used in BCI research because they are fast temporally, inexpensive and less cumbersome than other methods (MEG, fmri, SPECT, and PET require a room full of equipment), and non-invasive (the user wears a cap of electrodes) [4]. In addition, EEG signals can easily be combined with other techniques, such as fmri, for BCI methods that require fine grained spatial resolution. 2.2 Patient s Brain-Computer Interface The patient MS at IMPBN uses eye-brow movements and the Thought Translation Device, an EEG Brain-Computer Interface (BCI) developed by Birbaumer 3

4 (a) CT Scan [3] (b) MRI [7] (c) fmri [8] (d) SPECT [3] (e) PET [11] (f) PET/MRI (Multi-Modal) [15] (g) MEG (Superimposed MRI) [4] on (h) EEG [10] Figure 1: Example brain scans depicting spatial resolution. Computer Tomography (CT) is an X-ray imaging technique which does not provide functional information. et. al., to communicate. The Thought Translation Device (TTD) uses patient controlled slow cortical potential shifts for binary selections. Slow cortical potentials (SCP) are slow low frequency changes detected with scalp-recorded EEG that occur over seconds [16]. Figure 3 shows the results of users training to move a cursor to the top (more negative) and bottom (more positive) edge of a computer screen using SCP. In operation, the TTD presents a user with two alphabet halves. The user has a time limit to shift his SCP past a positive or negative threshold to select between the left and right alphabet halves or letter banks. In Birbaumer s experiments the time limit was 4.5 seconds for subject A and 6.0 seconds for subject B. When a letter bank is selected, it is split in half and presented to the user in the same fashion as above until the user selects a final letter. If the user does not select a bank after two successive tries a go back function appears as an option to go back to the previous letter bank. After training, patients with 65-90% spelling accuracy can write letters/min [2, 16]. 4

5 Figure 2: Spatial (mm) vs Temporal (sec) Resolutions for Brain Graphing Methods [4]. Figure 3: SCP of user learning to move a mouse cursor to top and bottom of a computer screen [16]. To improve the TTD s performance Perelmouter and Birbaumer [9] described a method for using a binary Huffman tree. A binary Huffman tree is a binary tree created using the Huffman algorithm: an algorithm to optimize the representation of finite messages in message ensembles based on the probability each message will occur in an ensemble [5]. Perelmouter and Birbaumer created a Huffman tree with probabilities equal to the frequency letters appear in common communication. The tree was used to order selection options presented to 5

6 the user. This ordering placed the most used options near the beginning of the selection choices and the least used options near the end. In addition to normal options, a special go back symbol was added to the tree weighted by the user s error rate. Unfortunately, his paper does not present experimental data on this technique. 3 Thesis This thesis will revolve around a brain-computer interface for the patient MS to play chess. In addition to the chess interface design and creation, I will explore methods to increase the bandwidth between the patient and interface. The methods include choice organization, intelligent choice prediction, and the degree to automate the system versus manual input. Ideas from the Human Computer Interface field using Intelligent Agents will also be researched. MS expressed an interest in iteratively testing the chess interface, which provides a unique opportunity to utilize end-user feedback during program development. However, communication with MS is through his caregivers in Germany, and in order to not overtax him and his caregivers, he will test infrequently. Given MS s inaccessibility, and that the apparatus to perform EEG experiments are not available at the Rochester Institute of Technology, healthy subject experiments will be performed using a simulator. The simulator will use a computer s mouse to simulate the bandwidth of a locked-in patient. By pushing a mouse right or left the subject will simulate positive and negative slow cortical potentials. The cursor will respond at the rate of SCP change. In addition, errors will be simulated by adding noise to the system. The SCP and error rates used will be determined during the course of this thesis. The simulator will be written such that future researchers may use it to test and prototype their BCIs before costly experiments are conducted. 3.1 Bandwidth Optimization Bandwidth optimization is an important concern in creating a BCI choice selection mechanism. In the chess domain the worst interface presents a choice for selecting a piece and placing a piece using all board positions: log 2 (64)+log 2 (64) = 12. A better interface only presents valid moves. For example, only considering valid opening moves reduces the bandwidth to log 2 (10) + log 2 (2) = 4, see Figure 4 for example openings. At 6 seconds per choice in TTD selection, the better interface offers a 48 second savings (worst 72 sec vs better 24 sec) [2]. Weighting the user s options in a Huffman tree is another optimization method. As described previously, Perelmouter [9] presented a method to use a Huffman tree for letter writing. A binary Huffman tree in conjunction with intelligent choice prediction can further optimize the bandwidth. By using AI techniques, such as Markov models, the system can learn the choices the user makes in certain situations and weight those choices to appear earlier in the option tree. 6

7 Figure 4: Example opening moves. Screen capture from Vektor One drawback to dynamically weighting the tree is that a user may find it more difficult to learn the options positions. For example, when using a word processing program the menus almost never change. Over time, and repeated use, a user will memorize the position of a choice in the menu, such as save or open, which increases their bandwidth with the application. Because both user adaption and program adaption are diametrically opposed, a balance must be found between them. Another technique for bandwidth optimization is to utilize the user s ability to make more than binary decisions. The user s interface my detect brain activity other than SCPs. For example, µ-rhythms and β-rhythms decrease during preparation for movement and increase after movement has ended. These rhythms also occur in imagined movement, allowing a BCI to interpret these rhythms in locked-in individuals as option selections. The P3 evoked potential is another signal the BCI selection mechanism may incorporate. The P3 is a peak in parietal cortex activity evoked when a person is presented infrequent or particularly significant auditory, visual, or somatosensory stimuli [16]. Bayliss and Ballard demonstrated P3 s BCI use in a virtual driving environment where the P3 was evoked by red traffic lights [1]. In addition to brain activity, non-brain signals, such as eye movements and residual voluntary muscle movements, may significantly widen the user s bandwidth. An n-ary Huffman tree should be constructed to utilize all of these information channels. In any bandwidth optimization scheme the inherent noise in the system and user errors must be considered. Users often achieve accuracies of 65-90% with BCI s. Without considering error rates bandwidth optimizations can become negligible. To ensure useful optimization a go back or undo feature will be added to the Huffman tree [2, 9] based on the user s error rate. As the user increases his accuracy with the system the weight of the undo feature may dy- 7

8 namically change. As with normal choices the balance between the dynamic undo option placement and the user s adaption to the system will be investigated. 3.2 Chess BCI Design Figure 5 depicts a high level chess brain-computer interface design. The chess application is separated into a Graphical User Interface (GUI), that displays the chess board and any information related to the game. The computer chess engine is the program which plays chess against the patient. This design is based on the majority of chess application designs that are in turn derived from XBoard and GNU/Chess. GNU/Chess is a text based command line chess program where a player issues move commands to the engine through standard input and receives opposing moves through standard output. XBoard is a graphical application that interfaces to GNU/Chess through pipes. XBoard provides a graphical chess board where moves are made using a mouse. GNU/Chess was ported to Windows and with it came the Winboard graphical chess interface which communicates using the text command protocol. Because Winboard is an open interface and Windows is ubiquitous, many chess engine programmers, more interested in strategy and tactics than GUIs, wrote their engines to use the Winboard protocol. There are now roughly 120 chess engines that support the Winboard protocol [12]. As with many de facto standards, there are other competing protocols. The most successful competitor is Universal Chess Interface (UCI), created for the commercial chess engine Shredder. However, there are chess interfaces which support both Winboard and UCI. Most graphical chess interfaces, such as Winboard and Arena, allow two engines to play against each other. This is beneficial as it allows engine authors an easy way to test their engines against other engines. The chess BCI is developed to leverage this ability. As shown in Figure 5 the proxy chess engine will interact with the chess GUI as a normal chess engine, but will proxy the patient s chess moves, communicated via the BCI, to the GUI. 4 Principal Deliverables The principal deliverables for this thesis are a research paper and chess application described below. Research Paper Paper will include research on Brain Computer Interface techniques and applications. Human Computer Interface applications of Artificial Intelligence. 8

9 Windows PC Chess Gui Winboard Protocol Winboard Protocol Proxy Chess Engine Computer Chess Engine Brain Computer Interface Person Figure 5: Chess Brain-Computer Interface High Level Design Experimental results and analysis of bandwidth optimization and automation for the chess application. Chess Application User Manual Design and Maintenance documentation Code If outside chess modules, such as GNU/Chess or WinBoard Chess Interface, are used they will be delivered. However, if the redistribution of a module violates the terms of its license a method to obtain that module will be provided or a comparable module will be delivered. 5 Chess Application Requirements The user requires the chess application to: 1. Play chess against the user at various levels of expertise. 2. Allow the user to play against a human using the same computer. 3. Optionally, allow the user to play against a human across a network, preferably interfacing with an online chess program such as 9

10 4. The program must interface with the user s current Brain Computer Interface. 5. The chess application must execute under Windows and compile under Borland C++ Builder Schedule January 13, 2003 February 3, 2003 February 24, 2003 March 10, 2003 April 7, 2003 May 5, 2003 May 6, 2003 May 21, 2003 Chess application and simulator prototypes completed Simulator completed, Chess application user test Chess application user test Chess application completed Experiments completed Research paper completed Thesis defense preparation begins Thesis defense References [1] J.D. Bayliss and D. H. Ballard. A virtual reality testbed for braincomputer interface research. IEEE Transactions on Rehabilitation Engineering, 8(2): , [2] N. Birbaumer, N. Ghanayim, T. Hinterberger, I. Iversen, B. Kotchoubey, A. Kübler, J. Perelmouter, E. Taub, and H. Flor. A spelling device for the paralysed. Nature, 398(6725): , [3] Ct and spect scans, cited in [4]. research/brains/brains.html, September [4] Simon Fisk. Music mind & matter. ~scf104/neuralmusic/overview.html, October [5] David A. Huffman. A method for the construction of minimum-redundency codes. Proceedings of the I.R.E., 40: , [6] Institute of medical psychology and behavioral neurobiology, university of tübingen, tübingen, germany. index.html, September [7] Keith A. Johnson and J. Alex Becker. Whole brain atlas. med.harvard.edu/aanlib/cases/casem/mr1/035.html, October [8] Introduction to fmri. gif, October

11 [9] J. Perelmouter and N. Birbaumer. A binary spelling interface with random errors. IEEE Transactions on Rehabilitation Engineering, 8(2): , [10] Renato M. E. Sabbatini. Mapping the brain. cm/n03/tecnologia/eeg.htm#topography, September [11] Jim Strommer. Tutorial: Clinical pet - neurology. ucla.edu/software/lpp/clinpetneuro/function.html, October [12] Aaron Tay. Chess engines - cutting through the confusion. October [13] Brain facts and figures. facts.html, November [14] J. J. Vidal. Towards direct brain-computer communication. Annual Review of Biophysics and Bioengineering, 2: , [15] Harm Johannes Wieringa. Meg, eeg and the integration with magnetic resonance images [16] Jonathan R. Wolpaw, Niels Birbaumer, Dennis J. McFarland, Gert Pfurtscheller, and Theresa M. Vaughan. Brain-computer interfaces for communication and control. Clinical Neurophysiology, 113(6): , [17] J.R. Wolpaw, N. Birbaumer, W. J. Heetderks, D. J.McFarland, P. H. Peckham, G. Schalk, E. Donchin, L. A.Quatrano, C. J. Robinson, and T. M. Vaughan. Brain-computer interface technology: a review of the first international meeting. IEEE Transactions on Rehabilitation Engineering, 8(2): ,