Brain Computer Interface for Virtual Reality Control. Christoph Guger

Transcription

1 Brain Computer Interface for Virtual Reality Control Christoph Guger

2 VIENNA Musical Empress Elisabeth Emperor s castle Mozart MOZART g.tec GRAZ

3 Research Projects #) EC project: ReNaChip - Synthetic system integration Rehabilitation of a discrete sensory motor learning function by a #) EC project: Sm4all Smart Home for all Brain-Computer Interface for smart home control #) EC project: RGS Rehabilitation Gaming System faster recovery from stroke with games #) EC project: BrainAble BCI with VR and social networks #) EC project: Decoder BCI for locked in patients #) EC project: CSI - Central Nervous System Imaging #) EC project: BETTER BCI for Stroke rehabilitation and rehabilitation robots #) EC project: VERE Virtual Embodiment Real Embodiment Dissolving the boundary between the human body and surrogate representation in virtual and physical reality. #) EC project: ALIAS Adaptable Ambient Living Assistant Robot system interacting with elderly people providing cognitive assistance and social interaction and inclusion

4 Our co-operations partners University of Barcelona, Spain Mel Slater, Chris Groenegress IDIBAPS, Barcelona, Spain Mavi Sanchez-Vives University College London (UCL), UK Anthony Steed, Angus Antely, University of Technology Graz, Austria Robert Leeb, Gert Pfurtscheller Wadsworth Center, New York, USA Gerwin Schalk, Eric Sellers Tel Aviv University Matti Mintz

5 Brain-Computer-Interface (BCI) Subject/ Patient EEG/ ECoG Brain- Computer Interface control signal Device Feedback A system for controlling a device e.g. computer, wheelchair or a neuroprothesis by human intention which does not depend on the brain s normal output pathways of peripheral nerves and muscles [Wolpaw et al., 2002]. HCI Human Computer Interface DBI Direct Brain Interface (University of Michigan) TTD Thought Translation Device (University of Tübingen)

6 Some examples of BCI applications BCI_ BCI

7 MATLAB and Simulink environment User-System #1 Subject, Patient Real-time system Real-time blockset biosignals Biosignal amplifier Custom feature extraction and classification Custom hardware e.g. orthosis MATLAB/Simulink feedback Stimulation unit DAQ board channel User-System #2 Personal Area Network (PAN) control unit

8 Influencing components adaptation to subject technical issues

9 How to record brain activity for BCI? Functional imaging techniques: FMRI, SPECT, PET Magnetencephalogram (MEG, SQUID) Near Infrared Spectroscopy (NIRS, fnir) Electrocorticogram (ECoG) Electroencephalogram (EEG)

10 Measuring brain electrical activity Electroencephalogram (EEG) 1 64 (128) channels, 1 µv 100 µv, DC up to mv-range, 0 40 Hz, low signal-to-noise ratio, moderate spacial resolution, high temporal resolution Surface electrodes: mm, mounted with conductive gel/paste Electro-corticogram (ECoG) closely spaced multi-electrode grids or strips applied directly to the cortical surface, electrode diameter ~ 4mm, up to 500 µv, Hz high signal-to-noise ratio, high spacial and temporal resolution highly invasive and limited study opportunities modified from University of Michigan

11 Changes of brain electrical activity and mental strategies - Slow cortical potentials (anticipation tasks) DC-derivation, artifact problem, difficult strategy, feedback method - Steady-State Evoked potentials (SSVEP, SSSEP) Flickering light with specific freuqency - Event-related, non-phase-locked changes of oscillatory activity ERD/ERS (motor imagery tasks) Changes of mu-rhythm, alpha activity and beta activity over sensorimotor areas, imagination of hand-,foot-, tongue- movements - Evoked potentials (focus on attention task) Thalamic gating, various methods of stimulation (visual, tactile, electrical, auditory,...), P300

12 Communication for the locked-in ALS patient in Germany using a BCI system for communication Birbaumer, Kübler, Hinterberger, Tübingen

13 Changes of brain electrical activity and mental strategies - Slow cortical potentials (anticipation tasks) DC-derivation, artifact problem, difficult strategy, feedback method - Steady-State Evoked potentials (SSVEP, SSSEP) Flickering light with specific freuqency - Event-related, non-phase-locked changes of oscillatory activity ERD/ERS (motor imagery tasks) Changes of mu-rhythm, alpha activity and beta activity over sensorimotor areas, imagination of hand-,foot-, tongue- movements - Evoked potentials (focus on attention task) Thalamic gating, various methods of stimulation (visual, tactile, electrical, auditory,...), P300

14 Physiological Background why does it work Left hand movement Right hand movement C3 RIGHT GND C4 REF Imagination of hand movement causes an ERD which is used to classify the side of movement. The desynchronization occurs in motor and related areas of the brain. Therefore, for analyzing and classifying ERD-patterns the electrodes must be placed close to sensorimotor areas.

15 C3 The ``Finger Movement Task`` Brisk movement of right index finger

16 Oscillatory Activity Paradigm for a simple motor imagery BCI experiment TRAINING left Recording of 40 trials minimum right beep Fixation cross CUE motor imagery s Offline data classification

17 Oscillatory Activity Paradigm for a simple motor imagery BCI experiment FEEDBACK left feedback (FB) right beep Fixation cross CUE FB s classifier

18 Patient Tom (C4/C5 lesion)

19 Case study: Electrocorticograms, ECoG Direct Brain Interface, rhythmic activity, Albany, USA With permission from Schalk 2007

20 With permission from Schalk 2007

21 Augmentation of neuronal population activity using a braincomputer interface g.tec's Spike & ECoG Workshop November 14 th, 2010 Kai J. Miller Physics, Medicine Neurobiology and Behavior, Neural Systems Lab University of Washington

22 Basic spectral changes with movement ERD

23 Real-time representation of cortical activity

24

25 Active Electrodes Advantages: - No preparation (abrasion) of the skin required - High signal quality and less 50/60 Hz noise with high impedance - Reduced artifacts from electrode and cable movements

26 Active versus passive electrodes Active Passive

27 EYE MOVEMENTS Channels closer to the eyes (1 and 4) show higher EOG artefacts than central and occipital channels. Both passive and active electrodes show a similar EOG contamination which is also clear because both pick up the same source signal.

28 BITING Biting produces EMG contamination almost equally on all channels No difference between active and passive electrodes because both pick up the same source signal.

29 CABLE ARTEFACTS Cable artefacts are produced by touching or shaking the cables. Active electrodes are almost unaffected while the passive electrodes show large movement artefacts.

30 ACTIVE HEAD MOVEMENTS Active head movements produce fewer artefacts with active electrodes compared to passive ones. Artefacts for both electrodes can occur because of skin-electrode movements. Passive electrodes are mostly affected by the cable movements initiated by the head movements.

31 PASSIVE HEAD MOVEMENTS Passive head movements have lower accelerations than active head movements and therefore the artefacts are smaller and mostly visible with passive electrodes.

32 BCI AND DRY ELECTRODES BCI use P300, motor imagery or steady-state visually evoked potentials (SSVEP) measured with the electroencephalogram (EEG) to control external devices. Evoked potentials, event-related desynchronization, power spectrum accuracies were calculated for dry and gel based electrodes to compare them.

33

34

35

36

37 Discussion: First dry electrode system that works for motor imagery, SSVEP and P300 (same accuracies reached for all) Whole frequency range available: Hz First dry electrode system that covers extended 10/20 system on frontal, central, parietal and occipital sites More low frequency components in the EEG spectrum below 3 Hz Careful montage required and more sensitive to surrounding noise Group studies submitted in March 2011 to Journal of Neural Engineering and BCI conference in Graz 2011.

38 Biosignal Analysis and Recording System in VE The recording system has to work in noisy environments CAVE system: creates a 3D Virtual World TRIMENSION ReaCTor 3 back projected screens (3m x 2.2m) 1 floor screen projected by ceiling mounted projector 3D effect with shutter glasses

39 Movie: Walking through a Virtual City by Thought

40 BCI Award 2010

41 Categorization of the 10 nominees Title Control signal Application fmri Spikes N200/ P300 SSVEP MI Stroke Spelling/ internet/ art A high speed word spelling BCI X X system based on code modulated visual evoked potentials Motor imagery-based Brain-Computer X X Interface robotic rehabilitation for stroke An active auditory BCI for intention X X expression in locked-in Brain-actuated Google search by using X X motion onset VEP Brain Painting - "Paint your way out X X Thought Recognition with Semantic X X Output Codes Predictive Spelling with a P300-based X X BCI: Increasing Communication Rate Innovations in P300-based BCI X X Stimulus Presentation Methods Operant conditioning to identify X independent, volitionally-controllable patterns of neural activity Algorithm development X Neurorehabilitation for Chronic-Phase X X Stroke using a Brain-Machine Interface Total

42 Categorization of the 57 submitted projects Property Percentage Property Percentage (N=57) (N=57) Real-time BCI 65.2 Stroke 7.0 Off-line 17.5 Spelling 19.3 algorithms P Wheelchair/rob 7.0 ot SSVEP 8.9 Internet/VR 8.8 Motor imagery 40.4 Control 17.5 EEG 75.4 Platform/Techn ology fmri 3.5 ECoG 3.5 NIRS

43 Stroke

44 Potential Users Worldwide Wadswor t h Cent er New York State Department of Health Cerebral palsy 16,000,000 Brainstem stroke 10,000,000 Other stroke 60,000,000 Spinal cord injury 5,000,000 Postpolio syndrome 7,000,000 Amyotrophic lateral sclerosis 400,000 Multiple sclerosis 2,000,000 Muscular dystrophy 1,000,000 Guillain-Barre syndrome 70,000

45 Brain Computer Interface (BCI) for Stroke Rehabilitation Virtual Reality BCI in combination with VR for stoke rehabilitation People can be motivated via VR to activate their motor cortex Simple bar feedback: Patients should try to increase the length of a virtual bar just by motor imagery tasks Rehabilitation games (e.g. RGS Rehabilitation Gaming System) Patients play games activated by the motor imagery BCI Patients can activate the virtual arm (picture: RGS project)

46 Brain Computer Interface (BCI) for Stroke Rehabilitation Virtual Reality BCI - VR System setup: EEG cap

47 Brain Computer Interface (BCI) for Stroke Rehabilitation Recent Results Motor imagery-based Brain-Computer Interface robotic rehabilitation for stroke (Cuntai Guan et al.)

48 Brain Computer Interface (BCI) for Stroke Rehabilitation Recent Results Study for comparison between BCI guided rehabilitation and non BCI guided rehabilitation (MANUS manual user) 26 patients were recruited, and randomized into two groups (15 in MANUS group, 11 in BCI group). The protocol of the rehabilitation is summarized as follows Patients performed 4 weeks rehabilitation training, 3 sessions per week, and each session lasted around 1 hour Clinical evaluation was done at the beginning of the training (week 0), mid of the training (week 2), end of the training (week 4) and following-up assessment (week 12).

49 Brain Computer Interface (BCI) for Stroke Rehabilitation Recent Results Clinical evaluation measures various outcomes for patients Fugl-Meyer Assessment (FMA) score is reported here as it represents an overall recovery of motor impairment. Patients in MANUS group performed 960 repetitions, while BCI group only performed 160 repetitions.

50 P300 Approach (EEG) Amplitude (µv) visual stimulation Time (ms) A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

51 The 6 x 6 matrix speller, single character flash concentrate on W Individual character intensifies for 60ms with 10ms between each intensification A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

52 The 6 x 6 matrix speller, single character flash A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

53 The 6 x 6 matrix speller, single character flash A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

54 The 6 x 6 matrix speller, single character flash A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

55 The 6 x 6 matrix speller, single character flash A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

56 The 6 x 6 matrix speller, single character flash [µv] Letter W Presentation Target:1 5 µv P300-8 Non-target: 1 µv time [s] Target NON Target

57 Rows- and Columns- Flashing A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

58 Rows- and Columns- Flashing A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

59 Rows- and Columns- Flashing A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

60 Rows- and Columns- Flashing A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

61 Rows- and Columns- Flashing A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

62 Performing real-time BCI experiments Hands on seminar P300 based speller Video

63 BCI goes BCI for social networking Tasks: P300 based BCI twitter interface Definition of the users needs reduced set of full functionality Adaptation of P300 interface

64 Twitter Control

65 Japanese version

66 intendix - g.tec integrates Brain-Computer Interface (BCI) technology into patients everyday life intendix is designed to be installed and operated by caregivers/patient s family at home enables the user to sequentially select characters from a keyboard-like matrix on the screen requires some training but most subjects can use intendix after only 10 minutes a spelling rate of 5 to 10 characters per minute can be achieved by the majority of healthy users the system can trigger an alarm, let the computer speak the written text, print out or copy the text into an or to send commands to external devices put on the cap, inject a drip of gel into each of the electrodes and start to spell! intendix includes the active electrode system, a portable EEG acquisition system and a notebook or netbook computer with the intendix software installed.

67 Live Experiment: Smart Home Control XVR and BCI

68 Requirements To use the P300 potential for smart home control we developed: Virtual Reality apartment in XVR (extreme VR) open source package to build VR The Smart home needed several controllable elements such as TV, music, windows, doors,... Special icons to control all the devices inside the apartment Portable BCI system to be inside the 3 D environment Reliable and fast BCI system

69 Concept

70 Concept Switch on the BCI system Head-tracker Extract the EEG information BCI-System Subject Highly immersive feedback Electrodes and biosignal amplifier Signal processing Send commands VR smart home with controllable elements Translate BCI commands UDP interface

71 The Virtual Reality apartment Designed by Chris Groenegress, Mel Slater

72 The Virtual Reality apartment Designed by Chris Groenegress, Mel Slater

73 The Virtual Reality apartment Designed by Chris Groenegress, Mel Slater

74 Control matrix for smart home with special icons Select music

75 Goto specific position The Beamer

76 Study Design 12 subjects, 8 EEG channels recorded Fz, Cz, P3, Pz, P4, PO7, Oz, PO8 Referenced to right mastoid, grounded to the forehead Data recorded with g.mobilab+ via Bluetooth Fa = 256 Hz, bandpass Hz 1st training run -> 7 icons per mask (7) Application runs -> 23 icons Spelling Device Application Single character flash experiment Total of ~ 2.5 hours incl. electrode montage and instruction of the subject

77 Control the Smart Home by your thoughts In cooperation with the Virtual Environments and Computer Graphics, UPC, Barcelona

78 Results - tested with 8, 4 and only 2 flashes per item - best result: subject 8 with 100 % accuracy for 8 and 4 flashes - worst result: subject 5 with only 30 % for only 2 flashes Best and worst selection

79 Comparison of different masks: accuracy and EPs Best accuracy Worst accuracy 11 µv max 15 µv max 8 µv max -> Accuracy depends on arrangement of characters, background,... and not on number of icons flashing in the matrix

80 Discussion The P300 works very well for spelling This is based on 85 subjects so far! Why does the GoTo perform worse?

81 What makes the difference?

82 Controlling Second Life with the BCI Allows patients to take part in social networks Control avatars just by thinking Patients appear as healthy person

83 Changes of brain electrical activity and mental strategies - Slow cortical potentials (anticipation tasks) DC-derivation, artifact problem, difficult strategy, feedback method - Steady-State Evoked potentials (SSVEP, SSSEP) Flickering light with specific frequency - Event-related, non-phase-locked changes of oscillatory activity ERD/ERS (motor imagery tasks) Changes of mu-rhythm, alpha activity and beta activity over sensorimotor areas, imagination of hand-,foot-, tongue- movements - Evoked potentials (focus on attention task) Thalamic gating, various methods of stimulation (visual, tactile, electrical, auditory,...), P300

84 Steady-State Visual Evoked Potentials (SSVEP) Frequency of stimulation Brain response Hz transient (single) VEP Hz undefined response Hz SSVEP

85 Methodology Visually Evoked Potentials (VEP) single VEP

86 Methodology Steady State Visually Evoked Potentials (SSVEP) SSVEP 7 Hz

87 Steady-State-Evoked Potentials EEG Power Spectrum Higher Frequency (e.g. 17 Hz) Lower Frequency (e.g. 14 Hz) EEG Power Spectrum

88 Steady-State Visual Evoked Potentials (SSVEP) A B C D E F up to 48 different frequencies possible!

89 Robot with video camera control Saumitra Dasgupta, Mike Fanton, Jonathan Pham, Mike Willard Faculty advisors: Deniz Erdogmus (BCI) and Bahram Shafai (irobot) Equipment used: g.usbamp with g.butterfly electrodes, irobot Four checkerboards flicker at: 13Hz (top left: means turn left) 11Hz (top right: means turn right) 9Hz (bottom left: means go forward) 7Hz (bottom right: means stop). Robots sends video back from webcam via Skype to operator Zero-order-hold fashion control - until a new/different command comes the robot continues to perform the latest received command. Welch periodogram + SVM classifier of frequencies

90 SSVEP + less training up to 48 classes fast response up to bit/min _ reduced responses for higher frequencies annoying stimulation the adaptation problem

91 BCI Award 2010