Brain Computer Interfaces Lecture 2: Current State of the Art in BCIs

Transcription

1 Brain Computer Interfaces Lecture 2: Current State of the Art in BCIs Lars Schwabe Adaptive and Regenerative Software Systems UNIVERSITÄT ROSTOCK FACULTY OF COMPUTER SCIENCE AND ELECTRICAL ENGINEERING INSTITUTE OF COMPUTER SCIENCE

2 Some Neuroscience Background: Sensory Systems and Motor Control (Prof. Angelucci, Univ of Utah, and Prof. Milan, EPFL)

3 Visual System Primer 2011 UNIVERSITÄT ROSTOCK FACULTY OF COMPUTER SCIENCE AND ELECTRICAL ENGINEERING INSTITUTE OF COMPUTER SCIENCE 15

4 Parallel Processing (Macaque monkey is a model system for invasive studies of visual cortical processing.) Where/action pathway What/perception pathway (Ungerleider & Mishkin, 1982)

5 The LGN has an orderly retinotopic map

6 Ramon Ramon y Cajal, y Cajal, The six layers of the neocortex The six layers of the neocortex

7

8 Pia Each layer has specific inputs & outputs in V1 White Matter - Layer 4C: Main input layer - M and P afferents from the LGN remain segregated in 4C: M --> 4Cα, 6 - V1 output layers: Π > 4Χβ, 4Α, 6 - Layers 2/3 and 4B ----> extrastriate cortex Ι/Κ > 2 3,1 - Interlaminar connections integrate activity within - Layer > SC, pulvinar, pons - Layer > LGN, claustrum V1: - from input layers upward to layers 2/3 - from layers 2-3 downward to > 6 - feedback 6 ---> 4C and > 2/3

9 Inter-areal Connections (Again: Macaque monkey as a model system) Feedforward Feedback

10 Orientation Tuning in V1

11 Orientation Tuning in V1 Current belief: Selectivity is in the feedforward inputs, intra-cortical and feedback connections modulate (feedback may carry attentional task-specific information).

12 Vision is not solved yet, neither in computer nor in biological vision. Receptive field and feedforward processing Feedforward (bottom-up) Feedback (top-down) Feedback is needed to perform global-to-local computations to recognize objects in cluttered scenes.

13 The Motor System: Types of Movements

14 The Motor System: Cortical and Subcortical Structures

15 The Motor System: Movement Planning

16 Somatotopic representation of motor and somatosensory system

17 Movement control (Interested in learning more about movement control? Ask me for more references and slides using a control engineering terminology.)

18 Basal Ganglia and Movement Initiation

19 Involved brain areas (metabolic activity) Executed: Imagined:

20 BCIs: Architecture and Sensing Modalities (Credit to gtec, see gtec.at. We use their devices.)

21 Brain Computer Interfacing for Neurofeedback

22 Brain Computer Interfacing for Interaction with the Environment EEG Signals EEG recording device Feature Extraction Feedback to user (C) Milan, EPFL Classification Action generation

23 Signal Flow (c) by gtec

24 Sensing Modalities: EEG and ECoG

25 Sensing Modalities: ECoG and LFPs (LFP=local field potential) Invasive devices have the best signal quality.

26 Real-time fmri

27 Real-time fmri: Clinical Implications

28 Can we use real-time FMRI for HCI? With FMRI we have to live with the hemodynamic delay (~6 sec).

29 Non-invasive sensing modalities applicable to humans fmri - good spatial resolution - poor temporal resolution - not mobile EEG - non-invasive - mobile and not expensive - good temporal resolution - poor spatial resolution - artifacts - mainly cortical sources, but still ill-posed problem For HCI, we focus on EEG-based BCIs.

30 Demo Video: P300 Speller

31 Demo Video: Control of a Smart Home with a P300-EEG-BCI

32 Demo Video: SSVEPs

33 BCI with Senorimotor Rhythms Left Relax Attention! You thought of Left Right

34 Established EEG-BCI Paradigms P300 => The BCI detects the surprise ~300ms later SSVEPs => The BCI isolated the dominant frequency frequency. Sensory-Motor Rhythms => The BCI compared power in ~10 Hz for left vs. right.

35 The P300 ( P300 = P3a+P3b ) is evoked by unexpected stimuli.

36 SSVEPs

37 SSVEPs Typical waveform of an EEG signal (Oz-Cz) acquired during visual light stimulation with a frequency of 15 Hz. Corresponding Powerspectrum. Choose frequencies, which are NOT harmonics of each other! (Zhu et al., Computational Intelligence and Neuroscience, 2010)

38 The state of a subject is reflected in the brain activity. Excited Relaxed Drowsy Asleep Deep sleep Compare Excited vs. Relaxed : This difference is used in BCI, which use sensory-motor rhythms.

39 Other Selected Advances in Non-Invasive EEG-based BCIs Decoding based on lateralized readiness potential. => Berlin BCI Using other modalities (such as touch) for P300-based BCIs. => Various groups Getting EEG-based BCIs to work in real-world settings => TOBI EU Project. Using the brain s error signal to correct the decoded action => Milan, EPFL Many new applications of the established paradigms => Various groups Cortically coupled computer vision => Columbia University, P. Sajda EEG-based adaptive HCI for time-critical decision making => Our own ongoing work.

40 Summary of the three main paradigms

41 Summary The neocortex has a generic (?) architecture with 6 layers. Sensory and motor areas in the neocortex. Movement initiation = removing the break on a selected candidate action. The front of the brain looks at the back. EEG (good temporal, but poor spatial resolution) vs. FRMI (vice versa) EEG-BCI-Paradigms: P300, SSVEPs, Sensori-Motor Rhythms UNIVERSITÄT ROSTOCK FACULTY OF COMPUTER SCIENCE AND ELECTRICAL ENGINEERING INSTITUTE OF COMPUTER SCIENCE 41