Brain-Machine Interface for Neural Prosthesis:

Transcription

1 Brain-Machine Interface for Neural Prosthesis: Nitish V. Thakor, Ph.D. Professor, Biomedical Engineering Joint Appointments: Electrical & Computer Eng, Materials Science & Eng, Mechanical Eng Neuroengineering Training Grant IEEE Transactions on Neural and Rehabilitation Eng

2 What is Neuroengineering? Neuroengineering is defined as the interdisciplinary field of engineering and computational approaches to problems in basic and clinical neurosciences. Thus, education and research in Neuroengineering encompasses the fields of engineering, mathematics and computer science on one hand, and molecular, cellular and systems neurosciences on the other hand. 1. Neural Cellular/Tissue Eng With N. L. Jeon, UCI

3 2. Neural Instrumentation and Implanted Devices Skull Telemetry chip Brain Gray Matter Nano Sensor array Micro Battery VLSI Potentiostat 3. Neural Prosthesis Program Mission Development of a fully neurally-integrated upper extremity prosthetic with appropriate documentation for clinical trials, FDA approvals, and manufacturing transition. DARPA Revolutionizing Prosthetics 2009

4 From Prosthetic Claw to Human Hand Ottobock Cylindrical Palmar Tip Spherical Motion Control Hand Vanderbilt University Hook Lateral Images from Next Generation Prosthetic Hand Michelangelo Hand developed by Otto Bock Weight = 400 g Speed of opening = 408 mm/sec Grip force = 120 N (27 lbsf) Width of opening = 102 mm (4 ) Powered by Lithium-Ion battery within the Dynamic Arm. Sufficient capacity to operate for 18 hours of usual everyday activities

5 The Classic Classical (EMG based) Approach and the New (Brain EMG wave) Control Control of of Prosthesis Signals for Brain-Computer Interface A direct communication and control channel from the brain to the external world, bypassing normal neuromuscular pathways Invasive: Cortical neuronal recordings, ECoG Non-invasive: EEG based Non-invasive methods (EEG): SCP (Slow Cortical Potentials) µ and β rhythm (motor imagery, ERD) Source Localization (motor imagery, visual imagery, abstract tasks) P300 potentials

6 Noninvasive Cortical Control In noninvasive BCIs, users learn to modulate various features of their EEG to convey their intent Imagined motor actions cause Event Related Desynchronization (ERD) in Mu (8-12 Hz) and Beta (18-25 Hz) rhythm power in sensorimotor cortex feedback Spatial Preprocessing Feature Extraction Classification A Spatiotemporal Filter for extracting activity from multiple cortical sources: Deconvolution using Information Maximization u(t) =Wx(t) Activity from Independent Cortical Sources x(t) = As(t) Observed multi-channel EEG by Maximizing Joint Entropy and Minimizing Mutual Information 1 2 3

7 Preprocessing Left Hand Motor Imagery µ-variance, DIPFIT FL Right Hand Motor Imagery Foot Motor Imagery Bandpass filter + Optimal PCA dimension reduction µ-variance, DIPFIT µ-variance, DIPFIT FR FF Independent Component Analysis Tongue Motor Imagery µ-variance, DIPFIT FT Problem: Separation of multiple brain signal sources Solution: The method of Independent Component Analysis

8 A Spatiotemporal Filter for extracting activity from multiple cortical sources: Support Vector Machine (SVM) classifier Activity from Independent Cortical Sources Subject to Equivalent Dipole Source Localization and selection of physiologically relevant dipoles 4 Spectral estimate by convolution with a Hanning windowed sinusoidal Wavelet. 5 Where y is the desired movement and x is the input feature 6 Demonstration of Cortical Control for Prosthesis Intelligent grasping of objects with varying compliance and size using local control and machine haptic feedback

9 Non-invasive Control of Prosthetic arm: a Proof of Concept Study Non-invasive Control of Multi-fingered Robotic Arm: a Proof of Concept EEG based control of a Robotic hand. (Johns Hopkins University)

10 Neurophysiological Basis of Motor Control in Hand/Fingers PRIMATE RESEARCH: Recordings of single neurons and multiple EMGs during individuated finger movements performed by monkeys Experiments by M. Schieber and team, URMC MONKEY: Recordings of single neurons with EMGs during individual finger movements

11 M1 Neural Response During Finger Movements Neuron Activity is Widely Distributed in M1 During Each Finger Movement (Schieber & Hibbard, 1993)

12 Interface for Dexterous Control of Individual Finger Movements Neural spike train from virtual microelectrode arrays (Randomly placed in M1 cortex.) N Gating decoder finger movement? Y Finger movement classifier max likelihood classifier (selected neurons based on information content) Dexterous Finger Control Single Unit Recordings Firing rates of neurons during 12 different movement types, illustrating highly tuned neurons (e.g K13409) and broadly tuned neurons (e.g K11404).

13 Maximum Likelihood (ML) Decoding m : Finger movements n: Neuron index, 1 N Define neural activity: x n (m) = x n (m) Difference between firing rate after movement and firing rate before movement - Firing rate: Poisson µ n (m) : mean firing rate Pr( x n m ) ML decoding: : the distribution of the difference of two Poisson distributions; the Skellam distribution Maximum Likelihood (ML) Decoding Skellam distribution of the difference of two Poisson distribution Pr( x n m) Firing rate I x (z) : modified Bessel function of the first kind Pr( x n m ) 2e 2e 2e

14 Maximum Likelihood (ML) Decoding Decoding performance of single finger movements Demonstration of Neural Spike Actuation of Multiple Finger Robotic Hand

15 Decoding Accuracy of Multi- Finger Movements The Cortical Piano M.Schieber

16 Looking Ahead Prosthetics Revolution is Underway 16

17 Steps to Neural Prosthesis: Implanted Microelectrode Arrays Each shank on each probe has four contacts. Multiple probes form a 3-D geometry Under Development at U of Michigan, U of Utah and Cal Tech Steps to Neural Prosthesis: A Distributed Wireless 3-D Implants Collaboration with Ghovanloo, Mehsani, et al

18 Human Robot Interaction Paradigm Neural Code Sensors Organic Electronics Skin Receptors Neuromorphic Circuits A National Science Foundation Proposal Andreou, Etienne-Cummings (ECE) Katz (Mat Sci Eng), Hsiao (Neuroscience) Haptic Study Overview Study Goal: To quantitatively assess whether haptic feedback of angle position and contact force through a sensory substitution scheme improves a prosthesis user s performance of a grasping task. Stage 1: Visual Feedback Only (learning) Stage 2: Visual + Haptic Feedback (mapping) Stage 3: Haptic Feedback Only (test) Visual Feedback Only Visual with Haptic With Vibratory Feedback Learning the task Mapping feedback Without Vibratory Feedback Control case Control case Haptic Feedback Only Test case Control (current method) with M. Bionics, A. Okamura

19 SenseNet for Temperature Feedback Tekscan -GripTM System Is this a cold, refreshing, not so heavy can of Coke? Proposed Haptics Approach Targeted Sensory Reinnervation Kinesthetic and tactile sensors Haptic display Colgate, Northwestern Univ Courtesy: T. Kuiken et al, Northwestern University

20 Sep 6, 2006 Direct 3D robot control: feeding Courtesy: A. Schwartz, U of Pittsburgh

21 Brain-Machine Interface in Humans Donoghue et al., Brown University Cognitive Control S. Harshbarger, JHU/APL Proprietary Provides hands free haptic based cognitive control for sample collection and mission support functions

22 Sun Rises on to the Field of Neural Interface/Prosthesis 2 N20 N35 Amplitude (uv) Blue <48hrs Green 72hrs Red 1week Time (msec) Post- ROSC Coma Patient Neuro Consult on Prognosis Neuro exam EEG Survivors Withdrawal Of Care

23 Building BME for the Future Multi-faceted, Multi-dimensional, Interdisciplinary