Design of Hands-Free System for Device Manipulation

Transcription

1 GDMS Sr Engineer Mike DeMichele Design of Hands-Free System for Device Manipulation Current System: Future System: Motion Joystick Requires physical manipulation of input device No physical user input

2 PROJECT OVERVIEW Goal: Create and demonstrate a system to relocate assets at a user s command without physical input from the user. Technological Readiness Level : 2-3 Based on DOD TRL Definitions Active R&D is initiated. This includes analytical studies and laboratory studies to physically validate the analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative. [0] DOD TRL Definitions 2 2

3 AGENDA 3 Motivation Problem Statement EEG Context Stakeholders and Gap Analysis CONOPS CONOPS Validation Project Management 3

4 PHYSICALLY DISABLED STATISTICS Approximately 1.9% of the U.S. population, or 5,596,000 people, reported some form of paralysis [1] 2012 study performed by the Christopher and Donna Reeve Foundation 4 4

5 HUMAN-MACHINE INTERACTION Process driven by input from human user Relies on interface to continue data exchange Challenges Systems requiring a huge amount of user input Time sensitive systems Systems requiring user accuracy Systems requiring frequent operator turnover [3] Robot FRIEND, Institute of Automation (IAT) of University of Bremen [2] FDA Device Regulations- Human Factors 5 5

6 AGENDA 6 Motivation Problem Statement EEG Context Stakeholders and Gap Analysis CONOPS CONOPS Validation Project Management 6

7 PROBLEM STATEMENT Hands-free Paralyzed people are unable to use any device which requires a physical interaction. Multi-functional The control system can be applied to the operation of a wheelchair, robot or computer in an effort to substantially increase the safety, quality of life and independence of a paralyzed person 7 7

8 SCOPE Before a new control interface can be selected for the robotic aid, we must examine the maneuvers which the robot will need to perform. This project will perform a simulation to collect and analyze the data crucial to creating a hands-free control interface. 8

9 TRADE SPACE ALTERNATIVES Methods to Perform Hands-Free Control Eye Tracking Software Head Movement (Gyroscope) Voice Control Muscle Contraction Detection Brain-Computer Interface -EEG [4]RoboNurse, Computers Making Decisions- Standford [5] Accompany Care-O-Bot, Fraunhofer Institute 9 9

10 BRAIN-COMPUTER INTERFACE (BCI) Direct communication pathway between the brain and a computing device BCI Type Signal Quality Sensor Location Cost Other Commitmen t 1. Invasive Best In Brain Matter High 2. PartiallyInvasive Medium Under skull High 3. Non-invasive Poor On Scalp Low Risk of scar tissue EEG is the most studied non-invasive BCI method due to its low cost and ease of use. The data quality is improving quickly

11 AGENDA 11 Motivation Problem Statement EEG Context Stakeholders and Gap Analysis CONOPS CONOPS Validation Project Management 11

12 STAKEHOLDER ANALYSIS 12

13 STAKEHOLDER INTERACTIONS 13 13

14 GAP ANALYSIS The necessity of physical movement of the control interface is a highly limiting factor is the system usability and user workload. For the 2,909,920 severely paralyzed persons in America, the necessity of physical motion to operate a device renders the system unusable. Win-Win - Create a hands-free control system which can match the quality and cost of a traditional control system 14 14

15 AGENDA 15 Motivation Problem Statement EEG Context Stakeholders and Gap Analysis CONOPS CONOPS Validation Project Management 15

16 CONOPS A paralyzed person cannot perform basic motor tasks without assistance. A robot, controlled by the person, can cross the room and retrieve an item (prepackaged meal, medicine, cell phone). User with HFCS 16 YouBot by Kuka [9] Youbot, Vertically-Integrated Projects, GT 16

17 COMMANDS FOR CONOPS Loop 1: Correct direct 1. Stop motion 2. Pivot to correct direction 3. Resume forward motion Assumption 1: Hand contacting object will result in grasp. (No slipping, etc) 17 17

18 USE CASE 18 18

19 MISSION REQUIREMENTS R.0 The HFCS shall be operable without physical user input. R.1 The HFCS shall direct the motion of a simulated robot in 3 dimensions. R.2 The robot shall perform motion directed by HFCS. R.3 The HFCS shall direct the movement of the simulated robot arm. R.4 The robot arm shall perform movement directed by HFCS R.5 The HFCS shall not harm the user in any way. R.6 The HFCS shall provide flexibility for use for a variety of functions. R.7 The HFCS system shall operate in real-time

20 FUNCTIONAL DIAGRAM 20

21 AGENDA 21 Motivation Problem Statement EEG Context Stakeholders and Gap Analysis CONOPS CONOPS Validation Project Management 21

22 VIRTUAL ROBOTIC SIMULATOR V-REP as a virtual robot simulator Physics based collisions Large number of ready made models Free for educational use YouBot will pick up an object and move it to another location Limited preprogrammed movement YouBot, a popular multi-function robot 22

23 YouBot VIRTUAL ROBOT[9] Platform Omnidirectional wheels Zero turn radius pivoting Surface for placing multiple objects Arm 5 joints for 6DoF movement (Arm is not capable of full pitch motion) Each joint has a limited range of movement Gripper Open and close To simplify the YouBot movement, we programmed a function to move all the joints according to the x-y-z position of the gripper in order to reduce user complexity 23

24 SIMULATION OVERVIEW Fetch item - an item the virtual robot is instructed to pick up Fetch point - the location of the fetch item Task - a randomly generated trip from a start point, to the fetch point, picking up the fetch item and carrying it to an assigned finish point Start location - initial (x,y) coordinates of the center of the robot s platform Finish location - final location of the center of the robot s platform Start Points (Green), End Point (Red) Fetch Points (Blue) 24

25 SIMULATION STATISTICS Statistics to Record per Task 1. Start location 2. Fetch item location 3. End location 4. Total time on task 5. Time spent on Path 1 6. Time spent on Path 2 7. Time spent picking up object (frame between Path 1 and Path 2) 8. Total distance traveled 9. # of platform rotations 10. degrees rotated for each rotation 11. # lateral arm movements 12. Arm displacement per movement (cm) 25

26 AGENDA 26 Motivation Problem Statement EEG Context Stakeholders and Gap Analysis CONOPS CONOPS Validation Project Management 26

27 FUTURE WORK Spring Semester: Create simulation Run and collect statistics Further Work: Identify the best interface design, given statistics from simulation 27 27

28 PROJECT BUDGET Labor cost - $40/ hr Hourly rate - $85.40 GMU Overhead rate 2.13 Planned hours - 1,576 Planned Equipment cost - $400 Planned Budget = $134,

29 WORK BREAKDOWN STRUCTURE 29 29

30 CRITICAL PATH TASKS Current Critical Path Analysis, performed in MS Project, returns the follow tasks as critical path tasks: 2.3 Problem Statement 3.1 Requirements 4.4 Physical Architecture 5.3 Perform Test Activities 8.2 Final Presentation 8.3 Final Report 30 30

31 Performance Metrics Week 1 through Week

32 PROJECT RISKS 32 Risk # Foreseeable Risk Mitigation Strategies 1 Delivery of Equipment 1a. Order equipment asap 2 Data Collection - quality of equipment, data quality 2a. Careful review of equipment reviews before ordering 2b. Research data collection techniques 3 Access to raw data 3a. Contact equipment company 3b. Find contacts within Mason community 4 Data analysis timeline driven by data collection/ equipment 4a. Develop test plan in line with system architecture 4b. Closely monitor progress towards project completion. 32

33 QUESTIONS? Current System: Future System: Motion Joystick Requires Physical Manipulation of input device 33 No Physical User Input 33

34 BACKUP SLIDES 34 34

35 References [1] "Prevalence of Paralysis." Christopher & Dana Reeve Foundation. The Reeve Foundation Paralysis Resource Center, n.d. Web. 19 Nov [2] "Human Factors and Medical Devices." U.S. Food and Drug Administration. U.S. Food and Drug Administration, n.d. Web. 19 Nov [3] "Care-Providing Robot FRIEND." Institute of Automation. University of Bremen, n.d. Web. 19 Nov [4] "Computers and Robots: Decision-Makers in an Automated World." Computers and Robots DecisionMakers in an Automated World. University of Stanford, n.d. Web. 19 Nov [5] "Care-O-bot 3." Fraunhofer Institute for Manufacturing Engineering and Automation. Fraunhofer Institute, n.d. Web. 19 Nov [6] "Brain Cells Chat, Even Without a Synapse." Science Magazine. AAAS, n.d. Web. 19 Nov [7]"Parts of Central Nervous System.", Control and Coordination, Science Help. Tutorvista, n.d. Web. 19 Nov [8] "A Closer Look at EEG." Epilepsy Society. Epilepsy Society, n.d. Web. 19 Nov [9] "YouBot." YouBot Store. Kuka, n.d. Web. 19 Nov [10] "Emotiv EPOC / EPOC+." Emotiv Epoc. Emotiv, n.d. Web. 19 Nov [11] "Quantitative EEG and Event-Related Potentials." Neuronetrix. COGNISION, n.d. Web. 19 Nov [12] T. Pierce, T. Watson, J. King, S. Kelly and K. Pribram, 'Age Differences in Factor Analysis of EEG', Brain Topography, vol. 16, no. 1, pp ,

36 AGENDA 36 Motivation Problem Statement EEG Context Stakeholders and Gap Analysis CONOPS CONOPS Validation Project Management 36

37 EEG TERMINOLOGY Signals - The electrical activity that travels through a user s brain Synapses - Spaces between neurons that conduct the electrical activity in the brain Noise - General term for unwanted and, in general, unknown modifications that a signal may suffer during capture, storage, transmission, processing, or conversion [6] AAAS, Brain and Behavior Event-Related Potential - electrophysiological response to a stimulus User command - The instruction a user gives to perform an action (user signal to EEG) Input Command - A collection of steps the end-device will perform in order to execute a single user command (digital signals to end-device) 37 37

38 PHYSIOLOGY OF THE BRAIN Human Brain - Comprised of 100,000,000,000 neurons - Neurons connect via synapses - When thought occurs, neurons generate spikes of electrical activity [7] Canadian Institutes of Health Research, Institutes of Neurological, Mental Health and Addiction 38 38

39 ELECTROENCEPHALOGRAHY (EEG) Offers a non-invasive method to detect electrical activity by using electrodes to measure the electrical pulses along the scalp to ultimately graph these impulses Predominantly used in medical field for sleeping/brain abnormalities and detection, stress evaluation, and prosthetics Event related potentials (ERP) Correspond to a brain response related to a sensory, cognitive or motor event The key to using EEG as a control method is to detect the desired ERP from other ERPs (noise) and map it to a command EEG is not widely used as a control method due to poor signal quality from being outside of the skull. Technological advances are improving quality

40 How the EEG Hands-Free System Works 1) When a user thinks, electrical pulses fire within synapses in lobes of the brain specific to that thought - user command 2) The EEG headset reads these signals as voltage and generates a graph indicating the intensity and pattern of the electrical signals (The same stimuli or thought should produce roughly the same EEG patterns when repeated) 3) These graphs will then be analyzed and chosen patterns will be stored in the database with it s corresponding thought or stimulus 4) These selected patterns will then be mapped to varying device actions through a system that this project will create - input command 40

41 EEG receives signals and captures them in the form of graphs 14 Sensors Raw signals are stored in database and inputted into the system EEG raw data stream [8] A Closer Look at EEG, Epilepsy Society System filters out noise and converts signals to digital format System sends digital signals to library to be matched to a command System outputs corresponding command to simulator User concentrates on a trained user command 41

42 PATTERN LIBRARY BLOCK DIAGRAM 42 42

43 OPERATIONAL BLOCK DIAGRAM 43 43

44 44

45 EEG SIGNAL PROCESSING The EEG signal will be composed of 14 continuous and numerical data streams, one from each of the EEG. General steps to preprocess the signal: 1. High pass filter to remove low frequency noise 2. Use a window function to group a variable length interval of data such as the Hann or Hamming window function and eliminate edge artifacts 3. Fast Fourier Transform 4. Convert the imaginary part of any complex numbers into amplitude values Use a machine learning approach such as neural networks to build a learning model, then continuously test in real time to measure accuracy 45 45

46 OPTIMIZING EEG Event-Related Potentials Goal: Determine largest limiting factor in EEG Control System reliability Use simulation to examining the effects of different variables on detection rate and reliability of ERPs such as: Optimal training procedures or increasing training time Use of different machine learning approaches to improve detection Tentative input commands to be detected: 6 directional commands for positive and negative displacement along each of the x, y, z axes 1 toggle command for switching platform control vs arm control May need to simplify the number of different commands to reduce user complexity 46 46

47 SIMULATION REQUIREMENTS S.0.1 Virtual Robot shall accept input from HFCS. S Virtual Robot model shall contain a robot with 2D movement capability S Virtual Robot model shall contain a robot capable of grasping and relocating S Virtual Robot model shall contain an object to be relocated S.0.2 Simulation shall accept EEG ERPs from an input device S Simulation shall respond to a detected ERP in time < 2 seconds S Simulation shall map ERPs to input commands and interface with VREP S Simulation should detect and respond to 2+ simultaneous ERP S.0.3 Virtual robot shall employ automated collision avoidance

48 EPOC Cognitive Suite 1) Create user account 2) Train neurological signal to a basic motion 3) Repeat thought to perform signal 4) Map signal to another vehicle motion 48 Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar Source: Emotiv EPOC 48

49 EEG Headset Alternatives Standard EPOC 14 EEG channels and two references for accurate spatial resolution High performance wireless device ios and Android compatible EPOC+ 9 axis-inertial motion sensors, Bluetooth capabilities, additional applications are enabled Raw Data Add-on Includes EEG firmware that allows the raw EEG data stream and marker events in TestBench software 49 49

50 PROJECT PLANNING, detailed 50

51 Statement of Work Objectives: Subset of brain signals mapped to the specific thought which generated the signal Documented process to identify and map new brain signals Increase value of a hands-free application Quality deliverables and products produced Improved realization of hands-free control systems Scope 51 Project Management Documentation Data Collection & Analysis Process Test Plan & Criteria Delivery of a Prototype Test Results Utility Analysis Plan for future expansion 51

52 52

53 53

54 54 54

55 55 55

56 56

57 Brainwaves Neural oscillations are rhythmic or repetitive neural activities in the central nervous system. Neural tissue generate oscillations through neuron interactions. A large number of neurons activating for a particular neural computation is called a neural ensemble. Neural oscillations can be categorized into 5 main frequencies Beta (14-40Hz) - Waking consciousness Alpha (7.5-14Hz) - Conscious relaxation Theta (4-7.5Hz) - Sleeping and light meditation Delta (0.5-4Hz) Deep sleep Gamma (above 40Hz) Insight Only recently discovered 57 57

58

59 Age A study conducted by Thomas W. Pierce et. al suggested that age has an effect on EEG Results showed that electrode groupings were higher in older adults than younger adults Older adults had more electrode locations that did not load than the younger adults 59 59

60 [1]T. Pierce, T. Watson, J. King, S. Kelly and K. Pribram, 'Age Differences in Factor Analysis of EEG', Brain Topography, vol. 16, no. 1, pp ,

61 Gender A study by Corsi-Cabrera et. al suggested differences in the brain waves between male and female Men showed higher beta power, women showed higher alpha power The alpha waves of men decreased during analytic,spatial, and mixed processing, while women decreased significantly only in analytic and mixed processing 61 61

62 Interface Diagram 62