T TH-Typing on Your TeetH: Tongue-Teeth

Transcription

1 T TH-Typing on Your TeetH: Tongue-Teeth Localization for Human-Computer Interface Phuc Nguyen, Nam Bui, Anh Nguyen, Hoang Truong, Abhijit Suresh, Matthew Whitlock, Duy Pham, Thang Dinh, and Tam Vu Mobile and Networked Systems Lab Department of Computer Science, University of Colorado Department of Computer Science, Virginia Commonwealth University ACM MobiSys 2018

2 Upper Jaw 1 4 GHI 7 PQRS 2 ABC 5 JKL 8 TUV 3 DEF 6 MNO 9 WXYZ 0 * + # Lower Jaw

3 It is challenging for ALS patients to interact with computing devices

4 Other Potential Usages A HAND-FREE INTERFACE FOR FACTORY WORKER TURNING MUSIC SHEETS FOR MUSICIANS HIDEN TEXT ENTRY INTERFACE CONTROLLING PHONE WHILE DRIVING USED IN TACTICAL SCENARIOS

5 TYTH Typing on Your TeetH Non-invasive, Continuous & long-term use, Socially acceptable

6 Experiment: Put your fingers at the back-of-the ear locations. Then, press any teeth using your tongue

7 Anatomical and Neurological Analysis of Tongue-Teeth Interaction Primary Motor Cortex Cortex Motor Cortex Sensorial

8 Brain sends commands out Primary Motor Cortex Cortex Motor Cortex Sensorial

9 Anatomical Analysis of Tongue-Teeth Movement Primary Motor Cortex EEG EEG EMG

10 Tongue is controlled by extrinsic and intrinsic muscles Styloglossus Dorsal surface of tongue Hyoglossus Mandible bone Genioglossus

11 TongueSee CHI 14

12 It is difficult to make the device socially acceptable TongueSee CHI 14

13 Tongue is controlled by extrinsic and intrinsic muscles Styloglossus Dorsal surface of tongue TYTH Hyoglossus Mandible bone Genioglossus Can we capture the tongue movement signal from this location?

14 Experimental Validation

15 Hardware Design

16 Sensing Techniques EEG Sensing EEG EEG EMG Sensing SKD Sensing EMG SKD

17 Sensing Techniques EEG Sensing EMG Sensing SKD Sensing

18 Sensing for EEG/EMG COTS Bioelectrical Sensing Circuit Programmable Gain Programmable Filters Low-Noise, 8 ADC Channel 16kSample/s uv sensitivity

19 Sensing Techniques EEG Sensing EMG Sensing SKD Sensing low amplitude low frequency signal

20 Skin Surface Deformation Sensing Technique Human skin Cooper (Gold plated) d Soft material (Silicon Dragon 0-10) Cooper tape (Gold plated) Capacitive sensing approach Permitivity Area size Capacitance Distance b/w 2 plates

21 Sensing Techniques EEG Sensing EMG Sensing SKD Sensing Challenge: These signals are extremely weak (mv/uv)

22 Software Components to: De-noising the signal Extracting EEG, EMG from bio-electrical sensing data Detecting when the tongue is typing/pressing Classifying where the tongue is taping on Recognizing untrained areas Challenge: These signals are extremely weak (mv/uv)

23 System Overview Electrical Sensor, Capacitive Sensor Analog Amplifier 60Hz Notch Filter Band Pass Filter BLE Communication Notch Filtering BP Filtering Low-rank Analysis EEG, EMG, SKD signals Tongue movement Detection Tongue Pressing Detection Wavelet STFT Feature Extraction SVM GMM Classification Regression Model Localization Key Mapping Feedback Key Generation TYTH wearable device Pre-processing Typing Detector Host Device Host Computer Typing Recognizer

24 Low-Rank Analysis Number of Gabor atoms in a dictionary Every bio-signal f(x) can be represented as: Or, coefficient atoms Hence, Building dictionaries to extract the main structures of the signal. Each dictionary Please represents refer the our key paper structure for of more each signal details type (EEG, EMG)

25 Low-Rank Analysis f(x) f(x main structure ) f(x EEG ) f(x EMG ) f(x detail structure ) f(x noise )

26 Pressing Detection How do we detect when a user is tapping? How do we detect when a user is pressing the teeth? Tongue movement Detection Tongue Pressing Detection Wavelet STFT Detecting the Tongue Movement by identifying the discontinuity of the signal. Detecting the Tongue Pressing based on the presence of the brain signal

27 Typing Area Classification: SVM - GMM Feature Extraction MFCC features delta double delta Initialization (UBM) Expectation Maximization mean vector SVM GMM Linear Cosine RBF Building a classification model to detect the trained typing areas

28 Typing Area Localization 42 dimensions GMM output PCA 3D space coordination During typing, typing location might not be exactly where it is trained. We build a regression model Please to detect refer the our untrained paper areas for more to recognize details these locations

29 Summary Anatomical and Neurological Analysis Hardware Sensing Design Software Components Let s put things together

30 TYTH s Prototype

31 TYTH s Prototype

32 TYTH s Prototype EEG Sensors EMG Sensors SKD Sensors

33 TYTH s Prototype

34 Performance Evaluation Teeth areas for evaluation

35 Typing Detection

36 Classified Area Recognition Performance Ground Truth Average Accuracy: 88.61%

37 User Study % 1 Extremely Difficult 2 Difficult 3 Normal 4 Easy 5 Extremely Easy Ease to use TYTH

38 Typing speed (s) User Study What are the speed of typing on the teeth?

39 TYTH s Sensing Techniques EEG Sensing EMG Sensing SKD Sensing

40 Conclusions We introduce TYTH-Typing on Your TeetH, A Non-invasive, Continuous and Long-term use and Socially Acceptable wearable device for Tongue-Teeth Localization Applications. The key contributions include: An analysis of brain, muscle, and skin deformation from behind the ears An algorithm to extract the EEG, EMG signals A novel method to sense a new type of signal, termed SKD signal A ear-mounted wearable prototype An evaluation of the system on 15 subjects

41 In progress Miniaturization Remove the impact of talking and body movement artifacts Improve the form factor for better contact quality

42 Danke schön!!! T TH-Typing on Your TeetH: Tongue-Teeth Localization for Human-Computer Interface Phuc Nguyen, Nam Bui, Anh Nguyen, Hoang Truong, Abhijit Suresh, Matthew Whitlock, Duy Pham, Thang Dinh, and Tam Vu Mobile and Networked Systems Lab Department of Computer Science, University of Colorado Department of Computer Science, Virginia Commonwealth University ACM MobiSys 2018

43 Thank you

44 Related Works TongueSee CHI 14 TongueWise EMBC 10 SITA - UIST '12 Tongue-in-Cheek CHI 15