On-line Gesture Recognition

Transcription

1 On-line Gesture Recognition Luis A. Leiva PRHLT Research Center Departamento de Sistemas Informáticos y Computación Universitat Politècnica de València Presentation Outline Introduction 1 Preliminaries 19 Some Techniques 31 Recap 59 References 63 Slides available at On-line Gesture Recognition 1

2 Introduction On-line Gesture Recognition 2 Definition Gesture / dzsts@/ noun The use of motions of the limbs or body as a means of expression. synonyms: signal, sign, motion, indication, gesticulation Gestures are hand-drawn strokes that do things. Lipscomb (1991) On-line Gesture Recognition 3

3 Definition Off-line gesture recognition: post-hoc, processed after user interaction static data, no temporal info available On-line gesture recognition: realtime, direct manipulation sequential, time series data On-line Gesture Recognition 4 Historical Precedents sketchpad RAND tablet Sutherland (1963) Davis and Ellis (1964) On-line Gesture Recognition 5

4 Gestures Today Minority Report. Image by 20th Century Fox & DreamWorks On-line Gesture Recognition 6 Input Devices Wii. Image by Nintendo Co., Ltd. On-line Gesture Recognition 7

5 Input Devices T(ether). Image by Massachusetts Institute of Technology On-line Gesture Recognition 8 Input Devices Kinect for Xbox 360. Image by Microsoft Corporation On-line Gesture Recognition 9

6 Input Devices Humantenna. Image by Microsoft Research On-line Gesture Recognition 10 Input Devices Skinput. Image by Carnegie Mellon University On-line Gesture Recognition 11

7 Input Devices Myo. Image by Thalmic Labs On-line Gesture Recognition 12 Input Devices Leap motion. Image by Leap Motion, Inc. On-line Gesture Recognition 13

8 Input Devices Air Clicker. Image by Yanko Design On-line Gesture Recognition 14 Input Devices TapTap. Image by Woodenshark LLC On-line Gesture Recognition 15

9 Input Devices Pen Tail gestures, by Tian et al. (2012) On-line Gesture Recognition 16 Advantages Natural communication Expressiveness Ergonomics Usability Fun On-line Gesture Recognition 17

10 Disadvantages May break fundamental interaction principles: Discoverability, Reliability, Scalability, etc. Ambiguity: non-deterministic decoding Lack of standards Cultural issues On-line Gesture Recognition 18 Trade-offs accuracy design setup performance On-line Gesture Recognition 19

11 Preliminaries On-line Gesture Recognition 20 Interaction Paradigms Mid-air Onscreen On-line Gesture Recognition 21

12 Definition stroke = pointer down pointer move pointer up s = {(x 1, y 1, t 1 ) (x j, y j, t j ) (x N, y N, t N )} On-line Gesture Recognition 22 A Taxonomy zero-order gestures On-line Gesture Recognition 23

13 A Taxonomy first-order gestures (unistrokes) On-line Gesture Recognition 24 A Taxonomy higher-order gestures (multistrokes) On-line Gesture Recognition 25

14 Processing Pipeline Input gesture Capture Preprocessing Output command Recognition Feature extraction Classifier selection Feature selection On-line Gesture Recognition 26 Capture Event-based Polling (constant frequency) 1 px x5 x6 x3 x8 low freq. high freq. Sampling rate matters! On-line Gesture Recognition 27

15 Preprocessing input capture segmentation noise removal resampling normalization *optional steps On-line Gesture Recognition 28 Features On-line Gesture Recognition 29

16 Recognition Techniques Hashing: dictionary lookup, chain codes Parametric: linear fitting, corner detection Matching: DTW, k-nn, dollar family Statistical: linear classifier, NNs, HMMs, CRFs Ad hoc: knowledge-based, decision trees, FSM On-line Gesture Recognition 30 Continuous Recognition OctoPocus, by Bau and Mackay (2008) On-line Gesture Recognition 31

17 Some Techniques On-line Gesture Recognition 32 Marking Menus by Kurtenbach (1991) On-line Gesture Recognition 33

18 Marking Menus Blender. Image by Blender Foundation On-line Gesture Recognition 34 Linear Fitting ŷ = a + bx minimize R 2 = N i=1 r i 2 y vertical offsets y perpendicular offsets x r i = y i (a + bx i ) r i = y i (a+bx i ) 1+b 2 x On-line Gesture Recognition 35

19 Corner Detection PDL, ShortStraw, Firefox s QuickGestures, etc. G = s 1,..., s n,..., s N s n {L, R, U, D} On-line Gesture Recognition 36 Graffiti & Unistrokes Comparison by Castellucci and MacKenzie (2008) On-line Gesture Recognition 37

20 Graffiti & Unistrokes Demo: mousegesture/gesturedemo.swf On-line Gesture Recognition 38 Rubine recognizer by Rubine (1991) On-line Gesture Recognition 39

21 Rubine recognizer linear classifier using F = 13 stroke features c = f ( w T g ) = w o + F Σ 1 µ i i=1 w 0 = 1 2 F w i µ i i=1 weight estimation: perceptron, LSBF, LDA, SVM, logistic regression, etc. On-line Gesture Recognition 40 Shapewriting SHARK 2, by Kristensson and Zhai (2004) On-line Gesture Recognition 41

22 Shapewriting sokgraph of hello Q W E R T Y U I O P sokgraph of there Q W E R T Y U I O P A S D F G H J K L A S D F G H J K L Z X C V B N M Z X C V B N M Ŵ = arg max P (W g) W Ŵ = arg max W P (g W )P (W ) P (g) = arg max P (g W )P (W ) W On-line Gesture Recognition 42 Euclidean Matching point-wise distances g t D(g, t) = 1 g g g i t i i=1 On-line Gesture Recognition 43

23 Elastic Matching dynamic programming g t W 1 D(g, t) = min W W W k=1 w k On-line Gesture Recognition 44 Elastic Matching W = w 1,..., w k,..., w K g i w k w K w 1 j t γ(i, j) = d(i, j) + min{γ(i 1, j), γ(i, j 1), γ(i 1, j 1)} On-line Gesture Recognition 45

24 The Dollar Family: $1 recognizer by Wobbrock et al. (2007) On-line Gesture Recognition 46 The Dollar Family: $1 recognizer Preprocessing: resampling, rotation, scaling, translation indicative angle (0,0) g t D(g, t) = arg min π 4 θ π 4 1 N N g i t i (θ) i=1 On-line Gesture Recognition 47

25 The Dollar Family: $1 recognizer On-line Gesture Recognition 48 The Dollar Family: $N recognizer by Anthony and Wobbrock (2010) On-line Gesture Recognition 49

26 The Dollar Family: $N recognizer $N = $1 with combinatory overhead: O(n s 2 s ) per template Memory drained out with 20 templates (n = 32 pts) in a quad-core computer with 4 GB RAM. On-line Gesture Recognition 50 The Dollar Family: $P recognizer by Vatavu et al. (2012) On-line Gesture Recognition 51

27 The Dollar Family: $P recognizer variation of the Hungarian algorithm D(g, t) = min { g t, t g } Haussdorf s alternatives: D(g, t) = max i min j g i t j D(g, t) = 1 N N i=1 min j g i t j On-line Gesture Recognition 52 The Dollar Family: $P recognizer On-line Gesture Recognition 53

28 The Dollar Family: Protractor by Li (2010) On-line Gesture Recognition 54 The Dollar Family: Protractor closed-form solution, minimum angular distance D(g, t) = 1 arccos(a cos ˆθ + b sin ˆθ) ˆθ = arctan b a a = b = N (x gi x ti + y gi y ti ) i=1 N (x gi y ti y gi x ti ) i=1 On-line Gesture Recognition 55

29 The Dollar Family: Protractor On-line Gesture Recognition 56 MinGestures for MIUIs disambiguate gestures from text with high accuracy & performace LABEL ACTION RESULT LABEL ACTION RESULT Substitute Lorem Ipsum Lorem Ipsan Split Lorem Lor em Reject Lorem Ipsum Lorem... Validate Lorem Ipsum Lorem Ipsum Merge Lorem Ipsum LoremIpsum Delete Lorem Ipsum Lorem Undo Lorem Lorem Ipsum Redo Lorem Ipsum Lorem Insert Lorem Ipsum Lorem et Ipsum Help Lorem Ipsum <help event> by Leiva et al. (2014) Demo: On-line Gesture Recognition 57

30 MinGestures for MIUIs disambiguating features: RMSE = 1 N x i y i N x = ϕ = i=1 N max(x i 1 x i, 0) i=2 max(x) min(x) max(y) min(y) classification rule: θ = ŷ b x ± ǫ On-line Gesture Recognition 58 MinGestures for MIUIs System E-pen Mouse training test training test $1 recognizer Marking Menus Modified $ Rubine MinGestures Error rates in % On-line Gesture Recognition 59

31 Recap On-line Gesture Recognition 60 Takeaways Gestures aim to improve human-computer interaction Many recognition techniques suitable for different input devices Recognition trade-offs: design, setup, performance, accuracy Gestures should be simple: For humans to perform and recall For computers to recognize On-line Gesture Recognition 61

32 TFM Proposals 1. Integration: free-form gestures in context 2. Error analysis & recovery: When should the recognizer ask the user? How much to ask? 3. Segmentation: automatic gesture parts identification 4. Generation: grammar-based, kinematic theory, etc. On-line Gesture Recognition 62 On-line Gesture Recognition 63

33 Bibliography L. Anthony and J. O. Wobbrock. A lightweight multistroke recognizer for user interface prototypes. In Proc. GI, O. Bau and W. E. Mackay. OctoPocus: A dynamic guide for learning gesture-based command sets. In Proc. UIST, S. J. Castellucci and I. S. MacKenzie. Graffiti vs. Unistrokes: An empirical comparison. In Proc. CHI, M. Davis and T. Ellis. The RAND tablet: A man-machine graphical communication device. In Proc. AFIPS, P. O. Kristensson and S. Zhai. SHARK 2 : A large vocabulary shorthand writing system for pen-based computers. In Proc. UIST, G. P. Kurtenbach. The design and evaluation of marking menus. PhD thesis, University of Toronto, L. A. Leiva, V. Alabau, V. Romero, A. H. Toselli, and E. Vidal. Context-aware gestures for mixed-initiative text editing UIs. Interacting with Computers, To appear. Y. Li. Protractor: a fast and accurate gesture recognizer. In Proc. CHI, J. S. Lipscomb. A trainable gesture recognizer. Pattern Recognition, 24(9), On-line Gesture Recognition 64 R. Plamondon and S. N. Srihari. On-line and off-line handwriting recognition: a comprehensive survey. IEEE Transactions On Pattern Analysis And Machine Intelligence, 22(1), D. H. Rubine. The Automatic Recognition of Gestures. PhD thesis, Carnegie Mellon University, I. E. Sutherland. Sketchpad: A man-machine graphical communication system. Tech. Report 296, Lincoln Laboratory, MIT, C. C. Tappert, C. Y. Suen, and T. Wakahara. The state of the art in on-line handwriting recognition. IEEE Transactions On Pattern Analysis And Machine Intelligence, 12(8), F. Tian, F. Lu, Y. Jiang, X. Zhang, X. Cao, G. Dai, and H. Wang. An exploration of pen tail gestures for interactions. Int. J. Human-Computer Studies, 71, R.-D. Vatavu, L. Anthony, and J. O. Wobbrock. Gestures as point clouds: a $P recognizer for user interface prototypes. In Proc. ICMI, J. O. Wobbrock, A. D. Wilson, and Y. Li. Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes. In Proc. UIST, S. Zhai, P. O. Kristensson, C. Appert, T. H. Andersen, and X. Cao. Foundations and trends in Human Computer Interaction. Foundational Issues in Touch-Surface Stroke Gesture Design An Integrative Review, 5(2), On-line Gesture Recognition 65

34 Videography $1 gesture recognition. Humantenna. Leap Motion. Marking Menus. Myo. OctoPocus. Pen Tail. RAND tablet. Shapewriting. Shuterland s Sketchpad. Skinput. Wacom gestures. On-line Gesture Recognition 66