On-line Gesture Recognition

Transcription

1 On-line Gesture Recognition Luis A. Leiva

2 Presentation Outline Introduction 1 Preliminaries 19 Some Techniques 31 Recap 59 References 63 Slides available at On-line Gesture Recognition 1

3 Introduction On-line Gesture Recognition 2

4 Definitions Gesture / dzsts@/ noun The use of motions of limbs or body as a means of expression. Synonyms: signal, sign, motion, indication, gesticulation Gestures are hand-drawn strokes that do things. Lipscomb (1991) On-line Gesture Recognition 3

5 Definitions 1. Off-line gesture recognition: post-hoc, processed after user interaction static data, no temporal info available 2. On-line gesture recognition: realtime, direct manipulation sequential, time series data On-line Gesture Recognition 4

6 Historical Precedents Sketchpad RAND tablet Sutherland (1963) Davis and Ellis (1964) On-line Gesture Recognition 5

7 Gestures Today Minority Report. Image by 20th Century Fox & DreamWorks On-line Gesture Recognition 6

8 Input Devices Wii. Image by Nintendo Co., Ltd. On-line Gesture Recognition 7

9 Input Devices T(ether). Image by Massachusetts Institute of Technology On-line Gesture Recognition 8

10 Input Devices Kinect for Xbox 360. Image by Microsoft Corporation On-line Gesture Recognition 9

11 Input Devices Humantenna. Image by Microsoft Research On-line Gesture Recognition 10

12 Input Devices Skinput. Image by Carnegie Mellon University On-line Gesture Recognition 11

13 Input Devices Myo. Image by Thalmic Labs On-line Gesture Recognition 12

14 Input Devices Leap motion. Image by Leap Motion, Inc. On-line Gesture Recognition 13

15 Input Devices Air Clicker. Image by Yanko Design On-line Gesture Recognition 14

16 Input Devices TapTap. Image by Woodenshark LLC On-line Gesture Recognition 15

17 Input Devices Pen Tail gestures, by Tian et al. (2012) On-line Gesture Recognition 16

18 Natural communication Expressiveness Ergonomics Usability Fun Advantages On-line Gesture Recognition 17

19 Disadvantages May break fundamental interaction principles: Discoverability, Reliability, Scalability, etc. Ambiguity: non-deterministic decoding Lack of standards Cultural issues On-line Gesture Recognition 18

20 Trade-offs accuracy design setup performance On-line Gesture Recognition 19

21 Preliminaries On-line Gesture Recognition 20

22 Interaction Paradigms Mid-air Onscreen On-line Gesture Recognition 21

23 Definition stroke = pointer down pointer move pointer up s = {(x 1,y 1,t 1 ) (x j,y j,t j ) (x N,y N,t N )} On-line Gesture Recognition 22

24 A Taxonomy 1. zero-order gestures On-line Gesture Recognition 23

25 A Taxonomy 2. first-order gestures (unistrokes) On-line Gesture Recognition 24

26 A Taxonomy 3. higher-order gestures (multistrokes) On-line Gesture Recognition 25

27 Processing Pipeline Input gesture Capture Preprocessing Output command Recognition Feature extraction Classifier selection Feature selection On-line Gesture Recognition 26

28 Capture Event-based Polling (constant frequency) 1 px Sampling rate matters! x5 x6 x3 x8 low freq. high freq. On-line Gesture Recognition 27

29 Preprocessing input capture segmentation noise removal resampling normalization *optional steps On-line Gesture Recognition 28

30 Features Feature Engineering is an art! On-line Gesture Recognition 29

31 Recognition Techniques Hashing: dictionary lookup, zone coding, chain codes Parametric: linear fitting, corner detection Matching: DTW, k-nn, dollar family Statistical: GLM, RF, ANN, HMM, CRFs Ad-hoc: knowledge-based, decision trees, FSM On-line Gesture Recognition 30

32 Bonus: Continuous Recognition OctoPocus, by Bau and Mackay (2008) On-line Gesture Recognition 31

33 Some Techniques On-line Gesture Recognition 32

34 Marking Menus by Kurtenbach (1991) On-line Gesture Recognition 33

35 Marking Menus Blender. Image by Blender Foundation On-line Gesture Recognition 34

36 Linear Fitting ŷ = a+bx minimize R 2 = N i=1 r i 2 y vertical offsets y perpendicular offsets x r i = y i (a+bx i ) r i = y i (a+bx i ) x 1+b 2 On-line Gesture Recognition 35

37 Corner Detection PDL, ShortStraw, Firefox s QuickGestures, etc. G = s 1,...,s n,...,s N s n {L,R,U,D} On-line Gesture Recognition 36

38 Graffiti & Unistrokes Comparison by Castellucci and MacKenzie (2008) On-line Gesture Recognition 37

39 Graffiti & Unistrokes On-line Gesture Recognition 38

40 Rubine recognizer by Rubine (1991) On-line Gesture Recognition 39

41 Rubine recognizer Linear classifier using N = 13 stroke features f ( w T g ) = w o + N i=1 Σ 1 µ i w 0 = 1 2 N i=1 w i µ i Weight estimation: perceptron, LSBF, LDA, SVM, logistic regression, etc. On-line Gesture Recognition 40

42 Shapewriting SHARK 2, by Kristensson and Zhai (2004) On-line Gesture Recognition 41

43 Shapewriting sokgraph of hello Q W E R T Y U I O P sokgraph of there Q W E R T Y U I O P A S D F G H J K L A S D F G H J K L Z X C V B N M Z X C V B N M Ŵ = argmax W P(W g) Ŵ = argmax W P(g W)P(W) P(g) = argmax W P(g W)P(W) On-line Gesture Recognition 42

44 Euclidean Matching point-wise distances g t D(g,t) = 1 g g i=1 g i t i On-line Gesture Recognition 43

45 Elastic Matching dynamic programming g t D(g,t) = min W W 1 W W k=1 w k On-line Gesture Recognition 44

46 Elastic Matching g i W = w 1,...,w k,...,w K w k w K w 1 j t γ(i,j) = d(i,j)+min{γ(i 1,j),γ(i,j 1),γ(i 1,j 1)} On-line Gesture Recognition 45

47 The Dollar Family: $1 recognizer by Wobbrock et al. (2007) On-line Gesture Recognition 46

48 The Dollar Family: $1 recognizer Preprocessing: resampling, rotation, scaling, translation indicative angle (0,0) g t D(g,t) = argmin π 4 θ π 4 1 N N i=1 g i t i (θ) On-line Gesture Recognition 47

49 The Dollar Family: $1 recognizer On-line Gesture Recognition 48

50 The Dollar Family: $N recognizer by Anthony and Wobbrock (2010) On-line Gesture Recognition 49

51 The Dollar Family: $N recognizer $N is $1 with combinatory overhead: O(n s 2 s ) per template Memory drained out with 20 templates (n = 32 pts) in a quad-core computer with 4 GB RAM. On-line Gesture Recognition 50

52 The Dollar Family: $P recognizer by Vatavu et al. (2012) On-line Gesture Recognition 51

53 The Dollar Family: $P recognizer Variation of the Hungarian algorithm: D(g,t) = min{ g t, t g } Haussdorf s alternatives: D(g,t) = max i D(g,t) = 1 N N i=1 min j g i t j min j g i t j On-line Gesture Recognition 52

54 The Dollar Family: $P recognizer On-line Gesture Recognition 53

55 The Dollar Family: Protractor by Li (2010) On-line Gesture Recognition 54

56 The Dollar Family: Protractor Closed-form solution, minimum angular distance D(g,t) = 1 arccos(acosˆθ +bsinˆθ) ˆθ = arctan b a a = N (x gi x ti +y gi y ti ) b = N (x gi y ti y gi x ti ) i=1 i=1 On-line Gesture Recognition 55

57 The Dollar Family: Protractor On-line Gesture Recognition 56

58 MinGestures for MIUIs Disambiguate gestures from handwritten text LABEL ACTION RESULT LABEL ACTION RESULT Substitute Lorem Ipsum Lorem Ipsan Split Lorem Lor em Reject Lorem Ipsum Lorem... Validate Lorem Ipsum Lorem Ipsum Merge Lorem Ipsum LoremIpsum Delete Lorem Ipsum Lorem Undo Lorem Lorem Ipsum Redo Lorem Ipsum Lorem Insert Lorem Ipsum Lorem et Ipsum Help Lorem Ipsum <help event> by Leiva et al. (2014) On-line Gesture Recognition 57

59 x = N i=2 MinGestures for MIUIs Three disambiguating features: RMSE = 1 N x i y i N max(x i 1 x i,0) i=1 Classification rule: θ = ŷ b x ±ǫ ϕ = max(x) min(x) max(y) min(y) On-line Gesture Recognition 58

60 MinGestures for MIUIs E-pen Mouse System training test training test $1 recognizer Marking Menus Modified $ Rubine MinGestures Error rates in % On-line Gesture Recognition 59

61 Recap On-line Gesture Recognition 60

62 Takeaways Gestures shortcut tedious commands Many recognition techniques for many input devices Trade-offs: design, setup, performance, accuracy Gestures should be fast and simple: 1. For humans to perform and recall 2. For computers to recognize On-line Gesture Recognition 61

63 Open Problems 1. Integration: free-form gestures in context 2. Error analysis and recovery: When should the recognizer ask the user? How much to ask? 3. Segmentation: automatic gesture parts identification 4. Generation: grammar-based, kinematic theory, VAEs, etc. On-line Gesture Recognition 62

64 On-line Gesture Recognition 63

65 Bibliography I. E. Sutherland. Sketchpad: A man-machine graphical communication system. Tech. Report 296, Lincoln Laboratory, MIT, M. Davis and T. Ellis. The RAND tablet: A man-machine graphical communication device. In Proc. AFIPS, D. H. Rubine. The Automatic Recognition of Gestures. PhD thesis, Carnegie Mellon University, S. Zhai, P. O. Kristensson, C. Appert, T. H. Andersen, and X. Cao. Foundations and trends in Human Computer Interaction. Foundational Issues in Touch- Surface Stroke Gesture Design An Integrative Review, 5(2), C. C. Tappert, C. Y. Suen, and T. Wakahara. The state of the art in on-line handwriting recognition. IEEE T. PAMI, 12(8), R. Plamondon and S. N. Srihari. On-line and off-line handwriting recognition: a comprehensive survey. IEEE T. PAMI, 22(1), J. S. Lipscomb. A trainable gesture recognizer. Patt. Recogn., 24(9), On-line Gesture Recognition 64

66 G. P. Kurtenbach. The design and evaluation of marking menus. PhD thesis, University of Toronto, O. Bau and W. E. Mackay. OctoPocus: A dynamic guide for learning gesture-based command sets. In Proc. UIST, S. J. Castellucci and I. S. MacKenzie. Graffiti vs. Unistrokes: An empirical comparison. In Proc. CHI, P. O. Kristensson and S. Zhai. SHARK 2 : A large vocabulary shorthand writing system for pen-based computers. In Proc. UIST, L. A. Leiva, V. Alabau, V. Romero, A. H. Toselli, and E. Vidal. Context-aware gestures for mixed-initiative text editing UIs. Interact. Comput., 27(1), J. O. Wobbrock, A. D. Wilson, and Y. Li. Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes. In Proc. UIST, Y. Li. Protractor: a fast and accurate gesture recognizer. In Proc. CHI, L. Anthony and J. O. Wobbrock. A lightweight multistroke recognizer for user interface prototypes. In Proc. GI, R.-D. Vatavu, L. Anthony, and J. O. Wobbrock. Gestures as point clouds: a $P recognizer for user interface prototypes. In Proc. ICMI, On-line Gesture Recognition 65

67 F. Tian, F. Lu, Y. Jiang, X. Zhang, X. Cao, G. Dai, and H. Wang. An exploration of pen tail gestures for interactions. Int. J. Human-Comput. Stud., 71, On-line Gesture Recognition 66

68 Videography Sketchpad. RAND tablet. Minority Report. Nintendo Wii. T(ether). MS Kinnect. Humantenna. Skinput. Myo. Leap Motion. On-line Gesture Recognition 67

69 TapTap. Intugine. Wacom gestures. OctoPocus. Marking Menus. Shapewriting. $1 recognizer. $N recognizer. $P recognizer. On-line Gesture Recognition 68