Enabling Natural Interaction. Consider This Device... Our Model

Transcription

1 Enabling Natural Interaction Randall Davis Aaron Adler, Christine Alvarado, Oskar Breuning, Sonya Cates, Jacob Eisenstein, Tracy Hammond, Mike Oltmans, Metin Sezgin MIT CSAIL Consider This Device... RANDALL DAVIS 2 Our Model The designer sketches with pen and paper The observer interprets the sketch The observer and designer interact RANDALL DAVIS 3

2 Sketches are Ubiquitous RANDALL DAVIS 4 Sketches are Dead RANDALL DAVIS 5 Magic Paper RANDALL DAVIS 6

3 Magic Paper RANDALL DAVIS 7 More Complex Example RANDALL DAVIS 8 More Complex Example RANDALL DAVIS 9

4 More Complex Example RANDALL DAVIS 10 Overview Architecture Early processing Learning new symbols from 1 example Sketching and talking Understanding gestures Overcoming error through context Other domains Reconceptualizing interaction: SkRUIs Reinventing desktop computing RANDALL DAVIS 11 A Prevasive Theme Natural interaction is knowledge-based. RANDALL DAVIS 12

5 Architecture Overview Shape & Seq. Description Symbolic Descr. Blackboard Early processing Hand-coded recognizers Define AndGate Recognition line L1 L2 L3 Engine arc A semi-circle A1 orientation(a1, 180) vertical L3 parallel L1 L2 same-horiz-position L1 L2 connected A.p1 L3.p1... RANDALL DAVIS 13 Shape & Seq. Description Architecture Overview Blackboard Early processing Hand-coded recognizers Learn icon Symbolic Descr. Code Gen. New recognizers 1 example Multi-modal recognizer RANDALL DAVIS 14 Architecture Overview Descriptions of Objects Application Appl n 23 Pixels Objects Sketch-Enabled Interaction RANDALL DAVIS 15

6 Early Processing Freehand stroke description in terms of geometric primitives (circles, lines, curves) RANDALL DAVIS 16 Vertex detection RANDALL DAVIS 17 Vertex detection Curvature RANDALL DAVIS 18

7 Vertex detection Curvature Speed RANDALL DAVIS 19 Results RANDALL DAVIS 20 Learning New Symbols From hand-drawn example to a shape description Define AndGate line L1 L2 L3 arc A semi-circle A1 orientation(a1, 180) vertical L3 parallel L1 L2 same-horiz-position L1 L2 connected A.p1 L3.p1 connected A.p2 L3.p2 meets L1.p2 L3 meets L2.p2 L3 RANDALL DAVIS 21

8 Learning New Symbols Goldmeir s singularities RANDALL DAVIS 22 The Description Learned STATISTICS: Number of initial relations: removed; 15 increased in relevance; 74 decreased in relevance; 16 redundant constraints removed Number of final relations: 83 RANDALL DAVIS 23 Multi-Modal Interaction Three identical, equally spaced pendulums Graphical vs verbal communication: What s best said, what s best sketched? RANDALL DAVIS 25

9 Multi-Modal Interaction Framework Nouns (pendulum) Adjectives as modifying actions to take (identical, touching, equally spaced) Next step: interactive acquisition of new nouns RANDALL DAVIS 27 Understanding Gestures Lexicon: What gestures do people make? Depends on task. Our task: explaining how something works. RANDALL DAVIS 28 Understanding Gestures 96% of gestures refer to the diagram. Two-handed gestures are common. Deixis is more frequent. Gesture units are longer. RANDALL DAVIS 29

10 Overcoming Error Domain: Sketching family trees Lexicon male female child-of married-to divorce RANDALL DAVIS 30 Overcoming Error RANDALL DAVIS 31 3 What s Hard? Task is incremental Signal is noisy Styles vary Segmentation is difficult The signal is 2-d, non-chronological RANDALL DAVIS 32

11 Use context What Do We Do? RANDALL DAVIS 33 Lexicon Context male female child-of married-to divorce Grammar RANDALL DAVIS 34 Using Context marriage child-of male married-to arrow female pentagon rectangle line ellipse RANDALL DAVIS 39

12 Challenges Controlling network growth heuristics for deferring, pruning Evaluating candidates Bayes net constructed dynamically from fragments RANDALL DAVIS 40 Bayes Net Example MD1 FD1 MS1 Mar1 Div1 PartF1 Other domain shapes for M1 F1 C1 C2 M1 O7 O8 Q1 Constraint nodes for Q2 Q2 ML1 A1 Constraint nodes for A1 E1 Constraint nodes for Q1 L1 L2 L3 L4 L6 L7 L8 L5 L9 L10 O1 Observations O2 O3 O4 O5 O6 Stroke1 Stroke3 Stroke8 RANDALL DAVIS 41 Domains Mechanical engineering Family trees Software design Circuit design PowerPoint: Exploring SkRUIs RANDALL DAVIS 42

13 Sketching Software RANDALL DAVIS 43 Sketching Software RANDALL DAVIS 44 Sketching Software RANDALL DAVIS 45

14 Sketching Software RANDALL DAVIS 46 Sketching Software //Source file: c:\\video\\deck.java public class Deck extends Hand implements Dealable { public Card thecard; public Game thegame; /** 3C21348C0257 */ public Deck() { } } //Source file: c:\\video\\game.java public class Game { public Graphics thegraphics; public Deck thedeck; public Player theplayer; /** 3C21348C02E3 */ public Game() { } } //Source file: c:\\video\\dealable.java public interface Dealable { } RANDALL DAVIS 47 Sketching Software //Source file: c:\\video\\deck.java public class Deck extends Hand implements Dealable { public Card thecard; public Game thegame; /** 3C21348C0257 */ public Deck() { } } RANDALL DAVIS 48

15 PowerPoint Diagrams RANDALL DAVIS 49 SkRUIs RANDALL DAVIS 50 SkRUIs RANDALL DAVIS 51

16 Circuit Design RANDALL DAVIS 52 Hardware RANDALL DAVIS 53 Hardware Kodak KLI linear image sensors at corners Pen: 200 Hz,.5mm, pressure sensitive Challenges: High data rate, blurred image at the sensor RANDALL DAVIS 54

17 Summary Once the sketch is understood, many things are possible. Break down the barriers: eliminate interfaces Natural interaction is enabled by intelligence About sketching, gesturing, speech About the domain About the task RANDALL DAVIS 55