GESTURES. Luis Carriço (based on the presentation of Tiago Gomes)

Transcription

1 GESTURES Luis Carriço (based on the presentation of Tiago Gomes)

2 WHAT IS A GESTURE? In this context, is any physical movement that can be sensed and responded by a digital system without the aid of a traditional input device such as keyboard or mouse.

3 HISTORY Sketchpad used light-pen gestures (1963). Teitelman in 1964 developed the first trainable gesture recognizer. A very early demonstration of gesture recognition was Tom Ellis' GRAIL system on the RAND tablet (1964, ARPA funded). A gesture-based text editor using proofreading symbols was developed at CMU by Michael Coleman in PLATO IV Touch Screen Terminal. Computer-based Education Research Laboratory, University of Illinois. 16-by-16 grid infrared touch panel Gesture recognition has been used in commercial CAD systems since the 1970s, and came to universal notice with the Apple Newton in 1992

4 HISTORY 1983 Video Place Myron Krueger Real time image processing Background subtraction and edge detection to create a silhouette of the user Introduce Pinch gesture

5 HISTORY 1991 Digital Desk Pierre Wellner Project a digital surface onto a physical desk Use optical and acoustic techniques to sense hands/fingers Demonstrate multi-touch concepts such as two finger scaling and translation

6 HISTORY 1992 Simon First smart phone Touch-screen

7 HISTORY 2006 Jeff Han Frustrated Total Internal Reflection Multi-touch touch-screen Created Perceptive Pixel (CNN) 5JcSu7h-I40

8 HISTORY 2007 iphone Soft touch-based interface Very smooth interaction Multi-touch capability to a limited degree

9 HISTORY 2007 Microsoft Surface Interactive table surface Capable of sensing multiple fingers and hands Capable of identifying various objects and their position on the surface

10 CHARACTERISTICS

11 TYPES OF GESTURES Touchscreen Aka TUI Single and multi-touch Free-form Wide variety of forms

12 STYLES OF GESTURES Deictic the gestures of pointing to objects in the environment Manipulative - direct relationship between the movement of the hand and the object to be manipulated Symbolic - system of gestures that use a dynamic or static dictionary Gesticulation - most natural ways of communicating, frequently used in multimodal interfaces

13 HAVING A GESTURAL INTERFACE! Why you should! More flexible Less visible hardware Hardware fits context better More natural More fun When you shouldn t Heavy data input Relies heavly on the visual Can be inappropriate for context More physically demanding

14 GOOD CHARACTERISTICS Discoverable (visibility and affordable) Trustworthy Responsive Appropriate Meaningful / Adequate Smart / Learnable Clever /Smart Playful Pleasurable Good!

15 THE ATTRIBUTES Presence (nahhh!) Position / Locale Duration Motion /Speed? Pressure (if touch) Size Form Orientation Objects /Context Number of touch/recognized points Participants Sequence

16 DRAW & DO

17 TECHNOLOGY

18 SENSORS Every system needs a gesture sensor Pressure Light Proximity Acoustic Tilt Motion Orientation

19 THE (REALLY) FREE ONES

20 THE (NOT SO) FREE ONES

21 TYPES OF TOUCHSCREENS Resistive Change in electrical field is noted and the coordinates of the point of contact are calculated

22 TYPES OF TOUCHSCREENS Capacitive A layer that stores electrical charge is placed in the monitor Some charge is tranferred to the user

23 TYPES OF TOUCHSCREENS Surface acoustic wave system Ultrasonic waves pass over the touchscreen panel When the panel is touched, a portion of wave is absorbed

24 TYPES OF TOUCHSCREENS Infrared Finger interrupts infrared light grid FTRI Frustrated Total Internal Reflection RDI Rear Diffused Illumination LLP Laser Light Plane DSI Diffused Surface Illumination

25 TYPES OF TOUCHSCREENS FTRI Frustrated Total Internal Reflection Infrared light is shined into the side of an acrylic panel Light is trapped inside the acrylic by internal reflection When a finger touches the acrylic surface this light is frustrated Picked up by na infrared camera

26 TYPES OF TOUCHSCREENS RDI Rear Diffused Illumination Infrared light is shined at the screen from below the touch surface Diffuser is placed on top or bottom of the touch surface When a finger touches the surface it reflects more light than diffuser Picked up by na infrared camera

27 TYPES OF TOUCHSCREENS LLP Laser Light Plane Infrared light from a laser(s) is shined just above the surface Laser plane of light is about 1mm tick When a finger touches it, it will hit the tip of the finger wich will register as a IR blob Picked up by na infrared camera

28 TYPES OF TOUCHSCREENS DSI Diffused Surface Illumination Uses a special acrylic to distribute the IR evenly across the surface This acrylic uses small particles that are inside the material, acting like thousands of small mirrors When a finger touches the surface it reflects more light Picked up by na infrared camera

29 AND 3D? Surface Drawing: Creating Organic 3D Shapes with the Hand and Tangible Tools Steven Schkolne, Michael Pruett, Peter Schr oder

30 HUMANS

31 THE ERGONOMICS OF HUMAN GESTURES Do: Avoid hyperextension or extreme stretches Avoid repetition Utilize relaxed, neutral positions Avoid staying in a static position No Gorilla Arm Unless

32 ARMS HANDS & FINGERS

33 HANDS & FINGERS Fingernails: Blessing and curse Fake fingernails: evil Finger oil Fingerprints (Left) Handedness Wrist support Gloves Inaccurate (when compared to a cursor) Attached to a hand (Screen Coverage)

34 SCREEN COVERAGE Avoid putting essential features or information like a label below and interface element that can be touched, as it may become hidden by the user s own hand

35 SCREEN COVERAGE Activity Zones for Touchscreen Tablets and Phone By Dan

36 TOUCH TARGET TRICKS Iceberg tips Adaptive targets

37 DRAW DO

38 PATTERNS

39 TRADITIONAL UI ELEMENTS TO WATCH OUT FOR Cursors MouseOvers and hovers Double-click Right-click Selected default buttons Undo

40 TOUCHSCREENS PATTERNS Tap to open/activate

41 TOUCHSCREENS PATTERNS Tap to select

42 TOUCHSCREENS PATTERNS Drag to move object

43 TOUCHSCREENS PATTERNS Slide to scroll

44 TOUCHSCREENS PATTERNS Spin to scroll

45 TOUCHSCREENS PATTERNS Flick to nudge

46 TOUCHSCREENS PATTERNS Pinch to shrink & Spread to enlarge

47 FREEFORM PATTERNS Proximity activates/deactivates

48 FREEFORM PATTERNS Move body to activate

49 FREEFORM PATTERNS Point to select/activate

50 FREEFORM PATTERNS Wave to activate

51 (NOT REALLY) FREEFORM PATTERNS Rotate to change state

52 (NOT REALLY) FREEFORM PATTERNS Step to activate

53 (NOT REALLY) FREEFORM PATTERNS Shake to change state

54 PROTOTYPING

55 PROTOTYPING GESTURES Low-fidelity: Paper prototyping

56 PROTOTYPING GESTURES Low-fidelity: The man behind the curtain

57 PROTOTYPING GESTURES Low-fidelity: Environments

58 PROTOTYPING GESTURES High-fidelity: Exact

59 PROTOTYPING GESTURES High-fidelity: Off-the-Shelf

60 PROTOTYPING GESTURES High-fidelity: Do it yourself

61 TURNING GESTURES INTO CODE Variables: what are you measuring? Data: get the data in from the sensor Computation: determine difference between data Patterns: what do the sums mean? Action: if a pattern is matched, do something

62 TO BEGIN WITH

63 COMMUNICATING INTERACTIVE GESTURES

64 DETERMINING THE APPROPRIATE GESTURE Four part equation The task that needs to be performed The available sensors and input devices The physiology of the human body The context This can be pretty straightforward or not The more challenging and complicated the gesture, the fewer people who will be able to perform it

65 DETERMINING THE APPROPRIATE GESTURE Context Different behaviors in different locations Avoiding accidental emotional weight Cultural issues Usability issues Avoid unintencional triggers via everyday actions Wide variation in performing gestures: need requisite variety Pick one: select then action, or selecting does action Gestures as command keys: Provide a normal means of performing the action (buttons, etc.) but have advanced gestures as shortcuts

66 HOW DOES IT WORK?

67 HOW DOES IT WORK Images captured by camera Tracking SW Id Position Area Send by protocol TUIO Tangible User Interface Objects Data processed by application and returns feedback

68 HOW DOES IT WORK

69 TRACKING Re-identify a given object for a series of video frames containing that object Tracking SW reactivision fiducial markers / basic multi-touch / cross-platform CCV multi-touch / cross-platform BBTouch multi-touch / Mac OS X

70 TUIO Open-framework that defines a common protocol and API for tangible multi-touch surfaces TUIO protocol allows transmission of na abstract description of interactive surfaces: Touch events Tangible object states

71 TOUCHPAD PROTOTYPES

72 MULTI-TOUCH TABLE PROTOTYPES