Touch & Gesture. HCID 520 User Interface Software & Technology

Transcription

1 Touch & Gesture HCID 520 User Interface Software & Technology

2 Natural User Interfaces

3 What was the first gestural interface?

4

5

6 Myron Krueger There were things I resented about computers.

7 Myron Krueger There were things I resented about computers. I resented that I had to sit down to use them.

8 Myron Krueger There were things I resented about computers. I resented that I had to sit down to use them.... that it was denying that I had a body.

9 Myron Krueger There were things I resented about computers. I resented that I had to sit down to use them.... that it was denying that I had a body.... that it wasn't perceptual it was all symbolic.

10 Myron Krueger There were things I resented about computers. I resented that I had to sit down to use them.... that it was denying that I had a body.... that it wasn't perceptual it was all symbolic. I started thinking that artists and musicians had the best relationships to their tools.

11 Myron Krueger There were things I resented about computers. I resented that I had to sit down to use them.... that it was denying that I had a body.... that it wasn't perceptual it was all symbolic. I started thinking that artists and musicians had the best relationships to their tools. As early as '74, the computer could see you. Krueger, 1988

12 [O Sullivan]

13

14

15 compared to. Pictures Under Glass

16 What makes an input method natural?

17 Top 8 images for natural interaction.

18 Top 8 images for natural

19

20 What makes an input method natural?

21 What makes an input method natural? An ill-posed question

22 A reasonable definition? A user interface is natural if: The experience of using a system matches expectations, such that it is always clear to the user how to proceed, and that few steps (with a minimum of physical and cognitive effort) are required to complete common tasks. Hinckley & Wigdor

23 A reasonable definition? A user interface is natural if: The experience of using a system matches expectations, such that it is always clear to the user how to proceed, and that few steps (with a minimum of physical and cognitive effort) are required to complete common tasks. Hinckley & Wigdor Q: Is this just usability by another name?

24 It is a common mistake to attribute the naturalness of a product to the underlying input technology. A touch screen, or any other input method for that matter, is not inherently natural. Hinckley & Wigdor

25 It is a common mistake to attribute the naturalness of a product to the underlying input technology. A touch screen, or any other input method for that matter, is not inherently natural. Hinckley & Wigdor Fluent experiences depend on the context and expectations of the user, often relying on prior learning and skill acquisition.

26 Touch Input

27 Touch Input Different types of sensors

28 Touch Input Different types of sensors Resistive: Pressure connects conductive and resistive circuits. Cheap, supports single touch.

29 Touch Input Different types of sensors Resistive: Pressure connects conductive and resistive circuits. Cheap, supports single touch. Capacitive: Layer holds electric charge, changed by touch at contact point. Supports multi-touch.

30 Touch Input Different types of sensors Resistive: Pressure connects conductive and resistive circuits. Cheap, supports single touch. Capacitive: Layer holds electric charge, changed by touch at contact point. Supports multi-touch. Surface acoustic wave: Measure changes to ultrasonic audio waves. Expensive, sensitive.

31 Touch Input Different types of sensors Resistive: Pressure connects conductive and resistive circuits. Cheap, supports single touch. Capacitive: Layer holds electric charge, changed by touch at contact point. Supports multi-touch. Surface acoustic wave: Measure changes to ultrasonic audio waves. Expensive, sensitive. Optical imaging: Use IR light and cameras to track touches (appear as shadows). Multi-touch.

32

33

34 In-Air Gestures

35 Kinect Sensor We continuously reference elements in the world in ambiguous ways, yet for the most part RGB infrared we seem to convey our intentions quite well. Deixis: Reference by means of an expression infrared whose interpretation is relative to the (usually) projector extralinguistic context. camera Common methods of physical reference: pointing & placing [Clark 2003] camera Microphones Motor USB

36 Depth Cameras Structured IR light cheap, fast, accurate missing pixels, shadows Structured IR Missing pixels (not IR reflective) shadow far RGB Depth near

37 How Kinect Works Structured Light 3D Scanner

38 RGB vs. Depth for Pose Estimation RGB Only works when well lit Background clutter Scale unknown Clothing, skin colour DEPTH Depth Works in low light Person pops from bg Scale known Uniform texture Shadows, missing pixels Much easier with depth

39 Human Pose Estimation x y z θ Kinect tracks 20 body joints in real time.

40 Skeletal Tracking top view Input depth image Inferred body parts & overlaid joint hypotheses front view 3D joint hypotheses side view

41 Kinect API Input Image Data Streams: RGB, Depth images Skeletal Tracking Audio (Microsoft Speech Platform) Constraints Latency: data analysis introduces lag 86cm to 4m range Not outdoors (too much IR noise) Not too close to other Kinects (IR interference) Track 1-2 people only; full bodies must be in view (?)

42

43 Gesture Design

44 Designing Gestural UIs A designer must consider: (a) the physical sensor

45 Input Device Properties Property Sensed: position, force, angle, joints States Sensed: contact, hover, Precision: accuracy of selection Latency: delay in property/state sensing Acquisition Time: get pen, move hand to mouse False Input: accidental touches

46 On clutches and live mics Device Property State Tracked

47 On clutches and live mics Device Property State Tracked Mouse 2D Position Hover, Button-Press

48 On clutches and live mics Device Property State Tracked Mouse 2D Position Hover, Button-Press Stylus 2D Position??

49 On clutches and live mics Device Property State Tracked Mouse 2D Position Hover, Button-Press Stylus 2D Position Hover, Contact

50 On clutches and live mics Device Property State Tracked Mouse 2D Position Hover, Button-Press Stylus 2D Position Hover, Contact Touch 2D Position??

51 On clutches and live mics Device Property State Tracked Mouse 2D Position Hover, Button-Press Stylus 2D Position Hover, Contact Touch 2D Position Contact

52 On clutches and live mics Device Property State Tracked Mouse 2D Position Hover, Button-Press Stylus 2D Position Hover, Contact Touch 2D Position Contact Gesture 2D/3D Position??

53 On clutches and live mics Device Property State Tracked Mouse 2D Position Hover, Button-Press Stylus 2D Position Hover, Contact Touch 2D Position Contact Gesture 2D/3D Position?? In-air gestures may involve a live mic, increasing chances of false positives and false negatives.

54 On clutches and live mics Device Property State Tracked Mouse 2D Position Hover, Button-Press Stylus 2D Position Hover, Contact Touch 2D Position Contact Gesture 2D/3D Position?? In-air gestures may involve a live mic, increasing chances of false positives and false negatives. Clutch: differentiate actions intended to drive the computing system from those that are not.

55 Managing a live mic Reserved Actions Design gestures that will not be triggered unless specifically desired by the user. Reserved Clutches Use a special gesture to indicate that the system should now monitor for input commands. Multi-Modal Input Use another modality such as buttons or voice input to engage tracking by the system.

56 Designing Gestural UIs A designer must consider: (a) the physical sensor

57 Designing Gestural UIs A designer must consider: (a) the physical sensor (b) the feedback presented to the user

58 Designing Gestural UIs A designer must consider: (a) the physical sensor (b) the feedback presented to the user (c) ergonomic and industrial design

59 Designing Gestural UIs A designer must consider: (a) the physical sensor (b) the feedback presented to the user (c) ergonomic and industrial design (d) the interplay between all interaction techniques and among all devices in the surrounding context

60 Designing Gestural UIs A designer must consider: (a) the physical sensor (b) the feedback presented to the user (c) ergonomic and industrial design (d) the interplay between all interaction techniques and among all devices in the surrounding context (e) the learning curve

61 How to design gestures? Observation: generate potential gestures by observing (and participating in) situated activity.

62 How to design gestures? Observation: generate potential gestures by observing (and participating in) situated activity. Participatory design: have representative users generate potential gestures for you.

63 How to design gestures? Observation: generate potential gestures by observing (and participating in) situated activity. Participatory design: have representative users generate potential gestures for you. One methodology [Wobbrock et al 2009] 1 Show participant start and end states of UI 2 Participant performs gesture for that effect 3 Analyze collected gestures from population

64 How to design gestures? Observation: generate potential gestures by observing (and participating in) situated activity. Participatory design: have representative users generate potential gestures for you. One methodology [Wobbrock et al 2009] 1 Show participant start and end states of UI 2 Participant performs gesture for that effect 3 Analyze collected gestures from population Must still consider interplay with task/context

65 User-Designed Gestures

66

67 Discussion

68 Discussion Questions Kristen: I found it interesting how the study was based around an effect/cause testing model (having users perform actions they think would result in the shown effect). Is this a popular method in other areas of hci? Lauren: I think showing users an effect, and then asking them to perform its cause, is an incredibly intelligent way to gather data on natural, intuitive gestures This method will undoubtedly decrease the cognitive load placed on the user and allow them to act naturally while interacting with the device, instead of requiring the user to memorize and recall several different learned gestures of interaction, creating a much more enjoyable user experience.

69 Discussion Questions Sara: How might gestures differ significantly across cultures (especially in ones that read from right to left or tend to prefer using the right hand instead of the left, for example)? Taysser: This reading says that users are not designers; therefore, care must be taken to elicit user behavior profitable for design what effect design has on developing a user-defined gesture set? This reading doesn t provide an example that clarifies this statement.

70 Discussion Questions Stuart: Morris, Saponas, & Tan posit that a glasses-shaped display is "the best candidate for always-available output in the near-term future." This paper was written in 2011, and since then, Google Glass has flopped tremendously. What went wrong? Why are glasses-based optical interfaces not as great an idea as they thought? Rick: With the announcement of Microsoft HoloLens, it seems that users can interact with virtual objects in their field of vision. I am very curious that whether the interaction mode of HoloLens will be more similar to a touch system or an in-air gesturing system?

71 Discussion Questions Acacio: Although the problems mentioned are somewhat obvious, I think I found a hidden conclusion where multi modal inputs make the most sense from a usability standpoint and have been the most successful in real world usage. In an ideal situation, each mode complements the other, adding functionality like the article mentions in the case of the keyboard and mouse. Another advance that I have found hard to live without is my apple magic mouse that incorporates touch as well. It has a clutch so when the mouse is actively moving the touch is disabled. This article opened my eyes to looking at these things around me.

72 The Kinematic Chain

73

74

75

76

77 Yves Guiard: Kinematic Chain Asymmetry in bimanual activities Under standard conditions, the spontaneous writing speed of adults is reduced by some 20% when instructions prevent the non-preferred hand from manipulating the page Non-dominant hand (NDH) provides a frame of reference for the dominant hand (DH) NDH operates at a coarse temporal and spatial scale; DH operates at a finer scales.

78 Proxemics

79

80

81

82

83 Proxemics Proxemics is the study of measurable distances between people as they interact. [Hall 1966] Taxonomy of Distance Intimate: embracing, touching or whispering Personal: interaction among friends / family Social: interactions among acquaintances Public: distance used for public speaking

84

85 We continuously reference elements in the world in ambiguous ways, yet for the most part we seem to convey our intentions quite well. Deixis: Reference by means of an expression whose interpretation is relative to the (usually) extralinguistic context. Vogel & Balakrishnan, 2004 Incorporating Marquardt et al, 2011 Common methods of physical reference: pointing & placing [Clark 2003] Proxemics

86

87 Final Thoughts Leverage the unique opportunities provided by a particular input technology. Don t shoehorn new modalities where old techniques excel. Consider perceptual vs. symbolic input. Prevent accidental (vs. intentional) input via unambiguous design and/or clutching. Respect existing conventions of spatial reference and social use of space.