Introduction to Tangible Interaction. Prof. Sergi Jordà

Transcription

1 Introduction to Tangible Interaction Prof. Sergi Jordà

2 Index Part Part Part Part Part I: II: III: IV: V: Defining TI & TUIs Thinking about TUIs Multitouch devices Tabletop devices Exploring and designing TUIs

3 Part I: Defining TUIs Defining Tangible Interaction Extending the Model-View-Controller Historical examples TI vs. AR

4 Tangible Interaction and Direct Manipulation Tangible interaction as an extension of Direct Manipulation (Shneiderman 83). Direct manipulation: HCI style that allow users to directly manipulate objects presented to them, using actions that (loosely) correspond to the physical world, assuming that realworld metaphors (e.g. desktop) for objects and actions, can make interfaces easier to learn and use. DM is closely associated with WIMP/GUI interfaces. WIMP/GUI [always] incorporate DM concepts, however DM does not imply the use of [w]indows DM can be more than [g]raphical (e.g. tactile control & feedback; sonic control & feedback vs. just visual feedback in GUIs)

5 Tangible Interaction? Mice and most alternative pointing devices are controlled with the hands Thus, they re tangible! So what s so novel about tangible interaction?!?

6 Definitions A Tangible User Interface (TUI) is a physical representation of digital information that one is able to touch in order to manipulate the digital data TUIs are UIs that augment the real physical world by coupling digital information to everyday physical objects and environments (Ishii & Ullmer 1997) (Ishii & Ullmer 97) is THE seminal tangibles paper, although it builds from ideas and concepts of (Fitzmaurice, Ishii & Buxton 95) and others

7 Tangible Interaction In TI, the paradigm is that of manipulating directly information, and not to use a controller to manipulate a display In TI, input and output tend to be strongly coupled This is why the Abacus is a reference often mentioned (e.g. Ishii, 97)

8 Ok with the abacus. but what about that? every speaker is also a microphone! (and vice versa) (Buxton 2005)

9 The traditional Model-View-Controller Control (up) is completely separated from Representation (down)

10 The Model-Control-Representation(p+d) suits better the TI paradigm

11 Not exactly TUIs. So what/how is a TUI!?

12 A precursor: The marble answering machine (Bishop 92, Royal College of Art, Interaction Design)

13 Wacom prototype (Fukuzaki 1993) Probably one of the first tabletop interfaces that were more than mere horizontal multitouch boards (although not credited by Ishii, this works was, according to Buxton, the originator of the BuxtonFitzmaurice-Ishii trend on TI)

14 A pioneer: Bricks (Fitzmaurice, Ishii & Buxton 1995) Bricks, introduces the concept of Graspable User Interfaces and explores two-hand interaction by means of bricks that can be coupled or attached to virtual object It works on the Active Desk, a desktop surface with rear projection, modelled after a drafting table (e.g. 30 degree angle)

15 Urban Planning (Underkoffler 1999)

16 TUI vs. AR There are also intersections between Tangible Interaction and Augmented Reality, since they both combine physical and virtual In AR, the focus is on how the virtual augments/adds up to the real In TI, the focus is on how the real allows us to better manipulate information (which tends to be virtual) Still, the frontiers, are not always clear: Golan Levin & Zach Lieberman, Manual Input Sessions YouTube demo, explanation (AR? TI?)

17 [Annex] Some augmented Reality YT Video Links From VR to AR (~2000) Paris + Phantom (AR version of the Cave) AR Indoor with fiducials Gizmondo Augmented Reality Game Total Immersion - Demo 2007 More fiducials / general tutorial Augmented Reality Encyclopedia (augmented books) AR Indoor with no fiducials logitech orbit cam (indoor, face recognition) Physics and Augmented Reality - Part 2 (indoor, surface recognition) de Pong Game I - Art Rock festival 2006 ( indoor -delimited-, contour recognition) AR with mobiles Arcade Reality (mobile) (outdoor, no recognition) Invisible Train (indoor, yes recognition) AR outdoor Augmented Reality GPS Tour Beyond Tomorrow (outdoor, GPS tour) ARQuake (2) Not interactive / using projectors Art: The Augmented Sculpture Project

18 Part II: Thinking about TUIs Expanding definitions Embodiment & metaphors Tokens & Containers

19 Extending the TUI concept / embodyment Several authors prefer a wider definition (than Ishii s) e.g. (Fishkin 2004) proposes a 2D taxonomy (embodiment & metaphor) that helps to determine the tangibility of a system (it increases with both attributes) Embodiment: How closely tied is the input focus to the output focus? To what extent does the user think of the states of the system as being inside the object they are manipulating? (in full embodiment, input and output devices coincide)

20 TUIs & Metaphors (1) We know the importance of metaphor in orthodox wimp interaction. Why is metaphor particularly appropriate for TUIs? (Fishkin 04) If an interface is made physically tangible, a whole realm of afforded metaphors becomes available: shape, size, color, weight, smell, texture of objects How does that apply-evaluate with TUIs? How analogous are the system effect of a user action and the real-world effect of similar actions?

21 TUIs & Metaphors (2) (Fishkin 04) distinguishes several types of metaphors for TUIs Metaphor of noun, appeal to the shape and physical properties of an object (e.g. Windows/desktop, buildings-objects in URP) Metaphor of verb, appeals to motion and actions on an object (e.g. moving a building-object in URP, shaking in order to ) Both (noun+verb) examples: dropping into the wastebasket (in GUIs) opening the bottle in Music Bottles URP (in TUIs) Full, means no analogy is needed, because virtual & physical systems seem to fully coincide (IO Brush)

22 Taxonomies: nouns & verbs, tools & containers Underkoffler & Ishii (1999) also propose to classify objects as nouns or verbs, depending whether they are more associated with data or with actions e.g. in URP, the buildings would be nouns, where the clock is more related to a verb, since its use is for setting the clock thus moving the sun (Holmquist et al. 99) and (Ulmer 2002) distinguish between 3 types of TUI artifacts: tools, tokens & containers Containers behave as if they contained the information (according to Fishkin they tend to be very embodied, because the data is considered to be within the object) Tools manipulate information (their metaphors are more verby) (cfg. The toolbox in ActiveDesk)

23 an informal TUI saga Some names have already appeared several times in this talk Buxton (Univ. Toronto) Underkoffler (Toronto/MIT/Hollywood) Ishii (MIT MediaLab) Ulmer Patten

24 From (Fitzmaurice, Ishii & Buxton 1995) In general, the Graspable UI design philosophy has several advantages: It encourages two handed interactions; shifts to more specialized, context sensitive input devices; allows for more parallel input specification by the user, thereby improving the expressiveness or the communication capacity with the computer; leverages off of our well developed, everyday skills of prehensile behaviors for physical object manipulations; externalizes traditionally internal computer representations; facilitates interactions by making interface elements more "direct" and more "manipulable" by using physical artifacts; takes advantage of our keen spatial reasoning skills; offers a space multiplex design with a one to one mapping between control and controller; and finally, affords multi-person, collaborative use.

25 Recapitulation: some TI Control Consequences The affordances of physical objects are richer than what virtual handles afford through conventional direct manipulation techniques In TI, control tends to be spatially multiplexed In TI, actions tend to be less constrained in time and in order (than with GUIs): anything can happen at anytime, anywhere, and simultaneously TUIs allow thus multidimensional control and two-hand interaction TUIs are thus ideal for collaborative work, as more than one person is able to manipulate the data TUIs are often associated with (but should not be restricted to) multitouch control and devices

26 Part III : Multitouch Definitions Brief history (Myron Krueger & more) Examples Standards (de facto) & clichés

27 Apple did not invent multi-touch!! (neither did Microsoft!)

28 Discrete multitouch?! Plus a mouse and a QWERTY keyboard, or a MIDI keyboard and a pedal Many devices (or combination of them) already allow multi-hand (multi-feet!) interaction

29 Multitouch / Multiuser (Diamond Touch) It s not the same to have 2 (or more) cursors than to do things with several fingers / both hands Yet sometimes, having 2 (or more) cursors is what we wanted (e.g. because there are 2 users) Diamond Touch (from Mitsubishi, 2001) is the only system that distinguishes between dif. users

30 Multiuser Devices / Size Matters (Buxton 2008) claims that the shift from the slate (~9 x 12 cm) to the blackboard in classrooms ca. 1850s could still arguably be the latest biggest change in education methods Yet, they both only differ in size, since they share exactly the same technology a slate and a blackboard

31 Points vs. Gestures and more Myron Krueger s Video Desk (1983) Although not sensing directly touch, Krueger s VideoDesk (1983) set the direction for the no-mice, no-gloves, no-glasses rich, free and two-hands interaction

32 [insert 1] Myron Krueger VideoPlace ( ) VideoDesk (left) comes from Krueger s earlier work VideoPlace which predates all the now so common silhouette or shadowinteraction (right) lately popularized by Sony s Eyetoy

33 [insert 2] Myron Krueger VideoPlace+VideoDesk Krueger also combined both models, creating novel telepresence environments

34 Krueger s links and bibliography Krueger, Wikipedia Krueger, Myron, W. (1983). Artificial Reality. Reading, MA: Addison-Wesley. Krueger, Myron, W. (1991). Artificial Reality II. Reading, MA: Addison-Wesley. Krueger, Myron, W., Gionfriddo, Thomas., & Hinrichsen, Katrin (1985). VIDEOPLACE - An Artificial Reality, Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 85),

35 Some multitouch research & implementations (80s-90s) 1984: Multi-Touch Screen (Bell Labs, Murray Hill NJ) 1985: Multi-Touch Tablet (Input Research Group, University of Toronto) : Bimanual Research (Alias Wavefront Toronto) ( demo video) 1995/97: Active Desk (Input Research Group / Ontario Telepresence Project, University of Toronto) (right pict.)

36 (Fitzmaurice, Ishii & Buxton 1995) Study two-hand control in many (non digital) tasks and export the conclusions to TUI

37 The multitouch vogue ( ) Jazz Mutant s Lemur (2004) (pics below) Jeff Han (NYU) The video that triggered the multitouch hype (2006) Video demo 2, conference video Read more, B.Buxton on Multitouch

38 Multi vs. Single touch selection From screens to tables In trad. GUIs and WIMP, selection is unique at every instant (selection N selection N-1 is unselected) Multiple selection seems related to temporary (or permanent) attachment: a selection could be made persistent without having to be kept physically connected This is where objects, and thus flat surfaces, appear There are other benefits from using objects instead of naked hands and fingers Tactile feedback is essential; it provides a way of safeguarding user intent. The bricks supply tactile confirmation and serve as a visual interaction residue Hand gestures lack very natural delimiters for starting and stopping points. This makes it difficult to segment commands and introduces lexical pragmatics. In contrast, the affordances of touching and releasing a brick serve as very natural start and stop points

39 Part IV : Tabletop (Orientation matters) History, examples, technologies Standards (de facto) & clichés

40 Tabletop Interfaces are promising Represent a natural medium for work, discussion, fun Promote user collaboration Allow two-hand manipulation and infinite simultaneous pointers Provide ample room for organising objects Physical objects can be more than just input devices, they can combine control and representation, maximizing bandwidth and minimizing indirection

41 Tabletop Interfaces are fashionable Specially since Microsoft announced (29th May 2007) the imminent production of the Surface, and Apple the IPhone but Mitsubishi, Panasonic, Philips, Samsung, Sony have shown their own interactive table prototypes for at least two years

42 Wacom prototype (Fukuzaki 1993) Probably one of the first tabletop interfaces that were more than mere horizontal multitouch boards (although not credited by Ishii, this works was, according to Buxton, the originator of the BuxtonFitzmaurice-Ishii trend on TI)

43 The SenseTable / the Audiopad (Patten 2001)

44 Diamond Touch (2001)

45 Microsoft s Surface ( ) The beggining: PlayAnywhere (Wilson 2005) Video The Surface official site The Official video demo The official video parody

46

47 Tabletop Interfaces are boring? Why do so many prototypes still rely on zooming on Google Maps or arranging photo collections? Myron Krueger`s Videodesk 1987

48 Part IV: Designing & Experimenting with TUIs TUIs: Control & Properties TUIs & board-games More examples TUIs pros & cons

49 TUIs Pros Control and interaction tends to be more natural (ideally, technology becomes transparent ) Favors exploration Can deal well with simple specific problems Phy. Objects are persistent TUIs Cons More physical controllers are needed Scalability problems (limited interaction size and limited objects/actions) limits available complexities Phy. Objects are persistent Difficult to create abstract/generic interfaces Diff. to store/retrieve states

50 Some TUI properties Persistence Connectivity Explorability Real-time interaction

51 (Appendix) Persistent and unmovable? The Actuated Workbench (Pangaro, DaynesAminzade, Ishii 2000) James Patten s Pico Webpage & YouTube Video

52 Generic vs. particular solutions TUIs and Games Tokens have been traditionally used in games Tokens in games tend not to be generic Boards in games tend not to be generic Board games is a good area to seek for inspiration, perhaps better that traditional WIMP interaction

53 Inspirations

54 Games: examples and discussion Chess Monopoly Poker

55 Conclusion / Some Prophecies It took more than 25 years for the mouse to become ubiquitous (between Engelbart and Windows 3.1) multitouch technologies have still some time left ;) New technologies will soon allow flat screens to be bi-directional prototypes already exist (see video after 4 30 ), in which each pixel is sensitive, like screen+scanner) tabletops will become cheap and portable Everything is best for something and worst for something else (Buxton) TI will not be THE mice-predator

56 Bibliography Fitzmaurice G, Ishii H, Buxton W (1995) Bricks: laying the foundations for graspable user interfaces. In: Proceedings of the CHI 95 conference on human factors in computing systems, Denver, Colorado, May 1995, pp Fukuzaki, Y. (1993) Electronic Pen According to the BTRON Guideline and its Background, Tronware, Vol. 4, Personal Media Publishers, Japan, pp (in Japanese). Ishii H, Ullmer B (1997) Tangible bits: towards seamless interfaces between people, bits, and atoms. In: Proceedings of the CHI 97 conference on human factors in computing systems, Atlanta, Georgia, March 1997, pp