LCC 3710 Principles of Interaction Design Readings Ishii, H., Ullmer, B. (1997). "Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms" in Proceedings of CHI '97, ACM Press. Ullmer, B., Ishii, H. (2000). "Emerging Frameworks for Tangible User Interfaces" in IBM Systems Journal, Vol. 39, Nos. 3&4, 2000. Research Motivation GUI Graphical User Interface vs. TUI Tangible User Interface Tangible Interfaces Graphical User Interfaces are not always an appropriate means of accessing digital information Tangible User Interfaces enable computationally mediated interactions in physical locales and social contexts where traditional computer use may be difficult E.g. meeting spaces, living spaces, and commercial, industrial or domestic contexts Collaborative browsing of a shared media content database. Users navigate a map surface with physical pawns that represent handles to a particular person's media collection. Some Definitions Giving physical form to digital information by seamlessly coupling the dual worlds of bits and atoms Use physical spaces, surfaces, and objects as both controls and representations of digital information Enable Co-located collaborative user interactions with digital information Tangible Interaction Model Key Characteristics [Ullmer, Ishii 2000] 1. Physical representations are computationally coupled to underlying digital information 2. Physical representations embody mechanisms for interactive control 3. Physical representations are perceptually coupled to actively mediated digital representations 4. The Physical state of the interface artifacts partially embodies the digital state of the system [Ishii & Ullmer, 1997] Two main categories: Tangible interfaces: Foreground interactions with graspable objects and augmented surfaces Ambient Displays: Background information displays using ambient media GUI interaction model Model-View-Controller TUI interaction model Model-Control-Representation (physical / digital)
Taxonomy & Applications Sensing Technologies Ullmer and Ishii [2000] provide the following categorization of TUI approaches: Some approaches for interactive surfaces and object tracking for TUI systems: 1. Object tracking using tag sensing 1. Spatial Systems Spatial configuration of objects in a reference frame is computationally interpreted and augmented. E.g. Interactive tables Urban Planning I/O bulb [Underkoffler 1999] Business Modeling Sensetable [Patten 2001] Multimedia Visualization TViews [Mazalek 2002] Active or passive tags are embedded within the interaction objects, and an antenna reader is placed underneath the interaction surface. RFID can track multiple objects at once, can be difficult to scale the interaction surface size. 2. Object tracking using computer vision Color/pattern recognition of labeled objects on a horizontal surface. The video camera can look at the objects from behind or in front. Requires clear line of sight from camera to objects. Obstruction (from hands, etc.) can pose problems. 2. Constructive Systems Modular electronically instrumented artifacts are used for constructing computationally interpreted physical structures. E.g. Blocks assembly Fluid mechanics GDP (Geometry defining processors) [Anagnostou 1989] Programming AlgoBlock [Suzuki 1993] Digital Storytelling Triangles [Gorbet 1998] 3. Object tracking using acoustics Fixed receivers on surface, active sonar pingers in objects. Modulation of signals can allow several objects to be tracked at once. 4. Bare hand tracking 3. Relational Systems Logical relationships between tokens are mapped onto more abstract computational interpretations. E.g. Tokens & constraints Media manipulation mediablocks [Ullmer 1998] Databases Tangible Query Interfaces [Ullmer 2002] Can be done with computer vision, capacitive sensing on a conductive surface, light curtains (IR LED arrays), or acoustic tap-tracking. The first three techniques are good for continuous movement of multiple hands, the latter handles only discrete taps. 5. Electrical contacts Electrical contacts can be used in cases where objects are snapped together or placed at fixed locations. Application Domains Across different physical contexts Homes: personal/home management, entertainment E.g. [Shen 2003] Personal Digital Historian on DiamondTouch Work: meetings, collaborative design, strategy, planning, control E.g. [Patten 2002] Supply Chain Visualization on Sensetable Schools: teaching tool, shared creative activities Public places: kiosks, table games, online access, art projects Tabletop Systems User Interaction Scenarios by Samsung Sending and receiving messages Digital Desk Building slideshows with images and music Listening to common music playlists Downloading images from digital camera Graspable Bricks [ Wellner, EuroParc, 1991] Allows user interactions with real paper on a physical desktop to be augmented with digital information Bricks: a graspable user interface on the ActiveDesk for direct control of virtual objects, e.g. drawing Top-projected video Camera tracking from above to recognize action gestures [Fitzmaurice 1995] Digitizing pen and tablet for computer input Microphone in table surface Application functionality: - Wired objects - Two-handed interactions Technical implementation: - Sensing: Ascension Flock of Birds 6D input devices - Display: rear-projected computer screen
Urp: Urban Planning Workbench Tangible Viewpoints Storytelling engine for multi-viewpoint stories Urp: a luminous-tangible workbench for urban planning and design [Underkoffler 1999] Character-driven narratives Application functionality: Audience interaction Physical pawns used as tangible embodiments of multiple character viewpoints in a story. - Shadows and time of day - Distance measurements Story segments appear as thumbnails around the pawns, fading in and out as viewers move forward through the story. Clips are displayed on external monitor. - Light reflections - Wind simulation Clip metadata Annotations based on character viewpoint and place in narrative flow. - Site views and snapshots Computational engine Technical implementation: Spreading activation network coupled with rule-based story structure. Allows the system to adapt to viewer preferences while still providing a coherent overall narrative. - Sensing: color/pattern recognition using computer vision (poses problems with occlusion) Character viewpoints - Display: front projected graphics Breadth vs. depth and thread intersections. Tangible Spatial Narratives Creating narrative landscapes in a physical form User Interactions Spatial structured narratives Audiences can collectively reflect upon and navigate complex spatially structured and multi-viewpoint stories. Audience interaction Visual landscapes displayed on the interaction surface provide a spatial framework on which the pawns are moved. Thumbnails indicate clip segments that can be selected and played. Clock tool used to adjust story time. Clip metadata Annotations based on location, character, and time. Kids sharing personal stories Artists browsing video collections Clip retrieval Computer Clubhouse, Boston Museum of Science Haystack School of Mountain Crafts Database queries based on character, location, time. Motivation TViews Table Can we put media tables in the home? Vision A sociable interface for the home environment Sociable media interaction platform for the home environment Table designed for scalability - Different table/display sizes - Large number of objects - Many applications on one table - Connect many tables together Objects designed for portability Reality Limitations of existing sensing and display technologies How to move forward from the current prototypes? Sensing larger scalable surfaces, track more objects, portability Display provide embedded display, avoid external infrastructure - Any object on any table Interactive tagged object (puck) Glass panel and flat-screen display Table enclosure containing master control board and PC - Customize objects with external I/O devices Industry Perspective Hardware Vendor E.g. Samsung TViews table for the home Software Vendor Application packages including software + set of objects
TViews Example Applications Photo browsing interactions Shared interactions, automatic spatial organization, power of digital content Considerations for Tabletop Applications Consideration Approach Less Constrained More Constrained Objects are generic controls Objects have fixed meaning Generic I/O add-ons Custom physical shape Free object movement Coordinated object movement Directionless POV / free rotation Fixed POV / automated reorientation Independent views Coordinated views Results of actions displayed All object movements displayed Object Design Interface and Control Networked Tables Ambient Displays Ambient Devices ambientroom [Ishii, 1997] An augmented architectural space that uses ambient media to convey information at the periphery of human perception Light, shadow, sound, airflow, water movement Physical handles such as bottles and a clock to control ambient display of bits Enabling seamless transitions between background awareness and foreground activity
Tangible Query Interfaces Information Visualization & Browsing [Ullmer, 2002] Physical tokens placed within constraints can be used to query relational databases. Token rotation maps to parameter selection Token adjacencies maps to Boolean AND/OR operations Token ordering maps to result sorting Individual racks map to parenthetical groupings Example domains: Media databases, network management, and bioinformatics Triangles [Gorbet, 1998] Physical/Digital Construction Kits Physical/digital construction kit based on identical, flat, plastic triangles that connect together physically and digitally with magnetic conducting connectors Connections can trigger specific digital events Allow 2D and 3D formations Example application spaces: Storytelling, education, rapid prototyping Topobo [Raffle, 2004] 3D constructive assembly kit with kinetic memory that serves as a learning tool for dynamic structures Combination of passive (static) and active (motorized) components Quickly assemble dynamic biomorphic forms Animate those forms by pushing, pulling, and twisting them Observe the system repeatedly play back those motions (narrated) (music) Arts and Entertainment
PingPongPlus [Wisneski, 1998] Digitally enhanced ping pong that is played with ordinary paddles and balls on a reactive table. Input: Acoustic sensing used to track ball hits on table Output: Projection of moving patterns and images on the table, acoustic feedback geniebottles [Mazalek, 2001] A story of entrapment and attempted escape told by three genies who live in glass bottles Bottles used as containers and controls for digital audio information I/O Brush [Ryokai, 2004] Drawing tool that uses the physical world as a color palette. Allows users to draw with colors, textures and movements from their physical surroundings. Paintbrush with embedded video camera, lights and touch sensors Spinning Dancers and Tilting Table [Lee, 2004] Physical interactions with the digital video screen; tangible interactions serve as a direct link to the media content