NUI. Research Topic. Research Topic. Multi-touch TANGIBLE INTERACTION DESIGN ON MULTI-TOUCH DISPLAY. Tangible User Interface + Multi-touch

Transcription

1 1 2 Research Topic TANGIBLE INTERACTION DESIGN ON MULTI-TOUCH DISPLAY Human-Computer Interaction / Natural User Interface Neng-Hao (Jones) Yu, Assistant Professor Department of Computer Science National ChengChi University HCI NUI December. 2, Research Topic Multi-touch Human-Computer Interaction / Natural User Interface NUI Tangible User Interface + Multi-touch 1 Jeff Han. TED Buxton & Smith. A multi-touch three dimensional touch-sensitive tablet. CHI '85

2 5 6 FTIR (Frustrated Total Internal Reflection) MS Surface DI (Diffuse Illumination) MS Surface 2.0

3 9 10 ITSO (Integrated Through Screen Optics) i-m-top <=>?=)!!" 58)!!" 9-""#':(");$##*#.*/"')!#*+",-*# 01)2'3"#')4)5!"#$%&"#'()!#*+",-*# 01)0((63$7'-*#)48 Izadi et al. ThinSight: a thin form-factor interactive surface technology. Communications of the ACM (2009) PerceptivePixel / Jeff Han

4 13 Capacitive Sensing Multi-touch DIY NUI Touch_Technologies_v1.01.pdf TUIO.org TuioPad ardumt Direct Manipulation Tangible User Interface (TUI) 1 Ishii & Ullmer, Tangible bits: towards seamless interfaces between people, bits and atoms. CHI 97

5 17 18 Urp SandScape Underkoffler, J., and Ishii, H. Urp: a luminous-tangible workbench for urban planning and design. In Proc. of CHI 1999, Piper et al., Illuminating Clay: a 3-D tangible interface for landscape analysis, CHI Ishii et al., Bringing clay and sand into digital design: Continuous tangible user interfaces. BT Technology Journal 22, 4 (2004), TinkerSheets metadesk Zufferey et al., TinkerSheets: Using paper forms to control and visualize tangible simulations in Proceedings of TEI09, pp , NY: ACM, 2009 Ullmer & Ishii, The metadesk: models and prototypes for tangible user interfaces. In Proc. of UIST 1997,

6 Hybrid surfaces (TUI + multi-touch) 3 Kirk et al. Putting the physical into the digital: issues in designing hybrid interactive surfaces. BCS-HCI ' reactable Lumino Jordà et al., The reactable: exploring the synergy between live music performance and tabletop tangible interfaces. In Proc. of TEI 2007, Baudisch et al., Lumino: tangible blocks for tabletop computers based on glass fiber bundles. In Proc. of CHI 2010,

7 25 26 Multi-display map touring system for interactive tabletop surfaces Demo video Street view Pawn (Tangible Widget) Map view! 28 Interior design system PART 2 DESIGNING TANGIBLE INTERACTIONS ON MOBILE DEVICES - TUIC & CLIP-ON

8 29 30 Vision-based sensing Mutual Capacitance Touch sensing MS Surface 1.0! Simulate finger touch Extend a user s touch If we can simulate capacitance change that is similar to a finger touch, we can use it to create tags that can be sensed electrodes (copper film) Jun Rekimoto, SmartSkin: An Infrastructure for Freehand Manipulation on Interactive Surfaces. CHI 02

9 33 34 Our solution Tag design based on spatial domain 3 capacitance tag designs: orientation d 2D Frequency Hybrid C0 B0 B1 B2 C1 d B3 B4 B5 B6 B7 B8 + QR code C2 9-bit TUIC-2D tag Implementation: TUIC-2D orientation d C0 C1 d B0 B1 B2 B3 B4 B5 B6 B7 B8 C2 9-bit TUIC-2D tag

10 37 38 TUIC-2D Advantages Passive (un-powered) Instant on Easy to build 4-bit TUIC-2D tag 5mm Tag design based on time domain Limitations Contact point size = 5mm (finger size) Distance between points = 5mm 5mm T Requires at least 3 touch points (3 registration points + payload bits). ipad supports an maximum of 11 touch points. W Implementation: TUIC-frequency

11 41 42 TUIC-frequency Hybrid: Combining spatial and frequency Advantages Single touch point Small size Programmable ID Limitations Requires power No support for orientation and movement Startup delay (= measurement window) frequency tag positioning-points TUIC-Hybrid Advantages Programmable ID Robust movement and orientation tracking Limitations Requires power Startup delay (= measurement window)

12 45 46 Discussion Discussion: replacing PIN numbers Tag design TUIC-2D TUIC-f TUIC-Hybrid Max # of IDs 2 n n: # of payload bits n m n: # of distinct intervals m: # of frequency tags Minimum Touch points Size Proportional to the minimum touch points and the resolution of touch sensors Power requirement Passive Active Orientation Yes No Yes Moveable Yes No Yes Robustness Instant on Startup delay = measurement window size Important features Unpowered Programmable ID. 48 Problems Using Touchscreens Clip-on Gadgets: Expanding Multi-touch Interaction Area with Unpowered Tactile Controls 1. Fat finger problem : Obscure the display

13 49 50 Problems Using Touchscreens 2.Lack of tactile feedback, requiring visual attention Problems using Touchscreens Occlusion + lack of tactile feedback Related works Reduce fat finger problem: Offset Cursor 1, Shift 2 Backside touch 3 Related works Augment tactile feedback: Vibration actuators TeslaTouch (low current) 1 Potter, R., Weldon, L., Shneiderman, B. (1988). Improving the accuracy of touch screens: an experimental evaluation of three strategies. In Proc. CHI 88, Vogel, D. and Baudisch, P. (2007). Shift: A Technique for Operating Pen-Based Interfaces Using Touch. Proc. CHI Baudisch, P. and Chu, G. (2009) Back-of-device interaction allows creating very small touch devices. In Proc. CHI Poupyrev, I. Maruyama, S. and Rekimoto, J (2002). Ambient touch: designing tactile interfaces for handheld devices. In Proc. UIST 02, Olivier Bau, Ivan Poupyrev, Ali Israr, and Chris Harrison (2010). TeslaTouch: electrovibration for touch surfaces. In Proc. UIST 10,

14 53 54 On-screen physical controllers Off-screen physical controllers Connect by wiring (Dock) wireless (Bluetooth) Tactile+Plus still occlude the display Fling require battery! Our goals Our solution: Clip-on Gadgets Minimize occlusion Tactile controls Simple (un-powered) 3~5mm Portable

15 57 58 Our solution : Clip-on Gadgets Our solution : Clip-on Gadgets buttons (conductive rubber) contact points Demo: clipper-style Demo: form-fitted

16 61 62 Software Architecture Software Architecture Application Button events (e.g. D-pad: left, right, up, down AB-button: A, B) 1 Clip-on Toolkit Event Dispatcher Clip-on object Clip-on D-pad Detector Object Clip-on AB-button Event Listener Touch events ios SDK Two connection modes: fixed-position mode Two connection modes: dynamic-position mode

17 65 66 Pairing procedure for dynamicposition mode Beyond the buttons Examples: Eye-free dialer Examples: Piano keyboard

18 70 Future work: Occlusion-free The touch sensing area can be off-screen, or on the back of devices. 72 Open Source SDK PART 3 TUIC+: ENABLING BIDIRECTIONAL TANGIBLE INTERACTION ON MULTI-TOUCH DISPLAYS

19 73 74 Motivation Extend Uni-direction to Bi-direction Bill Buxton s collections: Phantom Chess Future work Demo Bi-directional tag

20 77 78 Future work Entertainment music box dancing doll Future work Education storytelling toy!! Future work Advertising Promotional gifts THANK YOU Jones Yu