Improvisation and Tangible User Interfaces The case of the reactable

Transcription

1 Improvisation and Tangible User Interfaces The case of the reactable Nadir Weibel, Ph.D. Distributed Cognition and Human-Computer Interaction Lab University of California San Diego 1

2 Who am I? Originally from Southern Switzerland (Ticino) BSc. and MSc. in Computer Science and Engineering from ETH Zurich (Switzerland) PhD. in Computer Science (2009) from ETH Zurich Postdoctoral Fellow at UCSD in Cognitive Science and Computer Science since

3 UC San Diego Currently at home in Cognitive Science Distributed Cognition and Human-Computer Interaction Lab Affiliated with Computer Science and Engineering Ubiquitous Computing and Social Dynamics Research Group Collaborations with Calit2, Communication, School of Medicine, Political Science, Psychology, and many universities in the US and Europe 3

4 My HCI Research UbiSketch Client UbiSketch Server 4

5 Improvisation and Tangible User Interfaces The case of the reactable 5

6 Modular Synthesizers 6

7 Moog 55 Doepfer A-100 Fenix 7

8 Modern Hardware Synthesizer Sequen&al Circuits Pro- One, by Marko E,lich 8

9 Multitouch Synthesizer Magellan for ipad 9

10 Tangible Synthesizer reactable, by S. Jordà and M. Kaltenbrunner 10

11 11

12 What is Different? 12

13 reactable and Björk 13

14 What is really amazing about the reactable is how intuitive and visceral it is. The completely free way that I could access those different aspects of sound and synthesis were unlike anything I had ever experienced before or since. Demonstrating these capabilities to people who weren t as geeky as me was brilliant; they could grasp straight away the effects of what they were doing even if they didn t understand the theory. Performing with the reactable allows a different part of the brain to take over, and I found this more primal level of thought constantly led me to find new sounds, textures and experiences over and over and over throughout the whole tour. The reason Björk wanted the reactable with us was because it allows the audience to experience and understand electronic music and its performance on a whole new level. The reactable somehow allowed me to experience the audience and their response in a very different way to the other instruments in my live rig. Damian Taylor, live performer in Björk s Volta tour 14

15 The reactable 15

16 S. Jordà, M. Kaltenbrunner, M. Alonso, G. Geiger,

17 reactable controls A: sample player, sounding alone B: frequency modulator C: resonant filter, filtering the output of modulator B D: step- sequencer controlling modulator B E: sine wave oscillator modulanng filter C F: metronome (only affects A) G: tonalizer (global), corrects the notes generated by A and B 17

18 Demo reactable ipad App 18

19 Tangible User Interfaces 19

20 Physical representations are computationally coupled to underlying digital information Physical representations embody mechanisms for interactive control Physical representations are perceptually coupled to actively mediated digital representations Physical state of tangibles embodies key aspects of the digital state of a system Tangible Bits: giving physical form to digital information, making bits manipulable and perceptible. Hiroshi Ishii, MIT Media Lab,

21 TUIs affordances Social interaction Collaboration Ludic interaction Combine control and representation within a single physical artifacts Visualization capabilities enable shared experiences 21

22 Direct Manipulation 22

23 Direct Manipulation is defined by the information processing distance between user s intentions and interface facilities Directness is a property involving a relationship between the user and the task Directness is based on the qualitative feeling of engagement of effectively manipulating the object of interest Ed Hutchins, Jim Hollan, Don Norman, UC San Diego, 1985 Ben Shneiderman, University of Maryland,

24 Direct Manipulation Virtues Novices can learn quickly Experts can work extremely rapidly Error messages are rarely needed Users immediately see if their action further their goal 24

25 Distributed and Embodied Cognition 25

26 Conceptual frameworks that helps understanding interaction between human and the world (e.g. computers and interfaces) Extends cognition beyond the individual to encompass interaction with other people, materials and resources in the environment Cognition is delimited by the functional relationship among the elements that participate in it, rather then their spatial colocation. Cognition is embodied since it arises from bodily interactions with the world, and depends from perceptual and motor capabilities Cognition can not be separated by the cultural environment that an individual lives in, which shapes his/her experiences Ed Hutchins, Jim Hollan, David Kirsh, UC San Diego, A. Clark, University of Edinburgh,

27 Distributed and Embodied Cognition processes Cognitive processes may be distributed across members of a social group Cognitive processes may involve coordination between internal and external structures Processes may be distributed through time, in such a way that earlier events can transform later events 27

28 Improvisation? 28

29 Direct Manipulation Music Improvisation Distributed Cognition Tangible User Interfaces 29

30 References S. Jordà, G. Geiger, M. Alonso, and M. Kaltenbrunner, "The reactable Exploring the Synergy between Live Music Performance and Tabletop Tangible Interfaces", proc. TEI'07, pp , 2007 S. Jordà, "On Stage: the Reactable and other Musical Tangibles go Real", InternaNonal Journal of Arts and Technology, 1(3): , 2008 B. Ullmer and H. Ishii, "The metadesk: Models and Prototypes for Tangible User Interfaces", Proc. UIST'97, pp , 1997 H. Ishii and B. Ullmer, "Tangible bits: towards seamless interfaces between people, bits and atoms". Proc. CHI 97. pp , 1997 E. Hutchins, J. Hollan, and D. Norman, "Direct manipulanon interfaces", Human- computer interacnon, 1(4): , 1985 B. Shneiderman, "The future of interacnve systems and the emergence of direct manipulanon", Behaviour & InformaNon Technology, 1(3): J. Hollan, E. Hutchins and D. Kirsh, "Distributed cogninon: toward a new foundanon for human- computer interacnon research", ACM TransacNons on Computer- Human InteracNon (TOCHI), 7(2): ,2000 A. Clark, Being There: Pujng Brain Body and World Together Again. MIT Press 30

31 Further Resources TEI Conference (Tangible, Embedded and Embodied Interaction): Barcelona, Feb 10-13, (chair is S. Jordà...!) Studios Submissions, Student Design Challenge NIME Conference (New Interfaces for Musical Expression): Seoul, May 27-30, MIT, Tangible Media Group: UPF Barcelona, Music Technology Group: These slides: 31

32 Björk - Declare Independence 32