Displays Everywhere (only) a First Step Towards Interacting with Information in the real World Talk@NEC, Heidelberg, July 23, 2009 Prof. Dr. Albrecht Schmidt Pervasive Computing University Duisburg-Essen http://www.pervasive.wiwi.uni-due.de/ 2 how many digital displays have rconneyou seen today? 3 4 how many traditional displays have you seen today? how many interconnected displays will we encounter in 20 years from now? 5 6 1
Image Everything that is printed now will be an interactive and interconnected digital display in the future Ubiquitous Computing Enabling Intelligent Devices & Environments Processing cheap, fast, small, energy efficient Storage big and fast Networking global, local, ad-hoc, low-power The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it. (Mark Weiser) Displays projection, flexible materials, power consumption Sensors types, speed, accuracy, price Actuators many, computer controlled 7 8 New technologies what for? Easy involve the user :-) Focusing on user needs? User centered design process? Developing new technologies with a focus on human use Situating new technologies in the context of use Allowing technology to drive innovation Providing tools that users can use / abuse 9 10 Excurse: Maslows hierarchy of Human Needs From http://en.wikipedia.org/wiki/maslow's_hierarchy_of_needs technology creates user needs? Sketch Pad 1963 Mouse 1964 11 12 2
User Needs & Technology Drive Looks at user needs on a more general level (e.g. Maslow s hierarchy of needs) Successful designs have addressed Survival Safety Food Love Communication Recognition / admiration Allow technology to drive actual applications 13 14 Alan Kay on metaphors Magic on the screen Illusion 1.0 At PARC we coined the phrase user illusion to describe what we were about when designing the user interface. There are clear connotations to the stage, theatrics, and magic - all of which give strong hints as to the direction to be followed. For example, the screen as "Paper to be marked on" is a metaphor that suggests pencils, brushes, and typewriting. Fine as far as it goes. But it is the magic - understandable magic - that really counts. Should we transfer the paper metaphor so perfectly that the screen is as hard as paper to erase and change? Clearly not. (Alan Kay 1990, p. 199) 15 16 Illusion 1.1a Illusion 1.1b Quelle: gamespy.com Quelle: www.gemando.de Quelle: www.produktmeinung.com Quelle: www.absolute-playstation.com Quelle: www.coolest-gadgets.com Quelle: pcworld.com Quelle: redjar.org/jared/blog/ Quelle: www.welt.de/multimedia/ 17 18 3
Illusion 2.0 happens in the real world! some (technology) trends ingredients for Illusion 2.0 19 20 mobile (data) communication and electronic signage will be ubiquitous when electricity became ubiquitous there was more than the evolution of light bulbs 21 A question familiar to people who were born in the 60s and before If I only would know when the others come and where they are now will be completely alien to people born after 2000! Quelle: www.geo-reisecommunity.de 22 tracking and logging of location, activity, and interaction of people and objects will be common our understanding of privacy, and what we consider private will radically change Quelle: http://www.cartype.com/ and most of us will share this information with their friends and colleagues, just for pure convenience Quelle: [2] 23 our personal information will likely become a commodity that we trade 24 4
classical computing limitations will play very little role in the future for consumer devices magic beyond the screen Creating magical experiences with technology in our real world we will be so used to having bandwidth, memory and processing in excess that we may forget the terms Quelle: http://dcuser.net/others/ Quelle:http://www.themoviespoiler.com/ 25 26 imagination, user-illusion & magic Communicating with people on another continent Un-locking a door by one s voice Seeing and listening to a person that has lived before you and has died year ago Knowing who has touched an object you have just acquired Being able to acquire factual knowledge in a split second (basically knowing everything) I talked to a friend in Korea and saw at the same time Obama speaking in Chicago while my wife listened to Elvis imagination, user-illusion & magic Where are the borders? Never forgetting anything you see or hear for ever? Knowing what someone thinks and feels? Seeing things in front of you that relate to what you think? Triggering actions through subconscious channels Looking into the future? Time travel? 27 28 The next big thing? Making our life and the world in general more predictable Looking into the future http://img.dailymail.co.uk 29 30 5
Predicting the future: Imagine (1) each car, bus, train, truck,, object is tracked in real-time (2) each person is tracked (location, activity,, food intake, eye-gaze) in real-time (3) environmental conditions are continuously sensed globally and locally sensed (4) with have a complete (3D) model of our world (e.g. buildings, street surface, ) (5) based on the data you can predict a likely future short and long term (6) using a physics engine you can calculate the immediate future and displays provide the feedback loop 31 more realistic you buy the activity and location data of 5% of the people in Heidelberg you have access to 5% of the display time/space of all public displays in Heidelberg? what can you do? 32 Novel User Interfaces user interface paradigms Tangible and physical user interface Context-aware user interfaces and Implicit interaction Speech and gesture Physiological and emotional interaction Eye gaze interaction Interfaces ecologies (D. Bishop, 92) 33 Albrecht Schmidt, 2009 1.4.2009 34 Embedded Interaction Embedded Interaction describes the technological and conceptual phenomena of seamlessly integrating means for interaction into everyday artifacts. Technically this requires the embedding of sensing, actuation, processing and networking into everyday objects. Conceptually it requires that interaction is embedded into the user s primary task. The technical and conceptual perspective are interlinked and aim at providing optimal support for the user in interaction with the information that is not disrupting or distracting to the primary task. Design Space for Interactive Systems Implicit and explicit multimodal interaction modality command line GUI & direct manipulation gestures and speech tangible and physical UIs physiological and emotional eye gaze mode of interaction explicit implicit 35 36 6
Guidelines for designing Embedded Interaction Information when and where it is useful provide information to increase the user s ability to make an informed choice. (e.g. information is embedded at decisions points) Unobtrusive presentation - information should be presented without requiring direct attention (e.g. peripheral, ambient displays and haptic feedback) Preserve original functionality - augmented or adapted items should still behave as users expect to them at first glance; the original functionality of the artifact should not be compromised by making it interactive. Overprovisioning: allow many methods, input and output devices, and locations to achieve a task. The Internet of Things provides many opportunities and removes limits and constraints of the user. Guidelines for designing Embedded Interaction Understandability and Visibility: the design should allow easy identification of functionality and controls. Controls should be embedded in the design but still possible to spot for the user (e.g. visible and tangible) Information provision without explicit Interaction - account for providing relevant information without the user needing to explicitly trigger it. (e.g. phone s screensaver) Tackle the invisibility dilemma: transferring an object into the digital world should be done seamlessly, i.e. not change the original behavior of an object; however, there must be a perceived added value for the user. 37 38 Mobile Contextual Displays 39 40 Mobile Contextual Displays Many mobile displays E.g. people, objects, cars Source: http://www.thecoca-colacompany.com Proof of Concept Selecting and displaying content based on context and activity data Internal sensors: Bluetooth, GPS, acceleration External information: web services, etc. Source: http://www.stroeer.de Source: http://www.astramediagroup.com Prototype: Tablet PC integrated with Backpack GPS Receiver connected via Bluetooth Florian Alt Users View on Car Advertisements 41 42 7
Application Example WEB-BASED ADVERTISING CLIENT Specifying contextual advertising campaigns Context: Location Time Weather Temperature Income Population Density Traffic Density 43 A. Schmidt, F. Alt, C. Evers - Mobile Contextual Display Systems 44 A. Schmidt, F. Alt, C. Evers - Mobile Contextual Display Systems System Some findings 45 A. Schmidt, F. Alt, C. Evers - Mobile Contextual Display Systems 46 Customizing Environments Customizing Environments by just being there Basic Approach Users define their preferences (e.g. preferred music, news) by using a web interface preferences are stored in an encoded form on their mobile phones as Bluetooth friendly name Bluetooth friendly name can be read by a Bluetooth scanner Interests are decoded and content can be matched to the interests of the majority of users in the vicinity 47 48 8
Defining preferences Users define their interests on a web site (Step 2) The generated Bluetooth string (Step 3) is entered manually into the mobile phone Discovering Users Interests Mobile Phones are discovered by a Bluetooth scanner The users interests are decoded from the Bluetooth friendly name and sorted by preferences Scans are made continuously Devices which are not seen any longer are removed from the list after a certain amount of time 7/24/2009 49 7/24/2009 50 Displaying Content Depending on the interests of the majority of the users content is selected and displayed For the deployment in the cafeteria, news (derived from the Google news feed) are chosen Mobile Devices & Interactive Displays 7/24/2009 51 52 Multitouch interaction mobile interaction with interactive surfaces 9
Implicit interaction and logfiles for the real world? Integrating sensing and implicit interaction in displays 55 56 Bridging the Communication Gap V2 Automotive User Interfaces Without compromising driving performance 57 58 More Discussion Handwritten Text Input While Driving our world will more radically change as most people think there are still many hard technical challenges to solve experiments and deployment in the real world are essential we will depend on those systems 59 failure/malfunction can be fatal opt-in / opt-out what happens if all kids at school do have a memory prosthesis? Will it be acceptable to opt-out? there are so many opportunities :-) 60 10
http://auto-ui.org Questions? Comments? Ideas? Visit my websites at: http://www.pervasive.wiwi.uni-due.de/ http://albrecht-schmidt.blogspot.com/ 61 62 11