Augmented and Virtual Reality

Transcription

1 CS-3120 Human-Computer Interaction Augmented and Virtual Reality Mikko Kytö

2 From Real to Virtual [1] Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems, E77-D(12),

3 Augmented Reality (AR) According to definition: 1. AR application must mix real and virtual 2. Registered in 3D with physical world 3. Interactive in Real Time 3

4 Why AR? Enables providing augmented information and guidance in the physical context Enables investigating virtual 3D models in physical environment eographic.com/img/ bigidea/augmented/aug mented.jpg 4

5 Augmented Reality (AR) According to definition: 1. AR application must mix real and virtual DISPLAY TECHNOLOGIES 2. Registered in 3D with physical world TRACKING TECHNOLOGIES CAPTURING TECHNOLOGIES 3. Interactive in Real Time INTERACTION TECHNOLOGIES 5

6 AR display technologies Modalities Visual Aural Haptic Kinaesthetic Olfactory 6

7 Development of AR Displays ? 7

8 Hand-held displays 8

9 Wearable displays Two approaches 1. Optical see-through 2. Video see-through 9

10 Basic optical principle of near-eye display 10

11 Display comparison Optical see-through + Better visibility of surrounding real world Video see-through + Video image can be controlled à Matching graphics and video feed + Lighter - Perception of real world is distorted - Brightness (current displays difficult to use outdoors) - Latency 11

12 Social acceptability of wearable displays Lack of knowledge what the wearer is doing Glassholes Still bulky and interfere with face-to-face interaction Lighter and less noticeable wearable displays will open a lot new possibilities for AR 12

13 Spatial augmented reality Using projectors Augmented Climbing Wall 13

14 Future display technologies Retinal display Photons projected into eye Bright (works also outdoors in a day light) Challenge in viewing frustum Contact lens display Big challenges (e.g., power and resolution) to be solved Microvision Virtual Retinal Display 14

15 Augmented Reality (AR) According to definition: 1. AR application must mix real and virtual DISPLAY TECHNOLOGIES 2. Registered in 3D with physical world TRACKING TECHNOLOGIES CAPTURING TECHNOLOGIES 3. Interactive in Real Time INTERACTION TECHNOLOGIES 15

16 Tracking technologies Active (sending and receiving) Mechanical, Magnetic, Ultrasonic GPS, Wifi, Bluetooth, cell location Passive Inertial sensors (Compass, gyroscope, accelerometer) Optical (markers or natural features) Hybrid Combination of sensors 16

17 Active tracking Mechanical tracking: Measuring joints of mechanical arms Magnetic tracking: Measuring current generated from moving coil in magnetic field Ultrasonic tracking: Measuring changes of travel time of ultrasonic waves Wifi, Bluetooth, GSM cell location: Changes in signal strength GPS: Changes in radio signal travel time MOVEMENT RANGE INCREASES 17

18 Tracking with Inertial measurement units (INU) Accelerometers, gyroscopes (and sometimes magnetometer) combined + Good for tracking orientation + Cheap and small - Bad for tracking absolute position over time - Drift, requires calibration 18

19 Optical tracking (with markers) 19

20 Optical tracking (markerless) Tracking based on local image features (for example SURFfeatures) ch/~surf/download.html 20

21 Optical tracking (unknown environment) SLAM (Simultaneous Localisation and Mapping) Derived from robotics: Where am I? 21

22 Inertial and optical tracking combined Good accuracy Support by operating systems Android: ARCore SDK ios: ARKit part of ios 11 22

23 Scene capturing technologies Depth sensors (like Kinect) needed for improving interaction between virtual and physical Solving what is between image features (e.g., textureless areas) Solving changes in the scene (e.g., changes in occlusion) 23

24 Augmented Reality (AR) According to definition: 1. AR application must mix real and virtual DISPLAY TECHNOLOGIES 2. Registered in 3D with physical world TRACKING TECHNOLOGIES CAPTURING TECHNOLOGIES 3. Interactive in Real Time INTERACTION TECHNOLOGIES 24

25 Interaction technologies How do I interact with that thing? 25

26 Interaction technologies 3D gestural input Considered to be natural, but how natural it is to move your hands in the air? Can become tiring Haptic feedback? Tangible interaction Data gloves Head movements Eye movements Voice input 26 5DT data gloves:

27 3D gestural input Device Distance range (m) Purpose Leap m Hand tracking Kinect m Skeleton tracking, 3D reconstruction Laser scanners 1 approx. 200 m 3D reconstruction 27

28 AR experience challenges Display technology Especially in terms of depth perception, 3D stereoscopic displays used mainly in HMDs Tracking technology Capturing technology Lack of interaction between real and virtual (e.g., occlustions and shadows) 28

29 Applications so far Applications in industry Maintenance (e.g., X-Ray Visualisation), Design, Training Consumer applications Games (e.g., Pokemon GO), Social media Navigation, AR Browsers (Layar, Wikitude,..), language Interior design Advertising Education (e.g., medical training) 29

30 Social media Snapshat Cinco de Mayo lens Facebook Camera masks Not real time (yet) 30

31 X-RAY Visualisation

32 Maintenance 32

33 Interior design 33

34 Advertising 34

35 Possible Future applications Social interaction AR Sci-Fi video, Sight: AR-glasses concept videos: 35

36 From Real to Virtual [1] Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems, E77-D(12),

37 Augmented Virtuality com/product/vr 37

38 Virtual Reality The environment is purely virtual Two approaches 1. Cave automated virtual environment (CAVE) 2. Head-mounted display (HMD) 38

39 Key Characteristics of Virtual Reality Fast and accurate tracking More controlled environment than in AR 3D Stereoscopic display Wide field-of-view display 39

40 Virtual reality experience Immersion Presence Feeling of being there 40

41 Possibilities for VR Games Simulators Remote collaboration 360 Video recording 41

42 Challenges in VR experience Visual discomfort Accommodation and convergence mismatch, wearing display Motion sickness Mismatch between visual and vestibular systems Limited resolution Oculus Rift, Consumer Version 1080x1200, Human eye about ten times more Jason Geng, "Three-dimensional display technologies," Adv. Opt. Photon. 5, (2013) 42

43 Challenges in VR experience How to implement moving in virtual space? 43

44 Challenges in VR experience Content creation Virtual content creation 3D software (e.g., Unity3D) Content from real environment 360 Video recording 44

45 VR AR 45

46 Next week s lecture Prof. Antti Oulasvirta will talk about computational user interfaces Note: Location is AS1! 46

47 Conclusion Do not believe in hype, there is still a large gap between real and virtual environment In mixed reality the key is a meaningful interplay between physical environment and virtual environment 47

48 Further reading Course book (Benyon s) Chapter 13: pp Furht, B. Handbook of Augmented Reality, 2011, Springer Mark Billinghurst s lecture notes in SlideShare: =header&q=comp+4010&ud=any&ft=all&lang=**&sort= 48