Augmented and Virtual Reality - PDF Free Download

CS-3120 Human-Computer Interaction Augmented and Virtual Reality Mikko Kytö 7.11.2017

From Real to Virtual [1] Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems, E77-D(12), 1321 1329. 2

Augmented Reality (AR) According to definition: 1. AR application must mix real and virtual 2. Registered in 3D with physical world 3. Interactive in Real Time 3

Why AR? Enables providing augmented information and guidance in the physical context Enables investigating virtual 3D models in physical environment http://ngm.nationalg eographic.com/img/ bigidea/augmented/aug mented.jpg http://www.nlr.org/article/x-lab-frontrunner-augmentedreality-mro-aerospace/ 4

Augmented Reality (AR) According to definition: 1. AR application must mix real and virtual DISPLAY TECHNOLOGIES 2. Registered in 3D with physical world TRACKING TECHNOLOGIES CAPTURING TECHNOLOGIES 3. Interactive in Real Time INTERACTION TECHNOLOGIES 5

AR display technologies Modalities Visual Aural Haptic Kinaesthetic Olfactory 6

Development of AR Displays 1990 2000 2010 2020? 7 http://foonacha.blogspot.fi/2015/06/the-dangers-of-augmented-reality.html

Hand-held displays 8 https://binocular.io/insights/augmented-reality-for-retailersin-furniture-and-interior-design

Wearable displays Two approaches 1. Optical see-through 2. Video see-through www.youtube.com/watch?v=_x9462x6a0k 9

Basic optical principle of near-eye display 10

Display comparison Optical see-through + Better visibility of surrounding real world Video see-through + Video image can be controlled à Matching graphics and video feed + Lighter - Perception of real world is distorted - Brightness (current displays difficult to use outdoors) - Latency 11

Social acceptability of wearable displays Lack of knowledge what the wearer is doing Glassholes Still bulky and interfere with face-to-face interaction Lighter and less noticeable wearable displays will open a lot new possibilities for AR 12

Spatial augmented reality Using projectors Augmented Climbing Wall https://www.youtube.com/watch?v=kwticv9ai_q 13

Future display technologies Retinal display Photons projected into eye Bright (works also outdoors in a day light) Challenge in viewing frustum Contact lens display Big challenges (e.g., power and resolution) to be solved Microvision Virtual Retinal Display http://inhabitat.com/solar-powered-augmentedcontact-lenses-cover-your-eye-with-100s-of-leds/ 14

Tracking technologies Active (sending and receiving) Mechanical, Magnetic, Ultrasonic GPS, Wifi, Bluetooth, cell location Passive Inertial sensors (Compass, gyroscope, accelerometer) Optical (markers or natural features) Hybrid Combination of sensors 16

Active tracking Mechanical tracking: Measuring joints of mechanical arms Magnetic tracking: Measuring current generated from moving coil in magnetic field Ultrasonic tracking: Measuring changes of travel time of ultrasonic waves Wifi, Bluetooth, GSM cell location: Changes in signal strength GPS: Changes in radio signal travel time MOVEMENT RANGE INCREASES 17

Tracking with Inertial measurement units (INU) Accelerometers, gyroscopes (and sometimes magnetometer) combined + Good for tracking orientation + Cheap and small - Bad for tracking absolute position over time - Drift, requires calibration 18

Optical tracking (with markers) 19 http://www.hitl.washington.edu/artoolkit/documentation/userarwork.htm

Optical tracking (markerless) Tracking based on local image features (for example SURFfeatures) http://www.vision.ee.ethz. ch/~surf/download.html 20

Optical tracking (unknown environment) SLAM (Simultaneous Localisation and Mapping) Derived from robotics: Where am I? https://www.youtube.com/watch?v=5i5pbss-yru 21

Inertial and optical tracking combined Good accuracy Support by operating systems Android: ARCore SDK ios: ARKit part of ios 11 22

Scene capturing technologies Depth sensors (like Kinect) needed for improving interaction between virtual and physical Solving what is between image features (e.g., textureless areas) Solving changes in the scene (e.g., changes in occlusion) http://dx.doi.org/10.1016/j.neucom.2014.12.081. 23

Interaction technologies How do I interact with that thing? 25

Interaction technologies 3D gestural input Considered to be natural, but how natural it is to move your hands in the air? Can become tiring Haptic feedback? Tangible interaction Data gloves Head movements Eye movements Voice input 26 5DT data gloves: http://metamotion.com/hardware/motioncapture-hardware-gloves-datagloves.htm

3D gestural input Device Distance range (m) Purpose Leap 0.2 1 m Hand tracking Kinect 0.7 6 m Skeleton tracking, 3D reconstruction Laser scanners 1 approx. 200 m 3D reconstruction 27

AR experience challenges Display technology Especially in terms of depth perception, 3D stereoscopic displays used mainly in HMDs Tracking technology Capturing technology Lack of interaction between real and virtual (e.g., occlustions and shadows) 28

Applications so far Applications in industry Maintenance (e.g., X-Ray Visualisation), Design, Training Consumer applications Games (e.g., Pokemon GO), Social media Navigation, AR Browsers (Layar, Wikitude,..), language Interior design Advertising Education (e.g., medical training) 29

Social media Snapshat Cinco de Mayo lens Facebook Camera masks Not real time (yet) 30

X-RAY Visualisation https://www.youtube.com/watch?v=ocutlemtusu http://www.inautonews.com/x-ray-vision-givesvolkswagen-a-virtual-training-advantage 31

Maintenance https://www.youtube.com/watch?v=y5ywmb6segc 32

Interior design https://www.youtube.com/watch?v=witqcim0ntm 33

Advertising https://www.youtube.com/user/layarmobile 34

Possible Future applications Social interaction https://www.youtube.com/watch?v=tb0pmeg1un0 AR Sci-Fi video, Sight: https://www.youtube.com/watch?v=lk_cdkpazji AR-glasses concept videos: https://www.youtube.com/watch?v=fm-cqr4myoa 35

From Real to Virtual [1] Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems, E77-D(12), 1321 1329. 36

Augmented Virtuality https://www.leapmotion. com/product/vr http://basilic.informatik.uni-hamburg.de/publications/2009/sbrh09a 37

Virtual Reality The environment is purely virtual Two approaches 1. Cave automated virtual environment (CAVE) 2. Head-mounted display (HMD) 38

Key Characteristics of Virtual Reality Fast and accurate tracking More controlled environment than in AR 3D Stereoscopic display Wide field-of-view display 39

Virtual reality experience Immersion Presence Feeling of being there https://www.youtube.com/watch?v=pac5senh8jw 40

Possibilities for VR Games Simulators Remote collaboration 360 Video recording 41

Challenges in VR experience Visual discomfort Accommodation and convergence mismatch, wearing display Motion sickness Mismatch between visual and vestibular systems Limited resolution Oculus Rift, Consumer Version 1080x1200, Human eye about ten times more Jason Geng, "Three-dimensional display technologies," Adv. Opt. Photon. 5, 456-535 (2013) 42

Challenges in VR experience How to implement moving in virtual space? 43

Challenges in VR experience Content creation Virtual content creation 3D software (e.g., Unity3D) Content from real environment 360 Video recording 44

VR AR 45

Next week s lecture Prof. Antti Oulasvirta will talk about computational user interfaces Note: Location is AS1! 46

Conclusion Do not believe in hype, there is still a large gap between real and virtual environment In mixed reality the key is a meaningful interplay between physical environment and virtual environment 47

Further reading Course book (Benyon s) Chapter 13: pp. 291 293 Furht, B. Handbook of Augmented Reality, 2011, Springer Mark Billinghurst s lecture notes in SlideShare: http://www.slideshare.net/search/slideshow?searchfrom =header&q=comp+4010&ud=any&ft=all&lang=**&sort= 48