VR / AR Fundamentals

Size: px

Start display at page:

Download "VR / AR Fundamentals"

Dominic Pearson
6 years ago
Views:

VR / AR Fundamentals a practical guide to the big ideas INTM-SHU 280C - 1 Topics in New Media & Entertainment Fall 2017 Week 3: Spatiality & Immersion What

1 VR / AR Fundamentals a practical guide to the big ideas INTM-SHU 280C - 1 Topics in New Media & Entertainment Fall 2017 Week 3: Spatiality & Immersion What does it take to fool two eyes and ears and with an unframed image? Jiayi Wang Michael Naimark Visiting Associate Arts Professor Interactive Media Arts NYU Shanghai

2 Class Structure First half semester Week 1 - Introduction to VR and AR Week 2 - FUNDAMENTALS 1: Audiovisual Resolution and Fidelity Fooling a single static eye and single static ear experiencing a framed representation. Week 3 - FUNDAMENTALS 2: Audiovisual Spatiality and Immersion Fooling two static eyes and ears, then two moving eyes and ears, then removing the frame. Week 4 - FUNDAMENTALS 3: Other Senses (Haptic, Smell, Taste, Mind) Fooling the other senses. Week 5 - FUNDAMENTALS 4: Input and Interactivity Using our effectors as inputs and how they shape interactive experiences. Week 6 - FUNDAMENTALS 5: Social and Liveness Being immersed with other people and live events. Week 7 - Current Dilemmas This is a lead-in, pre-production, to the second half-semester.

3 VR / AR Fundamentals Spatiality and Immersion What does it take to fool two eyes and ears and with an unframed image? Jia Stereoscopic & Stereophonic Displays Parallax & Disparity Interpupillary Distance (IPD) Convergence Stereo & Binaural Sound Multiscopic & Multiphonic Displays Holography Head Tracking Volumetric Video Spatial Sound Jiayi Panoramic Displays History Monoscopic Panoramas & Nodal Points Stereoscopic Panoramas & Head Rotation Multiscopic Panoramas Jia Putting It All Together Hacks

4 VR / AR Fundamentals Spatiality and Immersion What does it take to fool two eyes and ears and with an unframed image? Stereoscopic & Stereophonic Displays Parallax & Disparity Interpupillary Distance (IPD) Convergence Stereo & Binaural Sound Multiscopic & Multiphonic Displays Holography Head Tracking Volumetric Video Spatial Sound Jiayi Wang Panoramic Displays History Monoscopic Panoramas & Nodal Points Stereoscopic Panoramas & Head Rotation Multiscopic Panoramas Putting It All Together Hacks

6 Distance & Parallax/Disparity

7 Depth Maps

8 VR / AR Fundamentals Spatiality and Immersion What does it take to fool two eyes and ears and with an unframed image? Stereoscopic & Stereophonic Displays Parallax & Disparity Interpupillary Distance (IPD) Convergence Stereo & Binaural Sound Multiscopic & Multiphonic Displays Holography Head Tracking Volumetric Video Spatial Sound Jiayi Wang Panoramic Displays History Monoscopic Panoramas & Nodal Points Stereoscopic Panoramas & Head Rotation Multiscopic Panoramas Putting It All Together Hacks

9 Elephant Eyes (a learning center installation by Edwin Schlossberg)

10 VR / AR Fundamentals Spatiality and Immersion What does it take to fool two eyes and ears and with an unframed image? Stereoscopic & Stereophonic Displays Parallax & Disparity Interpupillary Distance (IPD) Convergence Stereo & Binaural Sound Multiscopic & Multiphonic Displays Holography Head Tracking Volumetric Video Spatial Sound Jiayi Wang Panoramic Displays History Monoscopic Panoramas & Nodal Points Stereoscopic Panoramas & Head Rotation Multiscopic Panoramas Putting It All Together Hacks

15 Magic Leap story ending

16 14%

17 VR / AR Fundamentals Spatiality and Immersion What does it take to fool two eyes and ears and with an unframed image? Stereoscopic & Stereophonic Displays Parallax & Disparity Interpupillary Distance (IPD) Convergence Stereo & Binaural Sound Multiscopic & Multiphonic Displays Holography Head Tracking Volumetric Video Spatial Sound Jiayi Wang Panoramic Displays History Monoscopic Panoramas & Nodal Points Stereoscopic Panoramas & Head Rotation Multiscopic Panoramas Putting It All Together Hacks

18 Stereo = Amplitude Binaural = Amplitude + Phase (Amplitude = loudness) (Phase = speed of sound difference to each ear)

19 Dummy Heads Head-Related Transfer Functions (HRTFs)

20 VR / AR Fundamentals Spatiality and Immersion What does it take to fool two eyes and ears and with an unframed image? Stereoscopic & Stereophonic Displays Parallax & Disparity Interpupillary Distance (IPD) Convergence Stereo & Binaural Sound Multiscopic & Multiphonic Displays Holography Head Tracking Volumetric Video Spatial Sound Jiayi Wang Jiayi Wang Panoramic Displays History Monoscopic Panoramas & Nodal Points Stereoscopic Panoramas & Head Rotation Multiscopic Panoramas Putting It All Together Hacks

21 Holography and Holography

22 VR / AR Fundamentals Spatiality and Immersion What does it take to fool two eyes and ears and with an unframed image? Stereoscopic & Stereophonic Displays Parallax & Disparity Interpupillary Distance (IPD) Convergence Stereo & Binaural Sound Multiscopic & Multiphonic Displays Holography Head Tracking Volumetric Video Spatial Sound Jiayi Wang Jiayi Wang Panoramic Displays History Monoscopic Panoramas & Nodal Points Stereoscopic Panoramas & Head Rotation Multiscopic Panoramas Putting It All Together Hacks

23 Angular / Rotational / 3DOF (pan, tilt, roll) & Lateral / Positional / 6DOF (x, y, z) DOF = degrees of freedom

24 Inside Out & Outside In

25 VR / AR Fundamentals Spatiality and Immersion What does it take to fool two eyes and ears and with an unframed image? Stereoscopic & Stereophonic Displays Parallax & Disparity Interpupillary Distance (IPD) Convergence Stereo & Binaural Sound Multiscopic & Multiphonic Displays Holography Head Tracking Volumetric Video Spatial Sound Jiayi Wang Jiayi Wang Panoramic Displays History Monoscopic Panoramas & Nodal Points Stereoscopic Panoramas & Head Rotation Multiscopic Panoramas Putting It All Together Hacks

26 Active (IR emitter like MS Kinect and Google Tango, LIDAR) & Passive (cameras only and computer vision)

27 Active Rangefinding

28 Active Rangefinding Sweeps & Flashes

29 Passive Rangefinding Light Fields and Light Fields

30 Lytro first camera

31 Lytro first Immerge camera

32 Lytro current Immerge camera

34 VR / AR Fundamentals Spatiality and Immersion What does it take to fool two eyes and ears and with an unframed image? Stereoscopic & Stereophonic Displays Parallax & Disparity Interpupillary Distance (IPD) Convergence Stereo & Binaural Sound Multiscopic & Multiphonic Displays Holography Head Tracking Volumetric Video Spatial Sound Jiayi Wang Jiayi Wang Panoramic Displays History Monoscopic Panoramas & Nodal Points Stereoscopic Panoramas & Head Rotation Multiscopic Panoramas Putting It All Together Hacks

35 Ambisonic Binaural Speakers

36 VR / AR Fundamentals Spatiality and Immersion What does it take to fool two eyes and ears and with an unframed image? Stereoscopic & Stereophonic Displays Parallax & Disparity Interpupillary Distance (IPD) Convergence Stereo & Binaural Sound Multiscopic & Multiphonic Displays Holography Head Tracking Volumetric Video Spatial Sound Panoramic Displays History Monoscopic Panoramas & Nodal Points Stereoscopic Panoramas & Head Rotation Multiscopic Panoramas Jiayi Wang Putting It All Together Hacks

37 In 1792, English painter Robert Barker coined the word panorama to describe his giant cylindrical painting of Edinburgh, Scotland.

38 The following year, it was exhibited in London in a specially-built cylindrical building, with a skylight to provide even lighting and an elevated viewing platform.

39 Panoramas and Cycloramas were built all over the world. They were a most popular artform throughout the 19th Century.

41 Mesdag Panorama, The Hague, 1881 (restored) actual sand and 3D props were used in foreground

42 The Battle of Gettysburg, Boston Cyclorama, 1891

43 Painting dimensions: 27 by 259 feet (and VERY high resolution!) (note the narrow vertical field of view)

45 Less than 5 years after the birth of cinema, Cinéorama, used ten motion picture projectors below the audience, and a fake balloon above, for the 1900 Paris Exposition.

46 Napoleon, Abel Gance, x 35mm PolyVision

47 Napoleon, Abel Gance, x 35mm PolyVision (used for montage as well as panoramas)

48 Cinerama mm projectors 146 deg hor FOV

49 Cinerama camera - 3 x 35mm

50 Cinerama camera - 3 x 35mm

51 CircleVision, Disneyland, 1955 used nine 35mm motion picture projectors (note the narrow vertical field of view)

52 CircleVision camera system employed mirrors to prevent nodal point offsets

53 CAVE, Tom Defanti / Dan Sandin, University of Illinois, 1991 immersive, 3D projection of computer-based imagery (no cameras) (note the super large vertical field of view)

54 Be Now Here, Michael Naimark, Interval Research and UNESCO, D panning motion picture and rotating floor

55 Panoscope 360, Luc Courchesne, U Montréal, 1999 single 2D fisheye projection (hence not high resolution)

57 Place-Hampi, Jeffrey Shaw/Sarah Kenderdine, ZKM, RPI, UNSW, CityU HK, 2006 Rotating projection and floor, stereo-panoramic still images

58 Place-Hampi stereo panoramic still slit camera system (It takes about a minute to complete a rotation.)

60 Be Now Here Triptych USC Cinema 2008

61 Be Now Here Triptych USC Cinema 2008

62 Liquid Galaxy, Jason Holt, Google, D plasma screens (no camera)

DIVE into World Heritage 3D Panasonic and UNESCO, 2012 5

3D, motion, sound but narrow vertical field of view and nodal

64 DIVE into World Heritage 3D Panasonic and UNESCO, off-shelf Panasonic 3D video cameras simultaneous coverage, 3D, motion, sound but narrow vertical field of view and nodal point offsets non-orthoscopic display 2011 Then came VR headsets...

65 VR / AR Fundamentals Spatiality and Immersion What does it take to fool two eyes and ears and with an unframed image? Stereoscopic & Stereophonic Displays Parallax & Disparity Interpupillary Distance (IPD) Convergence Stereo & Binaural Sound Multiscopic & Multiphonic Displays Holography Head Tracking Volumetric Video Spatial Sound Panoramic Displays History Monoscopic Panoramas & Nodal Points Stereoscopic Panoramas & Head Rotation Multiscopic Panoramas Jiayi Wang Putting It All Together Hacks

66 Small 360Heros

67 Big 360Heros

69 Ricoh Theta

70 Gyroptic 360cam

71 Gyroptic io

72 VR / AR Fundamentals Spatiality and Immersion What does it take to fool two eyes and ears and with an unframed image? Stereoscopic & Stereophonic Displays Parallax & Disparity Interpupillary Distance (IPD) Convergence Stereo & Binaural Sound Multiscopic & Multiphonic Displays Holography Head Tracking Volumetric Video Spatial Sound Panoramic Displays History Monoscopic Panoramas & Nodal Points Stereoscopic Panoramas & Head Rotation Multiscopic Panoramas Jiayi Wang Putting It All Together Hacks

73 Samsung Beyond Google Jump / GoPro Odyssey Nokia Ozo Facebook Surround 360 x24 x6

74 Naimark 1995 Shaw / Kenderdine 2006

75 Head Rotation

76 VR / AR Fundamentals Spatiality and Immersion What does it take to fool two eyes and ears and with an unframed image? Stereoscopic & Stereophonic Displays Parallax & Disparity Interpupillary Distance (IPD) Convergence Stereo & Binaural Sound Multiscopic & Multiphonic Displays Holography Head Tracking Volumetric Video Spatial Sound Panoramic Displays History Monoscopic Panoramas & Nodal Points Stereoscopic Panoramas & Head Rotation Multiscopic Panoramas Jiayi Wang Putting It All Together Hacks

77 Lytro first Immerge camera

79 Intel demos world's first 'walk-around' VR video experience

80 Multiscopic Panoramic Video Rule of Thumb: Interpolate? Maybe. Extrapolate? No.

81 Or...

82 VR / AR Fundamentals Spatiality and Immersion What does it take to fool two eyes and ears and with an unframed image? Jia Stereoscopic & Stereophonic Displays Parallax & Disparity Interpupillary Distance (IPD) Convergence Stereo & Binaural Sound Multiscopic & Multiphonic Displays Holography Head Tracking Volumetric Video Spatial Sound Jiayi Panoramic Displays History Monoscopic Panoramas & Nodal Points Stereoscopic Panoramas & Head Rotation Multiscopic Panoramas Jia Putting It All Together Hacks

83 Camera-based Modeling

84 Models & Models (importance of semantic modeling)

85 VR / AR Fundamentals Spatiality and Immersion What does it take to fool two eyes and ears and with an unframed image? Jia Stereoscopic & Stereophonic Displays Parallax & Disparity Interpupillary Distance (IPD) Convergence Stereo & Binaural Sound Multiscopic & Multiphonic Displays Holography Head Tracking Volumetric Video Spatial Sound Jiayi Panoramic Displays History Monoscopic Panoramas & Nodal Points Stereoscopic Panoramas & Head Rotation Multiscopic Panoramas Jia Putting It All Together Hacks

86 Some Hacks (very partial list) Shooting in Parts Lytro s Hallelujah (but can be more DIY) CGI / Video Hybrids Giant VR Depthkit Blackout Directed Attention IMU VR camera test on YouTube

88 VR / AR Fundamentals Week 3: Assignment for Wednesday There s more material on: Select and post 3 more titles you d like to experience and why. Seeking more obscure and edgy.

89 VR / AR Fundamentals VR / AR News of the Week Mon 18 Sep ARKit and ARCore will not usher massive adoption of mobile AR (The AR Cloud Will) Report: Magic Leap To Cost Between $1,500 And $2,000, Limited Shipping Soon Google Earth Integrates With Street View In New Update (still clunky) Apple shows off breathtaking new augmented reality demos on iphone 8 (tabletop game) Adobe to introduce 360 VR editing, character animation to CC (video post-production ++) Chinese Pimax has an 8K VR headset coming to Kickstarter soon (impressive!) Felix & Paul s The People s House 360 Video Wins an Emmy (cinematic VR) This Robot Disarms Bombs Through Virtual Reality (HT to Bishka) THESE CRAZY HEADPHONES LET YOU RECORD IMMERSIVE 3D SOUNDSCAPES ANYWHERE (Binaural) Eye-tracking is virtual reality s next frontier The Everything Simulator (Thoughtful Medium story by VR VC)

Synthetic Stereoscopic Panoramic Images

Synthetic Stereoscopic Panoramic Images What are they? How are they created? What are they good for? Paul Bourke University of Western Australia In collaboration with ICinema @ University of New South