Virtual Reality I Visual Imaging in the Electronic Age Donald P. Greenberg November 9, 2017 Lecture #21
1968: Ivan Sutherland 1990s: HMDs, Henry Fuchs 2013: Google Glass History of Virtual Reality 2016: HTC & Valve s SteamVR Vive 25 Years
Types of Virtual Reality (VR) by Viewer Constraints Static location of viewer Movable location of viewer Augmented Reality
Static Location Known (or not considered)xx ee, yy ee, zz ee Unknown (Ɵ ee, ΘΘ ee, φφ ee ) How measured- (accelerometers) cameras
Movable Location e.g., within a room Unknown (xx ee, yy ee, zz ee ) Unknown (Ɵ ee, ΘΘ ee, φφ ee ) How measured- triangulation signals, light, timing I saw you cameras in goggles or cameras on wall
Augmented Reality No limitations Need GPS position relative to an outside environment > Not yet available- not accurate enough Current accuracy is unclear Can put virtual image in real space (MSFT Hololens)
Augmented Reality Augmented reality is the interpretation of digital graphical information with live video or the user s physical environment in real time. UNC
Microsoft s Hololens 3/30/16
Types of VR by Data and Image Generation Algorithms Environment Maps and Texturing Full 3D Geometric Environments Combination
Environment Maps and Texturing Can t get close to virtual surface Stationary position? Input- Photography Rendered image Both methods would be placed as a texture on the inside of a hemispherical dome (e.g., New York Times)
Full 3D Environment Required detail models (3D) Can move within constrained environment Can be tracked (Gaze Directions) Rendering- not fast enough Currently limited to diffuse environments Sometimes with shadow maps Limited light sources
Combination 2D + 3D Environment Maps & 3D Environments Can move within 3D virtual environments Need models and limitations Accuracy? Where do we need the 3D information?
Virtual Reality A term used to describe a computer generated environment which can simulate the perception of PRESENCE.
Virtual Reality A person immersed within this virtual world can manipulate objects, interact with the environment, and explore the virtual world in the same perceptual way as one interacts with the physical world.
Human in the Loop Abstract Interpretation Viewing a Picture on Television Cinema Viewing Presence
Microsoft s Hololens
Microsoft buys Minecraft
Microsoft s Hololens
Magic Leap
Oculus Rift DK2 2014
Oculus Rift March 28, 2016 Oculus VR PenTile OLED 2160x1200 (1080x1200 per eye) at 90 Hz Integrated 3D audio headphones (user removable/exchangeable) Wikipedia 6DOF (3-axis rotational tracking + 3- axis positional tracking) through USBconnected IR LED sensor, which tracks via the constellation method.
HTC and Valve s SteamVR Vive 2016 Available Tomorrow
Virtual Reality Speed (Refresh and update rates) Field of view and resolution Color and dynamic range Bandwidth
Refresh Rates 2016 Note that an insufficient refresh rate will cause flicker. For most cases the displays must be refreshed at least 50 cycles/second. Current virtual reality displays (2016) operate at 90 Hz.
Update Rates 2016 The human eye can process approximately 10-12 separate images per second (maybe even less). To simulate motion, update rates must be greater than this. The movie industry has historically used 24 fps and the television industry 30 fps. Since in VR/AR continuous motion is paramount, then rendering must be about 11 milliseconds (1000/90) or less.
Field of view Horizontal - 180-190 Vertical - 135 Wikipedia
Field of View Foundations of Sensation and Perception. Mather, George. 2009.
Field of View for Humans Humans have an almost 180 degree frontal horizontal field of view The vertical range of the visual field is approximately 135 degrees The resolution, color discrimination, and reaction times is not uniform across the field of view
Stereoscopic & Monoscopic Depth Information Note that only when the two fields overlap do we have the ability to determine depth stereoscopically Otherwise most of our depth cues are received monoscopically
Binocular Vision Binocular Vision, which is the basis for stereopsis is important for depth perception and covers 114 degrees (horizontally) of the human visual field. The remaining sixty to seventy degrees have no binocular vision (because only one eye can see those portions of the visual field)
Human Depth Perception Depth Perception Oculomotor Visual Binocular Monocular Binocular Monocular Convergence Accommodation Stereopsis Static Cues Motion Parallax Perspective Familiarity, Relative Size Motion, Position Occlusion Texture Gradient Shading, Shadows, Highlights Atmospheric Blur
Saccadic Motion The eye jumps, comes to rest momentarily (producing a small dot on the record), then jumps to a new locus of interest. - David H. Hubel. EYE, BRAIN, AND VISION, 1988 Scientific American Books, Inc. p. 80.
Resolution of the Human Eye Humans can tell visual details at distances larger than 0.3 arc minutes The Field of View (FOV) of the human eye can be generously estimated as 120 by 90 degrees
Resolution of the Human Eye (120 degrees x 60 arcminutes / degree x 1 pixel / 0.3 arcminutes) x (90 degrees x 60 arcminutes / degree x 1 pixel / 0.3 arcminutes) 431,568,000 pixels; 432 MegaPixels. A 1080p display is 2.1 megapixels.
Image from Eye, Brain, and Vision, David Hubel, 19
Opponent Color Theory Analog Computing Figure 12.9 Foundations of Sensation and Perception, George Mathe
Dynamic Range of Displays luminance (log cd/m 2 ) range of illumination displays -6-4 -2 0 2 4 6 8 starlight moonlight indoor lighting sunlight scotopic mesopic photopic poor contrast no color low acuity good contrast good color high acuity
Samsung AMOLED Displays Standard LCD RGB Display PenTile Display
Distorted Images
Truck Art http://www.wltc.org/documents/truckart.htm
Distorted Images DORSEY 1993
Distorted Images DORSEY 1993
Distorted Images DORSEY 1993
Technology of Current VR Systems
Oculus Rift
Oculus Rift Components Accelerometers and logic board Lenses Flat 1080P AMOLED Display
Oculus Rift Angular Rotation
Distortion Strategy
Oculus Rift Distortion Strategy
Oculus Rift Distorted Image 1080x1200 pixels per eye
Compensating for Lens Distortion The lenses in Head Mounted Displays (HMD s) apply a pincushion distortion to the image To correct the image to the viewer, the pipeline applies a barrel distortion as a post rendering process, to yield an undistorted image Since this operation requires a resampling, the image becomes slightly blurred
Compensating for Lens Distortion To overcome these deficiencies, frequently the image is rendered at a higher pixel resolution, but this requires more processing time Another method is to render the scene normally, then stretch it over a 3D polygon mesh as a template, and use a GPU to display it. (e.g. photographing a curved virtual screen)
End..