PERCEPTUAL INSIGHTS INTO FOVEATED VIRTUAL REALITY Anjul Patney Senior Research Scientist
INTRODUCTION Virtual reality is an exciting challenging workload for computer graphics Most VR pixels are peripheral Efficient peripheral rendering is poorly understood Quick perceptual experiments can expose opportunities for significant speedups 2
FOVEATED RENDERING Foveal Pixels Peripheral Pixels 3
THIS TALK Why foveated rendering? State of the art in efficient peripheral rendering How else could we reduce peripheral details? 4
WHY FOVEATED RENDERING? 5
PC Gaming : 60 MP/s (1920 x 1080 @ 30 Hz) 1080 1920 *VR render resolution for HTC Vive 6
PC Gaming : 60 MP/s (1920 x 1080 @ 30 Hz) Virtual Reality : 450 MP/s (3024 x 1680* @ 90 Hz) 1080 1680 1920 1512 1512 *VR render resolution for HTC Vive 7
VR IS UP TO 7X MORE DEMANDING TODAY PC Gaming : 60 MP/s (1920 x 1080 @ 30 Hz) Virtual Reality : 450 MP/s (3024 x 1680* @ 90 Hz) 1080 1680 1920 1512 1512 *VR render resolution for HTC Vive 8
MOST VR PIXELS ARE PERIPHERAL Column1 4% Peripheral 20 Foveal Peripheral 73% 96% iphone 7 Plus 27" Desktop Monitor 2016 VR HMD 9
MOST VR PIXELS ARE PERIPHERAL Peripheral Column1 20 Foveal Peripheral 4% Efficient peripheral 73% rendering can unlock significant performance gains 96% iphone 7 Plus 27" Desktop Monitor 2016 VR HMD 10
EFFICIENT PERIPHERAL RENDERING What is the ideal peripheral image? How to efficiently render the ideal peripheral image? 11
STATE OF THE ART IN EFFICIENT PERIPHERAL RENDERING 12
MULTI-PASS FOVEATED RENDERING [Guenter et al. 2012] 13
GPU-ACCELERATED FOVEATED RENDERING NVIDIA Multi-Res Shading 14
GPU-ACCELERATED FOVEATED RENDERING NVIDIA Lens-Matched Shading 15
BEYOND FOVEATED IMAGE RESOLUTION Peripheral vision shows non-uniform optical, retinal, and neural degradation Over an order of magnitude potential improvement Density (per mm2) 100K 10K 1K 100 Rods Ganglion Cells Cones 10 0 20 40 60 80 Eccentricity (degrees) 16
BEYOND FOVEATED IMAGE RESOLUTION LOW SENSITIVITY TOWARD HIGH SENSITIVITY TOWARD High Spatial Frequencies Color Flicker Motion Crowded Stimuli Faces http://pics.psych.stir.ac.uk/ 17
BEYOND FOVEATED IMAGE RESOLUTION LOW SENSITIVITY TOWARD HIGH SENSITIVITY TOWARD High Spatial Frequencies Color Flicker Motion Crowded Stimuli Faces http://pics.psych.stir.ac.uk/ 18
BEYOND FOVEATED IMAGE RESOLUTION LOW SENSITIVITY TOWARD HIGH SENSITIVITY TOWARD High Spatial Frequencies Color Flicker Motion Crowded Stimuli Faces http://pics.psych.stir.ac.uk/ 19
PERCEPTUAL FOVEATED RENDERING [Stengel et al. 2016] Acuity Fall-off, Eye/Object Motion, Contrast, Brightness Adaptation 2-3x lower fragment shading workload 20
PERIPHERAL BLUR PERCEPTUAL FOVEATED RENDERING [Patney et al. 2016] 21
PERIPHERAL BLUR + CONTRAST PRESERVATION PERCEPTUAL FOVEATED RENDERING [Patney et al. 2016] 22
PRACTICAL RENDERING SYSTEM WITH 2X-3X FEWER PIXEL SHADES PERCEPTUAL FOVEATED RENDERING [Patney et al. 2016] 23
HOW ELSE COULD WE REDUCE PERIPHERAL DETAILS? 24
PERCEPTUAL HYPOTHESES Can we reduce peripheral color saturation? Can we foveate alternate frames? Should we foveate symmetrically around the gaze center? 25
EARLY EXPERIMENTS Simulated foveation following each hypothesis Performed ad-hoc Yes/No tests with two subjects Estimated threshold using MATLAB psychtoolbox Total time per trained subject: < 1 hour 26
BASELINE: FOVEATED BLUR 27
CAN WE REDUCE PERIPHERAL COLOR SATURATION? 28
CAN WE FOVEATE ALTERNATE FRAMES? Even Odd Frame 29
CAN WE FOVEATE ALTERNATE FRAMES? Even Frame 30
SHOULD WE FOVEATE SYMMETRICALLY AROUND THE GAZE CENTER? Temporal Nasal Foveation 31
SHOULD WE FOVEATE SYMMETRICALLY AROUND THE GAZE CENTER? Temporal Foveation 32
RESULTS MAGNITUDE OF FOVEATION Foveation threshold 50 40 30 20 10 Subject 1 Subject 2 0 Baseline Color Flicker Nasal Temporal 33
RESULTS ESTIMATED PERFORMANCE 1.6 Relative performance 1.4 1.2 1 0.8 0.6 0.4 0.2 0 Color Flicker Nasal + Temporal 34
RESULTS ESTIMATED PERFORMANCE Relative performance 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 Potential Win! 0 Color Flicker Nasal + Temporal 35
SUMMARY Efficient peripheral rendering is fundamental for VR rendering But we have a lot to learn about efficient peripheral rendering What is the ideal peripheral image? How to efficiently render the ideal peripheral image? Perceptual evaluations can expose opportunities for significant speedups 36
ACKNOWLEDGMENTS Joohwan Kim Rachel Albert Peter Shirley Chris Wyman Marco Salvi Jason Paul David Luebke Aaron Lefohn 37
THANK YOU apatney@nvidia.com 38