PERCEPTUAL INSIGHTS INTO FOVEATED VIRTUAL REALITY. Anjul Patney Senior Research Scientist

Transcription

1 PERCEPTUAL INSIGHTS INTO FOVEATED VIRTUAL REALITY Anjul Patney Senior Research Scientist

2 INTRODUCTION Virtual reality is an exciting challenging workload for computer graphics Most VR pixels are peripheral Efficient peripheral rendering is poorly understood Quick perceptual experiments can expose opportunities for significant speedups 2

3 FOVEATED RENDERING Foveal Pixels Peripheral Pixels 3

4 THIS TALK Why foveated rendering? State of the art in efficient peripheral rendering How else could we reduce peripheral details? 4

5 WHY FOVEATED RENDERING? 5

6 PC Gaming : 60 MP/s (1920 x 30 Hz) *VR render resolution for HTC Vive 6

7 PC Gaming : 60 MP/s (1920 x 30 Hz) Virtual Reality : 450 MP/s (3024 x 90 Hz) *VR render resolution for HTC Vive 7

8 VR IS UP TO 7X MORE DEMANDING TODAY PC Gaming : 60 MP/s (1920 x 30 Hz) Virtual Reality : 450 MP/s (3024 x 90 Hz) *VR render resolution for HTC Vive 8

9 MOST VR PIXELS ARE PERIPHERAL Column1 4% Peripheral 20 Foveal Peripheral 73% 96% iphone 7 Plus 27" Desktop Monitor 2016 VR HMD 9

10 MOST VR PIXELS ARE PERIPHERAL Peripheral Column1 20 Foveal Peripheral 4% Efficient peripheral 73% rendering can unlock significant performance gains 96% iphone 7 Plus 27" Desktop Monitor 2016 VR HMD 10

11 EFFICIENT PERIPHERAL RENDERING What is the ideal peripheral image? How to efficiently render the ideal peripheral image? 11

12 STATE OF THE ART IN EFFICIENT PERIPHERAL RENDERING 12

13 MULTI-PASS FOVEATED RENDERING [Guenter et al. 2012] 13

14 GPU-ACCELERATED FOVEATED RENDERING NVIDIA Multi-Res Shading 14

15 GPU-ACCELERATED FOVEATED RENDERING NVIDIA Lens-Matched Shading 15

16 BEYOND FOVEATED IMAGE RESOLUTION Peripheral vision shows non-uniform optical, retinal, and neural degradation Over an order of magnitude potential improvement Density (per mm2) 100K 10K 1K 100 Rods Ganglion Cells Cones Eccentricity (degrees) 16

17 BEYOND FOVEATED IMAGE RESOLUTION LOW SENSITIVITY TOWARD HIGH SENSITIVITY TOWARD High Spatial Frequencies Color Flicker Motion Crowded Stimuli Faces 17

18 BEYOND FOVEATED IMAGE RESOLUTION LOW SENSITIVITY TOWARD HIGH SENSITIVITY TOWARD High Spatial Frequencies Color Flicker Motion Crowded Stimuli Faces 18

19 BEYOND FOVEATED IMAGE RESOLUTION LOW SENSITIVITY TOWARD HIGH SENSITIVITY TOWARD High Spatial Frequencies Color Flicker Motion Crowded Stimuli Faces 19

20 PERCEPTUAL FOVEATED RENDERING [Stengel et al. 2016] Acuity Fall-off, Eye/Object Motion, Contrast, Brightness Adaptation 2-3x lower fragment shading workload 20

21 PERIPHERAL BLUR PERCEPTUAL FOVEATED RENDERING [Patney et al. 2016] 21

22 PERIPHERAL BLUR + CONTRAST PRESERVATION PERCEPTUAL FOVEATED RENDERING [Patney et al. 2016] 22

23 PRACTICAL RENDERING SYSTEM WITH 2X-3X FEWER PIXEL SHADES PERCEPTUAL FOVEATED RENDERING [Patney et al. 2016] 23

24 HOW ELSE COULD WE REDUCE PERIPHERAL DETAILS? 24

25 PERCEPTUAL HYPOTHESES Can we reduce peripheral color saturation? Can we foveate alternate frames? Should we foveate symmetrically around the gaze center? 25

26 EARLY EXPERIMENTS Simulated foveation following each hypothesis Performed ad-hoc Yes/No tests with two subjects Estimated threshold using MATLAB psychtoolbox Total time per trained subject: < 1 hour 26

27 BASELINE: FOVEATED BLUR 27

28 CAN WE REDUCE PERIPHERAL COLOR SATURATION? 28

29 CAN WE FOVEATE ALTERNATE FRAMES? Even Odd Frame 29

30 CAN WE FOVEATE ALTERNATE FRAMES? Even Frame 30

31 SHOULD WE FOVEATE SYMMETRICALLY AROUND THE GAZE CENTER? Temporal Nasal Foveation 31

32 SHOULD WE FOVEATE SYMMETRICALLY AROUND THE GAZE CENTER? Temporal Foveation 32

33 RESULTS MAGNITUDE OF FOVEATION Foveation threshold Subject 1 Subject 2 0 Baseline Color Flicker Nasal Temporal 33

34 RESULTS ESTIMATED PERFORMANCE 1.6 Relative performance Color Flicker Nasal + Temporal 34

35 RESULTS ESTIMATED PERFORMANCE Relative performance Potential Win! 0 Color Flicker Nasal + Temporal 35

36 SUMMARY Efficient peripheral rendering is fundamental for VR rendering But we have a lot to learn about efficient peripheral rendering What is the ideal peripheral image? How to efficiently render the ideal peripheral image? Perceptual evaluations can expose opportunities for significant speedups 36

37 ACKNOWLEDGMENTS Joohwan Kim Rachel Albert Peter Shirley Chris Wyman Marco Salvi Jason Paul David Luebke Aaron Lefohn 37

38 THANK YOU 38

39