Predictive View Generation to Enable Mobile 360-degree and VR Experiences

Transcription

1 Predictive View Generation to Enable Mobile 360-degree and VR Experiences Xueshi Hou, Sujit Dey Mobile Systems Design Lab, Center for Wireless Communications, UC San Diego Jianzhong Zhang, Madhukar Budagavi Samsung Research America

2 Motivation: Towards a Truly Mobile VR Experience Goal: Enable wireless and light VR experience Observation: Existing head-mounted displays (HMDs) have limitations How to make it mobile and portable: wireless and lighter? Rendering with tethered PC Not mobile Rendering on mobile device attached to HMD Clunkyto wear Solution: Shifting computing tasks (e.g. rendering) to the edge/cloud, and streaming videos to the HMD Streaming only Field of View () 2 Example for Cloud-based Solution : 1. Transmit Head Motion and Control to Cloud 2. Field-of-View Rendering on Cloud 3. Transmit Rendered Video to VR Glass

3 Challenges of Cloud/Edge-based Wireless VR VR head-mounted devices make the requirements much steeper than cloud/edge-based video streaming - User experience much more sensitive to video artifacts à Significantly higher video quality needed - Head motion significantly increases latency sensitivity à Significantly higher frame rate and bitrate needed Experiment setup: - VR space created using Unity; VR HMD: Oculus Rift DK2; Video: H264, 1080p/4K, GOP=30 Display Device Head Motion Framerate & QP Virtual Classroom (Mbps) Bitrate (Mbps) (Racing Game) (Virtual Classroom) 1080p 4K 1080p 4K Acceptable Latency PC Monitor - 45fps, QP= ms(for VC) <100ms(for Game) Oculus - 45fps, QP= ms 75fps, QP= ms Oculus Racing Game Bitrate Note: For Virtual Classroom with 50 students, bitrate needed for 4k > 3.5 Gbps; For head motion, cloud/edge-based wireless VR will require very high frame rate and bit rate, and also needs to satisfy ultra-low latency! 3

4 Solution for Ultra-low Latency: MachineLearningBasedPredictive Pre-Rendering Possible Method 1: Render 360-degree video on cloud, transmit to RAN edge, and extraction at edge depending on head motion Advantage: lowcomputation overhead on edge device Problem: Very high (backhaul) data rate Possible Method 2: Render 360-degree video on edge device and extraction depending on head motion Advantage: theoretically low (backhaul) data rate Problem: Restricted to edge device with very high computation; ( Extraction) 4

5 Solution for Ultra-low Latency: MachineLearningBasedPredictive Pre-Rendering Solution: Based on head motion prediction, pre-render and stream predicted inadvance fromedge device Advantages: Latency: No rendering/encoding delay, minimal communication delay with significantly reduced bandwidth Edge can be RAN or local; can be mobile device System overview for proposed approach: Data Control Video Cellular Connection Data Control Video WiFi/Millimeter Wave Glasses Glasses Cloud Server MEC (Predictive Generation) Generation) Controller (a) Using Mobile Edge Computing node (MEC) Cloud Server LEC (Predictive (Predictive Generation) (b) Generation) Question: Is it possible to predict Head Motion? 5 Controller (b) Using Local Edge Computing node (LEC)

6 Predictive View Generation to Enable Mobile 360-degree and VR Experiences: // // Early experiments with Samsung Dataset y 90 Euler Coordinates Projection ~90 x ~90 in a 360-degree view Motivation: address both bandwidth & latency challenges Common approach to reduce bandwidth: streaming only à still cannot address latency problem System overview for proposed approach: Data Control Video Cellular Connection Data Control Video WiFi/Millimeter Wave Glasses Glasses Cloud Server MEC (Predictive Generation) Generation) Controller (a) Using Mobile Edge Computing node (MEC) 6 Cloud Server LEC (Predictive Generation) (b) Generation) Controller (b) Using Local Edge Computing node (LEC)

7 Idea: predictive view generation approach only predicted view is extracted (for 360-degree video) or rendered (in case of VR) and transmitted in advance (viewpoint refers to the center of ) y ~90 x 180 Tile (30 x30 ) Viewpoint 7 ~90-90

8 CDF Predictive View Generation to Enable Mobile 360-degree and VR Experiences: Early experiments with Samsung Dataset Setup: Samsung Gear VR, sampling frequency f=5hz Dataset: head motion traces from over 36,000 viewers for degree/VR videos during 7 days Tiles options: 12x6 tiles (30 x30 ), 18x6 tiles (20 x30 ), etc Video Duration (s) VR dataset statistics CDF # Viewers Over 80% of videos have >100s for duration Around 85% of videos have >1000 viewers Head Motion Speed ( /s) Time (s) Max 75 th Percentile Median 25 th Percentile Min Head motion speed versus time in Kong VR 8 This boxplot shows head motion speed distribution for over 1500 viewers during 60s; it presents the challenging situation of predicting head motion since viewers may change viewing direction fast as well as frequently

9 Predictive View Generation to Enable Mobile 360-degree and VR Experiences Attention heatmap is defined as a series of probability that a viewpoint is within a tile for n viewers during time-period from cts 1 to cts 2 Example of attention heatmap Brighter tiles attract more attention and viewers are more likely to lookat these areas Feasibility of performing viewpoint prediction (some areas attracting more attention than remaining areas within a 360-degree view) Multiple tiles (as high as 11 tiles) have relatively high probabilities (>5%), indicatingthe difficulties ofpredicting viewpointaccurately 9

10 Viewpoint Prediction using Deep Learning Goal: predict viewpoint position (tile) for 200ms in advance Model: multi-layer long short-term memory (LSTM) networks Input Features: tile-based one-hot encoding representation for viewpoint traces as 72x10 matrix (72 tiles, 10 timestamps in 2s) Label for training: whether viewpoint belonging to each tile as 72x1 matrix Output: probability of viewpoint belonging to each of the 72 tiles Viewpointtrace duringt (3,5], in seconds t=3s t=5s Where is the viewpoint when t= 5.2s (200ms afterwards)? 10

11 Viewpoint Prediction using Deep Learning Goal: predict viewpoint position (tile) for 200ms in advance Model: multi-layer long short-term memory (LSTM) networks Input Features: tile-based one-hot encoding representation for viewpoint traces as 72x10 matrix (72 tiles, 10 timestamps in 2s) Label for training: whether viewpoint belonging to each tile as 72x1 matrix Output: probability of viewpoint belonging to each of the 72 tiles Viewpoint traces during t (3,5) seconds <0.01 <0.01 <0.01 < <0.01 < <0.01 <0.01 <0.01 <

12 Viewpoint Prediction using Deep Learning LSTM Unit LSTM Unit Predicted Viewpoint Softmax Layer Fully Connected Layer LSTM Unit LSTM Unit Viewpoint Features LSTM Unit LSTM Unit Dataset: Head motion traces of 36,000 viewers during 7 days for degree/VR videos; Each trace point 200ms Training Data: 45,000 head motion sampling traces (each for 2s long) Test Data: 5,000 head motion sampling traces (where viewers are different from training data) Parameters: first layer: 128 LSTM units; second layer: 128 LSTM units; fully connected layer: 72 nodes; We explore four deep learning or classical machine learning models for viewpoint prediction: LSTM, Stacked sparse autoencoders (SAE), Bootstrapaggregated decision trees (BT), and Weighted k-nearest neighbors (knn) SAE: two fully-connected layers with 100 and 80 nodes respectively; BT: ensembles with 30 bagged decision trees; knn: 100 nearest neighbors 12

13 Predictive View Generation: Accuracy and Bitrate Selection: Accuracy and Bitrate generation generation method: Select m tiles with highest probabilities predicted by the LSTM model Compose the predicted as the combination of s for each selected tile Transmit the predicted with high quality while leaving the rest of tiles blank prediction accuracy: the probability that actual user view will be within the predicted depends on the LSTM model accuracy and generation method, and thus reflects both the performance of our LSTM model and generation method 13

14 Predictive View Generation: Accuracy and Bitrate Selection: Accuracy and Tradeoff Bitrate between size (bitrate) and prediction accuracy generation Prediction accuracy is 100% if predicted is the whole 360-degree but very high bitrate By selecting more tiles (i.e. larger m) with high viewpoint probability, we can achieve higher prediction accuracy but also higher bitrate Use choice of m to achieve the desired tradeoff between prediction accuracy and bandwidth consumed: Choice of larger m à higher bandwidth but better prediction accuracy Choice ofsmaller m à lower bandwidth but higher riskin prediction accuracy 14

15 Predictive View Generation: Accuracy and Bitrate prediction accuracy and pixel savings obtained when selecting different number of tiles (i.e. the choice of m) to generate Medium Motion Sequence SAE: Stacked sparse autoencoders; BT: Bootstrap-aggregated decision trees; knn: Weighted k-nearest neighbors 15

16 Predictive View Generation: Accuracy and Bitrate As number of tiles m increases, prediction accuracy continuously increases and pixel saving simultaneously decreases à tradeoff between prediction accuracy and pixel saving High Motion Sequence SAE: Stacked sparse autoencoders; BT: Bootstrap-aggregated decision trees; knn: Weighted k-nearest neighbors 16

17 Predictive View Generation: Accuracy and Bitrate Results below for one strategy show possible to achieve high prediction accuracy while significantly reducing bitrate (to rate comparable to real-time generated ) Selection: Accuracy and Bitrate We set prediction accuracy as ~95%, and compare pixel savings achieved for each model Model Kong VR Medium Motion Sequence High Motion Sequence Acc.(%) Pixel Saving(%) Acc.(%) Pixel Saving(%) SAE LSTM BT knn Model Fashion Show Acc.(%) Pixel Saving (%) Whale Encounter Acc.(%) Pixel Saving (%) Roller Coaster Acc. (%) Pixel Saving (%) SAE LSTM BT knn SAE: Stacked sparse autoencoders; BT: Bootstrap-aggregated decision trees; knn: Weighted k-nearest neighbors 17

18 Summary and Future Work We propose a predictive view generation approach in order to reduce the latency and bandwidth needed to deliver 360-degree videos and cloud/edgebased VR applications, leading to better mobile VR experiences We present a multi-layer LSTM model which can learn general head motion pattern and predict the future viewpoint based on past traces Our method presents good results on on a real head motion trace dataset and shows great potential to reduce bandwidth Future Work: Adaptively streaming using our training model (tiles with different video quality) 3DOF à 6DOF: View prediction considering body motion (6DoF) and hand motion *Photo Source: Qualcomm 3 Degrees of Freedom (3DoF) 18 6 Degrees of Freedom (6DoF)

19 Thanks!