Reinventing movies How do we tell stories in VR? Diego Gutierrez Graphics & Imaging Lab Universidad de Zaragoza

Transcription

1 Reinventing movies How do we tell stories in VR? Diego Gutierrez Graphics & Imaging Lab Universidad de Zaragoza

2 Computer Graphics Computational Imaging Virtual Reality

3 Joint work with: A. Serrano, J. Ruiz-Borau and B. Masia (Universidad de Zaragoza) V. Sitzmann and G. Wetzstein (Stanford University) Movie Editing and Cognitive Event Segmentation in Virtual Reality Video (SIGGRAPH 2017) How Do People Explore Virtual Environments? (IEEE-VR, submitted)

4

5 Open challenges Virtual Reality Field of View Resolution Size Latency Focus cues Augmented Reality #include <VR.h> Brightness Attenuation Occlusion Power Original list by Dave Luebke, NVIDIA

6

7

8

9 Like all new media, it still needs to find out and reach its true potential And for all its immersive and engaging capabilities.

10 it turns out we don t know how to tell stories in VR!

11 it turns out we don t know how to tell stories in VR!

12 it turns out we don t know how to tell stories in VR!

13

14

15 Challenge: Focus Cues Video from Narain et al., Optimal Presentation of Imagery with Focus Cueson Multi-Plane Displays, SIGGRAPH 2015

16

17

18

19

20

21

22

23 The world we perceive is a representation, a convenient and efficient mental construction

24

25

26 Ok, back to movies

27 MOTIVATION Traditional cinematography Matrix, 1999 Casablanca, 1942 Inception, The 2010 Kid, : A Space Odyssey, 1968

28

29 MOTIVATION Traditional cinematography Match on action Night of the Living Dead, 1968

30 MOTIVATION Traditional cinematography - Real world Continuous flow of information - Movies Discontinuous in space/time/action

31 MOTIVATION Traditional cinematography Movies are still perceived as a consistent sequence of events!

32 MOTIVATION The key to the illusion

33 MOTIVATION Traditional cinematography - Filmmakers and editors can make the cuts invisible

34 MOTIVATION Traditional cinematography - Filmmakers and editors can make the cuts invisible Continuity editing Set of editing tools to create situational continuity

35 MOTIVATION VR cinematography VR: Different paradigm User (partially) controls the camera New challenge for cinematography!

36 If one of the main pillars movie making is based on is not reliable anymore. How do we tell stories in VR?

37 MOTIVATION VR cinematography - Democratization of VR video content

38 MOTIVATION VR cinematography - Democratization of VR video content Final Goal: Build cinematographic language for VR Our contribution: First steps towards understanding user behavior in VR In particular, how do edits affect users in VR?

39 FIRST THINGS FIRST - What makes VR video different from traditional video?

40 UNDERSTANDING THE PROBLEM Traditional cinematography Frame-to-frame time Jessica Brillhart Filmmaker at google

41 UNDERSTANDING THE PROBLEM Traditional cinematography Frame-to-frame VR cinematography World-to-world time time Jessica Brillhart Filmmaker at google

42 UNDERSTANDING THE PROBLEM VR cinematography World-to-world Regions of Interest time Jessica Brillhart Filmmaker at google

43 MOTIVATION VR cinematography - Democratization of VR video content Final Goal: Build cinematographic language for VR Our contribution: First steps towards understanding user behavior in VR In particular, how do edits affect users in VR?

44 MOTIVATION VR cinematography - Democratization of VR video content Final Goal: Build cinematographic language for VR Our contribution: First steps towards understanding user behavior in VR In particular, how do edits affect users in VR? Does continuity editing hold in VR?

45 UNDERSTANDING THE PROBLEM Does continuity editing hold in VR? - Knowledge typically based on experience Books would tell you how, but not why!

46 UNDERSTANDING THE PROBLEM Does continuity editing hold in VR? - Knowledge typically based on experience

47 UNDERSTANDING THE PROBLEM Does continuity editing hold in VR? - Knowledge typically based on experience - Cognitive theory of event segmentation

48 UNDERSTANDING THE PROBLEM Does continuity editing hold in VR? - Knowledge typically based on experience - Cognitive theory of event segmentation This theory offers a plausible explanation! [Reynolds et al. 2007] [Zacks and Swallow 2007] [Kurby and Zacks 2008] [Zacks 2010] [Zacks and Magliano 2011]

49

50

51 UNDERSTANDING THE PROBLEM Similar to spatial segmentation, our brains seem to rely also on event segmentation in the temporal domain Our everyday experiences are organized as discrete events

52 UNDERSTANDING THE PROBLEM Event segmentation seems to be an automatic key component of our perceptual processing It is a very effective way to store and archive events in memory! Very efficient in terms of internal representation and memory It allows quick sorting mechanisms (this event occurred before/after this) It allows quick random access to a given memory

53 Cognitive theory of event segmentation This discrete mental representation is used as a basis for predicting the immediate course of events When these predictions are violated, it is an indication of a new (discrete) event It seems that unexpected changes lead to the perception of an event boundary

54 Cognitive theory of event segmentation In other words: event segmentation theory assumes that new events are registered when changes in action, space, or time, occur

55 Cognitive theory of event segmentation CONTINUOUS FLOW OF INFORMATION

56 Cognitive theory of event segmentation CONTINUOUS FLOW OF INFORMATION MEMORY SEGMENTATION

57 Cognitive theory of event segmentation CONTINUOUS FLOW OF INFORMATION MEMORY Conversation Subject A: Speaks Subject B: Answers SEGMENTATION

58 Cognitive theory of event segmentation CONTINUOUS FLOW OF INFORMATION MEMORY Conversation Subject A: Speaks Subject B: Answers SEGMENTATION Bike crossing Subject A: Looks Subject B: Stops talking

59 Cognitive theory of event segmentation CONTINUOUS FLOW OF INFORMATION Event Boundary SEGMENTATION MEMORY Conversation Subject A: Speaks Subject B: Answers Bike crossing Subject A: Looks Subject B: Stops talking

60 Cognitive theory of event segmentation [Magliano and Zacks 2011] Experiment showing that traditional cinematography techniques are surprisingly in-sync with how brain stores events

61 Cognitive theory of event segmentation [Magliano and Zacks 2011] Experiment showing that traditional cinematography techniques are surprisingly in-sync with how brain stores events When a cut introduces a major change, the brain does not try to explain the perceived discontinuity; instead, it adapts to the change, creates a new mental representation, and begins populating it with details

62 Cognitive theory of event segmentation This automatic mechanism might be a key process to explain why continuity editing works: the relation between edits and perceived event boundaries leads to perceived continuity

63 Cognitive theory of event segmentation This automatic mechanism might be a key process to explain why continuity editing works: the relation between edits and perceived event boundaries leads to perceived continuity Is this also the case in VR?

64 Cognitive theory of event segmentation Goal Investigate if continuity editing holds in VR

65 Cognitive theory of event segmentation Goal How Investigate if continuity editing holds in VR [Zacks and Magliano 2011] Replicate previous experiments in a VR environment

66 Cognitive theory of event segmentation Three types of edits: E1: action discontinuity E2: spatial/temporal discontinuity E3: continuity edit

67 MEASURING CONTINUITY IN VR E 1 ACTION DISCONTINUITY

68 MEASURING CONTINUITY IN VR E 2 SPATIAL/TEMPORAL DISCONTINUITY

69 MEASURING CONTINUITY IN VR E 3 CONTINUITY

70 Cognitive theory of event segmentation Participants were asked to segment the movie into meaningful events by pressing a button Movie edits had been previously identified by the authors

71 Results Timeline

72 Results Movie edits Timeline

73 Results Mark perceived event boundaries Movie edits Timeline Perceived event boundaries

74 Results Mark perceived event boundaries Movie edits Edits marked as event boundaries Timeline Perceived event boundaries

75 Results Mark perceived event boundaries Movie edits Edits marked as event boundaries Timeline Perceived event boundaries

76 Results Mark perceived event boundaries Movie edits Edits marked as event boundaries Timeline Perceived event boundaries

77 Cognitive theory of event segmentation As predicted, E1 edits had the largest correlation with perceived discontinuities, while E3 edits maintained continuity

78 EVENT SEGMENTATION IN VR Replicated this experiments in a VR environment with four VR movies Consistent results!

79 EVENT SEGMENTATION IN VR Replicated this experiments in a VR environment with four VR movies Consistent results! Continuity editing seems to hold in VR

80 GOAL First steps towards understanding user behavior in VR - Investigate continuity editing in VR - Analyze how different edits affect attentional behavior

81 GOAL First steps towards understanding user behavior in VR - Investigate continuity editing in VR - Analyze how different edits affect attentional behavior - Parameters of influence

82 GOAL First steps towards understanding user behavior in VR - Investigate continuity editing in VR - Analyze how different edits affect attentional behavior - Parameters of influence - Metrics describing user behavior

83 GOAL First steps towards understanding user behavior in VR - Investigate continuity editing in VR - Analyze how different edits affect attentional behavior - Parameters of influence - Metrics describing user behavior - Analysis and results

84 MEASURING CONTINUITY IN VR Experimental design Goal: Analyze how different edits affect attentional behavior

85 MEASURING CONTINUITY IN VR Experimental design Goal: Analyze how different edits affect attentional behavior - Monocular 360 o videos - 4 unedited videos Baseline - 12-second clips Two shots (edit) - Oculus DK2 with eye tracker - 49 users

86

87 MEASURING CONTINUITY IN VR First steps towards understanding user behavior in VR - Investigate continuity editing in VR - Analyze how different edits affect attentional behavior - Parameters of influence - Metrics describing user behavior - Analysis and results

88 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit

89 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, ACTION DISCONTINUITY

90 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } SPATIAL/TEMPORAL DISCONTINUITY

91 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } CONTINUITY

92 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } CONTINUITY 17 VR movies analyzed - 73% E1 edits - 25% E2 edits - 2% E3 edits

93 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } CONTINUITY 17 VR movies analyzed - 73% E1 edits - 25% E2 edits - 2% E3 edits = Traditional cinematography

94 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } - Alignment of ROI

95 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } - Alignment of ROI FOV

96 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } FOV - Alignment of ROI {A 0, ROI position before the edit

97 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } - Alignment of ROI {A 0, FOV ROI position before the edit ROI position after the edit

98 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } - Alignment of ROI {A 0, A 40, FOV ROI position before the edit ROI position after the edit

99 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } - Alignment of ROI {A 0, A 40, FOV ROI position before the edit ROI position after the edit

100 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } - Alignment of ROI {A 0, A 40, A 80 } FOV ROI position before the edit ROI position after the edit

101 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } - Alignment of ROI {A 0, A 40, A 80 } FOV ROI position before the edit ROI position after the edit

102 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } - Alignment of ROI {A 0, A 40, A 80 } - ROI configuration

103 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } - Alignment of ROI {A 0, A 40, A 80 } - ROI configuration {R b,0, {R a,0,

104 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } - Alignment of ROI {A 0, A 40, A 80 } - ROI configuration {R b,0, R b,1, {R a,0, R a,1,

105 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } - Alignment of ROI {A 0, A 40, A 80 } - ROI configuration {R b,0, R b,1, R b,2 } {R a,0, R a,1, R a,2 }

106 MEASURING CONTINUITY IN VR Parameters of influence - Type of edit {E 1, E 2 } - Alignment of ROI {A 0, A 40, A 80 } - ROI configuration {R b,0, R b,1, R b,2 } {R a,0, R a,1, R a,2 } 54 different conditions 4 clips for each condition 216 stimuli

107 MEASURING CONTINUITY IN VR Captured data

108 GOAL First steps towards understanding user behavior in VR - Investigate continuity editing in VR - Analyze how different edits affect attentional behavior - Parameters of influence - Metrics describing user behavior - Analysis and results

109 GOAL First steps towards understanding user behavior in VR - Investigate continuity editing in VR - Analyze how different edits affect attentional behavior - Parameters of influence - Metrics describing user behavior - Analysis and results

110 MEASURING CONTINUITY IN VR Metrics Frames to reach a Region Of Interest Indicative of the time taken to converge to the main action

111 MEASURING CONTINUITY IN VR Metrics Number of fixations Fixed or exploratory behavior

112 MEASURING CONTINUITY IN VR Metrics Number of fixations Fixed or exploratory behavior Percentage of total fixations inside the ROI Indicative of the interest of the viewer in the ROI

113 MEASURING CONTINUITY IN VR Metrics Divergence from baseline Indicates how gaze behavior is altered by the edit

114 MEASURING CONTINUITY IN VR Metrics Divergence from baseline Indicates how gaze behavior is altered by the edit State sequences Aggregated view of users fixations position along time: fixating on background, or ROI

115 GOAL First steps towards understanding user behavior in VR - Investigate continuity editing in VR - Analyze how different edits affect attentional behavior - Parameters of influence - Metrics describing user behavior - Analysis and results

116 GOAL First steps towards understanding user behavior in VR - Investigate continuity editing in VR - Analyze how different edits affect attentional behavior - Parameters of influence - Metrics describing user behavior - Analysis and results

117 ANALYSIS AND RESULTS Influence of ROI alignment

118 ANALYSIS AND RESULTS Influence of ROI alignment

119 ANALYSIS AND RESULTS Influence of ROI alignment

120 ANALYSIS AND RESULTS Influence of ROI alignment

121 ANALYSIS AND RESULTS Influence of ROI alignment Misalignment of ROIs non-linearly increases the time users need to find the ROI after an edit Guideline for adjusting the pace of the action?

122 Divergence from baseline ANALYSIS AND RESULTS Influence of ROI alignment

123 Divergence from baseline ANALYSIS AND RESULTS Influence of ROI alignment

124 ANALYSIS AND RESULTS Influence of ROI alignment Large misalignments alters viewer behavior also after the ROI is found. It fosters a more exploratory behavior

125 ANALYSIS AND RESULTS Influence of ROI configuration

126 ANALYSIS AND RESULTS Influence of ROI configuration Multiple ROIs before the edit elicit a more exploratory behavior after the edit.

127 ANALYSIS AND RESULTS Influence of type of edit

128 ANALYSIS AND RESULTS Influence of type of edit Edits with action continuity attract more attention to the ROI after the cut.

129 ANALYSIS AND RESULTS More in the paper! [Serrano et al. SIGGRAPH 2017]

130 CONCLUSIONS - Underlying principles of continuity in traditional cinematography hold for VR

131 CONCLUSIONS - Underlying principles of continuity in traditional cinematography hold for VR - Systematic analysis of viewer behavior and perceived continuity in VR video content. - Parameters of influence - Metrics to asses viewing behavior

132 CONCLUSIONS - Underlying principles of continuity in traditional cinematography hold for VR - Systematic analysis of viewer behavior and perceived continuity in VR video content. - Parameters of influence - Metrics to asses viewing behavior - Insights with direct implications in VR content generation

133 OUTLOOK - Some findings may not generalize to conditions outside our study - Exploration of other variables: - More cinematographic cuts - Longer movies - Complex visual content - Influence of sound

134 OUTLOOK How do people explore VR environments?

135 Thank you! Always interested in motivated students!