MoSculp: Interactive Visualization of Shape and Time

Size: px

Start display at page:

Download "MoSculp: Interactive Visualization of Shape and Time"

Randell Daniels
5 years ago
Views:

1 MoSculp: Interactive Visualization of Shape and Time Xiuming Tali Tianfan Andrew Qiurui Jiajun Stefanie William T. Zhang 1 Dekel 1,2 Xue 1,2 Owens 1,3 He 1,2 Wu 1 Mueller 1 Freeman 1,2 1 MIT CSAIL 2 Google Research 3 UC Berkeley

2 1

3 2

4 3

5 4

6 5

7 Video Courtesy of Tom Buehler (MIT CSAIL) 6

8 Outline Related Work System Walkthrough User Studies Approach Results Conclusion 7

9 Outline Related Work System Walkthrough User Studies Approach Results Conclusion 8

10 Motivation Muybridge, The Human Figure in Motion,

11 Motivation Muybridge, The Human Figure in Motion, 1901 Edgerton, Back Dive,

12 Motivation Muybridge, The Human Figure in Motion, 1901 Edgerton, Back Dive, 1954 Duchamp, Nude Descending a Staircase, No. 2,

13 Related Work Edgerton, Stroboscopic Photography, D 12

14 Related Work Edgerton, Stroboscopic Photography, D Freeman & Zhang, Shape- Time Photography, CVPR 03 Requires a depth camera 13

15 Related Work vs. Ours Edgerton, Stroboscopic Photography, D Freeman & Zhang, Shape- Time Photography, CVPR 03 Requires a depth camera MoSculp 3D w/ an RGB camera 14

16 Outline Related Work System Walkthrough User Studies Approach Results Conclusion 15

17 System Walkthrough 16

18 17

19 18

20 19

21 20

22 21

23 22

24 23

25 24

26 25

27 Outline Related Work System Walkthrough User Studies Approach Results Conclusion 26

28 User Studies: Design Choices With Floor Reflections Preferred by 82% Without 27

29 User Studies: Efficacy in Conveying Motion Baseline 1 (Stroboscopic) Baseline 2 (Shape-Time) MoSculp Preferred by 75% 28

30 Outline Related Work System Walkthrough User Studies Approach Results Conclusion 29

31 3D Shape & Pose Estimation Input Video Motion Sculpture Generation Depth-Preserving Compositing Overview 30

32 3D Shape & Pose Estimation Input Video Motion Sculpture Generation Depth-Preserving Compositing Overview 31

33 Approach: 2D Keypoint Detection Time: t [Cao et al., CVPR 17] 32

34 Approach: 2D Keypoint Detection Time: t + 1 [Cao et al., CVPR 17] 33

35 Hidden Markov Model 34

36 Approach: 3D Estimation Solve for the best shape and poses jointly for the clip 35

37 Approach: 3D Estimation Solve for the best shape and poses jointly for the clip Small reprojection error 2D Image 3D Model [Loper et al., ToG 15] 36

38 Approach: 3D Estimation Solve for the best shape and poses jointly for the clip Small reprojection error Large probability of the poses p( ) > p( ) 37

39 Approach: 3D Estimation Solve for the best shape and poses jointly for the clip Small reprojection error Large probability of the poses Smooth evolution of poses 38

40 Approach: 3D Estimation Solve for the best shape and poses jointly for the clip Small reprojection error Large probability of the poses Smooth evolution of poses Original Camera View Novel View Per-Frame Optimization [Bogo et al., ECCV 16] 39

41 Approach: 3D Estimation Solve for the best shape and poses jointly for the clip Small reprojection error Large probability of the poses Smooth evolution of poses Original Camera View Novel View Our Joint Optimization 40

42 3D Shape & Pose Estimation Input Video Motion Sculpture Generation Depth-Preserving Compositing Overview 41

43 Approach: Sculpture Generation 42

44 43

45 44

46 3D Shape & Pose Estimation Input Video Motion Sculpture Generation Depth-Preserving Compositing Overview 45

47 Approach: Compositing Key challenge: how to put together 3D sculpture and 2D video? 46

48 Approach: Compositing Naive Compositing: sculpture on top of the frames 47

49 Approach: Compositing Full 3D Rendering: texturing the 3D models Skirt Not Covered by 3D Model 48

50 Approach: Compositing Solution: depth-preserving composite Rendered Depth (Refined) 49

51 Approach: Compositing Solution: depth-preserving composite 50

52 Approach: Before Refinement 51

53 Approach: After Refinement 52

54 Outline Related Work System Walkthrough User Studies Approach Results Conclusion 53

55 54

56 55

57 Single-Frame Shape and Pose Estimation 56

58 Our Joint Estimation 57

59 58

60 59

61 60

62 61

63 Texture from Original Frames 62

64 63

65 64

66 Handling a Moving Camera 65

67 66

68 Outline Related Work System Walkthrough User Studies Approach Results Conclusion 67

69 Limitation: Repeated, Localized Motion 68

70 69

71 Conclusion 70

72 Please come to our demo D-12 for more! Thank you! Video Courtesy of Tom Buehler (MIT CSAIL) 71

Limitations of the Medium, compensation or accentuation

The Art and Science of Depiction Limitations of the Medium, compensation or accentuation Fredo Durand MIT- Lab for Computer Science Limitations of the medium The medium cannot usually produce the same