3D-Assisted Image Feature Synthesis for Novel Views of an Object

Transcription

1 3D-Assisted Image Feature Synthesis for Novel Views of an Object Hao Su* Fan Wang* Li Yi Leonidas Guibas * Equal contribution

2 View-agnostic Image Retrieval Retrieval using AlexNet features Query

3 Cross-view Image Comparison

4 Cross-view Image Comparison The comparison is between the underlying 3D objects

5 Reconstruct 3D and then compare? Su et al, SIGGRAPH 14 Kar et al, CVPR 15 Huang et al, SIGGRAPH 15

6 Single-image based 3D Reconstruction is hard Common dependencies: Many dependencies Not Robust Fg/bg segmentation Slow Keypoint detection 2D image part segmentation 3D shape part segmentation 2D-3D Correspondence Non-convex iterative optimization

7 Our Formulation: Novel View Feature Synthesis Observed view (HoG feature as an example)

8 Our Novel View Feature Synthesis Results (HoG feature as an example)

9 Outline Motivation Approach Applications Method Diagnosis Conclusion

10 Key idea Learn from a dataset of many objects with multi-view features

11 Key idea Learn from a dataset of multi-view features The dataset is generated by rendering 3D models d

12 Key idea Learn from a dataset of multi-view features The dataset is generated by rendering large-scale 3D models

13 3D-assisted Feature Synthesis: Nearest Neighbour Observed view image Novel view feature (HoG feature as an example)

14 3D-assisted Feature Synthesis: Nearest Neighbour Observed view image Strong assumption: very similar model exists Novel view feature (HoG feature as an example)

15 3D-assisted Feature Synthesis: Multiple Shapes Observed view image... Novel view feature (HoG feature as an example)

16 3D-assisted Feature Synthesis: Multiple Shapes Attention: Brain games start!

17 Pipeline Observed view image Novel view feature (HoG feature as an example)

18 Pipeline Observed view image Novel view feature (HoG feature as an example)

19 Pipeline Observed view image Novel view feature (HoG feature as an example)

20 Pipeline Observed view image + + Novel view feature (HoG feature as an example)

21 Pipeline Observed view image + + Novel view feature (HoG feature as an example)

22 Pipeline Observed view image Locally Linear Reconstruction Novel view feature (HoG feature as an example)

23 Pipeline Observed view image Locally Linear Reconstruction Novel view feature (HoG feature as an example)

24 Pipeline Observed view image Locally Linear Reconstruction Novel view feature (HoG feature as an example)

25 Pipeline Observed view image Locally Linear Reconstruction Novel view feature Inter-shape relationship (HoG feature as an example)

26 Surrogate Relationship Discovery Observed view image Locally Linear Reconstruction ? + + Novel view feature Inter-shape relationship (HoG feature as an example)

27 Surrogate Relationship Discovery Observed view Shape Collection Novel view

28 Surrogate Relationship Discovery Observed view Shape Collection Novel view Surrogate suitability matrix

29 Formal Definition of Surrogate Suitability Shape Collection Observed view Assume A, B are discrete random variables A Novel view B

30 Formal Definition of Surrogate Suitability Shape Collection Observed view Assume A, B are discrete random variables (a 1, b 1 ), (a 2, b 2 ), are i.i.d samples of (A, B) A Novel view e.g. a 1 a 2 b 1 b 2 B

31 Formal Definition of Surrogate Suitability Shape Collection Observed view Assume A, B are discrete random variables (a 1, b 1 ), (a 2, b 2 ), are i.i.d samples of (A, B) A Novel view e.g. a 1 a 2 Surrogate suitability: b 1 b 2 B γ A; B = log P(b 1 = b 2 a 1 = a 2 )

32 Formal Definition of Surrogate Suitability Shape Collection Observed view Assume A, B are discrete random variables (a 1, b 1 ), (a 2, b 2 ), are i.i.d samples of (A, B) How well can the sameness at A predict the sameness at B? A Novel view e.g. a 1 a 2 Surrogate suitability: b 1 b 2 B γ A; B = log P(b 1 = b 2 a 1 = a 2 )

33 Formal Definition of Surrogate Suitability Shape Collection Observed view Assume A, B are discrete random variables (a 1, b 1 ), (a 2, b 2 ), are i.i.d samples of (A, B) How well can the sameness at A predict the sameness at B? A Novel view e.g. a 1 a 2 Cross-view transfer of relationships B b 1 b 2 Surrogate suitability: γ A; B = log P(b 1 = b 2 a 1 = a 2 )

34 Estimation of Surrogate Suitability Derivation shows H R : Renyi-entropy

35 Estimation of Surrogate Suitability Derivation shows Sample complexity: tight bound Θ V A + V B where V A and V B are vocabulary size of A and B

36 Estimation of Surrogate Suitability Derivation shows Sample complexity: tight bound Θ V A + V B where V A and V B are vocabulary size of A and B Theoretically optimal algorithm is proposed that reaches the bound

37 Estimation of Surrogate Suitability Derivation shows Sample complexity: tight bound Θ V A + V B where V A and V B are vocabulary size of A and B Theoretically optimal algorithm is proposed that reaches the bound Strong connection with Mutual Information

38 More Visualization of Surrogate Suitability Matrix Novel view Observed view B

39 More Visualization of Surrogate Suitability Matrix Novel view Observed view B

40 More Visualization of Surrogate Suitability Matrix Novel view Observed view B

41 Review of Pipeline Observed view image Novel view feature

42 Inter-shape relationship Review of Pipeline Observed view image + Inter-shape relationship: Knowledge transfer from 3D shape database to+ new instance + Novel view feature

43 Intra-shape relationship Inter-shape relationship Review of Pipeline Observed view image Intra-shape relationship: + Inter-shape relationship: Knowledge transfer from observed view to novel view Knowledge transfer from 3D shape database to+ new instance + Novel view feature

44 Outline Motivation Approach Applications Method Diagnosis Conclusion

45 Application: Cross-view localized image comparison

46 Cross-view Image Retrieval

47 Application: View-agnostic Image Retrieval HoG L2 vertical bars swivel base Ours (combined HoG)

48 Application: View-agnostic Image Retrieval HoG L2 vertical bars swivel base Ours (combined HoG)

49 Application: View-agnostic Image Retrieval HoG L2 vertical bars swivel base Ours (combined HoG)

50 Part-based View-agnostic Image Retrieval

51 Generalizability to Many Feature Types Task: fine-grained retrieval (images and annotations are from ImageNet) Metric: Average Precision

52 Outline Motivation Approach Applications Method Diagnosis Conclusion

53 How many shapes are sufficient? 200 (Measured by Average Precision on Fine-grained retrieval for Chairs)

54 How many neighboring shapes for interpolation? 80 (Measured by Average Precision on Fine-grained retrieval for Chairs)

55 How well can one view predict another view? Controlled diagnosis on renderings Cross-view retrieval rank

56 Outline Motivation Approach Applications Method Diagnosis Conclusion

57 Conclusion A novel framework for synthesizing object features at novel views 3D shape database provides the knowledge of feature synthesis For relationship transfer, surrogate suitability is defined, which is a type of predictability between random variables. A theoretically optimal estimator is proposed

58 Thank you!

59

60