Image Analysis & Searching

Transcription

1 Image Analysis & Searching 1

2 Searching Photos Look for photos like this one: Look for beach photos Look for photos taken Sept. 15, 2000 Look for photos with: Look for photos with Aunt Thelma 2

3 Annotating Photos Tags (descriptive text) human or automatically generated free vocabulary vs fixed vocabulary include location within photo Features (image information) simple (color, brightness,...) complex (object class, person name,...) 3

4 What Simple Features Can We Extract? 4

5 Simple Features Turquoise Blue White Green Brown Black Yellow Pink... 5

6 Simple Features Turquoise Blue White Green Brown Black Yellow Pink... 6

7 Simple Features Turquoise Blue White Green Brown Black Yellow Pink... 7

8 Classifier 8

9 Classifier C indoor outdoor 9

10 Classifier C indoor outdoor beach meadow city 10

11 Decision Tree N pink > 30% N Y night, city night, people brightness > 50% Y turquoise > 60% N Y daytime, meadow daytime, beach 11

12 Decision Tree N pink > 30% N Y night, city night, people brightness > 50% Y turquoise > 60% N Y daytime, meadow daytime, beach need to learn questions to ask... 12

13 Weighted Score Example: Beach score = w 1 *Turquoise + w 2 *Blue + w 3 *Red +... If Beach score > 0.8 then add beach tag 13

14 Weighted Score Example: Beach score = w 1 *Turquoise + w 2 *Blue + w 3 *Red +... need to learn weights... If Beach score > 0.8 then add beach tag 14

15 How do we learn? Machine Learning often use training data beach=yes beach=yes beach=yes beach=no beach=no

16 How do we learn? Machine Learning feature #6 often use training data beach=yes beach=yes beach=yes beach=no beach=no Prob[ f6=1 beach ] = 1 Prob[ f6 = 0 not beach ] = 0 16

17 How do we learn? Machine Learning feature #6 often use training data beach=yes beach=yes beach=yes beach=no beach=no Prob[ f6=1 beach ] = 1 Prob[ f6 = 0 not beach ] = 0 Therefore, give f6 a high weight! beach score = w1*f1 + w2*f2 + w3*f

18 Machine Learning a.k.a Data Mining Has many applications: Recommending movies Finding pages on Google/Bing What to stock at Walmart Diagnosing diseases Big Data ML/DM Trends Recommendations Anomalies Predictions 18

19 More Complex Features Color/brightness by position in image blue white turquoise 19

20 More Complex Features Edges & Blobs 20

21 More Complex Features Textures, Patterns 21

22 Object recognition: what does it involve? slides courtesy: Fei Fei Li

23 Verification: is that a lamp?

24 Detection: are there people somewhere?

25 Identification: What is this? (A: Potala Palace)

26 Object categorization mountain tree banner building street lamp people vendor

27 Scene and context categorization outdoor city

28 Computational photography

29 Face Priority AF on Nikon D7100 Discuss contrast detection and phase detection AF 29

30 Assisted driving Pedestrian and car detection Ped meters Ped Car meters Lane detection Collision warning systems with adaptive cruise control, Lane departure warning systems, Rear object detection systems,

31 Improving online search Query: STREET Organizing photo collections

32 Challenges 1: view point variation Michelangelo

33 Challenges 2: illumination slide credit: S. Ullman

34 Challenges 3: occlusion Magritte, 1957

35 Challenges 4: scale

36 Challenges 5: deformation Xu, Beihong 19

37 Challenges 6: background clutter Klimt, 1913

38 Challenges 7: intra-class variation

39 How many object categories are there? Biederman 1987

40 How many object categories are there? Biederman 1987

41 Discriminative Model Direct modeling of p( zebra image) p( no zebra image) Decision boundary Zebra Non-zebra

42 Using Representation Generative / discriminative / hybrid

43 Representation Generative / discriminative / hybrid Appearance only or location and appearance

44 44

45 Annotating Photos Tags (descriptive text) human or automatically generated free vocabulary vs fixed vocabulary include location within photo Features (image information) simple (color, brightness,...) complex (object class, person name,...) 45

46 What tags are useful? Date Just month Time of day Location Weather Number of people Daylight Camera used Aperture Others??? Where do we get? 46

47 What tags are useful? 47

48 Next: What Can We Do With Tags? 48

49 Viewing Photos on Map 49

50 Searching Photos Direct: Example: Find photos with Zach Indirect: Example (1) Search my photos for Eiffel Tower My photo: Tag: none GPS: x1, y1 50

51 Searching Photos Direct: Example: Find photos with Chris Indirect: Example (1) Search my photos for Eiffel Tower (2) Search database for Eiffel Tower My photo: Tag: none GPS: x1, y1 Photo in database: Tag: Eiffel Tower GPS: x1, y1 51

52 Searching Photos Direct: Example: Find photos with Chris Indirect: Example (1) Search my photos for Eiffel Tower (2) Search database for Eiffel Tower My photo: Tag: none GPS: x1, y1 (3) Search my photos for nearby photos Photo in database: Tag: Eiffel Tower GPS: x1, y1 52

53 Organizing Photos How can we organize and view photos that have time and location tags? 53

54 54

55 55

56 Identifying clusters of related photos 56

57 57

58 Crowd Sourcing Paid, e.g., Mechanical Turk Games, e.g., gwap.com 58

59 59

60 60

61 grass 61

62 62

63 Summary Types of annotations Tags Features How to Obtain Image analysis Human input (crowd sourcing) How to Use Searching Organizing 63

64 Bonus: Entity Resolution a e f b d g i c h j 64

65 crowd powered 65

66 ER with Sport Photos 66

67 End