Image Analysis & Searching 1
Searching Photos Look for photos like this one: Look for beach photos Look for photos taken Sept. 15, 2000 Look for photos with: Look for photos with Aunt Thelma 2
Annotating Photos Tags (descriptive text) human or automatically generated free vocabulary vs fixed vocabulary include location within photo Features (image information) simple (color, brightness,...) complex (object class, person name,...) 3
What Simple Features Can We Extract? 4
Simple Features Turquoise Blue White Green Brown Black Yellow Pink... 5
Simple Features Turquoise Blue White Green Brown Black Yellow Pink... 6
Simple Features Turquoise Blue White Green Brown Black Yellow Pink... 7
Classifier 8
Classifier C indoor outdoor 9
Classifier C indoor outdoor beach meadow city 10
Decision Tree N pink > 30% N Y night, city night, people brightness > 50% Y turquoise > 60% N Y daytime, meadow daytime, beach 11
Decision Tree N pink > 30% N Y night, city night, people brightness > 50% Y turquoise > 60% N Y daytime, meadow daytime, beach need to learn questions to ask... 12
Weighted Score Example: Beach score = w 1 *Turquoise + w 2 *Blue + w 3 *Red +... If Beach score > 0.8 then add beach tag 13
Weighted Score Example: Beach score = w 1 *Turquoise + w 2 *Blue + w 3 *Red +... need to learn weights... If Beach score > 0.8 then add beach tag 14
How do we learn? Machine Learning often use training data beach=yes 0000111000011110000 beach=yes 0000110001001110000 beach=yes 0000110000011100000 beach=no 0101000111011010110 beach=no 1100000011100010000 15
How do we learn? Machine Learning feature #6 often use training data beach=yes 0000111000011110000 beach=yes 0000110001001110000 beach=yes 0000110000011100000 beach=no 0101000111011010110 beach=no 1100000011100010000 Prob[ f6=1 beach ] = 1 Prob[ f6 = 0 not beach ] = 0 16
How do we learn? Machine Learning feature #6 often use training data beach=yes 0000111000011110000 beach=yes 0000110001001110000 beach=yes 0000110000011100000 beach=no 0101000111011010110 beach=no 1100000011100010000 Prob[ f6=1 beach ] = 1 Prob[ f6 = 0 not beach ] = 0 Therefore, give f6 a high weight! beach score = w1*f1 + w2*f2 + w3*f3 +... 17
Machine Learning a.k.a Data Mining Has many applications: Recommending movies Finding pages on Google/Bing What to stock at Walmart Diagnosing diseases Big Data ML/DM Trends Recommendations Anomalies Predictions 18
More Complex Features Color/brightness by position in image blue white turquoise 19
More Complex Features Edges & Blobs 20
More Complex Features Textures, Patterns 21
Object recognition: what does it involve? slides courtesy: Fei Fei Li
Verification: is that a lamp?
Detection: are there people somewhere?
Identification: What is this? (A: Potala Palace)
Object categorization mountain tree banner building street lamp people vendor
Scene and context categorization outdoor city
Computational photography
Face Priority AF on Nikon D7100 Discuss contrast detection and phase detection AF 29
Assisted driving Pedestrian and car detection Ped meters Ped Car meters Lane detection Collision warning systems with adaptive cruise control, Lane departure warning systems, Rear object detection systems,
Improving online search Query: STREET Organizing photo collections
Challenges 1: view point variation Michelangelo 1475-1564
Challenges 2: illumination slide credit: S. Ullman
Challenges 3: occlusion Magritte, 1957
Challenges 4: scale
Challenges 5: deformation Xu, Beihong 19
Challenges 6: background clutter Klimt, 1913
Challenges 7: intra-class variation
How many object categories are there? Biederman 1987
How many object categories are there? Biederman 1987
Discriminative Model Direct modeling of p( zebra image) p( no zebra image) Decision boundary Zebra Non-zebra
Using Representation Generative / discriminative / hybrid
Representation Generative / discriminative / hybrid Appearance only or location and appearance
http://www.faceplusplus.com/demo-landmark/ 44
Annotating Photos Tags (descriptive text) human or automatically generated free vocabulary vs fixed vocabulary include location within photo Features (image information) simple (color, brightness,...) complex (object class, person name,...) 45
What tags are useful? Date Just month Time of day Location Weather Number of people Daylight Camera used Aperture Others??? Where do we get? 46
What tags are useful? 47
Next: What Can We Do With Tags? 48
Viewing Photos on Map 49
Searching Photos Direct: Example: Find photos with Zach Indirect: Example (1) Search my photos for Eiffel Tower My photo: Tag: none GPS: x1, y1 50
Searching Photos Direct: Example: Find photos with Chris Indirect: Example (1) Search my photos for Eiffel Tower (2) Search database for Eiffel Tower My photo: Tag: none GPS: x1, y1 Photo in database: Tag: Eiffel Tower GPS: x1, y1 51
Searching Photos Direct: Example: Find photos with Chris Indirect: Example (1) Search my photos for Eiffel Tower (2) Search database for Eiffel Tower My photo: Tag: none GPS: x1, y1 (3) Search my photos for nearby photos Photo in database: Tag: Eiffel Tower GPS: x1, y1 52
Organizing Photos How can we organize and view photos that have time and location tags? 53
54
55
Identifying clusters of related photos 56
57
Crowd Sourcing Paid, e.g., Mechanical Turk Games, e.g., gwap.com 58
59
60
grass 61
62
Summary Types of annotations Tags Features How to Obtain Image analysis Human input (crowd sourcing) How to Use Searching Organizing 63
Bonus: Entity Resolution a e f b d g i c h j 64
crowd powered 65
ER with Sport Photos 66
End