Face Recognition for Personal Photos using Online Social Network Context and Collaboration

Size: px

Start display at page:

Download "Face Recognition for Personal Photos using Online Social Network Context and Collaboration"

Tobias Davidson
6 years ago
Views:

1 Face Recognition for Personal Photos using Online Social Network Context and Collaboration Guest Lecture at KAIST 14 December, 2010 Wesley De Neve, Jaeyoung Choi, Yong Man Ro Image and Video Systems Lab Department of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) Yuseong gu, Daejeon, Republic of Korea e mail: wesley.deneve@kaist.ac.kr web:

2 Professional Background Licentiate degree in computer science (June 2002) at Ghent University, Ghent, Belgium Ph.D. degree in computer science engineering (February 2007) at Ghent University, Ghent, Belgium dissertation on Description driven media resource adaptation Postdoctoral researcher (September 2007) at Ghent University IBBT, Ghent, Belgium at Information and Communications University (ICU), Daejeon, Korea Senior researcher (March 2009) at KAIST, Daejeon, Korea research focus on combining visual content analysis and collective knowledge in social media applications 2/54

3 Outline Introduction Face recognition 101 Face recognition using online social network context Collaborative face recognition in online social networks Future applications Conclusions 3/54

4 Outline Introduction Face recognition 101 Face recognition using online social network context Collaborative face recognition in online social networks Future applications Conclusions 4/54

5 Introduction (1/2) The number of personal photos shared online keeps growing thanks to easy to use multimedia devices and online services thanks to cheap storage and bandwidth thanks to an increasing number of people going online Statistics Flickr (as of September 2010) hosts 5 billion images, with 3,000 new images uploaded every minute more than 40 million users Facebook (as of January 2010) more than 2.5 billion photos are uploaded each month more than 500 million active users 5/54

Introduction (2/2) Problem: digital information overload our ability to automatically organize photos does not keep up with our ability to create and

6 Introduction (2/2) Problem: digital information overload our ability to automatically organize photos does not keep up with our ability to create and store photos Promising solution automatic face detection, face recognition, and face annotation allows identity based organization and retrieval of photos 6/54

7 Face Detection, Recognition, and Annotation Barack Obama face detection Joe Biden face recognition face annotation Identity tags: Barack Obama, Joe Biden retrieval of photos based on identity tags 7/54

8 Lecture Goals and Main Sources To provide an answer to the following questions what is face recognition and why is it relevant? what is the value of face recognition using online social network context? collaborative face recognition? what are future applications of socially aware face recognition? Main academic sources Z. Stone, T. Zickler, T. Darrell, Toward Large Scale Face Recognition using Social Network Context, Proceedings of the IEEE, 2010 J.Y. Choi, W. De Neve, K. N. Plataniotis, Y.M. Ro, Collaborative Face Recognition for Improved Face Annotation in Personal Photo Collections Shared on Online Social Networks, IEEE Transactions on Multimedia, /54

9 Outline Introduction Face recognition 101 Face recognition using online social network context Collaborative face recognition in online social networks Future applications Conclusions 9/54

10 Application Areas of Face Recognition Identity verificiation face recognition is used to confirm the identity claim of a given person relevant to applications such as controlling access to buildings and computer terminals (e.g., Kinect) identity verification of passport holders (immigration) Identity recognition face recognition is used to identify an unknown person, by matching his/her face image against a gallery of known face images relevant to applications such as video surveillance face annotation in personal photo collections 10/54

11 Conceptual Design of a Face Recognition System Hillary Joe Barack Robert gallery of known face images face detection preprocessing (e.g., scaling and rotation to put eyes on fixed locations) input photo matching unknown probe face images ~ rank 1 rank 2 rank 3 rank 4 11/54

, grayscale pixel values) appearance based face recognition p r o

12 Matching Face Images Face feature vector a d dimensional vector of feature values (e.g., grayscale pixel values) appearance based face recognition p r o b e feature extraction x = [ x 1,..., x 72 ] g a l l e r y feature extraction y = [ y 1,..., y 72 ] x y n 12/54

13 Possible Outcomes of Face Recognition True negative (system correctly decides that the gallery does not contain the identity of the probe face image) False positive (system incorrectly matches the probe face image with one of the gallery face images) =? V = X False negative (system incorrectly decides that the gallery does not contain the identity of the probe face image) True positive (system correctly matches the probe face image with one of the gallery face images) =? X = V min( x - y n ) 13/54

Effectiveness of Face Recognition (1/2) Automatic

photos is a hard problem uncontrolled variations in

presence of heavy makeup, eye glasses, facial hair,

recognition is even more difficult in large online

14 Effectiveness of Face Recognition (1/2) Automatic appearance based face recognition for personal photos is a hard problem uncontrolled variations in expression, pose, lighting, and spatial resolution presence of heavy makeup, eye glasses, facial hair, and occlusions Automatic appearance based face recognition is even more difficult in large online photo collections may contain hundreds of millions of individuals 14/54

able to deal with a limited number of gallery face images the difference in appearance

15 Effectiveness of Face Recognition (2/2) probe face image rank 1 rank 2... first hit at rank 12 Appearance based face recognition for Facebook photos is only able to deal with a limited number of gallery face images the difference in appearance between individuals becomes very small relative to the difference in appearance of any particular individual 15/54

16 Room for Improvement... Gartner hype cycle describes the adoption of new technology (2) (1) appearance-based face recognition for personal photos (1) (2) appearance-based face recognition for personal photos using online social network context 16/54

17 Outline Introduction Face recognition 101 Face recognition using online social network context Collaborative face recognition in online social networks Future applications Conclusions 17/54

with their photographer are broadcasted out to the online contacts of the

18 Online Sharing of Personal Photos Photos on online social networks do not exist in isolation arrive in a batch of related photos from a trip or event are associated with their photographer are broadcasted out to the online contacts of the photographer join a collection of billions of other photos photographer sharing event 18/54

19 Online Social Network Context (1/2) Research question how to take advantage of online social network context to improve the accuracy of appearance based face recognition in large photo collections? contact list of the photographer (social network structure) availability of manual annotations (labeled face images) 19/54

20 Online Social Network Context (2/2) Contact list of the photographer used for reducing the number of known face images in the gallery from millions to hundreds of candidate identities Manually labeled face images used for learning the strength of social ties e.g., co occurrences of individuals are the result of social incentives to manually attach correct identity labels to face images ( human computation ) i.e., upon tagging, a notification is sent to the person tagged Use of a contact list and already labeled face images motivated by an empirical study for Facebook 20/54

21 Empirical Study for Facebook Collected labeled face images from 50 college aged volunteers and their friends by means of a Facebook platform application July 2009 Friends per volunteer (avg.) 645 Volunteers and friends (total #individuals) 22,108 Photos Identity tags (*) 7.7 million 8.1million (*) using an open-source frontal face detector, it was found that 32% of the 8.1 million manually attached identity tags could be reliably associated with a machine-detectable frontal face After pre processing, a collection was obtained of 2.5 million reliably labeled face images of 385,624 individuals 21/54

22 Observations (1/2) Most people can be associated with at least one identity tag out of the 22,108 volunteers and friends, 67% could be associated with at least one labeled face image Fraction of individuals tagged N times significant amount of labeled face images that can be used to train and test face recognition algorithms Number of tags (N) 22/54

23 Observations (2/2) About 30% of the tagged faces in a photographer s albums belong on average to the photographer him or herself a face recognition system can draw upon social context surrounding the photographer to reduce the set of possible identity labels People appear in photos with fewer people than they count among their Facebook friends (with less than 13%) photo co occurrence defines a subgraph of an individual s friend graph that may be more relevant for predicting co occurrence in new photos 23/54

24 Summary Empirical study for Facebook confirms the usefulness of online social network context for improving the accuracy of appearance based face recognition a contact list for recuding the number of candidate identity labels manually labeled face images for learning the strength of social ties Question how to integrate online social network context into appearance based face recognition? 24/54

25 Mathematical Modeling (1/2) Need for a mathematical tool that supports labeling of face images by combining two sources of information the appearance of each face (i.e., pixel data) as used by conventional face recognition techniques the strength of social ties (i.e., social network structure) learned from the manually labeled face images 25/54

26 Mathematical Modeling (2/2) Use of a probability model known as a Markov Random Field allows inferring an identity label y i for each face by combining node features φ i (baseline face recognition scores) edge features φ i,j (pairwise co occurrences of individuals) φ 2 y 2 φ 1,2 φ 2,3 φ 1 φ 3 y 1 φ 1,3 y 3 Markov Random Field (MRF) 26/54

27 Experimental Results (1/2) Probability proportion of face images with a correct label in the top R suggested labels Rank (R) number of suggested identity labels 27/54

28 Experimental Results (2/2) Additional observations and discussion the combined use of pixel data and social context yields better face recognition rates than the use of either information source alone the probability of having a correctly suggested label at rank one is low from a practical point of view, socially aware face recognition could be used to create a short list of R candidate identity tags room for use of other social signals social interaction (e.g., message exchanges and comments) gender and age how about relationships changing over time? 28/54

29 Outline Introduction Face recognition 101 Face recognition using online social network context Collaborative face recognition in online social networks Future applications Conclusions 29/54

30 Centralized and Decentralized Online Social Networks Centralized online social networks are highly popular (e.g., Facebook) present several problems information silos ownership of data privacy issues Decentralized online social networks are attracting more and more interest (e.g., Diaspora and Thimbl) present several advantages do not bind users to a particular online social network give users more control over data ownership and privacy 30/54

31 Architecture of Online Social Networks Centralized centralized server with all user data (e.g., contact list, photos) Decentralized each user has a personal server containing his/her user data technologies used FOAF, HTTP, SSL, OpenID, URI, etc. 31/54

32 Collaborative Face Recognition (1/2) Decentralized online social networks each user will have a personalized face recognition engine, optimized for recognizing face images of the user in question Research question given a user, how about using the face recognition engines of other users for face annotation in the personal photos of the given user? 32/54

Collaborative Face Recognition (2/2) photos FR

photos FR engine personal server photographer photos

photos FR engine personal server contact 3 (friend)

33 Collaborative Face Recognition (2/2) photos FR engine personal server contact 1 (family member) photos FR engine personal server photographer photos FR engine personal server contact 2 (family member) photos FR engine personal server contact 3 (friend) photos FR engine personal server contact 4 (co-worker) 33/54

Proposed Framework for Collaborative FR FR engine 1 Mark Zuckerberg

.. nametagged photo Research challenges FR engine K how to select

how to merge multiple face recognition scores into a single

34 Proposed Framework for Collaborative FR FR engine 1 Mark Zuckerberg Jet Li face detection FR engine 2 fusion input photo... nametagged photo Research challenges FR engine K how to select expert face recognition (FR) engines? how to merge multiple face recognition scores into a single decision? how to deal with heterogeneous FR engines? this is, FR engines using different face recognition techniques 34/54

35 Selection of Expert FR Engines Contact list contact 1 Weighted social graph model for the photographer contact 2 contact 3 contact 4 contact 5 contact 6 Labeled face images occurrence probabilities co occurrence probabilities the thicker the line, the stronger the social tie, the more important the personalized FR engine of the corresponding contact 35/54

36 Empirical Study for Cyworld Retrieval of 547,991 personal photos from four volunteers and their contacts on Cyworld, a Korean online social network ID Age Gender Contacts Years active Volunteer 1 28 Female Volunteer 2 29 Male Volunteer 3 30 Female Volunteer 4 27 Male 84 8 ID Photos Photos with tagged individuals Individuals tagged Detected face images Volunteer 1 251, ,422 2, ,363 Volunteer 2 109,021 81,211 1,834 94,452 Volunteer 3 117,772 94,297 2, ,408 Volunteer 4 69,987 59,753 1,302 64,412 36/54

37 Observations A lot of face images belong to the photographer numbers range from 3.4% for Volunteer 2 to 14.7% for Volunteer 1 A lot of face images belong to contacts of the photographer numbers range from 73% for Volunteer 3 to 93% for Volunteer 4 the identity of probe face images not belonging to individuals enrolled in the contact list of a volunteer needs to be asked to the volunteer Most of the face images only belong to a small number of contacts of the photographer e.g., 91% of the probe face images of Volunteer 1 belong to 28 contacts 37/54

38 Effectiveness of Selecting FR Engines (1/2) head of the distribution experimental results for Volunteer 1, having 165 Cyworld contacts Relevance FR engine 28 out of 166 appearancebased FR engines come with high relevance values ( inner social circle ) FR engine index (in decreasing order of relevance) 38/54

39 Effectiveness of Selecting FR Engines (2/2) Number of correctly recognized face images x number of correctly recognized face images when 28 FR engines are used experimental results for Volunteer 1, having 165 Cyworld contacts the collaborative use of 28 out of 166 appearancebased FR engines results in a maximum number of correctly recognized face images Number of FR engines used 39/54

40 Experimental Results (1/2) Probability (proportion of face images with a correct label in the top R suggested labels) Collaborative FR (Bayesian) Collaborative FR (Voting) Non-collaborative FR (Avg.) experimental results for the 28 FR engines selected for Volunteer 1, for both collaborative and noncollaborative FR Rank (R) 40/54

41 Experimental Results (2/2) Explanatory notes non collaborative FR accuracy is measured by averaging the face annotation accuracy of all FR engines used to perform collaborative FR collaborative FR weighting of the different FR scores is either done using a Bayesian decision rule or majority voting take into account the relevance of a FR engine Collaborative FR is more effective than non collaborative FR by virtue of a complementary effect caused by fusion of multiple face recognition scores 41/54

42 Outline Introduction Face recognition 101 Face recognition using online social network context Collaborative face recognition in online social networks Future applications Conclusions 42/54

43 Microsoft OneAlbum OneAlbum project allows users to find relevant photos across a social network (e.g., all photos taken by friends at a birthday party and shared on Facebook) makes use of unsupervised event recognition time stamps visual content socially aware face recognition the social graph (co )occurrence statistics 43/54

extracts a face feature vector and sends the feature vector to a server server matches the feature vector with a

44 Augmented Identity Augmented reality superimposes virtual objects and info on top of the real world, facilitating interaction between virtual and real objects Augmented identity user points a smart phone at a person software extracts a face feature vector and sends the feature vector to a server server matches the feature vector with a pre registered identity in a database server sends back the identity of the subject, as well as contact information 44/54

Socially Aware Advertisement Billboards Quotes Ray Ozzie (ex Microsoft) [We will see] service connected devices going far beyond just the screen, keyboard and mouse : humanly natural conscious

45 Socially Aware Advertisement Billboards Quotes Ray Ozzie (ex Microsoft) [We will see] service connected devices going far beyond just the screen, keyboard and mouse : humanly natural conscious devices that ll see, recognize, hear & listen to you and what s around you, that ll feel your touch and gestures and movement, that ll detect your proximity to others; that ll sense your location, direction, altitude, temperature, heartbeat & health. Nicholas Negroponte (MIT Media Lab) Every surface will be a display. Everything will be linked to every other thing. Things will know where they are and some may know who they are. Face recognition facilitates customized advertising 45/54

augmented identity) research challenges robust and large scale face recognition using the entire social graph

46 Socially Aware Video Surveillance Video surveillance used to prevent and detect crime used to identify terrorists Socially aware video surveillance identification by means of social network knowledge (cf. augmented identity) research challenges robust and large scale face recognition using the entire social graph as a gallery gathering of representative data, including access to a social graph privacy issues (cf. Google Goggles) 46/54

with will be able to learn about characteristics of each individual and treat them uniquely as individuals leads to

47 Socially Aware Robots Humanoid robots overall appearance is based on that of the human body, allowing interaction with made for human tools or environments need to have the ability to recognize and remember people they interact with will be able to learn about characteristics of each individual and treat them uniquely as individuals leads to complex social behavior, such as cooperation, dislike, loyalty, and affection prototypes are already accessing Facebook 47/54

48 Outline Introduction Face recognition 101 Face recognition using online social network context Collaborative face recognition in online social networks Future applications Conclusions 48/54

49 Conclusions Online social networks contain a vast collection of collective knowledge ( human computation ) allow researchers to test algorithms in realistic conditions without exceptional data collection effort Online social network context and collaboration allow for a substantial increase in the effectiveness of appearancebased face recognition for personal photos shared online Socially aware face recognition will enable applications in the (near) future that may have a tremendous impact on our daily lifes 49/54

50 Thank you! Any questions or comments? Contact information e mail: wesley.deneve@kaist.ac.kr web: 50/54

51 Video Demos Microsoft OneAlbum [online] Augmented identity [online] [online] 51/54

52 References (1/2) 1. Z. Stone, T. Zickler, T. Darrell, Autotagging Facebook: Social Network Context Improves Photo Annotation, Proc. of the IEEE Computer Vision and Pattern Recognition Workshops, Z. Stone, T. Zickler, T. Darrell, Toward Large Scale Face Recognition using Social Network Context, Proceedings of the IEEE, J.Y. Choi, W. De Neve, K. N. Plataniotis, Y.M. Ro, Collaborative Face Recognition for Improved Face Annotation in Personal Photo Collections Shared on Online Social Networks, IEEE Transactions on Multimedia, J.Y. Choi, W. De Neve, Y.M. Ro, K. N. Plataniotis, Automatic Face Annotation in Personal Photo Collections Using Context Based Unsupervised Clustering and Face Information Fusion, IEEE Transactions on Circuits and Systems for Video Technology, K. W. Bowyer, Face Recognition Technology: Security versus Privacy, IEEE Technology and Society Magazine, /54

53 References (2/2) 6. N. Mavridis, W. Kazmi, P. Toulis, Friends with Faces: How Social Networks Can Enhance Face Recognition and Vice Versa, Computational Social Network Analysis, L. Aryananda, Online and Unsupervised Face Recognition for Humanoid Robot: Toward Relationship with People, Proc. of the 2001 IEEE RAS International Conference on Humanoid Robots, C. Au Yeung, I. Liccardi, K. Lu, O. Seneviratne, T. Berners Lee, Decentralization: The Future of Online Social Networking, W3C Workshop on the Future of Social Networking, H. Sohn, W. De Neve, Y. M. Ro, Privacy Protection in Video Surveillance Systems: Analysis of Subband Adaptive Scrambling in JPEG XR, IEEE Transactions on Circuits and Systems for Video Technology, P. Levy (Author), R. Bononno (Translator), Collective Intelligence: Mankind's Emerging World in Cyberspace, Perseus Books, /54

54 Picture Credits Flickr: Barack Obama's Photostream [online] Flickr: Mynameisharsha's Photostream [online] 54/54

Experiments with An Improved Iris Segmentation Algorithm

Experiments with An Improved Iris Segmentation Algorithm Xiaomei Liu, Kevin W. Bowyer, Patrick J. Flynn Department of Computer Science and Engineering University of Notre Dame Notre Dame, IN 46556, U.S.A.