Community detection from Virtual identities

Transcription

1 Community detection from Virtual identities Prof Muttukrishnan Rajarajan Head of Information Security Group City University London, UK

2 Knowledge discovery and data mining Many data mining methods have been proposed for mining useful patterns in text documents These methods include identifying semantics of words, clustering and classification of text documents However, these methods have limited applications in names and social media analysis We have developed text mining techniques that perform address categorization based on semantics of surnames Community identification in surnames and forenames Establishing statistical inferences and interactions among the communities of forenames and surnames 2

3 Preprocessing Surname Correction: Typographical errors of surnames were corrected using Jaro-Winkler distance and corpus statistics Surname Identification: Surname in a substring was identified that has no identification mark for surname. It constructs suffix tree for a name that has no identification mark for surname. Further it identifies surname in a set of surnames that present as substring in name. 3

4 Knowledge discovery in names data set In general, a set of names preserve hidden structures (communities) which represent ethnic, geographic, cultural and genetic structure. In addition to this addresses often contains a surname or a forename as substring in it. The proposed address categorization based on semantics of surnames involves two phases Phase 1: Surnames collected at several locations were represented in a vector space model and clustered surnames that co-occur together at several locations. Phase 2: address categorization was performed if an address has surname as substring in it then categorize into the surname s cluster. 4

5 address categorization based on semantics of surnames Reference: Suresh Veluru, Yogachandran Rahulamathavan, P. Viswanath, Paul Longley, and Muttukrishnan Rajarajan, Address Categorization based on Semantics ofsurnames, IEEE Symposium Series on Computational Intelligence (IEEE SSCI 2013), pp ,

6 Experimental results for UK data set Cluster 9 Surname Location Knowles Outer hebrides rosalyneknowles167@ domain.com Hayes Bolton john_hayes_uk34@do main.com Lee Lincon lee.brown@domain.co m White Bath ler.white@domain.com Stuart Zetland stuartleemould@domai n.com Sheppard Walsall michaelsheppard1@d omain.com Dickinson Lancaster pamdickinson9834@d omain.com Hanson Huddersfield K_hanson231@domai n.com Marsden Halifax sandramarsden@dom ain.com Cluster 30 Surname Location Kaur Uxbridge kaurp_uk@dom.com Singh Uxbridge Cluster 10 Surname Location Cartwright Dudley nicola_cartwrig ht@dom.com Dodd Outer_he brides g_dodds@dom.com Lamb Durham lambrini_14@d om.com Brady Belfast Fleming Zetland Cluster 18 Surname Location Khan Bradford khan_tabs001@dom.com Mahmood Bradford asimmahmood@dom.com Iqbal Bradford Ali Bradford ali.mason@dom.com 6

7 Generative probabilistic models Latent Dirichlet Allocation (LDA) is a generative probabilistic approach which is an efficient method than Latent semantic analysis to identify topics in text documents. LDA model can be applied to identify communities in names data set. However, name contains forename and surname. Hence, finding correlation among forename and surname in each community or across communities is a challenging problem. 7

8 Generative Probabilistic models Several variants of LDA model are proposed to identify Communities in forenames and surnames Correlation among surnames and forenames in each community Correlation among communities of surnames and forenames. The basic idea is each location consists of proportions of communities (either surname or forename communities). Each community consists of a multinomial distribution of surnames or forenames. Communities can be identified by estimating hidden structure in either forename or surname. 8

9 Results Indian data set Forename Correlated LDA Community 1 Forenames Probability arun anoop sreejith biju aneesh divya anish prasanth abhilash ajith renjith Surnames Probability thomas joseph mathew george nair jose john abraham varghese Kerala Community Surname Correlated LDA Community 11 Surnames Probability das ghosh roy chakraborty mukherjee banerjee saha dutta sarkar chatterjee Forenames Probability amit abhijit sanjay arindam anirban abhishek suman subrata kaushik partha sandip santanu biswajit sourav Bengali Community UK data set Forename Correlated LDA Community 17 Forenames Probability eugen black rice yu dren brown hil srun E-4 hai E-4 feng E-4 Surnames Probability jones li smith david luo rosa E-4 liu E-4 zhao 7.993E-4 Chinese Community Surname Correlated LDA Community 6 Surnames Probability patel khan ali ahmed singh hussain Forenames Probability john mohammed david paul Michael sarah muhammad ali Asian Community 9

10 Indian names data set Testing was carried out on 27 held-out location names for Indian data set and 10 held-out locations for UK data set. If lower the perplexity then better captures communities Observation: In India data set, surnames capture communities better than forenames. 10

11 UK names data set Observation: In UK data set, forenames capture communities better than surnames 11

12 Results for UK data set English Community S-Community 15 Surnames Probability smith jones taylor brown williams johnson wilson wright F-Community Probability F-Community F-Community F-Community F-Community8 Forename Probability sian gareth rhian bethan ceri Surname Community Correlated LDA F-Community2 Forename Probability james william john margaret robert fiona F-Community1 Forename Probability david john paul michael sarah F-Community9 Forename Probability maria anna ali monika george Asian Community S-Community 5 Surnames Probability patel khan ali ahmed singh hussain shah begum F-Community Probability F-Community F-Community F-Community F-Community10 Forename Probability mohammed muhammad imran ali mohamed

13 Results for India data set Surname Community Correlated LDA Assam Community S-Community 1 Surnames Probability saikia gogoi baruah borah deka kalita sarmah E-4 hazarika E-4 F-Community Probability F-Community F-Community F-Community F-Community9 Forename miya showkat bibi barman manzoor emily emi Probability 2.48E E E E E E E-5 F-Community2 Forename Probability amit abhijit arindam anirban subrata kaushik Bengali Community S-Community 11 Surnames Probability das ghosh roy chakraborty mukherjee banerjee saha dutta F-Community Probability F-Community F-Community F-Community F-Community7 Forename Probability ram amit sanjay rajesh manoj anil F-Community4 Forename Probability rajesh sanjay amit vijay santosh sachin

14 Twitter Analysis for community detection

15 Twitter Languages

16 Twitter Ethnicity Maps English Italian German French Portuguese Spanish Greek Turkish Pakistani Indian Bangladeshi Sikh

17 Network Analysis

18 Network Analysis Development and application of spatio-temporal network analysis techniques to emerging social and geographic networks of individuals (and the devices used by them) Characterisation of the spatio-temporal profiles and activity patterns of individuals in physical and virtual spaces Study of the correlation of networks from physical and virtual interactions between individuals

19 Conclusions address categorization based on semantics of surnames has been developed. Statistical generative models are proposed to identify communities and correlation among communities in surname and forename. Forenames capture better communities for UK names data set and surnames capture better communities for Indian names data set using LDA model. Twitter data can be used to identify online communities and topics 20

20 Future work In the current model the number of clusters need to be specified in advance. Hence, more general approach needs to be developed that should not have any advanced constraints on the number of clusters Community modeling reveals community information of a person from his name, tweet or address. Hence, community information of a person needs to be protected Privacy preserving LDA model to be developed to preserve the privacy & identities of communities Currently working on the development of the Chinese Restaurant Process (Discrete time stochastic process) to eliminate the need for predefined number of clusters 21