CS 111: Program Design I Lecture 22: CS: Network Analysis, Degree distribution, Profiling

Transcription

1 CS 111: Program Desig I Lecture 22: CS: Network Aalysis, Degree distributio, Profilig Robert H. Sloa & Richard Warer Uiversity of Illiois at Chicago November 15, 2018

2 NETWORK ANALYSIS

3 Which displays a graph i the sese of graph/etwork aalysis? A: Left, B. Right, C. Both

4 Graphs Graph (or etwork) is collectio of odes (aka vertices) ad liks (aka edges). For the formal mided, lik is pair of odes A CS 151 (Discrete Math) textbook might make a formal defiitio like "A graph is a fiite set of odes N together with a set L of pairs of odes."

5 Graphs Graph (or etwork) is collectio of odes (aka vertices) ad liks (aka edges). For the formal mided, lik is pair of odes A CS 151 (Discrete Math) might make a formal defiitio like "A graph is a fiite set of odes N together with a set L of pairs of odes." Default case: liks are bidirectioal, udirected, ad what we assume if we just see "graph" or "etwork" Ca also have directed liks (e.g., road etwork with oe-way streets): "directed graph"

6 Example Network with 6 odes ad 7 liks (Udirected) Liks are: (6, 4), (4, 5), (1, 5), (1,2), (2,3), (2, 5) (3, 4)

7 Examples Social etworks people as odes; fried = (udirected) lik Web pages (directed) page = ode; lik = lik Web crawler: crawls aroud this etwork Computer etworks odes = computers liks = 2 computers that ca commuicate directly Chicago El Nodes = stops; liks betwee adjacet stops Collectio of Phoe Calls Nodes = phoe umbers; liks = phoe calls

8 Networkx To work with graphs i Pytho, especially for etwork aalysis: import etworkx (as x) Lear more at: Spyder almost certaily has versio 2.1, almost idetical to versio 2.2, released i mid- September. (Some importat differeces from old versios 1.x) etworkx provides Graph as basic data type ad ways to add odes ad edges ad do all sorts of thigs, icludig visualize

9 Simple graph example import etworkx as x g = x.graph() #Create a empty graph object #Add several odes g.add_ode('alice') g.add_ode('bob') g.add_ode('charlie') g.umber_of_odes() à 3 g.umber_of_edges() à 0

10 Simple graph example cotiued # Add a sigle edge I [9]: g.add_edge('alice', 'Bob') I [10]: g.umber_of_edges() Out[10]: 1 I [11]: g.odes() Out[11]: ['Alice', 'Charlie', 'Bob'] I [12]: g.edges() Out[12]: EdgeView([('Alice', 'Bob')]) # udirected

11 Drawig etworkx ca do simple drawig (workig with matplotlib.pyplot uder hood): x.draw(g, with_labels='true')

12 Drawig without ode labels x.draw(g) (or, for cotrol freaks or the pedatic) x.draw(g, with_labels=false)

13 Addig a bit to the graph # Add some more edges ad odes g.add_ode("david") g.add_edges_from([("alice", "Charlie"), ("Alice", "David")]) add_edges_from is a method whose argumet should be list of pairs of ode ames, each pair i ()s.

14 etworkx.draw(g) (as updated)

15 Key graph statistic 1: Degree Degree of ode = umber of eighbors ode has Rage of differet degrees discovered i last years to vary with ature of graph. etworkx has graph method degree that gives us special data structure easily coverted to a dictioary with all the degree iformatio for the graph

16 degrees from etworkx I [5]: d = etworkx.degree(g) I [6]: d Out[6]: DegreeView({'Alice': 3, 'Bob': 1, 'Charlie': 1, 'David': 1}) I [7]: ddict = dict(d) I [8]: ddict Out[8]: {'Alice': 3, 'Bob': 1, 'Charlie': 1, 'David': 1} Will explai dictioaries ext class; for ow: padas kows them!

17 etworkx degree data à padas I [7]: degree_data = padas.series(dict(x.degree(g))) I [8]: degree_data Out[8]: Alice 3 Bob 1 Charlie 1 David 1 Ad we kow from earlier i semester how to plot graphs from padas series: padas series has method.plot()

18 Plottig degree data We wat a histogram: Bar plot where the thigs o the x-axis have a specific meaigful order (e.g., umbers), as opposed to beig categorical (e.g., ames of justices) degree_data.plot(kid='hist') (Uimportat side ote: There is a abbreviatio for that. Ca write.hist() istead of.plot(kid='hist') )

19 To make plot of series look ice padas put i stuff automatically for some of the plots we did before from dataframes, but it does't always. If eed be: plt.xlabel('strig I wat to see below x axis') plt.ylabel('similarly for y') plt.title('strig I wat up top i title positio')

20 Plot with some appropriate labels

21 Remember what our graph looks like Bob Charlie Alice David

22 Cetrality Alice is coected to everybody else; Bob, Charlie, ad David are coected oly to oe ode each (Alice) Alice is obviously the most cetral Various cetrality measures to tell which odes are most cetral (Prof. Philip Yu of UIC CS foud to be "most cetral" computer sciece author by oe such measure)

23 What is maximum degree of ode i graph with odes? A. B. 1 C. 2 D. ( 1) / 2 E. 42

24 Oe simple measure of cetrality of a ode degree of ode / maximum possible degree of ay ode i that ode's graph For -ode graph: degree of ode / ( - 1) etworkx will give us (a dictioary of) the cetrality of every ode i graph g: cet = x.degree_cetrality(g)

25 For our little graph I [11]: cet = x.degree_cetrality(g) I [12]: cet Out[12]: {'Alice': 1.0, 'Bob': , 'Charlie': , 'David': }

26 Path legths How may edges do we eed to walk over to get from oe ode to aother? 0 to get from ode to itself 1 to get to immediate eighbor > 1 to get to all other odes

27 Gettig all path legths eworkx has operator for this; gives somewhat complex data structure back padas to the rescue: It kows how to hadle that data structure ad tur it ito a dataframe, which we already kow about: padas.dataframe(x.all_pairs_shortest_path_legth(g))

28 All path legths i our graph p = padas.dataframe(x.all_pairs_shortest_path_legth(g)) >>> p Alice Bob Charlie David Alice Bob Charlie David

29 Aother stat: Average shortest path legth etworkx will calculate the average over all shortest path legths for you: # Get the average path legth prit(etworkx.average_shortest_path_legth(g)) 1.5

30 Our 4 ode graph is kida dull Poit is to apply these sorts of techiues to e.g., graphs of various types of social etworks with thousads to 1 billio+ odes Our example data (real data): odes = twitter users edge = follows relatioship (could be directed; could igore directio) ~40,000 pairs of follower, followee (This particular bit of twitter formed by techiue called sowball samplig startig at Computatioal Legal Aalytics)

31 Large etworks Stored as text files Oe lie for each lik with lie cotaiig ames (strig or umber) of odes Notice that if we kow all the liks the we kow what the odes are Both comma ad space are commo delimiters for betwee the two odes of a edge i large etwork work Both are, broadly speakig, CSV We'll use padas to read these i

32 Recall: Plottig degree data (i ext lab!) Plottig histogram: Bar plot where the thigs o the x-axis have a specific meaigful order (e.g., umbers), as opposed to beig categorical (e.g., ames of justices) degree_data.plot(kid='hist') or ca use Abbreviatio: Ca write.hist() istead of.plot(kid='hist') )

33 matplotlib.pyplot ad labelig your plot padas ca put up plots without importig matplotlib.pyplot But do eed to import matplotlib.plyplot if we wat to be able to alter plot's labels, etc. Gives us a "hadle" for the plot

34 To make plot of series look ice padas put i stuff automatically for some of the plots we did before from dataframes, but it does't always especially for series. If eed be: import matplotlib.pyplot as plt plt.xlabel('strig I wat to see below x axis') plt.ylabel('similarly for y') plt.title('strig I wat up top i title positio')

35 Plot with some appropriate labels

36 Large etworks Stored as text files Oe lie for each lik with lie cotaiig ames (strig or umber) of odes Notice: if we kow all liks the we kow the odes Both comma ad space are commo delimiters for betwee two odes of edge i large etwork work Both are, broadly speakig, CSV We'll use padas to read i such files

37 Readig graphs from files padas (The joys of workig with real data! J) Ca still have ecodig issues for, e.g., Chiese ode ame CSV files ad csv_read default: rows separated by ewlies ad items i rows separated by commas, but ca specify item separator either Comma: padas.read_csv(<fileame>) Space: padas.read_csv(<fileame>, sep = ' ') # Project!

38 DEGREE DISTRIBUTIONS

39 How may close (real world) eighbors Estimate umber of other people livig withi 100 feet of where you sleep at ight: A. 0 7 B C D E. 65+

40 How may close eighbors Estimate highest umber of other people livig withi 100 feet of bed of ayoe i Chicago area ot i dorm, priso, military, or hospital A B C D E. 250+

41 How may FB frieds? Estimate your umber of FB frieds (or followers if larger) A B C D E

42 Maximum umber FB followers? Estimate maximum umber of followers of most followed perso o FB? A B. 50,000 C. 500,000 D. 5,000,000 E. 50,000,000

43 Roaldo, Shakira Vi Diesel has about 100 millio followers Shakira has about millio followers Cristiao Roaldo has about 122 millio followers

44 Power law degree distributios # of followers does ot approximate classic bell curve distributio of Heights of Homo sapies Times of ruers Number of real-world eighbors Perhaps: Number of FB frieds of people i this class? Compare Roaldo's FB followers to huma height: No 50-foot tall (much less 50 mile tall) outliers!

45 PROFILING: ALIEN INTELLIGENCE

46 What Is Profilig? Aalyzig people s characteristics i order to Classify Predict Digital techologies allow the creatio of profiles that icorporate patters discovered i large uatities of data

47 We All Profile. So Why Object? New factors A cultural chage New techiues Both leadig to icreased power. The cultural chage has a log history.

48 Cultural Chage May people Place vast amouts of iformatio i the hads of third parties (e.g., use of gmail) Detail evets i their lives o social etworks Provide details about their frieds lives o those etworks, Create (o social etworks) etworks of frieds, acuaitaces ad orgaizatios.

49 More ad More Accessible Iformatio The US Supreme Court recogizes this fact: It is a totally differet thig to search a ma s pockets ad use agaist him what they cotai, from rasackig his house for everythig which may icrimiate him.... If his pockets cotai a cell phoe, however, that is o loger true. Ideed, a cell phoe search would typically expose to the govermet far more tha the most exhaustive search of a house: A phoe ot oly cotais i digital form may sesitive records previously foud i the home; it also cotais a broad array of private iformatio ever foud i a home i ay form uless the phoe is. US v. Riley.

50 I 1840 Commercial credit bureaus compiled iformatio about the persoal life, habits, property, ad fiacial reputatio of all America merchats ad etrepreeurs.... Durig the late 1850s, some of these firms bega to publish ratig books that displayed the creditworthiess of each idividual i cryptic alphaumeric codes. I essece, what early commercial reportig firms sought to do was to covert a idividual s local reputatio ito a easily readable, cetralized summary of creditworthiess for remote leders. I the process they did somethig more profoud: they created the moder cocept of fiacial idetity. Lauer, Creditworthy: A History of Cosumer Surveillace ad Fiacial Idetity i America

51 Iformatioal Privacy Iformatioal privacy is the ability to cotrol what others do with iformatio about you.

52 Privacy I Public The family holiday dier example. Your cotrol over the collectio ad use of iformatio cosists i others volutarily refraiig from collectig ad usig that iformatio. 52

53 Why Privacy i Public Matters Vast amouts of persoal iformatio is i the hads of others. Adeuate privacy reuires cotrol over those others. That is what privacy i public gives. Are ew techiues of data collectio ad aalysis cosistet with privacy i public?

54 New Techiues We look at two: Predictive aalytics (Supreme Court database) Network aalysis.

55 Cell Phoe Calls

56 Machie Learig/Artificial Itelligece The alies lad. Imagie beeficet alies who come i peace. Oe of their first acts is to provide us with a aalytics/artificial itelligece program that predicts future thought ad actio. Call the program AI, for alie itelligece. We have o explaatio or uderstadig of why AI predicts what it does.

57 Alie Itelligece Eve the best huma computer sciece experts fid large parts of the AI program completely uitelligible. It appears to ivolve programmig ad statistical techiues ukow to us. Its predictios are more accurate tha ours but, like ours, still have a fairly high error rate.

58 Should We Use AI? Humas busiesses, govermets, ad idividuals embrace the program, ad May (humas) propose usig AI systematically i the widest possible rage of cotexts as a basis for predictio ad actio. Would this be a good idea?

59 What AI Would Do AI creates wiers ad losers a very large umber of wiers ad losers sice AI govers the widest possible rage of cotexts. Losers will face great difficulty i escapig the categorizatios that codem them to that role. The high error rate meas that may of the categorizatios are wrog, ad The lack of feedback esures that AI will ot correct its errors.

60 Ujust ad Destabilizig AI s uitelligibility meas that there is o way to explai to losers why such treatmet is ot capricious ad arbitrary. Such a predictive system is both profoudly ujust ad a serious threat to social stability.