Using mobile network big data for urban and transporta5on planning in Colombo

Transcription

1 Using mobile network big data for urban and transporta5on planning in Colombo Rohan Samarajiva & Danaja Maldeniya Sri Lanka Ins8tu8on of Engineers, 16 January 2015 This work was carried out with the aid of a grant from the Interna8onal Development Research Centre, Canada and the Department for Interna8onal Development UK..

2 Our mission Catalyzing policy change through research to improve people s lives in the emerging Asia Pacific by facilitating their use of hard and soft infrastructures through the use of knowledge, information and technology

3 Where we work Big data work in LK only in Hope to extend to BD and BT in

4 Smart ci8es The new buzz phrase IBM has been promo8ng smart ci8es and big data since 2000s But two visions New smart cities created on green fields, like S Korea s Songdo, OR Improving the functioning of existing cities 4

5 India appears to be going for green field option 5

6 Colombo We have to take the second option and improve the functioning of Greater Colombo We tried the green field option in the 1980s with the Eastward expansion Some possibilities exist with the 3Ks (Kadawata, Kaduwela & Kottawa) defined by the circular highway, but not easy given existing population densities 6

7 Why big data? Why now? Proximate causes Increased datafica8on : Very large sets of schema- less (unstructured, but processable) data now available Advances in memory technology: No longer is it necessary to archive most data and work with small subset Advances in so_ware: MapReduce, Hadoop 7

8 But more than that... James Beniger s Control revolu+on (1986) provides the theore8cal context Speed up of transporta8on caused crisis of control; remedied by early ICTs (telegraph, 8me zones) As mass produc8on brought down unit costs, another crisis of control emerged; this 8me remedied by mass media & adver8sing Now, a crisis of control over aeen8on; data analy8cs & micro targe8ng of audiences 8

9 Private & public purposes Data analy8cs in use within companies since 1990s American Express was using Cray Supercomputers in 1990s Big push by IBM, Cisco, etc. to use big data for smart ci8es in 2000s Rio as prototype in 2010; but used mostly video feeds and GPS These models rely on proprietary so_ware and installed- for- purpose sensors 9

10 Is there an alterna8ve approach? Can we have smart ci8es on the cheap? Transac8on- generated data ( data exhaust ) Ubiquitous mobiles make every ci8zen a mobile sensor: ci8zens are who we aim to serve If prepaid travel cards such as Octopus & Oyster in place and public transport is popular, they too can serve as sensors Open- source analy8cs; cheap hardware 10

11 But we need to take care Humans changing their behavior (Google flu trends) Problems of representa8veness 11

12 Bias in big data! why mobile network big data in developing countries Streetbump is a Boston crowdsourcing + big data applica8on that uses the natural movement of ci8zens to improve street maintenance Data generated from an app downloaded to a smartphone mounted in a car Can Streetbump be transplanted in Colombo at this 8me? Feature phones >> Smartphones Something beeer than nothing may not apply Bias toward roads traversed by smartphone owners è In condi8ons of limited resources, may skew resource alloca8on 12

13 Mobile network big data are more inclusive, especially in our ci8es Mobile SIMs/100 Internet users/100 Facebook users/100 Myanmar Bangladesh Pakistan India Sri Lanka Philippines Indonesia Thailand Source: ITU Measuring Information Society 2014; Facebook advtg portal 13

14 There is a role for other sources of big data But for smart ci8es, MNBD is the best Low cost, compared to fipng all vehicles with GPS or electronic toll cards/toll infrastructure But can/should be complemented with GPS and other sensor data Proposed two- year study to LK Ministry of Urban Development, based on first results with MNBD, a_er which appropriate sensors can be installed Global Pulse analysis of food- related Twieer content in Jakarta shows value in social media content, even if not as representa8ve Visitor Loca8on Register (VLR) data is best for physical mobility, but Call Detail Records (CDR) can serve as acceptable proxy Something we plan to explore in rela8on to infec8ous diseases in

15 Mobile network big data + other data è rich, 8mely insights Mobile network big data (CDRs, Internet access usage, air8me recharge records) Construct behavioral variables (i) Mobility variables (ii) Social variables (iii) Consump8on variables Other data sources (i) Census data (ii) HIES data (iii) Survey maps (iv) Transporta8on schedules (v) ++++ Analytics (i) (ii) Insights Urban & transporta8on planning Crisis management & DRR (iii) Health monitoring & planning (iv) Poverty mapping (v) Financial inclusion 15

16 Data used in the research Mul8ple mobile operators in Sri Lanka have provided four different types of meta- data Call Detail Records (CDRs) Records of calls SMS Internet access Air8me recharge records No Visitor Loca8on Register (VLR) data Data sets do not include any Personally Iden8fiable Informa8on All phone numbers are pseudonymized LIRNEasia does not maintain any mappings of iden8fiers to original phone numbers Cover 50-60% of users; very high coverage in Western (where Colombo the capital city in located) & Northern (most affected by civil conflict) Provinces, based on correla8on with census data 16

17 UNDERSTANDING CHANGES IN POPULATION DENSITY 17

18 Popula5on density changes in Colombo region: weekday/ weekend Pictures depict the change in popula5on density at a par5cular 5me rela5ve to midnight Sunday Weekday 18

19 Popula5on density changes in Jaffna & Kandy regions on a normal weekday Pictures depict the change in popula5on density at a par5cular 5me rela5ve to midnight Jaffna Kandy 19

20 Our findings closely match results from expensive & infrequent transporta8on surveys 20

21 UNDERSTANDING WHERE PEOPLE LIVE AND WORK 21

22 Methodology Based on extracted average diurnal mobility paeern for popula8on, choose 8me frames for home and work Home 8me: 2100 to 0500 Work 8me: 1000 to 1500 Calculate a home and work loca8on for each SIM: Match cell towers to Divisional Secretariat Division (DSD) Count each DSD at most once per day. Pick the DSD with the largest number of hits For work consider only weekdays that are not public holidays 22

23 46.9% of Colombo city s day8me popula8on comes from the surrounding regions Colombo city is made up of Colombo and Thimbirigasyaya DSDs Home DSD %age of Colombo s day5me popula5on Colombo city Maharagama Kolonnawa Kaduwela Sri Jayawardanapura Koee Dehiwala Kesbewa Waeala Kelaniya Ratmalana Moratuwa

24 40% of Jaffna s day8me popula8on comes from outside the city Home DSD %age of Jaffna s day5me popula5on %age of Home DSD popula5on that is in Jaffna during day5me Jaffna Nallur Valikamam South (Uduvil) Valikamam East (Kopay) Valikamam South - West (Sandilipay) Valikamam West (Chankanai) Thenmarachchy (Chavakachcheri) Island South (Velanai)

25 Implica8ons for public policy Urban Planning Current municipal boundaries are obsolete; those from outside city limits cause costs but do not contribute adequate revenues; our data indicate logical boundaries of metro regions Transporta5on Policy High volume transport corridors suitable for provision of mass transit Kaduwela DSD (now served by AB 10 & A0) (3) already iden8fied High Level Road (now served by A4 & rail line) to Maharagama DSD (1) 25

26 MOBILITY DURING AVURUDU 26

27 People travel greater distances during Avurudu Average: 11.6km A-4 A-3 A-2 A-1 A A+1 A+2 A+3 A+4 27

28 More people travel greater distances during Avurudu A -1 A A +1 28

29 How does the inflow in DSDs during the Avurudu weekend differ from other weekends? With the excep8on of Ampara, more Colombo city residents travelled to other DSDs during Avurudu, as compared to other weekends (some examples below): Nuwara Eliya: 315% Town & Gravets (Trincomalee): 100% Udunuwara: 93% Kalutara: 90% Jaffna: 80% Galle Four Gravets: 77% Kotmale: 233% Gangawata Korale: 71% Aeanagalla: 71% Mirigama: 66% 29

30 UNDERSTANDING LAND USE CHARACTERISTICS 30

31 Hourly loading of base sta8ons reveals dis8nct paeerns Type X:? Type Y:? We can use this insight to group base sta8ons into different groups, using unsupervised machine learning techniques 31

32 Understanding land use characteris8cs: methodology The 8me series of users connected at a base sta8on contains varia8ons, that can be grouped by similar characteris8cs A month of data is collapsed into an indica8ve week (Sunday to Saturday), with the 8me series normalized by the z- score Principal Component Analysis(PCA) is used to iden8fy the discriminant paeerns from noisy 8me series data Each base sta8on s paeern is filtered into 15 principal components (covering 95% of the data for that base sta8on) Using the 15 principal components, we cluster all the base sta8ons into 3 clusters in an unsupervised manner using k- means algorithm 32

33 Three spa8al clusters in Colombo District Cluster- 1 exhibits pazerns consistent with commercial area Cluster- 3 exhibits pazerns consistent with residen5al area Cluster- 2 exhibits pazerns more consistent with mixed- use 33

34 Our results show Central Business District (CBD) in Colombo city has expanded 34

35 Small area in NE corner of Colombo District classified as belonging to Cluster 1? Photo Senanayaka Bandara - Panoramio Seethawaka Export Processing Zone 35

36 Internal varia8ons in mixed use regions: More commercial or more residen8al? Blue dots: more residen8al than commercial Red dots: more commercial than residen8al 36

37 Plans & reality 1985 Plan 2020 UDA Plan 2013 reality 37

38 Implica8ons for urban policy Almost real- 8me monitoring of urban land use We are currently working on understanding temporal varia8ons in zone characteris8cs (especially the mixed- use areas) Can dispense with surveys & align master plan to reality LIRNEasia is working to unpack the iden8fied categories further, e.g., Entertainment zones that show evening ac8vity 38

39 UNDERSTANDING COMMUNITIES 39

40 Iden8fying communi8es: Methodology Social networks segregated so overlapping connec8ons between communi8es are minimized Strength of a community is determined by modularity Modularity Q = (edges inside the community) (expected number of edges inside the community) M. E. J.- Newman, Michele- Girvan, Finding and evalua8ng community structure in networks, Physical Review E, APS, Vol. 69, No. 2, p. 1-16,

41 Resultant communi8es The op8mal number of communi8es discovered by the algorithm was 11 41

42 How do these communi8es mesh with exis8ng administra8ve boundaries? Southern & Northern provinces mesh the best Surprisingly, also Uva, hitherto thought to have a district aligned with Central planta8ons & other with Southern Province Eastern Province most intriguing Trinco on its own Polonnaruwa in NCP (predominantly Sinhala) 8ed to Bapcaloa (Tamil/ Muslim) and Ampara (Muslim/ Sinhala) districts through rice economy Rest suggests administra8ve boundaries have been transcended 14

43 Work performed by collabora8ve inter- disciplinary teams LIRNEasia Sriganesh Lokanathan Kaushalya Madhawa Danaja Maldeniya Prof. Rohan Samarajiva Dedunu Dhananjaya (lost to industry in Nov) Nisansa de Silva (moved on to U of Oregon) LIRNEasia/ MIT Gabriel Kreindler (Economics) Yuhei Miyauchi (Economics) Technical partners: WSO2 (Dr. Srinath Perara) Auton Lab at Carnegie Mellon University University of Moratuwa Prof. Amal Kumarage (Transport & Logis8cs Management) Transport Dr. Amal Shehan Perera (Computer Science & Engineering) Data Mining Undergraduates working on projects Other US Universi8es Prof. Joshua Blumenstock (U Washington, School of Informa8on) Data Science Saad Gulzar (NYU Poli Sci) Poli8cal Science 43

44 Challenge Nego8a8ng access to data Addressing challenges Solu5on(s) Win- win; insights/ techniques for public policy outputs can be leveraged for operator s business interests Pro- ac8ve ac8on by operator(s) rather than reac8ve to growing government interest in using such data Minimizing harms from data sharing Development of self- regulatory guidelines for operators Skills Assemble interdisciplinary teams that are superior to what consultants can offer Research è policy Policy enlightenment as step 1 44

45 DRAFT GUIDELINES FOR THIRD- PARTY USE OF MOBILE NETWORK BIG DATA

46 Purpose Reduce transac8on costs of releasing mobile network big data (MNBD) to third par8es for public and commercial purposes First step in a process that will hopefully lead to the adop+on of a voluntary code of conduct by the region s mobile network operators (MNOs) that will be the most effec+ve in minimizing possible harms 46

47 Method Poten8al harms have been iden8fied through the literature (Annex 1) and engagement with ongoing analysis of MNBD at LIRNEasia Anchored on my work on u+lity transac+on- generated data since

48 Privacy and other harms, from the ground up Guidelines address harms that have emerged in society and recognized as worthy of remedy in the Common Law and not on abstract principles. Solove (2008: 174) argues that privacy as an abstract concept is difficult to pin down, since it involves a cluster of protec8ons against a group of different but related problems. He iden8fies 16 privacy problems, grouped into four general types: Informa8on collec8on; Informa8on processing; dissemina8on; and invasion 48

49 Considered harms Privacy (9 out 16 recognized in the Common Law in mul8ple countries) Surveillance Aggrega8on Iden8fica8on, individual and group Insecurity Secondary use Exclusion Breach of confiden8ality Disclosure Increased accessibility An8- compe88ve effects Marginaliza8on 49

50 Remedy Mobile Network Operators (MNOs) will not engage in ac8ve surveillance of their customers, except as required by applicable law. MNOs will desist from collec8ng more data than are needed for the efficient opera8on of the networks and the supply of good service to customers. To the extent feasible, data collec8on prac8ces will be transparent. Iden5fied poten5al harm Include in agreements transferring iden5fiable MNBD Ac#ve surveillance No. Applying only to MNOs, this need not be included in agreements. However, ac8ve surveillance is a root cause of problems that could be manifested in other forms at the subsequent informa8on processing and dissemina8on phases. Include in agreements transferring anonymized MNBD No. Applying only to MNOs, this need not be included in agreements. However, ac8ve surveillance is a root cause of problems that could be manifested in other forms at the subsequent informa8on processing and dissemina8on phases. 50

51 Remedy Best efforts will be made to prevent de- anonymiza8on. Working groups may be formed with data users to monitor the state of knowledge in techniques of anonymiza8on and de- anonymiza8on. Iden5fied poten5al harm Include in agreements transferring iden5fiable MNBD Include in agreements transferring anonymized MNBD De- anonymiza#on No Yes 51

52 Remedy Iden5fied poten5al harm Include in agreements transferring iden5fiable MNBD Include in agreements transferring anonymized MNBD Individually iden8fiable data will not be released to third par8es, unless the purposes are specified in the agreement and have been approved by an ethics review commieee, if one is available. If an ethics review commieee is not available, an equivalent third- party review should be sought. Individual iden#fica#on Yes No 52

53 Remedy Any agreement transferring iden8fiable data to a third party will also transfer responsibility to maintain safeguards to ensure security of individually iden8fiable data. Iden5fied poten5al harm Include in agreements transferring iden5fiable MNBD Include in agreements transferring anonymized MNBD Insecurity Yes No 53

54 Remedy Iden5fied poten5al harm Include in agreements transferring iden5fiable MNBD Include in agreements transferring anonymized MNBD The agreement governing the transfer will include provisions to minimize risks posed by increased accessibility when data are released to third par8es. Increased accessibility Yes Yes 54

55 Remedy The principle of non- discrimina8on shall govern the release of MNBD to third par8es who do not compete with the MNO. Those in the same class will be treated equally, subject to reasonable accommoda8on for resource constraints. Iden5fied poten5al harm An#- compe##ve effects Include in agreements transferring iden5fiable MNBD Yes Include in agreements transferring anonymized MNBD Yes 55