Cyber-physical Systems in Social Spaces: A Data Reliability Perspective

Transcription

1 Cyber-physical Computing Group Cyber-physical Systems in Social Spaces: A Data Reliability Perspective Tarek Abdelzaher University of Illinois at Urbana Champaign CPS Challenges? Embedded Computing Systems 1

2 CPS Challenges? Embedded Computing Systems CPS Challenges? Embedded Computing Systems 2

3 CPS Challenges? Embedded Computing Systems Data to Decision Building Reliable Systems on Unreliable Data Acknowledgements: Dong Wang, UIUC Tanvir Amin, UIUC Lance Kaplan, ARL Jemin George, ARL Hengchang Liu, HKUST Siyu Gu, UIUC Clare Voss, ARL Dan Roth, UIUC Jiawei Han, UIUC Raghu Ganti, IBM Research P. Mohapatra, UC Davis Bolek Szymanski, RPI 6 3

4 CPS in Social Spaces: Health and Fitness Data Disaster Recovery Energy Sustainability Transportation 7 CPS in Social Spaces: The Age of Crowd-sensing Disaster Recovery Health and Fitness Energy Sustainability Transportation 8 4

5 Social Sensing: A Confluence of Three Trends Mass Dissemination Media Sensors Cell-phones Glucose monitor Cars on Internet Connectivity Game Consoles on Internet Pulse oximeter Smart Meter GPS Sportsware The Research Crossroads Option 1: Ignore the potential of crowdsensing in CPS applications 10 5

6 The Research Crossroads Option 1: Ignore the potential of crowdsensing in CPS applications Option 2: Restrict use of crowd-sensing on simple applications where reliability is not an issue 11 The Research Crossroads Option 1: Ignore the potential of crowdsensing in CPS applications Option 2: Restrict use of crowd-sensing on simple applications where reliability is not an issue Option 3: Solve the data reliability problem deliver reliable systems on unreliable data 12 6

7 The Research Crossroads Option 1: Ignore the potential of crowdsensing in CPS applications Option 2: Restrict use of crowd-sensing on simple applications where reliability is not an issue Option 3: Solve the data reliability problem deliver reliable systems on unreliable data 13 A Wealth of Social Sensing Content Every Minute: More than 121,000 tweets made on Twitter Almost 700,000 status updates made on Facebook More than 3,500 images uploaded to Flickr More than 2000 check-ins on FourSquare More than 100 hours of video uploaded to YouTube

8 Significant Information on Ongoing Events Thousands of tweets per second during key events MTV Video Music Awards (Aug 28) Steve Jobs Resigns (Aug 25) End of FIFA Women's World Cup (Jul 17) BET Awards (Jun 27) NBA Finals (Jun 13) Raid on Osama bin Laden (May 2) UK Royal Wedding (Apr 29) Japan Earthquake and Tsunami (Mar 11) Superbowl (Feb 6) Average Tweets per Second Tweets per second, 2011 Japan s tsunami

9 Japan s tsunami Reconstructing Event Timelines A Twitter Example Do you know these individuals? Contact boston@ic.fbi.gov or CALL-FBI ( ), prompt #3 Unreal. Officer down in Cambridge. Scene centered around a MIT building. Does NOT appear related to #bostonmarathon. What happens when you give 4chan images of the Site of the bomb explosion on laurel st. bomb detectors are out #mitshooting #boston #mit Photos taken from State Police Air Wing on Watertown manhunt.media, please credit MSP for pics

10 Dow Jones Hickup Dow Jones lost 150 points on a rumor of two explosions in the White House on April 23 rd, 2013 Events Challenge: Reliable Information Distillation from Social Sensing Data Boston Bombing News and Public Sources Analytics Decision Support Hurricane Sandy People Data Sensors Egypt unrest 10

11 Reconstructing Event Timelines A Twitter Example Boston Bombing Clean Event Summary? Hurricane Sandy People Egypt unrest The Human Sensor Model Humans are better at binary observations. For measurements on a scale, use sensors Examples of actual Twitter feeds that can be thought of as binary observations : Crash blocking lanes on McBean Pkwy in Santa Clarita 105E past LakewoodB: traffic stopped to clear tire debris out of BREAKING NEWS: Shots fired in Watertown; source says Boston Marathon terror bomb suspect has been pinned down. The police chief of Afghanistan's southern Kandahar province has died in a suicide attack on his headquarters. Yonkers mayor has lifted his gas rationing order. Fill it up! #SandyABC7 11

12 Humans as Sensors The binary sensor property: Each person is a sensor that produces a stream of bits. Some are correct some are not. Humans as Sensors The uncertain provenance property: It is generally impossible to tell who originated a given bit of information. When a source issues a bit stream of binary measurements, even if the information is signed, there is no guarantee that the source observed all measurements first-hand, and did not copy some from another source. Implication correlated bit errors 12

13 Constructing Reliable State from Unreliable Observers Boston Bombing Hurricane Sandy Egypt unrest Sources Attribute: Reliability Claims Attribute: True/False Define a i as: P (source i makes an original observation it is true) Define b i as: P (source i makes an original observation it is false) What are the source reliability parameters that maximize the probability of received observations? A Maximum Likelihood Estimation Problem The expectation maximization algorithm computes: 13

14 Formulation: A Maximum Likelihood Estimation Problem Maximize log-likelihood by appropriate selection of truth values for claims: Log-likelihood Function of EM Scheme: M log (1 ) log(1 ) log N z j SiC j ai SiC j ai d i 1 lem ( x; ) M j 1 (1 z j ) SiC j log bi (1 SiC j ) log(1 bi ) log(1 d ) i 1 where z 1 when measured variable j is true and 0 otherwise j 27 Expectation Maximization Expectation Step (E-Step) Z t j f a b d j ( t) ( t) ( t) (, ) (,, ) Maximization Step (M-Step) Iterate 28 14

15 Derivation of Confidence Using the Cramer-Rao Bound Estimation and Statistic Background Fisher information is defined as T I( ) E X ( x; ) ( x; ) Score vector ( x; ) for a k 1 estimation vector [,,..., ] l( x; ) l( x; ) l( x; ) ( x; ) =,,..., 1 2 k Fisher Information Matrix can rewritten as (under regularity condition of EM) : 2 l( x; ) ( I( )) i, j EX i j Cramer-Rao Bound (CRB) is defined as the inverse of Fisher information CRB I 1 ( ) Cramer-Rao Bound Fisher Information T 1 2 k T 29 Evaluation Estimation Error More accurate than state of the art fact-finders 15

16 Evaluation Error Bound Empirical data suggests the confidence interval is accurate Evaluation Error Bound Accuracy of reliability estimation increases with scale Error in reliability estimation has good Gaussian properties 16

17 Expectation Maximization with Conflicting Claims Goal: Find Free Parking Lots on UIUC Campus Y: Free Parking lot - no charge after 5pm N: Not free parking lot Y N 33 Expectation Maximization with Conflicting Claims Goal: Find Free Parking Lots on UIUC Campus Results of Extended EM vs Baselines Experiment setup: 106 parking lots of interests, 46 indeed free 30 participants, 901 marks collected 34 17

18 Apollo Cleaning Noisy Speed Data Average Apollo Ground Truth Event Monitoring Boston Bombing Hurricane Sandy Egypt unrest Sources Attribute: Reliability Claims Attribute: True/False Define a i as: P (source i makes an original observation it is true) Define b i as: P (source i makes an original observation it is false) What are the source reliability parameters that maximize the probability of received observations? 18

19 Event Monitoring Boston Bombing Hurricane Sandy Egypt unrest Sources Attribute: Reliability Claims Attribute: True/False Define a i as: P (source i makes an original observation it is true) Define b i as: P (source i makes an original observation it is false) What are the source reliability parameters that maximize the probability of received observations? Reconstructing Event Timelines A Twitter Example Boston Bombing Social Network Hurricane Sandy Egypt unrest People Information (consistency) Network Clean Event Summary? 19

20 Accounting for Source Dependence in Estimation Accounting for observation forwarding Example: Distilling Information from Twitter Feeds 20

21 Use Cases Egypt Unrest 2011, 2013 Fukushima Nuclear Disaster, Japan Hurricanes (Irene, Sandy) Boston Marathon Bombing Kandahar, Afghanistan Syria Chemical Weapons Crisis Browsing the Data Stream 21

22 Hurricane Sandy (2M Tweets, Nov. 2012) Top correct tweets found by EM match well with Media Reports 43 Hurricane Sandy (2M Tweets, Nov. 2012) Media Report New Jersey and New York, struggling to recover from the wreckage of Sandy, were staggered today by a gas shortage Tweet found by EM Live gas chat: Hurricane Sandy causes massive lines at N.J. gas stations Top correct tweets found by EM match well with Media Reports 44 22

23 Hurricane Sandy (2M Tweets, Nov. 2012) Media Report New York Governor Andrew Cuomo says New York City subway, bus and train service will be suspended starting at 7pm ET ahead of Hurricane Sandy. Tweet found by EM Governor Cuomo Announces Public Transportation Shutdown At 7 PM Tonight Top correct tweets found by EM match well with Media Reports 45 Situation Awareness in the Aftermath of Hurricane Sandy Tweets found by Social EM (Apollo) 46 23

24 Situation Awareness in the Aftermath of Hurricane Sandy Finding gas in hurricane-stricken NYC % of unconfirmed tweets after filtering 47 Fact-finding Filtering fact from rumor Results from three datasets (Hurricane Sandy, Hurricane Irene, Egypt unrest) Filtered data contains fewer unverified claims 24

25 Conclusions The world s streaming content is growing Sensor devices are becoming increasingly available to the common user Real-time information services are needed to sift through the streaming human and sensor data and extract reliable state of the world models from unreliable inputs Analytic foundations are needed for data cleaning, disambiguation, privacy preservation, and search Further Info Dong Wang, Tarek Abdelzaher, Lance Kaplan, Raghu Ganti, "Exploitation of Physical Constraints for Reliable Social Sensing," in Proc. Real-Time Systems Symposium (RTSS), Vancouver, Canada, December Dong Wang, Lance Kaplan, Tarek Abdelzaher, "Maximum Likelihood Analysis of Conflicting Observations in Social Sensing," ACM Transactions on Sensor Networks, Accepted in Dong Wang, Lance Kaplan, Tarek Abdelzaher, and Charu Aggarwal, "On Credibility Estimation Tradeoffs in Assured Social Sensing," IEEE Journal On Selected Areas in Communication (JSAC), Vol. 31, Issue 6, pp , June Dong Wang, Tarek Abdelzaher, Lance Kaplan, and Charu Aggarwal, "Recursive Fact-finding: A Streaming Approach to Truth Estimation in Crowdsourcing Applications," In Proc. 33rd International Conference on Distributed Computing Systems (ICDCS) Philadelphia, PA, July Dong Wang, Lance Kaplan, Tarek Abdelzaher, Charu Aggarwal, "On Scalability and Robustness Limitations of Real and Asymptotic Confidence Bounds in Social Sensing," 9th Annual IEEE Communications Society Conference on Sensor, Mesh and Ad Hoc Communications and Networks (SECON), Seoul, Korea, June Dong Wang, Hieu Le, Lance Kaplan, Tarek Abdelzaher, "On Truth Discovery in Social Sensing: A Maximum Likelihood Estimation Approach," In Proc. 11th ACM/IEEE Conference on Information Processing in Sensor Networks (IPSN), April

26 Further Info Tomorrow afternoon! (Session 7) Dong Wang, Tarek Abdelzaher, Lance Kaplan, Raghu Ganti, "Exploitation of Physical Constraints for Reliable Social Sensing," in Proc. Real-Time Systems Symposium (RTSS), Vancouver, Canada, December Dong Wang, Lance Kaplan, Tarek Abdelzaher, "Maximum Likelihood Analysis of Conflicting Observations in Social Sensing," ACM Transactions on Sensor Networks, Accepted in Dong Wang, Lance Kaplan, Tarek Abdelzaher, and Charu Aggarwal, "On Credibility Estimation Tradeoffs in Assured Social Sensing," IEEE Journal On Selected Areas in Communication (JSAC), Vol. 31, Issue 6, pp , June Dong Wang, Tarek Abdelzaher, Lance Kaplan, and Charu Aggarwal, "Recursive Fact-finding: A Streaming Approach to Truth Estimation in Crowdsourcing Applications," In Proc. 33rd International Conference on Distributed Computing Systems (ICDCS) Philadelphia, PA, July Dong Wang, Lance Kaplan, Tarek Abdelzaher, Charu Aggarwal, "On Scalability and Robustness Limitations of Real and Asymptotic Confidence Bounds in Social Sensing," 9th Annual IEEE Communications Society Conference on Sensor, Mesh and Ad Hoc Communications and Networks (SECON), Seoul, Korea, June Dong Wang, Hieu Le, Lance Kaplan, Tarek Abdelzaher, "On Truth Discovery in Social Sensing: A Maximum Likelihood Estimation Approach," In Proc. 11th ACM/IEEE Conference on Information Processing in Sensor Networks (IPSN), April