OPEN SOURCE INDICATORS (OSI) Intelligence ARPA Jason Matheny
Program Goal Develop and test methods for continuous, automated analysis of publicly available data in order to anticipate and/or detect significant societal events: Civil unrest, political elections, economic crises, disease outbreaks Beat the news by fusing early indicators of events from diverse data. 2
Approach Identify changes in population-level behavior that precede events of interest Analyze publicly available data indicative of those changes: Blogs, microblogs, news, internet traffic, web search queries, public webcams, financial markets, Wikipedia edits, online reservation systems, others With big data volume and variety, train models to detect patterns that have historically preceded societal events Evaluate teams on the accuracy and timeliness of forecasts they deliver about real-world events in Latin America. 3
01 Civil Unrest 011 Employment & Wages 012 Housing 013 Energy & Resources 014 Other Economic Policies 015 Other Government Policies 0111 Non-violent Civil Unrest 0112 Violent Civil Unrest 0121 Non-violent Civil Unrest 0122 Violent Civil Unrest 0131 Non-violent Civil Unrest 0132 Violent Civil Unrest 0141 Non-violent Civil Unrest 0142 Violent Civil Unrest 0151 Non-violent Civil Unrest 0152 Violent Civil Unrest Event Typology 016 Other 0161 Non-violent Civil Unrest 0162 Violent Civil Unrest 017 - Unspecified 0171 Non-violent Civil Unrest 0172 Violent Civil Unrest 0211 President/Prime Minister 02 Vote 021 Election 022 Referendum 0212 Governor 0213 Mayor 0221 Yes 0222 No 03 Infectious Human Illness 031 Rare Diseases 032 Pandemic 033 Influenza Like Illness (ILI) 0311 Bolivian Hemorrhagic Fever (Machupo) 0312 Cholera 0313 Hantavirus 0314 Yellow Fever 04 Economy 041 Stock Index 042 Currency Exchange 0411 Stock Index Increases 0412 Stock Index Decreases 0421 Currency Exchange Increases 0422 Currency Exchange Decreases 4
Audit Trail
Audit Trail 6
Audit Trail 7
Audit Trail 8
Flu incidence Results 14-day lead-time, 60% accuracy Rare diseases 6-day lead-time, 75% accuracy, 80% recall, 40% precision Civil unrest 7-day lead-time, 75% accuracy, 85% recall, 70% precision Geolocation ~90% of daily twitter volume to the city-level 9
Foresight and Understanding from Scientific Exposition (FUSE) 10
Goal: Validated, early detection of technical emergence Reduce technical surprise via reliable, early detection of emerging scientific and technical capabilities across disciplines and languages found within the full-text content of scientific, technical, and patent literature Special focus from the outset on multiple languages, Phase 2 focus on English and Chinese Novelty Usage Discover patterns of emergence and connections between technical concepts at a speed, scale, and comprehensiveness that exceeds human capacity Alert analyst of emerging technical areas with sufficient explanatory evidence to support further exploration 11
Scientific and Patent Literature Growth estimated at ~35k unique docs/month for FUSE; worldwide ~800k docs/month 12
FUSE: Example Nominations Three year forecast for most prominent filers in Chinese Patent Office Two year forecast for most prominent terms in Englishlanguage scientific papers (Web of Science) Three year forecast for most prominent patent term in Chinese Patent Office 13
Storyboard Example 14
FUSE Research Thrusts 3000 Total Company 2500 Academic/Govt/Non-Profit Theory & Hypothesis Development Supports indicator development and explanation; a robust theory is unlikely Document Features Patents, S&T Lit # Granted US Patents Individual Unclassified 2000 1500 1000 500 0 1981-1985 1986-1990 1991-1995 1996-2000 2001-2005 2006-2010 Indicator Development Leading indicators System Engineering Evidence Representation Nomination Quality Forecast formulation Prominence of surrogate entities of emergence 15
Forecasting Science & Technology (ForeST)
ForeST Program Goal Develop and test methods for generating accurate and timely forecasts for significant S&T milestones, by efficiently aggregating the judgments of many experts Technical approach: Generate S&T forecasting questions from indicators within the scientific and patent literatures Elicit and aggregate forecasts from ~10,000 scientists and engineers, globally, using a combinatorial prediction market (SciCast.org) The ForeST Program directly leverages the programmatic and technical achievements of IARPA s ACE and FUSE Programs 17
18
19
20
Evaluation Develop and test methods in parallel. As with all IARPA programs, the program is evaluated every six months. Run the world s largest S&T forecasting tournament. Evaluate using real events (1,000 per year). E.g., Will $100 whole human genome sequencing costs be achieved in 2015? Evaluate against best known approaches: econometric models, trend extrapolation, deliberating experts, flat prediction markets. 21