Finding Patterns of Emergence in Science and Technology Evaluation Implications

Transcription

1 Understanding Federal R&D Impact Through Research Assessment and Program Evaluation Panel: Increasing Research Impact Through Effective Planning and Evaluation Finding Patterns of Emergence in Science and Technology Evaluation Implications Foresight and Understanding from Scientific Exposition (FUSE) Dewey Murdick, Program Manager Office of Incisive Analysis, IARPA 19 March

2 Invests in high-risk/high-payoff research programs that have the potential to provide our nation with an overwhelming intelligence advantage over our future adversaries 2

3 Goal: Validated, early detection of technical emergence Enable reliable, early detection of emerging scientific and technical capabilities across disciplines and languages found within the full-text content of scientific, technical, and patent literature Special focus from the outset on multiple languages Novelty Usage Discover patterns of emergence and connections between technical concepts at a speed, scale, and comprehensiveness that exceeds human capacity Alert analyst of emerging technical areas with sufficient explanatory evidence to support further exploration 3

4 Scientific Lit and Patent Sources (Selected) Growth estimated at ~35k unique docs/month for FUSE; worldwide ~800k docs/month 4

5 FUSE Approach Today, ad hoc technical horizon scanning already consumes substantial expert time, is narrowly focused on a small number of topics, and is subject to limited systematic validation. Analysts need to scan continually for signs of technical capability emergence. Today Manual Selected coverage Updated infrequently Months to produce (for one technical area) Ad hoc evaluation Complete, Continual, Unbiased FUSE Automatic Complete literature coverage Updated monthly 24hrs to produce (for all technical areas) Formal models of emergence 5

6 All teams are still pursuing all research areas FUSE Research Thrusts (Updated) 3000 Total Company 2500 Academic/Govt/Non-Profit Theory & Hypothesis Development Supports indicator development and explanation, unlikely to result full theory Document Features Patents, S&T Lit # Granted US Patents Individual Unclassified Indicator Development New focus of milestones System Engineering Evidence Representation Nomination Quality Reformulated as forecasts Prominence of surrogate entities of emergence 6

7 Text to Indicator Implementation Challenges New classes of FUSE-related features are being developed and validated Community response, role in community Topic stability Rhetorical stance for refs/citations Zoning the full-text content Methods moving from the topic to used to mentioned Quality and quantity of resources available to an investigator or inventor New terms introduction and adoption 7

8 What is technical emergence? Hypotheses from Phase 1 A concept has emerged if it has been accepted by others within and beyond one s community. ~Columbia A concept is emerging when its actant network is increasing in robustness. ~BAE A concept has emerged when evidence has appeared that the concept is new and unexpected, noticeable and growing. ~Raytheon BBN A concept is emerging when it is identifiable by its own practitioners, enables a capability that was not achievable previously, and persists. ~SRI Many ways to probe technical emergence Community of Practice Practical Application Debates Alternative Acceptance Interdisciplinarity Attention (Citation) Prediction Dominant sub-topic within set Commercial Application Infrastructure 8

9 Columbia Community of Practice Indicator Hypotheses (Ph 1) BAE BBN SRI Red edges connect data sources to data fields Blue edges connect BAE high-level indicators to BAE low-level indicators Line thickness between features and indicators, measures significance for the challenge 9

10 FUSE Validation / Metrics Evidence Quality Nomination Quality (Provided RDGs) Nomination Quality Multilingual Performance Chinese Phase 1: Base 60% PASS Precision: 55% Recall: 60% (10s of RDGs) N/A Concept (~10 RDGs, also German) Midterm Self-Score Participation Phase 2: Option Period 1 Phase 2: Final Option Period 2 80 % PASS 100% PASS N/A N/A N/A Functional Concept Self-Eval Precision: 70% Recall: 70% Functional Precision: 80% Recall: 80% English Parity Effective prioritization and nomination of forecasted surrogate entities of emergence as compared to real world prominence Nomination Quality Evidence provided in a clear and humanly usable form Evidence Quality System to perform at scale across multiple languages Computational Efficiency and Multilingual Performance 10

11 If you had a system that could (a) reliably identify what technical capabilities are emerging and (b) provide humanly understandable evidence explanation then how would you use this tool? 11

12 How might an IARPA PM use FUSE? Idea development for a relevant problem or problem space [IARPA uses the Heilmeier Questions, Hype versus reality New enabling component capabilities, signs of potential convergence Why is this innovative? Does it overlap with past and existing efforts? Does new capability or convergence of capabilities require further investment to motivate progress? Optimal emergence state for an R&D program investment Program impact assessment Observable impact within a research community, across research communities, beyond the research community? Are these focused programs yielding novel results or not? Is application development potential increasing as a result of this effort? Is there a gap between funding and output? Which areas are being driven by a focused research program? Which ones are not (but still have a vibrant community of practice)? Which ones are being driven by many research programs? 12

13 Would the components of FUSE enable a new capability or enhance an existing workflow? 13

14 Potential Component Services Data Transformation (e.g., PDF to XML) Forecast Nomination Engine XML Enrichment (e.g., new extractions, zoning) Evidence Explanation and Justification for Nomination Indicators (e.g., Confirming, Disconfirming) Indicator Fusion Interface 14

15 FUSE Status Update Four year, fundamental research program with four teams under contract since August 2011 Phases for exploring technical emergence Ph 1: Detect robust within a consistent theoretical construct / All teams met or exceeded Phase 1 metrics and targets Ph 2 Opt. Per. 1 (OP1) Forecasts (limited models) at scale in scientific and patent literatures in English and functional prototype in a second language Ph 2 Opt. Per. 2 (OP2) Forecasts (rich set of models) at scale in scientific and patent literatures in two languages 15

16 The FUSE Team 3 (+2) large businesses 3 (+3) small businesses 12 academic orgs 1 not-for-profit org Many data vendors Plus FFRDCs & gov orgs 16

17 Questions Dewey Murdick, Ph.D. Program Manager, IARPA 17