Early Stage Research and Technology at U.S. Federal Government Agencies Jonathan Behrens, Susannah Howieson, Vanessa Peña American Evaluation Association Evaluation 2017 Annual Meeting November 9, 2017 Acknowledgement to Bhavya Lal and Brian Zuckerman for their support in this study 1
Scope & Approach The National Aeronautics and Space Administration (NASA) asked the Science and Technology Policy Institute (STPI) to identify similarities and differences in how federal early stage research and technology development programs and portfolios are managed at other Federal departments and agencies Literature review, budget analysis, and interviews with program executives were conducted to develop case studies on 12 programs, agencies and topics 2
Programs, Offices and Topics Examined Department of Defense Air Force Office of Scientific Research (AFOSR) Office of Naval Research (ONR) Discovery and Innovation Portfolio Defense Advanced Research Projects Agency (DARPA) Defense Science Office (DSO) Multidisciplinary University Research Initiative (MURI) Army Research Office, Army Research Laboratory (ARO/ARL) Department of Energy Advanced Research Projects Agency-Energy (ARPA-E) Laboratory Directed Research and Development (LDRD) Intelligence Community Intelligence Advanced Research Projects Activity (IARPA) National Geospatial-Intelligence Agency (NGA) Research Directorate Science-Oriented Agencies National Science Foundation (NSF) Emerging Frontiers in Research and Innovation (EFRI) NSF Innovation Corps (I-Corps) National Institutes of Health (NIH) Common Fund Topics Examined Definition and Approach Budget Allocation of Funding Topic Selection Project Selection Funding Mechanisms Personnel Project Management Transition Evaluation of Success Metrics Targets 3
Early Stage Research Defined by Approach Technology and risk based approaches Basic/applied vs. technology development (e.g. DOD agency-level budget codes) Low risk vs. high risk (e.g. NSF/EFRI) Timeline oriented approach Far term vs. near term (e.g. DOD/ONR) Mission driven approach Opportunity vs. need (e.g. DOD/ARL/ARO) Source: Office of Naval Research (ONR). The Naval Science and Technology (S&T) Strategy. 4
OVERALL FINDINGS 5
Early Stage Research Program Budgets Rough Size of Program ($ Millions) Early Stage Program as a Percent of Department & Agency R&D DOD/AFOSR (6.1+) $540 AFOSR is 2.3% of AF RDT&E, 0.8% of DOD RDT&E DOD/ONR/D&I (6.1+) ~$1,000 DOD/DARPA/DSO (6.1) ~$400 DOD/ARL+ARO (6.1-6.2) $575 D&I is 45% of ONR 6% of Navy RDT&E 1.6% of DOD RDT&E DARPA 6.1 is 14% of DARPA 0.6% of DOD RDT&E ARL/ARO is 8.6% of Army RDT&E 0.9% of DOD RDT&E DOD/MURI (6.1) $160 MURI is 0.3% of DOD RDT&E DOE/ARPA-E (6.2-6.3) $280 ARPA-E is 2% of DOE R&D DOE/LDRD (6.1-6.2) $542 LDRD is 3.8% of Total Lab Budget (capped at 6% for each lab) NIH Common Fund (6.1-6.2) $200 CF is 0.7% of NIH Budget NSF EFRI (6.1-6.2) $31M EFRI is 3% of Engineering 0.4% of NSF Budget IARPA (6.1-6.2) $350-$700 (unclassified) Classified NGA Research (6.1-6.3) Classified Classified Takeaway: Early stage funding programs are a small fraction of overall RD&T budgets. 6
Allocation of Funding Topic Selection Project Selection Funding Mechanisms Practice (Example) External outreach (NSF/EFRI), site visits (DOE/ARPA-E), and international offices (DOD/AFOSR) Solicitations of white papers (NSF/EFRI), broad agency announcement (DOD/ONR) Tournament approach (IARPA) Mandate team composition (DOD/MURI) Employ internal and external reviewers (DOE/ARPA-E) Large award sizes (NSF/EFRI) Cooperative agreements (NGA) Use of prizes (multiple agencies) Intended Outcome Engage industrial/private, foreign, academic and governmental actors Identify priorities, emerging areas, and learn about user needs Identify multiple performers to solve the same challenge Encourage multi-disciplinary teams to foster new fields Identify high-risk, high-reward projects Solve large problems Enable strong government role Encourage unconventional participants and ideas Takeaway: Methods for allocating funding varies across and within agencies. 7
Personnel & Project Management Personnel models vary based on culture and context of work Permanent staff are used to retain institutional knowledge (e.g. DOD/ARO/ARL) or to support recruitment and retention (e.g. DOE/LDRD) Temporary staff of rotating PMs are used to gather an influx of new ideas (e.g. ARPAs), and can be hired through special authorities and at salaries above government scales (e.g. IARPA) Support staff hired to support program managers and officers (PMs) with contracting, evaluation and budgeting IARPA and ONR have approximately 3 support staff for every program manager Autonomy is given to PMs to make decisions DOD/AFOSR PMs are empowered and given autonomy to make funding and project management decisions Management of projects may be hands-on or minimal DOE/ARPA-E provides technical and administrative support; milestones are evaluated multiple times a year DOD/MURI provides minimal oversight after projects are funded Takeaway: A wide variety of personnel are employed to manage research portfolios. 8
Technology Transition Creation and evaluation of transition plans at project start (e.g. DOD/ONR) Engagement of transition partners early, to understand how to meet needs (e.g. DOD/MURI) Use of SBIR/STTR to leverage external organizations (e.g. DOD/AFOSR) Assignment of PMs to both early stage and later stage R&D programs to enable transition (e.g. DOD/ONR) Employment of dedicated transition managers to provide assistance for transition (e.g. DOE/ARPA-E) Measures of transition can be used as a program success metric (e.g. IARPA) Takeaway: Transition is an goal for many programs, but not always a metric for success. 9
Evaluation of Success Evaluative metrics employed by interviewed organizations include Qualitative reviews by external experts (e.g. DOD/AFOSR) Transition metrics such as number of MOUs signed with user organizations (e.g. IC/IARPA) Follow-on funding from private sector or government programs (e.g. DOE/ARPA-E) Use of bibliometrics, to identify outliers, but not as sole indicators of performance (e.g. DOD/AFOSR, DOD/MURI) Given the long timeline for return on investment, active resistance to measuring success at the project level was cited by multiple interviewees. The following are examples of alternative methods: Evaluation may be based on a portfolio rather than a single project (e.g. DOD/ONR) The rate of project success may be under-emphasized, providing PMs flexibility to manage a high-risk culture (e.g. IARPA) Takeaway: Specific target rates are uncommon, not all programs incorporate metrics. 10
QUESTIONS? Contact: Jonny Behrens, jbehrens@ida.org Publically Available Report: https://idalink.org/d-8481 11
BACKUP SLIDES 12
100% 90% Federal Basic and Applied Research as a Percent of R&D NSF DOA All Others 80% DOE 70% 60% 50% Total Federal 40% 30% NASA 20% 10% DOD 0% 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016** 2017** National Institutes of Health* National Science Foundation Department of Energy National Aeronautics and Space Administration Department of Defense Department of Agriculture All Other Total Federal Research 13 Source: AAS R&D analyses of OMB and agency R&D budget data. https://www.aaas.org/page/historical-trends-federal-rd#char
Less Common Practices Use of a tournament approach to encourage competition; routine performer attrition (IARPA) Seek/fund international input/researchers (AFOSR) Funding investigator-initiated research (NGA, DOE/LDRD) Mandating multidisciplinary teams (DOD/MURI) Extremely hands-on program management (ARPAs) Designating a transition role or responsibility (DOE/ARPA-E, NGA) Single manager moving research across TRLs (APRA-E) Focus on program manager quality rather than project-level performance indicators (DARPA, ONR) 14