A Theory-Based Logic Model for Innovation Policy and Evaluation

Transcription

1 A Theory-Based Logic Model for Innovation Policy and Evaluation Presented at Canadian Evaluation Society Conference Victoria, British Columbia May 2010 Gretchen Jordan, Sandia National Laboratories Portions of the work presented here were completed for the U.S. DOE Offices of Science and Energy Efficiency and renewable Energy by Sandia National Laboratories, Albuquerque, New Mexico, USA under Contract DE-AC04-94AL8500. Sandia is a multi program laboratory operated by Sandia Corporation, a subsidiary of Lockheed Martin Corporation. Opinions expressed are solely those of the author. SAND Number: C

2 The Challenge US national interest Policy evaluation cycle Models of what is known about innovation R&D Diffusion Putting these together Outline Leverage points for innovation policy Three levels, multiple institutions A solar energy example Implications for evaluation 2

3 White House S&T Priorities stress evaluation and developing policy tools Agencies should describe in their budget submission how they are prioritizing activities toward four challenges and strengthening four cross-cutting areas (which include productivity of institutions) Expecting outcomes of in above areas, providing quantitative metrics where possible Building capacity to rigorously evaluate programs, and how assessments have been used to eliminate or reduce programs Operating in the open innovation model and supporting long term high-risk, high payoff Agencies will: Develop outcome oriented goals for S&T, target investment toward high performers, develop science of science policy tools that can improve management and assessment of impact -Peter Orszag, John Holdren, August 4, 2009 (for the FY 2011 Budget) 3

4 Evaluation in the policy cycle starts with the rationale for policy Foresight Technology Roadmapping Technology Assessment Wolfgang Polt G. Jordan AEA November

5 Currently parts are studied and understood better than the whole! Blind Men and the Elephant G. Jordan AEA November

6 Most simple view of innovation is linear: R&D followed by adoption of the new product Inputs, Context Planning & management, skilled workforce & RTD infrastructure R&D Investigation, new concepts, understanding & tools New technology, practice, policy developed New technology, practice, policy demonstrated Market Diffusion Infrastructure for diffusion of the newly developed technology, practice, or policy New technology, practice, policy deployed, adopted 6

7 Often-used Stage Gate model explains more about process up to commercial launch Business Case is discussed at each gate Source: R. Cooper/ Exxon s Stage Gate 7

8 The Idea Innovation Network makes role of manufacturing, quality and commercialization R&D explicit Connectivity and Throughput 7 Production, Refinement 8 6 Engineering & manufacturing R&D Marketing R&D, Quality R&D The idea innovation network: Hage and Hollingsworth (2000), modifying Kline and Rosenberg (1986 8

9 Logic Model 1: Non linear R&D -- the Idea Innovation Network theory Six R&D arenas RTD advance can occur in one or more arenas Ideas move between arenas Arenas are increasingly differentiated Inter-organizational networks transfer tacit knowledge Large companies Quality Manufacturing Commercialization.. INNOVATION... Development An example Universities Basic Applied Small Tech firms... sub networks Jordan, Hage & Mote,

10 The logic of diffusion in a market: a system with four domains 4 Domains DOE Impact Evaluation Framework, Reed and Jordan 2007

11 Tassey s model stresses technology infrastructure Targets for Science, Technology, Innovation and Diffusion (STID) Policy Joint Industry- Government Planning Strategic Planning Interface Standards Tax Incentives Intellectual Property Rights National Labs Direct Funding of Firms, Universities, Consortia Measurement Standards Production Proprietary Technologies Generic Technologies Market Planning Assistance Market Development Technology Transfer (Universities, MEP) Entrepreneurial Activity Science Base Value Added Risk Reduction National Labs (NIST) Universities G. Tassey, The Technology Imperative, Edward Elgar, 2007 Value Added Acceptance Test Standards; National Test Facilities

12 Logic Model 2: Diffusion of a technology or practice or policy A system of Four market domains End User Business Government Information Supporting technology infrastructure Product refinement to get characteristics needed for diffusion (E. Rogers: Relative advantage, Compatibility, Complexity, Trialability, Observability) G. Jordan,

13 Logic Model 3: The Innovation Eco-System: R&D and Market Diffusion Manufacturing Business Infrastructure Development Quality Product Refinement Government Infrastructure Interactions Applied Basic Commercialization/ Utilization Technology Infrastructure End User Attitudes, Action Information Infrastructure G. Jordan,

14 Important to consider multiple levels within the system, to focus evaluation on technology sector Sectors differ in Amount of investment by R&D arena Rates of technical change Policy impacts differ by sector Mission and policy decisions are often sector specific bottleneck? Meso/sector level connects macro with micro National rules S&T sector Team/lab G. Jordan AEA November

15 Important to consider the institutions and actors as well as interactions, infrastructure A National Innovation System Model Demand Consumers (final demand) Producers (intermediate demand) Framework conditions Financial environment; taxation and incentives; propensity to innovation and entrepreneurship; mobility Industrial system Education and system Political system Large companies Mature small/ medium enterprises ( SMEs) Intermediaries Research institutes Brokers Professional education and training Higher education and Government Governance New, technology - based firms The potential reach of public policies Public sector RTD Policies Infrastructure Banking, venture capital IPR and information Innovation and business support Standards and norms Source: Arnold and Kuhlman,

16 Logic Model 4. The innovation eco-system with leverage points Macro institutions, interactions, infrastructure, actors Manufacturing Business Infrastructure Development Quality Product Refinement Government Infrastructure Interactions Applied Basic Commercialization/ Utilization Technology Infrastructure Meso/Sector End User Attitudes, Action Information Infrastructure Micro institutions, interactions, infrastructure, actors G. Jordan,

17 Policy rationale: Examples from DOE s Solar Program 17

18 Conclusion and implications for evaluation Innovation occurs within a multi-level, complex, dynamic eco-system. Policy rationale, objectives, and evaluation use at least an implicit notion of how the innovation system works. Looking at only part of the elephant may give incorrect answers. Evaluation using an agreed upon model of the innovation system could Better test existing theories, and synthesize theories and build new understanding of the underlying program theory. G. Jordan AEA November