The Three Methodologies & Three States of Knowledge. Underlying Technological Innovation

Similar documents
Three States of Knowledge in Technological Innovation

Managing & Communicating Knowledge in Three States

Technology Transfer Principles: Methods, Knowledge States and Value Systems Underlying Successful Technological Innovation

From Concept to Market: Linking Research, Development and Production Activities

Accomplishing Technological Innovation in AT: How the outputs from three methods can combine to generate beneficial socio-economic impacts.

Evidence-based Management of R&D Projects Intending Market Deployment

KT for TT Ensuring Technologybased R&D matters to Stakeholders. Center on Knowledge Translation for Technology Transfer University at Buffalo

Getting from Knowledge to Action: Effectively communicating Research & Development value to multiple Stakeholder Groups.

Tracking Evidence of Knowledge Use Through Knowledge Translation, Technology Transfer, and Commercial Transactions

RESEARCH, INNOVATION AND ENTREPRENEURSHIP: ESSENTIALS FOR SOCIO-ECONOMIC TRANSFORMATION

IP and Technology Management for Universities

ECU Research Commercialisation

Office of Science and Technology Policy th Street Washington, DC 20502

COMMERCIAL INDUSTRY RESEARCH AND DEVELOPMENT BEST PRACTICES Richard Van Atta

CRS Report for Congress

National Innovation System of Mongolia

Climate Change Innovation and Technology Framework 2017

Consultancy on Technological Foresight

M A R K E T L E D P R O P O SA LS

POLICY BRIEF. Defense innovation requires strong leadership coupled with a framework of

THE IMPLICATIONS OF THE KNOWLEDGE-BASED ECONOMY FOR FUTURE SCIENCE AND TECHNOLOGY POLICIES

The research commercialisation office of the University of Oxford, previously called Isis Innovation, has been renamed Oxford University Innovation

Arshad Mansoor, Sr. Vice President, Research & Development INNOVATION SCOUTS: EXPANDING EPRI S TECHNOLOGY INNOVATION NETWORK

Intellectual Property Initiatives

WPI Intellectual Property A day in the life of the tech transfer office. Todd Keiller Director, Intellectual Property and Innovation

Knowledge-Oriented Diversification Strategies: Policy Options for Transition Economies

Translating three states of knowledge discovery, invention, and innovation

Canada s Intellectual Property (IP) Strategy submission from Polytechnics Canada

Prof. Steven S. Saliterman. Department of Biomedical Engineering, University of Minnesota

Level Of Knowledge Use Survey (LOKUS) instrument: Documenting knowledge use by stakeholders

A POLICY in REGARDS to INTELLECTUAL PROPERTY. OCTOBER UNIVERSITY for MODERN SCIENCES and ARTS (MSA)

IUT QUIMPER UNIVERSITE DE BRETAGNE OCCIDENTALE. 3rd-year BUSINESS PROGRAMME IN ENGLISH PRESENTATION OF THE MODULES (1 st semester)

Brief to the. Senate Standing Committee on Social Affairs, Science and Technology. Dr. Eliot A. Phillipson President and CEO

Dynamic Cities and Creative Clusters

INTELLECTUAL PROPERTY POLICY

Intellectual Property Ownership and Disposition Policy

RECOMMENDATIONS. COMMISSION RECOMMENDATION (EU) 2018/790 of 25 April 2018 on access to and preservation of scientific information

Committee on Development and Intellectual Property (CDIP)

FACTORS INFLUENCING THE EFFECTIVENESS OF TECHNOLOGY TRANSFER

UW REGULATION Patents and Copyrights

University IP and Technology Management. University IP and Technology Management

CONSIDERATIONS REGARDING THE TENURE AND PROMOTION OF CLASSICAL ARCHAEOLOGISTS EMPLOYED IN COLLEGES AND UNIVERSITIES

The Intellectual Property, Knowledge Transfer: Perspectives

Gerald G. Boyd, Tom D. Anderson, David W. Geiser

EU Support for SME Innovation: The SME Instrument

Innovative Approaches in Collaborative Planning

Modern World History Grade 10 - Learner Objectives BOE approved

Creating Successful Public Private Partnerships Examining External Success Factors

COMPETITIVNESS, INNOVATION AND GROWTH: THE CASE OF MACEDONIA

executives are often viewed to better understand the merits of scientific over commercial solutions.

University of Dundee. Design in Action Knowledge Exchange Process Model Woods, Melanie; Marra, M.; Coulson, S. DOI: 10.

WORKSHOP ON BASIC RESEARCH: POLICY RELEVANT DEFINITIONS AND MEASUREMENT ISSUES PAPER. Holmenkollen Park Hotel, Oslo, Norway October 2001

Facilitating Technology Transfer and Management of IP Assets:

Building a Competitive Edge: Protecting Inventions by Patents and Utility Models

RFP No. 794/18/10/2017. Research Design and Implementation Requirements: Centres of Competence Research Project

Workshop on Enabling Technologies in CSF for EU Research and Innovation Funding

Technology Transfer & Inventing in Academia

TERMS OF REFERENCE FOR CONSULTANTS

WIPO REGIONAL SEMINAR ON SUPPORT SERVICES FOR INVENTORS, VALUATION AND COMMERCIALIZATION OF INVENTIONS AND RESEARCH RESULTS

Discovery: From Concept to the Patient - The Business of Medical Discovery. Todd Sherer, Ph.D.

THE AMERICAN INTELLECTUAL PROPERTY LAW ASSOCIATION RECOMMENDATIONS REGARDING QUALIFICATIONS FOR

The actors in the research system are led by the following principles:

Developing S&T Strategy. Lesson 1

Converting Research into Innovation & Growth: SBIR, the University, and the Park

Innovating together Collaborations between multi-national companies and academia in China

TOWARD THE NEXT EUROPEAN RESEARCH PROGRAMME

Innovation Management and Technology Adoption. Dr. Mircea Mihaescu, P.Eng. March 7, 2012

Marie Sklodowska Curie Actions. Business participation and entrepreneurship in Marie Skłodowska- Curie actions (FP7 and Horizon 2020)

Integrated Engineering Innovation and Entrepreneurship Environment

Getting Started. This Lecture

Working together to deliver on Europe 2020

President Barack Obama The White House Washington, DC June 19, Dear Mr. President,

Science and Innovation Policies at the Digital Age. Dominique Guellec Science and Technology Policy OECD

Sustainable Development Education, Research and Innovation

The Policy Content and Process in an SDG Context: Objectives, Instruments, Capabilities and Stages

Victor O. Matthews (Ph.D)

a) Core federal technology transfer principles and practices that should be protected, and those which should be adapted or changed;

Written response to the public consultation on the European Commission Green Paper: From

EL PASO COMMUNITY COLLEGE PROCEDURE

PROJECT FACT SHEET GREEK-GERMANY CO-FUNDED PROJECT. project proposal to the funding measure

Are large firms withdrawing from investing in science?

Innovation Management & Technology Transfer Innovation Management & Technology Transfer

Technology Transfer and the University: an orientation for new faculty at Johns Hopkins University

Lewis-Clark State College No Date 2/87 Rev. Policy and Procedures Manual Page 1 of 7

CAPACITY BUILDING INITIATIVE ON INCLUSIVE/COMMUNITY-BASED INNOVATION FOR AU MEMBER STATES

International comparison of education systems: a European model? Paris, November 2008

European Commission. 6 th Framework Programme Anticipating scientific and technological needs NEST. New and Emerging Science and Technology

TECHNOLOGY INNOVATION LEGISLATION HIGHLIGHTS

Innovative performance. Growth in useable knowledge. Innovative input. Market and firm characteristics. Growth measures. Productivitymeasures

Infrastructure for Systematic Innovation Enterprise

The Contribution of the Social Sciences to the Energy Challenge

Addressing the Innovation Imperative

Understanding DARPA - How to be Successful - Peter J. Delfyett CREOL, The College of Optics and Photonics

Science of Science & Innovation Policy (SciSIP) Julia Lane

Technology transfer industry shows gains

ROLE UNIVERSITIES PLAY IN INNOVATION AND ENTREPREURSHIP

Role of Knowledge Economics as a Driving Force in Global World

Policy 7.6 Intellectual Property Policy

Getting Value From Research:

Terms of Reference. Call for Experts in the field of Foresight and ICT

Transcription:

The Three Methodologies & Three States of Knowledge Underlying Technological Innovation Joseph P. Lane, Director Center on Knowledge Translation for Technology Transfer http://kt4tt.buffalo.edu University at Buffalo, SUNY, USA

3 Key Points - Technological Knowledge exists in Three States: Conceptual Discovery Prototype Invention Commercial Innovation Methodologies each generate Knowledge in One State: Scientific Research Engineering Development Industrial Production Opinion: Successful STI Policies require clarification between -- and parity among -- these basic concepts.

Why does this matter to you? Under SECIS, the CTI Renato Archer is tasked with creating CNRTA. This national network of cooperative Research, Development Innovation in AT, is supposed to have beneficial impacts! But... similar networks in North America and Europe are benefitting government and academia -- not industry or society! CNRTA can do better!

NIDRR/USDE 2013 Outputs

Seminar Organization 1. A Typology of Knowledge States in Ancient Greece. 2. The Evolution of Structured Methodologies to Generate Knowledge States. 3. The Gradual Co-Mingling of Sectors, Methods, Outputs and Terminology. 4. Resolving the Clash of Values through Enlightened Self- Interest. 5. The Muddled Mess of Technological Innovation in Theory and Practice. 6. Restoring Parity in Methods to Achieve Intended Results.

A Typology of Knowledge States in Ancient Greece

It is the mark of an educated mind to be able to entertain a thought without accepting it.

Aristotle s Knowledge Typology Five types virtues of thought (350 B.C.): Technê Epistêmê Phronêsis Sophia Nous

Three are the core elements of Technological Innovation Epistêmê represents the body of knowledge or a system of understanding that provides a basis for generating concepts and theories a fact. Technê represents the body of knowledge or system of understanding that provides the skill to do or create something tangible an artifact. Phronêsis - combines the facts of epistêmê and the skill of technê in a manner that improves quality of life a product and its experience.

The Evolution of Structured Methodologies to Generate Knowledge in Specific States.

Epistêmê Epistêmê Scientific Research Method. The Method s intended output is knowledge in the state of a novel conceptual discovery.

Scientific Research Methodology Designed to generates objectively observed outputs in the knowledge state of Conceptual Discoveries. The first new to the world articulation of a Conceptual Discovery a know what -- has primary value as novel intellectual property described in scholarly publications. Ownership and control of this IP is protected only by international copyright law.

Discovery State of Knowledge Purpose: Scientific Research methods create new to the world knowledge. Process: Empirical analysis reveals novel insights regarding key variables, precipitated by push of curiosity or pull of gap in field. Output: Conceptual Discovery expressed as manuscript or presentation the know what. Legal IP Status: Copyright protection only. Value: Novelty as first articulation of a new relationship/effect contributed to knowledge base.

Technê Technê Engineering Development Methodology. The Method s intended output is knowledge in the state of a tangible invention.

Engineering Development Methodology Designed to generate pre-determined functional outputs in the knowledge state of tangible Prototype Inventions. The primary value of a new to the world Prototype Invention is the demonstration of feasibility a know how. Ownership and control of IP is protected by international patent law.

Invention State of Knowledge Purpose: Engineering Development methods combine/apply knowledge as functional artifacts. Process: Trial and error experimentation/testing demonstrates proof-of-concept, initiated through opportunity supply or operational demand forces. Output: Prototype Invention claimed and embodied as functional prototype - the know how. Legal IP Status: Patent protection. Value: Feasibility of tangible invention as a demonstration of the Novelty of concept.

Phronêsis Phronêsis Industrial Production Methodology. The Method s intended output is knowledge in the state of a commercial product/service innovation.

Industrial Production Methodology Designed to create and deliver outputs in the knowledge state of Commercial Innovations. The primary value of a newly deployed Commercial Innovation is utility; monetary utility to the manufacturer and functional utility to the consumer a know why as motivation for the commercial transaction. Ownership and control of IP is protected by international trademark law.

Innovation State of Knowledge Purpose: Industrial Production methods codify knowledge in products/components positioned as new/improved products/services in the marketplace. Process: Systematic specification of components and attributes yields final form. Output: Market Innovation embodied as viable device/service in a defined context, initiated through a commercial market opportunity know why. Legal IP Status: Trademark protection. Value: Utility defined as revenue to company and function to customers + Novelty + Feasibility

Government (public) funding for Knowledge Creation followed Two Tracks Track 1: Grant-based Scientific Research Programs Exploration to discover new knowledge about physical world (Science/Medicine). Grant Scholarship Peer Review System Publish for Tenure. Track 2: Contract R&D for Production Programs Application of S&E to generate products with national value (Defense/Energy): Contract Production Performance Specifications Sell for Profit. - BOTH Tracks work very well - because their respective expectations, systems and incentives are closely and properly aligned.

Government Grants vs. Contracts Grants permit open exploration as deemed appropriate by investigator, while Contracts specific deliverables, milestones and requirements. Typically, Grants go to universities while Contracts go to corporations. Grants pay direct and overhead costs to conduct the work, while Contracts pay costs, a profit margin and a commitment to buy the output.

R-D-P Methods embody Values Each method has a unique purpose, underlying principles and assumptions, education and training requirements, certification of mastery and legal status. Each method s outputs are valued differently by sectors within society, as are society s expectations for their contributions. Aristotle s typology reflected pure states of knowledge, while in the modern world interest groups coalesced around and sought to influence each state of knowledge.

Marcos Pinotti Barbosa - Universidade Federal de Minas Gerais - UFMG

A Gradual Co-Mingling of Sectors, Methods, Outputs and Terminology.

1880 s 1930 s Inventor/Entrepreneurs established R&D laboratories to move beyond cut-and-try techniques: Eastern Dynamite Co. (AI DuPont, 1895); Menlo Park Lab (T. Edison, 1876). Scientists, engineers and managers moved between the industrial, academic and government sectors commensurate with the pace of technology-oriented discoveries and technology-based inventions. The private sector sponsored Managed R&D conducted by S&E personnel with practical and theoretical expertise.

AT&T s monopoly position allowed it to freely sponsor basic and applied science (Bell Labs) linked directly to New Product Innovations (Western Electric).

Pre-1940 strends in USA Leading experts readily moved between economic sectors by serving as New Product Development Managers, University Faculty, Deans & Presidents, or Government department heads, Cabinet officers & Presidential advisors. Many became independent inventors and some became serial entrepreneurs. Private sector activity drove most technological innovation as government was relatively small and universities were primarily privately funded. This revolving door of experts became more common as technology-oriented discoveries and technology-based inventions multiplied.

Trends in Europe European experienced the same co-mingling as individual nations sought to compete within the continent s markets and to compete globally. Germany s Physikalisch-Technische Reichsanstalt (1895) linked scientists (Albert Einstein & Max Planck) with major corporations (Siemens AG, Krupp & Zeiss). The United Kingdom created the Committee of the Privy Council for Scientific and Industrial Research (1915) to compete with S&E advances in Germany and in France.

WWII Period 1935-1945 Germany s rapid conquests demonstrated their superior military technologies (ballistics, aeronautics, communications, guidance, detection, telemetry etc.). Depend upon it, sir, when a man knows he is to be hanged in a fortnight, it concentrates his mind wonderfully. -- Samuel Johnson 1709-1784 The threat of global domination focused the minds of Allied leaders with little concern about the relative contributions of scientists, engineers and corporations.

U.S. National Defense Research Committee Seated, L-R: Brigadier General George Strong, James Conant, Vannevar Bush, Richard Tolman, Frank Jewett; standing: Karl Compton, Irvin Stewart, and Rear Admiral Harold Bowen. Missing is Conway Coe, the Commissioner of Patents. Success!! The mobilization of all sectors in the managed pursuit of clearly articulated goals.

Vision for Innovation Science: The Endless Frontier - Dr. Vannevar Bush (1945): Focus cross-sector resources on targeted socio-economic issues. What happened instead? Despite ancient and clear distinctions between Episteme & Techne, the two underlying Methodologies were merged as Research & Development (R&D) in the late 1940 s.

Special Interests Intervened The U.S. Congress set about establishing and expanding a plethora of agencies and programs lacking a central focus or guiding agenda. Pork Barrel politics grew. Each new Government agency and program obtained more funding and more staff. Government grew. The U.S. university system positioned itself through the Linear Model of Innovation to be the conduit for public revenues. Academia grew. President Eisenhower s caution about the Military-Industrial Complex went unheeded. Defense Industry grew.

Post WW II: 1945 1965 Economic boom in U.S. Reconstruction in Europe and Asia. Cold War between U.S. and Russia. Space Race. All accelerated pace of technological innovation for both national and consumer markets. Plenty of public and private money for expansion so no concern for distinctions between R and D.

Bureaucratic/Academic Complex Caption: Dr. Vannevar Bush (l), President Harry S. Truman (c ), James B. Conant, President, Harvard University, USA.

1960 s A Clash of Values Military & Industry - Contention over resource allocations caused the U.S. Defense Industry to challenge the Linear Model of Innovation: (DOD s Project Hindsight) Academia & Government The findings were successfully countered by a set of ad hoc studies disguised as scholarship: (NSF s TRACES)

Solution: Throw more money at problem! Government responds for calls to increase innovation by channeling even larger allocations to the academic sector -- doubling agency budgets over short timeframes. Established metrics for innovation were simply surrogate measures of other logic model elements: Government expenditures & University Awards = inputs. Level of sponsored research activity = process. Bibliometrics & Patents = Outputs Perhaps no one anticipated the long-term erosion to a nation s pace of technological innovation, especially because the effect was subtle and gradual.

The Muddled Mess of Technological Innovation in Theory and Practice.

Public Support for Knowledge Creation Grant-based Scientific Research Programs Exploration to discover new knowledge about physical world (science/medicine). Grant-based Scholarship Peer System Publish for Tenure. Contract R&D for Production Programs Application of S&E to deliver specified products with national value (defense/energy): Contract Production Performance Specs Sell for Profit. - BOTH of these programs work well - because their respective expectations, systems and incentives are closely and properly aligned. Sponsored R&D for S&T Innovation Generate S&E outputs for commercial exploitation to generate beneficial socio-economic impacts. Scholarly outputs for tenure Corporate requirements for profit - HYBRID programs have many problems because their expectations, systems and incentives are misaligned or even incongruent!

What are these Hybrid programs saying? That academia is better equipped than industry to deliver value for money? That tenured/career employees should dictate the rules of innovation for the private sector? That corporations are devoid of ideas for new products and services? That students and small businesses have the primary insight into societal needs? Yet these absurd premises remain unchallenged.

Hybrid Programs intending Impact United States All SBIR & STTR Programs; NSF Engineering Research Centers (ERC); Industry/University Cooperative Research Centers (I/U CRC); Innovation Corps (I-Corp); NIH Program on Public/Private Partnerships; NIST Technology Innovation Program (TIP); DoEd Rehabilitation Engineering Research Centers (RERC); Field Initiated Development (FID). Canada Natural Science and Engineering Research Council (NSERC); Canadian Institutes for Health Research (CIHR). European Union Research Framework Programme; Competiveness; Innovation Framework Programme. Latin America & Southeast Asia - CNRTA??

False Dichotomies/Erroneous Contractions Supply/Science/Technology Push vs. Demand/Market/Society Pull Research & Development (R&D) Science & Technology (S&T) Discovery/Insight/Invention/Innovation Scholarly vs. Societal: Outputs/Outcomes/Impacts Expenditures & Bibliometrics vs. New Net Wealth Counting what is countable vs. Counting what matters.

Silly Metrics based on Vague Models (R + D) / GDP = Innovation (95%R + 5%D) (5%R + 95%D) (X%R + Y%D) Products/Services Such measures co-mingle inputs, ignore key factors, and ignore causal links.

Newest government models lack utility (description, explanation, prediction, control). (http://www.ott.nih.gov/pdfs/nih-tt-plan-2013.pdf)

NIDRR/USDE 2013 Outputs

So why do they persist? Largesse of public funding since the 1940 s shifted power and influence over budgets from corporate to non-corporate sectors. Distortion of V. Bush s national R&D proposal by entrenched agency interests: Military/Industrial AND Academic/Bureaucratic Complex Perpetuation of false paradigms by beneficiaries in government and academia (Linear Model). Passive acceptance of aspirational language and unintended consequences by general public.

Restoring Parity in Methods to Achieve Intended Results.

Clarifying the Muddled Mess in STI Policy Establish Terms, Definitions & Proofs: These are essential yet currently absent from STI Policy. Acknowledge Knowledge States & Transitions: Methods of knowledge creation and output state attributes dictate opportunity and constraints for knowledge kernel. Apply proper transitions between Knowledge States: Ensure models, methods and metrics are congruent and designed to communicate information based on rigor and relevance, not on rhetoric. Apply the scholarly values of demonstration, replication, skepticism and peer review to all elements and actors.

Innovation & Impact Traditionally, each sector defined terms in own narrow context, unconcerned with downstream market activities or broader societal benefits, comfortable in status quo budgets and paradigms. But... U.S. National Science Board (2012) Innovation is defined as the introduction of new or significantly improved products (goods or services), processes organizational methods, and marketing methods, in internal business practices or in the open marketplace. (OECD/Eurostat, 2005).

Innovation Impact implies Utility Public support for investment in technology-based innovations grounded in 3 expectations: New/improved devices/services with economies of scale that contribute to societal quality of life. Sufficient return on investment through sales to sustain company, pay taxes and compete globally to generate new net wealth. Benefits realized in short-term (5 10 yrs). Innovation s context is Societal Impact via Commercial Marketplace.

Commercial Market is path to Utility Industry survives in competitive system by translating knowledge into market utility through Production methods (beyond R&D). Utility = Money to Seller / Function to Buyer. No $ale Research discoveries are freely published and globally disseminated, while Development prototypes lack commercial hardening or economies of scale. R and D outputs Market Innovation.

Importance of Untangling Innovation Terms Each Methodology has its own rigor and jargon. Actors are trained and operate in one Method and tend to over-value that one Method. Academic & Government sectors dominate STI Policy at the expense of Industry the only sector with time and money constraints... Methods are actually inter-dependent, while traditional dichotomies are all complementary factors supporting innovation outcomes/impacts.

Relational Attributes from Literature Episteme Techne Phronesis Know what Know how Know why Science Engineering Industry Research Development Production Intellectual Technological Commercial Long term Mid term Short term Concept Prototype Product Novelty Feasibility Utility Translation Transfer Transaction DISCOVERY INVENTION INNOVATION

The Way Forward: Integrate Conceptual but Differentiate Operational Consider three distinct states: Know role of Research, Development and Production methods in context of each project plan and budget accordingly. Engage Industry early: Government/Academic projects intended to benefit society fail to cross gaps (death valley vs. Darwinian sea) to business & open markets. Apply evidence-based framework: Link three methods; Communicate knowledge in three states; Integrate key stakeholder who will determine eventual success. Need to Knowledge (NtK) Model

Related Publications Lane,JP, Godin, B. (2013) Methodology Trumps Mythology, Bridges, The Translatlantic STI Policy Quarterly from the Office of Science & Technology, Embassy of Austria, Washington, DC, 36, December 2012/OpEds & Commentaries. Lane, JP, Godin, B, (2012) Is America s Science, Technology, and Innovation Policy Open for Business? Science Progress, June 12, 2012, http://scienceprogress.org/2012/06/is-america%e2%80%99s-science-technology-andinnovation-policy-open-for-business/ Flagg, J, Lane, J., & Lockett M. (2013) Need to Knowledge (NtK) Model: An Evidence-based Framework for Generating Technology-based Innovations. Implementation Science, 8, 21, http://www.implementationscience.com/content/8/1/21 Stone, V. & Lane J (2012). Modeling the Technology Innovation Process: How the implementation of science, engineering and industry methods combine to generate beneficial socio-economic impacts. Implementation Science, 7, 1, 44. http://www.implementationscience.com/content/7/1/44. Lane, J & Flagg, J. (2010). Translating 3 States of Knowledge: Discovery, Invention & Innovation. Implementation Science, 5, 1, 9. http://www.implementationscience.com/content/5/1/9.

Bibliography Aristotle (350 BC), Nicomachean Ethics, Book VI, Translated by W.D. Ross. http://classics.mit.edu/aristotle/nicomachaen.6.vi.html Bush, Vannevar (1945). Science, the Endless Frontier: a Report to the President. Washington, D. C.: U.S. Government Printing Office. Covey, Stephen R. (1998). The 7 Habits of Highly Effective People. Simon & Shuster. Gertner, Jon (2012) The Idea Factory: Bell Labs and the Great Age of American Innovation. The Penguin Press. Godin, B & J. Lane (2012). Research or Development? A Short History of Research and Development as Categories. Gegenworte. (See English translation: http://www.csiic.ca/pdf/gegenworte.pdf National Science Board (2012). Research, Development, Innovation and the Science and Engineering Workforce. National Science Foundation. http://www.nsf.gov/nsb/publications/2012/nsb1203.pdf OECD/EUROSTAT (2005). OSLO Manual: Guidelines for Collecting and Interpreting Innovation Data (3 rd Edition). http://epp.eurostat.ec.europa.eu/cache/ity_public/oslo/en/oslo-en.pdf

ACKNOWLEDGEMENT The contents of this presentation were developed under a grant from the U.S. Department of Education, NIDRR grant #H133A130014. However, the contents do not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal Government.