Assessing FP7-ICT research Performance indicators: patents and publications Carlos OLIVEIRA DG Information Society and Media Evaluation and Monitoring Unit (DG INFSO C3)
FP7-ICT assessment Overview 1) FP6-IST ex-post evaluation and FP7-ICT interim evaluation 2) Performance indicators whatare we «measuring» and why? 3) Outline of the study on "performance indicators" - methodology and findings 4) Plans for future activities - follow-up studies (data collection, analysis, etc.) 2
DG INFSO C3 Evaluation and Monitoring 3
FP7-ICT Interim Evaluation Legal base Financial Regulation (art 27 Sound and Financial Management, art. 60 Deelgation) FP7 legal base (art 7 of Decision 1982/2006 FP7 for research, tech. development and demonstration) Previous evaluations ( ex-post ) FP6-IST (Aho report) FP5-ICT (Gago report) Scope FP7-ICT ICT Theme of Cooperation art. 171 (JTIs) and 169 (AAL) Relationship FP7-ICT and einfrastructures (FP7 Capacities) FP7-ICT Overall budget: 9 billion EUR over the full duration of the FP7 (2007-2013) 4
FP7-ICT Interim Evaluation Evaluation questions Quality of research Are we doing the right kind of research? Are we mobilising the best people (Europe / worldwide)? Progress towards objectives ERA? Innovation? Knowledge triangle? EU policy goals? Quality of implementation; simplification WP formulation? Mix of instruments? Impact of Rules of Participation? Trust-based relationship? 5
Sources of evidence FP7-ICT information Work-programmes Implementation indicators (Success rate, TTG, TTP, etc.) FP7-ICT Self-assessment Challenges (technology, societal) Breakdown of funding / area Survey + interviews of participants Nature of the projects, Perceptions and expectations of participants Issues / barriers Specific studies Portfolio of participants (IPPA) Non-retained proposals Performance indicators 6
Study Performance Indicators What (objective): assess output performance of EU funded research in ICT using scientometric and bibliometric techniques Publications (scientific articles, conference papers) Patents Coverage 2005-2007 (FP5, FP6) Who: Universitá Luigi Bocconi Knowledge Internationalisation and Technology Studies Prof. Stefano Breschi How: data collection data cleaning and integration benchmarking methodology 7
Study Performance Indicators data collection: patents Survey of participants Excel files addressed to project coordinators 12
Study Performance Indicators data collection - patents 13
Study Performance Indicators data collection - patents 14
Study Performance Indicators data collection - patents 15
Study Performance Indicators patent analysis productivity Patent productivity by FP, funding instrument and strategic objective Descriptive and regression analysis (only FP6 projects) Analysis by country, region, and type of applicant Co-patenting analysis Identification of patenting-publishing researchers 16
Study Performance Indicators patent productivity regression analysis 1) Logit regression Dependent variable Binary outcome: 1 if a project produced a patent, 0 else 2) Negative binomial regression analysis Dependent variables Number of patents produced by a project Explanatory variables same as in the regression on scientific productivity plus number of articles (conference papers) 17
Study Performance Indicators patent analysis productivity IPs and NoEs significantly more likely to report a patent than STREPs No patent produced by CA, SSA and I3 Parallelism between patents and publications outputs at project level The level of regional concentration in patenting looks comparable to the one found for scientific publications activity 18
Study Performance Indicators patent analysis type of applicant FP-ICT projects are characterised by a significantly larger presence of patenting PROs and universities than in the benchmarking population They are also characterised by a larger fraction of jointly owned patents than in the benchmarking population Importance of patenting-publishing individuals (with a corporate affiliation) Around 28% of all EPO inventors are also authors of scientific articles and 15% of them have a corporate affiliation * analysis restricted to EPO patents 19
Study Performance Indicators issues Data collection tools (survey Excel files) is inefficient If survey is conducted annually, project co-ordinators have to fill same information several times It is time consuming to fill in the Excel templates provided, i.e. temptation to adopt shortcuts or to fill in information in a very incomplete way Moreover, much of the information requested is highly redundant Relevance and depth of analysis N of patents relatively modest => weakness of the programme? Patenting incentives depend on the area 20
Study Performance Indicators proposed solutions Collect data by setting up an online database Project co-ordinators and partners can enter data online through user id and password Less is more For articles, the only relevant information that project co-ordinators should enter is the DOI of the article (and the web link to the article) DOI = digital object identifier For patents the only relevant information is the application number of the patent (and the web link to the article in Espacenet) 21
FP-ICT output performance measurement Is the output of RTD limited to publications and patents? What are other elements of the research and innovation ecosystem that should be accounted for? Other outputs have been noted Grey literature Industry-wide specifications Software platforms / open source Human capital / research capacity building Leverage of public and private investment Mobilisation of stakeholders around major strategic initiatives 22
FP-ICT performance monitoring planned initiatives form and substance Form Improved data collection tools Systematic Rely on existing reporting mechanisms Substance Logic on intervention performance monitoring indicators Look for more sophisticated benchmarking mechanisms Causality relationship and patent relevance 23
Thank you! Questions? Contact: DG Information Society (and Media) INFSO-C3 Evaluation and Monitoring Unit Carlos Oliveira Carlos.Oliveira@ec.europa.eu 24
Europa web site 25