The European Research Council ERC Monitoring & Evaluation Strategy
Outline of the presentation European Research Council Strategy for Monitoring and Evaluation of ERC funding activities Implementing the strategy CSA projects 2
European Research Council Increasing pressure to account for public funds Accountability in Principal/Agent Model Principal : right to monitor (ensure that agent acts in the interest of Principal) Agent : obligation to account for resources Evolving accountability Core question : what have we done with the money Traditionally : Which projects do you fund? Currently : increased expectations which results have you achieved? what is their significance- for science, economy, society? Changing question not if R&D investments produce benefits but how are your investments produce benefits (or contribute to producing benefits) 3
European Research Council ERC Monitoring and Evaluation (M&E) Strategy ERC implements the "Ideas" Specific Programme of the EU's 7th Research Framework Programme. FP evaluated by EC. Roadmap of the Evaluation of the 7th FP by the EC Progress Report - 2009 Interim Evaluation - 2010 Ex-post Evaluation 2015 Coordination: EC - Unit A.5 Evaluation 4
Role of ERC M&E European Research Council Monitoring Inform decision making of the ERC council and ERCEA senior management Evaluation Provide evidence base for the ERC Assessment (mid-term review and the ex-post evaluation) Inform stakeholders, partners and the public on ERC activities, performance and impact (incl. reporting obligations of the ERC/ERCEA) 5
European Research Council ERC Approach I Dimensions Components Objectives Performance Accountability & Information Science Management Management Efficiency Stable, effective administration Direct impacts Advancing knowledge & dissemination Emerging research areas Training Reinforce excellence, dynamism and creativity Structural impacts Impact on researchers Impact on research organisations Impact on research policies and funding structures Attract best researchers and industrial research investment Derived impacts Economic benefits Societal benefits Better exploit research assets and foster innovation towards a dynamic knowledge based society 6
European Research Council ERC Approach - II It takes time for results to be produced and years for the impact to unfold. Only rigorous and carefully designed studies can help uncover the impact External Expertise Call for proposals Call for tenders Expert Groups Internal capacities Independent academics (to be explored) Good research information play a critical role 7
Implementation of ERC M&E Strategy Challenges European Research Council Managing the limited resources the ERC can devote to M&E activities Securing the high quality data required to deliver the strategy over-fishing and evaluation fatigue Data quality Adequate methodologies to measure impacts impact not easy to measure and/or to attribute impact of ERC funding will certainly need some time to emerge into recognizable effects 8
European Research Council Data sources Process-produced data (in-house data) External data: Collected by CSA funded projects Collected from specialized databases (e.g. bibliographic systems such as Web of Science, Scopus ) Data collected from other sources (Eurostats, Erawatch...) 9
European Research Council Data Analysis Tools AVEDAS ERC research information system: Integration of different (internal & external) data sources through user analytics module Enabling basic analysis in support of monitoring and evaluation of ERC operations Flexible and open for new data and analysis developments, for example integration of bibliometric tools developed by CSA projects 10
European Research Council Monitoring European Research Council s Implementation of Excellence [MERCI] Coordination: Professor Stefan Hornbostel (IFQ) Aim: to understand the impact of the grants in fours areas: (1) career development (2) institutional environments (e.g. on factors which favour or hinder the establishment of independent research groups); (3) structural effects in the European landscape (mobility) and (4) contribution to academic advancement (i.e. impact of publications of grant holders on the advancement of their fields). 11
Understanding and Assessing the impact and outcomes of the ERC funding schemes [EURECIA] European Research Council Coordination: Dr Maria Nedeva, Manchester Institute of Innovation Research (MIoIR), University of Manchester, UK Aim: develop and apply a methodology to measure and attribute impacts of the two ERC funding schemes in three areas: (1) on researchers (career path, skills, networking possibilities) (2) on research organisations and (3) on national as well as European research funding organisations. 12
Development and Verification of a Bibliometric model for the Identification of Frontier Research [DBF] European Research Council Coordination: Katy Whitelegg, Austrian Institute of Technology Aim: compare the ranking of applicants made by the panels (peer-review) and ranking based on the frontier research indicators measured through bibliometrics (publication records and citations from proposal submission) 13
Emerging Research Areas and their Coverage by the ERC-supported projects [ERACEP] European Research Council Coordination: Dr Thomas Reiss, Fraunhofer Institute Systems and Innovation Research, Karlsruhe, Germany Aim: to identify topically emerging researach areas and assess to which extend the ERC-supported projects cover and to contribute those emerging scientific areas * Identification of emerging areas on basis of bibliometric analysis (SCI, SSCI, A&HICI, years : 1998-2007) 14