Establishing a reference framework for assessing the Socio-economic impact of Research Infrastructures Survey of RI Managers and External Stakeholders OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 1
2 phases: (I) RI Managers (II) External stakeholders: Policy Makers Funders Local Authorities Hosting organisations A diverse and representative sample of RIs (21) Geographical coverage 2 global 9 international (7 pan-european) 10 national : EU (2) + North America (2) + Australia (2) + Asia (3) + Africa (1) Lifecycle Operation (10) Operation + major upgrade (3) Operation + in development (6) Construction (2 ) S&T Fields and Themes Analytical facilities (3)+Materials (1) Physical Sciences (6 ) + Energy (2) Health and Food (3) + Environment (4) Social Sciences (2 ) RI types 11 single-sited 9 distributed (incl. 4 e-ris) 1 mobile OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 2
General information MANAGERS: RI description including main missions/objectives EXTERNAL STAKEHOLDERS: Type of interviewee s organisation General role and responsibility towards RIs Specific role and responsibility towards the RI referred to Objectives that are expected to be fulfilled by this RI Strategic aspects Tools used to evaluate the strategic positioning Required investment and annual running costs Structural effects expected and generated by the RI (visibility, networking, collaborations) OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 3
Impact indicators Scientific output and attractiveness Technological output and attractiveness Training and education Direct economic impact Indirect economic impact (medium to long-term effects) Social and societal impact OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 4
QUESTIONS TO RI MANAGERS Indicators collected/planned, how (current management, separate process), periodicity Use of indicators: by whom (RI managers, external stakeholders), periodicity, for what purpose QUESTIONS TO EXTERNAL STAKEHOLDERS Usefulness level of each indicator (same list as for RI Managers) For the various relevant lifecycle phases Indicators not used / not planned Other existing data Other data/indicators that would be useful Indicators considered to be core indicators for each dimension of impact (meaningfulness vis-à-vis objectives) OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 5
Coverage of the RI sample: 52 % (11 responses 21 RIs) External Stakeholders (*) 2 Gov. Policy Maker 4 Funder 4 Host 3 Local Authority S&T Fields and Themes 1 Analytical facility 3 Physical Sciences ( Astro ) 3 Health and Food / 2 Environment 2 Social Sciences Geographical coverage 1 global 2 international 5 pan-european 3 national (2 EU + 1 North America) RI types 3 Single-sited 7 Distributed 3 e-ris 1 Mobile Lifecycle phases (the responses do not allow us to draw significant conclusions. To be further discussed) Operation (3) Operation + major upgrade (2) Operation + in development (5) Construction (1 ) OECD (*) An GSF external Workshop stakeholder on SEIRI can Paris, combine 19-20 March responsibilities, 2018 Survey e.g. results Gov. & Funder, or Host & Funder Slide 6
Main strategic objectives RI Managers - Be a scientific leading RI + Be an enabling facility to support S&T advances - For some RIs: Industry- and innovation-oriented strategies, incl. regional aspects Support to public policies: Grand Societal and Environmental Challenges International S&T cooperation Education outreach & KT Policy Makers - Science + Specific technological objectives - Science policy: internationalisation capacity building - Support to public policies: e.g. environment Funders - Science + Innovation potential - Skills Training Science outreach - Maximising scientific and socioeconomic impact RI long-term sustainability Local Authorities - Science + Specific technological objectives - Skills Training Science outreach - Regional development Impact of made in Hosts (e-ris) - Data quality collection provision - Fostering open science & research - Training (data) - Support to public policies: e.g. social sciences health OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 7
Main tools used by RI Managers to evaluate strategic positioning - Periodic (scientific) user surveys on the reasons why to choose the RI - Periodic Strategic Exercises - Evaluation by Government bodies/agencies and by Peers Policy Makers - All, except evaluation by consultants Local Authorities - Evaluation by peers - Perspective meetings - Results of the RI s evaluation processes addressing the various aspects Funders - All Hosts (e-ris) - Health: All - Social sciences: positioning vis-à-vis public policies + positioning of the nodes in a DRI OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 8
Main structuring effects RI Managers - Scientific networking Structuring of communities Sharing of values - Show-case for the development of international S&T cooperation - Visibility to general public Visibility of local scientific communities Policy Makers - Scientific networking Structuring of communities - Visibility of local scient. communities - Opening to new countries - Technological networking Funders - Scientific networking Structuring of communities and networks - Development of procurement policy - Visibility of local scient. communities - Opening to new countries - Technological networking Local Authorities - Scientific networking Structuring of communities and networks - Visibility of local scient. communities - Sharing of values - Visibility to general public - Show-case for the development of international S&T cooperation Hosts (e-ris) - Scientific networking Structuring of communities OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 9
Operation phase - Annual total running cost (broken down by various funders) - Cumulated investments (broken down by various funders) - Potential gains from new investments (in the facility s performance, in scientific excellence, in increasing innovation and boosting industrial capacities) Maturity/upgrade phase - Potential gains from new investments (in the facility s performance, in scientific excellence, in increasing innovation and boosting industrial capacities) - Cumulated investments (broken down by various funders) - Annual total running cost (broken down by various funders) Preparation and construction phases - Annual total running cost (broken down by various funders) - Cumulated investments (broken down by various funders) Fo For distributed RIs: the total budget covers the Central Hub / HQ and the Nodes: The core contributions to the central budget are: membership, host premium (cash and in-kind); incomes from projects, service provision, partnerships, donations The exact full contributions/budgets/hr of the nodes (in situ and transferred to the central HQ) are difficult to identify. the in-kind contributions (material & human resources, delivery of OECD services) GSF of Workshop the nodes on should SEIRI Paris, be included 19-20 March in the 2018 total budget Survey results Slide 10
RI Managers Bibliometric data (specific publications pattern of each field) Nr of (new) users Collaboration with leading teams worldwide Available experimental/observational instruments, products & services Access/use (incl. user visits) of experimental/observational facilities Access/use of research data, digital products, services (incl. web statistics) Available digital resources (data, databases, collections, storage, etc.) % of international manpower Nr of Nobel Prizes & other important awards linked to work carried out at the RI External Stakeholders Number of users Collaboration with leading teams worldwide Bibliometric data (specific publications pattern of each field) Access/use of research data, digital products, services (incl. web statistics) Access/use (incl. user visits) of experimental/observational facilities Available digital resources (data, databases, collections, storage, etc.) Available experimental/observational instruments, products & services o Specific data on citations (citation lifetime, database citation, etc.) Specific profiles of publications (for targeted types of users) o Impact on researcher productivity User perception OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 11
RI Managers Patents/licenses/copyrights, background IP used, etc. Prototypes/innovations, TT, codevelopment with various partners Proprietary use of the facility by industry Involvement of industry in collaborations Availability/access to technology platforms and specific products Involvement of industry in the financing of academic research at the facility Involvement in standardisation bodies External Stakeholders Prototypes/innovations, TT, codevelopment with various partners Involvement of industry in collaborations Patents/licenses/copyrights, background IP used, etc. Proprietary use of the facility by industry Availability/access to technology platforms and specific products Involvement of industry in the financing of academic research at the facility o Enabling technologies developed o Income earned from patents OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 12
RI Managers Various categories of HR who are/have been working with(in) the RI Training programmes Nr of masters, PhDs or postdocs trained within the RI and employed by other RIs or by industry Nr of graduates employed by the RI through relevant sponsorships External Stakeholders Training programmes Various categories of HR who are/have been working with(in) the RI Nr of masters, PhDs or postdocs trained within the RI and employed by other RIs or by industry Nr of graduates employed by the RI through relevant sponsorships o Key skills developed by users and staff o Use of the RI / RI data for teaching OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 13
RI Managers Total budgets + Total number of FTE Public procurements and contracts Industrial suppliers and users: nr and revenues generated (incl. regional dimension) Collaborative projects with industry Dedicated economic impact studies Purchases and industrial/commercial contracts in partner countries R&D time spent using RI provided data External Stakeholders Total budgets + Total number of FTE Nr of firms using the RI (incl. regional dimension) Collaborative projects with industry o R&D time spent using RI provided data (e-ri) o Identified impact pathways (on industrial research productivity) OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 14
RI Managers Innovative projects based on RI outputs Nr of start-ups around the RI and spinoffs generated by the RI Medium-/ long-term collaborative contracts with industrial partners Local/regional impact on environment, energy infrastructure, etc. Stat. on IP created by past collaborators & students from experiment collaborations Impact on the local/regional innovation ecosystem Indirect/induced job creation in the economy Economic impact on tourism Nr of PhDs/engineers from the RI employed by industry External Stakeholders Medium-/ long-term collaborative contracts with industrial partners Innovative projects based on RI outputs Impact on the local/regional innovation ecosystem o Revenues generated by IP OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 15
RI Managers Public awareness (in press and other media) Consultation of web pages / Web statistics Visits/visitors (nr, types, interaction with local public & students, etc.) Nr of people engaged in RI for educational & outreach activities Production of experimental/ observational data in support of public policies Production of open databases/biobanks/expert advice in support of public policies Impact on legislation and regulation External Stakeholders Public awareness (in press and other media) Visits/visitors (nr, types, interaction with local public & students, etc.) Nr of people engaged in RI for educational & outreach activities Production of open databases/biobanks/expert advice in support of public policies Production of experimental/observational data in support of public policies Impact on legislation and regulation Consultation of web pages / Web statistics o General profile of support provided to R&D related to a specific public policy OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 16
Priority Which dimensions of the RI s impact should the KPIs address? Please rank them in order of importance (0=not relevant; 1=1st priority; 2=2nd priority; 3=3rd priority), for each relevant lifecycle phase Preparatory Construction Operation Maturity/ upgrade 1 [5] [5] [10] [6] S &T Economic Social / societal 2 1 LO 3 [1] [3] [1] [1] 0 [1] 1 [2] [1] [2] 2 [3] [2] [5] [3] 3 [2] [1] [2] [1] 0 [2] [2] [2] [1] 1 [1] [1] [1] [1] 2 [4] [5] [8] [4] 3 [1] [1] [1] [2] 0 OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 17
For which aspects of the management / policy decision-making processes should these KPIs be used (for each lifecycle phase)? Preparatory Construction Operation Maturity/ upgrade Financial management and planning Strategic positioning / strategic evaluation [4] [5] [8] [5] [6] [6] [10] [7] OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 18
Data are increasingly taken into account, not only in genuine data RIs but also for the e-infrastructure part of most of the RIs Economic impact of a Data RI The concept of Efficiency gains could be developed, i.e. cost savings of having integrated data freely available, with all licensing and standards in place Time saved in researchers being able to spend more time doing additional research rather than data gathering and cleaning but also data improved with people reporting back data on the effect of R&D and policy programmes Indirect economic impact: some indications are also given by the typology of users Social and societal impact: gathering narratives nice stories and use cases that are indicative of this broader impact could be useful OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 19
In addition to the impact indicators, these KPIs are related to: the capability of the RI to deliver the services and to generate impact; the way the RI s mission is fulfilled and the RI s objectives are reached They address: Governance and Management Human resources Funding Tools and technical equipments Quality, efficiency, performance The survey highlighted the following (non financial) KPIs: Technical operating parameters of the facility, e.g.: Reliability Running time and availability Physical safety Data management capabilities S&T performance, e.g.: Experimental and observation efficiency Quality, quantity and availability of data Services User satisfaction Business development Programme & staff development R&D planning RI development Standardised/harmonised internal procedures (especially for a distributed RI) Contribution to policy decision Implementation of international agreements and commitments (inter-institutional, inter-organisational) Use of specific KPIs during the construction phase: Business plan Schedule performance (physical progress) Completion of high-level milestones Safety Quality management Etc. OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 20
JEAN MOULIN jean.moulin.ripolicies@gmail.com BELSPO HONORARY GENERAL ADVISOR OECD CONSULTANT OECD GSF Workshop on SEIRI Paris, 19-20 March 2018 Survey results Slide 21