D1.3 RESEARCH ASSESSMENT METHODOLOGY

Size: px
Start display at page:

Download "D1.3 RESEARCH ASSESSMENT METHODOLOGY"

Transcription

1 D1.3 RESEARCH ASSESSMENT METHODOLOGY Document Authors Document Contributors Jelmer Scholte, Joram Verstraeten, Alex Rutten (NLR) Mara Cole (Bauhaus), Simone Pozzi, Sara Silvagni (Deep Blue), Barry Kirwan, Dave Young (EUROCONTROL), Richard Shepherd (Rolls Royce). Abstract This OPTICS D1.2 document presents a methodology for evaluating the state-of-the-art of research and innovation addressing the ACARE safety related goals, and for identifying associated gaps and bottlenecks. Use of the methodology should enable providing strategic recommendations to the EC and/ or ACARE, including suggested corrective actions and priorities. The methodology is to be combined with OPTICS D1.4 s methodology for assessment of the socio-economic impact of aviation safety related R&I. The methodology is developed in sufficient detail to start the project assessment of OPTICS Work Package 2. During the OPTICS project, the methodology will be further improved and extended by developing templates for project assessment, and by using lessons learnt from the application of the methodology. This document is produced by the OPTICS Consortium. Framework Programme (FP7-AAT-2013_RTD-1) under Grant Agreement n ACS3-GA

2 Information Table Contract Number ACS3-GA Project Acronym OPTICS Project Coordinator EUROCONTROL Deliverable Number D1.3 Deliverable Title Research Assessment Methodology Version 1.00 Status Approved version Responsible Partner NLR Deliverable Type Report Contractual Date of Delivery 22 January 2014 Actual Date of Delivery 08 May 2014 Dissemination Level PU Document History Version Date Status Author Description /11/13 Draft /11/13 Draft /11/13 Draft /12/13 Draft /12/13 Draft /12/13 Draft /12/13 Draft /12/13 Draft Joram Verstraeten, Jelmer Scholte (NLR) Sara Silvagni, Simone Pozzi (Deep Blue) Jelmer Scholte (NLR), Simone Pozzi (Deep Blue) Simone Pozzi, Sara Silvagni (Deep Blue), Joram Verstraeten (NLR) Barry Kirwan, Dave Young (EUROCONTROL) Barry Kirwan, Dave Young (EUROCONTROL) Joram Verstraeten, Jelmer Scholte (NLR) Mara Cole (Bauhaus), Marcello Amato, Angela Vozella (CIRA), Joram Verstraeten, Jelmer Scholte (NLR) New document; Sections 1, 2 and 3. Added and Added 2.2; version for review of Sections 1 through 3. Expanded 2.2; processed review comments Added 4.3 Revised 4.3 Integrated and enhanced 4.2 and 4.3. Completed Sections 4.1, 4.3, 4.4, 4.5 and 4.7; introduced Appendix C. 2

3 /01/14 Draft /01/14 Draft /03/14 Draft Joram Verstraeten, Jelmer Scholte (NLR) Jelmer Scholte, Joram Verstraeten (NLR) Barry Kirwan, Dave Young (EUROCONTROL), Marcello Amato, Angela Vozella (CIRA), Jelmer Scholte (NLR) /02/14 Draft Jelmer Scholte (NLR) /05/14 Final Dave YOUNG (EUROCONTROL) Processed review comments MA and DY, and lessons from methodology test. Adapted in line with meeting results of 24/01/14. Major restructuring. Review and contributions CIRA & EUROCONTROL Processed reviews of Advisory Board, ACARE WG4, and Optics partners. Public Issue 3

4 Table of Contents 1. Introduction Background Objectives and scope Approach Structure of document Key elements of the methodology The challenging task of OPTICS The assessment approach The methodology steps The metrics for project assessment Selection of projects Assessment of projects Assessing the contribution to the SRIA Assessing the maturity Assessing the ease of adoption Synthesizing assessment results Synthesis onto capabilities Synthesis onto enablers Interpretation of synthesis results Synthesizing assessment results Use of assessment results Use of complementary workshop sessions Synthesizing assessment results Conclusions and recommendations References Appendix A Overview of the Safety Research and Innovation Agenda Appendix B Literature survey of previous projects B.1 AGAPE B.2 OPTI B.3 MEFISTO B.4 METRONOME B.5 EXCROSS Appendix C Literature survey: indicators for assessing projects C.1 Indicators from previous projects C.2 Indicators from other sources Appendix D Methodology requirements

5 Appendix E TRL levels NASA, AGAPE and OPTICS Appendix F Options for further development of methodology

6 List of Figures Figure 1: Simplified illustration of the structure of SRIA Volume Figure 2: Illustrational result of mapping projects to capabilities Figure 3: NASA Technology Readiness Levels (TRL) [30] Figure 4: The assessment cube for an individual capability Figure 5: The assessment cube for an individual capability Figure 6: Illustrational result of mapping projects to capabilities Figure 7: ACARE methodology Figure 8: Safety and security management lifecycle Figure 9: Quantitative goal completion assessment Figure 10: Qualitative goal completion assessment Figure 11: IRL metric Figure 12: OPTI assessment of completion of 7 ACARE goals Figure 13: Overview of Institutional Enablers IRL Figure 14: Sample questions from the questionnaire Figure 15: Technological roadmap of Framework Programme projects in environment Figure 16: Dissemination quality matrix Figure 17: Extent to which projects met the objectives (FP5 project only) Figure 18: Overview of the methodology Figure 19: An example of the infographic used by EXCROSS for the external dissemination Figure 20: Matrix to classify the impact depth and spread List of Tables Table 1: Flightpath 2050 goals for safety [17] Table 2: Overview of approach and inputs for assessment of the projects on the adopted dimensions Table 3: Example matrix for assessment of contribution to capabilities of project innovations23 Table 4: Scale for evaluation of maturity in in OPTICS Table 5: Classification scheme for assessment of ease of adoption Table 6: Overview of activities in the methodology Table 7: Overview of safety-related enablers in SRIA Table 8: Payback Framework categories and sample indicators Table 9: Australian Technology Network of Universities model and example indicators Table 10: Definitions of TRL levels by NASA, AGAPE and OPTICS Table 11: Example matrix presenting the indicators per enabler

7 Acronyms and Terminology Term ACARE AGAPE ANSP ATS CATER CAPPADOCIA CHASS CPP CSA EASA E-OCVM EUROCONTROL EXCROSS FP IndCo IRL MEFISTO METRONOME NASA OPTI OPTICS R&D R&I R&T SA SESAR SME SMS SRA Definition Advisory Council for Aeronautics Research in Europe ACARE GoAls Progress Evaluation Air Navigation Service Provider Air Traffic Services Coordinating Air transport Time-Efficiency Research Coordination Action Pro 'Production, Avionics, Design' On Costefficiency In Aeronautics Council for the Humanities, Arts and Social Sciences Community and Public Policies Coordination and Support Action European Aviation Safety Agency European Operational Concept Validation Methodology European organisation for the safety of air navigation EXploiting safety results across transportation modes Framework Programme Industrial Competitiveness Institutional Readiness Level MEthodology for Framework programmes' Impact assessment in Transport METhodology for evaluation of project impacts in the field of Transport National Aeronautics and Space Administration Observatory Platform Technological and Institutional Observation Platform for Technological and Institutional Consolidation of research in Safety Research & Development Research & Innovation Research & Technology Supporting Action Single European Sky ATM Research Small and Medium-sized Enterprises Safety Management System Strategic Research Agenda 7

8 SRIA SuD TCAS TRL WP Strategic Research and Innovation Agenda Sustainable Development Traffic Collision Avoidance System Technological Readiness Level Work Package 8

9 Executive Summary This OPTICS D1.2 document presents a methodology for evaluating the state-of-the-art of research and innovation addressing the ACARE safety related goals, and for identifying associated gaps and bottlenecks. Its use aims to provide oversight of progress in safety research and innovation in the context of the SRIA (Volumes 1 and 2). Use of the methodology will enable OPTICS to provide strategic recommendations to the EC and/or ACARE, including suggested corrective actions and priorities. The methodology is to be combined with a methodology for assessment of the socio-economic impact of aviation safety related R&I, which is currently being developed and which will be described in OPTICS D1.4. The methodology was developed in an iterative process building on the project objectives, lessons learnt from previous projects similar to OPTICS, and a survey of relevant indicators. Use was made of the specification of methodology requirements, feedback from other CSAs and internal and external reviews, and a test of the draft methodology. The key elements of the methodology are as follows: A bottom-up approach in comparing research and innovation to the SRIA is complemented with selected workshop sessions in which a top-down view is adopted. In the bottom-up approach, projects are individually assessed with respect to a number of metrics, and the assessment results are next aggregated at the level of capabilities and enablers. This way, the collected research and innovation projects are compared to SRIA Volume 2 s top-down structure of goals, constituting enablers, constituting capabilities, and estimated planned dates for achievement of the capabilities and the perceived needs in research and innovation. The complementary workshop sessions are used for gaining inputs from experts with an overview over the aviation safety R&I. The main steps of the methodology are as follows: 1. Selection of projects. Here, projects from the OPTICS D1.5 project repository are selected for assessment. This includes completed and ongoing projects, and FP7, Horizon 2020, national research, and international research projects. 2. Assessment of projects. Here projects are mapped onto the SRIA Volume 2, and assessed with respect to a number of project assessment metrics. 3. Synthesizing assessment results. Here, assessments are made of the status of the capabilities and enablers in the SRIA, using the results of the previous step. 4. Identification of issues and opportunities. Here opportunities and various types of issues are identified that can be used for strategic recommendations to the EC and/ or ACARE. These include gaps in performed research with respect to the SRIA, overlaps among research projects, potential supplements to the SRIA, bottlenecks to performing research, bottlenecks in the uptake and transformation of research results into products and services, and perceived delays in the realisation of the SRIA). These are identified based on the synthesized assessment results and the complementary workshops sessions. The project assessment uses metrics from three main dimensions. These are the contribution to the SRIA that a project delivers, the maturity of research in a simplified research and innovation lifecycle, and the ease of adoption, which represents the perceived complexity of what is still required to implement the innovation considered into the aviation system. Each R&I project is assessed on these dimensions using a three-point scale. To this end, information 9

10 is collected using questionnaires sent out to project coordinators, review of available project information, and (for specific topics) interviews with projects coordinators. Also, information will be collected regarding when the innovations targeted by the projects may be implemented (e.g., 2020, 3025, 2050). High importance is given to the documentation of the rationales for all qualifications. During the OPTICS project, the methodology will be further improved and extended. The first planned extension is the development of templates for questionnaires and interviews. These should support the retrieval of objective data, and minimise the use of subjective opinions of project coordinators. It is best coordinated with other Coordination and Support Actions how R&I projects are approached. Further options for extension and improvement of the methodology are considered when the deliverable of the first years has been completed. This document includes several ideas for further development of the methodology that should be considered. Also, the feedback and lessons learnt from application of the methodology during the first year should then be considered. One specifically important option to consider will be to compare the project assessment results not only with the SRIA capabilities and enablers, but also with the Flightpath 2050 goals. Furthermore, rationales of qualifications can be analysed to establish criteria to strengthen the consistency of project assessment results. 10

11 1. INTRODUCTION 1.1. Background In 2000 a Group of Personalities from key stakeholders was installed in Europe to agree on how aviation could better serve society s needs, and on how Europe can become a global leader in the field of aeronautics. The result was the European Aeronautics: A vision for 2020 report [12], which was published in January The Group of Personalities also agreed to establish a new Advisory Council for Aeronautics Research in Europe (ACARE) to develop and maintain a Strategic Research Agenda (SRA) [1] that would help achieve the goals of Vision ACARE has played a central role in providing support to the High Level Group on Aviation Research, convened by the European Commission that has now formulated a new vision beyond 2020: Flightpath 2050 [17], released in March In response to this new vision a new Strategic Research and Innovation Agenda (SRIA two volumes) [2] was elaborated by ACARE during The Coordination and Support Action (CSA) OPTICS aims to provide oversight of progress in Research and innovation (R&I) aiming to improve the safety of aviation in accordance with Flightpath 2050 challenges and goals, exploiting as far as possible the identified metrics, achievements, main topic areas and broad knowledge and expertise base established in the development of SRIA. OPTICS aims to implement sustainable processes supporting stakeholders with strategic recommendations and a comprehensive vision of the safety-oriented research landscape. The action will implement a reference base and methodologies to perform assessment of progress both from a technological perspective - are we doing the right research and the research right?, and from the societal and economic perspective - is it delivering societal and market benefit? The surveys are to be performed on an annual basis, in close collaboration with expertise drawn from the industry through a series of workshops, fully exploiting the network developed by ACARE. Assessments will be performed upon all on-going initiatives addressing safety research. The assessments will result in the provision of an annual report, identifying main performers, gaps and obstacles in the research landscape, formulating strategic recommendations, corrective actions and suggested priorities. The findings are to be presented and discussed with the aviation community at an annual safety conference, organised on the premises of EASA, EUROCONTROL, etc. The results of the annual state-of-the-art review, together with relevant basic data and project information will be made available on the OPTICS repository, accessible on a dedicated website. Finally, this action will ensure co-ordination and wherever possible, create synergies, with other actions supporting complementary challenges Objectives and scope This OPTICS deliverable D1.3 Research Assessment Methodology presents the research assessment methodology as developed in OPTICS Work Package 1.2. The methodology will serve to: evaluate the state-of-the-art of research and innovation addressing the ACARE safety related goals; identify issues in the realisation of the safety goals of Flightpath 2050 via the SRIA, such as gaps in research and bottlenecks to innovation. The methodology should allow keeping oversight of progress in safety research and innovation, exploiting as far as possible the SRIA (Volumes 1 and 2). Use of the methodology should enable 11

12 providing strategic recommendations to the EC and/ or ACARE, including suggested corrective actions and priorities. Taking actions based on these recommendations is not in scope of OPTICS. The safety research assessment will be combined with an assessment of the socio-economic impact of research and innovation, the methodology for which is documented in OPTICS deliverable D1.4. Further activities that are part of OPTICS WP1 Research Landscape and Methodology and that are explicitly out of scope of the current document are the consolidation of data regarding the SRIA (documented in D1.1), the development of a repository of relevant projects and other research and innovation activities (D1.2), and an annual release of such repository (D1.5). The research assessment methodology is developed in sufficient detail to start the project assessment of OPTICS Work Package 2. During the OPTICS project, the methodology will be further improved and extended by developing templates for project assessment, and by using lessons learnt from the application of the methodology Approach The research assessment methodology is developed as follows: A literature survey is conducted to identify lessons learnt from previous projects with comparable objectives, and to collect useful indicators for assessing research and innovation projects. Next, requirements for the methodology are formulated. These are based on the OPTICS project objectives [13], the literature survey, and discussions within OPTICS ([26] through [28]). Finally, the methodology is defined. Where possible use is made of existing material such as that identified via the literature survey. The methodology is defined in an iterative process using internal and external reviews, discussions with CSAs targeted at other performance areas than safety ([29],[33]), and results from a test of the draft methodology [25] Structure of document This document consists of nine sections, including this introduction, and six appendices. The core of the report documents the methodology, including: Key elements of the methodology (Section 2); Selection of projects (Section 3); Assessment of projects (Section 4); Synthesizing assessment results (Section 5); Identification of issues (Section 6); Overview of activities (Section 7); Conclusions and recommendations (Section 8); and References (Section 9). 12

13 The development of the methodology is documented in the appendices: Appendix A provides an overview of the SRIA. Appendix B documents the literature survey of previous projects with comparable objectives. Appendix C documents the literature survey of useful indicators for assessing projects and other research initiatives. Appendix D lists the methodology requirements. Appendix E provides an overview of scales for Technology Readiness Levels. Appendix F provides options for further development of the methodology. 13

14 2. KEY ELEMENTS OF THE METHODOLOGY This section summarizes the challenges of the task faced by OPTICS (Section 2.1), and discusses the key elements of the methodology: the assessment approach (Section 2.2), the main methodology steps (Section 2.3), and the metrics used for project assessment (Section 2.4) The challenging task of OPTICS Flightpath 2050 [17] provides Europe s vision for aviation, and defines a number of challenging goals for research and innovation. Table 1 lists the goals related to safety defined by Flightpath Table 1: Flightpath 2050 goals for safety [17] # Flightpath 2050 goal for safety and security Overall, the European air transport system has less than one accident per ten million commercial aircraft flights. For specific operations, such as search and rescue, the aim is to reduce the number of accidents by 80% compared to 2000 taking into account increasing traffic. Weather and other hazards from the environment are precisely evaluated and risks are properly mitigated. The European air transport system operates seamlessly through fully interoperable and networked systems allowing manned and unmanned air vehicles to safely operate in the same airspace. The SRIA [2] provides a strategic agenda for research and innovation in response to the challenges from Flightpath SRIA Volume 2 identifies and clusters meaningful enablers ( what is required to achieve the goals? ) to reach the goals, and decomposes these into main capabilities ( how can it be achieved? ) and concepts services & technologies ( with what can the goals be reached? ). Also, it provides a high level scheduling of which achievements should be realised by 2020, which by 2035, and which by Figure 1 illustrates the main elements of the SRIA structure, which are to be exploited in the methodology. The SRIA is realised via a multitude of research and innovation initiatives, programs and activities, for the main part realised through projects. These include: projects from FP7, Horizon 2020, national research, and international research; projects that have an explicit objective of improving safety and projects that implicitly have an effect on safety; projects funded by public bodies or by private bodies; and project which are completed or ongoing. 14

15 Goal Enablers Capabilities Achievement System-wide SMS SMS integrated in enterprise management Globally standardised regulations 2020: interfaces identified 2035: full integration Safety Goal etc. Safety radar Behaviour analysis Figure 1: Simplified illustration of the structure of SRIA Volume 2 OPTICS aims to evaluate the state-of-the-art of research and innovation addressing the safety related goals in Flightpath 2050, and to identify issues which impact the realisation of the safety goals of Flightpath 2050 through the implementation of the SRIA. The main challenges in conducting such assessment are: The research and innovation is being performed in a multitude of projects. To be able to provide strategic recommendations regarding the realisation of the safety goals of Flightpath 2050, it is required to consider how the contributions of all these projects combine and how they compare to the SRIA. There are various types of issues that OPTICS would have to identify in order to enable providing strategic recommendations. These include: a gap in the research being performed with respect to SRIA, an overlap among research projects, a gap in the SRIA with respect to real needs to achieve safety goals, a bottleneck to perform the needed research, a bottleneck in the uptake and transformation of research results into products and services, and a perceived delay in the realisation of the SRIA. Also, OPTICS may identify opportunities or positive developments that should be given further priority. There is no detailed roadmap that provides a schedule of expected outcomes in order to reach the implementations planned by the SRIA for 2020, 2035, and Obtaining useful information regarding the projects can be difficult; it includes dealing with completed projects and sensitive information. The methodology presented in this document aims to address these challenges The assessment approach The Flightpath 2050 goals with the SRIA (Volumes 1 and 2) provide a top-down structure of goals, constituting enablers, constituting capabilities, and estimated planned dates for achievement of the capabilities and the perceived needs in research and innovation. 15

16 The analysis of how actual research and innovation compares to these goals and agenda takes a bottom-up approach: Firstly, projects are selected and individually assessed with respect to a number of metrics. Next, the contribution of the project to the status of capabilities is evaluated. Then, the contribution of the assessed capabilities to the status of the enabler is evaluated. In a future version of this methodology, the assessment results may be finally considered at the level of the Flightpath 2050 safety-related goals. This bottom-up assessment of actual research and innovation versus the SRIA allows the identification of gaps, bottlenecks, and other issues. It is complemented with selected workshop sessions in which experts with an overview over the aviation safety R&I field take a top-down view in identifying issues and opportunities in aviation safety related research and innovation. The assessment process will inevitably involve a level of expert judgement. Therefore throughout the process attention will be paid to the documentation of rationales for all evaluations and results The methodology steps The main steps of the methodology are as follows: 1. Selection of projects (Section 3). Here, projects are selected for assessment based on a succinct review of their abstract and main objectives. It uses the project repository of OPTICS 1.5 as input. 2. Assessment of projects (Section 4). Here projects are mapped onto the SRIA Volume 2 and assessed with respect to a number of project assessment metrics, using also complementary inputs from these projects. 3. Synthesizing assessment results (Section 5). Here, assessments are made of the status of capabilities and enablers using the results of the previous step. 4. Identification of issues and opportunities (Section 6). Here opportunities and various types of gaps and bottlenecks are identified from the synthesized assessment results and complementary workshop sessions The metrics for project assessment As explained in Section 2.1, the application of the methodology necessitates the identification of key performance indicators regarding research and innovation in aviation safety. To this end, the methodology considers three main dimensions in the assessments: 1. Contribution. Assessment of how the scope of a project contributes to the implementation of the SRIA. More specifically, it is considered as to which capabilities each project contributes, and it is assessed which part of the scope of the capability is addressed. Once aggregated across all projects, this facilitates the assessment of SRIA coverage. 2. Maturity. Assessment of the level of maturity of the research results within a simplified Research and Innovation lifecycle, providing an indication of progress towards targeted delivery to the aviation system and what remains to be achieved. 16

17 3. Ease of adoption. This represents the perceived complexity of what is still required to implement the innovation considered into the aviation system. The complexity may be considered as e.g., magnitude of investments required, required time-to-market, changes required to infrastructure, policies, procedures, societal acceptance, etc. These three dimensions are defined as much as possible as independent from each other. When assembled they thus provide a good metric to monitor the progress towards completion with respect to the capabilities, the enablers, and the safety goals. They will provide insight into where R&I are likely to add safety value, where there may be gaps, and where priorities should be applied. They allow the identification of opportunities, gaps associated to the coverage of enablers and capabilities by projects, and bottlenecks which prevent or impede to deliver the expected outcome. The socioeconomic impact of R&I is considered using a separate methodology that is developed in Optics D1.4. The metrics are evaluated adopting a 3-point scale (low, medium and high). For easy management of data, numeric values are used in the project assessment files (i.e., low=1, medium=3, high=5). In order to evaluate the projects with respect to the identified metrics both questionnaires and interviews will be considered. Complementary to the three dimensions considered, also information will be collected regarding when the innovations targeted by the projects may be implemented (the SRIA distinguishes implementation by 2020, 2035, and 2050), and the perceived complexity to successfully achieve the results. For completed projects the research results are taken into account when evaluating projects; for ongoing projects the evaluation is based on project objectives and (if available) intermediate results. Care is taken in distinguishing plans and objectives from results already achieved. It is worth underlying that as the OPTICS process develops and rationales of scores accumulate, improvements to the methodology will be possible to increase robustness and reliability of results. 17

18 3. SELECTION OF PROJECTS This section describes how projects are to be selected for assessment throughout the four years of duration of the OPTICS project. The projects are selected from the repository developed in Optics WP1.3. The identification of projects and the collection of relevant project data are discussed in the associated deliverable OPTICS D1.2 OPTICS safety repository. In order to be able to select projects, a minimum of information needs to be collected for candidate projects. The information that should at least be obtained is: the acronym and the title of the project the Call identifier the abstract of the project Where available, additional information is considered, for example reports, the EC s website on R&D (CORDIS, [7]), the project website, etc. It is noted that it will require significant effort to collect the necessary information, potentially hampered by the reluctance of projects to share information and results. Therefore, direct contacts are to be established with the projects. Furthermore, direct interviews and/ or non-disclosure agreements may be used in case of information considered as sensitive. On the basis of the collected data a preliminary classification is performed allocating the projects either to the explicit safety research projects class or to the class of implicit safety research projects. Projects are classified as explicit safety research projects under the following conditions: Within the abstract it is declared that the project specifically addresses aviation safety; Within the abstract it is declared that the project addresses safety-related enablers or capabilities of the SRIA; or In additional available information (e.g. reports, website, etc.) the project addresses specific safety matters. Projects are classified as implicit safety research projects if the projects target other SRIA domains than safety, but do include safety related activities, e.g.: Projects involving design and development of concepts that can be safety critical (whose failure effects could threaten human life). These projects typically address safety assurance activities; or Projects involving design of new technology which cannot be easily certified using current standard rules and techniques. This includes projects which also develop innovative approaches and related methods for assessing safety of breakthrough concepts themselves, or study the adoption of consolidated technology normally used in different domains for aviation applications. It is to be considered whether and how projects that do not include safety related activities, but that do develop products that may affect safety are addressed. 18

19 Over the years, Optics will incrementally increase its analysis to more projects by applying the following division of projects: Year 1: FP7 projects with explicit safety part (and optionally FP6); Year 2: also include FP7 project with an implicit safety part and Horizon 2020 projects; Year 3: also include national research; and Year 4: also include international research. 19

20 4. ASSESSMENT OF PROJECTS This section describes how projects are assessed. Section 0 describes how the contribution of projects to the SRIA is assessed. Section 4.2 describes assessing the maturity of the research performed in projects. Section 4.3describes assessing the ease of adoption of the research results. The OPTICS WP2 team will perform the project assessment, using inputs such as generally available project information, questionnaires sent out to project coordinators, review of applicable project information, and interviews. The latter two means (review of applicable project information and interviews) are used for selected subjects where this is considered necessary for improving the project assessment. The rationales of all evaluations need to be recorded to assure transparency of results, and to allow future optimization of the methodology. The following table summarizes the approach and the inputs for the project assessment. Table 2: Overview of approach and inputs for assessment of the projects on the adopted dimensions. Dimension Approach Inputs Contribution a. Identifying project objectives/results b. Mapping these to applicable capabilities; c. Assessing the project contribution by qualifying the share of capability addressed General inputs retrieved about the projects, such as the project repository resulting from OPTICS WP1.3. Maturity Qualifying maturity of project results (1) answers to questionnaires sent out to project coordinators, Ease of adoption Qualifying ease of adoption of project results (2) review of available project information for selected projects (e.g., those with a high contribution and no questionnaire response), and (3) interviews with project coordinators for selected projects (e.g., those deemed highly interesting from a safety benefit point of view). Detailed questionnaires and templates will be developed to support these assessments. These questionnaires should be formulated in a way to get objective answers, and to minimise the use of subjective opinions of project coordinators. They should also include questions to seek information regarding when the innovations targeted by the project may be implemented (the SRIA distinguishes implementation by 2020, 2035, and 2050) and the perceived complexity to successfully achieve the results. In all assessments it is distinguished whether the evaluations are based on achieved results or on project objectives and plans. For projects that contribute to various capabilities, the evaluation of the project on the three dimensions may have to be done per capability. 20

21 4.1. Assessing the contribution to the SRIA The contribution of projects is assessed by first mapping the project onto the capabilities of the SRIA to which it contributes, and then qualifying for each relevant capability the magnitude of the contribution. Mapping projects to capabilities All selected research projects are allocated to the safety-related SRIA capabilities based on the overall project objectives. It is noted, that by this mapping to capabilities, the projects are automatically linked to the enablers via the structure of SRIA Volume 2. The mapping will be done by the OPTICS team, using the project repository of WP1.3 as a main source of information, which includes at least the project abstracts, but preferably also complementary information available in the public domain. To assure consistency in mapping of different projects by different partners of OPTICS the mapping results will be reviewed in workshops. To facilitate the mapping all SRIA information can be used; if a project can be mapped to an R&I need in the SRIA, it can also be mapped to the overarching capability. A mapping onto a capability is only made if there is evidence of a real link, and not simply a word scattered here or there in the description of the project which might potentially link it to the enabler. The aim in all cases is to identify the project s dominant capabilities, i.e. the ones it has the strongest link to, and so the ones it is most likely to deliver on. This is to avoid a dilution effect, whereby a project might seem to link to many capabilities, meaning it is unlikely to deliver strongly on any one of them. If no links can be established for a project with a clear safety contribution, then the project will not be mapped to the SRIA. This may be the case for national or international projects, but also for EC projects. This is an important result because it can be used to identify potential supplements to the SRIA. It does however not necessarily mean that a new SRIA enabler or capability is to be defined. The definition of new SRIA elements (or even new goals) can only be done after a thorough analysis of the projects without links to the SRIA, to identify if these projects represent commonalities and necessitate the establishment of a new enabler or capability. In such case OPTICS may provide recommendations regarding the definition of new SRIA elements; however the responsibility for adapting the SRIA accordingly is not with OPTICS. As an illustration, Figure 2 provides a possible result of mapping projects to capabilities. During the mapping at least the following is documented: the objectives of the project, the topic of the research programme addressed by the project, and the rationale of the mapping. The rationale can later be used to review the mapping and to establish criteria for mapping that can improve consistency between different experts. 21

22 Figure 2: Illustrational result of mapping projects to capabilities Qualifying the project contribution to the capability Next, the scope of the project is compared to the scope of the capabilities to which it has been mapped. For each capability, this provides an assessment of the share that the project may eventually deliver to the capability. This is however not to be mixed up by qualifying how mature this share of the capability will be made. The following scale is used: High: the project addresses the full scope of the capability; Medium: the project addresses a significant part of the scope of the capability; and Low: the project addresses a small part of the scope of the capability. Table 3 shows an example matrix for evaluating the projects against the capabilities based on the contribution of the innovations considered in the projects. The score in the cells indicate firstly that there is a link between the project and the capability identified, and secondly the weight of the contribution of the innovation considered to the capability. Two complementary approaches can be exploited: Per project, the contributions to all relevant capabilities are assessed at once. This way, the columns of the matrix are filled in. Per capability, the contributions of the innovations of the various projects mapped on it are considered. This works per row in the matrix. There are reliable methods (e.g. Paired Comparisons) to carry out the comparative evaluations using expert judgement. 22

23 Table 3: Example matrix for assessment of contribution to capabilities of project innovations Project A B C D E F G H n Capabilities 1 med high 2 low low low low low 3 low medium low 4 low low high 5 medium 6 low m low low The above approach facilitates the assessment of the contribution of individual projects towards individual capabilities. It should be acknowledged that the use of this scale provides a broad assessment of the weight of the contribution of a project to a capability. However, it does not identify whether the various contributions are complementary, overlap or duplicate other projects Assessing the maturity The maturity scale is a measure used to assess the maturity of the outcomes, or foreseen outcomes, of the research projects that are assessed. NASA s Technology Readiness Level (TRL) scale is a well-accepted industry-standard for measuring maturity of technological innovations (see Figure 3). A significant share of research and innovation does however not develop technology, but novel operations (including human roles and responsibilities), organisational and institutional approaches, novel safety methodologies in safety management systems, processes and procedures. 23

24 Figure 3: NASA Technology Readiness Levels (TRL) [30] Table 4 provides a classification scheme for maturity that includes both TRLs (adapted from NASA to exclude specialist terminology) and a generalisation that is also suitable for evaluating the maturity of non-technological innovations. The scale has been developed by OPTICS, aiming to provide a simple and generic scale that can be applied to a wide range of research projects. In the development, the NASA definitions and associated developments in AGAPE, OPTI, and E-OCVM [10] have been used as input (see also Appendix E). To facilitate scoring three bands are used covering two maturity levels each, and representing low, medium and high maturity. The classification scheme also includes indicators to aid in the assessment of maturity. The main rationale is that OPTICS should focus on determining maturity considering a number of objective indicators rather than on trying to use expert judgment to specify the level of maturity more precisely. The three bands are as follows: For low maturity research the impact is limited to specification of the problem and studying solutions in the academic and research world. For medium maturity research there is applied research, lab-testing and academic crossover. For high maturity research there are operational tests, and policy, industry uptake and enduser crossover impacts. 24

25 Table 4: Scale for evaluation of maturity in in OPTICS. Score Level Technological readiness General readiness Example indicators for maturity Low 1 2 Scientific research begins to be translated into applied research and development. The research is limited to paper studies of basic principles of the technology. Once basic principles are observed, practical applications are formulated. The concept of the application is defined and characteristics are described. There is no need for proof or detailed analysis to support the assumptions. Research is limited to analytic studies. A need is identified to solve a problem. Research is limited to paper studies of the problem, offering basic ideas for a solution. Once the basic idea is determined, practical applications are formulated. The concept of the application is defined and characteristics are described. Research is still limited to paper studies. Identification and specification of the problem. Academic impact: Papers in leading and respected journals Presentations academic audiences to Ongoing debate with other researchers Medium 3 Active Research and Development (R&D) is initiated with analytical and laboratory studies. The technical feasibility is demonstrated in proof of concept studies. The application for the technology results in the development of a system. Future research is likely to be impacted by this development. The concept is translated into product and functional descriptions e.g. new procedures, methodologies, management systems. The benefits and technical feasibility are assessed in paper studies. Applied research, lab-testing and academic crossover impact: Interdisciplinary contribution to academic research Generation of new research questions 4 Standalone components of the new system are validated in a laboratory environment. Experiments are carried out with integrated subsystems. Research has identified potential impediments for implementation, e.g. regulatory constraints. The new concept is tested in case studies. Research shows that there are no major problems for actual implementations, e.g. regulatory constraints. 25

26 Score Level Technological readiness General readiness Example indicators for maturity High 5 6 Thorough testing of prototyping (system, subsystems and components) in a representative environment. Basic technology elements are integrated with reasonably realistic supporting elements. Prototyping implementations conform to target environment and interfaces. Thorough testing of prototyping (system, subsystems) in a full-scale representative environment. Technology is partially integrated with existing systems. Engineering feasibility is fully demonstrated in actual system application. Concept is tested in representative environment, case study results are verified. Concept is picked-up by a wider range of disciplines, e.g. policy makers make draft regulations. The concept is deployed under testing condition at a selected number of pilot customers. Full feasibility is demonstrated. Operational tests, policy and end-user crossover impact: Constructive academicpolicy crossover Policy feedback into research Policy white papers and policy reports Constructive academicend-user crossover Feedback from end-users into research 7 Prototype is at or near scale of the operational system, with most functions available for demonstration and test in an operational environment. The concept is deployed in specific parts of the aviation system. Not applicable (out of scope of OPTICS) Out of scope of OPTICS 8 9 End of system development. System is fully integrated with operational hardware and software systems. Most user documentation, training documentation, and maintenance documentation completed. All functionality tested in simulated and operational scenarios. Actual application of the system in its final form and under mission conditions. Actual system has been thoroughly demonstrated and tested in its operational environment. All documentation completed. Successful operational experience. Sustaining engineering support in place. The concept is successfully deployed in multiple parts of the aviation system and does not need further development. The idea is embraced by rule-making bodies and the use is recommended. The concept is implemented throughout the total aviation system and, if applicable, mandated by rulemaking bodies. 26

27 4.3. Assessing the ease of adoption Ease of adoption is an indication of the effort required to introduce a new technology, concept or idea in the aviation system. It takes into account feasibility: concepts which are considered highly complex for their introduction into the aviation system will have a low ease of adoption value. Potentially, for some concepts and technologies the ease of adoption may improve over time. Others may be easily adoptable from the outset and accordingly have a high value. Others will retain a low level on the scale because introduction in the aviation system may remain highly complex. In project management five areas of feasibility are defined that serve as starting point for the development of the ease of adoption scale. The areas of feasibility are technical, economic, legal, organisational, and scheduling. Technical feasibility is implicitly evaluated under maturity (Section 4.2) while scheduling aspects are covered elsewhere in the project and capability assessments. Consequently, ease of adoption is assessed through the economic, legal/regulatory and organisational dimensions of complexity. Table 5 gives three bands of ease of adoption for each of these three areas of feasibility; it is worth noting that in Appendix F.3 an idea for further development of this scheme is presented. Table 5: Classification scheme for assessment of ease of adoption. Score Economic Legal Organizational Low The costs of adoption are very large, e.g. due to high technical complexity of the innovation. Medium The adoption will require investments that are not significantly high considering the magnitude of the change brought by the innovation. There are major legal constraints that need to be solved before adoption. Resolving these constraints requires a significant effort. There are minor legal constraints that must be solved before adoption. There are major organizational, institutional or political constraints for adoption by aviation stakeholders; in terms of e.g., operational, infrastructural or social limitations. Resolving these constraints would take a significant effort. There are organizational, institutional or political constraints for adoption by aviation stakeholders. Resolving these constraints will take time but is rather straightforward. High The costs involved in adopting the innovation are low compared to the magnitude of the change brought by the innovation. The innovation fits in the current legal framework. Adoption is therefore easy from a legal point of view. Aviation stakeholders are keen to adopt, because of the perceived benefits. Any constraints are minor only. Each project is assessed in all three areas. The results for the three areas are aggregated to result in one ease of adoption score per project. This is done by summing the scores on the three areas, assuming that a low score corresponds to 1, a medium score to 2, and a high score to 3. The overall ease of adoption score is low in case the total score is 3 or 4, medium for scores between 5, 6, and 7; and high for scores 8 and 9. 27

28 5. SYNTHESIZING ASSESSMENT RESULTS The project assessment of Section 4 provides detailed results for a multitude of individual projects. OPTICS however aims to compare the collected research and innovation to the level of the enablers identified in the SRIA and ultimately against the safety related goals of Flightpath 2050, to provide strategic recommendations. Therefore, this section explains how the assessment results are synthesized at a higher level allowing comparison with the SRIA capabilities and enablers. The synthesis is done bottom-up following the structure of the SRIA: First, the project results are aggregated at the level of capabilities (Section 5.1), and then the capabilities are aggregated at the level of the enablers (Section 5.2). Section 5.3 shortly discusses the interpretation of the synthesis results. Appendix F documents how this process may ultimately lead to an overall assessment of the progress of research in its contribution to delivering the ACARE goals Synthesis onto capabilities The assessment results for a specific safety-related capability level are synthesised by aggregating the assessment results for all projects contributing to the capability. Specifically, for each contributing project to the capability, the maturity of the research results, the ease of adoption of the research are considered, taking into consideration any complementary information obtained via questionnaires. The following steps are performed for each capability. The coverage of the capability is assessed. This is done by considering which share of the capability is addressed by the projects, applying the following criteria: o High: the projects address the full scope of the capability; o Medium: the projects address a significant part of the scope of the capability; and o Low: the projects address a small part of the scope of the capability. For some capabilities the coverage assessment may be based directly on the evaluation of the contribution of the projects that are assigned to it (cf. Section 0). For example, if only one project contributes to a capability, and the contribution of this project to the capability is assessed to be low, then the coverage of the capability is also low. For other capabilities it may be required to take a view at how the contributions of different projects combine in order to assess the coverage, potentially also considering the R&I needs that SRIA Volume 2 provides for the capabilities. The maturity of the capability is assessed. Here, only that part of a capability for which research or results exists are considered. Considering per project the share of the capability and the maturity, the average maturity of the capability may be evaluated as low, medium or high. The ease of adoption of the research for the capability is assessed. Similarly to the previous step the average ease of adoption may be provided. Finally, the status of the research results with respect to time progress for a capability is assessed. This is made considering the assessed maturity and ease of adoption with respect to scheduling in the SRIA, which distinguishes implementation by 2020, by 2035, and by

29 Answers from questionnaire results regarding scheduling are also exploited in order to perform this assessment. The following definitions are adopted: o Green: capability appears to be on schedule. o Yellow: some problems identified with respect to scheduling of this capability. o Red: major problems identified with respect to scheduling of this capability. These qualifications are provided for each capability for each planning deadline (2020, 2035, 2050). For one or more capabilities the assessment may be represented in a cube (e.g. capability assessment cube ) providing a good representation of the assessment. In such cube (see Figure 4) the assessment results of the individual projects mapped onto a capability can be plotted, considering the evaluation of the contribution to the capability, the maturity of research, and the ease of adoption of the results. H,H,H Contributions to capability Y 0,0,0 Maturity Ease of adoption Figure 4: The assessment cube for an individual capability A simple means of representing the results of the assessments of all capabilities is a table that provides for each capability the assessed coverage, the average maturity, the average ease of adoption, and the evaluation of progress with respect to time schedule in SRIA (e.g. 2020, 2035, and 2050). Alternative means of representing assessment results may be explored in case these presentation forms provides unsatisfactory results (e.g., [36]) Synthesis onto enablers The synthesis for an individual enabler is performed adopting an approach similar to the synthesis of the capabilities: 29

30 The coverage of the enabler is assessed by estimating the share of the constituting capabilities that is covered by research projects. This may be expressed in terms of the addressed percentage of capabilities. The maturity of each enabler is assessed. Considering the maturity of the covered relevant capabilities the average maturity of the enabler can be evaluated al low, medium or high. The ease of adoption of each enabler is assessed. Similarly to the previous step the average ease of adoption may be evaluated. Finally, the status of the relevant capabilities with respect to time progress is considered; this may provide an evaluation of the scheduling of the enabler (using green, yellow, and red) for each of the 2020, 2035, and 2050 timeframes. The status of each enabler may be represented into a cube (e.g. enabler assessment cube ) providing a visual representation of the assessment results. In such a cube the assessment of the individual capabilities constituting the overarching enabler can be plotted, considering the evaluation of the coverage, the maturity, the ease of adoption and status with respect to time progress of each capability. Figure 5: The assessment cube for an individual capability The figure above gives a simplified impression of how the cube may look and how it helps to interpret the aggregated assessment results. In terms of projects clearly P8 looks like a winner as it is high on all three dimensions. P16 is of interest, and the question would be whether the benefits obtained would be worth the significant investment that may be required to facilitate industrial adoption. P11 and P17 would also be of interest. P9 could be of interest for reaching achievements for a later timeframe (e.g., 2050). This figure only shows a 2D representation, hence there could be other projects of significant interest (e.g. with high contribution and ease of adoption but low maturity). A means of showing this either three dimensionally or in 2D (by showing different faces of the cube) will be developed during

31 5.3. Interpretation of synthesis results The enabler assessment cube is used for representing the assessment results of the relevant capabilities. Such a cube shows whether the capabilities are well-covered or poorly-covered, and how their maturity and ease of adoption are ranked. This section explains how the results in such cube may be interpreted. Effectively, the capabilities for which both maturity and ease of adoption receive the top score, represent project results that indicate high potential for early implementation and thus for becoming reality in the aviation system. While it is tempting to consider the projects that contribute to a capability occupying or approaching the top right hand vertex of the cube (H,H,H) as having most value, and projects contributing to capabilities closer to the origin point as having less value. However, the value of projects contributing to capabilities elsewhere in the cube should not be underestimated: Projects with low coverage of an enabler might still be an essential element for the achievement of a safety goal; projects may also deliver potential supplements to the SRIA (See Section 6.1). Low maturity projects or capabilities may be crucial for reaching safety goals further away in time, especially when having a large contribution to an enabler. A project with a low ease of adoption score may still deliver benefits that outweigh the required effort for adoption by far. 31

32 6. SYNTHESIZING ASSESSMENT RESULTS This section describes how to identify issues in the realisation of the SRIA and accordingly the safety goals of Flightpath 2050, and how to identify opportunities. Section 6.1 explains how to do this based on the results of the bottom-up assessment, while Section 6.2 describes the use of complementary workshop sessions in which a top-down view of the aviation safety R&I field is adopted. In order to enable EC and/ or ACARE to make strategic decisions, the reporting of OPTICS results and conclusions should focus on the main message. Therefore the collected issues and opportunities need to be clustered and structured to provide more high-level conclusions. Furthermore, results of the safety research assessment need to be combined with the results of the socio-economic impact assessment (cf. OPTICS D1.4). The rationales for the identified issues and opportunities are documented Use of assessment results This section explains the various issues that may be identified, and how these are identified from the results of the bottom-up assessment of previous Sections 3 through 5. The identification of the following types of issues is explained: a gap in research being performed; an overlap in research being performed; a potential supplement to SRIA; a bottleneck to implementation of research results; a bottleneck to perform research; and a potential delay in the realisation of a capability. OPTICS can provide recommendations to address the identified issues, such that their negative impact is mitigated. Complementary to these issues, OPTICS can identify opportunities and make positive observations from the collected assessment results. As one example, OPTICS may identify projects with high impact potential, that have a high coverage of a capability and a high ease of adoption. Then, OPTICS can recommend priority be given to the implementation of the results of these projects. Gap in research being performed Section 0 explained how research projects are linked to the safety-related SRIA capabilities based on their overall objectives. The result of such a mapping is depicted in Figure 6 Capabilities 1, 3 and n are all addressed by at least one project and it can thus be concluded that some research is conducted in order to advance towards the SRIA goals. It is noted that this does not necessarily mean that they fully cover the needs; there still may be gap in research to achieve the full capability. It is however obvious that Capability 2 is not addressed by any project. This provides an example of a gap in research. 32

33 Figure 6: Illustrational result of mapping projects to capabilities The assessment of the coverage of enablers shows for which enablers there is a gap in research. Overlap in research being performed Figure 6 also shows that Capability 3 is addressed by all relevant projects. Several factors may play a role in a high number of projects contributing to the same capability, such as: the capability is significant, requires large effort and high investments, and/ or is multifaceted; several projects delivering small contributions to multiple capabilities; and projects addressing the same capability and the same specific topics. It is aimed here to identify those overlaps in research that cause inefficiency or ineffectiveness of research; for this, some analysis will be required. Potential supplement to SRIA Figure 6 shows as well that the R&I activities of Project 2 are only partly covered by the capabilities in the SRIA. If a specific objective is highlighted in a number of project descriptions, but it cannot be related to any of the SRIA capabilities, then this suggests that there might be a gap in the SRIA. This may be identified from national or international projects, but also from EC projects. It is not the responsibility of OPTICS to determine whether this is indeed a gap in the SRIA. Instead, OPTICS will flag this as a potential supplement to the SRIA, leaving it to others (e.g., ACARE) to decide whether this research indeed contributes to the Flightpath 2050 goals, and whether it should be added as new enabler to the SRIA. It is noted that OPTICS WP1.1 already includes a consolidation of the SRIA. Bottlenecks to implementation of research results A low ranking of the ease of adoption metric is an indicator that there may be a bottleneck to take-up of research results in products or operations. First, this can be considered at the level of individual projects, and then at the synthesized levels of capabilities and enablers. The bottlenecks may be 33

34 related to technology, institutional matters and policies. For identifying what type of bottleneck is blocking implementation, the detailed assessment classification of projects need to be considered; see Table 5 in Section 4.3 for details. For a more detailed analysis of the nature of bottlenecks, questionnaire results and/ or interview results shall be used. Bottlenecks to perform research Here, it is considered why research being performed is not on track, or is not expected to deliver in a timely fashion. To determine the causes of bottlenecks, interviews and workshops are expected to be an appropriate means. The questionnaires for project assessment may provide additional input by specifically requesting information about experienced and/ or anticipated bottlenecks. These bottlenecks to perform research may be the cause for a gap or a delay. The identification of bottlenecks can consider, e.g., education and workforce, infrastructure (is there the right infrastructure?), investments (is the level of investments commensurate with the objectives?), and quality of achievements & capability to innovate (are the right projects funded?). Delay in capability completion A delay in the realisation of specific capabilities relates to the expected achievements with respect to the time frames (e.g. 2020, 2035 and 2050) identified in the SRIA. Such delay may be caused by the identified bottlenecks Use of complementary workshop sessions Complementary to the identification of issues via the bottom-up assessment approach, further issues and opportunities in aviation safety R&I may be identified using a more top-down view. To this end, dedicated sessions are organized in which experts with an overview over the aviation safety R&I field take such a top-down view in identifying issues and opportunities in aviation safety related research and innovation. The issues and opportunities of interest are similar as discussed in Section 6.1. The results of these sessions may be cross-checked with the assessment results in order to validate the findings, and to put the results in the perspective of the SRIA. 34

35 7. SYNTHESIZING ASSESSMENT RESULTS The following table provides an overview of activities to be performed when following the methodology, including a description of how it should be done. Care should be taken in dedicating questionnaires and interviews to several of these activities at once. Table 6: Overview of activities in the methodology Activity What and how OPTICS WP2 activities Project selection (Section 3) Project assessment (Section 4) Synthesizing assessment results (Section 5) A selection of research projects is made using publicly available project information such as abstracts. Assessment of contribution of projects to SRIA, by mapping selected projects to capabilities based on project objectives, and next assessing the project contribution to the capability. A rationale is given for each mapping and contribution score. Reviewing of initial mapping, contributions and rationales, for improving consistency of results and improving the methodology (e.g., by detailing mapping criteria). For each project, assessment of maturity and ease of adoption using questionnaires sent out to project coordinators. Project coordinators selfassess their project and include rationale of the scores. Further assessment of maturity and ease of adoption by reviewing publicly available project information for selected projects (e.g., those with significant enabler contribution and no questionnaire response). Further assessment of maturity and ease of adoption by interviews with project coordinators for selected projects (e.g., those deemed highly interesting from a safety benefit point of view). Review of project assessment results, for improving reliability and consistency of results, and improving the methodology (e.g., by detailing guidance). Synthesizing project assessment results to determine: coverage of capabilities and enablers by research projects, synthesized levels of maturity, ease of adoptions, and scheduling. Desk-top research Desk-top research Review meeting Analysis using questionnaire answered by project coordinators Desk-top research and/ or workshop Analysis using interviews with project coordinators Review workshop Desk-top research 35

36 Identification issues (Section 6) of Identification of issues (gaps in performed research, overlaps in research, potential supplements to SRIA, bottlenecks to implementation of research results, expected delays in the realisation of capabilities, bottlenecks to perform research) and of positive observations. This step is performed using the synthesis of assessment results, when needed further information by interviews, and complemented by workshops. Desk-top research and questionnaires, interviews and workshops. Review Review the complete assessment. Goal of review is two-fold: consolidate results and improve and standardize methodology. Review workshop Presentation results of Document the complete assessment and the improved methodology. Documentation 36

37 8. CONCLUSIONS AND RECOMMENDATIONS This document presents a methodology for evaluating the state-of-the-art of research and innovation addressing the ACARE safety related goals, and for identifying associated gaps and bottlenecks. The methodology aims to be used for oversight of progress in safety research and innovation, exploiting as far as possible the SRIA (Volumes 1 and 2). Its use should enable providing strategic recommendations to the EC and/ or ACARE, including suggested corrective actions and priorities. The methodology is to be combined with OPTICS D1.4 s methodology for assessment of the socio-economic impact of aviation safety related R&I. The methodology was developed in an iterative process, building on the project objectives, lessons learnt from previous projects similar to OPTICS, and a survey of relevant indicators. Use was made of the specification of methodology requirements, feedback from other CSAs and internal and external reviews, and a test of the draft methodology. During the OPTICS project, the methodology will be further improved and extended. The first planned extension is the development of templates for questionnaires and interviews. These should support the retrieval of objective data, and minimise the use of subjective opinions of project coordinators. It is best coordinated with other Coordination and Support Actions as to how R&I projects are approached. Further options for extension and improvement of the methodology are considered when the deliverable of the first year s results has been completed. At least the ideas for further development of the methodology documented in Appendix F should be considered, as should the feedback and lessons learnt from application of the methodology during the first year. Rationales of qualifications can be analysed to establish criteria to strengthen the consistency of project assessment results. One specifically important option to consider will be to compare the project assessment results not only with the SRIA capabilities and enablers, but also with the Flightpath 2050 goals. 37

38 9. REFERENCES [1] Advisory Council for Aeronautics Research in Europe, Strategic Research Agenda, Volumes 1 and 2, October 2004, [2] Advisory Council for Aviation Research and Innovation in Europe (ACARE), Realising Europe s vision for aviation, Strategic Research & Innovation Agenda, Volumes 1 and 2, September 2012, available on [3] AGAPE, D-4.1 Final Report, 30 June 2010 [4] AGAPE, Project Final Report, Publishable Summary, 30 June 2010 [5] AGAPE, The AGAPE Methodology, Section 2, provided by D. Young [6] Buxton, M. and S. Hanney, 1996, How can payback from health services research be assessed?, Journal of Health Service Research and Policy, Vol. 1, pp [7] Community Research and Development Information Service (CORDIS), [8] Council for the Humanities, Arts and Social Sciences (CHASS), Measures of quality and impact of publicly funded research in the humanities, arts and social sciences, [9] Duryea, M., Hochman, M. & Parfitt, A., Measuring the impact of research, 2007, retrieved from [10] EATMP, European Operational Concept Validation Methodology (E-OCVM), version 3.0, in two Volumes, available on: [11] ENQA report on Standards and Guidelines for Quality Assurance in the European Higher Education Area, [12] European Aeronautics: A Vision for 2020, Meeting society s needs and winning global leadership, report of the group of personalities, January 2001, [13] European Commission, Aeronautics and Air Transport Research, 7th Framwork Programme , Project Synopses Volume 2, Calls 2010 & 2011, 2012 [14] EXCROSS, D3.2 Final List of Selected Projects, 17 September [15] EXCROSS, D4.1 Preliminary List of Synergies and Opportunities, 28 April [16] EXCROSS, Publishable Summary 1st year, October [17] Flightpath 2050, Europe s Vision for Aviation Report of the High Level Group on Aviation Research, Publications Office of the European Union,

39 [18] Hanney, S., J. Grant, S. Wooding, M. Buxton, 2004, Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK s Arthritis Research Campaign, Health Research Policy and Systems, Vol. 2, No. 4. [19] Knott, J. and A. Wildavsky, 1980, If dissemination is the solution, what is the problem, Knowledge: Creation, Diffusion, Utilisation, Vol. 1, No. 4, pp [20] Lavis, J., et al., 2003, Measuring the impact of health research, Journal of Health Services Research and Policy, Vol. 8, No. 3, pp [21] MEFISTO, Final Publishable Summary, [22] MEFISTO, Final Report, Febuary 2010 [23] METRONOME, D5.1, Methodology for evaluation of FP5 and FP6 project impacts on community and public policies, June 2009 [24] METRONOME, Final Deliverable, METRONOME methodology for evaluation of research project impacts in the field of transport, July 2009 [25] Minutes for the teleconference held on 10th January 2014, Subject: OPTICS methodology test feedback, from R. Shepherd, 21 January [26] Minutes of OPTICS WP1 telecon of 13 December [27] Minutes OPTICS WP1.2 meeting of 12 December [28] Minutes OPTICS WP1.2 telecon of 29 November [29] Minutes of telecon OPTICS and CAPPADOCIA, 4 February [30] NASA, Technology Readiness Levels (TRL), [31] OPTI, Synthesis Report, February 2013 [32] OPTICS Description of Work, [33] Presentations used in meeting OPTICS and CATER, 24 January [34] RAND, Policy and practice impacts of research funded by the Economic and Social Research Council. A case study of the Future of Work programme, supporting data, [35] Scoble, R, Dickson, K, Fisher, J & Hanney, S. Research impact evaluation, a wider context: Findings from a research impact pilot, [36] Zoomable burst, mbostock s block, December 20,

40 APPENDIX A OVERVIEW OF THE SAFETY RESEARCH AND INNOVATION AGENDA The SRIA has been derived from the challenges and goals endorsed by the Aviation industry through Flightpath 2050 [17]. The SRIA is a collective effort from 350 experts from over 150 organisation engaged in Aviation and Air Transportation, including emergency services. A top down, layered approach was Enablers Interpreting & Undersatnding applied to facilitate the identification and construction of the content, Capabilities Knowledge & Functionality ensuring that the goals and challenges of the Safety and Security domain of Metrics Achievements & Timing Flightpath 2050 are addressed as completely and consistently as possible. Research Concepts, Services & Technologies The methodology applied is illustrated in Figure 7, consisting of the assessment of the Vision, the challenges and goals Figure 7: ACARE methodology contained within Flightpath 2050 by stakeholders from across the Aviation and Air Transport industry. Further steps build upon the initial assessment, identifying and clustering meaningful enablers ( what is required to achieve the goals? ), which are then further decomposed into the main capabilities ( How can it be achieved? ) and concepts services & technologies ( With what can the goals be reached? ). This further elaboration is facilitated by the use of a simplified safety and security management lifecycle, illustrated in Figure 8, whereby attention is paid to assessing and verifying the relevance of the identified capabilities, their coverage and applicability within each lifecycle phase. It should be noted that Data and Information Management is considered as fundamental transversal enabler to the entire management lifecycle process. Past experience from previous monitoring initiatives has highlighted the difficulties in monitoring progress against the goals. Consequently, during this exercise, attention has been paid to identifying meaningful and measurable metrics which will facilitate OPTICS to verify achievements over time (how much and by when). Aviation Services Airports European Commission Research & Academia The SRIA contains enablers for five challenges: meeting societal and market needs, maintaining and Figure 8: Safety and security management lifecycle extending industrial leadership, protecting the environment and the energy supply, ensuring safety and security, and prioritising research, testing capabilities and education. The relevant enablers for OPTICS are the safety-related enablers of Challenge 4 (ensuring safety and security); these are collected in Table 7. Vision Challenges Goals Manufacturing Industry Member States Data and Information Management Why Regulators What How When With what Prevention Detection Action Accommodate Recovery Adaptation Table 7: Overview of safety-related enablers in SRIA 40

41 Enabler name Further information 1. System-wide Safety Management Systems The identification and implementation of a Safety Management System to operate throughout the whole chain of Air Transport activities. 2. Safety radar Innovative methods, processes and services which ensure the identification and detection safety hazards to the total air transport system. 3. Operational mission management systems and procedures 4. System Behaviour Monitoring and self-healing Protection and responses which enable hazard risk management through appropriate tools including atmospheric models enabling the optimisation of trajectories to ensure hazard and collision avoidance throughout all flight phases and on the surface. Systems which enable the proactive detection of degraded and abnormal situations and optimised healing. 5. Diagnostic Analysis Tools, methodologies and processes which aim to automate the capture and analysis of aviation accidents, incidents and occurrences. Further improve the efficient identification of trends and emergent hazards, aiming to mitigate the risks to aviation safety. The application in current and new developments of technology, system designs and operations and necessary requirements and regulations and their effects on human performance. 6. Standardisation and Certification Innovative approach to standardisation, certification and approval processes. Advanced methodologies including simulation tools applied to compliance demonstration of safety requirements at component, product, system and system of systems level, including human, social and technical aspects, leading to efficiency and shorter time to market of new products, services and operations. Improved methodologies for standardised approval and licensing. 7. Resilience Methodologies and tools, products and services which ensure the air transport system is resilient by design and operation to current and predicted safety threat and hazard evolution. IT security concepts resilient against cyber-attacks applied throughout the global aviation system. 8. Human-centred automation Human-centred automation is concerned with the design of automation and information systems to support and optimise human roles across the ATS, including air and ground operations and maintenance. 9. New Crew and Team Concepts New crew and team concepts encompass both the functional interactions of all operators and users of the ATS and their culture. This delivers support optimised performance, monitoring, recovery from stress as well as effective change and evolution of the system. 10. Passenger Management Improved passenger management is based on better understanding of the characteristics, behaviours and cultures of passengers and leads to better prediction of threat, management of behaviour and recovery from stress. It is based on a firm understanding of the diverse dimensions of passenger culture built on measurement and analysis. 41

42 APPENDIX B LITERATURE SURVEY OF PREVIOUS PROJECTS By reviewing previous projects lessons learnt can be obtained. Relevant projects for the purpose of this review are those addressing an assessment of funded projects and developing specific methodologies. These methodologies include approaches for the consolidation of project results versus goals, the assessment of state-of-the-art, the identification of gaps and bottlenecks, and the reporting of results. The projects reviewed are: AGAPE ( ), which aimed to evaluate the progress being achieved by European R&T activity towards the Vision 2020 Goals during 2000 and 2009; OPTI ( ), which aimed to monitor the progress being achieved towards the ACARE Strategic Research Agenda (SRA) goals for both technological and institutional enablers during 2010 through 2012; MEFISTO ( ), which aimed to develop a process for conducting impact assessments of Framework Programmes in the aeronautical sector; METRONOME ( ), which aimed to develop a methodology for evaluating impacts of transport projects in Framework Programmes focusing on 1) strengthening industrial competitiveness 2) contributing to sustainable development, and 3) improving community and public policies. EXCROSS ( ), which aims to enhance cross-fertilization and synergies between safety research initiatives in the different transport modes. For each project a summary and a description of the adopted methodology is provided. The lessons learnt from each project are presented: first as identified in relevant project documents, and then as perceived by the OPTICS authors of the present report. B.1 AGAPE The following references have been used for this project: [3], [5] and [4]. AGAPE project summary ACARE (the Advisory Council for Aeronautics Research in Europe) is the European Technology Platform for aeronautics and air transport and its purpose is to provide guidance for the future of the European aeronautics research aiming at two general objectives: to meet society s needs for an efficient air transport system and to achieve global leadership for Europe in civil aviation. These two overarching objectives are the cornerstones of the Vision 2020 document produced back in Vision 2020 set ambitious goals addressing the breadth of the Air Transport challenges for a sustainable future: Environment, Safety, Air Transport Efficiency, Security and Quality & Affordability. As part of its mission of supporting the fulfilment of the Vision 2020 Goals (also known as the ACARE Goals), ACARE published a first Strategic Research Agenda (SRA) in October 2002 (SRA-1) and a second in October 2004 (SRA-2). The SRAs address the research needs of Europe in the field of air transport systems up to Years later, the Air Transport community considered it to be of prime relevance to take stock of the progress. Therefore, it was decided to include in the Work Programme of an EC FP7 Call a topic dedicated to Evaluation of the impact of FP5 and FP6 projects in the field of Transport. The AGAPE (ACARE GoAls Progress Evaluation) project answering to this topic was funded and ran for 2 years from 1st July 2008 to 30th June

43 In the Vision 2020 document goals have been identified for five high-level challenges, respectively: Environment 1) 50% CO2 reduction 2) 80% NOx reduction 3) Reduction of noise emission by half 4) No impacted people outside airport boundaries 5) Green manufacturing, maintenance and disposal ATS Efficiency 8) 3-fold increase in traffic 9) 99% flights with 15 min 10) Time in airport < 15 min (SR) or 30 min (LR) 11) Seamless ATM system Safety 6) 80% reduction of accident 7) Minimise human error Security 12) Zero successful hijack Quality and affordability 13) Fall in travel charges 14) Halved time to market and competitive supply chain 15) Increase passenger choice 16) Improve Air Freight Services The overarching objective of the AGAPE project was to define and apply a methodology to evaluate the progress being achieved in 2008/09 by European R&T activity towards the Vision 2020 Goals as defined in 2000 for the 2020 horizon. In other words the AGAPE project objective was to verify whether the current and planned R&T activities were meeting the ambitions of the Strategic Research Agenda. AGAPE methodology The guiding principle of the methodology was to generate for each goal the following overview: What are the results achieved from 2000 (until 2009) and what is the level of completion of ACARE goals? What are the results foreseen from on-going initiatives and what will be the level of completion of ACARE goals? What are the corresponding gaps? Gaps to be understood as the resulting deltas between [full goal completion] and [achieved + foreseen results] For the baseline a set of reference aircraft and a reference fleet are used; these are based on key statistics of the air transport system in 2001 (this baseline is equal to the SRA-1 baseline). The approach taken is to assess, at the time of the year under consideration (i.e. 2009) the contribution to ACARE goals of those technologies that are being considered at a secured TRL6 (Technology Readiness Level level 6) or as capable to reach TRL6 before TRL6 corresponds to a representative model or a prototype system being tested in a relevant environment. Examples include testing a prototype in a high fidelity laboratory environment or in a simulated operational environment. The definition of the level of completion of a goal in 2020 as the predicted level of performance of technologies available at TRL6 at that time is in line with the SRA-1 approach. This approach is dedicated to fostering the development of the range of enabling technologies that are needed to achieve the goals. This is necessary since the enter into service of an aeronautical product or system is driven not only by the availability of mature technologies but also by many other parameters which are outside of the research perimeter such as the market demand and the regulatory agenda, both of which have a strong influence on the characteristics and timing of aeronautical products and systems. 43

44 The results of the three questions posed at the beginning of this section are visualized in bar charts. For quantitative goals the bar chart of Figure 9 was used. For qualitative goals the bar chart of Figure 10 was used. To perform the assessment more than 150 experts were mobilised from the ACARE Stakeholder Community; they were divided over dedicated Goal Groups each in charge of one or more goal progress evaluations. The scope of the analysis included research projects undertaken in the period from 2000 (when the Vision 2020 was defined) to the current date. The analysis includes completed and running research projects. The project coordinators or involved experts provided relevant information on project results. Other data have been obtained from project reports or websites available in the public domain. The AGAPE experts reviewed material from research projects launched in Europe as part of: the European Commission Framework Programmes (in particular 5 and 6); the National programmes of the Member States; the privately funded projects at company level. It is noted by AGAPE that the goals are diverse in their nature as the range of expertise required to evaluate the technological progress. The main challenge was to perform the evaluation in a common framework that could be applied to each goal. Figure 9: Quantitative goal completion assessment Figure 10: Qualitative goal completion assessment Lessons learnt identified by AGAPE The lessons learnt listed below are taken from publically available AGAPE documentation. There is a need for better approaches to quantification in some areas (in the available AGAPE documentation this statement is not further detailed). The results of the research projects for the considered period must be made available for consultation to the AGAPE experts. Overall there is a need for improved access to project results and knowledge management. Evaluation experts should be mandated to perform the evaluation. Lessons learnt identified by OPTICS from reviewing AGAPE The lessons learnt listed below are distilled by the authors of this deliverable from publically available AGAPE material. 44

45 Not all goals lend themselves to direct assessment in terms of Technical Readiness Level. For those goals an indicative maturity scale can be adopted using expert judgment. To assess the contribution of projects towards meeting future goals, it works well to distinguish between the realised achievements towards meeting the goal and the expected future achievements towards meeting the goal. It should be taken into account during the evaluations that benefits from new technologies are not simply cumulative but are often subject to trade-offs. The presentation of the results of the state-of-the-art assessment and gap-analysis should be easy to understand for an uninitiated reader. AGAPE can be used as an example for wellpresented results. The projects AGAPE selected for the assessment constitute a representative set rather than a fully exhaustive basis. This way of working decreases required effort but gives satisfactory results, although the results are of a more indicative nature picturing trends, and may therefore not display precise results. B.2 OPTI The following references have been used for this project: [31] and [13]. OPTI project summary OPTI created a platform and an approach to monitor the progress being achieved towards the ACARE Strategic Research Agenda (SRA) goals for both technological and institutional enablers. An update was carried out on the state of progress towards meeting the technological objectives by adding two years to the period of analysis that was considered in AGAPE, thus the analysis involves evaluating the impact of European aeronautics research projects performed in the period This implies that also projects under FP7 were selected (launched before 2012). Clean Sky and SESAR are not yet fully accounted for in the assessment of technological completion with respect to ACARE goals. The institutional enablers were thoroughly analysed to assess the state of progress. The impact of specific institutional actions or initiatives in the period was evaluated. OPTI considers the following institutional enablers (these are detailed in SRA-2): Education, knowledge and workforce; research infrastructures; standardisation, regulation and certification; supply chain optimisation; European synergy in RTD; International collaboration. OPTI methodology OPTI uses the same methodology for technical progress evaluation as adopted by AGAPE. The approach taken is to assess, at the time of the year under consideration (i.e. 2012) the contribution to 45

46 ACARE goals of those technologies that are being considered as capable to reach TRL (Technology Readiness Level) 6 before OPTI adopted and updated the Institutional Readiness Level (IRL) metric for identifying the institutional enabler level of completion with respect to institutional SRA goals. Two main phases are identified in the IRL scale: definition/set-up phase implementation phase Each of these phases ranges from a low to a high readiness level, see Figure 11. Figure 11: IRL metric For each institutional enabler and for all actions identified (a specific activity defined under an institutional enabler, for example provide opportunities for student mobility under institutional enabler education ) an IRL is provided. The IRLs are determined by a combination of inputs from questionnaires and expert know-how. The ACARE- Monitoring Group played the role of the Advisory Board for OPTI, monitoring the progress of the project. An example of the results for the technical evaluation is given in Figure 12, showing the level of completion of ACARE goals; the bottom value indicates the achieved percentage, the middle value indicates the percentage that will be achieved by ongoing projects and before 2020, the top values indicates the gaps that new projects will have to fill. 46

47 Figure 12: OPTI assessment of completion of 7 ACARE goals In Figure 13 an overview of the Institutional Enablers status is provided presenting for each institutional enabler an average perceived level, and the minimum and maximum IRL for the specific actions. Figure 13: Overview of Institutional Enablers IRL Lessons learnt identified by OPTI The lessons learnt listed below are taken from publically available OPTI documents: The AGAPE final report presented lessons learned for next evaluations. These have resulted in the use of questionnaires in the OPTI project, in order to obtain a better insight in the relevant 47

48 R&T projects. However the response rate from the R&T projects was rather low, making it very difficult for the OPTI team members to draw conclusions. There is no unique and well assessed methodology to evaluate the impact of EU supported projects on improving the safety of air transport. A continuous monitoring/observation should be ensured during Horizon 2020 For future monitoring actions, measures should be taken to ensure: o there is a dedicated action to monitor achievements for each SRIA challenge; o the funding of activities to develop methodologies meant for the assessment of the technology progress and of the impact on societal and market needs; o a good database of information to assess the progress through workshops and wellstructured questionnaires; o a much higher response rate, if questionnaires will be used again; o the involvement of specialists for all the areas/topics to be assessed; Lessons learnt identified by OPTICS from reviewing OPTI The lessons learnt listed below are distilled by the authors of this deliverable from publically available OPTI material. While the focus of the OPTI assessment is aeronautics, it should be noted that the Strategic Research Agenda alludes to wider implications through its reference to "systems of systems", which takes on a much broader perspective when considering European Transport Policy and its greater emphasis upon interconnectivity between transportation modes. B.3 MEFISTO The following references have been used for this project: [22] and [21]. MEFISTO project summary The MEFISTO project had three main objectives: to develop a process for conducting impact assessments of Framework Programmes; to apply this process in the aeronautical sector, and; to propose how it could be used more widely across the transport sector. The first major output of the work was a description of the process methodology developed for assessing the impact of FP5 and FP6 on a variety of issues of importance to the aeronautical sector. The second major output was a demonstration of this process methodology in use. Results generated by the methodology application were also described. The third major output was a description of the required customisations to apply the process in other transport sectors. The MEFISTO methodology was used to collect the views of more than 350 respondents, at various levels and across many branches of aerospace research. Responses to 94 key questions to assess the impact of policies were gathered. MEFISTO also performed 53 interviews of selected executives in aviation and representatives of governments and the European Commission. More than 800 separate 48

49 comments were collected from these interviews and analysed against 20 key issues for the output impacts and input conditions. MEFISTO also considered how the impact assessment process could be transferred to and applied in other transport sectors. MEFISTO methodology MEFISTO identified 20 Key Policy Issues to be analysed. The 20 Key Policy Issues were arranged into four groups under the headings of: Driving Impacts, Structural Impacts, Leveraging Impacts, and Input Impacts. Driving Impacts were the direct impacts of the research work advancing technologies and increasing capability of the enterprises concerned through participation in projects. Structural Impacts were the effects of the FPs influencing the way in which enterprises collaborated and how their relationships developed through participation in the FPs. Leveraging Impacts were those effects of the FPs adding value by making the whole value of the research community more effective than could be attributed to the projects alone. Input Impacts were those effects that stemmed from the actions and structures provided by the Commission. The 20 issues were the following ones: Driving impacts: Improvement of the competitive position, Improvement of Mobility, Improvement of the environment, Improvement of Safety and Security, Stimulating new knowledge, Bridging the gap between research and application. Structural Impacts: Mobilising European research, Stimulating additional funding, Coordination with Member States programmes, Involving New Member States in aeronautics, Involving SMEs in the supply chain. Leveraging Impacts: Improving relations between research and SMEs, Stimulating Excellence, Benefiting education, Can Europe do without EU funding? Input Impacts: The efficiency of EU actions, Efficiency of the evaluation process, Efficiency of project work, Costs involved in European collaboration, Efficiency of EU international collaboration. For each important policy issue, the team drew up a series of questions and placed them into the categories of Driving, Structuring, Leveraging or Input Impacts. This large matrix produced a set of 214 background topics that seemed potentially of interest to MEFISTO. This number of topics was too large to be directly used for a questionnaire. A workshop was held to improve the focus and grouping of possible questions. A final list of 94 questions was obtained. The questions were grouped and colour coded so that, whilst all questions were visible respondents were guided to a more efficient use of their time and expertise. 49

50 Figure 14: Sample questions from the questionnaire In parallel with the survey MEFISTO carried out 53 interviews with people with different experience in the FPs. The set of 20 Key Issue questions that further encapsulated the original matrix and combined some questions together was used as a guide. Interviews were a mixture of face-to-face and telephone interviews. The survey response was processed numerically. The MEFISTO consortium derived conclusions and recommendations (i.e. suggestions) from the questionnaire analysis. The interview results were integrated. Lessons learnt identified by MEFISTO The lessons learnt listed below are taken from publically available MEFISTO documentation. It is impossible to measure the degree to which European policy objectives had been achieved. The main reasons why this simple process of measurement is inappropriate is that the impacts of research developed over time, in parallel with many other influences, contribute in many cases to long-term technical aims, and policies are not necessarily expressed as quantified goals. MEFISTO relied mostly on experts for impact assessment. There are few appropriate metrics for impact. The impact of research projects is often augmented by other influences, and some of the desired impacts are not susceptible to metrics. MEFISTO had originally intended to invite individual people to respond to the survey by means of a personal and direct invitation. This plan proved impractical. It was much harder than expected to generate lists of people needed to be approached and MEFISTO evolved the plan to be a mixture of direct invitation and roll-out from those invited to other colleagues. Lessons learnt identified by OPTICS from reviewing MEFISTO The lessons learnt listed below are distilled by the authors of this deliverable from publically available MEFISTO material. The lessons learnt can be seen as MEFISTO good practices that should be taken into consideration by OPTICS. Both practices are related to the presentation format of the results. 50

51 The questionnaire results were quantitative, but the MEFISTO consortium transformed them into text, with clear claims and argumentations. This step diminishes the objectivity of the assessment, but it greatly improves the readability of the results. For each of 20 issues, MEFISTO complemented the text summarising the results with more visual communication formats. Figure 15: Technological roadmap of Framework Programme projects in environment. B.4 METRONOME The following references have been used for this project: [24] and [23]. METRONOME project summary The METRONOME project (A METhodology for evaluation of project impacts in the field of Transport) had the objective of developing a methodology for evaluation of impacts of projects supported in the Framework Programmes (FP) 5 and 6 of the European Commission with particular focus on three themes. The themes were: Strengthening industrial competitiveness (IndCo); Contributing to sustainable development (SuD); and Improving community and public policies (CPP). In practice, the project aimed to fulfil this objective by: 1. Developing a methodology to evaluate the achievements of projects supported in FP5 and FP6 from the three above themes/perspectives. 2. Evaluating the performance of FP5 and FP6 in respect of the three themes by evaluating a sample of FP5 and FP6 projects with the developed methodology. Eventually the methodology was tested with a sample of 100 FP5 en FP6 project. 3. Producing recommendations regarding future transport research objectives based on the evaluation results. 4. Producing guidelines to the Commission for the further use of the methodology. 51

52 METRONOME methodology The METRONOME screening, selection and evaluation methodology has three main phases: 1. Identification of European transport research and policy objectives for Industrial Competitiveness (IndCo); Sustainable Development (SuD); and Community and Public Policies (CPP) 2. Screening and selection of FP5 and FP6 themes and projects for the evaluation 3. Evaluating project impacts through the METRONOME impact model, using a multifaceted approach Identification of European transport research and policy objectives for Industrial Competitiveness, Sustainable Development, and Community and Public Policies serves as the basis for both the project screening and impact evaluation in the METRONOME methodology (phase 1). The first task was to identify the relevant European research and policy documents for the three thematic fields. The METRONOME high-level European transport policy objectives for IndCo, SuD and CPP were derived from relevant European policy documents. In some cases, two or three of the objectives were merged into a single more comprehensive one. The project screening and selection in the METRONOME methodology (phase 2) follows three steps: (1) the identification of relevant Framework Programme themes and projects, (2) the acquisition of project outcomes (i.e. final reports), and (3) the selection of relevant projects using text mining software and a dedicated checklist. The METRONOME methodology takes a two-dimensional approach to project impact evaluation (phase 3). On the one hand, it evaluates the projects achievements against the FP5 and FP6 Work Programme objectives and targets set for IndCo, SuD and CPP themes. On the other hand, it evaluates, through the METRONOME impact model, the impacts of the FP research projects according to four impact groups with associated indicators: (a) impact indicators on management and coordination, (b) scientific impact indicators, (c) customer/end user impact indicators, and (d) societal impact indicators. Project impact evaluation is done via evaluation matrices, coordinator questionnaires, lead user interviews and workshops. Two types of evaluation matrices are used: a project evaluation matrix and a dissemination quality matrix. The former uses final project reports as input and shows the objectives and indicators as identified by METRONOME versus the contribution of project to meet those. This is done on a 4-point scale; from fully met to not at all. The dissemination matrix concerns the success of project result dissemination, see Figure 16. It enables evaluators to specify the characteristics of specific dissemination activities undertaken during and after the project lifetime in order to assess the potential effects of project results and indicate whether estimated impacts upon the objectives are likely to have been achieved in practice. The dissemination reports were reviewed to provide evidence of the scope and nature of dissemination activities conducted. In projects where no dissemination report had been produced, final project reports and project websites (where these had been produced and were still active) were reviewed. A complementary impact evaluation method to project evaluation matrices in the METRONOME approach was a questionnaire designed and distributed to a sample of Framework Programme project co-ordinators. The main aim of the questionnaire was to obtain information of the impacts of the research projects in each of the four indicator groups. The questionnaire was composed of four parts according to the impact groups. The METRONOME lead-user questionnaire was designed to assess lead-user (end-user) opinions of the impact and performance of FP5 and FP6 research projects. In order to maintain consistency between 52

53 the results of interviews by different evaluators (partners), a uniform template was developed and applied (see [6]). The questions in the template relate to the perceived impact of FP5 and FP6 research in general, the results of specific projects in which the respondents had been involved, the benefits for the respondent and his/her organisation and what did and did not work in FP5 and FP6 projects. Figure 16: Dissemination quality matrix In order to get feedback from the EC, but also from other end-users of the methodology, a workshop was arranged at the start of the METRONOME evaluation process. The workshop aimed to identify the needs and preferences of various types of end-users for the evaluation methodology and the results. The workshop participants were from the fields of policy, research and industry, which was considered useful in initiating new cooperation between e.g. the EC evaluation projects and technology platforms in the project evaluation. An example result obtained by METRONOME is given in Figure 17. The figure shows to which extent FP5 projects met the objectives. Four levels of objectives are defined: high-level work programme objectives, more detailed key actions of the work programme or programme subdivision objectives, strategic objectives of the projects themselves, and objectives of the theme community and public policies (one of the three themes in which METRONOME is divided). 53

54 Figure 17: Extent to which projects met the objectives (FP5 project only) Lessons learnt identified by METRONOME The lessons learnt listed below are taken from publically available METRONOME documentation. Impact and effectiveness evaluation differs from objective/target achievement evaluation in that the latter does not take into account the side effects or unanticipated effects that a research project or programme may have. Impact evaluation does not take into account the relevance of objectives set. This is why evaluation of both achievements and impacts (gained and expected) is needed. Testing the METRONOME methodology illustrated that different mixes of evaluation methods (both reviewing and surveying) are needed for evaluation of projects. A co-ordinator survey provides co-ordinators self-evaluation of the potential and actual impacts of the projects. The results were useful as supplements to other evaluation methods, but by themselves the risk of bias in co-ordinator responses, as always in self-evaluations, is present. The lead-user survey was found to be the most valuable source of information regarding the actual use of research results. This kind of survey should be promoted in the future, combined with in-depth, long-term project impact evaluations, in co-operation with technology platforms and EC officers. The main difficulty encountered during the METRONOME methodological development was the availability of project result (e.g. Final Report) data. A structured, up-to-date FP project result database that is ready and available for the evaluators would enable more reliable, less time-consuming and less costly FP impact evaluations. A major difficulty identified during the methodology development was the relatively low response rates in co-ordinator surveys. Sending the questionnaires officially by EC bodies could improve the response rate; the responses could even be demanded as a part of project proceedings. A major difficulty identified was the interpretation of the multi-level objective and target structures of the FPs as the basis for evaluation. In order to avoid missing or misinterpreted objectives and targets, input of strategic research objectives and targets from official EC data sources should be ready and available to the evaluators. The time reserved for methodology development (18 months) was too short. The input data gathering (project results), FP objective/target analysis and the coordinator survey all took much longer than expected. 54

55 An important aspect related to the timing of programme evaluation is the continuity of evaluations. METRONOME evaluation presents only a first phase in a FP impact evaluation process. As it often takes a long time for project impacts to materialise, only a repeated (and simultaneously elaborated) evaluation process can provide more detailed analysis of project or programme impacts. Evaluating the ultimate impacts, which might be realised 10 or more years down the road, is a very difficult task within all of the METRONOME evaluation themes. Finding the right time for FP evaluation is difficult. In our case, for example, FP5 and FP6 stand on a different line in the evaluation because of the temporal aspects. Later implementation of FP6 might have evoked, depending on the circumstances, more intense (positive or negative) responses in the surveys than the more distant FP5. Project evaluations need to be validated with selected experts, who are aware of the project results but did not participate in the projects being evaluated, in order to avoid biased opinions. An econometrics expert either within the consortium or as an external expert should be included in the evaluation team from the start of the project. Lessons learnt identified from reviewing METRONOME The lessons learnt listed below are distilled by the authors of this deliverable from publically available METRONOME material. Difficulties can be expected in finding information about project results on the internet after several years of termination. The methodology can therefore not rely on information provided via internet. Multiple types of analyses (e.g. project self-evaluation and expert assessment) should be included in the methodology in parallel to allow validation. The type of content in final project reports varies a lot, although they should give an accurate and comprehensive overview of project research activities and their outcomes. For certain projects it is therefore most likely needed to have additional documentation. Surveys/questionnaires should only be used to reinforce findings from other assessment approaches, but no evaluation results should be solely based on these. Review of dissemination activities supports the (final) report analysis by providing a more detailed indication of the likely impacts of research projects. Technology platforms are important bodies to consult with for the validation of the objectives and targets against which is assessed. The terminology used (e.g. output, outcome, impact, effectiveness) should be clearly defined to avoid inconsistent use. Different types of impact (i.e. the products, events, conditions and/or changes that follow from the research outcomes) should be considered: anticipated and unanticipated, inside and outside the target area (relevant and irrelevant), and productive, neutral and detrimental. 55

56 B.5 EXCROSS The following references have been used for this project: [16], [15] and [14]. EXCROSS project summary EXCROSS is a Supporting Action (SA) of the European Commission to enhance cross-fertilization and synergies between safety research initiatives in the different transport modes (e.g. road transportation, aviation, etc.). EXCROSS addresses the fragmentation that exists in Europe between safety initiatives in the different transport modes, with little cross-domain learning and sharing of experience. In particular, the objectives of this SA are: Identify synergies and opportunities for cross fertilization between different transport modes; Identify research gaps common to the different transport modes, to exploit synergies and remove discrepancies. The activities are organised around the following main streams: Establish a common understanding of safety principles across the different transportation modes, by identifying tacit assumptions and making underlying principles explicit; Collection, selection and analysis of past research actions dealing with safety. EXCROSS has identified more than 300 R&D safety projects and analysed 80 of them, across the four transport modes of aviation, maritime, railway, and road; Identification of the main stakeholders and preparation and dissemination of information tailored for their needs. The EXCROSS analysis covered topics such as incident and accident management, investigation, human factors, detection and management of fatigue, certification, safety assessment, dangerous goods transportation, training, technologies to detect dangerous situations and increase situational awareness. EXCROSS methodology The EXCROSS methodology was centred on the analysis of EU safety-related research projects and initiatives. The consortium collected more than 300 projects and initiatives, from different founding bodies like the European Commission, EUROCONTROL, EASA, SESAR JU, National Programmes, and so on. The consortium adopted a two pronged method of analysis, to combine the information gathered from the source projects, the understanding achieved in the first project year, and the rich expertise of all the partners, experts and steering group members [15]. The method aimed to combine the virtues of a bottom-up and a top-down approach, that is the thoroughness of analysing single project by single project, the expertise available in the consortium, the excellent network of external experts and steering committee members. A summary schema of both approaches is depicted here below, with more details in the following. 56

57 BOTTOM-UP APPROACH TOP-DOWN APPROACH Analysis of individual projects Identification of hot R&D topics Discussion with EXCROSS partners Input by partners/experts to create shortlist Creation of project groupings Aggregation of partners on topics Definition of topic / database search Database search to create project groupings Analysis of groupings Analysis of groupings Figure 18: Overview of the methodology Bottom-up approach: the EXCROSS consortium screened all the projects to identify the ones more promising from a cross-domain perspective and likely to present cross-fertilisation opportunities. The screening was performed by a quantitative ranking on the following categories: Availability of sufficient data, Relevance to EXCROSS, i.e. transferability of results to one or more other transport modes, Relevance to safety, Currentness of results, i.e. whether the results are still up-to-date. The iterative process ended up with a shortlist of 73 selected source projects, whereas another 32 projects were included in the Reserve List [14]. The selected source projects were extensively analysed by the consortium by reviewing the relevant documentation for each one of them and interviewing points of contact, whenever available. Each project was classified using the EXCROSS taxonomy and the main analysis results were captured in a working document, covering the following points: 57

58 Project abstract, Summary, Relevance for EXCROSS, with examples and pointers to any relevant document, Working hypothesis for cross-mode relevance, Target user(s), Input and clarifications from other partners, with pointers to related projects in other transport modes, Analysis follow-up, describing the actions to be performed to finalise and/or improve the analysis. The objective of this analysis session was to preliminary identify a set of safety hints potentially relevant for some of the other modes, and to collect feedback from the other partners on the identified hints. Partners feedback typically elaborated on the proposed cross-mode relevance, and/or pointed to other similar or complementary research projects in other transport modes. For some projects, the consortium reached the conclusion that little cross-mode relevance was present and the analysis was closed. Starting from this input, partners defined a more general topic to encompass all the mentioned projects. The topic was then used to search for other relevant projects. All the projects identified in this process constituted the grouping that was used to identify synergies and opportunities for crossfertilisation. The results of the grouping analysis were captured in a working document. The bottom-up approach just described was effective to identify various opportunities for crossfertilisation, but it was not the best approach to profit from the expertise of the partners and the knowledge available through the EXCROSS network (consolidated in the LinkedIn group). This expertise was especially important to identify the potential research gaps, which may not emerge from a bottom-up analysis, since this analysis by definition focuses on what is present in the research and less on what is missing. For this reason, it was decided to combine the bottom-up approach with a topdown one, starting from the knowledge and expertise of individuals (internal to the consortium and external). The process was organised in three steps. It was first asked partners to define a list of hot research topics in their domains. The same list was reviewed and discussed with external experts, to reach a more refined version. A shortlist was drafted in parallel, selecting only the key hot topics. The final list included topics such as (only shortlisted topics included here): Training, Human Factors, in particular, fatigue, workload and impairment seems to be the most relevant aspects to be investigated. Safety Assessment process, techniques, and methods, Certification process, Evacuation procedures, Intermodality stations/areas/nodes, 58

59 Incident/accident analysis and management, Dangerous Goods transportation, Embedded systems. Partners worked on each topic in pairs, or in a team of three, to make sure that the analysis of topics was happening across transport modes. Each group of partners searched the list of selected projects for relevant projects for the topic, and then extended the search to the whole list of projects. The process resulted in a list of relevant projects, i.e. a project grouping. From this stage on, the analysis proceeded similarly to the bottom-up approach. Each project was analysed individually, a summary included in the working document for the topic analysis, starting from which the consortium identified cross-domain synergies and opportunities. This process typically required much iteration among partners, with the topic scope being progressively refined, preliminary conclusions identified, discussed, and refined. The top-down approach was successful and the same underlying structure (i.e. projects grouped by topics) has been preserved for external dissemination. The results of the above analysis were captured in summary documents with the following sections: Topic Summary: the section captured WHAT can be transferred, at a high level. Safety hints: the applicable safety hint(s) for this topic analysis. Projects analysed: name and short description of the projects included in the grouping. Findings: this section described HOW transfer can happen from one mode to the other, including details on which mode. Input and clarifications requested from other partners. Results of the validation: expected results from the validation with external experts. Stakeholders: groups and audience that can be interested in the topic and its findings. Lessons learnt identified by EXCROSS The lessons learnt listed below are derived from personnel involved in the EXCROSS activities. The documentation of many projects was hard to access. This was partially mitigated by consortium members having direct access to some of these documents. However, it was not possible to explicitly refer to some important deliverables. The initial list of projects included only research projects. Safety-related initiatives were later added in order not to lose important information. With the aim of providing strategic recommendations, the analysis at the project level is sometimes too narrow in scope. On the other hand, the topic groupings were too large and with hard-to-define boundaries. The two pronged approach was devised to cover this middle ground, but not always successfully. Lessons learnt identified from reviewing EXCROSS The lessons learnt listed below are distilled by the authors of this deliverable from publically available EXCROSS material. 59

60 The EXCROSS methodology uses careful analysis of the source project results. This part of the work escaped an objective quantification and relied extensively on experts knowledge, in order not to waste relevant information. However, a common format for the analysis results was provided, to harmonise this part of the methodology. Working in small teams on the analysis of project increased the consensus around the analysis results and was often required by the research topics being addressed. The use of infographics to visually display results was greatly appreciated by the various stakeholders. It is likely that the evaluations will require much iteration among partners, with the topic scope being progressively refined, preliminary conclusions identified, discussed, and refined. Figure 19: An example of the infographic used by EXCROSS for the external dissemination. 60

61 APPENDIX C LITERATURE SURVEY: INDICATORS FOR ASSESSING PROJECTS This appendix discusses identified material regarding useful indicators for assessing projects. The following types of indicators are of use: Indicators to assess quality of research; Indicators to qualify output and outcomes for research; and Indicators or a framework to assess the maturity of project research results C.1 INDICATORS FROM PREVIOUS PROJECTS AGAPE The AGAPE project assesses technological progress achieved by projects both quantitatively and qualitatively. The indicators used are: OPTI Has the project developed (or helped in developing) technologies that have reached TRL6 today (and the achieved fulfilment of non-technological enablers where applicable). Has the project developed (or helped in developing) technologies expected to reach TRL6 by 2020 (including the effects of enablers that are decided but are still to be implemented before 2020). OPTI uses the same indicators for technical progress as used in AGAPE. OPTI has adopted the Institutional Readiness Level (IRL) metric for identifying the Institutional Enablers level of completion with respect to Institutional SRA goals. The indicator used is the perceived level identified by a questionnaire and working groups. The levels are shown in Figure 11. MEFISTO MEFISTO grouped 20 key policy issues under four impact categories. The 4 groups and related 20 issues were the following ones: Driving impacts: Improvement of the competitive position, Improvement of Mobility, Improvement of the environment, Improvement of Safety and Security, Stimulating new knowledge, Bridging the gap between research and application. Structural Impacts: Mobilising European research, Stimulating additional funding, Coordination with Member States programmes, Involving New Member States in aeronautics, Involving SMEs in the supply chain. Leveraging Impacts: Improving relations between research and SMEs, Stimulating Excellence, Benefiting education, Can Europe do without EU funding? Input Impacts: The efficiency of EU actions, Efficiency of the evaluation process, Efficiency of project work, Costs involved in European collaboration, Efficiency of EU international collaboration. Each issue was assessed by a set of questions, with experts expressing their level of agreement on a 6 steps scale. An average rating was calculated for each question. 61

62 As far as OPTICS is concerned, the 20 issues could be considered as indicators to be assessed. The corresponding questions could provide the metrics. At the time of writing, it was not possible to access the full MEFISTO questionnaire, to provide more details on this matter. METRONOME The METRONOME impact model proposes the following four indicator groups and associated indicators: a. Impact indicators on management and coordination reflect the enabling factors or tools for complementing the impacts measured in the other three groups below. Examples of indicators: Improved networks, new networks with public/private organisations Networks with global/eu/national partners Systematic dialogue with policymakers Customer orientation: customer involvement in project planning Efficiency of the research results (outcomes) versus resources used b. Scientific impact indicators These indicators reflect the quality and validity of research project results (outcomes) versus the project s own and FP objectives and targets set on different levels. Examples of indicators: Achievements of research projects - outcomes versus FP objectives/targets set Fit between framework and data The power to address previously unsolved questions Number of publications, number of patents c. Customer/ End user impact indicators These impact indicators reflect the (short-term) benefit of the research results to their actual end users (e.g. EC, industry, national governments, ministries, research organisations, etc.). Examples of indicators: Public-policy initiatives New business initiatives/activities Long-term product or service development Advantage and stability of the research results d. Societal impact indicators reflect the more long-term effects of the research on the society (e.g. on the transport system end-users: individuals, logistics companies, industry, etc.). Examples of indicators: Implementation of research output by policy field, industry or other societal stakeholders (Active) use of implemented research output by societal groups Contribution of priority setting, e.g. future research goals 62

63 Contribution to strategy processes of public and private organisations Norms, standards, regulation EXCROSS The EXCROSS project assesses the output of projects against the possibility for cross-fertilisation. The assessment was qualitative, based on experts review of the project results. Indicators were instead used for the preliminary screening of projects, to indentify the ones with the best cross-fertilisation potential. The following ones were used: Availability of sufficient data: yes/no; Relevance to EXCROSS, i.e. transferability of results to one or more other transport modes: assessed with a rank from 1 (lowest) to 3 (highest), Relevance to safety: from 1 (lowest - safety is not a focus of the project, but it is improved as a side-effect) to 3 (highest - safety is the focus of the project). Currentness of results: from 1 (lowest, results are outdated) to 3 (highest, results are still relevant). These criteria were summed up to get a total score and build a ranking among the projects. The ranking varied from a scoring of 0 (lowest) to 9 (highest). C.2 INDICATORS FROM OTHER SOURCES This appendix section includes indicators derived from initiatives and frameworks to assess the research quality and impact. This review covered the documents ([35],[11],[34],[8],[6][18][19][9][20]), looking for indicators to assess research impact and quality. The Payback Framework The most widely cited framework for research impact assessment is the Payback Framework [6][18]. It classifies the impact under five categories: Academic impact: knowledge. Defined as explicit and codified knowledge. Papers, books and book sections can be used as a proxy. Academic impact: impacts on future research. Defined as the generation of new research questions, development of new methods and/or datasets, capacity building, career development. Non-academic impact: impacts on policy. Defined as effects of research on policy at many levels, for example: national policy, the policy of professional bodies, the policies of departments of organisations. It includes effects on the ability, and propensity, of policy makers to use research. Non-academic impact: impacts on practice. Defined as effects on individual behaviour, which may or may not be in line with the policies of the organisation, or group to which the individual belongs. 63

64 Non-academic impact: wider social and economic impacts. Defined as social or economic effects that change society, including impacts on public opinion. Media coverage can be used as proxy for impact on public opinion. Example indicators for these five categories are reported in the table below. Table 8: Payback Framework categories and sample indicators. Impact category Academic impact: knowledge Academic impact: impacts on future research Non-academic impact: impacts on policy Non-academic impact: impacts on practice Non-academic impact: wider social and economic impacts Indicators Peer reviewed papers in journals in leading and respected journals Academic papers commissioned by the Government Book sections Book Presentations to academic audiences Other research outputs Citations and other evidence of use Ongoing debate Ongoing dialogue with other researchers Follow on research by self or others Interdisciplinary contribution to academic research Constructive academic-policy crossover feeding into policy Policy needs feeding back into research and interpretation of findings Scholarships and higher degrees awarded Promotions and subsequent employment of research staff Policy documents and reports Citation in policy documents Testimony of policy makers, managers or decision makers Research reports Patents Evidence of implementation or influence Definition of Key Indicators to inform policy making Findings adopted by practitioners Contributed to discussions on the implementation of policy Evaluation and research reports Articles in local newspapers Articles in national newspapers Magazine articles Radio interviews Television appearance Impact of socio-economic aspects 64

65 The list of indicators should be adapted to the research field being assessed. Impact assessment should also consider the impact depth and spread. This can be done by having experts assessing the indicators and classifying them according to a matrix like the one represented in Figure 20 (the numbers in each cell capture the collected indicators for each). Figure 20: Matrix to classify the impact depth and spread. The impact depth is classified by some practitioners on different levels. An example (from [19]) for the impact on policy makers is: Information received by the policy maker: presentations, papers (reception) Information understood by the policy makers (cognition) Information is referenced by the policy maker (reference) Information influences the action of the policy maker (effort) Information influences the policy outcomes (adoption) Information influences the policy implementation (implementation) The intended benefits are reached (impact) The ATN Model Another example can be taken from the Australian Technology Network of Universities (ATN). ATN developed a model with types of impact, on three levels [9]. 65

PROJECT FINAL REPORT Publishable Summary

PROJECT FINAL REPORT Publishable Summary PROJECT FINAL REPORT Publishable Summary Grant Agreement number: 205768 Project acronym: AGAPE Project title: ACARE Goals Progress Evaluation Funding Scheme: Support Action Period covered: from 1/07/2008

More information

Advanced Impacts evaluation Methodology for innovative freight transport Solutions

Advanced Impacts evaluation Methodology for innovative freight transport Solutions Advanced Impacts evaluation Methodology for innovative freight transport Solutions AIMS 3rd Newsletter August 2010 About AIMS The project AIMS is a co-ordination and support action under the 7th Framework

More information

International comparison of education systems: a European model? Paris, November 2008

International comparison of education systems: a European model? Paris, November 2008 International comparison of education systems: a European model? Paris, 13-14 November 2008 Workshop 2 Higher education: Type and ranking of higher education institutions Interim results of the on Assessment

More information

Horizon Work Programme Leadership in enabling and industrial technologies - Introduction

Horizon Work Programme Leadership in enabling and industrial technologies - Introduction EN Horizon 2020 Work Programme 2018-2020 5. Leadership in enabling and industrial technologies - Introduction Important notice on the Horizon 2020 Work Programme This Work Programme covers 2018, 2019 and

More information

Preparatory paper: food for thought

Preparatory paper: food for thought CNS SYMPOSIUM 2-3 October 2018 EUROCONTROL s Brussels HQ Preparatory paper: food for thought 1 Introduction EUROCONTROL will host a two-day interactive CNS Symposium on October 2 nd and 3 rd, 2018. This

More information

EUROPEAN GNSS APPLICATIONS IN H2020

EUROPEAN GNSS APPLICATIONS IN H2020 EUROPEAN GNSS APPLICATIONS IN H2020 Introduction to Call H2020-Galileo-2014-1 Marta Krywanis-Brzostowska Market Development European GNSS Agency www.gsa.europa.eu/r-d/h2020 Agenda R&D in the European GNSS

More information

CIVIC EPISTEMOLOGIES Civic Epistemologies: Development of a Roadmap for Citizen Researchers in the age of Digital Culture Workshop on the Roadmap

CIVIC EPISTEMOLOGIES Civic Epistemologies: Development of a Roadmap for Citizen Researchers in the age of Digital Culture Workshop on the Roadmap This project has received funding from the European Union s Seventh Framework Programme for research, technological development and demonstration under grant agreement no 632694 CIVIC EPISTEMOLOGIES Civic

More information

Dissemination and Exploitation under H2020

Dissemination and Exploitation under H2020 Dissemination and Exploitation under H2020 Vinnova Sweden, Stockholm Tuesday 20 November 2018 Katrien Rommens Notoglou European Commission, DG RTD H2020 Information and data Content I. Key concepts of

More information

Brief presentation of the results Ioana ISPAS ERA NET COFUND Expert Group

Brief presentation of the results Ioana ISPAS ERA NET COFUND Expert Group Brief presentation of the results Ioana ISPAS ERA NET COFUND Expert Group Mandate of the Expert Group Methodology and basic figures for ERA-NET Cofund Efficiency of ERA-NET Cofund Motivations and benefits

More information

Position Paper on Horizon ESFRI Biological and Medical Research Infrastructures

Position Paper on Horizon ESFRI Biological and Medical Research Infrastructures Position Paper on Horizon 2020 ESFRI Biological and Medical Research Infrastructures Executive summary The Biological and Medical Research Infrastructures welcome the European Commission proposal on Horizon

More information

An Introdcution to Horizon 2020

An Introdcution to Horizon 2020 TURKEY IN HORIZON 2020 ALTUN/HORIZ/TR2012/0740.14-2/SER/005 An Introdcution to Horizon 2020 Thies Wittig Deputy Team Leader Project "Turkey in Horizon 2020" Dr. Thies Wittig Ø PhD in Computer Science Ø

More information

THE ROLE OF TRANSPORT TECHNOLOGY PLATFORMS IN FOSTERING EXPLOITATION. Josef Mikulík Transport Research Centre - CDV

THE ROLE OF TRANSPORT TECHNOLOGY PLATFORMS IN FOSTERING EXPLOITATION. Josef Mikulík Transport Research Centre - CDV THE ROLE OF TRANSPORT TECHNOLOGY PLATFORMS IN FOSTERING EXPLOITATION Josef Mikulík Transport Research Centre - CDV Outlines European Technology Platforms ERTRAC Czech ERTRAC Other national TP Technology

More information

SESAR EXPLORATORY RESEARCH. Dr. Stella Tkatchova 21/07/2015

SESAR EXPLORATORY RESEARCH. Dr. Stella Tkatchova 21/07/2015 SESAR EXPLORATORY RESEARCH Dr. Stella Tkatchova 21/07/2015 1 Why SESAR? European ATM - Essential component in air transport system (worth 8.4 billion/year*) 2 FOUNDING MEMBERS Complex infrastructure =

More information

An ecosystem to accelerate the uptake of innovation in materials technology

An ecosystem to accelerate the uptake of innovation in materials technology An ecosystem to accelerate the uptake of innovation in materials technology Report by the High Level Group of EU Member States and Associated Countries on Nanosciences, Nanotechnologies and Advanced Materials

More information

FP7 AAT Level 0. FP7 AAT Level 0. Roberto Bojeri. Workshop ACARE Italia. Torino, 17 Maggio 2012

FP7 AAT Level 0. FP7 AAT Level 0. Roberto Bojeri. Workshop ACARE Italia. Torino, 17 Maggio 2012 FP7 AAT Level 0 Roberto Bojeri Workshop ACARE Italia Torino, 17 Maggio 2012 1 Content: questions answered by this presentation Perché? 1. Scope of the Level 0 Cosa? 2. The new "instrument" Level 0 Dove?

More information

Fact Sheet IP specificities in research for the benefit of SMEs

Fact Sheet IP specificities in research for the benefit of SMEs European IPR Helpdesk Fact Sheet IP specificities in research for the benefit of SMEs June 2015 1 Introduction... 1 1. Actions for the benefit of SMEs... 2 1.1 Research for SMEs... 2 1.2 Research for SME-Associations...

More information

Final Project Report. Abstract. Document information

Final Project Report. Abstract. Document information Final Project Report Document information Project Title Safety Research Project Number 16.01.00 Project Manager EUROCONTROL Deliverable Name Final Project Report Deliverable ID D04.017 Edition 00.01.00

More information

Scoping Paper for. Horizon 2020 work programme Societal Challenge 4: Smart, Green and Integrated Transport

Scoping Paper for. Horizon 2020 work programme Societal Challenge 4: Smart, Green and Integrated Transport Scoping Paper for Horizon 2020 work programme 2018-2020 Societal Challenge 4: Smart, Green and Integrated Transport Important Notice: Working Document This scoping paper will guide the preparation of the

More information

Rethinking the role of Social Sciences and Humanities (SSH) in Horizon 2020: toward a reflective and generative perspective

Rethinking the role of Social Sciences and Humanities (SSH) in Horizon 2020: toward a reflective and generative perspective THE EU FRAMEWORK PROGRAMME FOR RESEARCH AND INNOVATION Horizon 2020 Societal Challenge 6: "Europe in a changing world : inclusive, innovative and reflective society" Rethinking the role of Social Sciences

More information

OSRA Overarching Strategic Research Agenda and CapTech SRAs Harmonisation. Connecting R&T and Capability Development

OSRA Overarching Strategic Research Agenda and CapTech SRAs Harmonisation. Connecting R&T and Capability Development O Overarching Strategic Research Agenda and s Harmonisation Connecting R&T and Capability Development The European Defence Agency (EDA) works to foster European defence cooperation to become more cost

More information

EU-PolarNet connecting science with society

EU-PolarNet connecting science with society EU-PolarNet connecting science with society Co-designed research proposals, involving stakeholders from the outset, will ensure that scientific outcomes are directly relevant and beneficial to society

More information

Committee on Development and Intellectual Property (CDIP)

Committee on Development and Intellectual Property (CDIP) E CDIP/10/13 ORIGINAL: ENGLISH DATE: OCTOBER 5, 2012 Committee on Development and Intellectual Property (CDIP) Tenth Session Geneva, November 12 to 16, 2012 DEVELOPING TOOLS FOR ACCESS TO PATENT INFORMATION

More information

Robotics in Horizon 2020 IMPACT and Technology Readiness Levels

Robotics in Horizon 2020 IMPACT and Technology Readiness Levels Robotics in Horizon 2020 IMPACT and Technology Readiness Levels Franco Mastroddi Unit A2 - Robotics DG Communication Networks, Content and Technology European Commission v3 H2020 Robotics Info Day Luxembourg

More information

H2020 Theme Oriented Training on ICT. H2020 Overview. Thies Wittig. Deputy Team Leader Project "Turkey in Horizon 2020"

H2020 Theme Oriented Training on ICT. H2020 Overview. Thies Wittig. Deputy Team Leader Project Turkey in Horizon 2020 TURKEY IN HORIZON 2020 ALTUN/HORIZ/TR2012/0740.14-2/SER/005 H2020 Theme Oriented Training on ICT H2020 Overview Thies Wittig Deputy Team Leader Project "Turkey in Horizon 2020" Dr. Thies Wi:g Ø PhD in

More information

Empirical Research on Systems Thinking and Practice in the Engineering Enterprise

Empirical Research on Systems Thinking and Practice in the Engineering Enterprise Empirical Research on Systems Thinking and Practice in the Engineering Enterprise Donna H. Rhodes Caroline T. Lamb Deborah J. Nightingale Massachusetts Institute of Technology April 2008 Topics Research

More information

FP9 s ambitious aims for societal impact call for a step change in interdisciplinarity and citizen engagement.

FP9 s ambitious aims for societal impact call for a step change in interdisciplinarity and citizen engagement. FP9 s ambitious aims for societal impact call for a step change in interdisciplinarity and citizen engagement. The European Alliance for SSH welcomes the invitation of the Commission to contribute to the

More information

Air Traffic Soft. Management. Ultimate System. Call Identifier : FP TREN-3 Thematic Priority 1.4 Aeronautics and Space

Air Traffic Soft. Management. Ultimate System. Call Identifier : FP TREN-3 Thematic Priority 1.4 Aeronautics and Space En Route Air Traffic Soft Management Ultimate System Call Identifier : FP6-2004-TREN-3 Thematic Priority 1.4 Aeronautics and Space EUROCONTROL Experimental Centre EUROCONTROL Innovative Research Workshop

More information

Working together to deliver on Europe 2020

Working together to deliver on Europe 2020 Lithuanian Position Paper on the Green Paper From Challenges to Opportunities: Towards a Common Strategic Framework for EU Research and Innovation Funding Lithuania considers Common Strategic Framework

More information

An introduction to the 7 th Framework Programme for Research and Technological Development. Gorgias Garofalakis

An introduction to the 7 th Framework Programme for Research and Technological Development. Gorgias Garofalakis An introduction to the 7 th Framework Programme for Research and Technological Development Gorgias Garofalakis Contents What & why Potential impact Scope Inputs Framework Programme Budget and duration

More information

COST FP9 Position Paper

COST FP9 Position Paper COST FP9 Position Paper 7 June 2017 COST 047/17 Key position points The next European Framework Programme for Research and Innovation should provide sufficient funding for open networks that are selected

More information

Extract of Advance copy of the Report of the International Conference on Chemicals Management on the work of its second session

Extract of Advance copy of the Report of the International Conference on Chemicals Management on the work of its second session Extract of Advance copy of the Report of the International Conference on Chemicals Management on the work of its second session Resolution II/4 on Emerging policy issues A Introduction Recognizing the

More information

PROJECT FACT SHEET GREEK-GERMANY CO-FUNDED PROJECT. project proposal to the funding measure

PROJECT FACT SHEET GREEK-GERMANY CO-FUNDED PROJECT. project proposal to the funding measure PROJECT FACT SHEET GREEK-GERMANY CO-FUNDED PROJECT project proposal to the funding measure Greek-German Bilateral Research and Innovation Cooperation Project acronym: SIT4Energy Smart IT for Energy Efficiency

More information

Roadmap for European Universities in Energy December 2016

Roadmap for European Universities in Energy December 2016 Roadmap for European Universities in Energy December 2016 1 Project partners This project has received funding from the European Union s Seventh Framework Programme for research, technological development

More information

WG/STAIR. Knut Blind, STAIR Chairman

WG/STAIR. Knut Blind, STAIR Chairman WG/STAIR Title: Source: The Operationalisation of the Integrated Approach: Submission of STAIR to the Consultation of the Green Paper From Challenges to Opportunities: Towards a Common Strategic Framework

More information

RFP No. 794/18/10/2017. Research Design and Implementation Requirements: Centres of Competence Research Project

RFP No. 794/18/10/2017. Research Design and Implementation Requirements: Centres of Competence Research Project RFP No. 794/18/10/2017 Research Design and Implementation Requirements: Centres of Competence Research Project 1 Table of Contents 1. BACKGROUND AND CONTEXT... 4 2. BACKGROUND TO THE DST CoC CONCEPT...

More information

GEAR 2030 WORKING GROUP 2 Roadmap on automated and connected vehicles

GEAR 2030 WORKING GROUP 2 Roadmap on automated and connected vehicles GEAR 2030 WORKING GROUP 2 Roadmap on automated and connected vehicles Europe has a very strong industrial basis on automotive technologies and systems. The sector provides jobs for 12 million people and

More information

VSNU December Broadening EU s horizons. Position paper FP9

VSNU December Broadening EU s horizons. Position paper FP9 VSNU December 2017 Broadening EU s horizons Position paper FP9 Introduction The European project was conceived to bring peace and prosperity to its citizens after two world wars. In the last decades, it

More information

Projects for Policy (P4P) Aviation Safety

Projects for Policy (P4P) Aviation Safety Projects for Policy (P4P) Aviation Safety Is EU aviation safety research funding well-spent? Is it focusing on the right issues? Austrian Aviation Technology Days Wiener Neustadt, 17 October 2018 Andreas

More information

Report on the Results of. Questionnaire 1

Report on the Results of. Questionnaire 1 Report on the Results of Questionnaire 1 (For Coordinators of the EU-U.S. Programmes, Initiatives, Thematic Task Forces, /Working Groups, and ERA-Nets) BILAT-USA G.A. n 244434 - Task 1.2 Deliverable 1.3

More information

II. The mandates, activities and outputs of the Technology Executive Committee

II. The mandates, activities and outputs of the Technology Executive Committee TEC/2018/16/13 Technology Executive Committee 27 February 2018 Sixteenth meeting Bonn, Germany, 13 16 March 2018 Monitoring and evaluation of the impacts of the implementation of the mandates of the Technology

More information

Information & Communication Technology Strategy

Information & Communication Technology Strategy Information & Communication Technology Strategy 2012-18 Information & Communication Technology (ICT) 2 Our Vision To provide a contemporary and integrated technological environment, which sustains and

More information

Engaging UK Climate Service Providers a series of workshops in November 2014

Engaging UK Climate Service Providers a series of workshops in November 2014 Engaging UK Climate Service Providers a series of workshops in November 2014 Belfast, London, Edinburgh and Cardiff Four workshops were held during November 2014 to engage organisations (providers, purveyors

More information

Tuning-CALOHEE Assessment Frameworks for the Subject Area of CIVIL ENGINEERING The Tuning-CALOHEE Assessment Frameworks for Civil Engineering offers

Tuning-CALOHEE Assessment Frameworks for the Subject Area of CIVIL ENGINEERING The Tuning-CALOHEE Assessment Frameworks for Civil Engineering offers Tuning-CALOHEE Assessment Frameworks for the Subject Area of CIVIL ENGINEERING The Tuning-CALOHEE Assessment Frameworks for Civil Engineering offers an important and novel tool for understanding, defining

More information

CLEAN SKY 2 Impact Assessment. Final Report of the Expert Group

CLEAN SKY 2 Impact Assessment. Final Report of the Expert Group CLEAN SKY 2 Impact Assessment Final Report of the Expert Group 29 September 2012 Authors: BERTOLINI, Enzo EURY, Serge HECKER, Peter (Rapporteur) HUGUET, Michel (Chairman) SANNA-RANDACCIO, Francesca CS2

More information

Research Infrastructures and Innovation

Research Infrastructures and Innovation Research Infrastructures and Innovation Octavi Quintana Principal Adviser European Commission DG Research & Innovation The presentation shall neither be binding nor construed as constituting commitment

More information

Background T

Background T Background» At the 2013 ISSC, the SAE International G-48 System Safety Committee accepted an action to investigate the utility of the Safety Case approach vis-à-vis ANSI/GEIA-STD- 0010-2009.» The Safety

More information

Communication and Dissemination in HORIZON 2020 European Commission Directorate-General for Research and Innovation

Communication and Dissemination in HORIZON 2020 European Commission Directorate-General for Research and Innovation Communication and Dissemination in HORIZON 2020 European Commission Directorate-General for Research and Innovation Energy Infoday Brussels, 23 October 2017 Communication Communicating H2020 projects Strategic

More information

Participatory backcasting: A tool for involving stakeholders in long term local development planning

Participatory backcasting: A tool for involving stakeholders in long term local development planning Erasmus Intensive Programme Equi Agry June 29 July 11, Foggia Participatory backcasting: A tool for involving stakeholders in long term local development planning Dr. Maurizio PROSPERI ( maurizio.prosperi@unifg.it

More information

Access to Research Infrastructures under Horizon 2020 and beyond

Access to Research Infrastructures under Horizon 2020 and beyond Access to Research Infrastructures under Horizon 2020 and beyond JEAN MOULIN A presentation based on slides provided by: the European Commission DG Research & Innovation Unit B4 Research Infrastructures

More information

Common evaluation criteria for evaluating proposals

Common evaluation criteria for evaluating proposals Common evaluation criteria for evaluating proposals Annex B A number of evaluation criteria are common to all the programmes of the Sixth Framework Programme and are set out in the European Parliament

More information

Towards a Consumer-Driven Energy System

Towards a Consumer-Driven Energy System IEA Committee on Energy Research and Technology EXPERTS GROUP ON R&D PRIORITY-SETTING AND EVALUATION Towards a Consumer-Driven Energy System Understanding Human Behaviour Workshop Summary 12-13 October

More information

Committee on Development and Intellectual Property (CDIP)

Committee on Development and Intellectual Property (CDIP) E CDIP/6/4 REV. ORIGINAL: ENGLISH DATE: NOVEMBER 26, 2010 Committee on Development and Intellectual Property (CDIP) Sixth Session Geneva, November 22 to 26, 2010 PROJECT ON INTELLECTUAL PROPERTY AND TECHNOLOGY

More information

CO-ORDINATION MECHANISMS FOR DIGITISATION POLICIES AND PROGRAMMES:

CO-ORDINATION MECHANISMS FOR DIGITISATION POLICIES AND PROGRAMMES: CO-ORDINATION MECHANISMS FOR DIGITISATION POLICIES AND PROGRAMMES: NATIONAL REPRESENTATIVES GROUP (NRG) SUMMARY REPORT AND CONCLUSIONS OF THE MEETING OF 10 DECEMBER 2002 The third meeting of the NRG was

More information

GROUP OF SENIOR OFFICIALS ON GLOBAL RESEARCH INFRASTRUCTURES

GROUP OF SENIOR OFFICIALS ON GLOBAL RESEARCH INFRASTRUCTURES GROUP OF SENIOR OFFICIALS ON GLOBAL RESEARCH INFRASTRUCTURES GSO Framework Presented to the G7 Science Ministers Meeting Turin, 27-28 September 2017 22 ACTIVITIES - GSO FRAMEWORK GSO FRAMEWORK T he GSO

More information

Association of European Space Research Establishments (ESRE): Recommendations related to. Framework Programme 9

Association of European Space Research Establishments (ESRE): Recommendations related to. Framework Programme 9 November 2017 Association of European Space Research Establishments (ESRE): Recommendations related to Framework Programme 9 I. Budget, Structure of FP 9, European Innovation Council Despite significant

More information

ASCOS - Aviation Safety and Certification of new Operations and Systems

ASCOS - Aviation Safety and Certification of new Operations and Systems NLR-TP-2016-628 November 2016 ASCOS - Aviation Safety and Certification of new Operations and Systems CUSTOMER: European Commission NLR Netherlands Aerospace Centre Netherlands Aerospace Centre NLR is

More information

ccess to Cultural Heritage Networks Across Europe

ccess to Cultural Heritage Networks Across Europe A INTERVIEW Italy Rossella Caffo Germany Monika Hagedorn -Saupe ccess to Cultural Heritage Networks Across Europe Interview with the ATHENA project coordinator - Rossella Caffo, Ministry of, Italy by Monika

More information

Digitisation Plan

Digitisation Plan Digitisation Plan 2016-2020 University of Sydney Library University of Sydney Library Digitisation Plan 2016-2020 Mission The University of Sydney Library Digitisation Plan 2016-20 sets out the aim and

More information

FET Open in Horizon Roumen Borissov Scientific/Technical Project Officer Future and Emerging Technologies, DG CONNECT European Commission

FET Open in Horizon Roumen Borissov Scientific/Technical Project Officer Future and Emerging Technologies, DG CONNECT European Commission FET Open in Horizon 2020 51214 Roumen Borissov Scientific/Technical Project Officer Future and Emerging Technologies, DG CONNECT European Commission FET Open in FP7 a portfolio snapshot Evolutionary microfluidix

More information

The BLUEMED Initiative: RESEARCH AND INNOVATION INITIATIVE FOR BLUE JOBS AND GROWTH IN THE MEDITERRANEAN

The BLUEMED Initiative: RESEARCH AND INNOVATION INITIATIVE FOR BLUE JOBS AND GROWTH IN THE MEDITERRANEAN Maritime Spatial Planning in the, an occasion for Blue Growth Rome, 27 th February 2017 The BLUEMED Initiative: RESEARCH AND INNOVATION INITIATIVE FOR BLUE JOBS AND GROWTH IN THE MEDITERRANEAN Enrico Brugnoli,

More information

THE FUTURE EUROPEAN INNOVATION COUNCIL A FULLY INTEGRATED APPROACH

THE FUTURE EUROPEAN INNOVATION COUNCIL A FULLY INTEGRATED APPROACH FRAUNHOFER-GESELLSCHAFT ZUR FÖRDERUNG DER ANGEWANDTEN FORSCHUNG E.V. THE FUTURE EUROPEAN INNOVATION COUNCIL A FULLY INTEGRATED APPROACH Brussels, 30/08/207 Contact Fraunhofer Department for the European

More information

Centralised Impact Assessment of EU Co-Funded Projects. Angus Hunter Annual Joint Programming Conference November 2016 (Session 2)

Centralised Impact Assessment of EU Co-Funded Projects. Angus Hunter Annual Joint Programming Conference November 2016 (Session 2) Centralised Impact Assessment of EU Co-Funded Projects Angus Hunter Annual Joint Programming Conference 2016 22 November 2016 (Session 2) By the middle of 2015, around 4500 transnational projects had been

More information

. Faye Goldman. July Contents

. Faye Goldman. July Contents July 2018 Contents Background... 2 Introduction... 2 A new strategy for 2018-21... 2 Project overview... 2 Project partners... 3 Digital Product Development... 4 What we re looking for... 4 Deliverables...

More information

Terms of Reference. Call for Experts in the field of Foresight and ICT

Terms of Reference. Call for Experts in the field of Foresight and ICT Terms of Reference Call for Experts in the field of Foresight and ICT Title Work package Lead: Related Workpackage: Related Task: Author(s): Project Number Instrument: Call for Experts in the field of

More information

Development of the Strategic Research Agenda of the Implementing Geological Disposal of Radioactive Waste Technology Platform

Development of the Strategic Research Agenda of the Implementing Geological Disposal of Radioactive Waste Technology Platform Development of the Strategic Research Agenda of the Implementing Geological Disposal of Radioactive Waste Technology Platform - 11020 P. Marjatta Palmu* and Gerald Ouzounian** * Posiva Oy, Research, Eurajoki,

More information

D8.1 PROJECT PRESENTATION

D8.1 PROJECT PRESENTATION D8.1 PROJECT PRESENTATION Approval Status AUTHOR(S) NAME AND SURNAME ROLE IN THE PROJECT PARTNER Daniela De Lucia, Gaetano Cascini PoliMI APPROVED BY Gaetano Cascini Project Coordinator PoliMI History

More information

Score grid for SBO projects with a societal finality version January 2018

Score grid for SBO projects with a societal finality version January 2018 Score grid for SBO projects with a societal finality version January 2018 Scientific dimension (S) Scientific dimension S S1.1 Scientific added value relative to the international state of the art and

More information

A Research and Innovation Agenda for a global Europe: Priorities and Opportunities for the 9 th Framework Programme

A Research and Innovation Agenda for a global Europe: Priorities and Opportunities for the 9 th Framework Programme A Research and Innovation Agenda for a global Europe: Priorities and Opportunities for the 9 th Framework Programme A Position Paper by the Young European Research Universities Network About YERUN The

More information

COMMISSION STAFF WORKING PAPER EXECUTIVE SUMMARY OF THE IMPACT ASSESSMENT. Accompanying the

COMMISSION STAFF WORKING PAPER EXECUTIVE SUMMARY OF THE IMPACT ASSESSMENT. Accompanying the EUROPEAN COMMISSION Brussels, 30.11.2011 SEC(2011) 1428 final Volume 1 COMMISSION STAFF WORKING PAPER EXECUTIVE SUMMARY OF THE IMPACT ASSESSMENT Accompanying the Communication from the Commission 'Horizon

More information

FET Flagships in Horizon 2020

FET Flagships in Horizon 2020 HORIZON 2020 - Future & Emerging Technologies (FET) Paris, 21 st December 2017 FET Flagships in Horizon 2020 Aymard de Touzalin Deputy Head of Unit, Flagships DG Connect, European Commission 1 Horizon

More information

Impact and Innovation in H2020 Proposals and projects

Impact and Innovation in H2020 Proposals and projects Impact and Innovation in H2020 Proposals and projects Dr. Eugene Sweeney Brussels 16th September 2014 Get your ticket to innovation. Roadmap What to look for in a good proposal Managing impact and innovation

More information

Brief to the. Senate Standing Committee on Social Affairs, Science and Technology. Dr. Eliot A. Phillipson President and CEO

Brief to the. Senate Standing Committee on Social Affairs, Science and Technology. Dr. Eliot A. Phillipson President and CEO Brief to the Senate Standing Committee on Social Affairs, Science and Technology Dr. Eliot A. Phillipson President and CEO June 14, 2010 Table of Contents Role of the Canada Foundation for Innovation (CFI)...1

More information

Belgian Position Paper

Belgian Position Paper The "INTERNATIONAL CO-OPERATION" COMMISSION and the "FEDERAL CO-OPERATION" COMMISSION of the Interministerial Conference of Science Policy of Belgium Belgian Position Paper Belgian position and recommendations

More information

November 18, 2011 MEASURES TO IMPROVE THE OPERATIONS OF THE CLIMATE INVESTMENT FUNDS

November 18, 2011 MEASURES TO IMPROVE THE OPERATIONS OF THE CLIMATE INVESTMENT FUNDS November 18, 2011 MEASURES TO IMPROVE THE OPERATIONS OF THE CLIMATE INVESTMENT FUNDS Note: At the joint meeting of the CTF and SCF Trust Fund Committees held on November 3, 2011, the meeting reviewed the

More information

9 Vaccine SMEs' Needs

9 Vaccine SMEs' Needs 9 Vaccine SMEs' Needs As in most innovative sectors, SMEs play a critical role in bridging basic discoveries from academic research to clinical development 36. This could be seen over the last decade with

More information

EGS-CC. System Engineering Team. Commonality of Ground Systems. Executive Summary

EGS-CC. System Engineering Team. Commonality of Ground Systems. Executive Summary System Engineering Team Prepared: System Engineering Team Date: Approved: System Engineering Team Leader Date: Authorized: Steering Board Date: Restriction of Disclosure: The copyright of this document

More information

SCAR response to the 2 nd Foresight Expert Group Report

SCAR response to the 2 nd Foresight Expert Group Report SCAR response to the 2 nd Foresight Expert Group Report Teagasc 2030 One Year On Follow-up Workshop Dublin, 2 October 2009 François CONSTANTIN Scientific Officer DG RTD-E4 European Commission francois.constantin@ec.europa.eu

More information

Final Project Report. Abstract. Document information

Final Project Report. Abstract. Document information Final Project Report Document information Project Title Future Satellite Communication System Project Number 15.02.06 Project Manager THALES (TAS-I) Deliverable Name Final Project Report Deliverable ID

More information

Technology evaluation for longer-term coordination and prioritisation of research. - Noise - D. Collin Safran

Technology evaluation for longer-term coordination and prioritisation of research. - Noise - D. Collin Safran X-NOISE Technology evaluation for longer-term coordination and prioritisation of research - Noise - D. Collin Safran Technology evaluation for longer-term coordination and prioritisation of research -

More information

TECHNOLOGY WITH A HUMAN TOUCH

TECHNOLOGY WITH A HUMAN TOUCH TECHNOLOGY WITH A HUMAN TOUCH 2 SILVER TECHNOLOGY WITH A HUMAN TOUCH TECHNOLOGY WITH A HUMAN TOUCH The SILVER project searches for new technologies to assist the elderly people in their everyday lives.

More information

Pan-Canadian Trust Framework Overview

Pan-Canadian Trust Framework Overview Pan-Canadian Trust Framework Overview A collaborative approach to developing a Pan- Canadian Trust Framework Authors: DIACC Trust Framework Expert Committee August 2016 Abstract: The purpose of this document

More information

Exploring emerging ICT-enabled governance models in European cities

Exploring emerging ICT-enabled governance models in European cities Exploring emerging ICT-enabled governance models in European cities EXPGOV Project Research Plan D.1 - FINAL (V.2.0, 27.01.2009) This document has been drafted by Gianluca Misuraca, Scientific Officer

More information

European Commission. 6 th Framework Programme Anticipating scientific and technological needs NEST. New and Emerging Science and Technology

European Commission. 6 th Framework Programme Anticipating scientific and technological needs NEST. New and Emerging Science and Technology European Commission 6 th Framework Programme Anticipating scientific and technological needs NEST New and Emerging Science and Technology REFERENCE DOCUMENT ON Synthetic Biology 2004/5-NEST-PATHFINDER

More information

Examples of Public Procurement of R&D services within EU funded Security Research actions

Examples of Public Procurement of R&D services within EU funded Security Research actions Examples of Public Procurement of R&D services within EU funded Security Research actions Paolo Salieri 18 / 10 / 2017 paolo.salieri@ec.europa.eu PCP to steer the development of solutions towards concrete

More information

Outline. IPTS and the Information Society Unit IPTS Research Agenda on ICT for Governance

Outline. IPTS and the Information Society Unit IPTS Research Agenda on ICT for Governance EUROCITIES Knowledge Society Forum, Lille, 28-30 October 2009 IPTS Exploratory Research on ICT-enabled governance models in EU cities Gianluca Misuraca IPTS Information Society Unit EUROCITIES Knowledge

More information

Position Paper of Iberian universities. The mid-term review of Horizon 2020 and the design of FP9

Position Paper of Iberian universities. The mid-term review of Horizon 2020 and the design of FP9 Position Paper of Iberian universities The mid-term review of Horizon 2020 and the design of FP9 Introduction Horizon 2020 (H2020), the Framework Programme for research and innovation of the European Union,

More information

ESA Iris Programme Analysis & definition of the Satellite System Operations. Briefing 28 July

ESA Iris Programme Analysis & definition of the Satellite System Operations. Briefing 28 July ESA Iris Programme Analysis & definition of the Satellite System Operations Briefing 28 July 2009 - Nathalie.Ricard@esa.int 1 Analysis & Definition of Satellite Operations Study rationale ESA s involvement

More information

I. Introduction. Cover note. A. Mandate. B. Scope of the note. Technology Executive Committee. Fifteenth meeting. Bonn, Germany, September 2017

I. Introduction. Cover note. A. Mandate. B. Scope of the note. Technology Executive Committee. Fifteenth meeting. Bonn, Germany, September 2017 Technology Executive Committee 31 August 2017 Fifteenth meeting Bonn, Germany, 12 15 September 2017 Draft TEC and CTCN inputs to the forty-seventh session of the Subsidiary Body for Scientific and Technological

More information

Aviation Research Infrastructures networking

Aviation Research Infrastructures networking Aviation Research Infrastructures networking Mario Farioli EREA Executive Secretariat member (CIRA Delegate) Outline A general overview of the Association EREA Capabilities and Research Infrastructures

More information

Arshad Mansoor, Sr. Vice President, Research & Development INNOVATION SCOUTS: EXPANDING EPRI S TECHNOLOGY INNOVATION NETWORK

Arshad Mansoor, Sr. Vice President, Research & Development INNOVATION SCOUTS: EXPANDING EPRI S TECHNOLOGY INNOVATION NETWORK RAC Briefing 2011-1 TO: FROM: SUBJECT: Research Advisory Committee Arshad Mansoor, Sr. Vice President, Research & Development INNOVATION SCOUTS: EXPANDING EPRI S TECHNOLOGY INNOVATION NETWORK Research

More information

MINERVA: IMPROVING THE PRODUCTION OF DIGITAL CULTURAL HERITAGE IN EUROPE. Rossella Caffo - Ministero per i Beni e le Attività Culturali, Italia

MINERVA: IMPROVING THE PRODUCTION OF DIGITAL CULTURAL HERITAGE IN EUROPE. Rossella Caffo - Ministero per i Beni e le Attività Culturali, Italia MINERVA: IMPROVING THE PRODUCTION OF DIGITAL CULTURAL HERITAGE IN EUROPE. Rossella Caffo - Ministero per i Beni e le Attività Culturali, Italia Abstract The MINERVA project is a network of the ministries

More information

Climate Change Innovation and Technology Framework 2017

Climate Change Innovation and Technology Framework 2017 Climate Change Innovation and Technology Framework 2017 Advancing Alberta s environmental performance and diversification through investments in innovation and technology Table of Contents 2 Message from

More information

A Harmonised Regulatory Framework for Supporting Single European Electronic Market: Achievements and Perspectives

A Harmonised Regulatory Framework for Supporting Single European Electronic Market: Achievements and Perspectives A Harmonised Regulatory Framework for Supporting Single European Electronic Market: Achievements and Perspectives Irina NEAGA, Tarek HASSAN, Chris CARTER Loughborough University, Loughborough, Leicestershire,

More information

TERMS OF REFERENCE FOR CONSULTANTS

TERMS OF REFERENCE FOR CONSULTANTS Strengthening Systems for Promoting Science, Technology, and Innovation (KSTA MON 51123) TERMS OF REFERENCE FOR CONSULTANTS 1. The Asian Development Bank (ADB) will engage 77 person-months of consulting

More information

The main FP7 instruments. Aurélien Saffroy. 6 Dec

The main FP7 instruments. Aurélien Saffroy. 6 Dec The main FP7 instruments Aurélien Saffroy 6 Dec. 2006 www.euroquality.fr 1 Summary STRUCTURE OF THE 7 th Framework Programme STRUCTURE OF THE 7 th Framework Programme 2 The main instruments of FP7 Capacities;

More information

FP7-INFRASTRUCTURES

FP7-INFRASTRUCTURES FP7 Research Infrastructures Call for proposals FP7-INFRASTRUCTURES-2012-1 European Commission, DG Research, Unit B.3 FP7 Capacities Overall information Definition of Research Infrastructures The Research

More information

Score grid for SBO projects with an economic finality version January 2019

Score grid for SBO projects with an economic finality version January 2019 Score grid for SBO projects with an economic finality version January 2019 Scientific dimension (S) Scientific dimension S S1.1 Scientific added value relative to the international state of the art and

More information

HELPING BIOECONOMY RESEARCH PROJECTS RAISE THEIR GAME

HELPING BIOECONOMY RESEARCH PROJECTS RAISE THEIR GAME HELPING BIOECONOMY RESEARCH PROJECTS RAISE THEIR GAME An early glimpse into the lessons learnt from ProBIO 1 FOREWORD The fascinating experience of ProBIO This brochure comes as the ProBIO project is reaching

More information

HORIZON 2020 BLUE GROWTH

HORIZON 2020 BLUE GROWTH HORIZON 2020 BLUE GROWTH in Horizon 2020 Info-Day, Paris 24th January 2014 2014-2020 Christos Fragakis Deputy Head of Unit Management of natural resources DG Research & Why a Blue Growth Focus Area in

More information

Mainstreaming PE in Horizon 2020: perspectives and ambitions

Mainstreaming PE in Horizon 2020: perspectives and ambitions CASI/PE2020 Conference Brussels, 16-17 November 2016 Mainstreaming PE in Horizon 2020: perspectives and ambitions Giuseppe BORSALINO European Commission DG RTD B7.002 'Mainstreaming RRI in Horizon 2020

More information