Vojko BRATINA & Massimo CISCATO B1 - Space Research Unit, REA Space technologies, science and exploration SPACE-20-SCI-2018: Scientific instrumentation and technologies enabling space science and exploration
Purpose of this briefing To ensure a common understanding of the work programme's key elements in order to have a single and harmonised approach to the evaluation To review the evaluation criteria and provide guidelines, including what should be addressed under which sub-criterion
General info During the briefing and remote evaluation, all evaluators must remain anonymous in order to avoid any bias Evaluators are invited to ask questions by email (REA-SPACE-WEB- STREAMING@ec.europa.eu) Please specify the topic in the email subject: SCI20. We will reply to these questions either during the briefing or afterwards by email. An independent observer is attending the topic briefing to monitor the process The slides and the FAQs will also be available on the Space 2018 Evaluation website at https://ec.europa.eu/info/h2020-space-2018_en You have already received by email the IER Guidelines document that summarises the information on the evaluation criteria presented here
SCI-20 topic overview
SCI-20: Scientific instrumentation and technologies enabling space science and exploration Topic: SCI-20: Scientific instrumentation and technologies enabling space science and exploration What do we want to fund? - Scientific instrumentation and technologies enabling space science and exploration including planetary exploration, where advances are expected in support to on-site activities such as» Planetary landing» Planetary navigation» Sample collection and processing» In-situ analysis missions synergies between space and ground-based observations development of new and innovative approaches, such as» Cubesats and other small space platforms, including planetary entry probe» use of Commercial off-the-shelf (COTS) components
SCI-20: Scientific instrumentation and technologies enabling space science and exploration Funding Instrument: Research and innovation action (RIA) Description: Action primarily consisting of activities aiming to establish new knowledge and/or to explore the feasibility of a new or improved technology, product, process, service or solution. For this purpose they may include basic and applied research, technology development and integration, testing and validation on a small-scale prototype in a laboratory or simulated environment. Projects may contain closely connected but limited demonstration or pilot activities aiming to show technical feasibility in a near to operational environment.projects may include limited research and development activities. Funding rate: 100%
SCI-20: Scientific instrumentation and technologies enabling space science and exploration How much? Total budget: 10 million EUR Estimated amount per proposal: 2-3 million EUR Funding rate: 100% 32 proposals to evaluate Expected duration? The duration of the project is a decision of the applicants. Typically 2-3 years.
Individual Evaluation Report (IER)
Award criteria, scores and weighting AWARD CRITERIA Excellence Impact Quality and efficiency of the implementation Scoring: from 0 to 5*, weakness / shortcoming 0 The proposal fails to address the criterion or cannot be assessed due to missing or incomplete information. 1 Poor. The criterion is inadequately addressed, or there are serious inherent weaknesses. 2 Fair. The proposal broadly addresses the criterion, but there are significant weaknesses. 3 Good. The proposal addresses the criterion well, but a number of shortcomings are present. 4 Very Good. The proposal addresses the criterion very well, but a small number of shortcomings are present. 5 Excellent. The proposal successfully addresses all relevant aspects of the criterion. Any shortcomings are minor. *You can award 0.5 points Thresholds & weighting Threshold for individual criteria = 3 + Overall threshold, applying to the sum of the three individual scores, =10.
Criterion 1 - Excellence
Excellence Clarity and pertinence of the objectives How does the proposal address the specific challenge and scope of the work programme topic? In particular, the proposal is expected to develop Scientific instrumentation and technologies enabling space science and exploration including planetary exploration, where advances are expected in support to on-site activities such as Planetary landing Planetary navigation Sample collection and processing In-situ analysis missions. Are the objectives clear, measurable, realistic and achievable within project duration?
Excellence Soundness of the concept, and credibility of the proposed methodology Is the concept of the proposal sound (i.e. based on models or assumptions which are valid and convincing)? Is the proposal stimulating synergies between space and ground-based observations combining and reusing different technologies, techniques and methodologies? Does the activities target primarily European (even at national level) and European-led space science and exploration missions or internationally-led missions where the participation of European partners provides demonstrated added-value in terms of technological development and scientific output? Does the proposal show that the proposed methodology is appropriate and effective for achieving the stated objectives?
Excellence Extent that proposed work is beyond the state of the art, and demonstrates innovation potential (e.g. groundbreaking objectives, novel concepts and approaches, new products, services or business and organizational models) Is the state of the art well researched and presented and to what extent does the proposed work go beyond the state of the art in the proposed products and/or services? Does the proposal 'assess the current European contribution to the advances in the knowledge of the Universe and Space Science provided by implemented world-class space missions and ground-based infrastructure, in the context of national, ESA and ESO scientific programmes? Is the proposed work ambitious enough to be innovative, without being so ambitious that it is not feasible? Which are there ground-breaking objectives, novel concepts and approaches proposed and to what extent are they innovative? In the work programme, the development of new and innovative approaches, such as Cubesats and other small space platforms, including planetary entry probe, use of Commercial off-the-shelf (COTS) components is encouraged as long as it leads or contributes to the implementation of space science and exploration with significant scientific outputs."
Excellence Appropriate consideration of interdisciplinary approaches and, where relevant, use of stakeholder knowledge Are inter-disciplinary approaches (i.e. the integration of data, techniques, tools or concepts from two or more disciplines e.g. natural sciences, technology, engineering, economics, social sciences) considered in the proposed concept? Stakeholder knowledge: Who are the stakeholders (e.g. potential users or customers or other groups or organizations with an interest in the results of the proposed project)? How does the proposal demonstrate that the consortium will get relevant knowledgefromthem(e.g. organizingworkshops with users, discussing with customers, participating in relevant working groups etc.)? How will this knowledge be used? Stakeholders not to be confused with the partners of the consortium!
Criterion 2 - Impact
Impact The extent to which the outputs of the project would contribute to each of the expected impacts mentioned in the work programme under the relevant topic How much would the proposal contribute to each of the expected impacts mentioned in the work programme? (Here you should base your evaluation on the assumption that the proposed concept and methodology would work in order to avoid penalising the proposal twice for the same shortcomings.) Increased collaboration of scientific, engineering and industrial teams both within and outside Europe across different domains; Enable breakthroughs in terms of the capacity to exploit scientific space data and in terms of developing scientific instrumentation and technologies used in space science and exploration missions. Validation of novel human spaceflight and robotic space instrumentation and technologies through analogue tests. Does the proposal identify any relevant barriers/obstacles and any framework conditions (such as regulations, standards, public acceptance, financing of follow-up steps) that may affect the extent to which the expected impacts will be achieved?
IMPACT Any substantial impacts not mentioned in the work programme, that would enhance innovation capacity, create new market opportunities, strengthen competitiveness and growth of companies, address issues related to climate change or the environment, or bring other important benefits for society Are there any other substantial impacts not mentioned in the work programme which are described in the proposal and are relevant to the issues listed above? (The proposal should not be penalised if it does not have additional impact.)
Impact Quality of the proposed measures to: - exploit and disseminate project results (including management of IPR), and to manage research data where relevant - communicate the project activities to different target audiences Please assess the quality of the "draft Plan for the Exploitation and Dissemination of the project's Results (PEDR)" with respect to the specific measures to be implemented both during and after the end of the project. Is the plan to disseminate project results to audiences who might use the results in their own work (e.g. Presentations in conferences, publications in peer review journals etc.) appropriate? Are activities to communicate project activities to multiple audiences beyond the project's own community (media, public etc.) included? How are these tailored to the needs of various audiences? If research data is generated, does the proposal identify the types of data that will be generated, the standards that will be used, how the data will be exploited and/or shared/made accessible for verification and re-use? Is the strategy for the management of IPR appropriate? Does it describe the main background IPR needed to carry out the project, its ownership and accessibility, authorisation to use Third Party rights, and the ownership principles that will govern the project results?
Criterion 3 -Quality and efficiency of the implementation
Quality and efficiency of the implementation Quality and effectiveness of the work plan, including extent to which the resources assigned to work packages are in line with their objectives and deliverables How good is the overall structure of the work plan? Have the different work packages been scheduled and linked appropriately? Are their inter-dependencies clear? Are milestones and deliverables well defined and appropriately timed? Are the work packages and associated tasks sufficiently described to provide a good understanding of the work involved and justify the proposed resources to be allocated?
Quality and efficiency of the implementation Appropriateness of the management structures and procedures, including risk and innovation management Are the described organisational structure and decision making mechanisms appropriate to the scale and complexity of the project? Are the management procedures, including quality management and conflict resolution, appropriate to the scale and complexity of the project? Are critical internal and external risks related to project implementation described with appropriate mitigation measures and severity/probability estimates? has innovation management been appropriately addressed in the management structure and work plan (I.e. does the proposal show how any scientific or technological inventions or advances will actually be turned into relevant concrete innovations with specific practical or commercial advantages over existing solutions)?
Quality and efficiency of the implementation Complementarity of the participants and extent to which the consortium as a whole brings together the necessary expertise How is the complementarity of the participants described? Does the consortium bring together the necessary expertise for the project?
Quality and efficiency of the implementation Appropriateness of the allocation of tasks, ensuring that all participants have a valid role and adequate resources in the project to fulfil that role Does each partner have a valid role and adequate resources (both effort and budget) to fulfil that role? If "Other Direct Costs" exceed 15% of personnel costs: have they been explained adequately? Are these costs "reasonable, justified and comply with the principle of sound financial management, in particular regarding economy and efficiency"(art. 6.1.)? Please particular attention should be given for proposal with very high Other Direct Costs. If any of the work will be subcontracted to third parties, has this been justified convincingly?
Quality of the reports
Quality of the reports If this was my proposal, would I find this report fair, accurate, clear and complete? even if it brings bad news
Quality of the reports Avoid: Comments not related to the criterion in question. Double counting of same weakness under different criteria. Contradicting statements relative to strengths and weaknesses. Comments too short / too long / inappropriate. Categorical statements not properly verified and quoting text e.g. The proposal doesn t mention user requirements when there is a short reference Scores that don t match the comments. Making recommendations. Evaluating on potential, proposals need to be evaluated as they are. Discrimination/political incorrectness.
Quality of the reports Good vs. poor comments As Evaluator you have to ensure that Comments in IERs and CRs are: Specific to the relevant criterion Clear and substantial Facts, not opinions: "We think that " "This proposal is " Consistent with the score, balancing strengths and weaknesses Of adequate length: not just one sentence, not a booklet! Comments judge the proposal, they do not summarise it. Do not provide advice on improving the proposal.
Quality of the reports Poor comments merely echo the score The innovative aspects of the research programme are poor. Good comments explain it This proposal is not innovative in X or Y, and it does not take Z into account. Poor comments are ambiguous The resources for the project are unrealistic Good comments are clear The resources in WP 4 and 6 are seriously underestimated given the complexity of the activity proposed.
Quality of the reports Poor comments are vague, subject to interpretation Good comments are precise and final We think the management plan is probably inadequate given the duration of the project and the number of partners. The management plan is inadequate. It does not include clear risk assessment; it doesn't foresee a problemsolving mechanism in the event of disputes between partners.
Quality of the reports Poor comments are inaccurate and provide an opening for a complaint There is no discussion of a dissemination strategy. There is only one industrial partner in the consortium. The coordinator is not adequately experienced." Good comments close the question Dissemination activities are listed but the proposal doesn't have a clear dissemination strategy. The consortium doesn't include a sufficient industrial participation. The coordinator does not demonstrate in the proposal an adequate level of experience in this field.
Quality of the reports Poor comments include words like Good comments include words like Perhaps Think Seems Assume Probably Because Percent Specifically For example
Quality of the reports Useful vocabulary Insufficient, minimal, fails to describe, unacceptable, inadequate, very generic, not evident, unfocused, very weak, bad, does not meet the requirements, inappropriate, limited, unclear, not sound enough, not specified, no significant impact, unjustified, overestimated Extremely relevant, credible, very clear, precisely specified, realistic, very innovative, extremely well suited, timely, convincing, comprehensive, high quality, justified, very well identified, strong, highly effective, thoughtful, very promising, evidence, wellformulated, carefully prepared, very professionally prepared, fully in line, very profound, sound, very convincingly integrated, clearly articulated, coherent, well balanced, very plausible, ambitious, clear advances, well above average
FAQ
Frequently Asked Questions SPACE-20-SCI-2018 Q: Under call H2020-SPACE-2018, topic SPACE-20-SCI-2018, how to define 'early missions' in this topic? A: The term 'early missions' indicates missions at their early definition stages, such as conceptual and feasibility phases (Phase 0/A). The target should be European-led mission: ESA, national, international, or where there is a significant European participation. Published on: 27-02-2018 11:05 AM
Frequently Asked Questions SPACE-20-SCI-2018 Q: Under call H2020-SPACE-2018-2020, topic SPACE-20-SCI-2018, are astronomy missions coherent with the scope and objectives of the call topic? A: SPACE-20-SCI-2018 is a topic broad in scope and space exploration as well as science missions (including astronomy) are within the scope of this topic. Individual proposals are not obliged to cover the full scope of a topic and therefore not to fully address all of the expected impacts listed in the work programme. It is up to the proposers to clearly explain the relation of their proposal to the objectives of the topic and to argue the relevance. It is then the job of the independent evaluators, in line with the evaluation process and criteria set out in the General Annexes of the Work Programme, to assess how well the proposal is in line with the objectives of the call topic. Published on: 02-03-2018 10:40 AM
Final reminders
Documents to read http://ec.europa.eu/research/participants/portal/deskto p/en/opportunities/h2020/topics/space-20-sci-2018.html work programme. Work programme topic description Topic conditions and documents Point 8: Additional documents Frequently Asked Questions for topic SPACE-20- SCI-2018
Process reminders Evaluators: accept the proposals assigned in SEP (by EOB Monday 23.4.) - In case of non-acceptance provide a justification (e.g. CoI) Finalise and submit one individual evaluation report by Thursday 26.4. - So that quality controllers can check the completeness of your report and give you feedback on whether issues were commented under appropriate subcriteria - You will receive comments in your IER reports in SEP - In some cases your report may be re-opened by REA staff so that you may complete it Complete the Remote Evaluation by Friday May 11th - Necessary to allow the rapporteur to prepare a draft consensus report (CR) prior to the consensus meeting The draft CR will be available in the system a few days before the consensus meeting - Read the draft CR prior to the meeting to prepare for the consensus discussion
Thank you! In case of questions, do not hesitate to contact the secretariat with tag SCI-20: REA SPACE CALLS <REA-SPACE-CALLS@ec.europa.eu> (for technical issues: IT Helpdesk) EC HELPDESK<EC-HELPDESK-IT@ec.europa.eu> Disclaimer: This presentation is intended for information purposes only within the framework of the evaluation of the call H2020-SPACE-2018. Please refrain from sharing it with other parties not involved in this process or from making it publicly available.