H2020 Programme 2018-2020 For a better innovation support to SMEs Innosup-06-2018-2020 Supporting Experimentation in Innovation Agencies Background Note to the Call Topic Version 1.0 17 January 2018
1. Context and Concepts Background Note to the Call Topic "INNOSUP-06-2018: Supporting Experimentation in Innovation Agencies" This document provides background information to potential applicants for the Coordination and Support Action under the call "Supporting Experimentation in Innovation Agencies", as announced in the Horizon 2020 Work Programme 2018-2020 for "Innovation in SMEs". This action will provide grants to national and regional innovation agencies that wish to engage in policy experimentation to design and test new or significantly improved schemes in support of innovation in SMEs by using randomised control trials (RCTs). RCTs are still not widely used in the field of innovation and business support, so there are relatively few examples of best practice, with few innovation agencies or researchers in the field of innovation and business support having expertise and experience in using the RCTs in design and evaluation of the impact of their support programmes. In this context, this background note aims at providing further information on RCTs. Moreover, an event or webinar will be organised by the European Commission for the applicants invited to submit stage 2 proposals. Additional support and advice will be provided by a contractor selected by the European Commission as beneficiaries may lack capacities and expertise to successfully deliver their projects. Therefore, beneficiaries of the actions will be assisted from the onset of their projects by a contractor in the design and running of RCTs. 1 2. RCTs and evidence-based policymaking Whether adopting a new policy intervention is better than the status quo depends on a number of different factors: would stakeholders (for example SMEs) react to a given intervention as we expect? Will there be unforeseen consequences? Will there be any rebound effect or reduction of the expected gains over time? Evidence-based policymaking seeks reliable answers that help deciding on the suitability of a specific policy intervention. It is based on scientific proof and possibly on factual data, rather than on selfreported perceptions collected through surveys, as opposed as on assumptions or expert opinions. Evidence-based policymaking relies on special methods such as RCTs, ensuring the highest quality of evidence. 3. What are randomised control trials (RCTs)? RCTs are the best way to find out whether a given policy intervention works. This approach has for a long time been the gold standard in medicine and biology. It has recently become increasingly used in policy making as well. The basis of the RCT is that individuals or firms are randomly assigned to two or more groups. The units in the treatment group receive the policy intervention while the units 1 Please refer to the Horizon 2020 Work Programme "Innovation in SMEs"2018-2020, other action 7. Support to design and running of randomized control trials under INNOSUO-06-2017 1
in the control group receive a placebo or no treatment at all. RCTs provide a scientific proof because the control group represents the benchmark for evaluating the effectiveness of the intervention (see Figure 1). Having a control group including only SMEs that don t receive the policy intervention to be tested and a treatment group including only SMEs that receive the policy intervention allows for the identification of the net effect; that is the impact of the policy intervention, net of any additional factor. The control group serves as a counterfactual as far as all other characteristics of the SMEs are similar in the two groups. 4. Importance of randomisation Randomisation is key in setting-up of a robust RCT. If randomisation is done by using a robust randomisation methodology then a causal relationship between the treatment (the policy intervention) and the net effect observed can be established. With reference to Figure 1, three times as many units in the treatment group are affected by the intervention, compared to the control group. However, random assignment should not be confused with random sampling. Random sampling refers to the way a sample is drawn from the SME population. Random assignment refers to how SMEs are assigned to either a treatment or a control group. RCTs typically use both random sampling and random assignment. Randomisation ensures that characteristics that might have an impact on the relationship between the intervention and outcome will roughly be equal in both the treatment and control group. In doing so potential bias will be minimised. That means that the major characteristics of the SMEs in the control group and the treatment group should be comparable (for example in terms of size of SMEs, age of SMEs, experience in innovation, degree of internationalisation). 2
5. Essential steps for a robust RCT Setting up a sound RCT is not overly complex, and comes down to a few essential steps: 1. Identify one or more policy interventions to be evaluated: a. You may compare the new one against the status quo; b. You may compare two different versions of a new policy. 2. Define the expected outcome of the intervention and how this will be measured in the trial. 3. Determine the randomisation unit that is who or what is going to be randomised. It is usually individual SMEs but could also be groups of SMEs (for example in a sector, in a geographical area). 4. Determine how many SMEs are required for robust results. To draw conclusions, a sufficient sample size is required. 5. Randomly attribute the SMEs to the control or treatment group(s). As stated above, a robust randomisation method is essential because it makes RCTs superior to other types of policy evaluation by making the control and treatment group(s) equivalent with respect to all key factors such as size of SMEs, age of SMEs, experience in innovation, degree of internationalisation. That means that the control and treatment group(s) should ideally have SMEs with the same set of characteristics. 6. Collect the baseline data from both the SMEs in the treatment group and the ones in the control group. 7. Deploy the policy intervention to the SMEs in the treatment group(s). 8. Measure the impact and compare the results. The timing and the method of measurement should be decided before randomisation is done. 9. Should the new policy intervention work, consider extending it to all SMEs. If the results show that the policy intervention is ineffective, discontinuation or revision should be considered. 6. Preliminary review before designing an RCT As RCTs become more numerous, information about on-going, complete (or withdrawn) trials starts becoming available in public registries. The American Economic Association Registry for RCTs is possibly the most well-known of such registries. This may be a source of precious information and/or inspiration for new RCTs. You are most encouraged to browse their data base: https://www.socialscienceregistry.org. Acknowledgment: Foresight, Behavioural Insights and Design for Policy Unit, Joint Research Centre (European Commission) 3
7. Further literature Ambroz, Angela, and Marc Shotland, Randomized Control Trial (RCT), web page, BetterEvaluation, 2013. http://betterevaluation.org/plan/approach/rct. Behavioural Insights Team, Cabinet Office (UK) (2012), Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials, June 2012, www.gov.uk/government/uploads/system/uploads/attachment_data/file/62529/tla- 1906126.pdf. Bloom, Howard, The Core Analytics of Randomized Experiments for Social Research, MDRC Working Papers on Research Methodology, MDRC, New York, 2006. http://www.mdrc.org/sites/default/files/full_533.pdf. Duflo, Esther, et al., Using Randomization in Development Economics Research: A Toolkit, Department of Economics, Massachusetts Institute of Technology and Abdul Latif Jameel Poverty Action Lab, Cambridge, 2006. http://www.povertyactionlab.org/sites/default/files/documents/using%20randomization%20in% 20Development%20Economics.pdf. Gertler, Paul J., et al., Impact Evaluation in Practice, World Bank, Washington, DC, 2010. http://siteresources.worldbank.org/exthdoffice/resources/5485726-1295455628620/impact_evaluation_in_practice.pdf. Glennerster, Rachel, and Kudzai Takavarasha, Running Randomized Evaluations: A Practical Guide, Princeton University Press, Princeton, 2013. OECD (2017) Use of behavioural insights in consumer policy Directorate for Science, Technology and Innovation Committee on Consumer Policy February 2017 OECD/The European Commission (2014): The Missing Entrepreneurs: Policies for Inclusive Entrepreneurship in Europe, OECD Publishing. http://dx.doi.org/10.1787/9789264213593-en White, H., Sabarwal S. & T. de Hoop, (2014). Randomized Controlled Trials (RCTs), Methodological Briefs: Impact Evaluation 7, UNICEF Office of Research, Florence. 4