Findings from the ESRC s Impact Evaluation Programme Faye Auty, 21 st June 2011
ESRC Strategic Plan 2009-14- Extracts The ESRC expects that all the research it funds will be high quality and of scholarly distinction, but we are also committed to increasing its non academic impact, and benefit to the UK in public policy, economic prosperity, culture, and quality of life These include the close engagement with potential research users before, during and after the research process, and a flow of people between research and the worlds of policy and practice.
Evaluation Committee s Role Responsible for advising the ESRC on the successful achievement of its corporate strategy through a combination of policy, large investment, and project-level evaluations. Also leads the ESRC's work on developing methods for the evaluation of impact, and oversees its programme of International Benchmarking Reviews. Operates independently of the Policy Committees and provides Council with a comprehensive assessment of funded research as well as undertaking reviews in key policy areas; the evaluation results are incorporated into the ESRC's strategic planning and development. Committed to exploring new methods for assessing the impact of the research the Council funds on policymakers and practitioners. This helps demonstrate the broader contribution of research to society and the economy.
Project Level Impact Assessment At application stage, applicants asked to complete a Pathways to Impact attachment setting out how they will maximise the impact of their work Award holders are able to complete Impact Records during or after the lifetime of the project providing a simple, succinct way of recording any impacts that arise from their work Award holders are required to submit an Impact Report 12 months after their award end date Evaluated on basis of scientific quality and evidence of impact on policy and practice
ESRC Impact Toolkit
Evaluation Committee- Impact Evaluation Aims To identify and analyse evidence of research impact on policy and practice To understand how impact is generated, and help the ESRC to improve its performance in this area To develop impact evaluation methods
Impact Categories The impact of social science research can be categorised as follows: instrumental (for example, influencing the development of policy, practice or service provision, shaping legislation, altering behaviour) conceptual (for example, contributing to the understanding of these and related issues, reframing debates). capacity building (for example, through technical/personal skill development) (Nutley et al, 2007)
Overview of ESRC Impact Evaluation Programme
Case Studies Future of Work Programme (RAND Europe and HERG) Centre for Business Research (SPRU) Psychology projects (Technology Development Group) Politics projects (LSE Public Policy Group) Innovation Centres (Pricewaterhouse Coopers Ltd) Centre for Social & Economic Research on the Global Environment (CAG Consultants) People at the Centre of Information & Communication Technologies (Technology Development Group) Teaching and Learning Research Programme (HOST Policy Research)
Evidence of Impact from 2010 Programme (i) Significant evidence of impact of a limited number of researchers in the development of UK Child Poverty Policy. The previous Labour government enhanced Sure Start Children s Centres in the most disadvantaged parts of England and funded more outreach workers and training as a result of research that utilised MCS data showing the benefits to young people
Evidence of Impact (ii) The Welsh Government 2010 Child Poverty Strategy was influenced by internal and external social scientists working together effectively and with policy colleagues. Lessons for Learning: Using Lesson Study to innovate, develop and transfer pedagogic approaches and metapedagogy TLRP Research Training Fellowship established extensive links with the (then) Department for Children, Schools and Families (DCSF) and the National Strategies. The DCSF and National Strategies subsequently trained two thousand Leading Teachers in the Lesson Study pedagogy underpinning the research outcomes. The Leading Teachers carried out lesson study cycles with year six teachers in selected schools in 14 local authorities, with over 30,000 children engaged in the cohort. Subsequent KS2 results showed double the national improvements in mathematics test results and double the national improvement in the combined English and mathematics test score. This improvement was repeated in 2009 with roll-out funding made available to all local authorities.
Recent Findings(i) Conceptual impact hard to identify Two way relationship between academia and policy Hidden nature of research influence through formal and less formal interactions between academics and government Identifying that the research skills rather than the subject specific knowledge of social scientists working within the Welsh Assembly Government (WAG) had the biggest impact on its activities; and, that the presence of qualified social scientists within WAG helped ensure that the work of external academics was translated enabling maximum policy impact.
Recent Findings(ii) Need to determine the priority for research impact whilst developing and commissioning research investments and communicating clear expectations in terms of what can be reasonably be achieved Need for product strategies at project level to facilitate the production of usable and accessible materials and resources which are close to application Researchers need to systematically record activities which underpin impact (e.g. membership of advisory bodies, attending expert seminars)
Impact Evaluation- Common Issues What are the problems? Timing how long to wait? Attribution what role has research played in change? Complexity- what routes have led to change? Additionality what is ESRC s contribution? Measurement- quantification and qualification
Impact Determinants Established Relationships Co-production Contextual Understanding Timing Knowledge Translation Credibility Leadership Good Infrastructure and Management Support Use of Evidence
Future Activity Economic Impact Evaluation of Research Data Infrastructure Impact Evaluation of the ESRC s Energy Research Groups Sure Start: Case Study of Social Science Impact Case Studies: Research Impact on Practice
Contacts and Further Information Head of Evaluation and Impact Vicki Crossley (vicki.crossley@esrc.ac.uk) ESRC Principal Impact Evaluation Managers: Margaret Macadam (Margaret.Macadam@esrc.ac.uk) and Faye Auty (Faye.Auty@esrc.ac.uk) ESRC Website: http://www.esrc.ac.uk/impactsand-findings/impact-assessment/index.aspx