IMPACT AND REF New challenges for universities. Dr Molly Morgan Jones and Dr Catriona Manville

Transcription

1 IMPACT AND REF 2021 New challenges for universities Dr Molly Morgan Jones and Dr Catriona Manville

2 Session outline Introductions Why measuring impact is difficult but not impossible REF and Impact what did we learn last time? The path to REF 2021: Strengths, weaknesses and opportunities for ARU s case studies Coffee Institutional case studies breakout discussions Reflections and Next steps 2

3 Objectives Review the purpose of the case studies in the wider context and learning from REF 2014 Identify and understand the strengths, weaknesses and opportunities for ARU case studies Establish a forward plan for strengthening the case studies and gathering evidence with a particular focus on institutional case studies Your objectives? 3

4 INTRODUCING RAND EUROPE 4

5 RAND Europe Independent not-for-profit public policy research institute help improve policy and decision making through research and analysis Not a university and not a management consultancy but with the capabilities of both Part of the global RAND Corporation Work across the breadth and depth of government Strongly held values of quality and objectivity Provider of evidence 5

6 Our experience of research impact 6

7 CHALLENGES IN MEASURING IMPACT 7

8 There are many reasons to evaluate impact... Advocacy Make the case for research funding Accountability To taxpayer, employer, donors etc Analysis What works in designing or disseminating research? Allocation What to fund (institution, field, people ) 8

9 Impact is an important component of REF One 4* case study is (roughly) equivalent to thirteen 4* publications/outputs The back of the envelope 4 outputs/researcher vs. 1 case study/10 researchers 10 FTEs = 40 outputs The ratio of weightings is 65%/20% = 1/3 Therefore 1/3 weighting of impact x 40 outputs = 13 outputs/case study 9

10 Measuring and identifying impact is not easy 10

11 The challenge of attribution 11

12 Dealing with serendipity 12

13 Addressing time lags 13

14 Will users engage? 14

15 Examining the margin 15

16 Measuring impact is difficult, but not impossible 16

17 REF AND IMPACT WHAT DID WE LEARN LAST TIME? 17

18 Plurality of impacts 'For the purposes of the REF, impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia' (HEFCE et al 2012a). 18

19 The case study s job is to establish the impact pathway 19

20 Why the contribution of the research is important 20

21 The critical role of evidence Significance must be demonstrated through nature of benefits Reach must be demonstrated by the scale of impact Evidence will vary for different types of impact Quantitate and qualitative Value of fact vs evidence of opinion Breadth versus depth of case study Importance of the chain of evidence 21

22 What counts as evidence? All panels give examples of types of evidence The case study database is a great source of relevant examples International examples: Canadian Academy of Health Sciences (CAHS) Payback Framework Five categories of impact with indicators Australian Research Quality Framework provides list of indicators 22

23 RQF indicators Reduced pollution Regeneration or arrested degradation of natural resources Lives saved Reduced infection rates Reduced treatment time and cost Increased literacy and numeracy rates Positive reviews of creative publications and performances Increased cultural awareness Royalties Increased employment Reduced costs and resource usage Increased competitiveness of Australian industry Spin off companies New products & inventions Licences Creative works commissioned Change in procedures, behaviours, outlook etc New policies, guidelines, legislation etc Citations of research in legal judgements which become case law. Contracts and industry funding Repeat business Number of presentations involving contact with end-users Community awareness of research Non-academic publications & performances Collaborative projects with end-users Citations in Government reports, Hansard, etc Provision of expert advice and submissions to enquiries etc Invitations to be a visiting researcher or researcher in residence at an end-user institution (if based on research merit) Invitations to be on reference, advisory and/or steering committees (if based on research merit) 23

24 REF guidance on indicators/evidence Measures of improved clinical outcomes Documented changes to public health guidelines Evidence of adoption of best practice Evidence of use of process or technology Documented evidence of influence on policy process Economic impacts cost-effectiveness measures Priority shifts in expenditure profiles or quantifiable reallocation of budgets Documented evidence of policy debate Measures of improved international welfare or inclusion Evidence of sustained engagement with a group Evidence from clinical trials Traceable references to inclusion of research in industry standards or authoritative guidance Citation in a public discussion or consultation Measures of improved inclusion, welfare or equality Visitor or audience numbers, or number of participants Quantitative data relating, for example, to effectiveness or organisational performance Satisfaction measures Incorporation in training or CPD material Publication and sales figures, tourism data Critiques or citations in users documents, e.g. in the media, in teaching materials, for public or commercial bodies Number and profile of people engaged, types of audience Formal partnership agreements with major institutions, NGOs and public bodies Independent testimony 24

25 CONSIDERATIONS WHEN DEVELOPING CASE STUDIES 25

26 Overall observations from your case studies There is a need for better understanding of what impact stands for Understanding eligibility and key parameters for the REF is crucial for building the impact case study There is no one-size-fits-all approach either the case studies or assessment Collecting the evidence needs to be seen as a planned, systematic, and recurring activity Contributing to / influencing policy, public debate needs some forethought (and luck!) Avenues for dissemination and resources within ARU Mapping within and between units across ARU Going forwards the writing of the case study will be important Overall, ARU has a strong portfolio of research which actively engages with stakeholders already. This is an important first step on the pathway to impact, but impact activities will need careful support and planning between now and

27 Understanding of what impact stands for Impacts listed must be outside academia Note this is different from establishing the impact pathway Impact needs to have been realised, not potential Impact demonstrated needs to be within the impact window Dissemination at academic conferences can be an aspect of the case study but not impact itself Reach and significance are two sides of the impact' coin Both are equally important Don t forget about contribution! 27

28 Understanding eligibility and key parameters for the next REF Uncertainty around revision of existing impact: timelines for REF 2021and new contribution since 2014 for resubmission case studies. Impact stays with the institution at which the research was conducted and does not travel ARU s unique contribution in case of multi-institutional effort Stern review and its implications 28

29 Stern review recommendations about impact Recommendation 5: Institutions should be given more flexibility to showcase their interdisciplinary and collaborative impacts by submitting institutional level impact case studies, part of a new institutional level assessment. Recommendation 6: Impact should be based on research of demonstrable quality. However, case studies could be linked to a research activity and a body of work as well as to a broad range of research outputs. Recommendation 7: Guidance on the REF should make it clear that impact case studies should not be narrowly interpreted, need not solely focus on socio-economic impacts but should also include impact on government policy, on public engagement and understanding, on cultural life, on academic impacts outside the field, and impacts on teaching. 29

30 No one-size-fits-all approach This applies to putting the case study together and also to its assessment Important to recognise that the panel assessing the case study would take a subject/field-specific approach The panel's approach to assessment would be based on the strengths and weaknesses of the subject/field 30

31 Collecting the evidence Evidence needed to demonstrate reach and significance Reach achievable through (amongst other methods) public talks, writing in the trade press, the conversation, twitter/social media communication, download data, number of users, etc. Demonstrating significance requires evidence of change over a length of time. This change can be at an individual / collective level Significance can also be about preserving cultural heritage / or a unique contribution to the field or discourse Planning and persistence needed to collect evidence (e.g. feedback forms, twitter hashtags, s, feedback surveys, testimonials, written responses, book reviews, Amazon reviews, online comments/feedback) 31

32 Contribution / influence on policy, public debate Has the research fed directly / indirectly into policy debate or related outcomes? Testimonials from the stakeholders (written, , audio, or video) - government, NGOs, consumer bodies, or industry consortia Has the research been referenced in consultations, select committee hearings, and/or parliamentary debate? Look out for any citations in guidelines, reference documents, or policy papers published by the government 32

33 Avenues for dissemination and resources within ARU Consider using MSc/PhD students for communicating through social media and collecting evidence such as participant/audience feedback Engage with stakeholders Create forums for people to engage with research Using 'The Conversation' for reaching a non-academic audience, tracking feedback and debate Dissemination through relevant charities and public professional bodies Consider availability of centralised resources University press officer to publicise talks, book and article publications Technology transfer team to support commericalisation and discussions with industry Shared impact-oriented resources can help to provide practical and individual support across multiple UOAs 33

34 Mapping within and between units across ARU Different ways of clustering research and impact to frame a case study Maturity of research Chronological Type of impact Cross-cutting Type of stakeholder group Grand challenges Use of less developed or smaller scale impact alongside other elements of a case study 34

35 Going forwards the writing of the case study will be important Title Summary of impact Underpinning research References to research Impact section Corroborating sources 35

36 Title of the case study The first thing that reviewers will read A simple way to clarify and strengthen case studies Should include a clear statement of the impact Should express the focus of the case study Titles should be impact rather than research-led Also attention grabbing, where possible Do not make them too long 36

37 Examples of message driven titles from REF 2014 submissions Improving health access and equity in India through health financing reform Changing the way history of the Cold War is taught in schools across the United States and Europe A framework for establishing how to increase global food production at least cost to biodiversity Innovative chemistry reduces the environmental impact of mining and pharmaceutical manufacture The use of citizen science in recording wildlife: ladybirds and invasive alien species Attraction explained: The science of how we form relationships 37

38 Examples from your case studies Too long The prevalence of vision loss due to various causes had been occasionally analysed by the WHO in the past, however the outputs were limited in there was minimal attention to age-specificity and no gender-specificity to the results. Despite occasionally analysing the prevalence of vision loss, WHO's past studies have paid minimal attention to age and gender-specific causes. Too short Labour History Research Unit A new understanding of the civil wars in the Labour party and their influence on the livelihoods of the British working class No active verb Shiny Happy People Creating shiny happy people 38

39 Summary of the impact From the 2014 template: This section should briefly state what specific impact is being described in the case study Summary of key points, like an abstract Communicate three things: What the problem is/was (1 sentence) What the research was (1-2 sentences) What impact you had (2-3 sentences) Bullet points or numbered lists could aid clarity and help keep within word limit Use figures where possible 39

40 Underpinning research Needs to state what was unique about the particular research which contributed to the impacts claimed Distinct contribution c.f. research elsewhere Research should be clearly defined and bounded Focus on the impact, not career of particular researchers 40

41 References to the research From the 2014 template: This section should provide references to key outputs from the research described in the previous section, and evidence about the quality of the research Include: Author(s); Title; Year of publication; reference details (e.g. Journal title and issue); DOI or URL if required Need to provide evidence of quality, e.g.: rigorous peer-review process (e.g. Journal impact factors) end of grant reports referencing a high quality grading favourable reviews from authoritative sources prizes or awards made to individual research outputs cited evidence that an output is a reference point for further research beyond the original institution rating in output submission 41

42 Details of the impact Core of the case study Needs to present a clear narrative outlining how the impact came about, and how it is linked to the research From the 2014 template: This section should provide a narrative, with supporting evidence, to explain: how the research underpinned (made a distinct and material contribution to) the impact the nature and extent of the impact 42

43 Suggestions for a general approach to impact section Overview of the impact at the start of the section The narrative should outline the key steps which took place from the research being conducted to impact being realised NB: This may not be linear! Focus in this section on the impact Specific references and information on reach and significance Quantitative information can be useful here Always think one step further, beyond the initial impact Do not restate academic research, impacts Do not use the words will, could, may, etc 43

44 Sources to corroborate the impact From the 2014 template: This section should list sources external to the submitting HEI that could, if audited, provide corroboration of specific claims made in the case study... the information in this section will be used for audit purposes only Should provide evidence for each specific impact claimed What are the crucial steps in your narrative? Link evidence to impacts claimed in Section 4 Similar structure for each section Use a numbering system 44

45 ARU research already has the foundation for good impact Many of the case study groups we met with are already engaging with stakeholders and users of the research as part of the research itself This means that the foundations of impact are already present However, impact does not (always) just happen. It needs careful planning and active encouragement in order to be created. Our impact planning tools and ideas discussed in the workshops and memos can help map out your pathway to impact Stakeholder mapping Using a logic model to map backwards from desired impacts, to needed outputs, to activities and dissemination, to resources required Identification of impact creation steps you need to take 45

46 46

47 Breakout session: Introduction to institutional case studies Stern Recommendation 5: Institutions should be given more flexibility to showcase their interdisciplinary and collaborative impacts by submitting institutional level impact case studies, part of a new institutional level assessment. 47

48 REF 2014 case studies were (already) interdisciplinary in nature don t underestimate ARU s interdisciplinary strengths 48

49 Breakout sessions The purpose of the breakout sessions will be to: Identify areas of overlap between UOAs which could be developed into institutional case studies Start developing plans to create and track impact A discussion in three parts: Part 1: Reflect on learning and identify areas of synergy across UOAs (20 mins) (By doing this exercise in groups we hope to encourage and enable areas of overlap and synergy to come out) Part 2: Developing and mapping your impact strategy (20 mins) (Intention is to encourage people to think about cross-cutting case studies, but generally this mapping tool could be used at an individual case study level and overall we d like to get people comfortable with using it so you can go away from the day with a planning tool to hand). Part 3: Reporting back to the wider group (20 mins) 49

50 How to think about case studies Different ways to frame the case study Chronological Type of impact Cross-cutting Type of stakeholder group Grand challenges Ideas for institutional case studies Medical Teaching/schools/social work Veterans/military Digital technology Wellbeing 50

51 Part 1: Reflect on learning and identify areas of synergy across UOAs (20 mins) 51

52 What are the research strengths of my UOA? What are the impact strengths of my UOA? What are the research and impact opportunities looking forward? Who are our main external stakeholders? What other UOAs could our UOA work with? UOA W UOA X UOA Y UOA Z Grand Challenges: what are the big social, economic, political, health, cultural, or other issues which cut across these UOAs? Summary of overlapping areas of impacts across UOAs Realised impacts: Potential impacts: 52

53 Part 2: Developing and mapping your impact strategy (20 mins) 53

54 Area of exploration Grand challenge - What s the vision for the impact case study? What broader social challenges is it addressing? What impacts will be highlighted? - What are the 3-4 main areas of impacts which will be demonstrated? (e.g. improved social well being, changes in policy, changes in practice, improved health outcomes, economic growth, etc.) What outputs will lead to the impacts? - 2*+ research publications - Research networks and collaborations - Publications targeted at stakeholders What dissemination efforts will be needed and what audiences will you target? - Dissemination outputs (media, blogs, etc.) - Audiences (local councillors, professional bodies, general public, policymakers) What activities do we need to proceed with in order to produce our outputs/outcomes? - Proposed new research - Proposed new stakeholder engagements - Proposed new dissemination activities - Proposed new evaluations What resources are already available and what will be needed? - Existing research( mark with a *) - Investment required (time, money, people, space, project) - Existing networks/stakeholder relationships Notes This will allow researchers to identify their hook for the case study early on. This will encourage researchers to think about how they will group various impacts under different headings. There can be multiple kinds of impacts, but it is important to carve out a clear narrative. This will enable people to plan what research outputs might be needed to support the case study, but also what additional, more impact-focussed outputs might be required. By separating out dissemination as a unique step, this allows for the distinction between dissemination and impact to be reinforced, but will also encourage people to think about what dissemination activities are needed to create impact. This step will allow people to plan different activities. This step will allow people to think about what resources are required for each activity. 54

55 Logic model worksheet 55

56 Group 1: UoA 1, UoA 3, UoA 4, UoA 5 Strengths and areas for future collaboration/opportunity: Will use the process of identifying case studies to promote collaboration and discourse across the UoAs Areas of research activity which cut across the group: Ageing and dementia [links to other disciplines including law (dementia and rights) and business and management studies (work-life balance and silver economy)] Depression [supporting links to music (music therapy) and business and management studies (work life balance)] Use of medical technology [links to music therapy and computer science] 56

57 Group 2: UoA 11, UoA 15, UoA 16, UoA 17, UoA 26 Have a mix of single disciplinary focus and full interdisciplinary research; focus on problem solving; Have the ability to influence policy change and support citizen level understanding and development of practical solutions Main stakeholders include: government, industry (as bridge), citizens They seek to achieve impact through development of tools and products Grand challenges they could address cover many areas and bring together the following areas under the overarching umbrella of resilience and future living : Behaviour change Sport science [links to UOA35, Sport Law group, who do work on competition] Quality of life (inc behaviour change) Environment and natural capital Future cities Medical technology 57

58 Group 3: UoA 19, UoA 20, UoA 22, UoA 23, UoA 25 Strengths: significant core areas of research already well established eg early childhood research, education, social policy and all are interdisciplinary in nature so similar areas of impact can be readily identified Stakeholder engagement in research is strong: service users and carers in research, strong regional presence in East of England Core research themes (with cross-cutting links to other areas as above): Ageing Entrepreneurship Policing and innovation Social justice HE pedagogy Future cities Digital technology in education 58

59 Group 4: UoA 28, UoA 29, UoA 30, UoA 34, UoA 35, UoA 36 Core research areas have links with others and include: Age/dementia/depression Violence working with people in policing (UoA 36) [links to social policy and sociology] Migration Arts and humanities, plus sustainability and health issues (from UoAs 28,29,36 [links with sociology (urban multi culture); social policy (education and minority groups); and law (EU and migration policy)] Education and publishing (it was noted that these are more relationships between colleagues, rather than themes, but could be developed further) 59

60 What are the research strengths of my UOA? What are the impact strengths of my UOA? What are the research and impact opportunities looking forward? Who are our main external stakeholders? What other UOAs could our UOA work with? UOA 1 UOA 3 UOA 5 UOA 6 Grand Challenges: what are the big social, economic, political, health, cultural, or other issues which cut across these UOAs? Summary of overlapping areas of impacts across UOAs Realised impacts: Potential impacts: 60

61 What are the research strengths of my UOA? What are the impact strengths of my UOA? What are the research and impact opportunities looking forward? Who are our main external stakeholders? What other UOAs could our UOA work with? UOA 11 UOA 15 UOA 16 UOA 17 UOA 26 Grand Challenges: what are the big social, economic, political, health, cultural, or other issues which cut across these UOAs? Summary of overlapping areas of impacts across UOAs Realised impacts: Potential impacts: 61

62 What are the research strengths of my UOA? What are the impact strengths of my UOA? What are the research and impact opportunities looking forward? Who are our main external stakeholders? What other UOAs could our UOA work with? UOA 19 UOA 20 UOA 22 UOA 23 UOA 25 Grand Challenges: what are the big social, economic, political, health, cultural, or other issues which cut across these UOAs? Summary of overlapping areas of impacts across UOAs Realised impacts: Potential impacts: 62

63 What are the research strengths of my UOA? What are the impact strengths of my UOA? What are the research and impact opportunities looking forward? Who are our main external stakeholders? What other UOAs could our UOA work with? UOA 28 UOA 29 UOA 30 UOA 34 UOA 35 UOA 36 Grand Challenges: what are the big social, economic, political, health, cultural, or other issues which cut across these UOAs? Summary of overlapping areas of impacts across UOAs Realised impacts: Potential impacts: 63