Day 2- Edinburgh 2nd IESI Experts and Stakeholders Consultation Workshop Brussels, 24-25 February 2015 The mindmap shown in figure 6 highlights that a key topic of conversation at this session was the issue of who does social innovation, and in particular, that this could be people working in groups or as individuals, as well as people in all sectors. What was driving social innovation; questions of who decides - i-frame Outlining Designing a methodological framework for assessing the value and impacts of ICT-enabled social innovation initiatives promoting social investment similarly to day 1- and points about different ways in which social innovation could take place were also made. The corresponding word cloud is shown in figure 7. Gianluca Misuraca, IESI Project Leader Senior Scientist, European Commission, Joint Research Centre, Institute for Prospective Technological Studies 1 The views expressed by the author are not necessarily those of the European Commission
Policy Making Outcomes Problem Statement ICT-enabled Social Innovation Return on investment The evidence challenge Case studies Social Services INDEPENDENT SIP LIVING The master scholar should examine the present, ICT in Social Policy in light of the past and for the purpose of the future (John Maynard Keynes) SUPPORT Assessing impact Initiatives MAPPING Inventory Integrated care
Aim and objectives Development of a methodological framework to assess the impacts generated - from micro to macro level - by ICT-enabled social innovation initiatives promoting social investment (i-frame) to provide a framework of analysis of the initiatives collected through the Mapping activities and as a guide to conduct the in-depth analysis of Case studies structured and comparable approach to analyse such initiatives and provide insights for their replicability and transferability at policy/practice level across the EU to serve as a framework for conducting analysis of return on investment of initiatives, which have as key component ICT-enabled social innovation recommendations on how the European Commission and Member States could analyse (ex-ante, in-itinere and ex-post) the impact of ICT-enabled social innovations initiatives 3
Why measuring? If you can t measure it.. It doesn t exist! If a measurement matters, it is because it must have some conceivable effect on decisions and behaviour. If we can't identify a decision that could be affected by a proposed measurement and how it could change those decisions, then the measurement simply has no value`. Douglas W. Hubbard, How to Measure Anything: Finding the Value of Intangibles in Business
and evaluating? Evaluating means establishing a connection between cause AND effect and assessing the value both in financial and non-financial terms
Limitations and challenges Conceptual 'fuzzyness' and 'moving targets' Unknown universes and unclear 'units of analysis' Anecdotal evidence / Unavailability or poor quality of data Lack of scientific rigorous evaluation Limited assessment capacities of 'actors' involved Lack of robust methodological tools for data gathering / impact evaluation Often 'Evidence building' for policy/investment decision support rather than contribution to fill knowledge gaps different time-frames, expectations and approaches
A quick reminder about Impact Evaluation in theory NON CONTROLLED INTERVENING FACTORS CONTROLLED VARIABLES REACTIONS EFFECTS Ex Ante Evaluation (i.e. modelling) OBJECTIVES INPUT OUTPUTS DIRECT OUTCOMES INDIRECT OUTCOMES IMPACTS (SPECIFIC/ GLOBAL) Monitoring (using measurement indicators) In itinere Evaluation OPERATIONAL EVALUATION Ex Post Evaluation (e.g. counterfactual evaluation) SCIENTIFIC EVALUATION BROADLY DEFINED EVALUATION Source: Misuraca et al., 2014 JRC Report, MIREIA Ie2-IAF 7
and in practice Input: financial end other resources Output: trained people More employment More employability Outcomes/Impacts Unknown effects
and its fundamental problem: causality Fundamental problems of causal attribution/inference: outcomes produced by intervention and not by some other causes? Confounders, covariates, Unobservable/ omitted variables Outcomes can also be merely measured & monitored INPUT ( & other efforts) INPUT/OUTPUT EFFICIENCY OUTPUT (treatment). EFFECTIVENESS OUTCOMES COST-EFFECTIVENESS.. = controlled and direct = Domain of other forms of evaluation, and of mere measurement. = uncontrolled and mediated. = Intervening variables and measurement problems = Domain of Impact Evaluation (both ex post and ex ante)
a quick overview on methods and tools Source: Nesta Evidence Standard Framework Report on Innovation in the Public Sector, 2014 10
An example & starting point for building the i-frame Map einclusion intermediary actors across EU27 to better understand their characteristics and policy potential European Commission JRC FAQ Index Contact Search Services JRC websites Pri JOINT RESEARCH CENTRE The European Commission's in-house science focus on 'employability' First EU e-inclusion map measures the potential for improved digital literacy 14/11/13 http://is.jrc.ec.europa.eu/pages/eap/einclusion/mireia.html 07/11/13 Expand An EU-27 survey of intermediary organisations operating on the education, social and employment sectors and providing IT training has produced a first ever assessment of the e-inclusion intermediary sector. It accounts for a total of 250,000 organisations, or one e-inclusion actor per every 2,000 inhabitants. One in two employs 10 staff or less and operates on a budget smaller than 100,000. Half of the e-inclusion actors go further and offer employment related training. And for two out of three, local government funding is the main financial resource. Design and test a methodological framework to enhance capacity of einclusion intermediary actors and engage them to collect data and to measure their impacts 13/11/13 Expand rriers (such as high costs, restrictions to offer certain services) for electricity storage. First post-typhoon damage assessment for Tacloban City (Philippines) More than 700 residential buildings were completely destroyed and more than 1200 were damaged by the Haiyan typhoon in the Tacloban city alone (Philippines). This is the result of the first rapid damage assessment carried out soon after the event by the Copernicus Emergency Management Service 11/11/13 (EMS), Expand coordinated by the European Commission's Joint Research Centre (JRC). The damage assessment maps are expected to support response activities and, in the longer term, reconstruction efforts. JRC software to assess impact of cyber-threats against physical infrastructures The framework has been tested using counterfactual impact evaluation methodologies in four case studies at national and regional level in Ireland, Italy, Poland and Spain The JRC has developed innovative software to assess the cyber-security of connected critical infrastructures (CIs), such as railway systems, energy networks or power plants. The AMICI software, which stands for Assessment platform for Multiple Interdependent Critical Infrastructures, provides a novel experimental approach as it takes into account both the virtual and the physical aspects of modern interconnected CIs. New method to study mountainous terrains with satellite imagery JRC scientists obtained remarkable results in correcting surface shapes of satellite imagery to study mountainous terrains, with a combination of topographic correction algorithms and statistical methods. The JRC presented its new method in a recently published article in the IEEE Journal Sear The P prov publi Bu env fre dat Co and Far EC pot Me 40 res effi 14/ Sci hea cha add 29/ Rou Sci Eff 04/ 4th Sci Asp
Additional complications Beyond GDP measuring well-being and happiness
and balancing social and economic outcomes and value perspectives The headings in the photograph above reflect conversation about how whether a social innovation is an improvement, or not, depends on your perspective and that there is an issue over who gets to have a say in the goals or purpose of an innovation. Participants also highlighted how the relative success of a social innovation can look different over time. Further points included how social innovation can be accidental rather than by design. The word cloud for the Inverness Session (figure 4) particularly reflects a conversation about why social innovations are attempted, the desired outcomes, the aims and purpose. 13
SI is about change SI as a process Context Outcomes/ goals/ purpose? What determines social? Source: Melting Pot, Scottish Government, 2014 Report on How can we put social innovation to work for the people of Scotland? A collaborative enquiry addressing wicked problems in complex social systems igure 3 - Wendy with the output from Inverness 14
Filling the gap? Current frameworks and approaches are limited or solve only part of the problem: e.g. LM3, SROI, Social earning ratio, etc. Need to develop a meta-framework of social impact assessment for ICT-enabled social innovation initiatives promoting social investment in integrated approaches to social services delivery social innovation nature and elements specific role and contribution of ICT-enabled innovation characteristics of different welfare systems and social services delivery models needs of target groups and socio-economic evolving situation 15
i-frame: outline Draft Work in progress, Misuraca et al., 2015JRC-IPTS IESI Working Paper (D2.2.1) 16
Co-designing i-frame Review of the CURRENT state of the art based on PAST experience to propose solutions (in terms of concepts, methodologies and tools) for FUTURE development of the i-frame examine the present, in light of the past and for the purpose of the future (John Maynard Keynes) 17
More concretely 1. TODAY: start outlining a proposal of methodological approach for developing the operational components of the i-frame 2. By end of 2015: fully-fledged methodological approach and development of the operational tools composing the i-frame structured and comparable approach to analyse initiatives collected through the IESI Mapping and as a guide for Case study analysis 3. 2016: test and validation of the i-frame operational framework for analysis of social an economic return on investment of ICT-enabled social innovation initiatives, including e.g. Social Policy Experimentations and other case studies 18
Structure of the Workshop Day 2: Morning Session 08:30-9:00 Registration for Day 2 09:00-10:30 Session III. Assessing impacts of ICT-enabled social innovation for social investment 09:00-09:15 Towards a methodological framework for assessing impacts of ICT-enabled social innovation initiatives promoting social investment - IESI-FRAME outline Gianluca Misuraca EC JRC-IPTS 09:15-09:30 Assessing the returns of social investments Evelyn Astor DG EMPL 09:30-10:00 Impact assessment models and approaches for measuring the economic and social return on investment of ICT-enabled social innovations Karl-Heinz Richter Jenny Billings 10:00-10:30 Open discussion 'Initiators': Gwendolyn Carpenter, and Simona Milio 10:30-10:50 Coffee Break Engaged X University of Kent Danish Technological Institute (DTI) London School of Economics (LSE) 10:50-12:45 Session IV. Hands on Co-developing the IESI-FRAME 10:50-11:00 Objectives, structure and guidelines for working groups Clara Péron Value for Good 11:00-12:45 Working Groups (Break-out Sessions) All participants 12:45-13:45 Working Lunch 19
Structure of the Workshop Day 2: Afternoon Session 13:45-15:45 Session V. Discussion and validation of the results of the working groups 13:45-14:45 Presentation of the results of the working groups Rapporteurs 14:45-15:45 Open Discussion and validation Gianluca Misuraca 15:45-16:00 Coffee Break EC JRC-IPTS 16:00-17:30 Session V. Paving the way forward 16:00-16:20 Taking stock of the workshop's discussion and next Gianluca EC JRC-IPTS steps of the IESI research project Misuraca 16:20-17:00 Open discussion and suggestions Moderated by: Aurelio DG EMPL Fernandez Lopez 17:00-17:30 Closing remarks Egbert Holthuis Head of Unit, DG EMPL 20
gianluca.misuraca@ec.europa.eu 22