International Initiative for Impact Evaluation Farmer field schools: from agricultural extension to adult education Making Impact Evaluation Matter Manila, 1 st -5 th September 2014 Hugh Waddington, Birte Snilstveit, Jorge Hombrados, Martina Vojtkova, Daniel Phillips, Philip Davies, Howard White
Implementation history
FFS project objectives Most FFS projects focus on agricultural production and food security Increasing focus on empowerment
Polarised debate on FFS...the program did not have significant impacts on the performance of graduates and their neighbors Results demonstrated remarkable, widespread and lasting developmental impacts
What does a systematic look at the evidence say? Literature reviews are like sausages... I don t eat sausages as I don t know what goes into them Dean Karlan
A standard systematic review of effects 28,000 study titles and abstracts screened 460 relevant FFS studies screened at full text rigorous impact evaluations: 15 projects
3ie s integrated synthesis of systematic reviews 28,000 study titles and abstracts screened 460 relevant FFS studies screened at full text targeting: 92 projects rigorous impact evaluations: 15 projects
3ie s integrated synthesis of systematic reviews 28,000 study titles and abstracts screened 460 relevant FFS studies screened at full text targeting: 92 projects rigorous impact evaluations: 15 projects barriers and enablers: 20 projects
3ie s integrated synthesis of systematic reviews 28,000 study titles and abstracts screened 460 relevant FFS studies screened at full text global project portfolio: 327 projects targeting: 92 projects effects: rigorous IEs 15 projects barriers and enablers: 20 projects benefit/ cost estimates: 4 projects high risk of bias IEs (77 projects)
Farmer field school intervention Inception Targeting Farmer training Dissemination Training of FFS facilitators Curriculum development Finance & systems Effective targeting mechanisms Farmers are able to take part Farmer group formation Season-long training attended by farmers Facilitation through discovery-based group learning Field days, exchange visits Platform building Training of farmer trainers
Theory of change: intermediate outcomes Inception Targeting Farmer training Dissemination Capacity building Adoption Diffusion Legend: Intervention components Intermediate outcomes
Theory of change: impacts Inception Targeting Farmer training Dissemination Capacity building Adoption Diffusion Legend: Intervention components Intermediate outcomes Impacts Impact Sustainability
Policy: prices of inputs; regulations Relationship between input producers and extension/training system Market access for farmers Existing farmer practices Inception Targeting Farmer training Dissemination Capacity building Adoption Diffusion Legend: Policies & practices Intervention components Intermediate outcomes Impacts Impact Sustainability
Presentation 1: Systematic review of targeting
Presentation 2 Systematic review of impacts and barriers to and enablers of impacts Martina Vojtkova 3ie
3ie s integrated synthesis of systematic reviews 28,000 study titles and abstracts screened 460 relevant FFS studies screened at full text global project portfolio: 327 projects targeting: 92 projects effects: rigorous IEs 15 projects barriers and enablers: 20 projects benefit/ cost estimates: 4 projects high risk of bias IEs (77 projects)
Integrated synthesis along the causal chain Context Inception Targeting Farmer training Dissemination Capacity building Key Policy environment Intervention components Adoption Diffusion Intermediate outcomes Impacts Impact Sustainability
Improved outcomes for FFS participants
Risk of bias assessment: Included impact evaluations 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Overall risk of bias assessment 17 73 1) Selection bias & confounding 17 73 2) Spillovers & contamination 24 44 22 3) Outcome reporting bias 68 20 2 4) Analysis reporting bias 25 46 19 5) Other sources of bias 97 3 Low risk of bias Medium risk of bias High risk of bias
Critical appraisal: Studies of barriers and facilitators Research aim 100% Context 100% Sampling 80% 20% Sampling characteristics Data collection 90% 95% 10% 5% Data recording 35% 40% 25% Analysis 35% 40% 25% Link to relevant lit/ theory 85% 5% 10% Appropriate methodology 50% 45% 5% Appropriate sampling 40% 50% 10% Appropriate methods of data collection 40% 60% Appropriate analysis 15% 20% 55% 10% Triangulation 85% 15% Clarity of analysis and conclusions 25% 10% 65% Conflict of interest 15% 80% 5% Ethical considerations mentioned 25% 70% 5% Yes No Partially Unclear
Summary meta-analysis findings FFS participants
Barriers and facilitators of knowledge Training, skills, experience an characteristics of facilitators Top down vs participatory approach to training delivery Relevance and appropriateness of curriculum Language and terminology Resources for FFS schools Farmers motivation
Weak evidence of improved empowerment Only one quantitative study reported increased perceived ability to solve problems, express their opinion and participate in the community Qualitative studies: Self-confidence Improved social capital Leadership roles Gender relations
Summary meta-analysis findings FFS participants
Barriers and facilitators to adoption Approach to training delivery Relevant and appropriate curriculum Conflicting agricultural policies, institutional legacy and incentives Social capital Observability of benefits Complexity Access to inputs / capital Active follow up
Summary meta-analysis findings FFS participants
Improved yields may not be achieved at scale Study ID ES (95% CI) Pilot project Peru: DE Foundation Coffee Project China: FAO/EU IPM Programme for Cotton in Asia Pakistan: Cotton IPM programme (FAO, EU, GoP) Ecuador: Plataformas Program (FAO) Tanzania: East African Sub-regional Pilot Project Phase II (FAO) Pakistan: FAO/EU IPM Programme for Cotton in Asia Kenya: East African Sub-regional Pilot Project Phase II (FAO) Ethiopia: Participatory Forest Management Project in the Belete-Gera Regional Forest Priority Area Subtotal 1.13 (1.05, 1.22) 0.86 (0.63, 1.18) 1.08 (1.03, 1.14) 1.09 (1.03, 1.15) 1.22 (0.97, 1.53) 1.23 (1.00, 1.51) 1.24 (1.01, 1.54) 1.81 (1.15, 2.84) 2.71 (1.11, 6.60) 1.13 (1.05, 1.22) Scaled-up programme Vietnam: FAO Programme for Community IPM in Asia Indonesia: National IPM Training Project Phase II (FAO, World Bank, GoI) Subtotal 0.98 (0.96, 1.01) 0.97 (0.72, 1.31) 0.98 (0.96, 1.01) 0.98 (0.96, 1.01).5 1 2 3 6 FFS reduces yields FFS improves yields
No evidence of diffusion to FFS neighbours
Why no diffusion? Experience-based nature Complexity of practices Observability of benefits Social cohesion Socioeconomic differences
Funnel of attrition explains successes & failures
BCRs>1 require strong assumptions on adoption
Implications for policy FFS is effective in pilot projects Average improvement in yields 10% Average improvement in net revenues 18% FFS does NOT appear effective in scaled-up programmes Problems in recruiting and training facilitators Problems of price distortions and other policy factors Problems in targeting marginalised farmers Even in pilot programmes, FFS may NOT be cost-effective Evidence suggests there is little or no diffusion Very little evidence available on empowerment
Implications for programmes Recruitment and training of FFS facilitators Complementary interventions: up-stream/down-stream Targeting: location (areas with clear needs), participants (time availability, complementary inputs, relevant curricula) Sustainability: formal support/back-stopping of FFS alumni Scale-up: gradual approach favouring depth over breadth Intervention design?
Implications for research and evaluation Rigorous impact evaluations measuring empowerment outcomes Factorial designs examining training intensity Long-term follow-up and scale-up Factual data from FFS facilitators and extension workers
Thank you Policy-friendly report: Full technical report: www.campbellcollaboration.org/