31.01.2018 - "Filling the gaps"
With their presentation on “The Emerging Challenges in Generating and Using Evidence for Development Impact”, Emmanuel Jimenez and Marie Gaarder of the International Initiative for Impact Evaluation (3ie) sparked a lively discussion among evaluation experts at DEval on 29 January.
Jimenez and Gaarder presented two main “emerging challenges”: the use of evidence provided through systematic reviews and the filling of knowledge gaps.
The use of evidence:
As evaluators, we want evidence to be used in policy-making to improve development outcomes. To reach this goal, Emmanuel Jimenez stresses the importance of identifying and engaging stakeholders. Too often, evaluators make recommendations directed solely at implementing organisations, while neglecting to engage policy makers and other important stakeholders.
Identifying whether evidence is used can also present a challenge. To illustrate this, Marie Gaarder points to an “action bias”: “if an evaluation finds that something is working and you continue doing it, this continuation of a policy or project will often not be identified as an impact.”
Filling of knowledge gaps:
Jimenez and Gaarder present different kinds of “knowledge gaps”. One such gap is regional. A map of the regional distribution of impact evaluations shows that there have been very few impact evaluations studying West African nations as opposed to countries such as India or China.
Another such knowledge gap is in regards to thematic sectors. The 3ie “evidence gap maps” show a heavy concentration on four sectors; health, education, social protection and agriculture. Other sectors such as energy, though heavily funded, have been studied much less.
In the discussion with evaluation experts in the audience, questions of generalizability of findings were raised, as well as the relevance of systematic reviews to policy makers. Jimenez pleads for a shift from generalizability, i.e. “this finding is universal for all settings”, towards transferability; “how can this finding be applied in a different setting?” This approach is also recommended for policy makers trying to make use of the evidence from systematic reviews when shaping policies. Despite the value of evidence synthesised from some 4000 studies, Jimenez and Gaarder point to the importance of additional qualitative data to understand why certain interventions work, as well as a clear definition of the particular need being addressed. As an example, Jimenez and Gaarder present a systematic review on the impact of education programmes on school participation and student learning. The review identifies cash transfers as an effective intervention category for increasing school participation, but not for improving student learning. If a region shows high learning among students attending school but low participation, cash transfers could be a viable policy. If student learning also needs to be addressed, cash transfers can only be one part of a multi-pronged solution. In short: for this or any evidence to be of relevance, policy makers must apply the finding to the particular objective of a particular intervention.
The International Initiative for Impact Evaluation (3ie) is an international grant-making NGO promoting evidence-informed development policies and programmes.
Emmanuel (Manny) Jimenez is the executive director of 3ie. He came to 3ie after 30 years in different positions at the World Bank Group, among others as the director of public sector evaluations at the Independent Evaluation Group. He has published numerous monographs and articles in the fields of education, social protection, labour, health, urban development, public finance, environment and Population.
Marie Gaarder is the director of the evaluation office and global director of innovation and country engagement at 3ie. She recently joined 3ie from the World Bank’s Independent Evaluation Group where she was manager for social sector and corporate evaluations. Prior to that, she was the director of the evaluation department at the Norwegian Agency for Development Cooperation (NORAD) and the deputy executive director of 3ie.