ECG Good Practice Standards
Annex III.2: A Note on "Impact” and "Impact Evaluation” in the Public Sector GPS
Published by: Independent Evaluation Group (IEG), World Bank Group
Annex III.2: A Note on "Impact” and "Impact Evaluation” in the Public Sector GPS
1. The term "impact” has multiple meanings. It is understood to mean different things by different IFIs, and even within a given definition, there are sometimes multiple meanings. According to the OECD-DAC glossary of evaluation terms, "impacts” are defined as:
"positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended.”[1]
2. The OECD-DAC definition encompasses several concepts:
· a time dimension ("long-term effects”). The long-term nature of "impacts” is sometimes contrasted with the short- and medium-term nature of "outcomes;”
· two levels of results ("primary” and "secondary”);
· the pathways of transmission of results ("directly or indirectly”); and
· results captured in the project's results chain ("intended effects”) as well as others outside of the results chain ("unintended effects”).
3. Not surprisingly, and partly as a result of the multiple meanings in the OECD-DAC definition, various IFIs understand "impact” to mean different things.
· Sometimes it is understood to include both causal linkages (results attributable to the intervention) and long-term effects.
· Some IFIs consider "impact” to mean "the final level in the results chain” or "the highest level of objectives that an intervention has identified "(i.e., the ultimate intended results of the intervention).
· Some IFIs understand "impacts” to mean overall development goals (e.g., Millennium Development Goals in various sectors), and use the term "outcomes” to mean "project purposes/objectives.” This definition of the term "impact” is not the same as the "final level in the results chain.”
· Some IFIs define "impact” to mean specific types of results that are of particular interest to the IFI – such as institutional development, environmental impact, and/or social impact – that may or may not appear in the project's results chain.
· "Impact” sometimes also has the meaning of the unit of analysis examined (that is, results measured at the sector, region, or country level versus those measured for project beneficiaries).
These meanings are at variance with each other, and a number with the OECD-DAC definition, which includes both intended and unintended results.
4. Because of the multiple and sometimes conflicting meanings that are associated with the word "impact,” the GPS avoids using the term – either as a level in the results chain or as an evaluation criterion. Instead, the various meanings of the OECD-DAC and other definitions are stated directly – e.g., "long-term effects”'; "results attributable to the project”, and "broad social and economic goals”. The final level of the results chain – reflecting the ultimate intended results of the project – is generally called "intended outcome” in the GPS. Where relevant, the equivalent use of the term "impact” by some IFIs is shown in the Notes column.
5. Note that a results chain can have several levels of outcomes. In the evaluation literature, these are sometimes distinguished by using the terms "intermediate outcomes” and "final outcomes”. However, to avoid introducing new terminology, the GPS simply uses the word "outcome” to refer to levels in the results chain beyond "outputs.”
6. In avoiding the term "impact,” the GPS does not intend to suggest that IFI projects should not aim at achieving broad economic and social goals. However, the GPS calls for project objectives to focus on outcomes for which the project can reasonably be held accountable, avoiding objectives beyond the purview of the project. To the extent that higher-level social and economic objectives and corporate goals are included, they should be targeted at segments of the population that can reasonably be expected to be affected by the project, directly or indirectly (See OP #1.1 in GPS on Self-Evaluation of Public Sector in Chapter VI).
7. As with the term "impact,” there is confusion regarding the term "impact evaluation.” For some, "impact evaluation” means "an evaluation of impact”, however "impact” is defined. For example, some IFIs understand "impact evaluation” to mean an assessment of the achievement of the objectives reflected in the final level of the results chain; others interpret it to mean the project's effect on broad social and economic indicators that are not included in the results chain. For other IFIs, "impact evaluation” means an evaluation that establishes causality, i.e., attributes results to the project.
8. The OECD-DAC Glossary does not have a definition of "impact evaluation”. The definition used by the International Initiative for Impact Evaluation (3ie) is based on attribution and methodology:
"Impact evaluations measure the net change in outcomes amongst a particular group, or groups, of people that can be attributed to a specific program using the best methodology available, feasible and appropriate to the evaluation question(s) being investigated and to the specific context.”[2]
The 3ie definition is consistent with that of the Network of Networks on Impact Evaluation (NONIE), which focuses on two underlying premises for impact evaluation: (i) attribution: attributing impacts to interventions, rather than just assessing what happened; and (ii) counterfactual: an attempt to gauge what would have occurred in the absence of the intervention with what has occurred with the intervention implemented.[3]
9. Consistent with the 3ie and NONIE definitions, the GPS defines "impact evaluation” as:
An evaluation that quantifies the net change in outcomes that can be attributed to a specific project or program, usually by the construction of a plausible counterfactual.[4]
10. Under this definition, an impact evaluation usually employs quantitative methods to measure the net change in outcomes attributable to the project. However, qualitative methods can also be used to infer causality (see EP #5D and Annex A.3). Together, impact evaluation and qualitative methods to establish causality might be called "attribution analysis”. However, the GPS avoids introducing this new term.
11. Note that, in principle, impact evaluation can be applied to any of the levels of outcome in the results chain ("intermediate outcomes” and "final outcomes”). For higher-level outcomes, determining the appropriate counterfactual becomes more difficult because of the greater influence of factors external to the project. For lower-level outcomes (closer to the level of outputs), the counterfactual approaches the "before project” situation. In those cases, the results of an impact evaluation would be the same as the results of a naïve (before-and-after) approach.
[1] Organisation for Economic Co-operation and Development (2002). Glossary of Key Terms in Evaluation and Results Based Management. Paris. The Network of Networks on Impact Evaluation (NONIE) uses the same definition of "impacts”: see Leeuw, Frans and Jos Vaessen (2009), Impact Evaluations and Development: NONIE Guidance on Impact Evaluation.
[2] International Initiative for Impact Evaluation. 3ie Principles for Impact Evaluation. See www.3ieimpact.org.
[3] See Leeuw and Vaessen (op. cit.).
[4] See Guidance Note 1 on attributing outcomes to a project.