No Results Found.
ECG Big Book on Good Practice Standards
Self Evaluation
Good Practice Standards on Self-Evaluation
Background
The three approved GPS on evaluation methods[1] all contained similar discussions of self-evaluation. Accordingly, for this GPS Big Book this topic has been addressed in a separate GPS. According self-evaluation full weight as an important part of the overall evaluation system is consistent with the expressed view and initiatives of members on encouraging management towards greater harmonization of self-evaluation processes that is expected to broaden the coverage and quality of self-evaluation activities.
Jump to the Good Practice Standards (GPS) on Self-Evaluation
Self-evaluation of public sector operations is manifested through the preparation of completion reports (CRs) while in private sector operations it is through the expanded annual supervision reports (XASRs). For country strategy and program (CSP), self-evaluation generally takes three main forms: (i) brief summaries of lessons identified, which are included in a new country strategy; (ii) a country strategy completion report; or (iii) a country strategy progress report or midterm review. Self-evaluations are generally validated by the independent evaluation offices to ensure consistency and to encourage candid and critical evaluation by the operations departments.
Rationale for GPS on self-evaluation
The inclusion of self evaluation in the three approved methodological GPS stemmed from the discussion[2] on the post-Monterrey environment wherein multilateral development banks' (MDB) management adopted the results paradigm in organizing their administrative activities, including planning, results measurement, monitoring and self-evaluation. This was seen as an opportunity for fine-tuning the relationship between self- and independent evaluation. In 2003, members agreed that harmonization in self-evaluation practices, operational policies and processes facilitate the harmonization of independent evaluation. After more than a decade, members continue to recognize the importance of self evaluation to independent evaluation.[3]
Formulation Process
The GPS on self evaluation formed part of the formulation of the three approved GPS and did not have separate formulation processes. Brief description of the formulation processes for each category of operations (i.e., public and private sector operations and CSP) is provided in the preceding chapters.
Organization
The GPS for self-evaluation are presented by category of operations. Standards for self-evaluation of public sector operations cover ensuring evaluability, preparation of CRs, involvement of the central evaluation department (CED) in self-evaluation, and harmonization of self- and independent evaluation. For private sector operations, the standards define the scope of self- or indirect evaluation[4] which includes the executor of the evaluation, and report preparation. And lastly, standards for self-evaluation of CSP are limited to those most critical for the quality of independent evaluation, excluding, for example, topics such as the processing and review of self-evaluation reports.
The GPS for self-evaluation of public and private operations are grouped by standards defined by a number of elements that correspond to the evaluation principles (EPs). Each EP is supported by one or more operational practices (OPs) that describe the policies and procedures that would normally need to be adopted in order to be deemed consistent with the respective EP. The standards for self-evaluation of CSPs are also defined by the corresponding elements which are defined by core GPS. A total of 6 Standards and 10 Elements form the GPS on self evaluation. The summary of the standards and elements are in the Table below while details of EPs and OPs are presented in the next section.
Summary of Standards and Elements on EPs and Number of OPs on Self-Evaluation
| Evaluation Principles | Number of OPs | |
|---|---|---|
| Standards | Elements | |
| A. Public Sector Operations | ||
| 1. Self-Evaluation | A. Ensuring Evaluability | 11 |
| B. Completion Reports | 9 | |
| C. CED Involvement in Self-Evaluation | 1 | |
| D. Harmonization | 1 | |
| No. of Standards: 1 | No. of Elements: 4 | No. of OPs: 22 |
| B. Private Sector Operations | ||
| Planning and Executing a Project Evaluation Program | ||
| 1. Scope of indirect evaluation |
A. Self-Evaluation B. IFI Reporting |
1 |
| C. XASR Research | 1 | |
| D. Transparency | 1 | |
| No. of Standards: 1 | No. of Elements: 4 | No. of OPs: 3 |
| C. Country Strategy and Program (CSP) | ||
| Process-related GPS | ||
| 1. Advance preparations | A. Preparatory Step | 1 |
| 2. Coverage | B. Self-Evaluation Reports | 1 |
| No. of Standards: 2 | No. of Elements: 2 | No. of OPs: 2 |
| Total No. of Standards: 4 | Total No. of Elements: 10 | Total No. of OPs: 27 |
[1] GPS on Evaluation of Public and Private Sector Operations, and Country Strategy and Program Evaluation (CSPE).
[2] Minutes of ECG meeting in Fall 2002.
[3] Minutes of ECG meeting in Spring 2012.
[4] GPS on Evaluation of Private Sector Operations has adopted some flexibility in using the terms "indirect and direct evaluation” to mean self- and independent evaluation respectively, to incorporate the terminology used in the different IFIs (This is based on clarification of the consultant for GPS on Private Sector Operations­).
Good Practice Standards on Self-Evaluation
Public Sector Operations
|
Evaluation Principles (Standards and Elements) |
Operational Practices | Notes |
|---|---|---|
| 1. Self-Evaluation | ||
| A. Ensuring evaluability: International financial institution (IFI) policy requires that project design include a minimum set of elements to ensure evaluability. | 1.1 IFI policy requires that project design include a statement of objectives that is specific, realistic, has temporal characterization, is measurable, agreed upon, and clearly identifies the beneficiaries and those responsible for their achievement. | The outcomes in the objective statement are called "impacts” by some IFIs. |
| 1.2 The statement of objectives is based on a problem or diagnostic statement. | ||
| 1.3 The objectives statement focuses on outcomes for which the project can reasonably be held accountable. It does not encompass objectives beyond the purview of the project, nor is it a restatement of the project's components or outputs. | ||
| 1.4 To the extent that higher-level social and economic objectives and corporate goals (such as the achievement of Millennium Development Goals) are included, they are targeted at segments of the population that can reasonably be expected to be affected by the project, directly or indirectly. | ||
| 1.5 IFI policy requires that project design include a results chain that represents the underlying logic to achieve the objective(s). It shows the links between project activities, outputs, and intended outcomes, and describes the assumptions and risks in causal relationships. | The final level of the results chain is called "impact” by some IFIs. | |
| 1.6 IFI policy requires that project design include a reasonable number of key performance indicators that: are relevant to the project's objectives and span the range of the results chain from inputs to outcomes. | ||
| 1.7 IFI policy requires that performance indicators be clearly defined and measurable, and that they use available data where possible. | ||
| 1.8 IFI policy requires that any performance indicator target values be achievable .within the time-frame of the project. | ||
| 1.9 IFI policy requires that project design include a plan for continuous self- evaluation of key activities throughout the life of the project. | ||
| 1.10 IFI policy requires that data on the project's output and outcome indicators be collected, at a minimum, during project preparation/appraisal, and at project completion. | ||
| 1.11 IFI policy requires that output indicators be collected and monitored continuously throughout project implementation. | ||
| B. Completion Reports (CRs): Operational departments execute CRs in accordance with the IFI's self-evaluation guidelines, and ensure CR quality and timely delivery. | 2.1 Operational departments prepare a CR for every completed operation. | |
|
The CR summarizes the project's contribution to the intended outcomes contained in the project's statement of objectives. It assesses the following (2.2-2.6): 2.2 relevance of project objectives and design; |
For IFIs that define "impact” as the final level in the results chain, the CR summarizes the project's contribution to the intended impacts contained in the project's statement of objectives. | |
| 2.3 the degree to which the project achieved its objectives and delivered outputs as set out in the appraisal report; | ||
| 2.4 the efficiency of the project in converting resources into results; | ||
| 2.5 prospects for the project's sustainability; and | ||
| 2.6 IFI and Borrower performance. | ||
| 2.7 To the extent possible, the CR provides quantitative data to substantiate these assessments. | ||
| 2.8 The CR also identifies key lessons learned related to the achievement of outcomes. | ||
|
2.9 CRs are due within a fixed time (e.g., 6-12 months) after project closure. A longer time frame may be used in cases where outcomes are not observable within that time frame (e.g., some policy-based loans [PBL]). |
||
| C. Central evaluation department (CED) involvement in Self-Evaluation: The CED is involved in the IFI self-evaluation system to support project evaluability and CR quality, but CED involvement is limited to activities that do not compromise the CED's independence. | 3.1 The CED provides training to improve the evaluation capacity of Operations staff. | In addition, the CED may conduct evaluability assessments on projects at entry. This may include (i) preparing ex post evaluability assessments for a sample of projects that have been approved by the Board; and (iii) validating evaluability assessments conducted by Operations units. |
| D. Harmonization. The IFI's self-evaluation and independent evaluation systems are harmonized. | 4.1 The CED coordinates with IFI units responsible for self-evaluation to ensure consistency in evaluation scope, criteria, and rating scales between self-evaluation and independent evaluation. | |
Source: ECG Working Group on Public Sector Operations. GPS on Evaluation of Public Sector Operations. Revised Edition, 2012.
Good Practice Standards on Self-Evaluation
Private Sector Operations
|
Evaluation Principles (Standards and Elements) |
Standard Operational Practices | Element Link | Notes |
|---|---|---|---|
| Private Sector Principles: Planning and Executing a Project Evaluation Program | |||
| 1. Scope of indirect evaluation | |||
|
A. Self-Evaluation: Indirect evaluations are undertaken by operational staff in line with GPS guidance.
B. IFI Reporting: Findings from the indirect evaluations are reported in an Expanded Annual Supervision Report (XASR), which is signed off by operations' management.
C. XASR Research: The XASR is based on internal IFI data, staff consultations, market research and stakeholder meetings.
D. Transparency: The basis for findings are fully transparent in the XASR, including financial / economic calculations and environmental and social effects.
|
1.1 In an indirect evaluation, the project is evaluated by the IFI's operational staff. The scope of evaluation and indicator ratings should be consistent with the GPS.
Staff report their findings in an Expanded Annual Supervision Report (XASR). The XASR is issued only after it has received the approval of the responsible operations department manager. |
Self-Evaluation
IFI Reporting |
Note that the XASR is a once-only addendum to, or "expanded”, Annual Supervision Report. The Annual Supervision Report is the regular supervision report prepared by the IFI's portfolio staff or equivalent. |
| 1.2 The research for XASRs draws from a file review, discussions with other operational staff involved with the operation since its inception, and external market research. The XASR should reflect consultations (in the field as necessary) with stakeholders who are knowledgeable about the country, company and project.* | XASR Research | * Such stakeholders could include: IFI specialists, the company's management, employees, auditors, suppliers, customers, competitors, bankers, any relevant government officials, industry associations, community representatives and local NGOs. | |
|
1.3 The basis for findings and ratings are made fully transparent in the XASR. The XASR should also cite which stakeholder groups were consulted as part of the process. Where ex-post financial and/or economic rates of return for the project are cited, the document includes an attachment providing details supporting these calculations such as the key assumptions and underlying financial / economic time-series data.
The XASR should include a summary of environmental, worker health and safety, and social performance information, for each of the IFI's environmental and social safeguards that apply to the project. Evidence from on-the-ground observations and/or client reporting should be included to support the assigned outcome and IFI work quality ratings. The information can be incorporated as an attachment to the XASR if preferred. |
Transparency | ||
Source: ECG Working Group on Private Sector Evaluation. GPS on Evaluation of Private Sector Operations. 4th Edition, 2011.
Good Practice Standards on Self-Evaluation
Country Strategy and Program
|
GPS Category (Standards and Elements) |
Core GPS Description | Optional GPS Description | Notes |
|---|---|---|---|
| Process-related GPS | |||
| 1. Advance Preparations | |||
| A. Preparatory Step |
1.1 Evaluations of key projects, programs, and technical assistance operations should, if at all possible, be scheduled sufficiently in advance of the preparation of a CSPE. Operations personnel should also be encouraged to prepare self-evaluations in a timely manner |
||
| 2. Coverage | |||
| A. Self-evaluation Reports | 2.1 If self evaluation reports (i.e., country strategy completion reports) are properly done (and independently validated), this may reduce the need for in-depth independent CSPEs, particularly for small borrowers. | It can be difficult, however, for operations personnel to prepare candid and critical evaluations of country assistance performance, particularly in countries whose development results lag far behind what was expected. | |
Source: ECG. GPS on Country Strategy and Program Evaluations. 2008. Manila