ECG Good Practice Standards
Annex V.1: Evaluation Cooperation Group Progress Benchmarking Template – Country Strategy and Program Evaluation GPS
Published by: Independent Evaluation Group (IEG), World Bank Group
Annex V.1: Evaluation Cooperation Group Progress Benchmarking Template – Country Strategy and Program Evaluation GPS
ECG members agree that periodic assessments will be undertaken to assess the extent to which the GPS are being applied. The GPS have been summarized in tabular form below to assist each participating MDBs in progress benchmarking.[1] Some time will be required to adjust member practices to GPS, and thereafter to conduct at least one CSPE under the new GPS' regime. Benchmarking of MDBs against the CSPE GPS is under consideration.
|
(GPS Category |
Core GPS Description |
Degree of MDB Alignment[i] |
Optional GPS Description |
Degree of MDB Alignment |
Remarks[ii] |
|
A. Process-Related GPSs |
|||||
|
A.1. CSPE Goals, Objectives, Client Responsiveness, and Unit of Analysis |
(a) Provide credible and useful information on the MDB's performance at the country level (b) Used for both accountability and lesson-learning purposes (c) Designed to meet information requirements of main target clients (d) Focus on evaluating the results of the MDB's assistance, with the country strategy(ies) as the main reference point |
|
|
|
|
|
A.2. Country Selection and Mutual Accountability |
(a) Countries selected are those in which the findings and lessons will be most beneficial to the MDB and the country. (b) Efforts made to reduce potential bottlenecks in undertaking joint MDB CSPEs (c) Decision to pursue a multipartner CSPE made on a case-by-case basis |
|
(a) Covering all countries and treating all borrowers equally (b) Multipartner CSPEs extending beyond the MDBs to include all sources of external assistance to a country encouraged |
|
|
|
A.3. Timing |
(a) CSPE timed to feed into the preparation and review of the MDB's new country strategy |
|
(a) Could also be timed to contribute to strategic decision making of the government |
|
|
|
A.4. Preparatory Steps |
(a) Evaluations of key projects, programs, and technical assistance scheduled to precede the CSPE |
|
(a) Sector/thematic studies or impact assessments scheduled to precede a CSPE. (b) Application of the same evaluation criteria in sector/thematic studies as in the CSPE facilitates their use. |
|
|
|
A.5. Coverage |
(a) Coverage long enough to see results, but more emphasis put on the current strategy period (b) Newly initiated, completed, and ongoing operations covered (c) Full content of the MDB's assistance covered (d) Depth of coverage depends on client needs and those areas most likely to evoke lessons for future strategy (e) For second- or third-generation CSPEs, previous CSPE findings summarized, and use of previous CSPE lessons and recommendations assessed (f) Subsequent CSPEs will have an overlap in the period covered of a few years. (g) CSPEs may have limited scope if the MDB's role is minor, if there were few results, or if there is little likelihood of findings and lessons of broader impact. (h) Completion reports of country strategies independently validated; if the completion and validation reports are comprehensive and apply CSPE criteria, they may serve as a limited-scope CSPE. |
|
(a)In large country cases, a representative sample of assistance activities assessed (b) A limited-scope CSPE may also be needed to deliver evaluation findings to meet tight time-sensitive demands. |
|
|
|
A.6. CSPE Approach Paper |
(a) A CSPE approach (or position) paper prepared for each CSPE |
|
|
|
|
|
A.7. CSPE Preparation Period |
(a) A full CSPE implemented over |
|
|
|
|
|
A.8. Staffing |
(a) CSPE teams headed by an experienced evaluator with sufficient experience in MDB operations |
|
(a) A multidisciplinary team engaged to undertake the CSPE |
|
|
|
A.9. Guidelines |
(a) Each MDB will have CSPE guidelines that set out CSPE goals and objectives, methods, evaluative criteria, evaluation questions, procedures, reporting formats, quality control processes, and outreach and dissemination arrangements. (b) Quality control procedures will ensure that guidelines are followed. (c) While guidelines will be adhered to, the actual methods, scope, and approach may be tailored to the country setting. |
|
|
|
|
|
B. Methodology-Related GPS |
|||||
|
B.1. CSPE Methods and Approaches |
|||||
|
Overview |
(a) CSPE methods include steps to make the causal model explicit in the country strategy, analysis of country context, assessment of the validity of the MDB's diagnosis, and analysis of the strategy and program relevance in design and delivery. (b) Top-down, bottom-up, and attribution-cum-contribution assessments used to assemble information on performance in achieving strategic objectives (c) Evidence base analyzed to identify performance determinants (d) Evaluation criteria applied to assess performance in multiple dimensions (e) Findings and lessons drawn, and future-oriented recommendations provided |
|
|
|
|
|
|
(f) Methods explained in the CSPE report |
|
|
|
|
|
Evaluation Questions |
(a) General and country-specific evaluation questions posed to guide the assessment (b) Evaluation questions documented in the CSPE report |
|
|
|
|
|
Counterfactuals |
|
|
(a) Counterfactuals should be used only when they are possible and defensible. (b) Counterfactuals can be proxied through comparisons with similar countries, examination of those parts of the program for which a counterfactual can be more clearly identified, or for those parts of the program for which prior impact evaluations have been conducted. |
|
|
|
Attribution and Contribution |
(a) Since formal attribution is difficult to determine, assessment of program results will focus on determining whether the MDB has made a contribution to key results or outcomes, and identifying the main drivers of the outcomes.
|
|
(a) To characterize the nature of the MDB's contribution to results, the extent to which its assistance delivered additional value beyond the financing provided will be assessed. |
|
|
|
Evaluability |
(a) CSPE includes an assessment of the evaluability of the MDB's strategy and program of assistance. (b) Evaluability constraints overcome by reconstructing the program logic, retrofitting results frameworks, drawing on available information sources, and collecting performance information |
|
|
|
|
|
Multiple Evidence Sources |
(a) CSPE draws on the widest possible breadth of primary and secondary sources of information, and bases findings on information that has been successfully validated from multiple sources. |
|
(a) Use of client perception surveys can provide evidence about the MDB's performance. |
|
|
|
Client Participation |
(a) Participation of key stakeholders in the CSPE process encouraged |
|
|
|
|
|
Disclaimers |
(a) Limitations of the methodology and its application frankly acknowledged in the CSPE report |
|
|
|
|
|
B.2. CSPE Evaluation Criteria |
(a) Relevance, efficiency, effectiveness, sustainability, and impact considered mandatory criteria |
|
(a) Positioning, coherence, institutional development, borrower and MDB performance, and partner coordination considered optional criteria |
|
|
|
Relevance, Coherence, and Positioning |
(a) Relevance examined by assessing if the MDB's strategy and assistance program were consistent with the country context and the government's strategic priorities |
|
(a) The MDB's processes used to maintain relevance assessed (b) Criteria such as positioning and coherence used to assess the degree to which the design of the strategy and program harnesses positive synergies and builds on the MDB's core competence |
|
|
|
Efficiency |
(a) Efficiency assessed using indicators affecting cost-effectiveness, transaction costs, portfolio performance, monitoring and evaluation arrangements, and other project/program implementation |
|
|
|
|
|
Effectiveness |
(a) Extent to which strategic outcomes were achieved and sufficient development progress was made used to assess program effectiveness (b) Determinants of performance in achieving the MDB's objectives identified |
|
(a) The MDB's contribution to broader corporate objectives assessed, but distinction drawn between those |
|
|
|
|
|
|
thematic issues whose coverage is mandatory and those whose coverage is optional |
|
|
|
Sustainability |
(a) The degree to which the results of the MDB's assistance are likely to be sustained after the conclusion of the program assessed |
|
|
|
|
|
Impact and Institutional Development |
(a) Impact assessed relative to national goals and to |
|
(a) Extent to which the MDB has helped to develop institutional capacity separately assessed if not part of impact assessment |
|
|
|
Borrower Performance |
|
|
(a) Borrower performance, particularly the degree of program ownership, assessed but not formally rated |
|
|
|
MDB Performance |
|
|
(a) The MDB's performance formally assessed, including its responsiveness to client needs |
|
|
|
Partnership and Harmonization |
|
|
(a) CSPE examines the extent to which the MDB has been an effective partner in a multistakeholder development assistance effort.
|
|
|
|
B.3. Performance Rating |
|||||
|
Ratings Principles and Comparability |
(a) If quantitative rating is undertaken, the rating system should use well-defined criteria and be as simple as possible. (b) The manner in which ratings are derived is stated in the report. (c) Limitations of the rating system are acknowledged . (d) Rating system is uniform.
|
|
(a) A quantitative rating system is used to make the assessment process transparent and uniform across countries. |
|
|
|
Rating Criteria |
(a) If a quantitative rating is undertaken, ratings of the mandatory evaluation criteria (relevance, efficiency, effectiveness, sustainability, and impact) needed (b) If a quantitative rating is undertaken, ratings accorded for each criterion presented separately to make the performance assessment transparent |
|
(a) If a quantitative rating is undertaken, ratings of the additional evaluation criteria (positioning, coherence, institutional development, borrower and MDB performance, and partner coordination) considered optional
|
|
|
|
Rating Subcriteria |
|
|
(a) Defining subcriteria, if any, in a way that is applicable to specific country cases can help to provide an evaluative framework for more uniform, systematic, and comparable assessments. |
|
|
|
Weighting Criteria |
|
|
(a) If an overall rating is generated, more weight accorded to effectiveness, impact, and sustainability |
|
|
|
C. Reporting-Related GPSs |
|||||
|
C.1. Findings, Lessons, and Recommendations |
(a) Evaluation findings are country-specific, evidence-based, and follow from the evaluation questions. (b) Lessons are few in number and evidence-rooted, and have operational implications. (c) Recommendations are few in number, constructive, actionable, strategic, operational, and not obvious. |
|
|
|
|
|
C.2. Reporting and Review |
(a) Uniform formats followed with latitude to tailor to the country case (b) Report covers country context, country strategy, program implementation, program outcomes and impacts, partnerships, thematic issues, lessons, and recommendations. (c) Report presented in plain language and covers those issues that could be conclusively evaluated. (d) Draft report and supporting studies rigorously reviewed internally and externally. |
|
(a) Where there are substantive disagreements during the review process, these will be reflected in the final CSPE report. |
|
|
|
C.3. Making Findings Accessible |
(a) CSPE findings published (b) To spotlight the diversity with which CSPE findings can be interpreted, CSPE publications will include formal views of management, government, and the board.
|
|
(a) Outreach events may be held to boost the dissemination of CSPE findings. (b) A précis or other summary publication may be issued and findings translated into the local language to make CSPE findings more accessible. |
|
|
|
C.4. Generalizing Findings and Tracking Recommendations |
(a) Annual and/or biannual reviews of evaluation findings summarize and compare CSPE findings |
|
(a) Recommendation tracking systems or periodic reviews of the utilization of CSPE findings and recommendations prepared to track CSPE use |
|
|
CSPE = country strategy and program evaluation, GPS = good practice standard, MDB = multilateral development bank.
[1] A baseline of CSPE practices for AfDB, AsDB, IADB, and WBG is included in the self-assessment questionnaires provided in Tabor, Steven and Suganya Hutaserani. 2007. Phase I Background Report for the Preparation of GPS for CSPEs.
[i] Alignment refers to the extent to which the MDB evaluation practice is fully, partly, or not harmonized with the relevant GPSs.
[ii] The remarks section may be used to explain the reasons for divergence between the GPSs and MDB practice.
Source: GPS on CSPE, 2008. Manila. Appendix 1.