No Results Found.
ECG Big Book on Good Practice Standards
Private Sector Operations
Good Practice Standards on Evaluation of Private Sector Operations
Background
Formulation Process
The Evaluation Cooperation Group's Good Practice Standards for Evaluation of Private Sector Investment Operations (ECG-GPS) were originally formulated in response to the call for harmonization of evaluation methodologies by the Development Committee Task Force in 1996. In 2001, the ECG issued the first edition of the GPS, followed by second and third editions in 2003 and 2006 respectively. Each subsequent edition was informed by the findings and recommendations of a benchmarking exercise, which assessed members' practices against the GPS. Following the 2010 benchmarking exercise against the third edition of the GPS, the ECG has issued a fourth edition.
Jump to the Good Practice Standards (GPS) on Private Sector Operations Evaluation
Generic Principles
Private Sector Principles
The fourth edition builds and improves upon the previous GPS to reflect the evolution in evaluation practices and in the scope of investment operations undertaken by International Financial Institutions (IFIs). It takes into account feedback from the benchmarking exercise, and the experiences of members in implementing the previous set of standards. In particular, the fourth edition addresses the following issues identified with the GPS Third Edition:
some standards were too narrowly defined and did not recognize the variation in IFI mandates and operational procedures, particularly given the expanding membership of the ECG;
the GPS did not differentiate between standards that could be implemented unilaterally by the evaluation departments and those that relied in part or wholly on management action or cooperation;
some standards – the experimental standards in particular – were overly complicated both in design and in their implementation;
there was unnecessary overlap between different standards, which created ambiguity and/or duplication;
the GPS comprised a range of different types of standard (harmonization, other, good practice, best practice, and not universally applicable), which complicated both interpretation and periodic benchmarking;
the definition of rating benchmarks was in certain cases too complex and went beyond the objective of harmonization;
the GPS were narrowly focused on financial and non-financial sector investments (mainly project finance) and had limited relevance to the increased variety in investment operations (e.g., equity funds, working capital facilities, corporate finance, trade finance, political risk insurance etc.); and
there was scope for the GPS to promote more innovation in evaluation and dissemination through, for example, web-based media.
Objectives and Organization
Although the premise of the Task Force's 1996 decree was to enable comparability of results, the guidance stressed other objectives including identifying and disseminating best practices in evaluation, sharing lessons from evaluations, and describing results in a common language. The decree also acknowledged that harmonization efforts should take into account the differing circumstances of each institution. The GPS Fourth Edition is responsive to these wider objectives.
These GPS are organized into generic standards on reporting and corporate learning, evaluation guidance and rating systems that apply across all IFIs, and standards specific to IFIs supporting private sector investment.[1] The standards themselves are formulated as EPs. Each EP is defined in terms of its key components or "Elements”. To guide IFIs in their efforts towards meeting the Elements of the EPs, each EP is supported by a set of standard OPs. The OPs describe the policies and procedures that the IFI would typically need to adopt in order to be deemed compliant with the respective EP.
The EPs on evaluation of private sector operations comprise a total of 17 standards and 75 elements. The OPs include 63 policies and procedures.[2] The summary of the EPs, in terms of standards and elements, and OPs is presented below in a simplified and more logically consistent framework, as follows:
Summary of Standards and Elements on EPs and Number of OPs on Evaluation of Private Sector Operations
| Evaluation Principles | No. of OPs | |
|---|---|---|
| Standards | Elements | |
| Generic Principles: Reporting and corporate learning Concern the scope and timing of periodic reporting, the disclosure of evaluation reports and products, and the capture, dissemination and application of lessons from the evaluation system. |
||
| 1. Annual reporting of corporate results |
A. Corporate Results B. Reporting Rating Results C. Analysis D. Recommendations E. Disclosure |
5 |
|
2. Periodic reporting on evaluation systems
|
A. Periodic Reporting B. Quality Efficacy C. Alignment D. Evaluability E. Lessons Application |
5 |
|
3. Lessons and findings from evaluation
|
A. Coverage B. Relevance C. Accessibility |
3 |
| Generic Principles: Evaluation guidance and rating systems Concern the features of an evaluation rating system, and the preparation and dissemination of evaluation guidance. |
||
| 4. Guidance for project evaluation |
A. Preparation B. Content C. Dissemination |
2 |
| 5. Performance rating scales |
A. Range and Balance B. Descriptive C. Binary Reporting |
3 |
|
Private Sector Principles: Planning and executing a project evaluation program Concern the determination of when an operation is ready for evaluation, the selection of a sample from a defined population, and the process of direct evaluation and self-evaluation with independent verification. |
||
| 6. Defining the population of projects for evaluation |
A. Coherence and Objectiveness B. Qualifying Projects C. Screening D. Non-Qualifying Projects E. Exclusions F. Disclosure |
6 |
| 7. Selecting a sample of projects for evaluation |
A. Representative Random Sampling B. Sample Aggregation C. Disclosure D. Purposeful Sampling |
4 |
| 8. Process of direct evaluation by CED |
A. CED's Options B. Reporting C. Desk-Based D. In-Depth E. Transparency F. Review Process |
4 |
| 9. Scope of Independent verification by the CED |
A. Verification B. In-Depth Verification C. CED Reporting D. Review Process |
3 |
|
Private Sector Principles: Evaluation metrics and benchmarks Define the scope of measurement and benchmarks for rating each performance indicator within the evaluation framework. |
||
| 10. Rating project outcomes |
A. Synthesis rating B. Benchmark C. Financial Criteria D. Economic Criteria E. IFI Mandate Criteria F. E&S Criteria |
2 |
| 11. Outcome Indicator 1 - Financial performance and fulfilment of project business objectives |
A. Stakeholder Analysis B. Time Span C. Fulfilment of Project Business Objectives D. Methods E. Benchmarks |
5 |
| 12. Outcome Indicator 2 – Economic sustainability |
A. Stakeholder Analysis B. Time Span C. Net Benefits D. Methods E. Benchmarks |
5 |
| 13. Outcome Indicator 3 – Contribution to IFI mandate objectives |
A. Method B. Balanced C. Benchmark |
2 |
| 14. Outcome Indicator 4 – Environmental and social performance |
A. E&S Performance B. E&S Capacity C. Sub-Project Performance D. Benchmark |
3 |
| 15. Rating the IFI's investment profitability |
A. Scope B. Net Method C. Gross Proxy Method D. Benchmark |
5 |
| 16. Rating IFI work quality / bank handling |
A. Scope B. Stand-alone C. Pre-Commitment D. Post-Commitment E. Benchmark |
4 |
| 17. Rating the IFI additionality |
A. Counterfactual B. Financial Additionality C. Non-Financial Additionality D. Benchmark |
2 |
| Total No. of Standards: 17 | Total No. of Elements: 75 | Total No. of OPs: 63 |
Where appropriate, these GPS make reference to a project typology (Annex IV.2) so that OPs can be tailored to the wider range of projects and clients now supported by IFIs. The project typology is particularly important when defining early operating maturity, and rating the indicators of project business success, economic sustainability, and environmental and social performance. For projects exhibiting a mix of project types, it is recommended that IFIs use a range of metrics as appropriate.
[1] The generic standards on independence of evaluation departments were excluded from the GPS on Evaluation of Private Sector Operations as these already constitute Chapter II of the Big Book which contain GPS on Independence of IFIs' CED (June 2010). The same treatment was done on specific standards on indirect evaluation (otherwise referred to as self-evaluation) which are presented in Chapter VI on Self-Evaluation. Relatedly, GPS on Evaluation of Private Sector Operations use the terms "direct and indirect” evaluation to refer to self- and independent evaluation, respectively, to acknowledge the differences in terminology used in the different IFIs which reflect the nature of CED's interface with an actual project (based on clarification from consultant of ECG WGPSE.).
[2] GPS on Evaluation of Private Sector Operations, Fourth Edition is a refined and simplified version of GPS 3rd Edition where multi-faceted standards were disaggregated into their individual components resulting in a total of 22 EPs (or standards), each defined by between three and eight elements, making 101 Elements in total. The EPs and their Elements together cover much the same scope as GPS 3rd Edition.
Good Practice Standards on Evaluation of Private Sector Operations
Generic Principles: Reporting and corporate learning
|
Evaluation Principle (Standards and Elements) |
Standard Operational Practices | Element Link | Notes |
|---|---|---|---|
| 1. Annual reporting of corporate results | |||
|
A. Corporate Results: The central evaluation department (CED) reports to the Board annually on the IFI's independently verified outcome results. B. Reporting Rating Results: The CED reports the IFI's results in all rating dimensions and indicators. C. Analysis: The CED analyses the results to discern performance drivers. D. Recommendations: The CED formulates recommendations based on the findings. E. Disclosure: The CED discloses its synthesis evaluation results externally. |
On an annual basis, the CED reports to the governing Board on the international financial institution's (IFI) corporate-wide performance, based on the findings from project-level evaluations and, if required, thematic evaluations.* The report can be stand-alone or incorporated in other reports to the IFI's Board. The ratings reported should be those independently verified or directly assigned by the CED. | Corporate Results | * Thematic evaluations could include country, sector or other studies of the aggregate results across a defined group of projects. |
| For each rating dimension and indicator, the CED reports the number and proportion (by number of operations) of the evaluated cohort in each performance-rating category. | Reporting Rating Results | The CED may also choose to report the results weighted by project or investment size, to indicate the quantum of impact. | |
| The CED provides a synthesis description of the ratings patterns and their cross-cutting performance drivers under each indicator. It also provides the dimension and indicator ratings for the previous few years or cohorts thereof (where such data exists) to show how performance is evolving over time. | Analysis | ||
| Where feasible, the CED makes recommendations to the IFI's Management based on the evaluation findings. | Recommendations | The qualification here allows for an exception in the case of CEDs with insufficient evaluated projects to substantiate recommendations. | |
| The CED publishes its findings after appropriate redaction to protect commercial confidentiality, and posts on a webpage accessible via the IFI's external website the full text or an abstract of its report that accurately summarizes its essential findings. | Disclosure | The webpage can be on the CED's own site, provided that the CED's site can be accessed via a link on the IFI's main pages. | |
| 2. Periodic reporting on evaluation systems | |||
|
A. Periodic Reporting: At least once every three years, the CED reports on aspects of the IFI's evaluation systems, including: B. Quality & Efficacy: The CED reports to the Board on the quality and efficacy of evaluation systems. C. Alignment: The CED reviews and reports on the alignment of Management reporting systems with the evaluation framework. D. Evaluability: The CED reviews and reports on the evaluability of the IFI's operations. E. Lessons Application: The CED reviews and reports on the application of lessons learned from evaluation. |
The CED reports to the Board at least once every three years on the functioning and effectiveness of the IFI's evaluation systems, as detailed below. The report can be stand-alone or incorporated in other reports to the Board. | Periodic Reporting | The review of evaluation systems in the IFI could be undertaken by the CED directly, or by an external independent body under commission from the CED. |
| The CED reviews and reports on the quality and efficacy of the IFI's evaluation systems. As part of this reporting, the CED submits to the IFI's Management and Board the periodic benchmarking reviews of the consistency of the IFI's practices with the ECG Good Practice Standards (or provides a summary thereof). | Quality & Efficacy | ||
| The CED reviews and reports the extent to which internal Management and corporate reports (up to Board level) are broadly aligned with the evaluative framework. For example, the CED should review: (i) to what extent the IFI applies coherent and consistent benchmarks to gauge project performance at relevant stages of the project cycle; and (ii) whether Management's reporting of results includes project outcome and additionality ratings based on the ECG GPS. | Alignment | ||
| The CED assesses and reports on the evaluability of the IFI's operations i.e., the extent to which the value generated or the expected results of a project are verifiable in a reliable and credible fashion. In practical terms, the CED should assess whether the IFI had specified relevant indicators at approval and made sufficient provision to collect the data required for monitoring during project supervision. The CED need not report on every operation, or undertake such reviews at the time of project approval. | Evaluability | ||
| The CED assesses and reports evidence of the extent to which lessons of experience are being applied in new operations. It is not required that the CED report on every operation individually, or undertake such reviews at the time of project approval. | Lessons Application | Examples of methodologies for such an assessment include surveys or interviews of origination staff, or a CED review of appraisal documents. | |
| 3. Lessons and findings from evaluation | |||
|
A. Coverage: Lessons of experience are identified for all project-level evaluations. B. Relevance: Lessons are relevant to new operations. C. Accessibility: Lessons and evaluation findings are made readily available to IFI staff. |
All direct and indirect project-level evaluation reports should contain a prompt or template for the author(s) to identify and articulate one or more lessons from the operation. | Coverage | |
| Lessons should be concise, prescriptive, and placed in the context of a material issue that was encountered in the evaluation so that its relevance to new operations can be determined easily, on a stand-alone basis. The point of view and selectivity should focus on what the IFI might have done to obtain better results from the operation. | Relevance | ||
|
The CED maintains a database or library of operational lessons from project-level evaluation reports, which is freely accessible to IFI staff. Alternatively, the CED contributes lessons from project-level evaluations (or a summary thereof) to a database maintained by IFI Management.
The CED makes available to IFI staff a range of easily accessible dissemination products covering evaluation findings from projects and/or synthesis CED reports. This could include, inter alia, access to the full reports, electronic notification of new items, and presentations of findings. |
Accessibility | ||
Good Practice Standards on Evaluation of Private Sector Operations
Generic Principles: Evaluation guidance and rating systems
|
Evaluation Principle (Standards and Elements) |
Standard Operational Practices | Linked to | Notes |
|---|---|---|---|
| 4. Guidance for project evaluation | |||
|
A. Preparation: The central evaluation department (CED) develops guidance for staff undertaking direct and indirect project evaluations. B. Content: Guidance is self-standing, current, and comprehensive in key aspects of the evaluation process. C. Dissemination: Guidance is easily accessible and supplemented by training and/or good practice examples. |
The CED develops, in conjunction with Management as necessary, guidance for CED and operational staff undertaking direct and indirect project evaluations. The evaluation guidelines should be consistent with prevailing ECG Good Practice Standards and at a minimum include: (i) the key steps in the evaluation process, in the preparation and signing-off of reports, and in independent verification by the CED as necessary; (ii) the scope of measurement and the benchmarks for assigning ratings for each performance indicator and dimension; and (iii) standard reporting templates that include a performance ratings matrix. |
Preparation
Content |
Where separate guidance is prepared for self-evaluations and independent direct evaluations and/or verifications, these should be completely coherent in terms of the prescribed metrics and benchmarks. |
| The CED makes the evaluation guidelines and supporting information readily available on its website and/or the international financial institution's (IFI) website in respect of guidance for self-evaluation. The CED undertakes dissemination activities to familiarize staff preparing project evaluations with the requirements and supporting documentation. This may include the showcasing of evaluation reports regarded as good-practice examples. | Dissemination | ||
| 5. Performance rating scales | |||
|
A. Range & Balance: Each indicator is rated on a performance scale from most negative to most positive, with the scale balanced between positive and negative ratings. B. Descriptive: Each rating category accurately describes the extent of positive or negative performance. C. Binary Reporting: Binary ratings use the first positive rating within the performance scale as their benchmark. |
The rating scale for each indicator should encompass performance ranging from the most negative to most positive. There should be balance between positive and negative characterizations (i.e., if there are four ratings, two are less than good and two are good or better; or if there are six ratings, three are less than good and three are good or better). | Range & Balance | |
|
The words used to describe these ratings should accurately reflect whether the judgments are less than good or else good or better, and should clearly reflect the graduation from worst to best. For example: Four-point scale: unsatisfactory, partly (un)satisfactory, satisfactory, excellent; or unsuccessful; partly (un)successful; successful; highly successful. Six-point scale: highly unsuccessful, unsuccessful, mostly unsuccessful; mostly successful; successful; highly successful; or highly unsatisfactory; unsatisfactory; marginal; satisfactory; good; excellent. |
Descriptive | ||
| Where the CED reports success rates based on a binary simplification of the rating scale, the binary benchmark should be the first positive rating within the chosen scale i.e., a satisfactory or successful rating (in the case of the four-point scale cited above) or a mostly successful or satisfactory rating (in the case of the six-point scale cited above). | Binary Reporting | ||
Good Practice Standards on Evaluation of Private Sector Operations
Private Sector Principles: Planning and executing a project evaluation program
|
Evaluation Principle (Standards and Elements) |
Standard Operational Practices | Linked to | Notes |
|---|---|---|---|
| 6. Defining the population of projects for evaluation | |||
|
A. Coherence & Objectiveness: All projects in the population share common characteristics based on a coherent set of criteria. B. Qualifying Projects: The population includes all projects that have reached early operating maturity (or are unlikely ever to do so), and all closed projects. C. Screening: The CED determines projects' early operating maturity according to GPS criteria. D. Non-Qualifying Projects: Projects that are not operationally mature are reconsidered in subsequent years. E. Exclusions: The population may exclude other classes of projects where the CED determines that individual evaluations have limited utility. F. Disclosure: The CED discloses its criteria for defining the population and any excluded class of project. |
The central evaluation department (CED) defines the population of projects according to a coherent and objective set of criteria appropriate to the type of report. The full application of these criteria will determine whether or not an operation is to be included in the population. For example: Corporate reporting: the population should comprise projects with the same year (or defined range of years) of origin, based on the approval, commitment or disbursement date for the international financial institution's (IFI) associated investment. Alternatively, the population should comprise projects that have reached early operating maturity within a defined timeframe.* Part-portfolio reporting: the population should comprise projects sharing a common time-basis (as above), and the same country, region, sector or other thematic characteristics as desired. |
Coherence & Objectiveness |
While this operational practice (OP) allows CEDs to define a sub-portfolio within the overall corporate portfolio, evaluation principle (EP)6 continues to call for annual reporting of corporate-level results. The onus therefore rests with the CED to determine how best to comply with EP6 should it opt for sub-portfolio sampling in respect of this OP.
* This alternative approach is appropriate for CEDs that screen the entire portfolio every year to determine which projects have reached early operating maturity. |
|
Projects should be included in a designated population only once and only at such time as (but not necessarily as soon as) they have reached early operating maturity.* The population (the boundaries of which are defined under OP11.1) also includes all closed projects (i.e., where the associated IFI investment has been repaid, sold or written off, or the guarantee has been cancelled) regardless of whether or not they had reached early operating maturity by the time of closing. The population should also include projects that are deemed unlikely ever to achieve early operating maturity.** |
Qualifying Projects |
* This does not prohibit projects from being included in different populations relating to different studies, for example if the CED was to undertake both a corporate and country-level evaluation. ** This could include, for example, a project that has failed or stagnated such that it is unlikely ever to establish a trading record, yet the IFI's investment has not been sold, cancelled or written off and so has not been officially closed. In such cases, there is little value in postponing evaluation, hence they should be included in the current population. |
|
| The CED establishes which projects have reached early operating maturity, taking into consideration information on project status provided by operational departments and by applying the guidance in Annex IV.3 : Lookup Table for Determining Early Operating Maturity. | Screening | ||
|
Where the CED determines that projects have not yet reached early operating maturity (but are likely to do so in the future), they should be omitted from the current evaluation year's population. Instead, the CED should consider them for inclusion in the population in a future year when they will have reached early operating maturity. In cases where the IFI is involved in litigation, foreclosure or other legal process where evaluation could prejudice the IFI's legal position, the CED may choose to omit these projects from the current population and instead roll them forward for consideration in a future year. |
Non-Qualifying Projects | Rolling projects forward for consideration in future years' populations may render them incongruous with the year(s) of origin of those later populations. Where the CED reports an aggregation of several years of consecutive evaluation findings, such projects might legitimately be included. Otherwise, the CED should disclose the incidence of older projects in the reported population, or report their results separately. | |
|
The CED may choose to exclude altogether from the population the following classes of project: (i) those that did not proceed with IFI support and where the associated IFI investments were dropped or guarantees never signed, activated or utilized*; or (ii) those involving subscribed rights offerings or follow-up investments / guarantees undertaken for substantially the same purpose as before (e.g., to help finance cost overruns or restructurings).** |
Exclusions |
* CED may exclude projects where the IFI never incurred any exposure under the guarantee (e.g., because the beneficiary of the guarantee never made any advances to the end client).
** Repeat investments in an existing client company should normally be included in the population, unless they meet the criteria in (6). |
|
| The CED discloses how it defined the population and its criteria for including or excluding projects in line with the operational practices above. | Disclosure | ||
| 7. Selecting a sample of projects for evaluation | |||
|
A. Representative Random Sampling: Either all projects in the population are evaluated or the CED selects a random sample whose characteristics are representative of the population. B. Sample Aggregation: The CED reports the results of one or more years of evaluated random samples. C. Disclosure: The CED discloses its sampling methodology, how it defined the reported cohort, and sampling errors in reported results. D. Purposeful Sampling: The CED may self-select a purposeful sample to serve specific evaluative needs, but not for overall corporate reporting purposes. |
If evaluation coverage is less than 100%, the CED should select a random sample of projects for evaluation from the established population. The sample should be as representative as practicable insofar as it reflects the distribution of important characteristics throughout the population as relevant to each institution.* | Representative Random Sampling | * Relevant characteristics for testing the representativeness of the sample could include: industry sector; country; region; project size; investment size; IFI instrument of support; incidence of loan impairment or equity write-down. |
| For reporting purposes, the CED may report the results of a sample of projects evaluated in one year, or use a cohort comprising the evaluated samples from several consecutive years in order to increase the granularity of data and its statistical significance. In deciding how many years of data to combine, the CED should balance the desire to report on a meaningful number of observations against the currency of findings, particularly if using evaluation data more than three years old. | Sample Aggregation |
||
|
Where sampling is used, the CED should report details of the sampling technique used and the extent to which the sample's characteristics reflect those of the population. When reporting the aggregate results of a cohort comprising samples from more than one evaluation year, the CED should disclose how the reported cohort is defined. |
Disclosure | ||
| The CED should calculate and disclose the sampling errors (at the 95% confidence interval) in the reported success rates for each of the evaluated indicators and outcome ratings.* | * Disclosure of sampling errors enables observers to judge the relevance, usefulness and comparability of success rates reported by different institutions. | ||
| The CED may select a purposeful (self-selected) sample of projects to be evaluated.* The CED should not use the results of purposeful evaluations for overall corporate reporting purposes, unless projects in the purposeful sample are also selected as part of a random sample as determined under OP 7.1. | Purposeful Sampling | * Reasons for selecting a purposeful sample could include: the potential for learning; the high profile of an operation; credit and other risks; the sector is a new one for the IFI; the likelihood of replication; or the desirability of balanced country and sector coverage. | |
| 8. Process of direct evaluation by the CED | |||
|
A. CED's Options: At its own discretion, the CED can select projects on which to conduct its own direct evaluations. G. Reporting: The CED conveys its findings in a Performance Evaluation Report (PER). H. Desk-Based: As a minimum, the PER is based on internal IFI data, staff consultations and market research. I. In-Depth: For selected projects, the CED conducts on-the-ground research and stakeholder consultations. J. Transparency: The basis for the CED's findings are fully transparent in the PER, including financial / economic calculations and environmental and social effects. K. Review Process: Management and staff have the opportunity to comment on the draft PER, but the final assessment is determined solely by the CED. |
The CED can undertake a direct evaluation of a project on its own volition, acting with consideration to on-going legal process in line with OP 6.4 . The scope of evaluation and indicator ratings should be consistent with the GPS. The CED reports its findings in a Performance Evaluation Report (PER). |
CED's Options
Reporting |
|
| As a minimum, the research for PERs draws from a file review, discussions with available staff involved with the operation since its inception, and external market research. | Desk-Based | ||
| On a more rigorous basis, the CED may choose to conduct in-depth research (in the field as necessary) for the PER, based on consultations with stakeholders who are knowledgeable about the country, company and project.* | In-Depth | * Such stakeholders could include: IFI specialists, the company's management, employees, auditors, suppliers, customers, competitors, bankers, any relevant government officials, industry associations, community representatives and local NGOs. | |
|
The basis for the CED's findings and ratings are made fully transparent in the PER. The PER should also cite which stakeholder groups were consulted as part of the process. Where ex-post financial and/or economic rates of return for the project are cited in the PER, the document includes an attachment providing details supporting these calculations such as the key assumptions and underlying financial / economic time-series data. The PER includes a summary of environmental, worker health and safety, and social performance information, for each of the IFI's environmental and social safeguards that apply to the project. Evidence from on-the-ground observations and/or client reporting should be sufficient to support the assigned outcome and IFI work quality ratings. The information can be incorporated as an attachment to the PER if preferred. |
Transparency | ||
| The CED provides an opportunity to Management and operational staff to review and comment on the PER's draft findings, though the final content and ratings in the report remain the decision of the CED. Findings from the PER can be used in synthesis reporting without further verification. | Review Process | ||
| 9. Scope of independent verification by the CED[1] | |||
|
A. Verification: The CED conducts an independent review of expanded annual supervision reports (XASR) based on internal IFI data and independent research. B. In-Depth Verification: The CED conducts detailed verifications for selected projects. C. CED Reporting: The CED reports its independent findings in an XASR-Assessment (XASR-A), which records any rating differences to those in the XASR. D. Review Process: Management and staff have the opportunity to comment on the draft XASR-A, but the final content is determined by the CED. |
The CED conducts an independent review (which may be desk-based) of the XASR to verify its scope, responsiveness, evident reliability of the analysis, impartiality and consistency in ratings judgments, and appropriateness and completeness of the identified lessons. As a minimum, the independent review draws from a file review, discussions with available staff involved with the operation since its inception, and external market research. Depending on the coverage of the population by XASRs, either: (a) If the IFI has prepared XASRs for a representative sample selected in accordance with EP 7 , then the CED should conduct independent reviews for all the XASRs in such sample; or (b) If the IFI has prepared XASRs for all projects in the population of operationally mature projects (as defined in accordance with EP 6), then the CED may choose to conduct independent reviews either for all XASRs or for a representative sample of XASRs selected in accordance with EP 7. If a sample is preferred, only the ratings from CED-verified XASRs are valid for corporate reporting purposes. |
Verification | |
|
On a more rigorous basis, the CED conducts detailed reviews on selected XASRs to verify the self-evaluation findings. The CED should have a clear policy for selecting projects for in-depth verification and should implement the policy consistently. Selection criteria might include: poor quality / reliability of the XASR; apparent significant differences between self-evaluation ratings and CED ratings; projects exhibiting performance at the extremes; projects that will contribute to corporate learning; or projects of relevance to corporate strategy or development imperatives more widely. In-depth verifications have the same scope of research as in-depth PERs (per OP 8.2) and where deemed necessary by the CED are conducted through field-based research. The CED discloses its policy for selecting XASRs for in-depth verification along with the number and/or proportion of projects subjected to such a review. |
In-Depth Verification | Note that this OP does not prescribe the number or proportion of XASRs that should be subjected to in-depth verification. However, in the interests of evaluative rigor, it is desirable that the CED performs some degree of in-depth verification (see Annex Note EP 8/9). Depending on the availability of resources, in-depth verification could be focused on a project's environmental and social effects, (which are most reliably determined through field-based research), rather than on the full range of project impacts. | |
| The CED prepares an XASR-Assessment (XASR-A) on the final-edition XASR that records the CED's findings from its verification and its independent judgments on the project's results and appropriate ratings in relation to GPS guidelines. | CED Reporting | ||
| The XASR-A is shared in draft form with the XASR team and their comments solicited and considered by the CED. For transparency, the final XASR-A should communicate the CED's final independent judgments highlighting any differences between its performance ratings and those of the XASR, and cite the comments received from the XASR team. | Review Process | ||
[1] The generic standards on independence of evaluation departments were excluded from the GPS on Evaluation of Private Sector Operations as these already constitute Chapter II of the Big Book which contain GPS on Independence of IFIs' CED (June 2010). The same treatment was done on specific standards on indirect evaluation (otherwise referred to as self-evaluation) which are presented in Chapter VI on Self-Evaluation. Relatedly, GPS on Evaluation of Private Sector Operations use the terms "direct and indirect” evaluation to refer to self- and independent evaluation, respectively, to acknowledge the differences in terminology used in the different IFIs which reflect the nature of CED's interface with an actual project (based on clarification from consultant of ECG WGPSE.)