Private Sector Operations

Good Practice Standards on Evaluation of Private Sector Operations

Background

Jump to the Good Practice Standards (GPS) on Private Sector Operations Evaluation

Generic Principles

Private Sector Principles

 

Formulation Process

The Evaluation Cooperation Group's Good Practice Standards for Evaluation of Private Sector Investment Operations (ECG-GPS) were originally formulated in response to the call for harmonization of evaluation methodologies by the Development Committee Task Force in 1996. In 2001, the ECG issued the first edition of the GPS, followed by second and third editions in 2003 and 2006 respectively. Each subsequent edition was informed by the findings and recommendations of a benchmarking exercise, which assessed members' practices against the GPS. Following the 2010 benchmarking exercise against the third edition of the GPS, the ECG has issued a fourth edition.

The fourth edition builds and improves upon the previous GPS to reflect the evolution in evaluation practices and in the scope of investment operations undertaken by International Financial Institutions (IFIs). It takes into account feedback from the benchmarking exercise, and the experiences of members in implementing the previous set of standards. In particular, the fourth edition addresses the following issues identified with the GPS Third Edition:

¬∑ some standards were too narrowly defined and did not recognize the variation in IFI mandates and operational procedures, particularly given the expanding membership of the ECG;

¬∑ the GPS did not differentiate between standards that could be implemented unilaterally by the evaluation departments and those that relied in part or wholly on management action or cooperation;

¬∑ some standards ‚Äì the experimental standards in particular ‚Äì were overly complicated both in design and in their implementation;

¬∑ there was unnecessary overlap between different standards, which created ambiguity and/or duplication;

¬∑ the GPS comprised a range of different types of standard (harmonization, other, good practice, best practice, and not universally applicable), which complicated both interpretation and periodic benchmarking;

¬∑ the definition of rating benchmarks was in certain cases too complex and went beyond the objective of harmonization;

¬∑ the GPS were narrowly focused on financial and non-financial sector investments (mainly project finance) and had limited relevance to the increased variety in investment operations (e.g., equity funds, working capital facilities, corporate finance, trade finance, political risk insurance etc.); and

¬∑ there was scope for the GPS to promote more innovation in evaluation and dissemination through, for example, web-based media.

Objectives and Organization

Although the premise of the Task Force's 1996 decree was to enable comparability of results, the guidance stressed other objectives including identifying and disseminating best practices in evaluation, sharing lessons from evaluations, and describing results in a common language. The decree also acknowledged that harmonization efforts should take into account the differing circumstances of each institution. The GPS Fourth Edition is responsive to these wider objectives.

These GPS are organized into generic standards on reporting and corporate learning, evaluation guidance and rating systems that apply across all IFIs, and standards specific to IFIs supporting private sector investment.[1] The standards themselves are formulated as EPs. Each EP is defined in terms of its key components or "Elements”. To guide IFIs in their efforts towards meeting the Elements of the EPs, each EP is supported by a set of standard OPs. The OPs describe the policies and procedures that the IFI would typically need to adopt in order to be deemed compliant with the respective EP.

The EPs on evaluation of private sector operations comprise a total of 17 standards and 75 elements. The OPs include 63 policies and procedures.[2] The summary of the EPs, in terms of standards and elements, and OPs is presented below in a simplified and more logically consistent framework, as follows:

Summary of Standards and Elements on EPs and Number of OPs on Evaluation of Private Sector Operations

Evaluation Principles

No. of OPs

Standards

Elements

Generic Principles: Reporting and corporate learning
Concern the scope and timing of periodic reporting, the disclosure of evaluation reports and products, and the capture, dissemination and application of lessons from the evaluation system.

1.   Annual reporting of corporate results

A.     Corporate Results

B.     Reporting Rating Results

C.    Analysis

D.    Recommendations

E.     Disclosure

5

2.   Periodic reporting on evaluation systems

 

A.     Periodic Reporting

B.     Quality Efficacy

C.    Alignment

D.    Evaluability

E.     Lessons Application

5

3.   Lessons and findings from evaluation

 

A.     Coverage

B.     Relevance

C.    Accessibility

3

Generic Principles: Evaluation guidance and rating systems
Concern the features of an evaluation rating system, and the preparation and dissemination of evaluation guidance.

4.   Guidance for project evaluation

A.     Preparation

B.     Content

C.    Dissemination

2

5.   Performance rating scales

A.     Range and Balance

B.     Descriptive

C.    Binary Reporting

3

Private Sector Principles: Planning and executing a project evaluation program

Concern the determination of when an operation is ready for evaluation, the selection of a sample from a defined population, and the process of direct evaluation and self-evaluation with independent verification.

6.  Defining the population of projects for evaluation

A.     Coherence and Objectiveness

B.     Qualifying Projects

C.    Screening

D.    Non-Qualifying Projects

E.     Exclusions

F.     Disclosure

6

7.   Selecting a sample of projects for evaluation

A.     Representative Random Sampling

B.     Sample Aggregation

C.    Disclosure

D.    Purposeful Sampling

4

8.   Process of direct evaluation by CED

A.     CED's Options

B.     Reporting

C.    Desk-Based

D.    In-Depth

E.     Transparency

F.     Review Process

4

9.   Scope of Independent verification by the CED

A.     Verification

B.     In-Depth Verification

C.    CED Reporting

D.    Review Process

3

Private Sector Principles: Evaluation metrics and benchmarks

Define the scope of measurement and benchmarks for rating each performance indicator within the evaluation framework.

10.  Rating project outcomes

A.     Synthesis rating

B.     Benchmark

C.    Financial Criteria

D.    Economic Criteria

E.     IFI Mandate Criteria

F.     E&S Criteria

2

11.   Outcome Indicator 1 - Financial performance and fulfilment of project business objectives

A.     Stakeholder Analysis

B.     Time Span

C.    Fulfilment of Project Business Objectives

D.    Methods

E.     Benchmarks

5

12.   Outcome Indicator 2 ‚Äì Economic sustainability

A.     Stakeholder Analysis

B.     Time Span

C.    Net  Benefits

D.    Methods

E.     Benchmarks

5

13. Outcome Indicator 3 – Contribution to IFI mandate objectives

A.     Method

B.     Balanced

C.    Benchmark

2

14. Outcome Indicator 4 – Environmental and social performance

A.     E&S Performance

B.     E&S Capacity

C.    Sub-Project Performance

D.    Benchmark

3

15.  Rating the IFI's investment profitability

A.     Scope

B.     Net Method

C.    Gross Proxy Method

D.    Benchmark

5

16. Rating IFI work quality / bank handling

A.     Scope

B.     Stand-alone

C.    Pre-Commitment

D.    Post-Commitment

E.     Benchmark

4

17. Rating the IFI additionality

A.     Counterfactual

B.     Financial Additionality

C.    Non-Financial Additionality

D.    Benchmark

2

Total No. of Standards: 17

Total No. of Elements: 75

Total No. of OPs: 63

 

Where appropriate, these GPS make reference to a project typology (Annex IV.2) so that OPs can be tailored to the wider range of projects and clients now supported by IFIs. The project typology is particularly important when defining early operating maturity, and rating the indicators of project business success, economic sustainability, and environmental and social performance. For projects exhibiting a mix of project types, it is recommended that IFIs use a range of metrics as appropriate.


[1]  The generic standards on independence of evaluation departments were excluded from the GPS on Evaluation of Private Sector Operations as these already  constitute Chapter II of the Big Book which contain GPS on Independence of IFIs' CED (June 2010). The same treatment was done on specific standards on indirect evaluation (otherwise referred to as self-evaluation) which are presented in Chapter VI on Self-Evaluation. Relatedly, GPS on Evaluation of Private Sector Operations use the terms "direct and indirect‚Äù evaluation to refer to self- and independent evaluation, respectively, to acknowledge the differences in terminology used in the different IFIs which reflect the nature of CED's interface with an actual project (based on clarification from consultant of ECG WGPSE.). 

[2] GPS on Evaluation of Private Sector Operations, Fourth Edition is a refined and simplified version of GPS 3rd Edition where multi-faceted standards were disaggregated into their individual components resulting in a total of 22 EPs (or standards), each defined by between three and eight elements, making 101 Elements in total.  The EPs and their Elements together cover much the same scope as GPS 3rd Edition.

Good Practice Standards on Evaluation of Private Sector Operations

Generic Principles: Reporting and corporate learning

Evaluation Principle

(Standards and Elements)

Standard Operational Practices

Element Link

Notes

1. Annual reporting of corporate results

A.  Corporate Results: The central evaluation department (CED) reports to the Board annually on the IFI's independently verified outcome results.

B.  Reporting Rating Results: The CED reports the IFI's results in all rating dimensions and indicators.

C.  Analysis: The CED analyses the results to discern performance drivers.

D.  Recommendations: The CED formulates recommendations based on the findings.

E.  Disclosure: The CED discloses its synthesis evaluation results externally.

On an annual basis, the CED reports to the governing Board on the international financial institution's (IFI) corporate-wide performance, based on the findings from project-level evaluations and, if required, thematic evaluations.* The report can be stand-alone or incorporated in other reports to the IFI's Board. The ratings reported should be those independently verified or directly assigned by the CED.

Corporate Results

* Thematic evaluations could include country, sector or other studies of the aggregate results across a defined group of projects.

For each rating dimension and indicator, the CED reports the number and proportion (by number of operations) of the evaluated cohort in each performance-rating category.

Reporting Rating Results

The CED may also choose to report the results weighted by project or investment size, to indicate the quantum of impact.

The CED provides a synthesis description of the ratings patterns and their cross-cutting performance drivers under each indicator. It also provides the dimension and indicator ratings for the previous few years or cohorts thereof (where such data exists) to show how performance is evolving over time.

Analysis

 

Where feasible, the CED makes recommendations to the IFI's Management based on the evaluation findings.

Recommendations

The qualification here allows for an exception in the case of CEDs with insufficient evaluated projects to substantiate recommendations.

The CED publishes its findings after appropriate redaction to protect commercial confidentiality, and posts on a webpage accessible via the IFI's external website the full text or an abstract of its report that accurately summarizes its essential findings.

Disclosure

The webpage can be on the CED's own site, provided that the CED's site can be accessed via a link on the IFI's main pages.

2. Periodic reporting on evaluation systems

A.    Periodic Reporting: At least once every three years, the CED reports on aspects of the IFI's evaluation systems, including:

B.    Quality & Efficacy: The CED reports to the Board on the quality and efficacy of evaluation systems.

C.   Alignment: The CED reviews and reports on the alignment of Management reporting systems with the evaluation framework.

D.   Evaluability: The CED reviews and reports on the evaluability of the IFI's operations.

E.    Lessons Application: The CED reviews and reports on the application of lessons learned from evaluation.

The CED reports to the Board at least once every three years on the functioning and effectiveness of the IFI's evaluation systems, as detailed below. The report can be stand-alone or incorporated in other reports to the Board.

Periodic Reporting

The review of evaluation systems in the IFI could be undertaken by the CED directly, or by an external independent body under commission from the CED.

The CED reviews and reports on the quality and efficacy of the IFI's evaluation systems. As part of this reporting, the CED submits to the IFI's Management and Board the periodic benchmarking reviews of the consistency of the IFI's practices with the ECG Good Practice Standards (or provides a summary thereof).

Quality & Efficacy

 

The CED reviews and reports the extent to which internal Management and corporate reports (up to Board level) are broadly aligned with the evaluative framework. For example, the CED should review: (i) to what extent the IFI applies coherent and consistent benchmarks to gauge project performance at relevant stages of the project cycle; and (ii) whether Management's reporting of results includes project outcome and additionality ratings based on the ECG GPS.

Alignment

 

The CED assesses and reports on the evaluability of the IFI's operations i.e., the extent to which the value generated or the expected results of a project are verifiable in a reliable and credible fashion. In practical terms, the CED should assess whether the IFI had specified relevant indicators at approval and made sufficient provision to collect the data required for monitoring during project supervision. The CED need not report on every operation, or undertake such reviews at the time of project approval.

Evaluability

 

The CED assesses and reports evidence of the extent to which lessons of experience are being applied in new operations. It is not required that the CED report on every operation individually, or undertake such reviews at the time of project approval.

Lessons Application

Examples of methodologies for such an assessment include surveys or interviews of origination staff, or a CED review of appraisal documents.

3. Lessons and findings from evaluation

A.   Coverage: Lessons of experience are identified for all project-level evaluations.

B.   Relevance: Lessons are relevant to new operations.

C.   Accessibility: Lessons and evaluation findings are made readily available to IFI staff.

All direct and indirect project-level evaluation reports should contain a prompt or template for the author(s) to identify and articulate one or more lessons from the operation.

Coverage

 

Lessons should be concise, prescriptive, and placed in the context of a material issue that was encountered in the evaluation so that its relevance to new operations can be determined easily, on a stand-alone basis. The point of view and selectivity should focus on what the IFI might have done to obtain better results from the operation.

Relevance

 

The CED maintains a database or library of operational lessons from project-level evaluation reports, which is freely accessible to IFI staff. Alternatively, the CED contributes lessons from project-level evaluations (or a summary thereof) to a database maintained by IFI Management.

 

The CED makes available to IFI staff a range of easily accessible dissemination products covering evaluation findings from projects and/or synthesis CED reports. This could include, inter alia, access to the full reports, electronic notification of new items, and presentations of findings.

Accessibility

 

 

Good Practice Standards on Evaluation of Private Sector Operations

Generic Principles: Evaluation guidance and rating systems

Evaluation Principle

(Standards and Elements)

Standard Operational Practices

Linked to

Notes

4. Guidance for project evaluation

A.   Preparation: The central evaluation department (CED) develops guidance for staff undertaking direct and indirect project evaluations.

B.   Content: Guidance is self-standing, current, and comprehensive in key aspects of the evaluation process.

C.   Dissemination: Guidance is easily accessible and supplemented by training and/or good practice examples.

The CED develops, in conjunction with Management as necessary, guidance for CED and operational staff undertaking direct and indirect project evaluations. The evaluation guidelines should be consistent with prevailing ECG Good Practice Standards and at a minimum include:

(i) the key steps in the evaluation process, in the preparation and signing-off of reports, and in independent verification by the CED as necessary;

(ii) the scope of measurement and the benchmarks for assigning ratings for each performance indicator and dimension; and

(iii) standard reporting templates that include a performance ratings matrix.

Preparation

 

Content

Where separate guidance is prepared for self-evaluations and independent direct evaluations and/or verifications, these should be completely coherent in terms of the prescribed metrics and benchmarks.

The CED makes the evaluation guidelines and supporting information readily available on its website and/or the international financial institution's (IFI) website in respect of guidance for self-evaluation. The CED undertakes dissemination activities to familiarize staff preparing project evaluations with the requirements and supporting documentation. This may include the showcasing of evaluation reports regarded as good-practice examples.

Dissemination

 

5. Performance rating scales

A.   Range & Balance: Each indicator is rated on a performance scale from most negative to most positive, with the scale balanced between positive and negative ratings.

B.   Descriptive: Each rating category accurately describes the extent of positive or negative performance.

C.   Binary Reporting: Binary ratings use the first positive rating within the performance scale as their benchmark.

The rating scale for each indicator should encompass performance ranging from the most negative to most positive. There should be balance between positive and negative characterizations (i.e., if there are four ratings, two are less than good and two are good or better; or if there are six ratings, three are less than good and three are good or better).

Range & Balance

 

The words used to describe these ratings should accurately reflect whether the judgments are less than good or else good or better, and should clearly reflect the graduation from worst to best. For example:

Four-point scale: unsatisfactory, partly (un)satisfactory, satisfactory, excellent; or unsuccessful; partly (un)successful; successful; highly successful.

Six-point scale: highly unsuccessful, unsuccessful, mostly unsuccessful; mostly successful; successful; highly successful; or highly unsatisfactory; unsatisfactory; marginal; satisfactory; good; excellent.

Descriptive

 

Where the CED reports success rates based on a binary simplification of the rating scale, the binary benchmark should be the first positive rating within the chosen scale i.e., a satisfactory or successful rating (in the case of the four-point scale cited above) or a mostly successful or satisfactory rating (in the case of the six-point scale cited above).

Binary Reporting

 

 

Good Practice Standards on Evaluation of Private Sector Operations

Private Sector Principles: Planning and executing a project evaluation program

Evaluation Principle

(Standards and Elements)

Standard Operational Practices

Linked to

Notes

6. Defining the population of projects for evaluation

A.    Coherence & Objectiveness: All projects in the population share common characteristics based on a coherent set of criteria.

B.    Qualifying Projects: The population includes all projects that have reached early operating maturity (or are unlikely ever to do so), and all closed projects.

C.   Screening: The CED determines projects' early operating maturity according to GPS criteria.

D.   Non-Qualifying Projects: Projects that are not operationally mature are reconsidered in subsequent years.

E.    Exclusions: The population may exclude other classes of projects where the CED determines that individual evaluations have limited utility.

F.    Disclosure: The CED discloses its criteria for defining the population and any excluded class of project.

The central evaluation department (CED) defines the population of projects according to a coherent and objective set of criteria appropriate to the type of report. The full application of these criteria will determine whether or not an operation is to be included in the population. For example:

Corporate reporting: the population should comprise projects with the same year (or defined range of years) of origin, based on the approval, commitment or disbursement date for the international financial institution's (IFI) associated investment. Alternatively, the population should comprise projects that have reached early operating maturity within a defined timeframe.*

Part-portfolio reporting: the population should comprise projects sharing a common time-basis (as above), and the same country, region, sector or other thematic characteristics as desired.

Coherence & Objectiveness

While this operational practice (OP) allows CEDs to define a sub-portfolio within the overall corporate portfolio, evaluation principle (EP)6 continues to call for annual reporting of corporate-level results. The onus therefore rests with the CED to determine how best to comply with EP6 should it opt for sub-portfolio sampling in respect of this OP.

 

* This alternative approach is appropriate for CEDs that screen the entire portfolio every year to determine which projects have reached early operating maturity.

Projects should be included in a designated population only once and only at such time as (but not necessarily as soon as) they have reached early operating maturity.*

The population (the boundaries of which are defined under OP11.1) also includes all closed projects (i.e., where the associated IFI investment has been repaid, sold or written off, or the guarantee has been cancelled) regardless of whether or not they had reached early operating maturity by the time of closing.

The population should also include projects that are deemed unlikely ever to achieve early operating maturity.**

Qualifying Projects

* This does not prohibit projects from being included in different populations relating to different studies, for example if the CED was to undertake both a corporate and country-level evaluation.

** This could include, for example, a project that has failed or stagnated such that it is unlikely ever to establish a trading record, yet the IFI's investment has not been sold, cancelled or written off and so has not been officially closed. In such cases, there is little value in postponing evaluation, hence they should be included in the current population.

The CED establishes which projects have reached early operating maturity, taking into consideration information on project status provided by operational departments and by applying the guidance in Annex IV.3 : Lookup Table for Determining Early Operating Maturity.

Screening

 

Where the CED determines that projects have not yet reached early operating maturity (but are likely to do so in the future), they should be omitted from the current evaluation year's population. Instead, the CED should consider them for inclusion in the population in a future year when they will have reached early operating maturity.

In cases where the IFI is involved in litigation, foreclosure or other legal process where evaluation could prejudice the IFI's legal position, the CED may choose to omit these projects from the current population and instead roll them forward for consideration in a future year.

Non-Qualifying Projects

Rolling projects forward for consideration in future years' populations may render them incongruous with the year(s) of origin of those later populations. Where the CED reports an aggregation of several years of consecutive evaluation findings, such projects might legitimately be included. Otherwise, the CED should disclose the incidence of older projects in the reported population, or report their results separately.

The CED may choose to exclude altogether from the population the following classes of project:

(i) those that did not proceed with IFI support and where the associated IFI investments were dropped or guarantees never signed, activated or utilized*; or

(ii) those involving subscribed rights offerings or follow-up investments / guarantees undertaken for substantially the same purpose as before (e.g., to help finance cost overruns or restructurings).**

Exclusions

* CED may exclude projects where the IFI never incurred any exposure under the guarantee (e.g., because the beneficiary of the guarantee never made any advances to the end client).

 

** Repeat investments in an existing client company should normally be included in the population, unless they meet the criteria in (6).

The CED discloses how it defined the population and its criteria for including or excluding projects in line with the operational practices above.

Disclosure

 

7. Selecting a sample of projects for evaluation

A.    Representative Random Sampling: Either all projects in the population are evaluated or the CED selects a random sample whose characteristics are representative of the population.

B.    Sample Aggregation: The CED reports the results of one or more years of evaluated random samples.

C.   Disclosure: The CED discloses its sampling methodology, how it defined the reported cohort, and sampling errors in reported results.

D.   Purposeful Sampling: The CED may self-select a purposeful sample to serve specific evaluative needs, but not for overall corporate reporting purposes.

If evaluation coverage is less than 100%, the CED should select a random sample of projects for evaluation from the established population. The sample should be as representative as practicable insofar as it reflects the distribution of important characteristics throughout the population as relevant to each institution.*

Representative Random Sampling

* Relevant characteristics for testing the representativeness of the sample could include: industry sector; country; region; project size; investment size; IFI instrument of support; incidence of loan impairment or equity write-down.

For reporting purposes, the CED may report the results of a sample of projects evaluated in one year, or use a cohort comprising the evaluated samples from several consecutive years in order to increase the granularity of data and its statistical significance. In deciding how many years of data to combine, the CED should balance the desire to report on a meaningful number of observations against the currency of findings, particularly if using evaluation data more than three years old.

Sample
Aggregation

 

Where sampling is used, the CED should report details of the sampling technique used and the extent to which the sample's characteristics reflect those of the population.

When reporting the aggregate results of a cohort comprising samples from more than one evaluation year, the CED should disclose how the reported cohort is defined.

Disclosure

 

The CED should calculate and disclose the sampling errors (at the 95% confidence interval) in the reported success rates for each of the evaluated indicators and outcome ratings.*

 

* Disclosure of sampling errors enables observers to judge the relevance, usefulness and comparability of success rates reported by different institutions.

The CED may select a purposeful (self-selected) sample of projects to be evaluated.* The CED should not use the results of purposeful evaluations for overall corporate reporting purposes, unless projects in the purposeful sample are also selected as part of a random sample as determined under OP 7.1.

Purposeful Sampling

* Reasons for selecting a purposeful sample could include: the potential for learning; the high profile of an operation; credit and other risks; the sector is a new one for the IFI; the likelihood of replication; or the desirability of balanced country and sector coverage.

8. Process of direct evaluation by the CED

A.    CED's Options: At its own discretion, the CED can select projects on which to conduct its own direct evaluations.

G.   Reporting: The CED conveys its findings in a Performance Evaluation Report (PER).

H.   Desk-Based: As a minimum, the PER is based on internal IFI data, staff consultations and market research.

I.     In-Depth: For selected projects, the CED conducts on-the-ground research and stakeholder consultations.

J.    Transparency: The basis for the CED's findings are fully transparent in the PER, including financial / economic calculations and environmental and social effects.

K.    Review Process: Management and staff have the opportunity to comment on the draft PER, but the final assessment is determined solely by the CED.

The CED can undertake a direct evaluation of a project on its own volition, acting with consideration to on-going legal process in line with OP 6.4 . The scope of evaluation and indicator ratings should be consistent with the GPS. The CED reports its findings in a Performance Evaluation Report (PER).

CED's Options

 

Reporting

 

As a minimum, the research for PERs draws from a file review, discussions with available staff involved with the operation since its inception, and external market research.

Desk-Based

 

On a more rigorous basis, the CED may choose to conduct in-depth research (in the field as necessary) for the PER, based on consultations with stakeholders who are knowledgeable about the country, company and project.*

In-Depth

* Such stakeholders could include: IFI specialists, the company's management, employees, auditors, suppliers, customers, competitors, bankers, any relevant government officials, industry associations, community representatives and local NGOs.

The basis for the CED's findings and ratings are made fully transparent in the PER. The PER should also cite which stakeholder groups were consulted as part of the process. Where ex-post financial and/or economic rates of return for the project are cited in the PER, the document includes an attachment providing details supporting these calculations such as the key assumptions and underlying financial / economic time-series data.

The PER includes a summary of environmental, worker health and safety, and social performance information, for each of the IFI's environmental and social safeguards that apply to the project. Evidence from on-the-ground observations and/or client reporting should be sufficient to support the assigned outcome and IFI work quality ratings. The information can be incorporated as an attachment to the PER if preferred.

Transparency

 

The CED provides an opportunity to Management and operational staff to review and comment on the PER's draft findings, though the final content and ratings in the report remain the decision of the CED. Findings from the PER can be used in synthesis reporting without further verification.

Review Process

 

9. Scope of independent verification by the CED[1]

A. Verification: The CED conducts an independent review of expanded annual supervision reports (XASR) based on internal IFI data and independent research.

B. In-Depth Verification: The CED conducts detailed verifications for selected projects.

C. CED Reporting: The CED reports its independent findings in an XASR-Assessment (XASR-A), which records any rating differences to those in the XASR.

D. Review Process: Management and staff have the opportunity to comment on the draft XASR-A, but the final content is determined by the CED.

The CED conducts an independent review (which may be desk-based) of the XASR to verify its scope, responsiveness, evident reliability of the analysis, impartiality and consistency in ratings judgments, and appropriateness and completeness of the identified lessons. As a minimum, the independent review draws from a file review, discussions with available staff involved with the operation since its inception, and external market research. Depending on the coverage of the population by XASRs, either:

(a) If the IFI has prepared XASRs for a representative sample selected in accordance with EP 7 , then the CED should conduct independent reviews for all the XASRs in such sample; or

(b) If the IFI has prepared XASRs for all projects in the population of operationally mature projects (as defined in accordance with EP 6), then the CED may choose to conduct independent reviews either for all XASRs or for a representative sample of XASRs selected in accordance with EP 7. If a sample is preferred, only the ratings from CED-verified XASRs are valid for corporate reporting purposes.

Verification

 

On a more rigorous basis, the CED conducts detailed reviews on selected XASRs to verify the self-evaluation findings. The CED should have a clear policy for selecting projects for in-depth verification and should implement the policy consistently. Selection criteria might include: poor quality / reliability of the XASR; apparent significant differences between self-evaluation ratings and CED ratings; projects exhibiting performance at the extremes; projects that will contribute to corporate learning; or projects of relevance to corporate strategy or development imperatives more widely.

In-depth verifications have the same scope of research as in-depth PERs (per OP 8.2) and where deemed necessary by the CED are conducted through field-based research. The CED discloses its policy for selecting XASRs for in-depth verification along with the number and/or proportion of projects subjected to such a review.

In-Depth Verification

Note that this OP does not prescribe the number or proportion of XASRs that should be subjected to in-depth verification. However, in the interests of evaluative rigor, it is desirable that the CED performs some degree of in-depth verification (see Annex Note EP 8/9). Depending on the availability of resources, in-depth verification could be focused on a project's environmental and social effects, (which are most reliably determined through field-based research), rather than on the full range of project impacts.

The CED prepares an XASR-Assessment (XASR-A) on the final-edition XASR that records the CED's findings from its verification and its independent judgments on the project's results and appropriate ratings in relation to GPS guidelines.

CED Reporting

 

The XASR-A is shared in draft form with the XASR team and their comments solicited and considered by the CED. For transparency, the final XASR-A should communicate the CED's final independent judgments highlighting any differences between its performance ratings and those of the XASR, and cite the comments received from the XASR team.

Review Process

 

 

 


[1] The generic standards on independence of evaluation departments were excluded from the GPS on Evaluation of Private Sector Operations as these already  constitute Chapter II of the Big Book which contain GPS on Independence of IFIs' CED (June 2010). The same treatment was done on specific standards on indirect evaluation (otherwise referred to as self-evaluation) which are presented in Chapter VI on Self-Evaluation. Relatedly, GPS on Evaluation of Private Sector Operations use the terms "direct and indirect‚Äù evaluation to refer to self- and independent evaluation, respectively, to acknowledge the differences in terminology used in the different IFIs which reflect the nature of CED's interface with an actual project (based on clarification from consultant of ECG WGPSE.)

Good Practice Standards on Evaluation of Private Sector Operations

Private Sector Principles: Evaluation metrics and benchmarks

Evaluation Principle

(Standards and Elements)

Standard Operational Practices

Linked to

Notes

10. Rating project outcome

A.    Synthesis Rating: The project's outcome is based on a qualitative synthesis of underlying indicator ratings.

B.    Benchmark: The rating measures how well the project serves the international financial institution's (IFI) institutional mandate.

C.    Financial Criteria: It reflects the project / company's financial performance and achievement of project business objectives.

D.    Economic Criteria: It reflects the project / company's contribution to economic growth*.

E.    IFI Mandate Criteria: It reflects the project / company's contribution to the IFI's mandate objectives.

F.   E&S Criteria: It reflects the project / company's environmental and social performance.

Scope of Measurement: The rating of project outcome reflects summary qualitative performance judgments based on a synthesis of all the following underlying indicator ratings, taking into consideration the sustainability of results:

Synthesis Rating

See Annex IV.5: Guidance Paper: Technical Note on IFC's Methodology for Assigning Development Outcome Ratings.

- the project / company's financial performance (i.e., the project's contribution to the company's financial results, or the company's financial results where the project is indistinguishable from the company). This also considers the extent of fulfilment of project business objectives;

Financial Criteria

 

- the project / company's economic sustainability (i.e., the project and/or project company's contribution to growth in the economy)*;

Economic Criteria

* Not applicable to the European Bank for Reconstruction and Development (EBRD).

- the project / company's contribution to the IFI's mandate objectives, be they to stimulate development of the private sector, development of efficient financial / capital markets, or transition to a market economy;

IFI Mandate Criteria

 

- the project / company's environmental and social performance.

E&S Criteria

 

Binary Benchmark: As a minimum, for a positive project outcome rating, the project should have a clear preponderance of positive results (i.e., it may exhibit some minor shortcomings though these should be clearly outweighed by positive aspects). The guiding principle should be that if all the IFI's projects exhibited this level of performance, the IFI should be able to demonstrate the successful execution of its institutional mandate.

Benchmark

The Binary Benchmarks defined herein refer to the first positive rating within the chosen scale. In the example scales cited in OP 5.2, the binary benchmark therefore refers to that of a satisfactory / successful rating (in the case of a four-point scale) or mostly successful/satisfactory rating (in the case of a six-point scale).

 

See Annex IV.5, note OP 10.2 for guidance on an extended rating scale.

11. Outcome Indicator 1 – Financial performance and fulfilment of project business objectives

A.   Stakeholder Analysis: The indicator measures the incremental effect of the project on all key financial stakeholders in the project and/or company.

B.   Time Span: The rating is based on historic and projected future financial performance.

C.   Fulfilment of Project Business Objectives: The rating considers the achievement of process and business objectives articulated at approval.

D.   Methods: The central evaluation department (CED) applies a range of evaluation methods appropriate to the project type, with an emphasis on quantitative metrics wherever possible.

E.   Benchmarks: The rating is based on benchmarks appropriate to the project type and evaluation methods applied.

In evaluating financial performance, the incremental effect of the project on the company is assessed on a with vs. without project basis, or a before vs. after project basis. The effect of the project on all financial stakeholders in the project and/or company should be considered.* Both historic and, where relevant, projected performance should be taken into consideration. The rating also considers fulfilment of project business objectives, that is the extent to which the project has delivered on the process and business objectives stated at approval.

Stakeholder Analysis

Time Span

Fulfilment of Project Business Objectives

* Such stakeholders should include as relevant: the owners (shareholders); senior lenders; junior lenders; and trade creditors.

Scope of Measurement for projects of types A, B and C: The rating of financial performance and fulfilment of project business objectives is determined through the application of the methods set out below. The choice of method should be appropriate to the project type, and should use quantitative metrics wherever possible. At a minimum, methods 3, 4 and 5 should be used.

1. Quantitative Method: The rating is based on the project's after-tax financial rate of return in real terms (FRR), or on the time-adjusted after-tax return on invested capital in real terms (ROIC i.e., the costs and benefits to the whole company on a before vs. after basis).*

2. Achievement of Appraisal Projections: The evaluation compares actual performance with appraisal projections. This is only valid provided that the appraisal projections demonstrate sufficient profitability to: (i) service the project's debt obligations and meet creditor payments when due; and (ii) generate the minimally acceptable return to the project company's shareholders commensurate with the risk.**

3. Achievement of Project Business Objectives: The assessment concerns the extent to which the project has, or is judged likely to, fulfil the process and business goals that were articulated at approval.***

4. Analysis of Financial Statements: An appropriate range of performance indicators in project financing are considered such as: sales, net profit, debt service coverage, and financial internal rate of return (FIRR). Suitable project return analysis should supplement balance sheet and income related indicators.

5. Business Prospects: The project company's overall profitability, adaptability and prospects for sustainability and growth are considered, taking into account its performance relative to the market or sector peers.

Methods

For further guidance, see Annex IV.5, note OP 11.2.

* In general, an FRR should be calculated where the financial cashflows of the project can be separated from those of the company's other activities. A ROIC may be more appropriate in the case of corporate investments or expansion projects.

 

** The "minimally acceptable return” to shareholders can be derived from the IFI's own profit objectives if it is itself a shareholder. Alternatively, indicators could include the original financing plan or current expectations of investors in similar projects.

 

*** For example: business objectives could be those related to carrying out an investment plan in respect of plant and equipment and the establishment of a strong management team; process objectives could be the introduction of an IAS accounting system or for a financial institution the improvement of credit manuals and the training of staff. For EBRD, achievement of project objectives does not incorporate the transition impact objectives, which are captured separately.

Binary Benchmark for projects of types A, B and C: As a minimum, for a positive rating of the project / company's financial performance and fulfilment of project business objectives, it should achieve the following benchmarks. Where more than one method is applied, each of the relevant benchmarks should be met:

1. Quantitative Method: The project's FRR or ROIC is equal to or greater than the project company's Weighted Average Cost of Capital (WACC). The WACC should be calculated using accepted principles and based on company- or sector-specific data.* The use of fixed IFI-wide assumptions or hurdle rates in place of the WACC is not good practice.

2. Achievement of Appraisal Projections: Actual performance meets or exceeds appraisal projections such that the project has demonstrably met its obligations to lenders and creditors, and has yielded the minimally acceptable return to its shareholders commensurate with the project risk.**

3. Achievement of Project Business Objectives: The project's process and business goals articulated at approval are broadly achieved or are deemed within reach albeit with some risk to their realisation.

4. Analysis of Financial Statements: Performance indicators are in line with appraisal estimates.

5. Business Prospects: The project company's overall profitability and prospects for sustainability and growth are sound, such that it is expected to remain competitive in relation to the market and its sector peers.

Benchmarks

See Annex IV.5 , note OP 11.3 for guidance on an extended rating scale.

 

* See guidance paper: Using the FRR to Rate Project Business Success.

 

** The "minimally acceptable return” to shareholders can be derived from the IFI's own investment outcome rating if it is itself a shareholder. Alternatively, indicators could include the original financing plan, current expectations of investors in similar projects, or evidence that shareholders are satisfied with their returns (e.g., the company has attracted additional investment and/or executed a successful rights issue to fund future growth).

Scope of Measurement for intermediation projects of type D1 and D2: The rating of financial performance and fulfilment of project business objectives is determined through the application of the methods set out below. The choice of method should be appropriate to the project type, and should use quantitative metrics wherever possible. At a minimum, methods 3 and 4 should be used.

1. Performance of Sub-Portfolio: An assessment should be made of the financial impact of the sub-portfolio on the financial intermediary's viability. Where a calculation of the profit contribution of the sub-portfolio is not possible, proxies can be used, for example: sub-loan spreads (relative to the rest of the FI portfolio), FI sub-loan risk ratings, and/or incidence of arrears or write-offs among the sub-loans.*

2. Performance of Fund Portfolio: The rating is based on the project portfolio's projected or realized contribution to the fund's net return on equity (RoE) or net IRR to the investors (i.e., after management fees, carried interest and other administrative costs).

3. Achievement of Project Business Objectives: The assessment concerns the extent to which the project has, or is judged likely to, fulfil the process and business goals that were articulated at approval.** In particular, it should consider the project's success in reaching certain sub-borrower or investee groups if such groups were specified as targets at approval.

4. Performance of Intermediary: The financial intermediary / local fund management company's overall profitability, adaptability and prospects for sustainability and growth are considered, taking into account its performance relative to the market or sector peers.

Methods

* See guidance paper: Additional Business Indicators for Financial Intermediaries.

 

** For EBRD, achievement of project objectives does not incorporate the transition impact objectives, which are captured separately.

Binary Benchmark for projects of types D1 and D2: As a minimum, for a positive rating of the project / company's financial performance and fulfilment of project business objectives, it should achieve the following benchmarks. Where more than one method is applied, each of the relevant benchmarks should be met:

1. Performance of Sub-Portfolio: There is adequate evidence (quantitative or qualitative) that the sub-portfolio has had a positive effect on the financial intermediary's profitability, and helped improve its viability.

2. Performance of Fund Portfolio: The projected or realized net return on equity (RoE) or net IRR to the fund's investors is equal to or greater than the fund's weighted average cost of capital (FWACC)*.

3. Achievement of Project Business Objectives: The project's process and business goals articulated at approval are broadly achieved or are deemed within reach with some risk to their realisation. The intermediary has succeeded in reaching sub-borrowers or investee groups that were specified as targets at approval.**

4. Performance of Intermediary: The intermediary's overall profitability, adaptability and prospects for sustainability and growth are sound, such that it is expected to remain competitive in relation to the market and its sector peers.

Benchmarks

See Annex IV.5, note OP 11.5 for guidance on an extended rating scale.

 

* Annex IV.5, note OP 11.5 demonstrates how the FWACC is estimated for multi-country funds, using a combined project and equity risk premium of 600bpts over the cost of debt for the fund. Alternatively, the CED can establish its own RoE benchmark, provided that it too reflects an appropriate equity risk premium over the cost of debt to the fund.

 

** Since project type D1 concerns credit lines designed to target specific groups of sub-borrower (rather than a more general corporate investment in a financial intermediary – project type B), the IFI should make adequate provision to track at a minimum the broad sector groups reached through the intermediation. In the absence of such information, the CED may choose to assign a rating of No Opinion Possible for the project's business success.

12. Outcome Indicator 2 – Economic sustainability

A.    Stakeholder Analysis: The indicator measures the incremental effect on all key economic stakeholders in the project.

B.    Time Span: The rating is based on historic and projected economic effects.

C.    Net Benefits: The rating considers both benefits and costs associated with the project, including economic distortions.

D.    Methods: The CED applies a range of evaluation methods appropriate to the project type, with an emphasis on quantitative metrics wherever possible.

E.    Benchmarks: The rating is based on benchmarks appropriate to the project type and evaluation methods applied.

In evaluating the project's economic sustainability (i.e., the project and/or project company's contribution to growth in the economy), the incremental effect of the project on stakeholders is assessed on a with vs. without-project basis, or before vs. after-project basis. Both historic and, where relevant, projected economic effects should be taken into consideration.

The effect of the project on all key economic stakeholders (including and beyond the project company's owners and financiers) should be considered.* Economic distortions conveying trade protectionism should also be considered, for example: quotas; administrative barriers; import / export restrictions, tariffs or subsidies; anti-dumping legislation; exchange-rate manipulation; or protectionist use of patent systems.

Time Span

 

Stakeholder Analysis

 

Net Benefits

This EP is not relevant for EBRD.

 

* Such stakeholder should include, as relevant: customers; suppliers; producers of complementary goods; competitors; new market entrants; employees; tax-payers (government); and neighbours. This EP does not prescribe the methodology by which the CED should measure economic impacts on different stakeholders or how to verify the attribution of economic effects to the project. In practice, CEDs may wish to examine this in the form of a synthesis study or by assessing the aggregate effects of a group of related projects (e.g., in the same sector or geographic area). See also guidance paper: A Stakeholder Framework for Assessing Development Impact.

Scope of Measurement for projects of types A, B and C: The rating of economic sustainability is determined through the application of the following methods. The choice of method should be appropriate to the project type, and should use quantitative metrics wherever possible:

1. Quantitative Method: The rating is based on the project's net quantifiable economic benefits and costs, as measured by the project's real economic rate of return (ERR) or by the economic return on invested capital (EROIC) i.e., by the time-adjusted internal rate of return on the economic costs and benefits on a before-vs-after basis.* The analysis should also consider other material, but unquantifiable, costs and benefits to key economic stakeholders.

2. Qualitative Stakeholder Analysis: Where quantified estimates of the direct economic costs and benefits to all relevant economic stakeholders are not possible, each economic stakeholder group affected by the project should be identified and a judgment made broadly as to the magnitude and direction (positive or negative) of the impact on each.

Methods

* In general, an ERR should be calculated where the economic effects (cashflows) of the project can be differentiated from those of the company's other activities. An EROIC may be more appropriate in the case of corporate investments or expansion projects.

Binary Benchmark for projects of types A, B and C: As a minimum, for a positive rating of the project's economic sustainability, it should achieve the following benchmarks:

1. Quantitative Method: The ERR or EROIC is equal to or greater than the larger of either: (i) a multiple of 1.2 times the project company WACC*; or (ii) 10%. A positive rating may also be awarded if the ERR or EROIC falls short of the quantitative benchmark, but there are other material un-quantified net economic benefits that could be expected to raise the ERR or EROIC sufficiently.

2. Qualitative Stakeholder Analysis: Either: (a) the project meets the minimum standard for satisfactory financial performance** and there is evidence that it has generated a balance of benefits for its wider economic stakeholders (i.e., those other than the project company's owners and financiers); or (b) the project just fails to meet the minimum standard for satisfactory financial performance, but there is evidence that it has generated substantial net benefits for its wider economic stakeholders. In either case, the project should not rely on economic distortions to maintain its financial performance.

Benchmarks

See Annex IV.5, note OP 12.3 for guidance on an extended rating scale.

 

* The project company WACC should be calculated using accepted principles per OP 11.3.

 

** The reference here to project financial performance does not imply overlap between the two indicators, although the two are by definition linked. Financial performance is a measure of the project's impact on its financial stakeholders, who represent a sub-set of all of the project's economic stakeholders. It is therefore a starting point for assessing the project's overall economic impact. However, the metrics here consider stakeholders beyond the project company's owners and financiers and, therefore, the wider economic contributions of the project beyond those measured by project financial performance alone.

Scope of Measurement for projects of types D1 and D2: The rating of economic sustainability is determined through the application of either method D1 or D2 accordingly:

D1. Economic Activities of Sub-Borrowers: The rating is based on the economic activities of the sub-borrowers as the principal stakeholder group (i.e., customers of the financial intermediary). If a quantitative assessment of the net economic benefits generated by sub-borrowers is not possible, the analysis should consider the markets supported specifically by the project and/or more generally by the financial intermediary along with evidence of increased or decreased economic activity in these markets. The existence of economic distortions in these markets should also be considered.*

D2. Economic Viability of Fund Investees:** The rating is based on the economic viability of the fund's investee companies, a proxy for which is their individual and combined contribution to the fund's gross return (before management fees). The extent to which the commercial performance of the fund and its investee companies is influenced by economic distortions should also be considered.

Methods

* The criteria by which market sectors are deemed economically viable is left to the judgment of CED. Examples of the types of data that would support such an assessment would include country- and/or sector-level data on productivity, growth and competitiveness, and the existence of subsidy or other state support, and other macroeconomic factors that have affected the project's sustainability or could do so in the future.

 

** In rating the economic sustainability of a fund, the CED should look through the fund to the investee companies and assess their underlying economic viability. Economic rate of return calculations might be possible at the investee level. Otherwise it is possible to infer that if the investees have generated positive equity returns for the fund and are operating in competitive and non-subsidized markets, they are likely also to have generated positive economic contributions for their wider stakeholders.

Binary Benchmark for projects of types D1 and D2: As a minimum, for a positive rating of the project's economic sustainability, it should achieve the following benchmarks:

D1. Economic Activities of Sub-Borrowers:* Both (i) the project has succeeded in reaching targeted groups of sub-borrower; and (ii) there is direct evidence (from sub-portfolio data) that sub-borrowers are economically viable, or indirect evidence (from market data) that market sectors supported by the project and/or more generally by the financial intermediary are economically viable and do not rely on economic distortions to maintain their commercial viability.

D2. Economic Viability of Fund Investees: Either (i) the gross equity fund portfolio return (before management fees) is equal to or greater than the FWACC x 1.2; or (ii) at least half of equity fund investees have positive equity returns yet the gross portfolio return (before management fees) is less than FWACC x 1.2 but not less than the FWACC x 0.8.** In either case, there is direct evidence (from sub-portfolio data) that investees are economically viable, or indirect evidence (from market data) that market sectors supported by the project are economically viable and do not rely on economic distortions to maintain their commercial viability.

Benchmarks

See Annex IV.5, note OP 12.5 for guidance on an extended rating scale.

 

* Since project type D1 concerns credit lines designed to target specific groups of sub-borrower (rather than a more general corporate investment in a financial intermediary – project type B), the IFI should make adequate provision to track at a minimum the broad sector groups reached through the intermediation. In the absence of such information, the CED has the option of assigning a rating of No Opinion Possible for the project's economic development.

 

** The FWACC should be determined in accordance with OP 11.5.

13. Outcome Indicator 3 – Contribution to IFI mandate objectives

A.   Method: The scope of the indicator and scope of measurement is tailored to reflect the mandate of the IFI.

B.   Balanced: The rating considers both positive and negative contributions.

C.  Benchmark: The rating considers the degree of attribution and the quality of the project's contribution to the IFI's mandate objectives.

Scope of Measurement: This indicator measures the project's contribution to the IFI's mandate objectives,* be they to stimulate development of the private sector, development of efficient financial / capital markets, or transition to a market economy. The scope of measurement should be adjusted to match the scope of the IFI's mandate. It should consider, for example, the positive and negative contributions of the project in the following areas:**

Competition; market expansion; private ownership and entrepreneurship; frameworks for markets; transfer and dispersion of skills; demonstration effects; standards for corporate governance and business conduct; development of financial institutions and financial / capital markets; attracting FDI flows; and development of physical infrastructure.***

Method

 

Balanced

* Mandate objectives as set out in the IFI's Articles of Association or equivalent document.

 

** It is conceivable that the benefits or costs of the project's effects in these areas have already been quantified in economic terms and reflected in the rating of the project's economic sustainability. Where this is the case, the assessment should avoid double-counting and instead cross-reference the earlier economic calculations. In reality, quantification of the project's attributable value in these areas is unlikely to be possible and so the project's impacts can be considered here in qualitative terms without risk of overlap.

 

*** These components are defined in more detail in Annex IV.5 , note OP 13.1.

Binary Benchmark: As a minimum, for a positive rating of the project's contribution to the IFI's mandate objectives, the project should have:

(a) demonstrable effects consistent with the IFI's mandate objectives (for example, in furthering the country's private sector development, development of efficient financial / capital markets, or transition to a market economy); and (b) a clear preponderance of sustainable positive impacts in this respect.

A rating of "Neutral” is permitted for this indicator, to account for cases where a project has no observable or attributable impacts (positive or negative) of relevance to the IFI's mandate objectives.*

Benchmark

See Annex IV.5, note OP 13. for guidance on an extended rating scale.

 

* Note that that a "Neutral” rating is not a middle rating falling between satisfactory and partly (un)satisfactory. Rather, it signifies that this indicator should have no influence on the synthesis project outcome rating. Accordingly, it should also be discounted from both numerator and denominator in the calculation of success rate for this indicator.

14. Outcome Indicator 4 – Environmental and social performance

A.   E&S Performance: The indicator measures the project company or enterprise's overall environmental and social performance.

B.   E&S Capacity: The rating considers the environmental and social management capacity of financial intermediaries / fund managers.

C.   Sub-Project Performance: Where required, the rating considers the environmental and social performance of sub-projects / fund investee companies.

D.   Benchmark: The rating is based on the project company / enterprise / sub-projects achieving compliance with the IFI's specified standards at approval.

Scope of Measurement: The rating of environmental and social performance considers the project company's / enterprise's overall environmental and social performance in the area of influence of the project, as follows:

 

See also the ISO 14031 standard "Environmental Performance Evaluation” for additional guidance on the scope of measurement.

 

An optional supplementary indicator can be used to measure the extent of progress or regress in environmental and social performance since approval. See Annex IV.5 Note EP 14.

All Project Types: based primarily on the IFI's specified standards in effect at approval, and secondarily on the IFI's standards prevailing at the time of the evaluation. The assessment is based on the project company's management of its environmental and social aspects, (i.e., the elements of the organization's activities, products or services that can interact with the environment and society) and, to the extent covered by IFI's policies, includes pollution loads, wastes, energy and resource efficiency, biodiversity conservation, workers' and communities' health and safety, public consultation and participation, land acquisition and cultural heritage.

E&S Performance

 

Project Types D1/D2: In addition, the assessment should consider the adequacy of the financial intermediary's or fund manager's Environmental & Social Management System (ESMS) and its implementation. If so required by the IFI's specified standards at approval, the environmental and social performance of sub-projects / fund investee companies should also be considered.

E&S Capacity

 

Sub-Project Performance

Binary Benchmark: As a minimum, for a positive rating of environmental and social performance, by project type:

Project Types A/B/C: The project company / project enterprise should be in material compliance with the IFI's at-approval requirements including implementation of any environmental and social action program (ESAP).

Project Types D1/D2: The project company / enterprise has implemented an appropriate Environmental & Social Management System (ESMS), which has been functioning over the project life (as reflected also in environmental and social standards being applied to projects financed by the intermediary). If required by the IFI's specified standards at approval, the environmental and social performance of sub-projects / fund investees are in material compliance with the IFI's requirements.

Benchmark

See Annex IV.5, note OP 14.2 for guidance on an extended rating scale.

The project's environmental and social performance should be rated Not Applicable where, by virtue of the project's expected lack of environmental and social impacts, the IFI has not prescribed any at-approval environmental and social requirements, and the status of the project at evaluation remains the same. Evidence should be provided to support such a rating. Note, however, that should the project have subsequently changed in scope and given rise to environmental and social impacts, its performance should be rated accordingly against the standards that would have been prescribed had this been known at approval.

Benchmark

 

15. Rating the IFI's investment profitability

A.   Scope: The indicator measures the profitability of the IFI's investment(s) in the project. They may be reported separately or synthesized into a single rating.

B.   Net Method: The rating is either based on each investment's net profit contribution‚Ķ

C.  Gross Proxy Method: ‚Ķor on the quality of each investment's gross profit contribution.

D.  Benchmark: The rating is based on the investment(s) yielding a return commensurate with the IFI's targeted profitability or return on capital objectives.

Scope of Measurement: The indicator measures the profitability of each of the IFI's investment(s) in the project company. The rating of the IFI's investment profitability is based on either:

Scope

This EP is not relevant for MIGA.

 

While the scope permits the use of either net or gross profit contributions, the net contribution method is the more rigorous and should be favoured if cost accounting data are available. Gross profit contribution is applied in a largely qualitative manner as a proxy for likely investment performance, based on the incidence (or not) of loan impairments, called guarantees, or equity gains / losses.

(a) the investment's net profit contribution (the gross income less financing costs, loss provisions / write-offs, transaction and administrative costs), measured in risk-adjusted, discounted cash flow terms, provided reliable cost data are readily available from management information systems; or

Net Method

(b) the quality of the investment's gross profit contribution (i.e., its likely profitability net of financing costs and loss provisions / write-offs but before deducting transaction and administrative costs).

Gross Proxy Method

Binary Benchmark: As a minimum, for a positive rating of investment profitability for loans, either:

(a) the loan's net profit contribution is sufficient in relation to the IFI's target return on capital employed or overall profitability objectives; or

(b) the loan is expected to be paid, or has been paid, as scheduled (or rescheduled) or prepaid, with no loss of capital. In other words, the loan's gross profit contribution quality meets at-appraisal expectations.

Benchmark

 

Binary Benchmark: As a minimum, for a positive rating of investment profitability for financial guarantees, either:

(a) the guarantee's net profit contribution is sufficient in relation to the IFI's target return on capital employed or overall profitability objectives; or

(b) all guarantee fees have been received or are expected to be received, and the guarantee is not called, or is called but expected to be fully repaid in accordance with the terms of the guarantee agreement. In other words, the guarantee's gross profit contribution quality meets at-appraisal expectations.

Benchmark

 

Binary Benchmark: As a minimum, for a positive rating of investment profitability for equity investments, either:

(a) the expected or realized net profit contribution (or net RoE) is sufficient in relation to the IFI's overall profitability objectives or target return on capital employed; or

(b) the expected or realized gross profit contribution (or gross RoE) reflects an appropriate spread over actual or notional loan yields for the same credit risk, in line with the policy-defined at-entry approval standard.

In both cases, the valuations of active equity investments should be appropriately discounted to reflect the uncertainty of still-to-go dividend income or capital realization.

Benchmark

 

Where the IFI makes more than one type of investment in a single project, either:

(a) one rating is assigned on the basis of the combined net profit contribution of the investments; and as a minimum, for a positive rating of investment profitability, the net profit contribution is sufficient in relation to the IFI's target return on capital employed or overall profitability objectives; or

(b) ratings are assigned and reported for each investment instrument separately.

Benchmark

Since gross profit contribution quality is a predominantly qualitative concept within this EP, it can not be numerically aggregated for the purposes of a synthesis rating. Should the CED wish to report a synthesis investment outcome from gross profit contribution measures, it could use a lookup table based on the ratings for each underlying investment instrument, possibly applying a weighting in line with the size of each type of investment made in the project.

16. Rating IFI work quality / bank handling

A.      Scope: The indicator measures the quality of the IFI's pre-commitment work and on-going monitoring and supervision.

B.      Stand-Alone: The rating is independent of ‚Äì and so not directly influenced by ‚Äì the project's results.

C.     Pre-Commitment: The rating considers all aspects of the IFI's work in screening, appraising and structuring the project and the IFI's associated investment.

D.     Post-Commitment: The rating considers all aspects of the IFI's portfolio responsibilities in monitoring and supervising the project and the IFI's associated investment.

E.   Benchmark: The rating is assigned on the basis of the IFI having executed its responsibilities to an internally or externally recommended standard.

Scope of Measurement: The indicator considers both the IFI's pre-commitment work in at-entry screening, appraisal and structuring / underwriting, and its monitoring and supervision of the operation following commitment / guarantee issuance. These elements can be rated separately or in combination as IFI work quality / bank handling. The assessment should be made independently of the ratings assigned for the project's outcome and the IFI's investment profitability. It should reflect the quality of the IFI's contributions to good or bad outcomes, not the good or bad outcomes themselves.

Scope

 

Stand-Alone

 

Pre-commitment work quality assesses how effectively the IFI carried out its work prior to approval and commitment of the investment. It should consider all factors relevant to the institution's processing of the investment, for example:

- the quality of the IFI's assessment of the operation as being relevant to the IFI's corporate, country and sector strategies;

- the assessment of sponsors, company, management, country conditions, market dynamics, project concept, configuration and cost;

- the appraisal of the project financial plan, source of project funds, and assumptions used in the project's financial projections;

- the effectiveness of the IFI's assessment of project and political risks, and steps taken to mitigate them;

 

- the appraisal of project environmental and social risk, and inclusion of appropriate requirements in the legal agreement;

- investment instrument selection (as applicable), structure, pricing, exit / repayment mechanism, security, covenants and other terms and conditions; and

- the clients' satisfaction with the IFI's service quality up to commitment.

Pre-Commitment

 

Monitoring and supervision work quality assesses to what extent the IFI has adequately executed its portfolio responsibilities for the operation following commitment of the investment. It should consider all factors relevant to the institution's administration of the investment, for example:

- the completeness of supervision reports in documenting project status and risk;

- the monitoring of the client company's compliance with the terms of the investment, including financial, information and performance covenants;

- the monitoring of the client company's environmental and social performance, and adherence to relevant government regulations and IFI requirements;

- the adequacy and timeliness of the IFI's response to emerging problems or opportunities;

- the effectiveness of hand-over procedures should there be changes in IFI staff monitoring responsibilities;

- the clients' satisfaction with the IFI's service quality following commitment; and

- the conduct of and contribution made by a representative of the IFI (if applicable) on the client company's board.

Post-Commitment

 

Binary Benchmark: As a minimum, for a positive rating of IFI work quality / bank handling (or for its two components individually), the IFI should have materially met its prescribed operational procedures and quality standards throughout all stages of the operation. The IFI should have kept itself sufficiently informed to react in a timely manner to any material change in the project and/or company's performance (or any event or circumstance that could be the basis for a claim under an IFI's guarantee), and have taken timely action where needed.

Benchmark

As a point of reference, this rating uses the IFI's internally documented standards as the benchmark. However, the CED should check periodically that such standards are in line with any internationally recognised standards of good practice in commercial banking, investment or insurance institutions.

17. Rating the IFI's additionality

A.   Counterfactual: The indicator measures the IFI's additionality in supporting the project, based on the counterfactual of what would have happened without the IFI's support.

B.   Financial Additionality: The rating considers the IFI's financial additionality in providing funding and/or catalysing other funding.

C.   Non-Financial Additionality: The rating considers the IFI's non-financial additionality in improving the project's risk profile, design or functioning.

D.   Benchmark: The rating is assigned on the basis of the IFI having fulfilled its mandate-defined objectives as a financier of private sector projects.

Scope of Measurement: The rating of the IFI's additionality considers the IFI's value proposition in providing support to the project. It is based on the counterfactual assessment of how the project would have (or would not have) proceeded without IFI support. It should consider all factors relevant to the role and contribution of the IFI, for example:*

Counterfactual

 

* Depending on the IFI's mandate objectives or the scope of its engagement in project selection and structuring, some of the factors listed may not be relevant to the rating of additionality.

 

Financial Additionality:**

- Would the client have been able to obtain sufficient financing / insurance from private sources on appropriate terms? Judgments on this indicator consider pricing (including additional costs arising from IFI conditions that would not be imposed by a private investor), tenor, grace period, currency, and timeliness (i.e., the availability of financing without unduly delaying the project).

- Was the IFI catalytic in mobilizing funds from other investors and lenders, or was it merely helping to complete the financing package?

- Was the IFI (by virtue of its being an IFI) needed to reduce risks or provide comfort (i.e., improve the investors' perceptions of the risks involved) and, thus, to encourage other investors and lenders to proceed?

Financial Additionality

 

** For the purpose of the GPS, additionality factors are grouped into financial and non-financial types, though CED may choose different categories, for example: risk mitigation; policy setting; knowledge and innovation; and standard setting.

Non-Financial Additionality:**

- Was the IFI needed to bring about a fair, efficient allocation of risks and responsibilities e.g., between the public and the private investors?

- Did the IFI improve the project's design (through contributing knowledge or innovation), help the client's functioning in business (including adoption of new or better standards), or otherwise contribute to the client's capacity-building objectives?***

Non-Fi\nancial Additionality

*** These types of contribution could arise through a parallel or linked technical assistance project. The CED should determine whether to reflect this in the evaluation of the investment operation, or separately in an evaluation of the TA intervention.

Binary Benchmark: As a minimum, for a positive rating of additionality, it is evident that, absent the IFI:

(a) the project would not have gone ahead with financing on appropriate terms and/or without undue delays; or

(b) the project would have entailed an unfair or inefficient allocation of risks and responsibilities; or

(c) the project would have been weaker in design, business, developmental, transition, social or environmental terms.

Benchmark