Multilateral Development Banks' Evaluation Harmonization
The Harmonization Challenge
In March 1996, the Development Committee Task Force on Multilateral Development Banks (MDBs) [1] issued a report entitled Serving a Changing World, which called for harmonization of evaluation methodologies, performance indicators, and criteria by MDBs:
"… currently, it is not possible to compare their operational results, or even to describe them in a common language. Many public sector institutions like the MDBs must be able to account for their efforts in readily understood terms. A common methodology for evaluating their portfolios should be developed and kept up to date over time, with best practices in evaluation techniques being identified and disseminated. A determined effort should be made to harmonize performance indicators and evaluation criteria, taking into account the differing circumstances of each institution. The lessons learned from these evaluations should be shared among the MDBs with a view to applying them quickly in new operations.
The heads of the…MDB evaluation units…[should] be charged with elaborating common evaluation standards, including performance indicators; exchange experience with evaluation techniques [and] share results; and become the repository of best evaluation practices. The immediate task would be to develop, within a specified time period, methodology and criteria for assessing and rating the MDB's operational performance and development effectiveness.”[2]
As a matter of priority, the Development Committee invited the five major MDBs to act on the Report's relevant recommendations to strengthen their policies and practices and to advise the Committee in about two years, on the progress achieved in implementing the recommendations.[3]
MDBs' Response
In response to the task force's recommendations, the Evaluation Cooperation Group (ECG) was formed in 1996.[4] The ECG mandate embodies and endorses MDB evaluation harmonization and provides that:
"The ECG (i) works to strengthen cooperation among evaluators and (ii) seeks to harmonize evaluation methodology in its member institutions, so as to enable improved comparability of evaluation results while taking into account the differing circumstances of each institution. Harmonization in the ECG includes increased information sharing and improved understanding of commonalities and differences in evaluation policies, procedures, methods and practices and is not interpreted by members as ‘standardization of evaluation policies and practices'.” [5]
The ECG consisted initially of the heads of the evaluation units of the five MDBs referred to in the task force's report: the African Development Bank (AfDB), Asian Development Bank (AsDB), European Bank for Reconstruction and Development (EBRD), Inter-American Development Bank (IADB), and World Bank Group (WBG). A number of International Financial Institutions (IFIs) have joined the ECG since then and by 2011, total membership has reached nine (9) comprising of the heads of evaluation units of the five founding MDBs, the European Investment Bank (EIB), the International Monetary Fund (IMF), Islamic Development Bank (IsDB), and the International Fund for Agricultural Development (IFAD).[6] In 2012, Black Sea Trade and Development Bank (BSTDB) and the Council of Europe Development Bank (CEB) have been accepted as full members and membership processing is underway.[7] Meanwhile, ECG has three observers who are the heads of evaluation units of the United Nations (UN), Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD-DAC), and the Global Environment Facility (GEF).[8]
The ECG Chairperson is selected annually on a rotational basis from among the members, and a co-chairperson is likewise selected, if deemed desirable. ECG meetings are held twice a year where the ECG Chairperson proposes agendas that have benefited from consultation with ECG members. In executing its activities, the ECG may also convene working groups and technical groups, as needed, with specific terms of reference. It can also organize workshops on specific themes where participants from outside ECG members can be invited.
Evaluation Harmonization
In March 1998, the original five ECG members reported to the Development Committee on the implementation of the recommendations of the Task Force:
"The [Evaluation Cooperation] Group will continue its efforts to make evaluation results comparable and to have their findings properly translated into operational standards. Meeting in Hong Kong in October 1997, the MDB presidents … strongly endorsed further intensification of collaboration among MDB evaluation units in harmonizing evaluation standards and activities, defining more effective linkages between independent and self-evaluation …. The harmonization dialogue will be extended to country evaluations, nonlending services, and evaluation of private sector operations.”[9]
In almost one-and-half decades after its first Report to the Development Committee, the ECG has remained focused on its commitments and continues to pursue harmonization and improvements in evaluation process. ECG milestone accomplishments include the support to evaluation capacity development, conduct of joint evaluations and joint thematic workshops on major and timely topics, assessment of evaluability of operations of selected members, and strengthening its oversight function over member MDBs. The ECG has also established collaboration mechanisms with other evaluation networks by sitting as an observer in: (i) the OECD-DAC Network on Development Evaluation for the purpose of improving the effectiveness of international development assistance, and (ii) meetings of the United Nations Evaluation Group (UNEG).[10]But more importantly, ECG has developed and implemented the Good Practice Standards (GPS) for four categories of MDB evaluations covering governance and independence of evaluation function, public and private sector operations, and country strategy and program (CSP).[11]The goal of documenting these standards is to harmonize evaluation practice among ECG members, not to evaluate functions.[12] The GPS are therefore consistent with ECG's mandate on harmonization and on improving understanding of evaluation practices. Derived from the evaluation principles of the OECD–DAC, these GPS were built on good evaluation practices, and were designed to be consistent with the MDBs' operational policies.
Origin of Big Book
In view of the increased interaction between the various working groups on the GPSs as well as to achieve greater "cross fertilization”, the ECG members suggested in 2011 to combine the four GPS into a single document called the GPS "Big Book”. The "Big Book” contains the latest version of the approved GPS (Please see footnote#11) and will be presented for review in ECG's Fall Meeting in 2012. The "Big Book” shall be considered a living document that will be revised from time to time.
The Chapters of the "Big Book” attempt to organize the evaluation principles by type, i.e., general and specific, as well as to address overlaps noted in the GPSs and to resolve differences in terminologies, if any, without losing the original intent of the four GPS reports taken individually. Chapter II deals with the general evaluation principles and lays down the GPS on independence of MDBs' central evaluation department (CED) essential to ensuring MDB accountability and feedback mechanism from performance evaluation. Chapters III and VI present the specific evaluation principles. Chapters III to V cover the independent evaluation principles and standards by evaluation category, namely public and private sector operations, and CSPs, the results from which should be of value to further increase the effectiveness of MDB development interventions. Chapter VI provides key standards for MDB self evaluation aimed to help improve the evaluability of projects and investments and quality of completion reports (CRs), and to establish effective linkages with independent evaluation. The respective evaluation principles are grouped by standards, each of which is composed of a set of elements. The evaluation principles are also supported by the corresponding set of operational principles. In the case of GPS on evaluation at the country level, the operational principles are expressed in terms of Core-GPS (C-GPS) and Optional-GPS (O-GPS).[13]
[1] The Development Committee Task Force on MDBs was established in 1994 to undertake, for the first time, an assessment of the five major MDBs - the African Development Bank (AfDB), the Asian Development Bank (AsDB), the European Bank for Reconstruction and Development (EBRD), the Inter-American Development Bank (IADB), and the World Bank Group (WBG). (Source: Development Committee Task Force on MDBs. 1996. Serving a Changing World. Washington, DC, Foreword.)
[2] Development Committee Task Force on MDBs. 1996. Serving a Changing World. Washington, DC, p. 18.
[3] Development Committee Task Force on MDBs. 1996. Serving a Changing World. Washington, DC, Foreword.
[4] ECG was created by the 5 MDBs in October 1995 and was officially convened in February 1996. (Source: ECG Mandate in ECG website.)
[5] ECG. 2003. Amended ECG Mandate.
[6] EIB joined ECG in 1998, IMF in 2001, and IsDB and IFAD in 2010.
[7] Minutes of ECG Plenary Meeting in March 2012.
[8] The UN is represented by the Director of UNDP's Evaluation Unit and/or the Chair of the United Nations Evaluation Group, and OECD-DAC is represented by the Head of the Secretariat and /or the chair of the Network on Development Evaluation. (Source: ECG membership in ECG website.)
[9] Development Committee Task Force on MDBs. 1998. Implementation of the Major Recommendations of the MDB Task Force Report. p. 4.
[10] Based on traditional links to UNEG, some ECG members/observers participate in UNEG meetings.
[11] The approved GPS are on (i) Independence of IFIs' CED, June 2010, (ii) Evaluation of Public Sector, February 2012, (iii) Evaluation of Private Sector Operations, November 2011, and (iv) Country Strategy and Program Evaluation, 2008.
[12] ECG has developed a separate set of standards for the assessment of the evaluation functions of IFIs in ECG's Review of Framework for the Evaluation Function in Multilateral Development Banks, 2009.
[13] A core GPS is defined as one that establishes the key principles for CSPEs and is necessary to permit comparability of evaluation results, to the extent possible, among MDBs. While the core GPSs listed in this paper are currently in practice to some extent in all members, institutional differences may affect the pace at which harmonization can be achieved. An optional GPS is defined as one that is not strictly needed for comparability but is nonetheless designed to help improve accountability and learning within each institution.