ECG Good Practice Standards
Annex V.2: Subcriteria for Evaluating Country Strategies and Programs
Published by: Independent Evaluation Group (IEG), World Bank Group
1. What follows is a suggested list of possible subcriteria that multilateral development bank (MDB) evaluators can draw from in tailoring the interpretation of evaluation criteria to the circumstances merited by each particular country case. This is neither a comprehensive nor a minimal checklist. The subcriteria listed here have been found to be important determinants of country assistance performance in MDB evaluations. They can be used to select and define the subcriteria employed in evaluating specific country cases. This is aimed at providing the flexibility required in a country evaluation so that the evaluative criteria are interpreted in a way that is most suitable, given varying country contexts, assistance roles, and data availability.
2. These subcriteria are divided into two groups. The first group belongs to standard evaluation criteria that can be applied to the program as a whole, or to particular components (e.g., sectors or themes). The second group belongs to additional evaluation criteria. For each, an evaluative judgment is required to assess the degree to which each chosen subcriterion has been achieved.
Standard Evaluation Criteria
A. Relevance: the degree to which the design and objectives of the MDB's strategy and program of assistance were consistent with the needs of the country and with the government's development plans and priorities.
· Based on a valid diagnosis of the context for external assistance
® development context thoroughly reviewed
® adequate assessments of key sectors and thematic areas of the MDB's proposed intervention
® candid review and assessment of government policies and strategies
® robust consultative process to identify and validate priorities
® careful assessment of feasibility of using country systems
® careful review of lessons of past experience
® informed understanding of factors driving aid effectiveness
· Consistency with country's long-term development requirements (for each major objective)
· Consistency (i.e., alignment) with government's development (or poverty) strategy and priorities (for each major objective)
· Designed in a manner consistent with government's institutional capacity to absorb external assistance
· Consistency with global agreements (e.g., MDGs, Paris Declaration commitments, or World Trade Organization (WTO) regulations) for each major strategic objective
· Consistency with the MDB's corporate policy and strategy (for each major objective)
· Importance of program objectives addressed to meet critical development constraints (by category, such as macroeconomic management, structural reform, sector reform, private sector development, institutional development, human development, environmental reform, and infrastructure development)
· Any important objectives that, in hindsight, should have been pursued, but, in the end, were not (i.e., were any important development issues omitted or ignored in the diagnosis?)
· Program formulation and design were relevant to achieving objectives
® adequacy of external financing for program operation
® extent and appropriateness of medium-term framework
® consistency and coherence of the program logic (e.g., identification of goals to be achieved; specific purpose[s] of the MDB's assistance; and program measures, their expected outputs, outcomes, and impacts, together with key assumptions and risks to performance all identified)
® appropriate assistance instruments selected (e.g., assistance properly sequenced to reach targets, internally consistent, realistic/feasible, manageable, and with clearly defined targets and objectives)
® social consequences assessed, and suitable mitigation measures incorporated in overall program design
® performance risks (both internal and external) adequately identified, and suitable strategies for managing risk incorporated
® realistic time frame for results to be delivered, given institutional and other constraints
· Extent to which sector and thematic objectives were sufficient to achieve a level of critical mass, balanced among objectives, selective, and focused
· Extent to which dialogue and consultation ensured effective ownership of the program by government and by society at large
· Degree to which the MDB's program was built on lessons from past experience, was sufficiently focused and selective, and drew on areas of its core competency
· The MDB's program took into consideration, and was harmonized with, assistance provided by other development partners
· The MDB's responsiveness in designing and then adapting the assistance strategy to fundamental changes in client circumstances throughout the implementation period
· Extent to which the assistance strategy and program maintained relevance to the client's development constraints and priorities over time
· Assistance strategy and program could be readily evaluated
® targets well defined, links traced, baseline values provided, and performance targets specified
® reporting, monitoring, and evaluation responsibilities assigned, and funding provided
® knowledge gaps identified, and actions identified for securing information needed for decision making included
B. Efficiency: the extent to which the design and delivery of assistance were most cost-effective.
· Readiness for implementation of all products and services was secured
· Products and services were delivered in a timely manner
· Extent to which strategic objectives were achieved on time
· Were benefits gained from early completion of assistance (or costs incurred from late completion)?
· Benefits of major interventions are, or are expected to be, substantial, as demonstrated by
® positive economic rates of return for major investments
® positive financial rates of return and/or return on equity for MDB-supported private investments
® major policy or institutional reforms were undertaken that did ease critical constraints to improved socioeconomic performance and poverty reduction
® unambiguous evidence that benefits reached the poor
· Debt assumed and adjustment costs from MDB-supported reforms were relatively low compared with value arising from achievement of socioeconomic objectives (i.e., social benefits likely to exceed social costs)
· Overall program financing was provided in a timely manner through
® financing provided in sync with external financing requirements
® reasonable time for project design, negotiation, and effectiveness
® disbursements took place according to plan
· Costs of providing assistance were similar or less than those in comparator country programs and were kept in line with the MDB's norms
· Unit costs were reasonable for major investments
· Transaction costs of providing assistance were modest (in terms of time spent preparing projects, number of missions undertaken, extent to which efforts were made to combine or hold joint missions with other development partners, and time spent by key government officials in design and oversight of the MDB's program)
· Public expenditures made adequate provision to meet government's portion of program counterpart costs, and sufficient financing was provided for future recurrent cost requirements
C. Effectiveness: the extent to which the assistance instruments achieved the intentions and objectives set.
· Degree to which activities anticipated in strategy and program were actually undertaken
· Sufficient interventions were undertaken to generate outputs and outcomes identified in country strategy and/or program
· Performance of portfolio as a whole was satisfactory in comparison with MDB-wide averages
· Extent to which major issues arose during execution and were (or were not) resolved
· Extent to which main assistance program objectives achieved progress toward each of their stated objectives
· Extent to which results defined under country assistance program were actually achieved
· Extent to which there were major shortcomings, such as unintended social costs or environmental damage, in achieving program objectives
· Performance as assessed by rating of the MDB's projects (both self- and independent ratings) in terms of achievement of major objectives
· Project evaluation judgments regarding achievement of development objectives have verifiable claims
· Extent to which achievement of program objectives demonstrated best practices in some areas
· Extent to which factors beyond government's control influenced the outcome of program (including changes in world markets, natural calamities, war/civil disturbance)
· Were other performance assessments reviewed and presented for major components of the MDB's assistance (including those whose findings contradict the evaluation)?
· Extent to which actual performance met or surpassed benchmarks for financial performance of similar categories of private investment (for private sector operations)
D. Sustainability: the likelihood that actual and anticipated results will be resilient to risks beyond the program period.
· Absence of major policy reversals
· Continued borrower commitment to assistance program objectives demonstrated through postprogram implementation of related measures
· Sociopolitical support for main objectives of assistance program
· Adequacy of institutional arrangements for implementing agreed upon reforms and program measures
· Conducive macroeconomic and political setting (i.e., stable and supportive)
· Continued need for (i.e., ongoing relevance and value of) the results and benefits
· Ownership by government and other key stakeholders
· Financial capacity to address recurrent costs
· Degree of resilience to risk of development benefits of country assistance program over time, taking into account:
® technical resilience
® financial resilience (including policies on cost recovery)
® economic resilience
® social support (including conditions subject to safeguard policies)
® environmental resilience
® ownership by government and other key stakeholders
® institutional support (including a supportive legal/regulatory framework and organizational and management effectiveness)
® resilience to exogenous effects such as international economic shocks or changes in political and security environments
E. Impacts: the MDB's contribution to long-term changes in development conditions.
· Anticipated and unanticipated (positive and negative) impacts identified and adjusted to take into consideration unexpected shocks or other factors beyond program's control, such as
® country's macroeconomic balance
® country's economic performance
® poverty reduction
® social development
® governance
® environmental sustainability
® gender equality
® regional cooperation
® transition from central planning to market economy
® other major social, political, or institutional changes in context
· Extent to which the program has improved the government's capacity, in key sectors and thematic areas, to make effective and efficient use of its human, financial, and natural resources
· Commercial performance of the MDB's private sector operations, i.e., degree to which these have had wider impacts on private sector development and extent to which these have catalyzed private sector investment in the country
· Anticipated and unanticipated impacts from major projects or programs identified for illustration of magnitude and pattern of intervention effects (e.g., from impact studies or beneficiary surveys)
· Evidence that impacts attributable to country program have been, to the extent feasible, isolated from those caused by other factors
· The program's additional contribution to development impacts (e.g., delivering relevant knowledge or advice, catalyzing change, and fostering more effective use of external resources)
· Degree to which the MDB's assistance makes a meaningful contribution to the government's efforts to foster achievement of the following MDGs:
® reduce the proportion of people living in extreme poverty by half between 1990 and 2015
® enroll all children in primary school by 2015
® make progress toward gender equality and empowering women by eliminating gender disparities in primary and secondary education by 2015
® reduce infant and child mortality rates by two thirds between 1990 and 2015
® reduce maternal mortality ratios by three quarters between 1990 and 2015
® provide access for all who need reproductive health services by 2015
® implement national strategies for sustainable development by 2005 so as to reverse the loss of environmental resources by 2015
Additional Criteria
A. Positioning: a measure of how well the MDB responded to (or even anticipated) the evolving development challenges and priorities of the government, built on its comparative advantage, and designed the country strategies and programs in a manner that took into consideration the support available from other development partners.
· Country priorities and the MDB's corporate priorities were aligned
® country goals and the MDB's corporate goals were aligned
® strategic pillars were aligned to contribute to country strategic objectives
® strategic gaps and risks were identified and agreed upon with the government
· Timing and scope of the MDB's engagement were in what turned out to be major development priorities of country
· Program was results-oriented, coherent, and translated strategy into appropriate operations, which collectively addressed critical development constraints
· Strategic focus was appropriate by sector, target group, and geographic area
· Program provided critical mass of assistance, sufficient to generate sustained results
· Mix of lending and nonlending services, as well as operational approaches, were tailored to the particular conditions of the country
· Productive relationships were forged with other development partners within the wider framework of development cooperation in the country
· The MDB was well positioned to respond effectively to country priorities
® the MDB was structured, staffed, and managed to respond effectively to client requests
® institutional arrangements fostered the generation and use of new knowledge to spur innovation
® assistance was managed for delivery of development results
® corporate safeguards were adhered to
· Results were delivered, and the most strategic opportunities for assistance were exploited effectively
B. Coherence:[1] the extent to which there were measures aimed at fostering internal and external synergies within the MDB's program; this can include complementarity between different program elements, the extent to which the MDB's policies are self-reinforcing, and the extent to which external partnerships promote an efficient and effective division of labor in providing assistance that allows for complementarities and synergies with other development partners' programs.
· Country priorities served to establish main development objectives
· Country strategies were realistic for forging progress toward selected development objectives and were aligned with, and supportive of, implementation of national development strategies and policies
· Country assistance program was designed to make a substantial contribution to achievement of defined objectives
· Choice of assistance, across objective area, included measures that would be innovative and have positive synergies and demonstration spillovers and foster complementary activities, so that value of program as a whole would be greater than sum of its individual parts
· Strategies and assistance choices were aligned with and supportive of assistance provided by other development partners in an effective division of labor
· Assistance instruments were chosen, and effectively integrated, to ensure that response to development challenges was sufficient, complete, and cohesive
· Choice of sectors, regions, and target groups was consistent with needs identified to meet program objectives
C. Institutional Development: the extent to which the MDB's assistance improved or weakened the ability of the country to make more efficient, equitable, and sustainable use of its human, financial, and natural resources, for example through better definition, stability, transparency, enforceability, and predictability of institutional arrangements; and/or better alignment of missions and capacities of organizations with their respective mandates.
· Contribution toward improving/strengthening capacity of public institutions to ensure stable, transparent, enforceable, and predictable execution of their mandates:
® soundness of economic management
® structure of public sector, and, in particular, civil service
® institutional soundness of financial sector
® soundness of legal, regulatory, and judicial systems
® extent of monitoring and evaluation systems
® effectiveness of aid coordination
® degree of financial accountability
® informal norms and practices that govern social and economic interactions
® extent of building nongovernment organization capacity
® level of social and environmental capital
· Contribution toward improving organizational capacity (in planning, policy analysis, skills upgrading, public awareness building and consultation, management, restructuring, decentralization, management of information systems, financial controls, financial restructuring, regulatory enforcement, and agency governance)
· Contribution toward improving private sector capacity (i.e., improving rules of the game for efficient, broad-based private sector development)
· Contribution to improving stability, diversity, and growth potential of financial sector services
· Contribution to improving nongovernment organization and civil society capacity
· Contribution toward improving governance of public sector (i.e., transparency, checks and balances, public participation, improved fiduciary policies and practices, and accountability in discharge of public duties)
· Extent to which capacity has been developed within the government to manage formulation and implementation of suitable public policies and programs
D. Borrower Performance: focuses on the processes that underlie the borrower's effectiveness in discharging its responsibilities, with specific focus on the extent to which the government exhibited ownership of the assistance strategy and program.
· Shared ownership of the MDB's country strategy and program
· Maintained high-level dialogue with the MDB's personnel and management
· Consulted with civil society and other stakeholders on program implementation
· Supported high-quality preparation of MDB-assisted projects:
® degree of ownership and involvement in identification and design
® political support for project-related reforms secured
® adequate institutional arrangements for program implementation
· Provided sufficient counterpart funds and project personnel
· Followed procurement and safeguard (i.e., resettlement, environmental, indigenous peoples, and fiduciary) guidelines
· Carefully supervised project implementation
· Engaged in high-quality dialogue on policy matters with the MDB
· Implemented policy reforms, agreed upon between the MDB and government, in a timely manner
· Provided policy framework supportive of effective aid utilization (i.e., supportive macroeconomic policies and complementary structural and sector policies)
· Fostered public outreach, disclosure, and awareness building throughout program implementation
· Provided results-based monitoring, evaluation, and reporting
E. MDB Performance: focuses on the processes that underlie the MDB's effectiveness in discharging its responsibilities as a development partner, including compliance with basic corporate operating principles; consistency with furtherance of the MDB's corporate, country, and sector strategies; and its client service satisfaction.
· Quality of strategy and program at entry:
® appropriate degree of selectivity
® grounding in recent economic and sector work
® adequate economic and financial rationale
® adequate risk assessment
® realistic assessment of financial requirements and borrowing capacity
® incorporation of lessons identified in past evaluations
® adequate institutional analysis
® adequate poverty, social (including gender), environmental, and stakeholder analysis
® incorporation of monitoring and evaluation indicators and reporting procedures
® focus on areas of MDB comparative advantage
® appropriate mix of assistance instruments selected
® assistance strategy and program was suitable, given country context and institutional capacity of the government
· Quality of the MDB's supervision:
® degree to which supervision focused on achieving objectives
® degree to which civil society participation was fostered in program implementation
® problems identified during implementation were expeditiously assessed and resolved
® adequate resources devoted by the MDB to supervision
® attention paid to monitoring and evaluation data and processes
® quality and timeliness of self-assessment (i.e., country strategy completion reporting)
· Quality of other services:
® built client ownership of the assistance program
® built strong links between strategy and analytical and advisory services
® provided high-quality knowledge products
® maintained high-quality dialogue with government and civil society
® maintained high quality at entry for new projects
® explained and provided training in its policies, safeguards, and procedures
® provided personnel with appropriate skills mix to develop strategy and program
® strengthened the government's capacity for financial management and accountability
® enforced compliance with procurement guidelines, audit requirements, and other project cost controls
® managed portfolio effectively
® provided timely notice to the board of fundamental changes in its strategy
® solicited feedback on, and was responsive to, requests for ways of improving its performance
® provision of necessary long-term financing
® provision of suitable risk mitigation services
F. Partner Coordination: the contribution made by the MDB to coordinating external assistance and to building government and country ownership of external assistance processes.
· Degree to which assistance fostered government leadership of aid coordination
· Degree to which assistance built the government's capacity to plan its public investment and to mobilize and manage external assistance (including debt management) effectively
· Degree to which the MDB played a role in catalyzing or otherwise inspiring other stakeholders to cooperate toward achieving common development results
· Degree to which policies and strategies pursued by other partners were consistent with those pursued by the MDB (i.e., if there were major conflicts or inconsistencies, were steps taken to resolve these?)
· Extent to which assistance effort played catalytic role in resource mobilization
· Degree to which the MDB coordinated and mobilized aid resources effectively
· Degree to which the MDB served as an effective aid partner in terms of knowledge sharing, support for and participation in multipartner initiatives, design of complementary assistance initiatives, assistance provided to other partners to resolve problems of wider concern, and active participation in aid coordination arrangements
Sources: African Development Bank. 2004. Guidelines for Country Assistance Evaluation (prepared by O. Ojo). Tunis; Asian Development Bank. 2006. Guidelines for the Preparation of Country Assistance Program Evaluation Reports. Manila; Inter-American Development Bank. 2003. Protocol for the Conduct of Country Program Evaluations, Revised Version, Report RE-271-1. Washington, DC; Organisation for Economic Co-operation and Development–Development Assistance Committee. 1999. Evaluating Country Programmes. Report of the Vienna Workshop. Paris; World Bank. 2003. Country Evaluation Guidelines (Internet version) and Country Questionnaire. Washington, DC; and 2005. Country Assistance Evaluation Retrospective: An OED Self-Evaluation. Washington, DC.
[1] Coherence is a separate evaluative criterion used by the Office of Evaluation and Oversight of the Inter-American Development Bank. It is assessed as part of positioning by the Operations Evaluation Department of the Asian Development Bank.