ECG Good Practice Standards
Annex III.5: Guidance Note 3: Special Considerations for Policy-Based Lending
Published by: Independent Evaluation Group (IEG), World Bank Group
1. This guidance note suggests ways in which the EPs and OPs in this document may be applied to Policy-Based Lending. The guidance is taken from the PBL GPS,[1] the background paper for the PBL GPS prepared by the Asian Development Bank,[2] as well as current guidelines and practices of the four ECG members that provide policy-based lending (Asian Development Bank, African Development Bank, World Bank, and Interamerican Development Bank).[3]
2. Policy-based lending is a form of lending in support of a time-bound set of discrete changes in public policy. PBLs may be directed at the economy-wide or sectoral level to support global, regional, national, sector, or thematic objectives. PBLs may occur as individual operations or a series of operations (a programmatic series), and disbursements made in single or multiple tranches. The agreed-upon policy and institutional actions determine a set of conditions or prior actions, against which tranches are disbursed or subsequent operations in the series are approved. Functionally, two main categories of PBLs can be distinguished: (i) stabilization operations, focusing on macroeconomic measures aimed at restoring short-term balance of payments and fiscal equilibrium; and (ii) development policy operations, focusing on policy and institutional reforms aimed at improving the medium-term structural, sector, and sub-national enabling environment for growth, poverty reduction, and sustainable development.
3. PBL inputs are the agreed-upon policy conditions, finance, and technical assistance (TA); PBL outputs refer to the implementation of reforms and the disbursement and utilization of PBL finance. PBL intermediate outcomes are the changes in the policy or institutional environment that occur as a result of the implementation of reforms. PBL final outcomes are changes in economic, social, environmental, and human development performance attributable to PBL-supported policy and institutional reforms.[4]
Timing
4. The evaluation principle on the timing of ex-post evaluations is the same for PBLs as investment loans: PERs are timed to ensure that sufficient time has elapsed for outcomes to be realized and for the sustainability of the operation to be apparent. However, the outputs and outcomes generated by PBLs may become apparent only some time after a program has been completed. In addition, time is needed after an operation has been completed to assess whether or not the government has stayed the course and implemented agreed upon reforms. Thus, compared to investment loans, the appropriate time for post-evaluation of PBLs may be longer after project completion. Where there is a series of overlapping and related PBLs, the timing question is more complicated, since the policy reform effort will have been supported by several operations. In such instances, the appropriate timing of an evaluation depends on a judgment of when the outcomes of a sequence of related PBLs are likely to be realized.
Objectives and Results Chain
5. The statement of objectives in PBL appraisal or legal documents may be very general, and the evaluator may need to construct the operation's statement of objectives from various sources. The design logic for a PBL operation is often implicit in the policy matrix, the description of the reform measures, or the development policy letter. As with investment loans, the operation's key performance indicators may be used (with caution) to infer objectives or elements of the results chain.
Prior Actions
6. "Prior actions” or "triggers” in PBLs correspond to the inputs and outputs in the program's results chain. Applying the principle that evaluations focus on outcomes rather than inputs and outputs means that prior actions taken and triggers met do not by themselves provide sufficient evidence of achievement of program objectives. In assessing Effectiveness, PBL evaluations provide evidence on the achievement of intermediate and final intended outcomes, i.e., changes in the policy and institutional environment (rules of the game and incentives) and the resulting changes in the intended social, environmental, and human development outcomes contained in the program's statement of objectives.
Other Topics
7. PBL evaluations cover the following topics, either as part of the assessment of the six core criteria (Relevance, Effectiveness, Efficiency, Sustainability, IFI Performance, and Borrower Performance) or in other sections of the report:
· Macroeconomic stability: A PBL is evaluated with respect to its contribution to improved macroeconomic balances, whether or not this is stated in the operation's statement of objectives. This would include an assessment of the reasonableness of the macroeconomic assumptions and program and the performance of the authorities in correcting macroeconomic imbalances. It would also include any unintended outcomes.
· Political economy: PBL evaluations examine the degree to which the political economy of decision making was conducive to reform. Included in the assessment are the process of policy decision making, the role of reform champions, the likely winners and losers in the reform process, and the incentives facing those affected by reform. In addition, (i) PBL evaluations assess the Government's commitment to reform. A variety of methods can be used: leadership analysis, stakeholder analysis, institutional assessment, and reform readiness analysis; and (ii) PBL evaluations assess the adequacy of political support for reform, the degree to which reform objectives and likely effects were communicated to the public, and the extent to which PBL design reinforced national ownership.
· Complementary reforms: PBL evaluations include a thorough evaluation of the extent to which PBL outputs (the implementation of reforms) were achieved. Evaluators assess not only the extent to which inputs were delivered (i.e., agreed-upon reforms took place), but also the degree to which complementary measures necessary for their implementation occurred. These may include changes in legislation, regulation, public awareness, and institutional arrangements.
· Adjustment costs: For PBLs that earmark resources to defray budget or adjustment costs, the PBL evaluation assesses the extent to which adjustment costs materialized, and the extent to which PBL-provided resources were sufficient to meet these obligations.
· Institutional development: The extent to which a PBL and its associated TA contributed to fostering institutional development is covered in PBL evaluations. This can be evaluated in terms of whether or not improved governance practices or improved skills, procedures, incentives, structures, or institutional mechanisms came into effect. Evaluating the contributions made by a PBL to building the capacity to lead and manage the policy reform process is also important.
· Impact on the poor and other specific groups: PBL evaluations need to assess whether or not a reform operation could have been more pro-poor in its design and implementation. Intended and unintended socioeconomic impacts on the poor and other specific and /or targeted beneficiary groups are assessed. The adequacy of measures planned at appraisal to protect the welfare of vulnerable groups during the adjustment process is analyzed.
[1] ECG (2004), Good Practices for the Evaluation of Policy-Based Lending. Addendum to the 2002 GPS.
[2] Asian Development Bank (2005). Good Practices for the Evaluation of Policy-Based Lending by Multilateral Development Banks. Prepared for the Evaluation Cooperation Group of the Multilateral Development Banks, March.
[3] In particular, Asian Development Bank (2006), Guidelines for Preparing Performance Evaluation Reports for Public Sector Operations, Addendum 1 on "Evaluating Program Lending”; and African Development Bank (2001), Revised Guidelines on Project Completion Report (PCR) Evaluation Note and Project Performance Evaluation Report (PPER), section on "Special Considerations in Evaluating Adjustment Lending Operations”. These references also contain guidance on sub-criteria under Relevance, Effectiveness, Efficiency, and Sustainability for PBLs.
[4] ECG 2004, p. 2.