The Independent Evaluation Department (IED) of the Asian Development Bank (ADB) evaluated the use of policy-based lending (PBL) by ADB between 2008 and 2017. The design and reform focus of ADB PBL fundamentally changed over this period, and the success rates—as judged by project completion reports validated by IED—more than doubled (from about 33 percent to over 80 percent), a trend also experienced by other multilateral development banks (MDBs). Improved performance appears to have coincided with the growing use of single tranche PBL and, with it, the use of prior actions (actions completed before loan approval). These changes substantially reduced disbursement risks and increased the capacity of MDBs to provide more predictable and reliable budget support in response to country financing needs, the primary objective of the instrument. 

A key issue is whether the need to respond efficiently to country financing needs has encouraged support for less critical reforms. Over time, PBL reform topics seem to have shifted from more politically sensitive reforms (such as reform of state-owned banks) to more technical reforms connected with public financial management (PFM). PBL modalities also changed as the second tranches of loans, which often contained more difficult policy actions, were no longer part of PBL design. The policy actions in second tranches often required waivers or loan cancellations contributing to poor performance ratings. There appears to have been a trade-off between the emphasis on efficient, rapidly disbursing modalities to meet country financing needs and the emphasis on policy reform, suggesting that the two objectives are not always automatically compatible.

PBL performance dramatically improved over the evaluation period, but the evaluation identified several design issues. For example, PBL tended to be used in the region’s more developed economies (Pakistan is an exception) with greater capacity for reform. It rarely focused on policy reform in areas of infrastructure development, ADB’s main comparative advantage. Moreover, it was difficult to reconcile the high success rates in PBL project completion reports validated by IED with the evaluation’s finding of design shortcomings. A key reason for this discrepancy is that although PBL performance significantly improved over the evaluation period, the causal chain from policy actions to country-level results was often difficult to discern. Where there is doubt about whether a PBL outcome resulted directly from the policy actions taken, the responsibility often falls on the evaluator to prove the connection by, for example, constructing a counterfactual to show whether the result would have been achieved without the PBL. In practice, however, if outcome indicators are achieved, the PBL is usually rated successful even if causality between ADB-supported policy actions and the reform outcome is tenuous. 

PBL remains an efficient modality for supporting country clients through crisis periods. Its usefulness was shown by ADB’s rapid response to the global economic and financial crisis in 2007–09 and the COVID-19 pandemic in 2020. The use of PBL spiked during crisis years, breaking through the 20 percent ceiling imposed on this type of lending in the sovereign loan portfolio. The increase in PBL use beyond the ceiling was made possible only through the introduction of “reform-free” and rapidly disbursing budget support modalities to finance developing member countries’ (DMC) countercyclical public expenditure programs to mitigate the effects of the crisis. 

PBL has played several roles in the region. It supported countries through difficult periods, including economic downturns, natural disasters, and pandemics, and it supported broad public sector management and macroeconomic stability through noncrisis years. Other budget support mechanisms are also emerging, including results-based lending, which is more effective than PBL at directly improving service delivery. 

The evaluation published in 2018 made several recommendations, some strategic and others related to PBL design. At the strategic level, it recommended ADB make greater use of PBL to support policy reforms in sectors where significant project investments are also undertaken, to forge more integrated and sustainable solutions to public policy problems in these areas. Although ADB management accepted this recommendation, it is unlikely to be implemented in the short term, mainly because of the COVID-19 pandemic, which led to higher use of non-reform-based budget support responses in 2020. If such support continues, the opportunity to use PBL to support infrastructure-related policy reform is likely to be limited in the immediate term. 

The evaluation’s recommendation that ADB develop an operational plan on the scope, objectives, and articulation of public sector management interventions was not accepted formally, but ADB has moved in this direction. An operational priority plan for governance and institutional capacity has since been developed as part of ADB’s Strategy 2030. This plan provides corporate guidance on the conditions under which PFM loans should be provided.31
 

The evaluation recommendations that concessional assistance-only countries (Group A) be granted access to a countercyclical facility and that the use of contingent disaster financing be formalized were accepted. Since the outbreak of the pandemic, countercyclical support has been expanded to include Group A and non-OCR-eligible countries32
(Group B) as part of ADB’s pandemic response, which includes using both Asian Development Fund grant resources and ADB concessional loan resources. ADB formally approved contingent disaster financing soon after the evaluation was issued.

ADB did not agree to the evaluation’s recommendation that in the rare cases where a regional department’s view on the macroeconomic situation of a country diverges from that of the IMF the risks involved be assessed independently of the regional department. Still, it has since strengthened the capacity of the Strategy, Policy, and Partnerships Department (SPD) to oversee PBL design before board approval. SPD has revised the PBL provisions of the Operations Manual and the relevant staff instructions, which now include a specific loan approval template and a design and monitoring framework better suited to PBL. ADB’s relationship with the IMF has been clarified and ADB’s capacity to produce a clear macroeconomic assessment strengthened, helping it support the overall quality assurance mechanism for PBL. ADB management decided against the recommendation of a separate three-year PBL operational review like the one the World Bank produces. 33

ADB has strengthened PBL design, although the evaluation recommended ADB limit the use of process-oriented actions and articulate policy actions as substantive outputs. It recommended tailoring the design and monitoring framework (DMF)34
so policy actions, outputs, and outcomes are more clearly linked and the analytical work underpinning PBL design and policy actions clearly referenced. These recommendations are part of the revised Operations Manual and staff instructions, but the outbreak of COVID-19 and the need to respond quickly to DMC financing needs during the pandemic has meant that implementation of these changes was initially deferred.

ADB needs to strengthen its assessment of PBL design at program completion. It also needs to focus more sharply on the role and quality of technical assistance, given its central role in the preparation and implementation of PBL. As PBL requires its own template and DMF, new approaches for assessing PBL performance need to be introduced to make sure the success rating given to completed PBLs is based on a robust evidence-based assessment of the design, especially the relevance and criticality of policy actions to development outcomes. In a single tranche PBL, policy actions have been carried out at the time of board loan approval but their relevance and criticality to the outcome should still be assessed at completion. ADB’s policy-based and project lending (on commitment bases) for the period 2000–20 is shown in Figure 2. Although non-PBO lending fell slightly during 2019–20, PBO lending increased sharply in 2020, making ADB lending highly countercyclical compared to other MDBs and the EU. 

Note: PBL = policy-based lending

Source: ADB-IED.

Comments by Homi Kharas

He posed three questions: What issues affect the development effectiveness of PBL? Does the chapter capture the issues well? How is PBL linked with the Sustainable Development Goals (SDGs), which all member countries of ADB have signed on to? 

PBL provides rapid financing to governments along with support for a policy reform process. The chapter describes the trade-offs involved in this process well. Financial support sometimes needs to be large and rapidly disbursed to have impact, especially during a crisis. By contrast, policy reform is often a long, slow process of incremental change and institution building. Over time, PBL has tended to address the finance objective more than the policy reform objective, as evident in the greater reliance on prior reforms; the shift to public sector management reforms within control of the ministry of finance (which, in contrast to sectoral ministries, has an incentive to deliver on reforms, as it also gets to allocate PBL resources); and the delinking of loan volume from the difficulty or cost of reform implementation. 

This evolution of PBL may be positive, but it also changes the nature of the instrument. The recent trend in ADB is positive for several reasons. First, it puts countries firmly in the driver’s seat on the pace of reforms. As a development partner, it is appropriate for ADB to comment on and advise counterparts on the nature, pace, and sequencing of a reform program. It is not appropriate to use financing to bolster ADB’s own views over those of elected officials, unless there is a risk that the government program is so weak that a default could occur, or the economic context is so distorted that the loan could be immiserizing.

The chapter suggests that ADB should pay more attention to transport, energy, and water infrastructure reforms, to align with areas in which ADB has significant sectoral expertise. ADB has expertise in these areas it can and should share with governments, but it is not clear that using PBL as an instrument to force such reforms is appropriate. Adjustment loans are about providing liquidity, not forcing (or, more politely put, encouraging) specific policy reforms. Of course, a series of PBL operations can be used to structure reform incentives in the right way, but structuring the operation in a certain way is more about the pace, sequencing, and difficulty of reforms rather than the selection of one sector over another.

Second, one lesson of crisis management is that “too little, too late” has long-lasting harmful consequences. But how much is “too little”? It would be useful to have had some discussion of whether ADB’s PBLs always complemented International Monetary Fund (IMF) programs. The chapter has a brief discussion of the need for ADB to have in-house capacity to perform its own macroeconomic assessments. This recommendation may be correct, but a strong partnership, including shared analytical assessments, with other crisis lenders is possibly more important. Has ADB ever moved ahead absent an IMF program or an IMF letter of comfort on the macroeconomic front? How often is an ADB PBL part of a financing package to a government that also includes other development partners, notably the World Bank? 

Data from the International Aid Transparency Initiative indicate that ADB has one of the best records of disbursement against commitments of PBL operations in response to COVID-19 of all MDBs. That is a strong testament to the value of tilting toward finance.

The chapter correctly notes it is difficult, if not impossible, to develop a strong causal link between PBL operations and actual results, given so many other factors also affect the results. PBL is likely to become even more important than it is now, partly because it provides a unique source of affordable, flexible, countercyclical, long-term development finance. The form may change toward greater pooled funding, including through country platforms (it would have been useful had the paper commented on the ongoing pilots, which will probably be supported by sector development program loans), but the sharp focus on public finance will surely remain. 

• 31
  
  Asian Development Bank. 2019. Strategy 2030. Operational Plan for Priority 6. Strengthening Governance and Institutional Capacity 2019–2014. Manila.
• 32
  
  OCR refers to Ordinary Capital Resources, which are market-based resources.
• 33
  
  World Bank Group. 2022. 2021 Development Policy Financing Retrospective: Facing Crisis, Fostering Recovery, Operations Policy and Country Services, March 16, Washington, DC.
• 34
  
  Initial information about the project, results chain, performance indicators, data sources, and risks.

Clement Bansé and Stephanie Yoboué. The Independent Development Evaluation at the African Development Bank Group (AfDB) carried out two major independent evaluations of the policy-based operation (PBO) instrument. In 2011, it conducted an evaluation that covered 1999–2009; in 2018, it conducted an evaluation that covered 2012–17. The chapter draws on both evaluations, supplemented by recent data.

PBOs are fast-disbursing financing instruments AfDB provides to countries as loans or grants. They address the actual, planned, or unexpected development financing requirements of AfDB’s Regional Member Countries.

The evaluation found that PBOs remained a relevant and useful instrument for AfDB and its clients, although PBOs were challenging to design and manage effectively. It found the relevance of the PBOs in AfDB’s portfolio to be broadly satisfactory, based on their programming, design, and broad adherence to AfDB’s policy and guidelines as well as international good practice. As for the achievement of reform objectives, the overall picture was also satisfactory.

It was much harder to find evidence of AfDB’s influence on reform direction and speed. Even in the presence of strong ownership, concerns about the institutional and financial dimensions of sustainability meant that the outlook for sustainability in the sectors examined was unsatisfactory. 

AfDB deployed UA7.2 billion (about $10.8 billion) in PBOs in 2012–17, but it did not invest in its own institutional infrastructure to obtain maximum value from the instrument. As reflected in its 2012 policy, PBOs were expected to form part of a “package of support,” to ensure that they influenced and supported reform agendas while providing important funding. This package included analytical work to inform technical input, policy dialogue, and capacity support. In practice, AfDB underperformed with policy dialogue, despite its strong position as a trusted partner, partly because of its institutional arrangements; the lack of clarity about who was responsible for policy dialogue; the way the dialogue was conducted, reported, and coordinated; and a lack of investment in human resources to conduct it. In addition, AfDB underperformed in providing timely and adequate capacity support and specialized technical advice, partly because of the limited menu of instruments available with which to do so. These shortcomings had implications for how well AfDB influenced or added value to country reform paths. 

The evaluation examined a range of programming issues. Although the overall picture was assessed to be broadly satisfactory, the evaluation identified areas that could be strengthened. First, use of most of the PBOs reviewed was envisaged in either the relevant country strategy paper or the midterm review, in line with the policy. However, assessment against the eligibility criteria usually was made for the first time during the PBO preparation phase. The justification for the type of PBO chosen could also have been stronger, especially when the PBO did not use the recommended programmatic approach. 

Second, in two-thirds of the operations reviewed, the analytical underpinnings used were listed and relatively complete. However, exactly how this work informed or underpinned the design of the operation was not clear. 

Third, although risk assessment was assessed as satisfactory in two-thirds of the operations reviewed, reputational risk was rarely explicitly considered. The risk mitigation measures, such as future capacity support to address current risks, were generally not convincing within the time frame of a PBO. 

The evaluation cites many good examples of how AfDB coordinated with other development partners, notably during the identification and appraisal periods. AfDB staff took coordination seriously and invested in upfront work with other development partners. However, the in-depth assessment illustrated how difficult it was to sustain these initial high levels of coordination throughout the implementation phase. Moreover, following adoption of the G-20 Principles for Effective Coordination between the IMF and MDBs on Policy-Based Lending in 2017, in countries facing macroeconomic vulnerability MDBs needed to align with the IMF. 

Although two-thirds of the PBO appraisal reports examined stated that complementary inputs could play an important role, only a handful explained how they were to do so. All PBO results frameworks defined baselines, targets, and means of verification, and integrated prior actions and triggers. However, over one-third were less than satisfactory because of (a) weaknesses in presenting a convincing results chain; (b) the large share of process- and action-based indicators; and (c) a lack of realism, particularly for single-year operations. The use of conditions was suitably selective; in programmatic operations they linked from one phase to the next to plot a medium-term path and were linked to broader dialogue frameworks. 

PBOs were broadly disbursed and implemented promptly, although some receiving countries reported that disbursement was unpredictable. In line with expectations for the PBO instrument, the evaluation found that AfDB had disbursed funds fully and, compared with investment projects, quickly. In addition, implementation progress was rarely identified as a cause for concern. Nine of the 10 in-depth assessments were considered efficient in terms of transactions costs and the time taken to disburse the funds. Perceptions of timeliness and transactions costs varied among both staff and borrowers’ officials, however. 

Perceptions of the efficiency and transactions costs of technical assistance or institutional support provided to support PBOs were negative. When it was provided, such support was slow and tended to arrive toward the end rather than at the beginning of a PBO series, partly because capacity support tended to be designed in parallel with PBOs rather than in advance and partly because of the limited set of instruments AfDB had available to provide small pieces of technical assistance, all of which operated like full projects rather than as rapidly deployable expertise.

AfDB did not use policy dialogue sufficiently or make best use of its “African voice” to ensure PBO results. This finding echoes that of the 2011 evaluation, which described AfDB as “punching below its weight” with policy dialogue. Only 3 of the 10 in-depth assessments had satisfactory frameworks for policy dialogue in the targeted sectors. 

Some of AfDB’s practices were out of line with both its own policy and the practices of the World Bank and the European Union. First, PBO design and management remained somewhat centralized, led by either the Governance and Public Financial Management Coordination Office (ECGF) or sector departments. The extent to which country offices took up ownership varied significantly. Second, no centralized unit provided specialized support to PBO teams. ECGF staff task-managed most of AfDB’s general budget support. This lack of a central support unit, and the limited guidance and training provided to staff, starkly contrasted with the support available at the World Bank and the European Union. 

Overall, the assessment of PBO effectiveness, which focused on energy, private sector environment (PSE), and PFM, was broadly satisfactory. The evaluation highlighted areas in which AfDB could focus to strengthen results, indicating how it could contribute to the direction and pace of reforms. Data from project completion reports and country strategy and program evaluations by the Independent Development Evaluation indicated that satisfactory assessment was likely to reflect the effectiveness of the broader portfolio. All but 1 of the 10 cases achieved or partially achieved all or most of the reform actions in the results framework (in one case, 25 percent of outputs were assessed as not having been achieved). No sector performed notably better than any other. In 7 of the 10 countries covered by the in-depth studies, the overall effectiveness in terms of achievement of the objectives stated in the Results Measurement Framework was considered satisfactory. 

Across the 21 individually assessed components, two-thirds were assessed satisfactory in terms of achievement of “landmark policy changes.”35
In one-third of the components, AfDB’s influence on either the direction or pace of reforms was evident and was usually achieved, through analytical work, technical inputs, and policy dialogue. But AfDB staff respondents to the survey supported the view that AfDB’s influence was limited and strongest at the appraisal stage. 

The sustainability of PBOs in energy, PSE, and PFM was assessed as unsatisfactory, particularly in relation to the institutional and financial dimensions of sustainability. Only 4 of the 10 in-depth assessments had good prospects for sustainability. Almost all 10 had laid strong foundations for sustainability in terms of government ownership and leadership, which should be at the core of the decision to proceed with a PBO. However, weak institutional and financial sustainability undermined the positive assessments of ownership. This trend was clear for energy, PSE, and PFM; it cannot be generalized across the whole PBO portfolio. 

The evaluation evidence from AfDB and other institutions providing budget support in Africa indicates that the most often identified factors relating to country context were ownership, country capacity, and having a champion for reforms; the country’s socioeconomic status; and country systems. The most frequent factors relating to the budget support mechanism were the quality of design, programming, development partner coordination, and monitoring and the choice of indicators. The most often cited enabling factor was the quality of design. In terms of hindering performance, the most often cited factors were insufficient policy dialogue, high inefficiency and transactions costs, the poor choice of indicators, weak monitoring, and poor predictability. 

The most significant factors associated with achievement of landmark policy changes were programming, design, and efficiency factors; technical assistance; inclusion of the operation as part of a series; and the existence of a country office. These factors were even more important than the country’s socioeconomic status. AfDB’s policy-based and project lending (on commitment bases) for 2000–20 is shown in Figure 3. Although PBO lending increased during 2019–20, non-PBO lending fell sharply in 2020. 

Note: PBL = policy-based lending

Source: Independent Evaluation Department, African Development Bank.

Comments by Alan Gelb

The chapter provides a useful overview of the evolution of PBOs at AfDB, which are described in terms of their evolving portfolio share, country distribution, variants, and focus. The chapter stimulates thinking on the evaluation of PBOs and on the institutional requirements needed to help with AfDB’s transition from a project-based to a policy-based bank. 

These policy packages were contentious. Many saw them as impinging on national sovereignty—a sensitive issue, especially for newly independent countries. There were also genuine differences of view on what constituted an appropriate trade policy for African developing countries, some of which lacked a strong indigenous business sector, and on the role of the state in facilitating economic transformation. Still (and despite many critical views of the Washington Consensus), countries that stayed on track with macroeconomic and structural reform programs generally fared better than those that did not do so. Analysis based on the World Bank’s Country Policy and Institutional Assessment (CPIA) index suggested that countries with stronger macroeconomic and structural policies as well as more efficient resource allocations fared better than others. These findings suggest that many basic elements of economic management in these early PBOs were important, even if they were not sufficient, for a resumption of growth. 

Given the increasing emphasis on PFM and economic governance, PBOs have shifted toward areas on which there is more consensus. Experts may debate the appropriate degree of trade protection, but it is rare to find arguments against more accountable public spending. This trend may not translate into easier implementation, however, because PFM reforms often confront strong entrenched interests and political opposition. They have a mixed record and, in some countries, governments have bypassed the reformed systems. As illustrated in the chapter, the third stage in the evolution of PBOs has been increasing the emphasis on sector policy components across a wide range of critical areas, although most PBOs have kept a strong focus on PFM. 

The chapter suggests that PBOs tend to be provided mostly to middle-income countries (MICs), although the degree of concentration is difficult to assess without comparative data on the relative size of MIC economies in Africa or the cross-country distribution of the overall portfolio. As MICs generally have stronger PFM and likely greater policy stability, PBOs might seem more suitable in such countries, especially as programmatic operations or programmatic tranching. Offsetting this tendency, the chapter notes the use of the crisis window to provide quick-disbursing funds during the Ebola crisis and the COVID-19 pandemic, both of which had both health and economic crises. The chapter confirms the importance of having a mechanism available to combine policy, technical advice, and financial support at relatively short notice to countries beset by exogenous shocks. 

Two-thirds of the assessed operations appear to have achieved landmark policy changes; where they did not, reasons were advanced to help explain why. But defining a results framework for PBOs against a clear counterfactual remains a challenge.

How can impact be assessed? Attribution is a fraught area for PBOs, especially where there are multiple development partners and operations are designed to support reform measures with strong country ownership and be well coordinated with programs of other partners. PBOs are intended to provide quick-disbursing support for the budget and to encourage policy and institutional reforms. The balance between these objectives varies, and the compatibility of these twin goals cannot be taken for granted. When financing needs are pressing, policy elements may take a back seat. It is also possible that the policy reforms required by an operation may be measures that the country would have undertaken anyway. 

In these circumstances, it is probably best to recognize the problem and rest content with observing whether the operation came with the expected landmark policy changes, preferably ones set out in advance as part of a well-defined programmatic or tranching operation. One-third of AfDB PBOs do not fall into one of these categories, and the share of PBOs with full programmatic tranching is modest (less than one-quarter). As loan triggers tend to become more substantive in the later years of a program, more such operations will likely be truncated, with disbursement rates above zero but less than 100 percent. 

To be effective PBO partners, funding institutions need to have the capacity to engage in policy dialogue at a high level and across critical areas, including macroeconomic management (to complement the work of the IMF, public sector and budget management, and sector policy). The chapter paints a picture of the evolution of AfDB, from project lending to an institution balanced between projects and policy and program engagement. Acquiring and sustaining capacity requires a strong analytical focus, through economic and sector work, together with research and analytical support. Doing so is challenging for any institution; as the chapter notes, for AfDB that work is still incomplete. The resource requirements of achieving this analytical basis across the full range of development sectors and policies may mean a degree of operational selectivity focused on areas of traditional strength. Substantial and continuing investment in capacity seems to be essential for AfDB to exploit its “African voice” in policy dialogue, even though it is not necessarily the major player in the region. A policy-based bank that functions well will also need the expertise and flexibility to offer complementary and timely packages of technical cooperation, a shortcoming identified in the chapter. 

If policy dialogue and reform, rather than quick-disbursing funding, is to be the driver of non-project lending, it may be useful to consider other modalities to complement, or even replace, traditional PBOs, especially considering the shift toward a larger component of sector reforms. Results-based lending, such as the World Bank’s Program-for-Results (PforR) instrument is one possibility. Like PBOs, PforR financing is provided to the government treasury, disbursed using country systems, and not necessarily tied to program costs. The multiyear nature of PforR operations allows countries sufficient time to move beyond immediate outputs and toward measures of outcomes and impacts, allowing for a more substantive results framework than those of many PBOs. Experience with this new instrument is still limited, but it could become a successor to sector-based PBOs.

• 35
  
  Landmark policy changes are budgetary or institutional changes of substance and influence targeted by PBOs within the set of intermediate outcomes (induced outputs) identified in the theory of change.

Karolyn Thunnissen. The European Commission first introduced budget support in the 1990s. The approach evolved in the context of conditionality reform and in response to the evolution of the aid effectiveness agenda. The current approach has been implemented since the beginning of the 2000s. The period was marked by a gradual shift from using only project aid, whose effectiveness was often undermined by weak policy and governance contexts. The form in which EU budget support was implemented evolved to reflect changing policy contexts and to act on recommendations by external evaluations and by the European Court of Auditors.

Unlike projects, budget support addresses the partner country’s overall conditions for economic and social development. EU budget support has always been provided exclusively as grants. It is consistent with and complementary to other EU aid implementation modalities, including projects, technical assistance, delegated cooperation, cofinancing, blending, and guarantees for investment loans by financial institutions, humanitarian aid, and emergency assistance. 

The latest EU budget support policy was adopted in 2012. Its guidelines were revised in 2017 to take into account the new European Consensus on Development that followed the international adoption of the UN 2030 Agenda for Sustainable Development and the Addis Ababa Action Agenda. This policy notably introduced a new type of budget support, known as State and Resilience-Building Contract, to help countries in situations of fragility or facing the consequences of crisis and natural disasters. This type of budget support has increasingly been used and has proved instrumental in supporting countries during the COVID-19 pandemic lately.36

The EU’s approach to budget support has always involved four interrelated components acting together in support of partner countries’ policy implementation: 

• policy dialogue with a partner country to reach agreement on the policies and reforms to which budget support can contribute
• performance assessment to achieve consensus on expected results and to measure progress achieved
• financial transfers to the treasury account of the partner country once those results have been achieved and according to their degree of achievement
• capacity development to enable countries to implement reforms successfully and to sustain results.

EU budget support is thus a performance-based modality that provides a package of grant funding, capacity development, and a platform for dialogue to partner countries in support of the implementation of their policies. Funding is fungible: Budget support is used by the partner country’s government based on domestic budgetary planning, execution, and oversight processes and using domestic PFM systems. EU budget support grants can thus be used for both recurrent and investment expenditure. 

Policy dialogue is a fundamental component of EU budget support. The general conditions (regarding public policy, macroeconomic stability, PFM, and since 2012, budget transparency and oversight) provide the overall framework for dialogue with the government and other stakeholders; variable tranche indicators enable a more in-depth discussion on key reforms and policy results. Because funds are transferred to the budget, the EU can discuss general PFM issues, overall budget allocations, and sector spending as well as its results with the partner countries’ authorities and other stakeholders. Because of the grant nature of the funding, the EU is particularly concerned that budget support should not be considered a substitute for efforts to raise revenues. Domestic resource mobilization is systematically raised in policy dialogue and often supported through capacity strengthening or the use of performance indicators. Monitoring of general policy outcomes and sector-level policy processes, activities, outputs, and most important, outcomes are an essential input into the overall dialogue.

Although external experts hired for technical cooperation can never be responsible for achieving the targets set for the agreed performance indicators, the capacity building most often associated with budget support is used to enhance the government’s capacity to design, implement, monitor, and evaluate policies and to deliver public services. As EU budget support relies on the monitoring of performance indicators, preferably outcome indicators, the strengthening of national monitoring frameworks and associated statistical systems is a priority. Attention is also systematically paid to promoting the active engagement of nongovernment stakeholders in these monitoring frameworks. 

Independent evaluation teams have undertaken 17 general and sector budget support evaluations applying the universally accepted OECD-DAC methodology for budget support evaluations (the so-called “three-step approach”) since 2010.37
They were managed by evaluation management groups made up of representatives of partner countries and funding agencies, under Commission’s coordination. Of the 17 evaluations undertaken, 11 were multidonor evaluations assessing the joint effects of all the general and sector budget support operations financed by different development partners. Evaluation periods differed slightly across the evaluations, which stretched from 1996 to 2018.

The period being evaluated (roughly 2005–2015, although seven of the eight evaluations concentrated on 2005–10) was a period of high and increasing official development assistance (ODA), with budget support the preferred aid modality of the EU and multilateral development partners. Budget support provided a significant and predictable source of funding for recipient governments and created fiscal space for them to undertake discretionary expenditure. The scale of budget support in relation to public expenditure was significant in all countries. Budget support annual disbursements represented as much as 25 percent of public expenditure in Uganda in the first half of the period; 15 percent of public expenditure in Burkina Faso; over 10 percent in Mali, Mozambique, and Tanzania; 8 percent in Ghana; and 6.5 percent in Zambia. Even in Tunisia, where it represented only 1.4 percent of public expenditure, budget support was an important source of funding for discretionary expenditure.

The predictability of the amounts of budget support was high, with disbursements close to planned amounts in most cases, even though failure to meet eligibility conditions triggered temporary suspensions of budget support by the EU and other development partners in five of the eight countries during the evaluation period. In three cases, temporary suspension was linked to the government’s breach of principles (major corruption and fraud cases came to light in Tanzania in 2007 and 2008; Zambia in 2009; and Mozambique in 2009, 2011, and 2012). The EU’s general budget support was not yet explicitly linked to respect for fundamental values but only to the eligibility criteria, which continued to be satisfied. While corrective measures were discussed and then implemented, the EU continued to disburse funds, which eased the effect of these suspensions on the government’s treasury tensions. In two other cases, Uganda (2012) and Ghana (2013 and 2014), underperformance on results, a deteriorating macroeconomic situation, and serious concerns regarding PFM triggered all development partners, including the EU, to suspend budget support, because the key conditions were no longer being met.

With these temporary suspensions and deferred disbursements of budget support because of countries’ breach of mutual accountability, the predictability of disbursement timing could not be maintained. In Mali, Uganda, and Zambia, public expenditure was delayed and the government had to seek temporary domestic borrowing. 

In most EU budget support operations, capacity development complements funding, policy dialogue, and performance monitoring. Technical assistance is used to strengthen the country’s policy and PFM systems, improve the accountability of the government toward its citizens, and strengthen key institutions and policy making processes. Typical areas of support include external oversight; monitoring and evaluation; underlying statistical data systems and processes; PFM, including gender budgeting and monitoring; and the active engagement of stakeholders in policy design, implementation, and monitoring.

Technical assistance usefully complemented budget support in backing governance reforms and reinforcing capacities in PFM, audit, and statistics in six of the eight countries. Where sector budget support was provided alongside general budget support, sector capacities also benefited from technical assistance. In Ghana, major efforts were made to strengthen the capacities of civil society organizations and to enhance their role in policy processes. Overall, technical assistance remained a minor component of the budget support package; in many instances, evaluators estimated that more could have been done with better planning and a more flexible response to strengthen capacities at the subnational level, where policy implementation takes place. 

In every result identified in all eight evaluations as a direct or indirect effect of budget support, policy dialogue featured as a central element. Dialogue related to budget support was invariably a crucial factor in improving policies, governance, and policy decision making. Through their policy dialogue, development partners were able to put and keep specific issues on the government’s priority agenda, draw attention to governance matters, and propose and discuss policy options. Development partners also used performance monitoring and the variable tranche indicators to discuss results of policy implementation, corrective measures, and implementation challenges. 

Strong coordination of budget support donors within a structured framework increased the effectiveness of policy dialogue, facilitating harmonization, alignment, and the delivery of joint messages. During the period, temporary suspensions of budget support disbursements led to a severe deterioration of relations between the government and development partners in several countries. The overall positive assessment of budget support policy dialogue was tempered in several cases by a perceived lack of government ownership and leadership of the policy dialogue as well as by extending budget support areas of interest to ever wider governance and sector issues for which reform capacities were insufficient.

General budget support was found to have induced and sometimes helped to trigger positive and mostly lasting changes in four main areas: policy formulation and implementation, the composition of public spending, PFM, and transparency and external oversight. Also, general budget support accompanied improvement in policies in several areas, depending on the objectives pursued and the weaknesses to be addressed.

Improvements in sector policies and delivery processes were substantial when general budget support was paired with sector budget support. In several countries, budget support contributed to strengthening of sector policies, adoption of a sector-wide approach, and implementing sector policies. 

Discretionary funding enabled by budget support helped governments to significantly increase their social and pro-poor expenditure, in particular in the low-income countries (LICs), which led to expanded access and delivery of services in these priority sectors. The gains in social outcomes were momentous, but not always equitable. PFM, supported by policy dialogue, technical assistance, and the monitoring of PFM performance indicators, also improved in all countries except one as shown by repeated Public Expenditure and Financial Accountability reports. Evidence of strengthened transparency and external oversight as well as sector governance was also found in most countries receiving budget support. However, in fragile countries, where budget support contributed to fiscal stabilization and strengthening the capacities of vital state institutions, the strengthening of the institutions responsible for security, justice, peace, and democratic governance has been slower than expected.

The evaluation findings summarized here should be considered in their context. This synthesis was based on evaluations undertaken in 2011–20 of budget support programs implemented during 1996–2018. Looking across the 17 evaluations and with the benefit of hindsight, some of the progress to which budget support contributed was short-lived, especially when countries experienced drastic sociopolitical, economic, or security shocks. The risk of losing progress never disappears; it therefore needs to be monitored closely during budget support implementation. This finding may also indicate the need for deeper consideration of the factors that would help ensure the sustainability of outputs and induced outputs when designing budget support and evaluating its effectiveness.

Each of the 17 evaluations made recommendations to improve the use of budget support in the country concerned and, sometimes, to improve the specific programs, policies, and institutions supported. Recommendations that were less context-specific and could be applied to the management and use of budget support included: (a) establish new types of partnerships with partner countries, using cooperation modalities and tools differently; (b) use budget support as a complement to other aid modalities; (c) strengthen policy dialogue; (d) carefully consider the choice and use of performance indicators for the variable tranches; (e) improve technical assistance.

EU’s policy-based and project lending (on commitment bases) for 2005–20 is shown in Figure 4. Although non-PBO lending remained at high level in 2019–20, PBOs were not countercyclical as they had been during the previous crisis.

Note: ODA = official development assistance

Source: European Union.

Comments by Shanta Devarajan

This chapter is a useful description of the European Union’s budget support instrument and synthesis of the 17 independent evaluations of budget support operations. The EU’s budget support programs have many distinctive aspects, some of which the chapter highlight. The synthesis of the evaluations paints a generally favorable view of EU budget support, although without a counterfactual analysis of the true impact cannot be discerned. 

The two main distinguishing characteristics of EU budget support are that it is provided exclusively as grants rather than loans and is disbursed based on observable and monitorable indicators of performance, such as progress in implementing PFM reforms. EU budget support differs from budget support operations of the World Bank or AfDB, which mainly provide loans (some of which are concessional) and disburse based on prior policy actions rather than results. 

The fact that the EU provides grants has implications for the definition of the appropriate macroeconomic framework. While everyone agrees that budget support should be provided only in a stable macroeconomic environment (hence the EU’s collaboration with the IMF), the definition of “a stable macroeconomic environment” may be somewhat different if the country does not have to repay a loan. Hence, the macroeconomic framework for EU budget support may not necessarily be the same as the frameworks of the IMF or World Bank. 

Disbursement of EU budget support is based on progress in meeting certain benchmarks agreed upon at the beginning of the program. The disbursement is made of a fixed component (which requires general conditions to be met and whose amount do not vary) and a variable component, which serves as a performance top-up and is disbursed proportionally to a set of additional and specific performance indicators (provided the general conditions are met). The disbursement can therefore be full (fixed tranche and the totality of the variable tranche), partial (fixed tranche and part of variable tranche, if progress was only partial), or nil (one of the general conditions is not met). This system of disbursement contrasts with the approach taken by the MDBs, whose budget support is disbursed based on policies undertaken (prior actions) rather than on results. To the extent that there is a difference between ex ante policies and ex post performance, one wonders how countries can coordinate across their budget support donors. 

Performance-based conditionality (PBC) raises three issues of its own. First, because development is a risky business—whether a given policy reform will yield the expected outcome is not known—PBC puts more of the risk on the recipient. 

Second, other attempts at PBC, such as the World Bank’s Program-for-Results (PforR) financing, have found a tendency to “dilute” the performance criteria (in order not to risk failing to disburse) to the point where they resemble ex ante policy conditions.38
The reality is that both the donor and the recipient have an interest in seeing the operation disburse and therefore may, even subconsciously, nudge conditions in that direction. It is possible that this is happening with EU budget support as well. 

Third, the chapter notes approvingly, in addition to budget support, the EU provides technical assistance to countries to further progress on key areas such as PFM. Although it is desirable that EU technical assistance, budget support conditions, and policy dialogue all pull in the same direction, there may also be problems here. If the EU is providing technical assistance in an area that is also a performance criterion for tranche release, at least two things could happen. If the country fails to meet the performance criterion, it could blame it on the technical assistance, or the organization providing the assistance could try to influence the EU into certifying that the country had met the criterion, lest its own performance be judged as mediocre. Even if the technical assistance and budget support operation are kept independent, as both being provided by the same institution, it is difficult for the country to not perceive them as linked. 

The independent evaluations synthesized in the chapter all follow the same three-step framework. First, the effects of the budget support operation on policies and institutions are analyzed. Second, the outcomes and outputs in a country are related to policy and institutional changes. Third, the results of the first two steps are combined to provide a narrative of how the budget support operation, through its contribution to policy and institutional change, helped achieve outcomes and impacts.

There is an attempt to specify a counterfactual in some of the individual steps, but the overall narrative does not include one. One way of constructing a counterfactual is to compare one country with another with similar characteristics that did not receive a budget support operation from the EU. This cross-country analysis has been used in other evaluations of budget support operations.39
Another is to compare the same country’s performance in two periods, one in which there was a budget support operation and one in which there was not. Such an analysis would need to adjust for other factors, such as a terms-of-trade shock, that may have affected the economy during the budget support phase but were unrelated to the operation. If, for instance, a country experienced a favorable terms-of-trade shock during the period of the operation, the success of the operation may have been because of the shock rather than the operation.

The value of having a counterfactual goes beyond just having a better estimate of the project’s impact. It also helps disentangle the different components of budget support. As the synthesis notes, budget support operations have three components: the transfer of resources, technical assistance, and dialogue on policy reforms aimed at achieving the targets of the operation. What is the relative importance of each? 

The chapter notes that many evaluations discussed the relative contributions of the financial transfer (flow of funds), technical assistance, and policy dialogue, but this seems to be based on the relative magnitude of the financial transfer and the intensity of technical assistance and policy dialogue rather than on a model of the combined effect of the three on outcomes. Nevertheless, the chapter finds that the flow of funds played a greater role in the general budget support operations than in sector budget support. The reason could be that the sector budget support operations were concentrated in MICs, including some upper-middle-income countries; the financial transfer represented a tiny part of the government’s budget (0.6 percent as opposed to 15 percent for general budget support). In addition to their relative contributions, one needs to understand the interaction between financial and knowledge assistance. 

The bundling of finance and policy dialogue into budget support raises the broader question of why they should be bundled. If these policy reforms benefit the country, why do they not implement them anyway? Why is it necessary to incentivize them with funding? 

One answer is that the financial transfer acts as an encouragement for the government to undertake the reforms, suggesting that the reforms are not collectively owned by the whole of government. The financial transfer helps fix a fundamental political economy problem in the country (usually on a temporary basis). It is not clear that such solutions are sustainable. It is also not clear that external actors, such as the European Union or World Bank, can or should select reform champions in the country. 

The financial transfer associated with budget support can have two other effects that may not be conducive to better development outcomes. The synthesis in the chapter hints at some of them but does not develop their implications. The first is the fungibility of aid resources, which the chapter mentions. As the financial transfer goes directly to the government’s budget, it could in principle be used for any expenditure. Several evaluations speak favorably of the fact that pro-poor expenditures—on, for example, health, education, and social protection—rose during a budget support operation. But if the country was planning on increasing spending in these sectors anyway, then the EU’s funds were being used to finance other expenditures, about which little is known. This possibility is not just theoretical. There is evidence on the fungibility of aid in general. The second problem with the financial transfer links back to the political economy problem mentioned earlier. 

Most of the evaluations seem to equate increased public spending on health and education with improved health and education outcomes. Evidence for this link is weak at best because the delivery of basic services in health and education is poorly targeted and often ineffective (often because of absentee teachers or doctors). The chapter notes this discrepancy by pointing out that the gains were momentous but not always equitable… and gains in access have not always been accompanied by better quality of services. 

• 36
  
  European Commission. 2017. Budget Support Guidelines. Brussels https://ec.europa.eu/international-partnerships/system/files/budget-support-guidelines-2017_en.pdf.
• 37
  
  Independent evaluations of budget support using only steps 1 and 2 of the OECD-DAC methodology are more systematically undertaken at program level. See OECD. 2012. Evaluating Budget Support, Paris.
• 38
  
  Alan Gelb, Anna Diofasi, and Hannah Postel. 2016. “Program for Results: The First 35 Operations.” CGD Working Paper, 430. Washington, DC: Center for Global Development.
• 39
  
  William Easterly. 2003. “IMF and World Bank Structural Adjustment Programs and Poverty.” In Managing Currency Crises in Emerging Markets, ed. Michael P. Dooley and Jeffrey A. Frankel. Chicago: University of Chicago Press.

Monika Huppi and Gunnar Gotz. The Inter-American Development Bank (IDB) offers three broad lending categories among its sovereign-guaranteed loans: investment lending, policy-based lending (PBL), and lending for financial emergencies during macroeconomic crisis (called special development lending). PBL provides fast-disbursing financial assistance or country budget support conditional on the borrowing country fulfilling a set of agreed upon policy and institutional reforms. Investment lending disburses against specific predefined project expenditures. Special development lending also provides fast-disbursing support but is conditional on a country having been struck by a macroeconomic crisis, being supported by an active International Monetary Fund (IMF) program, and the special development lending being part of an international support package.

IDB introduced PBL in 1989, in response to the Latin American and Caribbean debt crisis. The instrument has evolved over time, leading to a decoupling from IMF support, the introduction of a programmatic variant and of a deferred draw-down option. The programmatic version consists of a series of single tranche operations set in a medium-term framework of reforms. The first operation identifies the policy conditions for the disbursement of that operation as well as indicative triggers for the subsequent loans in the series. Since the triggers can be revisited at the time of loan approval, programmatic PBL allows for conditions to be adjusted as circumstances change.

PBL has historically been subject to a lending limit which has changed over the years. In 2005–19, PBL accounted for about 28 percent of IDB’s sovereign-guaranteed approvals and amounted to $42.6 billion. About 80 percent of these resources were approved as programmatic operations supporting 124 programs, with the remaining 20 percent as individual single- or multi-tranche policy-based operations. All IDB borrowing member countries except one used PBL to varying degrees in the period. IDB’s PBL is rarely co-financed by other institutions and IDB tends to support reform processes in areas in which it has accumulated experience and knowledge. Emergency budget support has been provided through separate budget support instruments (currently called special development lending) that have also evolved. This form of support accounted for only 2 percent of sovereign-guaranteed approvals in 2005–19. During the first half of 2020, in response to the COVID-19 pandemic, policy-based and special development lending have spiked.

IDB’s Office of Evaluation and Oversight (OVE) has looked at PBL in several contexts, but a full-fledged evaluation of IDB’s PBL has yet to be undertaken. OVE routinely reviews the performance of PBL in its country program evaluations. It also reviews and validates IDB’s self-evaluations of completed programs and operations and assigns a performance rating to each completed PBL program or freestanding PBL operation. In addition, OVE undertook a thorough review of the design and use of PBL in 2015, which is summarized in the chapter.

The OVE review of the design and use of PBL found that although countries used PBL for various reasons, the predominant use was for budget support in time of need. While countries valued the policy dialogue and technical expertise that came with IDB PBL, the policy elements were usually secondary to the primacy of budget support. Although PBL provided important financial support, its ability to play a countercyclical role overall was limited because of the cap on PBL, the limited size of PBL operations compared to the economy in all but small countries, and because PBL could not be disbursed if borrowers did not have a positive macroeconomic assessment. 

The extent to which PBL is complementary to or a substitute for market financing has not been systematically assessed by OVE’s review. In some instances, OVE’s country program evaluations show that countries with ample access to financial markets used PBL as a liquidity management tool to complement market financing and fill short-term liquidity needs, particularly outside an economic crisis. 

In its 2015 review, OVE assessed the depth of the policy measurers supported by PBL and found that while they were generally relevant to the objectives of the reform programs they aimed to support, most policy conditions did not have enough depth to set in motion reforms that could by themselves bring lasting changes. Policy conditions were of higher depth in programs in the financial and energy sectors and during times of crisis.

Programmatic policy-based loans were found to allow for more sustained engagement. To the extent that policy measures became deeper as a programmatic series progressed, they were a useful tool to support reform programs while also helping borrowers meet financing needs. However, over one-third of programmatic PBL series active between 2005–19 were truncated before they reached their most consequential reform steps, raising questions about the ownership of the underlying reform programs that such lending sought to support. Truncation was more pronounced for PBL series in countries that resorted mostly to PBL to meet financing needs and did not seek technical assistance to accompany the underlying reform programs. That programs supported by technical cooperation grants had a lower truncation rate suggests the need for continuous engagement and technical cooperation to support borrowing countries in their reform efforts. 

The findings of OVE’s work undertaken so far invite questions. To what extent does PBL complement or substitute for funding from financial markets? Are IDB-supported policy measures complementary to, or do they overlap with those of other institutions providing budget support? What nonfinancial additionality does PBL provide? What results have PBL operations helped achieve and how sustainable will those results prove to be? OVE plans to undertake a full-fledged evaluation of PBL at IDB to try to answer some of these questions. 

IDB’s policy-based and project lending (on commitment bases) for the period 2005–20 is shown in Figure 5. Both PBO and non-PBO lending increased in 2020, though the ratio of policy-based lending to total lending fell slightly below IDB’s 40 percent cap. 

Note: PBL = policy-based lending 

Source: Office of Evaluation and Oversight, Inter-American Development Bank. 

Comments by Augusto de la Torre

The chapter is divided into two sections. The first neatly summarizes the evolution of PBL and its use by borrowing countries since its introduction at IDB in 1989. The second assesses PBL along several dimensions (based on well-focused findings), including the reasons countries demand PBL, complementarities between PBL and other IDB operations, and issues in design and implementation of PBL operations. The analysis part of the chapter is strong when it comes to findings; it falls short when interpreting the implications of such findings for IDB and borrowing countries. The observations of this commentator elaborate on these questions and issues and raise a few others.

The chapter provides significant evidence that borrowing countries’ predominant use of PBL was to fill their budget financing needs, with policy elements usually being secondary to the primacy of budget support. This suggests that MDBs recognize that financing needs are at the heart of the demand for PBL but point to policies and reforms as a key reason for offering PBL. Efforts to align these two motivations drive PBL preparation and design. These efforts succeed at times but not always. 

Not surprisingly, using a creative analytical approach, the OVE review found that most conditions in PBL were of low to medium depth. They tended to involve one-off and often reversible policy measures, to be process oriented, or to have good policy intentions that could not be implemented at the time. by themselves effect lasting change. The OVE review stressed that conditionality in PBL was generally relevant to the programs’ objectives but probably insufficient to attain the expected outcomes. The commentator concludes that the findings of the OVE report should lead MDBs to adjust expectations toward more realistic levels. PBL operations do not simply “buy” reforms, as is often believed. At best, they provide needed budgetary financing while recognizing—and helping fine-tune and strengthen the technical aspects of—reforms that would have been attempted by the country with or without the PBL. 

Still, the commentator finds that the rise in programmatic policy-based loans (PBPs) can be interpreted as a major step toward greater realism and frankness in PBL. PBPs have accounted for the lion’s share of IDB-originated PBL since 2005. He concludes that wisely, PBPs do not pretend to “buy reforms.” Instead, each single tranche loan in the program recognizes and gives credit to the country for policy actions and reforms that have happened before loan disbursement. Future reforms appear only as indicative guides for future operations under the multiyear program. 

As a result, PBPs avoid the time-inconsistency trap of traditional multi-tranche PBL operations, in which countries under duress may agree to conditions (reforms) with a low probability of being met (because the incentives to stick to the conditions diminish after the PBL is approved and the first disbursement comes in). 

Related to the question of ownership is the crucial question of whether PBL or PBPs can realistically be expected to generate policy additionality. Given the difficulties in identifying a counterfactual, it is difficult to attribute policy reforms to PBL or, equivalently, to reject the hypothesis that those policy reforms would have taken place even absent PBL. This calls for modesty by MDBs, whose role is not so much to tell countries what to do but to partner with countries in their quest for social and economic progress, including in design, implementation, and evaluation of reform. 

To mitigate the risks of low-depth reforms and truncation, a premium must be put on a continued and robust technical engagement and policy dialogue between the MDB and the country client. Doing so is particularly important considering several important findings in the OVE evaluation, including that “IDB tends to support policy reforms in sectors in which it had previously worked (usually through technical cooperation grants and investment loans) and thus has some country-level expertise that allows it to sustain a policy dialogue and provide relevant technical advice” and that “there was a significant positive relationship between technical cooperation support and the likelihood of a PBP series being completed.” 40
This point also argues for not evaluating PBL—or any particular financial product offered by an MDB—in isolation, but in context—that is, considering the entire portfolio of services the MDB offers to its country clients, including financial services, knowledge services, and convening services. 

The findings in the chapter raise questions about the complementarity and substitutability of PBL and market-based finance, an issue the chapter does not address but should. One hypothesis is that in countries with strong macro-financial policy frameworks, PBL is complementary to market-based finance—that is, these countries use PBL as part of their prudent management of the portfolio of public sector liabilities. In countries with weaker macro-financial policy frameworks, the hypothesis would imply that PBL is a substitute for market-based finance—that is, these countries resort to PBL because they lack access to market-based finance. The chapter should have explored this hypothesis and elaborated on the implications of what it found. 

• 40
  
  Chapter 4 in this volume.

James Melanson and Jason Cotton. The mandate of the Caribbean Development Bank (CDB) is to reduce poverty and transform lives by contributing to the sustainable, resilient, and inclusive development of its Borrowing Member Countries (BMCs). PBOs are financing instruments—loans, grants, and guarantees—that are used to incentivize the implementation of country-owned policy reforms and institutional changes aimed at advancing sustainable development goals. PBL provides fast-disbursing budget support to finance priority expenditures while helping to strengthen the effectiveness of public policy frameworks. It is disbursed following compliance with agreed policy actions and supports the process of good policy making and governance while reducing transactions costs and providing timely resources to national budgets. PBL is complementary to investment lending, as it helps establish an enabling environment for enhancing resilience, achieving economic growth, and reducing poverty. It is an important CDB intervention modality to enhance development effectiveness and responsiveness to the changing needs of members.

CDB PBOs can take several forms:

• Macroeconomic PBOs address external or internal economic imbalances. 
• Sector PBOs support reforms that help address critical sector issues and strengthen progress toward overall economic development. 
• Exogenous shock response PBOs provide resources in crisis situations to assist with the fallout from a shock; they can support reforms to enhance resilience. 
• Regional public goods PBOs help embed the policy and institutional frameworks necessary to advance regional cooperation and integration. 
• PBO guarantees guarantee a portion of debt service on a borrowing or bond issue by a BMC in support of country-owned policy reforms. 

PBL can be an important component of country financing strategies. At the country level, the size of the loan is related to development financing requirements, defined in terms of balance of payments, fiscal, sector, or other economic funding needs. 

CDB began participating in PBL operations in the late 1980s, with operations to support macroeconomic adjustment executed in collaboration with the IMF, World Bank, and IDB. These operations addressed complex development problems, made more acute by the increased frequency of natural disasters and the impacts of climate change, external shocks, relatively low growth, and high debt in Latin American and Caribbean states. 

In 2005, CDB formally introduced PBOs into its lending toolkit. Since then, it has sought to gradually strengthen the PBO instrument and the policy governing its use. Five external reviews or evaluations, as well as internal assessments by staff, have guided these efforts. Over time, they have revealed scope for improving the administration of PBOs, particularly in their design, supervision, and reporting, and the need to develop a more comprehensive and structured policy framework and guidelines. CDB has also experienced internal capacity building in results-based management, country fiscal diagnostics, and debt sustainability analysis. 

In 2013, a significant revision to the 2005 framework was undertaken to provide greater clarity on the principles, procedures, and guidelines for administering PBOs and to anchor them within CDB’s overall risk management and control framework. The new framework provided for an increase in the PBL limit from 20 percent of total loans and guarantees outstanding to 30 percent and subsequently, subject to further approval, to 33 percent. It also introduced risk-based and policy-lending allocation limits (from a credit risk, use, concentration, and capital adequacy standpoint) at the country level that align with, and preserve, the prudential soundness of CDB. Following a comprehensive review of operations and the establishment of a centralized Office of Risk Management in May 2013, the PBL limit rose to 33 percent in December 2015. 

In March 2020, CDB’s board approved an increase in the prudential limit to 38 percent, creating headroom for lending in response to the fallout from the COVID-19 pandemic. The increase is expected to be temporary, with a return to 33 percent by the end of 2023. 

The PBL framework encourages collaboration with development partners when they have PBOs that pursue similar expected outcomes to those of CDB. CDB seeks to harmonize appraisal, supervision, and monitoring using a common policy matrix. When CDB resources will not be able to close the financing gap, staff either appraise a PBO request as part of a joint operation with other development partners or consult closely with strategic partners to help mobilize resources. Staff must assess the adequacy of the macroeconomic framework for the conduct of a PBO. The views of the IMF and the existence of an IMF program or an Article IV assessment are important ingredients in the appraisal. Absent an IMF program or Article IV assessment in the preceding 18 months, an assessment letter of the macroeconomic framework is requested. For overseas territories of the United Kingdom, a letter of approval from the requisite authority in the United Kingdom is sought.

Between 2005 and September 2020, CDB undertook 27 operations, amounting to $944.7 million. In 2019, PBL represented 42 percent of CDB’s total loan approvals and 54 percent of its loan disbursements. PBL has financed emergency priority spending and helped preserve stability in BMCs, which are highly vulnerable to external shocks and natural disasters. This vulnerability derives from inherent structural characteristics, such as lack of economies of scale or economic diversification, export concentration; remoteness from global markets; dependence on external financing; and exposure to natural hazards and climate change.41

During 2006–2042
,PBL activity correlated closely with periods of economic and natural hazard shocks. In 2008, on the heels of the global financial crisis, CDB approved nine PBOs, totaling about $340 million (36 percent of the PBL portfolio). Given the increasing frequency and intensity of hurricanes in the region, PBL demand has remained strong since 2015, peaking in the 2017 and 2019 hurricane seasons. Since the introduction of PBL, practice has evolved, reflecting CDB’s learning and experience in managing the instrument and its observation of experience at other MDBs. 

The findings and conclusions of a 2017 evaluation conducted by CDB’s Office of Independent Evaluation, based on evidence from document review, case studies, and a wide range of interviews, suggests that several key factors increased the likelihood of PBL operations achieving their desired results. These included (a) clear objectives and results logic, with indicators and targets that can be measured and verified; (b) a selective focus on a manageable number of expected reform outcomes; (c) agreement on a small number of prior actions clearly linked to those outcomes; (d) good understanding of external risks and elaboration of mitigation strategies; (e) an engagement process with BMCs that engenders ownership and commitment by borrowers; (f) a menu of PBL options that offers the right instrument calibrated to borrowers’ reform readiness; (g) an understanding of national capacity constraints and, where needed, provision of affordable technical assistance to address them; (h) designation of an identified champion in the national public service with responsibility and authority for achieving reform results; and (i) consistent monitoring to identify when conditions are met, and the degree of progress toward reform outcomes. 

Although the evaluation found that CDB’s PBL was increasingly taking these factors into account, it offered several recommendations to encourage further progress: 

• CDB should review its practice of management for development results (MfDR) in the PBL program, making sure its design processes respect good MfDR practice, with clearly stated expected outcomes and indicators that are specific, measurable, achievable, relevant, and time-bound (SMART). 
• A corollary of more carefully stated results frameworks would be more tailored risk mitigation strategies. To date, such strategies have tended to be generic across PBOs. They should be more closely matched to the specific circumstances of the national context and reform program. 
• CDB’s PBLs should focus on a small number of key outcomes, with prior actions that are causally linked to them. The choice of outcomes should take account of the limited size of CDB’s PBL loans, BMC priorities and CDB’s own country strategy, and an agreed longer-term reform program. 
• This focus should be maintained over time, with prior actions in successive PBOs building incrementally on one another. 
• National ownership and leadership are indispensable to the success of development reform programs. CDB should facilitate these to the greatest extent possible through collegial engagement with BMCs in PBL design and implementation. Doing so will require consultations with a sufficient breadth of national stakeholders, at both leadership and implementation levels, to gain commitment and follow through on reform objectives and prior actions. A good practice to be encouraged is the designation of a “champion” from the BMC’s public sector for implementation of targeted reforms. Small economies experience serious capacity constraints to implement reform programs. These constraints need to be anticipated and responded to as part of an effective PBL program. Relative to other MDBs, CDB has a more intimate understanding of the contexts and constraints of its BMCs, but it has carried out only limited needs analysis and there has been limited uptake of CDB technical assistance in connection with its PBL loans. CDB should investigate the reasons for this and make sure potential technical assistance requirements are well analyzed at the design stage and that flexibility is shown when such assistance is implemented. 

CDB’s policy-based and project lending (on commitment bases) for the period 2006–20 is shown in Figure 6. While non-PBO lending rose sharply in 2020, PBO lending fell and the ratio of policy-based lending to total lending fell well below CDB’s temporary cap of 38 percent. For CDB the lending pattern has become increasingly reflective of the financing needs arising from natural disasters than global events.

Note: PBL = policy-based lending 

Source: Office of Independent Evaluation Office, the Caribbean Development Bank. 

Comments by Ali Khadr

The chapter provides an informative overview of the findings of five assessments, including two Office of Independent Evaluation (OIE) evaluations of CDB PBOs. Among many other findings, it conveys how the institution’s practice of PBL, as well as the associated framework and guidance, has evolved over the roughly 15 years since it was initiated. Even with the favorable evolution of CDB’s PBL practice, there is room for further improvement.

Like other MDBs, CDB has moved over time toward the body of good practice identified in an ever-growing PBL literature. Among the key elements of this emerging body of good practice are (a) more frequent use of the more adaptable programmatic PBL instrument variant rather than the more rigid multi-tranche variant; (b) a focus on fewer, “deeper” 43
prior actions in PBOs; (c) use of results frameworks with a tighter logic linking a small number of  prior actions to a manageable number of key outcomes sought, as well as associated use of specific, measurable, achievable, relevant, and time-bound (SMART) results indicators; (d) greater country ownership of proposed measures and outcomes sought, bolstered by broad prior consultation; (e) identification of capacity constraints to reform implementation and provision of parallel technical assistance as needed; and (f) identification and mitigation of risks adequately tailored to the specific operation. These elements raise questions about (a) the use of PBOs in small states and the shocks to which they are often subject; (b) the use of PBOs to strengthen fiscal management; (c) the analytical underpinnings of PBOs; (d) the quality of the results framework, including the depth of prior actions; and (e) establishment of attribution or contribution.

CDB is unique among MDBs in that its clients consist overwhelmingly of small states (formally defined as countries with fewer than 1.5 million inhabitants). Of the CDB’s 19 BMCs, 17 are small states (or dependencies), most of them islands or archipelagos.

As extensively documented in a burgeoning literature, small states as a group, especially small island states share several intrinsic characteristics and challenges. They include higher fixed costs (for instance, larger public expenditure, including public sector wage bills, as a share of GDP). The locations of these states also commonly entail high trade costs as well as extreme vulnerability to natural disasters and the deleterious effects of climate change. In addition, their exports tend to be very concentrated (usually in tourism and a few commodities), which makes them particularly vulnerable to trade shocks and contagion from trading partner downturns, including the downturn induced by the COVID-19 pandemic. 

These intrinsic characteristics and challenges—particularly the exposure to repeated economic and natural shocks that are large relative to GDP—have resulted in a greater volatility of growth in small states compared with larger ones. Together with the inherent stresses on public finances and limited borrowing opportunities, these repeated shocks have often entailed fiscal distress and rapid debt accumulation, making effective fiscal and debt management imperative. 

Given the shock-intensive country client context, PBL from MDBs has a clear role to play in CDB BMCs. It is especially encouraging to see that CDB has raised the prudential limit to 38 percent to create lending headroom to counter COVID-19-related fallout and offering exogenous shock response PBOs as a distinct instrument variant. Future evaluations of CDB PBOs can yield valuable lessons on how effectively such operations have supported small states, especially in helping mitigate the shocks to which they are subject and building resilience. 

In many small states drawing on PBOs, fiscal management is likely to be—or at least should be—a central component. One area of focus in future PBL evaluation work by CDB could therefore be the quality of PBOs’ macro-fiscal frameworks, given recent findings in the evaluation literature to that it is positively associated with loan outcomes.

In an earlier study, the Independent Evaluation Group (IEG) of the World Bank Group examined the quality of macro-fiscal frameworks in 390 World Bank PBOs completed during fiscal years 2005–13. It found that certain aspects of the quality of PBO macro-fiscal framework design were positively correlated with loan outcome ratings. Two aspects of the quality of the PBO framework showed a statistically significant association with loan outcome ratings: the credibility of the PBO framework given the country’s fiscal record and adequate coverage of quasi-fiscal risks (risks the government might need to devote public spending to off-budget items, such as an underfunded public pension system or state-owned enterprises in distress). 

There is emerging, although not conclusive, evidence that strong analytical foundations can be an important determinant of PBO effectiveness. IEG found generally solid links between World Bank PBL design and integrative analytical work on public expenditure, as well as continuity in policy dialogue from the latter to the former. However, it was difficult to establish a clear association between such links and PBO outcome ratings, although PBOs informed by analytical work on public expenditures showed slightly better outcome ratings over 2009–12. 

The depth of prior actions in CDB PBOs increased over time. Depth—the extent to which the reform measure on its own can bring about lasting change in the institutional and policy environment—is a key ingredient in the quality of the results framework. Noncritical, shallow, and process-related measures should be avoided. 

A common complaint in PBL evaluations concerns the difficulty of attributing medium-term country outcomes to the use of PBOs, including the prior actions they support and the financing they provide. The difficulty is compounded when several development partners deliver PBL simultaneously. It is therefore not surprising to read in the chapter that “it was not feasible for the evaluation to gather a sufficient amount of directly attributable evidence to support statements of causal linkage between CDB’s PBL support and higher-level medium-term outcomes.” 44

Given the concentration of CDB clients in small states, CDB PBL evaluation work can yield valuable lessons about how CDB PBOs support small states in dealing with shocks, particularly whether PBOs adequately cover the multiple drivers of fiscal and debt sustainability and foster systemic, rather than incremental, changes in disaster and climate resilience by targeting incentives and behaviors. Other issues on which future CDB evaluation work could usefully focus include the quality of CDB PBOs’ macro-fiscal frameworks and analytical underpinnings, the quality of PBO results frameworks (including depth of prior actions supported), and establishment of the plausible likelihood of PBOs contributing to country outcomes.

• 41
  
  Caribbean countries are seven times more likely than other countries to be affected by a natural hazard and to suffer damage that is six times greater. See Sebastian Acevedo Mejia. 2016. “Gone with the Wind: Estimating Hurricane Climate Costs in the Caribbean.” IMF Working Paper WP/16/199. Washington, DC: IMF.
• 42
  
  Data for 2020 cover the months January–September
• 43
  
  The concept of depth, used in several evaluations of PBL, can be traced back to the measure of “structural depth” developed and applied in Independent Evaluation Office. 2007. Structural Conditionality in IMF-Supported Programs, Washington, DC: International Monetary Fund
• 44
  
  Chapter 5 in this volume.

Zeljko Bogetić and Jeff Chelsky. International financial institutions use different names for policy-based financing (PBF) and PBOs/PBLs. The World Bank generally refers to this type of support as development policy financing (DPF), delivered through different types of the DPF instrument, depending on the nature of the financing provided. A development policy loan or development policy operation is the most common type. If the financing is a grant (typically provided to low-income country recipients), the PBF will usually be a development policy grant. 

Development policy financing can also be a guarantee, for which a policy-based guarantee (PBG) is used,45
or as financing contingent on the activation of an indicative trigger related to, for example, natural disasters or health crises. DPF can be provided through a single, stand-alone operation or through a programmatic series of operations, linked by indicative triggers. DPF is provided to sovereign national governments of World Bank member states and, sometimes, to subnational governments.

The World Bank’s policy on DPF states that “DPF is aimed at helping a Member Country address actual or anticipated development financing requirements that have domestic or external origins. The [World] Bank may provide a Bank Loan to a Member Country or to one of its Political Subdivisions; and it may provide a Bank Guarantee of debt incurred by a Member Country or by one of its Political Subdivisions.”46
DPF aims to help the borrower achieve sustainable poverty reduction through a program of policy and institutional actions. These may include, for example, strengthening PFM, improving the investment climate, addressing bottlenecks to improve service delivery, or diversifying the economy. DPF provides general budget support, meaning that the funds are disbursed into the general budget of the client government and are not earmarked for specific purposes. 

Approval by the Board of Executive Directors and implementation of a set of policy and institutional actions (prior actions) are required before financing can be disbursed. Prior actions are expected to advance, catalyze, or signal meaningful movement along the results chain toward the achievement of clearly specified objectives. The objectives of each development policy operation are expected to be consistent with the recipient’s national development goals and the World Bank–supported strategy for the country. Well-defined results indicators, with clear baselines and time-bound targets, monitor progress toward objectives along a credible results chain (or theory of change) that links prior actions, complementary other activities, and reforms to targeted outcomes. The policy framework underpinning a DPO is developed through a dialogue between the World Bank and the recipient government. 

Over the past decade and a half, DPF operations have accounted for one-quarter to one-third of World Bank financing commitments. The use of DPF increased during the global financial crisis in 2007–09, when it was used to provide countercyclical financing to many developing country recipients. It has also increased during regional crises and most recently, in response to the COVID-19 pandemic. 

This chapter reviews the evolution in DPF at the World Bank and provides evidence on the performance of the instrument between 2005 and 2019, as reflected in the recent literature and IEG evaluations. It also describes recent changes to IEG’s evaluation framework for PBF operations. The principal findings are: 

• Policy-based lending has been an important financing instrument of the World Bank, accounting for about one-quarter of its total financing during 2005–19 but increasing to about 40 percent in times of crises. It often plays an important countercyclical role in developing countries. Budget support operations have supported short-term and longer-term policy and institutional reforms geared toward poverty reduction and shared prosperity.
• The World Bank uses DPF as a key instrument in supporting country clients in crisis. During crises such as the global financial crisis, the focus on fiscal management, effectiveness of public expenditures, and targeted social programs supported countercyclical policies. 
• Several types of budget support are in use, including standard, stand-alone operations; programmatic series of operations; and PBGs and deferred draw-down options. This variety makes DPF a versatile financing instrument that can be deployed in a wide variety of contexts to support short-term goals (such as macroeconomic stabilization, natural disaster emergency, post-conflict budget financing support, and arrears clearance) and long-term goals (such as poverty reduction and shared prosperity). For this reason, governments have often requested this instrument, especially in times of crises, when national budgets are under stress and quick-disbursing financing, combined with corrective policy actions, is an economic and social imperative. 
• The COVID-19 crisis and its unprecedented global health, economic, and social impact prompted the World Bank to rapidly scale up its financing to developing country recipients to cushion impact. At the start of the pandemic, it committed to deliver $160 billion in overall financial support by the end of June 2021. In the event, $157 billion was delivered, including $28 billion in DPF. 
• IEG evaluations have generally assessed World Bank DPOs positively, with the overall outcomes of about four-fifths of operations rated moderately satisfactory or higher. However, within this, the share of operations rated satisfactory has declined while the share rated moderately satisfactory has risen. 

The framework for evaluating DPF at the World Bank was recently revised to produce more operationally relevant findings and lessons better tailored to this instrument. IEG has similarly revised its validation framework for evaluations of DPF operations to give greater attention to World Bank performance more generally, including the relevance and quality of prior actions, appropriateness of results indicators, and the adequacy of the ex ante assessment of risks to the achievement of objectives. IEG began using this new framework in late 2020.

World Bank policy-based and project lending (on commitment bases) for 2005-20 is shown in Figure 7. While both PBO (DPF) and non-PBO lending rose sharply in 2020, the ratio of policy-based lending to total lending at about 30 percent fell well below the World Bank’s ratios during earlier crisis (e.g., 2008–10) when ratio rose to 35 percent. 

Note: PBL = policy-based lending  

Source: IEG, the World Bank. 

Comments by Cheryl Gray

This chapter provides a useful, concise, and well-written summary of the evolution of the World Bank’s approach to PBF and methods to evaluate it. It shows the careful thinking undertaken by the World Bank as it has struggled to deliver effective support to countries, often in complex and difficult settings. 

As the chapter illustrates, PBF has long been a major instrument of international development support, valued in the hundreds of billions of dollars annually across development agencies. Its breadth and complexity have made it exceedingly difficult to study, and evidence of its results remains elusive. This chapter sheds light on what is known, although for perhaps unavoidable reasons the picture is still incomplete. 

The first point worth stressing is the practicality and value-added of the World Bank’s evaluation architecture. Over more than three decades the World Bank has designed, implemented, and continuously improved a cohesive structure to document results from all its operations—both investment and PBOs—in a practical and cost-effective manner. The process begins with a self-evaluation, using standard criteria applicable in all similar operations, by the task team, whose members know the operation best. IEG then reviews and assesses that self-evaluation.

The fact that 100 percent of self-evaluations are assessed by IEG creates an incentive for task teams to report accurately while also producing a complete database of operation-level reviews across the institution. IEG sometimes follows up with more in-depth evaluations or broader sector or thematic evaluations, adding further context and evidence on results. The entire evaluation architecture is oriented toward documenting activities and outcomes, creating opportunities for learning through self-evaluation and analysis. 

These routine World Bank evaluations are not in-depth impact evaluations with rigorous causal inference. They do not compare performance against counterfactuals to identify and measure cause and effect. Occasionally, it is possible to apply impact evaluation techniques to specific interventions in specific settings, but it is not possible to do so across the board, given the breadth and complexity of most World Bank operations, particularly PBF. 

The chapter describes the criteria for self-evaluation and validation of DPF (the World Bank’s term for PBF). These criteria have changed over time to reflect changes in the design of PBF. When policy-based lending began ay the World Bank, in the 1990s, loans were disbursed in several tranches triggered by successful completion of policy reforms and institutional milestones. Today, DPFs provide all the financing upfront, upon completion of a small number of key prior actions. This policy differs from the policy lending of the EU, which includes a performance element and disburses in part based on the achievement of results. 

The World Bank’s approach puts a heavy weight on a small number of upfront policy and institutional changes it considers key to the country’s success. Having a small number of prior actions simplifies the lending process and focuses the World Bank’s oversight, but it runs the risk that the assumptions regarding the impact of reforms may be wrong. 

Recently, the World Bank moved from rating the relevance of a DPF operation’s objectives (the standard approach in evaluations of investment operations) to rating the relevance of the prior actions, the only conditions for the operation directly within the World Bank’s control. The World Bank is also putting greater weight on evaluating the relevance and quality of the operation’s results indicators, World Bank performance, and the treatment of risk. These judgments are largely qualitative. To ensure these ratings are robust, it would help to track whether guidelines, dialogue, and practice are converging in reasonably common standards across operations and over time. 

An important aspect of DPF operations missed by the World Bank’s evaluation approach is the impact of the resource transfers themselves—that is, the impact of transferring hundreds of millions of dollars to recipient countries through DPF. Some have argued that the increased availability of funds for governments to spend may be the biggest impact of PBL, greater than the support to policy and institutional reforms provided by the loans. 

The chapter reviews the data on the results of DPF operations over time, highlighting several academic studies and thematic evaluations that have tried to draw conclusions from these data. In addition to the inherent limitations on results measurement noted above, a few points stand out. First, there is a high prevalence of “moderately satisfactory” ratings for outcomes and World Bank performance. 

Second, thematic evaluations emphasize the prevalence and salience of DPF prior actions related to PFM. Managing public finances is an important and powerful responsibility of government that strongly influences the distribution of resources and effectiveness of public programs. It is an area that the World Bank has focused on and influences relatively effectively through its operations. The World Bank can bring expertise and resources to the technical aspect of PFM, such as budgeting processes, computer systems, and auditing. Other areas of governance reform, such as election systems, public employment, or direct anticorruption efforts, may be as (or even more) important for development outcomes. They have been more difficult for the World Bank to address in the political environments in which it works. These sensitivities put limitations on what kinds of prior actions are feasible in DPF operations, limiting their potential development impact.

Third, one of the academic studies cited concludes that the level of macroeconomic stability is positively associated with the success of DPF operations.47
As noted in the chapter, it is impossible to untangle causation (whether the World Bank’s operation influenced the country’s policies or good policies made it possible for the World Bank to lend). The fact that government ownership is also key to achieving outcomes and that “the World Bank’s policy lending is significantly and positively correlated with the quality of social policies and institutions”48
 reinforce the overwhelming importance of committed country counterparts. 

Attributing causal impact to DPF operations themselves is not likely to be supported by the evaluation techniques available. But the evidence strongly supports the finding that enlightened leadership, pro-development policies, and effective World Bank support go hand in hand.

• 45
  
  In a PBG, instead of providing financing directly to the client government, the World Bank provides a guarantee for a portion of the principal or interest on a loan, or both, which is provided by commercial creditors, or the client government issues an international bond.
• 46
  
  Chapter 6 in this volume.
• 47
  
  Lodewijk Smets and Stephen Knack. 2014. “World Bank Lending and the Quality of Economic Policy.” World Bank Policy Research Paper 6924, Washington, DC.
• 48
  
  Željko Bogetić and Lodewijk Smets. 2017. “Association of World Bank Policy Lending with Social Development Policies and Institutions.” World Bank Policy Research Paper 8263, Washington, DC.