In 2015, the IDB Office of Evaluation and Oversight undertook an analysis of the design and use of policy-based lending at IDB. Although it found countries used PBL for various reasons, the predominant use was for budget support in time of need. While countries valued the policy dialogue and technical expertise that came with IDB PBL, the policy elements were usually secondary to the primacy of budget support. Although PBL provided important financial support, its ability to play a countercyclical role overall was limited because of the cap on PBL and because PBL could not be disbursed if borrowers did not have a positive macroeconomic assessment. The review assessed the depth of the policy conditions and found that most were of low- or medium-depth, meaning they helped set in motion policy reforms but could not by themselves effect lasting changes. Conditions tended to gain in depth in the second and third loan of a programmatic series, as the underlying reform program progressed. However, over one third of programmatic PBL programs active in 2005–2019 were interrupted, affecting the depth of supported programs. Policy conditions were of higher depth in programs in the financial and energy sectors and during times of crisis. Neither the number of conditions in a program nor the loan size were correlated with program depth

OVE has looked at PBL in several contexts, but a full-fledged evaluation of IDB’s policy-based lending has not been undertaken to date. OVE routinely reviews the performance of PBL in the context of its country program evaluations. OVE also reviews and validates IDB’s self-evaluations of completed programs and operations and assigns a performance rating to each completed PBL program or freestanding PBL operation. In addition, OVE undertook a thorough review of the design and use of PBL in 2015.7
This section will briefly discuss the performance ratings of PBL based on OVE’s validations of self-evaluations and then present key findings of OVE’s 2015 review of the design and use of PBL.

IDB’s current self-evaluation system was adopted relatively recently. Project teams are required to prepare a project completion report for a programmatic PBL series when the program has been completed or interrupted or, in the case of freestanding PBL operations, at the time of completion of the operation. These self-evaluations are then validated by OVE which assigns an outcome rating to each program or freestanding PBL operation. In the case of programmatic PBL series, the program as a whole is evaluated against a results matrix for the entire program rather than for each loan. In the case of a freestanding PBL, the operation is assessed against the results matrix for that particular operation. The assessment covers three dimensions: relevance, effectiveness, and sustainability. An overall performance rating is assigned based on a weighted average of the ratings achieved on each of these three dimensions.8
As this system has evolved over time, comparable performance ratings are available for only 4 years; for operations or programs that were validated by OVE in 2017–2020. A total of 26 programs, comprising 48 loans have been rated thus far. Four of these consisted of hybrid operations with a PBL and an investment lending component. Of the 26 validated programs, 15 (58%) achieved an overall outcome rating of partly successful or higher. Excluding the hybrid operations, 14 of the 22 programs (64%) achieved a rating of partly successful or higher (compared with 57% of investment loans). 
 

• 7
  
  IDB Office of Evaluation and Oversight, Design and Use of Policy-Based Loans at the IDB. Document RE-485-6. Washington, DC: IDB https://publications.iadb.org/en/ove-annual-report-2015-technical-note-design-and-use-policy-based-loans-idb
• 8
  
  Relevance, effectiveness and sustainability are each rated on a four-point scale. The overall performance rating is a weighted average of the scores on each of these three dimensions, with relevance and sustainability being given a weight of 20% each and effectiveness 60%. The overall performance rating uses a six-point scale ranging from highly successful to highly unsuccessful.

Among the key questions that OVE’s 2015 review of the design and use of PBL explored were how the design and implementation of PBL operations changed over time and why countries demand PBL. The following sections will briefly summarize the review’s findings in this respect. OVE’s 2015 analysis did not seek to evaluate the achievements of the outcomes to which the PBL sought to contribute. It covered the period 1989–2014, with an emphasis on the last decade of the period. 

To explore what drives countries’ demand for PBL, OVE looked at four dimensions: (i) the frequency and intensity of PBL use; (ii) the correlation between PBL borrowing and growth rates, fiscal deficits, and gross financing requirements; (iii) countries’ reliance on parallel technical cooperation grants; and (iv) countries’ tendencies to fully complete or interrupt (“truncate”) programmatic PBL series. OVE analyzed these dimensions by reviewing all relevant lending documents and country economic data, carrying out an econometric analysis of lending, and interviewing IDB staff and officials from borrowing countries. Through this analysis, OVE identified four main categories of PBL users (Box 4.1).

• Mostly as budget support. This category included countries that resorted to PBL mainly in a countercyclical fashion (for example, to deal with a crisis that had suddenly halted capital inflows) or as a swift source of liquidity to handle short-term needs, such as debt servicing. In general, these countries exhibited a negative correlation between policy-based lending and GDP growth rates, and a positive correlation between policy-based lending and fiscal deficits or gross financing requirements. Their programmatic PBL series exhibited relatively high rates of interruption. They did not rely much on parallel IDB technical cooperation grants to accompany the reform programs. Since their demand for PBL depended on economic needs, these countries were not among the most regular users of PBL. In the decade leading up to 2015, examples in this category included Dominican Republic, Honduras, and Jamaica.
• Mostly as seal of approval for reforms and to benefit from IDB’s technical advice. This group comprised countries that tended to resort regularly to PBL and did so mostly to help legitimize their policy reform process by getting a “seal of approval” from IDB and to benefit from technical discussions between country officials and IDB specialists. Their PBL operations tended to be relatively small, and the demand for PBL tended not to be correlated with growth, fiscal deficits, or gross financing requirements. Moreover, programmatic PBL series in these countries had low truncation rates (a reflection of reform program implementation over a more extended time period and arguably higher ownership of the underlying reform program). They relied significantly on parallel technical cooperation grants provided by IDB to support the PBL programs. Peru and Bolivia were examples of this country grouping.
• Mixed. These countries sometimes relied on PBL to cover financing needs and sometimes used them to benefit from IDB’s validation and technical inputs. Examples in the decade leading up to 2015 included Brazil, Colombia, Mexico, El Salvador, Nicaragua, and Panama.
• Preventive. Uruguay has used policy-based lending as part of the government’s precautionary borrowing strategy with multilateral development banks (MDBs). Since 2008, Uruguay has frequently postponed disbursements of approved PBL and used the proceeds only when it faced large financing needs. This practice was institutionalized with IDB’s introduction of the DDO modality in 2012. Recently, Uruguay resorted to drawing down resources from several DDO PBL operations to rapidly cover its financing needs to counter the COVID-19 pandemic and its associated social and economic effects. 

Overall, OVE concluded that, despite the range of reasons for using PBL across countries, the predominant use was for budget support in time of need. Its review found that, while countries valued the policy dialogue and technical expertise that came with IDB PBL, the policy elements were usually secondary to the primacy of budget support. Econometric analysis found that policy-based lending was negatively correlated with a country’s growth rate, and positively correlated with the size of fiscal deficits and gross financing needs (Figure 4.5, Figure 4.6). Drawing on the literature on early warning signals for economic and financial crises, OVE estimated fixed-effects panel regression models using PBL disbursements as a percentage of GDP as the dependent variable. The results confirmed that countries’ financing objectives were a key motivation for the use of PBL, both to handle short-term financing needs and to face contingent shocks. The use of PBL for budget support purposes was found to be particularly pertinent for small economies, which tended to be more vulnerable to external economic shocks and for which IDB financing could be decisive in helping them to weather a storm. While larger countries also made use of PBL for fiscal and liquidity management purposes, the instrument’s ability to affect macroeconomic conditions in these countries was limited by the small size of the loans in relation to their overall economies. While PBL played a major financing role, its countercyclical role overall was limited in most countries by the overall cap on PBL and the fact that PBL could not be disbursed if borrowers did not have a positive macroeconomic assessment.

An analysis of the extent to which PBL funding is complementary or a substitute for market financing was beyond the scope of OVE’s 2015 review and therefore remains an open question. OVE country program evaluations suggest that some countries with ample access to international financial markets tend to use PBL as a debt and liquidity management tool to complement market financing, particularly when borrowing during good economic times. A Colombia country program evaluation, for example, found that, as the country gained increased access to financial markets, PBL remained an attractive instrument because its large and predictable disbursements facilitated the Ministry of Finance’s financial planning, given that Colombia tended to issue bonds in January and September. Similarly, a recent evaluation of Mexico’s country program found that the Ministry of Finance sought regular and predictable disbursements for debt management purposes.9

GDP = gross domestic product, LAC = Latin America and the Caribbean, PBL = policy-based lending. 

Source: IDB Office of Evaluation and Oversight (OVE). Design and Use of Policy-Based Loans at the IDB. Document RE-485-6. Washington, DC: IDB.https://publications.iadb.org/en/ove-annual-report-2015-technical-note-…

• 9
  
  IDB Office of Evaluation and Oversight (OVE). 2015. Country Program Evaluation: Colombia 2011-2014. Washington, DC: IDB. https://publications.iadb.org/en/country-program-evaluation-colombia-2011-2014; IDB Office of Evaluation and Oversight (OVE). 2019. Country Program Evaluation: Colombia 2015-2018. Washington, DC. IDB https://publications.iadb.org/en/country-program-evaluation-colombia-2015-2018; IDB Office of Evaluation and Oversight (OVE). 2019. Country Program Evaluation: Mexico 2013-2018. Washington, DC. IDB https://publications.iadb.org/en/country-program-evaluation-mexico-2013-2018

OVE’s analysis of the design of PBL focused on the evolution and nature of policy conditions, the vertical logic of the programs, and the extent to which PBL was accompanied by other IDB support. When looking at the evolution of policy conditions, OVE considered all PBL operations approved between 1990 and 2014, in order to gain a longer-term perspective. For a more in-depth analysis of the nature of policy conditions, and the complementarity between PBL and parallel IDB support, OVE focused on PBL operations approved in the decade leading up to 2015. In order to conduct this analysis, OVE drew a stratified random sample of 40 policy-based programs from the universe of PBL operations approved between 2005 and 2014 in four thematic areas: public sector and economic management, social sectors, financial sector, and energy. The sample encompassed 70 multitranche and programmatic loans in 18 countries and covered 34% of all programs approved over the time period. Analysis of policy matrices was supplemented by information from pertinent OVE country program evaluations and interviews with IDB staff and country stakeholders. To review the complementarity between PBL and parallel IDB support through technical cooperation grants or investment loans, OVE focused on all 82 programmatic PBL series (equivalent to 144 PBL operations) approved between 2005 and 2014.

The number of conditions at the program level increased over time. In the early 1990s, the average number of conditions per loan was roughly 50, and that figure fell by half between the mid-1990s and 2004. In line with the Organisation for Economic Co-operation and Development (OECD) Paris Declaration in 2005, the introduction of the programmatic modality further streamlined conditionality at the loan level (Figure 4.7). However, at the program level, the average number of conditions increased after 2005, offsetting the streamlining gained at the loan level10
. There were no significant differences in the number of conditions across thematic areas or regions. 

PBL = policy-based lending.

Note: The graph shows the average number of conditions per multitranche policy-based loan for 1990–2004, and per programmatic PBL series from 2005 onwards, at both the loan and the program level (as originally expected). Figures at the program level are based on the programmatic PBL series’ initial year.

Source: IDB Office of Evaluation and Oversight (OVE).Design and Use of Policy-Based Loans at the IDB. Document RE-485-6. Washington, DC: IDB. https://publications.iadb.org/en/ove-annual-report-2015-technical-note-design-and-use-policy-based-loans-id

To determine to what extent PBL-supported policy conditions had sufficient depth to trigger long-lasting policy and institutional changes, OVE’s analysis reviewed the content of each policy condition and assigned one of three categories to it:

• Low-depth. Conditions that would not, by themselves, bring about any meaningful changes. Low-depth conditions are usually process-oriented and often involve the preparation of action plans or strategies and the announcement of intentions.
• Medium-depth. Conditions that can have an immediate but not a lasting impact. These include conditions calling for one-off measures that can be expected to have an immediate and possibly significant effect, but that would need to be followed by other measures for this to be lasting. Submission of draft legislation to Congress, reaching a target or benchmarks, and organizational changes are examples of medium-depth conditions.
• High-depth. Conditions that could, by themselves, trigger long-lasting changes in the institutional or policy environment. Conditions in this category include legislative changes, government decrees, or lower-level actions that complete a critical reform process. High-depth conditions also include measures that require that certain fiduciary measures be taken regularly or permanently, even when legislation is not needed.

This analysis was supplemented by an assessment of the programs’ overall vertical logic and coherence. OVE evaluated the sequencing of the conditions across PBL tranches or across individual loans in a programmatic PBL series by looking at the extent to which the policy conditions included in each tranche of a multitranche PBL, or in each loan of a programmatic PBL series, followed a logical sequence over time by supporting different stages of the reform process cycle (i.e., formulation or design, adoption or approval, implementation, monitoring and evaluation). OVE also assessed the program’s vertical logic: the coherence between conditions and the reform program objectives and expected results. While OVE’s analysis of the depth of policy conditions and their sequencing allowed it to gauge the progress of the reform program supported by PBL, the methodology did not measure IDB’s technical additionality to the reform program or the extent to which the impetus for reforms could be traced to IDB actions. 

Most conditions involved policy or regulatory measures, while a small proportion focused on organizational changes at public agencies. OVE found that almost 80% of conditions in the sample supported policy reforms, ranging from the design of a new payment scheme for a social program to the approval of fiscal responsibility legislation. The rest promoted changes in the structure, responsibility chain, and/or institutional capacity of public agencies, such as the creation of a public health unit in the Ministry of Health or the formulation of a code to define the structure and processes for a public agency.

Most conditions were low- or medium-depth; they helped set in motion policy reforms but could not by themselves effect lasting changes. Almost a third of conditions in both the multitranche PBL and the programmatic series reviewed were low-depth, calling for basic one-off measures or simply expressing intentions. For example, a condition would commit a line agency to an independent operational audit of a feeding subsidy or call on an agency to prepare terms of reference for the design of a methodology to analyze the outcomes of a national investment plan. It is questionable whether such measures are in line with IDB’s guidelines, which stipulate that PBL conditions should be essential for the achievement of expected results. Only 15% of the conditions in the sample were high-depth, for example the elimination of government budget support for state-controlled enterprises or the adoption of revised targeting mechanisms for a school feeding program. No major differences in the depth level of conditions were found between multitranche PBL and programmatic PBL series.

Sequencing of PBL conditions followed the stages of reform cycles and tended to gain in depth as the reform process advanced, but the monitoring and evaluation (M&E) stage was seldom included. OVE classified the conditions in each operation according to the milestones in a policy reform cycle (formulation, adoption, implementation, M&E) that they supported. Not surprisingly, conditions in the first tranche or loan tended to focus on earlier stages of a policy reform process, while a larger proportion of conditions in subsequent tranches or loans tended to focus on implementation (Figure 4.8). Conditions gained in depth as the program advanced. For example, 43% of conditions in the first loans of programmatic PBL series were low-depth, while this proportion decreased to 30% in the second loan and to 16% in the third (Figure 4.9). According to OVE’s analysis, the reviewed conditions were generally relevant to the programs’ objectives, although they were probably insufficient to attain the expected outcomes. Less than 6% of the conditions reviewed included provisions linked to the last stage of a reform process—M&E. Programs in the social sectors were more likely to include M&E conditions, especially when compared with those in the financial sector (0.8% of conditions).

Source: IDB Office of Evaluation and Oversight(OVE). Design and Use of Policy-Based Loans at the IDB. https://publications.iadb.org/en/ove-annual-report-2015-technical-note-design-and-use-policy-based-loans-idb

Programs with a larger number of conditions tended to have a greater share of low-depth conditions. This suggests that IDB could in many cases have been more parsimonious with policy conditions and focused on measures that were critical to achieving the desired results. Similarly, OVE found no correlation between loan amount per capita and how ambitious the reform program was, nor did it find any correlation between loan size and the number of policy conditions. For example, the first loan of a program to strengthen the public finance system in Mexico in the amount of $800 million had 15 policy conditions, while the second loan in support of a water resources reform program in Peru in the amount of $10 million had roughly the same number. These findings are consistent with IDB’s policy-based lending guidelines, which state that the size of the loan is not necessarily related to the cost of the policy reforms or institutional changes supported by the PBL, but rather to development financing requirements.

The depth level of reform programs varied across and within countries; in general, programs in the financial and energy sectors tended to have more depth. When analyzing differences in the depth level, sharp differences across countries were detected. For example, about 22% of the conditions in Peru had high depth, compared with 9% in Colombia programs and less than 5% in Bolivia programs. Moreover, there were substantial differences across programs within countries. In Peru, for example, fewer than 8% of the conditions in a social sector reform program were high-depth conditions, compared with almost 30% in the energy program. The most consistent differences appeared to be at the thematic level: almost a quarter of the conditions in programs in the financial and energy sectors were high-depth, compared with slightly above a tenth in the social and public sector and economic management clusters.

Programmatic PBLs in countries whose reform processes were further advanced at the outset tended to have higher depth conditions. For example, energy sector reforms in Surinam were only modestly advanced when IDB approved a programmatic PBL to help the country develop a framework for the energy sector. Most of the conditions consisted of one-off measures to help set building blocks, such as the preparation of diagnostic assessments and draft guidelines for future legal frameworks. In contrast, a programmatic PBL to support energy sector reforms in Nicaragua supported a reform process that was already advanced and in which the country already had experience. Although it had almost the same number of conditions as Suriname’s, Nicaragua’s PBL had a higher depth level, with conditions that included devising and implementing a new energy tariff structure.

Reforms supported at times of crisis were slightly deeper than those supported outside crisis periods. OVE examined whether PBL programs initiated in times of crisis (which tended to provide countercyclical funding) supported more ambitious reforms than programs started during less adverse economic times. When comparing the depth of the reform programs that were initiated during the global financial crisis (that is, programs for which the first loan was approved during 2008, 2009, or 2010) with those initiated either before or after, OVE found that, on average, programs initiated in times of crisis had slightly higher depth.

• 10
  
  For multitranche PBLs, the average number of conditions per loan was found to have increased from 23 to 32.

Over 80% of programmatic PBL series approved between 2005 and 2014 were accompanied by parallel technical cooperation grants. The grants supported policy dialogue, diagnostic work, and compliance with disbursement conditions and averaged $1.3 million per series. While the resources from a PBL go to the country’s Treasury, parallel technical cooperation grants provide direct support for the line ministries in charge of the reforms and can thus help incentivize them to proceed with reform implementation. While PBL programs supported by technical cooperation grants were not found to have deeper conditions than those without such support, OVE found that there was a significant positive relationship between technical cooperation support and the likelihood of a programmatic PBL series being completed, pointing to the importance of sustained dialogue and technical support by IDB to accompany countries’ reform efforts. The presence of technical cooperation grants was neither correlated with a country’s institutional capacity, nor with income per capita. Less frequently (in 15 of the 82 programmatic series), investment loans accompanied programmatic PBL series, with the PBLs either continuing a line of work initiated by previously approved investment loans or preparing the ground for subsequent investment operations.

Over one third of programmatic PBL series approved since 2005 have been truncated. OVE’s 2015 review of the design and implementation of PBL operations at IDB found that 32% of active programmatic PBL series between 2005 and 2014 had been interrupted, while 40% had been completed and the remainder were still active, resulting in a truncation rate of 44% (truncated series as a share of completed plus truncated series). An OVE update of this analysis to cover PBL programs approved through 2019 showed only marginal improvement in series completion. Of the 124 programs active between 2005 and 2019, 59 have been completed, 36 have been interrupted and 29 are still ongoing (Figure 4.10, Table 4.3), resulting in a truncation rate of 38%. The truncation rate increases with the number of operations in a series: it is 33% for series with two operations, but 43% for series with three or more operations. 

There are significant variations in truncations across countries. For example, Colombia had 16 series between 2005 and 2019 and a truncation rate of over 54%, while Peru with a similar number (15) of programs  had a truncation rate of 8%. OVE country program evaluations showed that, in countries with high numbers of truncated series (e.g., Colombia and Panama), IDB often engaged in a new series in a different sector after a series has been truncated. Since medium- and high-depth conditions tend to be concentrated in the second and third loans of a series, the truncation of a series impairs the program’s depth. OVE’s 2015 review of the design and use of PBL found that the truncation rate was higher when there was a change in government, yet almost 20% of programs had been started within a year of elections and over 40% had been started within 2 years of elections, raising questions about IDB’s timing of programs.                       
 

Table 4.3: Status of Programmatic PBL Series by Program Size, 2005–2019 Approvals

Number of Planned Operations in Series	Completed	Too Soon to Tell	Truncated	Total
2	34	22	17	73
3	24	7	18	49
4	1	0	1	2
Total	59	29	36	124

Source: IDB Office of Evaluation and Oversight (OVE), based on data from IDB databases.

In a programmatic PBL series the policy matrix of each operation outlines the conditions applicable to the loan in question, as well as indicative conditions (called triggers) for subsequent loans in the program. In its review, OVE compared the actual policy conditions in second and third loans in a sample of 28 programmatic PBL series to the most up-to-date indicative triggers and found that about half of the triggers had changed during implementation, reflecting the flexibility of the programmatic instrument (Table 4.4). In terms of policy and institutional depth, for about 14% of the triggers in the second loans, and 19% in the third loan, the depth was found to have been reduced when the loan was approved. Conversely, the depth of conditions rarely increased.

Table 4.4. Changes to Disbursement Triggers and Policy Conditions in 28 Programmatic PBL Series 

Source: Based on IDB Office of Evaluation and Oversight (OVE). Design and Use of Policy-Based Loans at the IDB. Document RE-485-6. Washington, DC: IDB. https://publications.iadb.org/en/ove-annual-report-2015-technical-note-design-and-use-policy-based-loans-idb