Background and Terms of Reference

Following CDB’s adoption of the PBL instrument in 2005, multitranche policy-based loans were made to Antigua and Barbuda, Belize, Grenada, Jamaica, St. Kitts and Nevis, St. Lucia, and St. Vincent and the Grenadines over the period 2006–2009, with an approved total of $242.8 million. In 2010 CDB’s Evaluation and Oversight Division9
commissioned a review of experience with the PBL instrument to date. The overall objective was to assess:

The terms of reference did not mention the Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD-DAC) evaluation criteria or require a rating of performance, but rather they focused on lesson learning and recommendations for improved implementation.  An individual consultant with a background in public finance, fiscal policy, and banking systems was engaged.  The consultant conducted a document review, interviews with Board members and staff, and country visits to meet with senior officials and heads of agencies charged with implementing PBL prior actions.

Review Findings

Context and analytic framework. The review observed that PBOs over the 2006 to 2009 period were transacted in part due to imminent fiscal crisis, and in part out of national political will for a longer-term reform and social protection. The main influencing conditions were:  

• the emergence of fiscal pressures as expenditure and debt commitments outstripped revenue by amounts in excess of available financing on prudent terms; 
• an assessment that the financing gap was not quickly reversible and that measures being taken to address it would take time to yield results; 
• a judgment that financial support already finalized or being discussed with other lenders and donors would not be delivered in time to avert a potential payments crisis; and 
• a conviction that a PBL would ease the immediate fiscal pressures, forestall a potential budget crisis, and allow time for the country to implement policies that would provide lasting stability and improved growth potential. 

CDB’s analytical basis for its PBL lending at the time, shared by other multilateral development banks (MDBs), recognized the pervasive role of the public sector in economic development in small developing countries, and particularly the way in which government policies affect private behavior, investment, and growth. Since these policies were implemented through the national budget and the activities of government-controlled entities, fiscal policy could have a profound influence on growth and the pace of development. 

At the same time, it was recognized that a focus on fiscal policy, public sector management and reform, and debt sustainability, while necessary, was not sufficient to achieve lasting stability and growth, particularly in the 2–3-year timeframes of PBOs. Other factors, including good governance and credible institutions governing law and order and property rights, can be equally important. 

Design process. Typically, a cross-sectoral team of CDB staff visited the country in question for one week. It reviewed the economic situation and policies, focusing on public finances, debt, and the government’s program of adjustment and reform. The macro-fiscal framework and projections were developed, often with inputs from the work of other institutions, including the Eastern Caribbean Central Bank (ECCB), the World Bank, the IMF, and other development partners. In addition, TA needs for diagnostic work or program implementation were assessed, taking into account TA from other providers, including the Caribbean Regional Technical Assistance Center (CARTAC). 

The review found that staff had collaborated closely with country officials and staff of other MDBs, the IMF, and the ECCB on program design, including the macroeconomic framework; projections over the medium term; debt sustainability analyses; and appropriate policies or actions to be incorporated in PBL conditions. It stated that analytical work on topics relevant to the objectives of the PBOs, such as the impact of debt restructuring and the effects of the global financial crisis had been noteworthy.10
The staff analysis of the likely impact of PBL on poverty and the social sectors was frank and well informed. 

All or most of the policy actions included in a loan proposal were expected to have originated from the country’s own economic reform program, with explicit consideration of the likely impact on social conditions and poverty. In cases where other lenders were present (usually the World Bank or IDB) disbursement conditions were to have been calibrated to achieve consistency across institutions and avoid “conditionality arbitrage.” 

However, two areas in program design needed attention: (i) the specification of PBL objectives and likely outcomes; and (ii) the treatment of assumptions and macroeconomic projections. Since clarity was essential for program design, conditionality, and monitoring and evaluation, there was a need for a clearer explanation PBL objectives, creating a stronger basis for the assessment of performance. Furthermore, further refinement of the techniques for macroeconomic and fiscal projection, including explanations of the basis for key assumptions and projections, was also needed to sharpen the analysis and bolster the credibility of the PBL instrument. 

Conditionality. The review summarized the guidance on conditionality from various CDB documents: 

The seven PBOs to 2009 were multitranche operations. Most conditions (about 80%) related to fiscal policy, public financial management (PFM), and debt management, with measures in other areas (economic management, investment and growth, and the social sector) accounting for an average of about 6% each (Table 5.2). Under the fiscal and debt umbrella, most types of conditions were in the revenue category followed by measures covering PFM, debt management, and expenditure. 

The average number of conditions for disbursement of the first tranche of PBL operations was 10, with the figure rising from five in the first PBL to Belize in 2006 to 15 in the case of Grenada in 2009. With regard to total conditions (covering first- and second-tranche disbursements), these increased from nine for Belize to an average of 30 for St. Lucia and Grenada, with a pronounced backloading of conditions, particularly in the case of St. Lucia. A large number of conditions were included in some of the PBOs.11
In small countries with limited capacity, the requirements were viewed as overly burdensome.

Table 5.2. Conditions in Caribbean Development Bank Policy-Based Loans, 2006–2009

* First tranche condition only has listed in Board documents

Country performance on conditions. Compliance with first-tranche loan conditions during 2006–2009 was mixed. Three of the six borrowing countries (Belize, Grenada, and St. Vincent and the Grenadines) met all their conditions without waivers, postponements, or adjustments and were able to draw down their first-tranche disbursements within about 2 months of the date of the loan agreement. Jamaica also secured a rapid disbursement after adjustments to its initial policy matrix, in part, to ensure consistency with the conditions of the World Bank’s development policy loan. Of the remaining countries, St. Lucia’s first-tranche disbursement took place following CDB’s agreement to re-program five conditions to the list of second-tranche conditions. In St. Kitts and Nevis, after a delay of more than 1 year from the signing of the loan agreement, the first-tranche disbursement took place following a waiver of one of the conditions.

 Observations on loan conditions. An important contribution of the review was to gauge the depth of “ownership” of PBL conditions and reforms, as expressed by senior national officials. Some key messages emerged: 

The evaluator added some summary reflections regarding CDB’s use of conditionality: 

Monitoring and supervision. Overall, the assessment found that CDB’s procedures for monitoring and supervising its policy-based loans had not kept pace with PBL operations. Procedures were built on pre-existing systems for investment lending and had been insufficiently adapted to the PBL instrument. Reporting by staff to senior levels in CDB was ad hoc. Reporting required of borrowers in loan agreements (on macroeconomic indicators every quarter for 5 years) was viewed by countries as burdensome and therefore not regularly submitted. 

Conclusions and Lessons 

The review noted that loans had facilitated improvements in frameworks for macroeconomic management, fiscal policy, debt management, and overall public financial administration. Also, revenue systems had been modernized and debt restructuring facilitated. In addition, through their TA components, PBOs had helped strengthen capacity in areas, including macroeconomic forecasting, budgeting, and debt management. 

The review observed that PBOs require careful consideration of feasible policy options, and analytical skills that can mold these into credible loan operations. They also require clear objectives and focused conditions, with specific, measurable goals, particularly in PBOs which are part of joint policy support operations with other MDBs. Goals need to be clear, realistic, and modest with greater consultation in setting loan conditions that are few in number and well defined. A series of discrete, well-defined steps toward reform, supported by a single-tranche PBL, might be more effective than a multitranche loan based on a hopeful set of longer-term commitments. A development bank with a commitment to improving social conditions should not shy away from incorporating social sector conditionality in its policy work. 

Recommendations 

The review offered four main recommendations: 

• 9
  
  The Evaluation and Oversight Division (EOV) was until 2011 a unit reporting to the Vice President of Operations. A new Evaluation Policy converted it into the Office of Independent Evaluation, reporting to the Board, beginning in 2012
• 10
  
  CDB’s analytical work on debt dynamics in Jamaica, for example, helped catalyze contributions from the IDB and World Bank in this area.
• 11
  
  By comparison, a $450 million development policy loan by the World Bank to El Salvador for public finance and social sector reform, approved in January 2009, contained a total of 14 disbursement conditions for two tranches.

Only one year after the first assessment of policy-based lending, the Evaluation and Oversight Division commissioned a second one.  However, rather than review operational experience as had been done previously, this second exercise was tasked with examining CDB’s overall framework for PBL operations, as a prelude to updating it.

The same individual consultant who had performed the first assessment undertook the second, employing the same methodology of document review and key informant interviews, but this time adding a survey.

Terms of Reference

The consultant’s terms of reference were as follows:

Background and Regional Context

By the end of 2010, the global financial crisis was taking firm hold in the Caribbean region:

• Low or negative rates of GDP growth had characterized many of CDB’s BMCs since the early 1990s, and in 2010 the region as a whole was estimated to have registered a contraction.
• A heavy debt burden derived from several years of weak fiscal performance continued to constrain growth and poverty reduction.
• Weak or declining growth had led to rising unemployment, social pressures exacerbated by rising food and fuel prices, and worsening poverty and social indices. 
• Growth was projected to be sluggish until tourism could rebound and was therefore anchored in a recovery in the United States and Europe that remained uncertain for 2011–2012. 

It was against this backdrop that the demand for policy-based lending was framed, to both stave off fiscal crisis and facilitate growth-oriented reforms.

Three new loans had been approved since the previous review, two being single-tranche operations     (Table 5.3).

TABLE 5.3:  PBLs by the CDB 2006 -2010: Basic Data

Source:CDB

a'  In millions of US Dollars

b'  Indicates loans composing three equal tranches

c'  Two interest rates are quoted. One for OCR one for SFR

d'  Actual date or latest estimate

e'  Original date by which loan was expected to b fully disbursed. In the case of Jamaica the date refers to the second tranche disbursement

… = not applicable, CDB = Caribbean Development Bank, IDB = Inter-American Development Bank, IFI = international financial institution, IMF = International Monetary Fund, OCR = ordinary capital resources, SFR = Special Funds Resources, WB = World Bank.  

Of the 10 PBOs listed in Table 4.3, several experienced delays and performed below expectations. There was an average delay of 6 months between Board approval and the signing of loan agreements. Implementation of loan conditions was delayed because of weak capacity in the BMCs, and unclear objectives and design issues (including, particularly in the earlier PBL operations, an excessive number of conditions in multitranche loans and in situations of rapidly changing economic circumstances). Timely implementation and disbursement were observed in only two of the seven multitranche loans. Disbursements of the remaining five loans were hampered by problems of implementation, with a need for adjustments, waivers, postponements or deferrals, or revisions of scope. By contrast, single-tranche loans were disbursed in a timely manner, without the need for waivers, shortly after finalization of the loan agreements.

Review Findings

Institutional and management arrangements. The review assessed the evolution of CDB’s arrangements to manage PBL and found a number of issues that needed to be addressed: 

• The “prudential limit” on policy-based lending of 20% of total loan disbursements had been reached by the end of 2010.  CDB either had to severely restrict further lending or raise the limit.
• CDB needed to clarify the role of the IMF and other MDBs if they were to be involved.
• CDB should clarify operating rules for the funding of PBL by OCR or a blend of OCR and concessional resources (a blend of funding should occur only where there was a social sector or poverty reduction component).
• CDB should establish criteria for recommendations to the Board for the approval of waivers, partial disbursements, and revisions of scope.
• Separate and specifically adapted documentation should be prepared for the appraisal, supervision, and review of PBL (rather than relying on existing investment loan procedures).
• The role of CDB in financial sector restructuring needs to be clarified. Experience with bank rescues in two BMCs suggested that restructuring only be done in coordination with other lenders and TA providers.
• Revised guidelines for sectoral PBL operations are needed, including the extent to which, like IDB, CDB plans to develop them to tackle the many challenges in the social sector.
• An appropriate balance needs to be struck between supporting home-grown reforms and undertaking lending operations in which the contribution of CDB is clearly identified.

Conclusions and Recommendations 

The review concluded that there was a pressing need for CDB to change its processes given the economic crisis facing the region at the time, and the fact that CDB had reached the 20% PBL lending limit set under the 2005 policy. 

After 5 years, important gaps had surfaced in CDB’s framework, indicating that it was no longer adequate to address recent developments in PBL activity or to serve as a comprehensive guide to future PBL operations.  There was a need for greater clarity on key aspects, including the review and supervision of PBOs, loan terms, waivers, TA, and the role of partner institutions, such as the IMF and World Bank. Given the uneven performance of PBOs over the first 5 years (as measured by disbursement delays and the incidence of requests for waivers and revisions of scope), the framework needed to be strengthened by updating the policy and operational guidelines.

The review made the following recommendations:

• 12
  
  IDB allows borrowers to select one of two interest rate options: (i) a pool-based adjustable lending rate, which is tied to the average cost of a pool of medium- to long-term borrowing, or (ii) a London Interbank Offered Rate (LIBOR)-based lending rate.

Notwithstanding the guardedly positive assessments of the 2010 and 2011 reviews, some CDB Board members still questioned the effectiveness of PBL and the extent to which outputs and outcomes were being achieved.  This concern was prompted in part by the waivers sought and granted to certain BMCs, as well as questions about whether the conditions attached to the PBOs had been commensurate with the gravity of the fiscal, debts and broader macroeconomic situations. 

Terms of Reference

A study was commissioned, again with an individual expert, with the following terms of reference:

• Review the rationale and considerations underpinning the current policy-based lending framework.
• Assess the effectiveness of CDB’s policy-based interventions (loans and TA) in support of policy reforms and institutional changes in its BMCs.
• Assess the institutional capacity of CDB to design and supervise effective policy-based interventions.
• Identify lessons learned and opportunities for improvement in policy-based operations and recommend other instruments CDB should consider in support of fiscal and debt management in its BMCs if policy-based interventions are not considered the most effective instrument.

The assessment was based on key informant interviews (CDB staff and members of the Board of Directors), a document review, and a comparison between the PBL policies at CDB and those at other MDBs.

Conclusions and Recommendations          

The review concluded that there was both demand- and supply-side appetite for further policy-based lending.  However, improvements were needed in supervision and monitoring, and questions had been raised as to whether in, certain situations, PBOs had delayed necessary IMF-supported adjustment by BMCs and/or whether the reform agendas as designed and implemented had been sufficiently robust, given the regression that appeared to have occurred in certain cases.  

The review recommended that;

Findings

The assessment summarized CDB PBL operations (Table 5.4).

Table 5.4. Summary of Caribbean Development Bank Policy Based Lending Operation

CDB = Caribbean Development Bank, IMF = International Monetary Fund, MDB = multilateral development bank, OCR = ordinary capital resources, PBL = policy-based lending, SBA = Stand-By Arrangement, SFR = Special Funds Resources, SVG = the Bank of Saint Vincent & the Grenadines was formerly known as the National Commercial Bank (SVG),

Source: CDB, SDF-8 A Framework for the Continuation of Resources to Address Fiscal Distress,          Annex, SDF 8/3 –NM-2-1

As part of a cluster evaluation of CDB’s country strategies and programs in the Organisation of Eastern Caribbean States (OECS), the Office of Independent Evaluation (OIE) undertook a review of PBL experience in these six countries, and three overseas territories of the United Kingdom.  The review drew heavily on the 2017 OIE evaluation of all CDB PBL (presented below) and distinguished between first- and second-generation loans.  First-generation loans were characterized by more numerous and diverse prior actions, while second generation loans were more focused on their reform expectations.

The OECS borrowers are among the smallest and most vulnerable of CDB’s members, with some in debt distress and others at moderate to high risk of becoming so. Over the review period, five OECS members received 10 PBL operations totalling $319 million.  The loans supported reforms in PFM; public debt restructuring and management; macroeconomic planning; public sector reform; social sector reform; sector reform (banking and finance, tourism, food safety, energy regulation); and trade facilitation.

As with other studies, this review confirmed the demand for the instrument in facilitating debt restructuring and averting banking crises in the Eastern Caribbean Currency Union, as well as in supporting “home-grown” reforms.The review probed some important design issues.Apart from noting the incidence of wide and unfocused reform plans in some early PBOs, it assessed the “quality” of prior actions across all 10 PBOs.

To do this, it applied the IDB classification of low-, medium-, and high-depth prior actions, according to the likelihood a given prior action would trigger lasting policy or institutional change. On this basis it found 25% were low-depth, 48% medium-depth, and 27% high-depth.In programmatic series, high-depth prior actions were observed in the later loans, evidence of good sequencing.  As CDB had not at that time undertaken  such an explicit analysis of prior action depth, the OES review suggested that it begin doing so.

The review found evidence that PBOs had facilitated bank resolutions in three members following the 2008 financial crisis, heading off potential contagion in the Eastern Caribbean Currency Area.Orderly debt restructuring and avoidance of default had also been accomplished in one of the smaller members.

The outcome of targeted reforms in PFM, public debt management, tourism, trade facilitation, disaster management (legislation, building standards and codes), and social safety nets was documented. However, there were numerous implementation delays across the portfolio, often as a result of insufficiently anticipated gaps in national capacity.

The review made a number of suggestions for improved PBL planning. CDB should:

This was the first fully independent overall evaluation of CDB PBL, carried out by its Office of Independent Evaluation (OIE).  It assessed the nearly $550 million in PBL to 12 borrowing members approved over the period 2006–2016, employing a significantly higher level of effort than the three earlier reviews of 2010–2012.

Objectives of the Evaluation

The evaluation’s objectives as stated in its terms of reference were to assess:

• the need for the PBL program,
• the relevance of the PBL program to BMCs,
• the achievement of results for BMCs,
• the design and implementation of the PBL program, 
• the extent to which the PBL compares with international experience, and
• ways in which the program can be improved to support CDB’s strategic objectives.

Evaluation Approach and Methodology

This was a theory-based evaluation, with a reconstructed theory of change for the PBL program (Figure 5.3), validated with stakeholders. It tested numerous assumptions that underlay the program.

BMC = borrowing member country, CDB = Caribbean Development Bank, PBL = policy-based lending, PS=public sector, TS=technical support

Figure 5.3 suggests that the logic of CDB’s PBL program has two parts: (i) the conditions set by CDB, and (ii) the conditions actually implemented by BMCs. CDB created the PBL initiative to enable BMC governance reforms that would not otherwise occur. Specifically, PBL was intended to assist “small and vulnerable economies with declining growth rates, persistent and growing trade deficits, high indebtedness, with significant public-sector capacity constraints.13
”To support such economies, CDB prepared funding contracts with conditions negotiated with borrowers to address policy-based reforms. CDB assessed whether BMCs were carrying out the conditions of these contracts through regular monitoring and oversight as shown by the cross arrows between the two causal pathways in the figure. For their part, BMCs accepted the conditions contained in those contracts with the long-term objective of ensuring macroeconomic stability and public capacity to meet their development goals.

In program theory literature, the change theory that best explains whether CDB can create such conditions is called “planned behaviour.” This theoretical framework suggests that if CDB creates appropriate application, review, and implementation processes for its PBL program, and there is a clearly stated need and rationale for the PBL intervention, then borrowers will utilize the program to buttress their own reform efforts and prevent breakdowns or crises in their local governance systems. For their part, BMCs will not successfully effect the reforms unless conditions are built that maximize their room or flexibility for programs of reform based on their own identification of needs. Such flexibility provides the local confidence and commitment needed to respect PBL agreements.

Extensive evidence was gathered to test the assumptions in the theory of change (Table 5.5).  This included in-depth case studies of PBL operations in Barbados, Jamaica, Grenada, and St. Vincent and the Grenadines; a meta-analysis of PBL experience at other MDBs; extensive interviews with BMC and CDB officials; and analysis of secondary (mostly macroeconomic) data. 

Table 5.5: Assumptions Tests by Evaluation Criterion

PBL = policy-based lending, PBL = policy-based operation.

Findings and Conclusions

Need, relevance, and rationale. It is beyond dispute that MDB lending has been important to BMCs, enabling them to address fiscal pressure and debt management, as well as to encourage economic and social sector reforms. However, different parties emphasized different aspects of the instrument. Borrowers tended to be driven by short-term fiscal pressures, particularly in the aftermath of the 2007–2009 financial crisis. PBL support played a role in helping some of them through that period, and at times they agreed to reform programs that they did not entirely buy into. For their part, lenders (including CDB), while recognizing fiscal exigencies, understood the PBL to be primarily an instrument that provided incentives to implement reforms. At times they required large numbers of “prior actions” from BMCs as conditions of PBL support.

To some extent this difference in perspective had to do with sequencing: in one view relieving fiscal pressure first to allow the space to eventually undertake reforms; and, in the other, adopting reforms that will eventually help open fiscal space. While these views could co-exist in the broad space of acknowledged need for PBL lending, their differences did have implications for the expectations and approach to PBL negotiations by the respective parties. 

A number of MDBs and other partner organizations—including the World Bank, IMF and IDB— brought significant funding to policy-based lending in the region. Respondents had clearly reflected on the appropriate role and value added of CDB among these larger players. They alluded to CDB’s more detailed understanding of BMC contexts, its closer working relationships with governments, and the potential for brokering harmonized reform packages that included non-economic governance elements.

Planning and design. The quality of the process by which borrowers and lenders came to an agreement on the design of an intended reform program was an important predictor of eventual success. The evaluation observed that in the first generation of PBL operations there was a perceived imbalance in negotiating leverage between CDB and borrowers (favouring CDB). As a result, the ownership of the prior actions by the BMCs and their commitment to expected reform outcomes were sometimes less than complete. This was compounded in cases where CDB’s consultation did not involve a sufficient range of stakeholders, particularly those who would either have a role in implementing reforms or would be affected by them. Not hearing these views at the outset came at the cost of lack of buy-in or even resistance to intended reforms during implementation. More recent PBL design processes had performed better in this regard.  

Apart from the process of arriving at a design, the actual nature and number of prior actions and expected reform outcomes were important determinants of effectiveness. Again, it was observed that there was an evolution from earlier to more recent PBOs. Pre-2013 PBOs tended to require larger numbers of prior actions across multiple sectors, and these often lacked a clear causal linkage to the higher-level expected reform outcomes. BMCs felt that prior actions did not always reflect national reform priorities, and that the cost of delivering on them sometimes exceeded the value of the PBL on offer. More recently, there have been examples of PBOs with streamlined prior actions in fewer areas. These actions have been better calibrated to the scale of assistance being offered, and more likely to be achieved. There has also been some evidence of successive PBOs building on earlier efforts, with prior actions requiring incremental progress from earlier to later loans.

Assessing at the outset whether borrowers had the capacity to implement intended reforms was a necessary element of good PBL planning. Providing TA responsively during implementation to address bottlenecks was also important. To date, this has not been an area of strength for CDB.

Harmonization of CDB PBOs with those of other MDBs became stronger over the evaluation period. However, an unanticipated consequence of this harmonization has been that closely synchronising CDB’s prior actions with those of other lenders has somewhat limited CDB’s flexibility to tailor its own offerings. Such tailoring could grow out of CDB’s particular understanding of BMC context, or its interest in promoting reforms focused on non-economic areas.  

Implementation. The timeliness of fund disbursement under the PBL mechanism was efficient. That said, there were some instances of tranche payment in the absence of all prior actions being met (which is likely to have been related to earlier findings regarding numerous conditions and national capacity constraints).  CDB’s monitoring of PBOs was inconsistent. Project supervision and completion reports were sometimes missing, and monitoring was more oriented toward verifying completion of prior actions than to assessing progress towards reform outcomes. Evidence was not always available to corroborate project completion report statements.

The quality of PBL results frameworks was not optimal. The link between prior actions (outputs), and economic, sectoral, and institutional reforms (outcomes) was not always clear. Proposed indicators and targets were not necessarily good measures of the outcomes with which they were (or should have been) associated.  BMCs lacked the capacity to report on the range of expected results. Statements of risk tended to be generic across PBOs, missing the need for mitigation strategies specific to each PBO’s expected outcomes and national contexts.

The revised framework document of October 2013 placed renewed emphasis on the longer-term reform orientation of policy-based lending, and the value of programmatic PBL. At the same time, there are varying stages of readiness for reform implementation across the region, and a menu of PBL instruments, including multitranche PBOs, may be needed to respond to different situations.

Results achievement. Completion of PBO prior actions for three of the four case study countries (Barbados, Jamaica, and Grenada) was verified.  This totaled 113 prior actions across five PBOs. In the fourth case study country (Saint Vincent and the Grenadines), completion of 19 out of 23 originally planned prior actions was verified; the other four were waived with Board approval to allow a second-tranche disbursement.14
Among the short-term outcomes of the PBL operations were:

• debt management improved;
• fiscal space created that allowed BMCs to bolster social program reforms or reduce economic stress on individuals and families;
• conditions for investment improved to bolster key industries (such as tourism, by reducing wait times at border crossings, which could be attributed in part to PBL); and 
• critical management systems such as audit, budgeting and planning improved, contributing to increased public sector management efficiency.

Because of the number of causal factors in play, including support from other PBL lenders and global economic events, it was difficult to attribute medium-term outcomes directly to CDB lending. Nonetheless, BMC officials across all case studies indicated that a coordinated, targeted, and ongoing program of reform supported by lenders such as CDB had ensured momentum, leading to improved economic and social program performance. For example, Jamaican respondents indicated that its 2008 PBL was, “a critically important intervention in Jamaica, and with the support of other MDBs helped to identify first generation structural reforms on which the recent fiscal gains have been premised.”

Generally, however, it was not feasible for the evaluation to gather a sufficient amount of directly attributable evidence to support statements of causal linkage between CDB’s PBL support and higher-level medium-term outcomes.  This is a common difficulty in PBL assessment across MDBs, although as mentioned above an improvement in CDB’s specifications, measurement, monitoring, and reporting on results would help.

Summary Comments and Recommendations

Over the 10 years since the introduction of PBL at CDB, there had been an evolution in practice that reflects CDB’s learning and experience in managing the instrument, and its observation of how other MDBs also manage PBL. The loans addressed an evident need among BMCs.

The findings and conclusions of this evaluation, based on evidence generated from the document review, documented case studies, and a wide range of interviews, suggested that several key factors increased the likelihood of PBL operations achieving their desired results.  These were:

• clear objectives and results logic, with indicators and targets that can be measured and verified;
•  a selective focus on a manageable number of expected reform outcomes;
• agreement on a limited number of prior actions that are clearly linked to those outcomes;
• good understanding of external risks, and elaboration of mitigation strategies;
• an engagement process with BMCs that engenders ownership and commitment on the part of borrowers;
• a menu of PBL options that offers the right instrument calibrated to borrowers’ reform readiness;
• an understanding of national capacity constraints and, where needed, provision of affordable TA to address them;
• designation of an identified champion in the national public service with responsibility and authority for achieving reform results; and 
• consistent monitoring to identify when conditions are met, and the degree of progress towards reform outcomes.

Although the evaluation found that CDB’s PBL was increasingly taking account of these factors, it offered the following recommendations to encourage further progress.

Management Response

Management expressed general agreement with the findings, conclusions, and recommendations of the OIE evaluation, with one area of exception.  It felt that evaluators had underappreciated the extent of staff engagement with borrowers in arriving at agreed prior actions and reforms, and thus also understated the degree of national ownership of PBL-facilitated reform programs.  It acknowledged, however, that consultation processes could have been better documented, that results frameworks should have more clearly established the logic of the links between prior actions and reforms, and that monitoring could be improved.

Management accepted all recommendations and provided a time-bound action plan for their implementation. The Oversight and Assurance Committee, a subcommittee of the Board of Directors, annually monitors completion of these actions.

• 13
  
  Caribbean Development Bank. 2013. Policy Paper: Framework for Policy-Based Operations - Revised Paper. BD_72/05 Add. 5
• 14
  
  In view of the challenging economic circumstances at the time, and the otherwise positive reform trajectory, the CDB Board authorized disbursement of the second tranche of the 2009 PBL, notwithstanding delays in completion of four prior actions. A separate 2010 PBL operation for St. Vincent and the Grenadines used an unusual formulation involving six prior actions and seven post-disbursement conditions or indicators. Completion of the prior actions was verified at the time of the evaluation, along with four of the post-disbursement conditions.