The first independent evaluation of PBOs was undertaken in 2011 and assessed AfDB’s use of PBOs over the period 1999–2009 (footnote 71). The evaluation examined how effectively AfDB had used PBOs to support national development objectives, with a focus on AfDB’s policies and procedures for PBOs. It was based on: (i) a review of the literature and comparative experience with PBOs in other development agencies, (ii) a review of AfDB’s institutional and policy framework, (iii) six country case studies, and (iv) four case studies of other significant operations. 

The evaluation concluded that AfDB had made substantial progress in its use of PBOs. In 1999, AfDB was heavily dependent on the IMF and the World Bank for the analysis and design of its engagement in structural and sectoral adjustment operations. As of 2011, AfDB operated as a significant partner in joint donor budget support arrangements, and the record of its engagement, as shown by the country case studies, was largely one of success. AfDB had developed a cadre of staff with strong experience in the design and management of budget support. The establishment of country offices had significantly improved AfDB’s ability to engage in national policy and budget processes and had strengthened its monitoring and supervision of PBOs (even though decentralization progressed far more slowly than planned). 

AfDB had strengthened its organizational capacity and structure for the design, appraisal, management, and monitoring of PBOs, although some aspects still required further development. It had proved highly responsive to the international economic and financial instability that affected RMCs during 2008 and 2009. AfDB was able to design and implement operations to meet the urgent financial requirements of its clients and these operations provided a platform from which longer-term structural reforms could be addressed. It also made important contributions to the development of budget support arrangements under the Fragile States Facility; in Liberia, for instance, AfDB played a leading role in moving other development partners toward budget support. 

The evaluation found that, while AfDB had succeeded in engaging effectively in joint budget support arrangements and in mobilizing rapid responses for fragile and crisis-affected countries, there were some shortcomings in its policies and practices. Its numerous policies and guidelines were not being consolidated or updated. Project procedures were not being fully documented, and were designed for investment operations rather than specifically tailored to PBOs. There was a lack of clarity about how results should be defined and measured for PBOs, information systems were weak, and audit and fiduciary risk issues needed to be addressed. 

The evaluation was based on seventeen background reports, as well as on a series of focus groups and reference group discussions (footnote 72). The evaluation scope covered the 9127
PBOs approved between 2012 and 2017, with a collective approval value of UA7.2 billion. 

The evaluation was designed to: (i) identify factors which had enabled or hindered good performance, (ii) draw lessons for AfDB, and (iii) identify specific recommendations to help AfDB to optimize the effective use of the PBO instrument in future. Specifically, the objectives of the evaluation were to provide credible evidence on: (i) how AfDB was programming, designing and managing its PBOs in accordance with the 2012 policy and other elements of good practice;28
(ii) the performance of PBOs in specific areas; (iii) the factors that had enabled or hindered achievement of PBO objectives; and (iv) lessons that could inform AfDB’s future use of PBOs to ensure consistent good practice and to support achievement of the High 5s. 

The evaluation addressed three overarching questions. 

These three questions were broken down into eight sub-questions, addressed through over 40 criteria.

• 27
  
  Sixty of these relate to programmatic operations, where two or three separate operations together are considered a series, but each phase in the series is approved as an individual operation. The number of single PBOs plus PBO series is 51.
• 28
  
  There are various sources of advice on “good practice.” The most relevant for AfDB is OECD-DAC. 2006. Harmonising Donor Practices for Effective Aid Delivery. Volume II. Paris: OECD, which laid out a set of principles for the provision of budget support consistent with the 2005 Paris Declaration.

A generic theory of change for AfDB PBOs was constructed based on AfDB documents, consultations with internal stakeholders, and the methodology endorsed by the Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD-DAC) for evaluating budget support operations. The theory of change helped to refine the evaluation questions. Individual theories of change were also developed for each of the 10 in-depth country studies based on the generic version.

The evaluation had seven components and each used different sources of data and analytical techniques. In addition, the evaluation was based on a thorough inception phase which included two staff focus groups and interviews with 42 internal stakeholders, including eight executive directors and 12 members of senior management. 

For the 10 in-depth studies, a cluster approach was used. This allowed strong evidence to be collected within specific areas but limited the extent to which the findings could be generalized across the full portfolio. The two focus areas of energy and the private sector enabling environment (PSE) were identified early in the design stage following analysis of: (i) the recent use of PBO funds, (ii) the availability of evidence, and (iii) the pertinence of these areas to future directions for AfDB, particularly in support of the High 5s. In addition to evaluating these two focus areas, the evaluation also examined the PFM components of the PBOs (present in nine of the 10 countries). It is important to note that, for the two general budget support (GBS) cases that included a broad range of targeted sectors in addition to energy, PSE, and PFM, those other sectors were not the focus of in-depth analysis, although the delivery of all reforms was assessed.

Within these two focus areas of energy and the private sector enabling environment, cases were selected according to the following considerations. 

• Evaluability The sample included countries with PBOs that were at a reasonably mature stage of implementation (least 12 months since approval) so some influence could be expected on intermediate outcomes (known as “induced outputs,” see Table 2). All 2017 approvals were therefore excluded.
• Contemporary relevance The sample covered countries with relatively recent PBOs, whose design and implementation should reflect the 2012 policy, and where the process of implementation could still be recalled by those interviewed. This meant most of the cases came from the 2014–2016 period. 
• Diversity in types of PBOs In selecting the cases, the goal was to include examples of SBS, GBS, single-standing operations, programmatic operations, and programmatic tranching.
• Diversity in country contexts The sample covered: (i) MICs, LICs, and transition countries; (ii) countries in at least four of the five sub-regions in which AfDB operates; and (iii) anglophone, francophone, and lusophone countries. 
• Diversity in size of PBOs The sample included some of the largest and most important PBOs, intermediate PBOs, and small PBOs. 

Ten countries and 16 operations were covered by the in-depth studies (Table 2.1). Collectively, they accounted for UA2,155,040 in approvals and 36% of PBO approvals by amount in 2012–2016. The assessments covered energy, PSE, and PFM29
. However, the sample was not designed to be generalizable across the full portfolio. In particular, PBOs with a focus on social sectors—which have also been an important part of the portfolio and are managed by a different department—were not covered.

• 29
  
  PFM is a cross cutting area and most PBOs (Energy and PSE) have a PFM component.
• 30
  
  AfDB. IDEV Approach Paper, IDEV Information Note to the Board, Final Inception Report(2017).

The evaluation found that PBOs remained a relevant and useful instrument for AfDB and its clients, although they were challenging to design and manage effectively. The evaluation found the relevance of the PBOs in AfDB’s portfolio to be broadly satisfactory, based on their programming and design and their broad adherence to AfDB’s own policy and guidelines and to international good practice. With regard to the achievement of reform objectives, the overall picture was also satisfactory. However, it was much harder to find evidence of AfDB’s influence on reform direction and speed. Regarding sustainability, even in the presence of strong ownership, concerns about the institutional and financial dimensions of sustainability meant that the overall outlook for sustainability in the sectors examined was unsatisfactory. 

Despite having deployed UA7.2 billion as PBOs 2012–2017, AfDB had failed to invest in its own institutional infrastructure to obtain maximum value for money from the instrument. As was well reflected in the 2012 policy, PBOs were expected to form part of a “package of support” in order to ensure that they influenced and supported reform agendas, while also providing important funding. This package included analytical work to inform technical input, policy dialogue and capacity support. In practice, while there was some variation across countries, overall AfDB had underperformed when it came to policy dialogue, despite its strong position as a trusted partner. This was partly due to its institutional arrangements; a lack of clarity about who was responsible for policy dialogue; the structure of how the dialogue should be conducted, reported, and  coordinated; and a lack of investment in human resources to conduct it. In addition, AfDB had underperformed in providing timely and adequate capacity support and specialized technical advice, partly due to the limited menu of instruments available to do so. These shortcomings had implications for how well AfDB was able to influence or add value to country reform paths. 

Programming issues. A range of programming issues were examined and, while the overall picture was assessed to be broadly satisfactory, the evaluation identified areas that could be strengthened. First, for the large majority of the PBOs reviewed (94%, excluding CRBS), their use was envisaged in either the relevant country strategy paper (CSP) or the midterm review (MTR), in line with the policy. However, in the majority of cases, the assessment against the eligibility criteria was made for the first time during the PBO preparation phase. In terms of the type of PBO, the justification for the type chosen could have been be stronger, especially when the PBO did not use the recommended programmatic approach. Second, in approximately two-thirds of the operations reviewed, the analytical underpinnings used were clearly listed and relatively complete. However, exactly how this work informed or underpinned the design of the operation was not clear. Third, while risk assessment was assessed as satisfactory in two-thirds of the operations reviewed, reputational risk was rarely explicitly considered. The risk mitigation measures, such as future capacity support to address current risks, were generally not convincing within the timeframe of a PBO. 

Alignment with country and AfDB priorities. This was assessed positively on the basis of the document review and in terms of stakeholder perceptions. All the PBOs could be mapped to at least one of the High 5s or supported crucial governance issues which cut across them. AfDB had also succeeded in expanding use of the instrument to support sector reforms in addition to economic and financial governance. Nearly 80% of survey respondents had positive views on the alignment of PBOs with country policy frameworks.

Coordination. There were many good examples of how AfDB had coordinated with other development partners, notably during the identification and appraisal periods. AfDB staff had taken coordination seriously and had invested in upfront work with other development partners. However, the in-depth assessment illustrated how difficult AfDB had found it to sustain these initial high levels of coordination throughout the implementation phase. Moreover, following the adoption of the G20 Principles for Effective Coordination between the IMF and MDBs on Policy-Based Lending in 2017, in countries facing macroeconomic vulnerability MDBs needed to align behind the IMF. 

Designing PBOs for results. The overall picture was satisfactory, although some shortcomings were identified. Although two-thirds of the PBO appraisal reports examined stated there was an important role to be played by complementary inputs, only a handful explained how this was to be achieved. All PBO results frameworks defined baselines, targets and means of verification, and integrated prior actions and triggers. However, over a third were less than satisfactory because of: (i) weaknesses in presenting a convincing results chain; (ii) high proportions of process- and action-based indicators; and (iii) a lack of realism, particularly for single-year operations. The use of conditions was suitably selective; in programmatic operations these linked from one phase to the next in order to plot a medium-term path, and they were linked to broader dialogue frameworks. However, weaknesses were noted when the number of prior actions was high, opportunities for identifying triggers were missed, or the level of ambition for prior actions was not appropriate. 

Gender and environment. The evaluation found that AfDB had missed valuable opportunities provided by the PBOs to address gender equality and environmental reform issues at the policy level. Just over a third of the PBO project appraisal reports (PARs) that were assessed included gender-related indicators and 7% included environmental or climate-sensitive indicators. The opportunity to push gender equality and environmental concerns varied according to the type of PBO. However, particularly in the energy sector, PBOs can provide valuable opportunities to shift national policies in support of AfDB’s ambitions of inclusive and green growth.

Efficiency. Broadly speaking, PBOs were broadly disbursed and implemented in a timely way, although some receiving countries said that disbursement was unpredictable. In line with expectations for the PBO instrument, the evaluation found that AfDB had disbursed the funds fully and, compared with investment projects, quickly.  In addition, implementation progress was very rarely identified as a cause for concern. Nine of the 10 in-depth assessments were efficient in terms of transaction costs and the time taken to disburse the funds. However, perceptions of timeliness and transaction costs varied among both staff and RMC officials.

Technical assistance. Perceptions of the efficiency and transaction costs of technical assistance or institutional support provided to support PBOs was negative. Such support, when it was provided, was slow and tended to arrive toward the end rather than beginning of a PBO series. This was partly because  capacity support tended to be designed in parallel with PBOs rather than in advance, and partly because of the limited set of instruments AfDB had available to provide small items of technical assistance, all of which operated like full projects rather than as rapidly deployable expertise.

Policy dialogue. AfDB did not use policy dialogue sufficiently or make best use of its “African Voice” to ensure PBO results. This finding is not dissimilar to that of the 2011 evaluation which described AfDB as “punching below its weight” when it came to policy dialogue. Only three of the 10 in-depth assessments had satisfactory frameworks for policy dialogue in the targeted sectors. The deficiencies that emerged in relation to policy dialogue can be broadly categorized as: (i) lack of clarity over who is leading and responsible for policy dialogue, especially after approval;     
(ii) limited capacity to engage in in-depth technical dialogue in some areas; (iii) lack of structured planning or reporting for policy dialogue efforts, including through AfDB’s normal supervision channels; and (iv) lack of a medium-term strategy to capitalize on doors that may be opened by a PBO after formal completion. In the survey, fewer than a third of respondents were clearly positive when asked about the extent to which AfDB mobilizes appropriate resources for policy dialogue. In the 10 countries investigated, only five had a satisfactory framework for policy dialogue. 

Policy guidance. The existence of the 2012 policy helped AfDB to improve its approach to PBOs and to make it more consistent; however, there were areas where implementation was wanting. The policy provided clarity on the authorizing environment and on a range of important issues, including the type of instrument, when it should be employed, on what basis and with what objectives. The policy was broadly aligned with good practices. Although it clearly set out activities or changes that needed to take place in order to facilitate implementation, not all aspects of implementation have gone as planned. For example, there have been delays in producing the supporting guidelines, a glaring lack of training, and unfinished business in ensuring an enhanced role for country offices. The guidelines, which are described as a living document, have not been updated and there has been no additional guidance on new reform areas, such as energy. The guidelines have not been following the adoption of the G20 principles. The survey and focus groups both also revealed staff demand for more guidance in areas such as policy dialogue, working in post-conflict contexts, and results measurement.

Institutional arrangements. Some of AfDB’s practices were out of line with both its own policy and the practices at the World Bank and the European Union. First, PBO design and management remained somewhat centralized and led by either the Governance and Public Financial Management Coordination Office (ECGF) or by sector departments. The extent to which country offices had taken up ownership varied significantly. Second, in practice, there was no centralized unit that provided specialized support to PBO teams. ECGF staff had been task-managing most of AfDB’s GBS. This lack of a central support unit, and the limited guidance and training provided to staff, was in stark contrast to the support available at the World Bank and the European Union.

Effectiveness. Overall, the assessment of PBO effectiveness, which focused on energy, PSE, and PFM, was broadly satisfactory. The evaluation highlighted areas where AfDB could focus attention in order to strengthen results and specifically how it could contribute to the direction and pace of reforms. Data from project completion reports and country strategy and program evaluations by the Independent Development Evaluation indicated that the satisfactory assessment was likely to reflect the effectiveness of the broader portfolio.

• All of the 10 cases achieved or partially achieved all, or the majority, of the reform actions listed in the results framework. In only one case were 25% of outputs considered to have been not achieved. In all other cases, at least 75% of the reforms were assessed to have been either fully achieved, partially achieved, or achieved with significant delay. With regard to the achievement ratios by sector covered, no clear pattern emerged. No sector performed notably better than any other. At an aggregate level, in seven of the 10 countries covered by the in-depth studies the overall effectiveness in terms of the achievement of the objectives stated in the results measurement framework (RMF) was considered satisfactory. 
• Across the 2131
  individually assessed components, two-thirds were assessed satisfactory in terms of the achievement of “landmark policy changes.” Within a PBO RMF, some actions can be much more important than others, although an RMF can include a large number of “tick-box” type items alongside more fundamental issues that have the potential to drive change and contribute to transformative outcomes. Such indicators were identified as “landmark policy changes.” Where these were not achieved, it is worth noting that the sample included a transition state (Comoros); two cases where the principal focus of the PBO was in another area (Tanzania and Seychelles); and one where the second part of the planned series never took place (Nigeria). 
• AfDB’s influence on the achievement of landmark policy changes was not always evident. In one third of the components, AfDB’s influence on either the direction or pace of reforms was evident and was usually achieved through analytical work, technical inputs, and policy dialogue. AfDB staff respondents to the survey supported the view that AfDB’s influence was limited, and strongest at the appraisal stage. 

Sustainability. The sustainability of PBOs in energy, PSE and PFM was assessed to have been unsatisfactory, particularly in relation to the institutional and financial dimensions of sustainability. Only four of the 10 in-depth assessments had good prospects for sustainability. Almost all of the 10 had laid strong foundations for sustainability in terms of government ownership and leadership, which should be at the core of the decision to proceed with a PBO. However, the weak institutional and financial sustainability undermined the positive assessments in terms of ownership. While this trend was clear for energy, PSE and PFM, it cannot be generalized across the whole PBO portfolio.

Contextual, design, and management factors. The evaluation evidence from AfDB and from other institutions providing budget support in Africa indicates that the most frequently identified factors relating to country context were: (i) ownership, country capacity, having a “champion” for reforms; (ii) the country’s socioeconomic status; and (iii) country systems. The most frequent factors relating to the budget support mechanism were: (i) quality of design, programming, development partner coordination; and (ii) quality of monitoring and choice of indicators. The single most frequently cited enabling factor was the quality of design. In terms of hindering performance, the most frequently highlighted were: insufficient policy dialogue, high efficiency and transaction costs, poor choice of indicators, weak monitoring, and poor predictability. 

The most significant factors associated with achievement of landmark policy changes,32
stronger even than the country’s socioeconomic status, were: programming, design, and efficiency factors; technical assistance; the operation being part of a series; and the existence of a country office.

• 31
  
  Separate assessments were made for each of the relevant sectors: PFM, energy, and PSE. The total number of assessments was 21. Where the PBO was part of a series, the assessment was for all parts of the series completed or well underway.
• 32
  
  Budgetary or institutional changes of substance and influence targeted by PBOs within the set of intermediate outcomes (induced outputs) identified in the theory of change.

Updating the guidance framework. Changes in the international context, calls for more internal guidance, and shortcomings in design and implementation indicate that the 2014 guidelines on implementing the 2012 policy need to be updated. The analysis indicates that a strong relationship between the government and the development partners and the presence of an ongoing IMF program are fundamental to the achievement of reforms.

Recommendation 1: Update or complement the PBO guidelines 

• Reflect AfDB’s response to the 2017 G20 principles on coordination with the IMF. 
• Provide detailed guidance to staff on the most challenging areas of the results frameworks, including: conducting effective policy dialogue, post-conflict concerns, and promoting reforms in support of gender equality and environment and climate change. 

Enforcing compliance. Although the assessment of programming and design was satisfactory (most of the PBOs in the sample were assessed to have been satisfactory against selected criteria), AfDB aims for every PBO to be satisfactory, especially in relation to 100% compliance with the provisions of AfDB’s own policy and guidelines.

Recommendation 2: Fully enforce the provisions of the 2012 policy 

• Use of non-programmatic operations or operations that are not already programmed in the country strategy paper (CSP) or CSP midterm review (MTR) should be done only on an exceptional basis as per the 2012 policy. Such operations should have a convincing rationale and should be based on sound analysis, including an evaluation of the alternative options. 
• Conduct fiduciary risk assessments when the decision is first made to use a PBO. The assessment should be updated at appraisal, and the proposed risk mitigation measures should be adequate to address the identified risks within the timeframe of the planned PBO. 

Focusing PBO ambitions. Some PBOs were spread over a broad range of reform areas. Moreover, analysis was not always undertaken to identify where AfDB could add most value, including through the expertise it could provide, or which reform actions would pave the way to “landmark policy changes.”

Recommendation 3: Design all future PBOs with a focus on a limited number of medium-term reform areas from within broader government reform plans

• Assess which reforms have the potential to pave the way to landmark policy changes. 
• Evaluate AfDB’s complementarity with other development partners and with its wider portfolio. 
• Judge the ability of AfDB to add value in these areas, especially in terms of analytical work, expertise, and policy dialogue. 

A tight focus should be combined with a strengthening of the medium-term dimension in the design, i.e., programmatic PBOs should follow a clearly defined multi-year reform path, as well as paying attention to how AfDB might accompany reform processes over the medium term over one or more PBOs. 

Prioritizing policy dialogue. Policy dialogue is a central part of how PBOs achieve results and how AfDB adds value to reform processes. Yet there was a lack of clarity and insufficient prioritization of policy dialogue in AfDB PBOs.

Recommendation 4: Reflect the vital role of policy dialogue in PBOs in practice 

• Make unequivocally clear at the design stage what policy dialogue will entail, what mechanisms will be used, what the priorities will be, how policy dialogue will be underscored by relevant technical expertise, and who will be responsible for conducting and reporting it. This can be done by including a standard annex on policy dialogue priorities and responsibilities in the PBO’s project appraisal report. This would provide a starting point that could be adapted over time to respond to new policy needs as they arise. 
• Align practices with plans in the 2012 policy and the development and business delivery model (DBDM) by more clearly allocating responsibility for PBO design and management to country offices and regions. Ensure country offices and regions have sufficient resources and the necessary reporting structures to take up this responsibility, and provide strong technical support from headquarters teams. (Alternatively, if AfDB prefers to operate a centralized model, the policy and DBDM documents should be adjusted to reflect this approach to remove any confusion). 
• Ensure that budget lines for PBO appraisal and supervision take account of the need to involve the appropriate range of expertise in the case of PBOs that cover a range of areas. 

Using technical assistance more efficiently. The other complementary input supporting PBOs was technical advice and capacity support. AfDB has tied its hands by relying on a limited menu of instruments on which it can call to provide this support, and some of these do not provide support in a timely or efficient manner.

Recommendation 5: Provide PBOs with appropriate and timely expertise and capacity support 

• Examine how to refine and expand AfDB’s menu of options when it comes to providing expertise and technical assistance. This should include: (i) reviewing how to make the MIC Trust Fund and other trust funds more flexible so as to improve their relevance; (ii) investigating other instruments, including short-terms options, such as framework contracts with specialist companies that can provide quick and high-quality technical expertise that is not available internally; and (iii) providing longer-term solutions such as a fast-track technical assistance scheme. 
• Require clear justification if relevant capacity support or expertise is not already in place or at least planned by the time approval for any PBO is sought. 

Investing in supporting institutional infrastructure for PBOs. AfDB has not appropriately invested in its most important tool for making PBOs an effective and value-for-money instrument: its people. It has no central support team charged with supporting the instrument at a technical level or for cross-learning purposes. It has established only minor differences between quality at entry and processes for PBOs as compared with those for investment projects.

Recommendation 6: Invest in the supporting infrastructure for PBOs 

• Invest in continuous training for staff involved in PBO design and implementation. Such training could take the form of an accreditation scheme and draw on the rich experience that has been gained internally, while also drawing on lessons from elsewhere. 
• Invest in upfront analytical work to support PBO design and the focus of policy dialogue and capacity support, which will require forward planning and resources to allow teams to conduct or commission it.eview the extent to which AfDB’s quality assurance processes are appropriate for PBOs, in particular the readiness review. Strengthen supervision and reporting of supervision.