The OECD-DAC methodological approach15
 to evaluating budget support, sometimes known as the three-step approach, was used in all the evaluations listed in Table 3.2. This approach acknowledges that budget support cannot deliver outcomes and impacts by itself, as it can only contribute to the outcomes and impacts that are achieved through the implementation of governmental policies and public spending. EU budget support focuses on the results and, in particular, on the outcomes of the policies it supports. It leaves the partner country to take full ownership of its policy process and full responsibility for the accountability of its results, while supporting it with discussions, advice, funding, capacity development, and results monitoring. Because of this approach, the EU cannot claim to have directly delivered any of the achieved outcomes but, since it has supported governments in reaching these results, it can claim that it has (or has not) contributed to the achievement of these results. 

The OECD–DAC methodology is particularly well suited to evaluating EU budget support because it unravels and assesses the paths through which budget support inputs may have contributed to the improvement of public policies and institutions and also the extent to which these improved public policies and spending actions have caused changes in social and economic development. 

In many cases, the EU is not the only development partner to have provided support for government policies. When evaluating budget support, the combined effects of all budget support operations in a given period of time in a country are considered. Given the influence of external factors, budget support evaluation cannot rely on a causality analysis and needs to differentiate between: the budget support’s direct outputs (which can be expected to be produced directly by the budget support’s inputs, e.g., funds, policy dialogue, technical assistance and performance measurement); and its induced outputs (which are situated at the level of public policy, institutional and spending changes, and which result from budget support direct outputs influencing and interacting with government processes). 

To accommodate this complexity, the OECD–DAC approach to budget support evaluation is undertaken in three steps (Figure 3.5)



Source: Adapted from https://www.oecd.org/dac/evaluation/evaluatingbudgetsupport.htm.

Each step follows a very distinct logic: 

  • Step 1. Identifies the combined effects of all budget support provided to the country on aid, policy and institutional processes. The causal relationships between the budget support provided and changes in public policies, institutions, services delivery, and spending are analysed, recognizing that these changes are determined by the government and its policies beyond the budget support package. This step analyses the contribution of budget support inputs to outputs and induced outputs. 
  • Step 2. Identifies changes observed in outcomes and impacts as regards social and economic development which were targeted by the government policies supported (e.g, use of public services, business confidence and other sector outcomes) and analyses the factors determining these changes. These determining factors include public policy actions and also factors outside the government’s control (e.g., private sector and civil society initiatives, other aid programs, and external factors). Step 2 links changes observed at outcome and impact levels to their explanatory factors. It usually involves an econometric regression analysis of change in two or more sectors supported by budget support. 
  • Step 3. Combines the results of step 1 and step 2. The analysis teases out the extent to which budget support, through its contribution to government policies and spending actions, may have contributed to the outcomes and impacts identified. 

This evaluation framework assumes that there are two main driving forces which generate most of the changes in induced outputs and outcomes: 

  • the flow-of-funds effects resulting from the provision of the budget funds; and 
  • the policy and institutional effects resulting from the interplay of budget support funding, policy dialogue, capacity building and disbursement conditions (performance indicators) with domestic processes of policy making, budget formulation and budget execution. 

Both streams of effects can be traced up to the induced output level, while also recognizing that other factors are at play. The contribution analysis of step 1, when confronted with the results of the attribution analysis undertaken in step 2, allows the evaluator to assess the contribution of budget support to the successes and/or failures of the government policies and strategies, in relation to the outcomes and impact that the budget support programs intended to promote. This last step 3 analysis is a qualitative contribution analysis.