Table of contents

1.Introduction

1.1    Purpose and scope of the meta-evaluation    

2.What was the expected outcome of the intervention?

2.1    Description of the intervention and its objectives    

2.2    Point(s) of comparison    

3.How has the situation evolved over the evaluation period?

4.Evaluation findings (analytical part)

4.1    To what extent was the intervention successful and why?    

4.2    How did the EU intervention make a difference?    

4.3    Is the intervention still relevant?    

4.4    Is the current framework for evaluating interventions appropriate?    

5.What are the conclusions and lessons learned?

5.1    Conclusions    

5.2    Lessons learned    

Annex I: Procedural Information

Annex II. Methodology and Analytical models used

Annex III. Evaluation matrix and, where relevant, Details on answers to the evaluation questions (by criterion)

Annex IV. Overview of benefits and costs and, where relevant, table on simplification and burden reduction.

Annex V. Stakeholders consultation - Synopsis report

Bibliography

Glossary

1.1    Purpose and scope of the meta-evaluation

Macro-financial assistance (MFA) is a form of EU financial aid for partner countries experiencing (or threatened by) a balance of payments crisis, helping to restore their external stability and to return their economies to a sustainable path. Taking the form of medium- or long-term loans or grants (or an appropriate combination of the two), and normally complements financing provided by the international community in the context of a financial arrangement with the International Monetary Fund (IMF). Support can be provided only if beneficiaries respect effective democratic mechanisms. Disbursements are conditional on the implementation of policy reforms agreed between the Commission on the EU’s behalf and the beneficiary country and laid down in a memorandum of understanding (MoU) (European Commission, 2021).

Since its inception 30 years ago, MFA has been increasingly used by the EU in its dealings with countries outside the EU and is an important part of its contribution to the Global Financial Safety Net under G20 rules. Initially MFA was extended mostly to countries of central and eastern Europe (some of which have since joined the EU), but it has progressively been extended to other countries in the former Soviet Union and the Mediterranean. MFA volumes increased substantially following the global economic and financial crisis of 2008-2009, which profoundly affected the emerging economies of EU neighbourhood countries. Throughout 2014-2020, MFA has been instrumental in supporting partner countries in the eastern and southern neighbourhood facing conflict and challenging political transitions. In 2020-2022, EUR 3 billion in MFA loans was made available to help 10 partner countries limit the economic fallout of the COVID-19 pandemic. More recently, in 2022, with the escalation of the conflict and the Russian war of aggression against Ukraine, Ukraine requested new MFA support. It received EUR 1.2 billion in emergency MFA (disbursed in March 2022) and, at the time of writing, EUR 6 billion in exceptional MFA (EUR 1 billion disbursed in August and EUR 5 billion disbursed from October to December 2022).

The geopolitical, security-related and economic challenges beyond the EU’s borders require a strong and determined action. To rise to new (and old) challenges, the current Commission has identified the need to strengthen the role of Europe in the world (Von der Leyen, 2019). To do this, the EU must act quickly and efficiently to support key partner countries and territories facing external financing difficulties. MFA is one way of doing this.

The COVID-19 pandemic illustrates this perfectly. The pandemic challenges partner countries and their financial capacity like never before. Against this backdrop, the EU granted a total of EUR 3 billion in MFA to 10 partner countries in spring 2020 (Decision (EU) 2020/701 1 ). This decision and the speed it was implemented with showed that MFA continues to be used extensively by the EU in its dealings with countries outside the EU, in line with its geopolitical ambitions.

MFA has also demonstrated its worth as a flexible crisis instrument enabling the EU to swiftly and substantially support Ukraine following Russia’s unprovoked and unjustified invasion of the country at the beginning of 2022. Already since 2014, Ukraine has received an unprecedented amount of MFA. In 2014-2019, a total of EUR 4.4 billion was made available to Ukraine in four MFA operations. This is the most EU MFA ever granted to a non-EU country in such a short period of time. Following Russia’s unprovoked and unjustified invasion of Ukraine, an additional EUR 7.2 billion of MFA loans were made available at record speed. This is testament to the EU’s unwavering support for Ukraine.

Unlike other EU external action instruments, MFA operations are adopted and implemented directly on the basis of the treaties, without a specific legal framework. Since the Treaty on the European Union (TEU) and the Treaty on the Functioning of the European Union (TFEU) entered into force on 1 December 2009, Article 212 TFEU (and for developing countries Article 209 TFEU) are the legal basis for MFA and govern it.. The European Parliament and the Council authorise each MFA operation on the basis of a dedicated Commission proposal, in accordance with the ordinary legislative procedure (OLP, as set out in Article 294 TFEU). In urgent cases, MFA operations can be authorised by the Council alone, in a non-legislative procedure, always on the basis of a Commission proposal, pursuant to Article 213 TFEU.

For a crisis instrument, the decision-making process for MFA is considered rather lengthy. The first meta-evaluation of past MFA operations, undertaken in 2009 2 , identified delays in adopting MFA decisions as a major shortcoming of the instrument. To remedy this, the Commission put forward a proposal for a Framework Regulation for MFA in July 2011, aimed at expediting decision-making and aligning the management of the MFA instrument with that of other external instruments (European Commission, 2011). However, the proposed approach was not acceptable to the co-legislators, which modified the proposal to the extent that it lost its “raison d’être”, and the Commission ultimately withdrew the proposal; lawfully, as the Court of Justice later held (ECJ, 2015). As a result, there is no framework regulation for MFA policy and legislative decisions on individual MFA operations continue to be adopted by the EP and the Council on a case-by-case basis in the OLP. Recent experience with the COVID-19 MFA package and Ukraine have shown that the current MFA set-up lends itself to swift adoption, if all institutions agree to make full use of the available procedural flexibilities in particularly urgent cases.

In line with the applicable legal and political framework 3 , ex-post evaluations of each individual MFA operation are undertaken. This means that a steady stream of evaluations, not necessarily in sync with the MFF cycle, is a distinct feature of MFA. By 2020, 15 such evaluations had been completed since the first meta-evaluation of MFA operations in 2009. In 2020/21, the Commission’s Directorate-General for Economic and Financial Affairs (DG ECFIN) carried out a new meta-evaluation covering 15 MFA evaluations completed in the last 10 years. The exercise, coinciding with the MFA instrument’s thirtieth anniversary, took stock of previous evaluation results and aims to support organisational learning both in terms of the design and implementation of the MFA instrument and the methodology used to evaluate individual operations. The meta-evaluation will also support transparency and accountability and help lay the groundwork for any possible improvements in the instrument’s design and/or implementation that might be necessary in order to even better support the need for a stronger ‘Global Europe’ in line with President von der Leyen’s vision.

An external evaluation study has been undertaken to help inform the meta-evaluation (DG ECFIN, 2020) 4 . The study began in December 2020 and ended in October 2021. Its outcome is supplemented by input from a high-level seminar on the experience of 30 years of MFA that took place in November 2021 to flesh out the evidence base for the meta-evaluation.

Against this backdrop, and on the basis of the 15 evaluations mentioned above, the meta-evaluation:

1.assesses the principles and characteristics of the MFA instrument in line with the Better Regulation criteria of relevance, effectiveness, efficiency, coherence, value added, and taking into account the instrument’s ability to respond to EU external action priorities, as well as the day-to-day management of MFA interventions; 

2.assesses the overarching methodological framework applied across the evaluations and identifies areas where there is room for improvement.

The meta-evaluation’s methodological approach is threefold:

1.participatory and inclusive data collection and analysis in a (key) stakeholder consultation programme (semi-structured interviews and a public consultation), a validation workshop preceded by a roundtable with the Commission staff in charge of MFA, and a high-level seminar;

2.a mix of qualitative and quantitative input, with qualitative input consisting mainly of document analysis (chiefly the evaluation reports of the 15 operations mentioned above) and stakeholder input, and quantitative data supporting the qualitative information as much as possible; 

3.triangulation: the information and data collected from a range of different sources collectively provide answers to DG ECFIN’s evaluation questions.

A number of limitations were experienced during the completion of the study and consequently the preparation of the meta-evaluation, including data availability and access to stakeholders (Valdani, Vacari e Associati & Centre for Strategy and Evaluation Services, Oxford Analytics, 2021). Such limitations have undoubtedly had a bearing on the quality of the exercise, but do not affect the robustness or reliability of the meta-evaluation’s conclusions, which serve as a useful basis for reflections on the instrument. The limitations and mitigating factors are elaborated on in Annex II.

2.1Description of the intervention and its objectives

Since 1990, the EU (and its predecessors 5 ) has been providing MFA to partner countries, primarily in its neighbourhood. Although the EU has been using the instrument relatively regularly, it continues to be meant to be an ad hoc financing tool in the EU’s external policy toolkit, triggered by sudden economic and financial developments in recipient countries. This means that instead of having a specific legal basis, operations have been based on the EU’s ‘flexibility clause’, requiring a unanimous decision in the Council without a formal say for the European Parliament (European Commission, 2021). In order to nonetheless provide a framework for those ad hoc decisions, the Council formulated a set of criteria to determine the basic features of MFA operations at an informal meeting in Genval, Belgium (the Genval criteria) in 1993. It revised these criteria in 1996 and 2002 (European Commission, 2011).

By a Joint Declaration in 2013 6 , the European Parliament and the Council has provided guidance on the MFA instrument. While not legally binding, it takes stock of the main features of MFA. It was drafted in the aftermath of the entry into force of the Lisbon Treaty in 2009, which gave the European Parliament full co-legislator powers on MFA. The Declaration, based on the Genval criteria, albeit clarified and adapted reflects the co-legislators’ shared understanding. In line with these criteria, MFA is an exceptional financing instrument of untied and undesignated balance-of-payments support for eligible third countries and territories. Its objectives are:

1.to restore a sustainable external finance situation for eligible countries and territories facing external financing difficulties;

2.to underpin the implementation of a policy programme that contains far-reaching adjustment and structural reform measures designed to improve countries’ and territories’ balance of payment positions, in particular over the programme period;

3.to strengthen the implementation of relevant EU programmes and agreements with the EU.

The combination of the objectives mentioned above makes MFA a singular instrument.

·On the one hand, it is a traditional instrument for addressing temporary external financing difficulties and crises. In that sense, it mirrors and is closely linked to IMF interventions with the same purpose, albeit intended as a complementary measure to create financial stability.

·On the other hand, unlike the IMF, the EU is an explicitly political actor, and in line with Article 205 TFEU and Article 21 TEU, MFA is ultimately a political action. As such, it is part and parcel of a framework of tools, instruments and policies designed to also achieve the EU’s broader objectives and promote its values.

It therefore follows that MFA combines parts of a balance-of-payments financing tool (such as the IMF’s) with traditional instruments of external financial cooperation, whilst being distinct from them both. In contrast to IMF support, MFA explicitly requires the fulfilment of political preconditions. In contrast to other EU external financial cooperation instruments, MFA operations continue to be authorised by means of the OLP, thereby remaining outside of the scope of the Commission’s implementing powers under the EU’s Neighbourhood, Development and International Cooperation Instrument – Global Europe (NDICI) 7 . 

MFA typically comes into play when an unsustainable mix of economic policies or an economic shock have given rise to external financing imbalances that result (or risk resulting) in an undesired currency depreciation. MFA operations therefore follow a traditional pattern of seeking to correct such an unsustainable policy mix (or other causes of a shock) whilst easing economic adjustment by means of financial assistance. It targets the policy areas that are relevant to macroeconomic stability in general and aims to improve them, in line with the following diagram.

Figure 1: MFA intervention logic

·The Commission (on behalf of the EU) provides financial assistance, having agreed with the beneficiary on a set of (structural) policy conditions underpinning the macroeconomic adjustment taking place. Technical assistance may also be agreed (Inputs).

·The beneficiary begins the process of implementing the reforms agreed on. In the 15 operations evaluated, reforms tended to focus on public sector governance, the fight against corruption and money laundering, the energy sector, the business climate, the financial sector, and the social safety net and labour market.

·During and after an MFA operation, the policy reforms and financial assistance should help stabilise the beneficiary’s macroeconomic situation, including with regard to their balance-of-payment position and (as necessary) public finances. Ideally, these effects should last for a long period, underpinned by structural improvements in the efficiency and effectiveness of the development and implementation of economically relevant policies.

However, the specific intended effects depend on the operation and the country/economy concerned. For reasons already mentioned, the intervention logic and the operation’s operational objectives need to be placed in the context of the MFA’s ends and purposes as a way for the EU to carry out external action. These include, among others, safeguarding EU values and fundamental interests, consolidating and supporting democracy and the rule of law, encouraging the integration of all countries into the world economy, and promoting an international system based on closer multilateral cooperation and good global governance (Article 21 TEU). These objectives (except in individual cases) are not part of the operational objectives of MFA operations and therefore of the intervention logic. Rather, they are preconditions for a country’s benefiting from MFA support.

It must also be stressed that MFA is usually part of a stabilisation operation incorporating, alongside the EU, at least the IMF and most often also the World Bank Group and in some cases other bilateral and multilateral international partners. In the case of the IMF, the link is particularly close given the need for parallel IMF support (Principle 6 of the Joint Declaration) and the requirement for the amount of assistance to be based on a country’s needs (Principle 5). At the same time, given the IMF’s own ceilings (as a multiple of a country’s share in its capital, the quota (IMF, 1944)), its interventions can equally be contingent on an MFA operation as it is necessary to ensure that the stabilisation operation is fully financed 8 .

This link also results in a certain division of labour: Commission services rely on the IMF its core areas of expertise, most notably in agreeing with the beneficiary country the macroeconomic stance required to underpin the adjustment to external financing constraints (the financial programming exercise). The IMF’s own policies require it to comprehensively cover all reform requirements (including structural ones) necessary for the standalone success of its intervention (IMF, 2005). Since MFA operations are conditional on a continuously satisfactory track record in implementing IMF-supported policy programmes, IMF conditions (unless waived) are in practice incorporated into the MFA requirements. For its part, the Commission provides the IMF with its country-specific expertise on structural reform priorities, as well as a set of draft reforms designed to support the implementation of the IMF’s conditions.

Similarly, MFA usually comes into play alongside other EU instruments. Following on from the European Neighbourhood Policy Instrument (ENPI) and the European Neighbourhood Instrument (ENI – 2014-2020), the Neighbourhood, Development and International Cooperation Instrument (NDICI) is to give neighbourhood partner countries support during the 2021-2027 period; in accession countries, the relevant instrument is the Instrument for Pre-Accession Assistance (IPA). The EU is also active regionally through EIB lending.

2.2Point(s) of comparison

As Section 1 explains, MFA operations are systematically evaluated after they end, notably to assess (i) the impact of MFA on beneficiary countries’ economies and in particular on the sustainability of their external position and (ii) the operations’ added value. These evaluations are also supposed to provide the following insights into the EU’s financial assistance:

·to what extent the ex ante considerations determining the design and terms of each operation have been appropriate, taking due account of the economic, political and institutional context;

·to what extent the outcome of the operations has achieved the objectives set in the legislative decisions.

The ex post evaluations consist of an ex post assessment of the MFA operations in question, focusing on the macroeconomic and structural adjustment measures that the Commission, on the EU’s behalf, agreed with each beneficiary country in an MoU.

The MoU’s policy conditions are based on the economic stabilisation and reform programme endorsed by the authorities of the beneficiary country and are consistent with its agreements with the IMF. In line with the division of labour mentioned above that has evolved over time between the Commission and the IMF, the MoU does not include specific macroeconomic indicators and targets. This is because the Commission relies for these on the IMF’s macro-framework agreed with the country receiving MFA. Instead, the MoUs specify that disbursement of MFA is conditional on a satisfactory track record of implementation of the policy programme agreed between the country and the IMF.

Given this is the case, a precise assessment of its standalone impact is difficult to make. As explained above, MFA is an external financial cooperation instrument (serving both economic and external policy objectives), implemented alongside an IMF intervention and the EU’s regular financial operations, typically as part of a stabilisation operation involving other international institutions. This means that the operations’ effects on macroeconomic variables over time cannot be attributed solely to MFA. In addition, as MFA is a short-term crisis-related instrument spanning a maximum of 2 to 3 years, it is not possible to quantify its impact in terms of indicators/milestones beyond the timeframe of the MFA operations themselves. This makes it difficult to capture the impact of some of the medium-term reforms.

Despite these constraints, past evaluations have traditionally looked at a certain number of macroeconomic indicators. These included:

·indicators for gauging external stability: developments of international reserves and current account balances; 

·indicators for gauging internal stability: inflation levels, GDP growth, government structural balances and debt levels.

Besides the constraints outlined above, some of these indicators (such as GDP growth, government debt levels, current account balances) also raise conceptual questions. Assessing significant impacts on GDP growth and government debt is therefore ambitious even for a combined MFA/IMF operation, while government structural balances are beyond the scope of MFA. Finally, current account balances also behave peculiarly, tending to exhibit some automatic self-correction over the cycle. This makes it difficult to assess and isolate the impact of MFA operations on such indicators as their evolution depends on multiple factors. Further analysis may therefore be warranted to fine-tune the Guidelines for the Ex Post Evaluation of Macro-financial Assistance Operations that will be included in the new ECFIN framework contracts for the provision of evaluation and evaluation-related services.

Since the 2009 meta-evaluation, the main features of the MFA instrument have remained unchanged, with some limited changes introduced following the adoption of the Lisbon Treaty. This section provides an overview of the operations evaluated since 2010, the applicable decision-making procedures, and the amounts and financial terms of the assistance. It also describes the budgetary arrangements governing the instrument as well as the conditions and compliance with them.  

MFA operations 2010-2020

MFA amounts made available to non-EU countries increased from EUR 1.2 billion in 2000-2008 to EUR 6.3 billion in 2009-2019. Political and economic upheaval in Arab Mediterranean partner countries has led to greater demand for MFA since 2012, with operations implemented in Jordan and Tunisia. From 2014, the conflict in eastern Ukraine and the marked deterioration of the country’s economic and financial situation triggered an unprecedented level of MFA support. In 2014-2019, a total of EUR 4.4 billion was made available to Ukraine under four MFA operations, the largest amount of financial assistance ever granted to a non-EU country in such a short period of time. All but two of these operations have already been evaluated and are included in the meta-evaluation.

During the 2010-2020 period, the bulk of MFA funds (71%) was used to support eastern neighbourhood countries, followed by southern neighbourhood countries (24%), (pre)-accession countries and other countries (5%). The bulk of MFA funds was provided in loans (95%), the rest in grants (5%).

Of these 17 MFA operations, in total 15 have been completed and evaluated since the last meta-evaluation in 2009/10.

Table 1: MFA operations evaluated over the 2010-2020 period

Many operations were also authorised in the wake of the COVID-19 pandemic, and more recently in the wake of Russia’s unprovoked and unjustified invasion of Ukraine. In response to the pandemic, the Commission adopted a proposal for a EUR 3 billion MFA package for 10 enlargement and neighbourhood partner countries in spring 2020. This was endorsed by the co-legislators in May 2020. At the time of its adoption, it represented the largest volume of MFA the EU ever made available to partner countries at a single time. In response to the 2022 invasion of Ukraine, an additional EUR 7.2 billion of MFA loans was made available to Ukraine at record speed. The COVID-19 package of operations was only fully completed by mid-2022 and support for Ukraine is ongoing. Both sets of operations therefore remain to be evaluated. They provide valuable insights into, for instance, the decision-making procedure used to initiate the package (discussed below).

As explained above, the Genval criteria applied to operations before 2013 and the Joint Declaration to operations after that, with the following implications.

·The change of regime increased the possible amount of assistance. The Genval criteria set a ceiling for the amount of assistance at 60% of the residual financing gap for (pre-) accession and association countries and 33% of that gap for neighbourhood countries. On the insistence of the European Parliament, the 2013 Joint Declaration removed these ceilings whilst introducing the principles of fair burden-sharing and EU value added. However, the Commission has continued to apply the 60%/33% ceilings as a rule of thumb, since they were considered to adequately reflect the Declaration’s principle of fair burden-sharing.

·The 2013 Joint Declaration showed a clear preference for assistance in the form of loans (instead of grants), but the Genval criteria were more neutral in this regard. As Table 3  shows, this has resulted in a marked decrease in the share of grants over time.

Procedures and decision-making

In the absence of a framework regulation that would attribute powers to the Commission, operations have to be authorised by means of decisions – acts of secondary EU law, on the basis of Article 212 TFEU. Until 31 November 2009, the Council adopted such acts alone 9 . Since then, the ordinary legislative procedure usually applies, with equal involvement of the European Parliament and the Council. The procedure follows the rules set out in Article 294 TFEU, as follows. The Commission triggers the procedure when it prepares and submits a proposal to the European Parliament and the Council, who reach their respective positions in ensuing internal deliberations before entering interinstitutional negotiations. In theory, the procedure could include a first and second reading in the Parliament and the Council, a conciliation procedure and, possibly, a third reading. However, in practice co-legislators usually reach early agreements during the first reading. When the situation in a third country requires urgent financial assistance from the EU, Article 213 TFUE allows the Council, acting alone in a non-legislative procedure, to adopt the necessary decisions on a proposal from the Commission.

Since the adoption of the Joint Declaration, the Commission/ EEAS and its services must complete a certain number of steps to prepare the operation. On receiving a potential beneficiary’s request, they carry out an ex ante evaluation to assess the amount of financing needed and whether or not the country in question fulfils the eligibility conditions for assistance, including the political preconditions. The Commission then submits a proposal to the co-legislators. After the operation is authorised by the co-legislators, an MoU is negotiated and concluded between the beneficiary country and the European Commission, on behalf of the EU, setting out the reform conditions for the operation. The financial terms are set out in a loan agreement and, if the operation also comprises a grant, a grant agreement. The MoU is adopted as an implementing act under comitology, in line with the Joint Declaration.

Figure 2: Process for granting and providing MFA

The time lag between request and first disbursement varies a lot. In the 15 operations evaluated, the average length between the recipients’ request for assistance and the first disbursement was roughly 28 months and the median 26 months. However, since these figures include operations authorised before the entry into force of the Lisbon Treaty and during the protracted negotiations on the Commission proposal for an MFA Framework Regulation, putting some operations on hold, they do not give an accurate picture of the current state of play.

This is why it is necessary to adjust the sample of operations in order to get a meaningful picture of the lags in the current decision-making procedures. Only 5 of the 15 operations evaluated took place within the current steady state (after the Joint Declaration entered into force in 2013). Excluding the special case of Ukraine (with three subsequent and sometimes overlapping operations), this would leave only the Tunisia and Jordan operations. It would therefore make more sense to describe the state of play on the basis of all operations since 2013.

On the basis of such an adjusted sample, the following observations can be made (see Table 2 ).

·Whilst remaining somewhat variable, the average/median overall period between a request and the first disbursement was roughly 17 months. In some cases there was an improvement over time, as both the EU institutions and beneficiary countries became more familiar with the procedures. On the basis of more recent cases since 2020, it can be seen that the standard procedures make it possible for the period between a request and the first disbursement to be less than 10 months.

·Thanks to the use of urgency procedures in the European Parliament, the Commission proposal for the COVID-19 package was adopted in a record 1 month. The one-time use of Article 213 TFEU for urgent cases (Ukraine 2014) made it possible for only the Council to decide, also expediting the process.

·Entry into force of the MoU’s (Step III) sometimes requiring ratification by the beneficiary’s parliament was the main drag in the process, but the OLP (Step II) also sometimes added to the overall lag between a request and disbursement.

Table 2: Time and steps (see Figure 3 ) from MFA request to (first) disbursement

* Jordan 20-2 was a top-up of Jordan 20-1, part of the COVID-19 package and authorised in a multicountry (omnibus) decision.

Amounts and financial terms

The 15 operations evaluated enabled around EUR 4 billion of support to be made available on very advantageous terms. In total, roughly EUR 3.3 billion were disbursed to the beneficiaries 10 . MFA most often takes the form of concessional medium-/long-term loans, the funds for which the Commission borrows on capital markets and lends on to the beneficiary country. MFA loans therefore had a typical maturity of around 15 years and were provided at the EU’s own borrowing costs over the same period – the costs of an AAA-rated debtor borrowing in their home currency.

In some cases, MFA took the form of grants financed by the EU budget, or a combination of loans and grants, with the share of grants changing over time. Before the adoption of the 2013 Joint Declaration, MFA disbursements tended to have a significant share of grants. However, reflecting the Joint Declaration’s stated preference for loans and the limited budget available, since 2013 MFA grants have been provided only for eligible countries with very high vulnerabilities and comparably low income per head. The Commission is currently looking into whether or not the methodology for deciding on the provision of grants needs to be updated to maximise their usefulness while respecting the limited budget available in the 2021-2027 MFF.

Table 3: Overview of financial terms of support provided

Budgetary arrangements

As a crisis instrument and unlike other EU external financing instruments, MFA does not provide financial support on a regular, programmable basis. When provided in the form of loans, its budgetary impact is limited to the provisioning of the External Action Guarantee in the Common Provisioning Fund financed from geographical budget lines under NDICI 11  to cover the EU budget in the event of potential defaults. At a provisioning rate of 9%, this mechanism creates a significant multiplier between budgetary expenses and the headline amounts provided 12 .

To streamline its external financing architecture 13 , the EU changed the budgetary set-up for MFA in 2021. In the 2021-2027 MFF, MFA grants continue to be budgeted separately (under budget line 14 20 03 01 Macro-financial assistance (MFA) – grants). However, the provisioning of the guarantee for MFA loans is now part of the budget allocated to NDICI in the External Action Guarantee, where the Regulation sets certain ceilings for the use of the budget for guarantees in general. Previously, MFA operations were provisioned in the Guarantee Fund for External Action (GFEA), which had a dedicated budget line. The assets constituted in the GFEA have now been transferred to the Common Provisioning Fund.

In recent years, the EU has increased the budgetary means available for the instrument. Faced with increasing demand for MFA support in the context of the 2014-2020 mid-term review of the MFF, the Commission proposed (and co-legislators endorsed) an increase in the volume of lending capacity from EUR 0.5 billion to EUR 2 billion annually for the remainder of the programming period. Similarly, the Commission proposed that the volume of MFA lending remain consistent with these agreed yearly volumes – at EUR 14 billion – in preparation for the 2021-2027 MFF 14 . During the NDICI negotiation process, the overall volume of guaranteed operations was reduced compared to the Commission proposal, with implications for the MFA budget as well.  The adopted NDICI Regulation does not feature a specific envelope for MFA loans under the External Action Guarantee. However, as result of the overall ceiling for the external action guarantee established in the adopted NDICI Regulation, the maximum volume of MFA loans was fixed at EUR 11.6 billion. In addition, outside of the NDICI Regulation, a budget of close to EUR 350 million is envisaged for MFA grants. The budget for provisioning the External Action Guarantee is EUR 10 billion, covering European Fund for Sustainable Development+ operations, including MFA operations and Euratom loans granted during the period 2021-27.  

MFA conditions and compliance

The 15 operations that form the basis of the meta-evaluation contained a total of 185 conditions, with compliance varying per operation as follows.

Table 4: MFA conditions compliance rate by operation

* Given the non-disbursement of the second tranche, compliance was not officially recognised.

The large majority of conditions covered budgetary reforms, including public financial management reforms. Almost a quarter of all conditions related to this policy area, followed by internal and external auditing (14%), social and healthcare policies (11%) and public procurement (10%).

Chart 1: Distribution of MFA conditions across policy areas

*Covers the financial management of the government, for example fiscal impact analysis, and everything related to the government budget. It also includes monetary policy and the organisation of the central bank, which is actually a separate topic but is taken into account here because there are too few reforms in this monetary policy area alone.

Evaluations and methodology applied

The ex post evaluations of MFA have drawn on the 2010 and 2015 Guidelines for the Ex Post Evaluation of Macro-financial assistance Operations drawn up by DG ECFIN. The guidelines take account of the very specific nature of MFA operations and the resulting challenges for assessing them. The 2015 Guidelines ushered in a number of changes in the way the ‘core questions’ for MFA ex post evaluations were formulated. The 2010 Guidelines formulated the core questions in terms of how effective MFA had been in relation to macroeconomic stabilisation, supporting structural reform, and returning external debt to a sustainable path, as well as examining indirect/unexpected effects and how the MFA’s design affected its effectiveness or efficiency. The 2015 Guidelines adopted a different approach, focusing on the broader Better Regulation themes of relevance, efficiency, effectiveness, coherence and EU added value. Of the 15 evaluations that form the basis of this meta-evaluation, 8 were done using the 2010 Guidelines and 7 done using the 2015 Guidelines.

Whilst all the ex post evaluations examined the contribution of MFA to short-term macroeconomic stabilisation, the impact on structural reforms, and various design and implementation issues, the precise mix of methodologies to do so varied from one evaluation to the next. In particular, there was a significant change between before and after 2015 in the evaluations’ suggested approach to counterfactual analysis. Until 2015, an essentially econometric modelling approach was suggested in the ECFIN Guidelines. However, it was judged that the economic models used for MFA did not produce reliable results. Since 2015, counterfactual analysis has therefore been carried out using relevant quantitative and qualitative techniques (Delphi technique and focus groups).

4.1    To what extent was the intervention successful and why? 

In line with the Better Regulations Guidelines, the performance of MFA should be analysed on the basis of its effectiveness, coherence and efficiency. Applied to MFA,

·“effectiveness” describes the instrument’s track record in meeting its stated objectives, i.e. contributing to macroeconomic stabilisation and promoting implementation of important structural reforms;

·“coherence” depicts the extent to which:

oMFA operations support the Union’s own broader objectives in its external policies (directly and indirectly) -> “external coherence”,

oa common framework of rules, policies and practices supports the instrument’s overall good and consistent functioning -> “internal coherence”: and

·“efficiency” relates to the ambition to deliver in the aforementioned dimensions at minimal costs to the overall resources available to the Union and the beneficiary country.

Effectiveness in stabilising macroeconomic developments

MFA has generally been effective in stabilising beneficiaries’ (external) macroeconomic environment, especially in the short and medium term. Besides the specific/individual assessments in the 15 evaluations previously mentioned, hard data back this statement up in the following areas.

·Despite a challenging external environment at the start of the respective MFA operations, market access could be maintained or eventually regained in all cases. This includes one beneficiary’s (Ukraine’s) restructuring its debt in good faith negotiations with private bondholders in 2015. With the exception of one strategic default (by Ukraine on a Eurobond of EUR 3 billion held by the Russian Federation that the Ukrainian authorities consider illegitimate), beneficiaries did not default.

·Reserve adequacy and current accounts mostly improved during MFA operations, and inflation subsided.

Table 5: Selected macroeconomic indicators before/during/after MFA operations

Note. Before – average of the 3 years before MFA. MFA – average during MFA. After – average of the 3 years after MFA. 
Source: IMF (April 2021), Worldbank

These findings are more nuanced for timeframes longer than the medium term, based on the external study. Some improvements in current account balances are visible during that time, but gains in reserve adequacy could usually not be extended further and current accounts remained in deficit (a few years after the completion of MFA operations). Participants at the validation workshop tended to confirm this interpretation of the data.

However, these conclusions, especially insofar as they concern current account balances, need to be assessed carefully. There is no consensus among professional economists on the right current account position for a given country. Clearly, not all countries can reduce their current account deficits at the same time or consistently run current account surpluses. Moreover, for potential beneficiaries of MFA, universally in their economic transition, standard economic theory would suggest from a normative perspective that surpluses are undesirable, whilst the size of sustainable current account deficits depends on a variety of country- and time-specific factors. 15  

In any event, it must again be stressed that the beneficial short-to-medium-term effects outlined above cannot be attributed to MFA alone, given its relatively limited size and given its operations are undertaken in conjunction with IMF operations. In this vein, several stakeholders (including stakeholders at the validation workshop) underlined that concomitance with IMF operations has been essential in producing the results mentioned before. Interviewees from EU institutions have also pointed out that the complementarity of expertise and policy conditions has made the overall stabilisation operation more impactful. Interviews with stakeholders also indicate that cooperation between the EU, the IMF and the World Bank has increased political leverage for the EU to push for reforms.

Table 6: Achievement of macroeconomic stabilisation according to the evaluation reports

At the same time, it appears that concomitance with IMF support can at times result in implementation difficulties, such as disbursement delays. The external study says that interviewees and some OPC respondents expressed this concern. Some even advocated a review of the Joint Declaration’s Principle (6) on this requirement. Some stakeholders also argued that the scope of the EU’s leverage in an economic adjustment programme predefined by the IMF was not clear. In this context, the study reports that some stakeholders in IFIs wondered whether the EU could be more influential, also considering the amount of financial capacity there is in MFA. Others expressed concern that the close link to IMF support could be detrimental to the perception of the EU as an autonomous global actor 16 . 

Effectiveness in incentivising reforms

In contrast to the obligatory concomitance with IMF support, the link with political preconditions 17 appears to be of less concern. In this regard, the external study focuses on one case (Moldova) in which their implementation caused some difficulties as the MFA operation was interrupted due to non-compliance. This in turn resulted in a desynchronisation with the continuing IMF operation. But otherwise, there is little reference in ex post evaluation reports to a (negative) impact of the political precondition on the outcomes, effectiveness and efficiency of MFA operations. OPC respondents also expressed little criticism in this regard, possibly because MFA is such an integral part of the EU’s external policy framework. For instance, the French senate confirmed this view in a 2017 report recommending a more prominent and explicit role for the political preconditions by including them, as well as the consequences of non-compliance, in the MoU (Marc, 2017).

That said, the external study sees some room for improvement with regard to the way the political preconditions are assessed in practice. It seems that ensuring compliance is at times very challenging, for two main reasons.

1.Countries that benefit from MFA have different political systems. As a result, political preconditions might be more easily and swiftly fulfilled in some countries than in others.

2.As several stakeholders in EU institutions and IFIs acknowledged, the political preconditions are rarely specified. This makes it hard to assess their fulfilment, also because different Commission services/the EEAS may differ in their assessment.

Like the link with political preconditions, the requirement to implement structural reforms enjoys relatively strong support among stakeholders. Furthermore, previous evaluations found that 81% of the requirements had been fully met, even if compliance rates differed between operations and targeted sectors. Despite this broadly strong track record, key stakeholders noted the prominent example of Ukraine (15), where excessive ambition led to a proliferation of requirements competing with each other in terms of implementation resources and political capital. The external study also identifies the 2014-2017 operation in Tunisia as having suffered from the same problem.

Table 7: MFA conditions compliance rate by thematic groups

Given the high compliance rates, the external study finds little conclusive evidence of backtracking. What evidence is there is mainly in the area of the fight against corruption and in cases in which there had already been problems during implementation.

These results are all the more noteworthy given previous analyses in this area. In 2007, the Independent Evaluation Office of the IMF found a compliance rate of just 54% in the IMF’s financial arrangements, with higher compliance by traditional interlocutors (finance ministries and central banks) in its core area of expertise (IEO, 2007). In comparison with this benchmark, MFA operations appear to be more successful in achieving implementation of the requested reforms.

Several features support the effectiveness of MFA in incentivising meaningful reform based on the external study

·The instrument’s geographical focus played a supportive role. Interviewees suggested that the EU’s strong political ties with countries in its direct vicinity help ensure that the MFA conditions are met and that the instrument achieves its goals. Conversely, interviewees stated that fewer political and economic connections result in significantly lower leverage and that this potentially reduces the impact of the MFA. A majority of the OPC respondents (90%) likewise argued that Principle 2 of the Joint Declaration in its current formulation suits objectives of the MFA instrument either very well or rather well.There is some evidence that the high share of loans rather than grants can support the implementation of some types of reforms. This is underpinned by the theoretical notion that a beneficiary country has more incentive to implement revenue-raising reforms because they improve its repayment capacity 18 . In this context, some shareholders support the Joint Declaration’s Principle 3 and its emphasis on loans.

·Similarly, there is broad support among interviewees from the EU institutions for the standard availability period of 2.5 years. This is considered sufficient to allow ambitious reform programmes to be implemented and has the benefit of allowing alignment with IMF arrangements (typically extending over 3 years). A shorter availability period could also mitigate the risk of a change of government and hence the loss of political will to implement a given reform programme, even if parliamentary ratification should to some extent help to avert such a scenario.

·The effectiveness of the MFA operations arguably stems at least in part from the political leverage they generate. This contributes to a more comprehensive reform effort that typically reinforces the effectiveness of the individual reforms undertaken.

Conversely, the external study also highlighted some features of the instrument that tend to reduce its effectiveness.

·The long procedure for authorising MFA operations undermines their effectiveness. As indicated in Section 3 above, the typical time lag between a recipient country’s request for assistance and the first disbursement was roughly 17 months, including 5 months for concluding the OLP approval process in the EU and sometimes including ratification by national parliaments. In particular, the time lag until the MoU (and therefore the MFA) enters into force may allow the economic situation to deteriorate further in the absence of full external funding support in cases where such a risk is typically mitigated by the timing of the underlying IMF assistance. Furthermore, the external study points to one case (i.e. Armenia 09), which was in urgent need of financial assistance and chose to seek front-loading from the IMF given MFA’s lengthy procedures. 19 In this context, only half of the OPC respondents consider the procedure for granting MFA (Principle 7) to be very suitable.

·Notwithstanding the relatively high compliance rates, the MoU’s inflexibility appears to have reduced MFA’s effectiveness in times of change. MFA instalments are disbursed in full or not at all – unlike EU budget support, which is disbursed in increments as and when conditions are fulfilled. This is in line with IMF policies but, unlike those policies, MFA practice is that policy conditions are fixed at the operation’s inception and remain unchanged once the national parliament has ratified the text and the MFA has entered into force. The MoU may be amended upon mutual agreement, but the time that would be needed to complete the respective approval processes within the EU and the beneficiary country makes it almost impossible for the amendment to enter into force before the end of the availability period. Furthermore, in order to encourage a comprehensive approach to reform implementation, MFA does not allow for partial disbursements. Flexibility is therefore limited to waivers that exceptionally allow disbursements to take place despite non-fulfilment of one or several conditions, but new conditions cannot be added ‘ad hoc’ in order to reflect changes in the economic environment or a change caused by other drivers to the priority or assessment of necessary policy reforms.

·Assuming all else is equal, (small) volumes also affect effectiveness, as the case of Kosovo shows 20 . The corresponding ex post evaluation report from 2012 stated that the overall contribution of MFA to macro-fiscal stability and structural reforms was limited due to a number of factors such as its volume. A total of EUR 30 million (0.6% of Kosovo’s GDP) was disbursed. Given the relatively small amount disbursed, the ‘real financial difference’ was moderate and so was the impact of the structural conditions linked to the MFA. The reference points for the amounts of the assistance (Principle 5) therefore appear to be somewhat controversial, with 70% of respondents judging them to be only ‘somewhat’ or ‘not very’ suitable.

In any event, ownership of the agreed reforms is of critical importance for the MFA’s effectiveness in promoting important reforms. This is a consistent finding of analyses of the reform-based financial assistance and is also reflected in the external evaluation 21 . Some factors (i.e. the lags in decision-making, and the relevance of reforms) therefore have an impact on the success of MFA operations through their impact on ownership. The external study similarly links cases where reform requirements lacked focus and were overly broad (Ukraine 10, 14) to a loss of ownership by the beneficiary country and therefore its compliance, which in turn reduces the MFA’s effectiveness in promoting reforms.

Additional analysis is required in order to properly judge how the choice of policy conditionality affected the MFA’s effectiveness. The country-specific ex post evaluations as well as stakeholder consultation generally indicate that the policy areas covered by the MFA conditions are relevant. However, the methodology for assessing whether the reform areas chosen were well-targeted, well-sequenced and sufficiently SMART (Specific, Measurable, Attainable, Realistic and Time-bound) to support the MFA’s overall objectives needs to be refined. Such analysis would not be straightforward, however, given the wide scope of the economic and structural MFA objectives on the one hand and the instrument’s contribution to the EU’s broader objectives (as reflected, for example, in the political preconditions) on the other. Further efforts to explore this issue could be useful.

Efficiency:

The external evaluation confirms that MFA is a relatively (cost-)efficient instrument for the EU. As explained above and in line with the provisioning policies of the External Action Guarantee (Article 31, NDICIR) and its predecessor the External Action Guarantee Fund, provisions are set for MFA loans at 9% of the principal amount to be disbursed. Given that no beneficiary has defaulted to date on an MFA loan, the related external guarantee provisioning has been routinely returned to the EU budget.

Some of the principles mentioned above further support this high cost-efficiency. Concomitance with IMF interventions provides some assurance concerning the country’s repayment capacity (European Commission, 2021) because the IMF does not usually provide funding to countries with an unsustainable debt (unless it is taking measures to address this issue). In addition, the synergies arising from relying on the IMF’s work generate important cost-savings on the EU side. The geographical focus similarly creates synergies in terms of expertise and human resources between and within different Commission departments.

MFA is also cost-efficient for the beneficiary countries because its loans are highly concessional. The EU’s high credit rating allows it to borrow money at much lower interest rates than beneficiary countries. The EU passes on these interest rates in its loan agreements, so the beneficiary countries receive a financial benefit that is often substantial. This benefit can be expressed as the difference between the nominal value of a credit (on the basis of market rates) and the present value of the debt service on the date of disbursement (calculated at a discount rate applicable to the currency of the transaction and expressed as a percentage of the nominal).  22  

Table 8: Concessionality of selected operations under the COVID-19 MFA package*

* The COVID-19 MFAs are cited as an example here because they consisted entirely of loans.

The current EU decision-making arrangements can cause some inefficiencies. By reaching agreement on a template for MFA decisions as a follow-up to the 2013 Joint Declaration, the EU’s institutions have revealed relatively little divergence in terms of the substance of MFA operations during the OLP approval negotiations. In the case of MFA, the main purpose of legislative negotiations (to resolve disagreement) was therefore largely absent, but the co-legislators were still required to go through the process required for a standard OLP negotiation. The co-legislators generally needed 5 months to adopt the Commission’s proposal for MFA operations under review. In addition and before MFA enters into force, the MoU has to be formally agreed (see Step III in Table 2 ) – which requires a Commission implementing act subject to Comitology (which is also generally used to decide on the EU’s budget support outside of emergencies). It took roughly 7 months to conclude the negotiations with the beneficiary country and adopt the MoU, but this can take less time if the negotiations are conducted largely in parallel with the OLP process. Besides, recent experience with the COVID-19 MFA package and with MFA support to Ukraine in the context of Russia’s unjustified invasion of the country has shown that MFA’s current set-up can allow very swift adoption if all institutions agree to make full use of the available procedural flexibilities.

Questions can also be raised over the efficiency of the decision-making process in beneficiary countries, which often also ratify the MoUs. This can create a stronger sense of ownership of the reforms to be implemented, but tends to delay the disbursement of the MFA and all but prevents MoUs’ subsequent adaptation (as discussed above). In addition, national ratification is not common practice for IMF programme documents or the conditions attached to the EU’s budget support under ENPI/NDICI or IPA III. 

Coherence:

1.Coherence with other EU instruments

As indicated above, MFA usually intervenes in tandem with other EU instruments, such as currently NDICI and IPA. This results in significant synergies, chiefly with budget support under these instruments. As Table 7 shows, MFA conditions often include reforms related to PFM/budget and auditing. These areas are also critical to the effectiveness of the budget support operations under these instruments, where the MFA operation requires implementation of all conditions before disbursement can take place, thus strengthening the incentives to carry out comprehensive reforms in these areas. MFA conditions therefore tend to be closely dovetailed with budget support conditions so that they can reinforce and complement them as much as possible.

In order to underpin such coherence with other instruments, coordination between Commission services (DG ECFIN, DG NEAR, DG INTPA and others) takes place on an ongoing basis. This reflects the need to address specific issues related to the MFA conditions, insofar as they are under the responsibility of the respective DGs.

The co-existence of intervention through MFA with intervention through other instruments such as ENPI has nevertheless produced some issues at times. For instance, the ex post evaluation of Moldova 07 found that it had resulted in modest additional administrative costs because two steering committees and two reporting mechanisms were run at the same time. It also found that MFA and ENPI policy conditions had overlapped, and that this had resulted in some synergies but also had caused some ‘confusion’. The ex post evaluation does not specify the consequences of this confusion, but the external study carried out for this meta-evaluation argues that the overlap of conditions probably made it harder for both the EU and the beneficiary country to keep track of the reforms.

By contrast, the example of Armenia stands out as an illustration of how a high level of coherence can be achieved and make a real difference. According to the corresponding evaluation 23 , the MoU sought to ensure consistency with the policy lines of the IMF-supported economic adjustment programme and the priorities of the ENPI and the Eastern Partnership. Alongside the ENPI, MFA conditions targeted socio-economic reforms and sustainable development, and supported good economic governance, efficient public spending, trade and investment. Reforms were selected which were likely to be achievable during the MFA operation and which non-governmental national and international stakeholders also identified as priority issues. In fact, those reforms also prepared the country to meet preconditions for the DCFTA negotiations between May 2012 and July 2013.

All in all, MFA’s coherence with other EU instruments appears to be good.

·As the stakeholders confirmed, the objectives, design and implementation of MFA are complementary and consistent with other initiatives providing financial support to beneficiary countries. MFA complements the budget support programmes in the beneficiary countries. The two instruments’ conditions are aligned as far as possible.

·There appears to be some potential for the MFA to provide additional benefit to other EU external policies, chiefly in conjunction with budget support operations.

It should nevertheless be noted that there are limits to the level of coherence that the MFA can deliver without losing its stand-alone rationale and nature. As the Joint Declaration explicitly states, MFA is not an instrument for long-term (‘development’) financing. It has a specific place and role in the EU’s external policy toolkit of focusing on relatively short-term and repayable balance of payment support with macroeconomic significance. Coherence must therefore not be understood as replicating the intervention model of budget support because that might compromise the complementarity of those instruments with each other.

2.Coherence with the interventions of international partners

Another dimension of the MFA’s coherence with other instruments concerns coordination between the Commission and the IMF. Synchronisation with the parallel IMF reviews was sometimes challenging and some conditions overlapped. That said, these coordination difficulties did not undermine the overall success of MFA  24 . Close coordination with the IMF and occasional interlinking with interventions by the World Bank enabled the Commission to draw on the expertise of these organisations. Coordination has therefore been reported by the stakeholders as ‘good’ between MFA, other EU support, the IMF and other sponsors. Complementarity between MFA and IMF support resulted in a joint impact that exceeded what both initiatives could have achieved alone.

The stakeholder consultation shows that external coherence with IFI interventions is generally seen as high and to be supporting the operation of the instrument. Stakeholders mainly referred to complementarity and the absence of overlap between the conditions of the different actors once their different interests, mandates and approaches had been taken into account.

Regarding internal coherence, stakeholder feedback suggests that MFA has been consistently and clearly developed, so that its implementation follows the same set of principles and rules in all beneficiary countries. This internal coherence is founded on the Joint Declaration, which provides a common framework despite the lack of a dedicated legal act.

However, when assessing Better Regulation criteria, the external study finds some variation in the outcome of individual operations. This variation may reflect specific country and/or time factors, but could also indicate that further standardisation work is needed.

When effectiveness, coherence and efficiency are all taken into account, the overall assessment of the MFA instrument’s success over the last decade is positive. The analysed evidence suggests that MFA helped stabilise beneficiaries’ economies and promoted the implementation of important policy reforms. The instrument’s external and internal coherence was satisfactory overall, even if external coherence with other EU external policy instruments could be further enhanced. Finally, MFA is very cost-efficient, especially when support is provided in the form of loans, although swifter upstream decision-making procedures could produce additional resource savings.

4.2    How did the EU intervention make a difference?

From the Member States’ perspective, EU MFA operations provide financial and political value added. The financing of MFA loans by means of EU borrowing does not affect Member States’ finances or debt ratios, but does allow them to provide support in regions that they have interest in. Consistency with the EU’s external policies allows more economies of scope (within the Commission and the EEAS) than smaller Member States in particular would otherwise be able to mobilise. This added value tends to increase with the volume of the operation.

From the beneficiaries’ perspective, all the consulted stakeholders recognised the value added by MFA. According to them, a majority of the reforms implemented as part of MFA’s reform requirements would not have taken place otherwise (or at least not as quickly) – e.g. reforms in the areas of public financial management, anti-corruption and the judicial system. This is despite the fact that the scope and overall impact of the MFA operations in their countries were relatively small compared with the overall EU presence there.

As some key stakeholders suggest, MFA supports the EU’s unique role in fostering reforms to strengthen the rule of law in particular. Interviewed stakeholders from both the Commission and the IFIs suggested that not all the MFA conditions were directly affecting macro-economic stability in the beneficiary countries. Stakeholders considered this to be an advantage, however, especially because MFA in the beneficiary countries works alongside the IMF and other EU instruments. Unlike the IMF, which is committed to strict neutrality, MFA promotes democratic values directly as well as other values that are of interest to the EU and the beneficiary countries. Rule of law and democratic values are not MFA operational objectives, however, but are rather pre-conditions for obtaining MFA.

This is despite the observation that the size of EU added value is perceived differently across operations, both by external evaluators and by beneficiary countries. According to the external study, this may reflect different levels of involvement by the EU (compared with the IMF and the World Bank) in the perception of both the external evaluators and the beneficiary countries. This could be because (i) IMF support has to be in place before the authorisation of an MFA and (ii) according to interviews with the IFIs, the IMF was usually ‘making a bit more noise’ about its support than the EU.

The interviewed stakeholders have noted in this context that MFA’s visibility was rather low in most beneficiary countries. This low visibility reduced the value added by the MFA because the instrument was under-utilised for the promotion of EU economic diplomacy in beneficiary countries and their geographical surroundings. The meta-evaluation finds that the EU could increase the value it adds through more active engagement with political leaders, greater media coverage and engagement with a wider group of stakeholders, including the private sector and civil society. Some stakeholders also mentioned the possibility of communicating more consistently on the whole package of the EU’s assistance, including MFA. Some key stakeholders were convinced that increased visibility can help the EU to reach out to non-traditional interlocutors and thus provide new opportunities to form ‘coalitions for reform’, increase the sense of ownership in the beneficiary countries, and thus make MFA more effective in promoting important reforms.

4.3    Is the intervention still relevant?

Continuing demand for MFA operations from beneficiaries demonstrates the instrument’s ongoing relevance. During the evaluation period, MFA was requested and deployed 15 times. Moreover, 10 operations were requested and authorised at the inception of the COVID-19 pandemic in 2020. The Commission received further requests by the end of 2021, despite the increased availability of global liquidity given the 2021 general allocation of special drawing rights (SDRs) by the IMF. The emergency and exceptional MFA provided to Ukraine in 2022 in the context of Russia’s unjustified war of aggression is further important testimony of the usefulness of MFA as a flexible crisis instrument.

On the basis of past operations, the external study also draws a positive conclusion on the relevance of policy reforms promoted through MFA. MFA had an operational and/or political reinforcing effect in all areas in which conditions were formulated (Moldova). It often therefore acted as a positive contributing factor rather than the sole cause of a reform (Ukraine 10.14 25 ). MFA similarly promoted actions that might not have been implemented otherwise while, in some other instances, the authorities would probably have taken action even without the MFA conditions (e.g. Moldova). In terms of scope, they addressed important issues, even if they did not always comprehensively cover all of the most pressing reform needs (e.g. Bosnia and Herzegovina 26 ). Reforms in Public Financial Management (PFM) and the social safety net, which are often the subject of conditions (see Table 7 ) were particularly relevant (Ukraine 10.14).

Against this backdrop, the external study concludes that the MFA instrument has a high level of relevance in most countries. Structural reform conditions enshrined in the MoU are relevant and most of them have been implemented as agreed. Only in a few cases was implementation of the agreed reforms slow and ownership of the reform programme limited (e.g. Tunisia 14).

4.4    Is the current framework for evaluating interventions appropriate?

With the 2015 ‘Guidelines for the Ex-Post Evaluation of Macro-Financial Assistance Operations’, the MFA ex post evaluations started to follow the Commission’s Better Regulation Guidelines (BRG). Previous evaluations did examine the five criteria of effectiveness, efficiency, coherence, relevance and EU added value but the BRG provide more detailed guidance on the general approach to be used. More specifically, BRG brought them together, gave them a more precise and standardised definition, and placed them at the heart of the evaluation framework for EU-supported interventions including the MFA operations.

The post-BRG ex post evaluation reports were therefore structured more closely around the five criteria after 2015 than had previously been the case according to the external study. The evidence demonstrates that the MFA evaluations throughout the period generally provided a good assessment of the key evaluation criteria set out in the 2015 BRG. These key criteria featured in the MFA ex post evaluations throughout the period, albeit to a varying degree. Within the overall BRG framework, each evaluation was meant to be tailored to the objectives of the specific programme and its expected outputs, results and impact. Whereas earlier, pre-2015 ex post evaluation reports had been structured around the main MFA objectives (macroeconomic impacts, contribution to structural reforms, and the design and implementation of the MFA operations), the later post-2015 ex post evaluation reports were largely structured around the five evaluation criteria.

In addition to the five evaluation criteria, the BRG encouraged a more in-depth assessment of social impacts than had previously been evident. The Social Impact Analysis (SIA) included in the MFA ex post evaluations was intended to improve the understanding of the extent to which MFA interventions, which primarily had an economic objective, also had effects – intended or otherwise - of a social nature and whether these were positive or negative, or a combination of these effects depending on geographical, social and other factors. This was a new type of analysis. In contrast, to the Debt Sustainability Analysis (DSA) introduced by the BRG was less novel to the ex post evaluations, because it was essentially an extension of the existing macroeconomic impact assessment.

Overall, the use of SIAs and DSAs made a significant contribution to the assessment of MFA efficiency and effectiveness, based on the external study. With the DSAs the difference between the pre- and post-BRG level of analysis was less pronounced because this type of analysis was already a key element in the earlier ex post evaluations. The DSAs improved the understanding not only of the macroeconomic impacts – which were central to all the ex post evaluations, whether pre- or post-BRG – but also of the longer-terms effects in terms of debt sustainability.

27 The BRG also placed increased emphasis on the systemic involvement of key stakeholders (both in terms of impact assessments and ex post evaluations). There had already been quite extensive consultations with stakeholders in Brussels, the beneficiary countries and the international financial institutions as part of the earlier MFA ex post evaluations. However, the 2015 BRG encouraged evaluators to further develop this aspect of their research, with examples including the Delphi technique successfully used to a greater extent in engaging a wide range of stakeholders with a knowledge of MFA. The 2015 DG ECFIN evaluation guidelines enshrined this principle further by de-emphasising econometric modelling for the purpose of counterfactual analysis. The analysis of high-frequency economic indicators has instead been supplemented by greater stakeholder consultation in the construction of counterfactuals. This decision was in line with a 2012 CEPS-report commissioned by the European Parliament  which concluded the following on this matter: ‘The meta-evaluation report [of 2010] devoted attention to the question whether econometric modelling techniques could materially help the evaluation process. Their conclusion, which this reviewer shares, is that there are both practical and theoretical reasons why this could add little value to the evaluations.’

The methodological approach to MFA counterfactual modelling is in line with how DG ECFIN has evaluated euro area economic adjustment programmes  28 where a crisis makes counterfactual scenarios underpinned by econometric modelling less reliable. Such an approach also has parallels in the work of other institutions (e.g. the IMF’s Independent Evaluation Office (IEO) recognises ‘significant empirical challenges involved in identifying appropriate counterfactuals and isolating the impact of programmes’  29 ).

When undertaking individual MFA evaluations, DG ECFIN does not prescriptively set an explicit methodology to be followed by external contractors. Instead, an overarching set of MFA evaluation methodological guidelines complements the Better Regulation requirements. This promotes competition and innovation among external contractors in devising suitable evaluation methodology, not just for counterfactual analysis but also for the overall assessment of the evaluation criteria. Examples include (i) analysis of fiscal savings relating to favourable debt terms used by one contractor that has subsequently been applied in other MFA evaluations, and (ii) the innovative use of social media analysis by a contractor in the assessment of relevance, effectiveness, efficiency and EU added value in the evaluation of MFA Ukraine I & II.

Against this backdrop, it seems that the standard of individual evaluations was good and therefore that, by extension, the reliability and relevance of the methodologies employed was appropriate. Evaluations followed the relevant methodological guidelines and were further enhanced by the introduction of the BRG into the MFA evaluation framework. The evaluations provide comprehensive and detailed insights into the MFA’s macroeconomic impacts, its contribution to structural reforms and other issues relating to the design and implementation of the MFA operations. In particular, it can be concluded that most of the MFA evaluations provided a good assessment of what would have happened without the MFA operations. The counterfactual analysis typically focused on macroeconomic issues but was also extended in a number of cases to the assessment of the MFAs’ contribution to structural reforms.

This overall positive assessment notwithstanding, the external evaluator has also found several points for improvement in the evaluation of the individual MFA programmes in the future:

1.The comparative dimension of the ex post evaluations was a relatively weak aspect of the assessments and ‘lessons learned’ were therefore not a prominent feature. More specifically, only two evaluations had dedicated sections on this matter, whereas around two thirds of the MFA ex post evaluation reports included ‘recommendations’. This was particularly the case in pre-2015/pre-BRG evaluations.

2.The assessment of coherence can be further improved by placing more emphasis on assessing all the EU-supported programmes as a whole in a particular beneficiary country. In addition, elements in MFA operations that contribute directly or indirectly to wider EU policy goals could be more clearly highlighted.

3.The issue of low MFA visibility also extends to its ex post evaluations. The evaluations’ findings were usually not disseminated systematically after the reports were finalised. Some stakeholders such as central banks, ministries of finances, IMF/World Bank did at times have access to the information because they participated in validation workshops, but this was not the case for other stakeholders such as local chambers of commerce and industry, some IFIs (e.g. the EBRD), representatives of civil society, academics, and the business and financial sector. 

5.1    Conclusions

The overall assessment of the MFA instrument over the meta-evaluation period is very positive. It contributed to operations that stabilised the economy of beneficiaries. It was very cost-efficient and provided an appropriate level of coherence with other EU instruments in neighbourhood and pre-accession countries. Continuing demand confirms its ongoing relevance and stakeholders widely confirm the EU value added of past operations.

Furthermore, the framework for MFA evaluations is assessed as appropriate and useful. Ex post evaluations were carried out to a high standard and were further enhanced with the introduction of the BRG into the MFA evaluation framework. More specifically, the BRG encouraged a more in-depth assessment of social impacts and a better appreciation of the five evaluation criteria of effectiveness, efficiency, coherence, EU value added and relevance. The evaluations have therefore provided comprehensive and detailed insights into MFA’s macroeconomic impact; its contribution to better governance, more economic resilience and/or sustainable growth models through structural reforms; and other issues relating to the design and implementation of the instrument. Most MFA evaluations have therefore provided a good indication of what would have happened without the MFA operations.

Several points for improvement have nonetheless been identified.

1.Concomitance with IMF support: concomitance with IMF support significantly improved MFA’s cost-efficiency and effectiveness, mitigating financial risk and allowing division of labour. At the same time, making the MFA strictly conditional on a satisfactory track record of implementing an IMF-supported programme has sometimes caused implementation delays at the start of and during operations, and this has prompted some stakeholders to call for more flexibility. Furthermore, some are concerned that the standard arrangements with regard to IMF involvement could compromise the EU’s standing as an autonomous global actor. A more flexible approach was employed in this respect in recent exceptional cases (the COVID-19 MFA package and 2022 MFA to Ukraine). While clearly exceptional, this experience could, once evaluated, usefully feed into further reflections on the pros and cons of potential alternatives to the current regime.

2.Decision-making procedure: the current standard procedures in the EU (the OLP) and the beneficiary country (parliamentary ratification) have been identified as a potential source of inefficiency and risk, thus reducing the instrument’s effectiveness in addressing an acute or impending balance of payments crisis and promoting reform implementation within the standard time window of IMF-supported economic adjustment programmes. It was the intention of the Commission in 2011 to establish a framework regulation for MFA that would make the instrument more effective by streamlining its decision-making process, but no agreement could be reached on a legal set-up that was acceptable to all EU institutions. Recent experiences with the COVID-19 MFA package and with MFA support to Ukraine in the context of Russia’s unjustified invasion of the country have shown that the current set-up of MFA decision-making can allow very swift adoption if all institutions agree to make full use of the available procedural flexibilities in particularly urgent cases.

3.Volume of financial assistance: although critically dependent on the design of its policy conditions, the impact of MFA operations (and hence their effectiveness and added value) is (if all other factors are the same) proportional to the level of the financial support. The 2013 Joint Declaration no longer specifies specific ceilings, but it is not yet fully clear to what extent the current policies provide sufficient flexibility. This issue may therefore need to be revisited at a later stage.

4.Policy conditions: policy conditions that are appropriate for the chosen policy areas, their degree of ambition and impact on long-term sustainability, as well as their sequencing over time and focus are important elements of a successful MFA operation. Individual ex post evaluations have generally been positive regarding the relevance and impact of MFA reform conditionality.

5.Flexibility: the inability to adapt the conditions listed in the MoU distinguishes MFA from IMF support and the lack of adaptable disbursement modalities distinguishes it from EU budget support. A more flexible, IMF-like regime appears to have the advantage of being able to effectively anchor and accompany important policy reforms, especially in a fast-moving economic situation which could influence decisions on the most suitable policy response and therefore the necessary reform commitments. In the context of MFA, some flexibility is provided through a limited number of waivers which exceptionally allow disbursements to take place despite the non-fulfilment of one (or a few) reform conditions. Furthermore, in the 2022 exceptional MFA package for Ukraine, allowances for the uniquely challenging war situation were made in advance when designing conditionality in the MoU. The increased flexibility enabling the use of MFA conditionality even in this context proves that, while recognising the exceptional character of these operations, solid reform conditionality remains a cornerstone of successful MFA operations.

6.Visibility of MFA operations and their evaluations: MFA’s low visibility meant that the instrument did not realise its full potential to promote EU ‘economic diplomacy’ in beneficiary countries. More specifically, greater visibility would make it possible to reach out to non-traditional interlocutors and thus create opportunities to form ‘coalitions for reform’ to increase a sense of ownership in the beneficiary countries and thus support the MFA’s effectiveness in promoting important reforms.

5.2    Lessons learned

The above-mentioned conclusions provide a basis for further reflection, but there are also a few ‘lessons learned’ which the Commission could consider acting upon in the future.

1.Political pre-conditions: as explained in Section 4.1, the external study finds some challenges in the way political pre-conditions are assessed in practice that could have a bearing on MFA’s effectiveness. The requirements for continued MFA support should therefore be communicated more clearly in cases where actual or possible disruptions emerge in relation to the partner country’s respect of effective democratic mechanisms, including a multi-party parliamentary system, the rule of law and human rights.

2.Decision-making: Like the external meta-evaluation of 2009/10, the current one points to the risks brought to the instrument’s effectiveness and efficiency from the standard decision-making procedures at the side of the EU and in the beneficiary country. Although the EU decision-making was greatly expedited in the most recent COVID-19-related operations as well as for the emergency and exceptional MFA to Ukraine, the use of urgency procedures, notably by the European Parliament, is fully within each institution’s prerogatives and hence highly uncertain as to its use in future cases. At this stage, however, there is no clear indication given that changing the legal set-up would significantly speed up MFA operations beyond what can be achieved when making full use of the available procedural flexibilities. Further reflections may be useful on potential alternatives to the current decision-making procedure. These aspects could usefully be assessed once the evaluations of the COVID-19 MFA package and the exceptional Ukraine MFA package have been carried out.

3.MFA visibility and dissemination of MFA evaluations’ findings: the low visibility of MFA operations is an important finding of the evaluation and touches on MFA outcomes in several dimensions (effectiveness, coherence, and EU added value). Possible improvements in this regard should therefore be explored, including with regard to the dissemination of ex post evaluation results.

4.‘Lessons learned’: future MFA evaluations could place more emphasis on ‘lessons learned’. This is in line with the November 2021 updates to the Better Regulation Guidelines and could be made a specific requirement in the terms of reference for final ex post evaluation reports. More emphasis on ‘lessons learned’ and methods of sharing these with key stakeholders would enhance the benefits of future evaluation activities.

In 2020, the Commission’s DG ECFIN took the initiative for a new meta-evaluation 30 covering 15 MFA evaluations completed in the last ten years. The exercise, coinciding with the MFA instrument’s 30th anniversary, takes stock of previous evaluation results and aims to support organisational learning both in terms of the design and implementation of the MFA instrument and the methodology used to evaluate individual operations. The meta-evaluation also aims to support transparency and accountability and to contribute to laying the groundwork for any possible improvements of the instrument’s design and/or workings as necessary. The Decide planning entry for the meta-evaluation is PLAN/2020/6586.

In line with the requirements set out in the Commission Better Regulation Guidelines 31 , the meta-evaluation considered the criteria of relevance, effectiveness, efficiency, EU value added, coherence with other international and Union instruments for financial assistance, social impact and impact on the sustainability of public debts. The exercise relies primarily on an external independent study commissioned by the Commission in December 2020 and concluded in October 2021. In order to ensure validity, the analysis and conclusions of the external study are based on the evidence obtained using several evaluation methods (document review, semi-structured interviews with targeted stakeholders, open public consultation, a validation workshop and a roundtable with relevant Commission staff). A number of limitations were experienced during the preparation and completion of the study. Annex 2 provides a detailed overview of these shortcomings, together with the mitigation strategies adopted.

The outcome of the external assessment was supplemented by input from (i) a high-level seminar discussing the experiences of 30 years of MFA that took place in November 2021 (see below for a report), and (ii) the Regulatory Scrutiny Board (RSB).

The lead DG to carry out and manage the meta-evaluation has been DG ECFIN. DG ECFIN also chaired the inter-service group (ISG) that was set up to manage the external evaluation. Apart from DG ECFIN, the ISG comprised of representatives of other Commission services (the Secretariat General, the Directorate-General for Neighbourhood and Enlargement Negotiations and the Directorate-General for International Partnerships). The ISG had the responsibility to:

a.Establish the evaluation roadmap for the external evaluation;

b.Establish the Terms of Reference (ToR);

c.Facilitate the evaluator’s access to the information needed;

d.Advise, monitor and comment on the work undertaken by the external contractor.

The call for tenders was launched in October 2020 and two applications were received. Following the verdict of an evaluation committee, the specific contract to undertake the external evaluation was awarded to a consortium led by OXFORD RESEARCH under Framework Contract number ECFIN-003-2017 - ref. Ares(2017)1628394. The consortium included VVA Brussels and the Centre for Strategy & Evaluation Services. The total amount paid under this specific contract was EUR 135,200.00 covering all tasks executed.

A kick-off meeting, where the ISG and the external contractor discussed the deliverables and the evaluation methods, took place in December 2020. The evaluation roadmap ran between 21 January 2021 and 18 February 2021, to seek wider feedback. This was followed by the launch of the OPC on 30 March, for a period of twelve weeks (OPC Decide planning entry PLAN/2020/6586). Meetings on the inception and interim reports were held in, respectively, January 2021 and April 2021.

In order to provide appropriate support and ensure the evaluator's access to the necessary information as comprehensively as possible, bi-weekly dialogues were organised between the evaluator and DG ECFIN from May 2021 onwards. These dialogues continued until the end of the exercise. To address identified knowledge gaps on the MFA instrument noted in the interim report, ISG members also set up a dialogue between the external contractor and DG ECFIN staff in charge of MFA. The purpose of this roundtable discussion held in June 2021 was to inform the evaluator on the specific features of the instrument and to allow her to gather further views on possible limitations and developments.

The draft final report was first submitted in July 2021. Subsequently, the report was revised four times, before a final version was approved in October 2021.

As with all ex-post evaluations related to programmes with a major impact on EU budget or which are of strategic importance for the Union, the meta-evaluation carried out by DG ECFIN is subject to scrutiny by the RSB.

An RSB upstream meeting was held in October 2021, followed by the official consultation on 19 February 2022. The board gave a positive opinion on the report.

Table A1.1 below summarises the main points of the RSB review and how they have been integrated into the meta-evaluation.

This annex presents the methodological approach to the meta-evaluation study. It describes the design of the methodology, the tools used for data and information gathering and the results obtained. It also provides insights on the limitations encountered during the study and the mitigation strategies adopted.

Evaluation design

The methodology of the evaluation study was designed to respond to (i) the evaluation questions detailed in the Terms of Reference (ToR) for the meta-evaluation 32 , and (ii) the Better Regulation Guidelines (BRG) evaluation criteria 33 . It rested on three pillars:

1.Participatory and inclusive data collection and analysis through a (key) stakeholder consultation programme (semi-structured interviews and public consultation), a validation workshop preceded by a roundtable with the Commission staff in charge and a high-level seminar;

2.A mixture of qualitative and quantitative input, with qualitative input obtained mainly through document analysis (chiefly of the evaluation reports of the 15 aforementioned operations) and stakeholder input and quantitative data supporting the qualitative information to the extent possible; and

3.Triangulation, i.e. the information and data collected from a range of different sources using a range of methods collectively provides answers to DG ECFIN’s evaluation questions.

Tools for information gathering and results obtained

The information and data required for the study were collected using the following methodological tools:

a.Review of internal documentation concerning all Macro-Financial Assistance (MFA) operations under study, including among others: the MoUs of the Macro-Financial Assistance operations under analysis; the relevant ex-post evaluations and Commission’s staff working documents; the 2009 MFA meta-evaluation report; budget support publications; and the Commission’s Better Regulation Guidelines and toolbox.

b.Semi-structured interviews with key stakeholders from the Union Institutions, national authorities of the beneficiary countries, consultancy companies in charge of the previous evaluations, and International Financial Institutions (IFIs). A total of 50 detailed interviews were carried out between February and June 2021, of which 30 were with staff from the Union Institutions. The remaining interviews ensured a geographical coverage of all concerned MFA countries. To implement the interview programme, the research team systematically contacted stakeholders from the list which was agreed with DG ECFIN during the inception phase. For this purpose, email invitations were sent out, giving details about the meta-evaluation and interview, and providing a targeted questionnaire. To ensure that the key information from each interview was well recorded, minutes of the conversations were taken during the interview. The highest response rate was found with the European Commission staff most directly involved in MFA operations (DG ECFIN, DG NEAR) and DG INTPA, together with the Commission Legal Service and DG BUDG. By contrast, staff and Members of the European Parliament showed a much lower response rate, with a moderate final outreach.

c.Open Public Consultation (OPC), to gather the views of civil society and the general public and ensure transparency of the study, in line with the requirements of the Commission's BRGs. The OPC was launched on 30 March 2021 and closed on 22 June 2021, collecting a total of 20 responses. To further increase the response rate, the external study team disseminated the link among stakeholders who were identified in the stakeholder mapping exercise conducted for the interview programme, asking them to respond to the OPC and further disseminate the information among their professional circles. In addition, DG ECFIN reached out to the EU Delegations in the beneficiary countries, as well as other stakeholders, to inform them about the consultation. The OPC was also promoted on the European Commission's social media accounts (Twitter and LinkedIn).

d.Roundtable discussion with the Commission staff in charge of MFA operations, to discuss more technical elements concerning the instrument and the interventions under analysis. A total of 12 people took part in the roundtable, which focused on the following themes: MFA conditionality and political pre-conditions; macroeconomic stability; reforms sustainability; coherence of MFA with parallel financial assistance instruments by the Union or by other international actors.

e.Validation workshop, with key stakeholders and previous evaluators, to validate the preliminary findings of the meta-evaluation. A total of 22 participants from 14 different organisations took part in the workshop, which took place on 22 June 2021. The main takeaway points of the workshop contributed to refining and validating the conclusions of the evaluation, thereby contributing to the final report.

f.High-level seminar to present the preliminary findings of the study to a wider audience, including beneficiary countries, and gather further input to inform the meta-evaluation. A total of around 200 participants attended the event held on 16 November 2021, with representatives from EU institutions, partner countries, IFIs and public policy consultancies. The event was kicked off by Paolo Gentiloni, Commissioner for the Economy at the European Commission. His keynote speech was followed by two panel discussions with high-level speakers, comprising: representatives of Ministries of Finance and Central Banks of MFA countries; Deputy Directors-General and Directors of the Commission; Permanent Representatives and Mission Chiefs of the IMF. The second part of the event included two case studies on MFA operations in Ukraine and Jordan, with related in-depth discussions and Q&As. Input from the seminar was used as evidence to inform the analysis undertaken in the SWD.

While the semi-structured interviews, the roundtable and the validation workshop enabled the collection of information of a highly technical nature, the OPC and the seminar extended the hearing to include less specialised interest groups.

The data collection tools (with the exception of the seminar) were used to inform the work of the external evaluator. DG ECFIN, together with the other members of the ISG, closely monitored the contractor's work throughout the study.

In addition to providing all necessary documentation to the evaluator in a timely manner, DG ECFIN held bi-weekly meetings with the external evaluator, starting in May 2021, to verify the quality of the information and analysis used to inform the external report. Furthermore, DG ECFIN initiated the roundtable with the Commission staff responsible for MFA, in order to give the evaluator a greater understanding of the instrument and its limitations and correct inaccuracies in knowledge.

At the conclusion of the external study the ISG completed a quality assessment of the final report. According to the assessment, the study addressed all key issues as required in the Terms of Reference, although some of the underlying analysis lacked depth. The evaluator provided all requested deliverables in line with the work plan. Some of the deliverables required several iterations before the quality was acceptable to the ISG. The assessment concludes that the report is broadly satisfactory and can be approved on that basis.

Limitations encountered and mitigation strategies

A number of limitations were experienced during the completion of the independent study and hence the preparation of the meta-evaluation.

The following list summarises the main challenges as well as the specific measures taken to address them:

1.Limitations on data availability, mainly due to a low response rate in the OPC. The consultation ran for a period of 12 weeks, collecting a total of only 20 contributions. One of the reasons for the low response rate is possibly the highly specialised nature of the topic, which requires a high level of expertise and knowledge. The low number of responses affected the representativeness of the sample. To mitigate this limitation, the results recorded in the consultation were supplemented by findings obtained through the use of other data collection tools. In particular, the seminar was beneficial in filling missing gaps after the conclusion of the external assessment, as it involved a wider and diversified audience including representatives of Ministries of Finance and Central Banks of MFA countries.

2.Difficulties in accessibility to stakeholders, particularly regarding interviews undertaken during the external study. The response rate was lower than expected, thus limiting the reach of the interviews. The study team addressed the issue by expanding the list of stakeholders according to priority groups assigned. After reaching out to priority 1 stakeholders (EU institutions and local authorities), the team followed up with priority 2 (international organisations and past evaluators) and, where necessary, priority 3 stakeholders (potential substitutes for each stakeholder group). This approach ensured that the final target of 50 stakeholders was reached.

3.Limited diversification of stakeholders consulted. The low response rates recorded in some cases led to a partial polarisation of the answers provided, creating an under-representation of some categories of stakeholders compared to others. In order to rebalance the input received and ensure the greatest possible defensiveness of findings, the study team extended the original list of people interviewed in a targeted manner. The November seminar also provided an excellent opportunity to collect input from external stakeholders and further diversify the sources of information.

4.Limited subject knowledge of some interview respondents. Many of the assessed MFA operations were carried out several years ago. Not all of the interviewed representatives of the EU delegations worked in the delegations at the time of the MFA implementation; therefore, their knowledge of the MFA was somewhat limited. The same was also true for external evaluators, who had often left their position in the consulting companies conducting the evaluations. To mitigate this limitation, the study team attempted to establish contact with and to interview officials and consultants with highest possible level of subject matter expertise. Furthermore, the team mitigated this difficulty by diversifying the content of the interviews and checking inputs collected against each other when on a similar topic or specific country.

5.Delays in some parts of the interview programme due to bureaucratic requirements of the respondent organisation. This limitation was mitigated by a work structure that allowed several parts of the study to run in parallel, avoiding stalemates that would impact on compliance with deadlines.

Whilst such limitations have undoubtedly had a bearing on the quality of the exercise, they do not impact on the robustness or reliability of the meta-evaluation conclusions which serves as a useful basis for reflections on the instrument.

This annex provides a detailed overview of the evaluation matrix used for the meta-evaluation study. It was developed by the external contractor, as a framework for the independent assessment. The matrix assigns each evaluation question outlined in the ToR to the relevant evaluation criteria and serves as general framework for the applied methodology.

Table A3.1: Evaluation questions matrix

Source: Valdani, Vicari e Assoiati & Centre for Strategy and Evaluation Services, Oxford Analystics, 2021

Answers to the evaluation questions

Relevance

On the basis of past operations, the evaluator draws a positive conclusion on the relevance of policy reforms promoted through MFA. As such, MFA had an operational and/or political reinforcing effect in all domains in which conditions were formulated (Moldova). In many cases, they hence acted as a positive contributing factor, rather than a sole causal factor of a reform (Ukraine). And in a similar vein, MFA promoted actions that might not have been implemented otherwise while in some other instances, the authorities would probably have taken action also without the MFA conditions (Moldova). In terms of scope, they addressed important issues, even if they did not always comprehensively cover all of the most pressing reform needs (Bosnia and Herzegovina). Reforms in Public Financial Management (PFM) and the social safety net, where conditions are frequent, were particularly relevant (Ukraine).

Against this backdrop, the evaluator concludes that the MFA instrument shows good relevance in most countries. Structural reform conditions enshrined in the MoU are relevant and most of them have been implemented as agreed. Only in a few cases did implementation of the agreed reforms prove slow and ownership of the reform programme limited (e.g. Tunisia 14).

Continuous demand for MFA operations from beneficiaries demonstrates the persistent relevance of the instrument (in the past and present). In the evaluation period, MFA has been requested and employed 15 times. Moreover, 10 operations were requested and authorised at the inception of the COVID-19 pandemic in 2020. The Commission received further requests by end-2021, despite the increased availability of global liquidity given the 2021 general allocation of special drawing rights (SDRs) by the IMF.

Effectiveness

According to the external evaluation study, MFA was overall effective in stabilising beneficiary’s (external) macroeconomic environment, especially in the short and medium-term. Despite a challenging external environment at the inception of the respective MFA operations, market access could be maintained or regained in all cases. This includes one beneficiary (Ukraine) restructuring its debt in good faith negotiations. With the exception of one strategic default (by Ukraine on the Russian Federation), there were no defaults by beneficiaries. Reserve adequacy and current accounts mostly improved during MFA operations, and inflation subsided.

These findings are more nuanced for time horizons beyond the medium term, based on the evaluator’s analysis. Whilst some improvements in current account balances are visible over that time window, she finds that gains in reserve adequacy could usually not be extended further and current accounts remained in deficit. Furthermore, participants at the validation workshop lent support to this interpretation of the data.

The instrument’s geographical focus, design and form played a supportive role in the effectiveness of MFA in incentivising meaningful reform.

On the other hand, the long procedure for authorising MFA operations adds risks to their effectiveness. In particular, the time lag up until the MoU (and thus the MFA) enters into force eats into the term available for implementing reforms and risks some unwanted economic corrections in the meantime since supportive external funding is not yet available. Furthermore, the evaluator points to one country (i.e. Armenia 09) in urgent need of financial assistance that chose to seek frontloading from the IMF given the MFA’s lengthy procedures.

Notwithstanding the relatively high compliance rates, the MoU’s inflexibility appears to have reduced the MFA’s effectiveness in times of change. In contrast to Union budget support, MFA instalments are not disbursed in part and proportion to fulfilled conditions but only at 100% or 0%. This reflects IMF policies, but in contrast to those policies, MFA practice is that conditions are fixed at the operation’s inception and remain unchanged later in the operation when the national parliament ratified the text.

In any event, ownership of the agreed reforms is of critical importance for the MFA’s effectiveness in promoting important reforms.

Efficiency

The external evaluation confirms that MFA is a relatively (cost-)efficient instrument for the Union. MFA loans are provisioned at 9% of the principal amount to be disbursed. Given the fact that no beneficiary ever defaulted on an MFA loan, these 9% have been routinely returned to the budget, bringing the budgetary impact of MFA loans over time down to zero. Grants, by contrast, have to be budgeted fully.

The parallelism with IMF interventions provides some assurance concerning the country’s repayment capacity, also since the IMF does not usually provide funding to countries with an unsustainable debt (unless it is taking measures to address this issue). In addition, the synergies arising from relying on the IMF’s work provides for important cost-savings on the Union side. Similarly, the geographical focus provides for synergies in terms of expertise and human resources across and within different DGs.

MFA is also cost-efficient for the beneficiary countries, since its loans are highly concessional. Due to its high creditworthiness, the EU can borrow money at much lower interest rates than the beneficiary countries would get if they were to go to the market themselves. As the Union passes on these interest rates in its loan agreements, the beneficiary countries receive an often substantial financial benefit. 

The current EU decision-making arrangements entail some inefficiencies. The time needed for the co-legislators to decide on the Commission’s proposal generally takes 5 months. In addition, before the MFA enters into force, the MoU has to be formally agreed; that requires a Commission implementing act subject to Comitology. It takes roughly 7 months to conclude the negotiations with the beneficiary country and adopt the MoU, which can be shorter when the negotiations happen for a large part in parallel to the legislative process.

Questions can also be raised over the efficiency of the decision-making process in beneficiary countries, where MoUs are oftentimes ratified. Whilst this can provide for stronger ownership of the reforms to be implemented, it tends to increase the period up to the disbursement of the assistance and all but prohibits adaptations.

Coherence

MFA usually intervenes in tandem with other Union instruments, such as ENI/ENPI/NDICI and IPA. This results in important synergies chiefly with budget support under these instruments.

The stakeholder consultation shows that external coherence with IFI interventions is generally seen as high and supporting the workings of the instrument. Stakeholders mainly referred to complementarity and lack of overlap between the conditions of the different actors after accounting for their different interests, mandates and approaches.

In what concerns the internal coherence, the stakeholder consultation suggests that the MFA has been developed in a consistent and clear manner, so that the implementation follows the same set of principles and rules in all beneficiary countries. Coherence with the Union’s external policies allows economies of scope (within the Commission and the EEAS) to an extent that in particular smaller Member States would have difficulties to mobilise otherwise. Such value added hence tends to increase with the volume of the operation.

EU value added

From the Member States’ perspective, Union MFAs provide financial and political value added. The re-financing of MFA loans by means of Union borrowing does not affect Member States’ finances or debt ratios whilst allowing support in regions that they have interest in.

From the perspective of beneficiaries an added value of MFA is recognised by all consulted stakeholders. According to them, a majority of the reforms implemented as part of MFA’s reform requirements would not have taken place otherwise, or not as quickly – e.g. reforms in the area of public financial management, anti-corruption and judicial system. This is notwithstanding the fact that the scope and the overall impact of the MFA operations in the countries was relatively small in comparison to the overall EU presence there.

As some key stakeholders suggest, MFA supports the Union’s unique role in fostering especially reforms to strengthen the rule of law. Interviewed stakeholders both from the Commission and from the IFIs suggested that not all conditions imposed by the MFA were affecting directly macro-economic stability in the beneficiary countries. However, stakeholders assessed this characteristic of the MFA conditions as an advantage, especially since the MFA in the beneficiary countries works alongside the IMF and other Union instruments. Unlike the IMF, which is committed to strong neutrality, MFA promotes democratic values directly as well as other values that are of interest to the Union and the beneficiary countries.

However, interviewed stakeholders have noted that, in most beneficiary countries, the visibility of the MFA was rather low. This lack of visibility reduced the added value of the MFA, as the instrument was under-exploited for the promotion of EU economic diplomacy in beneficiary countries and their geographical surroundings.

***

With the 2015 “Guidelines for the Ex-Post Evaluation of Macro-Financial Assistance Operations”, the MFA ex-post evaluations started following the Commission’s Better Regulation Guidelines (BRG). Whilst previous evaluations did examine effectiveness, efficiency, coherence, relevance and EU added value, the BRG provide more detailed guidance on the general approach to be used. More specifically, BRG brought them together, gave them a more precise and standardised definition, and placed them at the heart of the evaluation framework for EU-supported interventions including the MFAs. Consequently, the post-BRG ex-post evaluation reports were structured more closely around the five criteria after 2015 than was the case up to that point, according to the evaluator’s analysis.

In addition to the five evaluation criteria, the BRG encouraged a more in-depth assessment of social impacts than was evident before. With the DSAs the difference between the pre- and post-BRG level of analysis was less pronounced as this type of analysis was already a key element in the earlier ex-post evaluations.

The BRG also placed increased emphasis on the systemic involvement of key stakeholders (both in terms of impact assessments and ex-post evaluations).

The typology of costs in the Better Regulation toolbox (defined as compliance, enforcement and other indirect costs) is impossible to apply in the context of MFA operations, as such costs are not incurred here. As indicated in the Legislative Financial Statement accompanying Commission’s proposals for MFA, the budgetary impact at the level of the EU is limited to some administrative costs related to HR and outsourcing of operational assessments and ex-post evaluations.

The typology of benefits (improved welfare, market efficiency and wider economic effects) can seem to be more relevant. However, as shown in past country ex-post evaluations, which tried to quantify the macroeconomic effects of the MFA assistance, either by counterfactual modelling or by other more qualitative methods, it is almost impossible to disentangle the effects of MFA from IMF assistance (both disbursed at the same time) and from other macroeconomic events.

The Guidelines for carrying out ex-post evaluations of MFA operations of June 2015, acknowledged the problem and stated that:

[…] EU added value may be the results of different factors: coordination gains, legal certainty, greater effectiveness or efficiency, complementarities etc. In all cases, concluding on the continued need for the intervention at EU level may be difficult as measuring EU added value is challenging.

The analysis of EU added value is often limited to the qualitative, given the stated difficulties to identify a counterfactual.

More specifically in the field of ex-post evaluation of MFA, challenges can arise which cannot be easily discarded and should be taken into consideration:

·The nature of the instrument: MFA is not earmarked in any way. It is an instrument complementing an IMF programme. It cannot be linked directly to identifiable outputs as in the case of programme or project-linked aid;

·Objectives of MFA operations can be implicit for political reasons and when they are explicitly presented in an MoU, they can sometimes be not specific in nature;

·The size of the assistance: MFA grants/loans are provided in addition to the provision of an IMF loan and may represent a small proportion of the total funds accorded to the recipient country. Thus the financial components of both instruments cannot usually be easily or indeed meaningfully disassociated;

The consequence of these challenges is that:

·A classic assessment of efficiency (i.e. cost-effectiveness) is not feasible, though an examination of the suitability of the blend of grants and loans making up the assistance and other questions about its design and implementation and consequences for the achievement of objectives are pertinent issues.  

·Effects on macroeconomic variables over time cannot be uniquely attributed to MFA, requiring that analyses take into account the global package of which the assistance is a part.

Past country ex-post evaluations have taken this guidance into account and have come with an evaluation methodology suited for the MFA instrument.

This annex presents an outline of the consultation strategy adopted for the meta-evaluation. It is largely based on the consultation strategy developed as part of the independent study undertaken by Oxford Reseach, in collaboration with Valdani, Vicari e Associati and the Centre for Strategy and Evaluation Services. The annex details the objectives of the strategy adopted, the stakeholders and tools identified, and its final dissemination.

Objectives

The consultation strategy followed the indications provided by the Commission’s Better Regulation Tool #54.

The specific goals of the consultation activities were to:

1.Collect additional data on both MFA principles and characteristics and evaluation methodologies;

2.Engage with stakeholders involved in the delivery and evaluation of MFAs; and

3.Validate the preliminary findings of the meta-evaluation.

In order to ensure transparency and engagement, the process followed the standards and methods set out in the Better Regulation Guidelines (BRG). The various consultations complied with the Terms of Reference (ToR) and the following evaluation roadmap 34 .

Identified stakeholders and tools

The stakeholders targeted by the consultation activities were individuals or organisations who held a specific role in one or multiple MFAs and/or subsequent evaluations. The table below shows the stakeholder groups targeted through the consultation activities.

Table A4.1 Identified stakeholders and tools

Source: Valdani, Vicari e Assoiati & Centre for Strategy and Evaluation Services, Oxford Analystics, 2021

The consultation strategy employed five different consultation tools, following the ToR provisions and agreement with DG ECFIN. The table below illustrates the type of consultation tools used per each of the two parts of the external study, together with the seminar held in November.

Table A4.2 Key methodological tools and relevance for meta-evaluation

Source: Valdani, Vicari e Assoiati & Centre for Strategy and Evaluation Services, Oxford Analystics, 2021; EC

Putting together the two dimensions, the table below illustrates the type of consultation tool used per category of stakeholder.

Table A4.3 Key methodological tools and relevance for meta-evaluation

Source: Valdani, Vicari e Assoiati & Centre for Strategy and Evaluation Services, Oxford Analystics, 2021; EC

Final outreach

The total reach of the consultation activities, excluding the seminar, is shown below.

Table A4.4 Total reach of consultation activities

Source: Valdani, Vicari e Assoiati & Centre for Strategy and Evaluation Services, Oxford Analystics, 2021

Delivery of the consultation

This section presents the delivery of the completed consultations per type of activity.

A.Semi-structured interviews

A total of 50 in-depth interviews were conducted with key stakeholders, all of which were conducted via teleconferences. The interviews were distributed as follows:

·26 from the European Commission (DG ECFIN, DG NEAR, DG INTPA, DG BUDGET, Cabinet staff, Commission Legal Service (SJ);

·6 from the European External Action Service (EEAS);

·1 from the European Parliament;

·3 from the Council of the European Union and EU Member States;

·2 from authorities in beneficiary countries;

·4 from consulting companies/past evaluators; and

·8 from international financial institutions (IMF and World Bank).

Taking into account the restraints of varying responsiveness among stakeholders, the research team aimed to collect information from the most relevant stakeholder groups in a balanced way. While the primary focus of the interview programme has been on the role of the EU institutions in designing and implementing the MFA, the study team interviewed country experts (mainly from the Commission and EU Delegations) in all countries benefitting from MFA, hence ensuring a substantial geographical diversity.

B.Open public consultation

The aim of the open public consultation (OPC) was to collect views and opinions about the MFA operations from a wide range of stakeholders. Data collected through the OPC complemented the findings obtained through semi-structured interviews and document analysis.

The OPC was launched on 30 March 2021 and remained open until 22 June 2021, collecting views from 20 respondents, of which 12 came from officials at the European institutions and EU delegations. Public authorities in beneficiary countries provided one answer, while authorities and officials from EU Member States provided five replies. Two replies came from individuals, one from a beneficiary country and one from an EU Member State. No respondent chose the options: academic/research institution, business association company/business organisation, consumer organisation, environmental organisation, non-EU citizen or trade union.

C.Workshop

An online workshop with evaluation experts in the field of macro-financial assistance (MFA) was organised on Tuesday June 22, 2021. At this point, the research process was almost finalised, with the desk research, interviews, and OPC all completed. This meant that draft conclusions, for both Part A and Part B of the evaluation, were available for discussion and validation during the workshop.

The workshop gathered key stakeholders and previous evaluators of MFA operations in order to validate the preliminary findings of the study. The feedback gathered was used to fine tune the final report. Overall, the participants confirmed the findings and further developed some of them.

More specifically, the purpose of the validation workshop was to:

·Review and validate the overall draft findings of the meta-evaluation of the MFA;

·Discuss the overall draft conclusions of the evaluation, and their implications for the content of future MFA operations and related programmes; and

·Consider any lessons to be learned from the study for the Commission’s guidelines on programme evaluation, specifically the MFA operations.

To do so, the research team reached out to a total of 52 stakeholders from 27 different organisations, with a final attendance of 22 participants from 14 organisations.

D.Roundtable

On June 10, 2021, the meta-evaluation contractor and DG ECFIN co-organised a roundtable with selected Commission staff working on MFA operations. The roundtable saw the participation of a total of 12 people.

The focus of the discussion was on identifying the core limitations encountered in the MFA operations and the potential lessons learned for future interventions. In particular, the roundtable touched on four topics:

·Conditionality, including political pre-condition, complementarity with IMF financial arrangement, and structural reform MoU policy conditions;

·Macroeconomic stability;

·Reforms sustainability; and

·Coherence.

E.Seminar

On November 16, 2021, Directorate D organised an (online) event celebrating 30 years of the Macro-financial assistance (MFA) instrument. The event aimed at taking note of the evolution of the instrument over the past 30 years, give visibility and provide useful input for its future. Politically, the event was intended to raise awareness of the instrument; the complementarity and value added compared to other forms of external policy instruments; how its set-up provides particular leverage, allowing the EU to advance its values; and the MFAs importance in contributing to stability and prosperity in our neighbourhood. The event provided a forum for in-depth discussion on the MFA’s pros and cons by gathering key stakeholders; addressed limitations encountered so far and pointed to potential improvements. The event was comprised of two high-level panel discussions, case studies on the MFA operations for Ukraine and Jordan as well as discussions and Q & As.

Almost 200 participants took part in the event:

·EU Institutions: DGs ECFIN and NEAR as well as the European External Action Service were represented at senior-management level, the Council at technical level.

·EU Member States were represented at technical level.

·Partner countries in focus were represented at senior level.

·Public policy consultancies and independent international financial institutions took also part in the event.

General appraisal

Overall, the MFA has been rated very positively across all intervening stakeholders for both its ability to support the restoration of macro-economic stability as well as its role in promoting structural reforms in recipient countries. Moreover, it was praised for its complementing role to other EU foreign policy tools, constituting a powerful lever. Following senior officials from the IMF and COM services, to recipient countries, the MFA instrument, in its close alignment and collaboration with IMF missions, provides a trigger to enact and a framework to anchor reform policies. While being a very case-specific tool, general shortcomings were identified around the instruments inflexibility on conditionality, its visibility and its role as a crisis instrument that under the current structural set up requires – in some cases lengthy – internal as well as external processes to be enacted.

Role of MFA in EU external policies

Participants from COM services, an external evaluator and IMF officials, considered MFA to have an important role in the EU’s external policy tool kit, supporting and complementing other instruments. Central to the tool’s relevance and, according to a senior official from the COM services, one of the strongest features of the instrument is the successful close collaboration with the IMF, beneficiary countries authorities and other EU policies on the bilateral level, e.g. budget support efforts and technical assistance.

Moreover, the alignment of overall policy objectives and conditions with these instruments creates strong synergies and allows the EU to put weight on key priorities. In this regard, MFA conditions can support longer-term cooperation and structural reform efforts by enhancing incentives from an additional angle, also given the sheer and somewhat unique size and the resulting leverage, as senior officials from COM/EEAS services noted. Furthermore, the close collaboration and cross-conditionality with IMF interventions carries the message of a combined undertaking of the international community, further increasing the weight of conditions.

Conditions

Next to the financial assistance, the conditions attached to the disbursement of funds are an important pillar of MFA. The generally large pertinence of MFA conditions was highlighted by COM/EEAS and IMF officials and stressed by high-level officials of beneficiary countries authorities. They emphasized that through their conditionality, MFA operations add incentives for national decision-makers to also undertake unpopular reforms, creating a political “window of opportunity for home-grown reform efforts”.

On the all-important issue of potential backsliding on reforms, a high-level official from a beneficiary country authority pointed out that such incidences could be considered proof of the concerned condition’s necessity. At the same time, beneficiary countries authorities also argued that politically sensitive reforms, e.g. aiming at changes to the constitution or the passage of laws, should be avoided, as they are very hard to fulfil in changing political conditions and outside of the executive powers. Moreover, they highlighted that more focused conditions would lead to more success in meeting them – “rather two small steps than one failed big step”.

Noting that, frequently, key reforms, particularly regarding the rule of law, are politically sensitive but of utmost importance for overall reform efforts, COM & IMF officials and an external evaluator nonetheless supported more flexibility and focus of conditions for three reasons:

1.Beneficiary countries, by nature of the instrument, are usually in situations not only of financial, but also political instability that require the recalibration of conditions with changing environments.

2.As a crisis instrument, MFA is influential only during a limited time period and often met with scarce capacities in beneficiary countries, begging fewer and more nuanced conditions.

3. When facing political deadlock, applying leverage through alternative angles by reorienting conditions, e.g. transparency measures to increase public attention rather than changes to laws, can be more successful.

Processes and procedures (internal and external)

On the internal processes, the discussions revealed that the current institutional set-up involving different EU institutions and case-by-case decisions can lead to lengthy negotiations that collide with the instrument’s ability to function as a crisis instrument. However, as COM/Council officials argued, the underlying democratic process is a crucial measure of accountability towards EU taxpayers. Similarly, the alignment of overall policy objectives, cross-conditionality with internal and external partners is a crucial component for the instrument’s success that requires time. Another inherent difficulty to this process is the realization of compromises between the varying priorities among internal actors. Meeting these expectations risks overburdening local capacities and the long-term perspective of conditions sometimes deviates from the short-term nature of the instrument.

Nonetheless, as the recent Covid-MFA has demonstrated, this process can be completed sufficiently fast for the instrument to be rolled out in due time. Among participants, broad support was voiced for more flexibility in the internal process concerning the termination of operations when conditions are not fully met, as this often constitutes the collapse of the dialogue with beneficiary countries authorities when it is needed the most. The external processes, particularly the negotiating of Memoranda of Understanding, vary from case and size of the MFA operation. The continued collaboration and coordination with other EU organs and the IMF were highlighted as crucial to create synergies as well as to avoid redundancies and generally praised as successful in advancing common goals

MFA’s impact within the beneficiary country

Experience shared by COM/EEAS services officials and high-level participants from beneficiary countries authorities confirm the MFA’s significant positive macro-economic effects as well as a positive reinforcing role in promoting structural reforms. Whilst contingent on the size and scope of the operation, the highly concessional nature of the instrument regarding interest rates, grace periods and maturities allows recipient countries to restore financial and generally macro-economic stability, yielding positive effects on the overall macro-economic situation and generating EU added value. Being most effective in times of acute crisis, more moderate effects can be observed during recoveries as one country official noted. An external evaluator further added that while the medium-long term effects are also significant, these are hard to quantify. The absence of MFA operations would in most cases lead to cuts in public spending with severe knock-on effects on the overall economy and force beneficiary countries to borrow at higher rates, contributing to a further worsening of the external sustainability of public finances.

On the structural reform efforts inherent to MFA, senior IMF officials advance the same argument when considering cases of backsliding: Though performance on key structural reforms can be slow and incomplete in some cases, even fractional successes help to forestall worse outcomes and help to provide a framework for international and domestic support. MFA conditions have in many cases triggered home-grown reforms and delivered positive social impacts, e.g. improvements to the rule of law or more equitable tax systems, as one IMF official further elaborated. Key aspects to success are the political ownership in beneficiary countries, government capacity as well as the overall political stability. In this regard, senior level COM/EEAS officials have bemoaned the instrument’s low visibility in beneficiary countries and pointed to the need for more visible public political “naming and shaming” on the international stage to prevent backsliding and to increase local public awareness.

Results of the consultation

This section discusses the results of the completed consultation activities by each evaluation criterion.

Effectiveness

The MFA instrument has proved effective in stabilising macroeconomic fundamentals, especially in the short and medium-term, according to the majority of interviewees. Nevertheless, they also highlighted that effectiveness was a result of the burden sharing among international partners. On its own, without linkage to IMF disbursements, the MFA would have hardly resulted in the same effectiveness. The interview programme suggested that the link to a disbursing IMF financial arrangement should be kept, although a few argued that the EU had the necessary capacity and skills to grant MFAs on its own. Most of the respondents to the OPC also favoured keeping the link with the IMF. Participants at the validation workshop underlined that the MFA was particularly effective for restoring stabilisation in the short term, rather than in longer term.

The interview programme also showed that the effectiveness of the MFA is reduced due to inflexible conditionalities in the MoU. In contrast to the IMF, conditionalities enshrined in the MoU are fixed ahead of the programme and cannot be changed during the programme. In cases where the lack of reform implementation has delayed a disbursement, the effectiveness could have been increased had conditionalities been adapted. Another aspect highlighted by interviewees is that non-compliance with the political pre-condition has reduced the effectiveness of the MFA. This is mainly because the political pre-condition is unique to the MFA instrument. OPC respondents generally believed that reforms of the judicial system and on anti-corruption were very important. Moreover, the OPC shows that the political pre-condition remains important, and that reform priorities should also focus on areas linked to the political pre-condition.

During the validation workshop, a discussion on the added value of cooperation with the IMF arose. Participants were generally in favour of cooperation with the IMF.

Efficiency

The consultation confirmed the overall efficiency of the MFA instrument, although respondents emphasised that the degree of efficiency is difficult to assess. Efficiency relates to the costs and benefits of the MFA instrument, which are perceived to clearly exceed the costs, but the nature of the benefits makes them hard to quantify: they are linked to the mitigation of a balance-of-payments crisis that could have very far-reaching costs. Moreover, MFA operations promote structural reforms that aim at strengthening the economies and their growth potential, which only materialises in the long-term.

The efficiency of the MFA is nevertheless hampered by the long decision-making process.

The costs of providing MFA are assessed to be relatively low. Neither the stakeholder consultation nor the desk research indicated any late payment of loans from beneficiary countries to the EU. Loans are provisioned at a rate of 9% and hence have a limited budgetary impact. On the other hand, grants have a full impact on the budget. In recent years, MFA has predominantly been granted as loans, and hence had limited budgetary costs.

The validation workshop confirmed the efficiency of the MFA. Despite the lengthy decision-making process, participants at the workshop did not see late MFA approvals as a key factor affecting the efficiency of the MFA instrument.

In some cases, interviewees highlighted that the efficiency was relatively low in beneficiary countries outside of the standard geographical scope of the MFA. This was explained by the lack of country knowledge of EU institutions, which means it is necessary to rely on external expertise. This was also confirmed at the validation workshop, where some participants highlighted that the relevance of MFA is higher in countries usually not eligible for MFA, but its efficiency is lower due to the limited knowledge of structural features of the beneficiary country and the lack of long-term engagement.

Relevance

The consultations indicated that the MFA instrument shows good relevance in most countries. In some, the MFA operation was less relevant. In a few countries, the reform programme did not enjoy high ownership from the local government, reducing its relevance. The lack of ownership was reflected in the slow implementation of the reform programme or in the implementation of the reforms happening only on paper. The interview programme showed that weak ownership and changes in the political landscape have led to policymakers reversing some of the reforms or adopting other reforms that dilute the envisaged reforms.

The stakeholder consultation showed that structural reforms enshrined in the MoU are relevant. Most of the reforms has been implemented in line with the MoU. Most reforms were implemented in the area of public financial management, anti-corruption and judicial systems. In a few instances (e.g. Tunisia), reform implementation proved slow and ownership of the reform programme remained weak. However, the reform programme in Tunisia remained relevant. One country (i.e. Armenia) in urgent need of financial assistance chose to find alternative assistance. In this case the relevance of the MFA was hampered by the lengthy procedures.

The OPC found that the relevance of the MFA instrument could be increased, for instance, by expanding its geographical scope. This view was presented by a small share of respondents to the public consultation. This is broadly in line with the interview programme, where only a few interviewees supported the expansion of the scope and highlighted a trade-off between efficiency and relevance.

Coherence

Within the framework of this evaluation, two types of external coherence of the MFA were analysed: (1) coherence with other EU instruments present in the beneficiary countries; and (2) coherence with programmes of IFIs. The stakeholder consultation confirmed that both external and internal coherence of the MFA were in general very good.

With regard to external coherence at EU level, stakeholders confirmed that, to a large extent, the objectives, design and implementation of the MFA are complementary and consistent with other initiatives providing financial support to beneficiary countries. The MFA is complementary to the budget support programmes in the beneficiary countries, and conditionalities linked to the two instruments are aligned to the extent possible. For instance, COVID-19 Resilience contracts were aligned with the MFA conditionalities whenever relevant and possible.

With regards to external coherence with programmes implemented by IFIs, the stakeholder consultation shows that it generally works well. Complementarity and lack of overlap between conditionalities imposed by different actors is specifically sought at the stage of the instrument design. However, the objectives of the presence of IFIs and the EU in beneficiary countries are different; therefore full coherence among the instruments would not be possible.

In what concerns the internal coherence, stakeholder consultation suggests that the MFA has been developed in a consistent and clear manner, so that the implementation follows the same set of principles and rules in all beneficiary countries.

EU added value

All consulted stakeholders recognised an EU added value of the MFA. According to them, a majority of the reforms implemented following the MFA conditionalities would not have taken place otherwise, or not as quickly – e.g. reforms in the area of public financial management, anti-corruption and judicial system. However, stakeholders have noted that in most cases, the scope and the overall impact of the MFA operations in the countries was relatively small in comparison to the total EU presence there.

Interviewed stakeholders have noted that, in most beneficiary countries, the visibility of the MFA was rather low. Lack of visibility affected the added value of the MFA, as the instrument was under-exploited for the promotion of EU economic diplomacy in beneficiary countries and their geographical surroundings.

Bibliography

Boughton, James M. and Mourmouras, Alex. 2002. Is policy ownership an operational concept? International Monetary Fund. [Online] April 2002. https://www.imf.org/external/pubs/ft/wp/2002/wp0272.pdf.

DG ECFIN. 2020. European Commission. Meta-evaluation of Macro-Financial Assistance (MFA). [Online] 9 September 2020. tbd.

ECJ. 2015. Action for annulment — Macro-financial assistance to third countries — Decision of the Commission to withdraw a proposal for a framework regulation. C‑409/13, s.l. : European Court of Justice, 14 April 2015.

Ecorys, Economisti Associati, Case. 2010. Ex post evaluation of MFA operations in Moldova. European Commission - Completed evaluations. [Online] 12 February 2010. https://ec.europa.eu/dgs/economy_finance/evaluation/pdf/moldova_eval_en.pdf.

Ecorys, Wiener Institut für Internationale Wirtschaftsvergleiche, Case. 2012. Ex-post Evaluation of Exceptional Financial Assistance Operation to Kosovo. European Commission - Completed Evaluations. [Online] 30 Septrember 2012. https://ec.europa.eu/dgs/economy_finance/evaluation/pdf/efa_kosovo_eval_final_report_oct_2012.pdf.

European Commission. 2018. A Modern Budget for a Union that Protects, Empowers and Defends The Multiannual Financial Framework for 2021-2027. EUR-LEX. [Online] 2 5 2018. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM%3A2018%3A321%3AFIN.

—. 2021. Background Analysis per beneficiary country. EUR-Lex. [Online] 8 July 2021. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52021SC0178.

—. 2011. Commission Staff Working Document accompanying the Proposal for a Framework Regulation for Macro-Financial Assistance. EUR-LEX. [Online] 4 July 2011. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52011SC0865.

—. 2011. Proposal for a Regulation of the European Parliament and of the Council laying down general provisions for Macro-Financial Assistance to third countries. EUR-LEX. [Online] 4 July 2011. https://eur-lex.europa.eu/legal-content/en/ALL/?uri=CELEX:52011PC0396.

—. 2021. Report from the Commission to the European Parliament and the Council on financial instruments, budgetary guarantees, financial assistance and contingent liabilities - Situation at 31 December 2020. EUR-LEX. [Online] 5 November 2021. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52021DC0676.

European Parliament. 2012. Report on the proposal for a regulation of the European Parliament and of the Council laying down general provisions for Macro-Financial Assistance to third countries. European Parliament. [Online] 3 May 2012. https://www.europarl.europa.eu/doceo/document/A-7-2012-0157_EN.html.

GHK Consulting Limited & European Policy Evaluation Consortium. 2009. Completed Evaluations. European Commission - Completed Evaluations. [Online] 2 October 2009. https://ec.europa.eu/dgs/economy_finance/evaluation/pdf/final_report_meta_en.pdf.

ICF International. 2015. Ex-post Evaluation of the EU’s Macro Financial Assistance to Bosnia and Herzegovina (2009-2013) . European Commission - Completed Evaluations. [Online] January 2015. https://ec.europa.eu/dgs/economy_finance/evaluation/pdf/ex_post_evaluation_of_mfa_bosnia_summary_en.pdf.

ICF International, Cambridge Econometrics, Institute for Economic Research and Policy Consulting. 2017. Ex-post evaluation of Macro Financial Assistance Operations I & II to Ukraine. European Commission - Completed Evaluations. [Online] November 2017. https://ec.europa.eu/info/sites/default/files/economy-finance/evaluation_of_mfa_i_and_ii_to_ukraine_-_final_report_ecfin.pdf.

ICF International, GHK, Cambridge Economics. 2013. Ex-post Evaluation of Macro Financial Assistance (MFA) operation to Armenia. European Commission - Completed evaluations. [Online] October 2013. https://ec.europa.eu/dgs/economy_finance/evaluation/pdf/mfa__armenia_evaluation_final_report_en.pdf.

IEO. 2007. Structural Conditionality in IMF-Supported Programs. Independent Evaluation Office of the International Monetary Fund. [Online] 2007. https://ieo.imf.org/-/media/IEO/Files/evaluations/completed/01-03-2008-an-ieo-evaluation-of-structural-conditionality-in-imf-supported-programs/01032008sc-main-report.ashx.

IMF. 1944. Articles of Agreement of the International Monetary Fund. International Monetary Fund. [Online] 22 July 1944.

—. 2005. Review of the 2002 Conditionality Guidelines. International Monetary Fund. [Online] 3 March 2005. https://www.imf.org/external/np/pp/eng/2005/030305.htm.

Johnson, Omotunde E.G. 2005. Country Ownership of Reform Programmes. United Nations Conference on Trade and Development. [Online] January 2005. https://unctad.org/system/files/official-document/gdsmdpbg2420052_en.pdf.

Klein, Michael and Hartford, Tim. 2005. Grants or Loans? Development Finance and Incentive Effects. Worldbank Open Knowledge Repository. [Online] March 2005. https://openknowledge.worldbank.org/handle/10986/11230.

Marc, Francois. 2017. Rapport d'Information fait au nom de la commission des finances sur l’assistance macrofinancière de l’Union européenne aux pays tiers. Sénat. [Online] 28 June 2017. https://www.senat.fr/rap/r16-590/r16-5901.pdf.

OECD - DAC Working Party on Development Finance Statistics. 2018. Converged Statistical Reporting Directives for the Creditor Reporting. Organisation of Economic Cooperation and Development. [Online] 28 May 2018. https://one.oecd.org/document/DCD/DAC/STAT(2018)9/FINAL/en/pdf.

Phillips, Steven T, et al. 2013. The External Balance Assessment (EBA) Methodology. IMF. [Online] 31 December 2013. https://www.imf.org/en/Publications/WP/Issues/2016/12/31/The-External-Balance-Assessment-EBA-Methodology-41200.

Valdani, Vacari e Associati & Centre for Strategy and Evaluation Services, Oxford Analytics. 2021. European Commission. [Online] 8 October 2021.

Von der Leyen, Ursula. 2019. A Union that strives for more - My agenda for Europe. European Commission. [Online] 2019. https://ec.europa.eu/info/sites/default/files/political-guidelines-next-commission_en_0.pdf.

(1) OJ L 165, 27. 5.2020, p. 31.

(2)  

(3) MFA Decisions, the Financial Regulation applicable to the general budget of the European Communities and Better Regulation Guidelines.

(4)  

(5)  European Economic Community until 1993, then the European Community from 2010.

(6)  Decision No 778/2013/EU of the European Parliament and of the Council of 12 August 2013 providing further macro-financial assistance to Georgia (OJ L 218, 14.8.2013, p. 15).

(7) This distinction is political rather than legal. The adoption of the 2013 Joint Declaration and the Council Conclusions from 2002 reflects the co-legislators’ shared understanding that MFA is not part of the EU’s standard external financial cooperation toolbox.

(8) The IMF’s Financing Assurances Policy requires a recipient’s projected balance-of-payments position, improved as needed through external assistance, to be compatible with an assumed exchange rate that does not unduly jeopardise the repayment (repurchase) of IMF support. The required degree of certainty of these projections varies with the time horizon.

(9) This was also the case for Ukraine 15 (see table), pursuant to Article 213 TFEU.

(10) As Table 3 shows, the difference is mainly due to a non-disbursement of assistance under operation Ukraine 14. The operation expired on 13 January 2018 without the third and final disbursement having been released, given the lack of sufficient progress in fulfilling some of the required conditions (Commission Decision EU/2018/405 final).

(11) The 9% provisioning is paid from the EU budget into the Common Provisioning Fund after the loan is disbursed to the beneficiary country.

(12)  To date, no calls on the External Action Guarantee Fund have been recorded for MFA operations.

(13)  

(14) The Commission Proposal (COM(2018) 460 final) states: […] the External Action Guarantee will support EFSD+ operations, macro-financial assistance and loans to third countries on the basis of Council Decision 77/270/Euratom, for a proposed amount of up to EUR 60 billion, out of which EUR 14 billion are envisaged to cover macro-financial assistance loans.

(15) This is why analysts such as IMF staff and Commission services calculate structural – meaning sustainable current account – deficits for the economies of their interest, controlling for a certain set of econometric determinants .

(16)  .

(17) A precondition is that the beneficiary respects effective democratic mechanisms, including a multiparty parliamentary system and the rule of law, and that respect for human rights is guaranteed.

(18)  

(19) This particular case dates from before the Lisbon Treaty introduced the requirement to apply the OLP approval procedure. This nevertheless case demonstrates that delays (irrespective of their cause) can inhibit MFA’s effectiveness.

(20)  

(21)  Including, .

(22)  

(23)  

(24)  ,

(25)  

(26)  

(27) Emerson M., EU Macro-Financial Assistance: A Critical Assessment, Centre for European Policy Studies - CEPS, Belgium, February 2012, and p. 39.

(28)   Evaluation reports on economic and financial affairs policies and spending activities | European Commission (europa.eu)

(29)

 ‘Adjustment and Growth in IMF-supported Programmes’, Draft Issues Paper for an Evaluation by the Independent Evaluation Office (IEO), September 2019.

(30)  A first external meta-evaluation was published in 2009. The exercise covered MFA operations evaluated between 2004 and 2008: Microsoft Word - MFA Meta Evaluation - FINAL Report.doc (europa.eu)

(31) SWD(2021) 305 final

(32)  Evaluation questions detailed in annex III

(33) SWD(2021) 305 final

(34) The meta-evaluation roadmap ran from 21 January 2021 to 18 February 2021.