EUROPEAN COMMISSION

Brussels, 9.4.2021

SWD(2021) 73 final

COMMISSION STAFF WORKING DOCUMENT

Evaluation Report

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

Evaluation of Decision N° 601/2015/EU of the European Parliament and of the Council of 15 April 2015 providing macro financial assistance to Ukraine

{COM(2021) 169 final}

Table of contents

Glossary2

1.Introduction3

   Purpose of the evaluation    3

   Scope of the evaluation    3

2.Background to the intervention4

   Economic situation    4

   Description of the intervention and its objectives    5    

   Points of comparison    7    

3.Implementation / state of play8

Implementation of the MFA operation    8    

 Economic situation    9    

   MFA-III and MFA-IV    10

4.Methodology11

   Evaluation techniques used    11    

   Risks and limitations    12    

5.Analysis and answers to the evaluation questions13

   Evaluation Question 1: Relevance of the operation    13    

   Evaluation Question 2: Effectiveness    15

   Evaluation Question 3: Efficiency of the operation    17

   Evaluation Question 4: EU added-value of the operation    18    Evaluation Question 5: Coherence of the operation    19

   Evaluation Question 6: Social impact of the operation    20

   Evaluation Question 7: Public Debt Sustainability of the operation    21

6.Conclusions23

Annex 1: Procedural information27

   Organisation, design and timing    27        

Annex 2: Stakeholder consultation strategy28

    Mapping of stakeholder groups                         28        

   Methods and tools for engaging with stakeholders             29

ANNEX 3: METHODS AND DATA SOURCES32

Annex 4: Timeline of the UKRAINE MFA-III operation……………………….…………….38

Glossary

Term     Meaning or definition

1.INTRODUCTION

Purpose of the evaluation

This Staff Working Document (SWD) presents an evaluation of the third Macro-Financial Assistance (MFA-III) operation to Ukraine, provided by the European Union (EU) over the period 2015-2017. It largely draws on an independent, ex-post evaluation 1 conducted by an external contractor, and a consultation process that involved targeted stakeholders in Ukraine as well as EU staff.

The aim of the evaluation is to assess whether the MFA-III operation met its objectives to support Ukraine in addressing its balance-of-payments problems and implementing economic and structural reforms that would stabilise its economy and enhance the sustainability of its external position. The purpose of the evaluation is to support decision-making by identifying areas of improvement for similar on-going or future MFA operations, while also ensuring transparency and accountability.

Scope of the evaluation

MFA is a policy-based financial instrument of untied and undesignated 2 balance-of-payments support to partner countries. It is designed to assist third countries that are geographically, economically and politically close to the EU. MFA takes the form of medium/long-term loans, grants or a combination of the two. Unlike other, regular forms of aid provided by the EU, MFA is exceptional in nature and is mobilised on an ad-hoc case-by-case basis. Its objective is to help restore a sustainable external financial situation, while encouraging economic adjustment and structural reforms in the partner country. MFA always complements (and is conditioned on) financing provided in the context of a reform programme agreed with the International Monetary Fund (IMF).

This evaluation covers the third MFA programme for Ukraine, implemented in 2015-2017. Two preceding MFA operations for this country, disbursed in 2014 and early 2015 for a total of EUR 1.61 billion, were covered by a separate evaluation 3 .

In the context of a deep economic recession and a severe confidence crisis due to the protracted armed conflict in the Eastern part of the country, Ukraine requested additional financial assistance from the EU and other international partners in late 2014. In April 2015, the European Parliament and the Council adopted a decision 4 to provide EUR 1.8 billion of MFA-III to Ukraine as part of a wider package of international assistance. The objectives were to (i) alleviate Ukraine’s external financing constraints, balance of payment and budgetary needs, which remained sizable even after the effects of MFA-I and II 5 and other donor programmes; and (ii) to support the ambitious reform programme initiated under the first two MFAs. The assistance was to be disbursed in three instalments of EUR 600 million each 6 , alongside IMF assistance and contributions from other donors. Beyond some political and economic pre-conditions, disbursements were linked to the fulfilment of thirty-six specific structural reform conditions related to six policy areas, namely: public finance management; governance and transparency; business environment; energy sector; financial sector; and social safety nets.

In accordance with Article 34(1) of the Financial Regulation 7 , MFA operations in third countries are subject to an ex-post evaluation. In turn, the aforementioned MFA Decision for Ukraine stipulates that the European Commission is required to “submit to the European Parliament and to the Council an ex post evaluation report”.

To this end, the Directorate-General for Economic and Financial Affairs (DG ECFIN) engaged an external contractor to complete an independent assessment, which informs this Staff Working Document. The objectives of the ex-post evaluation were:

1.To analyse the impact of MFA-III on the economy of Ukraine and, in particular, on the sustainability of its external position;

2.To assess the added value of the EU’s intervention. In general, the evaluation seeks to draw lessons with respect to the EU’s financial assistance, i.e.

a.Whether the ex-ante considerations determining the design and terms of the operation were appropriate, taking due account of the economic, political and institutional context; and

b.Whether the outcome of the programme met the objectives.

These areas were assessed along the following key evaluation criteria: relevance, effectiveness, efficiency, EU added value, and coherence with both other EU policies and interventions from international donors. In addition, the evaluation assessed the social impact of the MFA and the impact on the sustainability of Ukraine’s public debt. This is further specified in the Evaluation Roadmap 8 .

2.BACKGROUND TO THE INTERVENTION

Economic situation

Following the Russian annexation of the Crimean peninsula, and due to conflicts in the eastern part of the country, the economic and political environment in Ukraine deteriorated sharply in 2014, resulting in a fully-fledged currency, banking and balance of payments crisis. Uncertainty and instability related to the confidence loss associated with the conflicts translated into restricted access to finance and drove the fall in investment activity and household consumption. As a result, real GDP contracted by 6.8% in 2014.

Weak economic activity, coupled with higher interest outlays on foreign currency denominated debt in light of strong currency depreciation, as well as sizable losses of tax collection in the eastern parts of the country, led to a widening of the budget deficit in 2014. The deficit worsened despite a number of austerity measures introduced by Ukrainian authorities in both March and July 2014. In the same year, a major additional drag on public finances came from the state-owned oil and gas company Naftogaz, whose activity was negatively affected by the depreciation of the hryvnia and the gas arrears to Russia. As a result, the State had to inject UAH 103 billion into Naftogaz by November 2014, an amount representing 6.8% of the yearly GDP. In this way, the public finance deficit run by Ukraine in 2014, including the deficit of Naftogaz, exceeded 10% of GDP, leading to a sharp deterioration of the country’s public debt metrics.

9 On the external side, the depreciation of the hryvnia, coupled with weak domestic demand, contributed to a significant adjustment of the current account at first. However, exports performance, despite benefitting from the weaker currency and the trade preference provided by the EU, was hurt by disruptions of the production chains and growing trade tensions with Russia. As a result, the current account deficit remained at 4% of GDP in 2014. This was accompanied by sizeable private sector financial outflows, due to dwindling confidence in an environment of high geopolitical uncertainty. In the context of a deepening economic recession and confidence crisis, the substantial official financial assistance provided to Ukraine in 2014 proved insufficient to offset capital flight and to stop the continuous drain on reserves. By December 2014, international reserves had dropped to USD 7.5 billion (corresponding to 1.8 months of imports only).

Description of the intervention and its objectives

In this difficult macroeconomic context, in April 2014, Ukraine entered into a USD 17 billion (800% of quota), two-year Stand-by Arrangement (SBA) with the IMF. Subsequently, in March 2015, a four-year Extended Fund Facility (EFF) of USD 17.5 billion was signed between the two parties, replacing the preceding SBA. To complement Ukraine’s arrangements with the IMF, in 2014 and early 2015, the EU pledged two macro-financial assistance (MFA I and II) programmes, of a combined amount of EUR 1.61 billion.

Given the worsening economic situation and continuing conflict in the eastern Ukraine, the Ukrainian authorities requested in late 2014 a third MFA. Upon an ex-ante evaluation 10 , the European Commission proposed 11  in January 2015 up to EUR 1.8 billion in MFA loans. The European Parliament and the Council adopted the related Decision 12 on 15 April 2015. The EU’s contribution through the third MFA was an essential part of programme financing to cover the identified financing gap under the IMF programme; without it, the gap would not have been filled and the IMF could not have proceeded with its own programme.

Following the consultation of the Member States' Committee, a Memorandum of Understanding (MoU) 13 and Loan Facility Agreement related to this assistance were signed, and the ratification by the Ukrainian Parliament took place in July 2015. Annex 4 of this report summarises the timeline of the operation.

The assistance was meant to be provided in three tranches of EUR 600 million each, disbursed between July 2015 and December 2017. Disbursements were conditional to good progress under the IMF's EFF, political prerequisites (the respect of human rights, effective democratic mechanisms, including a multi-party parliamentary system, and the rule of law), and to the fulfilment of the set of policy conditions specified in the Memorandum of Understanding. However, the MFA-III operation expired on 13 January 2018 without the third and final disbursement having been released, given the lack of sufficient progress in the implementation of some of the required conditions. The Commission formalised the cancellation in a decision dated 18 January 2018 14 .

The general intervention logic, which illustrates the objectives and expected results of the MFA-III operation, applicable to the 2015-2017 Ukraine programme, is summarised in the graph below.

Figure 2.1 Intervention logic of MFA operation

Source: ICF

Points of comparison

The points of comparison, against which the MFA operation of 2015-2017 is assessed, refer to the situation in Ukraine (1) prior to the intervention, on the one hand, and (2) during and immediately after the implementation of the MFA operation, on the other hand. As noted in the previous subsection, which describes the economic situation prior to the MFA operation, particular attention is paid to the external sector and the fiscal situation of the Ukrainian economy. The following section will describe the implementation of the    MFA-III operation, looking at the key developments in the policy reform areas supported by the programme, as well as in the economic situation of the country.

3.IMPLEMENTATION / STATE OF PLAY

Implementation of the MFA operation

To achieve the objectives detailed in section 1, and as per the usual MFA procedure, disbursements under this operation were tied to the fulfilment of the political pre-conditions 15 , as well as good progress with the implementation of the IMF programme. The disbursement of the second and third tranches was also subject to the fulfilment of a set of thirty-six country-specific policy conditions, specified in the MoU and related to six structural reform areas 16 .

MFA loan was provided on highly favorable terms. The first instalment was characterised by a coupon of 0.25% and a maturity of 5 years, while the second instalment had a coupon of 0.75% and a maturity of 14 years. For both tranches, a bullet capital repayment (i.e. a lump sum of the full outstanding amount) was envisaged in the last year of maturity. With respect to the interest rates, no grace period was applied.

Ukraine satisfactorily fulfilled all the policy conditions necessary for the release of the second instalment, with the exception of conditions 12 (on social safety nets) and 13 (on trade and customs), for which waivers were granted. These two conditions required Ukraine to: (1) ensure the effective provision of social benefits and services to internally displaced persons (IDPs) through adequate legislation and funding; and (2) to refrain from introducing new trade-restricting or trade-distorting measures, in line with Ukraine’s World Trade Organization (WTO) obligations. With respect to the first condition, the waiver concerned the suspension of social payments to around 500,000 IDPs living in the non-government controlled areas in Eastern Ukraine. Meanwhile, the waiver for condition 13 concerned a wood export ban 17 adopted by Ukraine in April 2015, which went against the standstill commitment on trade-restricting measures.

However, the third and last tranche was not released, due to four conditions that were not met by the Ukrainian government before the expiration of the MFA programme. More specifically:

ŸMeasure 5, on anti-corruption, whereby Ukraine committed to setting up an electronic asset disclosure system for public officials and a related verification mechanism. Ukraine did not fulfil this important condition, as the required verification mechanism was not established.

ŸMeasure 6, still on anti-corruption, whereby Ukraine committed to (i) putting in place mechanisms to verify (post-registration and following clear criteria) the accuracy of the information provided by companies; and (ii) enforcing compliance with this obligation. The country did not fulfil this condition, since the required system to ensure the verification of information submitted by companies was not implemented.

ŸMeasure 17, on trade and customs, linked to the preceding condition 13. Ukraine did not fulfil this condition, as it did not repeal the export ban on unprocessed timber of November 2015 and did therefore not refrain from introducing new trade-restricting measures in line with its obligations under the agreements of the WTO.

ŸMeasure 21, on the financial sector, whereby Ukraine committed to establishing a central credit registry. The country did not fulfil this obligation, since the necessary legislation to establish a central credit registry was not adopted.

The non-disbursement decision heavily depended on the two unmet conditions on anti-corruption. The main reasons for not implementing the two measures within the agreed timeframe were: (1) a lack of political will; (2) entrenched bureaucracy, which slowed the process; and (3) vested interest of some stakeholders. No evidence has been found regarding negative impacts on our bilateral relations originating from the non-disbursement. Indeed, the collaboration between the Commission and the Ukrainian authorities remained positive and constructive, with greater focus of the Ukrainian authorities on the implementation of the conditions and a more honest policy dialogue between the two sides.

Therefore, in light of the flexibility already granted with the two previous waivers and the prominent role given by the EU to transparency and the rule of law, the EU’s credibility appeared to be at stake and the non-disbursement decision is deemed justified.

Some achievements in the policy areas tackled by MFA’s conditionality deserve particular note. In the area of public finance management, MFA-III policy conditionality supported the implementation of an online public procurement platform (ProZorro) by 2016, ensuring open access to public procurement in Ukraine and increasing transparency in public finance management. Regarding governance and transparency, MFA conditions have helped in the adoption of legal foundations for establishing independent and professional supervisory boards at state-owned enterprises. With respect to the energy sector, thanks to MFA-III and other international donor programmes, the sector started to positively contribute to the State budget, reversing the negative trend recorded between 2011 and 2014.

Economic situation during the implementation of MFA-III

After a cumulated GDP contraction of about 16% in 2014-2015, Ukraine returned to growth in 2016 (+2.4% of GDP) as investment activity and household consumption started recovering. The recovery continued into 2017 and 2018, with GDP growth amounting to 2.5% and 3.3%, respectively. Inflation also started to decelerate. However, the depreciation of the local currency in the final months of 2017 contributed to soaring prices, resulting in inflation of 13.7% in December 2017, exceeding the 10% upper band of the central bank's inflation target. In order to tame inflation, the central bank reversed the monetary easing cycle in place since March 2015. It raised its policy interest rate by a cumulative 4.5 percentage points since October 2017, to 17% as of 2 March 2018.

With respect to the external sector, plummeting domestic demand, partly reflecting the sharp currency depreciation following the abolishment of the pegged regime by the National Bank of Ukraine, played a central role in reducing an unsustainable trade deficit and a rebalancing of the current account in 2015. However, Ukraine’s current account deficit started to widen gradually as of 2016 as a result of recovery in investment imports (reflecting growing business confidence) and domestic consumption. Nonetheless, a further widening of the current account deficit was contained by the parallel strong rebound in exports (17% year-on-year in 2017) and a gradual increase in remittances from Ukrainians working abroad. The support from Ukraine's multilateral and bilateral partners, coupled with a current account adjustment and a gradual return of private financial flows, helped Ukraine replenish its international reserves to USD 18.8 billion at the end of 2017 despite weakness in FDI.

Ukraine has also made significant progress in the consolidation of its public finances. The overall fiscal deficit was reduced from 4.5% of GDP in 2014 to 2.3% in 2016 (compared with the 3.7% deficit target agreed for 2016 under the IMF programme). Budget execution continued to be strong in 2017, also due to robust tax collection and rising dividend payments from state-owned enterprises. As a result, the consolidated state budget deficit in 2017 amounted to 1.4% of GDP. Thanks to the overall policy of fiscal consolidation, Ukraine also registered a reduction in gross government debt to an estimated 61.5% of GDP at the end of 2017 (down from the 69.2% of GDP at end-2016).

Figure 3.1 Selected macro-economic indicators for Ukraine

Sources: National Bank of Ukraine, Ministry of Finance of Ukraine

MFA-III and MFA-IV

The non-disbursement of the third tranche of MFA-III did not create an obstacle for the implementation of other EU programmes to Ukraine. Cooperation continued to be constructive reflecting also that a new MFA-IV operation of up to EUR 1 billion was proposed in March 2018, two months after the cancellation of the third disbursement under MFA-III. The operation was meant to accompany the IMF’s SBA operation of USD 3.9 billion, which was going to follow the Extended Fund Facility ending in December 2018. The EU adopted the fourth MFA in July 2018. To underline the importance of anti-corruption measures and safeguard the credibility of policy conditionality, the first disbursement under MFA-IV was only made in December 2018, after Ukraine fulfilled specific policy measures in relation to the unmet conditions of the previous programme. Notably, as part of the implementation effort of MFA-IV first tranche conditions, Ukraine made significant progress in the areas of anti-corruption policy 18 .

4.METODOLOGY

Evaluation techniques used

The methodology for evaluating the MFA operation in Ukraine over the period 2015-2017 was guided by the Commission’s Better Regulation Guidelines 19 and the Guidelines for the Ex-Post evaluation of Macro-Financial Assistance Operations 20 .

This evaluation was supported by an assessment carried out by an external contractor from December 2019 until July 2020 and was overseen by an Inter-Service Steering Group (ISG). The external evaluation comprised of the following four phases: (1) inception, (2) data collection, (3) analysis and judgement, and (4) finalization and feedback.

Evidence and the data were collected from a wide range of sources through a number of complementary quantitative and qualitative approaches aligned to the consultation strategy, including:

(I)Desk research and macroeconomic data analysis of publically available documents from a range of sources (European Commission, IMF, World Bank, National Bank of Ukraine and Ukrainian Ministry of Finance);

(II)Forty interviews with key informants, including representatives of the EU and Member State political landscape, Ukrainian national authorities and other stakeholders involved in the implementation or monitoring of MFA conditionality, as well as IFIs, the wider donor community, civil society organisations (CSOs) and businesses representatives. Interviews were also carried out with the European Commission (both in headquarters and the EU Delegation to Kyiv);

(III)A Delphi survey with a panel of experts. Twenty-one respondents provided valid feedback with a high degree of consensus amongst replies. Further details on the Delphi survey (which focused on the macroeconomic impact of MFA-III, the impact of structural reforms and design and implementation aspects of the operation) are included in Annex 2 21 .

(IV)An online focus group with attendees from Ukrainian CSOs, think tanks and academia. Further details on the focus group are included in Annex 2 22 ;

(V)Two in-depth case studies on anti-corruption reform and social safety net reform as well as a third case study on the use of the euro in three MFA operations in Ukraine;

(VI)A stakeholder validation workshop to test and validate the emerging findings.

Further information on the evaluation techniques is available in Chapter 4 of the external evaluation report. Overall, triangulation of findings, obtained using different techniques, has helped to increase validity of the evaluation results. The methodology employed is comprehensive and responded to the very specific and unique nature of the MFA operation 23 .

Risks and limitations

The overall reliability and validity of the evaluation is strong. Nonetheless, a number of methodological limitations and challenges affected the evaluation:

·While generally data coverage and quality is good, data referring to some key national statistical indicators have been frequently updated recently, and some gaps exists.

·Another limitation relates to the reliability of the judgment provided by some stakeholders. In a number of cases interviewees (not directly involved in the operation) had limited knowledge of specific MFA-III conditionality or were unable to recall the details of the operation and the general context. Additionally, the online format used for key stakeholder consultation, Delphi panel and focus group reduced the active participation and response rate of some actors involved.

·As occurred for previous MFA evaluations, the changing economic environment over an extended period and the fact that the MFA-III operation was implemented in parallel with IMF and other international support programmes made it difficult to disentangle the impact of the MFA from the impacts resulting from other factors. As a consequence, the counterfactual analysis was somewhat limited.

Nonetheless, the identified risks and limitations do not put into question the overall reliability of the evaluation analysis, as they were mitigated by the fact that information was obtained from a wide range of sources, using different evaluation techniques, alternative scenarios and multiple rounds of feedback. Therefore based on the methods employed, the conclusions reached in the evaluation are considered to be reliable and valid.

5.ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS

Relevance

1.1 The amount of the EU contribution under the first two tranches of MFA-III to Ukraine corresponded to circa 0.7% and 0.6% of the country’s GDP in 2015 and 2017, respectively. In absolute terms, this was, by far, the largest single operation in the history of the MFA instrument, justified by: (1) the financing needs of the country at that time; (2) the size of the country’s economy; (3) the important role of Ukraine for the stability of the European Neighbourhood; and (4) the importance attached to the integration of the country with the EU (in the context of the Association Agreement signed in 2014, following the “Revolution of Dignity”). The envelope covered 5% of Ukraine’s financing gap for the whole period 2015-2017 24 , compared to the 7% initially envisaged 25 . Such an unprecedented amount of MFA has further increased the importance of a careful design of the conditionality package and the implementation framework. However, some shortcomings were found in the design of the conditionality, as will be highlighted below.

Everything considered, the EU contribution is deemed appropriate (in terms of burden sharing with other donors) and proportional (necessary to achieve short-term macroeconomic stability in Ukraine).

1.2 The timing of MFA-III was particularly relevant to ease Ukraine’s financing needs and balance of payments constraints. In particular, the disbursement of the first tranche in July 2015 is deemed crucial for the country, given the high vulnerability of the Ukrainian economy at that time and the almost depleted foreign reserves. The EU decision-making process for MFA-III was extremely fast (the proposal submitted in January was adopted in April, with the MoU signed in May 2015). The second tranche disbursed in 2017 was not as critical, as the economy had begun to recover.

Regarding the form of the operation, the MFA financing was provided in the form of EUR 1.8 billion loan, on highly concessional terms that could not have been directly obtained on the market. The initial assessment of the main factors influencing the form of MFA instrument made the Commission excluding the possibility of a grant component 26 . Therefore, the adopted form (entirely loan) is considered and confirmed as appropriate. It was complementary to the EU grant support provided elsewhere by the EU to Ukraine in 2015-2017 totalling EUR 840 million.

1.3 With respect to policy conditionality, MFA-III contained 36 conditions (46 if sub-conditions are counted) related to six policy areas: (i) public finance management; (ii) governance and transparency; (iii) energy; (iv) financial sector; (v) social safety net; and (vi) business environment. Given the relatively short amount of time allowed for the implementation of the reforms 27 , and given the level of ambition of most of the conditions concerned, it is believed that the number of reform conditions should have been lower. Although the decision to adopt such an unprecedented number of conditions may reflect the exceptional size of the assistance (more money for more reforms), it also distracted the authorities from focus on key priorities and required substantial human resources for the implementation of these conditions. The non-disbursement of the third tranche was partially explained by the high number of conditions, as well as the design of the targets itself. Some reforms could have been broken down into smaller steps, envisaging a gradual approach for reform implementation, given the level of ownership and capacity of the Ukrainian authorities. This was particularly the case for condition 5 on asset declaration, and condition 6 on beneficial ownership. The high level of ambition of the conditionality is nonetheless considered appropriate in most areas.

The thematic areas addressed by MFA-III conditionality were relevant. Many of them were derived from the EU-Ukraine Association Agreement and thus aligned with the country’s priorities. However, the conditions related to avoiding an introduction of new trade-distorting measures proved problematic; trade-related disputes should and could be solved through trade-specific channels and not as part of MFA.

Overall, the design of MFA-III programme, both in terms of financing envelope and focus of reforms, was highly relevant in view of the targeted objectives. The unprecedented size of the disbursed assistance was crucial to support Ukraine in recovering from the economic crisis. The reform areas supported by the programme were highly relevant to the country's needs and well aligned with the country’s priorities. However, the high number of conditions and level of ambition, coupled with the relatively short time horizon for implementation, reduced Ukraine’s ability to deliver on all the policy reforms and limited the efficiency of the overall financial intervention. More sparing use of conditionality, with reforms milestones broken into smaller targets, might have sustained a better motivation and feasibility throughout the operation.

Effectiveness

2.1. The financial support provided under MFA-III has strongly contributed to the improvement of Ukraine’s economic performance from 2016. As described in section 3, Ukraine returned to modest growth in the first half of 2016, started to consolidate its public finance and regained access to the international debt markets. The majority of the experts interviewed in the evaluation study believe that the macroeconomic outcomes would have been less favourable in the absence of MFA-III. In particular, the highly favourable terms of the loan allowed for a reduction in the cost of public debt, as compared to alternative sources of financing. The magnitude of the intervention allowed the Ukrainian government to pursue fiscal savings, without burdening public spending cuts and creating room for much-needed structural reforms. MFA conditionality helped in identifying key priorities for fiscal governance, guiding Ukraine in its recovery path.

The results of the counterfactual analysis suggest that in the absence of the MFA’s first tranche in 2015, when Ukraine had no access to international debt markets, the most plausible course of actions for Ukrainian government would have been fiscal adjustment in the form of drastic expenditure cuts. Domestic borrowing possibilities were severely restricted by the ongoing restructuring of the banking sector; raising revenue through higher taxes and/or privatisation would have been similarly impossible. Likewise, increased financing from the IMF and World Bank would have been highly unlikely. However, substantial cuts in public spending could have jeopardised the country's path to economic recovery, with likely repercussions in terms of social tensions and political consensus.

Had the second tranche in 2017 not been disbursed, the analysis shows that the most plausible course of action would have been to raise the required financing from the domestic debt market, which had already recovered by then. The additional cost of debt, stemming from the higher interest rates in domestic borrowing compared to MFA financing, would have reached USD 30 million.

Overall, under a scenario assuming no MFA-III and no IMF support to Ukraine, due to lack of access to international markets and very limited domestic funding possibilities, Ukraine would have most likely defaulted on its debts in 2015-2017, with hard-to-predict but certainly disruptive economic, social and political implications.

Beyond financial contribution, MFA and other international donor programmes played a key role in renewing investors’ confidence and improving the country’s rating. This allowed Ukraine to regain access to international debt markets by 2017, which would have been unlikely in the absence of the above-mentioned interventions. Improved business confidence contributed to a strong rebound in investment activity and domestic demand, leading to economic recovery. Furthermore, the support granted by the EU and other international partners, coupled with a current account adjustment and a gradual return of private financial flows, helped Ukraine replenish its international reserves to USD 18.8 billion at the end of 2017.

2.2 With respect to the short and medium-term structural effects of the MFA operation of 2015-2017, there has been tangible progress beyond the fulfilment of the specific MFA-III conditions. Firstly, the energy sector is no longer causing budgetary problems to the Ukrainian government, helping the country to get on a path towards greater fiscal sustainability. Indeed, before 2014, the state-owned Naftogaz Group was a "black hole" in the Ukrainian government budget, with losses amounting to circa 6.2% of the country's GDP. In 2019, the restructured Naftogaz group accounted for nearly 16% of total revenues of the State budget 28 , registering a remarkable reverse of trend since 2014.

In the field of public finance management, the e-procurement system, ProZorro, has been widely acknowledged as a success, both domestically and internationally. Several steps have also been achieved in the field of taxation, including significant reduction of VAT refund arrears.

Some key milestones have also been achieved in the fight against corruption. Prior to 2015, Ukraine did not have dedicated institutions to fight corruption. The set-up of a new institutional architecture has therefore been a major undertaking and a breakthrough in constructing a systemic approach to tackle this phenomenon in the country. Given the early stage of Ukraine’s democratic transition, conditionalities agreed with the Ukrainian government required a split of competences among separate institutions, rather than opting for a single, central anti-corruption body. As part of MFA conditionality, the National Anti-Corruption Bureau (NABU), the National Agency for Prevention of Corruption (NAPC) and the Specialised Anti-Corruption Prosecution Office (SAPO) were established. Overall, NABU can be seen as a successful example of a broadly independent body, despite some limitations. On the contrary, the other two institutions still lack full political independence. An electronic asset declaration system for public officials also entered into force in 2016, but the actual verification initially lacked credibility, scale and speed. An automatic verification mechanism to check the declarations started operating only in 2019, as part of MFA-IV.

Overall, the EU MFA has proven to be essential for a consolidation of the country's balance-of-payments and public budget. The magnitude of the intervention allowed for fiscal savings, necessary for the implementation of needed structural reforms. The simultaneous commitment of the EU and IMF has also allowed a renewed increase in investors and business confidence. With regard to structural reforms, MFA-III was effective in promoting a variety of measures in the areas of public finance management, energy policy, business environment, financial sector, social policy and anti-corruption, although some of these reforms required subsequent efforts at a later stage.

Efficiency

3.1 To answer to this question, some key issues need to be addressed: (i) the ownership of the programme by the Ukrainian authorities; (ii) their capacity to undertake reforms; (iii) the effectiveness of monitoring activities; (iv) the effectiveness of the dialogue among concerned parties; and (v) the visibility of the MFA operation.

The level of ownership of the reforms underpinning the MFA operation turned out to be more fragile than initially anticipated by the European Commission and other donors between 2015 and 2018. Ownership was uneven across the various Ukrainian authorities and bodies in charge of the implementation of the specific conditions: while it was consistently high for the Ministry of Finance and the National Bank of Ukraine, some of the other institutions were much less involved. The factors explaining this drop in ownership were the resistance against some of the reforms from vested interests, also in the Ukrainian Parliament, as well as the uneven performance and frequent personnel changes in many parts of the Ukrainian public administration. In many areas, however, reforms were promoted thanks to the diligent monitoring and the continuous pressure applied by the Ukrainian civil society on the successive governments.

Even where present, capacity issues did not seem to cause the lack of progress on key reforms, as confirmed by the external evaluation report. However, the pressure placed on the Ukrainian authorities, due to the high number of conditions required in a limited time period, is believed to have caused an overloading of Ukrainian institutions. This may have led to consequent delays in some of the measures to be taken. On the other hand, the good collaboration among Ukrainian authorities and with the EU counterparts has been instrumental for a common understanding and commitment to the conditionality, as well as for a proper management of disbursements.

Regular progress reports submitted by Ukrainian authorities, missions of DG ECFIN staff to Kyiv, and exchanges with experts from the EU Delegation constituted the primary tool for monitoring the MFA operation. Given the complementarity with the conditionality of other donor programmes, there was regular and close collaboration with other international actors (in particular with the IMF). This contributed to a proper conceptualization and implementation of the monitoring activity.

Finally, the visibility of the MFA-III proved to be limited, as was the case for other MFA operations. To promote MFA-III, the Commission envisaged press conferences and press releases at the time of (i) the approval of the operation, (ii) the signature of the MoU, and (iii) the subsequent disbursements. The press releases described general budget allocations; particular conditions included in the operation; progress made by Ukraine in achieving agreed milestones; and the release of MFA disbursements. Contributing factors to a low level of MFA-III awareness may include: (i) complexity of public finance and macroeconomic issues for the general public and the way it was communicated; (ii) lack of publicity on MFA conditionality, financial aspects of the support and impacts of the programme; and (iii) confusion of MFA-III with other EU and IFI programmes. Feedbacks gathered from the key consulted stakeholders on this matter, during the external evaluation, confirmed the limited visibility. These findings highlight scope for improvement for future EU interventions.

Overall, the efficiency of the MFA-III operation was somewhat limited by the design of the conditionality package and other parallel shortcomings, related to reduced ownership and administrative capacity. The resistance from vested interest played a crucial role in the delays of many policy reforms, having relatively strong leverage over Ukrainian authorities. While there has been good collaboration between the EU and Ukrainian counterparts on the implementation process, the visibility and awareness of the conditions attached to MFA-III proved to be relatively limited.

EU added-value

The EU intervention primarily aimed to provide financing to Ukraine at a time of crisis when market-based sources of funding were inaccessible. MFA was an indispensable part of the overall external financial assistance package put in place by the international community in 2015. With the highly concessional terms of the loans, MFA also translated into fiscal savings for the government and a gradual adjustment of the public deficit. The financial added value of MFA operations also derives from the fact that the EU could quickly mobilise and coordinate a wider amount of resources, as compared to any other individual donor country 29 . Beyond the financial contribution, MFA conditionality granted a politically reinforcing effect that contributed to the sustained mobilisation of the authorities around crucial (and often new) reform areas. The anti-corruption field and public procurement reform are two areas where the EU stimulus was particularly helpful, as highlighted by the successful achievements reported before. Moreover, even for those areas in which the level of political ownership was already high (i.e. public finance management and financial sector), international support played a key role in facilitating the policy process at Parliamentary level.

Another (indirect) contribution of MFA-III relates to the increase of confidence among private investors. The MFA-III, together with other international donors’ support, helped to restore confidence in the economy, as confirmed by the majority of experts consulted in the external ex-post evaluation. The renewed confidence was also reflected in the appreciation of the hryvnia between February and mid-March 2015 (time when the IMF programme was negotiated and concluded and the parameters of MFA-III were finalized), and in the improvement of Ukraine’s rating 30 .

Overall, the ability of the EU to mobilise and coordinate significant financial resources at a relatively quick speed was a key rationale for this intervention, in a time in which Ukraine’s financing need was still extremely high. MFA conditionality granted a politically reinforcing effect that contributed to the sustained mobilisation of Ukrainian authorities around crucial reform areas. The programme also added value through its confidence-boosting effect on the private sector.

Coherence

5.1 The implementation period of MFA-III (2015-2017) coincided with the entry into force of the Association Agreement (AA) 31 , including a Deep and Comprehensive Free Trade Area (DCFTA). This agreement is the main tool for bringing Ukraine and the EU closer together, promoting deeper political ties, stronger economic links and respect for common values. The AA was signed in 2014; the DCFTA was applied since January 2016 and the full Association Agreement formally entered into force in September 2017. Title V, VI and VII of the AA outline a framework for an EU intervention in support to Ukrainian institutions, financial sector and economic activity. Mapping the conditions of the MFA-III against the short-term priorities established in the AA shows that MFA conditions were well aligned with the key priorities guiding EU-Ukraine relations.

5.2 MFA-III formed part of the wider package of EU support to Ukraine, including budget support, sectoral grant programmes and technical assistance that targeted largely the same areas as MFA-III (anti-corruption, public finance management, customs, public administration reform, energy). Over the period 2015-2017, EU deployed a total of EUR 840 million of grant support to Ukraine.

Out of this amount, EUR 150 million was disbursed as budget support (most of it as State Building Contract - SBC), on top of the EUR 265 million from SBC disbursed in 2014. At the time MFA-III was designed, the SBC programme was in place but the benchmarks for the second tranche were yet to be met. MFA-III contributed with a reinforcing effect on several key conditions, such as: the establishment of a verification system for asset declaration; the entry into force of the law on civil service; and the transparency around public procurement contracts.

5.3 MFA-III conditions were complementary to the reform packages included in the programmes of the IMF and the World Bank for Ukraine. For instance, in the field of public procurement (PP), the IMF introduced a condition calling for the adoption of a new PP law; the World Bank also urged the adoption of a new PP law reducing the exemptions in competitive procurements. MFA-III then followed up with the introduction of an e-procurement system.

In relation to reform of the National Bank of Ukraine (NBU), the EU put in place conditions on systemic banks and on related counterparties (MFA-III). The IMF tied its support to conditions related to internal control rules and governance and to autonomy framework of the NBU. This last condition was reinforced by the World Bank, which also introduced the implementation of a crisis management strategy (Development Policy Loan 1).

On energy sector reform, the IMF pushed for the adoption of a new Gas Market Law and MFA-III for the secondary legislation. In addition, both MFA and the World Bank supported a strengthening of the independence of the regulator. Some reforms were supported only by one of the donors: only MFA-III focused on social safety nets for IDPs, while monetary policy and pensions were only addressed by the IMF.

Overall, the measures of MFA-III were completely aligned with key principles and measures taken in the EU external actions towards Ukraine. The conditions attached to the programmes supported reforms in many of the areas covered by the AA/DCFTA between the EU and Ukraine. In addition, the operation was coherent with budget support commitments taken by the EU towards Ukraine, and with reform packages associated with the IMF’s and World Bank’s arrangements with the country.

Social Impact

6.1 The MFA operation included four conditions having a direct social impact. Two of them concerned a full functioning of the compensation schemes aimed at protecting poorer households from increasing energy prices, in view of the liberalisation of the energy market. Two further conditions covered an effective provision and evaluation of social assistance to internally displaced persons (IDPs). In order to cushion the social impact of the energy tariff increases introduced since 2014, the housing utility subsidies were introduced in 2017 with their targeting subsequently improved. Moreover, the social assistance programme for IDPs (the temporary assistance set up as an interim measure during the conflict in the East) was made permanent. However, the suspension of social payments to many IDPs has damaged the living standards of many Ukrainian citizens, and resumption process was slow and inefficient.

6.2 With respect to the indirect social impact of MFA operation, most of the social indicators such as average wages, households’ incomes and poverty rate have improved during and after the MFA-III operation; the unemployment rate slightly increased between 2015 and 2017, but went down from 2018. For instance, nominal wage growth and average household expenditure exceeded consumer price inflation in that period. Obviously, MFA-III was only one of many factors influencing the social situation in Ukraine.

As mentioned before, the results of the counterfactual analysis show that in the absence of the first tranche of the MFA, substantial reductions in government spending would have been necessary. In particular, the government would probably have responded by delaying increases in public pension payments (decreasing the value of these payments in real terms), negatively affecting household resources from 2015. An absence of the second tranche conditionalities would likely have had a negative impact on the pace and depth of reform of social safety nets. This would have affected the social situation in Ukraine, considering the need for compensation mechanisms to protect vulnerable households and the importance of the resumption of social benefits and services for IDPs. Increased domestic financing, with limited social impacts, would likely have offset the absence of budgetary support (provided by MFA).

Moreover, an absence of any support from the IMF and MFA would likely have created a sovereign debt crisis. Such a crisis typically blights the economy with issues such as a sharp economic recession with corresponding increases of the unemployment rate, high levels of inflation and/or cuts to public services and wages, all of which would have had a substantial negative effect on the social situation in Ukraine.

Overall, MFA-III had a positive social impact on Ukrainian population, strengthening the social safety net and protecting low-income households against higher prices resulting from energy reforms. The majority of social indicators also improved in this period.  

Public Debt Sustainability

7.1 

Figure 5.7.1 macroeconomic variables and debt indicators for Ukraine in 2014-2017

Sources: National Bank of Ukraine, Ministry of Finance of Ukraine; IMF; Dragon Capital. 

According to data, Ukraine’s debt-to-GDP ratio increased dramatically from 40% in 2013 to 70% in 2014, 80% in 2015 and 81% in 2016, before declining in the following years. Such a rapid increase in the ratio was largely driven by the depreciation of the hryvnia, as approximately 62% of Ukraine’s public debt was denominated in foreign currencies in 2014. As the hryvnia depreciated against the US dollar by 33% in 2014 and by a further 46% in 2015, foreign-currency denominated liabilities increased substantially in UAH terms. In addition, the combined public finance deficit, including the deficit of Naftogaz, exceeded 10% of GDP in 2014. The public gross financing needs relative to GDP were also impacted in 2015-2017. The ratio increased to 12.5% in 2015, primarily resulting from the currency depreciation, and it continued to rise in the following year, because of the increased financing needs related to resolutions and recapitalisations in the banking sector. Both indicators (debt-to-GDP and financing needs) improved in 2017 and the risk to debt sustainability moderated, thanks to the economic recovery, fiscal consolidation and the stabilization of the situation in the East.

MFA-III operation played a key role in smoothing Ukraine’s adjustment path. The substantial financial contribution provided at highly favourable terms (in a period in which Ukraine was not granted with access to international debt markets) allowed for fiscal savings, facilitating structural reforms and social spending. In the absence of the first tranche of MFA-III, the Ukrainian government would have been forced to make significant cuts in public spending. This would have led to an initial reduction in the amount of public debt, but could be politically difficult to accept and could jeopardize the upcoming reforms in the country. Had Ukraine not received the second tranche of MFA-III funding in April 2017, the most likely counterfactual - borrowing the equivalent amount from domestic debt markets to fill the financing gap - would imply additional interest payments estimated at some USD 30 million over the next period, but no major risk to debt sustainability.

In the scenario in which Ukraine did not receive funding from MFA-III nor the IMF, this would have severely impaired the country’s ability to attract funding from domestic and international debt markets and would have decreased the availability of assistance from sources such as the World Bank and the EBRD. As a result, Ukraine would most likely have defaulted on its debts over 2015-2017.

Overall, MFA-III had a positive direct and indirect impact on Ukraine's public debt sustainability through a variety of channels, including a signalling effect acting as catalyst for additional financial support and investor confidence. With the return of the economy to growth in 2016 and with quasi-fiscal deficits being nearly eliminated, Ukraine managed to put its debt ratio on a clear downward path and reduce its public debt-to-GDP ratio to 72% at the end of 2017.

6.CONCLUSIONS

In April 2015, the European Parliament and Council adopted a third MFA operation of EUR 1.8 billion to support Ukraine. The EU agreed on this operation relatively swiftly to support the country in addressing a growing balance-of-payment crisis, which was triggered by the economic and political crisis that followed the Russian annexation of the Crimean peninsula in 2014, and the subsequent conflicts in the eastern part of the country.

This Staff Working Document is informed by an external ex-post evaluation conducted by ICF, in collaboration with Cambridge Econometrics, which covered the relevance, efficiency, effectiveness, coherence, and EU value added of this MFA. It also explored the social impact of MFA and its effect on Ukraine’s public debt sustainability. Limitations encountered relate predominantly to the changing economic environment over the period during which the MFA-III took place, and the fact that MFA-III was implemented in parallel with other EU and international support programmes (making it hard to disentangle the impact of the MFA operation specifically). Furthermore, the outbreak of the Covid-19 pandemic made it somewhat difficult to guarantee an active participation and a high response rate of key consulted stakeholders because of the virtual contacts taken with online focus groups and interviews. Nevertheless, these limitations do not put into question the overall reliability of the evaluation analysis as the wide range of sources, the use of different evaluation techniques, alternative scenarios and multiple rounds of feedback mitigated them.

Overall, the evaluation found that MFA-III was highly relevant in terms of its objectives and form (loans). The timing of disbursements and the unprecedented size of the envelope (EUR 1.8 billion, of which EUR 1.2 billion disbursed) have largely facilitated Ukraine's adjustment path towards macroeconomic stability.

MFA policy conditionality covered the most relevant reform challenges in Ukraine, namely: (i) public finance management, (ii) social safety net, (iii) financial sector, (iv) governance and transparency; (v) energy; and (vi) business environment.

Out of three MFA tranches foreseen in the MoU, two were disbursed in July 2015 and April 2017 respectively. The third tranche was not released due to four unmet conditions. Two conditions were on anti-corruption, where Ukraine did not manage to set up two verification systems for asset disclosure and for the accuracy of the information provided by companies on beneficial ownership. A third condition was on trade, with Ukraine maintaining the export ban on unprocessed timber. The last condition referred to the financial sector, with the necessary legislation to establish a central credit registry not being adopted by the expiry date of the operation. The main reasons for not implementing these measures within the agreed timeframe were: (1) a lack of political will; (2) the bureaucracy, which slowed the process; (3) a (too) high ambition level for some conditions, outside the main focus area of the reforms (for instance, the condition on beneficial ownership) and (4) vested interest of some stakeholders. The evidence that the anti-corruption conditions were met at a later stage - for the disbursement of the first tranche of MFA-IV- highlights that the most complex reforms may require a longer time horizon for implementation than the two and a half years foreseen by the MFA. Therefore, particular attention is required while choosing the related targets, to ensure their feasibility already in the short term. In light of the flexibility already granted with two waivers for the disbursement of the second tranche and the prominent role given by the EU to fight corruption, the non-disbursement decision of the third tranche is deemed justified.

The evaluation found that MFA-III was effective in supporting fiscal consolidation and in helping Ukraine to improve its balance-of-payments position. In the period of the implementation of MFA-III, Ukraine returned to modest economic growth, consolidated its public finance and regained access to the international debt markets. The majority of the experts interviewed in the evaluation study believe that the macroeconomic outcomes would have been less favourable in the absence of MFA-III. In the counterfactual scenario without MFA-III, the Ukrainian authorities would most likely have had to further cut current budgetary expenditures, which could jeopardize the political acceptance of reforms. The simultaneous commitment of the EU and IMF has also allowed a return of investors’ and business confidence.

With respect to structural reforms, MFA-III was in general effective in promoting a variety of measures in the areas covered by the programme. Some of the conditions brought particularly satisfactory results, such as setting up and implementation of an online public procurement platform (ProZorro) and turning the state-owned Naftogaz Group from a loss-making company into the main contributor to the State budget. On the other hand, the implementation of some conditions met difficulties, especially in the field of anti-corruption and trade.

The efficiency of the MFA-III operation was somewhat limited by the complex design of the conditionality package and a reduced ownership of conditionalities by the Ukrainian side. This limited ownership, leading to delays in the implementation of the agreed policy actions, can be attributed to (i) limited institutional capacity; (ii) political and economic instability and (iii) vested interests opposing the reforms. A high number of conditions (36 in total), coupled with the relatively short time horizon for implementation, influenced Ukraine’s ability to deliver on all the policy reforms and limited the efficiency of the overall financial intervention. As a result and learning from this operation, the European Commission envisaged fewer conditions with reform milestones broken into smaller and achievable targets, for the following MFA-IV operation to Ukraine. While there has been good collaboration between the EU and Ukrainian counterparts during the implementation process, the visibility and awareness of the conditions attached to MFA-III was limited.

MFA-III was deemed coherent with the broad policy framework guiding the EU-Ukraine relations, notably the Association Agreement. There is a high degree of consistency in a number of reform areas between EU budget support programmes and MFA conditionality. Furthermore, as far as external coherence is concerned, the MFA has not only contributed to ‘burden sharing’ with the International Monetary Fund and other donors in financial terms, but also reinforced reforms promoted by the IMF and World Bank through the use of cross conditionality and complementary conditions.

The ability of the EU to mobilise and coordinate significant financial resources at a relatively quick speed was a key rationale for this intervention, at a time in which Ukraine’s financing need was still extremely high. MFA conditionality granted a politically reinforcing effect that contributed to the sustained mobilisation of Ukrainian authorities around crucial reform areas. The programme also added value through its confidence-boosting effect on the private sector.

MFA-III had both a direct and indirect impact on the social situation in Ukraine. The MFA operation included four conditions having a direct social impact, which concerned functioning and better targeting of compensation schemes aiming at protecting households from increasing energy prices, as well as provision of social assistance to internally displaced persons (IDPs). These conditions provided a tangible improvement in the situation of poorer households, in spite of a temporary suspension of social payments to many IDPs living in the non-government-controlled areas in Eastern Ukraine and their slow resumption. A counterfactual analysis shows that in the absence of MFA and IMF support, the indirect effects on the social situation in Ukraine would have been severe.

Finally, MFA-III had a crucial role in improving the sustainability of Ukraine’s public debt. The programme allowed for fiscal savings, stemming from the highly favourable terms of the loans, helping Ukraine to smoothen its adjustment path and to create fiscal space for reforms and sustained social spending. MFA had also a signalling effect, acting as catalyst for additional financial support and investor confidence.

In conclusion, MFA-III was highly relevant to support Ukraine in addressing its balance-of-payment problems and implementing key structural reforms to stabilise the economy and enhance the sustainability of its external position. It allowed for fiscal savings and financial benefits, and acted as catalyst for additional financial support and investor confidence. The MFA conditionality package was highly relevant to the country's needs and fully aligned with the related IMF programme. It created a politically reinforcing effect that contributed to the mobilisation of Ukrainian authorities around essential reforms, especially in areas not covered by other international donor programmes. However, the high number of conditions and level of ambition, coupled with the relatively short time horizon for implementation, hampered Ukraine’s ability to deliver on all the policy reforms and limited the efficiency of the overall financial intervention. More sparing use of conditionality, with reforms milestones broken into smaller targets, might have sustained a better motivation and feasibility throughout the operation.

Overall, the ex-post evaluation showed that MFA conditionality tends to be more effective when:

1.A high level of political ownership of the reform programme is ensured;

2.A strong alignment with other EU and international donors programmes is foreseen, especially for those interventions which appear to be more problematic;

3.Continuous technical assistance and advice is granted during the implementation process.

A number of the conclusions related to the assessment of the evaluation criteria will require further consideration and may benefit from cross-comparison with other completed MFA operations. A meta-evaluation is planned for end of 2021 and will explore these points in more detail. The meta-evaluation will provide a synthesis of the results of the evaluation studies of MFA operations carried out in the last ten years and will consider the reliability and relevance of the methodology, process, outputs, and outcomes of all the MFA operations undertaken. In addition, the exercise will assess the timeline of MFA operations and potential linkages to added value and operational visibility.

Annex 1: Procedural information

A1.1 Organization, design and timing

The ex-post evaluation assessed the EUR 1.8 billion MFA operation in Ukraine, which was approved in 2015 and implemented between 2015-2017. The assessment was in line with article 34(1) of the Financial Regulation 32 and the relevant MFA Decision 33 , which required the European Commission to submit an ex-post evaluation report to the European Parliament and the Council. The objective of the evaluation was to draw lessons with respect to the EU’s financial assistance, in particular the design and implementation of the programme and the way it contributed to achieving macroeconomic stabilisation and fostering structural reforms. Apart from identifying areas of improvement for similar on-going or future possible interventions, the evaluation also aimed at ensuring better transparency and accountability of the Commission’s activities.

The evaluation looked at various aspects of this particular EU intervention (relevance, effectiveness, efficiency, EU added value, coherence with other EU policies towards Ukraine, social impact and the impact on the sustainability of Ukraine’s public debt). In order to ensure validity, the analysis and conclusions are based on the evidence obtained using several evaluation methods (documentary review, macroeconomic data analysis, targeted stakeholder interviews, case studies, focus groups, qualitative counterfactual analysis, a Delphi survey and a social-impact analysis).

The lead DG to carry out and manage this evaluation has been the Directorate General for Economic and Financial Affairs (DG ECFIN). DG ECFIN chaired the ISG that was set up to manage the evaluation. Apart from DG ECFIN, the ISG comprised of representatives of other Commission services (the Secretariat General and the Directorate-General for Neighbourhood and Enlargement Negotiations) and the EEAS.

The Decide planning entry for the ex post evaluation of MFA-III for Ukraine is PLAN2020/6383 and the indicative roadmap for the evaluation was published in March 2020 34 to seek wider feedback. In the context of the framework contract for the provision of evaluation services related to MFA programmes, in November 2019 the Commission awarded the specific contract to undertake the external evaluation to ICF Consulting Services Ltd.

A kick-off meeting, where the ISG and the external contractor discussed the deliverables and the evaluation methods, took place in January 2020. This was followed by meetings on the inception and interim reports in, respectively, February 2020 and May 2020. The draft final report was submitted in June 2020 with updates provided for a final version approved in August 2020. In addition to meetings, ISG members were continuously informed and consulted (via email and by phone) during the evaluation. The work of the external contractor was complemented by internal analysis from Commission services.

Annex 2: Stakeholder consultation strategy 35

The stakeholder consultation strategy was developed with the overall objective to collect information and opinions on various aspects of the MFA operation in Ukraine, structured around the main evaluation criteria. It was designed to collect as much valuable and relevant information as possible from various groups and people involved. In addition to the information collected through key document reviews, the information obtained via consultation was used to support an ex-post assessment of the design, implementation and impact of the MFA-III operation by acting as source of evidence to address each of the evaluation questions.

The consultation was developed in line with the Better Regulation Guidelines on stakeholder consultation 36 and it focused on (i) extracting recollections from the period in which the operation was designed and implemented, and also (ii) on collecting views on the period after the MFA had ended, to assess its impact and sustainability.

Consultation tools were tailored to each targeted stakeholder group, to collect information most appropriate to their knowledge. These tools were also associated with different aspects of the MFA operation in Ukraine. As for primary data collection activities, these principally included: (1) a Delphi survey; (2) in-depth interviews with key (EU and Ukrainian) stakeholders; (3) an online focus group with civil society organisations and other non-governmental stakeholders. In addition, an evaluation roadmap was published from 6th March – 3rd April 2020 to seek wider feedback. Feedback received was considered as part of the evaluation process.

A2.1 Mapping of stakeholder groups

Consultation was targeted to specialists – either people who have been closely involved in the development and/or the implementation of the MFA-III operation, or people with expert knowledge in the areas related to the objectives of MFA-III. They were representatives of Ukrainian authorities and civil society, International Financial Institutions and EU institutions. In total, 101 people were consulted through a Delphi survey, in-depth interviews and an online focus group. Consultees can be grouped across the following four groups:

Ukraine public institutions:

-Ministry of Finance

-Ministry of Social Policy

-National Anti-Corruption Bureau

-National Bank of Ukraine

Ukraine civil society organisations:

-NGOs

-Think tanks

-Academics

-Media

Wider Donor community:

-IMF

-EBRD

-World Bank

-OECD

-GIZ

-Credit rating agencies

-Banks

European Union:

-European Commission

-European External Action Service

-Member State Permanent Representatives to the EU

-European business representatives

A2.2 Methods and tools for engaging with stakeholders

A2.2.1 Delphi Survey

The Delphi Survey seek to establish views on the role and contribution of the MFA in achieving macroeconomic stability, easing external financing constraints and alleviating Ukraine’s balance of payments and budgetary needs. Responses gathered from the Delphi panel fed into the analysis of the relevance, effectiveness, and efficiency of the MFA operation as well as the debt sustainability and social impact analyses.

Twenty-one respondents provided complete answers to the survey. Respondents included representatives from think tanks, academia, banks, credit rating agencies, investment funds, media and consulting.

The Delphi Survey was carried out on the basis of structured questionnaires. Specifically, participants were asked to elaborate on the plausible scenario(s) had (i) the EU MFA-III, or (ii) the joint MFA-IMF action not been implemented. As mentioned above, the Survey also covered aspects relating to the role of the MFA operation in promoting structural reforms and its social impact. A full analysis of the Delphi survey results is contained in Annex 4 of the external evaluation report 37 .

A2.2.2 In-depth interviews

ICF conducted forty semi-structured interviews with key stakeholders. An important aim of these targeted interviews was to explore selected stakeholders’ perspectives on the MFA operation in Ukraine. Specifically, detailed information was gathered on the operation, including (but not restricted to): its design and implementation; perceived/realised impact(s), notably its effectiveness in driving reform and / or macroeconomic stability; its ‘added-value’ when compared to other/similar support; and potential future improvements.

The objective was to interview various officials and representatives of the Ukrainian national authorities (and other public entities involved in the implementation of the operation), the wider donor community and the European Commission. Additional interviews were conducted with representatives of the business community, the research sphere and the general population (notably in the recipient country). The main focus of each interview varied by stakeholder type. Annex 2 of the external evaluation report 38 provides details on the main points of discussion by stakeholder group mapped across the evaluation criteria.

A2.2.3 Focus group

An online focus group discussion with non-government/private sector representatives, based in Ukraine, was organised.

The aim of the focus group was to capture additional insights, perspectives, and thoughts in relation to the MFA operation in Ukraine. The discussion was steered to specific topics, including (but not restricted to):

A summary of the focus group discussion is included at Annex 3 of the external evaluation report 39 .

A2.2.4 Stakeholder validation workshop

Towards the end of the evaluation process, a stakeholder validation workshop was held to test and validate the emerging findings with stakeholders closely involved in the negotiation and/or implementation of MFA-III. The workshop consisted of a short presentation of the main findings and conclusions, followed by discussion and feedback.

Participant groups included:

ŸEU: DG ECFIN, SGUA, EEAS, EU Delegation in Kyiv

ŸUkraine: NBU, Ministry of Finance

ŸIMF representative 

Annex 3: Methods and data sources

An evaluation matrix was developed to guide the choice and design of specific research methods, as well as to provide a framework for subsequent data analysis and interpretation. The table below provides a high-level overview of the data collection methods and analytical techniques that were used to address each evaluation criteria. Further details are provided in the sub-sections that follow.

Table A3.1 Overview of the methods and techniques used for the evaluation

●●● a very important method for addressing the evaluation criterion

●● an important method for addressing the evaluation criterion

● a complementary method

A3.1 Documentary review

All relevant literature and official documentation published by European Institutions, Ukrainian authorities and other international organizations, such as the IMF and the World Bank was reviewed to inform the evaluation.

Table A3.2 below provides an overview of the sources and types of documentary evidence assembled and reviewed, while table A3.3 details key macroeconomic indicators and data sources used.

Table A3.2 Documentary sources of evidence for the evaluation

Table A3.3 Key Macroeconomic Indicators and Data Sources

A3.2 Semi-structured interviews with key stakeholders

Forty key informants were consulted for the purposes of the external evaluation 40 . The interviews aimed at gathering information on the design, implementation and results of the MFA operation of 2015-2017. Interviewees received a copy of the semi-structured questionnaire in advance that was then used to guide the discussion. Interviews were held with relevant staff from the European Commission, the EU Delegation in Kyiv, officials from the relevant ministries and agencies in Ukraine, as well as representatives from International Financial Institutions. Annex II of the present document describes the strategy followed for stakeholder consultation.

A3.3 Focus group

An online focus group discussion 41 was organized, with representatives from Ukrainian civil society, think thanks and the Academia. It complemented and crosschecked information gathered from desk research and targeted stakeholder interviews.

A3.4 Delphi survey

A Delphi survey was undertaken with a panel of 54 experts, representing the business society, think thanks, financial and macroeconomic analysts, and the Academia. The scope of the survey was to assess on the contribution of the MFA in achieving macroeconomic stability, easing external financing constraints and alleviating Ukraine’s balance of payments and budgetary needs 42 . Experts were interviewed using a structured questionnaire and were asked to elaborate on plausible scenario would MFA-III not have been available, and the potential implications.

A3.5 Case studies

Two in-depth case studies on MFA-promoted reforms were developed in the areas of (1) anti-corruption (AC) reform and (2) social safety net reform.

The case studies addressed the following aspects:

ŸRationale behind the selection of specific MFA conditions in the above areas (for AC: #6 2nd tranche, on establishment of AC bodies, #5 3rd tranche, on asset declaration; for Social safety net: #12 2nd tranche, on provision of assistance to IDPs, #16 3rd tranche, on an evaluation of social service delivery to IDPs), as well as the relevance and added value of MFA conditionality;

ŸSignificance of MFA conditionality in the context of the overall need for reform in a particular thematic area/sector;

ŸHow the MFA conditions were implemented and whether the authorities encountered any obstacles in implementing these conditions (i.e. lack of capacity, political or public resistance to change etc.);

ŸRole and contribution of the MFA in promoting reform, including identification of key ‘causal links’;

ŸShort, mid and long-term benefits of the MFA conditions.

In addition, the case studies attempted to draw on some lessons learned from the design and implementation of similar previous reforms by two donors, the World Bank and the IMF.

The case studies were primarily based on desk research and stakeholder interviews, and took into account inputs from local economic experts.

Case study on the use of the euro

This case study examined whether the three MFA operations in Ukraine had any impact – via the above channels – in promoting the role of the euro in asset and debt management by Ukrainian authorities. The analysis was broken into two components and covered three dimensions: descriptive, exploratory and explanatory (see Table A3.4).

Table A3.4 Overview of the approach to case study analysis

A3.6 Qualitative counterfactual analysis

A qualitative counterfactual analysis was used by the evaluation team rather than a quantitative approach, due to the difficulties of isolating the effects of MFA from other interventions (i.e. the IMF programme, parallel EU interventions and supports from other donors) and other exogenous and/or unobservable factors. It applied a theory-based approach to draw inferences regarding the role and contribution of the MFA in promoting macroeconomic stabilisation. Information gathered through the other methods was used to deduce wat might have happened: in the absence of the first MFA disbursement (alternative 1); in the absence of the second disbursement (alternative 2); in case the third tranche would have been released (alternative 3); and in the absence of both MFA-III and IMF’s EFF operations (alternative 4).

A3.7 Social Impact Analysis (SIA)

For the Social Impact Analysis, the evaluation analysed trends of key indicators prior to, during and after the MFA operation: (i) wages, (ii) social services provided for IDPs, (iii) households expenditures, (iv) employment, (v) unemployment, (vi) pensions. Counterfactual reasoning was applied to deduce the extent to which the MFA operation contributed to the observed outcomes.

A3.8 Debt Sustainability Analysis (DSA)

The objective of DSA is to evaluate the contribution of the MFA-III operation (2015-17) towards the sustainability of Ukraine’s public debt. The DSA undertaken relies on modelling the key debt burden indicators and macroeconomic variables which affect the path of Ukraine’s debt and its capacity to manage its debt sustainably. A debt sustainability analysis tool developed by the IMF and World Bank for countries that can access international financial markets was used to support the assessment 43 .

The DSA tool was used to generate short-term and medium-term projections of the dynamics for debt sustainability indicators (the debt-to-GDP ratio and the gross financing need-to-GDP ratio) for years 2015-2017.

Annex 4: Timeline of the Ukraine MFA-III operation

Table A4.1

(1) Ex-post evaluation of the third Macro-Financial Assistance operation to Ukraine over the period 2015-2017, available at: https://ec.europa.eu/info/evaluation-reports-economic-and-financial-affairs-policies-and-spending-activities/ex-post-evaluation-mfa-operations-ukraine_en

(2) In terms of the use of funds.

(3) Ex-post evaluation and Commission SWD on MFA-I and II to Ukraine are available at: https://ec.europa.eu/info/evaluation-reports-economic-and-financial-affairs-policies-and-spending-activities/ex-post-evaluation-mfa-operations-ukraine_en

(4) Decision (EU) 2015/601 of the European Parliament and of the Council of 15 April 2015 providing macro-financial assistance to Ukraine: https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1476965532409&uri=CELEX:32015D0601

(5) For an assessment of the main outcomes of MFA-I and II to Ukraine, please refer to the related ex-post evaluation and Commission SWD, available at: https://ec.europa.eu/info/evaluation-reports-economic-and-financial-affairs-policies-and-spending-activities/ex-post-evaluation-mfa-operations-ukraine_en

(6) The MFA-III operation expired on 13 January 2018 without the third and final disbursement, given the lack of sufficient progress in the implementation of some of the related conditions.

(7) https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32018R1046

(8) https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/12118-Ex-post-evaluation-of-macro-financial-assistance-to-Ukraine-

(9) The net outflow from the financial account amounted to USD 4.8 billion in January-October 2014. Withdrawals of bank deposits by non-residents were the main factor behind the capital outflow. Both FDI and portfolio investments also recorded outflows in the period, although of a smaller size.

(10) Commission SWD/2015/0001 final, available at: https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1476965500936&uri=CELEX:52015SC0001

(11) https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1476965442393&uri=CELEX:52015PC0005

(12) https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1476965532409&uri=CELEX:32015D0601

(13) https://ec.europa.eu/economy_finance/international/neighbourhood_policy/doc/mou_eu_ukraine_en.pdf

(14) Commission Decision EU/2018/405 final. Available at: https://ec.europa.eu/transparency/regdoc/rep/3/2018/EN/C-2018-405-F1-EN-MAIN-PART-1.PDF

(15) https://ec.europa.eu/info/business-economy-euro/economic-and-fiscal-policy-coordination/international-economic-relations/macro-financial-assistance-mfa-non-eu-partner-countries_en

(16) For the detailed list of policy conditions, please refer to the Memorandum of Understanding, available at: https://ec.europa.eu/economy_finance/international/neighbourhood_policy/doc/mou_eu_ukraine_en.pdf

(17) The wood export ban refers to the temporary 10-year ban on export of unprocessed timber put in place by Ukraine through the Law of Ukraine No 325-VIII of 9 April 2015. The law imposed a ban on the export of all timber except the pine tree as of 1 November 2015, and as of 1 January 2017 of the pine tree as well.

(18)  For more information about the pre-conditions imposed within MFA-IV, please refer to: https://ec.europa.eu/info/business-economy-euro/economic-and-fiscal-policy-coordination/international-economic-relations/enlargement-and-neighbouring-countries/neighbouring-countries-eu/neighbourhood-countries/ukraine_en

(19)      European Commission, May 2015. Better Regulation Guidelines. Available at: http://ec.europa.eu/smart-regulation/guidelines/toc_guide_en.htm .

(20)      Available at: http://ec.europa.eu/dgs/economy_finance/procurement_grants/calls_for_tender/2015/015d/annex4-methodological_orientations_en.pdf .

(21) Detailed results are available in Annex IV of the external evaluation report.

(22) Detailed results are available in Annex III of the external evaluation report.

(23) Further information concerning methods used to build the evidence for the evaluation is available in Annex 3 of the present Staff Working Document.

(24) Gap remaining after the contributions from the IMF and the World Bank.

(25) The difference between 5% and 7% was largely caused by the non-disbursement of the third tranche.

(26) European Commission (2011). Criteria for determining the use of loans and grants in EU Macro-Financial Assistance. Available at: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52011SC0874&from=EN

(27) MFA-III initially meant to be completed by mid-2016.

(28)   https://naftogaz.com/www/3/nakweben.nsf/0/A09B58DD11619020C22584F1002A61B6?OpenDocument&year=2020&month=01&nt=News&

(29) https://ec.europa.eu/neighbourhood-enlargement/neighbourhood/countries/ukraine_en

(30) Moody’s, 2014-2018. Ukraine. Available at: https://www.moodys.com/credit-ratings/Ukraine-Government-of- credit-rating-600037040?stop_mobi=yes

(31) https://trade.ec.europa.eu/doclib/docs/2016/november/tradoc_155103.pdf

(32) https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32018R1046

(33) https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1476965532409&uri=CELEX:32015D0601

(34) https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/12118-Ex-post-evaluation-of-macro- financial-assistance-to-Ukraine-

(35) For more information about stakeholder consultation, please refer to Annex I, II, III and IV of the Ex-post evaluation available at: https://ec.europa.eu/info/evaluation-reports-economic-and-financial-affairs-policies-and-spending-activities/ex-post-evaluation-mfa-operations-ukraine_en

(36) https://ec.europa.eu/info/law/law-making-process/planning-and-proposing-law/better-regulation-why-and-how/better-regulation-guidelines-and-toolbox_en

(37) Please, refer to: https://ec.europa.eu/info/evaluation-reports-economic-and-financial-affairs-policies-and-spending-activities/ex-post-evaluation-mfa-operations-ukraine_en

(38) Please, refer to: https://ec.europa.eu/info/evaluation-reports-economic-and-financial-affairs-policies-and-spending-activities/ex-post-evaluation-mfa-operations-ukraine_en

(39) Please, refer to: https://ec.europa.eu/info/evaluation-reports-economic-and-financial-affairs-policies-and-spending-activities/ex-post-evaluation-mfa-operations-ukraine_en

(40)      See the list of completed interviews in Annex I and II of the external evaluation report.

(41) See the summary note from focus group discussion in Annex III of the external evaluation report.

(42)      Detailed results are available in Annex IV of the external evaluation report.

(43) IMF Staff Guidance Note for Debt Sustainability Analysis in Market-Access Countries, available at: https://www.imf.org/external/np/pp/eng/2013/050913.pdf. For the methodology of DSA calculations in the evaluation of the MFA-III operation 2015-2017 in Ukraine, please also refer to Annex 11 of the external evaluation report.