EUROPEAN COMMISSION

Brussels, 15.12.2017

SWD(2017) 463 final

COMMISSION STAFF WORKING DOCUMENT

Evaluation of the Instrument for Pre-accession Assistance (IPA II)

Accompanying the document

Report from the Commission to the European Parliament and the Council

Mid-term Review Report on the External Financing Instruments

{COM(2017) 720 final}
{SWD(2017) 600 final}
{SWD(2017) 601 final}
{SWD(2017) 602 final}
{SWD(2017) 604 final}
{SWD(2017) 605 final}
{SWD(2017) 606 final}
{SWD(2017) 607 final}
{SWD(2017) 608 final}
{SWD(2017) 609 final}

Contents

Executive summary    

1. Introduction    

Purpose of the evaluation    

Scope of the evaluation    

2. Background of the initiative    

Description of the initiative and its objectives    

Baseline    

3. Method    

4. Implementation state of play    

5. Answers to the evaluation questions    

6. Conclusions    

Annex 1. Intervention logic    

Annex 2. Evaluation Questions    

Annex 3. Procedural information    

Annex 4. Synopsis report of the stakeholders' consultation    

3.1 Introduction    

3.2 Stakeholder Mapping    

3.3 Stakeholder Consultation Strategy    

3.4 Stakeholder Consultation Framework    

3.5 Stakeholder statistics    

General    

Field missions    

3.6 OPC Contribution    

3.6.1 Introduction    

3.6.2 Summary of OPC contributions    

3.6.2.1.1 Question 1: Addressing IPA II objectives    

3.6.2.1.1 Question 2: Addressing inclusive public consultation    

3.6.3.1.1 Question 3: Addressing fight against corruption    

3.6.4.1.1 Question 4: Any other views on IPA II    

3.6.3 Face-to-face consultations    

3.6.3.1 Summary of contributions    

3.6.3.2 Response of the Commission    

Annex 5. Acronyms    

Annex 6. External evaluators' report, including its annexes and methods and analytical models used    

Executive summary

This Staff Working Document evaluates the Instrument for Pre-Accession Assistance (IPA II) at its mid-term, covering the period January 2014 to June 2017. With a budget of EUR 11.7 billion, IPA II is the main financial instrument supporting the implementation of the EU's enlargement policy in the period 2014-2020. It is one of the External Financing Instruments of the EU Multiannual Financial Framework 2014-2020. Its main objective is to support candidate countries and potential candidates in carrying out the reforms required for EU membership.

Though this assessment comes at an early stage of implementation, both the external evaluation and the broad consultation process carried out during 14 months confirm that the instrument is overall fit for purpose, and that it corresponds to EU priorities and beneficiary needs.

Compared to its predecessor, IPA II has become more strategic. In terms of priorities, it is increasingly focused on promoting key reforms, i.e. the three "fundamentals" of the enlargement strategy (rule of law, public administration reform and economic governance). It has also demonstrated its capacity to react in a flexible manner to emerging crisis situations, such as the refugee crisis or the floods in the Western Balkans, through the mobilisation of additional funding and the funding of new initiatives (e.g. the Regional Trust Fund in response to the Syrian crisis or the Facility for Refugees in Turkey).

The introduction of the sector approach, especially when implemented through budget support, has improved the strategic focus of IPA II and the focus on reforms, though its implementation remains uneven across sectors and beneficiaries.

The external evaluation has highlighted a mixed feedback about indirect management with beneficiaries: while its impact in terms of increased ownership is considered positive, there have been poor contracting performance and longer delays of implementation, especially in Turkey. The Commission's response consisted in adopting a more cautious and tailor-made approach in the move to indirect management by beneficiaries.

IPA II has an increased focus on performance and results, including through a newly established performance framework. However, there is some room for improvement, resulting in particular from the weak quality of indicators, the lack of a proper monitoring and evaluation frameworks at local level and the insufficient quality of data collection in beneficiaries.

Complementarity of IPA II with the actions of other Instruments, notably the European Instrument for Democracy and Human Rights (EiDHR) and the Instrument contributing to Stability and Peace (IcSP), is good but coordination during both the planning and programming phase has not been sufficient.

The findings and conclusions of the evaluation will feed into the reflection on how to improve the implementation of IPA II for the remaining period until 2020, and on the future set of Instruments for the next Multiannual Financial Framework.

1. Introduction

Purpose of the evaluation

This Staff Working Document presents an assessment of the Instrument for pre-accession assistance 1 (IPA II) 2014-2020 at mid-term, and is largely based on an external evaluation, as well as on a broad consultation process with different stakeholders. It also sets out the Commission's own views about the instrument and about the conclusions of the evaluations report. The aim is to assess whether the IPA II is appropriate and well suited for its designated purpose, based on its performance to-date, to deliver on its objectives of supporting candidate countries and potential candidates ("beneficiaries") in implementing the political, institutional, legal, administrative, social and economic reforms required for future membership 2 .

Its purpose is to inform future work on the instrument and its actions. In particular, this evaluation, which is part of a set of ten evaluations covering all the EU External Financing Instruments 3 , informs the Mid-Term Review Report 4 , which draws conclusions across the Instruments.

This document is largely based on an external evaluation by independent consultants provided in Annex 5.

Scope of the evaluation

This evaluation covers the period from 1 January 2014 to 1 June 2017. However, due to the length of the implementation cycle of IPA II, the availability of data on results is limited. Therefore the evaluation also looked at the previous Instrument for Pre-Accession Assistance covering the period 2007-2013 for some of the evaluation criteria (e.g. efficiency and effectiveness).

In order to feed into the Mid-Term Review Report, the evaluation is set at instrument level. By consequence it focuses, to the extent possible, on the IPA II Regulation (e.g. on its principles and flexibility) rather than the measures that have been put in place to implement the instrument. However, some information on programming has also been included to show progress in how the instrument has been implemented (see section 4).

The beneficiaries covered under the evaluation are those eligible under the IPA II Regulation: Albania, Bosnia and Herzegovina, the former Yugoslav Republic of Macedonia, Iceland 5 , Kosovo 6*, Montenegro, Serbia and Turkey.

In accordance with the EU Better Regulation requirements 7 , the following evaluation criteria are used: relevance, effectiveness, efficiency, coherence, and added value, complemented by impact, sustainability, consistency, complementarity and synergies and leverage.

2. Background of the initiative

The External Financing Instruments make up a major part of the Multiannual Financial Framework's 8 Heading IV "Global Europe" 9 which provides the EU with the tools necessary to reinforce its role on the world stage and to ensure that it is able to live up to its ambitions in promoting its interests and universal values and principles such as democracy, human rights, peace, solidarity, stability and poverty reduction and to help safeguard global public goods.

Adopted in early 2014, the Instruments were designed to facilitate and support policy implementation, with the intention of remaining relevant for the entire duration of the Multi-annual Financial Framework, thereby enabling the EU to implement external action policy as needed within the defined principles and objectives. The budget of IPA II amounts to EUR 11.7 billion for the period from 2014 to 2020.

Description of the initiative and its objectives

IPA II assistance is provided to IPA II beneficiaries in accordance with the general enlargement policy framework.

Assistance is predominantly focused on a selected number of policy areas to help beneficiaries strengthen democratic institutions and the rule of law, reform the judiciary and public administration, respect fundamental rights and promote gender equality, tolerance, social inclusion and non-discrimination. It is also aimed at enhancing their economic and social development with a view to attaining the targets set in the Europe 2020 strategy for smart, sustainable and inclusive growth. In addition, IPA II assistance supports the progressive alignment of beneficiaries with EU legislation and standards.

The reconstructed IL (Annex 1) for IPA II defines the impact, outcomes, outputs, activities and inputs contributing to or resulting from the implementation of IPA II (through its actions). These correspond to the different sequential levels necessary to bring about the desired results as defined by the IPA II Regulation. The reconstructed IL takes into account the provisions of the IPA II Regulation and the main policy orientations defined in the EU Enlargement strategies adopted in 2013-2015 (focus on fundamentals first in particular).

Baseline

As this is a mid-term evaluation, the baseline has been set at January 2014 when IPA II 2014-2020 was adopted. Therefore the evaluation compares, to the extent possible, the situation on 1 January 2014 with the current situation. For some evaluation criteria, where data is unavailable for this reference date, earlier baselines have been used, as described later in the document (see Evaluation questions on effectiveness and efficiency), considering that the main objectives of the instrument are still in line with those of the IPA Regulation.

At the beginning of 2014, accession status was quite diverse among IPA II beneficiaries, as shown by the table below.

State of play of the EU enlargement process at the beginning of 2014

In 2014, political stability, democracy, the respect of human rights and good governance still needed to be strengthened in all beneficiaries. Weak public institutions and administrative capacity affected negatively the effective and sustainable deployment of EU assistance. The level of socio-economic development was well below EU average, with a need for substantial investments to align their legislation and institutions to EU standards.

To respond to these challenges, IPA II was simplified, in particular by removing barriers linked to accession status 10 . IPA II was made more strategic and results-oriented through the introduction of a longer term planning and sector approach to address key reform challenges in a comprehensive way and with an increased focus on results and impact.

The IPA regulation does not include indicators at instrument level. Indicators with agreed targets are defined, for each beneficiary in the Indicative Strategy Papers, as well as in specific programmes, in particular when implemented through budget support. Indicators are linked to the specific situation in each country and progress can only be assessed in these sectors at country level. Therefore it is difficult to assess progress at the level of the instrument; assessment of progress is made at country or regional level for the different sectors.

3. Method

The present assessment is supported by an external evaluation carried out from August 2016 to May 2017. The external evaluation of IPA II was managed by the Commission services through the following steps: an inception report (which explained how the evaluation design would deliver the information required); a desk report (providing initial responses to evaluation questions); field visits to meet key interlocutors to obtain first-hand view in-country; a survey to EU delegations covering all instruments; an Open Public Consultation 11 on the draft report which comprised a 12 week online survey and targeted meetings with Member States in March 2017; and a final report.

Overall, the quality of the collected evidence (data, documentation, interviews and survey results) for this evaluation can be assessed as good. However, apart from the limited evidence on implementation, there was a limited feedback from the Open Public Consultation. This Staff Working Document largely concurs with the findings and conclusions of the external evaluation unless said otherwise.

Organisation

As part of a wider set of evaluations covering the eight External Financing Instruments (the instruments), the European Development Fund (EDF) and the Common Implementing Regulation (CIR), all the evaluations were carried out in an interlinked and co-ordinated manner. To ensure coherence of the different evaluations, relevant Commission services have worked closely together from the beginning of the process in June 2015. All reports prepared by the external contractor were discussed, reviewed and approved by the dedicated internal discussions set up by the Commission for the IPA II evaluation.

The draft final report prepared by the external contractor was placed, with the other instruments draft evaluation reports, on the web for the open public consultation (OPC) from 7 February to 3 May 2017. Thirty contributions were received. During that period targeted face to face meetings were organised with representatives of civil society organisations (CSOs), EU member states and the European Parliament. The external contractor delivered its final report at the end of June 2017.

Evaluation Design

The intervention logic (IL) forms the backbone of the analytical framework. It is visualised in an IL diagram, which brings together the most relevant elements (from inputs to impacts, but also assumptions) in a single framework. To take into account the Commission's Enlargement Strategy with a focus on 'fundamentals first', a reconstructed IL was prepared by the external consultants.

Covering the different levels of the IL, the evaluation questions (EQ) structure the analysis to gather evidence. Each question was structured into judgement criteria (JC) and indicators to provide an answer based on a synthesis of evidences.

Methodology and data

The EU evaluation criteria (relevance, effectiveness, efficiency, coherence, and added value, complemented by impact, sustainability, consistency, complementarity and synergies and leverage) were applied as an underlying basis. The evaluation questions gave rise to a number of judgement criteria and associated indicators. The external evaluation used a mixed methods approach blending quantitative and qualitative methods, with a focus on the latter. The main analytical tools consisted of rigorous assessment of documentation and consultation of stakeholders (semi-structured interviews, group consultations). The Commission services also organised an Open Public Consultation and a survey of beneficiaries and stakeholders in order to address key issues.

For all EQs, data collection included a mixture of desk review of documents, semi-structured interviews in Brussels by phone or face-to-face, as well as field visits to all seven IPA II beneficiaries.

Challenges and limitations

The process of the external evaluation is robust and the evidence reasonably solid. However, evaluators were faced with three main challenges:

- the implementation of IPA II was still in its early stages when the collection of evidence took place. Therefore, effectiveness/sustainability/impact criteria could not be assessed based on outcomes at this stage;

- the timeframe in which the evaluation took place was tight given the requirements imposed by the Common Implementing Regulation. This resulted in risks related to access to data, documentation and availability of key respondents;

- IPA I is still under implementation, and no ex-post evaluation has taken place yet.

The multiple rounds of comments by Commission services mean that facts were verified and this facilitated the cross verification of data from multiple sources. The evaluators were provided access to all relevant information and colleagues within the Commission services and in EU Delegations and Office. The Commission services also ensured that the evaluators would have access to representatives of the beneficiaries, in particular the National IPA Coordinators (NIPACs).

Overall, the quality of the collected evidence (data, documentation, interviews and survey results) for this evaluation can be assessed as good. The evaluators demonstrated a good understanding of the instrument and took a very positive and open attitude throughout the whole process. However, there was limited feedback from the Open Public Consultation.

As indicated by the Intervention Logic, the IPA II instrument can only be seen as a contributing factor towards any results achieved. First, the EU enlargement process is a highly ambitious transformative agenda. Second, many factors, both internal and external, affect developments in the beneficiaries. Ultimately, it up to the beneficiaries themselves to adopt and implement the necessary reforms and policies for meeting the accession criteria.

4. Implementation state of play

IPA II is still at an early stage of implementation as reform programmes are complex to develop, and require extensive stakeholder consultations. The first actions and programmes started in 2015.

Financial assistance is planned and programmed on the basis of indicative strategy papers setting out the priorities and objectives for each beneficiary and the multi-beneficiary programmes for the entire period up to 2020 12 and a breakdown of indicative financial allocations. The strategy papers define the results to be achieved by 2020 and the actions necessary to reach them, as well as indicators for monitoring progress and measuring performance. Strategy papers were adopted in 2014, and their mid-term revision has been launched in 2017. Priorities from the strategy papers are translated into detailed actions in the annual or multi-annual action programmes. Cross-Border Cooperation Programmes set out the priorities for assistance in the area of territorial cooperation between IPA II beneficiaries.

The implementation of IPA II is based on a sector approach. Reforms are promoted within the framework of pre-defined sectors closely linked to the enlargement strategy, such as democracy and governance, rule of law or growth and competitiveness. At the same time, IPA II assistance is tailored to the specific needs of each individual beneficiary, and takes into account the status of each partner in the overall accession process. To support structural reforms based on clear strategies, sector budget support has been introduced as a delivery modality. Sector reform contracts amounting to EUR 355 million are now in place in Albania, Kosovo, Montenegro and Serbia, in the field of public administration and/or public financial management, employment and skills, anti-corruption, transport, integrated border management and education. Policies supported under budget support must have targeted results and a robust monitoring and evaluation system for collecting, storing and evaluating data in place.

More generally, a considerable share of the assistance provided so far, EUR 2.27 billion has been focused on addressing the "fundamentals first", i.e. strengthening the rule of law and fundamental rights, public administration reform as well as economic development and competitiveness, in line with the priorities set out in the Commission's enlargement strategy.

IPA II commitments and payments from 2014-2016 (in million Euro)

Faced with the most important refugee inflows for decades, the EU reacted swiftly with the establishment of the Facility for Refugees in Turkey drawing funds from both Member States (EUR 2 billion), as well as from the EU budget – mainly IPA II and humanitarian aid (EUR 1 billion). The Facility became operational in February 2016 and is designed to ensure that the needs of 3 million Syrian refugees and host communities in Turkey are addressed in a comprehensive and coordinated manner. Despite the absence of a clear reference to migration in the thematic list of priorities in the IPA II Regulation, the mobilisation of IPA II fund is a unique example of EU reactiveness and joined-up action. Out of the EUR 3 billion, EUR 2.9 billion has been allocated, more than EUR 1.6 billion contracted and EUR 811 million disbursed at the end of June 2017, with a focus on six priority areas: humanitarian assistance, migration management, education, health, municipal infrastructure and socio-economic support.

In parallel, despite not being a key focus of IPA II assistance, the EU provided significant support to the countries along the Western Balkans migration route, in particular Serbia and the former Yugoslav Republic of Macedonia to help these countries to properly manage migration flows.

The implementation of financial assistance provided under IPA II is continuously monitored, in particular through the Results-Oriented Monitoring (ROM) 13 of projects and programmes by external experts, in addition to the internal monitoring by Commission staff. IPA II programmes are also subject to evaluations at policy, thematic, as well as at beneficiary or regional level 14 .

Facility for Refugees in Turkey: Amounts committed per priority area

In addition, joint monitoring committees and sectoral monitoring committees, including joint monitoring committees for cross-border cooperation programmes, bring together the Commission, beneficiaries and stakeholders several times a year and monitor the implementation of financial assistance programmes.

5. Answers to the evaluation questions

Evaluation Question 1: Relevance

1. To what extent do the overall objectives (IPA II Regulation, Article 1 and 2) and the design of the IPA II respond to:

(i) EU priorities and beneficiary needs identified at the time the instrument was adopted (2014)?

(ii) Current EU priorities and beneficiary needs, given the evolving challenges and priorities in the international context (2017)?

(iii) the need to set out a relevant system of indicators for programming?

Overall, as confirmed by the external evaluation 15 , IPA II appeared fit for purpose at the moment it was set out. The overall objectives and the design of IPA II respond to EU priorities and beneficiary needs. IPA II is in line with the Global Strategy for the EU's Foreign and Security Policy, which sets out the EU's core interests and principles for engaging with the wider world. In addition, IPA II supports the EU Enlargement strategy and policy frameworks stipulated in the IPA II Regulation. The Enlargement strategy is focusing on those areas in which all beneficiaries face severe challenges, with a strong emphasis on structural reforms as the basis for the accession process.

The planning of IPA II assistance is reflected in the individual Indicative Strategy Papers that ensure a close link between the political priorities and the priorities for financial assistance are properly set in these documents.

Strengthening the rule of law, including the fight against corruption and organised crime, good governance and public administration reform, as well as economic governance and competitiveness are identified in the IPA II Regulation as key challenges in most of the IPA II beneficiaries, and have been identified as the three fundamental priorities of the Commission's enlargement strategy. 16 The need to deal with these “fundamentals first” has been reflected in relevant programming documents, which form the basis for the allocation of IPA II funding.

As pointed out by the external evaluation 17 , clear links between the actual stage of EU accession and the IPA II Indicative Strategy Papers exist. Beneficiary-specific differences are clearly spelled out in the related Indicative Strategy Papers. A move towards sector approaches had already been launched during the late stage of IPA notably in the area of agriculture and rural development, and more clearly under IPA II, though this process is still on-going. The Commission services consider that, overall, the introduction of the sector approach has proved useful, having encouraged a more strategic orientation, also in budgetary terms, more sectoral focus and progress in the utilisation of budget support. This is clearly the case for example in the area of governance and public administration reform (with reinforced policy dialogue and budget support operations in Albania, Kosovo, Montenegro and Serbia). This has produced complementary benefits, in relation to public financial management reforms, increased policy dialogue and reduced transaction costs. However, the external evaluation highlights an uneven implementation. In some sectors, the move from the project-based programming approach towards true sector programming remains to be completed as many beneficiaries either do not have a proper understanding of the sector approach or the full capacities and willingness to develop such strategies 18 . In this framework an element of insufficient clarity and quality is represented by the Sector Planning Documents, prepared by the beneficiaries, in line with the programming guidelines issued by the Directorate General for Neighbourhood and Enlargement Negotiations of the Commission, but not fully understood by them, and of uneven quality 19 .

IPA II builds directly on the achievements made by the predecessor programme in many cases. The Rural Development component of the Instrument for Pre-accession Assistance (IPARD) and, to a slightly lesser extent, IPA Cross Border Cooperation, demonstrate a strong degree of continuity 20 . Thus, there is good potential for complementing and increasing the sustainability and impact of IPA I interventions. The management of the former IPA Components III (regional development) and IV (human resources development) has been unified within one Commission department for the current programming period (2014-2020), ensuring also continuity in these policy areas. Also, efforts are made to continue with the achievements made under IPA I, e.g. by promoting the same or similar sector policy objectives or to continue with local beneficiary institutions that developed their capacities under IPA I.

The overall objectives and the design of IPA II have been able to respond to a large extent to evolving challenges and priorities. The instrument has demonstrated a high degree of flexibility in cases of pressing emerging needs, such as the reaction to the 2014 floods in the Western Balkans and the refugee crisis. Examples include the introduction of special measures (10 in total between 2014 and 2017) 21 to address the consequences of the devastating floods in the Western Balkans in 2014, as well as the setting up of the Facility for Refugees in Turkey in 2015 22 , and the EU Regional Trust Fund in response to the Syrian Crisis (Madad fund) 23 .

The relevance of the instrument and of some of the changes introduced were overall acknowledged by the stakeholders consulted, both during the face to face interviews and in the Open public consultation 24 . By contrast, stakeholders criticised above all the insufficient involvement of non-state actors in the design and implementation of actions 25 .

Evaluation Question 2: Effectiveness, impact, sustainability

To what extent does IPA deliver results as compared to the instrument's objectives, and specific EU priorities?

Though no definitive conclusions can be drawn at this early stage of implementation, some positive elements seem to confirm the effectiveness and impact. The external evaluation points to a higher concentration of funds in priority areas, in line with programme objectives and the Commission's enlargement strategy 26 , in particular in the fields of political reforms and economic, social and territorial development, which constitute around three quarters of all IPA II allocations. A substantial portion of the IPA II funds is devoted to institution building in the sectors of democracy and governance, and rule of law and fundamental rights, in line with political priorities. For example, comprehensive sector budget support programmes in the area of public administration reform were launched for Albania, Kosovo, Montenegro and Serbia and some of them are already being implemented.

Another key element is the existence of mechanisms to support the programming and delivery of assistance. The introduction of budget support offers considerable potential for a more efficient and effective delivery of IPA II assistance and its focus on results. Budget support explicitly addresses key elements of political reforms and demands a complex set of preconditions to be met. Its introduction has generated greater policy dialogue not only between the Commission and IPA II beneficiaries 27 , but also among beneficiaries’ institutions involved in its delivery. This requires significant administrative and/ or legislative changes as well as governance reforms, a macroeconomic stabilisation programme and a credible public finance management reform programme. All of these effects can be observed in those IPA beneficiaries that have now introduced budget support and they can be considered a major indirect effect of the instrument 28 in the Western Balkans. So far, Turkey has not considered appropriate to use the budget support modality.

The sector approach has generally facilitated improved thematic coherence mostly evident in the areas of public administration reform and public financial management. It demands a new way of thinking among the institutions responsible for its programming and delivery. Although the sector approach is still in a transition stage as regards its full implementation in all beneficiaries and faced resistance at the beginning, evidence confirms that this approach has indeed moved IPA II towards a greater focus on reforms by concentrating funding on key sector reform priorities 29 . The sector approach has changed the approach of many of those involved in programming in the Commission services, National IPA Coordinators (NIPACs) and sector lead institutions away from the project-based, input-output philosophy, very much focused on the harmonisation of EU legislation ("acquis") that often prevailed under IPA I 30 .

It is difficult to assess the direct effects resulting from IPA II support. This is due to the fact that much of the assistance under the IPA II has only started to be implemented or is still in the contracting stage. Likewise, the impact of IPA II at this stage is not evident and any assessment of impact prospects would be unduly speculative. A key condition for impact is the existence of strong policy dialogue between the EU and IPA II beneficiaries, particularly as regards political reforms. This is evident in most IPA II beneficiaries, in particular where budget support has been introduced 31 , but less so in Turkey where policy dialogue has been more challenging in the last few years. Regarding multi-beneficiary programmes, outputs and some preliminary outcomes have already been achieved because they started early, as they do not require the signature of a financing agreement with beneficiaries. This is for example the case of the programmes for public administration reforms (e.g. improved legislation on public procurement, civil service, administrative procedures, public internal financial control through the support of the OECD/SIGMA programme) or public financial management (e.g. review and improvement of public investment management through the programme implemented by the International Monetary Fund).

The Commission services have promoted a high level of stakeholder involvement in the programming of IPA II. The extent to which this has happened in practice remains uneven and significant differences among beneficiaries exist in the quality of the programming documents. Several stakeholders have criticised what they see as a lack of involvement and support to both non-state actors and local authorities 32 . Insufficient involvement of stakeholders is also seen as a way that prevents a successful fight against corruption, which is considered not adequately addressed 33 . The modest involvement of stakeholders may have a negative effect on impact and sustainability.

Finally, Cross Border Cooperation (CBC) under IPA II has remained largely unchanged and is perceived in its focus and basic functioning. Also, CBC is recognised as being a complex instrument 34 requiring continuity.

Evaluation Question 3: Efficiency

To what extent is IPA delivering efficiently?

Considering the changes under IPA II and the short period of actual implementation, it is difficult to come up with conclusive evidence on efficiency. The cost-effectiveness of IPA II is confirmed by an administrative cost ratio in a range of 3-5% for commitments made by the Commission. Where continuity from IPA I to IPA II has been fully ensured, as in IPARD 35 , efficiency gains are more evident. The available indicators 36 suggest progress in the Commission’s administrative efficiency as well when comparing 2014 with 2015.

Overall, sound financial management has improved between 2014 and 2015 for most of the beneficiaries. The EU Delegations in Enlargement countries have performed well and without major differences between them.

However, the operational efficiency is difficult to assess properly at this stage. There is a widespread belief among the stakeholders that time lags between programming and implementation are too high 37 . Operational efficiency is also related to the applied management modes. Here a direct comparison with IPA is difficult as indirect management is no longer the rule under IPA II. IPA II allows, in principle, more flexibility in applying different management modes. At the same time, compared to the IPA period, the number of beneficiaries operating at least partially under the indirect management with the beneficiary country (IMBC) mode has increased to five.

In recent years, the stricter conditions set by the Commission in order for beneficiaries to move to an indirect management mode (notably in terms of management and control systems, frequency of controls) have slowed down implementation of the programmes in those beneficiaries. For many policy areas, inefficiencies in the IMBC have generated chronic and cumulative delays (in particular in the former Yugoslav Republic of Macedonia and Turkey). This is adversely affecting IPA II programmes and overall budget execution, notably in Turkey, where the majority of funding is implemented through IMBC and only a small percentage of funds have been contracted yet under the annual programmes 2014, 2015 and 2016 38 .

For the indirect management mode to function more effectively, significantly greater resources are required (in particular in terms of institution building, as shown by IPARD) 39 . There is currently no reliable data available to assess the cost-effectiveness of the indirect management mode applied by the IPA II beneficiaries. Lack of transparency with regard to absorption capacity of IPA funds, partly related to visibility, was pointed out both in the OPC 40 and in the specific consultation with stakeholders 41 .

In terms of coherence between strategies, instruments and procedures, efficiency is improving 42 . Improvements in the programming documents and a better appreciation of what the sector approach looks like in practice are materialising 43 . With the introduction of the sector-based approach, a major shift in thinking for all stakeholders has been initiated although some line ministries among beneficiaries still have difficulties to fully grasp and apply the new approach 44 . In-house expertise was strengthened with the creation of Centres of Thematic Expertise which enhance coherence in formulating sector and thematic IPA II interventions, drawing on the lessons of past projects and best practices from other beneficiaries/regions. In addition, increased attention has been given by the Commission, notably through a revised internal quality review process, to screening relevance and maturity of proposals, with those not meeting the necessary requirements being deferred to later programming years.

The use of sector budget support has been actively promoted by the Commission and is expanding across the Western Balkans. Four beneficiaries are now involved in such operations. With budget support, much less resources are required for the contracting and managing of contracts compared to classic technical assistance projects, but additional resources need to be allocated to policy analysis and dialogue with the beneficiaries both during the programming and implementation phases. Also, there are first indications that the introduction of this modality in itself is likely to improve the quality of the policy dialogue (e.g. in Albania, Serbia, Kosovo) 45 . Intensified policy dialogue works best with the various administrations involved in the preparation of the support – as effective cooperation and communication is a requirement already during the planning phase – but also between the Commission and the national stakeholders.

When an agreement on budget support is made, other positive processes take place: increased willingness to assess and follow-up closely public financial management issues, to engage in more in-depth dialogue on macroeconomic issues (in close link with the EU-supported efforts to improve economic governance through Economic Reform Programmes) but also implementation of prior actions before the signature of the Financing Agreement. Budget support has also prompted the preparation of sector reform progress reports by the government (e.g. in Serbia and Albania), demonstrating increased capacity in the area of monitoring and evaluation, creating space for more informed policy dialogue between stakeholders and constituting a tool for improving accountability and transparency 46 .

A major potential driver for increased efficiency can be seen in the monitoring processes for the measurement of IPA II performance which are being put in place both in the Commission and in beneficiaries. The setting up of IPA II Sector Monitoring Committees in the beneficiaries involves national stakeholders to a varying extent, in line with the dominant implementation modalities in the particular beneficiary. A set of common strategic indicators has been defined, though not all beneficiaries have agreed to define targets 47 . The assessment of ‘operational’ performance is based on indicators of outputs and immediate outcomes put forward in the action documents. The quality of action documents and the indicators (especially at outcome level) contained therein still need improvement. The Commission has developed internal Guidelines on linking planning/programming, monitoring and evaluation 48 , which are meant to help all involved parties in the preparation of action documents. In addition, the Commission has organised capacity building activities both for Commission staff and the beneficiaries, and it has revised the in-house quality review process to increase scrutiny and the focus on results.

Evaluation Question 4: Added value

To what extent do the IPA II actions add value compared to interventions by Member States or other key donors?

IPA II added-value is reflected in the IPA II Regulation through a reference to the principle of subsidiarity, as set out in Article 5 of the Treaty on the European Union. Overall, the Commission support is consistently aligned with actions by other donors, including EU Member States and non-EU donors, usually with the EU representing the most significant provider of grant funding.

For EU Member States representatives and stakeholders in the beneficiaries, the EU’s unique supranational nature is a key element of added value. Its political influence and leverage allows engagement with authorities or other donors with greater authority and legal certainty than individual EU Member States. 49

The intense use of both the Twinning 50 and TAIEX 51 initiatives, where valued European public expertise is made available to beneficiaries, confirms the added value of bringing particular EU Member States expertise into the IPA II beneficiaries to address specific needs. The fostering of long-term relations with a similar institution in an EU Member State is an intangible benefit explicitly ascribed to Twinning.

IPA II contributes to other donors' actions only to a limited extent, reflecting the lower engagement of other donors in the Western Balkans and Turkey. IPA II national funds are used for some multi-donor funds, both for longer-term programmes and for emergency response actions. The dominant multi donor fund in IPA II is the Western Balkans Investment Framework (WBIF), which provides financing and technical assistance to strategic investments in the energy, environment, social, and transport sectors, and also supports private sector development initiatives. With regard to added value, it is clear, for example, that without this facility, which represents a relevant and important response to the development needs of the Western Balkans region, 52 the majority of funded projects would be difficult, or impossible, to finance solely from EU Member States' funds.

It is also unique to the EU to actively promote territorial cooperation such as through IPA II Cross-Border programmes, transnational and interregional cooperation programmes, as well as macro-regional strategies. The added value is obvious: reconciliation and confidence building in the Western Balkans, overcoming of geographical and mental barriers and developing of good neighbourly relations – all these remain key aspects in the enlargement process, addressed solely by EU programmes and not by other donors 53 .

Coordination and division of labour as promoted in the aid effectiveness agenda show mixed achievements in IPA II beneficiaries despite the fact that it has been strongly supported by the EU 54 . The departure of most international and bilateral donors and the dominating role of EU grant funding mean that division of labour is de facto increasingly seen as less of an issue.

The EU Regional Trust Fund in Response to the Syrian Crisis (Madad fund) is a good example of the EU’s capacity to mobilise large amounts of funding from different sources in response to an emerging crisis. The contributions, including from IPA II, to the Madad fund are significant and allow policy dialogue at a higher level than EU Member States bilateral interventions. The Madad fund provides also an illustration for effective joint programming undertaken by the EU and other donors, including EU Member States.

Evaluation Question 5: Coherence, consistency, complementarity and synergies

To what extent does IPA II facilitate coherence, consistency, complementarity and synergies both internally between its own set of objectives and actions and vis-à-vis other EFIs?

As the main EU funding instrument for pre-accession support, IPA II finances actions required not only for aligning with the EU acquis, but also to promote the financial, social and institutional development of beneficiaries. However, the overall available funds of the Instrument are small compared to the actual needs required to comply with the obligations of EU membership 55 .

Besides IPA II, the European Instrument for Democracy and Human Rights (EIDHR) and the Instrument contributing to Stability and Peace (IcSP) also provide grants for the implementation of specific actions in the Western Balkans and Turkey. Although IPA II and the EIDHR provide support to local civil society organisations, for instance through a dedicated Civil Society Facility, many stakeholders consulted (notably from CSOs) during the Open Public Consultation and face to face interviews, noted the limited support of IPA II to civil society and local governments and the need to provide more support to civil society actors in beneficiaries 56 .

IPA II procedures for planning, programming and implementation of programmes and actions foresee the coordination and the stimulation of synergies with the above mentioned thematic Instruments, but there is still room for improvement 57 . Coordination of the ad-hoc (non-programmable) actions of the EIDHR and the IcSP with similar actions of IPA II requires programme or task managers in EU delegations to take the initiative. When this is not happening, there is a risk of duplication or a lack of flexible response to an emerging need 58 .

Financial support provided under IPA II contributes to the long-term objectives of EU accession by funding the social development of the beneficiaries 59 and urgent needs stemming from emergencies such as natural disasters (e g. the floods which occurred in Serbia and Bosnia and Herzegovina in 2014) or social problems (e.g. linked to the recent refugees crisis in 2015 and 2016) also through loans, co-financed where appropriate with loans provided by the International Financial Institutions.

The EIB is a key financing partner of IPA II beneficiaries, providing its banking products (mainly loans and guarantees) either alone or in the framework of wider financing Facilities created together with other International Financial Institutions and Funds. One example is the Western Balkan Investment Framework (WBIF). The EBRD and the Council of Europe Development Bank (CEB), although less active than the EIB, play a similar role.

In addition to the bilateral and multi-country programmes and actions, IPA II also finances cross-border cooperation (CBC) programmes implemented with EU Member States, with neighbouring beneficiaries and between IPA II beneficiaries and contributes to the participation of the IPA II beneficiaries in the transnational and interregional cooperation programmes of the European Regional Development Fund (ERDF). It also supports actions implemented through macro-regional strategy frameworks. Its contribution complements the financing of the ERDF.

Other donors active in the beneficiaries include a number of EU Member States, International Organisations and International Financial Institutions. There are continuous efforts from all donors, as well as beneficiaries to coordinate activities and to programme and implement actions in a way to further strengthen complementarity and synergies. Donor coordination in most of the IPA II beneficiaries is adequate 60 . It is ensured by a high-level National Authority, supported by the EU Delegation/Office on the basis of the National Development Strategy and its in-built sectoral priorities. Donor coordination still needs to be improved in some beneficiaries (e.g. in Kosovo). As a consequence, there is also a considerable risk of overlaps and gaps 61 .

Evaluation question 6: Leverage

To what extent has IPA leveraged further funds and/or political or policy engagement?

IPA II is used pro-actively for supporting negotiations with the governments of the beneficiaries for taking the necessary measures leading to reforms, although this applies more to the Western Balkans than Turkey. This includes rewards, but also the decommitment of funds, together with a number of other measures. IPA II has contributed to an increasing political and policy engagement by the respective beneficiaries, in particular in the Western Balkans.

The policy discussion platforms created for the coordination of the policies of the Commission, the EU Member States and other donors have contributed to the coordination and promotion of common policies and to the creation of greater policy leverage.

IPA II has also managed to create considerable financial leverage. As the biggest donor in the beneficiaries, IPA II funds offer a reliable financial basis to leverage funds from other donors. Financial leverage is mainly ensured by special financing vehicles which pool all these different financing sources. Such vehicles are the WBIF 62 , EDIF 63 , GGF 64 , but also the Madad fund and the Facility for Refugees in Turkey.

The amounts already mobilised by other donors to co-finance IPA II projects are considerable 65 . It was not possible to collect detailed data on the overall financial leverage in the framework of this evaluation, however, it has been estimated that: (1) under IPARD I, EUR 1.00 from the EU has attracted at least EUR 1.60 from other sources; (2) to date, the WBIF has allocated EUR 1.3 billion to the Western Balkans, out of which EUR 473 million represent technical assistance and investment grants aimed at leveraging EUR 13.5 billion investments 66 . The major part of these investment grants comes from IPA funds (the rest from EBRD, EIB, CEB and 19 EU Member States). The leverage of the grants provided is very significant. EU Member States have been very active and participated in many of these facilities and funds.

6. Conclusions

IPA II (2014-2020) is the main financial instrument supporting the implementation of the EU's enlargement policy.

It is yet difficult to assess with absolute certainty the effectiveness of the instrument as implementation started only in 2015 and most programmes adopted since then require time to be rolled out, let alone to produce outputs and outcomes.

The external evaluation finds that IPA II is overall relevant and well suited for its designated purpose. Its overall objectives and design are in line with EU priorities and beneficiary needs. The instrument has also, compared to IPA I, a stronger focus on key reforms required to ensure that candidate countries and potential candidates successfully meet the EU accession criteria. In addition, IPA II is more strategic and results-oriented, and has allowed greater leverage of other donors' funds.

In parallel to this focus on long term reforms, the instrument has also demonstrated some flexibility to face emerging priorities. Since 2015, the EU has provided significant support to the countries along the Western Balkans migration route to help them properly manage the influx of refugees trying to reach the EU. The most important response was in relation to refugees in Turkey; the EU established the Facility for Refugees in Turkey which draws funds from both Member States (EUR 2 billion), and from the EU budget – mainly IPA II and humanitarian aid (EUR 1 billion).

The introduction of the sector approach to programming has improved the strategic focus of IPA II over its predecessor. This more conceptual approach to programming, prioritising sector level change over projects is already observable in the Commission and some Institutions in beneficiaries. However, the implementation of the sector approach remains uneven across sectors and many beneficiaries have difficulties in the proper uptake of the approach, as well as in understanding the proper meaning of sector planning documents, as revealed by the external evaluation.

In support of the sector approach, the Commission has introduced sector budget support as a key delivery modality. The strict eligibility criteria linked to this approach have pushed the IPA beneficiaries to embark on new reforms in important areas of governance such as the strengthening of democratic institutions, the rule of law, public financial management and macroeconomic stabilisation (in close link with the EU-supported efforts to improve economic governance through Economic Reform Programmes). The intensive engagement between the Commission and IPA II beneficiaries on the introduction of this complex mechanism has enhanced the quality of policy dialogue between the Commission and IPA II counterparts.

Evidence from IPA I shows that the introduction of the Decentralised Implementation System / Indirect Management with the beneficiary country (IMBC) has a mixed impact. It offers improved ownership of the programme among beneficiaries, but in most cases overall efficiency suffers compared to the direct management mode. In some beneficiaries, notably Turkey, inefficiencies in the IMBC are evident for many policy areas and have generated chronic delays that have accumulated and led to the decommitment of funds. The Commission takes now a much more prudent approach in the framework of IPA II, compared to the predecessor instrument, in view of the identified weaknesses.

The monitoring and evaluation system embedded in the IPA Regulation, which involves setting up a performance framework, is promising, but the monitoring and evaluation systems at national level and many indicators used, especially at outcome level, are still weak. This implies the need of further strengthening statistical capacities, further clarification with regard to the functioning of the monitoring system, and a solid system of data gathering, encoding and analysis. In this context, the Commission is developing a new information management system (OPSYS), which will improve the data collection processes and hence the reporting on results.

Complementarity of IPA II with the actions of other Instruments, mainly the EIDHR and the IcSP is good, but coordination and cooperation during the planning and programming phases remains uneven.

Donor coordination is largely satisfactory in most of the IPA II beneficiaries. The EU is promoting the coordination of donors to be fulfilled by a National Authority. IPA II is making considerable efforts in strengthening public consultation with the civil society and other actors, in particular at the planning and programming stages. However, the process differs widely among beneficiaries and sectors, and the involvement of civil society organisations during the implementation of IPA II projects remains limited.

Annex 1. Intervention logic

Annex 2. Evaluation Questions

Relevance

1. To what extent do the overall objectives (IPA II Regulation, Article 1 and 2) and the design 67 of the IPA II respond to:

(i) EU priorities and beneficiary needs identified at the time the instrument was adopted (2014)?

(ii) Current EU priorities and beneficiary needs, given the evolving challenges and priorities in the international context (2017)?

(iii) the need to set out a relevant system of indicators for programming?

Information sought in this area includes:

· A timeline showing congruence/divergence of the instrument against an evolving context, including global challenges, and institutional policy changes e.g. to what extent does IPA II respond to the changes in the larger neighbourhood.

·To what extent has the new programming reflected the recent shift of the enlargement strategy towards the three fundamental pillars (i.e. rule of law, economic governance and competitiveness and public administration reform), in terms of priorities and modalities of intervention and planning of reforms (like Economic Reform Programmes)?

Effectiveness, impact, sustainability

2. To what extent does IPA deliver results as compared to the instrument's objectives, and specific EU priorities? 68

Information sought in this area includes:

·To what extent do IPA programmes contribute towards:

oSupport for political reforms;

oSupport for economic, social and territorial development, with a view to smart, sustainable and inclusive growth;

oStrengthening of the ability of the beneficiaries at all levels to fulfil the obligations stemming from Union membership;

oStrengthening regional integration and territorial cooperation (IPA II regulation Article 1 and 2).

oInstitution building

oImproving cooperation and good neighbourly relations among CBC partners

·To what extent has IPA contributed to the European Union's priorities for smart, sustainable and inclusive growth?

·To what extent does IPA II mainstream EU policy priorities (e.g. gender, climate change) and other issues highlighted for mainstreaming in the instrument, and, where relevant, deliver on the commitments including the financial allocations (IPA II Regulation, preamble)?

·To what extent does IPA II promote principles of aid effectiveness, such as coordinating assistance, partnership and ownership (IPA II Regulation, Preamble and Article 5)?

·To what extent are the processes conducive to programming, identification/formulation of effective actions (IPA II Regulation, Article 6-15)?

·How have the criteria for the performance reward been designed under IPA II?

·To what extent is IPA II flexible enough to respond to changing needs (e.g. changed policy priorities, changed contexts)?

Efficiency

3. To what extent is IPA delivering efficiently? 69

Information sought in this area includes:

·What is the ratio of administrative costs (as defined as “IPA II Support Expenditure” in the Draft General Budget of the EU 70 ) to overall budget?

·How efficient is budget execution in terms of time taken from commitments to payments?

·Have the changes made to IPA II 2014 – 2020 from the previous IPA I 2007 – 2013 brought efficiency gains? E.g. Has the introduction of sector approach and new aid modalities brought positive change in terms of efficiency of delivery?

·Are there areas, such as administrative/management procedures, where IPA II could be simplified to eliminate unnecessary burden?

·To what extent is IPA II in line with the implementing rules of the CIR? Specifically in terms of :

oImplementation

§Subject matter and principles

§Adoption of action programmes, individual measures and special measures

§Support measures

oProvisions on the Financing Methods

§General financing provisions

§Taxes duties and charges

§Specific financing provisions

§Protection of the financial interests of the Union

oRules on nationality and origin for public procurement, grant and other award procedures

oClimate action and biodiversity expenditure

oInvolvement of stakeholders of beneficiary countries

oCommon rules

§Eligibility under IPA II

oMonitoring and evaluation of actions

·To what extent are the following in place and functioning:

oappropriate monitoring processes and indicators for measurement of the performance of the IPA II instrument

orelevant strategic and operational indicators to measure results achieved by IPA II?

Added value

4. To what extent do the IPA II programmes add value compared to interventions by Member States or other key donors?

Information sought in this area includes:

·Where IPA II is operating in the same field as other donors, does it offer added value in terms of size of engagement, particular expertise, and/or particular weight in advocacy?

Coherence, consistency, complementarity and synergies

5. To what extent does IPA II facilitate coherence, consistency, complementarity and synergies both internally between its own set of objectives and programmes and vis-à-vis other EFIs?

Information sought in this area includes:

·To what extent are the different IPA II programmes coherent/complementing/overlapping with one another, including coherence between bilateral and multi-country programmes?

·To what extent are the different IPA II programmes aligned with EU enlargement policy?

·To what extent are the programmes consistent with EU external action policies?

·To what extent do the programmes complement/overlap/stimulate synergies with other external action financing instruments?

· To what extent does IPA II complement/overlap with other EU instruments outside of development and other external policies, notably the EU territorial cooperation programmes and the regional macro strategies?

·To what extent does the IPA II complement/overlap with interventions of other donors?

· To what extent policy dialogue and implementation of financial cooperation act in a synergetic way?

Leverage

6. To what extent has IPA leveraged

- further funds, and/or

- political or policy engagement?

7. How could IPA be enhanced to achieve its policy objectives more effectively and efficiently?

8. How can programming and implementation of IPA assistance be enhanced to improve the impact and sustainability of financial assistance?

Annex 3. Procedural information

Organisation

This evaluation assessed the Instrument for Pre-accession Assistance (IPA II) 71 ahead of the Mid-Term Review Report, as set out in Article 17 of the Common Implementing Regulation (CIR 2014). It will mainly be used to generate information for the Mid-Term Review Report requested by the CIR due end 2017. The evaluation provides information on relevance, EU added value, coherence and complementarity, effectiveness, efficiency, sustainability, leverage and impact of the instrument.

The Staff working document is based on the independent assessment carried out by an external contractor and complemented by further internal analysis.

The lead DG to carry out and manage this evaluation was the Directorate General for Neighbourhood and Enlargement Negotiations (DG NEAR). But as this evaluation was part of a wider set of evaluations covering the instruments under Heading 4 of the Multiannual Financial Framework 2014-2020 as set out in the CIR, as well as the performance review of the European Development Fund, and in view of ensuring a consistent European external policy, all the evaluations were carried out in an interlinked and co-ordinated manner. To ensure the coherence of the different evaluations from the beginning, the Directorates General in charge of external actions in the Commission worked closely together.

An Umbrella Inter-service Steering group (ISG) was set up to oversee the mid-term review process of the instruments. This ISG is chaired by Directorate General for International Cooperation and Development and is composed of members from relevant Commission services. In addition ISGs were set up for each individual evaluation. The process started in June 2015 and the Umbrella ISG met formally for the first time on 21 September 2015 and during the following weeks and months agreed on the joint elements which all the different evaluation should use in their specific roadmaps and Terms of reference.

The ISG for the IPA II evaluation was set up in October 2015, including relevant services of the Commission, and met for the first time on 15 October 2015. The roadmap for IPA II was published, like all the other instruments roadmaps, at the end of October 2015, no comments were received. The Terms of reference were approved by the ISG on 15 April 2016. A specific contract was awarded on 9 June 2016 to PARTICIP GmbH, using the framework contract set up for carrying out evaluations in the context of external relations, and the evaluators started their work on 28 June 2016.

As part of their work the evaluators produced an Inception Report, a desk report and a draft Report. All these reports were presented to the ISG meeting. ISG members then provided written comments to the evaluation manager in DG NEAR, who combined all the comments and communicated them to the evaluators. Revised versions were checked by the evaluation manager and further comments from the ISG were requested where necessary. Several ISG members also requested comments from their directorates/units where this was deemed necessary. During the evaluation process from October 2015 onwards the ISG met 7 times, in addition the members were consulted through email several times on draft and revised draft reports.

Following the approval of the Draft Report by the ISG after the meeting of 11 January 2017, this report was then placed, with the other instruments draft reports, on the Commission's website for the web based open public consultation (OPC) on 7 February 2017. The OPC concluded on 3 May 2017. During the OPC period targeted face to face meetings were organised with representatives of CSOs, both from the beneficiary countries and Europe, with representatives of EU Member States and European Parliament.

Following the closure of the OPC during which only 30 responses were received, the evaluators then submitted a Final Draft Evaluation Report on 15 May, this was discussed by the ISG on 18 May leading to the issuing of the final report.

Evaluation Design

Concerning the evaluation design, the intervention logic (IL) forms the backbone of the analytical framework. It is visualised through an overall IL diagram, which brings together the most relevant elements (from inputs to impacts, but also assumptions) in a single framework.

The critical analysis of the (reconstructed) IL helped to focus the evaluative framework (EQs, JCs and indicators) on key issues, particularly the underlying theory of change and the assumptions that ensure the operationalisation of the regulatory provisions.

Evaluation questions, which were very similar to all the simultaneously on-going instruments evaluation, structure the analysis to gather evidence. The evaluation questions, as well as more instrument-specific sub-questions, were provided in the Terms of Reference. Considering the objective of the various instruments evaluations launched under the mid-term review, consistency among the respective EQ/JC structure was prioritised up to the JC level. Each question was structured into judgement criteria (JC) and indicators required to provide an answer based on a synthesis of evidences.

Methodology and data

The evaluation of IPA II is evidence-based. The EU evaluation criteria (relevance, effectiveness, efficiency, added value of the EU, coherence, consistency, complementarity and synergies, as well as leverage) were applied as an underlying basis. The evaluation questions (EQs) from the Terms of Reference gave rise to a number of JCs and associated indicators. The evaluation is indicator-based. The evaluation used a mixed methods approach blending quantitative and qualitative methods, with a focus on the latter. The main analytical tools consisted of rigorous assessment of documentation and consultation of stakeholders (via interviews, group consultations, and the Open Public Consultation- OPC; key issues were also been addressed by a joint-instruments survey managed by one of the contractors in charge of preparing an overall coherence report).

For all EQs, data collection included various tools and methods. The combination of data collection methods and techniques varied according to the different EQs and JCs. Several methods and techniques were used to collect the necessary data to assess a given JC according to the nature of the set of indicators identified. Where possible, the Evaluation Team combined the use of qualitative and quantitative data and relied both on primary and secondary data sources while taking into account resource and time constraints. Priority was given to document review and interviews at HQ, in line with the methodological indications in the ToR. Data collection activities were carried out mainly during the desk phase, but also continued during subsequent phases.

The following analytical tools were applied in order to reply to the respective EQs:

EQ1 on relevance included an assessment whether 1) IPA II did (or did not) correspond to the objectives of the EU Enlargement strategy; and whether 2) IPA II did (or did not) correspond to the needs of beneficiary countries related to their stated EU priorities, given also the recent fundamental shift in the way how the programming of IPA II is conducted since 2014 (sector-based approach).

EQ2 on effectiveness examined the extent to which the conditions were in place for the achievement of programme objectives once implementation moves forward.

EQ3 on efficiency was also pitched at the instrument level and assessed the relative costs and outcomes of the current programming period. It also looked at process efficiency at the implementation level, including whether the recently enforced management and monitoring arrangements are designed and applied in a manner which ensures increased results orientation and allows for improved performance measurement.

EQ4 on value added was closely related to the idea of comparative advantage, particularly as regards EU MS. Value addition was also assumed to derive from complementarity and synergies with other donors, which were also assessed under EQ5.

EQ5 on the issues of coherence, consistency, complementarity and synergy was assessed at two levels: the conceptual level (scope and rationale of the instrument versus other instruments and EU policies) and the operational level (how is coherence, consistency, complementarity and synergy ensured at programming and implementation level).

EQ6 on leverage was assessed using two parameters: 1) IPA II financial assistance has been used for increasing political and policy engagement of the beneficiaries towards reforms but also in terms of better political coordination with EU MS and other donors; 2) the extent to which IPA II is able to leverage additional financial resources.

The evaluation of IPA II comprised three phases:

Desk phase:

Inception:

•    Kick off meeting and consultation with the EU in Brussels and consultations with other stakeholders, particularly in DG NEAR.

•    Presentation of the inception report to the EU in Brussels.

Desk:

•    Identifying and gathering information at the indicator level. Documents reviewed included policy and strategy papers, programme documentation from the CRIS/ MIS database, evaluations, studies and assessments. In addition to the document review, interviews were conducted by phone or in person with staff members from the European Commission and EU Member States to verify information, obtain leads to documents/new interviewees, and to discuss the Intervention Logic (IL).

•    Reviewing the EQs/ JCs to assess whether or not a revision was needed: The EQs and the JCs were not revised.

•    Refining the methods of analysis of the data collected, according to the preliminary finding and needs of the evaluation. Elaborating pathways of change for further testing.

•    Conducting the first step of the IL analysis by verifying whether: 1) the intended activities under the programme are actually implemented; 2) the IL, as depicted in the inception report, is agreed to by consulted stakeholders.

•    Elaboration of a detailed validation phase work plan and a list of people to interview.

Validation phase:

•    Further interviews with staff available in Brussels for DG NEAR, DEVCO, HOME, etc.

•    Interviews (phone) with EU MS members of the IPA II Committee.

•    Further systematic analysis and extractions from the documentation to fully answer JCs and indicators.

•    Targeted compilation of documentation directly related to each JC, in particular documents collected during the field visits.

•    Mission to all seven IPA II countries to conduct semi-structured interviews and/ or group discussions.

•    Drafting a separate CIR input according to the guidelines provided by DG DEVCO.

•    Based on the success of data collection, assessing whether there was need for further research and interviews to prepare the draft final report, and in particular the conclusions chapter.

Synthesis phase:

•    Preparing the (draft) final evaluation report, together with conclusions and recommendations.

•    Implementation of the Open Public Consultation. Results feed into the final report.

The main purpose of the field missions was to complete the data collection and to contribute to answering the EQs. The visits served to validate or revise the preliminary findings and hypotheses formulated during the desk phase and were in particular an opportunity to obtain more information on the actual status of implementation on the ground. The field missions were conducted as follows:

Issues related to the CIR were assessed in a common way across all external financing instrument evaluations. A questionnaire was developed and shared with the evaluation team, which was answered to feed into the CIR evaluation. Besides the review of documentation and interviews, a common survey was conducted for the different instruments evaluations. The purpose was to gather feedback from the EUDs on specific aspects of the CIR and on the coherence and complementarity of the instruments. All collected information was analysed in order to prepare a consolidated response to the CIR questionnaire.

Challenges and limitations

The main challenge of the evaluation was the very tight timeline dictated by the CIR, which resulted in risks related to access to data, documentation and availability of key respondents. The external evaluation team counterbalanced the risks by being highly flexible especially when gathering information. Countering the risks demanded considerable efforts from the external evaluation team, but the team managed to mitigate potential limitations by mobilising resources to carry out all tasks indicated in the methodology in parallel and analyse additional sources of information when available. Ultimately these challenges did not compromise the foundations of the evaluation findings. The quality of the collected evidence (documentation, interviews, data and survey results) for this evaluation can be overall assessed as good, demonstrating a satisfactory degree of confidence regarding the various findings of this evaluation.

The key limitation, which is inherent to any mid-term review, is that the outputs (i.e. IPA II funded programmes) of the Instrument have only recently entered the implementation stage. Therefore, effectiveness/sustainability/impact criteria could not be assessed based on results at this stage. The external evaluation team was, however, able to gain a first impression of these criteria based on the quality of the action documents and, for the five case studies, the congruence of the analysis of the context and the EU response strategy.

The ISG as well as other staff from DG NEAR, EEAS and IPA II delegations actively participated in providing missing data to the external evaluators. The multiple rounds of commenting also mean that facts have been verified and have facilitated the triangulation of data.

Overall, the quality of the collected evidence (documentation, interviews and survey results) for this evaluation can be assessed as good, demonstrating a satisfactory degree of confidence regarding the various findings of this evaluation.

Annex 4. Synopsis report of the stakeholders' consultation

3.1 Introduction

The evaluation of IPA II, together with the other independent evaluations of each External Financing Instrument, that of the Common Implementing Regulation (CIR) and the Coherence Report, are some of the sources of information to feed into the Mid Term Review Report (MTR) of the instruments. The MTR is required by the Common Implementing Regulation (CIR) Article 17, by end December 2017.

This Consultation strategy provides an overview of the approach that has been taken for consulting this IPA II evaluation with its main stakeholders. It contains two elements. The first is a stakeholder mapping for IPA II which outlines the main institutions or groups that are considered as ‘stakeholders’. The second is a stakeholder consultation strategy that the evaluation team has deployed, in consultation with the ISG, to engage with these stakeholders during the evaluation process, along with the timeline for its delivery. An important component of this consultation process has been the Open Public Consultation (OPC) done at the end of the synthesis phase of the evaluation. The OPC allowed acquiring feedback from all potentially relevant parties on the main evaluation findings, tough it was undermined by a low level of participation, which makes the feedback received not that significant.

3.2 Stakeholder Mapping

Below is a presentation of the main stakeholder groups in IPA II. They have been split into stakeholders operating at European level and those at national level.

Figure 1    Stakeholder mapping on European level

Figure 2    Stakeholder mapping on national level

3.3 Stakeholder Consultation Strategy

The stakeholder mapping for IPA II outlined the main institutions or groups that are considered as ‘stakeholders’. The developed stakeholder consultation strategy aimed at ensuring that the evaluation team could fully engage with all these stakeholders during the evaluation process. An important component of this consultation process was the open public consultation (OPC) done at the end of the synthesis phase of the evaluation to acquire feedback from all relevant parties on the main evaluation findings. Details on the implemented and completed approach are given below.

The approach taken by this evaluation to engaging with the scope of all these aforementioned stakeholders has been defined by their role in IPA II and their relative importance and influence over it. The consultation approach for the principal stakeholders identified in the above table has been as follows:

Commission Services and EEAS

The evaluation involved all the relevant DG NEAR geographical and thematic units throughout the desk and validation phases. DG AGRI has also been consulted throughout the evaluation in relation to IPARD II. Other DGs have been consulted during the desk and validation phases where specific instances required it and informed of evaluation results. EUDs have also been consulted and informed throughout all phases of the evaluation.

IFIs

A major IFI presence in IPA II is the WBIF. Its Secretariat has been consulted during the desk and validation phases to understand complementarity of this instrument with other related IPA II measures and has also been consulted in the synthesis phase. Other IFIs have been consulted in specific cases during the desk and validation phases, otherwise these have been informed of the evaluation preliminary results via the OPC.

International Organisations

Those international organisations implementing IPA assistance (CoE, UN agencies) have been consulted in-country in the validation phase as well as desk phase if judged necessary. The RCC was consulted in the validation phase and in the synthesis phase when preparing the draft evaluation report.

National Governments

The NIPAC, NAO and OS have all been consulted in the validation phase, as they have a central role in the delivery of IPA II in the beneficiaries.

Civil Society Organisations

These stakeholders have an active role in ensuring citizens are adequately represented in the formulation of IPA II actions and in overseeing as well as implementing them. They have been consulted at all relevant levels, particularly during the validation phase and the OPC.

3.4 Stakeholder Consultation Framework

Consultation with stakeholders took place via the following means:

·Interviews (face-to-face and via phone) and group discussions with various stakeholders at HQ level as well as via field missions to all seven IPA II beneficiaries;

·Instruments-wide survey to EU delegations (coordinated by the chapeau team);

·Open Public Consultation (OPC) via web and face-to-face;

Desk Phase    

The evaluators prepared the inception report and submitted it to the client for its approval. At this stage of the evaluation, no further stakeholder consultation took place. During the actual sub-phase of desk work, stakeholders as identified above have been closely consulted as appropriate, preferably via phone/email/ face-to-face /video-conference discussions. By the end of the desk phase, a Desk Report has been prepared and discussed with the ISG.

Validation Phase

In line with the assessment provided in the table above, the evaluation team has consulted with those stakeholders identified as having a direct role in IPA II. The evaluators contacted these stakeholders with a view to acquiring information for the evaluation via the appropriate evaluation tools e.g. interviews, focus groups, questionnaires.

Synthesis Phase

Towards the end of the synthesis phase, the evaluation team has prepared a set of key preliminary findings and conclusions based on its analysis from the validation phase. This formed a central part of the OPC process required by the ToR.

Report dissemination

Once the final evaluation report has been submitted, the ISG will decide on dissemination of the final report.

3.5 Stakeholder statistics

This section provides an overview of how many stakeholders have been consulted during the different phases of the evaluation, where they were consulted (HQ level or in the field), as well as which stakeholder groups they belonged to.

General

A total of 352 stakeholders have been interviewed to inform the results of this evaluation.

58 interviews took place at HQ level, namely in Belgium, Luxembourg or via phone. The remaining 294 interviews were conducted during the field missions to all seven IPA II beneficiaries.

The interviews conducted covered a large variety of stakeholders on EU level as well as international, regional and national level. The following stakeholder groups were consulted:

Field missions

The duration of the different field visits was between 2,5 and 4 days. The following table shows the number of stakeholders consulted per country:

3.6 OPC Contribution

3.6.1 Introduction

The draft evaluation report on the Instrument for Pre-accession Assistance (IPA II) 72 was posted on the website of the European Commission for an Open Public Consultation (OPC) between 7 February and 5 May 2017 73 , together with the evaluation reports of all other External Financing Instruments (instruments). All stakeholders in beneficiary and EU countries were welcome to participate in this process. The objective of the web-consultation was twofold:

·To gather feedback from the broadest possible range of stakeholders, including those in beneficiaries and in the EU Member States, on the emerging conclusions from the evaluations.

·To gather preliminary ideas on the future external financing instruments after the current ones have expired by 31 December 2020.

From the web OPC, a total of 30 contributions were received which related specifically to IPA II. Most contributions came from public authorities (12 in total) and organisations or associations (10). Further contributions were made by citizens/individuals (3), research/academia (3) and EU platforms, networks or associations (2), as illustrated by the graph below.

Figure 3    Type of contributors to IPA II via the web OPC

In the framework of the web OPC, contributors were invited to respond to the following four IPA II-specific questions:

·Question 1: How well do you think the IPA II has addressed its objectives? The main assessment criteria for the evaluation are: relevance; effectiveness, impact and sustainability; efficiency; EU added value; coherence, consistency, complementarity and synergies; and leverage. Feel free to comment on the findings, conclusions or recommendations for any/all of the criteria.

è24 answers received

·Question 2: To what extent do you think the strategic orientation of IPA II and the political dialogue between the EU and the Governments in beneficiary countries is adequately addressing the desired improvement of a truly inclusive public consultation?

è15 answers received

·Question 3: To what extent do you think the strategic orientation of IPA II and the political dialogue between the EU and the Governments in beneficiary countries are adequately addressing the key issue of fight against corruption?

è15 answers received

·Question 4: If you have any other views on the IPA II you would like to share, they are welcome here.

è22 answers received

The following summaries synthesise the main contributions received for each question from the web OPC.

3.6.2 Summary of OPC contributions

3.6.2.1.1 Question 1: Addressing IPA II objectives

3.6.2.1.2 Summary of contributions

Out of the 30 contributors which responded to the IPA II-specific questions, 24 provided a contribution to Question 1.

The contributions were categorized as being positive, mixed, negative or neutral. There is a balance between positive (6) and negative (5) assessments, with mixed (9) being the most used category. Four contributions were neutral.

The following bullet points provide a summary of the contributions received 74 :

Positive

·IPA II is well aligned with the EU’s enlargement strategy and increases its strategic relevance;

·IPA II has a better strategic focus than IPA I;

·The priorities set for IPA II reflect the priorities for the region;

·IPA II objectives are well-defined;

·IPA II addresses its objectives well and its priorities and selected projects are relevant;

·Security-related objectives and priorities appear in all relevant IPA II documents and are clearly defined.

Negative

·Lack of a suspension clause in the IPA II Regulation;

·Lack of transparency about IPA’s absorption capacities;

·Long administrative procedures;

·Absence of joint programming between IPA II, EIDHR and IcSP;

·The IPA Committee’s role is often too technical;

·CSOs have difficulties to obtain funding (e.g. co-financing difficulties, changing regulations for applying to IPA funding, procedures creating barriers for grassroots organisations, too little funding for CSOs);

·Lack of comprehensive horizontal issues (Roma, gender, child rights, disability aspects, climate action, security issues) in strategy papers and programming.

Recommendations

·Establishment of a comprehensive and transparent database to inform the MS about budget execution, implementation of measures and projects, absorption and backlogs;

·Better coordination and coherence between IPA II and the development cooperation of other MS and donors for more effectiveness;

·The integration of horizontal themes should be increased (e.g. for gender: gender analyses, gender disaggregated data for all sectors, gender responsive budgeting, alignment of IPA II with the GAP II);

·Coordination such as the WBIF and CoTEs should be well-resourced;

·IPA II after 2020 should not contain a stand-still clause, as this risks to become a tool for political pressure.

In summary, the comments received are largely in line with the findings of the evaluation.

3.6.2.1.1 Question 2: Addressing inclusive public consultation

3.6.2.1.2Summary of contributions

Out of the 30 contributors which responded to the IPA II-specific questions, 15 provided a contribution to Question 2.

The majority of contributions assessed the public consultation processes as negative (8). Only two contributors assessed this question positively, two gave a mixed assessment. Three contributions remained neutral.

The following bullet points provide a summary of the contributions received 75 :

Positive

·The strategic orientation of IPA II has improved over its predecessor due to the sector approach.

Negative

·Involvement of non-state actors in the implementation of IPA II significantly affects the level of public consultations, but there is only limited access to IPA II funds for civil society;

·IPA II and the political dialogue between the EU and governments lacks structure for truly inclusive public consultations, as public consultations and dialogue are still done ad-hoc, based on personal connections and initiatives of individual EU representatives, or thanks to CSO initiatives, rather than through a systematic approach;

·Processes are too complicated for a real inclusive public involvement;

·CSOs are often excluded from governmental consultations;

·The participatory processes are often of bad quality, e.g. when during the public consultation on IPA II SPDs only a limited number of NGOs are consulted over a short period of time;

·At national level, public consultations are sometimes completely omitted;

·As the subject of IPA II is highly political and has low impacts on the public’s everyday life, the majority of the population is not in a position to express a concrete opinion on the policy.

Recommendations

·The role of civil society in the beneficiaries should be improved to counterbalance governments when implementing reform programmes;

·The EU should reinforce its conditionalities; they should be concrete, measurable and realistic. They should be made public to the extent feasible;

·The political dialogue related to pre-accession negotiations should take into account non-discrimination, disability and accessibility issues;

·DG NEAR should not accept SPDs if they have not passed a truly inclusive public consultation and if they are not of sufficient quality and in line with EU policies and objectives;

·The EC should step up its efforts in order to maximize the EU’s support for civil society in Turkey via IPA II funds.

3.6.2.1.3 Response of the Commission

IPA II is making considerable efforts in strengthening public consultation with the civil society and other actors. Still, the process is at the beginning and differs widely among beneficiaries and sectors.

A new performance framework is being set up for the instrument, which gradually should also improve transparency of budget executions and a true results-oriented reporting.

With the performance reward the instruments already foresees a mechanism to reward/penalise beneficiaries having regard to actual performance.

Also based on past evaluations, consultation with CSOs has already improved. .

The political dialogue with beneficiaries is strongly based on the European values, inter alia also taking into account non-discrimination, disability and accessibility issues.

For Turkey, civil society support has been identified as one of the important areas of future IPA II funding (for instance increased support towards rights-based CSOs/ activists).

3.6.3.1.1 Question 3: Addressing fight against corruption

3.6.3.1.2 Summary of contributions

Out of the 30 contributors which responded to the IPA II-specific questions, 16 provided a contribution to Question 3.

The above graph shows that the opinions of the contributors are almost evenly divided in positive (5) and negative (7). Only one contributor gave a mixed assessment of IPA II’s contribution to the fight against corruption. Three contributors remained neutral in their answers.

The following bullet points provide a summary of the contributions received 76 :

Positive

·The strategic orientation of IPA II and political dialogue largely take account of the key issues of the fight against corruption;

·The financial support via PA II to improve the legislation against corruption is an important step to reduce corruption;

·IPA II funding has made a positive contribution towards the fight against corruption, and some progress has been made in the beneficiaries;

Negative

·The fight against corruption is still not adequately addressed;

·The fight against corruption is impeded by the fact that corruption is entrenched in the mentality in the beneficiary states;

·The increased use of budget support does not help fighting corruption; it rather increases;

·As long as CSOs are generally excluded from governmental dialogues and consultations & as long as governmental agencies are working with closed doors, it is difficult to fight against corruption in the candidate countries and potential candidates;

·The complexity of the tools and processes at hand does not facilitate the fight against corruption; it rather facilitates corruption;

·Weak impact on the fight against corruption in the field, including almost inexistent monitoring of corruption issues.

Recommendations

·All rule of law issues, including fight against corruption, should be a main focus of IPA II;

·The IPA Committee should be informed about cases of corruption. Reports should be linked to actions on operational level (e.g. amendments of programmes, action programme, performance reward, etc.).

3.6.3.1.3 Response of the Commission

In line with revised strategic approach for the instrument, addressing “fundamentals first”, also the fight against corruption is now more in the focus of pre-accession assistance compared to IPA I.

Corruption is a systemic problem for many beneficiaries. However, pre-accession support is only one tool among others to effectively address this problem.

Corruption is a not a particular problem for the actual delivery of pre-accession support, it is primarily related to the given beneficiary and its prevailing culture and public and private structures and systems. To some extent, the instrument tries to address these systemic issues also through increased political and policy dialogue but results have still to materialise.

3.6.4.1.1 Question 4: Any other views on IPA II

3.6.4.1.2 Summary of contributions

Out of the 30 contributors which responded to the IPA II-specific questions, 22 provided a contribution to Question 4.

The following bullet points provide a summary of the contributions received 77 :

·Strategic communication and public diplomacy should be increased and improved, regarding the values of the EU as well as the EU’s added value. The information strand projects under IPA II should be strengthened to make the IPA II activities of the EU more visible and to counterbalance the propaganda activities of third countries. A European narrative should be constructed which at the same time informs of the benefits and requirements of EU membership. Improving the visibility and transparency of EU measures and better communication, outreach and visibility are central points for all operations and documents. Visibility could also be improved through the establishment of “quick-impact flagship projects” in the candidate countries and potential candidates.

·Transparency of budget execution should be improved, incl. the establishment of an adequate database. The IPA Committee should be informed more timely about new policies, financial acts and draft action programmes.

·EU funding awarded through IPA should shift from small investments in many areas towards investing into systemic changes of a sector in order to build sustainable impact and the EU should apply the lessons learned during the previous EU accession process and support its Member States to share their transition experience.

·The IPA II Regulation lacks sufficient procedures to take into account negative tendencies in relation to the Copenhagen criteria, notably on democracy and human rights, because of the absence of a suspension clause.

·Consultation with CSOs is an important component of IPA assistance programming which should continue and be developed even further.

·Horizontal issues should be better addressed (e.g. disability issues, child protection)

·IPA funding to women’s CSOs is problematic, as it does not reach grassroots’ organisations and it is not flexible regarding methods and expected results; EU funding falls under similar smear attacks in several IPA beneficiaries and for that reason it is viewed as a problematic source; IPA administration requirements are too high/rigid; and some of the funding to women human rights defenders / women peacebuilders is channeled through UN Women in a re-granting scheme, but UN Women is neither an expert of the specific country / conflict contexts, nor of practice on the ground, and is therefore highly criticized by those CSOs that are in most need of IPA-support.

·IPA priorities should be evidence-base driven: the EU should support the candidate countries and potential candidates in areas where evidence shows that increased efforts are needed.

·The introduction of the Centres of Thematic Expertise (CoTE) in DG NEAR to mainstream horizontal issues already in early planning phases is crucial.

3.6.3.1.3 Response of the Commission

Based on the introduced changes and new orientations, IPA II is expected to strengthen also communication and visibility.

The instrument also recognises the need for effective policy and political dialogue in order to make pre-accession support fully successful.

A new performance framework is being set up for the instrument, which gradually should also improve transparency of budget executions but also the results-oriented reporting.

IPA II is more inclusive than its predecessor but this has so far mostly been limited to programming aspects.

IPA II puts emphasis on a limited number of key sectors, providing concentrated funding there, including sector budget support, to increase impact and sustainability.

The involvement of EU MS is taking place for instance through Twinning or TAIEX or delegation of management tasks to accredited MS agencies but could still be increased.

The need to strengthen the tackling of horizontal issues has been recognised by DG NEAR, particularly by introducing CoTEs and focal points. This process works already well in some sectors but not in all, including in areas like supporting women CSOs.

In conclusion, IPA II recognises many of the critical comments raised above and some of the raised aspects in line with the instrument’s implementation in the field.

3.6.3 Face-to-face consultations

As part of the public consultation, the findings of the IPA II draft report were discussed face-to-face on several occasions as follows:

·23 March 2017, Civil Society Organisations (CSOs) and Local Authorities (LAs) from the European Union and partner countries in the framework of the Policy Forum on Development Meeting;

·24 March 2017, with the council working groups COWEB and COELA;

·28 March 2017, with representatives from the European Parliament and the EU Member States.

3.6.3.1 Summary of contributions

During each meeting, a number of issues were raised with regards to the evaluation findings, but also to IPA II in general. Comments were made by different stakeholders from the Council of the European Union, European Parliament, Member States and Civil Society Organisations.

The following bullet points summarize the main issues raised during the face-to-face consultations:

·Visibility aspects of IPA II, such as communicating better the added value of the EU support within the beneficiaries;

·Transparency issues related to budget execution, absorption rates, allocated and paid amounts, etc. and the need to strengthen administrative data collection, monitoring systems and sector-based management information systems,

·The lag between programming and implementation;

·Coordination, synergies and knowledge sharing between IPA and other instruments, IFIs, MS and other donors and development actors;

·The need to take into account different actors in the field and to involve them into the implementation;

·The limited support of IPA II to civil society and local governments and the need to provide more support to civil societal actors in the beneficiaries;

·The lack of a suspension clause in the IPA II Regulation;

·Possibilities of monitoring the success of projects also after their completion;

·The response to low performance under the performance framework;

·Issues of corruption in the public sector and the private sector;

·The importance to focus on the rule of law and key reforms in economy and administration;

·Stability issues and how to build stability in the beneficiaries.

3.6.3.2 Response of the Commission

There is a clear requirement now under IPA II to strengthen also communication and visibility, revised procedures and intensified actions are being launched at the time of this evaluation.

A new performance framework is being set up for the instrument, which gradually should also improve transparency of budget executions but also the results-oriented reporting.

IPA II is more inclusive than its predecessor but this has so far mostly been limited to programming aspects. The inclusiveness of the implementation process still needs to materialise.

Measures to tackle low performance have been applied in the past and might be used even more in the future, based on objective performance data. Discussions about an increased use of sanction mechanisms are currently ongoing. Suspension, if it is to be more strongly implemented in future, will also need an accompanying strong policy dialogue.

Final/ ex-post evaluations are already important tools to supplement the performance review. Also, the Results-oriented Monitoring allows for the review of actions being at the final stage.

The focus on the rule of law and other key reforms is a major feature of the new instrument.

Stability is a basic requirement for effectively delivering pre-accession assistance but cannot be addressed solely by the instrument.

In conclusion, IPA II potentially offers many of the features required by the given comments. It needs still to be seen to which extent these requirements can be addressed in the given context of beneficiaries and their prioritised sectors.

Annex 5. Acronyms

List of acronyms and abbreviations

Annex 6. External evaluators' report, including its annexes and methods and analytical models used

"The external evaluation can be found here: https://ec.europa.eu/europeaid/public-consultation-external-financing-instruments-european-union_en "

(1)

 Regulation (EU) No 231/2014 of the European Parliament and of the Council of 15 March 2014, OJ L77, p. 11.

(2)

Articles 1 and 2 of the IPA II regulation

(3)

The Development Cooperation Instrument, the 11th European Development Fund which is outside of the EU budget, the European Neighbourhood Instrument, the European Instrument for Democracy and Human Rights, the Greenland Decision, the Instrument contributing to Stability and Peace, the Instrument for Pre-accession Assistance, the Instrument on Nuclear Safety Cooperation, the Overseas Countries and Territories Decision, the Partnership Instrument and the Common Implementing Regulation. For the purpose of this exercise, the evaluation of the Overseas Countries and Territories Decision is included within the evaluation of the 11th European Development Fund.

(4)

As provided for in Article 17 of the Common Implementing Regulation, Regulation (EU) No 236/2014 of the European Parliament and of the Council of 11 March 2014, OJ L77, p. 95.

(5)

Though there was no specific analysis for this country.

(6)

* This designation is without prejudice to positions on status, and is in line with UNSCR 1244/1999 and the ICJ Opinion on the Kosovo declaration of independence.

(7)

 Commission Communication Better regulation for better results – An EU Agenda, COM (2015) 215, and Commission Staff Working Document Better Regulation guidelines, SWD (2015) 111.

(8)

Council Regulation (EU, EURATOM) No 1311/2013 of 2 December 2013 laying down the Multiannual Financial Framework for the years 2014-2020, OJ L 347/884, p. 884.

(9)

The Multi-annual Financial Framework is divided into six broad groups of expenditure called "Headings". The External Financial Instruments make up the majority of Heading IV: Global Europe.

(10)

IPA was divided into five components, with the two first - on transition assistance, institution building and cross-border cooperation- open to all beneficiaries, the three others - on regional development, human resources development and rural development - only open to candidate countries.

(11)

https://ec.europa.eu/europeaid/public-consultation-external-financing-instruments-european-union_en

(12)

For Bosnia and Herzegovina only for the 2014-2017 period.

(13)

137 projects are subject to a ROM review per year, on average.

(14)

See https://ec.europa.eu/neighbourhood-enlargement/news_corner/key-documents_en for the evaluation plan and published report.

(15)

Evaluation report, page i.

(16)

JC 11, indicator 111 of the Evaluation report, page 9. See also https://ec.europa.eu/neighbourhood-enlargement/countries/package_en for the Enlargement Package.

(17)

Evaluation report, page 8.

(18)

JC 13, indicators 132-133 of the Evaluation report, pages 9-10. A dedicated strategic evaluation is ongoing to better apprehend the limitations and perspectives related to the implementation of sector approaches in the region.

(19)

Evaluation report, page 11.

(20)

JC 13, indicator 133 of the Evaluation report, page 10.

(21)

See for instance the Commission Implementing Decision C(2014)9797 final of 17.12.2014 adopting a special measure on flood recovery and flood risk management in Albania, Bosnia and Herzegovina, the former Yugoslav Republic of Macedonia, Kosovo*, Montenegro, the Republic of Serbia and Turkey: https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/pdf/financial_assistance/ipa/2014/multi-country/ipa_ii_2014-037-703_2014-037-788_2014-37-853_flood_recovery_and_risk_management.pdf

(22)

Commission Decision C(2015)9500 of 24.11.2015 on the coordination of the actions of the Union and of the Member States through a coordination mechanism – the Refugee Facility for Turkey, as amended by the Commission Decision C(2016)855 of 10.2.2016.

(23)

Commission Decision C(2015)9691 final of 21.12.2015 amending Decision C(2014)9615 final on establishment of a European Union Regional Trust Fund in response to the Syrian crisis, "the Madad Fund".

(24)

Evaluation report, vol. 1, Executive Summary, and vol. 2, par. 6.2.1

(25)

Evaluation report, vol. 2, page 248 and 250.

(26)

JC 21, indicators 211.1 and 211.2 (page 12), indicator 215.1 (page 14) of the Evaluation report.

(27)

As evidenced by the intensification of the meetings of the special groups on Public Administration Reform.

(28)

JC 21, indicator 211.2 (page 12) of the Evaluation report.

(29)

page 11 of the Evaluation report.

(30)

A specific evaluation on sector approaches has been launched in 2017 by DG NEAR and is at the desk phase.

(31)

page 11 of the Evaluation report, JC 21.

(32)

Evaluation report, vol. 2, page 255.

(33)

Evaluation report, vol. 2, page 252.

(34)

JC 21.6, indicators 216.1 and 216.2 page 14 of the Evaluation report

(35)

The rural development component managed, by DG AGRI. See Evaluation report, vol. 2, page 3 " As IPARD money can only flow after management powers have actually been conferred, the absorption rate has initially been low. However, as management for some measures has now been conferred for all of the three beneficiary countries, the overall uptake of IPARD funds is moving in an upward direction.”

(36)

JC 31, indicators 311, 312, page 19 of the Evaluation report

(37)

See evaluation report, vol. 2, page 255,

(38)

Nonetheless, available statistics do not show a significantly different path in terms of delays of payments between the two management modes (evaluation report, vol. 2, pages 114-115).

(39)

page 18 of the Evaluation report and JC 32, indicators 323 and 333.

(40)

Evaluation report, vol. 2, page 248.

(41)

Evaluation report, vol. 2, page 255.

(42)

page 20 of the Evaluation report.

(43)

This appeared also from comments in the framework of the OPC (evaluation report, vol. 2, page 250).

(44)

page 18 and page 20-21 JC 33, indicator 332 of the Evaluation report

(45)

page 19 of the Evaluation report. Moreover, “with the introduction of the modality of Sector Budget Support (SBS) (the Albanian authorities) have stopped requesting more indirect management, because the SBS under the sectoral approach provides more flexibility to the National Authorities to manage the EU funded interventions in a more coordinated way, servicing the National goals” (evaluation report, vol. 2, page 67.

(46)

Evaluation report, pages iii-iv.

(47)

These indicators address the impact dimension and are based on international sources, like World Bank, World Economic Forum, Transparency International, etc. They address the key areas of support (like Rule of law, public administration reforms, business environment, etc. The beneficiaries had been requested to set out targets representing the level of progress expected in the different areas. On 01/09/2015 the Turkish authorities confirmed the intention not to come up with targets with reference to some indicators (like the one on Press Freedom).

(48)

  https://myintracomm.ec.europa.eu/dg/near/whatwedo/monitoring-reporting-evaluation/Pages/Monitoring-evaluation.aspx

(49)

Evaluation report, vol. 2, page 166.

(50)

Twinning is a specific tool set out in the Enlargement area to support beneficiary administrations by means of experts coming from the European public administrations (https://ec.europa.eu/neighbourhood-enlargement/tenders/twinning_en).

(51)

Technical Assistance and Information Exchange instrument of the European Commission

(52)

page 23 of Evaluation report, and Evaluation of the Western Balkans Investment Framework (2015) . https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/pdf/financial_assistance/phare/evaluation/2015/2014_352812_1_final_evaluation_report.pdf

(53)

Evaluation report, vol. 2, page 77.

(54)

Chapter 3.4.2, pages 24-25 of the Evaluation Report

(55)

page 25 of the Evaluation report

(56)

Volume 2, annex 6, page 253-254 of the Evaluation report

(57)

page 26 and JC 52 page 28 of the Evaluation report

(58)

page 26 of the Evaluation report

(59)

For example, the regional housing programme financed by the multi beneficiary programme

(60)

page 25 and JC 54 of the Evaluation report

(61)

page 25 of the Evaluation report

(62)

Western Balkan Investment Framework

(63)

Enterprise Development & Innovation Facility

(64)

Green for Growth Fund

(65)

JC 63, indicator 631

(66)

Evaluation report, vol. 1, par. 3.6.

(67)

i.e. how it all fits together

(68)

Evaluators will need to look at both the current IPA II 2014-2020 and the previous IPA I 2007-2013 to respond to this question. Evaluators should distinguish the findings between the two periods.

(69)

Evaluations will need to compare, where possible, information from the current IPA II 2014-2020 with the previous IPA I 2007-2013.

(70)

See Title 22, item 22-01-04-01 of the latest, 2016 draft budget http://eur-lex.europa.eu/budget/data/DB/2016/en/SEC03.pdf

(71)

Regulation (EU) No 231/2014 of the European Parliament and of the Council of 15 March 2014, OJ L77.

(72)

Regulation (EU) No 231/2014 of the European Parliament and of the Council of 11 March 2014 establishing an Instrument for Pre-accession Assistance (IPA II).

(73)

  https://ec.europa.eu/europeaid/public-consultation-external-financing-instruments-european-union_en  

(74)

Eight of the 24 contributions were marked with „cannot be directly published but may be included within statistical data”; their content is not included in the summary.

(75)

Five of the 15 contributions were marked with „cannot be directly published but may be included within statistical data”; their content is not included in the summary.

(76)

Six of the 16 contributions were marked with „cannot be directly published but may be included within statistical data”; their content is not included in the summary.

(77)

Six of the 22 contributions were marked with „cannot be directly published but may be included within statistical data”; their content is not included in the summary.