EUROPEAN COMMISSION

Brussels, 1.4.2019

SWD(2019) 145 final

COMMISSION STAFF WORKING DOCUMENT

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS

European Structural and Investment Funds 2014-2020
2018 Summary report of the programme annual implementation reports
covering implementation in 2014-2017

{COM(2018) 816 final}

STAFF WORKING DOCUMENT

ACCOMPANYING THE 2018 SUMMARY REPORT

ON THE EUROPEAN STRUCTURAL AND INVESTMENT FUNDS 2014-2020

Table of Contents

1.INTRODUCTION

2.ERDF AND COHESION FUND

2.1. Synthesis of evaluation work completed by the Member States

2.1.1.Findings from evaluations on the 2014-2020 period

2.1.2.Findings from evaluations on the 2007-2013 period

2.2. Synthesis of ERDF/CF evaluation work by the Commission

3.ESF AND YEI

3.1. Synthesis of ESF/YEI evaluations completed by the Member States

3.1.1.Findings from evaluations on the 2014-2020 period

3.1.2.Findings from evaluations on the 2007-2013 period

3.2.Synthesis of ESF evaluation work by the Commission

3.2.1.Update of the 2007-2013 ex-post data

3.2.2.Launch of two thematic evaluations

4.EAFRD - EUROPEAN AGRICULTURAL FUND FOR RURAL DEVELOPMENT

4.1. Synthesis of evaluation work by the Member States

4.1.1.Period 2014-2020

4.1.2.Ex-post evaluations 2007-2013

4.2. Synthesis of EAFRD evaluation work by the Commission

5.EMFF - EUROPEAN MARITIME AND FISHERIES FUND (EMFF)

5.1. Period 2014-2020

5.2. Period 2007-2013

6.CONCLUSION

1.INTRODUCTION

This Staff Working Document provides a complementary synthesis of the evaluation work undertaken by the Member States and the Commission in order to assess the impact of the programmes supported by ESI Funds 1 .

The legislation for the period 2014-2020 sets out a stronger emphasis on the need to evaluate the contribution to growth, sustainable development and job creation of the policies co-financed through the ESI Fund programmes. The programmes define objectives that are specific and articulate the change sought in particular areas. Evaluations are essential to try establishing whether those changes have taken place and whether and how the programmes have contributed.

The Staff Working Document accompanying the 2017 Summary Report 2 synthesised the evaluation work expected under the national and programme evaluation plans, highlighting a good level of completeness and coherence with regard to coordination, management and communication, with improvements needed on skills, methods and data requirement. An updated analysis, focused on the revised evaluation plans received by the Commission until June 2018, shows that there are still shortcomings in the latter areas.

The number of completed evaluations published by Member States has increased significantly since last year, although at a lower level than planned both for implementation- and progress-oriented evaluations and, to a larger extent, for impact evaluations. Further increase of completed evaluations is expected in 2018-2019, although their number will be probably revised from what was initially planned in order to take into account the actual pace of implementation.

The impact evaluations that have been undertaken so far are aimed not only to assess the effects of programmes but also their implementation and progress towards achieving the targets set. Overall, if compared to planned evaluations, they show on average clearer evaluation design and more appropriate use of data.

However, an in-depth analysis of the reliability of impact evaluations, related to both programming periods, showed that only a limited number of them can be considered as fully reliable, which highlights the need for improvement in the quality of the works produced by the Member States, to which the Commission is committed to provide necessary support.

First evidence is available from the Member States on the period 2014-2020, mostly for ESF/YEI and show positive impact on employment of the various measures evaluated so far, while impact evaluations on the other Funds are still very limited, due to the scope and nature of the actions funded. For the 2007-2013 period, impact evaluations show that most investments would have not been carried out without EU support and their success requires strong coordination of different measures and cooperation among stakeholders. There is also evidence that investments have led to a greater attractiveness of the interested areas, although with some exceptions. Support to SMEs has increased employment, productivity and exports and can be more effective when complemented with non-financial support.

In the sections that follow an update and synthesis of the evaluations concluded by the Member States and the Commission are provided for each of the ESI Funds.

2.ERDF AND COHESION FUND

2.1. Synthesis of evaluation work completed by the Member States

The national and regional programmes devoted considerable attention to the topic of evaluation. The majority of the evaluations identified since January 2015 are ex-post evaluations of the 2007-2013 programming period, although the number of evaluations concerning 2014-2020 is now increasing faster, as almost 60% identified in the last 12 months until June 2018 are referring to 2014-2020, including those referring to both periods.

Table 1: Evaluations published since January 2015 on Cohesion policy interventions; breakdown by programming period

The evaluation effort is concentrated in some Member States. In particular, Poland (158), Spain (60), Hungary (26), UK (25), Lithuania (20), Italy (19), Slovakia (15), Czech Republic (14), Germany (12) and Romania (12) make up almost 90% of all evaluations identified. The high variability in the number of published evaluations among Member States depends not only on the different intensity of evaluations planned on the various thematic objectives but also on the different timing of the planned evaluations.

Figure 1: Evaluations published since January 2015 by Member State

Concerning the 2014-2020 period, most of the evaluations identified are either process/implementation oriented or look mostly at the progress of the programmes and do not yet allow the assessment of the contribution of the programmes to the expected results. For this reason, the following paragraphs will dwell mostly on the 2007-2013 evaluations.

Although the number of evaluations identified has increased substantially in the last year, it is still behind the level initially planned. Overall, the number of evaluations identified until June 2018 is approximately 50% of the planned value for the end of 2017, for impact evaluations, which require a more advanced level of implementation, the figures are even lower. From the evaluation plans, a more conspicuous number of impact evaluations related to 2014-2020 programmes is expected in 2018-2019, although these figures will be probably revised to take into account the actual pace of implementation.

As the goal of this document is to bring to light the effects of the interventions supported and the extent to which cohesion policy has contributed to economic and social development, the following synthesis is based on the findings of impact-oriented evaluations. These include evaluations dealing partly with process/implementation or monitoring/progress issues in addition to those focussed on impact alone.

In order to draw messages that can be relevant to the policy, an in-depth analysis was carried out to take into account the extent to which the findings of the reports appear to be reliable. The evaluations were inspected on the basis of a set of quality criteria, such as the clarity and suitability of the evaluation design, the appropriateness and correctness of the techniques applied, data quality and availability, the validity of the findings and of the conclusions 3 .

Overall, if compared to the quality rating of planned impact-oriented evaluations, the evaluations reviewed show on average clearer evaluation design and more appropriate use of data. The subset of the most reliable impact evaluations includes approximately 30% of those scrutinised. This proportion is based on a limited number of cases and therefore cannot be generalised, however it highlights the need for improvement in the quality of the works produced by the Member States.

Table 2: Quality rating of planned impact-oriented evaluations – Breakdown by Fund, including ESF/YEI

Table 3: Quality rating of impact-oriented evaluations reviewed, including ESF/YEI

The following summary of findings is based on a sample of 42 reliable impact evaluations, of which 41 related to the period 2007-2013 and one to the period 2014-2020.

2.0.0.Findings from evaluations on the 2014-2020 period

So far, the number of impact evaluations completed on ERDF/CF is extremely limited and only the findings of one evaluation can be reported entirely on this period 4 . Such evaluation spans across nine Swedish OPs and is focussed on the shift towards a low-carbon economy (TO4). The main findings show that while, in overall terms, projects were in line with the intervention logic, there was limited coordination with relevant actors, an excessive fragmentation of funding on small projects and, in some cases, the use of inadequate indicators.

2.0.1.Findings from evaluations on the 2007-2013 period

The 41 reliable impact evaluations of 2007-2013 programmes cover all Thematic Objectives except TO5 “Promoting climate change adaptation, risk prevention and management”, TO8 “Promoting sustainable and quality employment, supporting labour mobility” and TO11 “Enhancing institutional capacity of public authorities and stakeholders and efficient public administration”. They have been grouped into three main policy areas:

·R&I, ICT and SME competitiveness:

oTO1-strengthening research, technological development and innovation;

oTO2-enhancing access to, and use and quality of, ICT;

oTO3-enhancing the competitiveness of SMEs.

·Low carbon economy, climate change, environment and transport and energy networks:

oTO4-supporting the shift towards a low-carbon economy in all sectors;

oTO6-preserving / protecting the environment and promoting resource efficiency;

oTO7-promoting sustainable transport and removing bottlenecks in key network infrastructures.

·Social inclusion and education:

oTO9-promoting social inclusion, combating poverty and any discrimination;

oTO10    -investing in education, training and vocational training for skills and lifelong learning by developing education and training infrastructure.

In addition, a quarter of the evaluations were aimed at estimating the effects of the 2007-2013 programmes on the economy by using econometric models which implicitly cover all TOs but which do not provide any specific findings in relation to these. Such measures are included under a separate area ‘All/Multi TOs’. The reliability of the findings in these cases depends on the validity of the model used and its ability to represent the workings of the economy, which remains uncertain.

R&I, ICT and SME competitiveness

The Polish evaluation on the effects of interventions co-funded by the national OP Innovative Economy and aimed at increasing technical R&D and innovation (TO1) capacity of SMEs by upgrading research infrastructure, found that overall the support was one of the main elements which enabled modernisation of research infrastructure on a large scale and helped international research cooperation to be established and intensified. Nevertheless, this did not translate into a significant increase in the transfer of knowledge within the economy, although effects of this kind are expected only after some time. Equally, the maintenance of the infrastructure created is perceived as being potentially problematic in the long run.

The Scottish Co-investment Fund (SCF) supported by the regional investment bank and the OP ERDF Lowlands and Uplands, invested EUR 52 million in 139 business between 2009 and 2013 with the aim of promoting business investment in R&I. The evaluation assessing the effects of the investment fund found that it produced significant economic effects at regional and national level because of a high level of additionality, a low displacement rate and strong export performance.

Another UK evaluation focused on the support provided by the Welsh project “High Performance Computing (HPC)” to strengthen R&I and ICT. It found that the Special Purpose Vehicle provided an appropriate system on which to build a long-term sustainable model for increasing technological infrastructure, in-house expertise in applications and domain-specific support. It was also successful in providing services to businesses, students, researchers and academics. However, the project suffered an initial delay which affected the achievement of the targets set and arrangements had to be put in place to ensure a smooth transition between the first and the second phase.

Among evaluations which focused strictly on ICT (TO2), the evaluation of activities co-funded by the ERDF to develop this TO in Poland found that measures supported provided access to the internet for around 9 million people as a result of the creation of over 29 000 km of networks, increased ICT skills (90 000 young people enrolling in ICT studies), supported social inclusion (100 000 household in difficult situations receiving basic ICT training and 8 000 residents in Eastern Poland taking part in e-inclusion activities) and increased commercial opportunities for businesses. From the study, it was evident that it was important for these measures to be coordinated at central level and additional support to be provided for the development of e-skills and e-inclusion as well as of entrepreneurship. Another Polish evaluation of the Podkarpackie regional OP assessed 24 projects with a total investment of EUR 47 million, aimed at extending broadband and developing a number of ICT products and services. It found that the given support helped to create a number of e-services and to construct 59 km of broadband network, 206 km of new local-area networks (LANs) and the modernisation of 240 km of existing LANs. Around 12 300 households (accounting for 4% of the working-age population in the region) were connected to broadband, most of them in rural areas. E-government and e-health services increased the attractiveness of the region for investment and improved the quality and efficiency of healthcare facilities.

In the UK, the evaluation carried out on the “Next generation Broadband Programme” (Superfast Cymru), co-funded by both ERDF OPs in Wales, found that after the completion of the project around 520 000 premises/facilities were served by a superfast (24Mbps+) service. Take-up amounted to 23% (which was greater than similar interventions in the rest of the UK) and beneficiaries surveyed reported a high level of satisfaction in terms of time saved and increased access to educational opportunities. In addition, the projects were estimated to generate a net present value of EUR 1,3 million and to have a benefit-cost ratio of 6,7. It also led to the net creation of an estimated 350 jobs by 2015 and 1 050 by 2024 and to a reduction in CO2 equivalent emissions of 33 kilotons/year by 2024.

As regards SME competiveness (TO3), the evaluation of the projects Customer Engagement (CE) and New Business Start-up Support (NBSS) funded by the ERDF OPs in Wales found that the support helped to create or safeguard jobs, increase turnover, improve profitability and introduce new products or services. Allowing for some deadweight, displacement and multiplier effects which were identified, an estimated 75 net jobs were attributed to the support provided.

A Polish evaluation which used a macroeconomic model to assess the effects of business-support measures implemented by the Agency for Enterprise Development and funded by the Innovative Economy OP, estimated that the given support added PLN 92 billion (EUR 21,5 billion) to the GDP in 2007-2017, increased exports by PLN 38 million (EUR 9 million) and created 45 000 additional jobs, as well as raising productivity and earnings.

An evaluation of 45 projects supporting SMEs in the Southern Denmark (Syddanmark) region between 2010 and 2012 estimated, through counterfactual techniques, that these projects increased employment in the firms concerned by around 1 600 in full-time terms.

An evaluation of measures in Romania to support investment in business infrastructure of regional and local importance (TO3) and the reconversion of old industrial sites (IP6.5) concluded that these measures were consistent with the objectives, were effective overall and would not have taken place without EU funding. However, delays were experienced because of the nature of public procurement regulations, problems relating to property insurance and lack of companies specialised in soil decontamination.

In the case of evaluations on activities carried out by the Institute for Sustainable Design (ISD), co-funded by the West Wales and the Valleys OP and combining TO1-2-3, the measures were found to have helped develop a design-led cluster in the region, the creation of new products and services and new design attitudes and skills.

In Estonia, an evaluation of the schemes implemented by Enterprise Estonia (EAS), co-financed by the ERDF, found that these schemes helped to create business start-ups and to increase innovation and exports. In some areas, however, such as product development, deadweight was significant, and it was important for the future to combine financial support with mentoring, training initiatives and guidance to enterprises at the application stage in order to expand the effects of the schemes.

In Latvia, an evaluation of the impact of EU support on business development found that it was insufficient to have a perceptible effect on regional development problems. Projects aimed at boosting cooperation between businesses and research centres did not have a significant effect, except on small and recently established enterprises.

Low carbon economy, climate change, environment and transport end energy networks

An evaluation in Lithuania of the measures supported for housing renovation over the 2007-2013 and the 2014-2020 periods (though mainly in the earlier period) concluded that these measures helped to increase significantly the number of renovation projects but did not ensure that the largest possible energy savings were achieved at the lowest cost. In fact, construction costs increased to the point that the actual savings of heat energy during all the loan repayment period were lower than the total costs of renovation, thus reducing the attractiveness of the investment and lengthening the payback period. To improve this kind of support, the study suggested that calls for proposals should be better coordinated and more attention should be given to publicity and consultation.

In Poland, another evaluation in the same sectoral area, which assessed the contribution of EU funding in the 2007-2013 period to increasing energy efficiency and the use of renewables found that the supported measures helped increase the share of renewables in electricity consumption, but that the impact on the use of renewables in transport (e.g. biofuels) was marginal. However, there were no significant effects on cogeneration or the modernisation of heating and energy networks. Interventions reduced the impact of energy production on the environment but did not have a significant effect on the development of low-carbon technologies.

In Italy, an evaluation of the impact of a new tramway system in Florence concluded that there were gains to the local community who increased their use of public transport. This lead to an improvement in environmental quality (and an increase in the value of housing).

As regards environmental protection and preservation (TO6), water supply projects in Poland led to the construction and/or modernisation of 7 100 km of the supply network and to 354 100 people being connected to this, while waste management projects led to the construction, expansion and/or modernisation of 111 municipal treatment plants and 186 waste disposal facilities. Projects rehabilitated 38 km of the sea coastline and 339 municipal waste landfill sites and overall there was a significant reduction in emissions of air pollutants.

As regards conservation and promotion of the natural and cultural heritage (IP6.3), an evaluation in the Podlaskie region in Poland found that projects in this sector helped to increase the attractiveness of the region for tourists only slightly, largely because of the wide dispersion of projects, poor transport connections and a lack of a tourist management system at local level, as well as a lack of cooperation between stakeholders.

In respect of transport (TO7), an evaluation of projects implemented in Estonia over the 2007-2013 and 2014-2020 periods (though it largely covered projects undertaken in the earlier period, as relatively few projects in the later period had been completed) indicates that construction prices fell following the crisis and many projects were completed at lower cost than originally planned. The quality of rail transport improved significantly but overall the development of the transport infrastructure was not found to be a driver of local business activity.

In the UK, an evaluation of two rail projects implemented in Wales and funded by the ERDF West Wales and the Valleys OP (the intermodal facility at Energlyn and a passing loop at Tir-Phil station) found a reduction in car use, while the number of passenger rail journeys increased in line with, or above, forecasts, though the increase in passenger kilometres travelled was below expectations.

In Polska Wschodnia (Eastern Poland), an evaluation of EU-funded transport projects found that most projects achieved or exceeded their targets in terms of reduction in travel time for both passengers and freight. Effects on the economic development of the region, however, were not yet evident at the time the evaluation was carried out.

Another Polish evaluation of transport projects co-funded by the Infrastructure and Environment OP concluded that the road network across the country had been substantially developed, but the infrastructure at sea ports was still inferior to that in other Member States.

In Latvia, an evaluation of EU funding of infrastructure concluded that it had contributed significantly to the overall improvement in the of transport network. In particular, the railways had been modernised, resulting in a high-quality, competitive system for both freight and passenger transport and the conditions and safety of roads had been improved as had ports in the country as well as Riga airport.

Social inclusion and education

This sector includes thematic objectives that are only marginally funded by the ERDF.

As regards evaluations on social inclusion (TO9), an evaluation in the Lubuskie region in Poland assessed the effects of urban regeneration projects and their complementarity with similar projects funded in 2014-2020, finding that supported projects involved culture, tourism and roads and to a lesser extent to social infrastructure and increased the attractiveness of the areas concerned as places to live and work. Few projects would have been implemented without support or would have been on a smaller scale. Projects planned for the 2014-2020 period are in line with the main socio-economic and environmental needs of the region.

As regards education and training (TO10), a counterfactual analysis in Romania of support for investment in education infrastructure found that the consequent improvement in the quality of school buildings and equipment increased the standard of education (more in poorer localities than richer ones) and increased access for those with disabilities.

All/Multiple Thematic Objectives

As some evaluations focused on more than one TO or all TOs, their results are presented separately.

In Poland, an evaluation of the impact of measures supported by the regional Wielkopolska OP on employment, which partly involved the use of financial instruments, found that such measures helped to increase the number of permanent jobs created. Overall, the recipients of support were more resilient to the economic crisis and other market shocks. In the Pomorskie region, the support provided by the regional OP was estimated to have increased employment, GDP growth, SME development and innovation. In the Warmia i Mazury region, measures were found to have increased business competitiveness, regional socio-economic development and employment, although the region remained less developed than others in Poland. In Kujawsko-Pomorskie, Cohesion Policy support is estimated to have increased employment, income and living standards, as well as to reduce disparities in development within the region. In Podlaskie, it is estimated that Cohesion Policy support increased GDP growth and the regional employment rate between 2004 and 2015 as well as raising household income significantly. An evaluation of equal opportunity and non-discrimination measures supported by the Podlaskie regional OP found that these measures led to more equal access to the labour market and social services. They also helped to reduce intra-regional inequalities in access to tourist, sports and cultural facilities as well as to good quality education and healthcare services in both urban and rural areas.

A horizontal evaluation on the impact of EU funds on poverty and social exclusion found that overall there was a general reduction of people using community social aid, the scale of relative and absolute poverty, the proportion of young people not continuing education beyond compulsory schooling, the proportion of NEETs among young people and the scale of severe material deprivation.

In Italy, an evaluation of measures supported by the Lombardia regional OP, using a counterfactual approach, found that these measures increased RDI expenditure and enterprise competitiveness, as well as the business survival rate and employment. They also improved energy efficiency and increased the share of renewables in energy consumption.

An evaluation of urban development measures co-funded by the Regional OP in Romania concluded that these measures led to an improvement in the quality of life and an increase in job creation, a reduction of unemployment and an increase in the attractiveness of cities as well as high school enrolment rates.

In Hungary, an evaluation of integrated urban development policies in the regional OPs found that cities started to use more integrated approaches to development but there remains a lack of strategy and limited integration with other policies and involvement of stakeholders.

Evaluations using macroeconomic models to assess the effects of ESI Fund support, estimated that in Hungary, this support helped to improve education, the healthcare system and human capital, to increase the efficiency of public administration, to develop transport and environment infrastructure and to stimulate RDI. Also in Hungary, investment in construction and community services were estimated to have contributed most to GDP growth and employment, particularly of less-skilled workers, though it was also estimated that the additional demand generated led to increased costs of production, thus compromising the competitiveness of domestic producers and leading to a decline in exports and an increase in imports.

In Lithuania, the projects supported over the period 2007-2020 are estimated to have increased GDP by twice the amount of the expenditure involved.

In Slovakia, the funding provided is estimated to have increased economic development and convergence and to have helped alleviate the effects of the crisis. In the Visegrad countries, it is estimated that there were significant spill-over effects on entrepreneurship and innovation, transport infrastructure, research, energy and environmental protection.

The main findings of the evaluations undertaken by Member States (on ERDF/CF and ESF/YEI), including a description of the data used and the adopted methodologies are available in the Evaluation Library published on the Inforegio website. 5

2.2. Synthesis of ERDF/CF evaluation work by the Commission

The ERDF and Cohesion Fund 2007-2013 ex post evaluation was finalised in late 2016 and was extensively presented 6 at the time, and included a major communication campaign on the results of the Cohesion Policy on the ground.

In 2017 the Commission launched an evaluation to complement previous ex-post evaluations. The objective of the latest evaluation of major projects, supported by ERDF and the Cohesion Fund between 2000 and 2013, is to analyse the long-term contribution (direct and indirect, as expected and unexpected) of major investment projects on economic and environmental development, quality of life and well-being of citizens. The first study under the evaluation, finalised in June 2018, focused on 10 ERDF or Cohesion Fund transport projects. A Staff Working Document based on the findings of the study will be available in the first semester of 2019.

The findings show that major transport projects can positively contribute to increasing the transport efficiency within EU regions, supporting the shift to cleaner urban mobility and facilitating cross-border transport connections. Two key success factors were identified: a sound forecasting capacity driving the project design and selection process and excellent managerial capacity to keep the project on track. There is also evidence that major investments in the transport sector would benefit from strategic guidance by the Commission on the prioritisation and selection process towards projects that maximise EU objectives and are implemented according to well defined quality standards (e.g. environmental measures).

This evaluation is now continuing to cover major projects in the policy area of environment.

An ex post evaluation on investment projects in Research and Technological Development (RTD) infrastructures, competence centres and activities financed by the ERDF during the period 2007-2013 is currently under design. It follows and complements the work packages of the ex post evaluation of the period 2007-2013 and is aimed to obtain a deep understanding of the rationale for and effectiveness of ERDF support for research infrastructure and activities and to identify factors contributing to the success or failure of these investments under different socio-economic conditions.

During the 2014-2020 programming period, the main evaluation work undertaken by the Commission has been related to the accumulation of evidence from Member States' evaluations. To this end, the Commission has launched an "Evaluation Helpdesk" contract (covering ERDF/CF and ESF/YEI) to identify and synthesise, among other tasks, the evaluations carried out by Member States. Such evaluations are collected in the above-mentioned Evaluation Library that includes also a summary of the main findings for 75% of them (100% of those identified in the last year).

To provide support to the Member States, the Commission, through the Evaluation Helpdesk, organises evaluation training for the Member States through "summer schools" for managing authorities and others involved in the evaluation of programmes. It also organises peer reviews of selected evaluations (final reports or whole evaluation processes) in order to discuss the existing strengths and weaknesses in the practice of evaluation of cohesion policy and to help Managing Authorities in their work of designing and conducting evaluations. A targeted methodological support on issues concerning the planning and implementation of evaluations was also provided to selected Managing Authorities in Latvia (2016), Greece (2017), and Croatia (2018).

3.ESF AND YEI

By August 2017, evaluation plans have been submitted for all OPs. Nevertheless, evaluation planning has continued with a view to further specifying the already planned evaluations or catering for additional evidence needs, e.g. due to OP modifications. Since the last quarter of 2017 until the last quarter of 2018, 24 ESF evaluation plans have been received, including 21 revised substantially 7 and including 22 multi-fund plans.

With the completion of the initial phase of evaluation planning, the focus is moving on the planning of impact evaluations, i.e. the evaluation work that contributes most in the assessment of the fund's achievements. The impact evaluations of the latest evaluation plans have been assessed according to five standard quality criteria 8 . The pattern that emerges from these assessments is similar to the main findings about evaluation planning in general. Namely, the planning of the impact evaluations is relatively strong in organisational aspects, such as budgeting and timing. They also score relatively well in terms of defining the objectives of the evaluation and the evaluation questions. However, the specification of the evaluation design and methods as well as of data sources is less articulated. The main shortcomings in respect of the former are the lack of detail on the approach to be used, such as theory-based, counterfactual or cost-benefit analysis, and on the rationale for the adoption of the evaluation approach. In a number of cases, there are also question marks over the appropriateness of the approach indicated for answering the evaluation questions.

For the data and sources, the main shortcomings are the lack of detail about data sources to be used, the specific data to be taken from these, the quality of the data available and whether or not there is a need to take action in respect of missing data. Where a counterfactual approach is suggested, in the majority of cases, insufficient consideration is given to the data required to measure the performance of the control group. As Table 2 in Section 2.1 shows above, the weakness in the identification of data sources is particularly challenging in case of ESF planned evaluations.

Following the plans, the quality of the published evaluations is assessed through criteria 9 similar to those used in analysing the information about planned evaluations. That assessment confirms to what extent the shortcomings in evaluation planning translate to weaknesses in the evaluations themselves. The results show that especially the validity of the findings and the appropriateness of evaluation design are weak, both of which are closely related to problems in the planning stage (see Table 3).

These problems compromise the reliability of the findings and thus the use that can be made from evaluations. So far, altogether 22 ESF evaluations for the 2007-2013 and 7 for the 2014-2020 period have been regarded reliable, i.e. rated above an average score of 3 according to the five quality criteria.

Concerning methodological choices and the quality of the evaluations, direct feedback is provided to the Managing Authorities in the evaluation peer reviews of selected evaluations with the support of the Evaluation Helpdesk. The reviews appraise the evaluation dossier from the terms of reference to the final report with a particular view on methodological choices. The main general findings of the reviews will be summarised in a guidance.

Moreover, the Commission continues promoting the use of counterfactual impact evaluations by Member States, notably through periodic evaluation partnership and Community of Practice meetings and the data fitness initiative carried out jointly with JRC since 2017. Concerning better access to data sources, DG Employment is carrying out a feasibility study: the aim is to identify ways of accessing existing data sources, administrative registers, thereby enabling the execution of counterfactual impact evaluations. With a view to the future, the Commission's draft proposal for the ESF+ Regulation enables the access by Managing Authorities to data registers for monitoring and evaluation purposes.

3.1. Synthesis of ESF/YEI evaluations completed by the Member States

As shown last year in the Staff Working Document accompanying the 2017 Strategic Report on the implementation of the ESIF (SWD(2017) 452), the number of evaluations planned to be completed is showing a sharp increase compared to 2016-2017. The same applies to the specific group of impact evaluations. The data about the period between August 2017 and August 2018 confirm that expectation.

Table 4: Evaluations published since January 2015 – Breakdown by programming period, type, year of publication and Fund

Unsurprisingly, most evaluations relating to the 2014-2020 period are process/implementation-oriented, aimed at assessing the way the programmes are operated and implemented. Of the 164 ESF (and multi-fund) evaluations identified for the 2014-2020 period, 52% are ‘pure’ process/implementation-oriented evaluations, 23% are also concerned with assessing progress in achieving targets and 10% with assessing first results and outcomes.

Most evaluations identified apply a combination of relatively simple qualitative and quantitative methods. Of the 298 ESF (and multi-fund) evaluations identified 227 combine at least two methods. Among impact-oriented evaluations, 32 use counterfactual evaluation and 48 use theory-based evaluation techniques (alone or combined with other techniques).

It is evident that the share of evaluations using counterfactual techniques is larger for ESF OPs and OPs using mixed funding than for ERDF OPs.

Contrary to 2017 when the findings of all completed evaluations were synthesised, this year's synthesis uses findings only from evaluations deemed reliable 10 . This selection has resulted in 22 ESF evaluations for the 2007-2013 and 7 for the 2014-2020 programming period.

Table 5 - Impact evaluation identified, reviewed and considered reliable

3.0.0.Findings from evaluations on the 2014-2020 period

Six of those seven evaluations concerned mainly employment related findings and one looked at results in institutional capacity building.

An evaluation of the effects of Le Chéile’s mentoring services for young people in Ireland found that the model put in place was robust and worked efficiently with positive results, a social return of 4 times the cost of the project being estimated from cost-benefit analysis.

In Germany, an evaluation of the progress achieved and effects of 7 ESF-supported projects in the city of Hamburg to increase the employability of disadvantaged people and to help women into employment found that a close relationship between coaches and participants was created and the capabilities of participants improved. The main reasons for success lay in the adaptable and integrated approach adopted, the well-developed network of coaches and the relatively long duration of projects.

In Italy, an evaluation of the “Employment and Labour Policies Integrated Plan” (PIPOL), co-funded by the Friuli Venezia Giulia regional OP concluded that participants had a higher probability of finding a job than non-participants and that overall, traineeships were more effective than training programmes. The measures were also more effective for women, young people, the low-educated and people from abroad.

Another Italian evaluation of measures co-funded by the Piemonte regional OP found that 31% of participants were employed one year after the completion of training, an estimated difference of around 12-20 percentage points percentage as compared with non-participants.

In Estonia, the "My First Job" initiative to support the (re)entry of young people into the labour market through wage and training subsidies and training was found to have had a significant effect in reducing youth unemployment, though there were large numbers of dropouts and relatively few who continued working for their original employer over the long-term. It was recommended awareness-raising campaigns among young people and employers should be carried out and the selection criteria for participation broadened to include unemployed parents, in particular.

An evaluation of the Curriculum and Career Guidance Programme, also in Estonia, aimed at implementing the Estonian Lifelong Learning Strategy 2020, indicated that, while the operations supported were in line with the objectives, more attention needed to be paid to children with special needs. It also indicated that the sustainability of the programme was not guaranteed and effective financial plans needed, therefore, to be formulated.

An evaluation of four specific activities under the ESF Knowledge, Education, Growth OP in Poland, aimed at strengthening institutional capacity and efficient public administration (TO11), found that they did not appear to have a significant effect on international good governance indicators.

3.0.1.Findings from evaluations on the 2007-2013 period

Employment

In this theme, a number of evaluations reviewed carried out counterfactual analysis. In the UK, ESF supported measures by both ESF OPs in Wales were considered to have increased access to employment and raised skill levels of those in work (both in the convergence and competitiveness regions). The training provided had a positive impact, increasing the likelihood of participants (both unemployed and inactive) finding a job afterwards (by 26% for the unemployed and 50% for the inactive as compared with non-participants). Positive effects were also estimated for those participating in training to increase their skills (7% of whom indicated that improvements in their employment situation were directly attributable to the training). Another Welsh evaluation was based on five surveys carried out on those completing both ESF OPs, in particular those participating in training aimed at increasing skill levels and employability. Results show that the people concerned acquired a wide range of skills. Half of the participants had a positive transition from unemployment to paid work.

The evaluation of active labour market measures supported in Croatia between 2010 and 2013 found that participants in vocational training increased their medium-term employability by 40% as compared with unemployed non-participants. Measures were particularly positive for those covered by the Employment Promotion Act. Positive effects were also visible for the long-term unemployed as a result of public works measures and labour cost subsidies for small employers. Nevertheless, a sixth of the vocational training interventions did not provide participants with adequate work experience.

In Ireland, the evaluation of the Job Bridge Activation Programme co-funded by the ESF estimated that the support had significantly increased the likelihood of participants finding a job and more than other labour market activation programmes.

In Poland, the evaluation of the measures aimed at older workers and elderly in the Lublin region found that interventions had a positive effect on their employment though there was a relatively large deadweight. Another Polish evaluation of activities designed to increase the employability of young people in the Wielkopolskie region found that it improved the job search capability of participants and increased their motivation to look for a job. From the social point of view, participation in projects helped beneficiaries to gain independence and be more receptive to others.

The evaluation of the Employment, Human Capital and Social Cohesion OP 2007-2013 in Cyprus found that in general the OP was effective in achieving its objectives. Advisory guidance to enterprises and training of women were more efficient - in terms of unit costs - in 2007-2013 than the previous period, but projects for integrating participants into employment and getting socially vulnerable groups into work were less efficient.

Social inclusion

In this field, ESF-supported measures in the UK were found to have been successful in reducing social isolation and increasing work-related activity of disadvantaged families. Positive factors were the ongoing support to participants and the focus on a broad range of limitations which disadvantaged families experienced.

In Lithuania, the evaluation of Cohesion policy found that the Structural Funds helped to improve the quality of healthcare services across the country, as evidenced by a range of indicators, such as the mortality rate, the suicide rate, infant mortality, early diagnosis and oncological treatment. The healthcare system, however, still needed to be improved, (as regards, for example, training of staff, e-health systems and refurbishment of buildings) in order to ensure future sustainability.

Evaluation of the economic and social urban regeneration measured undertaken by the regional Lubuskie OP in Poland found that few projects would have been implemented without support or would have been on a smaller scale. Measures were mainly implemented in major urban centres in the South West of the province of Lublin and involved support for culture, tourism and road improvement and, to a lesser extent, for social infrastructure. Overall, support helped to increase the attractiveness of the areas concerned as places to live and work.

Education and training

In the area of education and training, the need for ICT improvements was identified in many regions. The evaluation of training activities in universities, co-funded by the ESF Human Capital OP in Poland, found an overall high level of satisfaction among students, though also more difficulty for small firms than for larger ones in establishing cooperation with universities.

Evaluation of the effectiveness, coherence and relevance of interventions co-funded by the federal ESF OP in Germany highlighted the overall coherence of the measures supported with other regional labour market policies and with specific local needs. Regions with high unemployment, therefore, focused mainly on employability and transition from school to work, while those in which it was low, focused mainly on lifelong learning and disadvantaged groups. The decentralised management and control model used was successful in enabling active participation of employment agencies, job centres and municipal offices providing social services and in ensuring effective coordination with federal authorities.

Another German evaluation carried out a counterfactual analysis of the impact of a specific action funded by the same federal ESF OP, Civil Work (Bürgerarbeit). It found that the activation phase increased access to employment, participants having an integration rate 35% higher than non-participants 6 months after the beginning of the programme, though after two years, there was no difference. In the employment phase, however, participation in the programme was estimated to have a negative effect, in that the integration rate was only half that of non-participants, which is partly explained by a locked-in effect, those participating in an employment programme not being able to look for a job.

A Latvian evaluation of measures implemented under the Human Resource Development OP to promote equal opportunities and combat discrimination found that the inclusion of gender equality as a requirement in calls for tender is not sufficient in itself and that there was a need for a proper assessment of compliance during project selection.

3.0.Synthesis of ESF evaluation work by the Commission 

3.0.0.Update of the 2007-2013 ex-post data 

The implementation of the programmes was not completed at the time of the Commission ex-post evaluation, and relied on the 2014 Annual Implementation Report data and national evaluations from the Member States conducted until 30 June 2015. As a result, the ESF 2007-2013 ex-post evaluation did not present the full picture of the ESF during that programming period.

Therefore the Commission carried out a study supporting the update of the data reported in the 2007-2013 ESF ex-post evaluation based on the Final Implementation Reports which were submitted to the Commission in the course of 2017 (without reviewing evaluation findings). The study will very soon be complemented by fact sheets for the entire EU as well as each of the 28 Member States to be used for dissemination to the public.

3.0.1.Launch of two thematic evaluations 

The Commission has recently launched two thematic evaluations; the first one focuses on the ESF and YEI support to youth employment and the second one covers Thematic Objective 8 with the exception of support to youth employment and mobility. Both of them will be completed in 2019. They will be followed by two evaluations on Thematic Objectives 9 and 10.

These studies will provide the Commission with evidence and analysis that will allow it building up its evaluation of the ESF support to employment and mobility. The results of the studies will be integrated into the staff-working document produced by the Commission annually.

On the one hand, these thematic evaluations aim at taking stock of the results of ESF and YEI related support until now, building in particular on the relevant evaluations carried out by Member States. On the other hand, the results of the evaluations should feed into the next programming period, by providing lessons, notably as regards cost-effectiveness, outreach and target groups most in need, thus supporting the negotiation of the operational programmes for the ESF+. It will also pave the way for the ex-post evaluation of the ESF by the Commission to be completed by December 2024.

4.EAFRD - EUROPEAN AGRICULTURAL FUND FOR RURAL DEVELOPMENT

4.1. Synthesis of evaluation work by the Member States

4.0.0.Period 2014-2020 

According to the information submitted by the Member States in June 2018 in 115 Annual Implementation Reports (AIRs), the number of completed evaluations (192) has increased by 28% compared to 2017 and the overall majority (89%) was published online. More than half of the completed evaluations (66%) were linked to the major EU evaluation milestones and requirements. Furthermore, self-standing evaluations on other specific Rural Development Programme (RDP) aspects, such as delivery mechanisms, financial instruments, and technical assistance were reported, as well as evaluations on other rural issues such as rural tourism and biodiversity.

The highest share of completed evaluations covered several of the six RDP priorities (40%), followed by evaluations covering only RDP Priority 4 ‘Restoring, preserving and enhancing ecosystems’ (14%). Relatively few evaluations have been completed in relation to Priority 1 ‘Knowledge transfer and innovation‘, Priority 2 ‘Farm viability and competitiveness’, Priority 3 ‘Food chain organisation and risk management’, Priority 5 ‘Resource-efficient, climate-resilient economy’ and Priority 6 ‘Social inclusion and economic development’. Completed evaluations were reported also on RDP cross-cutting objectives, such as innovation (e.g. BE Flanders), environment (e.g. ES Castilla-Mancha, FI Mainland), and climate change (e.g. UK England).

The extent of reporting on evaluation findings has overall increased: 79 AIRs contained summaries of completed evaluations (compared to 65 in 2017). Within these summaries, 471 distinct evaluation findings could be identified covering different topics, as shown in Figure 2.

Figure 2: Reported evaluation findings across topics (N=471)

Source: European Evaluation Helpdesk for Rural Development (2018)

Following the same trend observed in the completed evaluations above, 67% of the evaluation findings was linked to one or more RDP priorities and focus areas. Among these evaluation findings, the highest share concerned the RDP Priority 4 ‘Restoring, preserving and enhancing ecosystems’ (21%), followed by Priority 2 ‘Farm viability and competitiveness’ (11%), Priority 6 ‘Social inclusion and economic development’ (10%), Priority 1 ‘Knowledge transfer and innovation‘ (7%), Priority 5 ‘Resource-efficient, climate-resilient economy’ (6%), and Priority 3 ‘Food chain organisation and risk management’ (4%). Moreover, 14% of the evaluation findings was connected to other specific thematic areas, such as European Innovation Partnerships on Agricultural Productivity and Sustainability (EIP-AGRI), National Rural Networks, innovation, or RDP synergies with other ESI Funds. To a smaller extent, evaluation findings also covered the RDP management and delivery mechanism (8%), RDP monitoring and evaluation system (6%), and other topics (5%), such as ex post evaluation of RDPs 2007-2013.

The analysis of the evaluation findings related to the six RDP priorities grouped according to the three CAP Objectives can be summarised as follows:

·The evaluation findings linked to CAP Objective 1 of fostering the competitiveness (15% of the total findings), covered both the implementation and the achievements of RDP Priorities 2 and 3. Member States were able to capture the trends in farm productivity and modernisation in supported farms (e.g. ES Navarra, FR Bourgogne), the changes in the farm net value added (e.g. AT, BG, ES Galicia), the contributions to the generational renewal (e.g. ES Galicia, ES La Rioja, IT Bolzano), as well as observations on the level of programme implementation registered in 2017 (e.g. CZ, ES Extremadura, FR Auvergne).

·The findings concerning the CAP Objective 2 of sustainable management of natural resources and climate action (27%) addressed the implementation and achievements of RDP Priorities 4 and 5. In relation to RDP Priority 4, several Member States reported on the trends of the Farmland Bird Indicator (e.g. CY, EE, SI), the changes in the content and dynamics of soil’s nitrates and sulphates (e.g. EE, DE Saxony, IT Piedmont), and the implementation of agri-environmental schemes (UK England, FR Mayotte, ES La Rioja). For RDP Priority 5, the evaluations showed the RDP support to carbon conservation and capture (e.g. ES Galicia, ES Navarra, SK), the reduction of carbon emissions (e.g. IT Trento, PL), renewable energy (e.g. LV, ES La Rioja), and investments in energy efficiency and water management (e.g. ES Murcia, FR Bourgogne, IT Bolzano).

·The findings linked to CAP Objective 3 of ensuring balanced territorial development (10%) mainly described the implementation of Priority 6, with a focus on LEADER/Community-Led Local Development (CLLD) (e.g. PT Madeira, BG, EL, ES Valencia) and the support given to Local Action Groups for implementing the LEADER/CLLD strategies, monitoring and evaluation (e.g. PT Azores, IT Puglia, HR, CY).

·The findings related to the horizontal RDP Priority 1: knowledge transfer and innovation described the level of uptake (e.g. IT Puglia, ES Galicia), the selection criteria and aspects of the RDP delivery mechanism (IT Bolzano, ES Navarra, IT Trento), as well as the positive contributions towards all RDP priorities (e.g. PT Mainland).

Evaluation findings covering multiple RDP priorities and focus areas (8%) concerned more general aspects of the RDP 2014-2020, such as the internal and external coherence of the RDP intervention logic (ES Madrid), good practices observed across the supported projects (ES Aragon), and the challenges in achieving the targets set up in the RDP (SI).

The communication activities in relation to publicising evaluation findings have increased by 23% compared to the previous reporting period. 344 communication activities were reported in 2018, reaching out to 3,97 million stakeholders. These were reached mainly through websites (3,04 million stakeholders reached), publications and evaluation reports (0,23 million), but also through other channels (e.g. written consultations, newsletters etc.)

Follow-ups given to evaluation results have increased by more than three times compared to those reported in the AIRs submitted in 2017 (from 169 to 544). The follow-up activities concerned different areas as shown in Figure 3.

Figure 3: Reported follow-up activities given to evaluation results (N=544)

Source: European Evaluation Helpdesk for Rural Development (2018)

It is important to keep in mind that the enhanced AIRs submitted in 2019 will contain a significant update on the evaluations conducted and their findings as the programmes will have reported for the first time on the impacts of the 2014-2020 funds and assessed progress towards the objectives of the programme and its contribution to achieving the Union strategy for smart, sustainable and inclusive growth based on the relevant result and impact indicators 11 .

4.0.1.Ex-post evaluations 2007-2013 

For the EAFRD, Member States had to carry out by 31 December 2016 an ex-post evaluation of their RDPs 12 . Member States examined the degree of utilisation of resources, the effectiveness and efficiency of the programming of the EAFRD, its socio-economic impact and its impact on the EU priorities. They had to cover the goals of the programme and aim to draw lessons concerning rural development policy. They had to identify the factors which contributed to the success or failure of the programmes’ implementation, including as regards sustainability, and identify best practice. Based on these evaluations, a summary of these ex post evaluations was carried out by Commission services by the end of 2017. Furthermore, a synthesis of the ex-post evaluations was conducted, analysing the effectiveness, efficiency, coherence, relevance and EU added value of the 2007-2013 RDPs.

This work provided important insights into the contribution to policy objectives made by the 2007-2013 programmes. Regarding effectiveness, the evidence base allows to state that the implementation of EAFRD funding via RDPs has been effective in achieving EU-level objectives and supporting key EU priorities to a variable extent (albeit with the caveat of a better evidence base for Axis I and II than for Axis III objectives). Based on the overall judgments on the three criteria (effectiveness in achieving objectives, coherence with EU priorities and complementarity with other instruments, and subsidiarity), the evaluation found that EAFRD funding via the RDPs ensured EU added value to a moderate and variable extent, depending on the aspect considered. However, taking into consideration the limitations of the data, it was not possible to provide an overall judgment of proportionality of EAFRD expenditure with regards to benefits achieved at the EU level (efficiency). As to the extent to which RDPs have contributed to addressing the social, economic and environmental needs within the programmes' area, it can be concluded that RDPs have contributed to addressing the needs – across the social / socio-economic, economic and environmental dimensions – to an overall moderate extent.

4.2. Synthesis of EAFRD evaluation work by the Commission

As regards the evaluation work of the Commission, RDPs' effects are being also assessed via specific thematic evaluations, in which the Commission has to ensure that the combined impact of all CAP instruments, i.e. the first pillar (direct payments and market measures) and the second pillar (rural development), is measured and assessed 13 . To date, the following actives were conducted:

·The report of the evaluator of the "Evaluation study of the forestry measures under rural development" 14 was published in March 2018. It covered 16 evaluation study questions relating to: the drivers influencing the implementation choices taken by Member States/regions and beneficiaries, as well as the effects of the measures concerning production, geographical distribution; effectiveness (in relation to economic, environmental and climate objectives); efficiency; coherence; relevance; and EU added value. The EU Forest Strategy is implemented through a variety of policies (both at EU and national level) but the main EU-level funding to implement the Forest Strategy has been and remains the co-financing of forest measures under the Rural Development Regulation. Furthermore, as stated in the EU Forest Strategy, the Commission considers that rural development funding in particular should be used to support the implementation of sustainable forest management. Given that the evaluation covers a limited period of time (2014-2017), in some cases the 2007-2013 period was also examined to assess the level of implementation of the forest sub-measures and to capture their mid-term effects. This approach was facilitated by the high degree of similarity of the Rural Development forest-related measures in these two programming periods.

From the analysis of effectiveness, it can be concluded that the forest measures (representing, at EU-28 level, 4,8 % of the total planned public expenditure of RDPs) and horizontal RDP measures implemented in forests, have a generally very positive effect, even if often difficult to separate from other factors such as state aids and the operations funded by foresters on their own. The evaluation identified also some possible improvements for a more effective implementation of the forest measures.

The evaluation process will be finalised by the publication of a Commission Staff Working Document.

·"Evaluation of the impact of the CAP on climate change and greenhouse gas emissions" and the "Evaluation of the impact of the CAP measures towards the general objective of viable food production" covering policy instruments under both first (EAGF) and second pillar (EARDF).

·"Evaluation of the impact of the CAP measures on generational renewal, local development and jobs in rural areas" was launched with the aim to assess how various forms of support for young farmers as well as other support measures available under the CAP Pillar I and II affect the generational renewal in rural areas.

Moreover, "Evaluation on the impact of the CAP on water" was launched with the view to be published in 2020.

CMEF Report

By December 2018, an initial report on the implementation of the Common Monitoring and Evaluation Framework (CMEF) and first results on the performance of the CAP will be published. The framework covers both CAP pillars as well as horizontal measures (e.g. cross compliance) of the CAP. The report will describe the CAP overall achievements on the basis of the data and evaluation information available so far within the CMEF framework. The available evaluation information will also be taken into account.

5.EMFF - EUROPEAN MARITIME AND FISHERIES FUND (EMFF)

5.1. Period 2014-2020

The national European Maritime and Fisheries Fund (EMFF) programmes experienced a late start due to late adoption. In most Member States, the first operations started in 2017, with Managing Authorities intending to begin evaluations in 2018. Two Member States, the Czech Republic and Sweden performed process evaluations during 2017.

To support Member States with process and impact evaluations of the EMFF, the Commission provided a working paper on EMFF evaluation 15 and the EMFF Evaluation toolbox 16 .

5.2. Period 2007-2013

National interim evaluation reports on the European Fisheries Fund (EFF) (2007-2013) were conducted during the period 2007-2013. These were summarised in a Staff Working Document 17 on the ex-post evaluation of the EFF .

The EFF was launched in 2007 at the onset of the global economic crisis. The ensuing decline in demand for fisheries products and drop in prices, as well as reduced access to private financing put the sector under high economic pressure. Despite this difficult situation, the following results were achieved:

·Support provided by the EFF for the local development of fishing depending communities was a considerable success despite a slow start. It allowed for the creation and maintenance of a large number of jobs and tangibly improved the quality of life in these areas. This included the creation of some 10 000 jobs in the processing and marketing sectors.

·Competitiveness of the fleet was enhanced, through modernisation and, most importantly, support for changing fishing gears and methods. Ports and landing sites were improved, and investments in marketing and processing of fish increased its added value.

·The EFF contributed to reducing overcapacity in many fleets, although imbalances still exist between fleet capacity and fishing opportunities.

·The EFF contributed to the economic resilience of the aquaculture beneficiaries, especially in the shellfish sector. Due to lack of strategic orientation however, only a marginal impact was identified on finfish aquaculture, whose production increased but at a slower pace than production at the world scale, which has been an extremely rapidly growing sector since the beginning of the EFF period.

·For small scale coastal fisheries, EFF funds were made available for the management of fishing areas, the reduction of fishing effort, the organisation of production and innovation. However, uptake of this measure was low as it was unfocused and competing with other measures encompassing all fleet segments.

·The contribution of the EFF to broader conservation objectives, as well as weak complementarities and synergies with other funds (such as LEADER) were identified as areas to be strengthened.

Several shortcomings identified for the EFF were addressed by the EMFF which succeeded it for the 2014-2020 period.

6.CONCLUSION

An overview of the implementation progress shows that the reported expenditure of projects selected at the end of 2017 in the programmes supported by the European Structural and Investment Funds amounts to 15% of the total planned. Although it has more than doubled in 12 months, it still represents a low execution rate, especially if compared to the previous programming period.

The low pace of implementation has determined a limited availability of evaluations for the period 2014-2020 so far, behind the level initially planned. The evaluations mostly deal with process/implementation and monitoring/progress issues, and those oriented to evaluate the impact of the programmes and the policy are only a minor share of the total.

However, the number of 2014-2020 evaluations completed by the Member States is now increasing, faster than those referred to in 2007-2013 (the number of evaluations published in the last 12 months and referring to the period 2014-2020 nearly doubles the number of those referring to 2007-2013). There are positive signals also on the implementation, like the increase of the selection rate of the projects of 25 percentage points in 2017 alone: this is the largest annual rise so far, indicating that Member States are turning the planned investments into concrete projects.

Based on the forecasts of the evaluation plans and the fact that implementation is accelerating to reach its full pace, the Commission expects that a more conspicuous number of impact evaluations related to 2014-2020 programmes will be available in the next 12 months, so that more evidence from the Member States will be reported in the 2019 edition of this document.

(1)       https://cohesiondata.ec.europa.eu/  

(2)      SWD(2017) 452 final - https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=SWD:2017:452:FIN  

(3)      Each criterion is assigned a score ranging from ‘4’ strong/good or ‘3’ fairly strong/good to ‘2’ fairly weak/poor or ‘1’ weak/poor (i.e. many weaknesses and limitations are identified therefore findings are not considered reliable).

(4)      Two evaluations are related to both programming periods 2007-2013 and 2014-2020 (housing renovation in Lithuania and transport in Estonia) and their findings are reported in Section 2.1.2 because they mainly refer to the earlier period.

(5)      The Evaluation Library is available here: http://ec.europa.eu/regional_policy/en/policy/evaluations/member-states/  

(6)      The summary and full details of the ERDF/CF 2007-2013 ex post evaluation package is available here: http://ec.europa.eu/regional_policy/en/policy/evaluations/ec/2007-2013/  

(7)    Less significant, on-going adjustments in the evaluation plans are not considered as reviewed plans.

(8)      The criteria have been a) the clarity and pertinence of general background information, b) the objective of the evaluation and the evaluation questions, c) the evaluation design and methods, d) the data and data sources, e) the coherence of the budget indicated and the time allowed for the evaluation with the objective and scope.

(9)      a) The clarity and suitability of the evaluation design, b) the appropriateness and correctness of the techniques, c) data quality and availability, d) the validity of findings and e) the validity of conclusions.

(10)      The criteria have been the same as those listed in Footnote 11. The evaluation has been considered reliable in case the overall score of the five criteria has been above 3 in a range between 1 ('weak/poor') to 4 ('strong/good').

(11)       Commission Implementation Regulation (EU) No 808/2014, Annex I, Point 9, and Annex VII, Point 7 .

(12)      Article 86 of Regulation (EC) No 1698/2005.

(13)      Article 110 of Regulation (EU) No 1306/2013.

(14)     https://publications.europa.eu/s/ilDj

(15)       https://ec.europa.eu/fisheries/sites/fisheries/files/fame-working-paper-emff-evaluation_en.pdf  

(16)       https://ec.europa.eu/fisheries/sites/fisheries/files/fame-working-paper-emff-evaluation-toolbox_en.pdf  

(17)      SWD(2017) 274 final