EUROPEAN COMMISSION

Brussels, 15.1.2024

SWD(2024) 2 final

COMMISSION STAFF WORKING DOCUMENT

SYNTHESIS OF THE FINDINGS OF THE EVALUATIONS OF


EUROPEAN STRUCTURAL AND INVESTMENT FUNDS


PROGRAMMES 2014-2020

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS

European Structural and Investment Funds
2023 summary report of the annual programme implementation reports covering implementation in 2014-2020

{COM(2024) 6 final}

STAFF WORKING DOCUMENT

ON THE

SYNTHESIS OF THE FINDINGS OF THE EVALUATIONS OF

EUROPEAN STRUCTURAL AND INVESTMENT FUNDS

PROGRAMMES 2014-2020

accompanying the 2023 Annual Summary Report

Contents

1INTRODUCTION

2COHESION POLICY PROGRAMMES

2.1Evaluations by the Member States

2.1.1General overview

2.1.2Main findings of evaluations

2.2Evaluation work by the Commission

3EUROPEAN AGRICULTURAL FUND FOR RURAL DEVELOPMENT (EAFRD)

3.1Evaluations by the Member States

3.1.1General overview

3.1.2Main findings of evaluations

3.2Evaluation work by the Commission

4EUROPEAN MARITIME AND FISHERIES FUND (EMFF)

4.1Evaluations by the Member States

4.2Evaluation work by the Commission

5FINAL REMARKS

1INTRODUCTION

The European Structural and Investment (ESI) funds are managed jointly by the Commission and the Member States under shared management. This means that while the Commission remains responsible for implementing the funds, Member States oversee the day-to-day management of the programmes. Member States are required to evaluate each programme to improve the quality of the design and implementation of programmes and to assess their effectiveness, efficiency and impact.

In line with Article 53 of the Common Provisions Regulation 1 , this staff working document presents a summary of the findings of the latest available evaluations of the programmes supported by the ESI Funds 2 . It accompanies the 2023 summary report of the annual implementation reports of the programmes, covering the implementation of the 2014-2020 ESI Funds at the end of 2022.

So far, around 3 600 evaluations have been done by Member States of all ESI funds for the 2014-2020 programming period. As implementation progresses and more evidence on the results becomes available, Member States have continued to focus on impact evaluations. The Commission has also done a number of evaluations, including an evaluation on e-Cohesion, and it is doing an ex post evaluation of the funds for the 2014-2020 programming period, which will be completed in 2025. The European Agricultural Fund for Rural Development is not covered, as the duration of its programmes has been extended.

The 2014-2020 period is a particularly challenging period to evaluate. The ESI Funds, with their long-term planning and 10-year implementation period, have continued to offer a stable and predictable framework for public and private investment. However, their implementation, which spans until 2023, has been affected by major challenges, such as the COVID-19 pandemic, the energy crisis and Russia’s military aggression against Ukraine.

Evaluations done by Member States mostly aim to analyse features and results in a regional or national context, so it is sometimes difficult to aggregate the findings and make assumptions about their general validity. Nevertheless, these evaluations provide material that it is important to discuss in the meetings of the monitoring committees of the programmes.

This document is organised thematically according to the policy areas and reflects how these are covered in the evaluations the Member States did.

After the introductory Section 1, Section 2 covers cohesion policy programmes, supported by the European Regional Development Fund (ERDF), the Cohesion Fund (CF) and the European Social Fund (ESF), including the Youth Employment Initiative (YEI). Section 3 covers rural development programmes, supported by the European Agricultural Fund for Rural Development (EAFRD). Section 4 covers the European Maritime and Fisheries Fund (EMFF). Section 5 wraps up with some concluding remarks.

2COHESION POLICY PROGRAMMES

This section presents the main findings of the evaluations of the three cohesion policy funds (the ERDF, the CF and the ESF) done by the Member States and the Commission.

2.1Evaluations by the Member States

2.1.1General overview

Since January 2015, Member States have published more than 2 200 evaluations of 2014-2020 operational programmes co-financed by cohesion policy funds. This proves how committed they are to an evidence- and results-based approach to policymaking.

In 2022, Member States published around 350 evaluations, with the number of impact evaluations increasing from the previous year and the number of implementation evaluations slowly decreasing (-14%), as is to be expected towards the end of a programming period.

The number of evaluations of 2014-2020 cohesion policy operational programmes completed and published since January 2015 has reached 2 223 3 . On the basis of the information on the data and the methods used, the findings of the evaluations done by the Member States appear to be reasonably strong.

The overall distribution of evaluations across the funds remains at around 40% of evaluations covering the ERDF/CF, 40% covering the ESF/YEI, and almost 20% covering both the ERDF/CF and the ESF/YEI (labelled as ‘multi-fund’ in Figure 1).

Figure 1: Evaluations of cohesion policy operational programmes published since January 2015 – breakdown by year and fund

Source: Commission Evaluation Helpdesk for cohesion policy.

Almost 45% of the evaluations published in 2022 are on programmes in Italy and Poland, the main beneficiaries of cohesion policy funds in 2014-2020. Overall, six countries accounted for 73% of cohesion policy evaluations. Poland and Italy did the most evaluations, followed by Germany, Czechia, France and Spain. This does not necessarily reflect the evaluation effort, because evaluations vary significantly not only in terms of scope but also in terms of the degree of detail they go into.

Figure 2: Evaluations of 2014-2020 cohesion policy operational programmes published since January 2015 – breakdown by Member State

Source: Commission Evaluation Helpdesk for cohesion policy.

Types of evaluations

The number of evaluations on implementation and progress, which cover mainly procedural aspects or progress in reaching policy targets, has decreased since the 2019 peak. By contrast, the share of impact evaluations increased to 44% in 2022, up from around 30% in 2019-2020. All 11 evaluations Bulgaria, Estonia, Lithuania and Spain published in 2022 are impact evaluations. The number of impact evaluations will keep increasing in the coming years, even after the end of the programming period, when all projects have been completed and produced results.

Most evaluations combine relatively simple qualitative with quantitative methods. However, over half of all impact evaluations use more advanced research methods, with their share rising to 62% in 2022, up from 49% in 2019. Counterfactual analysis is used mostly in ESF/YEI evaluations, and theory-based techniques mostly in ERDF/CF evaluations, while cost-benefit analysis is used to a lesser extent, primarily in ERDF/CF evaluations. There are some shortcomings that should be remedied, in particular evaluations could spell out more clearly their methodological choices, especially their theory of change. Analysis should also be based on data that is as reliable as possible. This means using data from administrative sources more often.

Figure 3: Evaluations of 2014-2020 cohesion policy operational programmes published since January 2015 – by type and year

Source: Commission Evaluation Helpdesk for cohesion policy.

Thematic distribution

The distribution of evaluations across thematic objectives is uneven, with over half of the evaluations covering more than one thematic objective.

Figure 4 shows the thematic coverage of evaluations for the 2014-2020 programming period.

Figure 4: Evaluations of 2014-2020 cohesion policy operational programmes published since January 2015 – thematic coverage (TO – thematic objective)  4

Source: Commission Evaluation Helpdesk for cohesion policy.

2.1.2Main findings of evaluations

Thematic objective 1: Research and innovation

A recent evaluation in Romania shows that measures to boost research capacity have led to greater participation of the supported centres in international research groups, a higher publication rate of scientific articles co-authored with researchers from abroad, and the setting up of more new research projects than by non-supported bodies. An example of the long-term sustainability of the links established between the private sector and academia comes from Czechia, where around 70% of companies supported launched further joint research initiatives after support from EU funds ended.

In Germany, Poland and Lithuania, evaluations show an increased capacity to commercialise research and development (R&D) results and enter new markets. In Germany (Hamburg), a recent evaluation found that 81% of the start-ups surveyed had successfully launched their products on the market, and 82% of start-ups saw an average yearly increase in sales of 63% since the first investment supported by ERDF. In Italy (Lombardy), the involvement of large companies in research projects was particularly welcomed by smaller companies because larger companies tend to have greater capacity to commercialise the output of R&D. In Belgium (Wallonia), measures to promote innovation have also had positive effects in terms of the development of new skills and interactions between SMEs, but less positive effects in terms of innovative products.

Thematic objective 2: Information and communication technology (ICT)

Recent evaluations show that, in Spain, the expansion of the broadband network has gone hand in hand with an increase in the number of companies operating in the areas concerned. Also in Spain, a recent evaluation found that there were significant differences between urban and rural areas in access to high-speed internet, and that EU-funded investment in broadband had improved coverage in rural areas. Similarly, in Poland, ERDF funding has significantly improved ICT infrastructure, particularly in rural areas. However, an evaluation in Romania found that interventions were delayed because of the reorganisation of strategic public beneficiaries and the heterogeneity of local regulations and procedures.

In Poland, projects helped to significantly increase the use and quality of e-services in the public sector. However, low access to broadband and low digital literacy are obstacles to citizens’ uptake of e-government applications. Recent evaluations show progress in developing big data platforms and e-government services in Romania, and the acceleration of the digitalisation of public management procedures, and an increase in the availability of e-services, in Lithuania.

Thematic objective 3: Competitiveness of SMEs

Counterfactual analysis in Poland (Warmińsko-Mazurskie) found that companies with innovation and competitiveness support had slightly higher revenues or profits, an increase in employment that was 4 percentage points higher than for non-beneficiaries, and more such companies (11 percentage points more) invested in R&D activities than in the control group. Also in Poland (Małopolskie), a recent evaluation showed direct support (grants) to be more effective than promotional activities in increasing the internationalisation of SMEs. Recent evaluations show that, in Slovakia, the SMEs supported, mostly start-ups, had significantly higher growth rates in production, value added and employment than those not supported, and in France support for start-ups created new job opportunities and helped increase turnover.

Direct support for R&D and innovation has also boosted the capacity of companies to develop new products and processes across the EU. This in turn has increased turnover and employment, as in Germany (Saxony) for example. In Czechia, support was provided to increase the availability of infrastructure for companies (the ‘Real Estate’ programme). The support enabled recipients to innovate and expand their production.

Thematic objective 4: Low-carbon economy

Several evaluations show the positive results of these measures in Germany, Estonia, Lithuania, France and Poland. In Poland, financial instruments were used successfully. Also in Poland, a recent evaluation shows that support from the national programme ‘Infrastructure and Environment’ for the production of thermal and photovoltaic energy led to an increase in production roughly equivalent to the output of 700 new wind turbines. The modernisation of district heating networks has also reduced energy consumption.

While actions aimed at increasing energy efficiency in SMEs have been effective, the overall effects are sometimes limited because of the small size of the companies supported (Rhineland-Palatinate). When large companies were given support, in many cases they would have invested even without EU funding (Saxony-Anhalt). A recent evaluation in Belgium (Wallonia) shows that sustainable mobility projects have helped reduce energy consumption and CO2 emissions.

Thematic objective 5: Climate change adaptation and risk prevention

In several countries, investment support to react and adapt to climate change has focused on strengthening resilience to natural disasters and improving systems for managing the risks involved. Evaluations show that new flood protection measures have been effective in Romania, Germany (Bavaria and Saxony-Anhalt) and, most recently, France (Ile-de-France). However, the evaluation in France suggests that the success of the measures could be jeopardised by the limited funding available compared to the significant structural costs involved.

Funding has also been allocated to this broad area under many Interreg programmes. Joint measures for managing climate change have been implemented under the Italy-France Maritime programme, and the joint risk-management projects undertaken under the Czechia-Poland programme have increased the capacity of the authorities concerned to deal with critical events and emergency situations. A recent evaluation shows that better coordination of country responses to potential natural disasters has been established under the Indian Ocean Area Interreg Programme, under which almost 70 000 people have been trained or informed of natural hazards risks related to climate change.

Thematic objective 6: Environmental protection and resource efficiency

Recent evaluations in Poland (Lubelskie) suggest that environmental protection measures have resulted in greater use of drainage systems and water supply mains in the supported areas, and have been associated with less outward migration. In Pomorskie, an evaluation suggests that projects for water retention in natural or semi-natural eco-systems have been the most cost-effective. Evaluations show, however, that the installation of the necessary infrastructure is not always sufficient in itself to persuade households to connect to it, with maintenance costs and charges given insufficient weight in the initial investment decision. In Bulgaria, green investment projects have helped create new jobs and promote a culture of respect for the environment, improving people’s quality of life.

Actions in the area of cultural and natural heritage have also led to an increase in the number of visitors to the areas concerned, as in Italy (Campania), where the archaeological site of Pompeii was supported with a major project recently completed. Improvements in the urban environment are also evident in several places, with positive evidence of rehabilitated urban areas in Austria, Czechia, Poland and Portugal. In Germany (Hessen), the renovation of buildings and clean-up of contaminated sites has helped prompt companies relocate to urban areas.

Thematic objective 7: Transport and energy network infrastructure

Rail investments have helped increase the capacity of railway networks, reducing journey times and improving connections between cities. In Estonia and Spain, investment in rail infrastructure has made train travel more attractive and competitive, increasing railway use. Recent evaluations show that in Slovakia, the modernisation of tram lines has increased the frequency of connections, reducing travel time while improving safety and helping to reduce noise and vibration. Evaluations in Poland suggest the success of the measures may depend on addressing the lack of well-functioning multimodal nodes connecting roads to rail and other transport networks and that, in addition to investment in infrastructure, it is necessary to take measures to make public transport a more appealing way to travel.

Thematic objective 8: Sustainable and quality employment

Evidence of the effectiveness of the actions aimed at increasing employment opportunities is consistent in various Member States, including Italy, Germany, Poland, Slovakia and Ireland, but challenges remain for measures targeting long-term unemployed and unskilled people.

In Poland, a recent evaluation estimates that overall, in 2021, about 490 000 additional jobs (half of them in sustainable growth areas) were created as a result of support from cohesion policy, with 55 000 jobs created directly by the projects supported. These jobs were mostly in services and urban areas and required higher education. In Portugal, YEI measures have proved to be effective, especially traineeships and hiring incentives for employers, with participants in traineeships having an employment rate 37 percentage points higher than the control group, and, more generally, with support under YEI having increased the probability of a person being in employment after 3 years by 8 to 32 percentage points, depending on the measure.

Evaluations of the effects of company creation found that ESF support helped participants improve their entrepreneurial skills and launch their own company. Recently, an evaluation in France (Ile-de-France et Seine) found that 95% of the entrepreneurs supported were still operating businesses 3 years after a project had ended.

In Belgium (Wallonia), an evaluation assessing socio-economic and environmental impact, with a specific focus on the contribution of the funds to employment growth and greenhouse gas (GHG) reduction, estimated that ERDF support led to the creation of 4 675 jobs, reduced annual GHG emissions by 126 369 tonnes of CO2 equivalent, and helped increase the number of SMEs in the region, as well as boosting R&D intensity and innovation.

Thematic objective 9: Social inclusion

Evaluations also show that integrated activation pathways for social inclusion have been effective in increasing vulnerable people’s chances of finding a job. A counterfactual evaluation in Italy (Marche) shows that the percentage of disadvantaged people who were employed 6 months after participating in ESF-financed traineeships was 6 to 8 percentage points higher than that of equally disadvantaged participants in ESF traineeships without ESF financing.

A recent evaluation in Germany (Berlin) found that measures that were part of a comprehensive individual approach have helped increase the social integration of vulnerable groups. Similarly, in Italy (under the National Operational Programme for Metropolitan Cities OP), a recent evaluation found that a combination of welfare, social housing and training implemented through collaboration between public administration bodies, civil society organisations, the business community and the general public led to positive outcomes for the integration of vulnerable people.

In many less developed regions, significant ESF investment went into support for health services and was complemented, in several cases, by ERDF investment in related infrastructure and equipment. Evaluations show that these investments have increased access to and improved the quality of healthcare. In Poland, an improvement in social services has led to a significant reduction in access inequalities in deprived areas, as shown most recently by an evaluation in Podlaskie.

Thematic objective 10: Education and vocational training

The effectiveness of ESF support in improving the quality of education is demonstrated by several examples, including actions aimed at reducing early school leaving or increasing participants’ chances of going on to higher education. According to a counterfactual evaluation recently published in Hungary, the proportion of students in supported schools not wanting to continue their education fell more than in the schools of the control group, and the proportion of disadvantaged young people wanting to go on to vocational training also increased.

In Portugal, grants for less privileged people had a positive effect on first-year students in higher education by reducing drop-out rates, thereby increasing the probability of students graduating on time. For second-year students, the effects were more noticeable for those who had received grants in the first year. The effects were more noticeable still for women and students from less developed regions.

There have been positive results in facilitating participants’ transition from education to work, as shown by evaluations in Poland, Italy and Germany, where a recent evaluation in Hessen shows that 30% of participants in the ‘Practice and School’ scheme had a job or were in training 6 months after the scheme ended.

Thematic objective 11: Efficient public administration

A recent evaluation in Lithuania shows that support has helped improve the quality of public services, implement state reforms, including in the justice system, and increase the expertise of civil servants. Similarly, in Romania, the measures implemented have helped simplify legislation, improve the quality of judicial services and increase access to the justice system. However, institutional instability, complex procedures and administrative bottlenecks have meant that the measures were not as effective as they could otherwise have been.

In Latvia, a set of measures for developing the skills of staff in public authorities has led to a better understanding of ways of stimulating entrepreneurship in SMEs, while corruption prevention training has also led to more cases of money laundering being brought to light.

In Czechia, measures have been introduced to improve transparency in the implementation of the ESI Funds and to prevent conflicts of interest from arising during project selection. In Estonia, the quality of government decisions, exchange of information and transparency have improved.

Public administration digitalisation projects supported under this thematic objective have helped improve operational performance, as in Portugal, where procedures have become faster and better tailored to users’ needs, and in Lithuania, where digitalisation has helped shift the focus more towards results.

Some evaluations found the project selection process too long, with the procedures to apply for funding considered too complicated and responsible for delays. A recent evaluation in Germany reports contrasting findings: in Bremen, administrative and application procedures were generally considered to be simple and flexible, while in Bayern, they were considered complex, particularly for organisations lacking experience.

Several evaluations found that the administrative burden was still considered excessive, most recently in France (Grand Est), with the effect of deterring potential beneficiaries, and in Greece (in the North Aegean), where the administrative burden appears to have hindered collaboration between public authorities and local stakeholders.

Territorial instruments: integrated territorial investment, community-led local development and sustainable urban development

In general, territorial instruments and integrated approaches have been effective in stimulating cooperation at local level to carry out local strategies and in improving the attractiveness of the areas concerned. In Poland (Świętokrzyskie), the integrated territorial instruments approach to policymaking for investment in natural and cultural assets was found to have worked efficiently and effectively and to have helped increase the attractiveness of the areas concerned, reducing the pace of decline in biodiversity and increasing opportunities for tourism.

In the Netherlands, the integrated territorial investment-based approach taken in Amsterdam, the Hague, Rotterdam and Utrecht has led to greater policy integration of social and economic aspects and increased cooperation between municipalities, schools and companies. In France (Brittany), this approach has helped to improve cooperation between the regional council and local people and organisations.

In Poland (Podlaskie), community-led local development (CLLD) strategies were prepared with the close involvement of local people and organisations, with many of the projects submitted under CLLD coming from first-time applicants for EU funding. In Czechia, the local action groups responsible for implementing the CLLD approach were considered a useful means of establishing long-term collaboration between local actors and were successful in promoting social activities.

Evaluations of sustainable urban development strategies show mixed results. In Italy, under the National Operational Programme for Metropolitan Cities, the evaluation of the support for access to housing in Genoa, Turin, Bologna and Milan emphasises the need of the authorities concerned for staff with administrative, legal and managerial skills as well as the ability to provide psychological and social support. Measures for active inclusion show positive results with regard to housing autonomy, but this is less the case with regard to improving the employment situation of participants. Nevertheless, social enterprise measures in deprived areas have had positive effects on the social and employment inclusion of vulnerable people.

Financial instruments

A recent evaluation shows that in Germany (Berlin), support for investment through venture capital funds has helped create around 1 900 jobs. In Lithuania, the net benefit of the financial instruments in question was estimated to be an additional 1% on average of annual nominal GDP, a figure that includes financial instruments not supported by cohesion policy. In Poland, financial instruments were used to promote self-employment, with 84% of the self-employed people supported still active 2 years after the end of the project.

Financial instruments have been used to fund research and innovation in SMEs, with positive results reported in Italy (Lombardy), where a loan guarantee measure has helped SMEs invest in innovative products and increase revenue, and Poland, where different schemes supported the use of innovative technology, the development of new products and projects with high market potential. In some cases, the perceived complexity of the rules led to limited uptake of financial instruments. The rules were therefore significantly simplified for the 2021-2027 period.

European territorial cooperation

In the Germany-the Netherlands programme, projects to strengthen cross-border research, technological development and innovation have made a significant contribution in terms of creating new knowledge and improving technology transfer. Similar positive results were reported under the Germany-Denmark programme, which supported the development of technologies for sustainable decentralised energy sources, and the Italy-Austria programme.

A few evaluations cover measures aimed at protecting the environment and reducing CO2 emissions. For example, the Alpine Space programme has helped develop an ecosystem service concept in the Alps, while the Sweden-Finland-Norway Botnia-Atlantica programme has increased knowledge of how to better protect and manage the sea and coastal areas. The Estonia-Latvia programme has increased awareness of energy conservation, waste sorting and re-use, and the Croatia-Serbia programme has supported electricity production from renewables and helped increase energy efficiency.

Recent evaluations show successful cooperation on thematic issues, such as smart cities ICT projects in Greece and Cyprus, and natural heritage projects in Italy and Austria.

Among the crisis measures taken by Member States, only a few programmes have evaluated this area so far. Support for companies featured prominently in many countries, such as loans for SMEs to keep them in operation in Slovakia, Czechia, and the Centre Region of France; support for innovative companies in Italy; financing for working capital and support for job retention in Czechia; and, in Italy, lump-sum support for self-employed people, the development of distance learning and healthcare technology projects for monitoring patients, diagnostic testing and genetic studies.

In Czechia, an evaluation examined the impact of the measures implemented to tackle the consequences of COVID-19. Surveys of both supported and non-supported companies suggest that the decline in employment was less steep for those receiving support. In the Netherlands, Malta and France (Grand Est), the measures were considered relevant to the needs of people and companies. At the same time, the evidence suggests that the support was not particularly well targeted towards the sectors most affected by the crisis.

In Czechia, an evaluation found that, for a specific programme, only 293 out of 1 000 pre-identified companies in the most affected sectors had received support. In Italy (Veneto), most ESF projects supported were located in urban centres with high income levels, not in the areas that needed them the most, while in Abruzzo, there were difficulties in reaching potential beneficiaries of ESF measures aimed at training workers in companies affected by the crisis. In Hungary, half of the SMEs supported reported that the projects did not fully meet their needs because the level of funding was inadequate.

In the Netherlands, an evaluation assessed the application process for accessing ‘Recovery Assistance for Cohesion and the Territories of Europe’ (REACT-EU) funding and identifying the bottlenecks to be tackled next. It found evidence that the REACT-EU scheme was tailored to the needs of beneficiaries, mostly special education secondary schools and training institutes. Application procedures were found to be easy for applicants to use, and both the co-financing rate (increased from 75% to 100%) and the adoption of simplified cost options were welcomed by beneficiaries. However, according to applicants, they weren’t given enough time to implement projects.

The higher co-financing rate and the adoption of the simplified cost option in the Netherlands, and the simplification of administrative procedures in Italy (Lombardy), improved the implementation of the measures supported.

By contrast, the counterfactual analysis done in Czechia of the measures taken during the COVID-19 pandemic did not find statistically significant effects on company survival rates and employment, perhaps because of the short time that has elapsed since the measures were implemented, but also possibly because support was not given to the companies most in need of it.

2.2Evaluation work by the Commission

In March 2023, the Commission published the final evaluation report of the ex post evaluation of infrastructures, competence centres and activities for research and technological development (RTD) financed by the ERDF in the 2007-2013 period 5 . It found that ERDF support had positively contributed to catch-up in RTD capacities, particularly in regions that were lagging behind. ERDF support did this by helping to reduce disparities between EU regions in terms of performing quality research, as some key RTD performance indicators show. On the other hand, evidence shows that there is a lack of observable long-term impacts on the use of research results for technological development and innovation, which ultimately did not increase regional competitiveness. Smart specialisation strategies were introduced in 2014-2020 to address these issues and increase the market orientation of investments.

The Commission also published the evaluation report of e-cohesion in 2014-2020. E-cohesion refers to systems for electronic data exchange between beneficiaries and programme authorities 6 . The evaluation found that e-cohesion systems reached their goal of reducing the administrative burden on beneficiaries and programme authorities, particularly in the areas of the submission and management of payment claims and progress reporting. Further improvements could be made with greater interoperability between e-cohesion systems and national databases and systems.

The preliminary evaluation of support provided by the ESF and the Fund for European Aid to the Most Deprived (FEAD) under the Coronavirus Response Investment Initiative (CRII) and the Coronavirus Response Investment Initiative Plus (CRII+), was also published 7 . The results showed that, compared to the aftermath of the 2007-2008 financial crisis, the EU responded much more quickly and more extensively to the socio-economic challenges COVID-19 gave rise to. The CRII and CRII+ were effective in giving Member States flexibility to re-programme the ESF and redirect resources to where they were most needed.

The work on ex post evaluation of the ERDF/CF and ex post evaluation of the ESF/YEI is ongoing. These evaluations cover all investment priorities in order to obtain a complete overview of the funds, including CRII, CRII+ and REACT-EU support provided in all 27 Member States and the UK. They will be ready by the end of 2025.

Preliminary findings generally support the evidence from Member States’ evaluations. Findings on SME support suggest that investments have had positive effects on turnover, employment and productivity in supported SMEs. Quick project selection and a short time-to-grant were crucial for ensuring good results. Likewise, in ICT, the Commission’s findings corroborate the evidence from Member States that investment in the digital sector is crucial for bridging the emerging technological divide between more developed and less developed areas of Member States.

The Commission has also started the work on the mid-term evaluation of cohesion policy funds of the 2021-2027 programming period. For the ERDF, the CF and the Just Transition Fund and the ESF+, the staff working document will be finalised by the end of 2024.

The Commission also did a study 8 to assess the legal and practical challenges in accessing and using administrative data for the evaluations of the ESF+. The study provides recommendations for the Member States on how to facilitate access to administrative data while following data protection rules. 

3EUROPEAN AGRICULTURAL FUND FOR RURAL DEVELOPMENT (EAFRD)

3.1Evaluations by the Member States

3.1.1General overview

Overall, reporting on evaluations by the Member States in 2023 shows solid progress, but also a slight decrease in the number of completed evaluations compared to the previous year. On the one hand, Member States reported 160 evaluations completed, a decrease of 23% compared to the year before (209). This brings the cumulative number of evaluations completed in the 2014-2022 period to 1 320. These evaluations were reported on by 23 Member States and the UK, and most of them covered the common agricultural policy (CAP) general objective 2 (Ensuring sustainable management of natural resources and climate action). This was closely followed by evaluations covering several rural development programme (RDP) priorities and objectives. On the other hand, there has been an increase in the reported evaluation findings, suggesting that Member States are effectively capitalising on the groundwork done in previous years.

Figure 5: Completed evaluations (160 in total) across the main topics reported in annual implementation reports in 2023

Source: European Evaluation Helpdesk for the CAP (2023).

Many communication activities (271) publicising evaluation findings were reported in 2023, with Member States estimating that altogether around 3.3 million people were reached. Many of them were reached through websites (2.5 million people or 77%), followed by external publications or evaluation reports (438 000 or 13%). Fewer people were reached through a combination of various communication channels, such as meetings and written consultations, exhibitions, outreach events and social media.

Fewer follow-up activities on evaluation results (355) were reported compared to 2022 (471). Member States used the evaluation results mainly to improve the RDP delivery mechanism (35%) and to prepare the CAP Strategic Plans (27%). Evaluation results were also used to improve the intervention logic of the RDP (19%) and adapt monitoring and evaluation systems (11%).

Figure 6: Use of evaluation results (N=355 follow-up activities).

Source: European Evaluation Helpdesk for the CAP (2023).

3.1.2Main findings of evaluations

Despite a slight decrease in the number of evaluations reported on, compared to the previous year, the overall number of evaluation findings reported is steadily increasing. In 2023, it increased slightly by 9% from the previous year (433 compared to 397 in 2022, and 198 in 2021). Among them, the findings on the three CAP objectives and the cross-cutting objective of knowledge-transfer and innovation have increased even more (26%).

Figure 7: Evidence basis for evaluation findings reported on in annual implementation reports in 2023 (N=433 findings)

Source: European Evaluation Helpdesk for the CAP (2023).

Most findings reported in 2023 are based on the assessment of gross results, constituting 39% of the total. There has been a significant decrease in findings based on impacts, recording, at 9%, a decline from the previous year’s 16%. This could be because of the stage the evaluation cycle is at, with managing authorities preparing for the next major milestone – an ex post evaluation in 2026.

Further analysis of the evaluation findings reported from 2021 to 2023 on different CAP objectives (379 findings out of the total 433 reported) highlights several overarching trends.

CAP objective 1 – Fostering the competitiveness of agriculture

Over the last few years, the competitiveness objective registered a total of 195 findings, showing consistent annual growth in the number of reported evaluation findings. In 2023, 33 RDPs (from 11 Member States) reported 94 evaluation findings, an increase of about 25% from the previous year.

Figure 8: Evaluation findings under CAP objective 1 ‘Competitiveness’ reported in the 2021-2023 period

Source: European Evaluation Helpdesk for the CAP (2023).

Findings showing positive effects were dominant, up to 38 by 2023 (40%). Negative effects, initially absent, accounted for 6 findings (6%) in 2023. Mixed effects are also reported in a small number of cases (8 in 2023). Findings categorised as unspecified increased sharply, reaching 38 in 2023 (40.4%).

For 2023, the largest number of positive findings under this objective is associated with rural development (RD) Priority 2: Farm viability and competitiveness. Member States have demonstrated the positive effects of the support given on the competitiveness of farms, evidenced by an increase in economic performance, gross value added and farm incomes (Czechia, France-Pays de la Loire, Ireland, Italy-Calabria and Campania, Lithuania). It has also been shown that investments have been a catalyst for more efficient production processes and higher labour productivity (Germany-Sachsen, Italy-Emilia-Romagna, Lithuania).

In 2023, Member States also reported positive effects of the measures implemented under RD Priority 3, which centres on food chain organisation, animal welfare and risk management. These positive effects encompass various aspects, such as the rise in the number of farms participating in quality schemes and producer organisations, strengthened relationships between the parties involved, greater demand for regional agricultural products, and greater profitability of food production and processing (Czechia, Italy-Calabria, Emilia-Romagna, Lazio, Apulia and Veneto).

CAP objective 2 – Ensuring the sustainable management of natural resources and climate actions

In 2023, the environmental CAP objective was, as in previous years, the most evaluated, with the highest number of evaluation findings (132) of all CAP objectives. Altogether, 35 RDPs (17 Member States) reported findings. More than half of the findings were positive about the effects of the CAP on the sustainable management of natural resources and climate action in 2023 (54%), while 12% were negative, 11% mixed, 5% showed no effect, and in 18% of cases no conclusion could be reached. 

Figure 9: Evaluation findings under CAP objective 2 ‘Environment’ reported in the 2021-2023 period

Source: European Evaluation Helpdesk for the CAP (2023).

For 2023, RD Priority 4, on ecosystems related to agriculture and forestry, stands out as the priority with the most positive effects. Member States report examples showing a significant reduction in soil erosion due to RDP measures and a substantial increase in soil organic matter (Italy-Campania, Emilia-Romagna and Veneto, and Malta). They have also reported improvements in water quality in the agricultural areas supported (Spain-Navarra, Italy-Campania, Emilia-Romagna and Sicily). The support given has also played a major role in bolstering organic farming, as reflected in the increase in land under organic farming and improved soil sustainability (Germany-Saarland and Saxony, Spain-Navarra, Italy-Friuli-Venezia Giulia and Lazio). Member States have also reported that RDP measures have greatly enhanced high nature value areas and protected biotopes and species (Germany-North Rhine-Westphalia, Spain-Navarra, Italy-Calabria). Targeted funding allocation has preserved or improved the quality and connectivity, and increased the biological diversity, of agri-ecosystems (Italy-Emilia-Romagna).

Under RD Priority 5: Resource efficiency, a low-carbon and climate-resilient economy, in 2023 Member States have also reported such positive effects as an increase in the production of renewable energy in projects supported by RDP measures (Estonia) and the continued growth of carbon sink capabilities, contributing to carbon sequestration (Spain-Navarra). Investments facilitated by the RDP have led to higher organic carbon storage in forest biomass (Italy-Emilia-Romagna).

CAP objective 3 - Achieving a balanced territorial development

Over the last 3 years, a total of 188 findings were reported for territorial development, showing a consistent yearly increase. 35 RDPs from 9 Member States reported 113 findings. The share of findings with positive effects increased slightly to 45% (51) in 2023. The number of reported mixed and negative effects was small. Findings reporting unspecified effects remained significant, at 45 in 2023 (40%).

Figure 10: Evaluation findings under CAP objective 3 ‘Territorial balance’ reported in the 2021-2023 period

Source: European Evaluation Helpdesk for the CAP (2023).

Half of the findings pertain to LEADER 9 , followed by other topics under RD Priority 6, such as diversification, job creation and ICT in rural areas. Findings reported in relation to LEADER often show mixed or unspecified effects. Nonetheless, LEADER evaluations reported by Member States have highlighted several achievements. These include supporting the establishment and expansion of companies in rural areas, supporting job creation, promoting non-agricultural activities to diversify rural economies, increasing equal opportunities in rural regions, and improving public services for disadvantaged people and people at risk of social exclusion (Estonia, Spain-Andalusia, Castilla la Mancha and Extremadura, Italy-Calabria, Emilia-Romagna, Apulia and Veneto, and Malta). The LEADER interventions have had a positive impact on social capital 10 and helped to strengthen community bonds (Spain-Andalusia, Malta, United Kingdom-Wales). The development of joint actions, integrated territorial approaches and innovative collaborations in the areas in question also underscores the positive effects of LEADER (Italy-Apulia and Marche, Spain-Andalusia). The findings also suggest promising advancements in involving diverse socio-economic actors in decision-making processes and encouraging greater participation of women in local action groups (Germany-Lower Saxony-Bremen, Spain-Aragon and Extremadura, Malta). 

Horizontal Priority 1: Fostering knowledge transfer and innovation in agriculture, forestry and rural areas

Under this priority, 78 findings were reported, reflecting a pronounced increase after a modest start in 2021. For 2023, 20 RDPs from 7 Member States reported 40 evaluation findings. Of these, 38% indicate positive effects, primarily concerning the European Innovation Partnership for Agricultural Productivity and Sustainability (EIP-Agri), under the cross-cutting objective of knowledge-transfer and innovation. The remainder of the findings show mixed (7%) or zero (5%) effects, with none considered to have negative effects. However, half of the findings showed unspecified effects.

Figure 11: Evaluation findings under the horizontal priority reported in the 2021-2023 period

Source: European Evaluation Helpdesk for the CAP (2023).

The evaluation findings show that the funded actions centred on product and process innovations, as well as the facilitation of knowledge flows, through the European Innovation Partnership for Agricultural Productivity and Sustainability (EIP-Agri). For instance, supported projects have showcased the potential of scalability to offer significant benefits to the industry (Germany-Thuringia, Italy-Tuscany, UK-Wales). Italy-Tuscany and UK-Wales report that collaboration among members of operational groups (OGs) has been constructive, with signs pointing to sustained cooperation beyond the lifespan of individual projects. OG members have also reported on the success of their projects, highlighting tangible changes in practices, including advancements in farming techniques, better animal health, data-driven decision-making, and the generation of new income streams or cost reductions (Germany-Thuringia, Italy-Liguria, UK-Wales).

3.2Evaluation work by the Commission

In June 2023, the Commission published a study on the European Network for Rural Development (ENRD) and the National Rural Networks’ (NRNs) contribution to the implementation of EU rural development policy 11 . The study concluded that networking activities carried out by the ENRD and NRNs had increased stakeholder involvement in the implementation of the 2014-2020 rural development policy with respect to the previous period. The increased exchange of information and good practices facilitated by these networks ensured better quality administration and management of rural development programmes, strengthening coordination between all stakeholders and addressing the real needs of beneficiaries. The stakeholders involved represented a wide range of individuals, including policy and programme designers and implementers, organisations representing farmers, forest managers, environmental organisations, rural communities, and actors on the ground such as farmers, project beneficiaries and local action groups. Networks at EU and national levels have contributed to balanced territorial development through their mutually beneficial relationship in support of rural development, and through specific thematic support.

4EUROPEAN MARITIME AND FISHERIES FUND (EMFF)

4.1Evaluations by the Member States

By the end of 2022, a total of 72 evaluation operations have been reported 12  for 17 EMFF programmes. EUR 8.6 million of total eligible public costs (EMFF and national contributions) was committed to support this work. Of this amount, around one third was committed in Spain (EUR 3.1 million), followed by Sweden (EUR 1.3 million), Finland (EUR 1.2 million) and Italy (EUR 1.2 million). Spain, Czechia and Sweden reported the highest number of evaluations (18, 10 and 9 respectively). The Commission gives practical support to managing authorities to boost Member States’ evaluation capacity for the EMFF and to broaden the scope and the quality of the evaluations.

The information used for EMFF evaluations comes from the annual implementation reports (AIR). However, the information presented in them varies significantly from one Member State to another in terms of the level of detail, likely correlated to the scope and depth of the evaluation process itself. It should be borne in mind that for several Member States, EMFF budgets are rather small, a limiting factor for the scope of the evaluations.

Evaluations vary considerably by type. In recent years Member States’ focus has shifted from process and mid-term evaluations to effectiveness and impact evaluations. They cover a wide range of topics, considering effectiveness and efficiency at the operational programme level, at the level of the Partnership Agreement, and at the levels of EU priorities, specific objectives and measures (for example, community-led local development (CLLD) and permanent cessation) or strategies (for example, aquaculture strategy). Several impact evaluations were also done – for example, assessing the impact at the level of EU priorities and specific EMFF measures, the impact of the operational programme interventions on the environment and sustainability, and assessment of the impact of the COVID-19 pandemic and Russia’s military aggression against Ukraine on the fisheries sector.

Several Member States have mentioned in their annual implementation reports preparing for ex post evaluations, where a detailed assessment of the achievement of programme-specific objectives will be done, considering their relevance, effectiveness, efficiency and durability.

Recommendations in the evaluations cover a wide range of topics, with some recommendations common to several Member States. One such recommendation is that amendments to the national operational programmes and the re-allocation of financial resources should be pivotal to EMFF implementation; this is particularly important since EMFF expenditures are eligible only until the end of 2023.

Country-specific recommendations include suggestions to reduce the administrative burden; build the capacity of managing authorities; improve the efficiency of communication and publicity measures; introduce simplified cost options; make the instructions for applicants and beneficiaries provided by the managing authorities more clear; centralise information for applicants and beneficiaries to improve accessibility; establish a streamlined and permanent electronic system for data collection; support innovation in programmes; introduce more systematic monitoring of strategically important investments; enable the pre-financing of projects to encourage a larger number of potential beneficiaries to apply; facilitate active communication with commercial banks to offer suitable financial packages to top up the EMFF grant to support project implementation; streamline the number of interventions in the interests of efficiency and clarity; ensure clear targeting (to increase uptake) of areas with pressing needs.

EMFF evaluations have helped Member States to assess funding needs and facilitated preparations for the 2021-2027 period. The evaluations have confirmed the feasibility of the European Maritime, Fisheries and Aquaculture Fund programme drafts; the appropriateness of the intended goals, priorities and actions in the context of the sector’s needs; and the appropriateness of the financial allocations. Some Member States also did strategic environmental assessments for 2021-2027 programmes as part of evaluations.

Here is a non-exhaustive list of specific evaluation topics.

·Seals and cormorants damage compensation scheme.

·Review of the fisheries local action groups (FLAG) component of the programme (EMFF Union Priority 4);

·Cost-benefit analysis of the fishing vessels decommissioning scheme. 

·Ex ante assessment of the use of financial instruments. 

·Sustainable fisheries scheme.

·Communication means and channels for primary target audiences.

·Compensation of aquaculture environmental measures.

·Sustainability of the fishing industry in terms of the consumption of productive resources.

·Investment support for aquaculture and the processing industry. 

·Follow-up of investment support for aquaculture and nutrient reduction. 

·Testing of the model for the LEADER effects evaluation.

·Environmental benefits of the EMFF.

There is detailed information summarising the evaluations of each Member State and based on information provided in annual implementation reports in Section 7 of the EMFF implementation report for 2022, published in October 2023.

4.2Evaluation work by the Commission

The ex post evaluation of the 2014-2020 European Maritime Fisheries Fund (EMFF) under Article 117 of the EMFF Regulation and Article 57 of the 2014-2020 Common Provisions Regulation needs to be done by the end of 2024. The evaluation will aim to identify:

·how the planned objectives have been achieved in different sectors and in relation to topics for each Union Priority separately, using the theory of change approach;

·effectiveness, efficiency, relevance, coherence and EU added value of the support given through the EMFF programmes in the 2014-2020 programming period;

·aspects regarding the actions taken to respond to recent and ongoing crises and market disruptions (COVID-19, Russia’s military aggression against Ukraine, climate change, biodiversity, etc.).

The mid-term evaluation of the 2021-2027 European Maritime, Fisheries and Aquaculture Fund, also needs to be done by the Commission by 31 December 2024 in line with Article 45 of the 2021-2027 Common Provisions Regulation. The evaluation will cover all 26 EMFAF programmes implemented under shared management and the support under Title III of the EMFAF Regulation, managed under direct management in the 2021-2027 programming period.

·

5FINAL REMARKS 

In 2022, Member States continued to allocate a considerable amount of resources to evaluation, with an increasing focus on the impact of the actions supported. This confirms the commitment of the Member States to invest valuable resources in evaluation as a tool that promotes accountability, transparency and policy learning.

The Commission continues to provide support and collaborate with the Member States to help them produce quality evaluations. It has also increased its methodological support services for the 2021-2027 period.

The evaluation findings have been used for programme adjustments in the final part of the 2014-2020 programming period and for the design of the programmes for the 2021-2027 period. They will also feed into the ex post evaluations of cohesion policy funds the Commission is doing, which will produce several intermediate outputs, before the synthesis reports for each fund are due in 2025.

(1)    Regulation (EU) No 1303/2013 of the European Parliament and of the Council of 17 December 2013, https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32013R1303  

(2)     https://cohesiondata.ec.europa.eu/

(3)    This includes some evaluations for both 2014-2020 and 2007-2013.

(4)    Evaluations covering more than one thematic objective are reported under all the thematic objectives in question. Evaluations that broadly cover all thematic objectives have not been included in the chart.

(5)     https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52023SC0071  

(6)     https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52023SC0258  

(7)     https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8558&furtherPubs=yes  

(8)     https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8562&furtherPubs=yes  

(9)    LEADER (Liaison Entre Actions de Développement de l’Économie Rurale) promotes integrated rural community-led local development with key features of bottom-up and multi-sector approaches, local partnerships and place-based design and delivery, funded through the Common Agricultural Policy (Regulation (EU) No 1305/2013), under a common framework (Regulation (EU) No 1303/2013).

(10)    Social capital can be defined as the features of social organisation such as networks, norms and trust that can facilitate action and cooperation for mutual benefit. It is linked to social relations and participation in local community activities.

(11)     Study on the ENRD and the NRNs’ contribution to the implementation of EU rural development policy - Publications Office of the EU (europa.eu)

(12)    MS reports in line with Article 97(1)(a) of Regulation (EU) No 508/2014.