EUROPEAN COMMISSION
Brussels, 17.12.2021
SWD(2021) 384 final
SYNTHESIS OF THE FINDINGS OF THE EVALUATIONS OF EUROPEAN STRUCTURAL AND INVESTMENT FUNDS PROGRAMMES 2014-2020
COMMISSION STAFF WORKING DOCUMENT
Accompanying the document
REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS
European Structural and Investment Funds
2021 Summary report of the programme annual implementation reports
covering implementation in 2014-2020
{COM(2021) 797 final}
STAFF WORKING DOCUMENT
ON THE
SYNTHESIS OF THE FINDINGS OF THE EVALUATIONS OF
EUROPEAN STRUCTURAL AND INVESTMENT FUNDS
PROGRAMMES 2014-2020
accompanying the 2021 Annual Summary Report
Contents
1.INTRODUCTION
2.COHESION POLICY PROGRAMMES
2.1.Evaluations in the Member States
2.1.1.General overview
2.1.2.Main findings of evaluations
2.1.3.Support to the Member States
2.2.Evaluation work by the Commission
3.EUROPEAN AGRICULTURAL FUND FOR RURAL DEVELOPMENT (EAFRD)
3.1.Evaluations in the Member States
3.1.1.General overview
3.1.2.Main findings of evaluations
3.1.3.Support to the Member States
3.2.Evaluation work by the Commission
4.EUROPEAN MARITIME AND FISHERIES FUND (EMFF)
4.1.Evaluations in the Member States
4.2.Evaluation work by the Commission
5.FINAL REMARKS
1.INTRODUCTION
Under the rules of shared management, Member States are responsible for managing and implementing the programmes supported by European Structural and Investment (ESI) Funds. These programmes pursue the objectives of the EU strategy for smart, sustainable and inclusive growth.
The Common Provisions Regulation for 2014-2020(CPR) requires that evaluations are carried out to improve the quality of the design and implementation of programmes, and to assess their effectiveness, efficiency and impact. During the programming period, Member States are required to carry out evaluations for each programme and ensure that each evaluation is duly followed up. At least once during the programming period, an evaluation should assess how the support from the ESI Funds has helped achieve the objectives for each priority of the programme. So far, more than 2700 evaluatios have been carried out by Member States on ESI Funds for the 14-20 programming period.
In line with Article 53 of the CPR, this staff working document summarises the findings of the latest available evaluations of the programmes supported by the ESI Funds
. At this stage of the programming period, the Member States carry out most evaluations, while the Commission’s main responsibility is to carry out ex-post evaluations. This document outlines the current situation of ESI Funds evaluation in the Member States and of the Commission’s parallel work. It accompanies the 2021 summary report of the programme annual implementation reports covering implementation in 2014-2020.
Evaluation is key to ensuring accountability and effective performance. It supplies policymakers with robust analytical findings to ensure that action supported by the ESI Funds achieves the best possible results. This continuous learning process helps improve relevant EU policies. In addition, the Commission provides specific support by fostering evaluation in all Member States.
The following sections present the main findings of the evaluations of the ESI Funds programmes that Member States and the Commission have carried out, mainly in the past year. The structure follows the broad policy areas:
·Section 2 covers cohesion policy programmes, supported by the European Regional Development Fund (ERDF), the Cohesion Fund (CF) and the European Social Fund (ESF) including the Youth Employment Initiative (YEI);
·Section 3 is devoted to the rural development policy, supported by the European Agricultural Fund for Rural Development (EAFRD);
·Section 4 focuses on the European Maritime and Fisheries Fund (EMFF); and
·Section 5 contains some concluding remarks.
2.COHESION POLICY PROGRAMMES
Cohesion policy is one of the most evaluated policies in Europe, and rightly so, considering that it represents around one third of the total EU budget.
This section presents the Member States and the Commission’s work in evaluating the work and programmes supported by the three cohesion policy funds.
0.
1.
1.0.Evaluations in the Member States
1.0.0.General overview
Since January 2015, the Member States have published more than 1 500 evaluations of 2014-2020 operational programmes. Programmes and Member States have multiple approaches to evaluation. Nevertheless, the growing number of evaluations shows that the Member States are stepping up their work and commitment to strive for evidence-based policymaking and to focus on results.
Most evaluations deal with procedural aspects and examine progress in achieving policy targets. However, a significant and growing number of evaluations aims to assess the impact of EU support, and the mechanisms that influence the effectiveness of this support. These evaluations are typically carried out later on in the programming period, once the actions haves produced enough results to be assessed. The number of these evaluations is expected to increase further.
Summary of common evaluation findings
Analysing the impacts of EU support is very challenging. Firstly, because it is difficult to establish causality between the financial support and the results observed on the ground as many interconnected factors influence these results. Secondly, because evaluations are mostly designed to deal with the results of supported projects in a local context, so their findings are difficult to aggregate as they are mostly specific to this local context.
Nevertheless, despite the local relevance of evaluations, some common findings are starting to emerge in various contexts. These can provide important insights for the ongoing negotiations on the 2021-2027 programmes.
For example, funding allocated to research and innovation has strengthened cooperation between research centres and businesses in many cases. This helps overcome a fundamental obstacle to innovation, namely to translate research results into innovative products or services. Making these marketable is often even more difficult. In addition, in most cases where support has led to positive results, EU funding has brought real benefits to the projects. Without it, the projects would have been smaller or would not have been carried out at all.
Support to investment in infrastructure for information and communication technology has helped meet the needs of less developed regions at risk from digital divide. However, it takes time for results to materialise. On the other hand, support to the digitalisation of services tends to produce results more quickly and its positive effects for businesses and public authorities are clearly visible.
The whole range of measures to support SMEs has boosted innovation in businesses, strengthening their competitiveness and enabling them to enter new markets and expand their exports.
There is a wide range of measures to implement support to the shift to a low-carbon economy. Preliminary evidence shows that the measures are effective, but it will take more time for the effects to materialise fully.
Evaluations have shown that cross-border aspects are important in order for climate change adaptation and risk prevention support to be truly effective for the regions concerned.
Installing and improving infrastructure for wastewater treatment and municipal waste management does not always increase the number of connected users. This is caused by the potential high charges to be borne by the users, which, unless infrastructure is realised through a major project, sometimes do not carry enough weight in the initial investment decision.
Investments in the transportation sector led to increased capacity of railway network, even though their impact can only be assessed in the longer term.
Employment-related measures and training have proved to be most successful when tailored to both the needs of those receiving support and the skill demands of the labour market. Apprenticeships and traineeships have shown to be more effective than other measures. Support is also important to improve the skills of teachers and trainers.
Initiatives aimed to foster social inclusion have helped vulnerable and disadvantaged people find work.Evaluations have also found that projects have helped increase access to and the quality of healthcare.
The effectiveness of action to increase institutional capacity depends as much on the motivation and cooperation of the staff involved as on the specific design of the measures themselves. Structural reforms require willingness at the political level.
Evaluations sometimes find contrasting evidence in various regions. This suggests that analytical work must increase its focus on the contextual factors that affect implementation and consequently the results. Such an analysis will be part of the ex-post evaluation that the Commission will carry out in the coming years and whose preparatory work has already started.
Completed evaluations
The number of evaluations of 2014-2020 cohesion policy operational programmes completed and published since January 2015 has reached 1 506, including a few evaluations related to both the 2014-2020 and the 2007-2013 period. The increase in the past year (to over 1 500, up from over 1 000) continued in line with the trend observed the year before. The distribution of evaluations across the funds has not changed much in the past year, with almost 20% of the evaluations covering both ERDF/CF and ESF/YEI.
Table 1: Evaluations of cohesion policy programmes published since January 2015 – breakdown by fund
|
Funds
|
Number of evaluations
|
of which in the past year
|
|
ERDF+CF
|
609
|
194
|
|
Multi-fund
|
272
|
76
|
|
ESF+YEI
|
625
|
191
|
|
Total
|
1 506
|
461
|
Source: Commission evaluation helpdesk for cohesion policy.
The distribution of evaluations across the Member States is uneven. Six countries account for over 75% of the evaluations of cohesion policy published so far. Poland has produced the largest number, followed by Italy, Germany, Czechia, Spain and France. However, the number of evaluations alone is not necessarily representative of the evaluation effort made or evidence collected, since the scale and scope of evaluations (e.g. number of operational programmes covered, priorities, themes, etc.) vary widely.
Figure 1: Evaluations of 2014-2020 cohesion policy operational programmes published since January 2015 – breakdown by Member State
Source: Commission evaluation helpdesk for cohesion policy.
Types of evaluations
There are still fewer evaluations assessing impact than evaluations focusing on implementation and progress, which look mostly into procedural aspects or progress in achieving policy targets. This is mainly because impact evaluations are normally carried out late in the programme cycle, when the impacts start to materialise. The number of impact evaluations completed in the past year has slightly increased, and it is expected to grow further towards the end of the programming period, when more projects will have been completed and will have produced results.
In relative terms, the share of impact evaluations has topped 30% in the past two years and is expected to grow further as progress in implementation produces more results. Although the total number of evaluations in Estonia, Finland, Latvia, Lithuania and Portugal is not particularly high (between 9 and 28 in the 2014-2020 period), it is worth noting that more than 50% of them assess impact.
Figure 2: Evaluations of 2014-2020 cohesion policy operational programmes published since January 2015 – by type and year
Source: Commission evaluation helpdesk for cohesion policy.
Most evaluations combine relatively simple qualitative methods with quantitative ones. Around half of all impact evaluations use more advanced research methods, such as counterfactual evaluation techniques (mostly in ESF/YEI and multi-fund evaluations), theory-based techniques (mostly in ERDF/CF evaluations) and cost-benefit analysis.
Thematic distribution
The distribution of evaluations across thematic objectives (TOs) is uneven as the nature of the action funded and its financial weight vary. In fact, more than half of the evaluations cover multiple TOs and many of them are not focused specifically on each TO, but cover them in relation to measures supported under other TOs. This often happens for evaluations that span across an entire operational programme.
Figure 3 shows the thematic coverage of evaluations for the 2014-2020 programming period. Evaluations covering multiple TOs are reported under each of them. However, the chart does not include evaluations that broadly address all TOs.
Figure 3: Thematic coverage of evaluations of 2014-2020 cohesion policy operational programmes published since January 2015
Source: Commission evaluation helpdesk for cohesion policy.
Most evaluations cover the TOs related to inclusive growth, i.e. employment (TO8), social inclusion (TO9) and education (TO10). Typically, actions under these TOs, receiving ESF and YEI support, are implemented faster and generate results in a relatively short time.
1.0.1.Main findings of evaluations
This section summarises the main findings of those evaluations of 2014-2020 programmes assessing the impact of EU support. The findings appear to be reasonably robust based on the information on the data used and the methods applied. They are often specific to the individual evaluations, as they are strongly dependent on the local implementation context and may not be valid in all cases. Recently, the Commission has launched the ex-post evaluation of the 2014-2020 period to examine cross-cutting impact and draw policy lessons (see Section 2.2). The findings of Member States reported below are organised by TO. This section also specifically focuses on integrated territorial investment, financial instruments and European territorial cooperation. It also contains a review of the cross-cutting findings of evaluations assessing procedural aspects. In line with the 2021 annual aummary report, a brief overview of the achievements and financial evolution by the end of 2020 accompanies each headline policy investment discussed below.
Figure 4 shows the overall planned contribution of cohesion policy funds to the TOs by the end of 2020.
Figure 4: Total amount planned (EU and national) under 2014-2020 cohesion policy operational programmes – breakdown by TO
Source: European Commission.
·Smart growth: research & innovation, information and communication technology and SME competitiveness
The total investment in smart growth supported by the ERDF in the 2014-2020 period amounts to EUR 114.4 billion, of which projects had actually spent EUR 60.8 billion (52%) at the end of 2020. The total eligible cost of selected projects contributing to this headline priority is even higher than the allocations, i.e. EUR 127.3 billion. Programme authorities use this selection strategy to ensure that by the end of the programming period, resources are fully spent, including those that would otherwise remain unallocated.
ØTO 1: research & innovation
As the end of the programming period draws closer, the results of the funded operations move closer to their targets. By the end of 2020, the cohesion policy programmes had planned EUR 56.5 billion of investment in this area, of which they had spent EUR 26.2 billion (46%) already. Thanks to the ERDF investment in research and innovation, more than 49 000 enterprises cooperated with research institutions (76% of the target) and about 24 000 enterprises introduced new products to the market (79% of the target).
Evaluations give insight into the performance of the projects that contributed to these targets. Evaluations in various Member States show successful examples of cooperation between SMEs, universities and research centres. In Czechia, some EU-funded projects have exceeded their targets for cooperation between enterprises and research centres. Moreover, most of the companies involved (70%) have continued to cooperate after the support drew to a close. Fruitful cooperation has also helped develop cross-border technology transfers between Germany and the Netherlands, and improve the innovation environment in Austria.
Several evaluations have also found that the ability of businesses to develop new products and processes had increased. Thanks to EU support, companies in Germany have introduced new products and services, and improved the existing ones. SMEs in Poland have introduced innovative processes and increased their profitability, but these are largely based on the search for novelties and there is room to increase their added value.
In several cases, support to research and innovation has also been conducive to smart specialisation strategies. For example, in a region in Poland, the support has helped eliminate barriers to innovation. And in some regions of Portugal, Spain and Italy, supported companies have been able to increase their exports and participation in international markets.
ØTO 2: information and communication technology (ICT)
Supporting the digital transition in the EU is one of the Commission’s headline priorities. The planned total ERDF allocation to this objective for the 2014-2020 programming period amounts to EUR 14.4 billion. By the end of 2020, programmes had spent EUR 6.5 billion of it already. As a result, 5.5 million households had received improved broadband access, representing more than 45% of the target. Nearly 12 million households are expected to benefit from improved broadband access by the end of 2023.
Evaluations show that in the long term, cohesion policy support has helped increase broadband access in some EU regions. In France, for example, ERDF support has made it possible to provide high-speed broadband services. In many other EU regions, such as Sweden, Spain and Italy, cohesion policy has helped deliver major infrastructural projects. Also in Poland, cohesion policy support has significantly improved ICT infrastructure, notably in rural areas.
Evaluations have also found that support to digitalisation has created opportunities to develop ICT products and services and e-government. In Poland, for example, the wide use of e‑services receiving EU fund support has increased the transparency of public sector action and reduced digital exclusion. In Italy, digital platforms set up to enable communication between local administrations and enterprises have reduced administrative costs.
Investments in digitalisation have also helped improve digital skills of workers in Czechia and of young people and households in difficult socio-economic conditions in Poland. In other cases, evaluations have highlighted a lack of appropriate skills that prevented beneficiaries from making the most of funding, such as in the education sector in Poland and Sweden.
ØTO 3: SME competitiveness
Increasing the growth, productivity and competitiveness of enterprises is one of the best performing areas of ERDF support. This support proved to be vital, especially in the response to the global health crisis. By the end of 2020, cohesion policy programmes had planned EUR 43.5 billion of investment in this field, of which they had spent EUR 28.1 billion (64%) already. More than 1.4 million enterprises had received ERDF support in the form of grants, financial instruments or non-financial support. The supported enterprises had created 238 000 new jobs, while an additional 213 000 jobs are expected to be created by the end of 2023.
Evaluations of TO 3 have found that support in this sector has often achieved its objectives, such as increasing competitiveness and employment of enterprises. This was the case across regions of Poland and Germany, where companies also have better access to international markets thanks to additional capital provided by the availability of additional capital through grants and financial instruments.
Support has also boosted enterprises’ productivity in some Polish regions, and has improved the ability of German firms to undertake larger projects. Furthermore, supported enterprises have often grown, or at least maintained their level of activity, and have been able to consolidate their market position, for example in Romania.
In Portugal, Italy and Spain, cohesion policy funding has strengthened enterprise’ competitiveness by improving their ability to innovate. In the Italian case, supported businesses have also improved their ability to attract private investment.
Sustainability is an important aspect of improving SME competitiveness. First evidence suggests that the sustainability of the results depends in particular on the relationships developed between research centres and businesses and the ability to attract other co‑investors. Selection criteria that favour companies with an explicit, and seemingly viable, expansion strategy can improve sustainability.
·Sustainable growth: low-carbon economy, climate change, environment, transport and energy networks
The European Green Deal is at the forefront of EU investments and support policies. Cohesion policy funds play their part by supporting climate-neutral, clean and circular economy investment and investment in the areas of environment and climate change adaptation. The ERDF and the CF support this headline priority with more than EUR 136.5 billion in the 2014-2020 programming period, of which the Commission has paid EUR 66.6 billion (48%) to the Member States already.
ØTO 4: low-carbon economy
The low-carbon economy objective is primarily focused on investment in energy efficiency and renewable energy production. By the end of 2020, cohesion policy programmes had planned EUR 37.9 billion of investment in this field, of which EUR 16.9 billion (44%) had already been spent. By the end of 2020, the ERDF had supported more than 358 000 households to improve their energy consumption classification (up by 75 000 in the last year). It has also helped increase the renewable energy production capacity by 2 700 MW; the end‑2023 target being 8 000 MW.
Evaluations have found that measures to improve energy efficiency have been effective across the EU. A study in Italy has found that the support has helped reduce energy use in public buildings. Furthermore, energy efficiency projects in SMEs in Germany have led not only to cost savings but also to competitive advantages, including an improved company image.
Many countries invested significant sums in this type of projects. Despite this, evidence on the impact of the measures is somewhat limited because results take time to materialise and many projects are still ongoing. In some cases, for example in Germany, investment in renewable energy is associated with innovative projects and technologies. Tangible outcomes of these in terms of energy use and efficiency gains are not visible yet.
Cohesion policy support under this TO also aimed to improve urban transport systems and make them more environmentally friendly, for example helping reduce CO2 emissions in a German region.
ØTO 5: climate change adaptation and risk prevention
Cohesion policy funds also support investment in climate change adaptation and risk prevention. By the end of 2020, the Member States had already spent EUR 2.5 billion (41%) out of the total amount planned of EUR 6.0 billion. Thanks to these investments, 11.3 million people (more than 40% of the target) are now less exposed to flood risks.
Evaluations have found that support under this TO has helped improve the systems for monitoring risks and severe weather events. In Romania, supported projects to monitor such events have limited floods, reduced their damage and provided appropriate emergency equipment. In Poland, support has helped develop a disaster recovery system.
Italy and France have also taken joint measures to manage climate change under the European Territorial Cooperation goal. Furthermore, authorities in Czechia and Poland have cooperated to improve their ability to tackle emergency situations.
ØTO 6: environment protection and resource efficiency
EUR 29.6 billion are allocated through ERDF and CF to projects for environment protection and resource efficiency. Implementation is steadily progressing: by the end of 2020, projects had already spent EUR 11.5 billion (38%). Thanks to these investment, 3.5 million people benefit from an improved water supply already, with a target of 19.5 million benefiting people at the end of the period.
Evaluations of some environmental protection and resource efficiency projects have found the supported measures to be effective in improving wastewater management and the preservation of natural habitats. In France and Latvia, measures to improve wastewater management and preserve water basins have strengthened the conservation of habitats and biodiversity Territorial cooperation programmes have also supported the protection of ecosystems, such as the one between Italy and France, which aims to protect ecosystems by developing green infrastructure. In Romania, support helped expand the sewer network, increase the extent of recycling and close non-compliant landfills. However, the connection of households to main drainage is still limited.
Support under this heading has also helped preserve natural and cultural heritage by:
·contributing to the development of a network of tourism sites in Poland;
·safeguarding the archaeological site in Pompeii (Italy) and improving its accessibility; and
·fostering cooperation between Germany and Czechia.
Evaluation findings have found evidence that Cohesion policy investments have also helped rehabilitate urban areas in Austria, Czechia, Poland and Portugal. It has also helped to improve waste management systems in Czechia, Poland and Portugal and has boosted the circular economy in France and Belgium.
ØTO 7: transport and energy network infrastructure
Energy and transport network infrastructure receives EUR 63.0 billion of support by ERDF and CF in the 2014-2020 programming period. By the end of 2020, projects had spent EUR 35.7 billion, thus reaching an implementation rate of 56%. Investments in clean urban transport supported by the ERDF and CF had delivered 137 km of new or improved tram and metro lines by the end of 2020 (29% of the target). An additional 405 km of track is expected to be put into use in the coming years.
Support under this heading mainly aimed to develop road and rail networks. In Poland, Seeuch investments have improved road safety and reduced the number of accidents in several instances. It has also reduced travel times, noise and air pollution in cities. Furthermore, rail investments have helped increase the capacity of railway networks, reducing journey times and improving connections between cities. This has increased railway use.
Transport projects often take a long time to deliver and it is rarely possible to assess their impact within the same programming period. It is long-term data that gives relevant information on the impact of this type of investment. Moreover, to attract more passengers and reduce car use, investment in public transport infrastructure, notably rail, needs to be matched with high-quality transport services and commitment. For example, investment in tramway lines in Italy has helped make the regions concerned more attractive, and reduce car use in favour of public transport. Also in Estonia and Spain, support to rail infrastructure has made train travel more attractive and competitive, increasing passenger numbers.
·Inclusive growth: employment, social inclusion and education
The European Pillar of Social Rights, which is the basis of the EU’s social agenda, is one of the main driving forces of cohesion policy investment in sustainable and quality employment (EUR 45.5 billion), social inclusion (EUR 40.1 billion) and educational and vocational training (EUR 37.6 billion). By the end of 2020, the average spending rate for the three topics was 54% (EUR 66.8 billion of declared expenditure), a significant increase compared to 2019 (41%). Support to this policy area mainly comes from the ESF and the YEI.
ØTO 8: sustainable and quality employment
By the end of 2020, cohesion policy programmes had planned EUR 45.5 billion of investment in this area, of which they had spent EUR 25.5 billion (56%) already. The ESF and YEI had supported 45.3 million participants, including nearly 17.3 million unemployed and 17.2 million inactive participants. Moreover, 5.4 million people had found a job thanks to ESF and YEI support.
Evaluations of measures to support employability have found that these programmes, often combined in a tailor-made package of training, traineeships and work experience, have increased the chances of the beneficiaries of finding a job. In Italy, for example, traineeships and voucher schemes have increased participants’ chances of finding a job compared to non-participants. In Germany, more participants were employed 15 months after the end of the projects compared to non-participants. In Poland, such projects have proved to be successful also among long-term unemployed, young and low-skilled individuals.
Support under this heading also aimed to improve the skills of those already in employment and to develop entrepreneurial skills. In Poland and Italy, such projects have aided the creation of new enterprises and the survival of enterprises in the market after the support ended. In Germany, people who had attended training were able to adapt better to changes in market conditions and to improve their labour market situation.
ØTO 9: social inclusion
By the end of 2020, cohesion policy programmes had planned EUR 40.1 billion of investment in this area, of which they had already spent EUR 20.5 billion (51%). Cohesion policy investments have improved skills for the job market and employment opportunities for 3.1 million participants with disabilities, 6.8 million migrants, participants with a foreign or minority background and 8.5 million other disadvantaged people. In addition, 18.5 million people have access to expanded childcare and education infrastructure and 90 million people benefit from improved healthcare services across the EU.
Much of the ESF support under this heading aimed to foster social inclusion by supporting people into jobs. Evaluations of such initiatives have found instances where the support has helped vulnerable and disadvantaged people find work. In Spain, projects have increased the chances of vulnerable people of finding employment while also reducing their risk of suffering from mental illnesses. In Italy, 20% of the participants in a project supporting the employability of people with disabilities or other disadvantages had a job one year after the project ended.
A significant part of the ESF investment also supported healthcare services, sometimes accompanied by ERDF investment in healthcare infrastructure (particularly in less developed Member States). Evaluations of this investment area have found that projects have helped increase access to and the quality of healthcare. In Lithuania, such projects have helped reduce the risk of death from cardiovascular disease and reduce the suicide rate.
ØTO 10: education and vocational training
By the end of 2020, selected projects in educational and vocational training amounted to EUR 37.6 billion, of which EUR 20.8 billion (55%) were already spent. Thanks to ESF and YEI support, 21.8 million low-skilled people had participated in specific activities, 7.4 million had gained a qualification and 2.2 million had participated in education and training.
Evaluations have found that education and vocational training has reduced early school leaving and improved attitudes to schools, notably in Germany, where this training also improved the integration of migrants. In Germany, training was also open to teachers, who have gained communication and conflict resolution skills.
Support to higher education has made universities in Lithuania more international, increasing the share of foreign students and the number of Lithuanian students spending part of their studies abroad.
Projects have also improved the links between education and the labour market. In Poland, increased cooperation between vocational training schools and employers has improved teachers’ skills and helped students choose suitable courses. In Germany, cooperation with local entrepreneurs has enabled students to make informed career decisions, and in Italy, technical courses have improved the participants’ chances of finding a job.
·Strengthening institutional capacity and efficient public administration
ØTO 11: efficient public administration
By the end of 2020, the cohesion policy programmes had selected institutional capacity projects worth EUR 5.5 billion, corresponding to 96% of the planned allocation (EUR 5.7 billion). The implementation rate had reached 41%, with the Member States having already received EUR 2.3 billion of payments. Thanks to the investments under this TO, more than 720 000 people had participated in targeted activities of more than 2 600 supported projects.
Evaluations in institutional capacity have found that in many EU regions, investment under this TO has strengthened collaboration and networks among cohesion policy’s stakeholders. In Poland, an improved understanding of good governance practices has led public authorities to increase citizens’ involvement in spatial planning procedures. In Slovenia, supported projects have helped strengthen non-governmental organisation networks and connect them with public authorities. In Czechia, such investment has enabled collaboration between social partners.
In many instances, this investment has also made public administration more transparent, for example by improving the exchange of information on government decisions in Estonia. However, it has been found that public administrations also suffer from insufficient resources, hampering the implementation of communication strategies and evaluations.
Improved governance thanks to projects in this field has also improved the skills of staff in the managing authorities. In Czechia, staff have improved their know-how and capabilities, which has helped make the organisation more efficient. In Latvia, supported projects have given public authorities a better understanding of ways to promote entrepreneurship. In Lithuania, the projects have increased the efficiency of investment thanks to a stronger link with reforms. In Romania, investment has boosted staff’s hard and soft skills, improving decision-making and simplifying institutional procedures.
·Cross-cutting instruments
ØTerritorial instruments: integrated territorial instruments, community-led local development and sustainable urban development
The results of this investment are likely to become visible in the long term only. Nonetheless, a number of evaluations have shown instances across the EU where territorial instruments had been implemented successfully.
Some of these findings suggest that the instruments have encouraged close involvement of the local population and organisations, which has helped public authorities meet local needs in a more targeted way. For example, in the two Polish regions that used community-led local development, many people have become involved in the elaboration of local development strategies. Furthermore, many projects were submitted for funding from beneficiaries who did not previously receive EU support. Again in Poland, some integrated territorial instruments have supported the development of natural and cultural assets, which has made the concerned areas more attractive, increased their tourism potential, and protected biodiversity.
Clearly defined and appropriate procedures for creating local partnerships also seem to be important to manage and coordinate activities efficiently. In the Netherlands, for example, integrated territorial instruments have enabled policymakers to integrate better the social and economic aspects of local policies. They have also increased cooperation among public and private stakeholders. In France, they have improved the collaboration between public authorities and local community organisations.
ØFinancial instruments
Many evaluations have studied the use of financial instruments and have found several instances where these had been successful. Financial instruments can support different policy areas, such as research and innovation, SME competitiveness, energy efficiency, etc. Nevertheless, the reported findings mostly relate to their use for business support.
In Poland, financial instruments have supported businesses: loans have enabled firms to grow and develop new products and services or to maintain their operations. In some cases, financial instruments have been considered a significant incentive to innovate.
In Germany and Northern Ireland, financial instruments have increased business competitiveness, profits and turnover. In Italy, these instruments have enabled SMEs to carry out or extend research, technological development and innovation projects.
In some cases, however, financial instruments are considered difficult to navigate. In Czechia, the number of instruments creates fragmentation and overlaps. Polish authorities face difficulties arising from the dense national regulatory framework, complicated implementation model, lack of coordination and harmonisation of products.
Considering the potential competition from the financial markets or from national or regional support measures offering more favourable terms, including non-repayable grants, when choosing the use of financial instruments, the Member States and the managing authorities should assess various support schemes to ensure synergies and complementarities of support.
ØEuropean territorial cooperation
Evaluations of cooperation programmes have found that some of the supported measures have increased cooperation between local authorities across borders. They have also helped to involve more locals in the decision-making processes, leading to more inclusive and effective policies. For example, networking and increased institutional cooperation has taken place between Germany and Czechia, and Polish and Czech local authorities have embarked on long-term cooperation.
Member States have also benefited from successful cooperation on thematic issues, including:
·creating cross-border networks for technology transfer and promoting energy efficiency between Germany and the Netherlands;
·producing joint measures to protect biodiversity and manage climate change between France and Italy; and
·promoting cultural heritage in Estonia and Latvia.
In some instances, however, differences in regulatory frameworks and the limited involvement of policymakers restricted the extent of cooperation, highlighting that there is still some room to improve cooperation between the bodies that manage cooperation programmes.
·Horizontal findings on implementation
Although impact evaluations are very important, process and monitoring evaluations carried out during implementation provide operational findings that can directly inform the management of the programme. They can also inform supported projects to make them more efficient and effective.
In fact, around two thirds of the evaluations carried out in the Member States aimed to assess potential weaknesses and success factors of implementation processes, rather than or besides investigating the impact of supported investment. These evaluations provide mixed findings on horizontal aspects, at different stages of implementation, which is important information for the managing authorities.
Many Member States have encountered difficulties in starting the programmes due to:
·delays caused by the overlap with the 2007-2013 period;
·lack of clarity on the applicable legislation caused by an overlap of national and EU policies and complex rules; and
·difficulties in selecting managing authorities and setting up complex projects.
During the implementation phase, burdensome or changing management procedures have caused issues. This has limited the involvement of stakeholders and hindered the achievement of outcomes. For example, some regions of Central and Eastern Europe, Italy and Greece, have encountered such difficulties. In the same countries, however, other regions have improved their management procedures over time to smoothen operations and reduce complexity. This shows that managing authorities aim to reduce the burdens linked to programme implementation where possible.
In some cases, complex administrative procedures have had the negative effect of deterring applications for funding in Lithuania, Romania, Bulgaria and Italy. Other managing authorities, however, have adopted approaches that encouraged applications, for example in a German region and in Interreg programmes involving Poland, France and Switzerland.
Weak governance structures, fragmentation of the managing authority and insufficient cooperation between involved bodies have caused difficulties in regions of Belgium, Bulgaria, Denmark, Poland and Estonia (the latter for ESF interventions). However, in many cases, there was effective cooperation and positive governance. Italy, has strengthened governance procedures over the years, uses common structures for service provision across municipalities, and has benefited from successful public-private cooperation. In Sweden, Poland and Spain too, there were cases of fruitful cooperation among institutions and good management.
In some regions of central and eastern Europe and of Italy and Spain, managing authorities and intermediate bodies sometimes encountered difficulties because the available personnel lacked thematic or technical skills. Difficulties appeared in particular when using simplified cost options, notably in Spain and Poland. However, there has also been effective use of these options, for example in Italy and France.
In some regions of north, central and eastern Europe and in Spain, France and Italy, evaluations have sometimes highlighted inaccurate use of indicators and deficiencies in the monitoring systems. These systems are the primary source of information on implementation progress and data for impact evaluations. However, studies have also found examples of improving monitoring systems in Bulgaria, Poland and Italy, where IT systems supported and strengthened data reporting.
The new regulatory framework for the 2021-2027 programming period addresses some of the findings reported under the various TOs. Notably, sharp simplification measures have been introduced, e.g. deleting the designation procedure applicable to the management and control system or cancelling the obligation to set up a distinct certifying authority. Clarification of rules governing simplified cost options, regarding in particular the scope of audits, is also meant to overcome the observed difficulties. In order to deliver effective outcomes, this simplification will have to be accompanied by efforts to detect and reduce gold-plating practices in the Member States.
1.0.2.Support to the Member States
The Commission supports the Member States to enable them to carry out quality evaluations. This support includes methodological guidance, expert reviews, tailored assistance and discussion fora to exchange experience. The Commission runs the evaluation network (managed by the Directorate-General for Regional and Urban Policy) and the European Social Fund Plus (ESF+) evaluation partnership (managed by the Directorate-General for Employment, Social Affairs and Inclusion), which are the main platforms to exchange experience in monitoring and evaluating cohesion policy. These platforms bring together evaluation experts from the national authorities of the Member States. They meet two to four times per year to share experience and good practice and to strengthen evaluation capacity throughout the EU.
The Commission has continued to run its cohesion policy evaluation helpdesk to strengthen Member States’ evaluation capacity. The experts of the helpdesk collect, categorise, summarise and analyse the evaluations carried out by the Member States. The main findings on impacts are summarised by country and TO. All evaluations together with their summaries in English are uploaded in the evaluation library
. The helpdesk also provides customised support to the Member States on specific evaluation issues. Last year, 12 expert meetings took place providing support to eight Member States. The experts provided feedback and advice on evaluation questions, evaluation scope and choice of methodology. The Member States also received expert feedback on draft terms of reference, evaluation reports and evaluation plans.
In addition, the Commission organises evaluation summer schools for managing authorities and other public bodies involved in the evaluation of ERDF/CF programmes. The schools provide practical training in the areas most requested by the Member States. After the online edition of 2020, the latest summer school took place in a hybrid format in Tallinn, Estonia in October 2021. It focused on the general theme of evaluation design in relation to both the remaining ex-post evaluations for the 2014-2020 period and the evaluation plans that need to be prepared for the 2021-2027 period.
In cooperation with the Joint Research Centre, the Directorate-General for Employment, Social Affairs and Inclusion promotes the use of counterfactual impact evaluations
because these methods provide robust evidence of the effectiveness of the ESF at the micro-level . Joint Research Centre experts have provided customised methodological support to seven managing authorities during the last year. The questions discussed included the choice of supported projects to be evaluated using counterfactual impact evaluations, anonymisation, data linkage, access to data, data needs, evaluation design, selection of control group and other issues. Two community of practice meetings took place where managing authorities, evaluation practitioners and the Commission discussed the challenges of the counterfactual impact evaluations of the ESF.
A good monitoring system is key for quality evaluations. In July 2021, the Commission published a staff working document on Performance, monitoring and evaluation of the European Regional Development Fund, the Cohesion Fund and the Just Transition Fund in 2021-2027. The document illustrates how the new legislative package provides for a systematic approach to performance, based on monitoring and evaluation. It also sets out the context for the use of the common output and result indicators listed in the fund regulations.
To support the Member States in fulfilling the ESF+ monitoring requirements, the Commission has set up the ESF Data Support Centre
(DSC). It provides methodological support and guidance on data collection, storage and processing. The ESF+ common indicators toolbox and issue papers offer guidance and advice on the monitoring of the ESF+. In addition, the DSC provides answers to managing authorities’ questions on monitoring. The questions and answers are publicly available in a searchable database. The DSC also runs the meetings of the ESF+ Data network, during which Member States can exchange experience on ESF+ monitoring.
1.1.Evaluation work by the Commission
The Commission is completing i its ex-post evaluation of investment projects in infrastructures, competence centres and activities for research and technological development financed by the ERDF in the 2007-2013 period. Starting from the collection of detailed data on projects and beneficiaries
, the analysis has led to the production of a manual in the form of a cookbook. It details the underlying mechanisms of the main work funded by the policy and summarises the key ingredients for an effective policy. The Commission will publish the evaluation report in 2022.
At the end of 2020, the Commission launched the evaluation of e-Cohesion, the electronic data exchange system to make it easier to exchange documents and data between ESI Funds beneficiaries and the relevant authorities. This evaluation is progressing on schedule. The main data collection is complete and the analysis phase has begun. The Commission will publish the evaluation report in 2022.
In September 2021, the Commission launched a call for a study supporting the preliminary evaluation of the support from the ESF and the Fund for European Aid to the Most Deprived under the Coronavirus Response Investment Initiative and the Coronavirus Response Investment Initiative Plus.
On 16-17 September 2021, the Commission organised the ninth Conference on the Evaluation of EU Cohesion Policy in Porto, Portugal. More than 100 participants attended the conference in person and 400 more participated online in the first event organised by the Commission in hybrid format, one and a half years after the start of the COVID-19 pandemic. It was an occasion to discuss evaluation issues specific to the headline initiatives of the Commission and to disseminate the evaluation culture further among local stakeholders. The title of the conference, Shaping Transition with Evidence, referred to the guiding role that evaluation must play in shaping and informing the green and digital transitions under cohesion policy. However, although evaluation can support policy learning with findings on what works and the underlying mechanisms, it is the responsibility of the policymakers to set out the desired achievements and, most importantly, what can be considered as a success.
During the conference, the Commission presented its approach to the ex-post evaluation of the 2014-2020 period. The structure of the evaluation of ERDF/CF investment will consist of cross-cutting and thematic work packages covering the various sectoral domains supported by cohesion policy. The preparatory study on monitoring systems and data started in 2020. It is part of the cross-cutting work packages and will provide valuable data at the level of projects and beneficiaries to organise the thematic work packages.
The Commission will launch its ex-post evaluation of ESF investment in 2022. The evaluation will build on the findings of the four thematic evaluations published in 2020: (i) youth employment; (ii) employment and labour mobility; (iii) social inclusion; and (iv) education and training.
3.EUROPEAN AGRICULTURAL FUND FOR RURAL DEVELOPMENT (EAFRD)
2.
2.0.Evaluations in the Member States
2.0.0.General overview
The Member States provided very comprehensive information on the progress in implementing the evaluation plans (i.e. annual implementation report (AIR) Chapter 2). However, the 115 AIRs analysed in 2021 were slightly less comprehensive, suggesting that overall Member States have slightly reduced their evaluation activities.
According to the AIRs submitted in 2021, modifications to the evaluation plans for the rural development programmes (RDPs) have decreased significantly by 42% compared to 2020. Overall, the 12 AIRs that filled this sub-section in 2021 have reported 15 modifications. These mainly concerned the update of the planned evaluation timeline and of the evaluation work.
The number of evaluations (396 reported by 98 AIRs) has decreased by 21% compared to the previous year (499). This can be explained by the fact that all types of reported evaluation activities had a peak during the reporting years with major evaluation milestones (2017 and 2019) and decreased since then in line with the evaluation cycle. It can be expected that evaluation activity will increase again because of the ex-post evaluations in 2026. The Member States have carried out new evaluation work to prepare the new programming period (ex-ante evaluation for the strategic plan of the common agricultural policy (CAP)). Overall, the evaluations show good progress and concern a wide variety of topics (including the COVID‑19 pandemic). The most common topics of the reported evaluations are RDP priority 4 (restoring, preserving and enhancing ecosystems related to agriculture and forestry) and the assessment of community-led local development, LEADER and local action groups. There were also many multi-thematic evaluations.
Figure 5: Distribution of reported evaluations (N=396) in AIRs submitted in 2021
Source: European Evaluation Helpdesk for Rural Development (2021).
The 2021 AIRs have reported less data-related evaluations. The overall number of evaluations on data provision and management (312) have dropped by 13% compared to 2020. Most concerned the preparing and conducting phase. Data collection work (which is part of the conducting phase), however, has increased. Also, work to improve the IT system has increased, evidently to prepare for the ex-post evaluations and the stricter rules for performance monitoring in the next programming period.
The number of completed evaluations has dropped by 24% compared to the 2020 AIRs. In 2021, 85 AIRs reported 217 completed evaluations. As in the previous year, most related to multiple RDP priorities and objectives, and to CAP objective 2 (ensuring sustainable management of natural resources and climate action).
Figure 6: Completed evaluations across main topics (N=217) reported in AIRs in 2021
Source: European Evaluation Helpdesk for Rural Development (2021).
The main types of evaluations reported in 2021 were evaluations of achievements of results (36%), followed by impact evaluations (18%), out of which the number of evaluations with a clear counterfactual approach was rather modest in this reporting year (3% of total evaluations). This was not related to any major evaluation milestone. Furthermore, the AIRs contained process evaluations of the implementation system (7%) and research studies other than evaluations (e.g. monitoring studies) (11%). Various reported evaluations related to the ex-ante evaluations of the CAP strategic plans.
The 2021 AIRs have reported a substantial number of communication activities on the dissemination of evaluation findings (303). However, this is 68 activities less than reported in 2020. Altogether, these activities have reached 3.8 million stakeholders, which is less than in 2020 (5 million). Websites were most effective in reaching stakeholders (3.2 million, representing a large majority of 84%), followed by external publications or evaluation reports (0.6 million; 15%). Other communication channels, i.e. newsletters, social media and internal or external meetings (e.g. workshops, seminars, etc.), reached a smaller share of stakeholders.
About half of the communication activities were targeted either to the general audience (31%) or the RDP monitoring committee (21%). Other target groups included researchers and thematic experts, local action groups, national/regional authorities, evaluators, RDP beneficiaries and national rural networks.
Figure 7: Number of communication activities by type and stakeholders reported in the AIRs in 2021
Source: European Evaluation Helpdesk for Rural Development (2021).
The 2021 AIRs have reported much fewer follow-ups to evaluation results than in 2020, when exceptionally high numbers were observed. Evidently, the lower number of completed evaluations and the absence of a major evaluation milestone can explain this trend. In more than half of the cases, the follow-ups concerned the:
·improvement of the RDP delivery mechanism and implementation (52%);
·adaptation of the RDP monitoring and evaluation system (20%); and
·the preparation of the CAP strategic plan for the new period (11%).
To a lesser degree, reported follow-ups concerned the improvement of the RDP intervention logic (9%) and other topics (8%).
2.0.1.Main findings of evaluations
In total, 76 AIRs have reported 198 evaluation findings. The distribution of their topics is similar to that of the completed evaluations: findings for CAP objective 2 (ensuring the sustainable management of natural resources, and climate action) prevail with 38%. This is followed by findings covering multiple RDP priorities and objectives (15%), CAP objective 1 (fostering the competitiveness of agriculture) (13%) and CAP objective 3 (achieving a balanced territorial development of rural economies and communities) (8%). Other aspects (CAP strategic plan, governance, monitoring and evaluation system, technical assistance and national rural networks) are less represented in the reported evaluation findings.
Most of the findings are based on an assessment of gross results, gross impacts and input/output achievements of RDPs rather than of net impacts. Nearly 60% of the reported findings derive from the assessment of impacts and results, whereas 13% of the evaluation findings are mainly based on the analysis of inputs and outputs, which is considered a less robust approach. A significant share of the findings (10%) is based on an analysis of contextual trends at the macro level (i.e. no impact of the RDP was assessed) and an analysis of other issues (e.g. administrative obstacles). Therefore, the reported findings are not only based on RDP results and impacts, but also relate to a much broader range of evaluation and research outcomes.
Figure 8: Evidence bases of evaluation findings reported in the AIRs in 2021
Source: European Evaluation Helpdesk for Rural Development (2021).
With regard to the direction of the effects of the reported findings, a large number of positive effects (45%) and mixed effects (21%) were observed. Negative and zero effects concerned only 7% of the reported findings. However, for a large share of the findings (27%), it was impossible to determine the direction of the effect. This indicates that evaluations at this stage cannot provide clear findings on achievements in terms of positive/negative results or impacts. Such findings are, however, expected for the ex-post evaluation.
Figure 9: Direction of effects in reported evaluation findings (N=198) in AIRs in 2021
Source: European Evaluation Helpdesk for Rural Development (2021).
A further breakdown of the findings by CAP objective and priority shows the below picture.
CAP objective 1: fostering the competitiveness of agriculture
In this thematic area, 17 RDPs have reported 26 findings, of which around 62% are positive. However, there are also mixed (15%), unspecified (19%) and zero effects (4%).
On farm viability and competitiveness (RDP priority 2), mainly positive (gross) results are reported, thanks to investment in agricultural holdings (e.g. to optimise cultivation techniques and machinery). Also with regard to CAP impacts on the competitiveness of agricultural sectors, there are positive findings. These include improved participation of producers in the food chain, more added value in each food sub-sector, etc. On food chain organisation, animal welfare and risk management (RDP priority 3), measures promoting the processing and marketing of agricultural products show positive effects. However, these are limited by high deadweight losses (zero effects), i.e. projects would have been carried out even without promotion measures.
CAP objective 2: ensuring the sustainable management of natural resources and climate actions
Out of 75 evaluation findings reported under CAP objective 2, 53% findings from 29 RDPs are clearly positive, followed by unspecified (19%), mixed (16%), negative (9%) and zero effects (3%). Positive findings include better water and water management with reduced leaching of fertilisers and nutrients, reduced greenhouse gas emissions and protected carbon storage in soils.
Negative effects are reported mainly in the area of ecosystems related to agriculture and forestry (RDP priority 4) and also include an overall decline in bird species. Around 40% of the findings are clearly positive (e.g. better soil characteristics, less erosion, etc.).
On resource efficiency, low-carbon and climate-resilient economy (RDP priority 5) and the cross-cutting objectives environment and climate, the results are clearly more positive and measurable.
CAP objective 3: achieving a balanced territorial development
In this thematic area, 14 RDPs have reported 16 findings, of which 44% are positive. However, there are also mixed (12%) and unspecified effects (44%). Most of the findings relate to community-led local development (CLLD), LEADER or local actions groups and less to other priority 6 topics such as diversification, job creation or ICT in rural areas. There are only few assessments of the effects of LEADER, 40% of which show a positive direction demonstrating that CLLD projects create more added value than structural changes (entrepreneurship, jobs, etc.). The effective implementation of CLLD/LEADER still seems to be a major challenge. However, in other sections of the AIR, it was reported that the RDPs are still preparing a number of CLLD/LEADER evaluations. These may provide insights into the success of LEADER soon. Business investment under priority 6 has positive effects, in particular in terms of productivity and turnover.
Horizontal priority: knowledge transfer and innovation, European Innovation Partnership on Agricultural Productivity and Sustainability (EIP-AGRI)
In this thematic area, 8 RDPs have reported 8 evaluation findings, of which 75% are positive at all levels (input/output, results and impacts). They particularly relate to EIP-AGRI and innovation support but less to knowledge transfer. The predominantly positive results (such as improved interaction between companies and stakeholders in the agri-food value chain, better cooperation, increased knowledge, etc.) might not be representative for the total population of RDPs due the small number of reported findings. 13% of the findings show zero effects and 12% are unspecified.
National rural networks (other RDP aspects)
In this thematic area, a small sample of 5 RDPs have reported 7 findings, of which around 57% are positive (e.g. in identifying good practices, stakeholder participation or consolidation of thematic networks), but there are also mixed (14%) and unspecified effects (29%), e.g. limited effectiveness in engaging with other RDP schemes.
2.0.2.Support to the Member States
In 2021, the European Network for Rural Development Evaluation Helpdesk continued to assist Member States in their efforts to improve their evaluation capacities. Examples of specific activities included:
·organising yearly capacity building events for Member States (on evaluation, better data to evaluate the CAP, etc.);
·organising good practice workshops to discuss lessons-learned and experiences on selected topics;
·publishing guidance documents and tools; and
·collecting good practice and publishing it on webpages, in newsletters and in factsheets of the European Network for Rural Development Evaluation Helpdesk (e.g. on RDPs’ impact on the gross nutrient balance, etc.).
2.1.Evaluation work by the Commission
The Commission finances external studies that support specific thematic evaluations of the effects of RDPs. These assess the effectiveness, efficiency, relevance, consistency and EU added value of all relevant CAP instruments. Since March of 2021, the below evaluations have taken place.
·On 1 June 2021, the Commission published a staff working document on the CAP’s impact on climate change and greenhouse gas emissions was published. The evaluation concluded that although the climate footprint per unit of agricultural output continued to improve, EU greenhouse gas emissions from agriculture have stagnated since 2010. The CAP has helped mitigate climate change mainly by protecting existing carbon stocks, notably thanks to the maintenance of permanent grasslands supported by extensive livestock grazing systems. However, there is potential to increase carbon storage in EU soils further. Climate adaptation is mainly achieved by supporting diversity of crops and farming systems, supporting investment in adaptation to new climate conditions, limiting soil erosion and improving resilience to floods. However, better targeting of CAP support would increase efficiency. Rural development supports emission reduction in the livestock sector using various measures, such as certain agri-environmental commitments with respect to feed or manure management or investment in physical assets relating to manure storage, animal housing or installation of anaerobic digesters.
·On 12 May 2021, the Commission published a staff working document on the impact of the CAP measures on the general objective of viable food production. The evaluation concluded that direct income support played an important part in stabilising farm income, with direct payments and support to areas facing natural constraints playing a particularly important role. Market measures, EU quality schemes and various rural development measures enabling productivity growth also supported farm income by limiting price reductions and improving competitiveness. However, despite a steady and continuous improvement, the income gap between agricultural and non-agricultural income remains considerable.
·On 8 April 2021, the Commission published a staff working document on the impact of the CAP on generational renewal, local development and jobs in rural areas. The evaluation found that the CAP measures to support generational renewal helped increase the number of young farmers, although to a limited extent. This positive impact also depends on other factors. For instance, socio-cultural and wider economic incentives play a major role in deciding to get into farming or living in rural areas. In addition to support under other EU policies, such as regional and cohesion policy support to rural economic diversification, better services and infrastructure (including broadband), rural development support was vital to improve the broader economic climate, particularly in rural areas. Member States could improve the delivery and impact of CAP generational renewal measures by developing integrated approaches, consistently using various CAP and non-CAP instruments and broader legislative and fiscal provisions. The CAP on its own cannot remove the main entry barriers to farming, such as access to land and access to capital.
·An evaluation study on the impact of LEADER on balanced territorial development is due to be completed by the end of 2021. It takes a broader analytical perspective assessing not only the effects of local strategies on local development, but also the LEADER added value. To this end, the evaluation study factors in the principles that define the LEADER approach and make it distinct in terms of programme implementation (the ‘seven features’ of LEADER). The evaluation examines the relevance, effectiveness, efficiency, consistency and EU added value of LEADER. The study also assesses LEADER’s consistency with other EU policies, particularly CLLD under European Structural and Investment Funds (multi-funding CLLD), and other national and regional policies. The Commission will publish the evaluation report in 2022.
·Additional activities should be concluded before the end of 2021, with the publication of staff working documents on:
othe impact of the CAP on biodiversity, soil and water (natural resources);
othe CAP’s impact on territorial development of rural areas (socio‑economic aspects);
othe CAP’s impact on knowledge exchange and advisory activities.
oReport on the implementation of the common monitoring and evaluation framework including an assessment of the performance of the common agricultural policy 2014-2020.
4.EUROPEAN MARITIME AND FISHERIES FUND (EMFF)
3.
3.0.Evaluations in the Member States
By the end of 2020, programmes had committed EUR 4.1 billion of EMFF funding to operations in the Member States. This corresponds to 72% of the total EMFF budget of EUR 5.7 billion available under shared management. Since the beginning of the programming period, beneficiaries have declared EUR 2.4 billion of expenditure (41% of total EMFF funding).
During this time, Member States carried out a total of 54 evaluations on the 27 programmes, with EUR 5.1 million committed to support this work. The Commission has boosted Member States’ ability to evaluate EMFF programmes by providing practical support to managing authorities. This has helped to strengthen evaluations and to extend their scope.
The level of detail of the information varies significantly across Member States and correlates with the scope and depth of the evaluation itself. Several Member States’ EMFF budgets are rather small, which limits the scope of evaluations.
The number and type of evaluations vary significantly across Member States. Process evaluations are the most frequent type. Effectiveness and impact evaluations cover a wide range of issues. They assess effectiveness and efficiency at the operational programme level and in relation to EU priorities, specific objectives and measures (for example community‑led local development and permanent cessation).
The recommendations of the evaluations also cover a wide range of subjects. Many evaluations concluded that re-allocating financial resources is pivotal to successful EMFF implementation. Another recommendation was to reduce administrative complexity further and to digitise EMFF implementation as much as possible. For one Member State where EMFF implementation takes place at the regional level, the evaluation recommended to reduce the number of intermediate bodies and to improve communication between managing authorities, intermediate bodies and beneficiaries. Recommendations also encouraged additional networking of producers in producer organisations and the creation of new producer organisations and interbranch organisations. One evaluation underlined the need to focus on innovation in fisheries and aquaculture. Several overarching issues were also covered:
·promotion of cross-cutting principles, such as green public procurement;
·introduction of financial instruments to increase final recipients’ operations by combining grants with financial instruments; and
·use of simplified cost options.
Most commonly, Member States use the results of evaluations to inform amendments of national operational programmes and reallocations of financial resources within EU priorities. Several Member States mentioned that EMFF evaluations directly facilitate preparations for European Maritime, Fisheries and Aquaculture Fund programming. In addition, several Member States have carried out assessments of the use of financial instruments and environmental impact assessments to prepare the next programming period.
Specific evaluations include:
-aquaculture providing environmental services;
-multi-fund approach in fisheries local action group funding;
-impact of the COVID-19 pandemic on the fisheries sector;
-review of the fisheries local action group programme;
-cost-benefit analysis of the proposed decommissioning scheme;
-ex-ante assessment of the use of financial instruments;
-evaluation of the lobster V-notching scheme;
-evaluation of the sustainable fisheries scheme;
-assessment of the role of fisheries in the circular economy and the bio-economy;
-the effects of support for selective gear and predator proof gear;
-evaluation of the environmental benefits from the EMFF.
Section 7 of the EMFF implementation reports for 2019 (published in December 2020
) and 2020 (published in September 2021
) contains detailed summaries of each Member State’s evaluations based on information from the annual implementation reports.
3.1.Evaluation work by the Commission
The Commission has carried out a mid-term evaluation of the direct management portion of the EMFF budget under direct management. It published the resulting staff working document (SWD(2020) 221 final) in 2020. The evaluation found that the investment had provided value for money and recommended a framework of indicators compatible with shared management in the future. The European Maritime, Fisheries and Aquaculture Fund for 2021–2027 incorporates this framework. Preparatory work for the ex-post evaluation of the EMFF is underway.
5.FINAL REMARKS
The evidence collected via the evaluations carried out in the Member States has kept growing in the past year, whereas the number of evaluations has overall stabilised. This proves the extraordinary work of the managing authorities to dedicate resources to evaluations during the public health crisis. This came at the same time as the intense reprogramming of the resources under the Coronavirus Response Investment Initiative (Plus) and REACT-EU. Moreover, it shows how managing and coordination authorities are using evaluation as a working tool to ensure a result-oriented policy, and not merely to comply with regulatory requirements.
This shows that the evaluation culture in the Member States has improved, also thanks to the Commission’s commitment to providing dedicated services for methodological support and exchange platforms on evaluations issues.
Impact evaluations, which account for around one third of the total evaluations, provide useful insights for policy learning in the specific context of the supported projects. Although some common findings are starting to emerge from evaluations carried out in the Member States, a broader assessment of EU-wide results will be available with the Commission’s ex-post evaluation, whose preparation has already started. Most evaluations provide further useful information, including those that are more focused on implementation. The findings of these evaluations support the managing authorities in adjusting projects to maximise their benefits.
Finally, the COVID-19 pandemic has impacted many programmes. First the reallocations under the Coronavirus Response Investment Initiative (Plus) and then the programming of fresh REACT-EU resources have reshaped much of the 2014-2020 programmes’ work and targets. Still, only a very limited number of evaluations undertaken in the Member States cover this impact already. This number is expected to grow in the coming years and future analyses will factor in the results of these evaluations.
