Table of contents

1.Introduction

Purpose and scope of the evaluation    

Methodological approach    

2.What was the expected outcome of the intervention?

2.1    Description of the intervention and its objectives    

2.2 Point of comparison    

3.How has the situation evolved over the evaluation period?

4.Evaluation findings

4.1.To what extent was the intervention successful and why?

Effectiveness    

Efficiency    

Coherence    

User-friendliness    

4.2.How did the EU intervention make a difference?

EU added value    

4.3.Is the intervention still relevant?

Relevance    

5.What are the conclusions and lessons learned?

5.1 Conclusions and lessons learned    

5.2. Good practices for improving the functioning of e-Cohesion systems    

5.3 Implementation of e-Cohesion in the 2021-2027 programming period    

Annex I: Procedural Information

1.    Lead DG and Decide reference    

2.    Derogations granted    

3.    Organisation and timing    

4.    Evidence and sources    

5.    External expertise    

Annex II. Methodology and Analytical models used

Desk research    

Large-scale surveys    

Case studies    

Webinar with the representatives of authorities    

Annex III. Evaluation matrix and, where relevant, Details on answers to the evaluation questions (by criterion)

Annex IV. Overview of benefits and costs and table on simplification and burden reduction

Annex V. Stakeholders consultation - Synopsis report

1.    Strategy    

Objectives    

Tools    

Stakeholder participation    

2.    Delivery and results    

Roadmap    

Large-scale survey    

Interviews    

Webinar    

Glossary

Purpose and scope of the evaluation

“e-Cohesion” refers to electronic data exchange systems between beneficiaries and programme authorities in relation to cohesion policy funds. The Common Provisions Regulation 1 (‘the CPR’) governing the 2014-2020 programming period of the European Structural and Investment (ESI) Funds introduced, for the first time, requirements designed to simplify and streamline the implementation of cohesion policy programmes by facilitating their interoperability with national and EU frameworks for the exchange of information. The requirements were also intended to lower the administrative burden on beneficiaries by enabling them to submit information on programme implementation automatically. e-Cohesion systems were recommended for progress reporting, submitting and managing payment claims and exchanging information relating to management verifications and audits.

The evaluation of the introduction of e-Cohesion in the 2014-2020 period sought to:

·collect and provide up-to-date information on the implementation and performance of e-Cohesion systems in the Member States during the 2014‑2020 programming period;

·identify both good practices and challenges that would be useful for those Member States who have yet to implement e-Cohesion systems or plan to improve their existing systems;

·assess the extent to which the e-Cohesion requirement has delivered the intended policy results;

·analyse the options for increasing the effectiveness, efficiency, coherence and user-friendliness of the e-Cohesion systems during the 2021-2027 programming period.

In the 2014-2020 period, Member States were required to ensure that, by 31 December 2015, all exchanges of information between beneficiaries and programme authorities (managing authorities (MAs), intermediate bodies (IBs), certifying authorities (CAs) and audit authorities (AAs)) could be carried out through electronic data exchange systems. However, beneficiaries still had the option to exchange information with programme authorities in other ways (such as paper-based communication).

The CPR referred to two types of electronic systems: electronic data exchange systems, i.e. e‑Cohesion systems (Article 122(3)) and the information monitoring systems for recording and storing data (Article 125(2)(d)). The first is an information exchange interface between programme authorities and beneficiaries. The second is an internal system for programme authorities used for monitoring, evaluation, financial management, verification and audit activities. In practice, the two systems can be integrated into a single functioning module or they can interact as two distinct and complementary modules. The current evaluation focused solely on e-Cohesion systems.

The evaluation of e-Cohesion was carried out between November 2020 and May 2022 and covered the ERDF and CF operational programmes (OPs) from the 2014-2020 programming period 2 , including Interreg programmes. The evaluation thus examined 302 programmes funded by ERDF and CF in the 2014-2020 programming period, for which 108 e-Cohesion systems were mapped and analysed.

The new legislative framework governing the 2021-2027 programming period has retained and strengthened the requirements on e-Cohesion, by rendering the exchange of information outside e-Cohesion systems an exception, at the explicit request of beneficiaries.

Methodological approach

The evaluation followed the approach set out in the Better Regulation framework. The analysis was organised around the five evaluation criteria included in the regulation – effectiveness, efficiency, relevance, coherence, and EU added value – plus a sixth, namely the user‑friendliness of the IT systems.

The evaluation methods included 3 :

·desk research/documentary research; and

·field research based on:

olarge-scale surveys of all types of e-Cohesion system users (beneficiaries, managing authorities, intermediate bodies, certifying authorities, and audit authorities);

ocase studies and interviews with stakeholders; and

oa webinar with the representatives of the authorities.

The desk research consisted of online searches (using the websites of e-Cohesion systems and the websites of the programme authorities) and a review of numerous documents available in the System for Fund Management in the European Union (SFC), online or shared by stakeholders (e.g. user manuals or presentations for users; guidelines; IT system descriptions; relevant national evaluations and reports).

The organisation of a large-scale survey required significant effort given that hundreds of thousands of projects and beneficiaries were supported via ERDF and CF in the 2014‑2020 programming period across all Member States. In addition, the number of users of e-Cohesion systems varies significantly between Member States, given the difference in size of Member States, the different number of programmes and the total funding allocated. Moreover, the ERDF and CF support is implemented differently in each Member State: some countries implement programmes in a centralised way, others use a decentralised approach. Therefore, instead of constructing a representative sample of users of e-Cohesion, the surveys were opened up to all users. This approach resulted in the largest possible number of survey addressees and provided a rich body of evidence for the evaluation: 6 248 beneficiaries and 455 representatives of programme authorities from all 27 Member States replied to the surveys. Although the statistical significance could not be verified (as the total population of users of e-Cohesion remains unknown), the findings of the survey have been cross-checked against other data sources, such as desk research findings and case studies, in order to draw reliable conclusions.

The surveys were complemented by a webinar with the Member State authorities (managing authorities, intermediate bodies, certifying authorities and audit authorities), which enabled the contractor to follow up on certain survey questions and collect further opinions. The webinar took place on 23 November 2021 and was attended by 113 participants from 25 Member States.

To collect more detailed information on best practice and the challenges of introducing e‑Cohesion systems, the evaluators also carried out in-depth analysis of six e-Cohesion systems. 84 interviews with stakeholders were conducted as part of the field research for the case studies.

2.1    Description of the intervention and its objectives

The objective of e-Cohesion was to support the reduction of administrative burden for beneficiaries and costs for authorities, as an important part of the cohesion policy simplification effort.

During the lifetime of a project or operation supported by cohesion policy, beneficiaries and programme authorities must exchange information at certain moments during implementation. In the absence of a dedicated electronic data exchange system, these exchanges were carried out based on document sharing via emails or other means. If a document was required by more than one authority, the beneficiary had to submit it separately to each of them. The paper-based exchange of information was time-consuming and placed a significant burden on beneficiaries, in particular smaller ones with limited administrative capability.

The e-Cohesion systems were introduced to facilitate interoperability with national and EU frameworks and enable beneficiaries to submit the relevant information only once. These are the two key principles of e-Cohesion.

The provision envisaged a scenario in which, once project implementation starts, all exchanges of documents between the beneficiaries and the programme authorities take place electronically on a dedicated platform; the latter can store and make available the documents already submitted for other purposes (audits, verifications), without asking the beneficiary to resubmit them. In those cases where interoperability with other national databases (e.g. business or tax databases) is in place, the information allowing the identification of the beneficiary can also be retrieved from a common database. The administrative burden on the beneficiary is further reduced thanks to the interactive forms and system alerts, which guide them through the various steps of the implementation process. The beneficiary also takes advantage of the automatic calculations and embedded controls to reduce the risk of errors. The progress reports and payment claims are also processed electronically via the system, while the security and integrity of the data are ensured. Finally, interoperability with EC systems such as SFC 4 would further ensure the automatic extraction, aggregation and submission of the programme data to the EC.

To take advantage of the digitalisation of the exchange of information, additional simplification measures related to e-Cohesion systems were introduced at European level under Commission Implementing Regulation (EU) No 1011/2014 (‘the implementing Regulation’) 5 . The implementing Regulation detailed key processes, functionalities, and data security requirements that complement the key processes laid down in the CPR. These requirements are described in more detail in Table 1.

Table 1: The principles, key processes, functionalities, and data security requirements of e-Cohesion as listed in Commission Implementing Regulation (EU) No 1011/2014

Source: Evaluation of e-Cohesion 2014-2020, final report, 2022

The implementing Regulation clearly requires that, as a minimum, e-Cohesion systems are used for progress reporting, submitting payment claims and the exchange of information related to management verifications and audits between beneficiaries and programme authorities. However, there are earlier stages in the project lifecycle, such as the project application, selection and contracting, for which the legislation does not require the use of e-Cohesion systems. Thus, the programme authorities were able to choose whether or not to use e-Cohesion systems for the stages preceding project implementation.

Based on the legal acts described above, the objectives of the introduction of e-Cohesion systems can be summarised as follows:

·reduce multiple transmissions of the same information;

·improve data security (confidentiality, integrity, availability, non-repudiation);

·ensure data privacy;

·improve communication;

·improve transparency and accessibility of relevant information; and

·improve data quality (fewer calculation errors, fewer missing values, correct format, fewer other inconsistencies, high up-to-datedness).

Based on these objectives, the intervention logic of the e-Cohesion initiative is presented in Figure 1.

Figure 1: The intervention logic of e-Cohesion

Source: Evaluation of e-Cohesion 2014-2020, final report, (2022).

2.2 Point of comparison

The Impact Assessment (IA) accompanying the proposal for the CPR 2014-2020 6 analysed, among other aspects, the issues related to the delivery of the cohesion policy. It noted that, despite the progress in IT systems throughout the EU, the communication between beneficiaries and programme authorities was still largely paper-based. It also mentioned that for the delivery of cohesion policy investments, the heaviest costs for beneficiaries are linked to the processes of applying for funding, reporting and storing documents. Considerable costs were associated with the transcription and aggregation of financial and monitoring data on paper, while control costs were deemed to be higher than necessary, as supporting documents were not always easily available and accessible.

As such, one of the avenues for reducing the administrative burden for the beneficiaries was to do away with the need to copy, submit or resubmit and retain large volumes of documents on paper.

In this respect, the IA assessed several options for the introduction of a legal provision on the use of electronic exchange of data between the beneficiaries and programme authorities and retained the proportional approach (oblige the managing authorities to make available electronic data exchange systems and leave the option to beneficiaries to make use of them or choose other data exchange channels).

Consequently, making electronic data exchange systems available between the beneficiaries of cohesion policy support and the programme authorities was one of the responsibilities of the Member States under Article 122(3) of the CPR 2014-2020. This was the first time such a requirement was included in the legislation governing cohesion policy. The deadline provided by the legislation for doing so was 31 December 2015. Subsequently, the e-Cohesion requirement was retained in Article 69(8) of the CPR 2021-2027.

Having analysed the expected benefits of introducing e-Cohesion, the evaluation study provides evidence on how the benefits of e-Cohesion compared to its costs and whether it resulted in simplification and a reduction in the administrative burden.

One of the key outputs of the evaluation study, completed at the beginning of 2022, was the mapping of e-Cohesion systems.

108 e-Cohesion systems were identified for the 2014-2020 ERDF and CF operational programmes, in all Member States. The breakdown of these systems by Member States sheds light on the approach taken on the implementation of ERDF and CF, in a centralised or decentralised manner.

In most Member States (18) there is one single e-Cohesion system used by all operational programmes. The highest number of systems identified in a Member State was 32 (Italy), while in the remaining Member States the number of systems varied between 2 and 15. Figure 2 shows the overall mapping of e-Cohesion systems.

Figure 2: The number of e-Cohesion systems in each Member States, plus Interreg

Source: Evaluation of e-Cohesion 2014-2020, final report, (2022).

Most of the systems identified are used for either operational or for cooperation programmes. A minority (11 systems) are used for both types of programmes.

Table 2: Breakdown of e-Cohesion systems by type

Source: Evaluation of e-Cohesion 2014-2020, final report, (2022).

The evaluation study also revealed the coverage of programmes by e-Cohesion systems. Most systems (77) cover only one programme, while a few of them are used by more than 20 programmes. This is the case for the following e-Cohesion systems: Synergie (used in France, covering 37 programmes), eMS (used by and covering 36 Interreg programmes), SL2014 (used in Poland, covering 25 programmes), MIS (used in Greece, covering 22 programmes), Fondos2020 (used in Spain, covering 21 programmes).

33 programmes (from Greece, Spain, Italy, Austria and Germany) used several e-Cohesion systems. This was the case especially when different intermediate bodies were involved in the management of the programme, each with its own specific projects which required tailored e-Cohesion systems.

Despite an increase in the number of e-Cohesion systems, from 12 before 2013 to 108 in the 2014-2020 programming period, there are still a few programmes that do not have an e-Cohesion system: two in Germany, one each in Finland and France, and four Interreg ENI CBC programmes (out of 15 ENI CBC programmes). However, for the ENI CBC programmes there was no obligation under the 2014-2020 legal framework to set up e‑Cohesion systems. This changed in the 2021-2027 programming period. It is interesting to note that, even in the absence of a legal requirement, most of the ENI CBC programmes have already started using e-Cohesion systems in the 2014-2020 programming period.

This section presents the results of the evaluation study in relation to the evaluation criteria. It is divided into three parts: the first part offers insight into whether the intervention was successful and the reasons behind this. It covers the effectiveness, efficiency, coherence, as well as the user-friendliness evaluation criteria. The second part discusses whether the EU intervention made a difference (the EU added value criterion), while the last part examines the relevance criterion.

4.1.To what extent was the intervention successful and why?

Effectiveness

Under the effectiveness criterion, the evaluation aimed to establish:

-whether the introduction of e-Cohesion has resulted in simplification and a reduction in the administrative burden; and

-the extent to which the systems in place in the Member States comply with the requirements set out in the legislation in terms of the key principles and processes covered.

Simplification and reduction of administrative burden

The reason why the introduction of e-Cohesion systems was mandated by the CPR 2014-2020 was to simplify a critical aspect of the delivery of the cohesion policy, reducing the administrative burden related to the exchange of information between the beneficiaries and the programme authorities.

The results from the field research show a positive response from both beneficiaries and programme authorities on the reduction of administrative burden brought about by the use of e-Cohesion systems. The answer to this question is positive, as the surveys show that close to 90% of both the beneficiaries and the authorities surveyed replied that the introduction of e-Cohesion simplified the exchange of information and resulted in a reduction in the administrative burden.

Figure 3: the perception of e-Cohesion users on the impacts on simplification

Source: Evaluation of e-Cohesion 2014-2020, final report, (2022).

The high percentage of positive answers (‘strongly agree’ and ‘agree’), and the fact that they represent broadly similar proportions of the replies of both beneficiaries and programme authorities, supports the conclusion that the introduction of e-Cohesion has been successful in delivering simplification. The same conclusion emerged from the discussions in the webinar with programme authorities. Therefore, considering the assessment of the stakeholders’ responses, the introduction of e-Cohesion contributed to the simplification and a reduction in the administrative burden.

Compliance with the legal requirements

Another aspect analysed by the evaluation report under the effectiveness criterion was the extent to which the e-Cohesion systems in place in the EU include the required key principles, processes, and functionalities. The surveys and desk research carried out in the context of the evaluation study provide a more nuanced picture regarding the implementation and compliance with the legal requirements.

The two key principles to be embedded into e-Cohesion systems are: (1) interoperability and (2) ‘once-only’ encoding principle.

The principle of interoperability states that ‘all the bodies involved in the implementation of a programme should work together at the organisational and technical levels in ensuring effective communication between computerised systems, as well as the exchange and re-use of information and knowledge’ 7 . In relation to this principle, the legislation provides that e-Cohesion systems should facilitate interoperability with national and EU frameworks. Without obliging programmes to introduce e-Cohesion systems, the legislation thus encouraged their development with a view to facilitating the reuse of information across different existing databases at national and at European level.

The evaluation looked at the interoperability principle from two perspectives: between e‑Cohesion systems and registers/databases at national level and between e-Cohesion systems and management/monitoring systems at European level.

The management and monitoring system of ERDF and CF at European level for the 2014-2020 programming period was SFC2014. Its main function was the electronic exchange of information concerning shared fund management between Member States and the European Commission.

Looking at the national level, the desk research found that e-Cohesion systems are most commonly linked to the central information monitoring systems of EU funds (67 systems), and secondly to other national registers and databases (31 systems, of which 27 were also linked to the central information monitoring systems).

As regards the interoperability of e-Cohesion systems with European databases and monitoring systems, 24 systems were identified as having a direct interface with SFC2014 (with one exception, all were also linked to the central information monitoring systems of the Member State or region in question) 8 .

By design, SFC is expected to be connected to central monitoring systems for programme implementation (which stores the information to be sent to the EC from programme level reporting purposes), rather than directly to the e-Cohesion systems themselves. In fact, the e-Cohesion systems have information at individual beneficiary level, while the information sent to SFC by programme authorities is aggregated at programme level. This aggregation is normally carried out by central monitoring systems.

The desk analysis concluded that more than 65% of the e-Cohesion systems (72 systems) were interoperable with other databases at national or European level in the 2014-2020 period. For 30% of the systems (30 systems) this information could not be verified using documentary sources or via other interactions with stakeholders and therefore their interoperability status is unknown. Three systems were found to have no interoperable modules with other external databases.

The once-only encoding principle states that the ‘submission of documents and data through the electronic data exchange systems shall be made only once (…) as regards the same operation for all authorities implementing the same programme’ 9 . The evaluation found that, overall, the e‑Cohesion systems comply with the once-only encoding principle. Of the 108 systems identified, only one was non-compliant, while for 26 systems the implementation of this principle could not be established with any certainty on the basis of the available documentation. When this principle is analysed on the basis of whether the systems can reuse information provided by the beneficiaries during the earlier stages of project implementation or even during the project application stage, the survey shows that approximately 85% of beneficiaries and 90% of authorities gave a positive reply. It can therefore be concluded that the once-only encoding principle was successfully integrated into most of the existing e-Cohesion systems.

The key processes to be covered by e-Cohesion systems are: (1) reporting on progress, (2) payment claims and (3) exchange of information related to management verifications and audit.

The evaluation found that 75% of the 108 systems identified include all key processes and 92% of the systems include at least one process. When broken down by individual key processes, the analysis reveals that for 99 of the 108 systems identified, payment claims could be submitted through the system, while only 74 systems allowed for modifications of the progress reports and/or payment claims.

The analysis showed that the extent to which an e-Cohesion system is (exclusively) used to exchange information for all key processes varies. The exchange of information relating to payment claims and progress reports are the processes supported and used most extensively in e-Cohesion systems; the exchange of information relating to management verifications and on-the-spot checks is available in most cases (97 e‑Cohesion systems) but less used in practice. Most programmes still used parallel channels for exchanging information with beneficiaries, especially communications via emails (to a limited extent for progress reports and payment claims, but extensively so for data relating to audit and management verifications). This finding is confirmed by the survey replies received from beneficiaries: 76% indicated that they used e‑Cohesion for submitting payment claims, 70% used it for reporting of progress, while only 44% used it for providing documents related to management verifications and on-the-spot checks.

One interesting finding that emerged from the analysis is that almost 80% of the existing e-Cohesion systems had the possibility to support the application phase for the projects, even if this was not explicitly required in the CPR. Moreover, according to more than 75% of the replies provided by beneficiaries, the e-Cohesion system was used (in some cases with other parallel channels of communication) in the contracting phase of the projects.

The key functionalities of e-Cohesion systems are as follows: (1) interactive forms and/or forms prefilled by the system; (2) automatic calculations; (3) automatic embedded controls; (4) system-generated alerts; (5) online status tracking; and (6) availability of all previous data and documents processed by the electronic data exchange system (i.e. this information can be retrieved from the system).

The evaluation found that the majority of e-Cohesion systems (more than 80%) embedded all the key functionalities listed above. When looking at individual key functionalities, the most present in e-Cohesion systems were the interactive forms (96% of the systems), while system-generated alerts were included in only 82% of the systems.

Lastly, one essential feature of all e-Cohesion systems is related to data security. The data security features of each e-Cohesion system should be in line with data protection rules 10 to guarantee the security, integrity and confidentiality of data by means of the features of (1) data encryption, role-based access control in the form of (2) authentication and (3) authorisation and a (4) defined incident management process, in the event of technical issues or disruptions. Due to the sensitivity of the topic, it was not always possible to identify relevant data in the technical documentations of e-Cohesion systems (relevant information was identified for 84 out of 108 e-Cohesion systems). The analysis showed that in most cases the e-Cohesion systems presented a combination of three or four of the key features listed above. Considering that the evaluation found no indication of data security infringements and, taking into account the inputs of the respondents to the survey (over 90% of survey respondents agreed that data integrity and quality, as well as data security and privacy, had improved significantly due to the introduction of e-Cohesion systems), it can be stated that the implementation of e‑Cohesion has brought significant improvements in data security.

To conclude, the evaluation found that the majority of e-Cohesion systems complied with the legal requirements, although in certain cases some of the key requirements were still under development.

Efficiency

Under the efficiency criterion, the evaluation analysed whether:

-the introduction of e-Cohesion resulted in resource efficiency gains; and

-the benefits of e-Cohesion outweighed its costs, for both categories of users (programme authorities and beneficiaries).

In addition, a series of challenges, barriers and success factors that have a direct influence of the efficiency of e-Cohesion systems were also analysed.

The administrative costs and burden were analysed through qualitative analysis based on the relevant stakeholder responses. Thus, the evaluation of the efficiency of the e‑Cohesion initiative examined whether the beneficiaries and relevant public authorities perceived the introduction of e-Cohesion as having contributed to a reduction in costs and administrative burden for the delivery of ERDF- and CF-funded investments.

Resource efficiency

The data collected via the survey showed that, compared with paper-based processes or email exchanges, both the beneficiaries and the programme authorities agreed that the introduction of e-Cohesion resulted in a faster exchange of information (83% and 90% respectively), as well as in a reduction of the costs of project management (82% and 76%). The programme authorities who participated in the webinar provided similar responses and most frequently cited the reduction in costs relating to project management as having been less influenced by the use of e-Cohesion systems. This can be explained by the fact that the e-Cohesion covered only few processes of the project lifecycle, while the costs for project management extend beyond these.

Figure 4: Time and resource gains as a result of electronic data exchange (surveys of authorities and beneficiaries)

Source: Evaluation of e-Cohesion 2014-2020, final report, (2022).

In general, beneficiaries were the group of stakeholders that reported the highest perceived gains in terms of resources and time (96% of respondents) following the introduction and use of e-Cohesion systems.

Benefits compared with costs

When analysing the benefits of e-Cohesion systems compared to their costs in relation to the main types of processes in the implementation of projects, the results are overwhelmingly positive for both beneficiaries and programme authorities (with an average of 90% reporting that the benefits exceed the costs, in terms of the time and effort required to implement and use the e-Cohesion systems), as shown in the Figure 5 below.

Figure 5: Benefits versus costs of e-Cohesion systems

Source: Evaluation of e-Cohesion 2014-2020, final report, (2022).

Among the key processes supported by the e-Cohesion systems, both beneficiaries and programme authorities indicated the exchange of information related to payment claims as requiring the most effort. Respondents indicated that certain missing functionalities and technical features (e.g. incorrectly applied automatic calculations, no prefilling of information, file size and format limits for required attachments, etc.) required substantial additional time and effort.

The evidence collected by the evaluation for the efficiency criterion shows that users of e-Cohesion (both beneficiaries and programme authorities) perceived these systems as providing a faster and more cost-effective exchange framework throughout project implementation.

Challenges, barriers and external factors

In addition to the simplification and reduction in the administrative burden, under the efficiency criterion the evaluation sought to collect information on the administrative effort required by the introduction of e-Cohesion, as well as on the external barriers to the efficient functioning of e-Cohesion systems.

Table 3 below shows how managing authorities and intermediate bodies assess the types of effort associated with the introduction of new IT systems.

Table 3: Level of effort required by the introduction and operation of e-Cohesion systems 11

Source: calculated on the basis of the Evaluation of e-Cohesion 2014-2020, final report, (2022).

The replies show that the most resource-intensive phase is that of the setting up of e‑Cohesion systems, while their operation largely involves a manageable level of resources.

When it comes to external barriers to the efficient functioning of the e-Cohesion systems, the number one factor identified is a lack of interoperability (73% of the replies). It is followed by a lack of digital skills among the beneficiaries (54%), an insufficient digitalisation strategy at national/regional level (46%), and the limitations in terms of user-friendliness of the systems (44%).

Figure 6: External barriers to the efficient functioning of e-Cohesion systems

Source: Evaluation of e-Cohesion 2014-2020, final report, (2022).

Interoperability emerges as a key challenge for e-Cohesion systems. This finding is further analysed in light of the external coherence between e-Cohesion systems and national and European systems and databases. The detailed analysis is included in the ‘Coherence’ section below.

Coherence

Coherence can be defined as the alignment and cooperation between different policy frameworks, programmes and actions, leading to the better attainment of their objectives. The evaluation focused on coherence between the different authorities and systems for the electronic exchange of information, for the purposes of implementing EU cohesion policy. Thus, the coherence of the setting up of e-Cohesion systems was assessed at three distinct levels: (1) internal coherence at programme level; (2) coherence at national level; and (3) coherence with EU systems and databases.

Internal coherence at programme level

The analysis of coherence at programme level looked at whether all types of programme authorities had access to the e-Cohesion system in use for the programme they were responsible for. The desk research found that for 81 systems (75%), all types of relevant authorities had access to their e-Cohesion system. For one of the 108 systems, it was found that access was not granted to all types of authorities, while for 26 systems the evaluation could not establish with certainty whether all types of relevant public authorities had access to the e-Cohesion system.

Therefore, it can be concluded that internal coherence at programme level was ensured for the majority of e-Cohesion systems.

Coherence at national level

The analysis of coherence at national level looked at whether the e-Cohesion systems are compatible with other systems and databases, including the central monitoring system established at national or regional level.

The data collected indicated that it was more common for e-Cohesion systems to be connected to a central monitoring system (67 systems) than to national registers/databases (31 systems). A possible explanation for the limited uptake of interconnectivity with other national registers/databases could be a lack of harmonisation and integration between definitions of common concepts (for example, different terminology between European and national legislation) and protocols (for example, different sets of rules and guidelines for communicating data) at programme/cross-programme and national level, which reduce the complementarity and compatibility of the various systems. For example, some Member States do not have an integrated e‑government system 12 , thus preventing e-Cohesion systems from accessing national registers and databases. Authorities also mentioned that, in certain cases, beneficiaries had to use alternative methods, or to submit parts of an application manually, because the e‑Cohesion system had not integrated certain functionalities, such as tendering, contracting, invoicing, etc., as these were built on other national IT systems.

However, for 40 e-Cohesion systems (37%), the degree of coherence at national level could not be established by desk research or via the survey.

The difficulty in arriving at a satisfactory conclusion on the coherence of e-Cohesion systems at national level is, it itself, a relevant finding, and highlights the need for stronger integration of the IT systems and databases at national level to facilitate the management of EU funds. This conclusion is also borne out by the replies received to the large-scale survey, in which only 41% of respondents from public authorities indicated that the e-Cohesion system they use is connected to a national or regional database.

Coherence with EU systems and databases

With respect to coherence at EU level, the evaluation analysed whether e-Cohesion systems were linked to the relevant EU systems, such as SFC.

The desk research findings suggest that only a limited number of e-Cohesion systems were connected to SFC (24 out of 108), and only one system was connected to another European database 13 . In the context of the case studies, when asked about the reasons for the lack of connection, representatives of the managing authorities explained that this was not because of incompatibility or other technical reasons, but rather it had to do with the cost of developing this new functionality of the e-Cohesion system, as compared to transferring the data to SFC manually. The results of the research showed that central monitoring systems were more often connected to SFC – the central monitoring systems were outside the scope of this evaluation 14 . This specific finding may deserve a future in-depth analysis which should examine coherence between these three types of systems (e-Cohesion, central monitoring systems and SFC).

User-friendliness

In addition to the five Better Regulation evaluation criteria, due to the technical nature of the examined subject, an extra evaluation criterion was included in the analysis: the user-friendliness of e-Cohesion systems. The reason behind this choice is that regardless of the extent to which the various principles, functionalities and processes are implemented, the intended policy effects of information exchange systems can be enhanced or limited depending on how user-friendly those systems are.

The analysis looked at the level of agreement among the users of e-Cohesion in relation to a series of aspects which determine the user-friendliness of the systems. Figure 7 presents the responses of stakeholders that participated in the large-scale survey in relation to the user-friendliness of the e-Cohesion systems used.

Figure 7: Perceived user-friendliness of e-Cohesion systems

Source: Evaluation of e-Cohesion 2014-2020, final report, (2022).

On average, the replies indicate that most users in both categories (73% of beneficiaries and 75% of authorities) agreed that the e-Cohesion systems they worked with were user-friendly (strongly agree or agree). However, a quarter of them consider that the e‑Cohesion systems they used were not user-friendly. When analysed by the various aspects that make up the concept of user-friendliness, system response time and stability appeared to be the most problematic (over 40% of respondents across both groups of users disagreed or strongly disagreed that this aspect functioned properly). Other sources of discontent were the level of training required to use the systems and the logical sequence of steps included in each system. Nevertheless, the users tended to agree (90% of beneficiaries and 95% of authorities) that, with time, and having gained more experience with the system, they were able to carry out their tasks more efficiently.

An additional key aspect of user-friendliness is the help desk and the help functionalities embedded in the e-Cohesion systems. The overall perception of these help desk and help functionalities is positive (over 70% of users in both categories and for both aspects). The interviews with the users of e-Cohesion systems in the context of the case studies show that they attach significant value to well-developed help functionalities and comprehensive user documentation. The analysis also found that the collection of user feedback and its incorporation into system developments generated higher levels of perceived user-friendliness.

Based on the evidence presented in the evaluation report, it was found that the majority of e-Cohesion systems (58%) are moderately user-friendly and 40% are extensively user-friendly, while 2% are considered to have limited user-friendliness.

Figure 8: E-Cohesion systems by user-friendliness

Source: Evaluation of e-Cohesion 2014-2020, final report, (2022).

The conclusion of the analysis of the user-friendliness criterion is that, while the determinants of user-friendliness may vary, the e-Cohesion systems are overall perceived as user-friendly. Future developments, such as improvements in system response time and stability, improvements to the user interface and better user documentation to reduce the need for training, are expected to improve the perceived user-friendliness of the systems.

4.2.How did the EU intervention make a difference?

EU added value

The analysis of the EU added value criterion was carried out from two angles: (1) by examining whether the legal requirement to set up e-Cohesion systems was the main driver behind the introduction of the systems; and (2) by establishing if the introduction of e-Cohesion systems had any spillover effects in the Member States.

To determine the contribution of EU legislation to the setting up of the systems in the Member States, the evaluation assessed whether it was the legal provision that prompted the national authorities to set up e-Cohesion systems. Based on the desk research, before the entry into force of the CPR in 2014, only 12 systems were in place. During 2014, an additional 17 systems were deployed, while 51 systems became operational from 2015 15 .

Table 4: Year of start of operation of e-Cohesion systems

Source: Evaluation of e-Cohesion 2014-2020, final report, (2022).

The time needed to develop and deploy such systems varies, but it can be assumed that the systems that became operational before 2014, as well as in the first two years after that – i.e. up to end-2015 – were decided upon before the legal requirement was in place. This amounts to roughly half of the 108 systems. For a quarter of the systems – i.e. those developed after 2015 – the legal provision was most probably a determining factor in their development, while for 25% of the e-Cohesion systems, the year of deployment could not be identified. As regards these later cases, and the systems deployed before 2015 (75% of the systems), it can be safely assumed that the development of certain functionalities and modules was influenced by the European Commission’s advocacy for the introduction of e-Cohesion systems. In the overall context of the implementation of ERDF and CF 2014-2020, the Commission communicated, advocated for and promoted the e-Cohesion initiative to programme authorities, which might also have played a role in its mainstreaming across Member States.

There was therefore EU added value in the e-Cohesion initiative, as it triggered either the deployment of new systems or the further development of existing ones in order to integrate the required functionalities and processes.

The survey of authorities also aimed to establish whether the e-Cohesion systems had any spillover effects, e.g. by inspiring other non-cohesion policy systems, or by using lessons learned from e-Cohesion when setting up other new systems (in relation to other national or EU instruments). Most authority representatives assessed the spillover effects of the e‑Cohesion initiative positively (75% of respondents). This indicates that the e-Cohesion initiative supported innovation and the transfer of ideas between stakeholders.

4.3.Is the intervention still relevant?

Relevance

When analysing the relevance criterion, the evaluation study (1) investigated whether the needs of the different user groups (programme authorities and beneficiaries) were met and (2) tried to identify the external factors that may impact the relevance of the e-Cohesion system.

The different categories of user groups use the e-Cohesion systems to different extents –for example the beneficiaries would use the system for all key processes listed in the legislation, while the audit authorities only for one key process, which was related to exchange of information for audit verifications. Therefore, the systems are not equally relevant to all user groups. To determine if the needs of the user groups are met, the field research analysed the scope of the use of e-Cohesion systems by authorities and beneficiaries by key stages and processes in a project’s life.

The analysis showed that exchanges of information in relation to payment claims and reporting of progress were the most common, while the preparation and signature of financing contracts was the process that mostly took place outside the system. This is not surprising, as the latter was not a process listed in the implementing Regulation for e-Cohesion systems, but it was an optional module which was developed for many e-Cohesion systems. The e-Cohesion systems were not widely used for communication between programmes authorities and beneficiaries and for the planning and implementation of audits, even if the latter was part of one of the key processes associated with e-Cohesion systems (i.e. exchange of information related to audits).

In terms of frequency of use, the analysis showed that the systems were used most extensively by managing authorities and certifying authorities, and the least by audit authorities.

E-Cohesion systems were less commonly used for the processes outside the scope of the legislative provisions, such as for the application phase of the projects, including the preparation and signature of contracts, or for communications between the programme authorities and beneficiaries.

Figure 9: e-Cohesion coverage of key processes

Source: Evaluation of e-Cohesion 2014-2020, final report, (2022).

On closer analysis of the communications between beneficiaries and authorities, the evaluation found that e-Cohesion systems were used for this purpose by almost 90% of managing authorities, intermediate bodies and certifying authorities. Only 40% of the audit authorities mentioned that they used the e-Cohesion system to a certain extent for communicating with the beneficiaries.

Figure 10: Use of e-Cohesion for communication between authorities and beneficiaries

Source: Evaluation of e-Cohesion 2014-2020, final report, (2022).

From the evidence collected, it can be concluded that while, overall e-Cohesion systems were used by all types of users, the degree of relevance varied between the processes listed in the legislation (highly relevant) and other processes in the project lifecycle (less relevant), and between different types of users, with managing authorities, intermediate bodies and certifying authorities being the most frequent users.

The evaluation also examined possible external factors that limited the relevance of e-Cohesion systems. This aspect was explored at the level of the authorities, as they had a wider overview of these issues. The survey indicated the limited interoperability as being the most important challenge to the relevance of e-Cohesion systems (73% of the replies). This finding should be corroborated with one of the most important barriers identified by the surveys, more specifically the lack of digitalisation strategies sufficiently developed (46%). Another factor which influenced negatively the relevance of e-Cohesion systems was the lack of a nationally recognised, legally valid e-signature feature. In the absence of an e-signature feature the e-Cohesion system cannot function fully electronically, and beneficiaries are more inclined to use parallel channels for data exchange.

5.1 Conclusions and lessons learned

The main objective of introducing e-Cohesion in the 2014-2020 programming period was to simplify and streamline the implementation of cohesion policy programmes, by reducing the administrative burden for beneficiaries and authorities. The evaluation found that this objective was accomplished – both beneficiaries and programme authorities reported that e-Cohesion systems simplified the exchange of information and reduced the administrative burden for implementing ERDF and CF projects and programmes.

The ideas above emerged from the in-depth analysis performed for six evaluation criteria: efficiency, effectiveness, coherence, relevance, EU added value and user-friendliness.

From the effectiveness perspective, these systems were expected to facilitate the interoperability with national and EU frameworks and allow for the beneficiaries to submit the relevant information only once.

The majority of the systems integrated the modules and features expected, in particular ensuring availability of all previous processed data and documents, interactive forms and/or forms prefilled by the system and online status tracking. The e-Cohesion systems were extensively used for (1) submitting and managing payment claims and (2) for reporting on progress. The systems also played an important role in the communication between beneficiaries and programme authorities in relation to these two processes. Alternative communication means, such as emails between beneficiaries and programme authorities were still used in certain instances, mostly for the exchange of information related to audit verifications. In this respect, the evaluation found that one of the possible reasons for the use of alternative means in presence of e-Cohesion systems is the absence of e-signature features embedded in the systems.

In terms of efficiency, users’ perception of e-Cohesion systems is overall positive, as the systems have provided faster and more economical information exchange frameworks for the implementation of projects. Beneficiaries reported the highest benefit in terms of resources and time following the introduction of the e-Cohesion systems, especially for handling payment requests and reporting on progress.

e-Cohesion systems were reported as having a high level of internal coherence - all targeted categories of programme authorities (managing authorities, intermediate bodies, certifying authorities and audit authorities) had access to and were able to use the systems. As regards, the external coherence, the e-Cohesion systems were more likely to be connected to the central monitoring system than to other national registries/databases or European systems, such as SFC. A possible explanation for the limited uptake of interconnectivity with other national registers/databases could be a lack of harmonisation and integration between definitions of common concepts (for example different terminology between European and national legislation) and protocols (for example different sets of rules and guidelines for communicating data) between the programme and national systems.

As regards their relevance, e-Cohesion systems were relevant to all types of users (programme authorities and beneficiaries). The analysis showed that the systems were used most extensively by representatives of managing authorities and certifying authorities, and least by representatives of audit authorities. As highlighted above, this can be explained by the use of alternative means of communication (i.e. emails) for the audit verification process, which can be further linked to the possible limited accessibility to the e-signature feature for some of the beneficiaries.

From the perspective of EU added value, the e-Cohesion initiative triggered either the deployment of new systems or the further development of existing ones for integrating the required functionalities and processes. The EU legislation and the constructive dialogue between the EC and programme authorities had a determining role in the development and deployment of e-Cohesion systems in the 2014-2020 programming period.

e-Cohesion systems were perceived as being user-friendly. The help desk functionalities, the collection of user feedback and its incorporation into system developments were highly appreciated by the users.

In conclusion, during the 2014-2020 programming period, e-Cohesion systems were successful in contributing to the EU’s digital progress through simplifying the exchange of information and reducing the administrative burden. In the period 2021-2027 the requirement was further consolidated in the legislation 16 .

Areas where future developments could deliver further simplification and reduction of administrative burden include:

-The continuous developments of interoperability between e-Cohesion systems and national databases and systems;

-The integration of the e-signature feature in the e-Cohesion systems, which would further incentivise beneficiaries to use this electronic exchange channel and limit the parallel data exchanges;

-The integration of additional processes in the e-Cohesion systems, such as the application and contracting phases of projects, which were already included by some authorities in their e‑Cohesion systems in the 2014-2020 programming period, even though this was not a legal requirement;

-The improvement of response time, stability of the system, user interface and user documentation (guides, tutorials, questions and answer repositories) would reduce the training needs and increase user-friendliness;

5.2. Good practices for improving the functioning of e-Cohesion systems

The evaluation aimed also to collect and present the main lessons learned and challenges identified in setting up e-Cohesion systems. In relation to this, several good practices have been identified following the in-depth analysis of six e-Cohesion systems, which are listed in Table 5.

Table 5: Good practices in relation to e-Cohesion systems

Source: Evaluation of e-Cohesion 2014-2020, final report, (2022).

5.3 Implementation of e-Cohesion in the 2021-2027 programming period

Under Article 69(8) of the CPR 2021-2027, the implementation of e-Cohesion requirements remains, in the first instance, the responsibility of the Member States. The European Commission will continue collecting information about the level of implementation of the e-Cohesion requirements in the 2021-2027 programming period for cohesion policy programmes, in particular using the methods outlined below:

1.by monitoring and checking the existence of e-Cohesion systems during programmes and the negotiation of partnership agreements. Where such a system has not yet been set up, the Commission will continue monitoring its setting-up;

2.at the monitoring committee and annual review meetings with Member States, the Commission representatives will enquire about e-Cohesion systems and follow up if needed; and

3.the key requirements of the e-Cohesion systems will be checked by national audit authorities (in their audits of management and control systems) and by European Commission auditors throughout the entire programming period 17 .

1.Lead DG and Decide reference

The evaluation was managed by the European Commission’s Directorate-General for Regional and Urban Policy, DG REGIO Unit B2: Evaluation and European Semester.

Decide entry: PLAN/2018/4898.

2.Derogations granted

A derogation was granted from the obligation to carry out an open public consultation. The derogation was justified by the fact that the evidence underpinning the analysis could only be collected from the users of e-Cohesion (beneficiaries and programme authorities). The general public, not being users of e-Cohesion systems, could not provide any relevant input.

3.Organisation and timing

The evaluation roadmap was published on 11 September 2020 and was closed on 30 October 2020. The service contract started on 20 November 2020 and the final report 18 was delivered on 2 May 2022.

An Interservice Steering Group (ISG) was set up comprising the following EC services: Secretariat-General, Budget, and Home Affairs. Five meetings of the ISG were held on: 17 March 2021, 11 May 2021, 26 November 2021, 17 February 2022, and 14 March 2022.

4.Evidence and sources

Evidence was gathered from several types of sources: surveys and interviews with users of e-Cohesion from the four types of programmes authorities, as well as beneficiaries, case studies, desk research, and a webinar with the representatives of the Member States. The main body of evidence consisted of the 6 248 replies to surveys by beneficiaries and 455 replies to surveys by programme authorities. Although the respondents are not a representative sample of the users of e-Cohesion, the large number of replies make it possible to draw conclusions on the evaluation criteria.

5.External expertise

The evaluation study was carried out by a consortium composed of PPMI Group (Lithuania), Rechenwerk (Germany) and Ismeri Europa (Italy).

The evaluation of e-Cohesion posed significant challenges to constructing a representative sample of users from whom to collect evidence. To overcome this risk and to collect as much user feedback as possible to underpin the analysis, the Commission services and the contractor decided to take an approach based on large-scale surveys, which were complemented by desk research, case studies, interviews, and a webinar with the representatives of the authorities.

Desk research

The desk research consisted in identifying the systems using open sources and analysis of any relevant information that was readily available. The desk research benefited from input from the Member State level, collected via the network of country experts provided by the contractor.

Large-scale surveys

The main tools for gathering evidence were the large-scale surveys. Given the different needs of the beneficiaries and authorities when using the e-Cohesion systems and to maximise the relevance of the responses, instead of a single survey, the contractor designed two different surveys, one for beneficiaries and one for authorities. The survey for authorities was further tailored to each of the four types of programme authorities.

The survey questions were designed to both collect factual information on the e-Cohesion systems used by the respondents, as well as to obtain the opinion of the users regarding the various functionalities of the systems, their added value and the remaining challenges.

To ensure the relevance and ease of understanding of the survey for the various respondents, the surveys were first piloted on a small group of representatives of the authorities, beneficiaries and Commission staff. The surveys were translated into the official languages of the EU.

The authorities were contacted by email and invited to take the survey, while the beneficiaries were contacted via the managing authorities, given that the latter hold the most up-to-date lists of beneficiaries. The contractor sent reminders to and contacted several authorities by phone to increase the number of replies. When the surveys were closed, 455 replies from authorities and 6 248 replies from beneficiaries were received.

Although representativeness could not be established, responses were numerous enough to enable the analysis of the experience of users of e-Cohesion and address the evaluation criteria.

Case studies

To obtain greater analytical depth than it was possible to achieve via the large-scale surveys, the analysis included six case studies. The case studies were carried out on systems from Estonia, Greece, Portugal, Italy, Poland, as well as the eMS system dedicated to Interreg cooperation programmes. 84 interviews were conducted as part of the case studies.

Two case studies were run as pilots (Estonia and Greece). They were selected based on existing information at the Commission.

The remaining systems were selected on the basis of the survey results (sufficiently high number of replies from both beneficiaries and authorities, high overall scores, with both a centralised and a decentralised approach to e-Cohesion).

The case studies were prepared based on initial desk research and surveys and consisted of interviews with three groups of interviewees:

·Policy: representatives of the authorities who have designed the system and are responsible for operating the system.

·Technical: representatives of the authorities and contractors who have set up the system.

·Users: beneficiaries and institutional users.

This three-pronged approach made it possible to gather sufficiently detailed information, which is presented in the case study reports and in the cross-case analysis report, which draws conclusions based on all the case studies.

Webinar with the representatives of authorities

Following the completion of the surveys and the initial analysis of the findings, a webinar was organised with representatives of the authorities to discuss the findings and gather additional information on certain aspects. The webinar was attended by 113 participants (of whom 18 were staff from either the contractor or the Commission) and 25 Member States were represented.

Limitations

·The findings under the efficiency criterion rely on qualitative analysis (i.e. the perceptions of the stakeholders involved in the field research via the large-scale survey, case studies and webinar) of the costs and benefits of the implementation of e-Cohesion. 

·The survey was experimental in the sense that it was the first time that ERDF and CF beneficiaries had been targeted on such a scale. The survey aimed to cover all ERDF (including Interreg) and CF programmes, and targeted authorities and beneficiaries from all EU Member States. In relation to this attempt to target beneficiaries for the first time at scale, the survey has some limitations in terms of how far conclusions may be drawn, and to what extent certain conclusions are substantiated by the evidence. In addition, the subject of the survey – electronic data exchange systems (e-Cohesion systems) – is not a straightforward one. Based on the share of ‘Do not know / cannot answer’ answers, the survey results indicate that respondents found some of the survey questions difficult to answer.

·To reach the beneficiaries, the evaluation team relied on the help of managing authorities, who were asked to disseminate the survey to their beneficiaries. This approach resulted in the evaluation team not having any opportunity to impact the scope of beneficiaries targeted, except for constant communication and reminders to the MAs asking them to disseminate the surveys to their beneficiaries. The evaluation team did not have any information on how many, or what kind of beneficiaries (i.e. private, public), the MAs targeted. This approach of targeting beneficiaries through the MAs and giving the MAs full control of this process resulted in a rather uneven distribution of responses. For some countries, hundreds of responses were received, whereas for others there were only a few. For example, 40% of all beneficiary respondents represent Italy and Poland. Some types of authorities or beneficiaries are thus likely to be overrepresented in the survey, which in turn places certain limitations on the interpretation of the data.

·It should also be noted that the evaluation was carried out during the preparation of the 2021-2027 programmes, the Recovery assistance for cohesion and the territories of Europe (REACT-EU) 19 amendments to the 2014-2020 programmes, and the preparation of the Recovery and Resilience Facility (RRF) 20 programmes, which put considerable pressure on the relevant stakeholders and limited the time they could devote to responding to the surveys.

Effectiveness

Efficiency

Relevance

Coherence

EU added value

User-friendliness

Given the scope of the e-Cohesion initiative (hundreds of thousands of people at beneficiary and programme authority level use these systems), a quantitative analysis of the costs and benefits would have required significant resources in terms of time and budget which were not available for this evaluation. Due to the lack of quantitative data on the costs, benefits and savings associated with the introduction of e-Cohesion are described in the following tables from a qualitative perspective, as the perceived reduction of costs and reduced administrative burden by beneficiaries and programme authorities.

Table 1 Overview of the costs and benefits identified in the evaluation

Table 2 Simplification and burden reduction

1.Strategy

Objectives

The general objectives of the consultation activities were to:

·disseminate the activity and encourage stakeholder participation in the evaluation process;

·ensure transparency of the process;

·enhance accountability and transparency of European Commission activities;

·encourage stakeholders to respond to the enquiries and the other knowledge-gathering activities to fill information gaps;

·gather general public end expert stakeholder opinions on the principle evaluation findings; and to

·publicise the main evaluation findings.

Tools

The stakeholders targeted by the consultation activities were first mapped by and then matched to different consultation tools. In particular, four sets of activities, each with a different purpose, were organised:

·publication of the evaluation roadmap;

·large-scale survey of all users of e‑Cohesion systems

·semi-structured and in-depth interviews carried out by the contractor for use in case studies; and

·webinar organised by the contractor to discuss preliminary findings of the evaluation and collect additional input from programme authorities.

A derogation was granted from the obligation to carry out an open public consultation. The derogation was justified by the fact that the evidence underpinning the analysis could only be collected from the users of e-Cohesion systems (beneficiaries and programme authorities). The general public, not being users of e-Cohesion systems, could not provide any relevant input.

Stakeholder participation

A summary of the final stakeholder participation figures is provided in Table 1 below.

Table 1 – Stakeholder reach by activity

Source: Evaluation of e-Cohesion 2014-2020, final report, (2022).

2.Delivery and results

Roadmap

The roadmap of the evaluation was published in September 2020 on the ‘Have Your Say’ website of the European Commission 21 for four weeks.

The six replies received from stakeholders revolved around the functionalities of e‑Cohesion systems, including accessibility, some issues relating to data import from different file sources, the relevance of the information collected in the context of evaluation, the interoperability between e-Cohesion systems and SFC, and the requirements for using the systems.

Large-scale survey

The key findings of the large-scale survey are presented below.

Respondents are highly satisfied with the functioning of e-Cohesion systems

The survey results indicate the high overall satisfaction of both the authorities and the beneficiaries with the functioning of e-Cohesion systems. This high satisfaction rate relates to a broad range of factors that were explored in the surveys of authorities and beneficiaries, including the benefits of having the ability to exchange information electronically throughout the key processes and the user-friendliness of the systems.

When asked whether the benefits of the e-Cohesion system outweigh their costs, the results were overwhelmingly positive in both target groups (approximately 90% of respondents agree or strongly agree), with only minor differences between the perception of efficiency gains brought by e-Cohesion across different processes. This indicates that e-Cohesion systems help to efficiently support the operational processes of authorities and beneficiaries.

Figure 1 - % of respondents who strongly agree or agree that systems’ benefits outweigh their cost for tasks related to …

With regard to the e-Cohesion initiative’s objectives, the survey results show that authorities and beneficiaries alike agree that e-Cohesion systems have delivered improvements and simplification in a wide range of areas. The areas in which both groups of respondents say that electronic data exchange has yielded the greatest improvements are data security and privacy, data quality and integrity, and the transparency and accessibility of relevant information.

In terms of user-friendliness, the responses from authorities and beneficiaries suggest, overall, that e‑Cohesion systems are highly user-friendly and effective, though the responses do also indicate some areas of weakness.

The factors relating to user-friendliness that were rated most positively are as follows:

·With time and more experience using the system, it helps users to carry out their tasks more efficiently.

·The use of the e-signature and its associated reduction of costs and increased security.

·(Especially among authorities) the usefulness of the assistance provided by the help desk service.

Users are, however, comparatively less satisfied overall with the responsiveness and stability of e‑Cohesion systems. In addition, authorities, in particular, were more likely to perceive systems as less flexible regarding their preferred order of steps in the workflow. Lastly, around 35% of respondents said that the use of e-Cohesion systems requires extensive training (see Figure 2). 

Figure 2 - % of respondents who strongly agree or agree that …

e-Cohesion systems provide extensive support for the key processes but the parallel exchange of information still takes place outside the systems

The findings of the surveys show that the systems cover all key processes (i.e. application, implementation, auditing), albeit to a varying extent. The experience of the survey respondents reveals that the application process is extensively supported, even though this is not legally required (89% of respondents say that e-Cohesion systems support the application process). In terms of project implementation, the submission of payment claims and progress reporting are the core e-Cohesion processes, with 96% of beneficiary respondents stating that those core processes are supported by the system. Auditing, however, is less standardised and thus a considerable amount of data exchange takes place outside the system.

Even though the respondents indicate that the key processes are supported by e-Cohesion systems, including the application phase, parallel data exchanges outside e-Cohesion systems have not been eliminated altogether and remain commonplace, especially in relation to less standardised auditing activities:

·Application process: 26% of authorities answered that only up to 50% of all data exchanges related to application take place via the system; 32% of beneficiaries answered that they could apply through the system, but some documents are submitted through other channels.

·Implementation: 10% of authorities answered that only up to 50% of all data exchanges related to project implementation take place via the system; 20% of beneficiaries answered that information related to payment claims were submitted both through the system and via other channels; 25% of beneficiaries answered that information related to reporting on the progress were submitted both through the system and via other channels.

·Auditing: 26% of authorities answered that only up to 50% of all data exchanges related to auditing take place via the system; 43% of beneficiaries answered that information related to management verifications or on-the-spot checks were submitted both through the system and via other channels.

Individual responses by authorities to open questions highlight that the elimination of paper-based processes and the closely linked centralised access to all information is one of the most relevant improvements resulting from the electronic data exchange and can significantly contribute to the simplification of their key processes. The elimination of additional paper files is also among the key areas for development for the upcoming programming period mentioned by authorities in open-ended answers.

Interoperability is a key challenge

Regarding barriers relevant to the efficient functioning of e-Cohesion systems, the survey results show that authorities perceive the lack of interoperability with other systems and registers as the most significant challenge (73% of authority respondents indicated it as a very or somewhat relevant challenge), with all other challenges being indicated as significantly less relevant. This finding is also supported in answers provided by respondents to several open-ended questions, showing that even without prompts, interoperability is mentioned as one of the most important challenges indicated by authorities (see Figure 3).

Figure 3 - Key challenges for the efficient functioning of systems (% of authorities that selected very relevant or relevant)

A closer look at the results of the surveys shows that it is much more common for systems to reuse and prefill information that was previously entered in e-Cohesion systems rather than prefill information based on information available from external databases or registers: 82% of beneficiaries and 89% of authorities answered that the system prefills information based on what was entered in previous steps; 81% of beneficiaries and 88% of authorities answered that the system prefills information based on project application; 56% of beneficiaries and 46% of authorities answered that the system prefills information based on information available from various national/regional registers.

Figure 4 - % of respondents who selected that the e-Cohesion system pre-fills information for beneficiaries

Improvement of e-Cohesion systems is continuous process

The findings of the surveys also show that the improvement of e-Cohesion systems is a continuous process, requiring significant input from the authorities: most authorities say that their systems will either be further developed (55%) or replaced (36%) in the next programming period.

The ongoing improvement of the systems is also noticed and appreciated by the beneficiaries: 68% reported that the various functionalities and the overall operation of the systems are continuously improving, a finding which also provides a strong indication of their overall satisfaction with those systems.

The introduction and maintenance of such systems is nevertheless a resource-intensive activity for the authorities. Survey findings suggest that the initial deployment of e‑Cohesion systems requires more effort from authorities than their subsequent operation, with IT procurement and the tendering process requiring the most effort and resources, followed by the setting-up and deployment of the systems and the associated administrative costs and human resource requirements.

Interviews

The interviews were integrated into the six case studies conducted as part of the evaluation. The six case studies are listed in Table 2.

Table 2. e-Cohesion systems analysed in the case studies

Source: Evaluation of e-Cohesion 2014-2020, final report, (2022).

The in-depth analysis of the six selected e-Cohesion systems complemented the information previously gathered during the initial desk research-based mapping and further in-depth desk research of additional data sources acquired from relevant national authorities or the European Commission.

In many cases, the functionality of the e-Cohesion systems is an integral part of the transactional monitoring system. Transactional monitoring (activities related to checking, approving applications, progress reports and payment claims) in such cases is supported by a single, unified system sharing a common user interface and database. This is the approach of systems like eMS (Interreg), MIS (Greece), SFINGE2020 (Italy) and SL2014 (Poland). Consequently, the terms ‘e-Cohesion’ and ‘monitoring’ are often used interchangeably.

There are also situations in which the e-Cohesion system is a separate entity. In these cases, the front-office functionality is independent of the back-office functionality. ‘Independent’ in this context means that if even one system is down, the other can still operate as normal. Moreover, one system can be individually developed or even replaced if technical progress or additional requirements make this favourable. Both systems have their own database but they are connected by a bidirectional interface. In such cases, the transactional monitoring system pre-dates the e-Cohesion system, which was developed later to collect and exchange information, e.g. in the case of Balcão2020 and e-Toetus.

In the 2014-2020 CPR, e-Cohesion is described as a task of the Member States, while the monitoring system is described as a task of the managing authority. Except for Balcão2020, all the systems analysed were developed in the same organisational context as the monitoring system.

e-Cohesion is a complex environment that comprises various different aspects. Thus, similarities and differences exist at different levels.

The context in which e-Cohesion systems operate, especially the fact that they may be used for only one or for several programmes, or by only one Member State or across multiple Member States, is also significant. Equally, it is important to take into account whether the system is used only for programmes under a single fund or for programmes under multiple funds, e.g. for both ERDF and ESF programmes. There are certain fund-specific aspects, such as the handling of participant data, that affect the performance and complexity of the corresponding IT solutions. Since ESF programmes process sensitive data, essential measures are needed to ensure data security and meet the requirements of EU data protection rules: such data security measures will also affect ERDF financed projects if they are managed together with ESF programmes in a single, integrated IT solution. Only two of the six systems examined – eMS and SFINGE2020 – support only ERDF programmes. All the other systems support several different programmes.

The same is true if different groups of beneficiaries use the same IT solution. It is different if only public organisations from a single Member State use a system (as is the case for MIS, which is used for both national and regional programmes) or if a wide range of different types of public and private beneficiaries located in different countries use the same system (as is the case for eMS). Another important aspect is if programmes using the e-Cohesion system adhere to a common or at least similar management and control system, as these heavily define the structures, processes and rules that apply to the system.

Webinar

The main purpose of the webinar was to present the survey findings and gather additional feedback from participants on a few selected themes to complement those findings. Thus, during the webinar, the evaluation team collected reactions and thoughts on the survey findings and obtained additional input on the key selected themes that would later feed into the final evaluation reports.

The target group for the webinar consisted of MAs and any other authorities representing the Member States (intermediate bodies, certifying authorities, audit authorities). The webinar took place on 23 November 2021 via the Zoom platform.

After a presentation of the findings of the survey on the scope of the use of e-Cohesion systems, webinar participants were invited to rank the processes they believed were most burdensome for beneficiaries during the application and implementation phases. The purpose of this exercise was to establish whether their responses corresponded to beneficiaries’ views on which system processes are the most burdensome, and to complement beneficiaries’ views with those of the authorities, as the survey findings indicate that the authorities are more frequent users of e‑Cohesion systems. According to participants, the exchange of information relating to payment claims required by far the most effort. This process was ranked highest by over half of the webinar participants who voted on this question. Interestingly, the survey findings indicate that the payment claims process has benefited most from the introduction of the e-Cohesion system 22 . Furthermore, the survey findings show that the payment claims process is the one for which data exchange is most likely to be carried out exclusively in the system.

The survey findings indicate that the key processes are, to a large extent, being implemented via the e-Cohesion systems. However, results also indicate that information is still being exchanged in parallel via other channels (e.g. email, paper or similar). To explore the topic of parallel data flows, webinar participants were asked if any exchange of data still takes place outside the system. Almost a fifth of authorities in the webinar said that all data exchange takes place within the system, while most said that data exchange still takes place through other channels as well. When asked for further comments, one MA representative mentioned that participant information required for ESF is gathered outside the system. Another authority representative commented that information relating to procurement is still exchanged outside of the system, as these documents are large, and the system is limited in terms of the size of documents it allows users to upload.

Most webinar participants value the advantages of interoperability and the ‘once-only’ encoding principle. However, a greater proportion of participants felt that the ability to reuse information entered by beneficiaries in previous steps was more useful than the ability to reuse information from external databases. These findings can be explained by the perceived level of difficulty associated with the latter: while around one third of webinar participants felt the reuse of previously entered information is very difficult, almost 90% of them rated the reuse of information from external databases as very difficult.

The survey findings indicate that e-Cohesion systems brought simplification in a wide range of areas. All of those areas were assessed very positively and none stood out in particular. For this reason, webinar participants were asked to name a single key area in which authorities perceived the introduction of the e-Cohesion system had had the least impact. Most webinar participants who voted chose the reduction of costs relating to project management, closely followed by improvements in data quality and integrity. However, it is important to bear in mind that these areas were assessed positively in the survey.

As part of the survey, the authorities were asked one question regarding the main challenges of and significant barriers to developing an efficient and effective e-Cohesion system, and one question on the factors that are essential for success in developing an efficient and effective e-Cohesion system. On the whole, the challenges were not rated as particularly significant, except for the lack of interoperability with external applications and databases, which 73% of authorities considered to be a significant issue. All options concerning success factors, however, were rated as significant (at least 90% of respondents agreed, to varying degrees, with all the success factor statements).

Clearly, the survey findings are more complex when it comes to the challenges. For this reason, webinar participants were asked several questions designed to obtain a better understanding of the authorities’ perspective on this. Participants were asked to further specify which aspects of interoperability represent a challenge to an efficient and effective e-Cohesion system. Most webinar participants reported that a lack of common definitions (of concepts, information needs, data fields, forms, etc.) at programme and cross-programme level, was the main barrier to interoperability with other systems and registers.

To assess the preparedness of e-Cohesion systems for the 2021-2027 programming period, webinar participants were asked about their future plans for the development of their respective systems. The planned improvements to e‑Cohesion systems cited by the authorities can be grouped into three main categories:

·Interoperability – facilitate data exchange with other systems and/or national registers/databases.

·User-friendliness and usefulness – introduce new features such as dashboards, automatic controls (validation checks), better communication with beneficiaries, standardisation and simplification.

·Harmonisation on a national level – use e-Cohesion functionalities for national fund management, provide overviews of fund performance on national level.

(1) Regulation (EU) No 1303/2013 of the European Parliament and of the Council of 17 December 2013 laying down common provisions on the European Regional Development Fund, the European Social Fund, the Cohesion Fund, the European Agricultural Fund for Rural Development and the European Maritime and Fisheries Fund and laying down general provisions on the European Regional Development Fund, the European Social Fund, the Cohesion Fund and the European Maritime and Fisheries Fund and repealing Council Regulation (EC) No 1083/2006.

(2) The evaluation did not include any OPs implemented in the United Kingdom as it was no longer a member of the EU at the time the evaluation was commissioned.

(3) The evaluation methods are described in more detail in Annex II, ‘Methodology and Analytical models used’.

(4) SFC is the electronic exchange of information concerning shared management funds, between Member States and the European Commission, provided by Art. 74(4) CPR.

(5)  Commission Implementing Regulation (EU) No 1011/2014 of 22 September 2014 laying down detailed rules for implementing Regulation (EU) No 1303/2013 of the European Parliament and of the Council as regards the models for submission of certain information to the Commission and the detailed rules concerning the exchanges of information between beneficiaries and managing authorities, certifying authorities, audit authorities and intermediate bodies 
https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32014R1011

(6) Commission Staff Working Paper Impact Assessment accompanying the document Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL laying down common provisions on the European Regional Development Fund, the European Social Fund, the Cohesion Fund, the European Agricultural Fund for Rural Development and the European Maritime and Fisheries Fund covered by the Common Strategic Framework and laying down general provisions on the European Regional Development Fund, the European Social Fund and the Cohesion Fund and repealing Regulation (EC) No 1083/2006 {COM(2011) 615 final} {SEC(2011) 1142 final}

(7) Questions & Answers on e-Cohesion Programming period 2014-2020 (ERDF, Cohesion Fund and ESF), EGESIF_17-0006-00, 06/04/2017; Building Blocks for e-Cohesion: good practices from Member States, regions and programmes, Version 2, December 2013.

(8) In one case, a system was linked to the keep.eu European project database for territorial cooperation projects.

(9)  See Article 10(4) of Commission Implementing Regulation (EU) No 1011/2014.

(10) Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995 on the protection of individuals with regard to the processing of personal data and on the free movement of such data, repealed by Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data.

(11) The replies where the respondents indicated that they were not involved in these aspects were removed from the table.

(12) An integrated e-government system provides a wide range of public services to citizens and businesses digitally. It generally integrates services provided by various institutions in a country or a region.

(13) The European database in question is keep.eu which showcases information on Interreg projects.

(14) The external coherence between the central monitoring systems and SFC has already been analysed as part of the ex post evaluation of the ERDF and CF in 2014-2020 - Work package 2 - Study on the monitoring data on ERDF and Cohesion Fund operations, and on the monitoring systems operated in the 2014-2020 period. The evaluation concluded that interoperability between the central monitoring systems and SFC is a point for improvement which could generate efficiency gains and improve data consistency.

(15)  For 28 systems, the starting date could not be established.

(16) Regulation (EU) 2021/1060 laying down common provisions on the European Regional Development Fund, the European Social Fund Plus, the Cohesion Fund, the Just Transition Fund and the European Maritime, Fisheries and Aquaculture Fund and financial rules for those and for the Asylum, Migration and Integration Fund, the Internal Security Fund and the Instrument for Financial Support for Border Management and Visa Policy.

(17)  As laid out in Annex XI of the CPR 2021-2027, failure to comply with these requirements may lead to the conclusion that this key requirement ‘works partially and substantial improvements needed’ (category 3) or ‘essentially does not work’ (category 4). Where this conclusion is associated with deficiencies found in relation to other key requirements (also assessed in category 3 or 4), this may lead to interruption of the payment deadline, suspension of payments or financial corrections.

(18) See: https://op.europa.eu/en/publication-detail/-/publication/5f2b4c00-e79c-11ec-a534-01aa75ed71a1

(19)   https://commission.europa.eu/funding-tenders/find-funding/eu-funding-programmes/react-eu_en  

(20)   https://commission.europa.eu/business-economy-euro/economic-recovery/recovery-and-resilience-facility_en  

(21)   https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/12519-Evaluation-of-e-Cohesion-2014-2020/feedback_en?p_id=8509725  

(22) Question for beneficiaries: ‘Please assess the following statement: the benefits (e.g. reduced administrative burden, simplified procedures) of the introduction of the electronic data exchange system between beneficiaries and authorities exceeds the associated costs (e.g. the time and effort required to use it) for the following processes:…’.