EUROPEAN COMMISSION

Brussels, 17.12.2021

SWD(2021) 965 final

COMMISSION STAFF WORKING DOCUMENT

Final evaluation of the ISA² programme

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

Results of the final evaluation of the ISA² programme

{COM(2021) 965}

Table of contents

1.Introduction

2.Background on the intervention logic of the ISA2 programme

3.Implementation / State of Play

3.1.    How is the ISA2 programme implemented?    

3.2.    Important developments in the policy field    

4.Methodology

4.1.    Fact-finding    

4.2.    Limitations and robustness of findings    

5.Analysis and answers to the evaluation questions

5.1.    Relevance    

5.2.    Effectiveness    

5.3.    Efficiency    

5.4.    Coherence    

5.5.    EU added value    

5.6.    Utility    

5.7.    Sustainability    

6.Conclusions

6.1.    Successful elements of the intervention    

6.2.    Less successful elements of the intervention    

6.3.    Will issues be resolved over time or do they need to be addressed with specific measures?    

6.4.    Lessons learnt    

Glossary

1.Introduction

The  ISA2  programme – interoperability solutions for public administration, businesses and citizens – aimed to support the digital transformation of the public sector by providing digital solutions that enable public administrations, citizens, and businesses across the EU to reap the benefits of interoperable cross-border and cross-sector public services. ISA2’s primary beneficiaries are EU, national and regional public administrations: by reusing the solutions offered by ISA2 they can provide better – more interoperable, user-centric, and digital – public services. However, the programme also helps a broader group of stakeholders, namely EU businesses and citizens.

Established by the  ISA2 Decision 1  the programme was operational from 1 January 2016 to 31 December 2020 with a total budget of € 131 million distributed over the five-year period. It was open to EU countries, other EEA countries and candidate countries. In line with Article 13(3) of the ISA2 Decision, the Commission must carry out the final evaluation of the ISA2 programme by 31 December 2021. Besides fulfilling this legal obligation, the evaluation aims to improve the implementation of the  Digital Europe Programme  – which is funding public sector interoperability from 2021 onwards – and contribute to the development of  a new interoperability policy  for the EU’s public sector. 

The evaluation covers ISA2 activities from the programme’s start until October 2020 in all participating countries. The evaluation was based on an evaluation framework composed of seven evaluation criteria and 10 evaluation questions (see Annex 4). Five evaluation criteria stem from the  Commission’s better regulation  requirements and two additional criteria from the ISA2 Decision (see Box 3 ). 

2.Background on the intervention logic of the ISA2 programme

The ISA2 programme’s ultimate objective was to promote the ICT-based modernisation of the public sector in Europe and to help addressing the needs of businesses and citizens, via improved interoperability of European public administrations. End-users, citizens and businesses should benefit from common, re-usable and interoperable front-office services resulting from better integration of processes and exchange of data through the back offices of European public administrations.

More specifically the programme aimed to do the following:

Facilitate efficient and effective electronic cross-border or cross-sector interaction between European public administrations, businesses and citizens.

Contribute to the development of a more effective, simplified, and user-friendly e-administration at the national, regional, and local administration levels.

Promote a holistic approach to interoperability in the EU, by identifying, creating and operating interoperability solutions and facilitating their reuse by European public administrations. This will support the implementation of various EU policies and activities.

By working towards achieving the above objectives, the programme intended to address the problem of existing or emerging ‘electronic barriers that impede the proper functioning of the internal market’ 2 .

It is important to note that ISA2 was designed to be part of a wider policy framework related to the digitalisation of public administrations in the EU. In cooperation with the EU countries and the Commission, it promoted and supported the  European Interoperability Framework (EIF) 3 , which has been in place since 2010 and was revised in 2017. Since the start ISA2 was designed to be synergetic especially with the  Connecting Europe Facility (CEF) 4 . Operating from 2014 to 2020 CEF has been a key programme that supported the cross-border interaction between the digital services infrastructures of EU countries. Chapter  5  supplies further details of the links between ISA2 and the above-mentioned programmes and initiatives, under the coherence evaluation criterion (section  5.4 ).

The intervention logic presented in Annex 3 aims to clarify the reasoning followed by EU decision makers when setting up the ISA2 programme. It includes a detailed description of:

·the needs, problems and drivers that the programme intended to address, 

·the objectives set out for the programme (following a three-level hierarchy: global objective, specific objectives and operational objectives), 

·the inputs or activities of the programme, 

·the programme's expected results, and 

·the logical links between these components. 

The intervention logic elements and links supply a benchmark against which the final evaluation can assess the programme's achievements. The final evaluation of ISA2 follows the same intervention logic discussed in the  interim evaluation of the programme 5 . This approach ensures full comparability between the findings of the final evaluation and the interim evaluation of ISA. 

ISA2 is the fifth in a series of European Commission programmes 6  providing and promoting interoperability solutions for public administrations in the EU ISA2 succeeded the ISA programme 7 . The actions contributed to different kinds of interoperability solutions, among them open-source software, open specifications (e.g.: data models and frameworks) and managed services (e.g.: Joinup and EU Survey) based on the decision of the ISA2 committee. The results of the ISA programme represent the main baseline used for the purpose of this evaluation. More precisely, the below recommendations stemming from the final evaluation of the ISA programme 8 , serve as points for comparison:

Box 1 Conclusions and recommendations from the ISA programme evaluation 9

The interim evaluation of the ISA2 programme noted some progress concerning the above-listed areas and complemented them with recommendations to:

·go beyond national administrations when it comes to communication and awareness-raising activities thus target regional and local administrations directly.

·improve the quality of the interoperability solutions by better considering user needs.

·preserve and build on the ISA2 programme’s achievements.

3.Implementation / State of Play

How is the ISA2 programme implemented?

Up to the end of 2020, the ISA² programme supported a total of 54 actions, grouped in nine packages, which were defined on a yearly basis through an annual rolling work programme. 10  The nine packages include the following:

1.key and generic interoperability enablers;

2.semantic interoperability 11 ;

3.access to data / data sharing / open data;

4.geospatial solutions;

5.eProcurement / eInvoicing;

6.decision-making and legislation;

7.EU policies — supporting instruments;

8.supporting instruments for public administrations;

9.accompanying measures.

ISA2 managed 39 actions in its first year of operation, 43 actions in 2017, 53 actions in 2018, and 54 actions under both the 2019 and 2020 rolling work programmes. Table 1 and Table 2 provide a breakdown of the allocated budget and the number of actions per package and per year, respectively.

Table 1: Overview of allocated budget per package from 2016 to 2020 (in thousands of euros)

Note: This table presents the allocated budget per package as presented in the ISA2 Rolling Work Programmes. The Rolling Work Programmes also list non-allocated budget, which includes reserves.

Source: ISA2 rolling work programmes (2016, 2017, 2018, 2019 and 2020), indicative planning and financial overview.

Table 2: Overview of the number of actions per package from 2016 to 2020

Source: ISA2 rolling work programmes (2016, 2017, 2018, 2019 and 2020), indicative planning and financial overview.

The programme in its entirety was overseen by the Interoperability Unit of DG DIGIT (DIGIT.D2), while individual actions were implemented by different Commission services, depending on the thematic scope. The EU countries were also involved in programme governance through two channels: the ISA² Committee, the programme's high-level governing body, and the ISA² Coordination Group, a technical body mandated to ensure coherence between the programme’s actions 12 . All EU countries participated in the programme. Beyond the EU, ISA2 has four additional members — Iceland, Montenegro, North Macedonia and Norway — and an agreement of cooperation had been in place with Uruguay since March 2018 13  and a cooperation with Ukraine is in place since 2019 14 .

ISA2 actions were selected and implemented through a process consisting of four steps taken each year:

1.Submission: Commission services, EU countries, and other countries participating in ISA2 were invited to submit proposals for actions to be included in the rolling work programme via a call for proposals.

2.Evaluation: The submitted proposals were analysed by DIGIT.D2, which compiled a list of proposals that qualified to be included in the rolling work programme, based on the selection and prioritisation criteria as defined in the ISA2 Decision. At this stage, the ISA2 Committee gave an opinion on the proposed action list.

3.Adoption: The rolling work programme was adopted by the Commission and the budget got released.

4.Implementation: The actions in the adopted rolling work programme were implemented by the Commission services in charge.

The programme's overall performance and the progress of each action were frequently recorded through the quarterly and annual monitoring and evaluation reports, which fed into the  ISA2 dashboard . The dashboard is an online interactive tool that facilitates the dissemination of information about action activities and achievements, and their efficiency, effectiveness and coherence. The efficiency of actions is measured using the earned value management (EVM) analysis (see section  5.3   Efficiency ), effectiveness is presented in terms of performance indicators (see section  5.2   Effectiveness ), and the coherence of actions is mapped using network analysis (see section  5.4   Coherence ).  Figure 1  illustrates the performance until Q1 2021, as captured by the ISA2 dashboard based on the earned value management (EVM) mechanism.

Figure 1 Earned value analysis at programme level

Note: The cumulated planned value is the sum of the planned values of the programme's different actions for which the EVM is used. The cumulated earned value is the sum of the earned values of the programme's different actions for which the EVM is used. As the figure shows, the implementation of some actions continued in 2021 based on contracts signed in the previous year(s).

Source: Monitoring team of the ISA2 programme (see also the  Efficiency view of the ISA2 Programme ).

Important developments in the policy field

During the evaluation period from 1 January 2016 until 31 December 2020 digitalisation and more precisely interoperability have become more important at all levels of society and became a priority of the new elected Commission. The following developments can be highlighted:

Adoption of a new EIF

In 2017 EIF has been updated and extended as planned in the Communication on a  Digital Single Market Strategy for Europe 15  and the  eGovernment Action Plan 2016-2020 16 . The new EIF was issued through a Communication 17  and accompanied by an Interoperability Action Plan. ISA2 remained the main resource to ensure: 

·the implementation of EIF in the EU countries and sectors; 

·the implementation of the Interoperability Action Plan; 

·the monitoring of EIF’s implementation and of the Interoperability Action Plan;

·the “governance of these actions” through the ISA2. 

The new EIF gave the ISA2 programme a new policy framework with coherent objectives. The evaluation contributed to the assessment of the achievements of the EIF and more specifically its annex, the Interoperability Action Plan (IAP) ( Box 2 ).

Box 2: ISA2 contribution to the implementation of the IAP

Commitment of EU countries and the European Commission

In the evaluation period the political commitment from EU countries and European Commission has been steadily growing:

·The  Tallinn Declaration on eGovernment 18 , a declaration made by ministers in charge of eGovernment policy across the EU to spell out their commitment to a number of principles, including ‘interoperability by default’.

·The  Berlin Declaration on Digital Society and Value-Based Digital Government 19 , which shows the commitment of the EU countries, alongside EU institutions, to tap into the potential of digital public services and take a value-based approach – incorporating digital sovereignty and interoperability - to the digital transformation of the public sector. 

·In February 2020, the European Commission adopted the Communication ‘ Shaping Europe's digital future ’ 20  under the headline ambition ‘Europe fit for the digital age’. The Communication sets out as a key action the development of a ‘reinforced EU governments interoperability strategy’, aiming to foster coordination and the adoption of common standards for public services and data flows.

A growing EU regulatory environment

While the ISA2 Decision is already referring to a wide range of sectorial and cross-cutting interoperability activities on EU level, important initiatives have been brought in place after the programme’s start and more are to come in the coming months:

·2016 brought e.g. the  GDPR 21  with high impacts on processing of personal data for public services and the  Web Accessibility Directive 22 ; 

·The  Single Digital Gateway 23 , which facilitates online access to information, procedures and assistance regarding EU and national rules for citizens and businesses was introduced in 2018.

·In 2019 the  EU cybersecurity act 24  and other legislation in the field as well as the  Open Data Directive 25 . 

·In 2020 the European Commission proposed the  Digital Services Act 26 . 

Digitalisation investments

Public administrations across the EU are investing massively in the digitalisation of their services. With the experience of the COVID crisis, the need for digital solutions has been more prominent than ever. Besides ISA2 other EU programmes have contributed to these efforts in the evaluation period:

·The  Structural Reform Support Programme (SRSP) 27 , implemented from 2017 to 2020, provided technical support to the EU countries for designing and implementing reforms in various areas, including the modernisation of public administrations. Support continues in the next multi-annual financial framework through the Technical Support Instrument (TSI) for the period 2021 to 2027 28 .

·Several other EU funding programmes, like  Horizon 2020 29  (dedicated to research and innovation), the  European Social Fund (ESF) 30  and the  European Regional Development Fund (ERDF) 31 . 

4.Methodology

Fact-finding

To start the Commission outlined and agreed on the draft evaluation design (including the intervention logic and the evaluation questions). Then it reached out to external experts and asked them to support the evaluation process (see Annex 1). The consultant refined the evaluation design and — with the help and under the close monitoring of the Commission — moved to action. First, it collected data, then validated and analysed them, as explained in the following sections. At the end, the consultant summarised its findings in an  independent evaluation study  (referred to as the CEPS final study in this report) 32 .

Box 3 Evaluation framework

33 Data collection

The data collection phase involved a mix of quantitative and qualitative data collection methods and aimed to gather the evidence base to respond to the evaluation questions (EQs) specified in the evaluation framework (Annex 4). More specifically, data were drawn from two main sources: consultation activities (primary data) and desk research (secondary data). The data collection was complemented by an expert assessment, performed by five technical interoperability experts who were part of the consultant’s evaluation team. 

Primary data

Primary data were collected between December 2020 and April 2021 via the following consultation activities:

targeted in-depth interviews;

targeted online surveys;

public consultation;

feedback on the ISA2 evaluation roadmap 34 .

·two workshops, which aimed to raise awareness of the ongoing evaluation and discuss preliminary findings with stakeholders.

These activities resulted in a total of 102 responses and reached out to different types of stakeholders, ranging from actors involved in the programme’s governance to indirect beneficiaries (i.e., citizens and businesses). Overall, the respondents were characterised by a high level of expertise both generally in the field of digital public services and interoperability, as well as more specifically when it comes to the ISA2 programme, which contributed to the quality and reliability of primary data.

On average, consulted stakeholders across stakeholder groups have a very good knowledge of the field of digital public services and interoperability (with an overall average of 3.9 out of 5 for the 100 respondents who completed this question). On average the respondents involved in the governance of ISA2 are most familiar with the programme (with an average score of 4.11 out of 5, based on 9 responses), followed by ISA2 action owners (average score of 4.07 out of 5 based on 15 responses). At the opposite end of the scale, the wider public is least knowledgeable compared to the other stakeholder groups (average score of 2.94 out of 5, based on 36 responses).

Figure 2 Consulted stakeholders’ familiarity with digital public services and interoperability and the ISA2 programme

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely.

Note: 1) Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 2) Total number of respondents: 100 (Knowledge of digital public services and interoperability) and 102 (Knowledge of ISA2).

Source: CEPS final study.

Primary data provided inputs for the assessment of all evaluation criteria set out in this evaluation. In particular, the in-depth interviews as well as the targeted online surveys, which were tailored to targeted stakeholders, allowed for the collection of in-depth information for the evaluation process.

For more details, please consult Annex 2: Synopsis report of the consultation activities, which presents a breakdown of responses by consultation activity and stakeholder category along with respondent characteristics.

Chapter  5  of this report presents the aggregate results of the consultation activities using mainly bar charts showing the average scores of responses from each stakeholder group. 35  

Secondary data

The desk research reviewed the following resources:

publicly available documents and data sources, including the ISA2 annual rolling work programmes, the ISA2 dashboard, and the webpages dedicated to actions and solutions;

additional operational documents including monitoring and evaluation reports, overviews of communication activities, and lists of participants in the ISA2 Committee and Coordination Group;

·policy documents, studies and reports relevant for public sector interoperability.

Annex 6 provides an overview of the supporting evidence collected from desk research. Note that, in order to better guide the data collection activities, a sample of 21 actions was selected from the total of 54 actions included in the 2020 Rolling Work Programme 36 . The criteria used for sampling and the sampled actions are presented in Annex 5: Sample of actions. 

Data validation

The collected data were validated via triangulation in order to ensure the robustness of evidence. Tool #4 of the Commission’s Better Regulation Toolbox defines triangulation as ‘the application and combination of several research methodologies in the study of the same phenomenon’ 37 . In fact, for all evaluation criteria and questions, data were collected from multiple sources and via at least two different data collection methods (e.g.: interviews, targeted questionnaire, public consultation, desk research).

Quantitative methods of data analysis

Besides the statistical analysis of data collected via consultation activities and the qualitative analysis of open responses provided by consulted stakeholders and information contained in documentary evidence, three specific quantitative methods were used in order to evaluate the programme's efficiency:

The standard cost model (SCM) 38  is a method of assessing administrative costs imposed by rules or policies inter alia on businesses and public administrations. It is based on the identification of the basic components of a rule, the information obligations, whose costs for the addressees can be measured and quantified. An information obligation is a specific duty to gather, process or submit information to the public authority or a third party. The SCM was used to measure the costs borne by action owners in preparing and submitting proposals for ISA2 actions.

Cost-effectiveness analysis (CEA) 39  is a method of assessing the merits of a policy in an interim and ex post evaluation setting. In a nutshell, CEA measures the value-for-money of past policies, i.e. the amount of benefit generated by unitary costs. Costs are measured in monetary terms, whereas effectiveness is measured in ‘natural units’, and the unit of account varies depending on the nature of the problem addressed (e.g. the number of users of key and generic interoperability enablers). The heterogeneity of performance indicators available for ISA2 actions makes it difficult to draw conclusions about the programme's overall cost-effectiveness. Thus using CEA was only possible for certain action packages and indicators, as described in Chapter  5.3 .

·In line with the PM2 methodology developed by the Commission 40 , the earned value management (EVM) and earned schedule (ES) methods are currently used to monitor and assess the programme’s efficiency. EVM is a project management technique that helps determine work progress against a given baseline, so that costs, time, and scope of a certain activity are constantly tracked. In ISA2’s context, efficiency is assessed at action and programme levels. The implementation of EVM requires managers to calculate the earned value, i.e. a quantification of the ‘worth’ of the work done to date, and the actual costs, i.e. the executed budget for achieving the work, and to compare them with the planned value of such activity. This allows for a better understanding of the programme's performance. The ES is an extension of the EVM method which deepens the level of analysis to a ‘units of time’ layer. In the framework of the ISA2 programme, tailored versions of the EVM and ES approaches are adopted 41 .

Limitations and robustness of findings

The availability of a mix of primary and secondary data, gained through the consultation activities and the data collection (Annex 4) allowed the evaluation team to draw robust conclusions for each evaluation question. However, it is necessary to point out that existing caveats may have affected the main findings:

·One constraint consisted in the lack of direct contact with solution users. Only action owners had access to the contact details of their solution users due to confidentiality and data protection reasons. Therefore, action owners were requested to: i) invite users to complete the targeted online survey; and ii) share with DIGIT.D2 the details of those users who gave their consent to participate in interviews. This two-step approach may have limited the number of answers received from solution users. 

·Another limitation may be the potential 'consultation fatigue' of respondents and complexity of consultations. With the programme having come to an end, stakeholders may have been confronted with multiple requests for feedback. Given the technical nature of the field of interoperability, the fact that several consultations with various degrees of complexity occurred at similar times might have resulted in some stakeholders being more selective about the consultations they wished to engage in.

Compared to the interim evaluation, however, a higher number of respondents contributed to the public consultation (43 respondents in the final evaluation and 14 respondents in the interim evaluation). This result may be due to the fact that the consultation activities were run jointly with the consultations contributing to the evaluation of the EIF and the impact assessment for a future interoperability policy for the EU public sector. The joint consultations may have attracted more respondents to the ISA2 consultation. This outcome, in turn, counterbalances the low rate of responses from solution users (5 respondents in the final evaluation compared to 43 solution users in the interim evaluation).

5.Analysis and answers to the evaluation questions

In line with the evaluation framework (see Annex 4), the final evaluation focused on seven evaluation criteria: relevance, effectiveness, efficiency, coherence, EU added value, utility and sustainability. The overall analysis is based on evidence from both the external evaluation report and the Commission’s own sources. This chapter presents the analysis and provides the answers to the general evaluation questions.

Relevance

In the assessment of the criterion “relevance” the relationship between the needs and problems 42 of stakeholders and the programme’s objectives and actions is analysed as well as the question of the continued suitability of the programmes objectives. The consulted stakeholders confirmed that the original needs and problems as set out in the intervention logic (Annex 3) are still an issue for interoperability in the EU’s public sector. Almost all respondents across the stakeholder groups indicated that the needs are still present to a great extent or completely (based on average scores of 4.2 out of 5 for the need for public administrations to cooperate to enable more secure and efficient public services, and 4.1 out of 5 for the other two needs). 

Figure 3: Extent to which needs and problems originally addressed by ISA2 are currently experienced by European public administrations, businesses and/or citizens (breakdown by group of stakeholders; average score and number of respondents)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely.
Note: 1) Averages do not account for respondents answering “don’t know/no opinion” (DK/NO). 2) Total number of respondents for each need and problem from top to bottom: 101, 101, 101, 98.
Source:  CEPS final study.