EUROPEAN COMMISSION

Brussels, 23.9.2019

SWD(2019) 1615 final

COMMISSION STAFF WORKING DOCUMENT

Interim evaluation of the ISA² programme

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

Results of the interim evaluation of the ISA² programme

{COM(2019) 615 final}

Table of contents

1.Introduction

1.1.    Purpose and scope    

2.Background to the intervention

2.1.    Description of the intervention and its objectives    

2.2.    Baseline and points of comparison    

3.Implementation / State of Play

3.1.    Description of the current situation    

4.Method

4.1.    Short description of methodology    

4.2.    Limitations and robustness of findings    

5.Analysis and answers to the evaluation questions

5.1.    Relevance    

5.2.    Effectiveness    

5.3.    Efficiency    

5.4.    Coherence    

5.5.    EU added value    

5.6.    Utility    

5.7.    Sustainability    

6.Conclusions

6.1.    Context    

6.2.    Method    

6.3.    Findings    

6.4.    Next steps    

Annex 1: Procedural information

Annex 2: Synopsis report of the consultation activities

Annex 3: Intervention Logic

Annex 4: Evaluation Framework

Annex 5: Sample of actions

Annex 6: Supporting evidence from desk research

Glossary

1.Introduction

Purpose and scope

ISA2 (interoperability solutions for European public administrations, businesses and citizens) is an EU programme that supports the development of digital solutions and common frameworks that enable public administrations, businesses and citizens in Europe to benefit from interoperable cross-border and cross-sector public services.

The programme’s primary beneficiaries are European public administrations at all levels - EU, national, regional and local levels - by reusing the solutions offered by ISA2 they can provide better – more interoperable, user-centric and digital – public services. However, the programme also benefits a wider group of stakeholders, including EU businesses and citizens.

ISA2 is open to EU Member States, other countries of the European Economic Area and candidate countries. In addition to the 28 EU Member States, three other countries take part in the programme: Iceland, Norway (since 2016) and Montenegro (since 2018). The programme encourages cooperation with other non-EU countries and with international organisations and bodies.

The ISA2 programme — established by the ISA2 Decision 1 — has a budget of €130.9 million and runs for five years from 1 January 2016 until 31 December 2020. In line with Article 13(3) of the ISA2 decision, the Commission must carry out an interim evaluation of the ISA2 programme by 30 September 2019. Besides fulfilling this legal obligation, the evaluation aims to improve the implementation of ISA2 and any of its successor programmes.

The evaluation covers ISA2 activities from the programme’s start until the third quarter of 2018 in all participating countries. The evaluation framework builds on the five evaluation criteria stemming from the Commission’s better regulation requirements: relevance, effectiveness, efficiency, EU added value and coherence. As shown in Box 1 , it also addresses utility and sustainability in line with the ISA2 Decision.

Box 1: Evaluation framework

2.Background to the intervention

Description of the intervention and its objectives

The ISA2 programme’s ultimate objective is to promote the ICT-based modernisation of the public sector in Europe and to facilitate addressing the needs of businesses and citizens, via improved interoperability of European public administrations.

More specifically the programme aims to do the following:

Facilitate efficient and effective electronic cross-border or cross-sector interaction between European public administrations, businesses and citizens.

Contribute to the development of a more effective, simplified and user-friendly e-administration at the national, regional and local administration levels.

Promote a holistic approach to interoperability in the EU, by identifying, creating and operating interoperability solutions and facilitating their reuse by European public administrations. This will support the implementation of various EU policies and activities.

By working towards achieving the above objectives, the programme intends to address the problem of existing or emerging ‘electronic barriers that impede the proper functioning of the internal market’ 2 .

It is important to note that ISA2 is part of a wider policy framework related to the digitalisation of public administrations in the EU. In cooperation with the Member States and the Commission, the ISA2 programme promotes and maintains the European Interoperability Framework (EIF) 3 , which has been in place since 2010 and was revised in 2017. The revision of the EIF was called for in the Communication on a  Digital Single Market Strategy for Europe 4 , which recognised the role of interoperability in cross-border and cross-sector connections between communities, public services and public administrations. Other initiatives related to the digitalisation of public administrations to which ISA2 contributes include:

The eGovernment Action Plans (the current version of which runs from 2016 to 2020), which aim to make ‘public administrations and institutions in the EU open, efficient and inclusive, providing borderless, personalised, user-friendly, end-to-end digital public services to all citizens and businesses in the EU’ 5 .

The 2017 Tallinn Declaration on eGovernment 6 , a declaration made by ministers in charge of eGovernment policy across the EU to spell out their commitment to a number of principles, including ‘interoperability by default’, which aims to ensure that public services are designed to work seamlessly within the single market and across organisational silos, relying on the free movement of data and digital services in the EU.

The Connecting Europe Facility (CEF) 7 established in 2013 as a funding programme providing support to cross-border interaction between the digital services infrastructures of Member States. Based on interoperability agreements between Member States, CEF Digital provides building blocks that ensure that citizens, businesses and administrations can benefit from seamless digital public services across the EU.

The Single Digital Gateway 8 , to provide citizens and businesses with online access to information and procedures and to assistance and problem-solving services.

The Structural Reform Support Programme (SRSP) 9 , which provides support to Member States in five main reform areas, including IT capacity building in public administrations.

·Several other EU funding programmes, like Horizon 2020 10 (dedicated to research and innovation), the European Social Fund (ESF) 11 and the European Regional Development Fund (ERDF) 12 .

Chapter 5 provides further details of the links between ISA2 and the above-mentioned programmes and initiatives, under the coherence evaluation criterion (section 5.4).

The intervention logic aims to clarify the reasoning followed by EU decision makers when establishing the ISA2 programme. It includes a detailed description of:

·the needs, problems and drivers that the programme intended to address;

·the objectives set out for the programme (following a three-level hierarchy: global objective, specific objectives and operational objectives);

·the inputs or activities of the programme;

·the programme's expected results; and

·the logical links between these components.

The intervention logic elements and links provide a benchmark against which the interim evaluation can assess the programme's achievements in the first half of its period of operation. Figure 1 provides an overview of the elements of the intervention logic and the links between them, and further details are provided in Annex 3: Intervention Logic .

Figure 1: Intervention logic of the ISA2 programme

Baseline and points of comparison

ISA2 is the fifth in a series of EU programmes 13 providing and promoting interoperability solutions for public administrations in the EU. ISA2 succeeded the ISA programme, which implemented 51 actions during its 6-year period of operation (from 2010 to 2015) with a total budget of around €164 million 14 . The results of ISA represent the main baseline used for the purpose of this interim evaluation.

The end of the ISA programme was marked by a final evaluation 15 , which took stock of the programme’s achievements and put forward three recommendations for the future edition of the programme, focusing on the below areas:

communication with public administrations and raising their awareness of interoperability;

a holistic approach to and cross-cutting view of interoperability; and

·cooperation with other EU policies and initiatives.

Although the present interim evaluation focuses on results achieved under ISA2, it also addresses the above-listed recommendations. It is also worth noting that ISA2 aims to achieve its objectives through a number of actions that continue from the previous edition of the programme, as well as through new actions. Where relevant, the impact of ISA on ISA2 was assessed by comparing the performance of actions continued from ISA with that of actions started during ISA2.

3.Implementation / State of Play

Description of the current situation

ISA² has been operational from 1 January 2016 and will continue to run until 31 December 2020. It has a total budget of around €131 million. ISA2 consists of nine groups of actions, called packages, which are defined on a yearly basis through an annual revision of the rolling work programme (rolling work programme):

key and generic interoperability enablers;

semantic interoperability 16 ;

access to data / data sharing / open data;

geospatial solutions;

eProcurement / eInvoicing;

decision-making and legislation;

EU policies — supporting instruments;

supporting instruments for public administrations; and

·accompanying measures.

The 2018 rolling work programme consists of 53 actions, categorised into the nine packages mentioned above. ISA2 managed 39 actions in its first year of operation, 43 actions in 2017, 53 actions in 2018, and is currently managing 54 actions under the 2019 rolling work programme. Table 1 provides a breakdown of actions and allocated budget per package and per year. As ISA2 is scheduled to end on 31 December 2020, one additional rolling work programme remains to be adopted. The present evaluation is based on the rolling work programme implemented in 2016, 2017 and 2018, as the 2019 rolling work programme had not yet been adopted when the data collection took place.

Table 1: Overview of allocated budget and number of actions per package from 2016 to 2018

Source: ISA2 rolling work programmes (2016, 2017, 2018), indicative planning and financial overview.

The programme in its entirety is overseen by the Interoperability Unit of DG DIGIT (DIGIT.D2), while individual actions are implemented by different Commission services, depending on the thematic scope. The Member States are also involved in programme governance through two channels: the ISA² Committee, the programme's high-level governing body, and the ISA² Coordination Group, a technical body mandated to ensure coherence between the programme’s actions 17 . All 28 EU Member States participate in the programme. Beyond the EU, ISA2 has three additional countries — Iceland, Montenegro and Norway — participate in ISA2. In addition, an administrative agreement between DG DIGIT and the Agency for the Development of the Government of Electronic Management and the Information and Knowledge Society of Uruguay has been in place since March 2018 18 .

ISA2 actions are selected and implemented through a process consisting of four steps taken each year:

Submission: Commission services, Member States, and other countries participating in ISA2 are invited to submit proposals for actions to be included in the rolling work programme via a call for proposals.

Evaluation and consultation of the ISA2 Committee: The submitted proposals are analysed by DIGIT.D2, which compiles a list of proposals that qualify to be included in the rolling work programme, based on the selection and prioritisation criteria as defined in the ISA2 Decision. The ISA2 Committee gives an opinion on the proposed action list.

Adoption: The annual rolling work programme is adopted by the Commission and the budget is released.

·Implementation: The actions in the adopted annual rolling work programme are implemented by the Commission services in charge.

The programme's overall performance and the progress of each action are frequently recorded through the quarterly and annual monitoring and evaluation reports, which feed into the ISA2 dashboard 19 . The dashboard is an online interactive tool that facilitates the dissemination of information about action activities and achievements, and their efficiency, effectiveness and coherence. The efficiency of actions is measured using the earned value management (EVM) analysis (see section 5.3 Efficiency ), effectiveness is presented in terms of performance indicators (see section 5.2 Effectiveness ), and the coherence of actions is mapped using network analysis (see section 5.4 Coherence ). Figure 2 illustrates the performance of the ISA2 programme until Q1 2019, as captured by the ISA2 dashboard based on data from the EVM mechanism.

Figure 2: Earned value analysis at programme level

Note: The cumulated planned value is the sum of the planned values of the programme's different actions for which the EVM is used. The cumulated earned value is the sum of the earned values of the programme's different actions for which the EVM is used. The budget at completion (BAC) is the sum of all undergoing and completed specific contracts financed by the ISA2 budget.

Source: Monitoring team of the ISA2 programme (see also the Efficiency view of the ISA2 Programme ).

4.Method

Short description of methodology

As a first step, the Commission outlined and agreed on the draft evaluation design (including the intervention logic and the evaluation questions). It then reached out to external experts and asked them to support the evaluation process (see Annex 1 ). The consultant refined the evaluation design and — with the help and under the close monitoring of the Commission — moved to action. First, it collected data, then validated and analysed them, as explained in the following sections. At the end, the consultant summarised its findings in an independent evaluation study (referred to as the CEPS final study in this report) 20 .

Data collection

The data collection phase involved a mix of quantitative and qualitative data collection methods and aimed to gather the evidence base to respond to the evaluation questions (EQs) specified in the evaluation framework ( Annex 4 ). More specifically, data were drawn from two main sources: consultation activities (primary data) and desk research (secondary data).

Primary data

Primary data were collected between 29 November 2018 and 1 March 2019 via the following consultation activities 21 :

short questionnaire distributed during the ISA2 Mid-Term Conference and the kick-off workshop;

public consultation;

targeted online surveys;

targeted in-depth interviews; and

·kick-off workshop.

These activities resulted in a total of 129 responses from stakeholders, out of which 14 replies arrived during the public consultation. The low consultation response rate is mainly explained by two reasons:

ISA2 is a technical programme producing solutions addressed mainly to European public administrations. Understanding how the programme works requires specific knowledge in the field. Therefore, direct contact between the average citizen/business and the programme tends to be very limited, and this most probably led to the reduced response rate. In order to increase the participation of citizens/businesses, a short questionnaire with the same questions as those of the public consultation was distributed during the ISA2 Mid-Term Conference, which resulted in 15 additional responses. It is worth noting, however, that businesses and citizens benefit from ISA2 solutions and activities indirectly, through their interactions with public administrations. Thus, a low response rate among these groups of stakeholders does not reflect negatively on the programme.

·Stakeholder groups that are in direct contact with the programme (e.g. solution users, action owners, etc.) were consulted via targeted online surveys and interviews, which covered inter alia the same topics addressed by the public consultation. Therefore, stakeholders belonging to these groups were asked not to respond to the public consultation in order to avoid duplicate answers.

Despite the relatively low response rate, the consultation activities allowed for a wide coverage of the different stakeholders of the ISA2 programme, ranging from actors involved in programme governance to citizens, i.e. the indirect beneficiaries of ISA2. Moreover, on average, the respondents were characterised by a high level of expertise, which contributed to the quality and reliability of primary data collected. The in-depth interviews and targeted surveys provided particularly useful input to the interim evaluation. They were tailored to stakeholders closely related to the programme and allowed them to provide extensive details, thus making it possible for the evaluation to get deep insights into the subject areas covered. In general, primary data contributed to the assessment of all evaluation criteria. Overall, the answers collected are consistent across stakeholder groups.

For more details, please consult Annex 2: Synopsis report of the consultation activities , which presents a breakdown of responses by consultation activity and stakeholder category along with respondent characteristics.

Chapter 5 of this report presents the results of the consultation activities using mainly bar charts with averages. The results of the public and targeted consultation activities are combined and presented in aggregated form throughout the evaluation. For each question, average scores were computed for each stakeholder group that provided feedback on the specific question. Averages do not take into account those respondents who answered ‘don’t know/no opinion’ to the question under analysis. The data labels of the bar charts show the average score first, then the number of respondents in brackets.

Secondary data

The desk research reviewed the following resources:

publicly available documents and data sources, including the ISA2 annual rolling work programmes, the ISA2 dashboard, and the webpages dedicated to actions and solutions; and

·additional operational documents including monitoring and evaluation reports, overviews of communication activities, and lists of participants in the ISA2 Committee and Coordination Group.

Annex 6 provides an overview of the supporting evidence collected from desk research.

Note that, in order to better guide the data collection activities, a sample of 20 actions was selected from the total of 53 actions funded by the ISA2 programme up to 2018. The criteria used for sampling and the sampled actions are presented in Annex 5: Sample of actions .

The desk research, review of the programme's operational documents and targeted in-depth interviews focused on the 20 sampled actions. However, stakeholders involved in all sampled actions had the opportunity to respond to either the targeted online surveys or the public consultation, to participate in the kick-off workshop and to complete the short questionnaire distributed during the ISA2 Mid-Term Conference.

The data collection was complemented by an expert assessment of the programme, performed by four technical interoperability experts who were part of the consultant’s evaluation team.

Data validation

The collected data were validated via triangulation in order to ensure the robustness of evidence. The Commission’s Better Regulation Toolbox defines triangulation as ‘the application and combination of several research methodologies in the study of the same phenomenon’ 22 . In fact, for all evaluation criteria and questions, data were collected from multiple sources and via at least two different data collection methods (e.g. interviews, targeted questionnaire, public consultation, desk research).

Quantitative methods of data analysis

Besides the statistical analysis of data collected via consultation activities, desk research and the qualitative analysis of open responses provided by consulted stakeholders and information contained in documentary evidence, three specific quantitative methods were used in order to evaluate the programme's efficiency:

The standard cost model (SCM) 23 is a method of assessing administrative costs imposed by rules or policies inter alia on businesses and public administrations. It is based on the identification of the basic components of a rule, the information obligations, whose costs for the addressees can be measured and quantified. An information obligation is a specific duty to gather, process or submit information to the public authority or a third party. The SCM was used to measure the costs borne by action owners in preparing and submitting proposals for ISA2 actions.

Cost-effectiveness analysis (CEA) 24 is a method of assessing the merits of a policy in an interim and ex post evaluation setting. In a nutshell, CEA measures the value-for-money of past policies, i.e. the amount of benefit generated by unitary costs. Costs are measured in monetary terms, whereas effectiveness is measured in ‘natural units’, and the unit of account varies depending on the nature of the problem addressed (e.g. the number of users of key and generic interoperability enablers). The heterogeneity of performance indicators available for ISA2 actions makes it difficult to draw conclusions about the programme's overall cost-effectiveness and thus using CEA was only possible for certain action packages and indicators, as described in Chapter 5.

·In line with the PM2 methodology developed by the Commission 25 , the earned value management (EVM) and earned schedule (ES) methods are currently used to monitor and assess the efficiency of the ISA2 programme. EVM is a project management technique that helps determine work progress against a given baseline, so that costs, time, and scope of a certain activity are constantly tracked. In the context of the ISA2 programme, efficiency is assessed at action and programme levels. The implementation of EVM requires managers to calculate the earned value, i.e. a quantification of the ‘worth’ of the work done to date, and the actual costs, i.e. the executed budget for achieving the work, and to compare them with the planned value of such activity. This allows for a better understanding of the programme's performance. The ES is an extension of the EVM method that deepens the level of analysis to a ‘units of time’ layer. In the framework of the ISA2 programme, tailored versions of the EVM and ES approaches are adopted 26 .

Limitations and robustness of findings

The consultation strategy 27 and the data collection plan put forward in the Evaluation Framework ( Annex 4 ) were fully implemented. Therefore, the evaluation was able to draw robust conclusions for all EQs based on both primary and secondary data. Nevertheless, some emphasis needs to be placed on caveats that may have an impact on the main findings of the interim evaluation as well as on future evaluation exercises:

The timing of this evaluation is compliant with Article 13 of the ISA2 Decision, which requires the Commission to present an interim evaluation to the European Parliament and the Council by 30 September 2019. Given this constraint, the evaluation confined the analysis to actions funded during the programme's first three years (2016, 2017 and 2018). In particular, some actions under evaluation are still ongoing; hence, their outputs and outcomes can only be recorded to some extent. On a more general note, as discussed in the intervention logic ( Annex 3 ), impacts occur over the long term and it is difficult to capture them in an interim evaluation. In principle, in the coming years ISA2 actions may see improvements 28 in the way they achieve the programme’s objectives and deliver the expected results.

Only action owners had access to the contact details of their solution users due to confidentiality and data protection reasons. Therefore, action owners were requested to: i) invite users to complete the relevant targeted online survey; and ii) share with DIGIT.D2 the details of those users who gave their consent to participate in interviews. This two-step approach may have limited the number of answers received from solution users. In addition, some action owners explained that they were unable to contact their users for the purpose of evaluation, as the users’ consent to be involved in consultation activities was never collected. Against this background, ISA2 may consider requesting its users to provide their consent to be contacted by Commission staff and contractors for evaluation purposes in the future.

5.Analysis and answers to the evaluation questions

In line with the Evaluation Framework ( Annex 4 ), the interim evaluation focused on seven evaluation criteria: relevance, effectiveness, efficiency, coherence, EU added value, utility and sustainability. The below chapters present the following for each of them: a) the pre-defined evaluation questions (EQ) and related judgment criteria; b) the results of the analysis fed by the collected data; and finally c) the answer to the EQ.

Relevance

Evaluation framework for 'relevance'

Relevance refers to the alignment between the programme's objectives and the evolving needs and problems 29 experienced by stakeholders. As shown in Box 1 , the interim evaluation translated this criterion into the following EQ:

EQ1:To what extent are the objectives of the ISA² programme still pertinent in relation to the evolving needs and problems at both national and EU levels?

To answer this question, the evaluation assessed two main judgment criteria (JCs), by relying on data collected via consultation activities as well as on expert assessment:

JC1:Alignment between needs and problems addressed by the programme and current needs and problems experienced at the EU, national and subnational level; and

JC2:Alignment between the objectives of the programme and current needs and problems experienced at the EU, national and subnational level.

Analysis

JC1: Alignment between original and current needs and problems

The rapidly evolving nature of ICT and interoperability begs the question as to whether the needs and problems originally targeted by the programme are the same as the current needs and problems experienced by stakeholders.

Figure 3 shows the average scores derived from the answers received on this question, per stakeholder group. Overall, more than 70 % of respondents consider that the three original needs, as presented in Figure 3 , are still relevant to a high extent or to the fullest extent (93 out of 128 respondents for the first need, 99 out of 127 respondents for the second need, and 94 out of 125 respondents for the third need). The persistence of the problem of administrative e-barriers originally identified by the programme is confirmed as well. Two-thirds of the respondents (79 out of 118) consider this problem still relevant to a high extent or the fullest extent, in comparison to almost 13 % (15 out of 118) who believe the problem is relevant only to a limited extent or not at all.

Figure 3: Extent to which needs and problems originally addressed by ISA2 are currently experienced by European public administrations, businesses and/or citizens (average score of answers, number of respondents)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent.

Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO).

Source: CEPS final study

There are some additional needs and problems that the programme could aim to address, as pointed out by 39 % of the consulted stakeholders (46 out of 129). More specifically, several respondents converged on the following needs and problems related to interoperability of public services that European public administrations, businesses and/or citizens currently experience:

EU level:

The need for a more binding legal framework for interoperability 30 , including the need for legally binding standards or the general need for a more binding approach to interoperability in the EU.

The need for a more prescriptive approach to designing interoperable public services (‘interoperability by design’). It is worth noting, however, that the revised EIF adopted in 2017 explicitly mentions ‘interoperability by design’ as the conceptual model for interoperable public services 31 . Therefore, this need is taken into account through the wider framework of the EIF.

The need to improve the way administrations communicate with one another, given the different forms of organisation and functioning of public administrations in the Member States.

The need to share best practices.

·The need to account for new developments such as blockchain, privacy-by-design and self-sovereign identities 32 , which are changing the interoperability landscape.

National/subnational level:

Resource constraints, such as shortage of qualified IT staff, experienced by national and local public administrations.

The different political priorities among Member States hindering a consistent approach to interoperability in the EU.

·The limited awareness of ISA2 and other initiatives related to interoperability, especially at the regional and local levels 33 .

JC2: Alignment between needs and problems and the programme's objectives

After ensuring that all the original needs and problems are still relevant and identifying additional needs and problems experienced by stakeholders, the evaluation checked the degree to which the programme's objectives address these needs and problems.

The majority of consulted stakeholders agree that, through its general objective 34 , ISA2 addresses the main needs and problems identified in the field of interoperability of digital public services at the time the programme was set up, either to a high extent or to the fullest extent ( Figure 4 ). This result holds true across all stakeholder groups. Over 80 % of respondents indicate that, by achieving its general objective, ISA2 can address the original needs to a high extent or to the fullest extent (103 out of 125 respondents for the first need, 103 out of 123 respondents for the second need, and 103 out of 124 respondents for the third need). When it comes to the problem of administrative e-barriers, approximately two-thirds of respondents (77 out of 116) agree that ISA2 can address the problem through its general objective, while only 11 % (13 out of 116) believe that the programme can only address the problem to a limited extent or not at all. These results are also confirmed by the independent assessment carried out by technical experts, who pointed out that ISA2 highlights the topic of interoperability and helps build a community of professionals in this area, thus addressing the needs and problems identified in the intervention logic.

Figure 4: Extent to which achieving ISA2 objectives contributes to addressing the needs and problems originally addressed by the programme (average score of answers, number of respondents)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent.

Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO).

Source: CEPS final study

When it comes to the new needs and problems identified by consulted stakeholders, on average, respondents believe that the programme, through its general objective, could address them at least to some extent ( Figure 5 ). However, three out of the six respondents responsible for linked EU initiatives who indicated additional needs and problems consider that the general objective of ISA2 cannot address the additional needs and problems, or can only address them to a limited extent, as some of the issues mentioned fall outside the programme's scope (for instance, Member States' different political priorities).

Figure 5: Extent to which achieving ISA2 objectives contributes to addressing additional (current) needs and problems identified by consulted stakeholders (number of respondents by stakeholder category)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent; (DK/NO) do not know/no opinion.

Source: CEPS final study

Answer to the EQ on 'relevance'

EQ1: To what extent are the objectives of the ISA² programme still pertinent in relation to the evolving needs and problems at both national and EU levels?

The original needs and problems that the programme intended to address are still fully relevant and the objectives of ISA2 are pertinent to addressing them. In addition, consulted stakeholders have identified a number of new needs and problems related to the interoperability of digital public services. Some of these fall outside the programme's scope (such as resource constraints or different political priorities at national level). Others are already being addressed by existing ISA2 actions (for example, the 'Innovative Public Services' action provides support for identifying the innovation potential and framing conditions of emerging disruptive technologies). Finally, ISA² could (better) address some of these additional needs and problems by considering the below measures for future actions and future editions of the programme:

Increasing awareness of interoperability beyond national administrations by fostering more collaboration and exchanges of views with regional and local administrations. The recently launched Interoperability Academy action could help with this by developing training materials tailored to the needs of regional and local governments.

Performing studies to assess the costs and benefits of implementing ISA2 solutions. The findings of such studies could feed into promotional activities, targeting the sub-national layer of public administrations as well.

Improving the sharing of best practices among public administrations, academia and interested professionals thus raising awareness of interoperability and increasing the take-up of ISA2 solutions across Member States.

Relying more on potential influencers (like researchers, teachers or committed public servants) when communicating about interoperability. This could eventually create a community of interoperability ambassadors.

Building advisory capabilities around interoperability. These would help interested stakeholders to pick the appropriate interoperability solutions that match their needs, and would at the same time provide support services and technical assistance for effective implementation.

·Assessing the rationale and impacts of a possible binding interoperability instrument. Such assessment should rely on the findings of the final evaluation of the ISA2 programme and on the evidence that will be gathered when the implementation of the EIF is evaluated in 2021.

Effectiveness

Evaluation framework for 'effectiveness'

Effectiveness focuses on the extent to which the programme has achieved its objectives and generated the expected results. Analysing the effectiveness of ISA2 requires answers to two EQs:

EQ2:How far are the ISA² programme’s results in the process of achieving the programme’s objectives?

EQ3:Are there aspects that are more or less effective than others are, and if so, what lessons can be drawn from this?

To answer these questions, the evaluation applied the following judgment criteria (JCs):

JC1:Alignment between the programme's actual results, objectives and expected results;

JC2:Impact of external factors on the programme's performance;

JC3:Awareness of the programme;

JC4:Performance indicators; and

JC5:Alignment with principles specified in Article 4(b) of the ISA2 Decision.

The analysis relies on data collected via consultation activities, a desk review and an expert assessment.

Analysis

JC1: Alignment between the actual results, objectives and expected results of ISA2

The focus of this judgment criterion falls on the solutions developed by ISA2, as they represent the most tangible results achieved so far. Figure 6 shows the extent to which the programme's actual results are aligned with its objectives, according to the consulted stakeholders. 45 % of respondents (49 out of 109) agree that ISA2 solutions have contributed to achieving the programme's general objective to a high extent or to the fullest extent. In contrast, only 15 % of respondents (16 out of 109) believe that ISA2 solutions have made either no contribution or a limited contribution. The positive feedback is higher when it comes to the programme's specific objectives. For instance:

More than 56 % of respondents (59 out of 106) confirm that the solutions developed have contributed to developing, maintaining and promoting a holistic approach to interoperability in the Union (specific objective 1) to a high extent or to the fullest extent;

·More than 56 % of respondents (60 out of 108) indicate that the solutions developed have contributed to facilitating the reuse of interoperability solutions by European public administrations (specific objective 5) to a high extent or to the fullest extent.

However, respondents to the consultation activities express a lower level of agreement regarding the contribution to the development of a more effective, simplified and user-friendly e-administration at the national, regional and local levels of public administration (specific objective 3). Only 34 % of respondents (36 out of 106) consider that ISA2 solutions have contributed to this objective to a high extent or to the fullest extent, while 22 % (23 out of 106) believe that there has been either no contribution or only a limited contribution. This is aligned with the assessment of the EU added value criterion (see Figure 17 in section 5.5), which shows that Member States’ initiatives tend to perform relatively better when it comes to achieving specific objective 3.

Figure 6: Extent to which ISA2 solutions contributed to achieving the programme’s objectives (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

The desk review of secondary data also shows good progress towards the achievement of ISA2 objectives. The 20 sampled actions have already produced 35 solutions, and a further 21 solutions are under development. Most of the operational solutions are used across the 28 Member States and across different EU institutions, thus contributing to a coherent approach to interoperability in the EU ( Annex 6 : Table 9 , Table 10 and Table 11 ).

The Interinstitutional Register for Delegated Acts 35 is a good example that illustrates how the various ISA2 solutions help achieve the programme’ objectives. It was set up under the ‘RegDel’ action 36 in order to respond to the joint commitment of the Commission, the Council and the European Parliament to facilitate access to delegated acts through a dedicated register. By doing so, the register also ‘supports the implementation of Union policies and activities’ as envisaged by specific objective 4 of the ISA2 programme. Launched in December 2017, the solution has amassed 2202 active subscriptions and over 144 000 views (up to 1 February 2019) 37 .

Another example is the European Interoperability Reference Architecture (EIRA©) 38 , which helps classify and organise interoperability building blocks used in the delivery of digital public services. EIRA facilitates the reuse of interoperability solutions by European public administrations (specific 5 of ISA2) and is appreciated by its users, as shown in Box 2 .

Box 2: EIRA user testimonial

Source: ISA2 communication team

Other examples include the ‘Joinup’ solution 39 and events such as the ‘Sharing and Reuse Conference’ 40 that facilitate access to and the reuse of interoperability solutions, as well as the exchange of best practices.

Turning to the operational objectives ( Figure 7 ), the highest achievements of ISA2 solutions pertain to the following ones:

Operational objective 7 – 68 % of respondents (42 out of 62) confirmed that the existing solutions have contributed to supporting and promoting the maintenance, updating, promotion and monitoring of the implementation of the EIS, the EIF and the EIRA to a high extent or to the fullest extent, and only 1 % (1 out of 62) believe that there was a limited contribution. One good example is the European Interoperability Architecture (EIA), which has developed a conceptual reference architecture that is already deployed in eight Member States. Moreover, the European Interoperability Reference Architecture (EIRA) solution that is available on the ‘Joinup’ platform has been downloaded 1 364 times so far ( Annex 6 : Table 12 ). In addition, the National Interoperability Framework Observatory contributes to this objective by assessing interoperability developments in Member States and national alignment with the EIF.

·Operational objective 9 — according to 68 % of respondents (50 out of 73), ISA2 solutions have contributed to supporting and promoting the maintenance and publication of a platform allowing access to and collaboration with regard to best practices, functioning as a means of raising awareness and disseminating available solutions to a high extent or to the fullest extent, compared to 11 % (8 out of 73) who believe that the contribution has been limited. The ‘Joinup’ action, by maintaining and further developing the collaborative platform with the same name, is directly related to this operational objective. The platform counts over 13 000 members, namely professionals working in the field of eGovernment. It allows members to work together on over 109 thematic working spaces, called ‘collections’ ( Annex 6 : Table 12 ).

Based on the feedback received, there are two areas for potential improvement:

·Operational objective 4 — only 23 % of respondents (15 out of 64) believe that ISA2 solutions have contributed to supporting and promoting the identification of legislation gaps, at Union and national level, that hamper cross-border or cross-sector interoperability between European public administrations to a high extent or to the fullest extent.

·Operational objective 5 — only 25 % of respondents (17 out of 68) indicate that ISA2 solutions have contributed to supporting and promoting the development of mechanisms that measure and quantify the benefits of interoperability solutions including methodologies for assessing cost-savings to a high extent or to the fullest extent.

The fact that the ‘Legal Interoperability’ action, responsible for the advancement of these two operational objectives, has until now mainly focused on the Commission’s internal policy-making process, may explain the lower awareness levels among the programme’s stakeholders.

Figure 7: Extent to which ISA2 solutions contributed to achieving the programme’s operational objectives (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

As the programme has only just passed its midway point, it is reasonable to assume that its actual results might not fully match the expected results, since actions are ongoing and solutions will continue to be developed until the programme ends in 2020. Experts confirmed this assumption, pointing out that ISA2 solutions have contributed to, for instance, ‘a coherent interoperability landscape in the Union’, but more time is needed for the expected results to be fully realised. The feedback received as part of the consultation activities provides further evidence of this ( Figure 8 ).

Figure 8: Extent to which ISA2 solutions achieved the programme's expected results (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

The developed solutions are mostly aligned with the expected result of a coherent interoperability landscape in the EU based on a holistic approach to interoperability. In fact, around 50 % of respondents (37 out of 77) believe that ISA2 solutions have achieved this expected result to a high extent or to the fullest extent, in comparison to 8 % (6 out of 77) who argue that there has been limited achievement or no achievement in this area. The achievement of expected results is further assessed below, when the performance indicators are presented.

Turning to areas of potential improvement, only 30 % of respondents (23 out of 76) consider that ISA2 solutions led to improved efficiency and productivity of the European public administrations. Interestingly, during the interviews some respondents pointed out that it is difficult to reply to this specific question, as no specific studies have been carried out on the efficiency and productivity effects of ISA2 solutions, although such effects are expected. Future actions may aim to rectify this and ensure that impacts generated by ISA2 solutions for public administrations are identified and quantified.

JC2: Impact of external factors on the programme's performance

In line with the Evaluation Framework, assessing the programme's effectiveness also implies considering the positive and negative impacts of external factors on the programme's overall performance.

To this end, consulted stakeholders were asked to identify and rank the external factors that on the one hand contribute to and on the other hand jeopardise the programme's performance. There is a clear pattern in the answers of all groups consulted on this matter (programme governance stakeholders, action owners and solution users). The calls for common standards and frameworks from public administrations are considered particularly beneficial to the programme's performance ( Figure 9 ), as they can guide the development of new solutions to address the issues raised by public administrations, i.e. the direct beneficiaries of the solutions. Most respondents, 81 % (63 out of 75), believe that these calls contribute to the performance of ISA2 to a high extent or to the fullest extent. This also shows that interoperability is among the priorities of European public administrations.

Figure 9: Extent to which the following external factors contribute to the performance of ISA2 (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

On the other hand, institutional complexity is the external factor seen as having the greatest negative impact on the programme's performance ( Figure 10 ). 78 % of respondents (64 out of 82) believe that this factor jeopardises the programme's performance to a high extent or to the fullest extent. Institutional complexity comes in the form of multiple layers of governance and differences between national public administrations in various Member States as well as between local public administrations within Member States, all of which can cause coordination issues. Technical experts confirmed this result by identifying institutional and legal complexity as the factors that most jeopardise the programme’s performance.

Figure 10: Extent to which the following external factors jeopardise the performance of ISA2 (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

JC3: Awareness of the programme

When analysing the feedback collected via the consultation activities, each stakeholder group's level of awareness of interoperability issues and, more specifically, of the ISA2 programme, must be taken into consideration. As illustrated in Annex 2 ( Figure 30 and Figure 31 ), the consulted stakeholders were asked to assess their knowledge of digital public services and interoperability as well as their knowledge of ISA2. On average, the results show that most respondents have expert knowledge of interoperability and of ISA2. 71 % of respondents (91 out of 128) are familiar with digital public services and interoperability to a high extent or to the fullest extent, in comparison to 11 % (14 out of 128) who either are not familiar with the field or are familiar with it only to a limited extent. In addition, 63 % (81 out of 128) of respondents are familiar with the ISA2 programme to a high extent or to the fullest extent, compared to 20 % (25 out of 128) who report only a low degree of familiarity or no familiarity with the programme.

One factor that explains these positive awareness results is the ISA2 programme’s strong commitment to ‘enhance information and increase awareness with regard to the programme and its benefits’, as stated in Article 3 of the ISA2 Decision. In fact, raising public administrations’ awareness of interoperability was one of the three recommendations stemming from the final evaluation of the ISA programme 41 . The ISA2 programme addressed this by adopting its communication strategy and engagement plan in 2017 42 and by increasing its presence at conferences and other events, as shown in Box 3 .

The relatively low participation in the interim evaluation’s consultation activities and the need to better liaise with regional and local administrations, identified under the Relevance criterion (see section 5.1), call for continued and even improved awareness raising efforts.

JC4: Performance indicators

In line with the Evaluation Framework, the desk research collected data on several performance indicators ( Annex 6 : Table 12 ) in order to assess the extent to which the programme’s operational objectives had been achieved and to complement the data gathered via consultation activities.

When talking about performance indicators, it is important to distinguish between different categories of ISA2 actions:

actions continued from ISA and actions started during ISA2; and

·actions already completed at the time of the interim evaluation and actions still ongoing.

The different stages of an action imply different levels of achievement of objectives and expected results, as some solutions are operational while others are in the early proof-of-concept stage. This should be kept in mind when looking at the performance indicators’ results ( Box 4 ).

Data show that the 20 sampled actions have developed 35 solutions so far, out of which 10 emerged under ISA2. As shown in Table 2 , the solutions fall into one of the following four categories: common tools/services, common frameworks, common specifications/standards, and studies.

Table 2: Overview of solutions developed under the sampled actions, by type

Source: CEPS final study

Aside from these four solution types associated with ISA2 actions, there is also a particular case represented by the ‘Raising Interoperability Awareness — Communication Activities’ action, which is part of the ‘Accompanying measures’ package. There are no solutions associated with this action, and its outputs are of a specific nature, namely events organised or events to which ISA2 representatives actively contributed. Box 3 presents an overview of these communication activities.

Box 3: Communication activities

An array of metrics are in use to assess the effectiveness of solutions, for example, the number of users, instances of use, downloads and page views. When these performance indicators are assessed in a more in-depth way, it is clear that they support the feedback received from consulted stakeholders that the programme is on track to achieving the expected results. However, it is also clear that more time is needed to fully achieve these results. This pattern is also confirmed when the performance indicators for actions continued from ISA are compared with those for actions started during ISA2, as detailed in Box 4 .

Box 4: Comparing solutions and performance indicators of actions continued from ISA and actions started during ISA2

JC5: Alignment with principles specified in Article 4(b) of the ISA2 Decision

The evaluation of effectiveness also requires an assessment of the programme’s compliance with the principles listed in Article 4(b) of the ISA2 Decision. The rolling work programme provides details on how the ISA2 actions should consider these principles. The overall judgment stemming from the desk analysis of the rolling work programme is positive ( Annex 6 : Table 15 ). Supporting examples are provided by examining the contribution of ISA2 actions to the implementation of the EIF, as the principles in question also fall under the scope of the EIF ( Annex 6 : Table 23 ).

Answer to the EQs on ‘effectiveness’

EQ2: How far are the ISA² programme’s results in the process of achieving the programme’s objectives?

The results achieved by ISA2 so far are aligned with the programme’s objectives. However, they do not yet fully match the expected results, as most of the actions are still ongoing and solutions are still being developed. More time is needed in order to achieve all the expected results. This is confirmed by the assessment of key performance indicators and a comparison between actions continued from ISA and actions started under ISA2.

External factors could improve but also jeopardise the way in which the programme achieves its objectives and delivers its results. The main findings hold true across the different stakeholder groups and are further confirmed by expert assessments. The call for common standards and frameworks from public administrations is an external factor that contributes to the programme’s performance. In contrast, institutional complexity could harm the achievement of cross-border and cross-sectoral interoperability.

EQ3: Are there aspects that are more or less effective than others are, and if so, what lessons can be drawn from this?

While in general the programme is on track to achieving its objectives, it performs relatively less well when it comes to developing a more effective, simplified and user-friendly e-administration at the national, regional and local levels of public administration. This is an area where Member States can in fact have a greater impact. National initiatives related to the ICT modernisation of the public sector can also improve the overall performance of ISA2 by increasing the capacity for the take-up of ISA2 solutions.

Regarding the take-up rate of solutions, the evaluation found that the actions that continued from previous editions of the programme have produced solutions that are now more widely used than solutions resulting from newly established ISA2 actions.

As regards the specific request of Article 13(5) of the ISA2 Decision to assess the programme’s compliance with the principles set out in Article 4 of this decision, the evaluation found that the rolling work programme process effectively ensures that the ISA2 actions take into consideration the core principles listed in the ISA2 Decision.

When it comes to communication and awareness-raising activities, the evaluation results suggest that ISA2 duly acted upon the recommendation of the final ISA evaluation and increased general awareness of the programme. Still, some aspects of ISA2 could benefit from more promotion. For example, as legal interoperability is specifically mentioned among the operational objectives, the solutions developed for assessing the ICT and interoperability impacts of legislation could be more prominently featured in order to improve awareness among stakeholders. Moreover, ISA2 could better stress the benefits of implementing certain interoperability solutions. To this end, it could run studies that quantify the impact of interoperability solutions on the efficiency and productivity of public administrations. Such studies would improve the conditions for assessing the programme’s effectiveness and could have positive spillover effects on dissemination and communication activities: the monetary benefits deriving from ISA2 solutions would be better emphasised, thus providing a better basis for stakeholder support of future EU efforts promoting interoperability. Finally, as noted under section 5.1 Relevance , ISA2 communication activities should better target the sub-national layers of administrations.

Efficiency

Evaluation framework for ‘efficiency’

Efficiency relates to the extent to which the programme’s objectives are achieved at a minimum cost. This evaluation criterion includes two EQs and one sub-question:

EQ4:To what extent has the programme been cost-effective?

EQ5:Which aspects of the programme are the most efficient or inefficient, especially in terms of resources mobilised?

EQ5.1:    How is the programme performing relative to the planned work and budget?

To answer these questions, the evaluation focused on three judgment criteria, relying on data collected from the ISA2 dashboard, the monitoring and evaluation reports and consultation activities:

JC1:Efficiency of the selection process of the actions to be included in the rolling work programme;

JC2:Cost-effectiveness analysis to assess the ratio between allocated funds and actual results of the programme; and

JC3:Earned value management analysis.

Analysis

JC1: Efficiency of the action selection process

The ISA2 programme is implemented in four phases: i) submission; ii) evaluation; iii) adoption; and iv) implementation (further details are provided in Chapter 3). The process to select ISA2 actions includes the first three phases, i.e. submission, evaluation, and adoption. Note that this section does not assess the selection of contractors who will implement the actions, as this process follows the general European public procurement rules.

Several elements of the ‘submission’ phase need to be explained here. First, it is important to note that, in order to prepare and submit a proposal to be included in the ISA2 rolling work programme, the applicant must: i) find out about the call for proposals; ii) study the documents of the call for proposals and understand the rules and procedures; iii) prepare a concise description of the proposed action in compliance with the ‘work programme entry template’; iv) collect the required internal authorisations to submit the proposal; and v) submit the proposal by e-mail. Second, it is important to distinguish between two types of proposal, namely proposals for new actions and proposals for continuing actions that have already started. This distinction is relevant when it comes to assessing the overall workload required for submitting a proposal.

The efficiency of the action selection process (which includes the submission, evaluation and adoption phases) is assessed based on feedback received from programme governance representatives and action owners, as they are the stakeholder groups directly involved in the process. Most respondents consider the process fit-for-purpose at least to some extent: 16 out of 19 consulted programme governance stakeholders and 21 out of 23 consulted action owners ( Figure 11 ). However, 13 out of the 42 consulted stakeholders indicated that some measures could still be taken to further streamline the process. The measures mentioned by several stakeholders include:

Simplifying the rolling work programme, which can be done by:

oMaking the template more flexible to account for action specificities. This would ensure better input of information, which would make the process of drafting the proposals more efficient, and would also make it easier for anyone seeking specific information about ISA2 actions to read and understand the work programme.

oExtending the duration of a rolling work programme to cover more than one year, thus allowing for multiannual actions. In principle, actions to be run on a multiannual basis can be proposed, but the budget is allocated on an annual basis. Therefore, action owners must submit an updated proposal every year as part of the rolling work programme in order to access funding.

oSetting up an electronic data input system rather than full text editing.

·Simplifying the overall process by launching thematic calls for actions, thus ensuring that the selection is more objective-driven.

Figure 11: Extent to which the action selection process is fit-for-purpose (number of respondents by stakeholder category)

Source: CEPS final study

As regards the submission phase of the selection process, 26 out of the 42 consulted action owners and programme governance representatives indicated that they had prepared and submitted proposals. Out of these 26 respondents, 21 also specified the amount of time they spent on preparing the proposal. The answers provided vary between 1 to 30 person-days, with the average being 9 person-days. The large variation can be primarily explained by the differences in the efforts required to submit a proposal for a new action versus the amount of effort required to renew a proposal for an existing action.

As regards the time needed for submission, Figure 12 shows a clear concentration of answers: 14 of the respondents indicated that they finalised their proposal in 1 to 6 person-days. However, four respondents indicated that 20 or more person-days were needed. This distribution of answers and the qualitative inputs provided by some of the respondents during the consultation activities makes it possible to differentiate between preparing a renewed proposal and a brand-new proposal. Renewing a proposal for an action that has already been included in a previous rolling work programme tends to require between 1 and 6 person-days at most, while preparing a new proposal takes significantly longer, between 10 and 30 person-days. In most cases, stakeholders prepare the proposals themselves, without relying on external contractors.

Additional time is also required to merge similar proposals, upon request by the Commission team in charge of shortlisting the proposals to be funded.

Figure 12: Distribution of the number of days needed to prepare and submit a proposal for an ISA2 action (number of respondents)

Note: The histogram was constructed using a bin size of 2 days, which grouped the answers of respondents in intervals of 2 days. This bin size was deemed most appropriate after reviewing the answers provided.

Source: CEPS final study

The time spent to prepare and submit a proposal for an action to be included in the rolling work programme, whether it is a new or a renewed proposal, can be used to measure regulatory costs by applying the standard cost model 46 , presented in the Better Regulation Toolbox 47 . First, person-days are converted into person-hours by assuming that each person-day corresponds to eight hours 48 . Then, the overall number of hours is multiplied by a standard tariff, i.e. the 2017 hourly labour cost for the service sector at the Member State level (see Annex 6 : Table 22 ) 49 .

Two respondents indicated that they relied on external service providers to prepare the proposal. In this case, regulatory costs would also include out-of-pocket expenses incurred for such services. Table 3 presents the average regulatory costs both taking into account the fees related to external service providers and without accounting for these additional fees. The average respondent faced regulatory costs of approximately €1 077 for renewing a proposal and €5 651 for preparing a new proposal, without accounting for fees incurred when relying on external service providers. Considering the two instances when external service providers were used in the preparation of proposals, the average regulatory cost for a renewed proposal amounted to €1 138 and for a new proposal the average regulatory cost was €6 590. At the same time, the average funding allocated to an ISA2 action between 2016 and 2018 amounted to €1 217,566 50 . The costs incurred to prepare a proposal are thus very small, ranging between 0.09 % and 0.5 % of the potential funding that could be allocated to the proposal once accepted.

Table 3: Average regulatory costs incurred by respondents who prepared and submitted proposals for ISA2 actions

Note: The calculations are based on a sample of 21 respondents who specified the time spent on preparing proposals.

Source: CEPS final study

One respondent mentioned that the reason they relied on external service providers in this process was the need to tailor the proposal to the requirements of the rolling work programme template. As such, one potential cost-reduction method could be the simplification of the rolling work programme template, thus further corroborating the measures proposed to streamline the action selection process.

JC2: Cost-effectiveness analysis

Based on the performance indicators discussed in section 5.2 on effectiveness and on data related to the actual costs of actions, a cost-effectiveness analysis 51 could be performed. However, it is important to note the limitations of this method. First, the heterogeneity of performance indicators makes it difficult to draw conclusions about the programme’s overall cost-effectiveness ( Box 5 shows the variety of ISA2 performance indicators). Consequently, computing an aggregate value that is representative of the programme or action packages is quite difficult. Second, such data are only partially available, as some ISA2 solutions are still under development. In addition, as concluded in section 5.2, it is expected that the values of the performance indicators will improve over time, as solutions tend to require longer periods to fully show their results. Third, data on effectiveness are not yet available for a limited number of solutions developed so far.

Against this background, the cost-effectiveness analysis can be exemplified by using some of the ISA2 packages that rely on similar performance indicators. Two packages fit this profile: ‘Key and generic interoperability enablers’ and ‘eProcurement / eInvoicing — supporting instruments’. The sampled actions included in both packages use the number of public administrations (at EU, national and regional levels) that use the solutions generated by the actions as a performance indicator. Table 4 presents the estimated average cost per solution user based on the available data on the number of solution users (public administrations) and the actual costs of the action between 2016 and Q3 2018. For the ‘Key and generic interoperability enablers’ package, the average actual cost 52 per public administration using the solutions that are part of this package amounts to €19 420. For the ‘eProcurement / eInvoicing — supporting instruments’ package, the average actual cost amounts to €25 870. Note that these average costs are calculated per public administration using the solution rather than per single end-user. Each administration serves millions of end-users (citizens/businesses); hence, costs per end-user are estimated to be low.

Table 4: Costs per user of ISA2 solutions

Note: The calculations are based on the sampled actions of the indicated ISA2 packages over the period 2016 and Q3 2018. A full overview of the number of users and other performance indicators is presented in Annex 6 : Table 12 . The overview of costs per package is shown in Figure 13 .

Source: CEPS final study

The appraisal of the programme’s cost-effectiveness could be made more consistent by ensuring that monitoring and evaluation reports quantify some performance indicators common to all actions. Examples of such indicators include the number of EU public administrations using a solution, the number of public administrations participating in pilot projects, a consistent approach to reporting the number of downloads from the ‘Joinup’ platform for solutions that can be downloaded, and/or the number of single users per solution.

Box 5: Summary of the main costs and benefits identified

JC3: Earned value management (EVM) analysis

EQ5 and EQ5.1 focus on progress made in implementing the programme based on the EVM analysis. This analysis helps determine progress of work against a given baseline, so that costs, time, and scope of a certain activity are constantly tracked. To support this, the ISA2 dashboard provides quarterly data on the earned value, planned value, and actual costs of each monitored action 54 . As regards the sample of 20 actions grouped in the respective packages, Figure 13 presents the data at the end of Q3 2018 for the earned and planned value, as well as for the actual costs recorded. The earned value is close to equalling the planned value, which shows that the work is progressing as expected. Even where a bigger gap exists (e.g. in the ‘Key and generic interoperability enablers’ package), the indicators are expected to converge at the end of the year as the ISA2 budget year closes.

Figure 13: Earned value, actual costs and planned value sampled actions, by package (Q3 2018) 55

Source: CEPS final study

The data available also make it possible to perform a historical analysis of work progress for each package of actions. Progress can also be shown in the form of a schedule performance index (SPI), which is the ratio of earned value to planned value ( Figure 14 ). An SPI value above one indicates that the item analysed is ahead of schedule, while an SPI value below one suggests that the item analysed is behind schedule. Since the beginning of the programme and up until Q3 of 2018, the SPIs of the sampled actions grouped by packages have shown fluctuations, but now tend to converge on one, although values below one are still reported for almost all packages. These data show that the ‘Semantic interoperability’ and ‘Supporting instruments for public administrations’ packages are on track, while the others are close to achieving the planned level of work. Given that some actions started later on in the programme, as is the case of the ‘Geospatial solutions’ package, data for the EVM analysis are not always available from Q2 of 2016, but only from a later point in time.

SPI values above two were reported for the ‘Key and generic interoperability enablers’ package in Q2 and Q3 of 2016. This indicates that the work on the package was ahead of schedule in that period. While this result might seem surprising at first, it can be explained by the fact that some of the actions involved started under ISA and continued under ISA2. For these actions, processes had already been established during the previous programme, so work under ISA2 could start much faster than planned, resulting in a higher SPI value at the beginning of the process. In fact, both of the actions sampled from the ‘Key and generic interoperability enablers’ package (‘Trusted Exchange Platform e-TrustEx’ and ‘Catalogue of Services’) had been launched under the ISA programme.

Figure 14: Schedule performance indices of the sampled actions grouped by packages

Source: CEPS final study

Answer to the ‘efficiency’ EQs

EQ.4: To what extent has the programme been cost-effective?

The heterogeneity of performance indicators makes it difficult to draw conclusions about the programme’s overall cost-effectiveness. For those packages where it was possible to apply cost-effectiveness techniques, costs per end-user (e.g. business, citizens, etc.) have been estimated as low. In this respect, monitoring and evaluation reports could converge towards some common metrics allowing for a more consistent appraisal of cost-effectiveness. For example, the agreed performance indicators could comprise the number of EU public administrations using a given solution, the number of downloads from the ‘Joinup’ platform for downloadable solutions, and/or the number of single users per solution.

EQ5: Which aspects of the programme are the most efficient or inefficient, especially in terms of resources mobilised?
   EQ5.1: How is the programme performing relative to the planned work and budget?

The process of selecting actions to be funded by ISA2 is considered relatively efficient and fit for purpose. On average, six person-days at most are required to prepare and submit a proposal for actions that had already been included in previous rolling work programmes. On the other hand, between 10 and 30 person-days are required to prepare a proposal for a new action. The costs incurred in preparing a proposal are relatively small, ranging between 0.09 % and 0.5 % of the potential funds that could be allocated to the proposal once accepted. Nonetheless, this process could be further streamlined by simplifying the rolling work programme and launching thematic calls for actions, thus ensuring that the selection process is driven by objectives.

When looking at progress made in the programme’s implementation, it is clear that all packages are either on track or close to achieving the planned level of work. As some actions started during ISA and continued under ISA2, some processes had already been established during the previous programme. Therefore, work on these actions under ISA2 could start faster than planned.

Coherence

Evaluation framework for ‘coherence’

Article 13(4) of the ISA2 Decision requires that the coherence of ISA2 actions is assessed; this is referred to as the programme’s ‘internal coherence’. In addition, Article 13(6) pays specific attention to the programme’s ‘external coherence’ in order to ‘identify potential overlaps, examine coherence with areas for improvement, and verify synergies with other Union initiatives, in particular with the CEF’. The coherence criterion translates into two EQs:

EQ6:To what extent do the ISA2 actions form part of a ‘holistic’ approach within the framework of the programme? (Internal coherence)

EQ7:To what extent is the ISA2 programme coherent with other EU interventions, which have similar objectives and with global initiatives in the same field? (External coherence)

To assess both the internal and the external coherence of the programme, findings from desk research on links between ISA2 actions and the relationship between ISA2 and other relevant EU programmes/policies/initiatives were used in addition to the information provided by stakeholders. In this context, the following judgment criteria are considered:

JC1:Internal coherence:

-Degree of coherence among actions funded by the ISA² programme;

-Level of reuse of results of a funded action by another action within the ISA² programme;

JC2:External coherence:

-Degree of coherence between the programme and other EU supported programmes;

-Level of reuse of results delivered by ISA2 actions by other EU programmes;

-Degree of coherence between the programme and other EU policies; and

-Degree of coherence between the programme and global initiatives in the field.

Analysis

JC1: Internal coherence

Given that the ISA2 programme currently manages 53 actions 56 , it is important to ensure that they work together to create benefits and that work is not duplicated. The stakeholder groups consulted on this matter (programme governance, action owners, solution users) tend to agree that ISA2 actions generate synergies ( Figure 15 ). At the same time, most stakeholders indicate only limited overlaps between ISA2 actions. Stakeholder feedback on synergies and overlaps, as two sides of the same coin, paints a positive picture of the internal coherence of the ISA2 programme.

Only 4 % of respondents (2 out of 53) identify a high degree of overlaps between ISA2 actions, while the vast majority, 77 % (41 out of 53), consider overlaps as either not existing or being limited. In addition, respondents point to the existence of synergies: 41 % of respondents (27 out of 66) identify synergies as existing to a high extent or to the fullest extent and 17 % (11 out of 66) identify either no synergies or limited synergies. It is relevant to note that a significant number of solution users indicated that they were unaware of either synergies or overlaps between ISA2 actions. This feedback can be explained by stakeholders’ various levels of knowledge about ISA2 packages. As not all stakeholders are familiar with all ISA2 packages, they might not be in a suitable position to identify potential synergies or overlaps.

Figure 15: Extent to which there are synergies / overlaps between ISA2 actions (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

The review of documentary evidence confirms the results of the consultation activities, particularly regarding the existence of synergies between ISA2 actions. The indicator of the ‘network’ of relationships between the programme’s actions 57 , as shown on the ISA2 dashboard, is particularly relevant here. The quantitative assessment of the data provided online for the sample of 20 actions under assessment shows (see Annex 6 : Table 17 ) that there are multiple instances of internal reuse of ISA2 solutions, with actions like ‘SEMIC’ and ‘Joinup’ playing a particularly important role in the system.

At the same time, some actions tend to be less central in the network. This is the case particularly for the ‘EUSurvey’ 58 and ‘European Citizens’ Initiative and European Parliament Election’ 59 actions. The specificity of these actions makes it more likely that their solutions are reused by other programmes or initiatives, rather than by other ISA2 actions.

Throughout the consultation activities, additional inputs were collected from respondents concerning synergies between actions and examples of instances when results delivered by an ISA2 action were used by another action (see Box 6 ).

Box 6: Examples of internal coherence between ISA2 actions

JC2: External coherence

In order to assess the external coherence of ISA2, stakeholders were asked to assess the level of synergies and overlaps existing between ISA2 and four EU programmes identified as relevant for the field of digitalisation of public administrations and interoperability: i) CEF 60 , ii) ESF 61 , iii) Horizon 2020 62 and iv) SRSP 63 . The results are presented in Figure 16 .

Figure 16: Extent to which there are synergies / overlaps between ISA2 and other relevant EU programmes (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Out of the four programmes, respondents to the consultation activities identify the CEF as having the highest level of synergies with ISA2. According to 78 % of respondents (51 out of 65), ISA2 and CEF have a high level or a very high level of synergies. As an example, both programmes contribute to the Commission’s open data policy: the DCAT-AP solution of ISA2 provides a metadata specification used by open data portals across the EU and CEF ensures funding for the public open data infrastructure that aims to increase the reuse of open public data in the EU. However, 18 % of respondents (10 out of 57) also note the existence of overlaps to a high extent or to the fullest extent, which could be addressed by better delineating the scope of the concerned solutions.

When it comes to the relationship between Horizon 2020 and ISA2, half of the respondents (31 out of 63) identify synergies to a high extent or to the fullest extent, while only 12 % (6 out of 51) indicate the existence of overlaps to a high extent or to the fullest extent. The situation is similar for SRSP, with 52 % of respondents (22 out of 42) pointing out that there are synergies between ISA2 and SRSP either to a high extent or the fullest extent, and only 6 % (2 out of 32) indicating a high or very high degree of overlaps.

At the opposite end of the spectrum, out of the four programmes listed above, the ESF seems to be the programme with the lowest level of coherence with ISA2. Only 26 % of respondents (6 out of 23) indicate the existence of synergies between the ESF and ISA2 to a high extent or to the fullest extent.

Respondents also pointed to synergies and overlaps with other EU initiatives. More specifically, three additional EU programmes and initiatives were mentioned: the European Regional Development Fund (ERDF) 64 , corporate IT governance 65 and the Single Digital Gateway 66 . This feedback is discussed in the points below.

While the synergies between the ESF and ISA2 appear to be limited, two respondents pointed to the existence of a high synergy level between the ERDF and ISA2. At the same time, one respondent indicated that there could be overlaps between ISA2 and ERDF, possibly due to a lack of coordination. More specifically, national IT systems developed using ERDF funds could be incompatible with ISA2 solutions, thus preventing the adoption of these solutions.

A high degree of complementarity was also identified between the Commission’s corporate IT governance and ISA2, as explained by one respondent. The corporate IT governance team maintains that the Commission’s proposed IT investments take into consideration the reuse of IT solutions that have already been developed. In this sense, the work of this team supports synergies between ISA2 and other initiatives, ensuring that ISA2 solutions and any other IT solutions developed so far are reused in the future. In addition, the team helps mitigate the risk of developing very similar IT solutions under different spending programmes, which would create overlaps.

·One respondent also mentioned the existence of synergies with the Single Digital Gateway (SDG). In fact, this is clear given that ISA2 implemented an action dedicated to the interoperability requirements of the SDG in 2018 67 . ISA2 helped detail the IT architecture of the SDG and identify functional, technical and semantic interoperability challenges in its implementation. Further ISA2 solutions like ‘eForms’ 68 and ‘e-Documents’ 69 also support the SDG’s implementation.

Table 18 in Annex 6 shows that ISA2 actions have multiple links outside of the programme, as they rely on other EU programmes / policies / initiatives to deliver their results and their solutions are used by other programmes / policies / initiatives.

Beyond examining alignment with the initiatives mentioned above, it is worth analysing the degree of alignment of ISA2 with other initiatives or broader policies, like the Digital Single Market Strategy, the eGovernment Action Plan, ICT standardisation or the Tallinn Declaration on eGovernment.

When it comes to the Digital Single Market (DSM) Strategy 70 , the desk review of the rolling work plan reveals that ISA2 contributes to all three of the strategy’s pillars (see Annex 6 : Table 19 ): (i) better access for consumers and businesses to digital goods and services across Europe; (ii) creating the right conditions and a level playing field for digital networks and innovative services to flourish; and (iii) maximising the growth potential of the digital economy. Based on the 20 sampled actions, ISA2 in particular contributes to the strategy’s third pillar through:

the revision and implementation of the new European Interoperability Framework, which is explicitly mentioned in the DSM strategy;

work on supporting the implementation of the once-only principle, carried out as part of several actions such as ‘SEMIC’ and ‘European Interoperability Architecture’; and

·support for the implementation of the Single Digital Gateway.

Together with the launch of the DSM strategy for Europe, a new eGovernment Action Plan was announced for 2016-2020, aiming to ‘remove existing digital barriers to the Digital Single Market and to prevent further fragmentation arising in the context of the modernisation of public administrations’ 71 . The ways in which ISA2 contributed to implementing the DSM strategy also directly relate to the eGovernment Action Plan. In particular, the revised version of the EIF and the ISA2 actions monitoring its implementation in the Member States (‘EIF implementation and governance models’ and ‘NIFO’) address Action 4 of the eGovernment Action Plan. Furthermore, ISA2 is also contributing to Actions 1 and 6 through its work in the field of eProcurement. For instance, the ‘European public procurement interoperability initiative’ action facilitates the implementation of eProcurement in European public administrations and supports the EU’s TOOP 72 project by adjusting the necessary tools for the once-only principle.

In the field of ICT standardisation, the Rolling Plan for ICT Standardisation 73 provides a good overview of the EU’s main initiatives and priorities in this area. The Commission drafts this plan in collaboration with the European Multi-Stakeholder Platform on ICT Standardisation, which includes representatives of the Member States, EFTA countries, European and international standardisation organisations, industry, SMEs and consumers. The 2018 Rolling Plan on ICT Standardisation points out two main areas with highly relevant synergies between ISA2 and ICT standardisation:

1.eGovernment. The Rolling Plan emphasises that in order to leverage the applicability of technical specifications developed under ISA2, one solution would be to suggest using them as standards by the relevant standardisation organisations for the specifications in question. In this respect, it is worth noting that W3C, an international standardisation forum, has discussed some ISA2 solutions (like the ‘ADMS’ specification 74 ) and has published a W3C note 75 .

2.Public Sector Information, Open Data and Big Data. The Rolling Plan also mentions the usefulness of introducing a common standard for referencing open data in European open data portals. The ‘DCAT Application Profile’ 76 , an ISA2 solution that is already used by several open data portals across the EU, stands out as a candidate for this purpose. The Rolling Plan invites the CEN ‘to support and assist the DCAT-AP standardisation process’.
There are several potential issues with standardisation though. More specifically, the evaluation found that the coherence between intellectual property rights for ISA2 solutions and CEN/CENELEC standards needs to be clarified and resolved before the standardisation system could be fully implemented 77 .

Finally, in order to gain a comprehensive understanding of EU-level coherence, it is also useful to analyse the alignment of ISA2 with the priorities set at the highest political level in the Tallinn Declaration on eGovernment 78 . The assessment in Box 10 under Annex 6 concludes that there is no contradiction between the Tallinn Declaration and the ISA2 programme: even though they have different scope and aims, they proceed in the same direction and share common goals. However, strong coordination is required between them to build synergies and avoid overlaps and duplication.

Answer to the ‘coherence’ EQs

EQ6: To what extent do the ISA² actions form part of a ‘holistic’ approach within the framework of the programme? (Internal coherence)

ISA2 actions are characterised by substantial synergies with each other and limited overlaps. Stakeholder feedback on synergies and overlaps shows a positive picture of the internal coherence of the ISA2 programme. There are multiple instances of internal reuse of ISA2 solutions, with actions like ‘SEMIC’ and ‘Joinup’ playing a particularly important role in the system. At the same time, some actions tend to be less central within the network. This is for instance the case of the ‘EUSurvey’ and ‘European Citizens’ Initiative and European Parliament Election’ actions and can be explained by the particular nature of these actions.

This strong internal coherence is in line with the second recommendation of the final ISA evaluation, which called for taking a holistic approach on interoperability within ISA2.

EQ7: To what extent is the ISA² programme coherent with other EU interventions, which have similar objectives and with global initiatives in the same field? (External coherence)

When it comes to external coherence, ISA2 followed the third recommendation of the final ISA evaluation and developed close cooperation with other EU policies and initiatives. Respondents to the consultation activities identified the highest level of synergies between ISA2 and CEF. They also pointed out the synergies with ‘other’ EU initiatives such as the ERDF, corporate IT governance and the Single Digital Gateway. Synergies also exist between ISA2 and some broader EU initiatives and policies, such as the Digital Single Market strategy, the eGovernment Action Plan, the Tallinn Declaration on eGovernment, and the Rolling Plan for ICT standardisation.

In conclusion, ISA2 actions have multiple links outside of the programme, as they rely on other EU programmes / policies / initiatives to deliver their results and their solutions are used by other programmes / policies / initiatives. This further corroborates the positive feedback on the programme’s external coherence.

EU added value

Evaluation framework for EU added value

The EU added value captures the programme's impacts additional to those that would be achieved if the issues addressed by ISA2 were left solely in the hands of national and sub-national authorities. This evaluation criterion seeks to answer the following EQ:

EQ8:What is the additional value resulting from the ISA² programme, compared to what could reasonably have been expected from Member States acting at national, regional and/or local levels?

To answer this question, the evaluation analysed the following judgment criteria based on both primary and secondary data:

JC1:Achievement of objectives that could not be otherwise attained with national or sub-national interventions;

JC2:Achievement of objectives at a cost lower than what could be attained via national or sub-national interventions;

JC3:Achievements in terms of cross-border interoperability; and

JC4:The contribution to the advancement of common EU policies.

Analysis

JC1: Achievement of objectives that could not have been achieved without EU-level intervention

Most respondents believe that national or sub-national interventions would not be able to achieve the general and specific objectives in the field of interoperability that ISA2 is pursuing, or that they would achieve only be able to achieve them to a limited extent ( Figure 17 ). This observation is particularly evident when it comes to developing, maintaining and promoting a holistic approach to interoperability in the Union (specific objective 1), as 83 % of respondents (95 out of 115) believe that national or sub-national interventions would not be able to achieve this objective or would achieve it only to a limited extent. Additionally, 72 % of respondents (83 out of 116) point to limited achievements or no achievement from national or sub-national initiatives with regard to facilitating the efficient and effective electronic cross-border or cross-sector interaction between European public administrations, and between European public administrations and businesses and citizens (specific objective 2). These examples show that the coordination provided by a unique EU-level programme plays an important role in improving overall interoperability among European public administrations. The technical experts supporting the evaluation have unanimously confirmed this conclusion.

Figure 17: Extent to which national or sub-national interventions would be able to achieve the ISA2 objectives without the programme (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

There is, however, one specific area in which national or sub-national interventions would be able to contribute in the absence of an EU-level programme. A quarter of respondents (29 out of 115) believe that national or sub-national interventions would be able to contribute to the development of a more effective, simplified and user-friendly e-administration at the national, regional and local levels (specific objective 3) to a high extent or to the fullest extent. This result shows how well EU and national initiatives complement each other, since specific objective 3 is the one where the ISA2 programme shows the worst performance in terms of effectiveness (see Figure 6 in section 5.2).

It is worth emphasising that while national initiatives related to the ICT modernisation of the public sector cannot achieve the objectives pursued by ISA2, they are still very important to improving the programme’s performance as they increase the capacity for the take-up of ISA2 solutions. Initiatives pursued by Member States in this field show that interoperability is a priority, which encourages working together with ISA2. In addition, from the point of view of ICT development, national initiatives put Member States in a better position to use the solutions provided by ISA2.

JC2: Achievement of objectives at a lower cost

In addition, 83 % of respondents (91 out of 109) emphasised that ISA2 is probably or definitely able to achieve its objectives at a lower cost than comparable national or sub-national initiatives ( Figure 18 ). The main stakeholders of ISA2 appreciate the economies of scale that this EU-level programme generates.

Figure 18: Ability of ISA2 to achieve the relevant objectives at a lower cost than comparable national or sub-national interventions (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

JC3: Achievements in terms of cross-border interoperability

ISA2 focuses on providing both cross-border and cross-sectoral interoperability solutions. From the point of view of EU added value, the contribution to cross-border interoperability is particularly relevant, as it shows the extent to which the programme works towards achieving a holistic approach to interoperability in the EU.

Over half of the respondents (50 out of 93) believe that ISA2 has contributed to a high extent or to the fullest extent to improving cross-border interoperability in the EU. This result holds true across the consulted stakeholder groups, in comparison to less than 6 % (6 out of 93) who only see a limited contribution in this area ( Figure 19 ).

Figure 19: Extent to which ISA2 contributed to improving cross-border interoperability in the EU (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Several respondents also emphasised the following aspects:

Awareness: ISA2 raises awareness of interoperability in the EU and helps put the topic on national agendas; without this, little progress would be made in the field.

·Organisational contribution: ISA2 brings people together, creating networks and helping national organisations meet their counterparts in different countries, thus facilitating exchanges between Member States.

Desk research also supports the two examples mentioned above. First, ISA2 is committed to raising awareness about interoperability through its communication activities presented in Box 3 . Second, it contributes to the creation of a network of officials working on interoperability by holding regular ISA2 Committee and ISA2 Coordination Group meetings, which bring together representatives of Member States responsible for this topic. There were five ISA2 Committee meetings and three Coordination Group meetings between 2016 and 2018. In most of them, all 28 Member States were represented by at least one participant (with a maximum of three participants per meeting per Member State). In addition, Norway was represented in all Coordination Group meetings and in four of the five Committee meetings, Iceland was represented in three Committee meetings and one Coordination Group meeting, and finally Montenegrin representatives were present in three Committee meetings and one Coordination Group meeting (see Annex 6 : Table 20 and Table 21 ).

The fact that countries beyond the EU have either become members of ISA2 or cooperate with the programme shows that the programme’s benefits do not stop at the EU borders. An additional example is the administrative cooperation between DG DIGIT and the Agency for the Development of the Government of Electronic Management and the Information and Knowledge Society of Uruguay, officially started on 26 March 2018 and consisting of delegation visits, exchanges of experts, workshops, and sharing of best practices.

Further evidence comes from analysing the cross-border interoperability angle of the 20 sampled ISA2 actions: 29 out of the 35 operational solutions developed by these actions have a cross-border feature ( Annex 6 : Table 11 ). One example is the IMAPS solution 79 , which facilitates the assessment of public services at the local, national, and EU level from the point of view of their interoperability maturity. In practice, it consists of a compact and user-friendly self-assessment carried out through the IMAPS online survey. This 30-minute questionnaire looks at the context of the public service, how the service is delivered, how it consumes existing services and how it is managed. On completion of the IMAPS survey, the results give tailored recommendations and confidential guidance for improvement. 68 public administrations completed the assessment in 2017 and 51 in 2018 — at local, national and EU level. The solution has thus had a cross-border impact, since the assessments are directed at public administrations all across the EU as confirmed by the user testimonial in Box 7 .

Box 7: IMAPS user testimonial

Source: ISA2 Communication Team

Further key achievements include ( Annex 6 : Table 22 ):

The ‘e-TrustEx’ platform is a key enabler of secure information exchange for seven pan-European projects in which over 200 public administrations from all Member States participate.

The ‘Big Data for Public Administrations’ action created a working group to assess the needs in the area of big data analytics for policy making with the participation of the Czech Republic, Estonia, Hungary, Malta, Norway, Slovenia, Spain, and Portugal.

Several actions foster multilingualism:

othrough their work on providing common frameworks, the actions on semantic interoperability contribute to addressing multilingual challenges to the cross-border interaction of European public administrations; one example is the translation of the six ‘Core Vocabularies’ into 23 official EU languages 80 ;

otools have been developed by ISA2 with the specific purpose of being ready for use in all official EU languages; one example is ‘EUSurvey’; and

oCPSV-AP is used, for instance, in Finland and Estonia to create cross-border federated catalogues of public services.

One potential improvement would be to further increase the programme’s focus on multilingualism by providing more information about ISA2 in different EU official languages, as the current website is only available in English.

JC4: Contribution to the advancement of common EU policies

Half of the respondents (44 out of 84) indicate that ISA2 has contributed to a high extent or to the fullest extent to the advancement of common EU policies or initiatives ( Figure 20 ). For instance, stakeholders noted that ISA2 plays a central role in the implementation of the EIF by abiding by its principles and tracking its implementation. The review of secondary data provides additional evidence supporting this. The Communication on the revised EIF 81 is accompanied by an ‘Interoperability Action Plan, which sets out five focus areas and 22 actions to be undertaken in order to tackle existing interoperability issues during the 2017-2020 period. The Communication is also accompanied by 47 specific recommendations for improving interoperability in the EU. ISA2 actions can be clearly linked to the focus areas, actions, and recommendations outlined in the revised EIF (see Annex 6 : Table 23 ).

Figure 20: Extent to which ISA2 contributed to the advancement of common EU policies (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

By contributing to the implementation of the revised EIF, ISA2 is also contributing to the establishment of the Digital Single Market, as explained in section 5.4 on external coherence. The technical experts involved in the evaluation note that the programme has increased attention paid to the importance of interoperability and cross-border e-services for sustaining the Digital Single Market. In this way, the programme fully meets the aims of advancing common shared policies within the EU, as stated in the Tallinn Declaration.

Answer to the ‘EU added value’ EQ

EQ8: What is the additional value resulting from the ISA² programme, compared to what could reasonably have been expected from Member States acting at national, regional and/or local levels?

ISA2 ensures EU-level coordination, which is necessary for improving overall interoperability among European public administrations. Without the programme, national and/or sub-national interventions would not be able to achieve the interoperability-specific objectives that ISA2 is aiming at. In addition, ISA2 is able to achieve these objectives at a lower cost than comparable national or sub-national initiatives. These conclusions hold true across the different stakeholder groups and are further corroborated by expert assessment.

Nonetheless, national or sub-national interventions would be able to contribute to the development of a more effective, simplified and user-friendly e-administration at the national, regional and local levels even if there was no EU-level programme. This is the specific objective where the ISA2 programme appears to be less effective, thus showing how well EU and national initiatives complement each other. National initiatives on the ICT modernisation of the public sector can even improve the performance of ISA2 by increasing capacity for the take-up of ISA2 solutions.

ISA2 also creates additional value by improving cross-border interoperability in the EU, as well as advancing common EU policies. As regards the former, ISA2 raises awareness of interoperability across EU Member States and helps put the topic on national agendas. It also brings people together, thus creating networks, helping national organisations meet their counterparts in different countries and facilitating exchanges between Member States in the field of interoperability. As regards the latter, ISA2 plays a central role in implementing the EIF and supports the establishment of the Digital Single Market.

Utility

Evaluation framework for ‘utility’

Article 13(4) of the ISA2 Decision requires an assessment of the programme’s utility. Utility refers to (i) the extent to which the results of ISA2 meet stakeholders’ needs; and (ii) the level of stakeholder satisfaction with ISA2 solutions. This criterion translates into the following EQ:

EQ9:How do the ISA2 programme’s actions and results, achieved and anticipated, compare with the needs they are supposed to address?

The assessment of this EQ is based on primary data collected via consultation activities as well as on evidence gathered via desk research, and relies on the following judgment criteria:

JC1:Alignment between stakeholders’ perception of needs and problems at EU, national and sub-national levels and the results of the programme; and

JC2:User satisfaction, with a breakdown by stakeholder group.

Analysis

JC1: Alignment between EU- and Member State-level needs and problems and the programme’s results

Respondents from both EU and Member State public administrations confirm that the solutions developed or maintained by ISA2 have contributed to addressing the original needs and problems identified in the field of interoperability ( Figure 21 and Figure 22 ) 82 . Overall, 42 % of respondents at the EU level (38 out of 91) and 46 % of respondents at the Member State level (13 out of 28) consider that the solutions have contributed to addressing the needs and problems originally covered by the programme to a high extent or to the fullest extent. In comparison, only 5 % of EU-level respondents (5 out of 91) and 11 % of Member State-level respondents (3 out of 28) see a limited or no contribution.

Figure 21: Extent to which ISA2 solutions contributed to addressing the needs and problems originally covered by the programme: EU-level respondents (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Figure 22: Extent to which ISA2 solutions contributed to addressing the needs and problem originally covered by the programme: respondents from the Member States (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

As the programme is still ongoing, the ways in which solutions meet user needs may still improve. It is expected that the take-up rate of solutions will increase over time, especially for the programme’s newer solutions, a conclusion that was explained in section 5.2 on effectiveness. With a higher adoption rate of ISA2 solutions across European public administrations, the needs and problems would be increasingly better addressed, thus leading to higher user satisfaction.

The feedback from consultation activities is further supported by secondary data. When it comes to the need for cooperation among public administrations with the aim to enable more efficient and secure public services, one example of the contribution of ISA2 is the “Access to Base Registries” action. The action has developed and is constantly updating a “Catalogue of Solutions” 83 for base registries, facilitating interconnection and cooperation between public administrations. In addition, a set of guidelines for addressing potential challenges has been created. 84 Both outputs help implement the once-only principle thus increasing the efficiency and user-friendliness of public services. Another example of an ISA2 action fostering cooperation in order to enhance efficiency in the public sector is the “Communication and information resource centre for administrations, businesses and citizens (CIRCABC)” action and tool 85 . CIRCABC enables the efficient collaboration between EU institutions, European public administrations and other stakeholders, by offering a free-of-charge, trustworthy tool. Feedback is constantly collected from users in order to improve the way the tool meets their needs.

As regards the need for exchanging information among public administrations to fulfil legal requirements or political commitments, the “Open e-TrustEx” solution, for instance, provides a platform for the secure exchange of information between public administrations in the EU and in the Member States. It is used in seven pan-European projects, including the e-Prior Open Source e-Procurement platform 86 and the European eJustice Portal 87 , contributing to cross-border interoperability. Another example is the “Inter-institutional register of delegated acts (RegDel)” 88 , which directly responds to the commitment of the Commission, Council, and European Parliament to increase transparency and traceability of the delegated acts process. 89  

Turning to the need for sharing and reusing information among public administrations to increase administrative efficiency and cut red tape for citizens and businesses, the “Joinup” platform can be mentioned. It provides a collaborative space where solutions can be shared between stakeholders. Additionally, the platform also hosts a number of national repositories where five Member States have made national solutions available to other interested parties. An additional example in this regard is the “Development of an Open Data Service, Support and Training Package in the Area of Linked Open Data, Data Visualisation and Persistent Identification” action, which contributes to the sharing of best practices, by providing a set of reusable tools for data visualisation and data manipulation, available on the EU Open Data Portal. 90

As noted in section 5.1 on relevance, several respondents identified additional needs and problems to those explicitly mentioned in the programme’s intervention logic. In this respect, 27 % of EU-level respondents (11 out of 41) and 50 % of Member State-level respondents (6 out of 12) indicated that solutions have contributed to addressing these additional needs and problems to a high extent or to the fullest extent. In comparison, 22 % of EU-level respondents (9 out of 41) and 25 % of Member State-level respondents (3 out of 12) believe that this contribution has been either limited or absent ( Figure 23 and Figure 24 ).

Figure 23: Extent to which ISA2 solutions contributed to addressing additional needs and problems identified by consulted stakeholders: EU-level respondents (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Figure 24: Extent to which ISA2 solutions contributed to addressing additional needs and problems identified by consulted stakeholders: respondents from the Member States (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

JC2: Stakeholder satisfaction

The feedback received from respondents regarding user satisfaction tends to be positive ( Figure 25 ). More specifically, half of the respondents (57 out of 110) indicate that they are satisfied with the way solutions meet user needs to a high extent or to the fullest extent, while only 6 % (7 out of 110) indicate limited satisfaction.

Figure 25: Extent to which ISA2 solutions are meeting user needs in the fields of cross-border and cross-sectoral interoperability of digital public services (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Respondents also provided feedback on measures that could be taken to improve the programme’s utility. This feedback centres on the following aspects and shows significant complementarities with the measures proposed to improve the relevance of ISA2 (see section 5.1):

Best practices and user support. An increased focus on sharing best practices and providing support to users would have a beneficial effect on user satisfaction.

More focus on already existing solutions. According to the feedback from the consultation activities, another way to better address user needs would be to focus more on the already existing solutions and on improving their quality. This would improve the way in which user needs are considered in the further development of a smaller group of actions.

Promotion. Respondents indicated that they would like to see more activities promoting ISA2 at the Member State level in order to increase the level of take-up of solutions. An increased level of take-up would contribute to addressing the needs identified in the field of interoperability, thus improving overall satisfaction levels. Furthermore, feedback received from standardisation organisations shows that more awareness is needed among practitioners as well, particularly in the field of standards. Organising workshops targeted to specific groups of professionals could be useful. Formal presentations made at the plenary sessions of standards development organisations could also provide more exposure.

Co-creation. Another measure could be to involve users not only in the testing phase of solutions, but also in the design phase, and to establish a co-creation process. This includes more extensive on-boarding services (ensuring that users are provided with extensive information about the developed solutions), an increased focus on the sustainability of (re)user communities, as well as the integration of different solutions and services into one service, thus making it easier for users to navigate the landscape of available solutions.

·Commitment. The ISA2 Committee could reinforce the results of the programme by using it to set clear commitments to interoperability in the future. Member States could show their commitment to using the solutions stemming from ISA2 actions beyond simply participating in the selection of actions for the rolling work programme.

The ISA2 programme’s communication team and action owners already make an effort to capture users’ feedback and information on additional needs related to existing solutions. Their outreach activities made it possible to put together a number of user testimonials 91 , which support the promotion of solutions. For example, the user story presented in Box 8 encourages the use of Joinup. In the underlying detailed questionnaire, the interviewee shared her ideas about possible improvements, including more active marketing of the solution and the need to keep information on the Joinup platform up-to-date.

Box 8: ‘Joinup’ user testimonial

Source: ISA2 Communication Team

Answer to the ‘utility’ EQ

EQ9: How do the ISA² programme’s actions and results, achieved and anticipated, compare with the needs they are supposed to address?

According to consulted stakeholders, solutions developed or maintained by ISA2 have contributed to addressing the original needs and problems identified in the field of interoperability. As the programme is still ongoing, the way solutions meet user needs may still improve. In particular, these needs and problems are expected to be better addressed due to more extensive adoption of ISA2 solutions across European public administrations.

In line with the conclusion drawn on the relevance criterion (section 5.1), whereas some of the new needs and problems experienced by consulted stakeholders are addressed by the programme, some others fall beyond its scope as it is currently defined. Therefore, the solutions developed so far cannot respond to these particular needs and problems.

The feedback received from respondents in terms of user satisfaction tends to be positive. Nevertheless, both primary and secondary data show that additional measures could increase the programme’s utility. For example, ISA2 could place more emphasis on providing support to users of its existing solutions. Moreover, it could already involve users in the design phase of future solutions, thus increasing quality and user satisfaction.

Sustainability

Evaluation framework for ‘sustainability’

Sustainability is the final evaluation criterion that has to be assessed as part of the interim evaluation according to Article 13 of the ISA2 Decision. Sustainability measures the likelihood that the results of the ISA² programme last beyond its completion. The below EQ addresses this criterion:

EQ10:To what extent is the financial, technical and operational sustainability of the developed solutions — maintained and operated through the ISA² programme — ensured?

The evidence base for assessing the sustainability of ISA2 consists of primary data collected via the consultation activities, complemented by expert assessment. The evaluation relies on the following judgment criteria:

JC1:Extent to which the results achieved by the ISA2 programme are expected to last if funding for the actions covered by the programme would not be available in the future; and

JC2:Extent to which ‘cost recovery’ solutions could be introduced.

Analysis

JC1: Results would last if funding for actions would not be available in the future

This part of the assessment relies on the hypothetical scenario in which, after the completion of ISA2, no other similar programme is implemented at the EU level. Overall, the consulted stakeholders tend to have a positive outlook on the sustainability of results achieved so far ( Figure 26 ), with 79 % of respondents (66 out of 84) believing that the programme results probably or definitely would last in the absence of future funding for the actions currently supported by ISA2.

Figure 26: Likelihood that results achieved so far would last if funding for actions covered by the programme would not be available in the future (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Although the global results of the programme are expected to last (as the progress made in the field of interoperability will not be undone), certain issues may arise when considering the specificities of different solutions and future steps towards interoperability. In fact, 65 % of respondents (55 out of 85) consider that operations and maintenance costs 92 required to run the existing solutions would have a highly or fully negative effect on the ability of solutions to deliver their results once the programme ends. In addition, 56 % (48 out of 85) hold the same opinion about the technical and operational support 93 needed for the solutions ( Figure 27 ).

Beyond these two factors, respondents indicated other aspects that would have a negative effect on the solutions once ISA2 came to an end and no other programme replaced it:

Lack of further development for existing solutions. Given the rapid pace of changes in the field of ICT, solutions that do not benefit from continued development support and updating face the risk of becoming obsolete over time. This factor emerged from the consultation activities and was mentioned by the technical experts as well 94 .

·Lack of coordination. In the absence of a single programme overseeing the development of solutions, future changes to solutions might lead to diverging results between public administrations or bilateral solutions between countries and a large variety of outcomes. This would be contrary to the goal of achieving a holistic landscape of interoperability in the EU.

·Dissemination and communication. ISA2 also plays an important role in promoting interoperability and creating a space for discussions on the topic. Without such a programme, the interest in a common approach to interoperability would be jeopardised.

Figure 27: Extent to which financial, technical and operational aspects would prevent ISA2 solutions from delivering their results if the programme was terminated and not replaced (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

The elements presented above explain why 87 % of respondents (75 out of 86) believe that the achievement of the ISA2 general objective would — either probably or definitely — be jeopardised should ISA2 be terminated without being replaced ( Figure 28 ). This confirms that the ISA2 programme plays a central role in improving the interoperability landscape in the EU by promoting the ICT-based modernisation of the public sector in Europe and making it possible to address the needs of businesses and citizens via improved interoperability of European public administrations.

Figure 28: Likelihood that the ISA2 general objective would be jeopardised if the programme was terminated and not replaced (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

JC2: Cost recovery

Given the adverse effects of operational and maintenance costs as well technical and operational support, stakeholders were asked to assess the feasibility of introducing a fee for users in order to gain access to ISA2 solutions, in case funding would no longer be provided for the actions currently carried out by ISA2. Over half of the respondents (47 out of 79) would probably or definitely not pay such a fee, in comparison to 40 % of respondents (32 out of 79) who indicated that they would probably or definitely pay a fee in order to use the solutions ( Figure 29 ). The main reason underlying these responses is that users are accustomed to having access to ISA2 solutions free of charge. Changing this to a pay-for-access system may lead users to search for other solutions that are open source and free of charge, depending on the level of the fee requested. Nevertheless, for some of the more mature solutions, a fee could be acceptable if the level of take-up is already quite high, making the solution indispensable to some extent.

Figure 29: Likelihood that users would agree to pay a fee to continue accessing ISA2 solutions (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Answer to the ‘coherence’ EQs

EQ10: To what extent is the financial, technical and operational sustainability of the developed solutions - maintained and operated through the ISA² programme - ensured?

Overall, consulted stakeholders tend to have a positive view of the sustainability of results achieved so far. However, while the programme’s global results are expected to last (as the progress made in the field of interoperability will not be lost), certain issues may arise when considering the specificities of different solutions and future steps towards improved interoperability.

In fact, there are solutions that require both operational and maintenance costs as well as technical and operational support, which would render the solutions unable to deliver further results if the programme was terminated. Additional obstacles to the programme’s sustainability include the lack of further development for existing solutions; the lack of coordination between national administrations; and the limited dissemination of information and communication about the interoperability of digital public services. Consequently, ISA2 should investigate possibilities to increase the sustainability of the results of its existing solutions. It could assess the feasibility and costs of various sustainability measures and invest in them. The programme could for example consider transferring some of its solutions to open source communities or encouraging companies to build services around free ISA2 solutions under the European Union Public Licence (EUPL) 95 .

To conclude, the findings suggest, that the ICT-based modernisation of the public sector in Europe and the possibility to address the needs of businesses and citizens via improved interoperability of European public administrations — the general objective of ISA2 — would be jeopardised in the absence of ISA2 or any similar EU initiative. The consulted stakeholders’ reluctance to pay for ISA2 solutions makes this conclusion even more probable.

6.Conclusions

Context

ISA2 is a European funding programme offering interoperability solutions and common frameworks for European public administrations, businesses and citizens. It is the fifth in a series of European instruments fostering interoperability in the European public sector since 1995. It follows the ISA programme, which terminated in 2015, and covers the 2016-2020 period.

ISA2 was established by Decision (EU) 2015/2240 of the European Parliament and of the Council (‘ISA2 Decision’), which legally requires the European Commission to carry out an interim evaluation of the programme by 30 September 2019, and to communicate the results.

Method

Work on the evaluation started in Q2 2018 with the involvement of interested Commission services, who discussed and outlined the evaluation design. External experts supported the evaluation process as well by finalising the evaluation design, collecting, validating and analysing data, and summarising their findings in an independent evaluation study 96 .

The agreed evaluation framework ( Annex 4 ) covered all seven evaluation criteria identified by the ISA2 Decision, namely: relevance, effectiveness, efficiency, coherence, EU added value, utility and sustainability. It also described the evaluation questions and related judgment criteria for each evaluation criterion, together with the data sources and data collection methods. For data collection, the evaluation relied on both primary data gathered through stakeholder consultation activities and secondary data collected via desk research between November 2018 and March 2019 (Chapter 4).

Although the evaluation was able to draw robust conclusions for all evaluation criteria based on the collected data, there were two limitations:

1) Timing. As this is an interim evaluation, the analysis covered only the first three years of the programme (2016, 2017 and 2018). The ongoing ISA2 actions are expected to produce further results in the coming years, which will influence the programme’s overall achievement of objectives.

2) Reach. ISA2 is a very technical programme, and mainly involves interoperability experts. This explains the rather low participation in the stakeholder consultation activities. Another reason stems from the stringent confidentiality and data protection rules: without their specific consent, the evaluation team could approach ISA2 solution users only through the action owners acting as intermediaries.

Findings

Based on data collected from 129 consulted stakeholders, extensive desk research, and expert assessments, the interim evaluation confirms that ISA2 performs well in all the evaluation criteria. Nonetheless, some additional measures could be taken to improve the performance of ISA2 and any future EU programme dedicated to interoperability; these are presented below.

Relevance. The original needs and problems that the programme intended to address are still fully relevant and the objectives of ISA2 address them. Some stakeholders, however, experience a number of additional needs (like a more prescriptive approach to designing interoperable public services) and problems (including resource constraints), which ISA2 can address only partially. One additional need that deserves special attention is the request to ensure more collaboration and exchanges with regional and local administrations in order to increase awareness of interoperability and the take-up of ISA2 solutions at the sub-national level.

Effectiveness. The results that ISA2 has achieved so far are aligned with its objectives. The 20 sampled actions that were evaluated have so far generated 35 solutions, some of which are used by all 28 EU Member States as well as by EU institutions. Nevertheless, the programme has not realised all the expected results yet, because most of its actions are still ongoing and solutions are still being developed. The programme’s duration also influences the take-up rate of solutions. Actions continued from previous editions of the programme have produced solutions that are now more widely used than solutions resulting from actions started under ISA2.

There is one specific ISA2 objective where the evaluation found that the programme on its own is less effective: the development of a more effective, simplified and user-friendly e-administration at the national, regional and local level. Here Member States can in fact have a great impact, and complement the EU-level initiatives on interoperability and digitalisation.

As regards external factors, they have a mixed impact on ISA2. They can improve but also jeopardise the way in which the programme achieves its objectives and delivers its results.

When it comes to communication and awareness-raising activities, the evaluation results suggest that ISA2 duly acted upon the recommendation of the final ISA evaluation and increased the general awareness of the programme through its dedicated communication efforts. Still, some aspects of ISA2 could benefit from more promotion.

Efficiency. The programme’s implementation is progressing as planned, all actions being either on track or close to achieving the planned level of work. Nevertheless, the heterogeneity of performance indicators makes it difficult to draw conclusions about its overall cost-effectiveness. For those ISA2 packages where it was possible to aggregate performance indicators for different actions, costs per end-user (e.g. businesses, citizens) have been estimated as very low.

The selection process for actions funded by ISA2 is considered relatively efficient. Nonetheless, it could be further improved by simplifying the rolling work programme (e.g. making the template more flexible) and launching thematic calls for actions, thus ensuring that the selection process is driven by objectives.

Coherence. Substantial synergies and limited overlaps among ISA2 actions paint a positive picture of the programme’s internal coherence. This strong internal coherence is in line with the second recommendation of the final ISA evaluation, which called for taking a holistic approach to interoperability within ISA2.

ISA2 also followed the final third recommendation of the final ISA evaluation, on external coherence, and worked closely with other EU policies and initiatives. It established synergies with the CEF, Horizon 2020 and the Structural Reform Support Programme. ISA2 also interacts with initiatives such as the Single Digital Gateway, Digital Single Market Strategy, eGovernment Action Plan, Tallinn Declaration on eGovernment, and the Rolling Plan for ICT standardisation. Nonetheless, some overlaps were identified between specific solutions under ISA2 and the CEF; these could be addressed by better delineating the scope of each solution.

EU added value. ISA2 ensures EU-level coordination, which is necessary for improving overall interoperability among European public administrations. Without the programme, national and/or sub-national interventions would not be able to achieve the interoperability-specific objectives that ISA2 aims to achieve. In addition, ISA2 is able to achieve these objectives at a lower cost than comparable national or sub-national initiatives.

ISA2 also creates additional value by improving cross-border interoperability in the EU, as well as advancing common EU policies: it plays a central role in the implementation of the EIF and supports the establishment of the Digital Single Market.

Utility. User satisfaction with ISA2 solutions tends to be positive, with only 7 out of 110 stakeholders reporting ‘limited satisfaction’. As the programme is ongoing, it is expected that the take-up of ISA2 solutions will increase, thus improving the way solutions meet user needs and, in turn, overall user satisfaction. However, ISA2 could place more emphasis on providing support to the users of its existing solutions. Moreover, it could involve users already in the design phase of future solutions, which would increase both quality and user satisfaction.

Sustainability. The consulted stakeholders have mixed views on the sustainability of ISA2 solutions. While most of them believe that the programme results would last in the absence of future funding, they also recognise that ISA2 solutions may not be able to deliver their results if the related maintenance costs and support needs were not covered.

Stakeholders agree, however, that ISA2 plays a central role in improving the interoperability landscape in the EU and that its absence would jeopardise the efforts of European public administrations to improve interoperability and foster the ICT-based modernisation of the public sector in Europe.

Next steps

The interim evaluation identified several potential measures that may improve the performance of ISA2 and its successor instruments (see Box 9 ). It is up to the programme’s management to assess the feasibility of these actions and put forward specific recommendations, which the Commission could then put into practice in the future.

Box 9: Overview of identified measures that would improve the programme’s performance

Annex 1: Procedural information

1.Lead DG, Decide Planning/CWP references

DG Informatics (DIGIT), PLAN/2018/2654.

2.Organisation and timing

The Commission announced this interim evaluation in an evaluation and fitness check roadmap 97 on 29 May 2018.

Following the invitation by the Director-General of DIGIT in May 2018, the first Inter-service Steering Group (ISSG) meeting took place on 8 June 2018, with participants from DG CNECT, DIGIT, EMPL, FISMA, GROW, OP, SG, TAXUD and the JRC. The ISSG met five times between June 2018 and June 2019, as follows:

3.Exceptions to the better regulation guidelines

Not applicable

4.Consultation of the RSB (if applicable)

Not applicable

5.Evidence, sources and quality

Evidence & sources

Data have been drawn from two main sources:

1) Primary data were collected through the below consultation activities:

2)Secondary data were gathered by reviewing:

Quality

To support the ISA2 interim evaluation, in September 2018 the Commission tasked the Centre for European Policy Studies (CEPS) with delivering an independent evaluation study by May 2019. The ISSG members closely monitored the work of CEPS. They discussed the consultant’s various deliverables at the ISSG meetings and also commented on the documents in writing. The ISSG accepted CEPS final evaluation study by written procedure on 24 May 2019. This final ‘sign off’ by the ISSG also built on the quality assessment results, which concluded that the external contractor's study and work met the required quality standards.

Annex 2: Synopsis report of the consultation activities

Introduction

In line with the consultation strategy 98 , consultation activities were carried out to collect stakeholders’ views and opinions, thus contributing to the evidence base needed to achieve the two main objectives of the interim evaluation of the ISA2 programme:

Evaluating the implementation of the programme by considering seven evaluation criteria:

oRelevance ‒ the alignment between the programme’s objectives and the current needs and problems experienced by stakeholders;

oEffectiveness ‒ the extent to which the programme has achieved its objectives;

oEfficiency ‒ the extent to which the programme’s objectives are achieved at a minimum cost;

oCoherence ‒ the alignment between the programme and comparable EU initiatives as well as the overall EU policy framework;

oEU added value - the additional impacts generated by the programme, as opposed to leaving the subject matter in the hands of Member States;

oUtility ‒ the extent to which the programme meets stakeholders’ needs; and

oSustainability ‒ the likelihood that the programme’s results will last beyond its completion.

·Identifying areas of potential improvement and devising recommendations to improve the programme’s performance and bring more value to its stakeholders.

Feedback received on the evaluation roadmap

Before the interim evaluation was launched, one stakeholder provided feedback on the ISA2 evaluation roadmap93, noting that the ‘initiative is good, well planned, and funded’, but also that ‘it could have benefitted from more advertising’ 99 . This input was received in June 2018.

Types of consultation activities conducted

A mix of consultation activities was organised during the evaluation process to reach a variety of the programme's stakeholders. The following seven stakeholder groups were targeted:

Programme governance: Commission representatives from DIGIT’s Interoperability unit and members of the ISA2 Committee and Coordination Group;

Action owners: Commission representatives in charge of specific actions defined under ISA2;

Solution users: Commission and Member State representatives who are using/reusing ISA2 solutions;

Stakeholders responsible for linked EU policies/initiatives: Commission representatives who are in charge of EU initiatives linked to ISA2, and who are thus in a position to provide feedback, on the external coherence of the programme among other things;

·Experts: experts in IT and eGovernment able to perform an expert assessment of the programme and of the issues and developments in the field of interoperability;

Standardisation organisations: Representatives of standardisation organisations who can provide feedback on ISA2 activities related to standards and technical specifications; and

·Indirect beneficiaries and the wider public: Other programme beneficiaries or parties who are involved in or affected by the ISA2 programme in any way, including the private sector and the general public.

Between 29 November 2018 and 1 March 2019, five consultation activities were carried out in accordance with the Commission’s better regulation requirements:

Targeted online surveys (14.12.2018 – 23.1.2019)

Five online surveys were made available on EUSurvey, targeting the following stakeholder groups: programme governance, action owners, solution users, stakeholders responsible for linked EU policies/initiatives, and standardisation organisations.

The online surveys were initially set to close on 18 January 2019, but the deadline was extended to 23 January in order to accommodate some last-minute participants. Altogether 80 responses were collected through these surveys.

Targeted in-depth interviews (12.12.2018 – 23.1.2019)

Interviews were held with the same five stakeholder groups as the online surveys: programme governance, action owners, solution users, stakeholders responsible for linked EU policies/initiatives, and a standardisation expert. The interviews were based on questionnaires that were very similar to those used for the online surveys. They helped collect additional qualitative data from 20 stakeholders complementing the data collected from the surveys.

The in-depth interviews were initially supposed to run until 18 January 2019. However, as some stakeholders were not available in the original time-frame interviews continued until 23 January 2019.

Public consultation (7.12.2018 – 1.3.2019)

This activity gave all relevant parties — mainly the public, businesses, stakeholders' associations and public administrations, the opportunity to express their views on the ISA2 programme's achievements. Experts could participate in the public consultation and state their level of expertise in the programme and in the interoperability of digital public services. The public consultation 100 was available on EUSurvey in 23 EU languages and yielded 14 replies. This low response rate is due to two main reasons. First, ISA2 is a technical programme producing solutions addressed mainly to European public administrations; therefore, the direct contact between the average citizen/business and the programme is limited. Second, stakeholder groups that are in direct contact with the programme (e.g. solution users, action owners, etc.) were consulted via targeted online surveys and interviews.

ISA2 Mid-Term Conference short questionnaire (29 – 30.11.2018)

A short questionnaire based on the public consultation questionnaire was distributed during the ISA2 Mid-Term Conference (29 November 2018) 101 and during the Kick-off workshop (30 November 2018) of the interim evaluation. 15 stakeholders shared their views on the programme through this channel.

Kick-off workshop

The purpose of the kick-off workshop of 30 November 2018 was to present the interim evaluation and the planned consultation activities to the ISA2 programme’s stakeholder groups 102 . There were 40 participants in the workshop.

Methodology and tools used to process data

The consultation questionnaires were mainly based on ‘Likert scale’ questions, meaning that respondents were asked to give their opinion on a scale from (1) to (5) or from (1) to (4), depending on the type of question:

1 - not at all; 2 - to a limited extent; 3 - to some extent; 4 - to a high extent; 5 - to the fullest extent; or

·1 – definitely would not; 2 – probably would not; 3 – probably would; 4 – definitely would.

This approach helped the comparison of answers between respondents. In addition to the Likert scale questions, a number of open-ended questions were included to gather more in-depth information from consulted stakeholders. In addition, the interviews allowed more detailed feedback to be collected.

Characteristics of respondents

With a total of 129 respondents, the various consultations reached all of the following groups: programme governance, action owners, solution users, stakeholders responsible for linked EU policies / initiatives, standardisation organisations and the wider public ( Table 5 ). In addition, a team of technical experts assessed the programme and their conclusions are presented throughout the evaluation report.

Note that the results of the public and targeted consultation activities have been combined and analysed jointly in this report.

Table 5: Number of consulted stakeholders by consultation activity and stakeholder category

Note: Regarding the ‘Expert’ stakeholder group, four technical experts independently assessed the programme. In addition, most of the respondents from the consulted stakeholder groups are experts in interoperability and ISA2, as shown in Figure 30 and Figure 31 .

Source: CEPS final study

The consulted stakeholders claimed to have high levels of expertise both on the interoperability of public services ( Figure 30 ) and on the ISA2 programme as a whole ( Figure 31 ).

Figure 30 Knowledge of digital public services and interoperability (average score of answers, number of respondents)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent.

Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO).

Source: CEPS final study

Figure 31 Knowledge of ISA2 (average score of answers, number of respondents)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent.

Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO).

Source: CEPS final study

One slight exception is solution users who reported a relatively more limited knowledge of the entire programme compared to the other stakeholder groups consulted. Even so, most respondents are familiar with at least 5 packages of the 9 ISA2 packages of actions ( Figure 32 ).

Figure 32 Knowledge of ISA2 action packages (number of respondents familiar with a given number of packages) 103  

Source: CEPS final study

Regarding geographical distribution, solution users from various Member States provided input in the in-depth interviews and the online surveys:

At EU level, solution users who participated in the consultations work in EU institutions located in a number of EU countries: Belgium, Czechia, France, Hungary, Italy, Luxembourg, Poland, Portugal, and Sweden;

·At national and regional levels, solution users (including respondents who answered as individuals) who provided feedback come from the following 13 countries: Belgium, Bulgaria, Croatia, Czechia, France, Germany, Greece, Hungary, Italy, Luxembourg, the Netherlands, Portugal, and Sweden.

Feedback on the public consultation was provided by individuals in six Member States (Belgium, France, Germany, Greece, Spain, and the United Kingdom) and by public administrations from three Member States (Czechia, Greece, and Spain).

Results

The consultation activities revealed a general consensus among the different stakeholder groups consulted and confirmed that ISA2 performs well in all the seven evaluation criteria under investigation. The feedback from the consultation activities is summarised as follows:

Relevance

The respondents to the consultation activities emphasised that ISA2 is fully relevant with respect to the evolving needs and problems in the field of interoperability.

The majority of respondents indicate that: i) the needs and problems originally addressed by the programme are still experienced by European public administrations, businesses and/or the wider public; and ii) by achieving its general objective, ISA2 can address the needs and problems identified when the programme was established.

Respondents pointed out that there are other needs and problems currently experienced by stakeholders in the field, namely:

the need for a more binding legal framework for interoperability and a more prescriptive approach to designing interoperable public services;

the need to improve the way administrations communicate with one another;

the need to share best practices;

the resource constraints experienced by national and local public administrations;

the different political priorities of Member States that prevent a consistent approach to interoperability in the EU; and

·the limited awareness of ISA2 and other interoperability-related initiatives, especially at regional and local level.

In general, respondents to the consultation activities believe that the programme, through its general objective, the programme could address these new needs and problems to some extent, as some of the issues mentioned fall outside the scope of the programme.

Effectiveness

The results achieved so far by ISA2 are aligned with the programme’s objectives according to the consulted stakeholders, in particular when it comes to:

developing, maintaining and promoting a holistic approach to interoperability in the EU; and

·facilitating the reuse of interoperability solutions by European public administrations.

However, the results to date still fall short of the expected results, as most of the actions are ongoing and solutions are still being developed. For instance, respondents emphasised that relatively more could be done to achieve the expected result of improved efficiency and productivity of the European public administrations.

External factors could improve but also jeopardise the way in which the programme achieves its objectives and delivers its results. The vast majority of respondents confirmed that the calls for common standards and frameworks from public administrations are important external factors contributing to the performance of the programme. In contrast, institutional complexity in the form of multiple layers of governance and differences between national public administrations from various Member States and local public administrations within Member States could lead to coordination issues that hamper cross-border and cross-sectoral interoperability.

Efficiency

Respondents from the programme governance and action owners gave feedback on the efficiency of the process for selecting actions funded by ISA2. The process is considered relatively efficient and fit-for-purpose by the respondents from these two stakeholder groups. However, some measures could be taken to further streamline the selection process, such as:

simplifying the rolling work programme; and

·simplifying the overall process by launching thematic calls for actions.

Respondents from the same two groups also provided information about the amount of time they spent on preparing and submitting a proposal for an action to be considered by the ISA2 programme. On average, six person-days are required to renew a proposal for actions already included in previous rolling work programme. In contrast, between 10 and 30 person-days are required to prepare and submit a proposal for a new action.

Coherence

The consultation activities focused on both the internal coherence of the programme, i.e. the synergies or overlaps existing between the ISA2 actions, and its external coherence, namely the synergies or overlaps between ISA2 and other EU initiatives, policies, or programmes.

Internal coherence

ISA2 actions are characterised by substantial synergies with each other and limited overlaps. During the consultation activities, respondents provided examples of synergies to support their assessment. Examples mentioned include:

solutions developed under the ‘SEMIC’ actions are reused by the ‘Catalogue of Services’ and the ‘Access to Base Registries’ actions;

the ‘Re3gistry’ solution, part of the ‘ELISE’ action, is used to share the codes for the ‘Core Vocabularies’, part of the ‘SEMIC’ action; and

·The solutions developed as part of the ‘e-TrustEx’ actions are reused by the action ‘European Public Procurement Interoperability Initiative’.

External coherence

On external coherence, respondents to the consultation activities identified most synergy between ISA2 and CEF. In contrast, of the EU programmes interacting with ISA2, the ESF tends to have the least synergies. Respondents indicated that Horizon 2020 (in particular the work related to the once-only principle) and the SRSP (through the support given to Member States to align and integrate with cross-border services) also have some synergies with ISA2.

Respondents also pointed at synergies and overlaps with other EU initiatives such as ERDF, Corporate IT Governance and the Single Digital Gateway.

EU added value

Most respondents believe that national or sub-national interventions could not achieve or could only slightly achieve the general and specific interoperability objectives that ISA2 is pursuing. The consulted stakeholders consider that ISA2 plays an important role in ensuring a level of coordination that improves interoperability among European public administrations.

Respondents also emphasised that ISA2 can achieve its objectives at a lower cost than comparable national or sub-national initiatives. Most also believe that ISA2 has helped improve cross-border interoperability in the EU. Examples frequently mentioned by stakeholders are the work done in the field of semantic interoperability and the fact that ISA2 raises awareness about interoperability in the EU, without which little progress would be made in the field.

When it comes to advancing common EU policies, consulted stakeholders noted that ISA2 plays a central role in implementing the EIF (including its 2017 revision) by abiding by its principles and tracking its implementation. Stakeholders emphasised that ISA2 has also contributed to specific areas of broader EU policy, namely the Digital Single Market.

Utility

Respondents in both EU-level and national public administrations confirm that the solutions developed or maintained by ISA2 have contributed to addressing the original needs and problems identified in the field of interoperability. Respondents emphasised that while some of the new needs and problems experienced in the field are addressed by the solutions provided by ISA2, others, such as changing political priorities in the Member States, fall beyond the scope of the programme as it is currently defined.

The feedback received from respondents in terms of user satisfaction tends to be positive. Nevertheless, a number of measures were identified to increase the utility of the programme, namely:

placing more emphasis on sharing best practices and providing support to users;

establishing a co-creation process with users;

improving the quality of existing solutions by better considering user needs;

ensuring the Member States’ commitment to using ISA2 solutions; and

·strengthening the promotion of ISA2 solutions at Member State level as well as among specific groups of professionals (e.g. standards development organisations).

Sustainability

Overall, consulted stakeholders have a positive view of the sustainability of results achieved so far. Nevertheless, while the programme’s global results are expected to last (as the progress made in the field of interoperability will not be lost), certain issues may arise when considering the specific aspects of different solutions and future steps towards increased interoperability. In fact, there are solutions requiring both operations and maintenance costs as well as technical and operational support, which would be unable to deliver further results if the programme terminated. Additional obstacles to the sustainability of the programme include:

lack of development for existing solutions;

lack of coordination between European public administrations; and

·limited dissemination and communication related to interoperability.

Furthermore, introducing a fee for users to gain access to ISA2 solutions is not considered a feasible solution, especially in the event that funding would no longer be provided for the actions currently carried out by ISA2. Finally, the vast majority of respondents, from all stakeholder groups, stressed that the ICT-based modernisation of the public sector in Europe and the possibility to address the needs of businesses and the public via improved interoperability of European public administrations would be jeopardised, if ISA2 were terminated without any similar EU programme being established in its place.

Figures presenting the results of the consultation activities per evaluation criterion

Relevance

Extent to which needs and problems originally addressed by ISA 2 are currently experienced by European public administrations, businesses and/or citizens (average score of answers, number of respondents)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent.

Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO).

Source: CEPS final study

Extent to which achieving ISA2 objectives contributes to addressing the needs and problems originally addressed by the programme (average score of answers, number of respondents)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent.

Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO).

Source: CEPS final study

Extent to which achieving ISA2 objectives contributes to addressing additional (current) needs and problems identified by consulted stakeholders (number of respondents by stakeholder category)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent; (DK/NO) do not know/no opinion.

Source: CEPS final study

Effectiveness

Extent to which ISA2 solutions contributed to achieving the programme’s objectives (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study  Extent to which ISA2 solutions contributed to achieving the programme’s operational objectives (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Extent to which ISA2 solutions achieved the programme's expected results (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Extent to which the following external factors contribute to the performance of ISA2 (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Extent to which the following external factors jeopardise the performance of ISA2 (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Efficiency

Extent to which the action selection process is fit-for-purpose (number of respondents by stakeholder category)

Source: CEPS final study

Distribution of the number of days needed to prepare and submit a proposal for an ISA2 action (number of respondents)

Note: The histogram was constructed using a bin size of 2 days, which grouped the answers of respondents in intervals of 2 days. This bin size was deemed most appropriate after reviewing the answers provided.

Source: CEPS final study

Earned value, actual costs and planned value sampled actions, by package (Q3 2018) 104

Source: CEPS final study

Schedule performance indices of the sampled actions grouped by packages

Source: CEPS final study

Coherence

Extent to which there are synergies / overlaps between ISA2 actions (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Extent to which there are synergies / overlaps between ISA2 and other relevant EU programmes (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

EU added value

Extent to which national or sub-national interventions would be able to achieve the ISA2 objectives without the programme (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Ability of ISA2 to achieve the relevant objectives at a lower cost than comparable national or sub-national interventions (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Extent to which ISA2 contributed to improving cross-border interoperability in the EU (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Extent to which ISA2 contributed to the advancement of common EU policies (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Utility

Extent to which ISA2 solutions contributed to addressing the needs and problems originally covered by the programme: EU-level respondents (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Extent to which ISA2 solutions contributed to addressing the needs and problem originally covered by the programme: respondents from the Member States (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Extent to which ISA2 solutions contributed to addressing additional needs and problems identified by consulted stakeholders: EU-level respondents (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Extent to which ISA2 solutions contributed to addressing additional needs and problems identified by consulted stakeholders: respondents from the Member States (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Extent to which ISA2 solutions are meeting user needs in the fields of cross-border and cross-sectoral interoperability of digital public services (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Sustainability

Likelihood that results achieved so far would last if funding for actions covered by the programme would not be available in the future (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Extent to which financial, technical and operational aspects would prevent ISA2 solutions from delivering their results if the programme was terminated and not replaced (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Likelihood that the ISA2 general objective would be jeopardised if the programme was terminated and not replaced (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Likelihood that users would agree to pay a fee to continue accessing ISA2 solutions (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: CEPS final study

Annex 3: Intervention Logic

This Annex presents the ISA2 programme’s intervention logic, which comprises the rationale, the inputs, and the expected results of the intervention. This step aims to clarify the logic followed by EU decision-makers when establishing the ISA2 programme. The intervention logic includes a detailed description of the needs, problems and drivers that the programme intended to address, the programme’s objectives (with a three-level hierarchy: global, specific and operational objectives), its inputs/activities, its expected outputs, outcomes and impacts and the logical links between these components.

1.The rationale for the intervention: ‘why did the EU establish the ISA² programme?’

The intervention logic starts by understanding the rationale of the ISA² programme. At this stage, the reasons for its establishment were identified by (i) assessing the needs and problems that it aims to address; and (ii) outlining EU decision-makers’ main objectives.

Needs and problems 105

As highlighted by the Digital Agenda for Europe 106 , interoperability plays a key role in maximising the social and economic potential of ICT. Against this background, three specific needs can be identified in the field of interoperability 107 :

the need for cooperation among public administrations to enable more efficient and secure public services;

the need for exchanging information among public administrations to fulfil legal requirements or political commitments; and

·the need for sharing and reusing information among public administrations to increase administrative efficiency and cut red tape for businesses and the public.

The Digital Single Market (DMS) Strategy 108 confirmed these needs later on by recognising interoperability as a prerequisite for ‘efficient connections across borders, between communities and between public services and authorities’.

When it comes to problems, the ISA2 proposal 109 outlines one main issue, namely the existence of administrative e-barriers leading to a fragmented market.

This particular problem is affected by the following eight drivers (as pointed out in a series of documents including the ISA2 proposal, the Final Evaluation of the ISA programme 110 , the Commission Staff Working Document accompanying the DSM Strategy 111 ):

limited governance and coordination among Commission DGs and between Member States and the Commission on programmes and initiatives related to interoperability;

limited cooperation among dispersed institutional entities;

resource constraints in relation to interoperability;

internal organisational complexity;

limited ICT architectures and tools enabling interoperability;

limited availability of common frameworks, guidelines, and specifications;

limited awareness of the benefits of interoperability; and

·cultural fragmentation. 112

Objectives

The ISA2 objectives are presented in a hierarchical order, where the achievement of lower level objectives is normally a pre-condition for attaining the higher-level ones.

In this respect, the following three levels of objectives have been identified:

general objectives, concerning the overall rationale of an intervention and its longer-term and more diffuse effects, i.e. pertaining to the questions: ‘why has the ISA² programme been set up?’ and ‘what ultimate goal was it expected to contribute to?’;

specific objectives, providing a basis for assessing an intervention in relation to the short-term or medium-term results that occur at the level of direct beneficiaries/recipients of the intervention; and

·operational objectives, providing a basis for assessing an intervention in relation to its direct outputs, i.e. ‘what is directly produced/supplied during the ISA² programme’s implementation?’.

In concrete terms, the ISA2 programme’s objectives are the following:

General objective:

·To promote the ICT-based modernisation of the public sector in Europe and to facilitate addressing the needs of businesses and citizens via improved interoperability of European public administrations, thus contributing to the completion of the Digital Single Market and, ultimately, to economic growth and the global competitiveness of the European Union. 113  

Specific objectives:

Specific objective #1: To develop, maintain and promote a holistic approach to interoperability in the Union in order to eliminate fragmentation in the interoperability landscape in the Union (Article 1(a) of the ISA2 Decision).

Specific objective #2: To facilitate efficient and effective electronic cross-border or cross-sector interaction between European public administrations on the one hand, and between European public administrations and businesses and citizens on the other (Article 1(b) of the ISA2 Decision).

Specific objective #3: To contribute to the development of a more effective, simplified and user-friendly e-administration at the national, regional and local levels of public administration (Article 1(b) of the ISA2 Decision).

Specific objective #4: To identify, create and operate interoperability solutions supporting the implementation of Union policies and activities (Article 1(c) of the ISA2 Decision).

·Specific objective #5: To facilitate the reuse of interoperability solutions by European public administrations (Article 1(d) of the ISA2 Decision).

Operational objectives:

The operational objectives of the ISA2 programme are listed in the first column of Table 6 . The additional columns are relevant for the programme’s expected results (see below for further details).

Table 6: Overview of operational objectives and expected outputs of the ISA² programme 

Source: CEPS final study

2.The intervention: ‘what actions are supported by the ISA² programme?’

After identifying the needs, problems and objectives underlying the ISA² programme, the intervention logic approach requires the assessment of the main features (activities/inputs) of the programme itself. Within the intervention logic, activities and inputs represent the means to address the needs and problems, and to meet the identified objectives. In this context, these means largely correspond to the actions of the ISA² programme, as defined in the ISA2 Rolling Work Programme 116 . The ISA2 actions are grouped into the following nine packages.

Key and generic interoperability enablers are actions that develop interoperability solutions to support public administrations in providing services, ranging from the secure exchange of files to improving cross-border access to government data and information. Examples include TESTA 117 and Open e-TrustEx 118 .

The semantic interoperability package consists of initiatives to establish and increase semantic interoperability among public administrations through information standards and specifications. Examples include Core Vocabularies 119 and DCAT Application Profile for data portals in Europe 120 .

Under the access to data / data sharing / open data package, the ISA² programme funds actions that help facilitate the reuse of national data repositories, of data across borders and sectors and widen access to data created by the public sector. Example include ‘Providing big data opportunities for public administrations’ 121 and ‘FISMA: Financial Data Standardisation’ 122 .

The geospatial solutions package consists of legal/policy, organisational, semantic and technical interoperability solutions that aims to facilitate efficient and effective electronic cross-border and cross-sector interaction between European public administrations, and between them and citizens and companies, in the field of location information and services. This package led to the Re3gistry 123 solution.

The eProcurement/eInvoicing package helps update and improve existing EU tools for e-procurement. It aims to develop a common public procurement knowledge base to facilitate the creation, exchange, dissemination and reuse of procurement data. Examples include Open e-Prior 124 , ESPD 125 and eCertis 126 .

The decision-making and legislation package covers actions that support the decision-making process, in particular by improving interoperability between EU institutions and between them and the Member States. These actions address several processes – ranging from collecting feedback from various stakeholders to drafting new legislation and monitoring that this legislation is implemented. The package led to the LEOS 127 solution.

The EU policies — supporting instruments package covers actions that support the implementation of EU policies for which interoperability aspects need to be considered. The package has led to the following solution: Online Collection Software to support European Citizens' Initiatives. 128

The supporting instruments for public administrations package maps the EU’s interoperability landscape and provides solutions to support greater interoperability, ranging from sharing best practices to supporting the reuse of solutions.  Examples include EUSurvey 129 , Joinup – European Collaborative Platform and Catalogue 130 and the National Interoperability Framework Observatory (NIFO) 131 .

·The accompanying measures package consists of activities to raise awareness of interoperability and monitoring and evaluating the programme’s implementation. Examples include the annual SEMIC Conference 132 , the ISA2 Mid-Term Conference 133 and the ISA2 Dashboard 134 .

3.The expected results of the intervention: ‘what are the expected effects of the actions supported by the ISA² programme?’

When it began, the ISA² programme was expected to have certain effects (i.e. to cause changes) on certain categories of stakeholders. These effects fall under three categories (outputs, outcomes and impacts) depending on when they occur and the groups they address. Note that external factors and other EU policies may influence the performance of the ISA² programme.

Outputs (expected)

The outputs of the ISA² programme are its most immediate effects, i.e. the deliverables/products of the funded actions. As the intervention logic looks at a certain intervention by simulating an ex ante perspective, one must identify the outputs that were expected to stem from the programme when it was initially adopted. Note that expected outputs usually reflect the operational objectives identified in previous analytical steps. In this respect, the expected outputs and related indicators are listed in the ex-ante evaluation of ISA². Table 6 above summarises the expected outcomes and a series of key performance indicators for each operational objective of the ISA² programme.

Outcomes (expected)

The outcomes are the short-term and medium-term changes for the groups the programme directly addresses. As outcomes are usually connected to the specific objectives, they can be summarised as follows:

a coherent interoperability landscape in the Union based on a holistic approach to interoperability;

efficient and effective electronic cross-border or cross-sectoral interactions between European Public Administrations as well as between European public administrations and businesses/citizens;

more effective, simplified and user-friendly e-administrations in European public administrations;

advancement of Union policies and activities by supporting their implementation; and

·take-up and re-use of the ISA2 programme’s results by European public administrations.

Impacts (expected)

As mentioned above, the intervention logic also includes the impacts, i.e. the changes caused by an EU intervention over a longer period and also affecting the society as a whole rather than only the direct addressees of the intervention itself. These changes are related to the general objectives of a certain intervention. By analysing the ISA2 Decision and the ISA2 proposal, the following impacts are expected:

increasing the speed, efficiency and quality in the creation and delivery of cross-border and cross-sector electronic public services to meet the needs of businesses and citizens;

improving the efficiency and productivity of the European public administrations, which can be a strong driver of economic growth through its support for, and governance of, the private sector;

reducing the cost and administrative burden of cross-border interaction, removing administrative e-barriers and contributing to the swift implementation of ICT systems supporting EU legislation; and

·contributing to the successful achievement of the DSM.

Annex 4: Evaluation Framework

This annex presents the evaluation framework 135 on which the entire evaluation is based.