EUROPEAN COMMISSION

Brussels, 18.11.2022

SWD(2022) 720 final

COMMISSION STAFF WORKING DOCUMENT

Final evaluation of the European Interoperability Framework (EIF)

Accompanying the document

Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL

laying down measures for a high level of public sector interoperability across the Union (Interoperable Europe Act)

{COM(2022) 720 final} - {SEC(2022) 720 final} - {SWD(2022) 721 final} - {SWD(2022) 722 final}

Table of Contents

Executive summary

1.Introduction

2.WHAT IS THE EXPECTED OUTCOME OF THE INTERVENTION?

2.1Introduction to interoperability

2.2Introduction to the EIF

2.3Description of the intervention and its objectives

2.4Points of comparison

3.How has the situation evolved over the evaluation period?

3.1Policy context

3.2State of play

4.Evaluation findings (analysis)

4.1To what extent was the 2017 EIF successful and why?

4.2How did EU intervention make a difference?

4.3Is the EIF still relevant?

5.What are the conclusions and lessons learnt?

5.1Conclusions

5.2Lessons learnt

Annexes

Annex I. Procedural information

Lead DG: Directorate General for Informatics    

Organisation and timing    

Consultation of the RSB    

Evidence, sources and quality    

Annex II. Methodology and analytical models used

Overview of information and data sources    

Primary data    

Overview of the level of knowledge of respondents    

Secondary data    

Data validation    

Methods    

Main limitations    

Annex III. Evaluation framework for the EIF evaluation

Annex IV. Overview of benefits and costs

Annex V. Stakeholder consultation

Summary of the stakeholder strategy    

Summary of the feedback on the Inception impact assessment    

Consultation activities for the EIF evaluation and Impact assessment    

Results: EIF Evaluation    

Input of the Expert Group co-design process    

Annex VI. Supporting evidence from desk research

Annex VI.1. Scoreboards monitoring the implementation of the EIF

Annex VI.2. Overview of the adoption of NIFs

Annex VI.3. Overview of the mentions of EIF in a sample of EU pieces of legislation and policy documents

Annex VI.4. Evolution of the quality of digital public services and citizens’ use of the Internet to interact with public administrations between 2012 and 2019

Annex VI.5. Academic and grey literature outlining the needs and problems in the field of interoperability

Annex VII. Solutions contributing to the EIF implementation

Annex VIII. Mapping of legal initiatives to the EIF

Executive summary

Interoperability enables data exchanges and facilitates connections between IT systems and platforms in the delivery of digital services. It covers broader organisational, legal and governance aspects enabling fundamental user-centric principles like the once-only principle.

The EU has been supporting interoperability between public administrations since the 1990s through a series of funding programmes such as IDA, IDABC, ISA, ISA² and now Interoperable Europe . As part of this support, a European interoperability framework (EIF) was first produced in 2004 and then updated in 2010 and 2017 . With the introduction of the EIF for the first time in 2004, there was a first agreement on a common vision and non-binding guidelines to support data exchanges and digitalisation in public administrations in Europe. The implementation of a new EIF, adopted in 2017 as COM(2017) 134, was recognised as a key element in enhancing interoperable public services, especially in a cross-border context. It contains a set of principles, models and recommendations to guide public administrations across the EU and at all levels of administration in the design and provision of key digital and interoperable public services.

The EIF recognises the importance of cross-border interoperability as a means of implementing EU-wide policies. It provides guidance to public administrations on the design and update of national interoperability frameworks (NIFs), or national policies, strategies and guidelines promoting interoperability.

The evaluation shows that the EIF has certainly helped inspire European public administrations, on a strategic level, to design and deliver digital public services, and in the design and updating of (NIFs) or digital strategies. Quite a lot of Member States have aligned their digitalisation policies and plans with the EIF, or at least with its guiding principles. The framework has provided high-level advice through concepts and models to help guide the creation of interoperable digital public services. However, there is a disconnect between high-level guidance and actual execution or operational implementation, particularly at regional and local level.

The concepts, layered interoperability architecture, and conceptual model for integrated public service delivery set out in the EIF are generally synergic, and the suggestions mutually reinforce each other.

The significance of the EIF is attested to by the direct references to it in EU policy and strategy documents, but it is less often cited in EU legal acts like directives and regulations. Furthermore, the EIF is found to be consistent with initiatives such as Single Digital Gateway Regulation, the digital single market strategy, and the data strategy.

The voluntary nature of the framework is reflected in its uneven and ad hoc uptake at EU and Member State level. As a result, the availability of comparable data is limited, making comparisons between countries and sound cost-benefit analysis difficult.

Several steps for improving the EIF can be derived from experience with implementation:

·clarify the recommendations with a better structure, clarity, granularity, and implementation guidance, reducing some overlaps between them;

·promote the framework across polices and Member States, encouraging best-practice sharing and awareness-raising activities;

·reinforce the EIF’s role as a strategic guide by:

oenhancing EU interoperability governance and its fundamental role;

oincreasing the references to it in related EU legal acts;

ofostering coordination across EU and national polices through common interoperability rules;

oreinforcing EU policy for interoperability with more binding requirements;

ousing conditionality in EU funding instruments supporting digitalisation, enforcing interoperability by design and reuse of interoperability solutions;

·promote sound monitoring of progress in the field of interoperability and of the EIF’s effects.

A future enhanced EU policy for public administration interoperability could address those issues by:

ocreating binding interoperability rules;

oestablishing a system of governance that supports a more practical and actionable EIF;

oidentifying common reusable specifications and standards, tools, and solutions to support digital checks for new EU policies.

1.    Introduction

Digitalisation holds significant promise for improving public service delivery in Europe and generating benefits for public administrations, individuals, and businesses alike, such as time savings. Reforming internal processes, transforming the design and delivery of services, boosting data sharing and collaboration, adopting a user-centric approach, and adopting timely policies that keep up with technological changes are all part of the path to digital government 1 . All these changes and digital transformation are highly dependent on the level of interoperability of the underlying IT systems and platforms supporting digital services and data exchanges. If services are digitised in silos, data cannot be reused, shared and exchanged.

Interoperability can be defined as ‘the ability of organisations to interact towards mutually beneficial goals, involving the sharing of information and knowledge between these organisations, through the business processes they support, by means of the exchange of data between their ICT systems’ 2 . Legal, organisational, semantic and technical interoperability is crucial in the implementation of digital government policies. It enables the design of reusable specifications for data exchanges, or the agreement to use existing ones, and facilitates connections between IT systems. Both are key in the delivery of digital services. It goes beyond the design of IT systems, also covering broader organisational, legal and governance aspects. Interoperability is the foundation for the digitalisation of services, helping to put fundamental user-centric principles such as the once-only principle into practice. The once-only-principle is an  e-government  concept that aims to ensure that citizens, institutions, and companies only have to provide certain standard information to the authorities and administrations once 3 . The once-only principle is part of the European Union’s plans to further develop the  Digital Single Market  by reducing the administrative burden on citizens and businesses.

As emphasised in various EU-level initiatives in recent years, ensuring the interoperability of digital government has become a top priority, to maximise the potential of digital solutions for the single market. The European interoperability framework (EIF) adopted in 2017 as COM(2017) 134 4 (‘the EIF Communication’), was recognised as a key element in enhancing interoperable public services, especially across borders. It contains a set of principles, models and recommendations to guide public administrations across the EU and at all levels of administration in the design and provision of key digital and interoperable public services. Ultimately, it supports the free movement of goods, people, services and data throughout the EU.

The EIF is a set of guidelines – principles – similar to the better regulation guidelines, aimed at guiding and enhancing interoperability of digital public services at all levels across borders in the EU and within countries (at national, regional, local level). It is meant to be used by administrations at every level from EU-wide to local, to design and deliver seamless, integrated digital public services to be used by other public administrations, individuals and businesses. Furthermore, the EIF Communication was accompanied by support for implementation of the framework from 2016 to 2020, drawing on the ISA² and the Connecting Europe Facility programmes. Through the ISA² Programme, the Commission created and promoted interoperability solutions that support implementation of the EIF by the Member States, as described in the ISA² final evaluation report 5 .

According to the 2017 Communication on the new EIF 6 , its implementation needs to be evaluated to assess whether relevant recommendations should become part of a mandatory instrument. This need for a back-to-back evaluation and impact assessment was reaffirmed in a series of Communications in 2020, stating the need for a reinforced interoperability strategy for EU governments to foster coordination and the adoption of common standards for public services and data flows 7 .

Following the ‘evaluate first’ principle, this evaluation focuses on the new EIF introduced in 2017 and assesses its implementation, seeking to understand the achievements of the framework and the lessons that can be learnt to support the overarching goals of enhancing interoperability to support the digital transformation of the EU’s public sector.

The evaluation was based on five evaluation criteria stemming from the Commission’s better regulation requirements and nine evaluation questions in Annex III. A mix of data collection and desk research was used to gather both primary and secondary data (for further details on the methodology, see Annex II). In this, the Commission was assisted by a team of researchers led by the Centre for European Policy Studies (CEPS), which summarised its findings in an independent evaluation study 8 .

The collected data were validated via triangulation to ensure the robustness of the evidence 9 . Data were collected from multiple sources, and several different tools were used to analyse the data according to the evaluation criteria. Data were also used from the EIF monitoring mechanism exercise that takes place annually as part of the National Interoperability Observatory (NIFO). The monitoring mechanism is meant to provide a macro-level view of how Member States are implementing the framework. More specifically, as the evaluation relies on both qualitative and quantitative data, a variety of analysis methods was used. Qualitative information was aggregated, compared and summarised to substantiate the evidence resulting from the evaluation criteria. To analyse the data collected during the consultation activities, descriptive statistics were applied. Finally, to complement the findings, other quantitative analyses, including a standard cost model, an econometric model and text mining, were carried out when evaluating the effectiveness, efficiency and coherence criteria.

The mix of data sources and methods provided for solid conclusions. Nevertheless, several limitations may have affected the findings. These include the scarcity of data for assessing the costs and benefits of implementing the EIF at national level, the need for more time to be able to discern the longer-term results, and potential consultation fatigue among respondents.

2.    WHAT IS THE EXPECTED OUTCOME OF THE INTERVENTION?

2.1Introduction to interoperability

Interoperability is defined as ‘the ability of organisations to interact towards mutually beneficial goals, involving the sharing of information and knowledge between these organisations, through the business processes they support, by means of the exchange of data between their ICT systems.’ 10 Legal, organisational, semantic and technical interoperability is crucial in the implementation of digital government policies.

Interoperability is what makes digitalisation of government services efficient, enables the seamless and trusted exchange of data and connectivity of services across levels of administration, borders and sectors. It is a foundational element of all modern digitalisation strategies.

Digitalisation of government services without interoperability is possible, however, and unfortunately still very much the norm. Governments can certainly be digitalises in a way that does not allow for data exchange across borders or even inside the same country. However, this means foregoing the benefits of learning from best practices, and of reusing existing tools and solutions for lower development and maintenance costs; the wheel must be reinvented again and again.

For example, a well-designed online website for a national tax-return system may not be able to connect to the population registry, which means that taxpayers need to fill in the same data every year. And, possibly, if their signature is not valid in an electronic format, they will have to print out the declaration and go in person to the tax office to file it. And, if the tax database is not connected to a separate VAT database, businesses will need to enter the same information yet again, losing precious time and money.

But there are also digitally transformed systems where interoperability is built in by default, as in Estonia 11 . Individuals and businesses there spend an average of 3 minutes a year on their tax declaration. All the necessary information is already available, reused and exchanged between systems seamlessly thanks to the interoperability of data and systems.

Another example are the national mobile contact tracing and warning applications, which were set up in record time and work seamlessly for citizens thanks to a European interoperability gateway. At its peak, 19 European countries out of 22 that deployed national contact tracing and warning applications were connected to this gateway and were able to exchange information 12 .

The EU Digital COVID Certificate is also an outstanding example. It is based on common interoperability data models to enable a common reading, interpretation and verification of the data inside. It enables the verification of data originating from different national/regional health systems; semantic alignment (first, second, third dose, type of vaccine, types of tests); as well as organisational and legal cooperation certainty (authorised entities, business rules related to the acceptance of certificates). The system has been a global success, as in addition to EU/EEA Member States, also many third countries from five continents have joined it.

Interoperability enhances to the design of reusable specifications, the quality of seamless data exchanges, and facilitates connections between IT systems. All three are key in the delivery of digital services. Interoperability goes beyond the design of IT systems; it also covers broader organisational, legal and governance aspects. Interoperability is the foundation for the digitalisation of services, upholds fundamental digital rights and principles (like the once-only principle), and supports the design, interconnection, and delivery of digital services in a coordinated way so that data can flow across different domains and across borders. However, this requires strong interoperability foundations. Sound interoperability governance at both national and European level is a fundamental part of the legal, organisational and technical specifications towards those common goals.

The ambitious Communication on a digital single market strategy, put forward by the Juncker Commission in 2015 13 , the interoperability agenda needed to be revised and brought into line with the emerging challenges in the field of ICT and European public services. Among other relevant digital initiatives to interoperability, there is the Directive on reuse of Public Sector Information 14 (subsequently revised and renamed as the Open Data Directive in 2019), the INSPIRE directive 15 , the eIDAS Regulation 16 , the EU eGovernment action plan 2016-2020 , the single digital gateway (proposed in 2017 and subsequently adopted as a Regulation in 2019) 17 , and the European Cloud initiative .

2.2Introduction to the EIF

The EU has been supporting interoperability between public administrations since the 1990s through a series of funding programmes such as IDA, IDABC, ISA 18 , ISA² , and now Interoperable Europe . As part of this support, the first EIF was produced in 2004 and then updated in 2010 and 2017 . With the introduction of the EIF in 2004, there was for the first time an agreement on a common vision and non-binding guidelines to support data exchanges and digitalisation in Public Administrations in Europe.

Responding to the call for a revised framework for interoperability in the EU, a new EIF was adopted in 2017 as part of the Communication from the European Commission adopted on 23 March 2017 19 . The new 2017 EIF expanded on the previous 2004 version adopted by the PEGSCO committee of the IDABC programme and the 2010 version adopted by the Commission 20 . It brought more targeted recommendations, taking new technological and policy developments into account. The number of recommendations increased from 25 to 47 in the 2017 EIF and several recommendations were updated or newly developed to support relevant EU policies and initiatives.

The EIF establishes a set of guidelines and principles, similar to the better regulation guidelines. It aims to guide and enhance the interoperability of digital public services at all levels across borders in the EU and within countries (at national, regional and local level). The framework has three goals:

·to inspire and guide European public administrations in designing and delivering seamless European public services to other public administrations, individuals and businesses in a way that is digital, cross-border by default (i.e. accessible to the public in the EU) and open by default;

·provide guidance to public administrations on the design and update of national interoperability frameworks (NIFs), or national policies, strategies and guidelines promoting interoperability;

·contribute to the establishment of the digital single market by fostering cross-border and cross-sectoral interoperability for the delivery of European public services.

The EIF Communication was accompanied by an interoperability action plan to support the implementation of the EIF from 2016 to 2020, with a series of solutions supported by the ISA² and the Connecting Europe facility (CEF). Through the ISA² programme, the Commission created and promoted interoperability solutions that support implementation of the EIF by the Member States as described in the ISA² final evaluation report 21 .

The EIF focuses on cross-border interoperability between governments needing support to implement EU-wide policies. Given that digitalisation applies to all public administrations in all domains and at all levels, it is also applied directly in the countries for their own digital services and serves as inspiration for the creation of national interoperability frameworks (NIFs). Positive spillover effects were anticipated from providing Member States with common guidance to start working on interoperable services within their borders. Common interoperable governance structures, tools, specifications and services support seamless interactions across the various administrative levels in the country (central, regional, local levels), but also across some domains (justice, education, health and taxation). Most of all, the framework is a step towards developing interaction and cooperation required in a cross-border setting. Having a proper interoperability set-up within each Member State will help mitigate the complexities of coordinating interactions across borders that involve several domains and competent authorities.

The EIF helps remove barriers to the exchange of data, connect digital services across the various administrative levels of the country and across borders and sectors in the EU.

The framework is focused on supporting public organisations in working together on defining, designing and implementing digital services in a coordinated and efficient manner – starting by commonly designing and aligning back-office protocols, specifications, infrastructures, platforms and IT tools. However, it also covers fundamental issues for individual and corporate end users, such as accessibility, user-centric design and the once-only principle.

At EU level, relevant stakeholders are the EU institutions and policy units responsible for formulating, designing and implementing EU policies with digital implications. This includes units in DG GROW, DG CONNECT and the Joint Research Centre (JRC). As an example of EU policies, the Single Digital Gateway Regulation led by DG GROW implements a pan-European technical system using the principles of the EIF and some solutions from the interoperability action plan, like eDelivery and common vocabularies for cross-border and cross-sector evidence exchanges.

Member States’ digital government authorities and experts (government chief information officers, practitioners and academic experts) are longstanding close stakeholders, but the public bodies (ministries and agencies) mainly work on digitalisation and interoperability of digital services. In its factsheets on digital public administration and interoperability (Chapter 5), NIFO provides regular and up-to-date information on those authorities 22 . The information is also available in a dedicated section of the NIFO website 23 . Regional and local authorities have recently become more involved, particularly through the definition of an EIF-inspired framework for smart cities and local communities in the context of the Living-in.EU movement. The EIF, through the ISA² programme, has supported and encouraged the take-up of solutions by private companies involved in public sector development. Other important stakeholders are academia, and standard-setting, consulting and other private sector organisations (e.g. GovTech) that work closely with governments providing technical support.

The revised version of the EIF, which is subject of this evaluation, was created in a collaborative way with public administration experts and the national authorities in the Member States that are involved in designing and coordinating implementation of digital service policies or strategies. The text was discussed and agreed upon in several meetings between Member States’ representatives for the ISA² programme, from national digitalisation departments, and policy departments (Directorates-General) responsible for single market digitalisation. In addition, academia, standardisation organisations and ICT companies gave their views in several workshops. Finally, a public consultation was held 24 .

Their needs were considered and are an integral part of the EIF adopted in 2017.

2.3Description of the intervention and its objectives

To address the challenges identified, the EIF outlined several key objectives together with a set of inputs that comprise the main components of the EIF: the underlying interoperability principles of European public services, the layered interoperability model and the conceptual model for integrated public services provision, as well as the recommendations attached to each of the three components. The Communication on the EIF also included an interoperability action plan, detailing a set of key actions to be undertaken between 2017 and 2020 to support implementation of the EIF. As set out in the Communication, the Commission ensured the promotion, support and implementation of the Interoperability Action Plan mainly through the ISA² programme 25 . At the time of its adoption, the framework was expected to produce a series of outcomes and impacts.

The logical links between the needs and problems, objectives, inputs, and expected outcomes and impacts are summarised in Figure 1 (EIF intervention logic). The operational objectives and outputs are not included, as they were assessed during the parallel running evaluation of ISA² 26 .

The needs, problems 27 and drivers that the EIF intended to address were identified by analysing the 2017 EIF Communication 28 , the accompanying Staff Working Documents 29 , and the results of the consultation activities carried out in 2016 to inform the revision process 30 . The EIF was established to address one key need experienced by public administrations in the EU:

·the need for more specific guidance for public administrations on how to improve the governance of their interoperability activities 31 .

The Staff Working Document accompanying the 2017 EIF Communication 32 outlined two main problems in the field of interoperable digital public services:

·the fragmented delivery of digital public services in the EU and

·fragmentation in the organisation and format of public data in the EU.

These problems were attributable to five main drivers:

·incoherent governance of action to implement interoperability;

·cross-organisational barriers;

·resource constraints in relation to interoperability;

·scarcity of tools and solutions for interoperability in the delivery of public services; and

·legal issues affecting interoperability.

Source: Study supporting the evaluation of the implementation of the EIF .

The objectives of the EIF were presented in a hierarchical order; furthermore, the achievement of specific objectives was seen as a pre-condition for attaining the general objective.

General objective

·To help public administrations provide key interoperable, user-centric, digital public services to businesses and individuals, at EU, national, regional and local levels 33 , thus supporting the free movement of goods, people, services and data throughout the EU 34 .

Specific objectives 35

·Specific objective 1: to inspire European public administrations at all levels in their efforts to design and deliver seamless European public services to other public administrations, individuals and businesses which, where possible, are digital by default (i.e. providing services and data preferably via digital channels), cross-border by default (i.e. accessible for all in the EU), open by default (i.e. enabling reuse, participation/access and transparency), and which follow the once-only principle.

·Specific objective 2: to provide guidance to public administrations on the design and update of NIFs, or national policies, strategies and guidelines promoting interoperability.

·Specific objective 3: to contribute to the establishment of the digital single market (DSM) by fostering cross-border and cross-sectoral interoperability for the delivery of European public services.

·Specific objective 4: to account for technological developments and trends in the development of interoperability guidance and tools 36 .

As mentioned, the intervention logic of the EIF does not cover operational objectives, as they are covered in a parallel evaluation study on ISA² 37 .

What activities are supported by the 2017 EIF?

The EIF provides 47 recommendations that are grouped under the three main components of the EIF (an overview of the interrelations between the components is provided in Figure 2):

·Underlying principles of European public services. The EIF puts forward 12 interoperability principles to guide the design of interoperable European public services. The principles include the following:

othe principles of subsidiarity and proportionality (a case for EU action has to be made);

oa set of core interoperability principles: openness, transparency, reusability, technological neutrality, and data portability;

oprinciples on user needs and expectations: user-centricity, inclusion and accessibility, security and privacy and multilingualism; and

oprinciples for cooperation among public administrations: administrative simplification, preservation of information and assessment of effectiveness and efficiency.

·Layered interoperability model. The layered interoperability model includes the four layers of interoperability (organisational, semantic, legal, and technical interoperability) as well as a crosscutting layer relating to integrated public service governance, brought together into a comprehensive approach through a final and crucial component: interoperability governance.

·Conceptual model for interoperable public services. The concept is that European public services should be interoperable by design. The model consists of several basic components: coordination function, internal information sources and services, base registries, open data, catalogues, external information sources and services, and security and privacy protection.

Source: COM(2017) 134 final, Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, European Interoperability Framework - Implementation Strategy, Annex 2.

What was the expected outcome of the 2017 EIF?’

When created, the EIF was meant to deliver short-, medium- and long-term effects for certain categories of stakeholders. While the most immediate and operational results (the ‘outputs’) of the EIF were expected to emerge from relevant ISA² measures, it was thought that implementation of the EIF should in itself be able to generate results affecting European public administrations at all levels, as well as the public and businesses in the EU. These expected results are summarised below.

Outcomes (expected) 38

·European public administrations at EU, national, regional and local levels use the guidance provided by the EIF to effectively deliver digital public services responding to the needs of people and businesses, and to establish cross-organisational relationships supporting cross-border and cross-sector cooperation.

·National policies, strategies and guidelines consider and promote interoperability. EU legislation and policies consider interoperability implications, increasing the interoperability potential of the national follow-up measures.

·A coherent cross-border and cross-sector interoperability environment is achieved, supporting the digital single market in the EU.

·New technological developments are considered when designing interoperability guidance and tools.

Impacts (expected)

·Available interoperable, user-centric public services in the EU 39 .

·Developing a European public services ecosystem in which: owners and designers of systems and public services become aware of interoperability requirements; public administrations are ready to collaborate with each other and with businesses and the public; information flows seamlessly across borders, supporting the single market, particularly the digital single market 40 .

2.4Points of comparison

This evaluation focuses on the achievements and areas for improvement of the 2017 EIF since its adoption. By 2016, there was a relatively good alignment between the Member States’ NIFs and the guidance provided by the 2010 EIF. The average EIF-NIF alignment level across the EU in 2016 was estimated at 75%. However, at the level of actual implementation, more work remained to be done. The average NIF implementation level across the Member States in 2016 was estimated at 56% 41 .

In the run-up to the revision of the EIF, a public consultation was carried out to gather the views of stakeholders on existing problems and the future of the EIF. Among the key problems identified by stakeholders were 42 :

·lack of a comprehensive overview of existing interoperability initiatives;

·limited uptake of common standards in the development of solutions;

·lack of monitoring of implementation of interoperability initiatives;

·lack of both financial and human resources to support implementation of interoperability initiatives and solutions; and

·no assessment of costs and benefits of interoperability in the process of developing legislation at national level.

In addition, the public consultation brought to light the main expectation of consulted stakeholders for the future EIF: time and cost savings. In the consultation, 49 stakeholders (74% of the respondents) said they expected time savings and 46 (70%) expected cost savings and greater transparency 43 .

3.    How has the situation evolved over the evaluation period?

This chapter gives an overview of how the interoperability landscape has changed since the new EIF was adopted, both in terms of legislation and in terms of monitoring. This Chapter examines the policy context surrounding the adoption of the EIF before addressing the level of implementation.

3.1Policy context

The policy context of the EIF has been very dynamic since the adoption of the 2017 version. In recent years, digitalisation of public administrations and services has risen up the political agenda at EU and national level, as demonstrated by the Resilience and Recovery Funds and the national plans: EU Member States are planning to invest more in the digitalisation of their public administrations.

Back in 2017, the priorities for interoperability were reinforced through the Tallinn Declaration on eGovernment 44 . The high-level Ministerial Declaration emphasised the role of interoperability as a strategic component for ensuring the effective delivery of public services in the EU. A similar emphasis on interoperability is seen in a European Parliament resolution of 2017 45 . It underlines that the digitisation of public administrations should aim to ‘promote the better exercise of citizenship, improve the quality of life for citizens and the social and economic development of the regions, enhance citizens’ understanding of and involvement in public services’, while improving their efficiency and cost-effectiveness.

The digitalisation of public services has been progressing relentlessly, gaining even more momentum due to the pandemic. T he digital economy and society (DESI ) index and eGovernment benchmark report show that all EU Member States have made progress in the area of digitalisation, but the overall picture across Member States is mixed, and despite some convergence, the gap between the EU’s frontrunners and those with the lowest DESI scores remains large.

In 2019, the Council underlined in its conclusions on the future of a highly digitised Europe beyond 2020 46 that interoperability will continue to play a key role in the EU’s digital policy. The Von der Leyen Commission has set strong ambitions for digital transformation in the EU, as emphasised in the following Communications published in 2020 and 2021.

·Shaping Europe’s digital future, which sets out the Commission’s vision for the next five years on harnessing the potential of the digital transformation for the EU and its citizens, underpinned by European values. One of the key actions in the Communication is the development of a ‘reinforced EU governments interoperability strategy’ to foster coordination and the adoption of common standards for public services and data flows by 2021 47 .

·A European strategy for data, which proposes nine European data spaces that rely on interoperability to enable seamless data exchange 48 .

·The white Paper on artificial intelligence (AI) 49 and the subsequent proposal for a regulation laying down harmonised rules on AI 50 .

·A new industrial strategy for Europe for innovation and a greener and more digital industry, for which building EU leadership in key areas including data applications is essential 51 .

·The SME strategy for a sustainable and digital Europe, which identifies the opportunities that digitalisation holds for small and medium-sized enterprises (SMEs) in the EU; wider access to data and the reusability of data can support the development of innovative SMEs 52 .

·The long-term action plan for better implementation and enforcement of single market rules, focusing on better using digital tools to improve access to information, fostering cooperation between the Member States, and testing and applying advanced digital tools for public services 53 .

·The 2030 Digital Compass: the European way for the digital decade 54 , setting key priorities and targets for the digital transition in the EU over the next decade. When it comes to delivering digital public services, the Communication emphasises the need to ‘ensure interoperability across all levels of government and public services’ 55 .

In 2020, the Member States built on the Tallinn Declaration with the Berlin Declaration on digital society and value-based digital government 56 . The Member States and the EU institutions committed themselves to developing a more vigorous and more interoperability-focused digital transformation policy for government and to supporting the digital transformation of public administrations. In achieving these ambitions, interoperability plays a key enabling role, ensuring a level of coordination, common processes and models for the achievement of further goals.

The commitments included in these documents are also reflected in the budgetary allocations for the next several years. In particular, the new Multiannual Financial Framework includes a new programme to bolster the digital agenda in the EU called the Digital Europe Programme. With a planned budget of EUR 7.5 billion (at current prices) from 2021 to 2027, this new financial instrument will support five key areas of the digital transition: supercomputing, artificial intelligence, cybersecurity, advanced digital skills and European digital innovation hubs (EDIHs) 57 . The Digital Europe programme will ensure continued of funding for measures to increase interoperability in the EU.

3.2State of play

To understand the current state of play in the implementation of the EIF, it is important to note that the EIF is a non-binding instrument providing guidance that can be followed voluntarily by the Member States without a precise timeline. Therefore, delays in the implementation of the EIF are difficult to identify, given the lack of a specific timeframe (for more details, see Annex VI, Supporting evidence from desk research). Most of the Member States (15) have created an NIF based on the EIF, namely Belgium, Bulgaria, Estonia, Greece, Spain, Croatia, Italy, Cyprus, Luxembourg, Malta, Netherlands, Austria, Romania, Slovenia and Sweden. The others have an NIF or similar strategic documents partially inspired by the EIF. Only three countries, Ireland, Lithuania and Finland, have not implemented an EIF-inspired NIF.

Figure 3 reflects the extent to which the EIF has been taken up, at least conceptually, across the EU, based on data collected in January 2022.

Note: The legend corresponds to the extent to which EU countries have adopted NIFs that are fully or partly based on the EIF, as follows: (0) Light blue: countries that have not adopted an EIF-inspired NIF; (1) Middle blue: countries that have adopted NIFs or similar strategic documents partly inspired by the EIF; and (2) Dark blue: countries that have adopted NIFs based on the EIF. For each country, the year of the first adoption is displayed.

Source: Study supporting the evaluation of the implementation of the EIF .

The EIF monitoring mechanism (NIFO) delivers a high-level assessment of the progress in implementing the three main components of the EIF. It examines how the components have been adopted as guidance principles in the national strategies and polices.

Three scoreboards have been developed: the principles of interoperability, the interoperability layers, and the conceptual model for integrated public services provision. A set of 68 key performance indicators (KPIs) is used in compiling the scoreboards. The final output (see Annex VI.2) consists of a ‘traffic light’ assessment whereby the relevant KPIs are aggregated for each component of the scoreboard as follows:

[1] for the first scoreboard, the aggregation is done for each of the 12 principles;

[2] for the second scoreboard, the relevant KPIs are aggregated for each layer of the EIF interoperability model; and

[3] for the third scoreboard, the aggregation of the KPIs is carried out for each component of the conceptual model for integrated public services provision.

The ‘traffic light’ assessment produces a score value scale between 1 and 4, based on the aggregation of the KPIs mentioned above. The first results of the EIF monitoring mechanism are based on data collected in 2019. The results are presented in the 2020 digital public administration factsheets for each Member State 58 .

To gain an overview of the implementation of the EIF at EU level, the results of the monitoring mechanism exercise can be presented based on the progress in each scoreboard for all EU Member States (the full overview is presented in Annex VI.2). Overall averages per Member State per scoreboard were calculated, to provide a panoramic overview of those areas of the EIF in which most Member States score well and the areas where further efforts would be needed (see Figure 4, Figure 5 and Figure 6).

Note: For a full description of the principles, see Annex 2 to the 2017 EIF Communication.

Source: Study supporting the evaluation of the implementation of the EIF based on the data available on Joinup concerning the EIF monitoring mechanism and the underlying database provided by DIGIT.D2.

Source: Study conducted by Wavestone, 2022

Note: Countries are marked in grey where missing data for some sub-indicators make it impossible to calculate an overall average (see Annex VI.1 for a full overview).

Source: Study supporting the evaluation of the implementation of the EIF  based on the data available on Joinup concerning the EIF monitoring mechanism and the underlying database provided by DIGIT.D2.

Note: Countries are marked in grey where missing data for some sub-indicators make it impossible to calculate an overall average (see Annex VI.1 for a full overview).

Source: Study supporting the evaluation of the implementation of the EIF on Joinup concerning the EIF monitoring mechanism and the underlying database provided by DIGIT.D2.

Implementation of the 2017 EIF over time

The pre-2017 EIF version was much more theoretical, with narrower scope. At that time, the monitoring 59 mainly involved checking how well aligned the main principles in the national interoperability strategies were. Under the 2017 EIF, the level of implementation is also assessed. This is one of the reasons why it is not possible to make a direct comparison of progress in the Member States’ performance under the previous framework with their progress under the current one.

Nevertheless, small and positive improvements are apparent from a high-level comparative analysis of results from the EIF monitoring mechanism for 2019 and 2020.

Figure 4.b presents the improvements made by the Member States in implementing the recommendations linked to the principles of interoperability.

Several countries have improved their implementation of the principles. Portugal has seen the biggest improvement. Implementation is assessed by gauging how well aligned NIFs and interoperability strategies are with the EIF and, if needed, tailored and extended to address the national context and needs. Along with Portugal, the number of countries reporting a high level of alignment increased from 14 in 2019 to 16 in 2020.

With regards to interoperability layers, Greece and Poland have made most progress in implementing the recommendations, through increased efforts on governance. Results show significant improvements, particularly on holistic governance across all administrative levels and sectors, and the implementation of well-defined processes for selecting and adopting standards and specifications.

In the implementation of the recommendations related to the conceptual model, Sweden and Lithuania have made particular efforts to increase their level of alignment with the EIF conceptual model for integrated public services provision. They both increased their score by an average of 0.57 points, by putting in place catalogues of public services, open data and interoperability solutions.

Several external factors can affect the overall performance and implementation of the EIF, either positively, supporting the overall goals of the framework, or negatively, hindering the achievement of expected results. The factors were selected based on findings from the relevant literature and analyses of key trends relevant for interoperability initiatives in the EU. The following external factors were found to have a potential positive impact on the implementation of the EIF:

·technological advances bringing new opportunities to improve and foster the delivery of public services, which can make the need for interoperability more salient and thus provide a further incentive to ensure that interoperability principles and models are taken into account;

·the development of national initiatives on interoperability in the public sector, which can show the commitment to making progress on interoperability; and

·the deployment of cross-sectoral initiatives such as smart city initiatives, showcasing the relevance of interoperability at the intersection of different sectors.

Other external factors were found to have a potential negative impact and thus jeopardise implementation of the EIF 60 . These factors were:

·technical challenges, such as legacy systems;

·limited resources, reflecting, for example, difficulties in accessing funding and attracting skilled workers in the field of interoperability;

·institutional complexity, referring to the complex system of roles distributed between the different levels of public administration and of complexity stemming from legal requirements; and

·changing political priorities, which may impact measures to improve interoperability in a given country.

The impact of these factors on the performance of the EIF was tested through consultations with targeted stakeholders as described in Annex V.

EIF supporting instruments

The EIF is a theoretical framework, not a set of solutions. However, as a result of the associated interoperability action plan, a set of EU reusable IT solutions have been developed under the ISA² and CEF funding programmes to support the EIF implementation. Furthermore, the EIF toolbox 61 published on Joinup provides a catalogue of reusable solutions linked to the main areas, principles and recommendations of the EIF. Moreover, any interested stakeholder can take them up as they are based on open source specifications.

In fact, the EIF Communication was accompanied by action to support the implementation of the European interoperability framework over 2016-2020, through a series of solutions supported by the ISA² and CEF programmes. The Commission, through the ISA² programme, then created and promoted interoperability solutions that support implementation of the EIF by the Member States as described in the ISA² final evaluation report 62 .

Up to the end of 2020, the ISA² programme supported a total of 54 actions, grouped in nine packages, which were devised on a yearly basis through an annual rolling work programme. The nine packages are:

1.key and generic interoperability enablers;

2.semantic interoperability;

3.access to data / data sharing / open data;

4.geospatial solutions;

5.eProcurement / eInvoicing;

6.decision-making and legislation;

7.EU policies – supporting instruments;

8.supporting instruments for public administrations; and

9.accompanying measures.

ISA² managed 39 actions in its first year of operation, 43 actions in 2017, 53 actions in 2018, and 54 actions under both the 2019 and 2020 rolling work programmes.

In its digital policy hub , the National Interoperability Framework Observatory provides examples and case studies of national IT solutions and services implemented in public administration following the EIF principles. EU solutions that follow the EIF principles can be found in the EIF toolbox, which is now also starting to include national solutions.

Annex VII gives a comprehensive overview of solutions supporting EIF implementation that are available to Member States, as part of the interoperability action plan described above. Most of them are also accessible through the EIF toolbox 63 created to that end.

4.    Evaluation findings (analysis)

4.1To what extent was the 2017 EIF successful and why?

To assess whether the EIF has been successful, this chapter assesses how well it meets the following three criteria: i) effectiveness; ii) efficiency; and iii) coherence. The assessment draws on desk research, stakeholder and public consultations, and data collected from annual EIF monitoring across the 27 Member States.

The EIF has certainly helped to inspire European public administrations at all levels to design and deliver digital public services, as well as providing guidance to public administrations on the design and updating of NIFs or digital strategies.

Figure 3 in Chapter 3 shows that most Member States have drawn up a NIF or a similar national strategic or policy document that incorporates, at least in part, the EIF’s recommendations, principles, and models for delivering digital services 64 .

However, the EIF’s final achievement in helping to provide interoperable, user-centric public services in the EU and the development of a European public services ecosystem were limited, highlighting the need for more action and practical insights.

EIF Monitoring mechanism

The EIF monitoring mechanism, used for the annual monitoring exercise by the National Interoperability Framework Observatory 65 , is one of the Commission’s most important data sources for tracking roll-out of the EIF across the Member States. It provides an overview of NIFs (see Figure 3 in Section 3), which currently present a positive picture of EIF implementation progress. However, there is an important caveat: the EIF monitoring mechanism focuses on the higher level of conceptual take-up of the guidance and commitment to the framework’s principles, recommendations, layers, and high-level components.

The depth and breadth of EIF monitoring is limited by the cross-cutting nature of interoperability, deeply embedded in broader digitalisation efforts, and by the fact that public services are implemented and delivered by hundreds, or even thousands, of different authorities in some countries. Monitoring 66 is mainly done by gathering data directly from Member States through an online questionnaire and transforming the replies into score values for the various KPIs. The methodology for this is established under the guidance of the JRC’s Composite Indicator Unit. For some few KPIs only, the data are collected directly from other EU observatories that target specific domains or areas overlapping with the EIF, such as DESI, the eGovernment Benchmark report, the open data portal, or the Accessibility Directive.

Similarly, the overview of NIFs captures only the conceptual commitment, while stakeholders’ expectations are focused on the implementation level – implementing interoperability guidance at the service level. This is where a gap is apparent: between the conceptual level, where progress is generally positive, and practical implementation, where there is room for improvement in specific areas and where additional efforts are deemed necessary.

Evaluation of EIF’s main components

The EIF’s main components – the conceptual model, the layered interoperability model and the recommendations – are all useful, but are perceived as difficult to implement. As a result, more assistance appears to be needed for Member States in the form of guidance and systematic steps toward implementing the EIF’s recommendations. The risk is that the lack of granularity will allow for multiple interpretations.

There is undoubtedly a divide between acceptance of the EIF’s conceptual aspects and the need for more practical guidance in implementing interoperability requirements. There is also a need to raise awareness, particularly at local level. National public authorities are relatively confident in the EIF’s ability to provide public administrations with guidance on the design and updating of NIFs or strategies. However, they report that the framework’s voluntary nature and the scope of its recommendations restrict its ability to foster cross-border and cross-sector interoperability or take technological developments into account. This limits its contribution to the establishment of the digital single market.

Two factors can explain the limited success in achieving the objectives.

1.The EIF’s objectives are extremely broad and the recommendations usually high-level. Further clarity and guidance would be required to ensure that they are implemented by Member States.

2.In some sectors, a lack of collaboration between the private and public sectors impedes the growth of synergies.

To begin, the EIF’s principles are applicable to the development of interoperable digital public services in general. On the plus side, the principles of openness, technology neutrality, and data portability, as well as the reusability of IT solutions, transparency, and administrative simplification are deemed particularly critical, based on input from both targeted and open consultations, and are crucial for enhancing interoperable digital services.

Indeed, the EIF monitoring mechanism’s findings suggest that progress has been made toward implementing several of the principles identified as beneficial by stakeholders consulted. Most Member States are particularly aligned with the EIF in areas such as principle 2: ‘Openness’, principle 8: ‘Security and privacy’ and principle 10: ‘Administrative simplicity’ (see Annex VI.2).

However, several concepts are either too abstract (information preservation, effectiveness and efficiency assessment, and subsidiarity and proportionality), or they require additional clarification and practical focus to maximise their usefulness (transparency, technological neutrality and user-centricity). Further guidance is needed for a number of principles (including the assessment of effectiveness and efficiency, multilingualism, and subsidiarity and proportionality), as demonstrated by monitoring mechanism finding that better implementation is called for at Member State level.

There are also principles from the EIF that require further development 67 , such as multilingualism, information preservation, effectiveness and efficiency assessment, and subsidiarity and proportionality. This feedback is also corroborated by the EIF monitoring mechanism’s results, particularly in relation to principle 1: ‘Subsidiarity and proportionality’, principle 9: ‘Multilingualism’, principle 12: ‘Effectiveness and efficiency assessment’, and principle 7: ‘Inclusion and accessibility.’ Numerous countries have scores on the scale’s lower limit, (1 or 2), indicating the need for additional efforts to comply with the EIF’s principles and recommendations in their particular areas (see Annex VI.2).

While layered interoperability and the conceptual model are deemed beneficial for enhancing interoperable digital public services and are adequately addressed by the EIF, additional guidance could make them more actionable, as confirmed by both consulted stakeholders and EIF monitoring mechanism findings.

Layered interoperability could be further enhanced by applying the recommendations on the governance, organisational, and legal levels of interoperability. The evaluation study 68 demonstrates that greater emphasis should be placed on the non-technical aspects of interoperability, acknowledging that interoperability is a multifaceted notion, not limited to technical difficulties. Certain features of the semantic layer could also be addressed, such as the low rate of semantic adoption due to a lack of awareness or knowledge about the services end consumers.

Moreover, it emerged from the evaluation study that some stakeholders judged the organisational and legal layers to be incomplete and under-implemented. The organisational layer, in particular, requires clarification because it is believed to be too basic for practical applications in its current state. Finally, the layered model could be improved by elaborating on the interactions between the technological, organisational, and legal layers.

The monitoring mechanism’s findings corroborate these observations. The scorecard for the EIF’s layered model of interoperability demonstrates that Member States have made significant efforts to implement the EIF and its recommendations in this area, most notably technological interoperability at EU level. However, the scoreboard identifies a need to improve implementation of suggestions relating to the governance levels of interoperability, organisational interoperability, and legal interoperability.

Stakeholders contacted about the conceptual model confirm that it has helped to enhance interoperable digital public services to some extent. However, the conceptual paradigm is regarded as overly abstract and inoperative. As a result, Member States may interpret and hence implement it inconsistently, jeopardising the goal of interoperability. As such, experts and representatives from public administrations call for the establishment of an end-to-end operational approach to assist Member States in implementing interoperability by design in order to develop a comprehensive interoperability approach.

As regards the EIF’s coherence, the EIF’s principles, layered interoperability model, and conceptual model for integrated public service delivery are generally synergic. To improve the framework’s overall cohesiveness, the conceptual model might be more closely related to the other two components. According to the research, the strength of the national frameworks is contingent on an in-depth understanding and design of these three EIF components. The components’ interdependence is exemplified by the connection between the semantic layer, the ideals of openness and transparency, and the section of the conceptual model on openness. The principles and layered interoperability model are extremely synergistic, while enhancements to the conceptual model may result in more synergies.

The recommendations made throughout the EIF are largely self-reinforcing. Nonetheless, there are several instances where EIF guidelines overlap, at least thematically. While the overlaps may not cause problems, by better organising the recommendations, the important messages could be clarified, making the advice more actionable. On a more detailed level, the EIF monitoring mechanism’s findings suggest that various enhancements might be made to further support the conceptual model’s implementation in some Member States, notably, Germany, Ireland and Romania (see Figure 6 in Chapter 3).

Almost all countries do admirably in the field of ‘internal information sources and services’ (with scores of 4 almost everywhere), but efforts in some countries like Germany, Ireland and Poland would need to be stepped up (as signalled by scores of 1; see Annex VI.2). Additionally, progress on open data is encouraging, with all countries rating towards the top of the scale (scores of 3 and 4). Improvements could be made to cataloguing, external information sources and services, as well as security and privacy.

External drivers

External drivers such as technological advancements and national and cross-sectoral interoperability initiatives have contributed to some extent to the EIF’s observed achievements. The main constraining factors, to be minimised, are seen as limited resources and institutional complexity.

According to respondents in targeted consultations, external drivers do, on average, contribute to the framework’s implementation (see Figure 7).

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely.

Note: Averages do not include respondents answering ‘don’t know/no opinion’ (DK/NO).

Source: Study supporting the evaluation of the implementation of the EIF

However, the low average score assigned to cross-sectoral projects by national public agencies is particularly noteworthy. This score indicates primarily a lack of coordination between the public administrations of the various Member States. The COVID-19 pandemic has highlighted the critical nature of interoperable solutions. Nonetheless, several expert studies determined that cross-sectoral initiatives can play a significant catalytic role in advancing interoperability adoption. In this context, smart city efforts are instructive. There is significant opportunity to enhance such projects by ensuring that interoperability is a topic of discussion and, more significantly, a requirement, thereby reinforcing the EIF’s fundamental messages. There are considerable opportunities to achieve economies of scale for smart cities by reusing tools, data, and applications 69 .

Some unfavourable external influences have a similar impact on the EIF according to the respondents in the EIF evaluation consultations (see Figure 8). These are described below.

1.Scarce resources. budgetary limits, organisational difficulties, and a skills shortage. These obstacles stem mostly from a lack of human and financial resources, but also from the incompatibility of national databases 70 .

2.Institutional complexity 71 . This is undoubtedly the most challenging factor to solve, as it may be necessary to adjust the legislative framework in order to execute the EIF successfully, and institutional changes are slow to materialise. A new analysis on the influence of open source software on technological independence, competitiveness, and innovation in the EU economy identifies a fragmentation of duties in the field of digital policy, impeding collaboration on digital law-making 72 . This issue has a range of consequences for different countries, as some countries’ legal systems are more amenable to reform than others. Interoperability and institutional complexity are inextricably linked; this issue can be partially resolved by enhancing interoperability.

3.Limited awareness is another barrier that is deemed to jeopardise the EIF’s take-up, particularly at regional and municipal level and in the design of digital public services.

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely.

Note: Averages do not include respondents answering ‘don’t know/no opinion’ (DK/NO).

Source: Study supporting the evaluation of the implementation of the EIF .

EIF beneficial effects

The framework has had some beneficial effects. The EIF has to a certain extent contributed to the improvement of the quality of public administration services and to the promotion of free movement of products, services, capital, and labour across Member States. A strong alignment between the NIF and the EIF would surely boost the direct and indirect benefits for stakeholders (i.e. residents, enterprises, and public administrations) through better digital public services, particularly at cross-border level.

As explained above, some Member States have created an NIF based on the EIF, such as Belgium, Estonia, Greece, Spain, Cyprus, Luxembourg, Malta, Netherlands, Austria, Portugal, Romania and Slovenia.

Spain for example has passed an EIF-based NIF into law through a Royal Decree 73 . The purpose is to ensure that the systems and applications used by public administrations are technically, semantically and organisationally interoperable enough to enable rights to be exercised and duties fulfilled through electronic access to public services, while also enhancing effectiveness and efficiency. Thanks to the NIF, sound governance has been stablished across various administrative levels, and a set of supporting common services (eID, eSignature, eDelivery, eInvoicing…), infrastructures and technical specifications 74  have been devised for common and more efficient digital service provisioning. Thanks to this, public services were not disrupted during Spain’s first COVID-19 lock-down (the most severe in Europe along with Italy’s). The Spanish Chief Information Officer (CIO) highlighted common infrastructure and services as being among the key elements for the smooth delivery of public services during the first wave of the COVID-19 crisis. See below further details. In addition, thanks to Spain’s EIF-based NIF, data on taxation, social security, justice etc. data can be exchanged through a common interoperable platform 75 , used by all digital services in the country on a daily basis, reducing the administrative burden on people and businesses.

Likewise, on the 1 March 2019, Luxembourg’s Government Council adopted an NIF 76 which guides public sector bodies within its remit on how to attain a higher level of interoperability. This was Luxembourg’s first NIF bringing together in the same text all the elements needed for a coherent and clearly structured base for public sector interoperability. The NIF builds on the 2017 EIF whilst taking account of context and specific national needs. The NIF thus complies with the requirements set out in the Commission’s EIF and enables the Luxembourg government to make progress towards greater interoperability.

The Slovenian framework provides for a website 77 for publishing public sector interoperability solutions and products. It connects a catalogue of interoperability solutions with best practices for reusing its content.

Some NIFs set up the necessary governance to work together in a coordinated matter on interoperability-related topics supporting common goals and priorities laid down by the national digitisation policies, as well as relevant EU legislation like eIDAS, the Single Digital Gateway Regulation and the Open Data Directive. Such governance arrangements typically bring on board all ministries at central level, the regions and even the local government in the design and implementation of common processes, IT systems and technical specifications.

The report on public administrations’ digital responses to COVID-19 in Europe 78 shows that five countries (Czechia, Denmark, Estonia, Spain and the Netherlands) reported no disruption to their public services. This appears to be in line with what was outlined in the 2020 edition of the European Commission eGovernment benchmark 79 as four of these five countries are considered to have highly digitalised public services, scoring well above the European average. The delivery of public services in four of these five countries is also considered to be highly interoperable with scores above the European average on the EIF. This is also further corroborated by the Danish, Estonian Spanish and Dutch national CIOs, who stated that the already high level of digitalisation in public services delivery helped to ensure the continuity of these services throughout the first wave of the COVID-19 crisis.

An example of an EU solution which puts these generic interoperability guidelines into practice is the EU interoperability gateway for COVID-19 contact-tracing and vaccines. A national/bi-national solution is Estonia and Finland’s X-Road 80 platform , linking all public digital services.

Estonian experience demonstrates that the EIF aided in the development of a wide variety of digital and cross-border public services relating to company and property registration (for more details, see also the above section covering the impacts of external factors). Additionally, the EIF contributes to the free movement of commodities, services, capital, and labour throughout Member States. Experts and academia, as well as businesses and business organisations, are generally more optimistic about the framework’s potential benefits for research, development, and innovation.

With regard to implementation of the EIF, IMAPS is an online survey developed under the ISA² programme, helping public service owners evaluate, consider and improve all key interoperability aspects of their digital public service (legal, semantic, organisational, or technical), in compliance with the EIF . Some public bodies have used the tool on a voluntary basis to make their administrative procedures fit for purpose for the single digital gateway, assessing the level of digital maturity of a public service that needs to be provided online by 2023 and accessible via the gateway. Examples include online submission of corporate tax declarations or applications for a European health insurance card (EHIC). Some countries (Belgium, Czechia and Slovenia) and the city of Valencia have used the survey to detect areas for improvement and to make digital services more mature from the EIF perspective.

IMAPS publishes an annual report on the interoperability maturity of European digital public services assessed using IMAPS. The latest IMAPS report (2020) reveals that all assessed digital public services are on average below the essential interoperability level (IMAPS maturity level 3).

However, this EIF evaluation shows the limitations on providing a deeper and wider analysis of the level of EIF implementation across the hundreds and thousands of bodies in the 27 Member States. To do systematically, sound interoperability governance would be needed, involving the main national authorities dealing with digitalisation, in close coordination with the regions, local authorities and the European Commission. A common process for apply IMAPS would be beneficial, or a similar tool aligned with main interoperability and digitalisation goals stemming from the national and |EU strategies and policies. The aim would be to ensure that any interoperability maturity level assessment helped to monitor, rationalise and steer all implementation plans and investments, and find gaps, overlaps, inefficiencies and ways to bridge them.

The EIF can serve as a genuine catalyst for public-government dialogue. It is possible to quantify both the impact of adopting an EIF-based NIF and the maturity level of digital, online public services to gain additional insight into the range of possible EIF impacts and improvements in digital public services.

An economic analysis demonstrates that an EIF-based NIF, or an NIF with EIF-based components, is more conducive to public-government interaction than no EIF-based NIF. Additionally, a 10% improvement in the quality of online public services would encourage an additional 4.3 million to 5 million EU residents to connect with their governments via the internet each year.

Figure 9 and Figure 10 respectively report the results obtained under two different regressions:

1.a ‘complete’ EIF-based NIF compared to an ‘incomplete’ EIF-based NIF or no EIF-based NIF; and

2.a ‘complete’ or ‘incomplete’ EIF-based NIF compared no EIF-based NIF.

Note: The results (i.e. when NIF ‘complete’=1 and NIF ‘incomplete’=0) compare countries having a fully EIF-based NIF with countries having a partially EIF-based NIF or no EIF-based NIF at all. The results assume linearity over time, i.e. the average effect of having an NIF on online interaction with public administrations is assumed to be constant over time.

Source: Study supporting the evaluation of the implementation of the EIF

Note: The results (i.e. when NIF ‘complete’=1 and NIF ‘incomplete’=1) compare countries having a fully or partially EIF-based NIF with countries having no EIF-based NIF or similar strategies. The results assume linearity over time, i.e. the average effect of having an NIF on online interaction is assumed to be constant over time.

Source: Study supporting the evaluation of the implementation of the EIF .

This analysis is conducted at the macroeconomic level and is based on binary variables indicating whether or not the countries analysed have approved an NIF or comparable national strategic documents based entirely or partially on the EIF. As a result, the analysis does not account for the effects of the practical implementation of specific interoperability standards, but rather for the effects of strategic advice in general. The analysis determines the average impact throughout the sample examined. The magnitude of the effects may vary by country.

Additionally, the findings confirm the association between increased service quality and the fact that improved eGovernment services enable people to interact more with their government via the internet. More precisely, a 10 % increase in the eGovernment index results in an increase of 1.5-1.7 % in people’s use of the internet to connect with public authorities while all other variables remain constant.

On the efficiency front, the data indicate that benefits may outweigh costs in some cases. While public administrations bear the costs of implementing the EIF, streamlining procedures and facilitating data exchange are seen as benefiting a broader range of stakeholders by improving interactions (i.e. saving time and money) between public administrations on the one hand and between public administrations and users of public services on the other.

Most of the EIF’s benefits stem from two factors.

1.Affirming the importance of administrative simplicity. By streamlining and simplifying the digital delivery of public services, EIF implementation costs were negligible in comparison to the cost savings associated with interactions between public administrations, and between administrations and their constituents.

2.Encouraging data reuse. The EIF’s goal is to assist public administrations in avoiding duplication of efforts and effectively leveraging existing resources and information, resulting in better service quality.

Concerning expenditure by public administrations in implementing the EIF, national and EU public administrations gave consistent responses. Supporting the framework’s roll-out is regarded to be reasonably inexpensive, at both national and EU level. In the Netherlands, for example, an independent and external advisory body (ACTAL, the Dutch advisory board on regulatory burden) has been set up to assist the government and parliament on minimising regulatory burdens.

Additionally, short-term advantages must be weighed against long-term benefits. While initial investments can be costly, as procedures are optimised, operating costs tend to decrease over time.

The deployment of NIFs inspired by the EIF clearly also resulted in various benefits.

1.The EIF facilitates synergies in the implementation and design of new services, contributing to the reuse of a diverse range of software programmes collected in the Italian reuse catalogue, for example.

2.Interoperability has resulted in cost savings associated with various ICT projects. Centralised authorisation of ICT projects is estimated to have saved Czechia more than EUR 39 billion in public investment in 2019. Additionally, some ICT projects filed by public agencies were rejected due to their failure to meet interoperability criteria. Notably, public administrations were able to rethink and adapt their project ideas, resulting in further savings, by depending on the sharing and reuse of existing digital solutions and shared government services.

While voluntary adoption of the EIF and its recommendations has undoubtedly resulted in benefits recognised by public administrations, more should be done to foster an EU-wide strategy that is really integrated. However, a comprehensive evaluation is impossible to make due to the framework’s broad scope and non-binding character.

Cost-benefit analysis

In measuring and comparing the expense of adopting the EIF and applying these costs to NIFs, many difficulties arise due to the voluntary nature of the framework. These difficulties are due to the varying cost of digitalisation across different public sector bodies and the different approaches taken by countries in implementing the EIF and the varying degrees of digitalisation. Comparisons between Member States are complicated, as costs may vary in terms of coverage and over time, even when cost differences are clear (see Box 1).

Moreover, interoperability is an underlying support function for digital transformation; for this reason, many of the assessments conducted by Member States are based on digital government transformation costs. Defining specific interoperability costs requires complex research and analysis. The consultation with the stakeholders nevertheless provided some reference points for the costs linked to producing a strategy or an NIF document and bound mainly to consultation services.

The complexity of EIF implementation make it hard for Member States to estimate the associated costs. Many different authorities and public services are involved and it is difficult to isolate EIF implementation from the overall digital transformation processes. This is also the case for the implementation of EU policies. The team working on the single digital gateway conducted a study 81  to estimate the cost of implementing the once-only principle where interoperability plays a major role. Its conclusions, summarised below, show that the costs were hard to quantify.

1.The scarcity and low comparability of data makes it difficult to assess the costs and benefits of incorporating the EIF into national frameworks. The voluntary nature of the framework is reflected in its uneven and ad hoc uptake across the EU, limiting the availability of data and making comparison between countries difficult. Furthermore, implementation of the EIF has not followed the same timeline in the different countries. To overcome this limitation, the study team contacted the national public authorities that responded to the consultation activities and indicated their availability to participate in a follow-up interview, to gain insight on the time spent (in terms of person-days) on the incorporating the EIF, or part of it, into their national framework.

2.The overarching scope of the EIF hinders the assessment of the direct costs and benefits stemming from its implementation. Costs and benefits deriving from the EIF do not relate exclusively to implementation of the framework; costs and benefits also depend Member States’ levels of digitalisation, the degree of centralisation and varying organisational structures involved in delivering public services. The costs related to interoperability initiatives are borne at different levels of administration in different countries. The cost discrepancies are due to the following factors.

·The extent to which information technology has been digitised. Member States that have already deployed interoperable solutions in line with the EIF can quickly build on pre-existing solutions, resulting in reduced costs and responsibilities. Additionally, governments that rely on open source solutions mitigate the risk of lock-in. As a result, solutions can be more easily transferred or altered, resulting in cost savings.

·Catch-up effects. Because public services have differing degrees of centralisation and organisational structures, the costs of interoperability projects are borne at varying levels of administration in different countries, as stated in the expert assessments accompanying this study. In Sweden, for example, costs are borne primarily by individual agencies. In comparison, the role of national government initiatives and support is substantially stronger in neighbouring Norway and Denmark. It is often difficult to assess the cost of ICT projects since they can contain significant hidden costs such as the involvement of existing workers in modifications required for the new operation design, such as training, or time spent on transitional operations.

These constraints underline the need to identify common criteria so that Member States’ costs can be assessed consistently.

Furthermore, incorporating EIF guidance into the national framework is likely to have delivered different benefits in the different Member States (see Box 2).

The expense of translating the EIF, or parts of it, into national frameworks is difficult to assess and ranges from EUR 24 000 to EUR 169 000 (based on the estimates of time translated into costs provided by representatives of several public authorities in the Member States).

The impacts of a public sector interoperability policy are more difficult to identify and quantify than those of many other EU initiatives. A report by the JRC 89  analysed the economic impact of public sector interoperability overall. It found that, due to its very nature as an enabling factor across many policy fields, the impact of interoperability is hard to isolate from other aspects of a policy. Interoperability in its many forms is an important enabler but it is interoperable policies and public services that deliver benefits (and consume resources).

Nevertheless, the evidence is quite conclusive. For instance, in terms of economic impact, the study by the JRC found that, at a conservative estimate, the public could save up to 24 million hours a year through improved interoperability. This would represent monetary savings in the order of EUR 543 million annually. For businesses, improved interoperability would lead to time savings of 30 billion hours. In monetary terms, this translates into savings of EUR 568 billion annually. Improvements in public sector performance due to interoperability could lead to an increase of 0.4% in GDP. Overall, the potential positive benefits derived from increased interoperability for the public sector in the EU are considerable. For instance, converting these estimates into monetary terms, a 1 % improvement in the indicators used, through greater interoperability, would boost EU GDP by EUR 21 billion - EUR 56 billion.

Future monitoring should focus on reusing developed technologies and providing better estimates of costs. In this regard, a set of EIF-specific criteria could make it easier to compare costs across the EU (including, for instance, the daily cost of work by the team tasked with implementing the EIF at national level, the cost of partial implementation of the EIF, the number of consultations between the European Commission and the Member States when the EIF was agreed on and introduced, the time spent by annual reporting under the NIFO initiative, etc.).

The present evaluation study also shows the current limits to proper correlational analysis between digitalisation and interoperability, and the difficulties in decoupling the two effects in the estimates. Future work is needed to come up with sound methodologies for estimating the cost of interoperability, as opposed to the costs of digital transformation in a broader setting.

Coherence

The EIF is found to be consistent with initiatives such as the Single Digital Gateway Regulation, the digital single market strategy, and the data strategy. Both strategies highlight the EIF’s role in digitalising EU public services. In addition, the 2017 EIF and measures to monitor its implementation reflected measures outlined in the eGovernment action plan 2016-2020. Moreover, the 2015 DSM strategy 90  called for revision of the EIF to better complement the strategy’s goals of digitalising public services in the EU. The 2017 EIF heeded this call. The interoperability action plan 91 included key steps to be taken to further support the vision of digital and interoperable public services. The ISA² final evaluation study highlights the programme’s contributions (as the main EIF implementation instrument).

Moreover, the adoption of a revised EIF Communication in 2017 helped with the successful completion of two actions in the e-government action plan for 2016-2020, namely actions 4 and 6. It was developed as part of the NIFO initiative, which is funded by the ISA² programme to monitor the framework’s implementation in the Member States. The NIFO action also included an EIF toolbox . The toolbox maps possible interoperability solutions produced as part of the ISA² and CEF programmes, which public administrations can use to implement EIF recommendations. The toolbox also links the EIF’s suggestions to their accompanying solutions.

Many of the EIF principles or 47 recommendations have been targeted by other provisions (examples given already include open data, SDG for once-only, privacy and data protection (GDPR), accessibility, etc.) This can be seen from a first mapping of digital-relevant EU initiatives onto the EIF pillars and individual recommendations, part of the EIF toolbox resources (see Annex VIII). However, some parts of the EIF are not yet specifically targeted by the EU regulator, such as interoperability governance or reusability.

As examples of coherence at EU level, the SDG Regulation mentions a number of ideas and fundamental concepts promoted by the EIF, such as the potential for reusing an ISA² solution and the core public services vocabulary application profile (CPSV-AP). The Regulation itself implements a pan-European technical system using EIF principles and some solutions from the interoperability action plan, such as e-delivery and common vocabularies.

To implement the once-only principle, Article 14 of the Regulation obliges Member States to exchange evidence for 21 administrative procedures listed in Annex II to the Regulation. The EIF is fundamental to this endeavour. Implementation of harmonised common data models has already started for eight evidence types, to enable data to be shared according to semantic interoperability principles and fully in line with EIF guidance (semantic layer principles). These evidence types include the birth certificate, the absence of criminal records and academic qualifications. At Member State level, this involves firstly the central state digitalisation units. The last mile is taken care of by regional and local governments who host digital services in most cases and are strongly impacted by EU policies. Solutions for eDelivery and for interconnecting authoritative sources of data on people and businesses are also designed in line with principles and solutions stemming from the EIF conceptual model for integrated digital public service delivery.

The data.europa.eu portal has also been set-up using the EIF as inspiration, reusing a data model 92   created under the ISA² programme, to catalogue data and link up the open data sets from EU institutions, agencies and bodies, as well as European countries national and regional catalogues in an interoperable manner, according to the principles of the EIF conceptual model

Data portability is one of the main driving forces of the EIF. As another example, Article 28 of the data act draft proposal provides for common specifications and core vocabularies to underpin the interoperability of cross-sector and cross-border data spaces, fully in line with EIF principles and recommendations, to ensure interoperable data sharing services and flows. To ensure strong interoperability governance, the draft assigns a major role to the European Data Innovation Board, at least for public sector services. Where there are different approaches between sectors and even Member States on privacy, data protection, responsibilities, common use cases, data flows and data specifications, this can result in a lack of legal certainty for the sharing of data across domains, policy areas and borders. The data act proposal is meant to address this lack of legal certainty, in conjunction with EU data policy and data governance. In this regard, it is aligned with the legal interoperability concept promoted in the EIF, ensuring that organisations operating under different legal frameworks, policies and strategies can work together.

Health is another practical example. Patients typically have medical records that are hosted in a hospital or medical centre as a digital record. If the person moves to another location or country, transferring the file to another hospital digitally can be problematic since there are no, or few, common formats or solutions widely used in Europe. The solution is to agree on the pieces of information for the record, and then on common formats for the records.

This will be part of the European Health Data Space 93 . It has to be done in a way that not only health service providers, such as hospitals, but also other data users, for instance research institutions conducting medical research, can easily interpret the data and combine it with other data sources, such as mobility, socio-economic or environmental. This is one of the reasons why data spaces should be interoperable across themselves, and minimum technical requirements are needed. Cooperation and ongoing work in the area of digital health and health data use is applying EIF principles and actively seeking interoperability support.

ISA² solutions also help to implement the EIF and more broadly interoperability initiatives at EU and Member State level.

Other than those mentioned, the EIF aligns with Horizon 2020, the INSPIRE Directive, and the smart cities/communities projects. Moreover, the EIF’s overlaps and inconsistencies with other projects are restricted.

More direct references to the EIF could help reinforce the idea that such an EU-level framework has broad consequences, and that interoperability is vital for projects wanting to leverage the benefits of digital transformation.

An examination of rules, directives, decisions, and Commission communications from 2004 to 2021 demonstrates that the EIF’s interconnections with other EU programmes might be strengthened. We discovered that:

1.the EIF has been considered in the preparation of some legislation in areas where interoperability is vital;

2.the EIF is mentioned in relevant communications, with a spike in 2020, indicating that public sector interoperability is becoming a more important policy issue. Most references are found after the 2010 EIF was adopted, although a few more references have been identified since the 2017 EIF was published.

Figure 11 lists the EIF and related keywords in regulations, directives, and decisions from 2004 to April 2021.

Source: Study supporting the evaluation of the implementation of the EIF based on the EurLex dataset and the EurLex website

Finally, more could be done to strengthen the position of the EIF as a guiding, strategic document. It is important to ensure that alignment with the EIF is not limited only to the declarative level, but that it is fully taken on board in implementing new policies. A future interoperability strategy and policy could definitely contribute in this regard.

4.2How did EU intervention make a difference?

National or sub-national initiatives would make a very limited contribution to the EIF’s aims. Consulted public authorities saw only limited potential for national and sub-national efforts to contribute to the EIF’s existing aims, with substantial reliance on national expertise and improvements in digital public services. Indeed, there are countries at both ends of the spectrum: those that are more advanced and experienced and those that are less advanced and experienced. This heterogeneity may have a detrimental effect on interoperability at EU level.

Heterogeneity of solutions and methods may impede EU-wide interoperability. More crucially, the cross-border dimension cannot be addressed sole through the efforts of national or sub-national governments. Even if national, regional, and local public administrations strive to improve their approach to digital and interoperable public services, without coordination, the strategies and solutions selected may impede cross-border interoperability. Thus, the EU can serve both as a coordinator, ensuring a consistent approach to interoperability throughout the EU, and as a stimulant, offering direction and support.

The Tallinn Declaration of 2017 and the Berlin Declaration of December 2020 94  both emphasised the importance of a unified strategy to interoperability. Additional evidence from desk research corroborates these conclusions. The list of Member States that have adopted an NIF or comparable strategic documents (Figure 3 and Annex VI.2) demonstrates that several countries have incorporated the EIF into their national frameworks, and in some cases, the EIF served as the starting point for such papers (as it is the case, for instance, for Croatia).

Benefits at EU level

By using economies of scale (via shared technologies) and economies of learning, the EIF is almost certain to achieve its objectives at a lesser cost than equivalent national or sub-national projects (dialogue and shared solutions).

The EIF decreases costs through economies of scale (via common tools) and economies of learning, as confirmed by responses to the evaluation consultations (dialogue and shared solutions). Initiatives at the national and/or local level would be substantially more expensive. Additionally, the EIF generates positive externalities as a result of stakeholder collaboration. Notably, the ISA² programme has aided in the development and sharing of tools and solutions by serving as one of the primary instruments for executing the EIF, namely the interoperability action plan 95 .

A uniform EU strategy eliminates the costs associated with each Member State assessing and devising actions to address challenges on an individual basis. Additionally, a lack of coordination is likely to result in divergent paths being taken, resulting in increased fragmentation rather than a shared vision.

On average, the EIF has aided in the advancement of shared EU policy. However, more could have been accomplished. The EIF is underutilised across sectors and by Commission departments. Cooperation between other Commission departments should be strengthened to enable the EIF to produce stronger outcomes. A future interoperability governance, which is now being evaluated as part of the back-to-back study of a future interoperability policy, may help to close this gap.

Although the EIF has given value to the EU, cross-border interoperability remains limited and driven by sectoral needs. The benefits of cross-border interoperability, as promoted by the EIF, have not been shared equitably throughout Member States.

Without a doubt, the framework might make a greater contribution to the development of a European interoperability environment. Identifying and addressing significant cross-border demands would very certainly result in increased EIF take-up and consequently increased EU added value.

The EIF’s primary contribution to cross-border interoperability is in raising awareness of the need for interoperability and in encouraging countries to implement NIFs based on a common model. The EIF contributes to cross-border interoperability in three ways.

1.It has increased awareness of the multi-tiered approach to interoperability that goes beyond technical considerations. The EIF establishes a consistent vocabulary and conceptualisation of interoperability (including the interoperability layers and conceptual model), which can serve as a common starting point for future projects in the field.

2.It encourages Member States to implement NIFs based on a common model, fostering an EU-wide approach to interoperability. The EIF promotes conversation and cooperation. eIDAS, the electronic personal identity recognition and management system, has been adopted by 15 Member States, supporting people when moving permanently or temporarily, or completing official procedures. E-invoices are another example of an EU-developed interoperable solution successfully reused at national level.

3.Specific major digitalisation polices in the EU – such as the Single Digital Gateway Regulation (SDGR) or the Open Data Directive – have been designed in the implementation phase of the polices, following the guidance of the EIF and reusing some solutions developed under the EIF, such as eID, eDelivery and interoperable data models. A first mapping of digital-relevant EU initiatives onto the EIF pillars and individual recommendations shows this internal coherence. The SDGR is contributing to the interoperability of digital services and implementation of the once-only principle in many countries, such as Germany, Greece, Italy and Portugal. They have national plans – sometimes funded through the resilience and recovery plans (in Italy's case) – to develop interoperable and data sharing platforms, infrastructure and portals, common semantic models, and internal coordination and cooperation across various administrative levels, in the spirit of implementing the once-only principle in parallel at home. With regards to the Open Data Directive, the national open data portals – that have a set-up following the Directive in Member States such as in Belgium, Spain, Ireland, Italy, the Netherlands, and Sweden 96 – are based on a common EU interoperable semantic model developed under EIF principles, guidance and solutions. Thanks to the semantic model, Member States can easily foster their national portals as interoperable hubs connecting other data repositories in the country, mainly from regions and big cities, so that data sets can be found easily and reused in an interoperable manner. Likewise, these portals are interoperable and integrated since April 2021 in data.europa.eu, the official portal for European data, https://data.europa.eu , which relies on the same specification and their counter partners in the other countries in a cross-border setting, allowing citizens and businesses in Europe to search for data sets hosted in any Member States of competent authority in a user centric way.

Alignment at international level

On a global scale, the Organisation for Economic Cooperation and Development (OECD)’s work on digital government lends itself to comparison with the EIF. The ‘Recommendation on Digital Government 97 ,’ approved in 2014 and consisting of high-level recommendations, and the ‘digital government toolkit’ 98 in particular, are relevant, with the toolkit giving assistance in implementing the high-level proposals. The OECD’s approach is, however, larger and less granular than what the EIF sought to do.

The EIF is strongly aligned with OECD recommendations on interoperability; nevertheless, when it comes to recommending the implementation of standards, it does not expressly refer to standards development organisations (SDOs).

Most open data standardisation activities occur at the international level through the W3C and OASIS, as documented in the chapter on ‘Big data, open data, and public sector information’ in the 2022 rolling plan for ICT standardisation 99 .’ Additionally, it should be emphasised that the rolling plan now makes a reference to the EIF. Moreover, the Commission states in its Communication on European data strategy 100 that ‘the use of standard and shared compatible formats and protocols for collecting and processing data from diverse sources in a consistent and interoperable manner across sectors and vertical markets should be encouraged through the rolling plan for ICT standardisation and (in the case of public services) a strengthened European Interoperability Framework.’

To summarise, the EIF could promote further adoption of standards created by standardisation bodies. The ICT standardisation rolling plan could go beyond just referencing the EIF and specify possible actions involving standards development organisations. Finally, given the EU’s uniform approach to interoperability, the EIF may help ensure that standardisation initiatives support the European data strategy in line with the notion of digital sovereignty for Europe.

4.3Is the EIF still relevant?

The EIF mainly aims to assist public administrations in providing vital interoperable, user-centric digital public services to businesses and residents at EU, national, regional and municipal level.

Some issues with EIF design persist. Among these are the need to raise awareness about interoperability, solve skills shortages in public sector IT departments, enhance investment capacity, keep up with rapid technological change, and support public-private collaborations.

Nonetheless, the EIF appears to have attained some of its key objectives based on input from the targeted consultation.

The EIF’s early goals were to inspire and guide European public administrations at all levels to develop and provide seamless European public services. While the EIF has issued recommendations, it is insufficiently implemented.

Public administrations would still benefit from a common, more practical, end-to-end approach from the EIF, from policy to implementation. This would increase its relevance. Stakeholders seem to agree that more direction and awareness-raising is required among public authorities, especially at sub-national level. More guidance goes hand in hand with promoting interoperability at all levels. More ambitious measures are expected to raise public administrations’ knowledge of the EIF’s potential benefits, especially at sub-national levels 101 .

Cultural characteristics, especially the diversity of the public sector and associated actors 102 , can play a major role in implementing interoperability solutions. A silo-based strategy, for example, produces fragmentation owing to user-unfriendly systems, which hinders the exploitation of synergies 103 , according to the impact assessment accompanying the SDG proposal. Some stakeholders mentioned the linguistic barrier. Interoperability is thus not primarily a technology issue, but an organisational and informational one, influenced by conventions and values. Taking cultural issues into account, especially the diversity of public sector and associated actors, can play a vital role in developing interoperability solutions (see also Box 3).

Relevance at EU level

Digital (and interoperable) transformation of the EU public sector requires greater EU cooperation, common rules and initiatives. The COVID-19 epidemic has undoubtedly hampered cooperation and necessitated stronger governance mechanisms. The crisis has highlighted the significance of digitisation and interoperability in the public sector, not just to improve basic public services but also to build resilience and capability to deal with unexpected emergencies. Studies into the pandemic’s impact and the EU’s response provide evidence. During the epidemic, countries with more digitalised and interconnected governmental administrations and services reported less service disruption (Charay et. al., 2021). The pressing need for data transfer across countries has helped raise awareness of the importance of interoperable systems, particularly for crisis management and response. The field literature supports targeted interoperability approaches.

The pandemic also highlighted the need for better governance of pandemic response mechanisms across the EU. Data exchanges, compatible systems and processes, and shared standards all play a role. The lack of data consistency hampered the EU’s capacity to respond cooperatively early in the outbreak (Renda and Castro, 2020). Interoperability by design was emphasised as vital for effective cross-border contact-tracing apps in the EU (Ciucci and Gouardères, 2020). The argument around the EU Digital COVID certificate included discussions about standardisation and interoperability.

To promote digitalisation initiatives and the creation of a cohesive interoperability environment across the EU, a more binding approach to interoperability requirements may be required in future. Interoperability by design and reuse of interoperability solutions could potentially be explored as ways to improve the cohesiveness of the interoperability environment across the EU. The usage of conditionality and the need for more interoperability collaboration are among the conclusions of the ISA² interim and final evaluations.

Stakeholders emphasised the EIF’s potential benefits in other areas like mobility, environment, and smart cities. Encouraging public-private partnerships is also possible, as public services are not always generated and delivered solely by the public sector.

5.    What are the conclusions and lessons learnt?

5.1Conclusions

Interoperability makes for efficient digitalisation of government services, and enables the seamless and trusted exchange of data and connectivity of services across levels of administration, borders and sectors. It is a foundational element for any modern digitalisation strategy.

The 2017 EIF is a non-binding framework for interoperability in the EU public sector (a first version of the EIF was developed in 2004; a new version EIF was formally adopted in 2010 through a Commission Communication). The EIF has issued suggestions and guidance on interoperability to enable public administrations to establish digital and interoperable public services. A whole set of interoperable and reusable solutions have been created as part of the interoperability action plan, and supported by the ISA² and CEF programmes to make the EIF more actionable and implementable. This report has assessed the framework’s achievements and faults, as well as lessons learnt, four years after its introduction.

Successful elements of the 2017 EIF

Because it represents an outstanding (but non-binding) reference approach to interoperability (see Section 4.2) the EIF brings EU added value. National or sub-national initiatives would contribute very little to the EIF’s goals. Particularly, national or sub-national governments cannot achieve cross-border interoperability alone. Economies of scale (through shared technologies) and experience enable the EIF to achieve its goals more cheaply than equivalent national or sub-national projects (dialogue and shared solutions).

The framework has provided high-level advice through concepts and models to help guide the creation of interoperable digital public services. In this regard, the EIF monitoring mechanism shows that some of the EIF’s recommendations have been conceptually implemented by the Member States, such as openness, technological neutrality, data portability, reusability of IT solutions, and the layered interoperability model (except for the interoperability governance layer), including open data and base registries.

In terms of efficiency, the EIF’s benefits may outweigh the expenditures incurred by public agencies to implement it (see Section 4.1). Implementing the EIF guideline, or parts of it, through national frameworks may cost between EUR 24 000 and EUR 169 000 per Member State (based on estimates of time translated into costs provided by representatives of several public authorities in the Member States). Also, short-term investments must be weighed against long-term rewards. While initial expenditure can be costly, implementing EIF advice is projected to reduce costs over time as processes are optimised.

The concepts, layered interoperability architecture, and conceptual model for integrated public service delivery defined in the EIF are generally synergistic, and the suggestions mutually reinforce each other (see Section 3.1.3). In terms of external coherence, the examination indicates that the EIF has been considered in areas where interoperability is critical (see Section 3.1.3). Public and targeted dialogues as well as desk research have identified synergies between the EIF and the single digital gateway, the digital single market strategy, the data strategy and the upcoming data act. The EIF’s role in digitalising EU public services is highlighted by the digital single market and data strategies. While the Single Digital Gateway Regulation does not expressly address the EIF, it does mention several of its major principles and concepts. However, these are more commonly mentioned in contexts other than the SDG and the two strategies mentioned, demonstrating that public sector interoperability is becoming more of a policy debate topic.

Similar to the EIF, the OECD’s Digital Government initiative provides advice on setting up digital and interoperable public services. The OECD’s methodology is wider and less specific than the EIF’s. The EIF thus adds value in the EU.

Less successful elements of the 2017 EIF and areas for improvement

Whilst the EIF as an EU-level intervention has an added value to the interoperability environment (see Section 3.2), its ultimate impact is dependent on national and sub-national implementation across all sectors with digital public services. The EU’s solution and strategy heterogeneity may impede interoperability.

Given the EIF’s low overall success and the need for additional practical direction, it is judged just moderately useful. The EIF recommendations have only partially helped meet the goals set when the framework was adopted in 2017. This is because the EIF has wide aims and the suggestions are often high-level. More granularity is required for efficient framework implementation, especially at the local level. A few concepts (evaluation of efficacy and efficiency, subsidiarity and proportionality, multilingualism, and information preservation) are too abstract or require additional elaboration to be effective. Also, the principles of openness, technological neutrality, and user-centricity require more clarification and practical direction. The EIF monitoring mechanism found that certain principles need to be improved at country level, indicating the need for further guidance (including the assessment of effectiveness and efficiency, multilingualism, and subsidiarity and proportionality). Further advice is needed to the make the layered interoperability and conceptual models more actionable, according to the study. For example, the layered interoperability model’s advice on interoperability governance, organisational, and legal interoperability should be improved (as emphasised throughout the consultation activities and through the results of the EIF monitoring mechanism).

Internal coherence could be improved by better organising the components. Assemblage of the three components should be improved. The EIF’s suggestions typically reinforce each other. Nonetheless, numerous EIF recommendations overlap, at least thematically (e.g. Nos 2, 42 and 43; and Nos 21, 22, 23 and 24). The overlaps are not a problem, but better organising the advice would make the big ideas clearer and more actionable. In terms of external coherence, the EIF is openly mentioned in policy documents and less so in EU legal acts like directives and regulations. On coherence, the EIF is found to be consistent with initiatives such as the Single Digital Gateway Regulation, the digital single market strategy, and the data strategy. To increase interoperability governance and the EIF’s role, more references in associated legal acts are needed.

While, as an EU-level intervention, the EIF has added value to the interoperability landscape (see Section 3.2 and Annex VIII), ultimately its impacts depend on the level of practical implementation at national and sub-national level and across all sectors with digital public services. Heterogeneity of solutions and strategies across the EU may hinder interoperability at EU level.

Will the identified needs be met, and problems resolved, over time?

The EIF has only partially addressed the demands and challenges indicated by the stakeholders (see Section 3.3). Public administrations emphasise the need for further cooperation, citing inadequate EIF implementation guidance. In this context, a more practical, end-to-end EIF approach should deliver additional benefits. The digital (and interoperable) transformation of the EU public sector requires unified norms and policies. Other public sector interoperability needs include raising public knowledge of the EIF, addressing staffing shortages in IT departments, increasing investment capacity, and encouraging private-public cooperation.

In this respect, as stated before external factors such as technical improvements and national and cross-sectoral interoperability initiatives help the EIF to deliver. At the same time, negative aspects such as limited resources and institutional complexity must be mitigated (see Section 4.1). Finally, the COVID-19 epidemic has heightened the need for greater coordination and cooperation to improve interoperability in the EU public sector.

In summary, the needs and issues identified are expected to persist. To satisfy the needs and overcome the challenges outlined, more cooperation, common rules, and coordinated EU initiatives are required. It is worth exploring the possibility of putting in place sound governance to enhance cooperation in a future interoperability policy.

5.2Lessons learnt

The EIF’s actual results only partially match the predicted outcomes and consequences mentioned in the EIF’s intervention rationale (see Section 2). As a model for national interoperability policies and plans, the EIF has given high-level guidelines promoting the development of interoperable digital public services across the EU. Most Member States have adopted NIFs or equivalent national strategic papers that incorporate EIF suggestions, principles, and models. Over the last two years, a small but a steady improvement has been observed in general in the level of implementation of the framework across the three main areas, namely the principles, the interoperability layers, and the conceptual model.

However, there is a disconnect between high-level guidance and actual execution, particularly at regional and municipal level. So far, the EIF has helped build the digital single market by promoting cross-border and cross-sector interoperability, but more can be done. Developing a European public services ecosystem and making interoperable, user-centric public services widely available in the EU are key priorities.

Improving interoperable digital public services would surely boost direct and indirect benefits for stakeholders. An economic analysis reveals that adopting an NIF based on the EIF, or only part of it, is preferable to not adopting it at all. A 10% improvement in online public service quality would also attract 4.3 million to 5 million more EU residents each year to interact with public authorities online.

In connection with the European Commission’s REFIT project, enhancements to the EIF methodology could reduce administrative costs by streamlining processes and assist the production of more efficient digital public services through improved interoperability.

Another difficulty is the scarcity of data, especially comparable data, for assessing the costs and benefits of translating the EIF into national frameworks. The voluntary nature of the framework is reflected in its uneven, ad hoc uptake across the EU, limiting the availability of data and making comparison between countries difficult. Furthermore, implementation of the EIF has followed different timelines in different countries.

Several lessons can be derived from the experience of EIF implementation in recent years. These lessons will be helpful in further enhancing interoperability activities at EU level.

1.Clarify the recommendations that are generally sound, although they might be enhanced with better structure, clarity, granularity, and implementation guidance. The recommendations could be enhanced by: i) arranging them by target stakeholders (national government, data owners, and local agencies); ii) distinguishing between basic and advanced recommendations; iii) bridging the conceptual and practical gaps. The recommendations could be refined to better support public administrations in implementing the EIF.

2.Give practical guidance by strengthening the links between EIF and ISA² solutions. The EIF Toolbox helps to connect the EIF’s suggestions to available solutions. Future EU public sector interoperability policies should provide an integrated approach, integrating strategic initiatives with activities to increase adoption of standard interoperability solutions. To make the EIF more actionable in the future, it must be linked to specific solutions.

3.Encourage best-practice sharing. Given the EIF’s strategic and overall high-level recommendations, best practices for implementation might be added. In this context, the NIFO collection on Joinup’s digital policy hub presents examples of trends across the Member States. The hub might be maintained in the future, and more examples of Member States’ specific implementing solutions could be collected, building on the EIF’s guidelines.

4.Foster coordination through common rules. Common rules could improve digital public services across the EU and strengthen the public sector’s resilience to unanticipated shocks. The EIF’s voluntary framework has helped promote interoperability in public service design, but an EU public sector interoperability policy might benefit from a compulsory approach and might consider the following.

a.Enhanced EU policy for interoperability with more binding requirements. Coordination is required to create an EU-wide digital public service landscape. Having EU-wide laws would help to ensure that cross-border aspects of digital public services are considered.

b.Use of conditional interoperability solutions and interoperability by design in the development of digital public services could help achieve interoperable cross-domain and cross-border services throughout the EU. Such a framework may combine, for example, EU financing for public sector transformation and digitisation with interoperability standards.

5.Strengthen the EIF’s role as a strategic guide. Enhancing synergies between the EIF and EU interoperability, and digitalisation efforts, is critical for a cohesive EU interoperability environment. In this regard, more direct references to the EIF may assist raise awareness of its broad implications. It is also vital that other EU programmes are completely aligned with the EIF in terms of policy implementation.

6.Educate and involve stakeholders. Interoperability has long been a technical topic, but its broader ramifications are becoming apparent, with the EIF’s help. To keep EU action in this subject relevant, it is necessary to:

a.promote interoperability as a fundamental enabler for public sector digitisation.

b.ensure that stakeholders are aware of the benefits stemming from interoperability and thus incentivise public administrations to carry out coordinated initiatives for digital public services that are compatible, so contributing to a more cohesive public sector interoperability environment across the EU.

7.Ensure monitoring of progress in the field of interoperability and the EIF’s effects, to keep building a solid understanding on the EIF’s effectiveness, especially at the local level. Monitoring initiatives should thus continue to gather data on interoperability guidance implementation and costs, and encourage reuse of developed technologies. In addition, the EIF monitoring mechanism might be improved to better capture practical acceptance of recommendations in developing digital public services.

8.The EIF could promote the adoption of open specifications and standards created by standardisation bodies. The EIF is already referenced in the ICT Standardisation rolling plan, and some semantic interoperability specifications are mentioned. Finally, given the EU’s uniform approach to interoperability, the EIF may help ensure that standardisation initiatives support the European data strategy and the data act proposal, in line with the notion of digital sovereignty for Europe.

9.Future monitoring should focus on reusing developed technologies and on providing better estimates of costs. In this regard, a set of EIF-specific criteria could make it easier to compare costs across the EU (including, for instance, daily work of the team tasked with implementing the EIF through official documents, the degree of implementation of the EIF, the number of consultations between the European Commission and the Member States when the EIF was agreed on and introduced, the time spent by annual reporting under the NIFO initiative, etc.).

10.In terms of efficiency and the quantification of cost-benefits stemming from interoperability, future sound methodologies to estimate the costs of interoperability, as opposed to the costs of digital transformation in a broader setting, must be implemented and applied in the field.

For all of it, a future enhanced EU policy for public administration interoperability is now being assessed along with this EIF evaluation (back-to-back impact assessment for a future interoperable Europe policy). The policy will address binding requirements and a governance supporting a more practical and actionable EIF. In addition, it will identify common specifications and standards, reusable tools, and solutions to support at least digital checks for new EU policies.

Annexes

Annex I. Procedural information

Lead DG: Directorate General for Informatics

Decide Planning: PLAN/2020/7507 - ‘Evaluation of the implementation of the European Interoperability Framework (EIF) and proposal for an EU governments interoperability strategy’.

The Initiative is part of CWP 2022 - REFIT initiatives, point 12, and is referenced under the Policy objective ‘A Europe fit for the Digital Age’. The adoption is planned for Q2 (June 2022).

Organisation and timing

Consultation of the RSB

An upstream meeting with the RSB took place on 12 March 2021, whose recommendations were duly taken into account.

This draft EIF evaluation SWD was submitted to RSB on 17 December 2021 for the RSB hearing that was held on the 19 December 2022, getting a positive opinion with some comments for improvement.

The Table below summarises the changes introduced to this Staff Working Document (SWD) in response to the Board’s main comments

Evidence, sources and quality

The Commission has been working for more than 25 years now on interoperability at EU level with different stakeholders involved in the digital transformation of the public sector and creation of EU-wide digital public services.

The evidence used in the Impact Assessment have been collected from the following main sources and has been used at different steps in the preparation of the initiative:

·National Interoperability Framework Observatory (NIFO) data collection:

oThe Digital Public Administration Factsheets collected and published yearly; available on the Joinup platform 107 ;

oThe European Interoperability Framework (EIF) Monitoring Mechanism 108 ;

·Other databases including: Eurostat, the World Bank Worldwide Governance Indicators, United Nations e-Government survey, and the Organisation for Economic Co-operation and Development (OECD) Digital Infrastructure indicators; the datasets related to the e-Government Benchmark reports 109 ;

·Expert Group on interoperability of public services inputs and recommendations

oThe interoperability expert group was created on 14/02/2020 with representatives of the national administrations CIO offices, members appointed for all 27 Member States and observers from EFTA and Western Balkans countries

oPosition papers from 18 Member States: In December 2020 the Commission kindly asked the members of the expert group to react on a short policy paper with some initial ideas for the next interoperability policy (vision, general approach, policy instruments, etc.);

oCo-creation workshops – five informal discussions and brainstorming sessions took place between January and December 2021 with more than 200 participants;

oBilaterals – the Commission organised 26 bilateral calls with 26 different Member States between July and September 2021;

o Policy recommendations – they were formally endorsed by consensus of the whole group on the 5th of October 2021. Those recommendations served as a basis to build the draft legal text and the two dedicated workshops.

oTwo workshops on legal draft – They took place in November 2021 gathering both around 40 external participants each.

·CEPS external supporting studies - The Commission contracted the Centre for European Policy Studies (CEPS) to conduct three independent supporting studies between September 2020 and October 2021.

o Supporting study for the evaluation of the ISA² programme

o Supporting study for the evaluation of the EIF

o Supporting study on the impact assessment for a future interoperability strategy

·Joint Research centre (JRC) analysis – JRC has provided extensive support in preparation of the Impact Assessment, in particular with a study on quantifying the benefits of Location Interoperability and general costs of interoperability in the EU 110

·Input received to the Inception Impact Assessment feedback period 111 , 13 responses that are summarised in Annex V.

·Targeted online survey 94 responses that informed the analysis and the conclusions of the study, summarised in Annex V.

·In-depth interviews with 23 experts and stakeholders

·Input from Public Consultation launched for 12 weeks, from 01/02/21– 26/04/21, summarised in Annex V.

·4 public workshops: 1 inception workshop and 2 validation workshops in the framework of the DIGIT all conference, 1 public expert panel on options for a legal instrument

The Impact Assessment was based on certain assumptions, namely:

·It was estimated that the number of cross-border cases of usage of digital public services would be growing with the increase of the number of people living and working in a Member State different from the one of their origins and with the increase of the number of people traveling for tourism purposes.

The data limitations encountered in this Impact Assessment were the following:

·Lack of data focusing solely on the benefits and costs of interoperability

·Lack of comparable data on the costs and benefits of transposing the EIF into national frameworks, due to the voluntary nature and heterogeneous uptake of the framework

·Relatively low number of respondents to the different surveys

To mitigate the impact of the data limitations (to the extent possible), the external contractor followed up directly with some of the stakeholders to clarify certain aspects. The Joint Research Centre kindly offered their expertise and has developed a more specific study that took both a qualitative approach (20 use cases on location data interoperability) and quantitative (estimation of general costs of interoperability at EU level). In addition, where quantification of costs and benefits was not feasible, a qualitative approach was chosen instead (description of practices, processes and types of costs and benefits deriving from the options). To increase the number of stakeholders, involved an important number of public and targeted consultation activities were put in place.

Annex II. Methodology and analytical models used

This annex presents the methodology employed for the back-to-back evaluation and impact assessment. It details the types of data collected and their validation, the methods used, and the main limitations of the analysis.

Overview of information and data sources

In the Data Collection Phase, a mix of data collection methods and desk research was employed to gather both primary and secondary information and data, as follows:

·Primary information and data were collected via the following consultation activities:

oWork on recommendations with the Expert Group on Public Sector interoperability

oTargeted in-depth interviews;

oTargeted online survey;

oPublic consultation.

ofeedback received on the Combined EIF Evaluation Roadmap and Inception Impact Assessment 112

·Secondary information and data were collected by reviewing:

oThe Digital Public Administration Factsheets, developed as part of the NIFO action and available on the Joinup platform 113 ;

oThe EIF Monitoring Mechanism 114 ;

oDatabases including Eurostat, the World Bank Worldwide Governance Indicators, and the Organisation for Economic Co-operation and Development (OECD) Digital Infrastructure indicators;

oThe datasets related to the e-Government Benchmark reports 115 ;

oRelevant literature;

oOfficial EU-level documents such as communications, regulations, and directives;

oOfficial national-level documents such as national interoperability frameworks.

The above-mentioned sources were complemented by expert assessments conducted by five independent experts who were part of the Team on the supporting studies conducted by CEPS, in accordance with the Technical Proposal. The independent experts were tasked with completing the questionnaire that served as the basis for the in-depth interviews and with providing additional expert input to support the evaluation and impact assessment.

Primary data

The consultation activities were launched in December 2020 (with a kick-off workshop to raise awareness on the evaluation process) and lasted until the end of April 2021 (marked by the end of the public consultation). The duration of each consultation activity was planned to ensure that stakeholders had enough time to provide their inputs. This was complemented with a co-creation process with the Expert Group on interoperability of public services (see Annex V).

The general consultation activities resulted in a total of 134 responses for the impact assessment and 112 responses for the EIF evaluation, which can be divided into six stakeholder groups as presented in Table 2 . The stakeholder categories were defined on the basis of the assessment of relevant stakeholders conducted in the inception phase and by taking into account the need to ensure the comparability of results between the three strands of the consultation activities: in-depth interviews, online survey, and public consultation (considering in particular the mandatory template for background information used for public consultations).

Note: x/y:x:Number of participants in impact assessment, y:number of participants for EIF evaluation.

* In-depth interviews conducted with experts and academia include four independent expert assessments.

**One follow-up interview to the targeted online survey was conducted.

Source: Study supporting the impact assessment for a future interoperability strategy

In order to facilitate the comparison of stakeholders’ responses, the questionnaire of each consultation activity relied on Likert scales. Respondents were thus asked to provide their feedback by referring to a scale from (1) to (5) or (--) to (++), depending on the type of question 116 :

1.(1) – not at all; (2) – to a limited extent; (3) – to some extent; (4) – to a great extent; or (5) – completely;

2.(--) very negative; (-) negative; (0) neutral; (+) positive; or (++) very positive.

The SWD presents the aggregate results of the consultation activities using bar charts showing the average scores of responses from each stakeholder group. The average scores do not account for ‘don’t know/no opinion’ answers. The data labels of the bar charts display the average score first, then the corresponding number of respondents in brackets.

In order to assess the reliability of primary data, Figure 12  provides an overview of the level of knowledge in the field of digital public services, as well as the knowledge of the EIF among the participants who contributed to the consultation activities. Overall, stakeholders reflect a high level of knowledge, strengthening the reliability of the primary data collected.

Overview of the level of knowledge of respondents

Secondary data

The impact assessment builds primarily on the evaluations of the ISA² programme and the EIF 117 . These evaluations were conducted simultaneously with the impact assessment as key sources to assess achievements accrued so far in the field of public sector interoperability in the EU as well as lessons learnt.

The EIF Evaluation relies on an extensive review of available secondary data. One of the main sources of data is the National Interoperability Framework Observatory (NIFO) action supported by the ISA² programme and its outputs that are available in the Joinup collection with the same name 118 . The particular sources developed as part of NIFO and used as evidence feeding into the evaluation of the EIF include:

·The Digital Public Administration Factsheets, contributing to the assessment of the effectiveness of the EIF by providing an overview of the development of national interoperability frameworks or similar strategies 119 ;

·The 2020 Report on the State-of-Play of Digital Public Administrations and Interoperability, detailing the progress made by the EU Member States in enhancing the provision of digital and interoperable public services (the report also covers nine additional countries: Iceland, Liechtenstein, Norway, Switzerland, the United Kingdom, Ukraine, Montenegro, Turkey and the Republic of North Macedonia) 120 . This Report contributes to the assessment of the effectiveness and coherence of the EIF by providing an overview of key developments across the EU and the level of implementation of the EIF;

·The 2019 EIF Monitoring Mechanism, tracking the implementation of the EIF and its principles, model and recommendations across the EU, thus contributing to the assessment of the effectiveness of the EIF 121 ;

·Other reports and documents developed as part of NIFO and available on Joinup, including the Report on Public Administrations’ Digital Response to Covid-19 in the EU 122 , contributing in particular to assessing the relevance of the EIF.

·The webpages of ISA² actions and the ISA² Rolling Work Programmes.

Relevant indicators measuring digitalisation in the public sector were considered from a variety of databases, namely: Eurostat, the World Bank Worldwide Governance Indicators, and the Organisation for Economic Co-operation and Development (OECD) Digital Infrastructure indicators. In addition, the datasets related to the e-Government Benchmark reports 123 were used, contributing in particular to the assessment of the effectiveness and efficiency of the EIF and for the impact assessment contributing in particular to quantify the existing problems and the assessment of the potential impacts stemming from the policy options considered for this impact assessment;

Official documents such as communications, regulations, directives and decisions were reviewed contributing in particular to the assessment of the coherence and EU added value of the EIF. In addition, official national-level documents such as national interoperability frameworks were also reviewed, primarily as part of the analysis of the effectiveness of the EIF, while also contributing to understanding the EU added value of the framework. For the impact assessment these data sources contributed across the study, from depicting the policy context to comparing the policy options considered for this impact assessment;

In building the evidence base, the relevant literature in the field of interoperability was reviewed and considered throughout the evaluation process. The key insights from literature contribute particularly to the assessment of the relevance of the EIF and for the impact assessment particular the existing problems in the field of public sector interoperability.

Data validation

To ensure the robustness of evidence, the collected data were validated via triangulation. Tool #4 of the Better Regulation Toolbox on evidence-based better regulation defines triangulation as ‘the application and combination of several research methodologies in the study of the same phenomenon’ 124 . Validation requires checking ‘whether the findings of a study are true and certain: ‘true’ in the sense that research findings accurately reflect the situation; and ‘certain’ in the sense that research findings are supported by evidence’ 125 . To ensure that the findings are supported by well-founded evidence, data were collected from multiple sources and using different tools to analyse the evaluation criteria and questions. This approach allows for increasing confidence in collected data, comparing and contrasting findings, and providing a clearer understanding of the issues at stake. The analysis relies on three different types of triangulation to provide a solid basis for drawing robust conclusions:

·Triangulation of data, relying throughout the impact assessment on data collected from multiple sources and stakeholders.

·Triangulation of methods, based on data collected via at least two different data collection methods (e.g., interview, targeted questionnaire, public consultation, desk research).

·Triangulation of evaluators. Several members of the Core Study Team and Support Team were involved in data collection activities. In addition, each element of the impact assessment was addressed by at least two members of the Core Team. Hence, conclusions were agreed upon by at least two researchers.

Methods

The evaluation and impact assessment combines both quantitative and qualitative data, thus relying on a variety of methods and tools to analyse the collected data. In particular, the qualitative information was aggregated, compared and summarised to substantiate the evidence feeding into evaluation and impact assessment. To analyse the findings from the consultation activities, the Study Team employed descriptive statistics. To complement these methods, other specific quantitative methods were used to evaluate the effectiveness, efficiency and coherence of the framework:

Cost-benefit analysis (CBA) 126

CBA is a method of comparing the costs and benefits associated with a certain initiative, typically a government policy, programme or project, in order to assess whether the initiative is expected to bring about a net improvement. A key feature of CBA is that all costs and all benefits are quantified and expressed in monetary terms, and are adjusted for the ‘time value of money’ (through some discounting mechanism), so that all benefits and costs that occur at different points in time are expressed on a common basis in terms of their ‘present value’. The main limitations of CBA concern the difficulties in attributing a monetary value to items for which no market exists and the fact that certain non-monetisable effects simply cannot be taken into consideration. CBA was used to assess of the potential impacts stemming from the proposed policy options.

The standard cost model (SCM) 127

SCM aims at assessing administrative costs imposed by rules or policies inter alia on businesses and public administrations. It is based on the identification of the basic components of a rule, the Information Obligations, whose costs for the addressees can be measured and quantified. An Information Obligation is a specific duty to gather, process or submit information to the public authority or a third party. The SCM is employed to measure the costs borne in terms of days spent by national public administrations to transpose the EIF into national frameworks as well as the costs that would be borne by public administrations when implementing the proposed policy options.

The Multi-criteria analysis (MCA) 128

MCA is a technique for comparing and ranking different courses of action that yield multiple outcomes expressed in different units of accounts (i.e. some are expressed in monetary terms, others in physical units of different nature, e.g. number of lives saved and quality of gaseous emissions). In some respects, it can be said that MCA goes one step further than CBA, as it gives explicit recognition to the fact that a variety of both monetary and non-monetary objectives (i.e. social, environmental, technical, economic, and financial) may influence policy decisions. At the same time, this kind of analysis inevitably includes a higher degree of subjectivity compared to CBA, especially regarding the weighting and ranking of different criteria. Therefore, unlike CBA, a basic feature of MCA is its implementation through participatory processes, involving policymakers, experts (e.g. through Delphi panel method), stakeholders, etc. This method was employed to compare the proposed policy options based on their effectiveness, efficiency, coherence and feasibility.

Text mining

Text mining is a technique referring to a set of data-processing operations that extract knowledge according to a criterion of novelty or similarity in texts. In practice, this technique is based on an algorithm that counts the number of times a specific reference is made in a given text corpus. In this evaluation, text mining is used to evaluate the degree of external coherence of the EIF, together with its principles, interoperability layers and conceptual model. In particular, this approach allows checking, throughout the corpus of EU legislation and Commission communications, the extent to which the EIF and its components are taken into consideration by other EU initiatives in the field of interoperability that were adopted since the first version of the EIF (2004).

This analysis can be done in two steps:

·Searching through the corpus of EU legislation and Commission communications since the adoption of the first version of the EIF (2004) to obtain an overview of the number of times the EIF and keywords related to the EIF are mentioned in official EU documents;

·Taking a more in-depth look at a selection of relevant official documents and conducting a more granular analysis to understand which elements of the EIF are referenced.

While the first approach provides a quantifiable overview of the references to the EIF in different pieces of legislation and communications over time, the second approach provides more insights into the take-up of the EIF and its specific elements.

To quantify the number of mentions of the EIF and keywords related to the EIF in pieces of EU legislation, the Study Team relied on an extensive dataset, the ‘CEPS EurLex dataset’, customised to facilitate text mining 129 . The dataset contains 142,036 pieces of EU legislation adopted between January 1952 and August 2019, representing almost the entire corpus of the digitally available EU legal acts from this timeframe. The dataset covers three types of legally binding acts adopted by the EU institutions: regulations, directives, decisions, and implementing and delegated acts. When it comes to quantifying the number of references to the EIF and keywords related to the EIF in Commission communications, the analysis was run for documents including green and white papers, communications, reports (categorised as ‘DC’ on EurLex ) 130 . Importantly, the analysis excluded self-references to the EIF Communications.

The datasets were used to search for specific keywords in the timeframe 2004 (the year of the adoption of the first EIF) to August 2019. In addition, to cover the period from August 2019 and April 2021, the EurLex website was used. The keywords, selected based on their connection to the EIF and the extent to which they would be expected to yield results in a text mining exercise, were grouped into three clusters as follows 131 :

·Cluster 1: European Interoperability Framework. This cluster refers exclusively to the EIF and contains three keywords: ‘European Interoperability Framework’, ‘EIF’ and ‘interoperability framework’;

·Cluster 2: Interoperability layers. This cluster gathers together the following keywords: ‘semantic interoperability’, ‘technical interoperability’, ‘organisational interoperability’ and ‘legal interoperability’;

·Cluster 3: Key interoperability concepts. This cluster contains the following keywords: ‘once-only’ (keyword related to the ‘once-only’ principle), ‘digital by default’, ‘privacy by design’, ‘user-centricity’, ‘re-use of data’, ‘interoperability principle’, ‘interoperable e-Government service’.

The datasets of EU legislation and communications were searched for the keywords, resulting in a list of pieces of legislation and communications where the keywords were found. The results were thus summarised by year and number of mentions per cluster of keywords.

Econometric modelling

The econometric model statistically tests pre-determined assumptions regarding the relationship between two variables. The econometric tool is more powerful than a simple correlation because it unilaterally determines and quantifies the impact of an independent variable on a dependent variable. By isolating the effects of the variables of interest (i.e. independent variables) on the main variable (i.e. the dependent variable), the model specifies the statistical relationship existing (or not) between two variables. In order to test for the effectiveness of the EIF, the model is used to estimate the effects of adopting an EIF-based NIF and improving the quality of digital public services on the share of citizens using the Internet to interact with their public administrations. The model is estimated by relying on a regression with fixed effects:

(Eq. 1)

where the indexes and denote the country and the year, respectively. In the model, is the intercept, are the coefficients associated to each variable accounted for in the model, defines the fixed effects per country, and is the error term. This model controls for additional country-level variables that are difficult to measure or observe and that do not vary over time (such as cultural factors). Fixed effects isolate the impacts of time-invariant characteristics so that we can assess the net effect of the predictors on the outcome variable. In other words, this first model estimates an additional and single effect () per country, but the latter is assumed to be common across time.

Several additional variables (control variables), which vary over the time, are taken into consideration in order to isolate the effects of the variables of interest (i.e. NIF and e-Government index). These control variables ensure that other potential factors are properly captured in the analysis: 132 i) the real GDP per capita captures any differences that may stem from the economic development of a country; ii) the level of infrastructure development accounts for effects that may result from limited access to the Internet; iii) public spending on research and development related to general public services considers all the effects that can arise from discrepancies between countries’ expenditures in innovative public services; and iv) government effectiveness measures any effects that may arise from the perceived qualit13y of public services. All the variables used in the model are described in Table 3 . Moreover, Table 4 provides descriptive statistics of the analysed variables.

Source: Supporting study for the evaluation of the EIF

Source:

https://op.europa.eu/en/publication-detail/-/publication/29d694d4-4696-11ec-89db-01aa75ed71a1/language-en/format-PDF/source-search .

Before carrying out the analysis, it is important to ensure that the independent variables (i.e. the explanatory variables) are not highly correlated with each other. The simultaneous introduction of two highly correlated variables in the model (i.e. collinearity issue) could lead to erratic results. For this purpose, the correlations should be lower than 0.8 134 . Table 5 presents the pairwise correlations between the variables used in the analysis. Beyond ensuring the soundness of the results, the table also shows how variables are linked to one another (whether they evolve in the same or opposite direction, or whether they are independent). In Table 5 , three values illustrate important positive correlations:

·The correlation between the online interaction between citizens and public administrations (index_pa) and government effectiveness (gov_eff). However, the online interaction between citizens and public administrations variable is not an independent variable, but a dependent variable (i.e. the explained variable). Therefore, the two variables are not simultaneously introduced in the explanatory side of the model.

·The correlation between the two binary variables (NIF ‘complete’ and NIF ‘incomplete’’). It is no surprise, since these two variables are built on similar criteria, one following stricter assumptions than the other. These two variables are introduced alternatively (see regressions (1), (2), (3) and (4)) in order to test for the soundness of the results. Furthermore, the low values of the correlation with the other variables can be explained by the nature of the variable, which is binary.

·The correlation between the government effectiveness (gov_eff) and the real GDP per capita (GDP_cap). Although the value is slightly below 0.8, additional regressions are performed to ensure the robustness of the results when the two variables are dropped from the model (see Table 5 , regressions (1), (2), (3) and (4)).

Notes: All significance levels are at the p>0.05, except where denoted by: *=p<0.05.

Source: Supporting study for the evaluation of the EIF

The analysis tests two hypotheses:

·NIFs foster the use of online public services;

·Better e-Government services encourage citizens to interact with their public administration via the Internet.

By testing these relationships using a fixed-effects model and ensuring that the independent variables meet the assumptions required to run the model, the results of the analysis support the above two hypotheses.  Table 6 presents the results of the main model specification when the two variables, NIF ‘complete’ and NIF ‘incomplete’, are alternatively included as explanatory variables. To ensure the soundness of the results, additional results from different model specifications are provided in Table 6 .

Notes: Robust standard errors in parentheses

*** p<0.01, ** p<0.05, * p<0.1

Source: Supporting study for the evaluation of the EIF

Table 7 also presents the results of the analysis when new countries are introduced (see regressions (5) and (6)). This is essential in supporting the robustness of the results. Specifically, regression (5) drops the variable capturing the level of infrastructure, which constrains the model to 168 data observations. By removing this variable, four new countries can be introduced into the analysis. Finally, regression (6) replaces the variable capturing government spending on research and development to support the development of general public services with a broader variable, namely total government spending on general public services, thus introducing an additional country into the analysis.

Notes: Robust standard errors in parentheses

*** p<0.01, ** p<0.05, * p<0.1

Source: Supporting study for the evaluation of the EIF

Main limitations

The mix of primary data and information, obtained through consultation activities, and secondary data and information, mainly obtained through desk research, was crucial in allowing to draw conclusions and, more importantly, to provide estimates of costs and benefits. Nevertheless, the following limitations may affect the main findings:

1.The scarcity and difficult comparison of data for assessing the costs and benefits of transposing the EIF into national frameworks. The voluntary nature of the framework reflects a heterogeneous uptake of the framework across the EU, limiting the availability of data and making the comparison between countries difficult, as the framework is adopted on an ad hoc basis. Furthermore, the implementation of the EIF has not followed the same timeline in the different countries. To overcome this limitation, the Study Team contacted the national public authorities that responded to the consultation activities and indicated their availability to participate in a follow-up interview in order to gain insight on the time spent (in terms of person-days) on the transposition of the EIF, or part of it, into their national framework.

2.The overarching scope of the EIF hinders the assessment of the direct costs and benefits stemming from its implementation. Costs and benefits deriving from the EIF do not just pertain to the implementation of the framework, as costs and benefits depend on the levels of digitalisation of the Member States or on the different degrees of centralisation and varying organisational structures involved in delivering public services. The costs related to interoperability initiatives are borne at different levels of administration in several countries.

3.The conclusions drawn in the Impact Assessment are built upon the EIF Evaluation, which evaluates the latest implementation of the EIF (i.e. from 2017 to 2020). However, the framework seeks to achieve overarching goals and impacts that require time to show results.

Another limitation may be the potential ‘consultation fatigue’ of respondents. In particular, this Impact Assessment was carried out in a context in which the feedback of stakeholders may have been sought for several relevant developments: the end of the ISA² programme, the rollout of the new Digital Europe Programme, the 2030 Digital Compass and the new policy initiatives related to digitalisation in the EU.

Annex III. Evaluation framework for the EIF evaluation

Source: Study supporting the evaluation of the implementation of the EIF

Annex IV. Overview of benefits and costs

Source: Study supporting the evaluation of the implementation of the EIF

Annex V. Stakeholder consultation

Summary of the stakeholder strategy

A continuous and active stakeholder consultation strategy was designed and followed in preparation for the Impact Assessment on the EU public sector interoperability policy. It was complemented by active communication activities on our dedicated social media channels ( Twitter , LinkedIn , YouTube ).

A broad range of activities were put in place to ensure that all interested parties and stakeholders would have the opportunity to provide feedback on the various policy options that the Commission has identified with regard to its initiative, and their likely impacts, as well as on the relevance, effectiveness, efficiency and the added value of the initiative. In that context, the Commission reached out to a broad range of stakeholders, including Member State national authorities, non-governmental organisations, professional associations, business organisations and individual citizens.

To involve a broader range of stakeholders the following consultation activities were conducted:

·Public consultation on the inception impact assessment (15/10/21): The key ideas for the review of the European Interoperability Framework and a future policy were outlined in an Inception Impact Assessment (IIA). The published IIA informed citizens and stakeholders about the Commissions’ plans in order to allow them to provide feedback on the intended initiative. This fed into the subsequent consultation activities that ensured an inclusive process with all interested parties being actively invited to contribute.

·Online kick-off workshop ‘How interoperability can achieve seamless data flows and services for the EU’s public sector’, conducted on 3 December 2020. It aimed at raising awareness about the EIF evaluation and the impact assessment process and engaging stakeholders in the process.

·Online survey (19 January – 7 March 2021), targeting specific stakeholder groups.

·In-depth interviews (1 February – 8 March 2021) with 19 selected stakeholders to collect detailed data and information contributing to the EIF evaluation (12 stakeholders) and the impact assessment (7 stakeholders), respectively. The interviews are complemented by expert assessments conducted by independent experts who were tasked with, inter alia, completing the questionnaire that served as the basis for the in-depth interviews 138 .

·A 12-week long, Internet-based public consultation (1 February – 26 April 2021), open to the wider public and available in English, German, and French.

·Validation workshop in the digitalALL conference 139 on 22 April 2021 presenting and validating the outcome of the consultation activities with 126 registered participants.

·An innovative co-creation, co-design process has been put in place with the interoperability expert group with Member States. Workshops and roundtables following design-thinking methodologies and participatory practices were led with the experts to discuss and develop the needs for a future interoperability policy: 5 brainstorming sessions with more than 200 representatives from Member States, 5 formal expert group meetings with also more than 200 participants overall, 18 position papers issued by the Member States, 26 Member States bilaterals throughout the summer of 2021.

·The topic was also presented and discussed in two meetings of the Chief Information Officers network organised by the Presidencies of the European Council (Portugal - May 2021, Germany - December 2020) and presented to the digital attachés in the Telecom Working Party of the Council in April and November 2021. Member States have provided us with positive and constructive feedback which has been taken into account in the construction of the proposal.

Summary of the feedback on the Inception impact assessment

In total there were 13 feedback replies on the IIA from 8 different EU countries (3 from Germany, 2 from Spain and Belgium, 1 from each Sweden, Italy, Greece, France and Finland) and one non-EU country (Norway) and different stakeholder groups (see Figure 14 )

Source: Have Your Say Portal

The input was relevant for the formulation of problem and objectives as well as for the policy options and the impacts.

On the problems stakeholders highlighted the effects of competing and non-open standards. One stakeholder suggested that the EIF evaluation should include an analysis on the reasons that led to some recommendations to be better implemented than others. This suggestion was taken up in the EIF evaluation.

On the objectives several stakeholders draw links to public values beyond digitalisation as such. They suggested to highlight the value of interoperability for the democratisation of public knowledge, civil rights, digital involvement of all citizens and diversity.

One stakeholder suggested to strengthen the ambition of EIF beyond guaranteeing the necessary state tasks, public services should support the optimal processing and easy usage of digital data. In general, most stakeholders advocate to strengthen the principle of openness for the reuse of technical solutions as a core principle. One suggests including codified design principles and to encourage an API centric approach. Another stakeholder suggests that the EIF should more explicitly be also addressed to regions and sectors.

Eight of the 13 contributions mention explicitly the need for agreed and open standards. Two stakeholders request to include specific reference to interoperability between blockchains into the EIF with a clear definition and open standards. One stakeholder highlights that this should include standards around metadata, another the importance of unique and trusted digital identifiers. Two stakeholders ask for more and coordinated involvement of the public sector in standardisation organisations.

Five stakeholders bring forward the idea of fostering and EU-wide sharing and reuse of mature, reusable and open source interoperable solutions for public administrations. One stakeholder asks to facilitate the multiple use of the same privacy by design infrastructure at no extra cost. The future policy should incentivise the sharing of design costs around usable, trusted and secure solutions also between public and private stakeholders.

Three stakeholders mention the need for more guidance on the use of open standards and open source, including clear definitions. One stakeholder asks for a dedicated task force for immediate, middle term and long-term technical assistance at all levels to support any interoperability action, including relevant training resources in the Interoperability Academy . One stakeholder asks for a certification programme for interoperability trainers.

Three stakeholders highlight the importance of making the interoperability policy consistent with other EU policies like the EU AI strategy. Two stakeholders brought forward the idea of a reference implementation on the city level published under a free software license which would act as formal specification and can be implemented directly. This idea is mentioned linked to EU policies but also to EIF standards, that should come with at least one Open-Source implementation to confirm its implementability.

For the policy instrument two stakeholders suggest a more binding instrument, one of them a consolidated EU Regulation while another sees the risk that a strict legal framework might hinder interoperability – but harmonisation of organisational procedures could be beneficiary.

One stakeholder highlights the importance for the different sectors on the example of author’s rights. It asks not only for effective cooperation mechanism but also for effective safeguards to ensure their respect. Another stakeholder has a similar idea in developing a process for assessing compliance with the EIF.

Stakeholders highlighted the potential impact of a holistic transformation of (digital) public services on the private ICT sector with the potential to create an ecosystem of related apps and services.

Consultation activities for the EIF evaluation and Impact assessment

The consultation activities led by the contractor targeted several groups of stakeholders. The following grouping is used to analyse the feedback to the consultation activities:

oCivil society;

oCompanies and business associations;

oEU and non-EU citizens;

oEU public authorities;

oExperts and academia (including the independent expert assessments);

oNational and sub-national public authorities in the Member States.

The questionnaires used throughout the consultation activities mainly used Likert scale responses, referring to a scale from (1) to (5), (-2) to (2) or (--) to (++), depending on the type of question 140 :

3.(1) – not at all; (2) – to a limited extent; (3) – to some extent; (4) – to a great extent; or (5) – completely;

4.(-2) – definitely would not; (-1) – probably would not; (1) – probably would; (2) – definitely would;

5.(--) – very negative; (-) – negative; (+) – positive; (++) – very positive.

With a total of 112 respondents for the EIF Evaluation and 134 respondents for the Impact Assessment for a Future Interoperability Strategy, the consultation activities reached all types of stakeholders described in Table 9 . In what follows, the feedback received during the consultation activities are presented according to the type of consultation, i.e. public versus targeted consultation 141 . Feedback received during in-depth interviews are grouped together with those from the targeted online survey since both consultation activities were based on the same questionnaire; only more qualitative feedback was sought in the in-depth interviews.

*The six in-depth interviews include the five expert assessments.

**One follow-up interview to the targeted online survey was conducted in the scope of the EIF Evaluation and one was conducted in the scope of the Impact Assessment.

Source: Study supporting the impact assessment for a future interoperability strategy

In the public consultations, feedback was received from 13 Member States, with the highest number of received answers from Spain (see Figure 15 ). In addition, six respondents are non-EU citizens. For the work with the Expert Group all 27 Member States provided input.

Source: Study supporting the impact assessment for a future interoperability strategy

The group of respondents to the public consultations from public authorities includes a mix of administrations exercising at the international (3 respondents), national (4 respondents), regional (6 respondents) and local (3 respondents) levels (see Figure 16 ).

Source: Study supporting the impact assessment for a future interoperability strategy

On average, respondents have a good level of knowledge in the field, noting they are familiar to a great extent with digital public services and interoperability (see Figure 17 ). When it comes to specific knowledge about the EIF, the difference between the respondents to the targeted consultation and those contributing to the public consultation is more pronounced, with respondents to the public consultation reflecting a relatively lower level of knowledge (see Figure 18 ). Nevertheless, the consulted stakeholders are on average familiar at least to some extent with the EIF.

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely.

Source: Study supporting the impact assessment for a future interoperability strategy

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely.

Source: Study supporting the impact assessment for a future interoperability strategy

Results: EIF Evaluation

The framework is deemed to be relevant to some extent. The needs and problems originally identified by the EIF continue to be experienced across the relevant stakeholders and the framework has only partially addressed them (see Figure 19 ). Respondents to the targeted consultations call for more guidance for public administrations to improve the governance of their interoperability activities and address the problems linked to the fragmentation in the delivery of digital public services and the organisation and format of public data. Public administrations particularly point to the need for more cooperation, especially in specific sectors (e.g., health). Several other needs were highlighted, such as the need to increase awareness, for staff with IT skills in public administrations and to build investment capacity to keep pace with rapid technological change and bridge technological barriers.

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely.

Source: Study supporting the evaluation of the implementation of the EIF

Stakeholders participating in both consultations consider that the recommendations of the EIF have been effective to some extent in achieving the objectives (see Figure 20 ). However, the EIF has been seeking to achieve broad objectives and the recommendations require more granularity. The targeted consultation generally emphasises the contribution of the framework to raising awareness of the importance of cross-border interoperability, although more needs to be done at the sub-national level. The principles set out by the EIF have enhanced interoperable digital public services but are difficult to assess due to their abstract nature or still limited achievements. Stakeholders call for more clarification when it comes to transparency, technological neutrality and user-centricity. The layered interoperability and the conceptual model could be improved with more practical guidance for implementing the models. Overall, positive impacts have been experienced across stakeholders, in particular in enhancing the quality of services provided by public administrations and in fostering the free movement of goods, services, capital and workers across the EU. Nevertheless, more could be achieved to increase the direct and indirect benefits gained by stakeholders.

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely.

Source: Study supporting the evaluation of the implementation of the EIF

Although costs and benefits are difficult to assess due to a plethora of factors, stakeholders participating in the targeted consultation acknowledge that, in the long run, the EIF brings benefits that exceed the cost of implementation. The EIF results in efficiency gains by supporting data re-use, enabling synergies in implementing and designing new services and streamlining administrative procedures. A representative from a national public administration noted that the costs of several ICT projects have been eliminated due to timely considerations of interoperability. The costs of implementing the framework are relatively moderate, both at the national and EU levels; initial investments can be challenging, but over time the benefits outweigh the costs and expand beyond a single administration.

In terms of internal coherence, the respondents consider that the components of the EIF are generally synergetic (see Figure 21 ); the framework could be improved by better linking the conceptual model with the principles and the layered interoperability model. In addition, clarification of some thematically overlapping sets of recommendations would make the framework more actionable. At the level of external coherence, synergies are identified with the Single Digital Gateway, the Digital Single Market Strategy and the Data Strategy; overlaps and inconsistencies remain limited, but challenges may arise within own frameworks that are developed as part of sectoral initiatives (see Figure 21 ).

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely.

Source: Study supporting the evaluation of the implementation of the EIF

Finally, respondents to both types of consultations generally agree that national or sub-national initiatives would bring only limited contributions to the objectives pursued by the EIF, confirming its EU added value (see Figure 22 ). In particular, the cross-border dimension cannot be achieved solely by national or sub-national administrations. The EIF is not sufficiently used across areas and by Commission services. Furthermore, cross-border interoperability remains limited and is driven by specific sectoral needs.

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely.

Source: Study supporting the evaluation of the implementation of the EIF

Input of the Expert Group co-design process

We have put in place an innovative co-creation, co-design process with the interoperability expert group with Member States that encourages open cooperation and transparency. This has led to a constant increase in the engagement and interaction in meetings. Moving all discussions online since more than 18 months now has also led to an increase in the number of participants. In physical meetings we would have had 1 or maximum 2 representatives per Member State for a maximum of 35 approximately, whereas online we regularly have more than 54 participants. Workshops and roundtables follow design-thinking methodologies and participatory practices. We organised since mid-2020: 5 brainstorming sessions with more than 200 representatives from Member States, 5 formal expert group meetings with also more than 200 participants overall. Member States issued 18 position papers and we met them throughout the summer of 2021 in 26 bilaterals to discuss around the recommendations on the future policy.

The main result of this process with the expert group are the recommendations on a future interoperability policy. Other intermediate results can be interesting to understand the view of the stakeholders that are likely to be the most affected by the policy initiative.

The co-design process with the expert group was structured around five work phases:

·Position papers from 18 Member States – CIOs office – In December 2020 the Commission kindly asked the members of the expert group to react on a short policy paper with some initial ideas for the next interoperability policy (vision, general approach, policy instruments, etc.). This was only on a voluntary basis, however two-third of the members replied positively and shared position papers explaining their views on the upcoming policy and the proposed instruments;

·Co-creation workshops – five informal discussion sessions took place between January and December 2021 (see the list below), involving around 40-45 members of EU national administrations each time. The working methodology was based on design-thinking principles, including ‘diverge’ and ‘converge’ working phases, and participants were sometimes asked to fill polls (e.g. see below a poll from the workshop on the 30th of April). These meetings were moderated using Chatham House rules (no recording, participants do not formally engage their country), in order to foster interactions between people and ideation. Several working papers were circulated to stabilize ideas along the process and the subjects of these workshops helped build the structure of the draft policy recommendations of the group (see point d.). All papers were available to all the participants and we encouraged transparent feedbacks directly on a Teams space.

o04/02/2021 – General debate on the policy approach for the next interoperability policy following the position papers (e.g. vision, priorities, etc.)

o14/04/2021 – Communities of Practitioners (e.g. topics, organisation and composition, etc.)

o30/04/2021 – Synergies and Funding (e.g. links with other EU digital policies, financial support to Member States for interoperability implementation)

o20/05/2021 – Digital Checks and digital-ready policy (e.g. experience from Member States, replicability at the EU institutional level)

o10/06/2021 – Governance (e.g. composition, structure and mandate of a strengthened EU interoperability governance)

·Bilaterals – the Commission organised 26 bilateral calls with 26 Member States between July and September 2021 (only one country did not reply because of a lack of internal resources at that time). Each of these informal discussions lasted between 45 and 120 minutes. They were very rich in terms of content and feedback received, while helping the Commission to identify points of tension, test ideas with the Member States and build mutual trust;

·Policy recommendations – this is the main outcome of the expert group since its creation in February 2020. They were formally endorsed by consensus of the whole group on the 5th of October 2021. The document includes 27 recommendations structured around three chapters, each of them reflecting the common ideas expressed by the Member States in the position papers, the co-creation workshops and the bilaterals:

oChapter 1 – A strengthened governance

oChapter 2 – Interoperability for better EU policies

oChapter 3 – Upgrade EU interoperable solutions

Those recommendations served as a basis to build the draft legal text and the two dedicated workshops;

·Two workshops on the legal draft – They took place in October and November 2021. The first one gathered around 40 external participants who expressed their views on the legal concepts introduced by the Commission and discussed the composition and mandate of the future strengthened EU interoperability governance. Following the first workshop, the Commission circulated a first draft of the legal text with the members of the expert group and organised a workshop to collect preliminary ideas and suggestions to be integrated in the impact assessment and the upcoming legal proposal. Member States provided written feedback by the 3rd of December 2021.

Annex VI. Supporting evidence from desk research

This annex presents a series of sub-annexes containing supporting evidence for the assessment of the EIF.

Annex VI.1. Scoreboards monitoring the implementation of the EIF