EUROPEAN COMMISSION

Brussels, 1.9.2016

COM(2016) 550 final

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

Results of the final evaluation of the ISA programme

{SWD(2016) 279 final}

Contents

1. Introduction    

2. Interoperability as an enabler of public sector modernisation    

3. Background    

4. Methodology    

5. Evaluation issues and questions    

6. Findings    

Achieving Programme’s Objectives    

6.1 Relevance    

6.2 Efficiency    

6.3 Effectiveness    

6.4 Utility    

6.5 Sustainability    

6.6 Coherence    

6.7 Coordination    

7.    Conclusions of the Final Evaluation    

8. Recommendations    

8.1     Communication with public administrations and raising their awareness on interoperability    

8.2    Enhancing the holistic approach and cross-cutting view of interoperability    

8.3    Cooperation with other EU policies and initiatives    

9. Next steps    

1. Introduction

By decision of the European Parliament and the Council (the ISA Decision), 1 the six year programme on interoperability solutions for European public administrations (the ISA programme) was launched on 1 January 2010 as a follow-on to the programme on the interoperable delivery of panEuropean egovernment services to public administrations, businesses and citizens (IDABC). 2 It was implemented from 1 January 2010 to 31 December 2015.

As mentioned in Article 1 of the ISA Decision, the objective of the ISA programme was to support cooperation between European public administrations by facilitating efficient and effective electronic cross-border and cross-sectoral interaction between them (including bodies performing public functions on their behalf), with a view to enabling the delivery of electronic public services supporting the implementation of EU policies and activities. 3  

This Report presents the main results of the final evaluation of the programme 4 , which was carried out in accordance with Article 13(3) of the ISA Decision. According to this provision, the final evaluation had to be communicated to the European Parliament and the Council by 31 December 2015. The final evaluation was performed by the Commission from December 2014 until February 2016 when the draft Commission report was accepted by the steering group. However, the recent changes in the Commission evaluation procedures resulted in a significant delay in meeting the deadline provided in the legal basis.

During the final evaluation the Commission was assisted by a team of independent experts from a consultancy company (the final evaluation team). Representatives of relevant Commission services oversaw the final evaluation through a Commission steering group. 5

2. Interoperability as an enabler of public sector modernisation

The modernisation of public administrations is one of the key priorities for the successful implementation of the Europe 2020 Strategy. 6 The annual growth surveys published by the Commission from 2011 to 2015 7 show that the quality of European public administrations has a direct impact on the economic environment and is, therefore, crucial to stimulating productivity, competitiveness and growth.

As mentioned in its 2013 annual growth survey, 8 the Commission considers the cross-border interoperability of online services and the digitalisation of European public administrations to be important factors for growth and increased efficiency. Interoperability between administrations is a key enabler in achieving a more efficient and effective delivery of digital services.

The Digital Agenda for Europe (DAE) 9 outlines how interoperability is essential in helping to maximise the social and economic potential of ICT and, consequently, the DAE can only succeed if interoperability is ensured. Its dedicated pillar on ‘interoperability and standards’ urged Member States to undertake specific actions related to interoperability, 10 namely the implementation of the commitments laid down in the Granada and Malmö ministerial declarations including the adoption of the European Interoperability Framework (EIF) 11 in national legislation.

On 24 and 25 October 2013, the European Council adopted conclusions 12 stressing that the modernisation of public administrations should continue, with the swift implementation of services that rely on interoperability for effective functioning, such as e-government, e-health, e-invoicing and e-procurement. This will lead to more and better digital services for citizens and enterprises across Europe, cost savings and greater efficiency, transparency and quality of service in the public sector.

However, endeavours from Member States often take place at a national level without following a common European approach, compromising European-level interoperability and the availability of cross-border digital public services. Therefore, there is a high risk that new electronic barriers (‘e-barriers’) are created that prevent citizens and businesses from using public services in countries other than their own and impede the smooth functioning of the internal market.

3. Background

The ISA programme was established building on progress under its predecessor (IDA and IDABC) programmes. As the ISA Decision3 made clear, the main objective of the ISA programme was to support cooperation between European public administrations with a view to enabling the delivery of electronic public services supporting the implementation of EU policies and activities. The intermediate objective was to facilitate efficient and effective electronic cross-border and cross-sectoral interaction between those administrations.

To meet these objectives, facilitate the activities set out in Articles 1 and 3 of the ISA Decision, and provide ‘common and shared solutions facilitating interoperability’, 13 the programme involved launching studies, projects and accompanying measures, by means of specific actions, to support:

the operation and improvement of existing common services, and the establishment of new ones;

the improvement of existing re-usable generic tools and the establishment of new ones;

the establishment and improvement of common frameworks in support of cross-border and cross-sectoral interoperability; and

the assessment of the ICT implications of EU legislation (ISA Decision, Article 3).

As defined in Article 2 of the ISA Decision, ‘common services’ are operational applications and infrastructures of a generic nature, which meet common user requirements across policy areas. ’Generic tools’ are reference platforms, shared and collaborative platforms, common components and similar building-blocks that meet common user requirements across policy areas. ‘Common frameworks’ are strategies, specifications, methodologies, guidelines and similar approaches and documents.

The ISA programme is executed by means of implementing acts (ISA Decision, Article 9) on the basis of a rolling work programme, which is revised at least once a year and includes actions for funding relating to one or more layers of interoperability (technical, semantic, organisational and legal). 14 Annual work programmes are determined following annual calls for proposals to Commission services and Member States, endorsed by the ISA Committee and adopted by the Commission before entering into force. The specific actions to be funded are listed in the rolling work programme.

4. Methodology

The final evaluation of the ISA programme was based on quantitative and qualitative data collected in a variety of ways (desk research, online surveys, interviews, written requests for additional information and case studies), involving a broad range of representatives from Member State and EU services, and a limited number of stakeholders from civil society and private organisations with which the programme interacted.

5. Evaluation issues and questions

The final evaluation used the same set of criteria as the interim evaluation, 15 which serves as a benchmark of the results. The results, conclusions and recommendations of the final evaluation are based on a broad range of data, which constitute the hard evidence required to verify the validity of the programme’s intervention logic and the achievement of its objectives. It focused on the following seven main criteria:

– relevance — to what extent were the programme's objective(s) pertinent to meeting evolving needs and priorities at national and EU levels?

– efficiency — how economically were the various inputs converted into outputs, and with what impacts in terms of the best possible use of resources? Which aspects of the programme were the most (in)efficient , especially in terms of resources mobilised?

– effectiveness — to what extent did the programme's results and impacts achieve its objectives? Are there aspects that were more or less effective than others and, if so, what lessons can be drawn from this?

– utility — how did the programme's actual and anticipated results and impacts compare with the business needs they were intended to address? What measures could be taken to improve the utility of actions under the next programme?

– sustainability — to what extent is the financial, technical and operational sustainability of the developed solutions, as maintained and operated through the programme, ensured?

– coherence — to what extent did the ISA actions form part of a ‘holistic’ approach (in terms of coherence with each other and with other EU policies and initiatives)? How well were synergies achieved between programme actions and with other EU activities?

– coordination — to what extent were activities involving Member States, including those of the ISA Committee, coordinated to ensure stakeholders' involvement in the ISA programme? To what extent were activities coordinated or aligned with the needs of other stakeholders with whom the Commission was supposed to interact in the ISA framework?

6. Findings

Achieving Programme’s Objectives

The final evaluation found that the ISA programme was effective in achieving its objectives and delivering operational solutions to “support cooperation between European public administrations by facilitating the efficient and effective electronic cross-border and cross sectoral interaction between such administrations, [….] enabling the delivery of electronic public services supporting the implementation of Community policies and activities”.3 This is particularly the case for common frameworks, reusable generic tools and common services and for the assessment of ICT implications of new EU legislation. Overall, the achieved and anticipated results and impacts of the ISA programme largely address the business needs that they intended to although some actions have more utility than others.

The evaluation identified a number of key areas where changes in the successor ISA² programme have the potential, collectively, to be more relevant to the current needs on interoperability and to improve its utility. It also confirms that the ISA programme contributed to sharing and re-use, through the establishment of common frameworks. Finally, the ISA programme had the opportunity to revise the ‘Better Regulation’ guidelines 16 to ensure that the assessment of ICT implications is clearly part of European Commission’s Impact Assessment process.

The overall findings are summarised below:

6.1 Relevance

The ISA programme’s objective (‘to support cooperation between European public administrations by facilitating the efficient and effective electronic cross-border and cross-sectoral interaction between such administrations’), is still pertinent to meeting evolving needs at both national and EU levels. This was demonstrated by the programme’s final evaluation on the impact of the ISA results but also from the political agendas of Member States clearly indicating the need for efficient and effective digital public service delivery.12 In addition, the EIF (European Interoperability Framework) was developed, implemented and monitored by the ISA programme. The EIF’s recommendations and guidelines demonstrate how interoperability should be implemented by Member States in the public administrations and they aim to:

a)promote and support the delivery of European public services by fostering cross-border and cross-sector interoperability;

b)guide European public administrations in their work of providing services to businesses and citizens; and

c)complement and tie together the various national interoperability frameworks (NIFs) at European level.

The EIF provides a common understanding on interoperability in the Union, which is acknowledged by the DSM strategy that includes a particular action on interoperability between public administrations (action 15: adoption of a priority ICT standards’ plan and extending the European Interoperability Framework for public services).

The ongoing revision of the EIF builds on work undertaken over the past few years by the National Interoperability Framework Observatory (NIFO), 17 which monitors and supports its implementation in the Member States.

The ISA programme contributes to other DSM actions where interoperability acts as an underlying enabler. 18 The programme’s objectives are well aligned with the DAE and the 2010 – 2015 European eGovernment Action Plan, which both identify interoperability as a condition for success. In addition, the programme aligns well with the CEF, which aims to develop connectivity and interoperability of European digital services, as some of its building blocks 19 (e.g. e-delivery, translation, e-invoicing) rely on ISA results. 20 The CEF can provide operational support to solutions developed and matured under the ISA programme.

The ISA programme actively contributed to a number of EU policies such as the Internal Market, 21 the Infrastructure for Spatial Information in the European Community (INSPIRE), 22 the Common Information Sharing Environment (CISE), 23 the Pan-European Open Data Portal, 24 and competition law, 25 which have benefited from its support in the form of dedicated actions and shared solutions. 26 These policies have identified goals and objectives that rely on interoperability for their implementation.

The needs of European public administrations, such as the modernisation of public service delivery, the improvement of coordination between Member States’ administrations and Commission services, the organisational alignment in public administrations and the greater reusability of solutions, identified at the time of the ISA2 ex-ante 27 and ISA interim evaluations, 28 still remain. Facilitating cross-border and cross-sector service delivery in a uniform and resource-efficient way requires greater reuse of existing interoperability solutions and increased collaboration between Member States' administrations and between them and the Commission. Common approaches ensure no duplication of efforts. 29

6.2 Efficiency

Overall, the ISA programme can be considered to have been efficient. First, the evidence from the monitoring and evaluation action 30 found that it has been considered ‘on track’ throughout the 2010 - 2015 period, with an average delay of less than 5 %. 31 Second, the total executed budget for the 2010 – 2014 work programmes amounted to EUR128.2 million, approximately 2 % less on average than the cumulative budget allocated for the period. 32 Given these findings, the programme can be deemed efficient in budgetary terms.30

In addition, 90 % of surveyed Member State representatives and Commission officials from relevant services affirmed that the programme delivered on time and within its original scope. 33 As mentioned in the ISA final evaluation, a majority of these stakeholders felt fully or somewhat involved in the annual revisions of the programme. 34

The procedure for admitting proposals for new actions and the funds’ release process can be regarded as efficient. 35 The efficiency in terms of the allocation of human resources is confirmed by the fact that the actual allocation was always lower than that envisaged in the original ISA legislative proposal.

6.3 Effectiveness

The ISA programme has delivered operational solutions, i.e. common frameworks, re-usable generic tools and common services that help to achieve the objective of facilitating effective collaboration between European public administrations. Overall, the majority of interviewees and online survey respondents felt that all ISA solutions facilitate interoperability between those administrations. 36

The establishment of common frameworks such as the EIF, also supported the sharing and re-use of solutions with its dedicated recommendations. 37 A sharing and reuse framework started to be developed under ISA and is currently being finalised under the ISA2 programme.

The programme started work on the top two layers of interoperability, the organisational and the legal interoperability layers, which are areas with great potential impact on Member States’ public administration modernisation and need to be further developed and consolidated.

The assessment of the ICT implications of new EU legislation, such as impact related to the implementation of new or the adaptation of existing ICT solutions was included in the Commission’s impact assessment process, under the Better Regulation guidelines.16 The ISA programme has gone further by promoting such assessments and supporting DGs and services undertaking them.

The effort that the programme invested in the area of semantic interoperability and also in developing stronger links with standardisation organisations represented important contributions to increased cross-sector interoperability.

Although the programme and users of ISA results re-use existing common frameworks, common services and reusable generic tools, the programme should further promote their re-use as pointed out by surveyed Member State representatives and Commission officials. 38  

6.4 Utility

The achieved and anticipated results and impacts of the ISA programme largely address the business needs identified. 39 It led to the production of 23 solutions, made freely available on its website 40 or provided as shared services. These solutions vary from systems supporting implementation of EU policies such as the Internal Market Information System (IMI), to common specifications such as the Data Catalogue Application Profile (DCAT-AP) used in the Pan-European Open Data Portal. Of the 23 solutions, 15 are used by some or all Member States, 14 are used by several Commission services, and five are used by a handful of other EU institutions. This success is even more striking given that Member States are not obliged to reuse the solutions.

The method developed for the assessment of ICT implications of new EU legislation included in the Commission’s Better Regulation Toolbox, is being widely reused in the context of the impact assessment process.16 

Other actions have proven their utility through their widespread usage as shown by the statistics collected by the monitoring and evaluation action. 41

Member State representatives and Commission officials identified the following as the top four benefits of the ISA programme for Member States: 42  

1)the use of ISA solutions; 43  

2)the existence of the programme itself in raising awareness of the importance of interoperability and keeping it on the political agenda; 44  

3)the provision of references supported by NIFO, such as the EIF; and

4)the services provided, such as the sTESTA network and the CIRCABC platform.

The final evaluation highlights the low level of awareness of the programme among the key stakeholders who were interviewed and surveyed, Member State representatives and Commission officials. 45 This is confirmed by the fact that the additional measures suggested by interviewees and survey respondents are all either in place or being developed or expanded. 46  

The programme’s contribution to other EU policies and initiatives and to the modernisation of the public sector in Europe, improved its utility.30 Lastly, actual use (including the number of users) of each ISA solution should be systematically measured and the benefits of the solutions promoted.

6.5 Sustainability

The final evaluation found that the developed solutions, as maintained and operated under the programme, were sustainable from a financial, technical and operational point of view.

The technical and operational sustainability of an ISA solution is assessed indirectly by its operation and usage, as measured under the monitoring and evaluation action.

In financial terms, sustainability is guaranteed primarily by the follow-on ISA² programme or other EU initiatives and programmes, e.g. the CEF. In certain cases, funding options such as cost sharing and paying for use could be envisaged as a way to ensure long-term financial sustainability.

Organisational sustainability is also fundamental to overall sustainability. Long-term governance of solutions and services should be systematically planned to ensure their continuity. 47 It is important to note that information on the governance of solutions funded under the programme is always requested as part of the annual application process.

6.6 Coherence

The final evaluation found that the actions of the ISA programme feed into a holistic approach. This holistic approach, presented to the ISA Committee in January 2013, 48 recognises that genuine interoperability among public administrations requires working on various lines: strategic objectives, frameworks and architectural guidelines, development of specifications and the implementation, operation and dissemination of solutions. In addition, monitoring and mapping the interoperability landscape in Europe and awareness-raising are considered important enablers of interoperability. A holistic approach allows for gaps to be identified and relevant activities or projects to be launched to address them.

This holistic approach is not apparent to ISA stakeholders, 49 who are more familiar with the individual actions than with the programme itself. Low awareness of programme synergies was also confirmed among the interviewees and survey respondents, although the final evaluation pointed to an improvement since the interim evaluation.

Stakeholders mentioned only one ISA action (EUSurvey) 50 significantly more than others in the interviews and online surveys as not being coherent with the objectives of the programme.36 

Coherence was enhanced in the course of the programme by a considerable number of synergies that emerged between its actions and other EU policies and initiatives where interoperability is a key enabler for their implementation. What was clearly demonstrated and tackled through the programme was the need to act horizontally to ensure interoperability across policy domains. This horizontal dimension is missing in the domain specific implementations. ISA created synergies with areas like internal market, competition law, INSPIRE, CISE, Pan European Open Data Portal and, more recently, the CEF, which builds, to some extent, on the programme’s results.

A cross-cutting approach to interoperability brings tangible benefits that could not be achieved otherwise in sector specific approaches and solutions. The areas of public procurement, public sector information, ICT standardisation, justice and home affairs, and taxation would benefit from closer alignment with cross-sector interoperability solutions at EU level and the possible ensuing cooperation.

In the latter stages of the ISA programme, synergies were developed with Commission services responsible for the European Structural and Investment Funds (ESIF). Under the thematic objective 11, which aims to enhance the institutional capacity and efficiency of public administrations, the ESIF can support national, regional and local administrations’ efforts to improve interoperability as a means of modernising their public administrations. 51 Closer cooperation under the ISA2 programme could bring mutually beneficial results for the programme and ESIF implementation in Member States. In addition, active involvement of DG DIGIT in the recently launched Inter-service Group on Public Administration Quality and Innovation 52 ensures cooperation between the programme and the ESIF.

Efforts under the programme and by DG DIGIT led to closer links with Commission’s IT governance.

6.7 Coordination

During the implementation of the programme, a good coordination of formal activities was established with the Member States and with Commission services in line with the ISA Decision (Articles 12 and 14). Two thirds of the Member State representatives and over half of the Commission officials agreed fully or somewhat that the interaction between ISA and its stakeholders was effective to ensure consistent exchange of information, views and best practices. 53

In addition, a good coordination of activities with stakeholders outside public administrations was established under the programme to ensure their involvement where appropriate. These include, where relevant and appropriate, standardisation organisations, academics, private sector organisations and non-EU countries, e.g. the United States, Japan and Australia.

The final evaluation recognised the concerted efforts that have been invested in coordination at EU level with undeniably positive results. 54 The Inter-service Group on Public Administration Quality and Innovation will be instrumental in improving the coordination of all EU-level initiatives that contribute to the modernisation of public administrations in Europe.

The evaluation concluded that there is scope for Member State representatives on the ISA Committee, in the Coordination Group and the Working Group to coordinate better with their national colleagues on interoperability issues to contribute to the follow-on programme’s success. 55

The programme provides a good overview of interoperability developments at national level by consolidating the sets of NIFO and e-government Member States’ factsheets in one location on the Joinup platform. 56  

A majority of Member State representatives and Commission officials agreed fully or somewhat that the electronic platforms used by ISA to deliver consistent interaction between stakeholders and with the Commission (e.g. Joinup, the ISA website and dashboard) were adequate. However, there was a lack of awareness among stakeholders of the results achieved at national (e.g. the reuse of the programme) and EU level, and of the use of ISA results by Commission services. 57

7.    Conclusions of the Final Evaluation

The final evaluation was largely positive, concluding that the ISA programme was aligned with the Union policy priorities and Member States’ needs. 58 It found that it was implemented efficiently and coherently, delivering results that are re-used by both Commission services and Member States. The ISA programme has been effective in achieving its objectives and delivering operational solutions facilitating effective collaboration between European public administrations, including the assessment of ICT implications of new EU legislation. Overall, the achieved and anticipated results and impacts of the ISA programme largely address the business needs that they intended to and it coordinated its activities well with its stakeholders to ensure their engagement.

The conclusions found that there is a need to focus on improving the reuse of ISA solutions and addressing new evolving needs. In addition, there is a need to build on the coordination of initiatives at EU level, in particular among DGs, in the framework of the ‘Inter-service Group on Public Administration Quality and Innovation’, and between Member States and the Commission to ensure greater coherence with other EU and national initiatives and coordination with stakeholders.

The overall strategic approach taken by the programme since 2013 demonstrates that a coherent holistic approach within the framework the ISA programme exists. Synergies were achieved between programme actions and with other EU activities but scope for further synergies remains.

8. Recommendations

It is important to note that the ISA Decision focused the programme’s activities on achieving interoperability between public administrations. This is likely to have had an impact on levels of utility, awareness and re-use because the scope of the programme did not include public administrations’ links with businesses and citizens. Nevertheless, in some instances, the programme extended its stakeholder base in a pragmatic way to involve businesses and citizens in the development of interoperable solutions and put together a more holistic view than would otherwise have been possible.

The need, when developing interoperable solutions, to consider the link between public administrations, on the one hand, and, on the other hand, citizens and business and their particular needs, becomes increasingly apparent, given that the latter are the end-users of all public administration services. 59 The ISA Decision did not provide for action to address the needs of end-users, but experience of implementing the programme and consultation with action ‘owners’ and Member States made it clear that this was necessary. This limitation is addressed in the recommendation to extend the scope of the new ISA² programme to businesses and citizens.

The evaluation found that the ISA² programme should develop a more systematic ‘business-case’ approach, which would also serve to highlight further the role of interoperability in producing economic and societal benefits. 60 In addition, in order to ensure the full efficacy of the ISA² programme, the Commission should endeavour to respect its staffing targets, 61 particularly as the actual allocation of human resources was always lower than that originally envisaged in the programme.

As regards the recommendations of the interim evaluation, the programme sufficiently addressed that on improving interaction with stakeholders and presence at major events, and made consistent progress in addressing the others. However, the evaluation concluded that the following areas require ongoing effort under the new ISA² programme:

communication with public administrations and raising their awareness on interoperability;

enhancing the holistic approach and horizontal cross-cutting view of interoperability; and

cooperation with other EU policies and initiatives.

In addition, the final evaluation team considers that two of the 11 recommendations from the final evaluation of the IDABC programme are still applicable to the ISA programme; these are: 62

a common ‘promotion’ document should be produced, focusing on policy alignments and synergies between Union programmes and initiatives. (IDABC recommendation 5); and

information on projects’ financial and operational sustainability should be rendered more visible and explained better to external stakeholders. (IDABC recommendation 8)

The main recommendations are below:

8.1                    Communication with public administrations and raising their awareness on interoperability 

Between 2010 and 2015, approximately two thirds of the events that the ISA programme organised and held with Member State representatives, and 63 % (25) of the 40 events organised by Member States in which the ISA programme participated, were held after 2013, i.e. after the interim evaluation. The programme increased its presence at conferences and other events; of over 60 events in which it was involved, three quarters took place after the interim evaluation. 63  

The ISA² programme should update and implement a communication strategy, with a focus on targeted engagement, including with sector-specific stakeholders. This would address the lingering gap between the actual performance of the programme and stakeholders’ perception of it. 64

8.2    Enhancing the holistic approach and cross-cutting view of interoperability

The holistic approach taken since 2013, should be continued in the ISA² programme, as legal and organisational interoperability are still in their infancy. The European Interoperability Reference Architecture (EIRA) 65 and the European Interoperability Cartography (EICart), 66 documenting the existing services and tools, will be instrumental in supporting this holistic approach by helping to identify and re-use existing solutions and highlighting where interoperability enablers are lacking. 67 EIRA and EICart have a vital role in the dissemination of results and better identification of needs. They allow for more awareness, improved coordination and the emergence of greater synergies.

Now that the assessment of the ICT implications of new EU legislation has been included in the impact assessment process, the ISA² programme should continue to play its intended role and deploy its expertise to promote this exercise and support the DGs and services undertaking such assessments. 68  

Lastly, the ISA² programme should develop a more systematic approach to supporting and monitoring the use of common services and generic tools, but also the application and implementation of common frameworks. 69

8.3    Cooperation with other EU policies and initiatives

Efforts to align the ISA/ISA² programme with other relevant EU policies continue, taking as an example the CEF, which follows up on its results. Also, from 2016 onwards, account should be taken of the revised EIF, when solutions are implemented in support of other EU policies.

The ISA² programme should build on the clear improvements in the coordination of activities relating to interoperability and e-government across the Commission that the ISA programme has brought about since the interim evaluation. 70 This includes the ongoing cooperation with DG CNECT on the CEF and the new 2016-2020 e-Government Action Plan. In addition, there are plans to cooperate with DG EMPL and DG REGIO to identify existing or new ISA/ISA2 actions that have the potential to contribute to the ESIF thematic objectives 2, (‘enhancing access to, and use and quality of ICT’), and 11, (‘enhancing institutional capacity of public authorities and stakeholders and efficient public administration’). This is in addition to work on evaluating national operational programmes and identifying conditions. Lastly, the ISA programme has shown strong potential for greater cooperation with, inter alia, DG HOME, DG JUST, DG MARE, DG FISMA, DG GROW and the JRC.

The final evaluation sees the new Inter-service Group on Public Administration Quality and Innovation as the main driver to achieve consistent cooperation and coherence among EU policies and initiatives contributing to public sector modernisation. 71

9. Next steps

In implementing the follow-on ISA² programme, the Commission will pay the utmost attention to the above findings and recommendations, analysing them to validate and address issues raised in close cooperation, wherever appropriate, with the Member States and other stakeholders, including businesses and citizens.

(1)

  Decision No 922/2009/EC of the European Parliament and of the Council of 16 September 2009 on interoperability solutions for European public administrations (ISA), OJ L 260, 3.10.2009, p. 20.

(2)

  Decision 2004/387/EC of the European Parliament and of the Council of 21 April 2004 on interoperable delivery of pan-European eGovernment services to public administrations, businesses and citizens (IDABC), OJ L 144, 30.4.2004 (see OJ L 181, 18.5.2004, p. 25 ).

(3)

Article 1(2) of the ISA Decision.

(4)

Final Evaluation of the ISA programme, available at: http://ec.europa.eu/isa/library/official-documents/index_en.htm

(5)

Composed of officials from DG for Informatics (DIGIT), DG for Internal Market, Industry Entrepreneurship and SMEs (GROW), DG for Taxation and Customs Union (TAXUD), the Joint Research Centre (JRC) and the Secretariat General.

(6)

 Europe 2020: a strategy for smart, sustainable and inclusive growth, Commission Communication (COM(2010) 2020 final, 3.3.2010, p.12).

(7)

  http://ec.europa.eu/europe2020/index_en.htm  

(8)

Commission 2013 annual growth survey, COM(2012) 750 final, 28.11.2012, p. 13.

(9)

Commission Communication to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions (COM(2010) 245 final/2), p. 5

(10)

  http://daeimplementation.eu/dae_actions.php?action_n=26&id_country=1  

(11)

‘Towards interoperability for European public services’, Commission Communication to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions (COM(2010) 744 final, Annex 1).

(12)

http://data.consilium.europa.eu/doc/document/ST-169-2013-INIT/en/pdf

(13)

Article 1(1) of the ISA Decision.

(14)

See European Interoperability Framework (EIF) for European public services (COM(2010) 744 final, Annex 2), p. 21.

(15)

Interim evaluation of the ISA programme, 2012,

http://ec.europa.eu/isa/documents/interim_evaluation_of_the_isa_programme.pdf  

(16)

 Better Regulation guidelines, Commission staff working document (SWD(2015) 111 final);    
Guidelines on impact assessment (COM(2015) 215 final) (SWD(2015) 110 final), p. 18.

(17)

Developed under the IDABC programme, the NIFO supports the implementation of the EIF at Member State level, through the transposition of the EIF into national interoperability frameworks. http://ec.europa.eu/isa/actions/04-accompanying-measures/4-2-3action_en.htm  

(18)

Examples are its contributions to the interconnection of business registers, to the implementation of the ’once only’ principle (DSM action16) through its activities in the area of base registries and to the ‘digital single gateway’ objective, with the work on semantics and catalogue of services.

(19)

  https://joinup.ec.europa.eu/community/cef/og_page/catalogue-building-blocks

(20)

Final evaluation of the ISA programme, table 10: ‘Alignment of the ISA programme with CEF’, p. 54

(21)

 Working together to create new growth’ – the Single Market Act: 12 levers to boost growth and strengthen confidence, Commission Communication to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions (COM(2011) 206 final),p. 10.

(22)

Article 7 of Directive 2007/2/EC of the European Parliament and of the Council of 14 March 2007 establishing an Infrastructure for Spatial Information in the European Community (INSPIRE).

(23)

Draft roadmap towards establishing the Common Information Sharing Environment for the surveillance of the EU maritime domain, Commission Communication to the European Parliament (COM(2010) 584 final), , p. 3

(24)

 Open data: An engine for innovation, growth and transparent governance, Commission Communication to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions (COM(2011) 882 final), p. 11.

(25)

 Report on Competition Policy 2012, Commission report (COM(2013) 257 final), p.12.

(26)

Final evaluation of the ISA programme, table 21: ‘Level of adoption of ISA solutions by Commission services’, p. 131.

(27)

Document accompanying proposed ISA2 Decision (COM(2014) 367 final).

(28)

Interim evaluation of the ISA programme, 2012, p. 172

http://ec.europa.eu/isa/documents/interim_evaluation_of_the_isa_programme.pdf

(29)

 Final evaluation of the ISA programme, ‘Pertinence of the ISA objectives to respond to needs of EU public administrations’, p. 45

(30)

  http://ec.europa.eu/isa/dashboard/isadashboard  

(31)

Final evaluation of the ISA programme, ‘ISA programme efficiency’: % of delay, p. 82.

(32)

Final Evaluation of the ISA programme, ‘    Budget allocation and execution’, p. 67.

(33)

Final evaluation of the ISA programme, ‘Conclusion 4’, p. 86.

(34)

Final evaluation of the ISA programme, ‘Revision process of the ISA Work Programme’, p. 59.

(35)

Final evaluation of the ISA programme, ‘Allocation of financial resources’, p. 66.

(36)

Final evaluation of the ISA programme, ‘Conclusion 6’, p. 111.

(37)

Final evaluation of the ISA programme, ‘Establishment and improvement of common frameworks’, p. 94.

(38)

Final evaluation of the ISA programme, ‘Conclusions 7 and 8’, p. 112 and 140.

(39)

Final evaluation of the ISA programme, table 18 ‘Utility scores at action level', p. 116.

(40)

  http://ec.europa.eu/isa/ready-to-use-solutions/index_en.htm  

(41)

Final Evaluation of the ISA programme,     ‘Facilitation of effective interaction between European public administrations’, p. 88 and 89.

(42)

Final evaluation of the ISA programme, Main ISA results perceived, p. 88.

(43)

Final evaluation of the ISA programme,     ‘General perceptions on the benefits delivered by the programme’, p. 113 and 114.

(44)

Final evaluation of the ISA programme, figure 37 ‘Respondent’s agreement levels on the awareness raising of the ISA programme', p. 171

(45)

Final evaluation of the ISA programme, ‘Conclusion 9’, p. 141.

(46)

Final evaluation of the ISA programme, ‘Coherence’, p. 17.

(47)

Final evaluation of the ISA programme, ‘Conclusion 10’, p. 148.

(48)

ISA Committee meeting, 23 January 2013 – presentation of the results of the interim ISA programme evaluation.

(49)

Final evaluation of the ISA programme, ‘Conclusion 11’, p. 165.

(50)

EUSurvey is an open source, free-of-charge, web-based application which makes it easier both for respondents to answer the consultations and for survey managers to analyse the results.

(51)

Final evaluation of the ISA programme, ‘Internal coordination within the European Commission’, p. 167.

(52)

 Inter-service Group on Public Administration Quality and Innovation (Ares (2015) 4431139, 20.10.2015).

(53)

Final evaluation of the ISA programme, ‘Extent to which interaction between ISA and its stakeholders was effective’, p. 172.

(54)

Final evaluation of the ISA programme, ‘Internal coordination within the European Commission’, p. 167

(55)

Final evaluation of the ISA programme, ‘Conclusion 13’, p. 187.

(56)

  https://joinup.ec.europa.eu/community/nifo/home

(57)

Final evaluation of the ISA programme, ‘Reuse by ISA of results delivered by other EU initiatives’, p. 163.

(58)

Final evaluation of the ISA programme, ‘Conclusions 1, 2 and 3’, p. 56, 57 and 58.

(59)

Final evaluation of the ISA programme, ‘Consideration of results achieved by Member States in the establishment of ISA solutions’, p. 184.

(60)

Final evaluation of the ISA programme, ‘Recommendation 5’, p. 200.

(61)

Final evaluation of the ISA programme, ‘Recommendation 6’ p. 201.

(62)

Final evaluation of the ISA programme, ‘Recommendations from previous evaluations’, p. 190.

(63)

Final evaluation of the ISA programme, ‘Conclusion 14 and extent to which the ISA programme raised awareness of the ISA solutions’, p. 188.

(64)

Final evaluation of the ISA programme, ‘Recommendation 4, p. 199, and ‘Perceived performance of the ISA programme’, p. 83.

(65)

The EIRA is a reference architecture for delivering interoperable digital public services across borders and sectors.

(66)

The EICart tool is a tool that can be used to describe and catalogue the Solution Building Blocks (SBBs) of enterprise architectures in line with the EIRA.

(67)

Final evaluation of the ISA programme, Recommendation 8 of 9, p. 202.

(68)

Final evaluation of the ISA programme, Recommendation 3 of 9, p. 198.

(69)

Final evaluation of the ISA programme, Recommendation 9 of 9, p. 202.

(70)

Final evaluation of the ISA programme, Recommendation 7 of 9, p. 201.

(71)

  Opinion of the European Economic and Social Committee on the Communication from the Commission - Annual Growth Survey 2014 (COM(2013) 800 final); see OJ C 214, 8.7.2014, p. 46–54.