EUROPEAN COMMISSION

Brussels, 1.9.2016

SWD(2016) 279 final

COMMISSION STAFF WORKING DOCUMENT

Final evaluation of the ISA programme

Accompanying the document

Report from the Commission to the European Parliament and Council on the results of the final evaluation of the ISA programme

{COM(2016) 550 final}

Table of Contents

Executive summary    

Introduction    

1    Context    

1.1    Background of the ISA programme    

1.2    Objectives of the ISA programme    

1.3    ISA programme governance    

1.4    ISA Work Programme    

2    Methodology    

2.1    Inception (Evaluation structuring)    

2.2    Data collection and analytical tools    

2.3    Data analysis and synthesis    

2.4    Formulation of judgments and reporting    

3    Findings and results    

3.1    Relevance    

3.2    Efficiency    

3.3    Effectiveness    

3.4    Utility    

3.5    Sustainability    

3.6    Coherence    

3.7    Coordination    

4    Recommendations    

4.1    Recommendations from previous evaluations    

4.2    Recommendations from the final evaluation    

Table of tables

Table 1 Evaluation questions related to the ISA programme’s performance    

Table 2 Tabular overview of the actions    

Table 3 Mapping of evaluation criteria with main evaluation questions    

Table 4 Key stakeholder groups    

Table 5 Data collection and analytical tools    

Table 6 Data analysis and synthesis steps    

Table 7 Formulation of judgments and reporting    

Table 8 Alignment of the ISA programme with the DAE    

Table 9 Alignment of the ISA programme with the European eGovernment Action Plan 2011-2015    

Table 10 Alignment of the ISA programme with CEF    

Table 11 Alignment of the ISA programme with the DSM Strategy    

Table 12 Funds released and withheld    

Table 13 ISA unit resources - turn over in the period January 2010 to mid-2015    

Table 14 Operational ISA Solutions    

Table 15 Common frameworks within the ISA Programme    

Table 16 Common Services within the ISA Programme    

Table 17 Reusable generic tools within the ISA Programme    

Table 18 Utility scores at action level    

Table 19 ISA and related other EU initiatives, legislation or funding tools    

Table 20 Level of adoption of ISA solutions by Member States    

Table 21 Level of adoption of ISA solutions by Commission's services    

Table 22 ISA website statistics for the three years between June 2012 and June 2015    

Table 23 Public consultations launched by ISA    

Table 24 Validity of recommendations of the final evaluation of the IDABC programme    

Table 25 Validity of recommendations of the interim evaluation of the ISA programme    

Table of figures

Figure 1 Hierarchy of objectives of the ISA programme    

Figure 2 ISA Governance    

Figure 3 Methodology of the final evaluation    

Figure 4 Intervention logic of the ISA programme    

Figure 5 Needs of EU public administrations    

Figure 6 Needs addressed by the ISA programme    

Figure 7 Alignment of ISA objectives with EU initiatives    

Figure 8 Selection of proposals for new actions in the different Work Programme revisions    

Figure 9 Overview of ISA action’s budget deviation    

Figure 10 Overview of the 6 most significant ISA action’s budget deviation    

Figure 11 Total budget and estimated financial cost for the period 2010 – 2015    

Figure 12 ISA unit activities and tasks by types of post    

Figure 13 Number of HR allocated in 2010 - 2015    

Figure 14 ISA actions delayed and ahead of schedule    

Figure 15 ISA programme efficiency: Schedule Variance    

Figure 16 ISA programme efficiency: % of delay    

Figure 17 Most frequently mentioned measures to improve the ISA programme    

Figure 18 Main ISA results perceived    

Figure 19 ISA contribution to common frameworks    

Figure 20 ISA contribution to common services    

Figure 21 ISA contribution to reusable generic tools    

Figure 22 ISA contribution to the assessment of ICT implications    

Figure 23 ISA actions measured for Utility    

Figure 24 Use of ISA solutions by Member States    

Figure 25 Use of ISA solutions by Commission's services    

Figure 26 Number of views extrapolated to daily averages per year    

Figure 27 Number of downloads extrapolated to daily averages per year    

Figure 28 The changes’ potential to improve the benefits of the ISA programme    

Figure 29 The extent of agreement on the sustainability benefits of the proposed measures    

Figure 30 ISA Contribution to interoperability in Europe    

Figure 31 Mapping of the ISA actions to overall approach taken by the programme    

Figure 32 Summary of the links among ISA actions    

Figure 33 Synergies between ISA actions    

Figure 34 Interrelation between ISA actions    

Figure 35 Clusters of the ISA Programme    

Figure 36 Synergies between ISA and other EU initiatives    

Figure 37 Respondent’s agreement levels on the awareness raising of the ISA programme    

Figure 38 Respondents’ agreement levels on the interaction of their country and ISA    

Figure 39 Member States with/without ISA event occurrence    

Figure 40 Respondents’ agreement levels on the ISA Coordination Group coordination with the ISA Working Groups in communication of the ISA results at national level    

Figure 41 Extent of agreement on ISA’s effectiveness in engaging relevant stakeholders    

Acknowledgments

This assignment was completed by a Kurt Salmon and KPMG team, the evaluation team, guided by Commission through a steering committee. The team performed its task under the lead of Alessandro Zamboni, and supported by the engagement partner Jean Diederich, along with Ludovic Mayot, Michael Scanlon, Natalia Petrovova and Arianna Volpe. The production of this report would not have been possible without the efforts of the many stakeholders that we have interviewed and surveyed. The authors would like to express their gratitude to all of them.

Lastly, the evaluation team would like to thank all the Commission officials and especially the members of the unit DIGIT.B6 and the steering group who have been helpful and co-operative in providing guidance, information and feedback during this assignment.

List of acronyms

Executive summary

1. Context & evaluation objectives

The legal decision on ‘Interoperability Solutions for Public Administrations’, which led to the creation of the ISA programme, was adopted by the European Parliament and the Council on 16 September 2009 1 with a budget of €164,100,000 for its implementation from 1 January 2010 to 31 December 2015.

Building on the advances made by its predecessor IDA and IDABC programmes, the ISA programme was established against the background of major EU policies, to which it needed to be aligned. These included the Europe 2020 Strategy 2 , developed as a response to the financial and economic crisis, the Digital Agenda for Europe, one of the seven flagship initiatives of the Europe 2020 Strategy and the eGovernment Action Plan 2011-2015, which aimed to achieve a new vision of European public administrations through a more open model of design, production and delivery of online services to deliver greater value to citizens with fewer resources. Although other major policies have been launched since the inception of the ISA programme, notably the Connecting Europe Facility (CEF) and the Digital Single Market (DSM) strategy, the final evaluation finds that it is well aligned with both.

As stipulated under Article 1(2) of the legal decision, the programme aims ‘to support cooperation between European public administrations’ with a view to enable ‘the delivery of electronic public services supporting the implementation of Community policies and activities including bodies performing public functions on their behalf’ by ‘facilitating the efficient and effective electronic cross-border and cross-sectoral interaction between such administrations’. To achieve such objectives, the programme has been implemented by means of actions, i.e. studies and projects, as well as by accompanying measures to support the:

Creation and improvement of common frameworks in support of interoperability across borders and sectors;

Improvement of existing reusable generic tools as well as the establishment of new ones;

Operation and improvement of existing common services as well the establishment of new ones; and

Assessment of ICT implications of proposed or adopted Community legislation.

Currently, 38 actions are being implemented under the ISA Work Programme (2015) and are aligned with the clusters defined in the EIS; Trusted information exchange, Interoperability architecture and Assessment of ICT implications of EU legislation. 12 of these actions correspond to common frameworks, 8 to common services and 13 to generic tools. One action corresponds to the assessment of ICT implications of EU legislation and the remaining four actions concern accompanying measures and monitoring of the ISA programme.

The European Commission contracted KURT SALMON and KPMG to conduct an independent, external final evaluation of the ISA programme 3 , based on Article 13(3) of the legal decision with the four main objectives:

Objective 1: Evaluate the implementation of the ISA programme from its commencement in January 2010 until the time that this evaluation takes place.

Objective 2: Assess the relevance, effectiveness, efficiency, utility, sustainability, coherence and coordination of the ISA actions as well as assess performance, qualitatively and quantitatively, when possible, against its objectives and rolling Work Programme.

Objective 3: Examine the benefits of the ISA actions to the EU for the advancement of common policies.

Objective 4: Identify areas for potential improvement and verify synergies with other EU initiatives 4 in the area of cross-border and cross-sectoral interoperability as well as sharing and reuse.

2. Methodology

In order to satisfy the objectives of this final evaluation, data was collected using a bespoke methodology comprising of several techniques and quantitative and qualitative analysis. The final evaluation’s methodology 5 conforms to the methodological guidance for evaluations provided by DG Budget 6 and adheres to developments in the Smart Regulation policy 7 . During the data collection phase, the evaluation team used a combination of desk research, surveys (73 8 ), phone or face-to-face interviews (37) and also case studies of specific actions of the Work Programme (2) in order to fulfil the objectives of this evaluation. Furthermore, written requests were addressed to specific stakeholders to gather additional information on specific sub-issues of the final evaluation.

3. Findings and results

The ISA programme’s objective ‘to support cooperation between European public administrations by facilitating the efficient and effective electronic cross-border and cross sectoral interaction between such administrations’ is still pertinent to meet evolving needs at both national and EU levels. The ISA programme facilitates interaction between European public administrations. The modified scope of the follow-on ISA² programme will, most likely, enhance this objective. The follow-on ISA² programme’s scope aims to facilitate electronic cross-border or cross-sector interaction between European public administrations, businesses and citizens as well as between European public administrations (Conclusion No 1).

The needs of European public administrations, identified at the time of the ex-ante and interim evaluations of the ISA programme, are still considered as the current needs of European public administrations. In particular, the need to have ISA solutions more reused as well as the need for an increased collaboration between the programme and both the Member States and Commission services are as valid now as they were at the time of the interim evaluation. The evaluation notes that, as a voluntary policy, the ISA programme has no capacity to guarantee greater reuse of its solutions and increased collaboration depends on the willingness of all stakeholders (Conclusion No 2).

The programme has already started to take into account new evolving needs of European public administrations on governance and coordination, primarily through the revision of the EIS and through enhanced internal coordination within the European Commission. Finally, the ISA programme’s objectives are well aligned with the Digital Agenda for Europe, the European eGovernment Action Plan 2011-2015, the Connecting Europe Facility and the recently launched Digital Single Market Strategy (Conclusion No 3).

The ISA programme can be considered as efficient, overall. First of all, the evidence from action 5.01 Monitoring & Evaluation found that the programme has been considered on track during the entire period from 2010 to 2015 with an average delay of less than 5%. In addition, 90% of surveyed Member States’ representatives and EC officials affirmed their view that the programme has delivered on time and within its original scope. Furthermore, a majority of these stakeholders (68% of Member States’ representatives and 58% of EC officials) felt fully or somewhat involved in the annual revisions of the ISA programme (Conclusion No 4).

The evaluation found that the selection procedure of proposals for new actions and the funds release process can be regarded as efficient. Moreover, the total executed budget for the 2010 – 2014 Work Programmes is equal to EUR 128.2m, only slightly exceeding its planned budget of EUR 127.4m by less than 1 per cent (0.63%), yet still within the overall financial envelope. Given these budgetary findings, the evaluation confirms the programme has been efficient in this regard. The same can be concluded for the allocation of human resources, even though the actual allocation was always lower than what was foreseen in the ISA legislative proposal.(Conclusion No 5).

The ISA programme has delivered operational solutions, i.e. common frameworks, reusable generic tools and common services, contributing to the achievement of the programme’s objective to facilitate effective collaboration between European public administrations. Member States’ representatives and EC officials affirmed their view that the programme contributes more to the establishment of new common frameworks, common services and reusable generic tools than in improving existing ones. Overall, more interviewees and online survey respondents believed that all the ISA solutions facilitate interoperability between European public administrations (41%) than did not (30%). The remaining respondents said that they did not have sufficient overview of the programme to comment on all solutions. The evaluation found, nevertheless, that the need for the further development of the EUSurvey tool in the ISA² programme has to be appraised given respondents’ specific feedback on this (Conclusion No 6).

The evidence from the evaluation confirms that the programme contributed to sharing and re-use, through the establishment of common frameworks. The programme also contributed to organisational and legal interoperability, although the results from the ISA actions are scattered in this regard, which indicate space for improvement. Now that the assessment of ICT implications of new EU legislation has been included in the Impact Assessment process (Better Regulation Guidelines), the ISA programme has a defined role to promote this exercise and support the DGs and services undertaking these assessments with guidance as part of this internal EC process (Conclusion No 7).

The evaluation concludes that, overall, the achieved and anticipated results and impacts of the ISA programme largely address the business needs that they intended to. Some ISA actions have led to the production of 23 solutions. Of these 23 solutions, 15 are used by up to all Member States; 14 are used by several Commission services, and 5 are used by a handful of other EU institutions. Nevertheless, other actions have also produced results if not specifically related to solutions. For example, action 3.01 Assessment of ICT implications of new EU legislation has achieved its main objective as it has succeeded in being included in the formal Impact Assessment process even though there were not so many assessments conducted during the ISA programme.

Member States’ representatives and EC officials considered the top three benefits delivered by the ISA programme to Member States to be the use of ISA solutions, the existence of the programme itself (to raise awareness about the importance of interoperability and to keep it on the political agenda) and the provision of references, such as the EIF, supported by action 4.2.03 NIFO, networks and solutions, such as action 2.04 sTESTA and action 2.06 CIRCABC. In addition, the evaluation concludes that the ISA programme contributes to the modernisation of the public sector in Europe. The evaluation notes that the ISA programme does not coordinate, in a systematic way, with Commission DGs and services that run sectoral programmes, within which many cross-sector and/or cross-border services are managed and to which interoperability could greatly contribute (Conclusion No 8).

The evaluation confirms the existence of low levels of awareness about the programme among its interviewed and surveyed key stakeholders, Member States’ representatives and EC officials. Paradoxically, the additional measures, as suggested by the same interviewees and survey respondents, are all either in place or being developed or expanded. The evaluation, therefore, concludes that the follow-on ISA² programme should raise awareness about these existing measures and improve them by adapting them to the needs of targeted/specific stakeholders (Conclusion No 9).

The evaluation found that the financial, technical and operational sustainability of the developed solutions, maintained and operated through the ISA programme, is ensured. From a technical and operational standpoint, the annual ISA Work Programme summarises the actions that are funded based on numerous criteria, such as the technical need for an action to deliver a solution and the operational role that it fulfils. From a financial standpoint, sustainability of the developed solutions will be guaranteed, primarily, by the follow-on ISA² programme, although not after that. However, this may change subject to an impact assessment or other policy options, which may be pursued, during the period of the follow-on ISA² programme. Some solutions, created by the ISA programme, are now funded, operationally sustained and further developed outside of the framework of the programme by other EU initiatives and programmes.

To address the long term sustainability of the solutions, the ISA² programme should explore how to financially re-engineer certain services, which it provides, to ensure that users pay for them. In addition, the importance of focusing on the longer term organisational sustainability of ISA actions cannot be underestimated. Beneficiaries should formalise governance of services supported by these ISA actions to ensure their continuity.

The actual use and benefits of the solutions as well as the numbers of users of each ISA solution, which are not currently tracked in a systematic way, should be measured by the programme, to help identify options to ensure their sustainability outside of the programme’s framework or without its funding (Conclusion No 10).

The evaluation found that a holistic approach within the framework the ISA programme does exist, even if the programme does not convey this sufficiently. This holistic approach is based on the strategic approach of the ISA programme, created in January 2013, including the mapping of the ISA actions to the various cycles (Inception, Execution, and Operation), the interrelation between the various actions, as well as its global approach as contributor to interoperability in Europe. The evaluation found that the ISA programme houses independent actions, with which stakeholders are often more familiar than the programme itself. Levels of awareness about programme’s synergies are also confirmed to be lacking among the interviewees and survey respondents. In the context of the low levels of awareness, there is a need to provide greater clarity on the documented process about the selection of proposals for new actions (Conclusion No 11).

Although a considerable amount of synergies exists between the programme’s actions and other EU activities, and that synergies were achieved between programme actions quite well and other EU activities fairly well, the evaluation concludes that further synergies could have been achieved. The analysis showed that 25 ISA actions have more than five links with other ISA actions; including six ISA actions that are linked to more than ten other ISA actions. 24 actions have between one to five links with other actions. Given the ISA programme’s holistic approach, it is normal that some actions have more synergies with other actions, which does not reduce the value of the latter actions. The analysis showed that the ISA programme has achieved synergies with several activities at EU level, including CEF.

The ongoing efforts of the ISA programme and DG DIGIT to work with relevant DGs to increase the EC’s coherence on interoperability and eGovernment services, through enhanced coordination are satisfactory, for example, in the following separate activities; in the Inter-service Group on Public Administration Quality and Innovation and in its publications, such as the NIFO and eGovernment factsheets (Conclusion No 12).

The ISA programme coordinates formal activities well between the Member States, including the ISA Committee (Conclusion No 13); as well as with EC officials (Conclusion No 14) to ensure their engagement in the programme. Two-thirds of Member States’ representatives and just over half of EC officials fully or somewhat agreed that the interaction between ISA and its stakeholders was effective to ensure consistent exchange of information, views and best practices. In addition, the ISA programme coordinates activities well with stakeholders outside of the context of European public administrations during the course of its activities to ensure their engagement in the programme; when appropriate (Conclusion No 15).

In fact, the evaluation found that the increase in coordination with stakeholders since the interim evaluation has been a major success of the ISA programme although potential to build on this success with more targeted efforts does exist. Between 2010 and 2015, approximately two thirds of all events, which the ISA programme organised and held with Member States’ representatives, and 63% (25) of the 40 events organised by Member States, in which the ISA programme participated, were held since 2013, i.e. after the interim evaluation. With regard to Member States, the evaluation concluded that there is scope for individual national representatives on the ISA Committee, Coordination Group & Working Group to coordinate better with their own national colleagues on interoperability issues to better realise the programme’s achievements.

The ISA programme increased its presence at conferences and other events run by the EC, having been involved in over 60 events since its inception, with three-quarters of these events taking place since the interim evaluation. This is a clear indication of the focus of the ISA programme to increase interaction with its stakeholders and their increasing interest in the programme.

A majority of the Member States’ representatives and EC officials fully or somewhat agreed that the electronic platforms (i.e. Joinup, the ISA website and dashboard), used by ISA to deliver consistent interaction between stakeholders and with the European Commission, were adequate. However, there was a disappointing lack of knowledge among the stakeholders about the results achieved at national and EU level, i.e. about the reuse of the ISA programme at national level, as well as about the use of ISA results by DGs. 

4. Recommendations

The ISA² programme, which serves an EU policy, should continue to align itself with other relevant EU policies (Conclusions No 1, 2 & 3). It should also reflect the current needs of stakeholders to have more emphasis on having solutions operational and ready-for-use (Conclusion No 6).

The ISA² programme should support the review of the EIS on interoperability and to ensure its success. The alignment between the programme and the future strategy should be monitored regularly, as recommended in the EIS implementation review 2012 (Conclusion No 11).

The ISA² programme should continue to focus on the current ISA activities but place more emphasis on achieving legal and organisational interoperability. Now that the assessment of ICT implications of new EU legislation has been included in the Impact Assessment process (Better Regulation Guidelines), the ISA² programme should exploit its defined role to promote this exercise and support the DGs and services undertaking these assessments with guidance as part of this internal EC process (Conclusion No 7).

The ISA² programme should update and implement its communication strategy for the follow-on programme, with a focus on targeted engagement, including sector-specific stakeholders, by building on its substantial efforts since the interim evaluation. This would address the lingering gap between the actual performance of the programme and the stakeholders’ perception of it. This should be supported by the new Inter-service Group on Public Administration Quality and Innovation as the basis for improving the adoption of operational ready-for-use solutions (Conclusions 9, 13 – 15).

The ISA² programme should develop a more systematic business-case approach based on the Monitoring & Evaluation reporting, and the study on the cost-benefits of interoperability, both of which are ongoing. A clear business-case approach that satisfies defined eligibility criteria, as is likely to be the case in the ISA² programme based on its legislative proposal, would provide such an approach and clear guidance for stakeholders (Conclusions 5, 8 & 10).

DIGIT should, within its means and insofar as is possible, respect the targets of the follow-on programme’s envisaged staff levels (Conclusion 5).

DIGIT should build on the improvements in the coordination of activities related to interoperability and eGovernment across the Commission that it has achieved since the interim evaluation (Conclusion N° 12).

The ISA unit should document the existing services and tools established or improved by the ISA programme (Conclusion N°6) within the European Interoperability Cartography (EUCart) and Joinup, by applying the principles and guidelines from the European Interoperability Reference Architecture (EIRA). This would allow the programme to further clarify its benefits and its increasing levels of interaction with stakeholders (Conclusion No 13 – 15).

The ISA unit should develop a more systematic approach to support and monitor the use of common services and generic tools, but also the application and implementation of common frameworks (Conclusion No 6).

Introduction

Kurt Salmon and KPMG, as part of the KonSulT consortium, were mandated by the European Commission to conduct the final evaluation of the Interoperability Solutions for European public administrations (ISA) programme. This is based on Article 13(3) of the Decision No 922/2009/EC 9 , which states that ‘the ISA programme shall be subject to an interim evaluation 10 and a final evaluation, the results of which shall be communicated to the European Parliament and the Council by 31 December 2012 and 31 December 2015 respectively’.

As stated in the same Article 13(3), the final evaluation aims to ‘examine issues such as the relevance, effectiveness, efficiency, utility, sustainability and coherence of the ISA programme's actions’ and to ‘assess performance against the objective of the ISA programme and the rolling Work Programme’. The final evaluation should also specifically ‘examine the extent to which the programme has achieved its objective’.

In addition to the six aforementioned issues required by the legal decision, coordination is also considered as an additional issue, also called evaluation criteria, to be examined by the final evaluation. In fact, the evaluation should also, as stated in Article 13 (3) of the ISA legal decision, analyse the ‘benefits of the actions to the EU for the advancement of common policies, identify areas for potential improvement and verify synergies with other EU initiatives in the area of cross-border and cross-sectoral interoperability’.

In accordance with these seven evaluation criteria, Table 1 presents the main evaluation questions driving this evaluation.

Table 1 Evaluation questions related to the ISA programme’s performance

In order to monitor the progress of the final evaluation, a Steering Committee was specifically appointed. It is composed of the following members:

Ms Margarida Abecasis, European Commission, DG DIGIT B.6;

Mr Ioannis Sagias, European Commission, DG DIGIT B.6;

Mr Konstantinos Bovalis, European Commission, DG DIGIT B6;

Mr Hans Vanderbeke, European Commission, DG DIGIT B6;

Mr François Kodeck, European Commission, Former SG – IT Governance task force;

Mr Paul-Hervé Theunissen, European Commission, DG TAXUD A5;

Ms Natalia Aristimuno Perez, European Commission, DG DIGIT B2;

Ms Kate Anderson, European Commission, DG GROW B1; and

    Mr Michel Millot, European Commission, DG JRC H6.

This report has been structured in the following manner:

Chapter 1 is an introduction that provides insights on the background and objectives of the ISA programme as well as its governance and Work Programme;

Chapter 2 contains a presentation of the methodological framework for the evaluation;

Chapter 3 discusses in detail findings and presents the related conclusions for each evaluation criteria;

Chapter 4 Assesses the extent to which the recommendations from previous evaluations have been taken into account and includes, based on conclusions presented in Chapter 3, practical recommendations for the future.

Annexes referred to in this report, which contain more detailed evaluation findings, are available in a separate document called ‘Final report – Annexes’.

1Context

This chapter presents the background as well as the objectives of the ISA programme, respectively in sections 1.1 and 1.2. The ISA programme governance is described in section 1.3 whereas section 1.4 provides insights on its Work Programme.

1.1Background of the ISA programme

In 1995, the Interchange of Data between Administrations (IDA) programme was established to promote the development and operation of trans-European telematic networks for data interchange between Member State administrations and/or EU institutions. It ran from 1995 to 1999.

Following the first IDA programme, its successor programme, IDA II, was launched in 1999 and operated until the end of 2004. This programme differentiated from its predecessor by being more directed towards the market and interoperability, with the aim of increasing the efficiency of the delivery of eGovernment services to European businesses and citizens.

Decision No 2004/387/EC of the European Parliament and of the Council, adopted on 21 April 2004, concluded negotiations on the Interoperable Delivery of Pan-European eGovernment Services to public administrations, Business and Citizens (IDABC) programme, which succeeded to IDA II and entered into force on 1 January 2005.

The ISA programme aimed to build on the experience gained from the IDABC, IDA II and IDA programmes, which made important contributions to ensure interoperability to support the electronic exchange of information between European public administrations.

As mentioned in Article 17(1) of the ISA programme’s legal decision, ‘the financial envelope for the implementation […] for the period from 1 January 2010 to 31 December 2015 shall be EUR 164,100,000, of which EUR 103,500,000 is for the period from 1 January 2010 until 31 December 2013. For the period following 31 December 2013, the amount shall be deemed to be confirmed if it is consistent for this phase with the financial framework in force for the period commencing in 2014.’

As further detailed in section 1.4, the ISA programme is based on a rolling Work Programme that has to be revised and approved by the Commission at least once a year (cf. Article 9(2) of the legal decision).

1.2Objectives of the ISA programme

The hierarchy of objectives is the first step to design the different elements of the intervention logic. The identification of these objectives is indispensable for assessing, primarily, the relevance, effectiveness and utility of the programme. As defined in the framework of the interim evaluation, Figure 1 depicts the hierarchy of objectives of the programme.

Figure 1 Hierarchy of objectives of the ISA programme

Source: Decision No. 922/2009/EC of the European Parliament and of the Council of 16 September 2009 on interoperability solutions for European public administrations (ISA) (Decision located in OJ L 260, 3.10.2009, p.20).

Global objectives provide a basis for assessing the ISA programme in relation to the longer term effects (global impacts).

As stipulated under Article 1(2) of the Decision N°922/2009/EC, the objective of the ISA programme is to ‘support cooperation between European public administrations by facilitating the efficient and effective electronic cross-border and cross-sectoral interaction between such administrations, including bodies performing public functions on their behalf, enabling the delivery of electronic public services supporting the implementation of Community policies and activities.’

With reference to Article 1(1) and Article 1(2), European public administrations are defined as local and regional administrations and EU institutions (formerly called Community institutions) and bodies, including bodies performing public functions on their behalf. This definition applies to the remaining parts of this document.

As defined in the framework of the interim evaluation of the ISA programme, the global objective of the ISA programme can be expressed as ‘to support cooperation between European public administrations, with a view to enable the delivery of electronic public services supporting the implementation of Community policies and activities.’

Intermediate objectives provide a basis for assessing the ISA programme in relation to the medium term effects regarding both direct (public administrations) and indirect beneficiaries (citizens and enterprises). They are used as the means of attaining the global objective of the ISA programme.

The evaluation considers the following statement, ‘to facilitate the efficient and effective electronic cross-border and cross-sectoral interaction between European public administrations’, as the intermediate objective. This is fully in line with the legal decision and with the definition agreed at the time of the interim evaluation.

Specific objectives provide a basis for assessing the ISA programme in relation to its short term results for the direct beneficiaries. These objectives are equivalent to the deliverables of the ISA actions.

With reference to Article 3 of the legal decision, the ISA programme specific objectives could indeed be grouped into four broad categories, according to the different activities that the ISA programme shall support and promote:

-Common frameworks;

-Existing and new Common services;

-Existing and new reusable Generic tools;

-Assessment of ICT implications.

Indeed, the global and intermediate objectives of the ISA programme are realised through six specific objectives, which are constructed based on the above mentioned categories and correspond to the direct effects of the actions.

Operational objectives are related to the ISA programme actions aimed at operating the Work Programme. This relates primarily to the resources allocated, i.e. budget and human resources, which are required to accomplish the actions. With reference to Article 4, this is related to the objectives of the actions, which are specified in the Work Programme entries.

1.3ISA programme governance

The European Commission's Directorate General for Informatics (DIGIT) is responsible for coordinating the implementation of the ISA programme.

As stated in Article 12 of the ISA legal decision, ‘the Commission should be assisted by the Committee on Interoperability Solutions for European public administrations (the ISA Committee)’.

Indeed, DIGIT is assisted by a management Committee (ISA Committee), composed of representatives of each EU Member State that participate in the programme.

Article 4 and Article 7 of Decision 1999/468/EC [‘Comitology decision’ of 28 June 1999] are used to regulate the role of the ISA Committee and the procedure to be followed.

Regulation (EU) No 182/2011 11 (new comitology Regulation) was adopted on 16 February 2011. It repeals Decision 1999/468/EC (the old ‘Comitology Decision’).

There are now two comitology procedures - the advisory procedure and the examination procedure. The examination procedure replaces the management and regulatory procedures. The examination Committee can approve or reject the implementing measure with qualified majority (the same voting rules as those that apply in the Council).

In case no decision is taken, depending on the subject matter of the implementing measure, the Commission can either adopt the measure or submit it to an appeal Committee, where a new final voting takes place. Figure 2 illustrates the current governance structure of the ISA programme.

Figure 2 ISA Governance

Source: Regulation No 182/2011 of the European Parliament and of the Council of 16 February 2011 laying down the rules and general principles concerning mechanisms for control by Member States of the Commission’s exercise of implementing powers - (Regulation located in OJ L 55, 28.02.2011, p.13).

The roles and responsibilities of the various stakeholders involved in the management and delivery of the ISA programme can be summarised as follows.

The ISA Committee is composed of representatives of Member States. Its role is to assist the Commission in the implementation of the ISA programme by delivering opinions on and amendments to the Commission’s proposals. Therefore, the ISA Committee enables the supervision of the Commission’s executive power. In addition, the Commission’s draft implementing acts are discussed and debated during the ISA Committee meetings.

In accordance with Article 15 of the Decision No 922/2009/EC, ‘the ISA programme shall be open to participation, within the framework of their respective agreements with the Community 12 , by the countries of the European Economic Area and the candidate countries’. Representatives of third parties, as well as experts, can also attend the Committee meetings but cannot be present at or participate in the vote.

According to the Article 7 of the Decision 1999/468/EC (old ‘Comitology decision’), ‘each Committee shall adopt its own Rules of Procedure’.

In that regard, in order to ensure continuity and consistency at working level, the ISA Committee created the ISA Coordination Group.

The ISA Coordination Group, established by the adoption of the Terms of Reference of the ISA Coordination Group, is composed of high level experts, appointed by the ISA Committee and sometimes members of the ISA Committee itself. Its role is to apply the European Interoperability Strategy (EIS 13 ) and its related priorities into concrete actions for the ISA Work Programme, based on the expertise of the Member States, so that the ISA Work Programme actions are coordinated and aligned with the initiatives at national level. Indeed, the ISA Coordination Group ensures consistency in Member States’ contributions across the ISA Work Programme’s actions and specialists groups 14 .

The ISA Coordination Group reports to the ISA Committee on a regular basis, more specifically upstream to the ISA Committee meetings, in order to prepare their discussions.

At the suggestion of the ISA Coordination Group, the ISA Committee can appoint the ISA Working Group Members in order to delegate work on specific scopes to experts.

According to the Terms of Reference of the ISA Working Group, the role of the Working Groups is ‘to assist the Commission in implementing concrete actions of the ISA Work Programme and possibly other EU initiatives’, while facilitating the cooperation between the Commission and the Member States at the same time.

Thus, they can contribute to ISA actions in different ways: ‘by national information gathering, by providing input to, reviewing and commenting on draft documents and deliverables, by communicating and explaining ISA results, by contributing to the identification of relevant stakeholders and by informing Working Group members about national interoperability initiatives’.

Each Working Group is established on the basis of a call for nominations; as a matter of fact, they may include members of the ISA Committee. In addition, the members of the Working Group can also include representatives of other DGs, aside from DG DIGIT, and the appointed Member States high level experts.

Four Working Groups have been established within the ISA programme. Three of them follow the EIS clusters, namely one on Trusted Information Exchange, one on Interoperability Architecture and a last one on Exchange of Best Practices. These three groups were established in 2010 following the adoption of the ISA programme. They include stakeholders from different Member States as well as DG DIGIT officials and occasionally have representatives from the JRC or DG CNECT.

The fourth one was established more recently - in 2013, in the context of the ISA actions 1.17 Reusable Inspire Reference Platform and action 2.13 Development of a European Union Location Framework. It is an ISA-INSPIRE Working Group on Spatial Information and Services. The reason for establishing it was the importance of interoperability of geospatial information. Creation of such group aimed to facilitate the dialogue between the geospatial and egovernment communities in general; and particularly between the INSPIRE and ISA communities. Its members are a mix of INSPIRE and eGovernment representatives. European Commission’s stakeholders include representatives from DG CNECT, DG DIGIT, DG Environment and the JRC.

Each ISA Working Group addresses specific issues and, consequently, prepares the work of the ISA Coordination Group. In addition, members of the ISA Working Groups are meant to report nationally to the ISA Coordination Group members.

1.4ISA Work Programme

Article 9 of the ISA legal decision requires that the Commission establishes a rolling Work Programme for the implementation of the ISA programme for its full duration. The Commission must approve, at least once a year, any modifications of the ISA Work Programme, adopted for the first time by the Commission in July 2010. Amendments can be related either to minor or to significant changes of existing ISA actions 15 or to proposals for new ISA actions. Five revisions of the rolling Work Programme have occurred to date. 

Its main goal is to define the actions which will be implemented throughout the year. The ‘ISA actions’ can be of different kinds; namely studies, projects and accompanying measures. The policy context of those actions corresponds to the vision and goals outlined in the European Interoperability Strategy (EIS) 16 and the European Interoperability Framework (EIF) 17 .

The programme’s budget finances a series of ISA actions aimed at establishing or operating three different types of solutions, but also assessing the ICT implications (with reference to Art. 3(b)) of proposed or adopted EU legislation and planning for the introduction of ICT systems to support the implementation of such legislation.

With reference to the ISA legal decision:

Article 2(b): ‘Solutions means common frameworks, common services and generic tools’;

Article 2(c): ‘Common Framework means strategies, specifications, methodologies, guidelines and similar approaches and documents’;

Article 2(d): ‘Common Services means operational applications and infrastructures of a generic nature which meet common user requirements on policy areas’;

Article 2(e): ‘Generic tools means reference platforms, shared and collaborative platforms, common components and similar building blocks which meet common user requirements across policy areas’.

As of today, 38 18 actions are being implemented under the ISA Work Programme (2015) 19 , out of which 12 correspond to common frameworks, 8 to common services, 13 to generic tools and 1 to the assessment of ICT implications of EU legislation. The rest of the ISA actions (4 actions) refer to ‘Accompanying measures’ (Article 2(g): ‘Accompanying measures means strategic and awareness-raising measures, measures in support of the management of the ISA programme and measures in relation of the sharing of experience and the exchange and promotion of good practices’), ‘Awareness’ and ‘Monitoring’ of the ISA programme.

The design of the actions is based on proposals made on an annual basis by the Commission services and Member States through the ISA Committee. Actions relevant to specific policy areas are designed in close coordination with the Commission service(s) responsible for the policy area.

The ISA programme is articulated around the following three clusters:

1.Trusted information exchange;

2.Interoperability architecture; and

3.Assessment of ICT implications of EU legislation.

A tabular overview of the actions is proposed in Table 2 and includes the action number, the type of action (i.e. study, project or accompanying measure), the type of activity (common services, reusable generic tools, common frameworks and assessment of ICT implications) and the year of inclusion into the rolling Work Programme as well as the year of conclusion, for actions having their funding concluded yet.

Table 2 Tabular overview of the actions

2Methodology

The final evaluation’s methodology 21 conforms to the methodological guidance for evaluations provided by DG Budget 22 and adheres to developments in the Smart Regulation policy 23 . It incorporated and expanded on the main evaluation questions, listed in the ‘ Introduction ’ section. It is composed of the following four phases, which are elaborated on in this chapter:

Figure 3 Methodology of the final evaluation

Source: European Commission, Evaluating EU Activities: A practical guide for the Commission Services, Office for Official Publications of the European Communities, Luxembourg, 2004.

2.1    Inception (Evaluation structuring)

The inception phase defined the methodology ( Figure 3 ), based on the main evaluation questions and the intervention logic ( Figure 4 , below) from the interim evaluation of the ISA programme. It also identified the relevant stakeholders. A major part of this inception phase was the choice of adequate data collection methods and the design of processes to support their subsequent implementation. In particular, these activities included:

The development of specific questions, linked to the main evaluation questions, for the data collection phases, including the identification of related judgement criteria and indicators; and

Planning and creation of questionnaires/guides for the interviews and online surveys containing the specific questions.

The evaluation team’s significant investment in the organisation of the data collection activities as part of the inception phase was to ensure the reliability of the information on which a sound analysis could be built. The evaluation team drew on its experience and expertise to determine these elements and construct the overall methodology to conduct the final evaluation.

2.1.1Evaluation criteria & main evaluation questions

The table below maps the evaluation criteria and their definitions, as contained in the EC’s official guidelines22, to the main evaluation questions of the final evaluation of the ISA programme.

Table 3 Mapping of evaluation criteria with main evaluation questions

Once these main evaluation issues and associated questions were agreed, the evaluation team defined specific sub-questions, linked to each main evaluation question, to be addressed by the principal data collection methods, the interviews and online surveys as well as by desk research. These interview and online survey guides are contained in the Annex, which accompanies this report.

2.1.2Intervention Logic

The EC’s official guidelines22 define the intervention logic of a programme as ‘the conceptual link from an intervention's inputs to the production of its outputs and, subsequently, to its impacts on society in terms of results and outcomes.’ As agreed with the European Commission, the intervention logic of the interim evaluation of the ISA programme was reused for the final evaluation 24 . The examination of the programme's intervention logic with data collected during the evaluation is of central importance to identify its success in achieving its objectives.

Figure 4 Intervention logic of the ISA programme

Source: European Commission, Evaluating EU Activities: A practical guide for the Commission Services, Office for Official Publications of the European Communities, Luxembourg, 2004.

2.1.3Stakeholders

The ISA unit and the Steering Committee identified the following eight main stakeholder groups as those involved in, or impacted by, the ISA programme. Representatives of these stakeholder groups were involved in the final evaluation’s data collection activities.

Table 4 Key stakeholder groups

2.2Data collection and analytical tools

The data collection methods and analytical tools chosen to answer the evaluation questions are both qualitative and quantitative. The table below provides a summary of these. The Annex accompanying this report contains details on each of these.

Table 5 Data collection and analytical tools

2.3Data analysis and synthesis

Data analysis and synthesis is the major activity that assessed the data collected during the research phase. It translates data into understandable findings, from which specific conclusions can be drawn as the basis of final recommendations. The following two steps ensured this.

Table 6 Data analysis and synthesis steps

2.4Formulation of judgments and reporting

The activity to formulate robust conclusions on the implementation of the ISA programme and provide useful and practical recommendations on future Work Programmes, in particular the ISA² programme, involved the following three activities:

Table 7 Formulation of judgments and reporting

3Findings and results

This chapter is divided into seven sections in accordance with the seven aforementioned evaluation criteria.

3.1Relevance

The evaluation of the relevance of the ISA programme is driven by the following evaluation question (EQ):

EQ1: To what extent were the ISA programme's objective(s) pertinent to meet evolving needs and priorities at both national and EU levels?

As mentioned in section 1.2, the global objective of the ISA programme is to support cooperation between European public administrations, with a view to enable the delivery of electronic public services supporting the implementation of EU policies and activities. The intermediate objective is to facilitate the efficient and effective electronic cross-border and cross-sectoral interaction between European public administrations.

To evaluate whether these objectives were pertinent to meet evolving needs and priorities at both national and EU levels, the evaluation verified the extent to which the needs expressed by European public administrations, at the time of the ex-ante and interim evaluations of the ISA programme, can still be considered as current needs related to interoperability. New evolving needs were also identified in order to assess the extent to which the ISA programme’s objectives have addressed them. On the other hand, the evaluation also verifies the pertinence of the programme’s objectives to main EU priorities such as the Digital Agenda for Europe.

The remainder of this section is divided into different subsections in accordance with the different judgement criteria allowing the evaluation to answer EQ1.

3.1.1ISA programme objectives

With regard to these aforementioned objectives, there was a clear agreement on their continued validity. In fact, 95% of the interviewees and survey recipients agreed (64) or somewhat agreed (41) that the objectives of the programme were still valid today. Moreover, 11 respondents specifically highlighted that there is still work to be performed to fully reach these objectives. It has also to be noted that some respondents (3) clearly suggested that the objectives of any successor programme should not only focus on public administrations but also on public services targeting citizens and businesses. This recommendation has already been taken into account in the proposal for a decision establishing a programme on interoperability solutions for European public administrations, businesses and citizens (ISA²); the proposed general objective of ISA² is to facilitate interoperability between European public administrations, and also between them and citizens and businesses.

After having verified the level of validity of the programme’s objectives, the evaluation also addressed a question particularly focused on the intermediate objective of the programme. In fact, all the respondents were asked about their level of agreement on the extent to which the programme facilitates efficient and effective electronic cross-border and cross-sectoral interaction between European public administrations.

40% fully agreed that the ISA programme facilitated efficient and effective electronic cross-border and cross-sectoral interaction between European public administrations (44), whereas 53% of respondents somewhat agreed (58). The respondents primarily commented that the programme aims to and does facilitate interaction between European public administrations but room for improvement exists to concretely achieve this objective (11).

3.1.2Needs of EU public administrations

In order to assess the extent to which the objectives of the ISA programme were pertinent to meet evolving needs at national and EU levels, the evaluation firstly identified the main needs of public administrations at both national and EU levels.

In this regard, the evaluation verified the extent to which the needs expressed by European public administrations at the time of the ex-ante 26 and interim evaluations of the ISA programme, shall still be considered as current needs related to interoperability:

Existing generic tools, services and infrastructures have to be consolidated and re-used;

ICT implications have to be taken into account upfront in the legislative process;

Data and information exchange has to be secured;

Common frameworks, guidelines and specifications have to be established;

The benefits of interoperability have to be more communicated;

Coordination between the ISA programme and both the Member States and Commission services has to be increased;

ISA solutions have to be more reused.

Figure 5 depicts the views of the interviewees and survey recipients on the extent to which these needs should still be considered as current needs of EU public administrations.

Figure 5 Needs of EU public administrations

As depicted in Figure 5 , a large majority of respondents fully and somewhat agrees that each of the proposed needs should still be considered as current needs of EU public administrations. As a result, the evaluation draws a conclusion that these aforementioned needs have still to be considered as current needs of EU public administrations.

In addition to these aforementioned needs, eight additional ones were identified as potential new needs during the interviews.

Survey respondents have thus been asked to provide their level of agreement to the question ‘To what extent do you agree that each of the following [eight potential] needs should be considered as a new or evolving need since the adoption of the ISA programme?’

Out of a total of 67 answers, these eight needs are presented in a descending order based on the number of 'Fully agree' answers by EC officials and Member States to the aforementioned question (indicated in brackets following each one):

1.Improvement of the governance and coordination among Commissions DGs and services on programmes and initiatives related to interoperability (40);

2.Improvement of the governance and coordination between Member States and the Commission on programmes and initiatives related to interoperability (32);

3.Adaptation and update of the focus of the ISA programme to incorporate evolutions in technology that could support interoperability (e.g. Big Data and cloud computing) (27);

4.Need to address security and cyber-security of Member States as part of work related to interoperability (26);

5.Adaption and update of the focus of the ISA programme to involve businesses, citizens, civil society as stakeholders for the development and use of solutions (26);

6.Incentives for the use of the ISA programme's solutions (e.g. financial support to Member States for adoption and implementation of solutions until operational) (26);

7.Definition and promotion of key interoperability enablers to foster standardisation at national and cross-border levels (25);

8.Improvement of the coordination and cooperation between Member States on programmes and initiatives related to interoperability (23).

Based on this ranking, the evaluation concludes that an improvement of the governance and coordination among Commissions DGs and services but also between the Commission and Member States on programmes and initiatives related to interoperability are perceived as the main two new needs to be considered. The evaluation notes that these two new needs are beyond the scope and control of the existing ISA programme alone and that the need for an ‘increased coordination between the ISA programme and both the Member States and Commission services’ was already identified at the time of the interim evaluation. The establishment of the CEF programme, subsequent to the launch of the ISA programme, was frequently mentioned by interviewees and online survey respondents as a reason to ensure ‘increased coordination between the ISA programme and both the Member States and Commission services’.

One should note that the opinions on potential needs leading to the aforementioned ranking were homogenous among both EC officials and Member States for most of the needs, except for two of them. In fact, on the one hand, the ‘Adaptation and update of the focus of the ISA programme to incorporate evolutions in technology that could support interoperability’ was mainly seen as an important need for the EC officials whereas it was the one least cited by Member States. On the other hand, Member States consider the ‘Incentivisation of the use of the ISA programme's solutions’ as one of the most important needs whereas it was the one that was considered as the least important by EC officials.

3.1.3Pertinence of the ISA objectives to respond to needs of EU public administrations

The list below presents the extent to which the ISA programme has addressed each of the needs identified at the time of its ex-ante and interim evaluations. These needs are presented in an ascending order based on the number of times that they were recognised as addressed by the ISA programme by the 96 respondents who provided an opinion on this matter as shown in the Figure 6 .

Figure 6 Needs addressed by the ISA programme

On the one hand, it appears that the needs of having ISA solutions more reused as well as an increased collaboration between the programme and both the Member States and Commission services are the two needs that have been the least addressed by the ISA programme. In fact, these needs are not seen as addressed by the ISA programme by more than half of respondents.

On the other hand, the following two needs are considered as addressed by the ISA programme by a large majority of respondents (respectively by 72% and 82%):

Existing generic tools, services and infrastructures have to be consolidated and re-used (69);

Common frameworks, guidelines and specifications have to be established (79).

With regard to the needs considered as new ones, i.e. improvement in governance & coordination, since the adoption of the programme (cf. previous subsection 3.1.2), respondents were asked to provide their opinion on how the ISA programme has taken them into account.

Based on a total number of 23 answers, the programme has partially taken the aforementioned new needs of European public administrations into account for a majority of respondents (61%) and fully taken them into account for 26% of respondents who provided their opinion on this question. A mere 13% considered that the programme has not taken these new needs of European public administrations into account at all. Nevertheless, 22% of these aforementioned respondents highlighted that the proposed ISA² programme is a good opportunity to take the new evolving needs of European public administrations, i.e. improvement in governance & coordination, more into account.

The evaluation confirms that the current programme is anyway currently taking new evolving needs on governance & coordination into account. In fact, the European Commission is in the process of revising, in close cooperation with the Member States, the European Interoperability Strategy (EIS) and European Interoperability Framework (EIF) 27 . Indeed, as part of the EIS Revision, the interoperability governance organisation & processes are in the process of being established. This is seen as a fundamental opportunity to address these new evolving needs on governance and coordination.

3.1.4Alignment of the ISA objectives with EU priorities

In order to assess the extent to which the objectives of the ISA programme were also pertinent to meet evolving priorities at EU level, the evaluation verified the alignment of the ISA programme with the following EU initiatives:

Digital Agenda for Europe (DAE);

European eGovernment Action Plan 2011-2015;

eCommission 2012-2015;

Commission IT Rationalisation Initiative;

Connecting Europe Facility (CEF);

Digital Single Market (DSM) Strategy.

Of 98 EC officials and Member States’ representatives who responded, most believed that the ISA programme objectives are primarily aligned with the DAE and with the European eGovernment Action Plan 2011-2015 as well as, to a lesser extent, with the CEF programme and DSM strategy, which were established much later than the DAE and the European eGovernment Action Plan, respectively on 11 December 2013 and 6 May 2015.

In fact, with regard to the alignment between the ISA programme and the DSM strategy, the majority of the interviews were conducted before the establishment of the DSM strategy. This fact explains the reason why the number of ‘Don’t know/No opinion’ answers is higher for this new strategy compared to the number of ‘Don’t know/No opinion’ for DAE, European eGovernment Action Plan or CEF. 80% of respondents were however able to provide their opinion considering that the draft DSM strategy went through inter-service consultation within the European Commission and was made available to Member States as well.

Figure 7 describes the level of alignment of the ISA programme objectives with the aforementioned EU initiatives as perceived by Member States and European Commission officials. These are sorted by the overall number of occurrences of the fully agree answers.

Figure 7 Alignment of ISA objectives with EU initiatives

The objectives of the programme are perceived as less aligned with the eCommission 2012-2015 and Commission IT rationalisation initiative. However, as highlighted by several respondents, the eCommission 2012-2015 as well as the IT Rationalisation initiative are strategies more focused on internal systems of the European Commission whereas the scope of the programme is broader by nature.

In fact, the eCommission 2012-2015 is the European Commission’s IT strategy whereas the Commission IT Rationalisation Initiative is an overall strategy established in October 2010 by the European Commission to rationalise and streamline the IT systems it develops, maintains and operates.

The evaluation found that the highest level of disagreement was related to the alignment between ISA and the CEF programme. The majority of respondents in disagreement with the alignment between CEF and ISA singled out a lack of coordination with DG CNECT. This aspect will be further detailed in Section 3.6.3.

Nevertheless, the factual evidence that several actions previously funded by the ISA programme are now part of CEF Work Programme, e.g. MT@EC, STORK, implies a good alignment between these two programmes and counterbalances the respondents’ opinions.

In addition to the EU initiatives depicted in Figure 7 , respondents were also asked about the alignment of ISA with any other EU initiatives. In that regard, one main additional initiative was identified by five respondents, namely INSPIRE.

With regard to the perceived alignment with INSPIRE, the evaluation considers that this can be primarily due to two main reasons. Firstly, in 2012, two new actions linked to the INSPIRE directive, namely AR3NA (1.17) and EULF (2.13) were included in the revised ISA Work Programme. Secondly, as explained in section 1.3, the ISA programme also established, in 2013, in the context of these two ISA actions, a dedicated ISA-INSPIRE Working Group on ‘Spatial Information and Services’. This Working Group plays an important advisory role, helping to define requirements and priorities and to establish the direction of the two aforementioned ISA actions. It aims to help to facilitate dialogue between the e-government and European geospatial communities, in general, and between ISA and the community implementing the INSPIRE directive in particular.

The establishment of this Working Group can be seen as a positive result in the light of the new evolving needs identified in subsection 3.1.2, especially the new evolving need for    improvement of the governance and coordination among Commissions DGs and services on programmes and initiatives related to interoperability.

The remainder of this subsection contains further details on the alignment of the programme with major EU initiatives based on desk research in order to clarify the reality. Specifically, these major EU initiatives are the Digital Agenda for Europe (DAE), the European eGovernment Action Plan 2011-2015, the Connecting Europe Facility (CEF) and the Digital Single Market (DSM) Strategy.

Digital Agenda for Europe (DAE)

On 19 May 2010, the Commission adopted a Communication proposing a Digital Agenda for Europe, one of the seven flagship initiatives of the Europe 2020 Strategy, set out to define the key enabling role that the use of ICT will have if Europe wants to succeed in its ambitions for 2020.

Considering that the Digital Agenda for Europe is composed of 132 actions divided into seven pillars, the evaluation aims to verify the pertinence of the ISA programme to the DAE pillars. In this respect, the ISA programme is primarily aligned with the following three DAE pillars:

Pillar I: Digital single market;

Pillar II: Interoperability and standards;

Pillar VII: ICT-enabled benefits for EU society.

Based on evidences provided by the ISA Work Programme revisions and the interim evaluation of the programme, the evaluation confirms that several ISA actions are pertinent to DAE actions under these pillars. For each identified pillar, the specific DAE action is linked with the relevant ISA action(s) in Table 8 . Further details about this alignment are available in Annex.

Table 8 Alignment of the ISA programme with the DAE

The Digital Agenda for Europe was updated at the end of 2012 28 and additional actions have been included under its umbrella as of 2013. Compared to the alignment that was already observed at the time of the interim evaluation between the ISA programme and the first set of actions of the DAE, it should be noted an alignment between ISA actions and the aforementioned additional DAE actions. In fact, ISA actions included in the first ISA Work Programmes (e.g. ISA action - 1.07) as well as new actions included after 2012 are aligned with the initial (e.g. alignment between ISA action 4.2.06 and the DAE action 77) as well as updated version of the DAE (e.g. ISA action 1.21 and 1.22 with the new DAE action 110).

European eGovernment Action Plan 2011-2015

In order to support and complement the Digital Agenda for Europe, the European Commission proposed a European eGovernment Action Plan 2011-2015 29 aimed to support the provision of a new generation of eGovernment services for businesses and citizens.

The Action Plan identifies four political priorities (user empowerment, internal market, efficiency and effectiveness of eGovernment and administrations, pre-conditions for developing eGovernment) based on the Malmö Declaration, agreed on 18 November 2009 at the 5th Ministerial eGovernment Conference in Malmö, Sweden, and proposes concrete actions and measures to be implemented by the Commission and the Member States by 2015.

As described in Table 9 , the ISA programme is also aligned with these four priorities. Further details about this alignment are available in Annex.

Table 9 Alignment of the ISA programme with the European eGovernment Action Plan 2011-2015

Looking forward, it is important to note that the ISA programme is in the process of being aligned with the future eGovernment Action Plan 2016-2020 30 .This is being developed within the framework of the Digital Single Market strategy, which is covered below in this sub-section. As part of this, the ISA programme participated in two workshops on the new EU eGovernment Action Plan in June and September 2015 31 . The eGovernment Action Plan 2016-2020 will include an initiative on the application of the ‘once-only principle’ (OOP), which means that citizens and businesses supply information to public administrations only once.

The application of this principle is quite difficult in a cross-border context, considering that it is already challenging for Member States to apply in a national context, which is obviously a pre-requisite. Nevertheless, it can be achieved technically through a pilot with Member States to achieve the interconnection of databases, i.e. national registers, to avoid asking citizens and businesses for information more than once. Evidently, the ISA programme has a role to play here, in particular, given its work under action 1.02 Access to Base Registries. In addition, another intended pillar of the future eGovernment Action Plan 2016-2020 is to accelerate Member States' transition towards full e-procurement and interoperable e-signatures, where the work done under actions 1.07 and 1.09 respectively provides the ISA programme with great potential to contribute.

Connecting Europe Facility (CEF)

The Connecting Europe Facility (CEF) is the common financing instrument of Trans-European networks for the period 2014-2020. During this period, CEF will help to complete the European single market by making available EUR 33.24 billion in the form of procurement, grants and innovative financial instruments.

CEF will finance projects of common interest in three different sectors: transport (EUR 26.25 billion), energy (EUR 5.85 billion) and telecommunications (EUR 1.14 billion) 32 .

Within the telecommunications area, 170 million euros are earmarked for Broadband activities, while 970 million euros are dedicated to Digital Service Infrastructures (DSIs) delivering networked cross-border services for citizens, businesses and public administrations. These projects are to:

improve the competitiveness of the European economy;

promote the interconnection and interoperability of national, regional and local networks; and

facilitate access to such networks, thus supporting the development of a Digital Single Market 33 .

After careful consideration of the respective Work Programmes, the evaluation confirms that several actions included in ISA’s 2015 Work Programme are pertinent to 7 out of 11 planned CEF’s digital service infrastructures (DSIs) under its 2014 and 2015 Work Programme. Table 10 summarises the alignment of the ISA programme with the old and new DSIs as included in the Work Programmes (WP) for 2014 or 2015 respectively. Further details about this alignment are available in Annex. As shown in this table, most of the link existing between ISA actions and DSIs included in the CEF 2014 WP are due to the fact that several mature ISA solutions have been taken over by the CEF programme as of 2014.

Table 10 Alignment of the ISA programme with CEF

Digital Single Market (DSM) Strategy

The Digital Single Market Strategy, recently adopted by the Commission on 6 May 2015 34 , aims to open up digital opportunities for people and business and enhance Europe's position as a world leader in the digital economy.

The DSM strategy includes a set of 16 targeted actions to be delivered by the end of 2016 and is built on the following three pillars:

Pillar I: Better access for consumers and businesses to digital goods and services across Europe;

Pillar II: Creating the right conditions and a level playing field for digital networks and innovative services to flourish; and

Pillar III: Maximising the growth potential of the digital economy.

The evaluation highlights that the ISA programme remains relevant primarily for pillar III as some of its initiatives have links to ISA actions. Table 11 summarises the pertinence of the ISA programme towards this pillar. However, the evaluation would like to raise the following caveat. Given that the DSM strategy was only adopted in May 2015, after the adoption of the latest ISA Work Programme, the alignment of the ISA programme with its actions and priorities is difficult to capture. Further details about this alignment are however available in Annex.

Table 11 Alignment of the ISA programme with the DSM Strategy

Overall, the most significant relevance is related to action 15 from the DSM: in the roadmap of the recently published Communication, it is mentioned the need to revise and extend the European Interoperability Framework (EIF). The ISA programme is, at the time of this report, in the process of revising the European Interoperability Strategy and, through the NIFO action, extending the European Interoperability Framework. On the other hand, the results of action 1.02 Access to Base Registries, i.e. best practices and recommendations, are in line with the future initiative on the 'Once-Only' principle, which is part of action 16 of the DSM.

3.1.5Conclusions

The following box summarises the main conclusions related to EQ1: ‘To what extent were the ISA programme's objective(s) pertinent to meet evolving needs and priorities at both national and EU levels?’

Conclusion N°1 As mentioned, the ISA programme aims to support cooperation between European public administrations, with a view to enabling the delivery of electronic public services supporting the implementation of EU policies and activities. The evaluation considers this objective as still valid today. Even if room for improvement exists to concretely achieve this objective (see subsections 3.2 to 3.4), it is confirmed that the programme does facilitate interaction between European public administrations. The modified scope of the follow-on ISA² programme will, most likely, enhance this objective. The follow-on ISA² programme’s scope aims to facilitate electronic cross-border or cross-sector interaction between European public administrations, businesses and citizens as well as between European public administrations.

Conclusion N°2 The needs of European public administrations, identified at the time of the ex-ante and interim evaluations of the ISA programme (cf. subsection 3.1.2), are still considered as the current needs of European public administrations today. However, the follow-on programme should further address the need to have ISA solutions more reused as well as the need for an increased collaboration between the programme and both the Member States and Commission services. The opinion of interviewees and survey respondents was clear in this regard. The current usage levels of ISA solutions, as documented by the desk research contained in section 3.4, show that there is considerable potential for their greater levels of adoption and use, which reinforces this conclusion. The evaluation confirms that the programme should focus more on the reuse of the existing solutions but that, as a voluntary policy, the ISA programme has no capacity to guarantee greater reuse of its solutions and increased collaboration depends on the willingness of all stakeholders.

3.2Efficiency

The evaluation of the efficiency of the ISA programme is driven by the following evaluation questions:

EQ2: How economically were the various inputs converted into outputs?

EQ3: Which aspects of the programme were the most efficient or inefficient, especially in terms of resources mobilised?

The remainder of this section is divided into different subsections in accordance with the different judgement criteria allowing the evaluation to answer EQ2 and EQ3.

3.2.1Revision process of the ISA Work Programme

As mentioned in section 1.4, the Commission must approve at least once a year any modification of the ISA Work Programme. In this respect, DIGIT requests the members of the ISA Committee, as well as the different Commission services, to submit their proposals for new actions to be included in the rolling Work Programme.

Proposals may either be for projects, studies or accompanying measures and should be related to the activities defined in Article 3 of the Decision:

Establishment and improvement of common frameworks in support of interoperability across borders and sectors. Common frameworks are strategies, specifications, methodologies, guidelines and similar approaches and documents.

Assessment of ICT implications of EU legislation as well as the planning of the implementation of ICT systems in support of the implementation of such legislation.

Establishment, industrialisation, operation and improvement of common services that are operational applications and infrastructures of a generic nature.

Establishment and improvement of reusable generic tools that are reference platforms, shared and collaborative platforms, common components and similar building blocks.

These proposals must comply with the ISA legal decision’s objectives as stipulated in its Article (1). In addition, in accordance with Article 2 36 of the Decision, proposals for common services or generic tools should demonstrate their readiness to cover multiple sectors; for example by being jointly introduced by several Directorates-General or by one Directorate General (DG) acting on behalf of a group of DGs sharing common requirements.

Every year, once proposals are received by the ISA unit, the revision process is launched and follows three main steps:

A first draft revision of the Work Programme is elaborated and sent to the ISA Coordination Group members. The rolling Work Programme should include a description of each action, as stated in Article 9(4) of the ISA legal decision. In case an action is selected for funding, most of the information provided through the proposals will be integrated into the rolling ISA Work Programme, which will be presented to the ISA Committee for its opinion and subsequently will be adopted by the Commission.

Based on the deliberations of the ISA Coordination Group, a second draft revision is designed and sent to inter-service consultation.

Based on the replies submitted by the Commission Services, the final draft revision of the Work Programme is presented to the ISA Committee that must deliver its opinion, prior to the adoption of the Work Programme by the Commission.

The comments from both ISA Coordination Group members (first draft revision) and Commission services (second draft revision), through inter-service consultations, are taken into account in the final draft submitted to the ISA Committee. These two steps should be sufficient to ensure that the final draft of the Work Programme receives a favourable opinion by this Committee.

Based on the adopted Work Programmes in the period 2011-2015, the evaluation concludes that the revision process is efficient, since each revision of the Work Programme received a favourable opinion from the ISA Committee 37 . However, as further detailed in sub-section 3.3, the inclusion of EUSurvey in each of the Work Programme revisions was questioned by four respondents. The evaluation acknowledges that, even if this ISA solution supports different EU policies and contributes to eParticipation 38 , it does not facilitate interaction between European public administrations.  Figure 8 describes in details the number of proposals for new actions included and not included during the different Work Programme revisions.

Figure 8 Selection of proposals for new actions in the different Work Programme revisions

Concerning the proposals included in the final draft, proposals were in some cases integrated into existing actions, and in other cases included as new action within the amended Work Programmes 39 .

However, considering that some proposals were not included in the Work Programme revisions, it might be that either the selection criteria were not clear enough or not enough attention was given to these criteria when writing the proposals (e.g. proposals were rejected in 2013 due to the fact that their scopes were more related to national needs only and, thus, not aligned with the scope and objectives of the programme).

Respondents were thus asked to provide their feedback on the selection procedures applied by the ISA programme.

Overall, the selection procedures of proposals are indeed considered as fully or somewhat appropriate, so that the ISA actions selected for funding comply with the overall objectives of the programme, by 74% of all interview and survey respondents (35), with only 2% (1) of respondents finding them somewhat inappropriate. Minor deficiencies in the selection procedures were usually caused by a lack of involvement of Member States due to the complexity of the process and, hence, a need for extra support or guidance. However, this need is not major as most Member States’ representatives (68% (28)) felt fully or somewhat involved in the annual revisions of the ISA Programme. It is interesting to note that, if they did not, it was more often than not due to internal issues (e.g. in Denmark, or Sweden).

Interestingly, even though the involvement of Commission services was not a key concern in the open-ended questions, the analysis showed that a lower percentage of respondents of this group felt fully or somewhat involved (58% (28)). The evaluation confirms that both Member States’ representatives and EC officials could participate in the selection process through different means: inter-service consultations, management meetings or by provision of direct comments/inputs to consultations.

Overall, both Member States’ representatives and EC officials expressed satisfaction with their levels of involvement in the consultations. Only two respondents believed that further guidance and support on the selection procedure would be beneficial. The overall level of satisfaction about the provision of guidance and information from the Commission in regard to the selection procedures is good, with 57% (52) of the participants being fully or somewhat satisfied.

During the interim evaluation, the Member States’ representatives expressed the need for further guidance and support to enable them to better participate in the proposals selection. Along with Member States’ representatives, the EC officials expressed similar needs at that time.

The evaluation highlights that, compared to the findings of the interim evaluation, progress has been made in relation to stakeholders’ levels of satisfaction regarding the selection of proposals for new ISA actions.

One should note that, at the time of the final evaluation; EC officials were less likely to be fully or somewhat satisfied (53% (26)) than the Member States’ representatives (63% (26)) with the information and guidance provided by the Commission. Furthermore, none of the Member States’ representatives was fully or somewhat dissatisfied with the level of information and guidance provided by the European Commission, whilst 12% (6) of EC officials expressed some sort of dissatisfaction. Nevertheless, EC officials did not mention any specific problem on the provision of information and guidance during the selection of the proposals. On the other hand, although a majority of Member States’ representatives were fully or somewhat satisfied, they presented several measures that could even ease more the process of selection of proposals for them and, as a result, increase the overall efficiency of this process. These included the following measures:

(I) Increase the time to read the necessary documentation and the time for the development of the proposals; with regard to this point, the evaluation acknowledges that DG DIGIT provides both Member States and Commission services with over two months for the development and submission of a proposal. For example for the round of 2014, the Commission services were advised on 26th June 2014 with deadline for submission on 1st September, and for the round of 2015, the Commission services were advised on 22nd June 2015 with deadline for submissions on 17th August 2015. The evaluation confirms that the given time, of at least 8 weeks, shall be sufficient.

(II) Improve the precision and clarity of the documented process about the selection of proposals for new actions; including the improvement of clarity of the selection criteria related to the coordination of activities between the stakeholders. This shall be done in order to avoid inefficiencies, such as duplication of the work or other procedural misunderstandings, in the coordination of activities between ISA and the Member States in the process of preparation and approvals of proposals.

With regard to this point, the evaluation acknowledges that DG DIGIT provides both Member States and Commission services with Work Programme template, as well as explanatory notes to help them in proposals’ preparation. Furthermore, three main contacts from DG DIGIT are available to provide the necessary guidance and clarifications. The template includes all necessary elements to evaluate a proposal. This is judged by the evaluation as satisfactory even if, as mentioned by four Member States’ representatives as well as by one EC official, a more precise and clear documented process about the selection of proposals for new actions could be helpful; particularly for the new or recent stakeholders.

3.2.2Implementation rules

Each action funded by the ISA programme should comply with the six implementation rules defined in Article 8 of the ISA legal decision:

1.In the implementation of the ISA programme due consideration must be given to the EIS and the EIF.

2.Involvement of the largest possible number of Member States in a study or project must be encouraged. A study or project must be open for accession at any stage, and Member States not involved in a study or project must be encouraged to join at a later stage.

3.In order to ensure interoperability between national and EU systems, common frameworks, common services and generic tools must be specified with reference to existing European standards or publicly available or open specifications for information exchange and service integration.

4.The establishment or improvement of solutions must, where appropriate, build on or be accompanied by the sharing of experience and the exchange and promotion of good practices.

5.In order to avoid duplication and to speed up the establishment of solutions, results achieved by other relevant Community and Member State initiatives must be taken into account, where appropriate.

6.The initiation of actions, the definition of the phases of such actions and the establishment of project charters and execution reports must be carried out and monitored by the Commission as part of the implementation of the rolling Work Programme established in accordance with Article 9.

Accordingly, a majority of respondents (66%) fully or somewhat agreed that due consideration is given to the EIS and the EIF in the implementation of the ISA programme (implementation rule No1), whilst only 4% of respondents (4) fully or somewhat disagreed. Overall, 66% (64) of respondents fully or somewhat agree that the involvement of the largest possible number of Member States in ISA actions is encouraged (implementation rule No2), whilst only 5% (5) of them fully or somewhat disagree with the statement. This opinion is shared by both EC officials and Member States’ representatives, which confirms that the implementation of the ISA actions adheres to the second implementation rule. With regard to the involvement of Member States in ISA actions, the lack of financial or human resources at national level was mentioned by three respondents as a barrier limiting their involvement.

After further analysis of respondents’ answers, the evaluation concludes that there is a general satisfaction among stakeholders on the application of implementation rules No3, No4 and No5 for selecting new proposals. Firstly, an overwhelming majority of respondents (77%) fully or somewhat agree that the ISA solutions are specified with a reference to existing European standards or open specifications (third implementation rule), whilst only 2% of respondents (2) fully or somewhat disagree. Secondly, 66% of respondents (64) fully or somewhat agree that the establishment or improvement of solutions is, where appropriate, built on/accompanied by the sharing of experience and the exchange and promotion of good practices (fourth implementation rule). Thirdly, the fifth implementation rule is also regarded as sufficiently adhered to by the different ISA actions as 63% (63) of respondents fully or somewhat agree on this issue. However, with regard to the fifth implementation rule, the ISA programme was criticised by seven respondents who considered it primarily not sufficiently proactive and lacking the general overview of all of the results achieved by other relevant EU and Member State initiatives. Nevertheless, a few successful examples were mentioned by the respondents. For example, work undertaken separately by several Member States on automatic translation was subsequently integrated by the ISA programme into the action 2.08 MT@EC.

Lastly, the perception of respondents is positive about the ISA programme’s performance regarding the sixth implementation rule, which aims to maximise synergies and ensure complementary and combined efforts. A small minority (3) of respondents would prefer the process of monitoring to become more open and transparent. A majority of respondents (74%) fully or somewhat agreed that the initiation of the action, the definition of the phases of such actions and the establishment of project charters and execution reports is carried out and monitored by the Commission, whilst no respondents fully or somewhat disagree on this matter.

Regarding the evaluation of the implementation rules by the respondents, no significant differences exist between the two surveyed stakeholder groups; the Member States’ representatives and EC officials. Nonetheless, Member States tended to express higher levels of agreement for the all of the implementation rules except for the sixth. EC officials expressed lower levels of agreement on the adherence to implementation rules but were also more likely than Member States’ representatives not to have an opinion on them. For the EC officials, the ‘Don’t know / No opinion’ was expressed by at least 19% (11) of respondents for all the six implementation rules, which is noticeably higher than for the Member States. In fact, the percentage of ‘Don’t know / No opinion’ for Member States did not exceed 12% (5) for any of the implementations rules.

3.2.3Allocation of financial resources

Funds Release Process

As mentioned in the previous section the Commission amends the rolling Work Programme (Work Programme revisions), at least once a year. Funds are released to the ISA action on the basis of the achievement of specific milestones:

Inclusion of the action in the Work Programme, for the initiation of a study, an accompanying measure or the inception phase of a project;

Delivery of the project charter, for the initiation of the execution phase of a project; and

Delivery of the execution report, for the initiation of the subsequent operational phase of a project).

In this respect, each ISA action owner can decide to delegate the budget fully or partially to other Commission services.

Table 12 Funds released and withheld

For each ISA Work Programme in the period 2010 - 2015, Table 12  shows the date of adoption of a new Work Programme and date of first funds release after the adoption, as well as the total released amount for each Work Programme revision. The last column of this table indicates the withheld amounts that were subsequently released. The first release of funds occurred, in average, five working days after the adoption of the Work Programme (with significant deviation observed only for the adoption of the second revision of the ISA Work Programme 2012). Since 2012, the funds release time has been primarily on a decrease. Overall, the funds release process can be considered as working smoothly with no significant delays.

A slight deficiency was highlighted by one of the stakeholders. The use of the programme’s funds would be more efficient if the time schedule of the release of funds would occur earlier in the year. Currently, the funds for the respective year are released at the end of February or at the beginning of March. The situation could be improved if the ISA Work Programmes would be adopted earlier then, as a result, would provide reassurance on the availability of the funds earlier in the year. However, considering that the Work Programmes are reviewed on a yearly basis, the evaluation accepts that end of February is the earliest possible time for the release of funds considering that the execution of the budget of the previous year has to be presented to ISA Committee members before submitting a new revised Work Programme for their approval.

Budget allocation and execution

According to Article 17 of the ISA legal decision, the financial envelope for the implementation of the ISA programme for the period from 1 January 2010 to 31 December 2015 is equal to EUR 164.1m, of which EUR 103.5m is for the period from 1 January 2010 until 31 December 2013. Accordingly, the allocated budget for the period from 2010 until 2013 was equal to EUR 103.25m with executed amount of EUR 103.47m (EUR 0.03m less than the amount of the financial envelope foreseen for this period).

The Annex of this report provides a tabular overview of the ISA actions including the annual allocated budget for 2010 - 2015 and the annual executed budget for 2010 - 2014. The evaluation also verifies the percentage of budget executed versus allocated for years 2010 - 2014. The total executed budget for Work Programme 2010 – 2014 is equal to EUR 128.2m, therefore only slightly exceeding the allocated budget of EUR 127.4m by less than 1 per cent (0.63%). The evaluation confirms that the allocated budget, set at less the amount of the financial envelope, does not mean that it exceeds the financial envelope allocated to the programme. Other than that, almost the same number of actions executed less and more of the budget than was allocated (22 vs. 20 actions respectively) with seven actions executing the exact amount of allocated budget. Even though the overall executed budget almost matches the allocated one, it is also useful to analyse the budget deviations 41 on a per action basis.

Figure 9 shows the budget deviations of the 49 actions included in the ISA Work Programme 2014. The evaluation team observed that most of the actions (i.e. 33 out of 49) have no significant deviations from their allocated budgets with deviations between -10% and +10%. In addition, 10 actions incurred only moderate deviations (i.e. with executed budgets varying from -10% to -30% or from 10% to 30% compared to the allocated ones). Overall, the budget allocation can thus be considered as adequate since it appears that the budget forecasts are in line with the actual costs.

Figure 9 Overview of ISA action’s budget deviation 42

It is worthwhile to note that six ISA actions had major budget deviations, with their expenditures being significantly lower than their allocated budgets, whereas there was no ISA action with expenditure significantly above its allocated budget.

The aforementioned six actions are the following (listed in order of deviation from highest to lowest):

Figure 10 Overview of the 6 most significant ISA action’s budget deviation 43  

The evaluation team contacted the responsible Project Officers (within or outside the ISA unit in order to investigate the reasons for these budgetary deficiencies. Often, these variations occurred due to administrative proceedings. For example, actions 1.03, 1.05 and 1.19 were not able to commit the budget because they were setting their objectives or have undergone major reorientation of the objective or scope that delayed the need for commitment of the budget; this has automatically translated into the levels of deviation mentioned above.

The budgetary deviation of the action 2.03 PKI Services is explained by the nature of the action itself. The key purpose of the action was to establish a contract service with a PKI provider for a purchase of PKI certificates by all the European Institutions. The allocated budget was, therefore, meant to fund the establishment of this Local Registry Infrastructure (LRA) by equipment of each DG. The total number of DGs that expressed interest was, however, low and therefore overall 98% (332k EUR) of the allocated budget was not executed.

Organizational issues prevented timely use of the budget, i.e. the allocated budget of action 2.11 (200k EUR) was not executed due to a major delay in the delivery of the initial study 44 that was conducted with the budget allocated in 2011. The allocated budget of 2012 was not committed as the delayed study of year 2011 was still being conducted and financed with the allocated budget of year 2011. Further delays were caused by the study as it only concluded at the end of July 2013. The budget initially allocated in 2013 was thus similarly transferred to the following year (2014).

The budget deviation of the action 3.01 Assessment of ICT implications of EU legislation is rather explained by a change in the scope of the action. The objective of the action, which is to ensure that ICT implications of EU legislation are taken into account in due time to allow timely, efficient and effective ICT support for the implementation thereof, has remained the same over the years. However, the means to ensure that ICT assessments are performed, when necessary, significantly changed in 2013.

As mentioned in the ISA Work Programme 2nd revision 2012, before 2013, the action aimed at (i) testing the existing method on real-life cases with Commission services being in the process of drafting legislation (2010 – 2012); and (ii) offering it to all Commission services from 2012. Although this method was indeed offered to numerous DGs in the period of 2010-2012, none of them came forward to test this method on their legislative drafts. The budget allocated in 2010 was thus partly executed (EUR 42k out of EUR 150k were committed to finalise the method developed under the IDABC programme) and the budget related to 2011 (EUR 400k) and 2012 (EUR 300k) not executed at all.

In 2012, the ISA unit realised the need to (i) drastically review the existing method, perceived as too complex; (ii) support the Commission services performing this assessment by making a pool of resources (ICT experts) available to them; and (iii) promote the method and the service within the European Commission. The budget allocated in 2013, 2014 and 2015 was fully executed to serve these activities: a new ICT assessment method, which was released in July 2015, based on the lessons learned from three real-life cases, is currently being tested on three other real-life cases.

In parallel, efforts were made to ensure that ICT assessments become part of the EU Policy cycle. The importance of assessing ICT impacts of EU legislation is now highlighted in the Better Regulation guidelines 45 and the process to assess the ICT impacts of EU legislation referenced in the Better Regulation Toolbox 46 .

In order to verify whether the ISA budget is efficiently distributed across each type of ISA activity, the evaluation also compares the initial estimates from the legislative proposal for a decision on ISA 47 , with the executed budget for the period 2010 - 2015.

As shown in Figure 11 , overall, the total executed budget and the estimated financial costs do not vary significantly with negative difference between the two of approximately 5%, which amounts to EUR 7.788m. Even though the overall picture does not clearly show a large difference, more striking differences are shown within the respective types of activity.

There are in fact several types of activity that record a significant difference between the total budget and estimated financial cost; for example the Establishment and improvement of reusable generic tools records a difference of around 100%, meaning that the budget executed for this type of activity was doubled compared to the estimated costs from the legislative proposal). Another striking case is the difference between the total budget and estimated financial cost related to the Assessment of ICT implications of EU legislation (solely action 3.01) where, out of an estimated financial cost of EUR 12.000m, EUR 11.058m less has been budgeted for the period of 2010 – 2015. It represents a variation of -92% between the estimated financial cost and the total budget allocated to this type of activity. In the first ISA Work Programme, the allocated budget foreseen for this type of activity was already reduced to EUR 2.250m. This allocated budget was not fully executed primarily because the first results of the pilot with some Commission's services were not satisfactory and in line with the expectations. The orientation and therefore the scope of the action 3.01 had to be redefined as mentioned earlier in this sub-section.

Figure 11 Total budget 48 and estimated financial cost 49 for the period 2010 – 2015

With regard to the adequacy of the allocation of financial resources, Member States’ representatives and EC officials were asked to give their opinion on whether the results of the ISA programme could have been achieved with less budget. Out of a total of 98 respondents, 50% (49) did not express any opinion on this matter, due primarily to a lack of overall view of all of the ISA actions. However, out of the 98 respondents who could have expressed an opinion, 33% (32) believed that the current results of the programme could have been achieved with less budget. 28 respondents provided us with further justifications to their opinions: the current results are not high enough compared to the budget spent (25% (7/28)).

The budget allocated to sTESTA is too high compared to the overall budget of the programme and should have been reduced ((14% (4/28)). The evaluation confirms that almost 70% of the budget of this activity is allocated to action 2.04 Data Communication Network Service (sTESTA). It is also important to mention that the budget allocated to this specific action is equal to 36.7% of the total allocated budget of the ISA programme.

65% believe nevertheless that the current results of the ISA programme could not have been achieved with less budget. This majority of respondents indirectly acknowledges that the allocation of budgetary resources was adequate as a result of this analysis. The evaluation team also verified the adequacy of the budgetary resources to cover the entire scope of the ISA programme. First of all, it is worth noting that out of 47 responses 21% of respondents (10) did not express an opinion on this matter and 19% (9) were indifferent in their response by choosing neither to agree nor disagree on the adequacy. This can be linked to the lack of knowledge or general overview on the programme.

Furthermore, 55% (26) of those who responded fully or somewhat agreed that the budgetary resources were adequate to cover the entire scope of the ISA programme. Furthermore, only two respondents somewhat or fully disagreed on the adequacy of the budget of the ISA programme; representing 4% of total responses.

One of the obstacles preventing the respondents from answering was their lack of knowledge or of a general overview. Therefore, the evaluation team directly questioned the Project Officers for the respective ISA actions about their opinion on the adequacy of the budget’s annual breakdown to achieve the desired results. 68% (13) of the Project Officers agreed upon the adequacy of annual budget splits, whilst only 16% (3) did not agree with this. The perception of the annual budget split was, therefore, rather positive; with minor comments from Project Officers who suggested that these budget splits have a tendency to reduce flexibility regarding their use or create a situation where budgets are being spent unequally between actions.

3.2.4Allocation of human resources

As regards the efficiency of resources mobilised within the ISA unit, the evaluation is based on two main aspects:

Distribution of human resources in function groups (administrators and assistants) to ISA unit activities and tasks in comparison to the distribution of resources given by the legislative proposal 50 and the different structures of the ISA unit available for 2010 - 2015 (i);

Human resources allocation to total number of ISA actions, but also to the actions under the responsibility of DIGIT.B6, i.e. the ISA unit (ii); and

Human resources turnover rates (iii).

I.The proposal for the ISA programme gives a tabular overview of the number of staff (Commission officials) to be assigned to the ISA unit by type of function group. Indeed, the posts covered by the Staff Regulations 51 are classified in an administrators' function group (hereinafter ‘AD’) and an assistants' function group (hereinafter ‘AST’).

The AD posts should cover the actual management of the programme 52 , whereas the AST posts provide support to transversal and the rolling of the Work Programme activities 53 . In addition, the AD posts also include the Head of Unit, the legal advisor to the programme and the secretariat of the Management Committee. In addition, the Seconded National Expert (SNE) posts support the actual management of the programme, complementing the AD posts mainly in areas related to coordination with the Member States, the follow-up of projects and studies and the organisation of expert meetings, workshops and conferences. The functions of a SNE can be regarded as equivalent to those of function AD.

Since April 2010, the ISA unit maintains and updates the structure of the ISA unit, i.e. allocation of the staff to the ISA unit activities. The structure of the ISA unit has been updated five times in years 2010 - 2012, four times in 2013, three times in 2014 and the staffing rates remained unchanged since June 2014 for year 2015.

Each structure of the ISA unit includes activities divided in six categories: horizontal activities, finance and contracts, strategy, planning & reporting, coordination and ISA programme activities. The management of the ISA actions will be treated separately in the next part of this subsection.

Figure 12 compares the actual distribution of resources in different function groups to ISA unit activities and tasks since January 2010 until mid-2015, with those estimated in the ISA legislative proposal.

   Figure 12 ISA unit activities and tasks by types of post

In the period 2010-2015, the distribution of resources in AD and AST groups is in line with the planned allocation from the legislative proposal. Therefore, it can be concluded that the allocated resources have the adequate skills to cover the actual management of the programme and other support activities. It shall be noted, however, that the number of the HR resources for each type of post has always been below the estimated number of the HR resources. Therefore, there has always been more workload per resource than what was foreseen at the time of the establishment of the programme.

II.Another important aspect to evaluate is the increase/decrease in the number of human resources in the ISA unit (ISA staff) in the period from 2010 until mid-2015. This metrics should be evaluated against the number of human resources planned in the legislative proposal and the estimation of the ISA unit workload. In this respect, we consider the number of actions in the ISA Work Programme as a proxy 54 indicator to measure the ISA unit workload. From 2010 to 2015, the number of ISA actions increased by 41% (11 additional actions 55 , out of which seven have to be managed by the ISA unit); it is thus assumed that the workload of the unit has increased. In case when the new actions are managed by other Commission service (4 out of 11 new actions), the increase of the workload is then related to the support activities deriving from the actions (e.g. managements of contracts, payment of contractors, call for tenders, etc.); and to the fact that at least one resource of the ISA unit is allocated to the action follow up and supervision.

Figure 13 Number of HR allocated in 2010 - 2015 56

As shown in Figure 13 , the number of human resources is less than the one planned in the legislative proposal for each of the years between 2010 and 2015. The increase in the number of ISA actions shall be reflected upon the number of ISA human resources; however it is not being done so on a regular basis. Overall, the number of actions has been steadily increasing from 27 in 2010 to 38 in 2015. However, the number of ISA human resources has been growing only very slightly, creating thus an increasing pressure on the existing human resources within the Commission.

In total, there were 18 ISA actions under the responsibility of the ISA unit and 20 actions managed by other Commission services in 2015. After a consideration of the increase in the number of ISA actions over time in contrast to the less significant and unmatched increase in the ISA unit’s human resources, the evaluation team remarks that current human resources of ISA unit are managing relatively higher number of actions over time than was envisaged at the time of the inception of the programme. The number of ISA actions under the responsibility of ISA unit has increased by 64 % (from 11 to 18) between 2010 and 2015, whilst the number of ISA human resources has only increased from 15 to 16 57 (representing approximately increase of 7%) during the same time frame.

III.Having analysed the efficiency in terms of distribution of resources in AD and AST function groups and the allocation of resources to ISA activities and tasks, the evaluation team analysed the turnover of the ISA staff from the beginning of the programme. The retention of knowledgeable resources and a low rate of undesired replacements of resources are elements of prime importance to maintain a certain level of quality and keep the knowledge acquired during the lifetime of the programme. Leaving aside sectors in which continuously replacing resources is a need (e.g. highly innovative industries), the above statement is applicable to most of the sectors and therefore to the ISA unit as well.

In order to evaluate this criterion, the ISA unit turnover 58 in the period 2010-15 is evaluated to verify the qualitative aspect of its human resources 59 . Table 13 shows the turnover of the ISA unit resources from January 2010 until mid-2015. It displays the number of resources at the beginning and end of every year (except 2015) of the unit. The highest turnover (43%) was registered in 2013 being significantly higher than any other registered turnover. From year 2013, the turnover rates have drastically decreased, reaching 0% in 2015 (as of mid-2015). Overall, the risk related to high turnover rates from the beginning of the programme has been reduced, even if the knowledge and experience gained from the previous programmes was only partially kept. However, the programme draws on the competences of a wide variety of Commission officials and Seconded National Experts, particularly dealing with technical, semantic and organisational interoperability, but less with legal interoperability issues. The aspect of legal interoperability has indeed not been really explored. In fact, the evaluation highlights that the focus was not on this aspect, primarily because of the investment that it represents compared to the aforementioned lack of resources.

Table 13 ISA unit resources - turn over in the period January 2010 to mid-2015

Perceived efficiency of the allocation of HR

Overall, 47% (22) of respondents fully or somewhat agreed that ‘the allocation of human resources from the European Commission is adequate to ensure a smooth delivery of the ISA actions’. However, the EC officials and Member States’ representatives vary in their opinion on this statement even though it is important to acknowledge the difference in the absolute numbers of EC officials (6) and Member States’ representatives (41) who responded to this question. Whilst 52% (21) of the Member States’ representatives fully or somewhat agreed with the statement, only 17% (1) of the EC officials agreed that the allocation of human resources from the European Commission is adequate to ensure a smooth delivery of the ISA actions. Furthermore, 22% (2) of the EC officials fully or somewhat disagreed with the statement, whilst only 13% (4) of the Member States fall into this category.

The results of open-ended questions revealed the reasons for the perceived higher level of adequacy of the budgetary allocation than the human resources allocation. Most of all, EC officials and Member States’ representatives shared the view that the ISA unit seems have less human resources than it requires in order to ensure the smooth delivery of the ISA actions. Secondly, the human resources allocation tends to reflect the budgetary allocation of the ISA actions more than their human resources’ need. Thirdly, the EU practice of staff rotation seemed to have affected the ISA unit’s operations. Some responses also suggested an inadequate amount of human resources within Member States’ administrations to deal with interoperability matters.

3.2.5Overall performance of the programme

Time efficiency at action level

In the framework of the monitoring of the ISA programme, efficiency is measured at action and programme level. In this regard, two techniques are being used - the Earned Value Management (EVM) and Earned Schedule (ES) techniques. EVM is a systematic approach to the integration and measurement of cost, schedule and technical (scope) accomplishments of an action and its related pieces of work (i.e. milestones). ES is an extension of EVM, given that it completes the EVM approach by going deeper into performance in a time perspective 60 .

In order to verify whether each ISA action has been progressing to schedule within the ISA programme, the evaluation has calculated how their progress compares with their schedule between 2011 and 2015 61 , as detailed in the Annex. Figure 14 below provides the status of all the ISA actions, in terms of percentage of delay/ advance based on their status in May 2015. It should be noted that the delay/advance value is calculated based on the schedule variance of the action, i.e. the difference between the earned value of the work performed and the planned value of the work scheduled.

Figure 14 ISA actions delayed and ahead of schedule

Actions not included in Figure 14 , are actions for which data could not be reported at the concerned period, for the following reasons:

Data were not provided to the contractor in charge of the monitoring of the programme;

No contract was running in May 2015;

Funding of the action had concluded by May 2015.

As further detailed in the Annex, the majority of actions have been progressing to schedule over the years. Even though there have been some slight delays, overall they have occurred only occasionally and the actions were back on track the following year. The issues related to these delays always managed to be resolved and the risks were mitigated.

However, as depicted in Figure 14 , at the end of May 2015, the following two actions had a significant delay (greater than 10% of their total duration):

Action 1.05 – STORK Sustainability: The original eIDAS version of the PEPS, which aimed to be deployed in September 2015, will only be compatible with the Member States that are eIDAS compatible by this date. As no Member States aims to be eIDAS compatible by this time, DG CNECT decided to postpone the deployment (delaying the progress of the action). At the time of this report, the development team is awaiting specifications from Germany and Austria.

Action 2.04 – Data communication network service (sTESTA / TESTA NG): The time needed to perform some tasks on the new TESTA NG network was underestimated (e.g. encryption), delaying other dependent tasks, such as the testing, but also the roll-out and actual migration of the current operational sTESTA network to TESTA NG. As a result, the current sTESTA network needs to be kept in operation for longer than expected.

In comparison to the total number of actions in the ISA programme (51); having only two significantly delayed actions may appear to be insignificant. One should however take into account that action 2.04 represents almost one third of the overall budget of the ISA programme (59,043 million EUR). Although the sTESTA contract ended in Q3 2013, a migration period of more than 18 months starting in 2013 was foreseen due to the complexity of the provided services and the multiple communities that are served. During the migration period, the continuity of the current sTESTA services had to be guaranteed. Therefore, critical service elements of sTESTA and its successor, TESTA-ng, have to coexist. The financing that is required for this parallel operation was already budgeted in previous work programmes. At the time of writing the final report, a new project schedule is being finalised to address the issues encountered by this action. Greater clarity on this will be available from autumn 2015.

Time efficiency at the programme level

Overall, from a time perspective, the progress of the ISA actions can be considered as efficient as of the end of May 2015, however it is beneficial for our final judgement to also focus on the time efficiency of the programme overall.

Before analysing the programme time efficiency, mostly based on Earned Schedule metrics, the two following points should be noted:

Given the non-exhaustive number of actions in the scope of the programme aggregation (42 actions out of 51), one should keep in mind that the results displayed in this part of the final evaluation are only indicative and do not represent the progress of the entire ISA programme.

The information included below is based on the last Monitoring & Evaluation semester report (January 2015 – June 2015). In this regard, the latest data available at programme level is as of the end of June 2015.

The evaluation technique aims to compare the actual value produced by the programme (Earned Value (EV)) and the value initially forecasted until the end of June 2015 (Planned Value (PV)). In this regards, when comparing EV (EUR 76,663k) with PV (EUR 79,725k) at the end of June 2015, and taking into account a Budget at Completion (i.e. sum of the budgets of all the ISA actions monitored using EVM) which is around EUR 88,590k; the delta (EUR -3,062k) can be considered as minor from a value perspective. In order to delve deeper into the evaluation of the ISA programme’s performance, the impact of this delta was measured in terms of time.

Figure 15 introduces the Schedule Variance (SV) indicator, which aims to measure whether the work accomplished is ahead or behind schedule (in number of days).

Figure 15 ISA programme efficiency: Schedule Variance

The EUR -3,062k delta represents, in terms of time, a delay of 30 working days. Put into perspective with the length of the ISA programme ( Figure 16 ), this delay represents less than 5% of the programme’s duration. Therefore, it can be considered as minor (see time scale in the preceding part of this subsection). In addition to that, it can be seen in Figure 16 that the programme’s progress has always been on track. Overall, from a time perspective, the progress of the ISA programme can thus be considered as efficient, as of the end of June 2015.

Figure 16 ISA programme efficiency: % of delay

Perceived performance of the ISA programme

First of all, the evaluation team investigated whether the interview and survey respondents believed that the programme was on track. Almost 69% (67) of respondents agreed that the ISA Programme is on track, whilst only 4% (4) of respondents did not agree that this was the case. A large part of respondents (27% (26)) did not have a clear view on this issue. It is clear that, even though there are several ways in which to improve the overall performance of the programme, in general, it is perceived to be on track.

Further analysis revealed that 90% (42) of respondents fully or somewhat agreed that the ISA programme has been delivered within its original scope (stated by all EC officials (6) and by 88% (36) of the Member States’ Representatives).

Just 23% (23) of respondents stated that the current results of the ISA programme could have been achieved faster whereas 34% (33) of respondents did not agree with this viewpoint and    a    large portion (43% (42)) of respondents did not have a clear opinion on the pace of the process. Furthermore, the respondents have a tendency to explain the slower pace by the overall structural reality of the EU. European public administrations operate in a deliberative fashion and need to respect procedures and protocols of coordination in order to work together to produce results. This reality often constrains the pace of developments of programmes. At EU level, this reality is particularly acute. Coordination of 28 different Member States and the different entities of the EC were widely recognised by interview and survey respondents as major challenges that naturally slow the pace of the programme and, consequentially, the rate at which it can produce. Member States’ representatives voiced their own national experience of where institutional reality affects the pace of policy programmes. Furthermore, this was compounded by changes of personnel, institutional re-organisation programmes and the arrival of new political administrations/ governments.

In order to further investigate the timeliness of the programme, several Project Officers had a chance to share their experiences of the delays with actions under their purview. While a few of them did not experience any delays, most of them did. The reasons for the occurrence of delays were often related to the nature of the solution/action. Often, the key reason for delay was rooted in the inability to communicate with all of the relevant stakeholders (such as other DGs, Member States or Commission’s contractors) or the existence of administrative barriers that delay the process (e.g. acceptance process of deliverables and other specificities of EC’s internal processes). Overall weaknesses in the coordination between key stakeholders was also frequently mentioned reason for delays.

17 Project Officers also provided us with a range of reasons to explain any of the deficiencies in the overall performance of the action under their purview. Lack of awareness, communication and engagement of the interested parties (often the Member States) occurred most often in the responses of the Project Officers (6). Lack of focus of the programme resulting from too many actions is another reason for the deficiencies in the performance mentioned by two respondents. Budgetary resources were once again not the key issue, and the focus was much more on the lack of skills, competences or expertise at both EC and Member State level.

Having analysed perceptions regarding the ISA programme’s performance, the evaluation verified suggestions from the respondents for improvement of the ISA programme, Overall, 62% (61) of respondents ((Member States’ representatives and EC officials)) confirmed that there are measures to improve the overall performance of the programme.

Figure 17 provides a summary of the most frequently desired suggestions for improvement. The most frequently suggested measure was to increase the promotion of the ISA programme (10 occurrences), followed closely by the need for greater involvement of the stakeholders (9 occurrences) such as Member States as well as citizens, businesses and other DGs at various stages of the ISA programme. Third most frequently recommended measure was a need for revision of the relevance of the ISA solution/actions, including stricter criteria for the inclusion of new actions (7 occurrences). Several further measures were suggested, i.e. to improve/increase communication with the stakeholders (4 occurrences), to better align the programme with other EU initiatives (4 occurrences) or to allocate more human and financial resources to the ISA programme (4 occurrences).

Figure 17 Most frequently mentioned measures to improve the ISA programme

3.2.6Conclusions

The following box summarises the main findings related to EQ2: ‘How economically were the various inputs converted into outputs? and EQ3: ‘Which aspects of the programme were the most efficient or inefficient, especially in terms of resources mobilised?’.

Conclusion N°4 Based on the evidence provided by action 5.01 Monitoring & Evaluation, the evaluation judges the progress of the ISA programme as efficient. With an overall delay lower than 5% of the programme’s total duration (30 working days), the programme has been considered as on track during the whole 2010-2015 period. The progress of each individual ISA action can also be considered as efficient as the large majority of the actions have a delay 62  of less than 5%. In addition to this evidence, 90% of surveyed Member States’ representatives and EC officials affirmed their view that the programme has delivered on time and also within its original scope. A majority of these stakeholders (68% of Member States’ representatives and 58% of EC officials) felt fully or somewhat involved in the annual revisions of the ISA programme. Given that the programme should have followed specific implementation rules to ensure its smooth implementation, the evaluation judges this aspect as satisfactory as well. Once again, the perception of respondents of the ISA programme’s adherence to its implementation rules is aligned with the data that confirms the overall efficiency of the programme.

Conclusion N°5 The following aspects of the ISA programme can be regarded as efficient by the evaluation: Selection procedure of proposals for new actions: It is considered appropriate as the ISA actions selected for funding comply with the overall objectives of the programme to support the interaction between administrations with the exception of EUSurvey. Funds release process: It is judged as efficient given that the first release of funds occurs, on average, five working days after the adoption of the Work Programme. Allocation of budgetary resources: Overall, budgetary resources were adequate to cover the entire scope of the ISA programme. On the other hand, the ISA programme can be regarded as less efficient in the following aspects: Precision and clarity of the documented process about the selection of proposals for new actions: The evaluation found that the stakeholder involvement in the consultations with ISA was considered satisfactory and there was generally a correlation between the level of involvement of Member States in the selection of proposals and their levels of satisfaction with this process. However improving the precision and clarity of the documented process about the selection of proposals, as indicated by the proposal for the ISA² programme, would improve its efficiency and transparency 63 . Member States also recognised that internal issues within national and local administrations can hinder their capacity to be involved in and fulfil their obligations with regard to the process. Allocation of human resources: The allocation of human resources in the European Commission has always been lower than what was foreseen in the ISA legislative proposal47. However, the evaluation recognises that the ISA programme is subject to the general contraction of staff levels as part of the EC’s 5% staff reduction target by 2018 64 . A wide variety of Commission officials and Seconded National Experts are dealing, in particular, with technical, semantic and organisational interoperability, but less with legal interoperability issues. The programme has, indeed, not yet focused much on the aspect of legal interoperability.

3.3Effectiveness

The evaluation of the effectiveness of the ISA programme is driven by the following evaluation questions:

EQ4: To what extent did the ISA programme's results and impacts achieve its objectives?

EQ5: Are there aspects that were more or less effective than others, and, if so, what lessons can be drawn from this?

To evaluate the extent to which the ISA programme’s results and impacts achieve its general and specific objectives, the evaluation looks at the expected results for each programme’s activity based on both ISA legal decision and the ex-ante evaluation of the IDABC follow-on programme. The latter provides clear expected results for each of the specific objectives of the programme, which are its activities to be supported and promoted, as stipulated in Article 3 of the legal decision.

In addition to the establishment of new solutions, the scope of programme’s activities also covers existing solutions established under the IDA and IDABC programmes, which came to an end on 31 December 2009. For this reason, the evaluation of the programme’s activities provides an overview of these carried forward actions for each programme’s activity. A tabular overview of the actions carried-forward from IDA and IDABC including details on the budget for each action within the IDABC and IDA programmes, the corresponding activity for each ISA action and the IDABC action type as well, can be found in the Annex.

The remainder of this section is divided into different subsections in accordance with the different judgement criteria allowing the evaluation to answer EQ4 and EQ5. 

3.3.1Facilitation of effective interaction between European public administrations

Solutions established or operated under the ISA programme should achieve the important results of facilitating interaction between European public administrations, which is the main objective of the ISA legal decision, as stipulated in its Article 1.

Before looking at the results delivered by the ISA programme in practice, the evaluation team verified the level of awareness of the ISA programme’s results from the interviewees and respondents to the surveys. They were requested to express their opinion on what they considered as the main results of the ISA programme. This is along the lines of the interim evaluation of the programme that confirmed a low level of awareness and visibility of ISA programme’s results.

Many Member States’ representatives and EC officials named the results of several specific actions as the main results of the programme. In that regard, Figure 18 presents the most commonly cited actions in descending order based on the number of times that they were mentioned as a main result by the respondents. The evaluation confirms that the work on semantic interoperability, Joinup, and the European Interoperability Architecture are ‘’perceived’’ as the top three main results of the ISA programme.

Figure 18 Main ISA results perceived 65

In addition, 11 interview and survey respondents said that the existence of the ISA programme itself, as a means to better raise awareness about the importance of interoperability, was the main result of the programme. Also, the European Interoperability Framework (EIF) was cited ten times by the respondents. Although developed under the IDABC programme, ISA Action 4.2.3 National Interoperability Framework Observatory (NIFO) actively supports the implementation of the EIF at Member State level, through the transposition of the EIF into national interoperability frameworks. It can be deduced, therefore, that the support provided by the programme to Member States on the EIF implementation is a major result of the ISA programme.

The level of alignment of the results delivered by the programme with the expectations from respondents was also assessed. In this regard, out of 76 answers from respondents expressing their opinion, these results are aligned with their expectations for a clear majority, namely 83% of them, and not aligned for the remaining 17%.

Based on the identified results, respondents were asked to assess how well the ISA programme has facilitated efficient and effective cross-border and cross-sectoral interaction between European public administrations. Out of 77 respondents expressing an opinion on this matter, a majority judged that the programme has well (66%) or even very well (5%) facilitated an efficient and effective cross-border and cross-sectoral interaction between European public administrations. Only 3% expressed the opinion that the programme has quite poorly facilitated this interaction whereas 26% believes that it has neither well nor poorly facilitated it.

Even if less frequently mentioned by respondents, the programme has delivered other significant results facilitating effective collaboration between European public administrations. In that regard, the evaluation went beyond the pure perceptions from end-users (which are important for measuring the levels of awareness on interoperability and on the programme itself) and identified the key results delivered by the programme in terms of ISA solutions established, operated or further improved since IDA, IDABC in the period 2010-2015, as shown in Table 14 .

Table 14 Operational ISA Solutions

The majority of solutions included in Table 14 have been made available, in 2015, in a dedicated section of the official ISA website called ‘Our ISA solutions for you’. In fact, this section aims to provide details on the purpose of each operational ISA solution. In addition to key facts and figures about each of these solutions, it also gives direct access to them (e.g. link to download tools or to access services and documentation for common frameworks) as well as to related promotional material. The creation of this section is in line with one of the recommendation from the interim evaluation of the programme stipulating that ‘the ISA programme should reinforce promotion and communication activities regarding the ISA solutions that have produced concrete results’.

Furthermore, respondents from Member States as well as European Commission officials were requested to provide their views on whether all the ISA solutions facilitate interoperability between European public administrations.

In that regard, no clear consensus was reached amongst the respondents. In fact, out of a total of 66 answers, 41% of them considered that all the ISA solutions facilitate interoperability between European public administrations whereas 30% of them did not believe that this is the case for all the ISA solutions. 29% did not express any opinion on this matter, primarily due to a lack of overview of the full range of solutions.

20 respondents who considered that not all the ISA solutions facilitate interoperability between public administrations were asked to justify their answers. In that respect, specific solutions were mentioned and one of them was named more than all the others. In fact, 20% specifically mentioned EUSurvey as an ISA solution that does not facilitate interaction between European public administrations 70 . The evaluation confirms that the available market solutions do not cover all EU official languages, but they are in the process of leveraging more and more existing language translation software and services 71 . Several respondents also commented that a greater level of adoption of the different solutions will increase the facilitation of interoperability between European public administrations.

Based on the key results delivered by the programme in terms of ISA solutions, the next four subsections will focus on the contribution of the programme to the achievement of each of its specific objectives (cf. subsection 1.2), i.e. to support and promote:

the establishment and improvement of common frameworks in support of cross-border and cross-sectoral interoperability;

the operation and improvement of existing common services;

the establishment, industrialisation, operation and improvement of new common services, including the interoperability of public key infrastructures (PKI);

the improvement of existing reusable generic tools;

the establishment, provision and improvement of new reusable generic tools ; and

the assessment of the ICT implications of proposed or adopted Community (EU) legislation and planning for the introduction of ICT systems to support the implementation of such legislation.

3.3.2Establishment and improvement of common frameworks

‘Common frameworks’ means strategies, specifications, methodologies, guidelines and similar approaches and documents, which should have been established and improved by the programme by means of studies in accordance with Article 3 of the legal decision.

The ex-ante evaluation set specific expected results for this activity of the programme.    First of all, the establishment and improvement of common frameworks should create a common context in which Member States and EU institutions can discuss cross-border and cross-sectoral interoperability. Second, this activity should lead to an    refreshed set of concrete frameworks and methodologies, when needed.

This subsection verifies the extent to which the ISA programme has contributed to the improvement of existing common frameworks as well as to the establishment of new ones keeping, as a reference, the expected aforementioned results. Before looking at the results from surveys and interviews, the evaluation highlights that four common frameworks, carried-forward from the IDABC programme, were improved by ISA, whereas 12 new common frameworks were established as shown in Table 15 below.

Table 15 Common frameworks within the ISA Programme

In that regard, Figure 19 presents the answers provided by respondents who were requested to provide their level of agreement on the extent to which the ISA programme contributed to the improvement of existing common frameworks as well as to the establishment of new ones.

Figure 19 ISA contribution to common frameworks

As depicted in Figure 19 , a large majority of respondents fully and somewhat agreed that the ISA programme has contributed to the improvement of existing common frameworks as well as to the establishment of new common frameworks. The programme is perceived as contributing more to the establishment of new common frameworks than to the improvement of existing ones.

This perception is confirmed by desk research. In fact, based on findings presented in Table 14 , the ISA programme has contributed, on the one hand, to the establishment of new common frameworks in support of cross-sector and cross-border interoperability by delivering the following frameworks:

Interoperability Maturity Model (IMM), published in 2013, aims at providing public administrations insight into two key aspects of their interoperability performance: the current interoperability maturity level of a Public Service (i) and improvement priorities to reach the next level of interoperability maturity (ii). The IMM measures how well a public administration interacts with external entities in order to organise the efficient provisioning of its public services to other public administrations, businesses and or citizens.

Assessment of Trans-European networks supporting EU policies, ongoing since 2013, is an expanding cartography of the Trans-European Systems (TESs), which currently includes more than 80 solutions to support the European Commission in implementing an overall strategy to rationalise and streamline the IT systems it develops, maintains and operates with the goal of proposing rationalisation recommendations. 

Sharing and Reuse strategy aims to develop a holistic approach to sharing and reuse across border and sectors with a view to helping public administrations all over Europe to share and reuse solutions related to public services delivery in an efficient and effective way. In 2013, the programme identified a number of quick wins to be implemented having significant impact on sharing and reuse of assets. On the other hand, a framework is in the process of being finalised in 2015 together with the governance, the processes and the instruments to optimise the potential of sharing and reuse activities and increase the savings they can bring to public administrations.    

EULF Strategic Vision outlines a vision and framework for 'location-enabled government', based on applying good practice in a number of 'focus areas': policy and strategy alignment, e-government integration, standardisation and interoperability, return on investment and effective governance and partnerships. It identifies the objectives, transition strategy and high-level actions needed in each of these focus areas.

On the other hand, the programme has contributed to the improvement of existing common frameworks: Semantic standards 72 (action 1.01 Semantic Interoperability) Common Assessment Method for Standards and Specifications (action 2.2 CAMSS), but in particular the European Interoperability Reference Architecture (EIRA) in the framework of action 2.01 EIA.

Semantic standards: Core vocabularies which are simplified, re-usable and extensible data models that capture the fundamental characteristics of an entity in a context-neutral fashion. The core vocabularies developed under the ISA programme, and endorsed by Member States in May 2012, promote the alignment of concepts relevant for public administrations such as a person (Core Person), a public service (Core Public Service), a location (Core Location), a registered business (Registered Organisation);

Semantic standards: DCAT Application Profile for data portals in Europe which provides a common specification for describing public sector datasets in Europe to enable the exchange of descriptions of datasets among data portals. It now makes the following possible:

oData catalogues can describe their dataset collections using a standardised description, while keeping their own system for documenting and storing them.

oContent aggregators, such as the pan-European data portal, can aggregate such descriptions into a single point of access.

oData consumers can find datasets more easily from a single point of access.

Semantic standards: Asset Description Metadata Schema (ADMS) which is a simple specification used to describe interoperability solutions, making it possible for everyone to search and discover them. The ADMS was endorsed by Member States in May 2012 and primarily allows:

oSolution providers to describe their interoperability solutions using the standardised descriptive metadata terms of ADMS, while keeping their own system for documenting and storing them;

oContent aggregators, such as Joinup, to aggregate such descriptions into a single point of access;

oICT developers to more easily explore, find, identify, select and obtain interoperability solutions from a single point of access.

With regard to CAMSS, the Commission has developed a method to assess standards and specifications in the field of ICT based on the best practices of the Member States and aligned with the Regulation on European Standardisation 73 . A first unofficial version of the method was developed by the Commission and Member States under the IDABC Programme. Through discussions with Member States, standardisation bodies and other stakeholders, the Commission identified a need for revising the first version of CAMSS and developing a widely adoptable official version. In May 2012, a first revised version of CAMSS was finalised and validated by the Commission and Member States. At the time of writing this report, the CAMSS consisted of a documented reference assessment process, a set of quality requirements (criteria), and an assessment library and tools that were updated to v1.0 in March 2015 and made available on Joinup.

In 2013, the action 2.01 EIA released the first version of the European Interoperability Reference Architecture (EIRA) describing a common reference architecture for delivering interoperable digital public services across borders and sectors. The EIRA 74 is an architecture content metamodel defining the most salient architectural building blocks (ABBs) needed to build interoperable e-Government systems. The EIRA provides a common terminology that can be used by people working for public administrations in various architecture and system development tasks. The first version of EIRA was delivered along the European Union Cartography (EUCart) which was a result of mapping existing Trans-European Solutions (TES) onto EIRA. This mapping exercise was implemented in a proof-of-concept software application, the Cartography Tool (CarTool). In June 2014, the ISA Coordination Group endorsed the current versions of EIRA and CarTool stating that they are mature enough to go to public consultation and to be used in pilot projects. In the period September 2014 – May 2015, a number of pilots were conducted using the EIRA with public administrations in Denmark, Estonia, and the Netherlands, and with European Commission DGs DIGIT, CNECT (e-SENS project), and MARE. At the time of writing this report, the latest version of EIRA is under public consultation. Overall, it is worth highlighting that even if action 2.01 EIA was slow to start and get off the ground, significant progress has been made since 2014.

However, the evaluation noticed that the common frameworks delivered by the programme did not focus much on legal and organisational interoperability, even if some progress has been made through action 4.2.05 - Sharing & reuse strategy, as well as in action 2.14 - Assessment of TES. action 4.2.05 - Sharing & reuse strategy produced several deliverables on legal and organisational interoperability including sharing and re-use guidelines and templates for implementing agreements (including service conditions) and governance models for common IT solutions. In addition, common "standard" clauses for contracts, which public administrations could use when procuring services, were made available on Joinup in February 2013.

As part of action 2.14, the current state of play on organisational interoperability is in the process of being completed. The outcome of this activity could also be a valid input to the extension and update of the EIF in the context of the NIFO Action.

3.3.3Establishment, industrialisation, operation and improvement of existing and new common services

‘Common services’ means operational applications and infrastructures of a generic nature which meet common user requirements across policy areas, which should have been established, operated and improved by the programme by means of projects in accordance to Article 3 of the legal decision.

As with the frameworks, the ex-ante evaluation set specific expected results for this activity of the programme as well.    First of all, the programme should improve services and infrastructures and ensure continuity and professionalism in the delivery of the common services portfolio. Second, the programme should ensure an increased availability of services that meet the needs of sectors and Member States.

This subsection verifies the extent to which the ISA programme has contributed to the establishment, industrialisation, operation and improvement of existing and new common services, keeping as a reference the expected results mentioned above.

Before looking at the results from the interview and online surveys, the evaluation highlights that four common services, carried-forward from the IDABC programme, were improved by ISA, whereas six new common services were established as shown in the Table 16 .

Table 16 Common Services within the ISA Programme

In that regard, Figure 20 presents the answers provided by respondents who were requested to provide their level of agreement on the extent to which the ISA programme contributed to the operation and improvement of existing common services as well as to the establishment and industrialisation of new ones.

Figure 20 ISA contribution to common services

As depicted in Figure 20 , a large majority of respondents fully and somewhat agrees that the ISA programme has contributed to the establishment and industrialisation of new common services as well as to the operation and improvement of existing ones. The programme is perceived as contributing slightly more to the establishment and industrialisation of new common services than to the operation and improvement of existing ones.

Based on desk research as well as on the findings presented in Table 14 , the ISA programme has contributed, on one hand, to the establishment and industrialisation of new common services in support of cross-sector and cross-border interoperability by delivering the following three ISA solutions:

Joinup 75 which is a collaborative platform, launched on 9 December 2011, offering several services that aim to help e-Government professionals share their experience with each other. It aims to support them to find, choose, re-use, develop and implement interoperability solutions. It was established as a merger of the Open Source Observatory and Repository (OSOR.eu) and the Semantic Interoperability Centre Europe (SEMIC.eu) platforms. In December 2014, the ePractice platform was also integrated into Joinup which is now a single-access point to more than 5000 interoperability solutions for public administrations, included in the collections of more than 40 standardisation bodies, public administrations and open source software repositories.

MT@EC which aims to provide a new service relying on a system based on SMT (Statistical Machine Translation). SMT provides an improved machine translation service both in terms of quality of output and number of supported languages. A total of 552 language pairs covering all of the EU official languages are currently provided. This service generated 7195735 of machine translated pages in 2014.

Federated Managed Authentication Services for ECAS which aims to allow public officials to log into EC applications and to be granted access based on their role or position in a national administration.

On the other hand, it has contributed to the operation and improvement of existing common services by delivering the following three ISA solutions:

sTESTA which is a network allowing exchange of data between European and national administrations by guaranteeing security and availability of these data. sTESTA replaced the first TESTA (Trans European Services for Telematics between Administrations) network that was established under the IDA programme. A migration from sTESTA to TESTA NG (New Generation) is currently on-going under the ISA programme.

CIRCABC which is an evolution of the previous CIRCA tool that was firstly established under the IDA programme. CIRCABC (Communication and Information Resource Centre for Administrations, Businesses and Citizens) provides a web-based application that is used to create collaborative workspaces. It is divided into categories and interest groups, allowing people to manage content, users and communication features. In addition to further development and maintenance of CIRCABC, the migration from CIRCA to CIRCABC has been conducted within the ISA programme. In fact, a migration tool has been developed allowing for the smooth migration of the complete information including users, rights, content and metadata as well as full history from CIRCA to CIRCABC. In 2014, more than 4.5 million documents were uploaded and downloaded.

EUSurvey which is a web-based application available in 24 languages that aims to allow the creation and conduct of online surveys. EUSurvey is a new version of the IPM (Interactive Policy Making) tool that was established under the previous IDABC programme. IPM was replaced by EUSurvey in October 2013. This first release included a newly designed user interface, offering additional features and different enhancements improving the usability and stability of the software compared to IPM. In 2014, more than 2800 surveys were created with EUSurvey.

3.3.4Establishment, provision and improvement of existing and new reusable generic tools

‘Reusable generic tools’ means reference platforms, shared and collaborative platforms, common components and similar building blocks that meet common user requirements across policy areas, which should have been established, provided and improved by the programme by means of projects in accordance with Article 3 of the legal decision.

The expected results set by the ex-ante evaluation of the ISA programme are also taken into account for this activity. As for the common services, an increased availability of tools that meet the needs of sectors and Member States is expected by the programme, as well as availability of new tools in time.

This subsection verifies the extent to which the ISA programme has contributed to the establishment, provision and improvement of existing and new reusable generic tools keeping as a reference the expected results mentioned above. Before looking at the results from the interview and online surveys, the evaluation highlights that six reusable generic tools, carried-forward from the IDABC programme, were improved by ISA, whereas 13 new reusable generic tools were established as shown in Table 17 .

Table 17 Reusable generic tools within the ISA Programme

In that regard, Figure 21 presents the answers provided by respondents who were requested to provide their level of agreement on the extent to which the ISA programme contributed to the improvement of existing reusable generic tools as well as to the establishment and provision of new ones.

Figure 21 ISA contribution to reusable generic tools

As depicted in Figure 21 , a large majority of respondents fully and somewhat agrees that the ISA programme has contributed to the establishment and provision of new reusable generic tools as well as to the improvement of existing ones. The programme is perceived as contributing even more to the establishment and provision of reusable generic tools rather than to the improvement of existing ones.

Based on desk research as well as on the findings presented in Table 14 , the ISA programme has contributed, on one hand, to the establishment and provision of new reusable generic tools in support of cross-sector and cross-border interoperability by delivering the following three ISA solutions:

Open e-TrustEx which is an open-source tool that facilitates the secure exchange of structured and unstructured documents between Public Authorities at European, national and local level via standardised interfaces. In 2014, e-TrustEx transmitted more than 510000 electronic messages (around 45000 messages per month on average);

Online Collection Software (OCS) for European Citizens' Initiatives (ECIs) which is an open-source software providing all the basic functionalities to collect statements of support online through forms compliant with the ECI Regulation, securely store signatories' data and export the data to the competent national authorities (in accordance with Article 6(2) of the ECI Regulation). As an evidence of its success, even though ECI organisers have the choice to use the software developed by the European Commission or any other software available on the market (as long as it complies with the ECI Regulation and related Commission Implementing Regulation N°1179/2011), it should be noted that, at the time of writing this report, the ECI Online Collection Software developed by the European Commission has been used or was planned to be used by all ECI organisers so far;

Re3gistry which is a tool to manage and share 'reference codes' that provides a central access point allowing labels and descriptions for reference codes to be easily looked up by people or retrieved by machines.

On the other hand, it has contributed to the operation and improvement of reusable generic tools by delivering the following five ISA solutions:

ECAS STORK Integration aims to enable access to European Union information systems using user's national eID solution with minimal impact on the information systems themselves. In the framework of the ISA programme a pilot interconnection between ECAS and STORK was undertaken to demonstrate that ECAS was able to consume identities provided by STORK. This proof-of-concept was then transformed into an official STORK pilot (i.e. CIRCABC was used for demonstration purposes). Following this pilot, the next ISA activities were focused on the generalisation of the service to all information systems relying on ECAS for authenticating users.

Open e-PRIOR is an open-source e-Procurement solution. The ‘e-PRIOR’ project was initially funded by the IDABC to show that emerging standards to enable interoperability in a cross-border environment could be used in a real-life pilot of Business to Government eInvoicing. This first pilot paved the way for further development of e-Prior under the umbrella of the ISA programme. Open e-Prior now covers not only e-Invoicing but the whole post-awarding cycle of e-procurement: e-Catalogue, e-Ordering, e-Fulfilment and e-Invoicing. In addition, since March 2014, e-PRIOR has also provided an e-Submission module as part of the pre-award cycle, with tools allowing economic operators to prepare and submit tenders for a published call for tender, and contracting authorities to receive and handle such electronic tenders.

SD-DSS and TLManager are tools for e-Signature creation and validation that Member States could use at national level. The SD-DSS tool aims to facilitate Member States to render their public e-services more efficient and to ensure interoperability for one of the key-enablers for secure electronic services, i.e. e-signatures. DSS (Digital Signature Services) provides all the functionalities for the automated creation and validation of e-Signatures, checking them against the European Member States’ so-called ‘Trusted Lists’ (each Member State keeps a public list of supervised/accredited service providers issuing qualified certificates to the public).In addition, SD-DSS can enable the electronic signing of documents in different portals and relies on Trusted Lists. ‘TLManager’, a complementary piece of software was developed to enable the creation, editing and maintenance of these Trusted Lists.

The Internal Market Information (IMI) System aims to provide a flexible administrative cooperation platform amongst different policy area. It facilitates the exchange of information between public administrations across Europe involved in the practical implementation of EU law. The first premises of the development of the platform happened under the umbrella of the IDA programme in 2005. Over the last six years, the number of policy areas covered has been constantly extended so that IMI now supports 15 administrative procedures relating to nine different policy areas under the Internal Market. IMI connects around 13,000 users in more than 7,000 public authorities in all 31 EEA Member States.

Generic and Interoperable Notification Services (GENIS) which provides IT solutions to support the State Aid Notification Process in a more complete and better way than before, while keeping a significant level of data security, confidentiality and System reliability. In fact, GENIS allows development and management of large web forms with complex validation requirements. GENIS also saves times in creating new forms, updating existing forms and in deployment of such web form services.

Overall, the evaluation confirms that the programme made available tools meeting the needs of specific sectors such as administrative cooperation among several internal market policies, state aid, eProcurement and European Citizens Initiative. The programme contributed significantly in making available tools needed across multiple sectors, such as    SD-DSS and TLManager for e-Signature and validation, ECAS-STORK integration and Open e-TrustEx. Most of the new reusable generic tools were also delivered in due time by the programme, with the exception of STORK sustainability which suffers from a significant delay (cf. details included in subsection 3.2.3).

3.3.5Assessment of ICT implications of proposed or adopted EU legislation

In order to assess the effectiveness of the assessment of ICT implications, the evaluation looked at the following expected results documented by the ex-ante evaluation of the ISA programme:

Smooth implementation of EC legislation;

Identification of the needs for services and tools in time so that requirements are fulfilled when the legislation comes into force;

Understanding of ICT aspects of EU policies.

As we can deduce from the expected results set by the ex-ante evaluation and as stipulated in both the ISA legal decision and EIS, carried forward from the IDABC programme, the assessment of ICT Implications of new EU legislation is an important activity of the programme. The scope of this activity goes even beyond the scope of the ISA programme, by covering any legislation having ICT implications (Article 3(b) of the ISA legal decision).The importance of assessing ICT impacts of EU legislation is now reflected in the Better Regulation guidelines 80 and the process to assess the ICT impacts of EU legislation referenced in the Better Regulation Toolbox 81 . From now on, new EU initiatives should all be ‘digital and internet-ready’ and operate both in the digital and physical world.

This political achievement was made possible thanks to the change in the scope of the action and intensive communication efforts from the ISA unit to the different Commission DGs and services.

As mentioned in the ISA Work Programme 2nd revision 2012, before 2013, the action initially aimed at (i) testing the existing method on real-life cases with Commission services being in the process of drafting legislation (2010 – 2012); and (ii) offering it to all Commission services from 2012. Although this method was indeed offered to numerous DGs in the period of 2010-2012, none of them came forward to test this method on their legislative draft.

In 2012, the ISA unit realised the need to (i) drastically review the existing method, perceived as too complex; (ii) support the Commission services performing this assessment by making a pool of resources (ICT experts) available to them; and (iii) promote the method and the service within the European Commission. For this purpose, the ISA unit adopted a collaborative approach and consulted 38 Commission officials from 11 different Commission DGs to gather their needs on the subject and collect their views and opinions on the existing method and on the proposed changes to make. In parallel, the first ICT Assessments were conducted for several DGs including DG HOME, DG JUST, SECGEN and DIGIT to test this new method and continuously improve it, based on the lessons learned from each assessment. These assessments came from a direct request from the DGs to which the initiative was presented during the consultation or from the regular screening of new Commission initiatives (roadmaps), as performed by the programme to anticipate the needs for new ICT Assessments.

Commission DGs can now use the method to perform their assessments, either on their own or with the help of DIGIT; a specific service is indeed provided for free by the programme to Commission DGs willing to assess the ICT impacts of EU legislation. Taking into account that the ICT Assessment method is now included in the Better Regulation toolbox as a tool for policy-makers aiming at designing and assessing new initiatives 82 , the contribution of the programme to the Better Regulation can be considered as significant from a strategic point of view, and to a lesser extent in terms of operational results since not many ICT Assessments were conducted compared to what was foreseen.

Besides the practical results achieved by the programme on the assessment of ICT implications, especially in the period 2013-2015, the evaluation assessed the views from ISA stakeholders on the key results achieved by the programme on such an important programme activity. In that regard, the figure below presents the answers provided by respondents who were requested to provide their level of agreement on the extent to which the ISA programme contributed to the assessment of ICT implications of new EU legislation.

Figure 22 ISA contribution to the assessment of ICT implications

As depicted in Figure 22 , 14% of respondents fully and 34% somewhat agreed that the ISA programme has contributed to the assessment of ICT implications of EU legislation. However, this does not represent a majority of respondents. Indeed, 21% neither agreed nor disagreed and 5% and 3% respectively somewhat and fully disagreed whereas 23% did not know or did not express any opinion on this matter.

In order to improve the effectiveness of this action, comments were made on the fact that the method, currently under revision, should on one hand be more used by the European Commission but remain applicable at Member States’ level. At the moment, the evaluation notes a lack of awareness of the results of this action, especially at Member States’ level. The need for more awareness on the results delivered by this activity was indeed raised by seven interviewees, six of whom were Member States’ representatives.

Overall, the evaluation confirms that the programme achieved the expected results set for this activity from a strategic point of view, but to a lesser extent from an operational standpoint:

The real-life ICT assessment cases performed in 2014 and 2015 contributed to the smooth implementation of EC legislation;

Having a tool dedicated to ICT assessments as part of the Better Regulation toolbox facilitates the identification of the needs for services and tools in time so that requirements are fulfilled when the legislation comes into force;

The reviewed ICT assessment method ensures a better understanding of ICT aspects of EU policies, as it target policy-makers before going into technical considerations.

The ISA programme should continue to support Commission DGs and services in their ICT assessment so as to multiply their experience on this subject and promote the need for assessing ICT implications of new or revised legislation. Once the method is tested and a proven tool to successfully perform ICT assessment, efforts should be dedicated to developing a method targeting national public administrations.

3.3.6Conclusions

The following box summarises the main conclusions related to EQ4: ‘To what extent did the ISA programme's results and impacts achieve its objectives?’ and EQ5: ‘Are there aspects that were more or less effective than others, and, if so, what lessons can be drawn from this?’

3.4Utility

The evaluation of the utility of the ISA programme is driven by the following evaluation questions:

EQ6: ‘How did the ISA programme's achieved and anticipated results and impacts compare with the business needs they intended to address?’

EQ7: ‘Which measures could be taken to improve the utility of the next programme’s actions?’

The remainder of this section is divided into different subsections in accordance with the different judgement criteria allowing the evaluation to answer EQ6 and EQ7.

To answer these evaluation questions, the evaluation refers to ‘impacts’ achieved as the benefits generated by the programme for its targeted stakeholders, i.e. European public administrations. Since one of the main impacts expected by the programme is to provide interoperable solutions available for use, the evaluation looks at the use of ISA solutions by the targeted stakeholders. Since the evaluation question EQ7 aims to gather potential measures to be taken to improve the utility of the next programme’s actions, the evaluation verified this aspect with the programme’s stakeholders who provided valuable insights on this based on their experience within ISA, previous programmes, and other EU initiatives.

3.4.1Benefits generated by the ISA programme

In accordance with the ISA legal decision, and in particular Article 1 and Article 13, the programme should have supported the delivery of electronic public services, and generated benefits for the implementation of EU policies. This subsection appraises the extent of these achievements as well as analyses the general perception of the Member States (targeted stakeholders) on the benefits delivered by the programme.

General perceptions on the benefits delivered by the programme

The evaluation verified with the Member States’ representatives what they considered as benefits delivered by the ISA programme to their countries. Based on 97% (40) of respondents who agreed that the programme delivered benefits, the most commonly cited examples, by 51% (21) of surveyed respondents, were the reuse of ISA solutions. Interaction, networking and sharing was cited by 29% (12) of respondents. A further 29% (12) identified the existence of the programme itself as a means to raise awareness about the importance of interoperability and to keep it on the political agenda. 22% (9) of respondents said that the ISA programme provided their countries with good reference points, networks and solutions (e.g. EIF/NIFO, sTESTA and CIRCABC) from which it could develop interoperability nationally.

Respondents provided specific areas or ISA actions as evidence in addition to the general comments above. ISA’s work related to INSPIRE/geo-spatial data (linked to action 1.17) and action 2.05 CIRACBC were mentioned by 7% (3) of respondents. e-Signatures (linked to actions 1.04 &1.09), action 2.02 CAMSS, action 2.04 sTESTA and action 4.2.01 Joinup were each mentioned by 5% (2) of respondents.

The EIF (liked to action 4.02.03 NIFO) was cited by 10% (4) of respondents as an inspiration for national initiatives. However, the evaluation notes that the expected benefits of the EIF have not been fully achieved. The State of Play of Interoperability in Europe – Report 2014 83 found that while national interoperability frameworks (NIFs) are aligned with the EIF at a rate of 72%, the average implementation rate of the EIF at national level is 28%. A large part of this low rate of implementation is the fact that the EIF, an EU Communication, is not binding for European public administrations. Furthermore, the choice of verb in Article 1 of the legal decision captures the relatively limited extent of the programme’s capacity to ensure adoption of its solutions, including the EIF through action 4.02.03, by stating that ‘The objective of the ISA programme is to support cooperation between European public administrations by facilitating the efficient and effective electronic cross-border and cross sectoral interaction between such administrations.’

The evaluation confirms that the reuse of ISA solutions is an important aspect as acknowledged by the interim evaluation of the programme, which recommended the establishment of a control mechanism to ensure the reuse of operational ISA solutions. However, as specified in the ISA legal decision, the use of ISA solutions should be financed by the end users, and therefore it is not under the direct responsibility of the programme itself.

Contribution to the delivery of electronic public services

Establishing the perceived contribution of the programme to the delivery of electronic public services was an essential part of the evaluation. Member States’ representatives, EC officials and other ISA stakeholders agreed at a rate of 84% (92) that the programme enables ‘the delivery of electronic public services supporting the implementation of EU policies and activities'. 5% (5) did not believe this to be the case whereas those who did not know or did not express an opinion amounted to 11% (13).

Further analysis of the findings of the interviews and online surveys shows it was the perception of respondents that action 1.07 e-Prior/e-Procurement was the main electronic public service supported by the ISA programme. It occurred in 28 responses followed closely by e-Identification (or e-ID; 24 occurrences) and e-Signature (20). Less common, but nevertheless frequently mentioned, were the electronic public services delivered by action 2.04 sTESTA (7), input for e-Invoicing (7) and action 1.04 ECAS-STORK (5).

The perceptions of the two different surveyed stakeholder groups, EC officials and the Member States’ representatives, did not vary substantially. Action 1.07 e-Prior/e-Procurement was listed most frequently by both groups as the main electronic public service supported by the ISA programme. It is important to note that e-ID and e-Signature (linked to action 1.09) occurred in the responses of the Member States’ representatives exactly as often as the most commonly mentioned option, action 1.07 e-Prior/e-Procurement. However, EC officials clearly consider action 1.07 e-Prior/e-Procurement as the main electronic service, which was mentioned in one in every four responses. Furthermore, a similar response pattern was observed between the two groups of respondents in terms of the remaining most frequently mentioned main electronic public services; these were e-ID, e-Signature, e-Invoicing and action 2.04 sTESTA.

Given that it is not sufficient to rely solely on the varying levels of knowledge of interviewees and online survey respondents when discussing a topic as objective and as important as utility, the evaluation team complemented the aforementioned findings with further desk research. The 2015 First Semester Monitoring & Evaluation Report, compiled by CGI-Accenture, provides valuable desk research in this regard. This biannual report measures the progress made and results obtained on the availability, effectiveness, efficiency, perceived quality and utility evaluation criteria at action level through a survey of users and beneficiaries of each action. It measures the utility of 13 ISA actions, the average score for which is 4.07/5. ISA actions’ Project Officers nominate users and beneficiaries of each action to participate in each biannual survey. It can be inferred that this survey group is more knowledgeable about, and familiar with, these actions and, therefore, is better placed to provide feedback on their utility.

Utility is measured using an adaptation of the VAST (Value Assessment Tool) methodology 84 , considering an additional dimension related to the Global and Intermediate objectives of the ISA programme. The assessment is based on several dimensions: value for the European Union 85 :; value for the European Commission 86 and value for cross-border and cross-sector interoperability 87 ,

The findings demonstrate that six actions, shaded below in grey, have an above average utility score of the 13 assessed. It is worth mentioning that actions with a low sample size, namely action 1.8 E-TrustEx (3 respondents), action 1.12 OSS-ECI (5 respondents) and action 2.1 ((6 respondents) represent the highest utility scores. Due to the small sample size for these actions, the analysis presented for them is not valid to calculate statistically meaningful results and has an informative purpose only. However, there is no common trend or impact detected on the metrics average scores (i.e., common trend for perceived quality and utility) when reducing these values from the average calculations.

It is also worth noting that all of these actions have a utility score of over 3.5/5, which is 70% and reiterating that these are the scores provided by users and beneficiaries of these actions rather than a wider audience, many of which are unaware of their work.

Table 18 Utility scores at action level

The figure below displays these 13 ISA actions on the basis of the relation between each action’s weight ratio and the utility score that the action has attained. Furthermore, the size of each bubble within the figure illustrates the size of the allocated budget of the action.

Figure 23 ISA actions measured for Utility

Returning to final evaluation of the ISA programme’s online survey, respondents provided insights into what they considered the most and least valuable actions for ‘the delivery of electronic public services supporting the implementation of EU policies and activities'.

As regards the most valuable actions, 50% (16) of respondents named action 1.01 Semantic interoperability. Action 2.01 Elaboration of a common vision for a European Interoperability Architecture (EIA) was mentioned by 41% (13) of respondents and action 1.02 Access to base registers was mentioned by 34% (11) of respondents. These were followed by action 1.03 Catalogue of services and action 1.17 Reusable Inspire Reference Platform (ARE3NA), which were each cited by 31% (10) of respondents.

On the other hand, action 2.06 EUSurvey, mentioned by 37% (7) of respondents, was considered the least valuable ISA action by those who answered 89 , 90 , easily distancing the second least valuable action, action 5.01 Monitoring and Evaluation, which was cited by 26% (5) of respondents. Although the latter action is indeed needed as part of the programme management as a support activity, when taken on its own by respondents, it was not perceived as contributing, as such, to delivering electronic public services supporting the implementation of EU policies and activities. Nevertheless, the evaluation considers this activity to be of fundamental importance to provide insight into the programme’s performance as proven by the desk research presented in the subsection 3.6.2.

With regard to the most and the least valuable actions, some respondents noted that it is difficult to have the overview of all the actions to be able to knowledgably state which ones were valuable and which were not. It is clear from the low numbers of answers to the questions on the most valuable (32) and least valuable (19) that this posed a challenge for respondents. However, with two-thirds more answers on the former question than the latter, it would suggest that more stakeholders considered the ISA programme and its actions to be more valuable than those who did not.

Translation of needs into ISA solutions

Subsequent findings to similar questions reinforce this view. Member States’ representatives and EC officials who participated in the online surveys and interviews largely agreed that needs expressed by European public administrations have been successfully translated into ISA solutions. 12% (12) fully agreed and 54% (53) somewhat agreed whereas 7% (6) somewhat disagreed, 13% (13) neither agreed nor disagreed and 14% (14) did not express an opinion.

Comments from the interviews generally acknowledge the ISA programme’s success at translating needs expressed by European public administrations into solutions but offer some caveats. One official, who somewhat agreed, opined that some European public administrations sometimes struggle to express their needs, rendering it difficult for the ISA programme to translate the needs into solutions. Another official, who somewhat disagreed, said that the ISA programme successfully translated needs into actions but not into solutions, adding that, in his view, there had not been enough delivery in several ISA actions. Another official fully agreed that the needs of European public administrations had been successfully translated into ISA solutions but that they were not sufficiently reused.

Support the implementation of EU policies

Furthermore, the evaluation team conducted desk and field research on EU public policies and activities that were supported by the ISA programme. Recital 2 of the ISA legal decision 91 recalls the European Council’s conclusion that “more focused, effective and integrated policies regarding information and communication technologies (ICT) at both European and national levels are essential to achieving the goals of economic growth and productivity”. The Commission “was invited to encourage the efficient use of ICT in public services through the exchange of experience and to develop common approaches on key issues such as interoperability and effective use of open standards”.

The Commission considers the cross-border interoperability of online services and the digitalisation of European public administrations to be important factors for growth and increased efficiency. It has consistently demonstrated its commitment to interoperability solutions over a remarkably long period 92 .

The table below provides an overview of how the ISA programme advances or is implementing other EU initiatives, legislation or funding tools, which, in turn, provide interoperable solutions for policy challenges at both national and European levels. In June 2014, the Commission adopted a proposal for the successor of the ISA programme, ISA², the objectives of which are to capitalise and expand on the results of the ISA programme’s contribution to the advancement of EU policies, such as those listed below.

Table 19 ISA and related other EU initiatives, legislation or funding tools

As mentioned in the recent Communication on the Digital Single Market Strategy (DSM), cross-border interoperability can contribute to the modernisation of public administrations. Modernising public administrations at all levels (between the different layers of government) was one of the five main priorities at EU level set out in the Annual Growth Survey 2014 113 .    

Therefore, the evaluation verified how the ISA programme, being one of the contributors to the modernisation of public administrations, has contributed to this area as well. With regard to the overall numbers for Member States’ representatives and EC officials who participated in the online survey and interviews, 41% (40) fully agreed and 46% (45) somewhat agreed that the ISA programme contributed to the modernisation of the public sector in Europe. 4% (4) somewhat disagreed, 6% (6) neither agreed nor disagreed and 3% (3) did not express an opinion.

In addition, respondents offered different types of comments on the ISA programme’s contribution to the modernisation of the public sector in Europe. The majority of the 12 respondents who commented said that the ISA programme was an enabler of the modernisation of the public sector in Europe given how it is trying to ensure that interoperability and the programme’s solutions can enhance the operation of European public administrations. However, several respondents said that the ISA programme could do more to promote itself to contribute to this and that interoperability was only one part of this large question.

The interview and survey respondents largely considered that the ISA programme contributed to the modernisation of the public sector in Europe. Nevertheless, the desk research has identified the following barriers that the follow-on programme should address. These include 114 :

Legal and political – lack of availability of enforcement measures, etc.

Organisational – size of Member State, institutional complexity, lack of resources (time, budget, skills), etc.

Technical – legacy, replacement of the older systems, etc.

Other – lack of visibility of existing solutions, language, lack of information/documentation, lack of trust, lack of technical and semantic interoperability, etc.

The ISA programme has acknowledged these barriers and aimed to overcome them by establishing initiatives such as European Interoperability Reference architecture and EU cartography in order to support the European public administrations in their further modernisation. These initiatives aim to accelerate the design of the support systems of digital public services; to provide a reference model for comparing existing architectures, to help document the most salient interoperability elements, to facilitate the sharing of reusable solutions and to ease the discovery and reuse of interoperability solutions (e.g. via EUCart in Joinup website). Furthermore, the evaluation acknowledges that the ISA programme has contributed to the modernisation of public administrations area through the action 4.2.03 NIFO.

In this respect, since 2013, the ISA programme, through action 4.2.03 NIFO, focused on the identification of existing or new ISA actions that have the potential to contribute to the European Structural and Investment Fund (ESIF) Thematic Objective (TO) 2 “enhancing access to, and use and quality of ICT” and TO 11 “enhancing institutional capacity of public authorities and stakeholders and efficient public administration”. One of the objectives of the ESIF is to support Member States' efforts to improve the quality of public administration and governance by a provision of funds to acquire the necessary administrative and institutional capacities to support their structural reforms.

3.4.2Use of the ISA solutions by Member States

As previously mentioned, the adoption and use of the ISA programme’s solutions by Member States’ administrations is voluntary. Consequentially, Member States’ representatives and EC officials were questioned about the use of ISA solutions by Member States. Out of a total of 98 respondents, 8% (8) fully agreed and 46% (46) somewhat agreed that the results obtained by the ISA programme are currently used by Member States. 1% (1) fully disagreed and 12% (12) somewhat disagreed. 19% (18) neither agreed nor disagreed and 13% (13) did not express an opinion.

In the course of the online surveys, Member States representatives were also asked more precisely, based on a predefined list of ISA solutions 115 , to choose one of the following options to indicate whether their own country:

is currently using [the proposed] ISA solution (option 1); or

intends to use [the proposed] ISA solution in 2015/2016 (option 2); or

uses a similar solution but intends to replace it by [the proposed] ISA solution (option 3); or

uses a similar solution but has not yet decided to replace it (option 4); or

uses a similar solution but does not intend to replace it by [the proposed] ISA solution (option 5); or

does not need [the proposed] ISA solution (option 6).

The figure below presents, for each ISA solution, the number of times each of the aforementioned options was selected by the Member States’ representatives responding to the survey.

Figure 24 Use of ISA solutions by Member States

It is the perception of the survey respondents that the ISA solutions most reused by Member States are the following: action 2.05 CIRCABC (14 occurrences), action 4.2.01 Joinup (11 occurrences) and action 2.04 sTESTA (8 occurrences). Action 1.07 Open e-Prior is the only ISA action that is currently not being used by any of the surveyed Member State representatives. However, it is important to note that there is a negative correlation between the level of use of an ISA action and the occurrence rate of ‘Don’t know / No opinion’ responses regarding that action. Thus, the low levels of reuse of these actions, that survey respondents perceive, could partially be explained by low levels of awareness at individual or Member State level.

Several Member States intend to use an ISA solution in 2015/2016. In this regard, the ISA actions with highest occurrence rates are: the DCAT Application Profile for data portals in Europe (4 occurrences), ADMS (3 occurrences), Core vocabularies (3 occurrences) and action 4.1.02 Interoperability Maturity Model (IMM) (3 occurrences). However, the aforementioned ADMS can also be considered as the ISA action with lowest likelihood of being reused in the future by Member States as it has the highest occurrence rate for the Option 5 and Option 6 (3 occurrences), i.e. classifying the solution as not intended to be used or as not needed.

Given that it is not sufficient to rely solely on the varying levels of knowledge of interviewees and online survey respondents when discussing a topic as objective and as important as reuse, the evaluation team complemented the aforementioned findings with desk research. This triangulation exercise completed the overview of the ISA solutions that are reused by Member States. In this regard, the evaluation confirms that 15 ISA solutions are actually being reused by Member States. A precise description of these solutions is available in subsection 3.3.2. However, the table below provides an overview of the level of adoption of each of these aforementioned solutions by Member States.

Table 20 Level of adoption of ISA solutions by Member States

The evaluation emphasises that two other ISA actions have partially reached their objectives: ISA action 3.01 – Assessment of ICT implications of EU legislation and action 5.02 – EIS Governance support. As mentioned in the ISA Work Programme 2011 116 , the former action aimed at performing the following activities between 2010 and 2015:

3 Pilot assessments Refinement of method and elaboration of tools in 2010-2011;

9-18 assessments with different Commission Services in 2011-2013; and

6-12 assessments with further Commission Services 2013-2015.

In reality no effort was allocated to this action between 2011 and 2012. The low adoption of the existing ICT implications assessment method (developed in 2010) indeed prevented the conduct of pilots or assessments with different Commission services. Taking into account this low uptake and the recent updates on the way legislation is prepared by the Commission (Better Regulation guidelines), the existing method was reviewed in 2013 and six assessments have been performed since that year 117 . The evaluation concludes that, overall, the action has been in the right direction since 2013 to reach part of its objectives but the expected results were not delivered in the period 2010-2012

With regard to the ISA action 5.02, one of the action’s main objectives, as stated in the ISA Work Programme 2011, was that the EIS stays aligned with the EU political agenda and with the priorities and initiatives of the Member States regarding European Public Services and interoperability activities. While the EIS implementation review was performed in 2012, the revision of the strategy has not yet been conducted, putting into question the extent to which the strategy has remained aligned with these initiatives. The EIS aims now to be reviewed in 2015 and completed by 2016.

3.4.3Use of the ISA solutions by Commission DGs and other Institutions

EC officials were questioned about the use of ISA solutions by their DGs. Out of 57 respondents, a majority fully or somewhat agreed, respectively 23% (13) and 44% (25), that the results obtained by the ISA programme were being used by their DGs. 4% (2) fully disagreed and 12% (7) somewhat disagreed. 11% (6) neither agreed nor disagreed and 6% (4) did not express an opinion.

In the online surveys, EC officials were also asked more precisely, based on a predefined list of ISA solutions, to choose one of the following proposals to indicate whether their own DG:

1.is currently using [the proposed] ISA solution; or

2.intends to use [the proposed] ISA solution in 2015/2018; or

3.uses a similar solution but intends to replace it by [the proposed] ISA solution; or

4.uses a similar solution but has not yet decided to replace it; or

5.uses a similar solution but does not intend to replace it by [the proposed] ISA solution; or

6.does not need [the proposed] ISA solution.

The figure below presents, for each ISA solution, the number of times each of the aforementioned proposals were selected by the EC officials responding to the survey.

Figure 25 Use of ISA solutions by Commission's services

It is the perception of the survey respondents that the ISA solutions most reused by Commission's services are: action 2.05 CIRCABC (15 occurrences), action 4.2.01 Joinup (13 occurrences), action 2.06 EUSurvey (12 occurrences) and action 2.04 sTESTA (11 occurrences). However, it is important to note that there is a negative correlation between the level of use of an ISA action and the occurrence rate of ‘Don’t know / No opinion’ responses regarding that action. Thus, the low levels of reuse of these actions, that survey respondents perceive, could partially be explained by low levels of awareness at individual or EC DG/service level.

On the other hand, ISA actions perceived as currently the least used show the highest levels of intended use in 2015/2016, i.e. Core vocabularies (3 occurrences for Proposal 2), action 2.02 CAMSS (2 occurrences for Proposal 2) and action 4.1.02 IMM (2 occurrences for Proposal 2). Similarly, ISA actions with higher levels of current use show lower levels of intended use for 2015/2016 by Commission's services. In contrast to that, actions with lowest perceived current use are scoring highest occurrence rates for Proposal 6, i.e. no intention to use the ISA action. The actions with highest occurrence rates for Proposal 6 are: action.1.10 IMI (5 occurrences), action 1.07 e-Prior (2 occurrences) and action 1.08 e-TrustEx (2 occurrences). The evaluation team notes that it is normal that certain or some EC services do not foresee a use for action 1.10 IMI or Open e-Prior developed under ISA action 1.07 since they are not the target.

With regard to action 1.07, all DGs (73 Commission Services 118 ) already use ePrior as several EC processes are based on this. As well as the ePrior service for all DGs, a tool, Open ePrior, was developed for European public administrations during the course of the ISA programme to boost the use of eProcurement by enabling them to connect to the PEPPOL network easily by making use of the e-PRIOR platform.

Given that it is not sufficient to rely solely on the varying levels of knowledge of interviewees and online survey respondents when discussing a topic as objective and as important as reuse, the evaluation team complemented the aforementioned findings with desk research. This triangulation exercise completed the overview of the ISA solutions that are reused by Commission DGs and services. In this regard, the evaluation confirms that 14 ISA solutions as currently used by European Commission DGs and services.

A precise description of these solutions as well as details on the DGs and services using them are available respectively in Table 21  with further details included in the Annex. However, the table below provides an overview of the level of adoption of these aforementioned solutions by European Commission DGs and services.

Table 21 Level of adoption of ISA solutions by Commission's services