EUROPEAN COMMISSION

Brussels, 19.4.2016

SWD(2016) 108 final

COMMISSION STAFF WORKING DOCUMENT

Implementation and Evaluation report

Accompanying the document

Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions

EU eGovernment Action Plan 2016 - 2020
Accelerating the digital transformation of government

{COM(2016) 179 final}
{SWD(2016) 109 final}

Table of contents

1.Introduction

1.1.The purpose of the evaluation

1.2.The scope of the evaluation

2.Background to the EU eGovernment Action Plan 2011 - 20156

2.1.Description of the European eGovernment Action plan 2011 – 2015 and its objectives6

2.2.Instruments8

2.2.1.Political Activities including exchange of experience and Commission Communications8

2.2.2.Funding for eGovernment9

2.2.3.Legislation10

2.3.Intervention logic11

2.4.Situation in 2010/Baseline12

2.4.1.Baseline of target: 50% of citizens (Digital Agenda) and 80% of businesses make use of eGovernment services13

2.4.2.Baseline of target: a number of key cross-border services be offered online by 2015 (Digital Agenda)13

2.4.3.Baselines in the four policy priority areas13

3.Evaluation Questions/Criteria15

4.Method and sources16

4.1.Mid-term evaluation16

4.2.Annual benchmarking reports16

4.3.The online public consultation17

4.4.Limitations18

5.Implementation of the EU eGovernment Action Plan 2011 – 2015 – state of play of the policy priorities19

5.1.General target: 50% of citizens (Digital Agenda) and 80% of businesses make use of eGovernment services19

5.2.General target: a number of key cross-border services be offered online by 2015 (Digital Agenda)19

5.3.The four priorities and 40 actions of the EU eGovernment Action Plan22

5.4.Governance22

6.Answers to the evaluation questions24

6.1.Effectiveness24

6.1.1.Objective on using eGovernment (take-up)24

6.1.2.Key cross-border services online25

6.1.3.Effectiveness of the four policy priorities26

6.2.Efficiency36

6.2.1.Efficiency of the related budget36

6.2.2.Costs and benefits to stakeholders38

6.3.Relevance39

6.3.1.Relevance: User Empowerment40

6.3.2.Relevance: Internal market42

6.3.3.Relevance: "Efficiency and Effectiveness of Governments and Administrations"45

6.3.4.Relevance: Pre-conditions48

6.4.Coherence49

6.5.EU-added value51

7.Conclusions54

8.ANNEXES57

Annex 1.Procedural information57

Annex 2.The Online public consultation59

Annex 3.The individual actions of the eGovernment Action Plan 2011 - 201589

Annex 4.Implementation of the individual actions of the eGovernment Action Plan 2011 – 2015 – State of play of the actions93

Priority: User Empowerment93

Priority: Internal Market96

Priority: Efficiency and Effectiveness of Governments and Administrations98

Priority: Pre-conditions for developing eGovernment100

Annex 5.Acronyms103

Annex 6.REFERENCES AND DATA SOURCES105

1.Introduction

1.1.The purpose of the evaluation

In December 2015, European Commission's Directorate-General Communications Networks, Content and Technology launched an implementation and evaluation assessment of the European eGovernment Action Plan 2011 – 2015 1 ("EU eGovernment Action Plan 2011 – 2015"), adopted on 15th December 2010. The purpose of this evaluation is to describe the progress of implementation of individual actions of the Action Plan and assess its functioning. In the broadest sense, this evaluation exercise will provide answers to how well the European public administrations have progressed towards meeting the Malmö vision (November 2009 Malmö Ministerial declaration 2 ), which is included in this eGovernment Action Plan:

"by 2015 European public administrations will be recognised for being open, flexible and collaborative in their relations with citizens and businesses. They use eGovernment to increase their efficiency and effectiveness and to constantly improve public services in a way that caters for user's different needs and maximises public value, thus supporting the transition of Europe to a leading knowledge-based economy."

The EU eGovernment Action plan 2011-2015 did not indicate the need to carry out a final evaluation as such. According to the eGovernment Action Plan, the overall progress would be measured annually, using an appropriate mix of instruments and methods. However, according to the Commission's Better Regulation guidelines, retrospective performance evaluations support policy preparation 3 . "Evaluations gather evidence to assess how well a specific intervention has performed (or is working) - -. An evaluation also draws conclusions on whether the EU intervention continues to be justified or should be modified to improve its effectiveness, relevance and coherence or to eliminate excessive burdens or inconsistencies or simply be repealed." This evaluation is carried out in so far as possible in this context, in order to provide a set of lessons learnt to guide the work of the new eGovernment Action Plan 2016 – 2020 announced in the Digital Single Market Strategy 4 .

1.2.The scope of the evaluation

The scope of the evaluation is the objectives and measures set out in the EU eGovernment Action Plan 2011 – 2015. It covers the level of progress in eGovernment policy in the EU in the period starting from the beginning of 2011 (Action Plan was adopted 15 December 2010) until and including 2015, subject to the availability of most recent data.

The Action Plan 2011 – 2015 aimed to contribute to achieving two important targets of the Digital Agenda in Europe 5 : 1) "By 2015, 50% of EU citizens will have used eGovernment services"; and 2) offering a number of cross-border services online. The Action Plan also aimed that "by 2015, 80% of enterprises will have used eGovernment".

Furthermore, the evaluation focuses on the four policy priorities (agreed by Commission and Member States in the Malmö Ministerial Declaration) of the EU eGovernment Action Plan 2011 – 2015:

1.User Empowerment

2.Internal Market

3.Efficiency and Effectiveness of Governments and Administrations

4.Pre-conditions for developing eGovernment

The Action Plan included 40 individual actions (listed in Annex 4) divided amongst these four priorities. Five additional actions were related to the governance.

The evaluation focuses on the Action Plan as a political instrument to achieve its objectives. It does not evaluate in detail individual actions identified in the Action Plan. An assessment of the completion of actions will be done separately via a study planned for 2016. The results of the study will be used to update to complete this evaluation. Furthermore, since the scope of the evaluation is the policy priorities eGovernment Action Plan 2011 – 2015, it will not cover eGovernment related activities in Member States at all levels of government and all sectors.

For the evaluation of the eGovernment policy priorities in the EU it is important to note that eGovernment has also been supported through a number of funding programmes (listed in section 2.2.2). The evaluation of these funding programmes falls outside the scope of this evaluation but their findings are referred to in this evaluation when relevant to address the evaluation criteria.

The evaluation criteria used are in line with the Commission's Better Regulation Guidelines and assess the effectiveness, efficiency, relevance, coherence and the EU added value of the EU eGovernment Action Plan 2011 - 2015.

2.Background to the EU eGovernment Action Plan 2011 - 2015

2.1.Description of the European eGovernment Action plan 2011 – 2015 and its objectives    

The Europe 2020 strategy that was put forward by the Commission on 3 March 2010 6 set out a strategy for smart, sustainable and inclusive growth to face the immediate challenges resulting from the financial crisis and long-term challenges such as resource scarcity, ageing and globalisation. One of the so-called seven flagship initiatives, the Digital Agenda for Europe 7 , outlined a number of actions supporting the further development of eGovernment in the EU. The actions can be found from the eGovernment chapter of the Digital Agenda Communication and from the eGovernment Action Plan that was the successor of the i2010 Action Plan 2006-2010 8 .

The Action Plan 2011 – 2015 aimed to contribute to achieving two important targets of the Digital Agenda in Europe: 1) that 50% of citizens make use of eGovernment services; and 2) that a number of key cross-border services will be available online by 2015. The Action Plan also aimed that "by 2015, 80% of enterprises will have used eGovernment". Furthermore, the objective was to translate into concrete measures the four political priorities of the Malmö Ministerial Declaration on eGovernment of December 2009. Subsequently, the eGovernment Action Plan had the following four policy priorities (see also Figure 1 ):

"1.    User Empowerment

Empowerment means increasing the capacity of citizens, businesses and other organisations to be pro-active in society through the use of new technological tools. Public services can gain in efficiency and users in satisfaction by meeting the expectations of users better and being designed around their needs and in collaboration with them and third parties whenever possible. Empowerment also means that governments should provide easy access to public information, improve transparency and allow effective involvement of citizens and businesses in the policy-making process.

2.    Internal Market

Most public online services do not work across borders or involve cumbersome procedures to be accessible. People from one EU country cannot easily apply for public services in another country than the one in which they are established, using for instance national electronic identity cards. This seriously reduces the mobility of businesses and citizens. To support the Internal Market, governments should develop ‘seamless’ services for entrepreneurs to set up and run a business anywhere in Europe and allowing individuals to study, work, reside, receive health care and retire anywhere in the European Union.

3.    Efficiency and Effectiveness of Governments and Administrations

The aim was to use ICT and enabling organisational changes to deliver better, less intrusive, more sustainable and faster public services, by reducing the administrative burden, improving organisational processes and promoting a sustainable low-carbon economy.

4.    Pre-conditions for developing eGovernment

A number of technical and legal pre-conditions need to be put in place to enable the implementation of the actions that will enhance eGovernment services in Europe. These include the promotion of interoperability across borders, which would allow - among others - sharing of information, deployments of one-stop-shop approaches, Europe wide use of (national) electronic identity solutions and payment schemes. Interoperability is supported through open specifications and the development of key enablers such as electronic identity management and stimulation of innovation in eGovernment."

Figure 1 The four policy priorities of the eGovernment Action plan 2011 – 2015 and their related objectives.

To achieve these objectives, the Action Plan proposed 40 individual actions (listed in Annex 4 to the present document) distributed under each of the policy priorities. According to the Action Plan, "the actions can also be categorised in three groups, depending on the actors involved and their competence defined in the Treaty:

1.Where Member States are leading and rely on their own resources, the Commission will help by supporting and coordinating activities. The measures proposed will focus on setting targets with the Member States and on how to achieve these targets by means of measures such as exchanging best practice and information, conducting studies and benchmarking.

2.Where the Commission and the Member States work jointly to develop, deploy or improve cross-border services, the Commission will take the lead in activities where joint resources are used, while the Member States will bear the final responsibility for implementing activities using their own resources. The measures proposed will include research and development, pilot projects, collaborative development of services by Member States and transfer of knowledge to the market.

3.Where the Commission can create enabling conditions, the measures proposed will include adopting legal instruments, setting standards, formulating common frameworks, implementing generic tools, providing (re-usable) technical building blocks and ensuring interoperability."

Of the 40 actions in the eGovernment Action Plan, the Commission was exclusively responsible for 23 and the Member States for 10. Seven actions were under shared responsibility between the Member States and the Commission. Five additional actions, two for the Commission and three for the Member States, were included for the governance.

2.2.Instruments

The instruments used to implement the Action Plan were political, financial and legislative activities.

2.2.1.Political Activities including exchange of experience and Commission Communications

The political activities refer to dialogue with Member States, Ministerial meeting, exhibitions, awards and benchmarking. Their aim was to provide political steer, enhance mutual learning through exchange of experience and awareness rising.

Dialogue with Member States

eGovernment High-Level Expert Group of Member States representatives (activity under governance): The Commission and Member States have engaged in collaborating and in regular dialogue during the eGovernment Action Plan 2011 – 2015 through the eGovernment High-Level Expert Group. In addition, there have been other fora to engage with Member States' representatives: including the ISA (Interoperability Solutions for European Public Administrations) Programme Management Committee.

Ministerial meetings

Ministerial Conferences (together with a series of eGovernment Presidency events/conferences) helped to take stock of where eGovernment was and provide a vision for where it should go in the coming years, mainly in the form of Ministerial Declarations. The 2009 Malmö Ministerial Declaration resulted in the current eGovernment Action Plan. The conferences also provided exhibitions of projects and awards for best eGovernment projects. The 6th Ministerial meeting was held on November 2011 in Poznan. As well as the conference and the exhibition, it featured an Informal Council of Ministers meeting.

Benchmarking

Since 2001, the EC has been measuring the progress made in on-line public services. This annual benchmarking has become a worldwide reference for international comparison of the development of eGovernment. Since 2012 a new benchmarking framework has been in place, aligned to the four policy priorities of the current Action Plan.

The European Interoperability Framework and the European Interoperability Strategy

The European Interoperability Strategy 9 (EIS) and a European Interoperability Framework 10 (EIF) provide guidance in delivering services across borders and sectors. The EIF, in particular, has been translated into National Interoperability Frameworks in more than 20 Member States. It is now being reviewed and extended in the context of the Digital Single Market Strategy.

Cloud computing

The Commission Communication "Unleashing the Potential of Cloud Computing in Europe" 11 , provided a European strategy designed to speed up and increase the use of cloud computing across all economic sectors.

ICT standardisation

A number of ICT standardisation activities supported the EU policy activities, including the areas of eGovernment, eProcurement, Public Sector Information, Open Data and Big Data.

2.2.2.Funding for eGovernment

The Action Plan did not have funding, but served as a guide to prioritise a number of investments from EU funding programmes:

The past Competitiveness and Innovation Framework Programme ICT Policy Support Programme 12 (CIP ICT-PSP) has supported (some projects on-going) piloting of cross-border eGovernment solutions and supported piloting of concepts, like the "cloud of public services" 13 ;

Under the Connecting Europe Facility (CEF), Digital Services Infrastructures (CEF DSI) 14 deploy digital public services that work across borders;

Horizon 2020's Societal Challenge 6 'Europe in a changing world – inclusive, innovative and reflective societies' funds research and innovation actions on ICT-enabled public sector innovation 15 ;

The past Seventh EU Framework Programme for Research and Technological Development (FP7) supported eGovernment-related projects;

The ISA programme 16 (Interoperability solutions for European Public Administrations) provides a framework that allows Member States to work together to create efficient and effective electronic cross-border public services for the benefit of citizens and businesses;

The European Structural and Investment Funds (ESIF as of 2014 17 ) provide investments in the field of eGovernment in less favoured regions, as this is seen as a strategic component of their economic and social development. One of the thematic priorities 18 for the 2014 – 2020 programming period is enhancing access, use and quality of ICT (thematic objective 2), including strengthening ICT applications for e-government. Another thematic objective is on administrative capacity building (thematic objective 11).

The exact amounts invested to policy priorities of the Action Plan are in some cases challenging to measure. Sometimes investments were done directly by the Member States without reporting to the Commission. Furthermore, it is not always obvious what would be considered an eGovernment–related investment in programmes with various information technology or information society projects.

2.2.3.Legislation

Legislative instruments have been used to achieve the eGovernment Action Plan's objectives where needed. Some of these legislative instruments existed already in the beginning of the Action Plan and needed amendment, while new legislation was also required, these included:

The planned revision of the PSI (Public Sector Information) Directive was indicated in the eGovernment Action Plan. On 12 December 2011 presented a proposal in this respect 19 . The revised Directive was adopted on 26 June 2013. 20 The reuse policy of the Commission was revised by a Decision of 2011 21 .

The revision of the eSignature Directive and the proposal for a Decision for mutual recognition of eIdentification and eAuthentication were indicated in the eGovernment Action Plan and led to the eIDAS Regulation on electronic identification and trust services (Regulation 910/2014).

The update of the Public Procurement Directives presented by the EC in the end of 2011 proposed a gradual but ambitious transition towards e-procurement in the EU. In February 2014, two Public Procurement Directives 22 were revised and a Directive on concession contracts was adopted 23 . The new rules will enter into force as of 18 April 2016.

The CEF regulation in the area of telecommunications infrastructure 24 (CEF Telecom Regulation).

The reform of the EU data protection legal framework – the General Data Protection Regulation and the Data Protection Directive for Police and Criminal Justice Authorities. Both will be adopted in early 2016 and followed by a two year transition period 25 .

The proposal for a directive on the accessibility of public sector bodies' websites, which has not yet been adopted 26 .

The new ISA2 programme adopted by the co-legislators in November 2015 on interoperability between public administrations and extended to local and regional administrations and the links to businesses and citizens is a key instrument for the development of eGovernment in Europe. 27

2.3.Intervention logic

As stated in the Staff Working Document of 2010 accompanying the eGovernment Action Plan 2011 - 2015 28 , this eGovernment Action Plan 2011 – 2015 responded to a call from Member States, made in the Malmö Declaration, for a shared eGovernment policy in the European Union in order to build on past achievements and increase collaboration on eGovernment. This included joint action between Member States and close collaboration with the European Commission The simplified intervention logic figure ( Figure 2 ) explains how the different parts of the Action Plan fit together.

Figure 2 Simplified intervention logic.

2.4.Situation in 2010/Baseline

The 2010 Staff Working Document, referred to above, provides an overview of the situation in 2010 29 , focusing at the time on the availability of basic services online:

"The online delivery of basic services 30 has continued to increase steadily in recent years: their full online availability 31 went from 21% in 2001 to 71% in 2009 32 . However, there is a marked difference between services for businesses and services for citizens. Services for businesses reach 83% availability whereas services for citizens are still only at 63% availability."

There were considerable discrepancies between Member States. The gap between the best (with 100% availability of basic services) and worst performer (40% availability) was large. The EU average figure was around 70%. Four Member States had a full online availability for all the 20 basic services considered in 2009 in the benchmark measurement report.

2.4.1.Baseline of target: 50% of citizens (Digital Agenda) and 80% of businesses make use of eGovernment services

The Eurostat figure of the percentage of Europeans (EU27) aged 16 to 74 who had used the Internet in the last 3 months for interaction with public authorities was 31 % in 2010 33 . If extending to cover the last 12 months, the Eurostat figure collected in 2010 was 41 % 34 .  

Regarding advanced modes of interacting with public administrations, the Staff Working Document of the 2011 – 2015 Action Plan noted that in 2009 only 17% of EU citizens had downloaded official forms from a public authority website and only 12% had used the Internet to send back completed forms.

In 2010 76% of enterprises used at least one eGovernment service 35 . Take-up was also relatively high for advanced ways of interacting with the public administrations: 54% of enterprises used the Internet to return completed forms and 43% interacted with the public administration entirely using electronic transactions. 36

2.4.2.Baseline of target: a number of key cross-border services be offered online by 2015 (Digital Agenda)

The eGovernment Action Plan 2011 – 2015 Communication stated, describing the situation in 2010, that cross-border eGovernment services are few and, even where eGovernment services were offered, the majority of EU citizens were reluctant to use them. (In the Digital Agenda Communication in 2010 no specific baseline was given for the target of "by 2015 online availability of all the key cross-border public services contained in the list to be agreed by Member States by 2011".)

In 2010 the work towards cross-border services had started with three Large Scale Pilots (LSPs) co-funded by the EC under the "ICT Policy Support Programme" 37 as part of the Competitiveness and Innovation Framework Programme (CIP ICT PSP). PEPPOL 38 had started in 2008 to work towards a European system of eProcurement; STORK 39 had started in 2008 and aimed to enable citizens and businesses to identify themselves when interacting with their own and other national administrations over the internet; SPOCS 40 had started in 2009 and was related to the Services Directive. Around 36 million Euros 41 investment had been planned for in these three pilot projects at the time.

2.4.3.Baselines in the four policy priority areas

Data from 2010 is not available for the four policy priority areas of the Action Plan, because the benchmarking measurement of the four priorities started only with the new method as of 2012 with a first complete measurement available in 2013.

Some data relevant to the Action Plan 2011 – 2015 were given in the eGovernment Benchmarking report of 2010 42 that reported that Europe had made substantial advances in the 20 basic services. For businesses, a traditional 20 services assessment covered eight services for businesses. Out of these, the services ‘Registration of a company’ and ‘Obtaining an Environmental Permit’ were highly available online and displayed sophistication and full online availability scores of 90%/77% and 78%/63% for the EU27+ respectively. Furthermore, the eGovernment Benchmarking report 2011 measured the availability of different enablers in the EU27+ countries ( Figure 3 below)

Figure 3 Frequency of enablers in EU27+. Digitizing Public Services in Europe: Putting ambition into action. 9th Benchmark Measurement , 2010.

3.Evaluation Questions/Criteria

In order to structure the evaluation exercise, the current analysis sets out to answer a series of questions that will provide insight into the effects produced by the EU eGovernment Action Plan 2011-2015. The evaluation criteria used are from the EC’s Better Regulation Guidelines: effectiveness, efficiency, coherence, relevance and EU added value

Effectiveness analysis considers how successful the eGovernment Action Plan has been in achieving and progressing towards achieving the targets on the usage of eGovernment services and the availability of cross-border services, in addition to achieving the objectives of the four policy priorities on user empowerment, internal market, efficiency and effectiveness and pre-conditions. This criterion assesses the progress made and the role of the Action Plan in delivering the changes and/or the extent to which progress has fallen short of the target and what factors have influenced why something has not been achieved. Have users' needs been sufficiently catered for under the lifetime of the Action Plan?

The EU eGovernment Action Plan 2011 – 2015 did not have funding, but it served as a guide to prioritise investment from a number of funding programmes. How does the development and deployment of these projects support the overall aim of the eGovernment Action Plan? The efficiency analysis aims to, to the extent possible; consider the relationship between the resources used by the Action Plan and the changes generated by the funding intervention.

Relevance looks at the relationship between the needs and problems in society and the objectives of the EU eGovernment Action Plan: Was the vision in the Malmö declaration on which the eGovernment Action Plan is based appropriate for developing the EU eGovernment 2011 - 2015? Do the original objectives still correspond to existing needs within the EU, which of the objectives of the Action Plan continue to be still relevant and identifies also some new or changed needs.

The evaluation of coherence looks at how well different actions of the eGovernment Action Plan work together (internal coherence) and how the Action Plan relates to activities outside the Action plan (external coherence), including other EU policies. To what extent have Member States developed eGovernment services and policies in line with the overall principles set out by the current eGovernment Action Plan?

What is the EU added value resulting from the EU intervention compared with what could be achieved by Member States at national and/or regional levels? This assessment relates to the added value of the EU eGovernment Action Plan, why action at the EU level was necessary and what would have happened without an Action Plan? Which of the eGovernment policy priorities could not have been achieved without an EU-level Action Plan?

4.Method and sources

The evaluation has been coordinated by the EC's Directorate-General Communications Networks, Content and Technology with the support of an Inter-Service Group (with representatives of Commission Directorate-Generals Agriculture and Rural Development; Communication; Competition; Informatics; Economic and Financial Affairs; Employment; Social Affairs and Inclusion; Energy; Environment; Internal Market, Industry, Entrepreneurship and SMEs; Migration and Home Affairs, Joint Research Centre; Publications Office; Justice and Consumers; Mobility and Transport; Regional and Urban Policy; Research and Innovation; Health and Food Safety; Secretariat-General; Taxation; and Customs Union and Trade), which steered and monitored progress of the exercise, ensuring the necessary quality, impartiality and usefulness of the evaluation.

For the evaluation, a broad literature review was conducted covering various reports, EU studies and a number of international references. Important sources in the desk research were the mid-term evaluation of the eGovernment Action Plan 2011 - 2015 43 and the annual eGovernment Benchmarking reports. Furthermore, additional information and data (listed in the in the footnotes and in the "References and data sources" part in the end of this document) were used to answer the evaluation questions. Their methods are available in the original referenced reports. The eGovernment factsheets published on the Joinup platform 44 give an overview of eGovernment in each Member State.

A 12 week online public consultation was conducted to collect primary data and complemented by statistical data (Eurostat data and Digital Economy and Society Index DESI 45 ) and Digital Agenda Scoreboard reports 46 .

An effort was made to “triangulate” the data used throughout the analysis. This means that findings presented in the evaluation are supported by evidence from different data sources whenever additional data was available. Any contradicting evidence has been weighed according to its strength and quality before reaching conclusions.

4.1.Mid-term evaluation

The first monitoring of the implementation of the EU eGovernment Action Plan was done through the mid-term evaluation that carried out a monitoring and stocktaking exercise (progress assessment), with the purpose of providing an overview of the extent and modalities of implementation of the eGovernment Action Plan, its four priorities, 40 actions and governance chapter.

The Mid-term evaluation of the eGovernment Action Plan was carried out between December 2013 and June 2014 by external contractors. The eGovernment Action Plan states that "A midterm evaluation of implementation of this Action Plan will be conducted in 2013. "

4.2.Annual benchmarking reports

Since 2001, the Commission measures the progress made in the availability of on-line public services.. In 2012, this benchmarking was aligned 47 with the priorities of the eGovernment Action Plan 2011-2015. The benchmarking method examines a set of indicators related to each of the four political priorities of the Malmö Ministerial Declaration and the Action Plan. The latest benchmarking report 48 dates from June 2015. This 12th eGovernment Benchmark report is the third edition of the measurement made according to the new eGovernment Benchmark Framework 2012-2015 . Unless otherwise specified, the benchmarking report references in this evaluation refer to the "EU28+", which refers to 33 European countries: the EU, Iceland, Norway, Serbia, Switzerland and Turkey.

With regard to the methodology used for the 2015 benchmarking report, each of the priority areas of the eGovernment Action Plan was assessed by one or more top level benchmarks. Two methods were used for data collection: Mystery Shopping and User Survey 49 .

4.3.The online public consultation

The online public consultation gathered input from citizens, businesses and civil servants over 12 weeks from 30 October 2015 till 22 January 2016. Its objective was to inform and help define the new eGovernment Action Plan 2016 - 2020. The consultation was promoted through the eGovernment network (comprising 5000 entities), the EC website and on Twitter. 

The questionnaire also addressed the EU eGovernment Action Plan 2011 – 2015 as regards the improvement of cross-border eGovernment services and the success of the four political priories. The respondents were asked to provide answers in the form of agreeing to predefined statement on a scale from "not likely" to "very likely", in addition to a few open questions.

The results are provided in annex 2.

4.4.Limitations

The evaluation faced some limitations in the collection of data, whose impact was mitigated to a maximum possible extent:

Measuring the impact of the eGovernment Action Plan is a challenging exercise as it is almost impossible to isolate the impact of the Action Plan from other developments in the public sector, such as ICT-policies or the increased use of the Internet in general. Furthermore, some objectives of the Action Plan, such as increased collaboration, are non-tangible and, as a result, difficult to quantify. The evaluation has used benchmarking to measure the progress in the different policy priority areas of the Action Plan and assessed the effectiveness criteria qualitatively.

The Action Plan did not have a dedicated budget but relied on funding from other programmes for a large part of its actions. Other actions were financed by the Member States. As a result, the efficiency analysis (in particular the value for money assessment) was difficult to carry out.

Despite being prompted in a number of occasions by the EC, there is no systematic measuring or reporting requirement for the Member States of cross-border public services use. To overcome this, the evaluation used data from other sources, including case studies.

Given the multiplicity of the tools used to gather evidence, the results obtained are of different nature (for instance, eGovernment related data stems from the mid-term evaluation of the Action Plan and the benchmarking reports while the online public consultation provided insights in order to prepare for the new  eGovernment Action Plan 2016-2020).

The evaluation takes into account the inherent limitations of the findings of public consultations. Firstly, as in all surveys, the answers received reflect the views of a sample of relevant stakeholders and not those of the entire population who has a stake in this domain. Secondly, stakeholders' views convey an individual rather than a holistic perspective.

The mid-term evaluation did not systematically cover the five evaluation criteria. Furthermore, the methodology had strict requirements. For an action to qualify as "delayed" it was sufficient that one indicator is negative (e.g. one Member State answered ‘no’ to the question, or one action was delayed in one Member State) can sometimes give only a partial overview of progress and it should be understood as an evaluation of the actions as defined by the action plan, not an assessment of the policy priority as such. In addition, the mid-term evaluation did not capture the last 18 months of the Action Plan. This is partly mitigated using the 2015 benchmark study and partly the evaluation questions in the 2015/2016 online consultation on the eGovernment Action Plan and other recent reports and information.

Based on the elements above, the evaluation has been carried out on the basis of the best available data. Whenever reliable quantitative data is lacking, this is indicated as appropriate and possibly counter-balanced with qualitative data and considerations. 

5.Implementation of the EU eGovernment Action Plan 2011 – 2015 – state of play of the policy priorities

The monitoring of the implementation of the Action Plan was carried out by mid-term evaluation and supported by the annual eGovernment benchmarking. An assessment of the completion of actions will be done separately in a planned 2016 study on the assessment of the implementation of the Action Plan. The benchmarking reports are not intended to act as a benchmarking of the implementation of the Action Plan as such, but its results are informative for monitoring the progress of eGovernment in the EU.

5.1.General target: 50% of citizens (Digital Agenda) and 80% of businesses make use of eGovernment services

In 2014 eGovernment services were used by 47% of the EU population (38 % in 2009 50 ) 51 . The Eurostat figure 52 for individuals (aged 16 – 74) having used the Internet for interaction with public authorities within the last 12 months was 46 % in 2015 (41 % in 2010).

Figure 4 Individuals in the EU that had used the Internet in the last 12 months for interaction with public authorities.

eGovernment usage by firms (EU 28) has increased steadily from 76% in 2010 to 84% in 2011 53 and 88 per cent in 2013 54 .

5.2.General target: a number of key cross-border services be offered online by 2015 (Digital Agenda)

The eGovernment Action Plan was to contribute to two key objectives of the Digital Agenda: by 2015 a number of key cross-border services will be available on line – enabling a) entrepreneurs to set up and run a business anywhere in Europe independently of their original location, and b) allowing citizens to study, work, reside and retire anywhere in the EU. The Digital Agenda had specified that the key cross-border public services would be contained in a list to be agreed by Member States by 2011.

The identification of the key cross-border services was discussed at the Ministerial Roundtable in Poznan (Poland) in 2011 55 . EU Ministers responsible for eGovernment discussed a number of areas of the Digital Single Market from which potential cross-border services could identify for future roll out. The conclusions 56 noted that:

The Ministers concluded that cross–border eServices should be built from components that can be shared and re–used. The Large Scale Pilots have already contributed to this goals by developing building blocks.

The Ministers concluded that at least five key cross–border services should be selected and implemented between 2012 – 2014 where possible domains have been put forward for consideration. These cross–border eServices should be built from components that can be shared and re–used.

The Member States' eGovernment Expert Group later agreed to endorse a number of focus areas and key enablers for digital cross-border public services. Because of divergent views, Member States did not reach formal consensus on the exact services. However, the work continued through the The Competitiveness and Innovation Programme ICT Policy Support Programme 57 (CIP ICT-PSP) LSPs (STORK, SPOCS, PEPPOL, epSOS 58 , eCodex 59 and eSENS 60 ) towards the development of the building blocks that can be used by Member States to produce their own cross-border services according their needs. Most of the Member States have participated with co-funding in one or several of these projects to develop cross-border solutions for the public services.

At the mid-term evaluation it was noted that, "while not all Member States have identified their priority key cross-border public service, and not all Member States have implemented or are planning to implement cross-border services, there are several examples of Member States implementing cross-border services."

To support the work on cross-border services a detailed study 61 was concluded by the Commission in 2013. The study identified the cross-border services with the highest potential impact. In determining these services Member States were recommended to refer to the PESTEL (Political, Economic, Social, Technological, Environmental and Legal factors) analysis and estimated volume of users. Furthermore, the study noted that quick wins for Member States can be achieved by re-using the building blocks from the CIP ICT-PSP large-scale pilots.

The CEF Telecom Regulation 62  acknowledges the role of the CIP LSPs such as "PEPPOL, STORK, epSOS, eCODEX or SPOCS, in validating key cross-border digital services in the internal market, based on common building blocks, which are being consolidated by the project eSENS". The regulation's annex specifically identifies a number of building blocks:

Electronic identification and authentication

Electronic delivery of documents

Automated translation

Critical digital infrastructures support

Electronic invoicing

The CEF building blocks 63 that are now available are: eID, eSignature, eDelivery, eInvoicing and Automated Translation. The deployment of eDelivery, eID and eSignatures at the end of 2015 64 is described in the table below (excluding countries outside the EU and EEA):

Furthermore, the "Cross Border Mobility" benchmark of the eGovernment Benchmarking indicates to what extent European users can use online services in another country. This top level benchmark, which is split into Citizen Mobility and Business Mobility, measures the availability and usability of cross border services in a number of life events under four sub-criteria: availability, usability, ease of use and speed of use. According to the eGovernment Benchmarking Report 2015 65 , across Europe, on average, the score for the indicator of cross-border mobility for business is better than the citizen mobility indicator (58% as opposed to 43% for citizens). There is some improvement from the previous measurement (2012/2013, data not available earlier) from, respectively, 53% and 38%.

According to the eGovernment Benchmarking Report 2015, only 57% of the assessed services (life events) are available to cross border businesses and only 41% are available to cross border citizens 66 in 2014. This represents an improvement of 5 points for both groups in the assessed life events (compared to previous measurement 2012/2013, data not available earlier).

5.3.The four priorities and 40 actions of the EU eGovernment Action Plan

The 40 actions of the Action Plan (listed and numbered in Annex 4 to this document) were grouped under four priorities. The answers of each Member State (that answered to the survey) for each of the action in the mid-term evaluation of the eGovernment Action Plan are provided in a separate dashboard 67 . The last updates on the dashboard were in the first quarter of 2014. Non-implementation by one Member State was sufficient to mark an action either as delayed or at risk of delay. The mid-term evaluation was finished in June 2014.

The detailed implementation status of the 40 actions is provided in Annex 4 to this report. The status is provided on basis of the mid-term evaluation report findings. A number of actions were completed after the end of the evaluation and new progress in Commission actions is reported in Annex 4; detailed data from each Member States' progress after the mid-term evaluation was not available.

The mid-term evaluation concluded that of the 40 actions, 16 were completed at the time and six were on track. Three actions were at risk of delay and 15 delayed. The delay meant that an action had not been completed by one or several Member States or Commission within the deadline year and there was thus a delay from one to two years. The reasons and the possible impact (where an analysis was already possible) of these delays are assessed under the evaluation questions in part seven. The large majority of the EC’s 30 Actions were completed or were on track. Of these, there were 23 for which the EC has exclusive responsibility, while, for the remaining seven, responsibility was shared with Member States.

5.4.Governance

The High-Level Expert Group of Member State representatives was set up, and rules of procedures were adopted in 2011 (action 41).

According to the mid-term evaluation (2014) of the Action Plan, almost all Member States have a National eGovernment Strategy, which incorporates priorities from the Malmö Declaration and the eGovernment Action Plan (action 42). Only two Member States had not yet updated their national eGovernment strategies.

According to the mid-term evaluation, most of the Member States (17) have incorporated the political priorities of the Malmö Declaration and eGovernment Action Plan in their national strategies (action 43) and only four countries indicated in 2014 that this action was delayed in their case.

A mid-term evaluation of the Action Plan was carried out and its results and the results of this evaluation accompany the Communication on the EU eGovernment Action Plan 2016 – 2020 (action 44). Furthermore, Member States have informed the EC and the High-Level Expert Group on how the political priorities of the Malmö Declaration have been achieved, while six Member States (of the twenty-four responding in the mid-term evaluation) have not yet done so. Many countries have already provided some lessons learned within the implementation process (action 45).

6.Answers to the evaluation questions

6.1.Effectiveness

The effectiveness analysis assesses how successful the EU eGovernment Action Plan has been in achieving or progressing towards its objectives. What was the role of the EU eGovernment Action Plan 2011 – 2015 in delivering the changes? What factors have influenced the progress? To what extent progress fell short of the target?

The objectives to be analysed in this part are, first, the two targets of the Digital Agenda in Europe: (1) 50% of citizens make use of eGovernment services by 2015; and (2) that a number of key cross-border services be offered online by 2015. The Action Plan also aimed that "by 2015, 80% of enterprises will have used eGovernment". Secondly, the analysis includes the four policy priorities.

The mid-term evaluation of the Action Plan described a wide range of eGovernment effects across the different Member States with various case studies as examples, including examples in the area of user empowerment, open data, interoperability, eID and transparency.

However, the eGovernment Action Plan was a strategic plan to give the policies of EU Member States a direction towards commonly agreed goals. Without a detailed individual assessment of each action in each Member State it is not possible to associate the quantitative and qualitative effects on the Member States level directly to the Action Plan.

6.1.1.Objective on using eGovernment (take-up)

The 80% target for eGovernment usage by firms (EU 28) was achieved already in in 2011 (84%) 68 and it grew to 88 per cent in 2013 69 .

The action plan objective of "By 2015, 50% of EU citizens will have used eGovernment" was nearly achieved. In 2014 eGovernment services were used by 47% of the EU population 70 . The Eurostat figure 71 for individuals (aged 16 – 74) having used the Internet for interaction with the public authorities within the last 12 months was 46 per cent in 2015.

The achievement of the full 50 % usage by citizens was still expected in 2011. In the Digital Agenda Scoreboard report 2011 72 it was assessed that "the take-up of eGovernment in 2010 has reached 41 % of the EU population and, given the current trends, could be quite in accord with the target of the Digital Agenda of 50% of population using it by 2015." At the same time, the rise of the use of eGovernment services has also been driven by the increase of regular Internet users.

6.1.2.Key cross-border services online

During the Action plan duration, cross-border eGovernment services were addressed in the high-level political discussions and in the work of the LSP projects. This was a direct result of the eGovernment Action Plan. The Action Plan played a key role as a basis and political framework for bringing Member States together to discuss and agree on these initiatives.

The Action Plan objective was to contribute towards fulfilling the Digital Agenda objective: "By 2015, a number of key cross-border services will be available on line – enabling entrepreneurs to set up and run a business anywhere in Europe independently of their original location, and allowing citizens to study, work, reside and retire anywhere in the European Union."

The contribution towards this objective was achieved by LSP projects implemented under the CIP ICT PSP programme. The evaluation 73 of the ICT PSP Programme underlined, among other aspects, its innovative role as its projects design new platforms for innovative cross-border services in areas of public interest, in particular in areas where there are continuing systemic and organisational risks.

Through the financial support from the CIP ICT-PSP Programme to the LSPs' work on what has become the CEF building blocks, the Action Plan has contributed to achieving also regulatory objectives: the eIDAS Regulation and transition to the electronic public procurement.

The eIDAS Regulation 74 got important leverage from the STORK project that piloted an interoperability solution and that is referred to in the Regulation recitals.

In the new public procurement directives, e-procurement was one of the important changes introduced. The PEPPOL LSP (Pan-European Public Procurement Online) sought to make it easier for companies to bid for public sector contracts throughout the EU and provided supporting experiences and solutions for cross-border e-procurement.

In the context of cross-border services, the Points of Single Contact, required by the Services Directive, cover an increasing number of administrative formalities for businesses. The SPOCS LSP (Simple Procedures Online for Cross-Border Services) sought to improve the cross-border electronic procedures for businesses. In the environmental area, the EC has developed an EU "Shared Environmental Information System" (SEIS).

The aim of the Action Plan was to contribute towards the availability of cross-border services. To this date, life events for businesses and citizens are not yet covered by cross-border services and not all Member States have rolled-out the building blocks. For example, the objective that entrepreneurs should be able to set up and run business anywhere in Europe independently of their original location has not been met.

In the online public consultation (Annex 2) the majority of the respondents did not know whether the current eGovernment Action Plan has improved cross-border eGovernment services overall or did not agree that the Action Plan has improved them services. Around 25% of the respondents agreed that the Action Plan has improved cross border services overall.

Figure 5 Online public consultation: Has the current eGovernment Action Plan improved cross-border eGovernment services overall?

6.1.3.Effectiveness of the four policy priorities

The progress in four of the main areas (User empowerment, Internal Market and pre-conditions, key enablers/pre condition) is visible in the below figure ( Figure 6 ) from the 2015 eGovernment Benchmark background report "Future-proofing eGovernment for the Digital Single Market".

Figure 6 Future-proofing eGovernment for the Digital Single Market, eGovernment Benchmark report 2015. Top-level benchmarks EU28+ 2012/2013 versus 2013/2014 (%). (The top level benchmarks Effective Government and User Centricity (user survey) were measured once,

6.1.3.1.Effectiveness of the User Empowerment policy priority

In the Action Plan, empowerment was referring to increasing the capacity of citizens, businesses and other organisations to be pro-active in society through the use of new technological tools. Governments should provide easy access to public information, improve transparency and allow effective involvement of citizens and businesses in the policy-making process.

According to the mid-term evaluation 75 , the actions under the User Empowerment priority require further attention. Important targets have been achieved in implementing inclusive services, in involving citizens and business in the policy-making process, and notable progress has been made in creating collaborative services. There are also good achievements of the Action Plan targets under sub-priority on re-use of public sector information. At the same time, there are delays in general in drawing up and adopting common targets, indicators and measurement frameworks. Details of the achievements under each of the 14 actions under this priority are in Annex 4.

Services designed around users' needs and Inclusive Services

The Action Plan has supported exchanges of practice in the field of user-centred, inclusive and accessible eGovernment services (through the Joinup 76 platform), though no agreement had been reached with Member States on common targets and evaluation criteria for those services.

The Commission adopted its proposal for a directive on the accessibility of public sector bodies' websites on 3 December 2012 77 . The MeAC (Measuring progress of eAccessibility in Europe) reports provided evidence and analysis to help understand and compare the approaches followed by the European countries, with a view to identifying issues and challenges, good practices and future priorities in the web accessibility field 78 .

According to the mid-term evaluation report, most Member States have introduced personalised services in their One-Stop-Shops for business and for citizens, as well as in the area of health and tax services. The majority of the Member States that have already introduced personalised services offer them via multiple channels, such as mobile access via mobile apps and helplines.

Re-use of Public Sector Information (PSI)

Under the Action Plan priority on re-use of public sector information, among other things (details in Annex 4), the Commission has reviewed and subsequently revised the PSI Directive improving re-usability of data held by Member States' authorities and has put in place an Open Data Portal for data of the EU institutions 79 . The portal, launched in 2012, has currently more than 8000 datasets available.

The Action Plan achievements under the sub-priority Re-use of Public Sector Information (see Annex 4 and achievements under Action 6, 7 and 8, including the review of the PSI Directive and the set-up of the Open Data Portal) are good, apart from the formal achievement on agreement on indicators. The Action Plan has supported the achievements and has given them political visibility. There appears to be increased awareness of the importance, including economic importance, of open data as shown, for example, by the adoption of the G8 Open Data Charter and the commitment of various EU Member States to the Open Government Partnership.

The study "Open Data Maturity in Europe 2015 80 " reported that 27 countries (of EU28+) have a national Open Data portal and an open data policy is in place in 71% of the countries, often as part of a more generic Digital Strategy or eGovernment programme. It is also worth noting that beyond the objectives of the Action Plan, but supporting the sub-priority on re-use of public sector information, the EU endorsed the G8 Open Data Charter in 2013 and committed to promoting the application of the principles of the G8 Open Data Charter to all EU Member States within the context of a range of ongoing activities 81 .

Improvement of Transparency

On the sub-priority on Transparency the mid-term evaluation found that the actions under transparency were delayed. Fourteen Member States were using some transparency targets and most Member States provide online access to information on government laws and regulations (see Annex 4 for details). Member States also provide access to citizens' to their personal data, but not all provide it in electronic form as seen in the Action Plan (30% of Member States did at the time of the mid-term evaluation).

At the same time, the eGovernment benchmarking report 2015 82 stated that the transparency benchmark shows improvement from the previous measurement, but is still quite unsatisfactory, as it is at 51%. The Transparency indicator of the benchmarking report examines the extent to which governments are transparent about their own responsibilities and performance, the service delivery process, and the personal data involved 83 .

It is also positive that users have gained better access to personal data that is handled on the governments’ websites, but they still face considerable barriers when it comes to the clarity of the service delivery process. The Action Plan supported the delivery of a number of transparency objectives, but this remains an area for continued work.

Involvement of citizens and businesses in policy-making processes

In the sub-priority Involvement of citizens and businesses in policy-making processes progress is very good according to the mid-term evaluation (see Annex 4 for details). The Action Plan supported and gave a guiding framework for the work for governance and policy modelling under the Seventh EU Framework Programme for Research (FP7) and for eParticipation projects that were supported under the Competitiveness and Innovation Framework programme (CIP).

Not reaching an agreement on common targets was reported by the mid-term evaluation to be an issue requiring further attention in relation to the collaborative production of services, PSI re-use indicators and targets and indicators on transparency.

When looking into reasons of why common targets or indicators were not agreed in the different areas, it should be noted that sometimes similar activities were already taking place elsewhere, sometimes making the agreement under the Action Plan unnecessary, or similar issues were discussed elsewhere:

The Web Accessibility Directive proposal (2012) aims to approximate the laws, regulations and administrative provisions of the Member States on the accessibility of websites of public sector bodies. A directive proposal goes beyond the (non-binding) Action Plan target of agreeing on common targets.

For the collaborative production of services, workshops were organised to discuss recommendations but these did not lead to agreement on common targets for the roll-out.

For the PSI re-use indicators, the majority of the Member States agreed as to the usefulness of the indicators of the PSI Scoreboard 84 . However, given the existence of other benchmarking tools (Global Open Data Index, Open Data Barometer) and the fact that the PSI Directive does not mandate adopting specific indicators, it was difficult to ensure a uniform adoption of any given set of indicators for all EU Member States. Furthermore, the Europan Data Portal 85 , has developed its own approach and conducts a 'landscaping study' every year to measure open data maturity across Europe.

For transparency, the mid-term evaluation found that some Member States were using the Open Government Partnership indicators.

Looking beyond the specific actions of the eGovernment Action Plan to the broader development of user empowerment in the EU with regards to eGovernment services, the eGovernment Benchmarking Report 2015 indicates that governments’ efforts to improve the quality of the online experience have focused on increasing the availability of their online services and on improving the mechanisms for online support and feedback. Users are also more empowered with medium increases in User Centricity and Transparency. Their performance is weaker at the user friendliness and time efficiency of the online services.

The eGovernment Benchmarking report 2015 indicates that tests of 'user centricity' (to what extent a service is provided online and how it is perceived) show good results with a score of 73 out of a possible 100 points 86 across the EU28+. There is, however, a big difference between the compound indicators of user centricity, with much better performances for usability and online availability of services than for the ease and speed of using those services. While availability was up six points in 2014 (48% of all services are completely online and available through a portal 87 ), ease and speed of use did not change between 2012 and 2014. This indicates that many Member States are not focusing enough on the quality of the user’s experience 88 . This again suggests, according to the 2015 benchmark report, a trend of quantity over quality. The differences between citizens and businesses are not too large, suggesting that while there are significantly more information and services available to businesses, the quality of these does not differ much from citizen oriented eGovernment.

The online public consultation results reflect low awareness about the work carried out in the User Empowerment area. For User Empowerment, 27% of the respondents assessed that the measures had been successful; 48% said that these measure of the action were not successful and 24% did not know. It should be noted that the Action Plan activities were typically not directed directly towards citizens and businesses, but activities were carried out by the Member States and the Commission with a view to, for example, creating enabling conditions which are not directly visible to individual citizens as eGovernment Action Plan activities.

6.1.3.2.Effectiveness of the internal market policy priority

The internal market dimension of the Action Plan covers some of the most relevant accomplishments in the advancement of the cross-border delivery of services for citizens and businesses. The Action Plan supported the internal market objective through activities in "seamless services for businesses", "personal mobility" and in "EU-wide implementation of cross-border services". The detailed results are included in annex 4 of this report.

According to the mid-term evaluation of the Action Plan, most of the Member States have incorporated the political priorities of the Malmö Declaration, which is the basis of the Action Plan, in their national eGovernment strategies. As a consequence, aspects of internal market related eGovernment activities have been included in national strategies.

The role of the Action Plan in driving the cross-border services was discussed above in 7.1.2. The effectiveness of the work of the LSP projects was acknowledged also in the Commission report 89 on "Evaluations of the Competitiveness and Innovation Framework programme" where the evaluation panel recommended that the top-down policy-driven approach in the LSP projects (Pilot A) should be continued, as it is a working model of how to enhance more widespread uptake of new innovative services at EU level. 90

Also in e-procurement the achievements went beyond the objectives of the Action Plan. In addition to the objective stated in the latter, the mandatory transition to e-procurement is driven by the new procurement directives and facilitated with a wide array of policies and support measures.

Consecutive assessments of the Points of Single Contact in 2011/12, 2013 and 2014/15 show continuous but slow progress towards fulfilling the requirements of the Services Directive and becoming an effective tool for further integration of the Single Market. The most visible improvement has been made with regards to the availability of online procedures, although the gap is still significant. Furthermore, efforts to improve general usability of the web portals and accessibility to foreign users are recorded, but still insufficient to produce needed effects. 91

As for the effectiveness of cross-border eEnvironment services, the Mid-Term technical evaluation 92 on the implementation the INSPIRE Directive lists a number of effective eEnvironment cross-border services already at the mid-term implementation stage of INSPIRE. It emphasises that effective cross-border eEnvironment services will depend to a large extent on further ensuring the coherent and timely implementation of INSPIRE across the Member States in the 2016-2020 period.

Going beyond the actions of the Action Plan, the eGovernment Benchmarking Report 2015 measured the cross-border mobility of businesses through studying the life event "Starting up a business". Scores on all four indicators (speed and ease of use, online availability and usability) are generally much higher than those for other life events. The score on Usability is excellent at 81% but the scores for Online Availability, Ease of Use and Speed of Use leave room for improvement. 93

Personal mobility

In the Action Plan sub-priority on personal mobility activities 94 have been undertaken to support exchange of good practices and to coordinate the Member States’ efforts to jointly develop and set-up cross-border and interoperable eDelivery services. The Action Plan set-out the framework and commitment for this voluntary work. According to the mid-term evaluation of the eGovernment Action Plan, more than 50% of the Member States already provide or are planning to provide those services. The CEF Telecom programme has financed the deployment of the cross-border eDelivery service.

According to the eGovernment Benchmarking Report 2015 95 , eGovernment services are still far from giving adequate support to citizens moving to another EU country. The cross-border mobility compound benchmark (beyond the actions of the Action Plan)_measures the ease and speed of use, online availability and usability of online public services for nationals of another EU country giving a score of maximum 100. This stands at 58% for businesses and at 43% for citizens 96 . Although there is improvement from previous year's benchmarking report (from, respectively, 53% and 38%), these figures are lower than the corresponding ones for domestic services. Cross-border transactional services (where a citizen can complete an entire process online) are rare. At the same time, significant increases of both the Business Mobility and Citizens Mobility score of the benchmarking report indicate that it is gradually becoming easier for people working across borders to access eGovernment services. 

6.1.3.3.Effectiveness of the "Effeciency and Effectiveness" policy priority

According to the eGovernment Action Plan mid-term review significant progress was being made in improving organisational processes with a "great deal of activity within Member States both at national and local level, focusing on better coordination and cooperation across different public organisations, and therefore improved governance" 97 . The Action Plan played an essential role in supporting the exchanges of experience and the sharing of new approaches through the Expert Group meetings, workshops and the Joinup (ePractice) portal. The programme for staff exchanges between administrations in different Member States was not carried out given the current budgetary cuts due to the economic situation.

In the area of the reduction of administrative burden, the Action Plan included a number of actions related to the once-only principle. The sharing of experiences was done in a dedicated workshop and at the High-Level Expert Group meetings. A cost-benefit analysis study was carried out 98 and it gave important input to continuing the work on the once-only principle. The study provided also a roadmap for implementation of the once-only principle. The once-only policy gained further momentum through its inclusion in the Council Conclusions 2013 99 . This supported further steering of Commission policy and resources towards the once-only principle. The principle was included in the Digital Single Market strategy and a call for a new LSP project focusing on EU-wide once-only principle for businesses was included in the H2020 Work Programme 2016 – 2017 100 adopted in 2015. The effects of the Action Plan in went beyond the original objectives, and were influenced by the increased political visibility.

In the area of Green Government, the Action Plan did not succeed in achieving its objectives in assessing the possibilities of eGovernment in reducing the carbon footprint. The study was not carried out due to insufficient data (study on the potential of eGovernment to reducing carbon footprint of government). The mid-term evaluation also found that although there were interesting developments in the area of sustainability, most Member States did not have indicators or evaluation procedures in "Green Government". An agreement on indicators and evaluation procedures for measuring the reduction of carbon footprint as a result of eGovernment between Member States would not have been realistic.

In 2015 the REFIT Fitness Check of the environmental reporting process 101 was launched contributing to the reduction of the carbon footprint of governments. It aims to “Launch a broad review of reporting requirements to see how burdens can be alleviated. This review will have a particularly strong focus on areas where stakeholders have recently indicated their concerns, such as agriculture, energy, environment and financial services." With regard to environment the technical Mid-term evaluation report 102 on the INSPIRE implementation emphasised the need to prioritise in the further implementation period of INSPIRE the connection between EU environmental reporting obligations and INSPIRE so as to reduce overall administrative burden and make reporting easier and more efficient for the Member States.

6.1.3.4.Effectiveness of the preconditions policy priority

The EU eGovernment Action Plan's policy objectives for the preconditions priority supported putting in place technical and legal pre-conditions for eGovernment. The mid-term evaluation noted that coordination of national efforts to support the development of interoperable solutions and key enablers is ongoing. The actions that have been supported by the Action Plan include:

European projects (such as PEPPOL and other LSPs) piloted shared cross-border solutions that many Member States are implementing;

The alignment of Member States with the European Interoperability Framework (EIF) (combined with a revision of the European Interoperability Strategy (EIS)); The European Interoperability Framework promotes and supports the delivery of European public services by fostering cross-border and cross-sectoral interoperability and has strongly contributed to the Action Plan objectives in the preconditions priority.

The adoption of a Regulation on Mutual recognition of Identification and Authentication; with the eIDAS Regulation and the STORK solution the eGovernment Action Plan proved very effective. To complete the legal framework, the Commission adopted several implementing acts under eIDAS Regulation.

Launch of pilot projects for innovative architecture and technologies in eGovernment; under sub-priority Innovative eGovernment, the mid-term evaluation assessed the progress as very satisfactory.

On-going work on the deployment of the CEF building blocks.

The Action Plan pushed Member States administrations towards the deployment of the pre-conditions, as shown, for example, in the EIF alignment figures ( Figure 7 ) below and in the eID roll-out figures that indicate that 13 Member States are connected to the eID network via STORK (details are in Annex 4 under action 37).

However, in the online public consultation, the majority of the respondents did not know the measures in the pre-conditions area or rated them not successful. At the same time, 46% of public administrations that responded to the survey rated the measures related to the preconditions for developing eGovernment successful.

As an example of effective work supported by the Action Plan, the Member States national interoperability frameworks were aligned to the EIF ( Figure 7 ). This has helped in achieving a common understanding of interoperability in the public sector and how to achieve it.

Figure 7 NIF-EIF alignment overview for each country assessed. State of play of interoperability in Europe – Report 2014.

Some of the above interoperability activities, like the implementation of the EIF, STORK or PEPPOL were supported also by the ISA programme which has also funded various interoperability activities. It has contributed substantially also to the implementation of a number of activities in the Maintenance and Implementation Work Programme of INSPIRE. INSPIRE is an important element in cross-sector interoperability between public administrations.

The eGovernment benchmarking report 2015 found that in the key enablers measured in the report (measuring also enablers beyond the Action Plan), there was only a marginal gain in 2013/2014 compared to 2012/2013 103 in the EU28+. Europe has been slow in terms of adopting key enabling technologies for supporting public online services. Some Member States score well with some of technologies but less for others showing that countries have different priorities in the adoption of these enablers. Most countries showed almost no progress from 2012-13 to 2013-14. The most common of these technologies (eIdentification) was deployed in 63% of the cases examined in the eGovernment benchmarking report in 2014.

6.1.3.5.Effectiveness of the "Governance" actions of the Action Plan

Important effects of the Action Plan arise from the governance chapter of the eGovernment Action Plan. These included the creation of an eGovernment High-Level Expert Group and the update of national eGovernment strategies incorporating the principles of the Malmö declaration. Actually, in most Member States, the national eGovernment strategies incorporate the Malmö priorities (see above 6.4). The effect has been that in most Member States the national eGovernment strategies thus include Action Plan's political objectives. The mid-term evaluation gives some examples that show how the national eGovernment work has been influenced by the Action Plan 104 . The Expert Group has been a forum for exchange of information and discussion on eGovernment EU-wide.

6.2.Efficiency

6.2.1.Efficiency of the related budget

The eGovernment Action Plan had no dedicated budget, therefore efficiency related to budget may only be linked to actions under the Action Plan for which spending has been allocated under an EU funding scheme. For the same reason, no systematic monitoring of resources has been carried out for each of the priorities of the eGovernment Action Plan. Likewise, across the 28 Member States the implementation of the Action Plan has been carried out in a variety of different manners on the national level.

eGovernment is supported through a number of funding programmes (see also under 3.2.2.):

The past Competitiveness and Innovation Programme ICT Policy Support Programme 105 (CIP ICT-PSP);

CEF, Digital Services Infrastrctures (CEF DSI) 106 ;

Horizon 2020's Societal Challenge 6 107 ;

The past Seventh EU Framework Programme for Research (FP7);

The ISA programme 108 ; 

The European Structural and Investment Funds ESIF (ERDF, CF, ESF, YEI and EAFRD 109 ); before 2014: Structural Funds.

CIP LSP projects

The CIP ICT Policy Support Programme funded the following LSPs projects (with the approximate total costs, including the 50 percent EU co-funding): STORK (eIdentification – 26 + 18,6 M EUR), eCodex (e-justice – 24 M EUR), PEPPOL (e-procurement – 30,8 M EUR), SPOCS (Points of Single Contact – 24 M EUR), ePSOS (e-health – 36,5 M EUR) and eSENS (consolidation of results – 27 M EUR). In the analysis below, the eHealth related activities and pilot are excluded as they are covered by a separate eHealth Action Plan 110 .

The CIP Implementation reports 111 provide overviews on the LSPs. The total cost of the eGovernment LSPs (excluding ePSOS) is around 150 million EUR. The Commission has co-funded around 50 per cent of this amount the other half was invested by the participating Member States (and countries associated to the CIP programme). The Commission report 112 on "Evaluations of the Competitiveness and Innovation Framework programme" referred to the final evaluation 113 of the CIP programme: "The evaluation underlined the improvements arising from the delegation of the management of substantial parts of CIP to the EACI 114  concerning the efficiency of the programme management, both in relation with costs to the Commission and efficiency of the services provided, as measured in terms of number of contracts signed, the period to contract and payment delays."

The results achieved through these pilots in the form of re-usable technical building blocks that are now starting to be deployed in the EU have been described above in 7.1.2. The work contributed towards having key cross-border services online. The participation to these pilots was (and is) voluntary of which reason the costs varied among the Member States. The pilots typically involved between 10 – 20 Member States.

The 2013 study "The feasibility and scenarios for the long-term sustainability of the LSPs" included a cost-benefit analysis of the large-scale pilots. Cost-benefits analysis calculations were underpinned by an evidence based estimation of the potential use levels for the nine core service platforms (CSPs) and building blocks (BBs) - eID, eProcurement, eBusiness, eHealth, eJustice, eSignature, eDocument, eDelivery. The role of the public sector in financing the building blocks and CSP/BBs may diminish where a business for private sector involvement is more apparent.

ESIF

ESIF funding (as from 2014) for eGovernment can be assessed by extracting the amount of potential investments (this does not necessarily reflect final investment figures) in the intervention category "eGovernment services and applications" (including e-procurement, ICT measures supporting the reform of public administration, cybersecurity, trust and privacy measures, e-justice and e-democracy) 115 . For 2014 Operational Programmes, the amount is 3,427 million EUR potential investments.

Research and Innovation funding

The Action Plan has also led to financing initiatives under the Research and Innovation programmes of the Commission. In particular, research and innovation actions were targeting the understanding the transformation of public administration in engaging more stakeholders in the policy making process, the reduction of administrative burden and the service delivery of public services. Under the FP7 the programme invested over 75M EUR between 2011 and 2013 in co-financed research and innovation actions. Since the launch of the new programme, H2020, over 27M EUR were invested in co-financed research and innovation actions.

6.2.2.Costs and benefits to stakeholders

The efficiency can also be assessed as the capacity of the Action Plan to mobilise resources (human and financial) versus the potential savings in the intervention domain. Investments in the modernisation of the European public sector have to be seen also in the wider context.

In the EU, public expenditure accounts for almost 50% of GDP and the public sector represents about 17% of total employment 116 .

Public administrations have a powerful means to pull innovation; in the EU, in 2010, public authorities in the EU spent over € 2 400 billion on supplies, works, and services – amounting to around 19% of EU GDP 117 .

The potential of these investments for savings can also be seen in different sectors mentioned in the Action Plan, for example:

eIdentification
It has been assessed that replacing the current paper based identification elements used across the private sector could reduce costs by 90% if eID replaced them 118 . In addition, eID is also the most important service to access other digital public services, a key enabler.

e-procurement
Contracting authorities and entities that have successfully completed the transition to e-procurement commonly report savings between 5 and 20%; whereby experience also shows that investment costs related to transition to e-procurement can be rapidly recouped. Given the size of the total procurement market in the EU, each 5% saved could return around €100 billion to the public purse 119 . Based on the extrapolation of the national evaluations of benefits, the EC estimates that the adoption of e-invoicing in public procurement across the EU could generate savings of up to €2.3 bn 120 . Switching from paper to fully automated e-invoicing can cut the costs of receiving an invoice from €30-50 to €1 121 .

The once-only principle

The extension of this principle, in compliance with data protection legislation, would likely generate an annual net saving at the EU level of around EUR 5 billion per year by 2017 122 .

As reported in the mid-term evaluation, for the Member States public administrations, achieving this type of financial gains and improving the efficiency of administrative processes were among the main drivers for eGovernment projects, even more so given the current economic climate and budget pressures. Financial savings are expected to translate into efficiency gains in a variety of ways, including a reduction of administrative burden and savings in opportunity costs.

The mid-term evaluation found in the eGovernment case studies that financial impacts are not always clear before implementing a project. In general, the mid-term evaluation found that the decision to implement a project depended in only a few cases on a clear business case.

At the same time, the mid-term evaluation found that it was uncommon to have users of the service paying a fee for it. In fact, financial benefits for the public sector are expected to come mostly from savings generated by reducing the production costs for a service and/or by the rationalisation of public expenditure. This meant that for the other stakeholders, citizens and businesses, the Action Plan activities did not incur direct costs and they were able to benefit from, for example, improved cross-border availability of eGovernment services.

6.3.Relevance

The public sector represents a significant and unique part of Europe’s economy, has a key responsibility for delivering citizens' quality of life and is also a vital actor in efficiently and effectively delivering policy objectives.

Economic and budgetary pressures force governments to be ever more efficient, effective, reduce costs and be more competitive in a multi-polar world 123 . The, contribution of eGovernment to jobs and growth remains a very relevant high-level objective. Repeated analysis during the European Semester has shown that the modernisation of public administrations can contribute to tackling economic challenges. The recent Annual Growth Surveys 124 recognise the importance of a modern and efficient public administration for a business-friendly environment and to ensure fast and high-quality services for citizens.  

Additionally, in many EU countries the work to implement digital end-to-end services and to achieve cross-border interoperability continues, so further pursuing the eGovernment activities remains relevant. Online public services continue in relevance for reducing business costs and increasing the efficiency and the quality of the services provided to citizens and businesses 125 . According to a recent study 126 , if interactions with public authorities can be made as transparent, as fast and as cost-efficient as in the private sector, then the potential benefits will materialise.

6.3.1.Relevance: User Empowerment

As noted earlier, according to the 2015 eGovernment Benchmarking report, user centricity is confirmed as the most advanced indicator. However, many Member States are not focusing enough on the quality of the users' experience.

According to the mid-term evaluation (on Action 2), most Member States have introduced personalised services in their One-Stop-Shops for business and for citizens, as well as in the area of health and tax services. The relevance of one-stop-shops continues with new needs for cross-border approaches..

The Action Plan objectives on re-use of public sector information have been found to be relevant. The Action Plan's objectives on user needs, inclusive services, collaborative services, transparency and the involvement of citizens and businesses in policy-making were assessed to be appropriate for user empowerment. This is also due to advancing technology that brings, for example, increased interactions through new tools and social networking.

Transparency and participation continue as important principles in the modernisation of public administrations. According to the eGovernment benchmarking study users are provided with information about the duration of the process in only 39% of measured cases. The figure for maximum delivery timelines for government is 46%. At the same time, administrations rarely give an account of their service performance (in 38% of cases). Only 1 in 3 websites inform visitors about their ability to participate in policy-making processes 127 .

Users of Member States' justice systems are an example of user empowerment through new technological tools Here, there are currently gaps regarding the deployment of e-justice or more specifically the use of ICT at courts, for example, concerning the possibility to communicate electronically between the courts and the parties as well as the use of ICT in court management system as identified by the EU Justice Scoreboard 128 .

The online public consultation also asked how stakeholders should be enabled to contribute to, make proposals on and publicly exchange their views on new initiatives emerging under the eGovernment Action Plan. Most respondents said that using an online social media platform was a good idea. Using a social media platform was viewed as useful to reach out to a large audience, citizens as well as businesses. 

The Action Plan objective on the electronic service to support citizens' initiatives was achieved. However, the general goal of involvement of citizens and the rights of citizens continues to be relevant. According to the Citizenship Report 2013, only one in three citizens (36%) say they are well informed about their EU rights and just under a quarter (24%) feel fairly or very well informed about what they can do when their EU rights are not respected 129 .

Building on the achievements of the Action Plan on the sub-priority on the re-use of public sector information further work was relevant and continues to be so. The mid-term evaluation finding of the lack of agreement on the common set of Public Sector Information (PSI) re-use indicators can also feed into future work, including in possible future adaptations of the PSI Directive 130 . The policy evaluations of the PSI Directive 131 and the Infrastructure for Spatial Information in the EU (INSPIRE) Directive 132 demonstrated the need for public authorities to continue improving the sharing and re-use of their data.

Despite challenges remaining in implementing the INSPIRE Directive, both the Member States and the consulted stakeholders reported qualitative benefits ranging from more efficient information services to citizens and the improved collaboration between public authorities, as well as building up skills, to cost savings in various areas and improved data quality. Estimates are available for a number of countries 133 , 134 . Other illustrations are more effective environmental risk management 135 , improved cross-border and cross-sector collaboration, spatial planning and lower costs for environmental impact assessments 136 . Moreover, synergies 137 were reported between the EU and national strategies on Open Data 138 and eGovernment where the availability of INSPIRE spatial data and services is used increasingly for a wide range of government-to-government and government-to-public applications 139 .

The Commission has put in place an Open Data Portal where re-usable data from the EU institutions is are referenced 140 and has currently more than 8000 datasets available. As a next step to opening up its own data, the Commission is actively promoting the discoverability and re-use of PSI in a cross-border and cross-lingual context, notably by the creation of the European Data Portal 141 , a digital service infrastructure under CEF. The importance of open data and re-use of data continues relevant for contributing to transparency and for economic growth. According to a recent study "Creating Value through Open Data" 142 the market size for Open Data is expected to increase by 36,9 % between 2016 – 2020 to a value of 75,7 bn EUR in 2020. The study also forecasts and a 32 % growth in open data jobs in the 5-year period and important cost savings.

The relevance of the Action Plan sub-priority on transparency, including data protection has continued to increase. In addition to the consent-based approaches, users demand to be in control of their data and provide only the amount of data necessary for a certain transaction. The future General Data Protection Regulation on which co-legislators reached a political agreement 143 in December 2015 updates and modernises the Data Protection rules currently contained in the 1995 Data Protection Directive and will have an effect also on the public sector.

6.3.2.Relevance: Internal market

The Action Plan objectives of seamless services for businesses, personal mobility and EU-wide implementation of cross-border services still correspond to the needs of the single market. The Digital Single Market of over 500 million people cannot be completed without public services. In 2014, there were 17.9 million persons who had been born in a different EU Member State from the one where they were resident 144 . From the perspective of a citizenship, the figure is still high, in 2014 more than 14 million EU citizens lived in another EU Member State than where they had their citizenship 145 . The trends of intra-EU mobility are steadily increasing 146 . According to a 2013 study 147 , over 1,5 million citizens and 314,000 businesses are likely to use cross-border eGovernment services each year in 2020. The same study estimates that the potential volume of the top five public services for businesses is 2,490,000 cross-border users annually.

The mid-term evaluation of the Action Plan noted that under half of Member States had rolled out LSP projects, which highlights the need for continued action with the CEF DSIs.

There are also increasing needs for cross-border company procedures and the related objectives are still highly relevant, for example in the areas of e-procurement, company law, e-justice, Points of Single Contact and information portals:

E-procurement

As outlined above, transition to the e-procurement presents a potential of significant savings enabled by streamlining, simplification and opening up of public procurement procedures. Only adoption of e-invoicing in public procurement across the EU could generate savings of up to €2.3 billion 148 . The work in this area delivered beyond the objectives of the Action Plan. In 2014, the European Commission adopted new public procurement directives, which enter into force in April 2016 149 and make e-procurement gradually compulsory, with e-Submission being mandatory for all public buyers as of October 2018. The policy aiming at supporting transition to full e-procurement is also building on a 2012 Communication on "A strategy for e-procurement" and a 2013 Communication on "End-to-end procurement to modernise public administration" 150 . A wide range of support measures have been put in place to facilitate the transition to e-procurement in all Member States. This will allow the businesses, in particular, SMEs, to benefit as early as possible from new business opportunities opening up at the European Single Market thanks to the electronic public procurement, in particular simplification of the pre-qualification phase.

Company law

The objective of the 'internal market' priority of the action plan aimed at facilitating the running of companies anywhere in the EU. The Digital Single Market Staff Working Document 151 noted that existing rules under company law do not sufficiently integrate the benefits of digital technologies. Companies are still faced with paper-based formalities. For example, online registration of companies is only possible in 16 Member States. Though a number of measures have been taken at national level, a national approach does not remove the obstacles that companies face if they consider setting up or operating a company across borders 152 . In this context, the Single Market Strategy 153 further mentioned that the Commission will consider further ways of achieving simpler and less burdensome rules for companies — while continuing to act against letterbox companies — including making digital solutions available throughout a company’s lifecycle, in particular in relation to their registration and to the filing of company documents and information.

E-justice

The action plan foresaw in the 'EU wide implementation of cross border services' activities in the eJustice area as there are continuing needs in cross-border e-justice. The Digital Single Market strategy (Staff Working Document) noted that European e-Justice 154 facilitates access to justice and cross-border judicial proceedings and makes it easier for citizens to find a lawyer/notary in the EU and for businesses to search for insolvent entities through interconnected insolvency registers. However, in a cross-border context issues identified at national level are amplified, as cross-border cooperation in this area entails that the Member States are at equal levels of development.

In this context, the work started by the CIP pilot e-CODEX remains highly relevant. Although the pilot project has created a strong backbone for communications between justice administrations, which can be used for many different purposes, extension of this work will be needed in three dimensions. Firstly, the number of Member States which currently participate in this work remains limited, and should be expanded to cover all Member States. Secondly, the e–CODEX backbone provides many opportunities to further support cross-border exchanges, including between judicial professionals such as laywers, notaries and judicial officers. Thirdly, expansion can also take place in terms of the number of judicial cooperation instruments covered. A first pilot project implementing this technology on the European e-Justice Portal will go live in 2016. Further work on this topic could include efforts to realise the cross-border elecronic service of documents. Furthermore, as of 2017, the business registers interconnection system, which will be composed of Member States' business registers, a European central platform, and the European e-Justice portal serving as the European electronic access point will facilitate access to information on companies for citizens and businesses throughout the EU 155 . In addition, the interconnection of national land registers will be developed to go live on the European e-Justice Portal in the course of 2017.

Points of Single Contact

Extending the scope of the Points of Single Contact as indicated in the Action Plan could further contribute to simplification, savings for public administration and more coherent approach in providing information and e-services to businesses. Information gaps increase costs for business, in particular for SMEs 156 . Procedural streamlining via Points of Single Contact (PSC) could generate up to 0.15% of GDP in the medium run (5-year horizon) and up to 0.21% of GDP in the long run 157 .

In line with Action Plan action 18 to transit into "Second Generation" Points of Single Contact (PSC), that would function as fully-fledged eGovernment centres, a first assessment of the PSCs was carried out in 2011-2012 158 . The results revealed a hybrid landscape, with some PSCs more advanced than others in providing the information and services required. This first assessment resulted in the Communication on the implementation of the Services Directive "A partnership for new growth in services 2012-2015” 159 . This Communication encouraged Member States to develop by the end of 2014 Second Generation Points of Single Contacts, which should “1) cover all procedures during the business life cycle, 2) be multilingual, and 3) be more user-friendly”.

These recommendations were subsequently included into the PSC Charter 160 , endorsed by the Council in 2013. The Charter encourages the Member States to bring additional features into their PSCs so that the latter respond more effective to the business life cycle of a company in the services sector. It also includes the benchmarks facilitating the assessment of the PSCs with a view to ensure their continuous improvement and ultimately, to provide better public online services to business. The assessment of the PSCs performance against of the criteria set out in the Charter was carried out in 2014 and published in 2015 in a study 161 . The results of the study indicate that PSCs are still far from delivering what is expected from them. The overall score of the PSCs on the EU28+ level indicates moderate performance with considerable room for improvement. Although PSCs provide basic information on general requirements, information on sector specific requirements is insufficient. The study reveals also a considerable gap in online availability of the procedures associated with general requirements and the ones associated with specific requirements. Although the general requirements are fully transactional on about half of the PSCs. The access to PSC services by foreign users were assessed as poor. The transactionality of online procedures for foreigners is one of the weakest points of the PSC.

Portals and networks

Furthermore, there are a number of existing portals and networks for finding information and initiate and complete transactions with Member States’ administrations across the EU (including Points of Single contact, Product contact points, Construction product contact points, Your Europe, Your Europe Advice, SOLVIT and Enterprise Europe Network). However, currently Single Market-related information, problem-solving mechanisms, contact points and procedures do not operate as a whole but are dispersed and not sufficiently inter-connected – both at EU and national level. It is therefore difficult for users to find the right information and assistance required. This implies that higher transaction costs have to be incurred before engaging in a cross-border activity. More synergies between actions, such as providing information, assistance and e-Government services, can be achieved at EU, national and regional level if efforts are better coordinated. An area for further work is to link up relevant EU and national content, services and procedures and offer users a streamlined, comprehensive portal to find information and initiate as well as complete transactions with Member States’ administrations across the EU (Single Digital Gateway as included in the Digital Single Market strategy).

The online public consultation (Annex 2) reflected a need for increasing awareness about the on-going work and availability of cross-border public services. The majority of the respondents did not know whether the current eGovernment Action Plan has improved cross-border eGovernment services overall or did not agree that the Action Plan has improved them.

6.3.3.Relevance: "Efficiency and Effectiveness of Governments and Administrations"

Under the sub-priority on reduction of administrative burden, the eGovernment Action Plan initiated the work towards the implementation of the once-only principle.

This objective is ever more relevant as a policy priority. The European Council Conclusions of October 2013 stated that "EU legislation should be designed to facilitate digital interaction between citizens and businesses and the public authorities. Efforts should be made to apply the principle that information is collected from citizens only once, in due respect of data protection rules."

Citizens and businesses are still all too often required to submit information that governments already hold. Only in 48% of cases do public administrations reuse information about the citizen that is already in their possession without asking for it again 162 . According to the eGovernment benchmarking report 2015 163 , the use of Authentic Sources has actually decreased. The use of authentic sources means that public administrations get data from sources such as National register, Tax registers, Company registers etc., instead of asking citizens and businesses to provide the data (again). The decrease in their use is disappointing as it also indicates that European governments still struggle to re-use data. As a result, citizens and businesses are still all too often required to submit information that governments already hold.

The amount of data that is pre-filled in Public Services' online forms (one way to implement the once-only principle) varies to a great deal within the EU 164 :

Figure 8 Pre-filled forms as an indicator of once-only. Amount of data pre-filled in public services forms in seven life events. (Digital Economy and Society Index 2016.)

The percentage of individuals who used the Internet to submit completed forms to public authorities was on average 33% of Internet users within the EU 165 in 2014. In 2015 this was slightly lower at 32% 166 :

Figure 9 Percentage of Internet users submitting completed forms to public authorities. (Digital Economy and Society Index. 2016.)

The eGovernment benchmarking report measures the availability of "authentic sources", which are base registries used by government to automatically validate or fetch data relating to citizens and businesses. It implies governments re-use data to deliver and facilitates the implementation of the once-only principle. The figure of 2013/2014 in the eGovernment benchmarking 167 assessment for EU28+ was 45%, meaning that in 45% of the assessed cases do public administrations reuse information that is already in their possession without asking for it again.

In consideration of these findings and the potential savings, implementing the once-only principle across borders continues to be a highly relevant priority in Europe and could further contribute towards the efficiency of the European public sector.

The relevance of the principle was acknowledged also by the European Parliament in its report Towards a Digital Single Market act in January 2016 168 : "a step-by-step sectoral approach to apply the ‘once only principle’ in public administrations according to which citizens and businesses should not be asked for information already provided to a public authority, whilst ensuring citizens' privacy and a high level of data protection - -;"

In addition to the once-only principle, the sub-priority of the Action Plan on improving organisational processes remains relevant. The eCommission Action Plan has led also the EC towards, amongst other things, electronic procurement. The digitalisation of procedures (including digital by default) is also a means for administrative burden reduction and remains relevant. Toolboxes and innovative architectures are also important measures where the EC to promote and present pilots and practices on the quality, efficiency, transparency of public services. The Joinup collaborative eGovernment platform has grown to be a platform for thousands of users.

The mid-term evaluation also recommends that although the action under sub-priority Green Government was not carried out it should be reconsidered: "Looking at eGovernment as an enabler to lower the carbon footprint of administration makes sense in looking at eGovernment in a more holistic way." Furthermore, the mid-term evaluation recommends that the staff exchange programme between administrations in different Member States, though not carried out, could be re-launched and it could be an excellent learning and sharing accelerator for Member States' eGovernment agencies.

6.3.4.Relevance: Pre-conditions

This Action Plan objectives on technical pre-conditions is still very pertinent. However, with many of the technical and legal pre-conditions achieved during the eGovernment Action Plan, the benefits to citizens, businesses and public administrations themselves need to be demonstrated. The take-up of these solutions could be increased, making a real impact on people's lives - reducing administrative burden, facilitating mobility in the EU and improving the user experience when dealing with public administrations.

The eIDAS Regulation on electronic identification and trust services for electronic transactions in the internal market was an objective of the Action Plan. It ensures that people and businesses can use their own national electronic identification schemes (eIDs) to access public services in other EU countries where eIDs are available. In addition, it helps to create a European internal market for eTS (Trust Services).  The take up of these services demonstrates the important potential for the future and the continued high relevance. In the end of 2015, 15 countries, including 13 Member States, were connected to the eID network via STORK (see further details in Annex 4 under action 37). In addition, the potential of a trusted identification mechanism goes also beyond the public sector, such as for the financial sector.

The eGovernment Benchmarking Report 2015 suggests that building blocks, such as electronic identity (eID), that enable online authentication of persons and companies, or authentic sources that encompass registers of personal and other data, and that would allow re-use of that data for other service processes, will boost cross-border and national online services. 

A lot was achieved under the interoperability objectives of the Action Plan. However, working towards interoperability between Member States at European level continues to be a relevant objective. Interoperability is needed to enable public services to work across borders. 1,500,000 citizens and 300,000 businesses are likely to use cross-border online services each year by 2020 169 which will presuppose the existence of interoperable solutions for which reason the creation of interoperable solutions will continue to be of high relevance.

As stated in the DSM staff working document, the lack of interoperability among public entities and private operators restricts the potential for digital end-to-end services, One Stop Shops, the once-only principle, the single data entry principle, the transparency of public services and the full exploitation of public open data. Further progress is needed to improve the interoperability of systems for cross-border delivery of goods and services, as well as for the mobility of people and businesses and for cooperation between public authorities, at national and EU level.

In the area of public services, the European Interoperability Framework, adopted by the EC in 2010 170 , promotes and supports the delivery of European public services by fostering cross-border and cross-sectoral interoperability. The majority of the Member States have transposed this Framework, which has brought a common understanding of the basic requirements for interoperability between public services. This common understanding is still relevant and requires updating and extending with other concrete and practical instruments to possibly be shared by national administrations such as the European Interoperability Reference Architecture (EIRA) and the European Interoperability Cartography (EUCart) 171 . The revision of the EIF is included in the list of actions of the Digital Single Market Strategy.

The ISA2 programme, recently adopted by the co-legislators, is a key instrument for the development of interoperability of public administrations in Europe and between public administrations and businesses and citizens.

Standardisation is another aspect that remains relevant in the future. Digitisation of administrative formalities offers opportunities to, for example, standardise the documents that businesses have to present to national authorities in different Member States, yielding additional cost savings 172 .

6.4.Coherence

The EU eGovernment Action Plan was based on a collaborative design by the EC and Member States, starting from a common vision (Malmö Declaration), containing agreed policy objectives, a monitoring mechanism and holistic approach with the relevant EC services working together. This wide coordinated approach helped to ensure coherence right from the beginning. This coherence was further strengthened by the inclusion of priorities of the Malmö Declaration and Action Plan in national eGovernment strategies.

The various parts and actions of the Action Plan included several different policy areas of the EU. Coherence at the time of the Action Plan preparation was focusing on identifying impactful areas where the introduction of ICT and related interoperability measures across the EU could yield the greatest impact, for example on eID and eProcurement. The coherence of the numerous different areas together was supported through the holistic governance structure of the Action Plan with the involvement of various Commission services.

The governance through the different groups with Member States, including the eGovernment High Level Expert Group, CEF, H2020, FP7, CIP and ISA Committees contributed to the avoidance of overlaps. The governance activities provided for regular meetings and exchanges of experience between Member States and the EC.

In order to achieve a snowball effect on investments the Action Plan successfully made coherent use of different funding instruments related to the eGovernment Action Plan. The work carried out on key enablers under the various programmes aimed also at the subsequent deployment of mature services under the CEF Telecom programme. The ISA programme played also a key role in ensuring the maturity and implementation of enablers piloted and/or developed under the H2020 and the CIP programme. The uptake could then be initiated with funding from the CEF Telecom programme. The figure below demonstrates the moving from CIP pilots towards deployment in CEF.

Figure 10 Coherence through CEF. Moving from piloting to roll-out; from large-scale pilots to the CEF.

For the ESIF funding, an ex-ante conditionality on Digital growth 173 was established to ensure coherence of the investments with other EU or national funding source. This conditionality aims to foster the development and implementation of national and regional digital growth measures and to assess their consistency with the Digital Agenda for Europe's goals that include e-government. In practice this meant checking against the eGovernment Action Plan priorities to ensure complementarity.

Coherence analysis can also assess the eGovernment related activities of the 28 Member States. In the eGovernment Action Plan, the Member States were responsible for 10 actions while seven actions were under the shared responsibility of the Member States and the EC.

In the Commission, several different policy areas were involved in the eGovernment Action Plan. The different services took part also in the eGovernment Action Plan governance activities, including the Member States groups. This increased the coherence between the different priorities of the Action Plan. Public sector modernisation was discussed in various fora and considering different aspects during the Action Plan duration.

However, as the eGovernment Action Plan was based on voluntary activities of the Member States, full coherence could not be assured, also because of the different implementation approaches used in the Member States.

The aim was not necessarily to achieve full coherence as various other initiatives had to be considered. On the international level, the EC followed the policies of the UN and the OECD and the policies of third countries that sometimes were also taking aboard the Commission approaches. The Member States also followed and participated in various international eGovernment activities.

6.5.EU-added value

The EU eGovernment Action Plan 2011-2015 as well as its predecessor have been political instruments for the Commission to advance the agenda of Public Sector Modernisation across the EU. They have been supporting European coordination, collaboration and joint actions on eGovernment, helping to use public resources more efficiently and reducing public expenditures by coordinating and pooling public and private resources. The Action Plan also helped inside the Commission to provide a political context and objectives for many different policy areas and helped to focus investments from different programmes.

The Action Plan has also been an example for establishing national eGovernment strategies in the Member States. The Action Plan has helped to coordinate investments, share and re-use infrastructures, processes, data, resources, content and tools to avoid duplication and waste and to speed up the roll-out of eGovernment. These types of initiatives continue to be required at EU level.

Some of the actions of the Action Plan stem from mandatory requirements set down in EU law, which went through the subsidiarity and proportionality test in the legislative process. Here the Action Plan helped to support the collaboration between the Member States. On the other hand, some actions were relevant to or contributed towards new legislation, examples of this are e-procurement (public procurement legislation 174 ) and eID-activities leading to the eIDAS Regulation.

The cross-border elements of EU legislation relating to public services (e.g. eProcurement, eIDAS, Public Sector Information, Services Directive) continue to justify, and require collaborative activities at EU level, for example, to support the implementation of legislation through the common development and piloting of technical solutions.

The eIDAS Regulation ensures secure and seamless cross-border electronic transactions in the EU by promoting the widespread use and uptake of electronic identification and trust services (eIDAS services) across borders. It ensures that people and businesses can use their own national electronic identification schemes (eIDs) to access public services in other EU countries where online public services are available. In addition, it helps to create a European internal market for electronic trust services by ensuring that they work across borders and have the same legal status as traditional paper based processes.

In the eGovernment policy the Commission's responsibility is limited to the cross-border aspects. Yet the Action Plan included various actions that Member States committed to in a voluntary manner and where the Commission did not have a formal legal basis to act. The Action Plan fully respected the organisational set-up of eGovernment activities at national, regional and local levels and was complementary to national Action Plans under the responsibility of the Member States at the same time being instrumental in motivating and leveraging implementation in Member States of voluntary actions.

The mid-term evaluation of the eGovernment Action Plan acknowledges that the Action Plan "has a positive impact on the development of eGovernment at the European and Member State level even if progress has been better in some areas compared to others. This type of Action plan can be a perfect ‘mobilizer’ instrument in order to help the European Commission and the Member States coordinating their actions". Several examples of this "mobilizer" effect have been given in this evaluation.

Most of the countries also stress that the cross-border dimension of the eGovernment Action Plan has had the strongest impact on their national strategies. It is this effect that makes up the EU-added value. Interoperable cross-border digital solutions can prevent the emergence of new obstacles to the Single Market. The building block approach created in the large-scale pilots and continued under the CEF Telecom programme is doing exactly that by promoting and supporting the re-use of such solutions in support of interoperability across borders.

While the Action Plan did not have a dedicated budget or funding instrument, it has served as a catalyst to coordinate resources. The research, innovation and piloting projects carried out under the umbrella of the eGovernment Action Plan in FP7, H2020, CIP, ISA and CEF programmes helped to pool public resources and coordinating with private resources in order to develop and pilot innovative eGovernment as well as pilot solutions to be deployed as key enablers for eGovernment.

Without the intervention of the eGovernment Action Plan these achievements may not have realised or would have been only partly achieved. Acting at EU level through the eGovernment Action Plan helped the Commission and Member States to coordinate and collaborate in order to ensure interoperability between national systems towards seamless access to digital public services across borders. Continued work is required to encourage the opening up between public administrations at all levels, the re-use of existing solutions to avoid duplication, the reduction of costs multiplication and the optimisation of investments. Coordination of efforts and resources contributes to strengthening the EU as a provider of benefits to citizens and businesses and to meeting their digital expectations of modern governments.

7.Conclusions

The findings of this report allow for a relatively clear assessment of the key evaluation criteria and questions despite the limited availability of recent quantitative data and certain limitations regarding the representativeness of responses to the stakeholder consultations.

Effectiveness

Throughout the analysis it was often impossible to isolate the impact of the Action Plan from other developments in the public sector and therefore to assess its direct impacts. However, the eGovernment Action Plan has been effective in achieving most of its objectives. Already the mid-term evaluation of the Action Plan recognised that the Action Plan had a positive impact on the development of eGovernment at the European and Member State level.

Despite there being varying degrees of progress in the Member States and in the different policy priorities, the Action Plan demonstrated the importance of having common European goals in eGovernment. The Action Plan acted also as a "mobiliser" instrument. According to the mid-term evaluation of the Action Plan, the national eGovernment strategies in almost all Member States incorporate priorities from the Malmö Declaration and the eGovernment Action Plan. Furthermore, most of the Member States (17) have incorporated the political priorities of the Malmö Declaration and eGovernment Action Plan in their national strategies.

Furthermore, in 2012 the annual eGovernment Benchmark reporting was aligned with the priorities of the eGovernment Action Plan 2011-2015. This provided a robust tool to monitor and measure the effectiveness of the results, but the Action Plan had no mechanism to iteratively improve the process and fine-tune the actions according to the findings.

The eGovernment Action Plan has acted as a focal point for actions on the EU level, in particular related to the support for the development of enablers for cross-border services. The Action Plan had a key role in pushing for the availability of key cross-border services. Since the adoption of the Action Plan in 2010, large-scale piloting actions have been co-funded by the Commission to achieve the transition towards a Digital Single Market. The effects of these pilots have been quantitative (take-up) and qualitative (increased awareness and coordination) and have, for example, contributed towards the adoption of the eIDAS Regulation. The e-procurement piloting through the PEPPOL project has contributed towards the take-up of e-invoicing in some Member States. The work on the large-scale pilots led to the support of the Member States of the CEF Telecom Regulation, aiming at financing the deployment of the cross-border services and the building block approach. This has also had an impact in the work of other sectors, for example, e-justice and e-health. The public online consultation indicated that awareness of cross-border services availability could be further increased.

Some of the eGovernment Action Plan actions have not been implemented. The mid-term evaluation acknowledges that in a rapidly changing world with very fast evolving technology, a static five years period seems too rigid for an Action Plan. This static nature has affected the effectiveness of the Action Plan as it was not possible to identify new actions needed, to adapt actions to changed circumstance or to remove actions that became obsolete, sometimes due to activities taking part elsewhere.

Moreover, the Action Plan, while prioritising user empowerment, did not have a mechanism to enable stakeholders to be part of the decision-making process that identified actions to be taken by Member States or the Commission during the duration of the plan. This could have led to greater acceptance of the results. The online public consultation showed the interest of stakeholders in an online social media platform to facilitate the sharing of ideas.

Areas of eGovernment can gain political visibility arising from political decision-making, which increases the overall effectiveness. On the other hand, effectiveness can suffer from too specific objectives if they cannot be updated during the duration of the Action Plan.

Efficiency

The eGovernment Action Plan had no dedicated budget.. but it mobilised resources in the Member States and the Commission to develop solutions and common understanding in several internal market policies, whicn in turn hase the potential to unleash long term savings from digitalisation.

Relevance

The Action Plan priorities (user empowerment; internal market; efficiency and effectiveness of government and administrations; and pre-conditions for developing eGovernment) remain relevant today. The modernisation of the European Public Sector remains an important goal in the EU.

However, given the changing environment with economic challenges, increasing use of online means and increasing intra-EU mobility the evaluation found that further work is needed, sometimes with evolving new needs. For example, there is a continued need to pursue the re-use of common solutions and to increase the take-up of both cross-border and national eGovernment services. Another example is the indicator "online service completion" being over 80% on average in the EU in 2015 175 , the relevance has moved from mere online availability of public services to, for example, quality of the services and cross-border availability.

There are various examples of topics with continued or increased relevance in the future. Some key enablers have been successful and the rollout looks promising, but the strengthening of the key enabler approach, including the take-up, remains relevant for future work. The Action Plan areas of Citizens rights, re-use of public sector information, transparency and data protection are important principles also in the future when modernising public administrations.

The relevance of eGovernment services for the internal market continues, driven also by the increased intra-EU mobility of citizens and businesses. Advancing the once-only principle continues to be a relevant target beyond the duration of the Action Plan. There are also continued needs to pursue the re-use of common solutions for rationalisation, savings and interoperability and to increase take-up.

The online public consultation confirmed the relevance of a number of these areas. There was a clear consensus (supported by 80% or more of the respondents from all categories) on the importance of applying the principles of Privacy by default, Digital by default, Cross-border by default, open by default, the Once-only principle, Online end-to-end services, Inclusive by default and Transparency to eGovernment policies and services in the EU.

Coherence

The governance structure of the Action Plan with regular meetings and exchanges of experience between different Member States and the Commission contributed to coherence and avoidance of overlaps. The Action Plan's coherence benefited from a political consensus (Malmö Declaration) between the Member States. However, full coherence could not be assured, as the Action Plan was based on voluntary activities of the Member States that also had different implementation approaches.

The coherence of the different funding mechanisms for eGovernment key enablers benefited from a common aim towards the deployment of mature services under the CEF Telecom programme.

A wide variety of eGovernment initiatives at various levels were taken during the Action Plan's five year duration. An Action Plan specific coherence was not fully achievable.

EU-added value

The Action Plan led to increase the coordination of investments in the Member States and in the Commission in order to focus on the modernisation of public administrations. An EU level action plan was probably the only way to start to realise the potential of cross-border eGovernment services and to achieve interoperability while avoiding fragmentation. If the eGovernment Action Plan was not to be renewed, there would be a risk of fragmentation of public services solutions, possible overlap of efforts and a risk of low availability of cross-border public services in the Digital Single Market.

8.ANNEXES

Annex 1.Procedural information

In December 2015, European Commission's Directorate-General Communications Networks, Content and Technology (DG CNECT) launched an implementation and evaluation assessment of the European eGovernment Action Plan 2011 – 2015 ("EU eGovernment Action Plan 2011 – 2015"), adopted on 15th December 2010, first and foremost in order to provide a set of lessons learnt to guide the work of the new eGovernment Action Plan 2016 – 2020 announced in the Digital Single Market Strategy.

The evaluation has been coordinated by the EC's Directorate-General Communications Networks, Content and Technology with the support of an Inter-Service Group with representatives of Commission Directorate-Generals Agriculture and Rural Development; Communication; Competition; Informatics; Economic and Financial Affairs; Employment; Social Affairs and Inclusion; Energy; Environment; Internal Market, Industry, Entrepreneurship and SMEs; Migration and Home Affairs, Joint Research Centre; Publications Office; Justice and Consumers; Mobility and Transport; Regional and Urban Policy; Research and Innovation; Health and Food Safety; Secretariat-General; Taxation; and Customs Union and Trade. The Inter-Service group steered and monitored progress of the evaluation exercise, ensuring the necessary quality, impartiality and usefulness of the evaluation. The group met three time during the evaluation process.

For the evaluation, a broad literature review was conducted covering various reports, EU studies and a number of international references. Important sources in the desk research were the mid-term evaluation of the eGovernment Action Plan 2011 - 2015 176 and the annual eGovernment Benchmarking reports (for further details see section 4). Furthermore, additional information and data (listed in the in the footnotes and in the "References and data sources" part in the end of this document) were used to answer the evaluation questions. Their methods are available in the original referenced reports. The eGovernment factsheets published on the Joinup platform 177 give an overview of eGovernment in each Member State.

A 12 week online public consultation was conducted to collect primary data and complemented by statistical data (Eurostat data and Digital Economy and Society Index DESI 178 ) and Digital Agenda Scoreboard reports 179 .

An effort was made to “triangulate” the data used throughout the analysis. This means that findings presented in the evaluation are supported by evidence from different data sources whenever additional data was available. Any contradicting evidence has been weighed according to its strength and quality before reaching conclusions

The evaluation faced some limitations in the collection of data, whose impact was mitigated to a maximum possible extent:

Measuring the impact of the eGovernment Action Plan is a challenging exercise as it is almost impossible to isolate the impact of the Action Plan from other developments in the public sector, such as ICT-policies or the increased use of the Internet in general. Furthermore, some objectives of the Action Plan, such as increased collaboration, are non-tangible and, as a result, difficult to quantify. The evaluation has used benchmarking to measure the progress in the different policy priority areas of the Action Plan and assessed the effectiveness criteria qualitatively.

The Action Plan did not have a dedicated budget but relied on funding from other programmes for a large part of its actions. Other actions were financed by the Member States. As a result, the efficiency analysis (in particular the value for money assessment) was difficult to carry out.

Despite being prompted in a number of occasions by the EC, there is no systematic measuring or reporting requirement for the Member States of cross-border public services use. To overcome this, the evaluation used data from other sources, including case studies.

Given the multiplicity of the tools used to gather evidence, the results obtained are of different nature (for instance, eGovernment related data stems from the mid-term evaluation of the Action Plan and the benchmarking reports while the online public consultation provided insights in order to prepare for the new eGovernment Action Plan 2016-2020).

The evaluation takes into account the inherent limitations of the findings of public consultations. Firstly, as in all surveys, the answers received reflect the views of a sample of relevant stakeholders and not those of the entire population who has a stake in this domain. Secondly, stakeholders' views convey an individual rather than a holistic perspective.

The mid-term evaluation did not systematically cover the five evaluation criteria. Furthermore, the methodology had strict requirements. For an action to qualify as "delayed" it was sufficient that one indicator is negative (e.g. one Member State answered ‘no’ to the question, or one action was delayed in one Member State) can sometimes give only a partial overview of progress and it should be understood as an evaluation of the actions as defined by the action plan, not an assessment of the policy priority as such. In addition, the mid-term evaluation did not capture the last 18 months of the Action Plan. This is partly mitigated using the 2015 benchmark study and partly the evaluation questions in the 2015/2016 online consultation on the eGovernment Action Plan and other recent reports and information.

Based on the elements above, the evaluation has been carried out on the basis of the best available data. Whenever reliable quantitative data is lacking, this is indicated as appropriate and possibly counter-balanced with qualitative data and considerations.

Annex 2.The Online public consultation

REPORT

on the public consultation and other consultation activities of the European Commission for the preparation of the EU eGovernment Action Plan 2016-2020

1.INTRODUCTION

The Commission launched an online public consultation on 30 October 2015 for 12 weeks to seek views on a forthcoming EU eGovernment Action Plan 2016-2020, one of the actions for the completion the Digital Single Market . The consultation collected EU citizens' and businesses' needs and expectations from eGovernment services in the EU, and what public administrative bodies can or plan to deliver. It was accompanied by a roadmap for a new EU eGovernment Action Plan.

In addition, the Commission organised workshops with stakeholders. The consultation, those events, the annual eGovernment benchmark survey and the mid-term review of the previous action plan underpin the preparation of a new EU eGovernment Action Plan 2016–2020.

2.WORKSHOPS AND CONFERENCES WITH EXTERNAL STAKEHOLDERS

2.1. Riga, Digital Assembly 2015  

The first of these events was organised as part of the Digital Assembly in Riga. The panellists at the workshop pointed to some urging user expectations, resulting in an initial list of possible digital rights, further expanded during the discussion with the audience. The list stresses possible rights related to user-friendly public services:

digital – including the ability to receive and submit documents electronically and eventually through one-stop-shops

intuitive

inclusive

accessible

fast

efficient

multilingual

automated services or at least reducing the number of interactions with public administrations, for example through the once-only principle.

Some of the suggested rights linked to modernising public administration, making them open, transparent and collaborative, allowing for citizen involvement and eDemocracy. Another set of suggested rights related to mobility within the single market, ensuring that data and digital services moved seamlessly across borders and the right to do business anywhere in the EU. Another group of rights pointed to basic pre-conditions that would make this happen i.e. privacy / confidentiality, the right to exist digitally, to control access to personal data, security, access to cheap and fast network, digital literacy and the quality of and access to machine readable data.

Participants ranked the 'once only principle', 'user friendly / intuitive public services' and 'digital literacy' as the three most important rights.

2.2.Workshops on a new EU eGovernment Action Plan, Brussels: 1 July, 5 November 2015, 15 December 2015

The first workshop on a new EU eGovernment Action Plan focused on the overall policy framework (modernising public administration, helping public administrations cooperate with one another and offering better service, citizens' involvement). Many pre-conditions must be achieved before actions are implemented such as privacy, quality and opening up of data, interoperability and security. The current Action Plan's mid-term review showed, that five-year static plans were not fit-for-purpose amidst rapidly changing technology.

At the second Join-up workshop, the Commission focused on the public consultation questions and how to engage citizens so that they propose ideas for the new Action Plan. Participants discussed setting up an online stakeholder platform, raising awareness about it, the need for political support, the use of multiple languages and maintaining contributors' interest in the platform.

The third workshop on a new EU eGovernment Action Plan focused on the online platform and how it could support a new eGovernment Action Plan.

2.3. Luxembourg eGovernment conference , 1st December 2015

Organised by the Luxembourg presidency, the event showed EU countries' progress on public sector modernisation. Important steps have been taken since the Malmö Ministerial Declaration on eGovernment of November 2009 and the EU eGovernment Action Plan 2011-2015 which lead to concrete measures on the political priorities of the Declaration.

Defining and launching a new coherent EU eGovernment Action Plan was central, aiming at increasing citizen and business engagement in the design of public services and policy-making, among others.

The Luxembourg Presidency concluded that improved coordination between EU countries is essential. An effective European Action Plan must consider all actors and measures affecting the EU eGovernment agenda, yet be centred on real user needs in EU countries.

3.THE ONLINE, PUBLIC CONSULTATION

The 12-week consultation was launched on 30 October 2015 until 22 January 2016. It covered the following topics related to the development of eGovernment services in the EU:

Lessons learned for the current Action Plan

Factors hampering the use of public services

Improving eGovernment services

Mobility and cross-border public services in the EU

Modernizing eGovernment services in the EU

The role of the European Commission

Citizen involvement

Policy principles

3.1.Overview of the respondents 180

The online questionnaire received 365 replies; 12 position papers came from organisations and government representatives:

65% of the replies came from EU citizens

13% from businesses and organisations

22% from public administrations

Most replies were received from Germany (112), Romania (59), Italy (37), Czech Republic (32) and Spain (29). In the citizens group most replies came from Romanians (46), Italians (31) and Germans (31). In the Institution/Organisation/Business category, most replies were received from Germany (26) and Belgium (21).

3.2.The topics 181  

3.2.1.Lessons learned from the current Action Plan 182

This section with mandatory questions addressed the eGovernment Action Plan 2011-2015 (on-going at the time).

Has the current eGovernment Action Plan improved cross-border eGovernment services overall?

43% of respondents replied that they don’t know; 28.5% said that it has not; 24.9% agreed that the plan has improved cross-border services overall. There are not significant differences between the respondent groups with regards to the perceived cross-border effects of the current Action Plan. The answers point to the need to increase awareness of on-going work and availability of cross-border public services in the EU.

How do you rate the measures comprising the current eGovernment Action Plan?

This section had 4 sub-questions addressing the measures of the previous action plan, asking about their success.

The respondents mostly replied that they were either unaware of the measures comprising the Action Plan or rated them as not successful, "not successful" comprising the largest group. For instance, on the User Empowerment measure of the action plan on inclusive services, collaborative production of services, re-use of public sector information, transparency and the involvement of citizens and businesses, 48.1%, said that it was not successful; 24% said that they don’t know; 27% of respondents found it successful. Increasing the efficiency and effectiveness of government and administrative bodies was viewed as the least successful, although by a small margin. Public administrations view the success of the measures addressing user empowerment and preconditions for developing eGovernment as much higher than the overall average. For instance, 46% rated the measures related to the preconditions for developing eGovernment successful, against a 28% average for the 3 respondent groups.

Action Plan activities did not target directly citizens and businesses. They were carried out by the Member States and the Commission to create for example enabling conditions not directly visible to the citizens as eGovernment Action Plan activities.

3.2.2.Factors hampering the use of digital public services 183

Your contacts with the public administrations in your home country

This section addressed the factors that may/may not hamper the use of digital public services in the respondent's home country.

Citizens highlighted 3 factors not likely to hamper the use of digital public services: limited Internet access (63%), lack of digital skills (74%) and preference to interact with public administrative bodies through traditional channels (55.5%).

Concerning the factors likely or very likely to hamper the use of digital public services, 73% pointed to the lack of user-friendly public services (73%) and that public administrations require the same information more than once ('Once-only principle')(66%); 56% stressed meeting individual needs and 57% - the lack of online feedback mechanisms.

For the other factors e.g. accessibility on mobile devices, replies were more equally divided between factors that were likely and very likely to hamper the use of digital public services in the respondent's home country on the one hand and factors that were less likely and not likely to do so on the other.

Businesses also pointed towards the absence of the 'Once-only principle' as most likely to hamper the use of digital public services (82%). Other hampering factors businesses mentioned were the difficulty to find relevant information (75%), and the poor quality of pre-filled forms (71%). The factor least probable to stop businesses using digital public services in their contact with the public administrations is the preference for personal interaction (71%). One may conclude that businesses prefer to interact digitally with public sector administrations, but that the current state of online services is not fully satisfactory.

In addition to the pre-defined factors listed in the questionnaire, the lack of transparency and the insufficient respect of privacy were mentioned as other factors that could hamper the use of digital public services.

Public administrations listed 3 factors most likely to hamper the use of digital public services: the lack of accepted electronic identities (eIDs), the need to provide the same data more than once and poor quality and user unfriendly online public services. The lack of digital skills in public sector organisation staff was said to be the factor least likely to hamper the use of digital public services.

The questionnaire addressed the supply side of eGovernment services, asking about the likely factors to hold the organisation back from providing more or better government services online. Complex legislation making it difficult to digitise administrative procedures rank as the number one factor most likely to do this, followed by the lack of interconnections between public sector data sets and interoperability. Lack of political priority and leadership does not appear to be a major stumbling block.

Contact with the public administrations in another EU country

Regarding cross-border services, the need to provide paper documents in addition to online information were seen as the factors most likely to stop citizens from using digital public services in their contact with public administrations in another EU country (75%).

3.2.3.Improving eGovernment Services 184

This section listed measures with the potential to improve eGovernment services, posting the same list of measures for citizens, businesses and public administrations to compare.

Citizens consider most of the listed measures important or highly important, with minor differences between the categories. Improving digital skills, both in business and the general public were seen as the least important factor to improve.

On the measures rated as highly important, making all online public services inclusive and accessible to all got the highest score (64%), giving users access to public services online (63%) and making online public services more trustworthy and secure.

Public administrations emphasised the personalisation of all online public services and making them user-friendly (98% in favour); the once-only provision of data got 90%, giving users access to services online (89%) and 83% for the acceptance of EU wide electronic signatures.

Inclusiveness and accessibility received 100% positive replies from business respondents, 93% to giving users access to public services online. There are hardly any differences between the answers to the different measures.

The different respondent categories rate some measures higher than others, but the differences do not conclusively point towards one or a few measures to address more than others. All measures with the potential to improve eGovernment services should be equally tackled.

Ensuring cross-border availability of public services is important for the smooth functioning of the internal market. 63% of citizens indicate that that the measures promoted at national level also should be promoted at the European level. That is positive for the continuation of eGovernment initiatives at European level. 4.9% said that this should not be the case; 32.1% replied that they don’t know. Businesses and the public sector also believe that national level measures should also be promoted at European level.

3.2.4.Mobility and cross-border services in the EU 185

Citizens

This section addressed the cross-border aspect of eGovernment services, the expectations and difficulties citizens, businesses and public sector bodies may have had when interacting online with public authorities in another EU country.

54.1% of the respondents (citizens) said that they had had contact with or needed to engage with public authorities in an EU country other than their own.

Among the reasons for that contact education ranked highest (29.5%), followed by moving (22.5%), health-related services (20%), finding a job (16%), declaring taxes (11%), retiring and starting a judicial procedure (both 3.6%). Multiple answers were possible.

Many had indicated other reasons for the need to contact public authorities in another EU country i.e. setting up a business and contact with the European institutions. Most did not explain more about the other reasons they referred to.

When interacting online with a public authority in another EU country, many expected to be able to access all relevant information and start the procedure online using a "one-stop" shop (43%) and that the information would be provided in a language they understood (40%). Many also expected to be able to complete procedures, get help and communicate in a language they understood (37.5%) and use electronic signatures and electronic identifications from their home country (32%). Fewer expected to be able to give electronic access to personal data already provided in the home country (26%). Multiple answers were possible.

On the specific difficulties in transferring information/document/data between the public authorities of the home country and those of another, not accepting national information/documents/data was ranked the highest. In the free text under the "other" option, the difficulty to translate documents for official purposes and the lack of direct exchange of documents between public sector bodies in different countries were mentioned.

Businesses

Over half (52%) of the businesses replied that they had had contact with or needed to engage with public authorities in another EU country. Some reported no difficulties, others said they had had problems with the resubmission of documents already provided to a national authority or that national documents were not accepted.

Public sector bodies

The majority (62%) of public sector bodies reported that they had had contact with or needed to engage with public authorities in another EU country, and more so with authorities from more than one EU country in citizens-related cases. In doing so, 58% said contact was made using other means than the existing EU legal framework (e.g. under EU law on cooperation between national authorities) and in the absence of an established bilateral channel (62%) e.g. Memorandum of Understanding. Concerning potential difficulties in the transfer of information, most respondents replied that they had had problems. Few respondents referred to specific problems. Among those who did, language issues came on top.

3.2.5.Modernising eGovernment services in the EU 186  

Please indicate how important it would be to improve or apply eGovernment services over the next 5 years (for each of the following areas):

According to the respondents (citizens), most listed areas require improving i.e. eGovernment services for healthcare, filing tax declarations, enrolling in higher education, looking for a job and applying for a passport. eGovernment services related to buying and selling a home were seen as the least important.

Another element for the modernisation of eGovernment - the improvement of eGovernment procedures related to businesses - was addressed in the following question. None of the procedures listed were viewed predominantly as less or not important meaning that respondents would like to see all of them addressed. The procedures marked as more important compared to others were improving online services for tax- and insurance-related matters and obtaining government certificates.

When asked if measures to modernise eGovernment services should be proposed for the areas listed, public sector bodies mostly replied yes to all with minor differences in weighting between them.

Respondents would like to see eGovernment services improved for most eGovernment services and online procedures, not only a select few.

3.2.6.The role of the European Commission 187 and the new eGovernment Action Plan

Section 6 of the online questionnaire asked respondents how the Commission can help improve public administrations at all levels –national, regional and EU-wide (in free text).

The answers suggest various ways for the Commission to help improve public administrations: imposing EU legislation in specific areas e.g. establish an EU authority for digital certificates, setting standards for cross-border interoperability e.g. by consolidating existing national electronic registers, setting up an EU platform for e-participation, ensuring the exchange of good practice and cross-border cooperation, increase funding to eGovernment development in the regions including financing for the deployment of broadband infrastructures, design eGovernment policies to avoid social exclusion (universal design), harmonise the use of electronic identities including the issuing of an EU eIDs and make mandatory the use of electronic signatures, increase collaboration between governments and citizens in the design of eGovernment services and ensure the availability of open data-sets and open format documents (mandatory).

The setting of standards, interoperability at all levels and the exchange of good practice were mentioned by many as priority areas for the Commission in the future eGovernment Action Plan. The Commission was also asked to lead by example and improve its own internal and external services. Several respondents also underlined the need for the Commission to push for the application of the 'once-only principle' in order to reduce the administrative burden for citizens.

Ideas on the Commission's role and the focus of the new eGovernment Action Plan were contained in more detail in some of the position papers Member States submitted. For instance, the Danish Government sees the new eGovernment Action Plan as an important tool to move public sector digitisation in Member States forward and implement initiatives that will improve cross-border collaboration on public service delivery. The UK government believes it is crucial to improve access for businesses to the single market and that the role of the Commission is to support Member States in the delivery of cross border services, coordinate activities and monitor progress. Estonia also said that EU level activities should target the better functioning of the internal market. On the role of the Commission, the Polish government believes that it should coordinate and contribute to the creation of standards relevant to the digital single market; solutions related to the recognition of qualifications, data exchange between public registers in Member States, defining common document models and imposing interoperable solutions. The Commission should also provide open source solutions to handle electronic signatures, ePrior etc. under the Join-up initiative and lead by example. La poste (France) said that the eGovernment policy set forth by the Commission should facilitate digital interaction between governments and citizens/businesses reduce the administrative burden and leave the citizens in control.

In a follow-on question, all 3 respondent categories ranked accepting electronic signatures for the most significant communication flows as the most urgent. That would allow Commission suppliers and grant participants to send the relevant data and documents once only, and make e-invoicing and pre-award e-Procurement mandatory for all the new Commission market procedures.

3.2.7.The 2016 – 2020 eGovernment Action Plan, Citizen involvement 188  

This section asked how people should be enabled to contribute to, make proposals on and publicly exchange views on new initiatives under the eGovernment Action Plan e.g. should the Commission set up an online social media platform to facilitate sharing of ideas (free text).

A clear majority said that an online social media platform to reach out to a large audience is a good idea, although views were divided on whether the Commission should set up a dedicated one or use existing ones. Some viewed using exiting platforms as the more efficient option. Examples of comparable online platforms were provided e.g. the co-creating platform for budgetary spending in New York, and similar initiatives in Germany and France (Paris). With regards to the setting up of a dedicated online platform, respondents said that it should be designed with the users in mind, allow all languages, include feedback mechanisms, be backed up by e-signatures and allow users to compare eGovernment solutions in different EU countries.

Some argued that using an online social media platform is not sufficient. It has to be linked to going local initiatives and existing national mechanisms for reaching out to citizens. Due to the very technical nature of eGovernment, most likely the platform will only be used by experts and not ordinary citizens.

3.2.8.Policy Principles 189  

How important are the following strategic policy principles?

This section addressed policy principles that may be applied to the development of eGovernment services in the EU, asking respondents to rate them by importance.

Over 80% of the respondents agreed that these principles are important, including to have online end-to end services, that data should be provided only once, that services should be inclusive by default, transparent, privacy by default, No-legacy with regards to the technical infrastructure, cross-border and open by default. Privacy by default was said to be more important relative to the other principles but only marginally so. The replies do not provide evidence that one or several of the policy principle are viewed as significantly more important than any of the others.

Views cautioning the application of some principles referred to the need to adapt technology to the service in question and to user needs e.g. some user groups would be excluded if services were only provided digitally.

In the position papers, Member States referred to the reduction of the administrative burden and the 'Once-only principle' as important to the development of eGovernment services in the EU.

Respondents from the public sector were less favourable than citizens to the No legacy principle which would require governments to renew IT systems and technologies after a certain period of time to keep in line with the changing environment and technology.

Respondents agree that eGovernment policies should be designed with these policy principles in mind.

IV    CONCLUSIONS

In view of the replies on the previous action plan, the new eGovernment Action Plan should be more visible if it is to reach a broader audience.

Stakeholders responding to consultation report that many factors still hamper the use of eGovernment services. The consultation and the workshops confirm the need to do more to improve eGovernment services in the EU at national and EU level. The consultation also confirms the Commission's role e.g. by promoting standards and ensuring interoperability at all levels. Certain principles should be applied to the setting up of eGovernment policies, including the "Once Only" principle to reduce the administrative burden to businesses and citizens. Overall the responses support the approach taken in the roadmap for the new eGovernment Action Plan.

Public consultation Report Annex 1:    A: Graphs and associated questions

                       B: Free text questions

A: Graphs and associated questions

1)Information about the respondents

a)Nationality

b)Place of business

c)Age    

2)Lessons learned for the current Action Plan

a) Has the current 2011-2015 eGovernment Action Plan improved cross-border eGovernment services overall

b)How do you rate the measures comprising the current eGovernment Action plan?

3)Factors hampering the use of public services, 1)Citizens, 2) Cross-border, 3)Businesses, and public administrations

a)For each of the following measures how likely is it to stop you using digital public services in your contact with the public administration in your country?

b)For each of the following factors how likely is it to stop you using digital public service in your contact with the public administrations in another EU country

c)For each of the following factors how likely is it to stop you and your business using digital public services in your contact with the public administrations in your country?

d)For each of the following factors how likely is it to stop you and your business using digital public services in your contact with the public administrations in another EU country?

e)As a public administrative body and a user of eGovernment services, please rate for each of the following factors how it is likely to hold back the use of public services at your level of government (EU level, national, regional or local)

f)As a public administrative body providing online public services, please state for each of the following factors how likely it is to hold your organization back from providing more and better government services online?

4)Improving eGovernment services

a)How important are the following measures in terms of improving eGovernment services?

b)Promoting at EU level the measures you have selected above as important or highly important?

5)Mobility and cross border services in the EU

a)Have you ever had contact with or needed to engage with

b)For what purpose?

c)When interacting online with a public authority in another EU country, expected?

d)Have you ever had difficulty transferring information/documents/data between the public authorities of your home country and those of another EU country?

e)Have you ever had contact with, or needed to engage with public authorities in an EU country other than your own? 

f)If you have tried to engage with public authorities In another EU country (e.g.) for business purposes, have you ever had difficulties transferring documents/data between the public authorities of your home country and the country where you intended to do business?

g)To provide services to citizens, businesses or other public administrative bodies in your home country or abroad, have you had contact with, or needed to engage with, public authorities in an EU country other than your own? 

h)With authorities from? 

i)In cases concerning:            

j)Did you use existing mechanism (under the EU law on cooperation between national authorities, IMI, Solvit) 

k)Is there an established bilateral channel of communication with the authorities in the other EU country? (e.g. under a Memorandum of Understanding)     

l)If you have tried to engage with public authorities in another EU country have you had difficulty transferring information?

6)Modernising eGovernment services

a)Please indicate how important it would be to improve eGovernment services over the next 5 years for each of the following areas? 

b)Please indicate how important it would be to further improve or introduce each of the following eGovernment related procedures (local, regional, national and EU) over the next 5 years? 

c)Should measures be proposed in the following areas?

7.The role of the European Commission

Please indicate for each of the following areas the level of priority for action by the European Commission.

8.Policy principles

How important are the following strategic policy principles?

B: Free text questions

Section 6 – Modernising eGovernment services in the EU

In which areas should eGovernment at all levels (local/regional/national and EU) be improved over the next 5 years?

Section 7 of the online questionnaire – The role of the European Commission 

How can the European Commission help improve public administrations in the EU at all levels – regional, national and EU wide?

Section 8 – The 2016 2020 eGovernment Action Plan, Citizens involvement

How should people be enabled to contribute to, make proposals on and publicly exchange views on new initiatives emerging under the eGovernment Action Plan? For instance, should the Commission set up an online social media platform to facilitate the sharing of ideas?

Annex 3.The individual actions of the eGovernment Action Plan 2011 - 2015

Below is a description of the actions per priority and sub priority. Some of the actions were already defined in the Digital Agenda for Europe, they are indicated with the footnotes. The actions are numbered for the evaluation purposes. Of the 40 actions, the Commission was exclusively responsible for 23 (1, 3, 4, 6, 7, 8, 12, 13, 16, 19, 21, 25, 26, 27, 28, 30, 32, 33, 35, 36, 38, 39, 40 +41, 44) and the Member States for 10 (2, 5, 11, 17, 18, 20, 22, 31, 34, 37 +42, 43, 45). Seven actions (9, 10, 14, 15, 23, 24, and 29) were under shared responsibility between the Member States and the Commission.

1. User Empowerment

2. Internal Market

Seamless Services for Businesses

3. Efficiency and Effectiveness of Governments and Administrations

Improving Organisational Processes

4. Pre-conditions for developing eGovernment

Open Specifications and Interoperability

Governance

In addition to the above policy priorities, the Action Plan also includes a section on its governance with five actions. This reflects the close partnership between Member States and the Commission. The Governance chapter is an element of the eGovernment Action Plan that was added to the four priority areas of the Malmö Declaration.

Annex 4.Implementation of the individual actions of the eGovernment Action Plan 2011 – 2015 – State of play of the actions

Priority: User Empowerment

The Action Plan priority on user empowerment included 14 actions (listed in Annex 4) across five sub-priorities. The figure below 198 provides an overview of the status of implementation of these 14 actions until the beginning of 2014.

Overview of progress of the User Empowerment priority. Status 6/2014.

Priority: Internal Market

Overview of progress of the internal Market priority. Mid-term evaluation of the eGovernment Action Plan. Status 6/2014.

Priority: Efficiency and Effectiveness of Governments and Administrations

The figure below from the mid-term evaluation of the eGovernment Action Plan provides an overview of the seven actions included in the Efficiency and Effectiveness of governments and administrations priority.

Overview of progress of the Efficiency and Effectiveness priority. Mid-term evaluation of the eGovernment Action Plan. Status 6/2014. Action 26 was completed 2014. Action 29 (EC) was completed in 2014 – 2015.

Priority: Pre-conditions for developing eGovernment

Overview of progress of the Pre-conditions priority. Mid-term evaluation of the eGovernment Action Plan. Status 6/2014. Action 32 has been completed 2014 – 2015.

Annex 5.Acronyms

AGRI        The Directorate General responsible for the Common Agricultural Policy.

BBs        Building Blocks

CEF        Connecting Europe Facility

CF        Cohesion Fund

CIP         Competitiveness and Innovation Framework Programme

CIPA        Common Infrastructure for Public Administrations Sustainability

CONNECT    The European Commission Directorate General for Communications

Networks, Content & Technology

COM        Communication from the Commission 

COMM    The Directorate General responsible for Communication

COMP    The Directorate General in charge of Competition Policy in the European Union

CSP    Core Service platforms

DAE         Digital Agenda for Europe

DESI        Eurostat data and Digital Economy and Society Index

DIGIT        Directorate General for Informatics 

DG        Directorate General

DSI         Digital Service Infrastructure

DSM         Digital Single Market

EAFRD     European Agricultural Fund for Rural Development

ECFIN    Directorate General for Economic and Financial Affairs

DG CNECT     Communications Networks, Content and Technology DG (ex DG INFSO)

EACI        Executive Agency on Competitiveness and Innovation

EC         European Commission

ECAS        European Commission Authentication Service

ECI         European Citizens’ initiative

e-CODEX     e-Justice Communication via Online Data Exchange

e-Sens         Electronic Simple European Networked Services

EASME    Executive Agency for SMEs

EEA        European Economic Area

EFTA        European Free Trade Association 

EID        Electronic Identity 

eIDAS    Electronic identification and trust services for electronic transactions in the internal market

EIF         European Interoperability Framework

EIS         European Interoperability Strategy

EIRA         European Interoperability Reference Architecture

EMPL        Directorate for Employment, Social Affairs & Inclusion 

ENER        Directorate General for Energy

ENV        Directorate General for the Environment

epSOS     European Patients - Smart Open Services

ERDF         European Regional Development Fund

ESF         European Social Fund

ESIF         European Structural and Investment Funds

ESPD        European Single Procurement Document

eTS        electornic Trust Services  

EU         European Union

EULF        European Union Location Network

FP7        EU 7th Framework Programme for Research and Technological Development 

GDP        Gross Domestic Product

GROW    Directorate General for Internal Market, Industry, Entrepreneurship and SMEs

HOME    Directorate General for Migration and Home Affairs

ICT         Information and Communication Technology

ICT PSP     Information and Communication Technology Policy Support Programme

INSPIRE    Infrastructure for Spatial Information for Europe Directive

IPv6        Internet protocol version 6

IRM        Independent Reporting Mechanisms

ISA         Interoperability Solutions for European public administrations

ISA²     Interoperability solutions for European public administrations, businesses and citizens

JRC    EC Joint Research Centre

JUST    Directorate for Justice and Consumers

KPIs    Key Performance Indicators 

LSPs     Large Scale Pilots

NIF         National Interoperability Framework

NIFO         National Interoperability Framework Observatory

MeAC        Measuring progress of eAccessibility in Europe

MOVE    Directorate General for Mobility and Transport

NGOs        Non-Governmental Organisations 

OECD        Organisation for Economic Co-operation and Development

OP        Publication Office of the European Union 

PEPPOL    Pan-European Public Procurement OnLine

PESTEL    Political, Economic, Social, Technological, Environmental and Legal factors

PSC         Point of Single Contact

PSI         Public Sector Information

REFIT    Regulatory Fitness and Performance of EU Legislation

REGIO    Directorate General for Regional and Urban policy

SANTE    Directorate General for Health and Food Safety

SEC    Register of Commission Document: Documents which cannot be classified in any of the other series.

SEIS        Shared Environmental Information System

SG        Secretariat-General of the European Commission

SMEs        Small and Medium Size companies 

SOA        Cloud and Service Oriented Architectures

SOLVIT    Service provided by the national administrations in each EEA EFTA country

SPOCS    Simple Procedures Online for Cross-border Services

STORK     Secure Identity Across Borders Linked

SWD        Staff Working Document 

TAXUD    Directorate for Taxation

TRADE    Directorate for Trade

UN        United Nations 

VAT        Value Added Tax

WP         Work Programme

YEI        Youth Employment Initiative

Annex 6.REFERENCES AND DATA SOURCES

-Ashfield District Council, INSPIRE and Open Data, March 2015 [available at: http://www.ashfield-dc.gov.uk/residents/democracy,-elections-and-legal/data-information-data-protection,-freedom-of-information-etc/inspire-and-open-data.aspx ]

-Austrian Institute of Economic Research (WIFO), Center for European Economic Research and IDEA Consult for the European Commission, Study Excellence in public administration for competitiveness in EU Member States, 2012 [available at: http://ec.europa.eu/growth/tools-databases/newsroom/cf/itemdetail.cfm?item_id=7970]

-Capgemini Consulting for the European Commission, Creating Value through Open Data: Study on the Impact of Re-use of Public Data Resources, 2015 [available at: http://www.europeandataportal.eu/sites/default/files/edp_creating_value_through_open_data_0.pdf ]

-Capgemini Consulting for the European Commission, Open Data Maturity in Europe 2015 Insights into the European state of play, 2015 [available at: http://www.europeandataportal.eu/en/content/open-data-maturity-europe ]

-Capgemini et al. for the European Commission, Delivering the European Advantage? How European governments can and should benefit from innovative public services, 11th eGovernment Benchmark Report, 2014 [available at: https://ec.europa.eu/digital-single-market/en/news/eu-egovernment-report-2014-shows-usability-online-public-services-improving-not-fast]

-Capgemini et al. for the European Commission, eGovernment Benchmark Framework 2012 – 2015. Method paper, July 2012 [available at: https://ec.europa.eu/digital-single-market/sites/digital-agenda/files/eGovernment%20Benchmarking%20method%20paper%20published%20version_0.pdf]

-Capgemini et al. for the European Commission, Study on Analysis of the Needs for Cross-Border Services and Assessment of the Organisational, Legal, Technical and Semantic Barriers, 2013 [available at https://ec.europa.eu/digital-single-market/news/final-report-study-analysis-needs-cross-border-services-and-assessment-organisational-legal ]

-CapGemini et al. for the European Commission, Smarter, Faster, Better eGovernment: 8th Benchmark Measurement, 2009 [available at https://joinup.ec.europa.eu/sites/default/files/files_epractice/sites/Smarter%2C%20Faster%2C%20Better%20eGovernment%20-%208th%20Benchmark%20Measurement.pdf ]

-Capgemini et al. for the European Commission, Digitizing Public Services in Europe: Putting ambition into action. 9th Benchmark Measurement , 2010 [available at: https://ec.europa.eu/digital-single-market/sites/digital-agenda/files/egov_report.pdf ]

-Capgemini et al. for the European Commission, Future-proofing eGovernment for the Digital Single Market, eGovernment Benchmarking report, 2015 [available at: https://ec.europa.eu/digital-single-market/news/eu-egovernment-report-2015-shows-online-public-services-europe-are-smart-could-be-smarter ]

-Capgemini Consulting and Eurochambres for the European Commission, The Performance of the Points of Single Contact. An Assessment against the PSC Charter, 2015 [available at: http://ec.europa.eu/growth/tools-databases/newsroom/cf/itemdetail.cfm?item_id=8342 ]

-Carlyle, S. Environment Agency for England and Wales, Reducing environmental risk through INSPIRE (quantified benefits) 2013 [available at: http://www.poweredbyinspire.eu/documents/0403-sustainability-carlyle.pdf ]

-Centre for strategy & Evaluation Services for the European Commission (CSES), Study Final Evaluation of the Competitiveness and Innovation Framework Programme, December 2011 [available at: http://ec.europa.eu/cip/files/cip/cip_final_evaluation_final_report_en.pdf ]

-Craglia M., Pavanello L. and Smith R. S., European Commission, Joint Research Centre Institute for Environment and Sustainability, The use of spatial Data for the Preparation of Environmental Reports in Europe, 2010 [available at: http://ec.europa.eu/environment/eia/pdf/jrc_technical%20report_2009%20eia-sea%20survey.pdf ]

-Ctrl-Shift, The economics of identity, June 2014 [ https://www.ctrl-shift.co.uk/news/general/2014/06/09/the-economics-of-identity/ ]

-Deloitte for the European Commission, The feasibility and scenarios for the long-term sustainability of the Large Scale Pilots, including 'ex-ante' evaluation, 2013 [available at: https://ec.europa.eu/digital-single-market/news/final-report-study-feasibility-and-scenarios-long-term-sustainability-large-scale-pilots ]

-Deloitte for the European Commission, Study on cloud and service oriented architectures for eGovernment, Final report, 2011 [available at https://ec.europa.eu/digital-single-market/sites/digital-agenda/files/smart2010-0074finalreport.pdf]

-Deloitte Consulting and Tech4i2 for the European Commission, Mid-term Evaluation of the e-Government Action Plan 2011–2015, 2014 [available at: https://ec.europa.eu/digital-single-market/news/mid-term-evaluation-e-government-action-plan-2011-2015-implementation-smart-2012-006020 ]

-Deloitte and Tech4i2 for the European Commission, Pricing Of Public Sector Information Study on Models of Supply and Charging for Public Sector Information, 2011 [available at:  https://ec.europa.eu/newsroom/dae/document.cfm?doc_id=1158 ]

-Deloitte Consulting and Tech4i2 for the European Commission, Study on The functioning and usability of the Points of Single Contact under the Services Directive, 2012 [available at:  http://ec.europa.eu/internal_market/services/docs/services-dir/study_on_points/final_report_en.pdf ]

-E-Codex project [ http://www.e-codex.eu/about-the-project.html ]

-ECORYS Nederland BV & Grontmij Nederland BV, Costs-benefits analysis INSPIRE in The Netherland, 2009. [available at: http://www.geonovum.nl/sites/default/files/nkba_engelse_vertaling.pdf ]

-EEA Technical report 17/2014. Mid-term evaluation report of INSPIRE evaluation. Joint EEA-JRC report. [available at: http://www.eea.europa.eu/publications/midterm-evaluation-report-on-inspire-implementation]

-Ernst & Young and Danish Technological Institute for the European Commission, Study on eGovernment and the Reduction of Administrative Burden, 2014 [available at https://ec.europa.eu/digital-single-market/en/news/final-report-study-egovernment-and-reduction-administrative-burden-smart-20120061 ]

-EY for the European Commission, Study on Evaluation of the impact of the free movement of EU citizens at local level, 2014 [available at http://ec.europa.eu/justice/citizen/files/dg_just_eva_free_mov_final_report_27.01.14.pdf ]

-European Commission, A vision for public services, 2013 [available at: https://ec.europa.eu/digital-single-market/news/vision-public-services ]

-European Commission, Better Regulation Guidelines [available at: http://ec.europa.eu/smart-regulation/guidelines/ug_chap1_en.htm ]

-European Commission, CEF Monitoring report Q4/2015 [available at: https://joinup.ec.europa.eu/community/cef/news/first-cef-monitoring-report-available ]

-European Commission, Charter for the electronic Points of Single Contact under the Services Directive [available at: http://ec.europa.eu/internal_market/services/docs/services-dir/psc-charter_en.pdf ]

-European Commission, Communication to the Institutions - A Digital Agenda for Europe [COM(2010) 245 final; available at: http://eur-lex.europa.eu/legal-content/EN/ALL/?uri=CELEX:52010DC0245R(01) ]

-European Commission, Communication to the Institutions - A Digital Single Market Strategy for Europe [COM(2015) 192; available at: http://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1447773803386&uri=CELEX:52015DC0192 ]

-European Commission, Communication to the Institutions - A strategy for e-procurement - [COM/2012/0179 final; available at: http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2012:0179:FIN:EN:PDF ]

-European Commission, Communication from the Commission - Annual Growth Survey 2013 [COM(2012) 750 final; available at: http://ec.europa.eu/europe2020/pdf/ags2013_en.pdf ]

-European Commission, Communication to the Institutions - Annual Growth Survey 2015 [COM(2014) 902 final; available at: http://ec.europa.eu/europe2020/pdf/2015/ags2015_en.pdf ]

-European Commission, Communication to the Institutions - Annual Growth Survey 2016 [COM(2015) 690; available at: http://ec.europa.eu/europe2020/pdf/2016/ags2016_annual_growth_survey.pdf]

-European Commission, Communication to the Institutions - Better regulation for better results - An EU agenda [COM(2015)215 final of 19 May 2015; available at: http://ec.europa.eu/smart-regulation/better_regulation/documents/com_2015_215_en.pdf ]

-European Commission, Communication to the Institutions - End-to-end e-procurement to modernise public administration [COM/2013/0453 final; available at: http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2013:0453:FIN:EN:PDF ]

-European Commission, Communication to the Institutions – Europe 2020 strategy [COM(2010)2020; available at: http://ec.europa.eu/eu2020/pdf/COMPLET%20EN%20BARROSO%20%20%20007%20-%20Europe%202020%20-%20EN%20version.pdf ]

-European Commission, Communication to the Institutions, Towards interoperability for European public services [COM(2010) 744 final, available at http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2010:0744:FIN:EN:PDF]

-European Commission, Communication to the Institutions, European Interoperability Framework (EIF) for European public services [COM(2010) 744 final, Annex 2; available at http://ec.europa.eu/isa/actions/04-accompanying-measures/4-2-3action_en.htm ]

-European Commission, Communications to Institutions - Evaluations of the Competitiveness and Innovation Framework Programme [COM(2013)2; available at: http://ec.europa.eu/cip/documents/implementation-reports/index_en.htm ]

-European Commission, Communication to the Institutions - i2010 eGovernment Action Plan: Accelerating eGovernment in Europe for the Benefit of All [COM(2006) 173 final; available at: http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52006DC0173&from=EN ]

-European Commission, Communication to the Institutions - On the implementation of the Services Directive. A partnership for new growth in services 2012-2015 [COM(2012) 261 final; available at: http://ec.europa.eu/internal_market/services/docs/services-dir/implementation/report/COM_2012_261_en.pdf ]

-European Commission, Communication to the Institutions - Open data - An engine for innovation, growth and transparent governance [COM(2011) 882 final; available at http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2011:0882:FIN:EN:PDF ] 

-European Commission, Communication to the Institutions - The European eGovernment Action Plan 2011-2015, Harnessing ICT to promote smart, sustainable & innovative Government [COM(2010) 743; available at: http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2010:0743:FIN:EN:PDF ]

-European Commission, Communication to the Institutions - Towards interoperability for European public services [COM(2010) 744 final; available at: http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2010:0744:FIN:EN:PDF ]

-European Commission, Communication to the Institutions - Unleashing the Potential of Cloud Computing in Europe [COM(2012)0529 final; available at: http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2012:0529:FIN:EN:PDF ]

-European Commission, Communication to the Institutions -Upgrading the Single Market: more opportunities for people and business [COM(2015)550; available at  https://ec.europa.eu/transparency/regdoc/rep/1/2015/EN/1-2015-550-EN-F1-1.PDF ]

-European Commission, Competitiveness and Innovation Framework Programme ICT Policy Support Programme [Archived site: http://ec.europa.eu/cip/ict-psp/index_en.htm ]

-European Commission, Competitiveness and Innovation Framework Programme (CIP)- Implementation Reports [available at: http://ec.europa.eu/cip/documents/implementation-reports/index_en.htm ]

-European Commission, Connecting Europe Facility [https://ec.europa.eu/digital-single-market/connecting-europe-facility]

-European Commission, Contributions and preliminary trends of the public consultation on the eGovernment Action Plan 2016 – 2020, 2016 [available at: https://ec.europa.eu/digital-single-market/news/contributions-and-preliminary-trends-public-consultation-egovernment-action-plan-2016-2020 ]

-European Commission, DG Internal Market and Services: Management Plan 2013, 2012 [ http://ec.europa.eu/dgs/internal_market/docs/amp_2013_en.pdf ]

-European Commission, Digital Agenda, Action 91: Member States to agree a common list of key cross-border public services [https://ec.europa.eu/digital-single-market/pillar-vii-ict-enabled-benefits-eu-society/action-91-member-states-agree-common-list-key-cross]

-European Commission, Digital Agenda Scoreboard report 2011, Pillar 7 [available at: https://ec.europa.eu/digital-single-market/download-scoreboard-reports ]

-European Commission, Digital Agenda Scoreboard report 2012. [available at https://ec.europa.eu/digital-single-market/download-scoreboard-reports ]

-European Commission, Digital Agenda Scoreboard reports [available at: https://ec.europa.eu/digital-single-market/download-scoreboard-reports ]

-European Commission, Digital Agenda Targets Progress report, 2015 [available at:  https://ec.europa.eu/digital-single-market/download-scoreboard-reports ]

-European Commission, Digital Economy and Society Index DESI [available at https://ec.europa.eu/digital-single-market/digital-economy-and-society-index-desi]

-European Commission, Digital Single Market, Connecting Europe Facility [ https://ec.europa.eu/digital-single-market/connecting-europe-facility ]

-European Commission, e-Commission 2012 – 2015, Communication from VP Šefčovič to the Commission -Delivering user-centric digital services [SEC(2012) 492; available at:  http://ec.europa.eu/dgs/informatics/ecomm/doc/communication_sefcovic_tothecom.pdf ]

-European Commission, eGovernment Action Plan 2016-2020, Roadmap [available at: http://ec.europa.eu/smart-regulation/roadmaps/docs/2016_cnect_006_e_government_action_plan_en.pdf ]

-European Commission, eHealth Action Plan 2012-2020: Innovative healthcare for the 21st century [https://ec.europa.eu/digital-single-market/news/ehealth-action-plan-2012-2020-innovative-healthcare-21st-century]

-European Commission, e-PRIOR Supplier Portal [ http://ec.europa.eu/dgs/informatics/supplier_portal/index_en.htm ]

-European Commission, ePSIplatform [ http://epsiplatform.eu/ and http://www.epsiplatform.eu/content/european-psi-scoreboard ]

-European Commission, EU eGovernment Report 2015 shows that online public services in Europe are smart but could be smarter, 23rd June 2015 [https://ec.europa.eu/digital-single-market/news/eu-egovernment-report-2015-shows-online-public-services-europe-are-smart-could-be-smarter]

-European Commission, EU implementation of the G8 Open Data Charter [https://ec.europa.eu/digital-single-market/news/eu-implementation-g8-open-data-charter]

-European Commission, European citizens' initiative (ECI) [ http://ec.europa.eu/citizens-initiative/public/welcome ]

-European Commission, European Union Open Data Portal [ https://open-data.europa.eu ]

-European Commission, Europe's Digital Competitiveness Report 2010 [available at: https://ec.europa.eu/digital-single-market/sites/digital-agenda/files/european_competitiveness_report_2010.pdf]

-European Commission, Eurostat, E-government usage by individuals [available at: http://ec.europa.eu/eurostat/en/web/products-datasets/-/TSDGO330 ]

-European Commission, Eurostat, Foreign citizens living in the EU Member States, 2015 [available at: http://ec.europa.eu/eurostat/documents/2995521/7113991/3-18122015-BP-EN.pdf/d682df12-8a77-46a5-aaa9-58a00a8ee73e]

-European Commission, Eurostat, ICT survey of Households and Individuals, 2014 [available at: http://ec.europa.eu/eurostat/statistics-explained/index.php/Information_society_statistics_-_households_and_individuals ]

-European Commission, Eurostat, Information society indicator "Enterprises using the internet for interacting with public authorities"

-European Commission, Eurostat, Individuals using the internet for interaction with public authorities [available at: http://ec.europa.eu/eurostat/tgm/table.do?tab=table&init=1&plugin=1&pcode=tin00012&language=en ]

-European Commission, Eurostat, Migration and migrant population statistics [available at: http://ec.europa.eu/eurostat/statistics-explained/index.php/Migration_and_migrant_population_statistics]

-European Commission, GROWTH, E-Certis [https://ec.europa.eu/growth/tools-databases/ecertis/]

-European Commission, GROWTH, E-procurement [ http://ec.europa.eu/growth/single-market/public-procurement/e-procurement/index_en.htm ]

-European Commission, GROWTH, European Single Procurement Document (ESPD) [ https://ec.europa.eu/growth/tools-databases/espd ]

-European Commission, GROWTH, New rules on public procurement and concession contracts as of April 2016 [ http://ec.europa.eu/growth/single-market/public-procurement/new/index_en.htm ]

-European Commission, GROWTH, Single Market and Standards – New rules on Public procurement and concession contracts as of April 2016 [ http://ec.europa.eu/growth/single-market/public-procurement/new/index_en.htm ]

-European Commission, Guidance on Ex ante Conditionalities for the European Structural and Investment Funds, 2014 [available at: http://ec.europa.eu/regional_policy/sources/docgener/informat/2014/eac_guidance_esif_part2_en.pdf ]

-European Commission, High Level Group on Business Services, Final Report, 2014 [available at: http://ec.europa.eu/growth/industry/policy/renaissance/high-level-group/index_en.htm ]

-European Commission, Horizon 2020 Work Programme 2016 – 2017, Europe in a changing world – inclusive, innovative and reflective Societies [European Commission Decision C (2015)6776 of 13 October 2015; available at: http://ec.europa.eu/research/participants/data/ref/h2020/wp/2016_2017/main/h2020-wp1617-societies_en.pdf ]

-European Commission, INSPIRE, MIG, National Implementation Seminars [ https://ies-svn.jrc.ec.europa.eu/projects/mig-inspire/wiki/National_implementation_webinars ]

-European Commission, Interoperability Solutions for European Public Administrations [ http://ec.europa.eu/isa/ ]

-European Commission, ISA programme, Common Infrastructure for Public Administrations Sustainability [ http://ec.europa.eu/isa/actions/01-trusted-information-exchange/1-6_1-19action_en.htm ]

-European Commission, Joinup platform [ https://joinup.ec.europa.eu/ ]

-European Commission, Joinup, Connecting Europe Facility, Catalogue of Building Blocks, 2014 [ https://joinup.ec.europa.eu/community/cef/og_page/catalogue-building-blocks ]

-European Commission, Justice, Protection of personal data [ http://ec.europa.eu/justice/data-protection/ ]

-European Commission, Justice, Reform of EU data protection rules [ http://ec.europa.eu/justice/data-protection/reform/index_en.htm ]

-European Commission, Justice, EU Justice Scoreboard [http://ec.europa.eu/justice/effective-justice/scoreboard/index_en.htm]

-European Commission, MEMO/10/200, 19th May 2010, Digital Agenda for Europe: key initiatives [available at: http://europa.eu/rapid/press-release_MEMO-10-200_en.htm?locale=en ]

-European Commission, Mid-Term Evaluation of the eGovernment Action Plan - Evaluation of the eGovernment Action Plan [ http://www.egovap-evaluation.eu ]

-European Commission, National Interoperability Framework Observatory, eGovernment Factsheets 2015 (published in 2016) [available at: https://joinup.ec.europa.eu/community/nifo/og_page/egovernment-factsheets ]

-European Commission, National Interoperability Framework Observatory, Factsheets (published in 2015) [available at: https://joinup.ec.europa.eu/community/nifo/og_page/nifo-factsheets ]

-European Commission, National Interoperability Framework Observatory (NIFO), JoinUp community [ https://joinup.ec.europa.eu/community/nifo/home ]

-European Commission, Performance of the Points of Single Contact - an assessment against the PSC Charter, 29.6.2015 [ http://ec.europa.eu/growth/tools-databases/newsroom/cf/itemdetail.cfm?item_id=8342 ]

-European Commission, Press Release IP/11/1524, 12 December 2011 [available at: http://europa.eu/rapid/press-release_IP-11-1524_en.htm?locale=en ]

-European Commission, Press Release IP/13/608, June 2013 "E-invoicing in public procurement: another step towards end-to-end e-procurement and e-government in Europe" [ http://europa.eu/rapid/press-release_IP-13-608_en.htm ]

-European Commission, Proposal for a Directive of the European Parliament and of the Council on the accessibility of public sector bodies' websites [COM(2012) 721 final; available at http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2012:0721:FIN:EN:PDF ]

-European Commission, PSI Scoreboard Indicator List [ http://www.epsiplatform.eu/content/psi-scoreboard-indicator-list ]

-European Commission, Public Services, Cross-border solutions [https://ec.europa.eu/digital-single-market/node/77121]

-European Commission, Public Services, eParticipation [https://ec.europa.eu/digital-single-market/eparticipation]

-European Commission, Public Services, ICT-enabled public sector innovation in Horizon 2020 [https://ec.europa.eu/digital-single-market/ict-enabled-public-sector-innovation-horizon-2020]

-European Commission, Public Services, Towards a Cloud of Public Services [https://ec.europa.eu/digital-single-market/towards-cloud-public-services]

-European Commission, Regional Policy- European Structural and Investment Funds [ http://ec.europa.eu/regional_policy/en/funding/ ]

-European Commission, Regional Policy, Thematic objectives [ http://ec.europa.eu/regional_policy/en/policy/what/glossary/t/thematic-objectives ]

-European Commission, Staff Working Document – A Digital Single Market Strategy for Europe - Analysis and Evidence – [SWD (2015) 100 final; available at: http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52015SC0100&from=EN ]

-European Commission, Staff Working Document – i2010 eGovernment Action Plan: Accelerating eGovernment in Europe for the Benefit of All [SEC(2006) 511; available at: http://ec.europa.eu/smart-regulation/impact/ia_carried_out/docs/ia_2006/sec_2006_0511_en.pdf ]

-European Commission, Staff Working Document, Impact assessment, Proposal for a Directive of the European Parliament and of the Council on Single-member private limited liability companies [SWD(2014) 124 final; available at: http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52014SC0124&qid=1458297454242&from=EN ]

-European Commission, Staff Working Document - The European eGovernment Action Plan 2011-2015 –Harnessing ICT to promote smart, sustainable & innovative Government [SEC(2010) 1539 final; available at http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=SEC:2010:1539:FIN:EN:PDF ]

-European Commission, The European e-Justice Portal [ https://e-justice.europa.eu/home.do ]

-European Commission, Workshop "Improving Transparency in Government Decision Making" 6 November 2012 [https://ec.europa.eu/digital-single-market/news/workshop-improving-transparency-government-decision-making-6112012-results-presentations]

-European Council, Conclusions, 24/25 October 2013; [EUCO 169/13, available at: http://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/ec/139197.pdf ]

-European Parliament, European Parliament resolution of 19 January 2016 on Towards a Digital Single Market Act [2015/2147(INI); available at: http://www.europarl.europa.eu/sides/getDoc.do?pubRef=-//EP//TEXT+TA+P8-TA-2016-0009+0+DOC+XML+V0//EN ]

-Fornefeld, M. et al., Assessment of the Re-use of Public Sector Information (PSI) in the Geographical information, Meteorological Information and Legal Information Sectors, MICUS Management Consulting GmbH for the European Commission, 2008 [available at: https://ec.europa.eu/digital-single-market/news/assessment-re-use-public-sector-information-psi-geographical-information-meteorological ]

-Funka et al. for the European Commission, Study Measuring Progress of eAccessibility in Europe – MeAC, 2013 [available at: http://www.funka.com/en/our-assignments/position-of-trust/archive---commission-of-trust/measuring-progress-of-eaccessibility-in-europe---meac2/ ]

-GEN6 - Governments ENabled with IPv6 [ http://www.gen6.eu/home ]

-Geopunt, the Flemish Geoportal [ http://inspire.ec.europa.eu/events/conferences/inspire_2013/pdfs/26-06-2013_ROOM-4_09.00%20-%2010.30_95-Joeri%20Robbrecht_Joeri-Robbrecht.pdf ]

-KPMG for the European Commission; Gatti, R. Carbone, L. Mezzapesa,V, Study State of play of interoperability in Europe – Report 2014 [available at:  http://ec.europa.eu/isa/documents/publications/2014-report-on-state-of-play-of-interoperability.pdf ]

-London Economics for the European Parliament study, A European Single Point of Contact, 2013; [available at: http://www.europarl.europa.eu/RegData/etudes/etudes/join/2013/507453/IPOL-IMCO_ET(2013)507453_EN.pdf ]

-Ministerial Declaration on eGovernment, Malmö, Sweden, on 18 November 2009 [available at: https://ec.europa.eu/digital-single-market/sites/digital-agenda/files/ministerial-declaration-on-egovernment-malmo.pdf ]

-Open Data Barometer [http://www.opendatabarometer.org/]

-Open Government Partnership, Independent Progress Mechanism Reports [ http://www.opengovpartnership.org/independent-reporting-mechanism ]

-Organisation for Economic Co-operation and Development (OECD)- Recommendation on Digital Government Strategies, 2014 [available at: http://www.oecd.org/gov/public-innovation/recommendation-on-digital-government-strategies.htm ]

-Sixth European Ministerial eGovernment Conference “Borderless eGovernment Services for Europeans”, Conference proceedings, November 2011 [available at: https://mac.gov.pl/files/wp-content/uploads/2011/12/6th-BeGSE-folder-A4-CP-EN.pdf ]

-STORK2 project [ www.eid-stork2.eu ]

-Tech4i2 and Deloitte for the European Commission, Study on collaborative production in eGovernment, 2012 [available at: https://ec.europa.eu/digital-single-market/news/study-collaborative-production-egovernment-smart-20100075 ]

-TNS Opinion & Social for the European Commission- Flash Eurobarometer 365, European Union citizenship, 2013 [available at:   http://ec.europa.eu/public_opinion/flash/fl_365_en.pdf ]

-Tumin, Z, Fung, A. From Government 2.0 to Society 2.0: Pathways to Engagement, Collaboration and Transformation, October 2011 [available at: http://belfercenter.ksg.harvard.edu/publication/21475/from_government_20_to_society_20.html ]

-UK Environment Agency, Reducing environmental risk through INSPIRE (quantified benefits), 2013 [available at: http://www.poweredbyinspire.eu/documents/0403-sustainability-carlyle.pdf]

-UK Location Programme - Benefits Realisation Strategy Final, 2012 [available at: https://data.gov.uk/sites/default/files/Benefits%20Realisation%20Strategy%20v2.0%20Final_10.pdf ]

-Vickery, G. Review of recent studies on PSI re-use and related market developments, Information Economics for the European Commission, 2011 [available at: https://ec.europa.eu/digital-single-market/news/review-recent-studies-psi-reuse-and-related-market-developments ]

-Vickery,G. Luukkonen,T. Radosevic,S and Fisher, R, for the European Commission, CIP ICT PSP Final (Second Interim) Evaluation-Final report, 2011 [available at: http://ec.europa.eu/cip/files/cip/docs/cip_ict_psp_interim_evaluation_report_2011_en.pdf ]

-World Wide Web Foundation, Open data barometer Second Edition, January 2015 [ http://www.opendatabarometer.org/ ]

Legislation:

-Decision (EU) 2015/2240 Of the European Parliament and of the Council of 25 November 2015 establishing a programme on interoperability solutions and common frameworks for European public administrations, businesses and citizens (ISA2 programme) as a means for modernising the public sector

-Directive 2004/18/EC of the European Parliament and the Council of 31 March 2004 on the coordination of procedures for the award of public works contracts, public supply contracts and public service contracts

-Directive 2007/2/EC Of the European Parliament and of the Council of 14 March 2007 establishing an Infrastructure for Spatial Information in the European Community (INSPIRE)

-Directive 2012/17/EU Of the European Parliament and of the Council of 13 June 2012 amending Council Directive 89/666/EEC and Directives 2005/56/EC and 2009/101/EC of the European Parliament and of the Council as regards the interconnection of central, commercial and companies registers. 

-Directive 2014/25/EU of the European Parliament and the Council of 26 February 2014 on procurement by entities operating in the water, energy, transport and postal services sectors

-European Commission, Decision of 12 December 2011 on the reuse of Commission documents [2011/833/EU]

-Regulation(EU) No 283/2014 of the European Parliament and of the Council of 11 March 2014 on guidelines for trans-European networks in the area of telecommunications infrastructure and repealing Decision No 1336/97/EC 

-Regulation (EU) No 910/2014 of the European Parliament and of the Council of 23 July 2014 on electronic identification and trust services for electronic transactions in the internal market and repealing Directive 1999/93/EC

(1)

COM(2010) 743

(2)

https://ec.europa.eu/digital-agenda/sites/digital-agenda/files/ministerial-declaration-on-egovernment-malmo.pdf

(3)

http://ec.europa.eu/smart-regulation/guidelines/ug_chap1_en.htm

(4)

COM(2015) 0192

(5)

COM(2010) 245

(6)

COM(2010) 2020

(7)

COM(2010) 245

(8)

COM(2006) 173 final.

(9)

COM(2010) 744

(10)

COM(2010) 744 final, Annex 2

(11)

COM(2012) 0529 final

(12)

  http://ec.europa.eu/cip/ict-psp/index_en.htm and https://ec.europa.eu/digital-single-market/node/77121

(13)

https://ec.europa.eu/digital-single-market/towards-cloud-public-services

(14)

https://ec.europa.eu/digital-single-market/connecting-europe-facility

(15)

https://ec.europa.eu/digital-single-market/ict-enabled-public-sector-innovation-horizon-2020

(16)

http://ec.europa.eu/isa/

(17)

Before 2014 "European Structural funds"

(18)

http://ec.europa.eu/regional_policy/en/policy/what/glossary/t/thematic-objectives

(19)

COM(2011) 877 final.

(20)

Directive 2013/37/EU amending Directive 2003/98/EC on the re-use of public sector information, OJ L 175/1.

(21)

 European Commission, Decision of 12 December 2011 on the re-use of Commission documents,, repealing Decision 2006/291/EC, Euratom.

(22)

Directive on procurement in the water, energy, transport and postal services sectors; and Directive on public works, supply and service contracts

(23)

  http://ec.europa.eu/growth/single-market/public-procurement/new/index_en.htm  

(24)

Regulation (EU) No 283/2014..

(25)

http://ec.europa.eu/justice/data-protection/

(26)

COM(2012) 721 final

(27)

Decision (EU) 2015/2240

(28)

SEC(2010) 1539 final

(29)

 Source: Smarter, Faster, Better eGovernment: 8th Benchmark Measurement, 2009.

(30)

Basic Services refer to the 20 services (12 for citizens, 8 for businesses) used to benchmark online availability of public services. These are: income taxes, job search, social security benefits, personal documents, car registration, building permission, declaration to police, public libraries, certificates, enrolment in higher education, announcement of moving, health-related services (citizens), social contributions, corporate tax, VAT, company registration, statistical data, customs declaration, environment-related permits, public procurement (businesses).

(31)

Full online availability is an indicator which measures whether the service is delivered in a completely electronic way without need of interacting through traditional (i.e. paper, face-to-face) channels. It corresponds to level 4 and above of the sophistication indicator introduced below. The composite indicator is an average of the values taken in the 20 services.

(32)

The small decrease in 2004 was due to an enlargement of the sample to New Member States: until 2003 the sample included only EU15 countries.

(33)

Eurostat. E-government usage by individuals.

(34)

Eurostat. Individuals using the internet for interaction with public authorities and Digital Agenda Scoreboard report 2011. Pillar 7

(35)

Digital Agenda Scoreboard report 2012.

(36)

SEC(2010) 1539

(37)

Archived site: http://ec.europa.eu/information_society/activities/ict_psp/index_en.htm

(38)

Pan-European Public Procurement OnLine

(39)

 Secure idenTity acrOss boRders LinKed

(40)

 Simple Procedures Online for Cross-border Services

(41)

This figure is based on the total estimated eligible costs of STORK, PEPPOL and SPOCS for the entire duration of the projects including the share of co-funding of participating Member States and other partners.

(42)

 Digitizing Public Services in Europe: Putting ambition into action. 9th Benchmark measurement. 2010.

(43)

 Mid-term Evaluation of the e-Government Action Plan 2011–2015. 2014.

(44)

NIFO eGovernment factsheets (edition 2015 published in 2016).

(45)

http://ec.europa.eu/digital-agenda/en/digital-economy-and-society-index-desi

(46)

https://ec.europa.eu/digital-agenda/download-scoreboard-reports

(47)

 eGovernment Benchmark Framework 2012 – 2015. Method paper, July 2012.

(48)

 Future-proofing eGovernment for the Digital Single Market. eGovernment Benchmark report, 2015.

(49)

 The Mystery Shopping had consultants simulate the user journey of citizens dealing with the public administration in order to get some services. The tested services (both for services and enterprises) were seven different families of services grouped around a specific user need (e.g. services around starting a business) embodied by the concept of life event. The User Survey was conducted on an online panel of 26000 citizens across 32 countries, and participants were asked about their level of interaction with public administrations both online and offline and about their degree of satisfaction with the online channel. The User Survey has been conducted only in 2012, while the Mystery Shopping took place in all the years, but alternating the measurement of half of life events each year.

(50)

Digital Agenda Europe baseline: in 2009, 38% of individuals aged 16 – 74 had used eGovernment services in the last 12 months.

(51)

 Digital Agenda Targets Progress report. 2015.

(52)

Eurostat. Individuals using the internet for interaction with public authorities.

(53)

 Digitizing Public Services in Europe: Putting ambition into action. 9th Benchmark measurement. 2010; Digital Agenda Scoreboard Report 2012.

(54)

Eurostat: "Enterprises using the internet for interacting with public authorities".

(55)

https://ec.europa.eu/digital-single-market/pillar-vii-ict-enabled-benefits-eu-society/action-91-member-states-agree-common-list-key-cross

(56)

Sixth European Ministerial eGovernment Conference “Borderless eGovernment Services for Europeans”, Conference proceedings, November 2011.

(57)

ICT Policy Support Programme (ICT PSP) was one of three specific programmes of the Competitiveness and Innovation Framework Programme (CIP).

(58)

European Patients Smart Open Services project

(59)

E-Justice Communication via Online Data Exchange project

(60)

Electronic Simple European Networked Services

(61)

 Study on Analysis of the Needs for Cross-Border Services and Assessment of the Organisational, Legal, Technical and Semantic Barriers, 2013.

(62)

Regulation (EU) No 283/2014..

(63)

EC, Joinup, Connecting Europe Facility, Catalogue of Building Blocks, 2014.

(64)

CEF Monitoring report Q4/2015.

(65)

 Future-proofing eGovernment for the Digital Single Market, eGovernment Benchmark report, 2015

(66)

 Future-proofing eGovernment for the Digital Single Market, eGovernment Benchmark report, 2015. Background report, p. 32.

(67)

http://www.egovap-evaluation.eu/

(68)

Digital Agenda Scoreboard Report 2012.

(69)

As reported by Eurostat on data "Enterprises using the internet for interacting with public authorities"

(70)

 Digital Agenda Targets Progress report, 2015.

(71)

Eurostat, Individuals using the internet for interaction with public authorities

(72)

Digital Agenda Scoreboard report 2011, Pillar 7.

(73)

CIP ICT PSP Final (Second Interim) Evaluation-Final report, 2011 and COM (201302.

(74)

Regulation (EU) No 910/2014

(75)

 Mid-term evaluation of the eGovernment Action Plan 2011 - 2015 (p. 34).

(76)

 https://joinup.ec.europa.eu/

(77)

COM(2012) 721 final

(78)

Study Measuring Progress of eAccessibility in Europe – MeAC, 2013 

(79)

https://open-data.europa.eu

(80)

 Open Data Maturity in Europe 2015 Insights into the European state of play, 2015

(81)

  https://ec.europa.eu/digital-single-market/en/news/eu-implementation-g8-open-data-charter

(82)

 Future-proofing eGovernment for the Digital Single Market, eGovernment Benchmarking report, 2015. Insight report, p. 25

(83)

The data is collected through life event assessment by "Mystery Shoppers". More details of the method are in chapter 5.2.

(84)

http://www.epsiplatform.eu/content/european-psi-scoreboard

(85)

http://www.europeandataportal.eu/

(86)

 Method used: "Mystery Shopping". This top level (compound) benchmark assesses the availability and usability of public eServices.

(87)

 Future-proofing eGovernment for the Digital Single Market. eGovernment Benchmarking report 2015, Background report, p. 23

(88)

 Future-proofing eGovernment for the Digital Single Market. eGovernment Benchmarking report 2015, Insight report, p.19.

(89)

COM(2013)2

(90)

 The feasibility and scenarios for the long-term sustainability of the Large Scale Pilots, including 'ex-ante' evaluation, 2013

(91)

 The Performance of the Points of Single Contact. An Assessment against the PSC Charter, 2015, p. 50

(92)

  EEA Technical report No 17/2014 . Joint EEA-JRC report.

(93)

 Future-proofing eGovernment for the Digital Single Market, eGovernment Benchmarking report, 2015. Background report. Pp. 46 - 47

(94)

For example, the ISA programme activity on Common Infrastructure for Public Administrations Sustainability

(95)

http://ec.europa.eu/digital-agenda/en/news/eu-egovernment-report-2015-shows-online-public-services-europe-are-smart-could-be-smarter

(96)

 Future-proofing eGovernment for the Digital Single Market, eGovernment Benchmarking report, 2015. Insight report, p. 25.

(97)

Mid-term evaluation of the eGovernment Action Plan 2011 – 2015, 2014, p. 41

(98)

The study extrapolated that the extension of the Danish approach to implement the “once-only” principle is likely to generate an annual net saving at the EU 28 level, amounting to around € 5 billion per year by 2017. This highly positive impact depends on a complex system of registries being freely accessible by users (citizens and businesses) for commercial purposes, which additionally might foster growth in some economic sectors.

(99)

European Council, Conclusions, 24/25 October 2013

(100)

H2020 Work Programme 2016 – 2017, Europe in a changing world – inclusive, innovative and reflective societies. Co-creation-05-2016: Co-creation between public administrations: once-only principle.

(101)

COM(2015)215

(102)

  EEA Technical report No 17/2014 . Joint EEA-JRC report.

(103)

 Future-proofing eGovernment for the Digital Single Market, eGovernment Benchmarking report, 2015. Background report. Insight report, p. 30

(104)

 Mid-term Evaluation of the e-Government Action Plan 2011–2015, pp. 48 – 49.

(105)

  http://ec.europa.eu/cip/ict-psp/index_en.htm and https://ec.europa.eu/digital-single-market/en/node/77121

(106)

https://ec.europa.eu/digital-single-market/en/connecting-europe-facility

(107)

https://ec.europa.eu/digital-single-market/en/ict-enabled-public-sector-innovation-horizon-2020

(108)

http://ec.europa.eu/isa/

(109)

European Regional Development Fund, Cohesion Fund, European Social Fund, Youth Employment Initiative and European Agricultural Fund for Rural Development

(110)

https://ec.europa.eu/digital-agenda/en/news/ehealth-action-plan-2012-2020-innovative-healthcare-21st-century

(111)

http://ec.europa.eu/cip/documents/implementation-reports/index_en.htm

(112)

COM(2013)2

(113)

 Final Evaluation of the Competitiveness and Innovation Framework Programme , 2011

(114)

Executive Agency on Competitiveness and Innovation, as of 2014, Executive Agency for SMEs (EASME)

(115)

 Tool available at http://ec.europa.eu/regional_policy/en/funding/   

(116)

Annual Growth Survey 2013, COM(2012) 750 final

(117)

DG Internal Market and Services: Management Plan 2013, 2012

(118)

 The economics of identity, June 2014

(119)

A strategy for e-procurement, COM(2012)0179 final; http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2012:0179:FIN:EN:PDF  

(120)

End-to-end e-procurement to modernise public administration, COM(2013)0453 final

(121)

 E-invoicing in public procurement: another step towards end-to-end e-procurement and e-government in Europe (Press Release, June 2013)

(122)

 Study on eGovernment and the Reduction of Administrative Burden, 2014

(123)

Excellence in public administration for competitiveness in EU Member States, 2012

(124)

COM(2014) 902 and COM(2015) 690

(125)

 A Digital Single Market Strategy for Europe – Analysis and Evidence, SWD(2015) 100

(126)

 Study on eGovernment and the Reduction of Administrative Burden, 2014

(127)

 Future-proofing eGovernment for the Digital Single Market, eGovernment Benchmarking report, 2015. Insight report. 

(128)

The forthcoming EU Justice Scoreboard 2016.

(129)

Flash Eurobarometer 365, European Union citizenship, 2013

(130)

Directive (2013/37/EU), article 13.1.

(131)

 Review of recent studies on PSI re-use and related market developments, 2011

(132)

 Assessment of the Re-use of Public Sector Information (PSI) in the Geographical information, Meteorological Information and Legal Information Sectors, 2008

(133)

UK Location Programme - Benefits Realisation Strategy Final, 2012 - estimates the quantifiable INSPIRE benefits at 70-130 m£/year to UK environmental protection activities and the quantifiable benefits across UK government departments at 470-510 m£/year.

(134)

 Costs-benefits analysis INSPIRE in The Netherland, 2009

(135)

Reducing environmental risk through INSPIRE, UK Environment Agency, 2013

(136)

 The use of spatial Data for the Preparation of Environmental Reports in Europe, 2010 EUR24327 EN - 2010  

(137)

For example: 57% of INSPIRE data is OPEN in Finland and served through INSPIRE services (INSPIRE, MIG, National Implementation Seminars); Ashfield District Council (INSPIRE and Open Data, 2015) publishes data under the European INSPIRE Directive and as Open Data.

(138)

COM(2011) 882 final

(139)

For example: INSPIRED eGovernment Apps - Geopunt, the Flemish Geoportal.

(140)

  http://open-data.europa.eu/

(141)

http:// www.europeandataportal.eu

(142)

 Creating Value through Open Data: Study on the Impact of Re-use of Public Data Resources, 2015

(143)

http://ec.europa.eu/justice/data-protection/reform/index_en.htm

(144)

Eurostat, Migration and migrant population statistics, 2015

(145)

Eurostat, Foreign citizens living in EU Member States, 2015

(146)

 Study on Evaluation of the impact of the free movement of EU citizens at local level, 2014

(147)

 Study on Analysis of the Needs for Cross-Border Services and Assessment of the Organisational, Legal, Technical and Semantic Barriers, 2013, p. 45 and figure 26.

(148)

COM(2013) 453 final

(149)

  http://ec.europa.eu/growth/single-market/public-procurement/new/index_en.htm  

(150)

http://ec.europa.eu/growth/single-market/public-procurement/e-procurement/index_en.htm

(151)

SWD (2015) 100 final

(152)

SWD(2014) 124 final, p. 27 and examples of differing national reforms in company law p. 21 et seq.

(153)

COM(2015)550, p. 5.

(154)

The European e-Justice Portal is available at: https://e-justice.europa.eu/home.do

(155)

In line with Directive 2012/17/EU.

(156)

European Parliament study, A European Single Point of Contact, 2013; European Commission, High Level Group on Business Services - Final Report, 2014

(157)

  SWD(2012) 148 final

(158)

 Study on The functioning and usability of the Points of Single Contact under the Services Directive, 2012

(159)

http://ec.europa.eu/internal_market/services/docs/services-dir/implementation/report/COM_2012_261_en.pdf

(160)

http://ec.europa.eu/internal_market/services/docs/services-dir/psc-charter_en.pdf

(161)

 The Performance of the Points of Single Contact. An Assessment against the PSC Charter, 2015

(162)

 Delivering the European Advantage? How European governments can and should benefit from innovative public services, 11th eGovernment Benchmark Report, 2014, p. 49

(163)

 Future-proofing eGovernment for the Digital Single Market, eGovernment Benchmarking report, 2015, Background report, p. 35

(164)

 Future-proofing eGovernment for the Digital Single Market, eGovernment Benchmarking report, 2015

(165)

Eurostat, ICT survey of Households and Individuals, 2014; European Commission, Digital Agenda Scoreboard – the indicator has a 12-month reference period and concerns the use of web applications for uploading completed forms or transmitting web forms with details filled in directly.

(166)

EC, Digital Economy and Society Index DESI, February 2016

(167)

 Future-proofing eGovernment for the Digital Single Market, eGovernment Benchmarking report, 2015, p. 35.

(168)

European Parliament resolution of 19 January 2016 on Towards a Digital Single Market Act

(169)

   Study on Analysis of the Needs for Cross-Border Services and Assessment of the Organisational, Legal, Technical and Semantic Barriers, 2013

(170)

   COM(2010) 744 final

(171)

The Interoperability Solutions for European Public Administrations (ISA) programme (2010-2015) monitored and supported the EIF implementation in Europe; it was followed by the ISA² programme

(172)

Digital Single Market Communication 2015. Staff Working Document, SWD (2015) 100 final.

(173)

Guidance on Ex ante Conditionalities for the European Structural and Investment Funds, 2014

(174)

http://ec.europa.eu/growth/single-market/public-procurement/new/index_en.htm

(175)

Digital Economy and Society Index DESI, 2016. Future-proofing eGovernment for the Digital Single Market, eGovernment Benchmarking report, 2015. Share of the steps in a public services life event that can be completed online, seven life events.

(176)

 Mid-term Evaluation of the e-Government Action Plan 2011–2015. 2014.

(177)

NIFO eGovernment factsheets (edition 2015 published in 2016).

(178)

http://ec.europa.eu/digital-agenda/en/digital-economy-and-society-index-desi

(179)

https://ec.europa.eu/digital-agenda/download-scoreboard-reports

(180)

     See Annex 2, report annex 1 A, section 1

(181)

     The evaluation takes into account the inherent limitations of the findings of public consultations. Firstly as in all surveys, the answers received reflect the views of a sample of relevant stakeholders and not those of the entire population who has a stake in this domain. Secondly, stakeholders views convey an individual rather than a more general and representative perspective.

(182)

     See Annex 2, report annex 1 A, section 2

(183)

   See Annex 2, report annex 1 A, section 3

(184)

     See Annex 2, report annex 1A, section 4

(185)

     See Annex 2, report annex 1A, section 5

(186)

     See Annex 2, report annex 1A, section 6

(187)

     See Annex 2, report annex 1A, section 7

(188)

     For the full question see Annex 2, report annex 1B

(189)

     See Annex 2, report annex 1 A, section 8

(190)

   This action is defined in the Digital Agenda for Europe

(191)

   This action is defined in the Digital Agenda for Europe

(192)

   This action is defined in the Digital Agenda for Europe.

(193)

   This action is defined in the Digital Agenda for Europe.

(194)

   This action is defined in the Digital Agenda for Europe.

(195)

   This action is defined in the Digital Agenda for Europe.

(196)

   This action is defined in the Digital Agenda for Europe.

(197)

   This action is defined in the Digital Agenda for Europe.

(198)

 Mid-term Evaluation of the e-Government Action Plan 2011–2015, 2014.

(199)

https://joinup.ec.europa.eu/

(200)

COM(2012) 721 final

(201)

Study Measuring Progress of eAccessibility in Europe – MeAC, 2013 

(202)

 Study on collaborative production in eGovernment, 2012

(203)

In many cases using the PSI Scoreboard Indicators

(204)

 Pricing Of Public Sector Information Study on Models of Supply and Charging for Public Sector Information 2011

(205)

http://www.europeandataportal.eu/sites/default/files/edp_creating_value_through_open_data_0.pdf

(206)

  http://epsiplatform.eu/

(207)

  European Commission, Decision of 12 December 2011 on the reuse of Commission documents [2011/833/EU]

(208)

COM(2011) 882 final

(209)

  http://open-data.europa.eu/

(210)

http:// www.europeandataportal.eu

(211)

http://www.opengovpartnership.org/independent-reporting-mechanism

(212)

https://ec.europa.eu/digital-agenda/en/news/workshop-improving-transparency-government-decision-making-6112012-results-presentations

(213)

  http://ec.europa.eu/citizens-initiative/public/welcome

(214)

https://ec.europa.eu/digital-agenda/en/eparticipation

(215)

http://www.peppol.eu

(216)

http://www.eu-spocs.eu

(217)

 The feasibility and scenarios for the long-term sustainability of the Large Scale Pilots, including 'ex-ante' evaluation, 2013

(218)

  http://ec.europa.eu/growth/single-market/public-procurement/new/index_en.htm  

(219)

http://ec.europa.eu/growth/single-market/public-procurement/e-procurement/index_en.htm

(220)

http://ec.europa.eu/growth/single-market/public-procurement/e-procurement/index_en.htm

(221)

 Mid-term Evaluation of the e-Government Action Plan 2011–2015, 2014, p. 36 and country-specific details at http://www.egovap-evaluation.eu  

(222)

 Study on The functioning and usability of the Points of Single Contact under the Services Directive, 2012

(223)

COM(2012) 261

(224)

http://ec.europa.eu/internal_market/services/docs/services-dir/psc-charter_en.pdf

(225)

 The Performance of the Points of Single Contact. An Assessment against the PSC Charter, 2015

(226)

http://www.e-codex.eu

(227)

Note: the progress under the sub-priority on EU-wide implementation of cross-border services is linked to the general objective of the Action Plan on a number of key cross-border services to be offered online by 2015, see above at 5.2.

(228)

 Study on Analysis of the Needs for Cross-Border Services and Assessment of the Organisational, Legal, Technical and Semantic Barriers, 2013

(229)

http://www.e-codex.eu

(230)

The topic was included in the H2020 WP 2016-2017 that was adopted in 2015.

(231)

More information at http://inspire.ec.europa.eu/ and Directive 2007/2/EC of the European Parliament and of the Council of 14 March 2007 establishing an Infrastructure for Spatial Information in the European Community (INSPIRE). The INSPIRE Directive entered into force on the 15th May 2007

(232)

ISA Action 1.17 Improving the cross-border exchange of spatial information : A Reusable INSPIRE Reference Platform.

(233)

  Providing INSPIRE-compliant access to utility services: the case of sewage networks in Flanders, Belgium

(234)

  Dutch cadastre INSPIRE conform , The German Marine Data Infrastructure , UK National Biodiversity Network to deliver INSPIRE compliance for species data , Geodateninfrastruktur Deutschland: INSPIRE success story – Implementing e-reporting of air quality based on INSPIRE at national level

(235)

  One-Geology Europe , European Location Framework, Reporting and exchanging air quality information using e-Reporting

(236)

ISA ACTION 2.13 European Union Location Framework (EULF)

(237)

  EEA Technical report No 17/2014 . Joint EEA-JRC report.

(238)

 Study on eGovernment and the Reduction of Administrative Burden, 2014

(239)

https://joinup.ec.europa.eu/

(240)

e-Commission 2012 – 2015. SEC(2012) 492.

(241)

  http://ec.europa.eu/dgs/informatics/supplier_portal/index_en.htm - link to Supplier portal requires ECAS login.

(242)

 Study on eGovernment and the Reduction of Administrative Burden, 2014

(243)

 European Council, Conclusions, 24/25 October 2013 

(244)

Horizon 2020 Work Programme 2016 – 2017, Europe in a changing world – inclusive, innovative and reflective Societies

(245)

  http://ec.europa.eu/isa/actions/04-accompanying-measures/4-2-3action_en.htm and NIFO JoinUp community https://joinup.ec.europa.eu/community/nifo/home

(246)

http://ec.europa.eu/isa/

(247)

 State of play of interoperability in Europe – Report 2014

(248)

https://joinup.ec.europa.eu/community/nifo/og_page/nifo-factsheets

(249)

Regulation (EU) No 910/2014. On eIDAS, see http://ec.europa.eu/digital-agenda/en/trust-services-and-eid

(250)

www.eid-stork2.eu

(251)

https://joinup.ec.europa.eu/community/cef/og_page/catalogue-building-blocks

(252)

CEF Monitoring Report Q4/2015

(253)

 Study on cloud and service oriented architectures for eGovernment, Final report, 2011

(254)

A vision for public services, 2013

(255)

http://www.gen6.eu/home

(256)

https://ec.europa.eu/digital-agenda/en/towards-cloud-public-services