EUROPEAN COMMISSION

Brussels, 15.12.2021

SWD(2021) 383 final

COMMISSION STAFF WORKING DOCUMENT

Final evaluation of the European statistical programme 2013-2020

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

on the final evaluation of the implementation of the European statistical programme 2013-2020

{COM(2021) 794 final}

Table of contents

1.Introduction

1.1 Purpose and scope    

2.Background to the intervention

2.1 Description of the intervention and its objectives    

2.2 Baseline and points of comparison    

3.Implementation/State of Play

3.1 Description of the current situation    

4.Method

4.1 Short description of methodology    

4.2 Limitations and robustness of findings    

5.Analysis and answers to the EQs

5.1 Relevance    

5.2 Effectiveness    

5.3 Efficiency    

5.4 Coherence    

5.5 EU added value    

5.6 Evaluation of the points raised in Article 15 of the ESP    

6.Conclusions

Annex 1: Procedural information

Annex 2: Stakeholder consultation

Annex 3: Methods and analytical models

Annex 4: Final Report of the Contractor

Glossary

Definitions

1.Introduction

1.1 Purpose and scope

Article 13 of Regulation (EC) 223/2009 1 on European statistics states that ‘the European statistical programme shall provide the framework for the development, production and dissemination of European statistics, setting out the main fields and the objectives of the actions envisaged for a period corresponding to that of the multiannual financial framework’.

The European statistical programme 2013-2020 was created by Regulation (EU) 99/2013 2 for the years 2013-2017. It was extended by Regulation (EU) 2017/1951 3 to cover the remaining period of the previous EU multiannual financial framework, i.e. 2018-2020. Throughout this document, it is referred to as ‘the programme’ or simply ‘the ESP’, while the Regulation creating it is referred to as ‘the amended Regulation’.

The general objective of the ESP is for the European Statistical System (ESS) to continue to be the leading provider of high-quality statistics on Europe. As a spending programme, the ESP constitutes the overall framework for developing, producing and disseminating European statistics. The ESP therefore provides the financing for developing and maintaining Eurostat’s statistical infrastructure and the ESS as managed by Eurostat. The ESP also provides financial support to Member States.

The requirement to carry out a final evaluation and present a report on the ESP is enshrined in Article 15 of the amended Regulation. Article 15(3) states:

‘By 31 December 2021, the Commission shall, after consulting the ESSC and the European Statistical Advisory Committee, submit a final evaluation report on the implementation of the programme to the European Parliament and to the Council. The report shall in particular evaluate:

a)the outcome of the reprioritisation and cost evaluation of statistical products;

b)the actions taken by the ESS to reduce the implementation and production costs for Member States and to limit the overall burden stemming from the statistical projects and fields covered by the programme;

c)the progress on rendering access to official statistics easier and more user‑friendly, including the provision of data on the Eurostat website; and

d)the progress on the improvement of data availability, including on social economy activities and on the Europe 2020 indicators.’

The evaluation also serves to fulfil the requirements of the Commission’s push for better regulation , and it must therefore follow the better regulation guidelines for evaluations . The evaluation therefore covers the five compulsory evaluation criteria (effectiveness, efficiency, relevance, coherence and EU added value).

The conclusions of the evaluation could also be useful when performing the mid-term evaluation of the single market programme (SMP). The SMP has an objective of producing high-quality European statistics and includes the successor of the ESP.

Even though the current evaluation focuses on the 2018-2020 period (because 2013-2017 has already been covered by two previous mid-term evaluations of the ESP 4 ; 5 ), the conclusions of the current evaluation cover the implementation period of the programme in its entirety (2013-2020).

The current evaluation encompassed all Member States and other countries covered by the programme. Participation in the programme was open to countries in the European Economic Area (EEA)/European Free Trade Association (EFTA), Switzerland, and countries to which the EU’s enlargement policy applies. Regulation (EU) 2017/951 on the ESP has no related implementing/delegated acts.

2.Background to the intervention

2.1 Description of the intervention and its objectives

The general objective of the ESP is for the ESS to continue to be the leading provider of high-quality statistics on Europe. Official European statistics 6 provided under the ESP are indispensable for EU decision-making and for measuring the performance and impact of EU initiatives. High-quality European aggregates, together with comparable statistics on all individual Member States, underpin the investment plan for Europe; the European semester exercise; the European pillar of social rights; the European agenda on migration; and the energy union. Statistics are also key instruments which strengthen the transparency and accountability of EU policies and which enable EU citizens to engage with – and democratically participate in – political life. European statistics serve the needs of a wide range of users, including: (i) decision-makers at EU, national and local level; (ii) international organisations; (iii) academia and researchers; (iv) businesses; (v) NGOs; (vi) the media; and (vii) the European public in general.

As a spending programme, the ESP constitutes the overall framework for developing, producing and disseminating European statistics. Its duration was originally planned to be 5 years but this was extended to correspond to the duration of the multiannual financial framework. The ESP provides the financing for developing and maintaining the statistical infrastructure of Eurostat and the ESS as managed by Eurostat. The ESP also provides financial support to Member States. Member States use this support to: (i) improve their national statistical systems; (ii) implement measures to initiate new data collections; and (iii) strengthen the quality and efficiency of statistical production through innovative statistical methods and tools.

The statistical infrastructure enables statistics to be regularly produced and disseminated in the policy fields. Depending on the specific needs and uses in these policy fields, the statistical infrastructure also enables new statistics to be developed. The programme’s budget can be roughly split between: (i) developing and maintaining the general methodological and IT infrastructure on the one side; and (ii) providing support to the production, dissemination and development of statistics in the specific policy domains on the other. The total budget allocated for the implementation of the ESP in 2013-2020 was EUR 489 million.

The ESP has four specific objectives:

·‘Objective 1: provide statistical information in a timely manner, to support the development, monitoring and evaluation of the policies of the Union properly reflecting priorities, while keeping a balance between economic, social and environmental fields and serving the needs of the wide range of users of European statistics, including other decision-makers, researchers, businesses and European citizens in general, in a cost-effective manner without unnecessary duplication of effort;

·Objective 2: implement new methods of production of European statistics, aiming at efficiency gains and quality improvements;

·Objective 3: strengthen the partnership within the ESS and beyond in order to further enhance its productivity and its leading role in official statistics worldwide;

·Objective 4: ensure that delivery of such statistics is kept consistent throughout the whole duration of the programme, provided that this does not interfere with the priority-setting mechanisms of the ESS.’

The programme is divided into three parts, each focusing on a specific priority area:

I. statistical outputs;

II. production methods of European statistics;

III. partnership.

Objectives 1 and 4 are covered by the measures undertaken in priority area I; objective 2 corresponds to priority area II; and objective 3 to priority area III.

The three priority areas are themselves divided into one or two levels of sub-areas.

There is a total of 23 second-level, detailed, more operational objectives, spread across the three priority areas. A set of achievement indicators is provided for each of these objectives.

Financial contributions in the form of grants to Member States and procurements made by Eurostat are used to achieve the programme’s objectives. Eurostat mainly uses procurement to develop and maintain the general infrastructure, whereas grants help the national statistical systems to develop statistics in specific policy domains.

A schematic view of the ESP’s intervention logic is presented in Figure 1 below. The key elements included in the intervention logic are set out in the following bullet points (working from the bottom to the top of Figure 1).

·Needs and challenges: The programme cannot be sufficiently achieved by the Member States alone and can therefore be better achieved at EU level. The ESP addresses a number of needs and challenges observed in the societal, economic and policy spheres. The regulations follow the principle of proportionality and do not go beyond what is necessary to achieve that objective. The annex of the ESP Regulation discusses the needs and challenges faced by the ESS. The diagram in this annex presents a summary version of the needs and challenges encountered by the ESS. Other factors to take into consideration are the evolving nature of complex and multidimensional statistics and the increasing availability of data in the digital age. In addition, the ESP is challenged by the decentralisation of statistics production and the coexistence of multiple statistics producers. Rationalising the production and use of statistics – and improving their efficiency – are key goals to ensure the ESP does not create a disproportionate administrative burden for statistics producers (notably Member States, which are facing significant financial constraints).

·Inputs: EU legislation is an input, a resource provided by the EU. The resources needed to meet the legislative requirements (such as management effort and time spent on the ESS by the Member States, other statistics producers and Eurostat) have also been categorised as inputs. The funding available (through grants and procurements, to: (i) help the national statistical systems develop statistics in specific policy domains; and (ii) develop and maintain the general infrastructure) is also an input.

·Activities: The activities resulting from the implementation of the legal acts (such as the collection, processing and validation of data by Member States, statistics producers and Eurostat) are included in this category. This category also includes activities to communicate and disseminate statistics. Lastly, an ‘overall management’ category includes: (i) the provision of methodological guidance; (ii) the sharing of good practices and standards (including exchange of knowledge between regional, national, European and international levels); (iii) training; (iv) stakeholder dialogues; and (v) participation in international activities.

·Outputs: The outputs of the ESP are the immediate products of the activities implemented. The outputs are also called ‘detailed objectives’ in the Regulation, and they are categorised into three priority areas: statistical outputs, production methods of European statistics, and partnerships. Overall, the three categories have remained stable over time. Figure 1 includes a summary of the outputs/detailed objectives, structured along the three priority areas. Table 1 provides a comprehensive list of the 23 detailed objectives included in the Regulation.

·Outcomes/objectives: In the medium-term, the ESP is expected to achieve four ‘objectives’ (that is the terminology used in the Regulations), also called ‘outcomes’ (Article 4(2) of the 2013 Regulation). These four outcomes are structured along the three priority areas previously identified. The causality within each priority area between outputs/detailed objectives and outcomes/objectives is described in the annex to the Regulation.

·General objective: As per the 2013 Regulation (Article 4.1), the ESP aims to enable the ESS to continue to be the leading provider of high-quality statistics on Europe.

·Strategic objectives: Deciding the strategic objectives goes beyond the ESP, its functioning and governance. The strategic objectives must be chosen to suit the different types of users of the ESP. Through providing high-quality statistics on Europe, the ESP is expected to contribute to: (i) improving policy-making and the measuring of the performance and impact of initiatives; (ii) strengthening the transparency and accountability of policies; and (iii) enabling EU citizens’ civic engagement and democratic participation in political life. The ESP should contribute to these strategic objectives without creating a disproportionate administrative burden – in particular for statistics producers. In Figure 1 , we have differentiated the result levels to indicate when the results are internal to the ESP (outputs, outcomes and impacts) and when they are external to the ESP (strategic objectives).

Figure 1: ESP intervention logic

Table 1: Detailed list of ESP objectives

2.2 Baseline and points of comparison

Start of the programme

When the ESP’s impact assessment was carried out in 2011, Eurostat found that the ESS faced a number of challenges:

(1)    The need for European statistics had been constantly increasing over the preceding years, and it was unlikely that this trend of increased need would change in the future.

(2)    The nature of statistics was changing – there was a growing need for complex multi‑dimensional statistics of even higher quality which had to be provided within shorter periods of time than before.

(3)    Due to new participants appearing on the information market providing new types of statistics (including participants providing information almost in real time) the future priority for the ESS would be the freshness of data, in particular for short-term economic information.

(4)    The situation for statistics producers has become more challenging due to budget constraints both at national and European level. These constraints had become more pertinent with the economic crisis, as had the need to further reduce the burden of compiling statistics on businesses and citizens.

The ESP was designed to face all those challenges. For this reason, its main objectives were combining the production of more statistics (including statistics that are more complex and of a higher quality) with the modernisation and reengineering of the processes used to produce these statistics.

As a step further in modernising the production of statistics, the ESSC agreed in early 2015 to implement the ESS Vision 2020 7 , a strategic vision for how the ESS should develop by 2020. The implementation initially involved eight implementation projects for ESS Vision 2020.

First mid-term evaluation4

The ESP’s state of implementation was evaluated for the first time in 2015, taking into consideration the activities carried out and the results obtained in the first 2 years of the programme – 2013 and 2014. The first mid-term evaluation concluded that the implementation of the ESP was advancing well, with 17 of the programme’s 23 detailed objectives well on track to be completed. This first evaluation provided a detailed summary of the ESP’s produced outputs. Good progress had also been made on projects related to modernising the production systems for statistics. The first evaluation stated that the ESP continued to be relevant, as its objectives still corresponded to the EU’s needs and it continued to provide clear EU added value. The evaluation concluded that Eurostat had made efficient use of its resources, both financial and human, with improved productivity at a time when staff numbers were being reduced. The first evaluation said that the ESP was well coordinated with other EU initiatives in the same field, both other Eurostat programmes and initiatives on statistics produced by other Commission Directorate-Generals.

The first mid-term evaluation also made the following recommendations:

1.to give special attention to the objectives where problems have been encountered;

2.to try to secure sufficient resources to maintain the necessary level of investment to modernise the production of European statistics; and

3.to identify and implement projects at EU level which could maximise EU added value.

Second mid-term evaluation5

A second mid-term evaluation of the ESP was performed in 2018, covering implementation in 2015-2017. The results of the second mid-term evaluation indicated that the ESP had shown a high degree of effectiveness. The few problems registered in the first mid-term evaluation had been tackled successfully. The percentages of planned outputs that were achieved or on target in the 3 years under examination were constantly over 90%. 20 of the 23 detailed objectives of the ESP could be considered on track to be accomplished by the end of the extended programme. The second mid-term evaluation said that the remaining three objectives ((i) objective 2 on economic governance; (ii) objective 4 on economic globalisation; and (iii) objective 18 on dissemination and communication) only showed limited problems, which would have to be taken care of in the remaining period of the programme.

The second mid-term evaluation indicated that the programme: (i) was run efficiently; (ii) provided clear EU‑added value; and (iii) was coherent internally and consistent externally with other initiatives to produce statistics. The results also showed that the ESP had contributed – and continued to contribute – to satisfying user needs and to the design and monitoring of policies. Nevertheless, the results also showed that ESP needed to do more. Eurostat and the ESS were modernising their statistical production processes to respond to user needs while taking advantage of new technologies. The second mid-term evaluation said that the ESP needed to focus on implementing its plan to modernise statistical production processes, saying that this might require: (i) changes in the way statistical authorities gather and process data; and (ii) sufficient resources. The second mid-term evaluation also showed that the ESP’s structure was fairly complex in general, with objectives and sub-objectives covered by areas and measured by indicators which sometimes were not clearly distinct from one another. This made it difficult to understand and monitor the ESP. Therefore, the second mid-term evaluation recommended simplifying the structure of the programme for the future post-2020 ESP.

Programme extension 2018-2020

The 2013 programme was extended in 2017 by Regulation (EU) 2017/1951 to cover the period 2018-2020 (the remaining period of the multiannual financial framework up to 2020). The impact assessment accompanying Regulation (EU) 2017/1951 noted limitations in the ESP’s structure relating to: (i) a lack of flexibility to deliver new statistics, while also limiting the associated cost and administrative burden; and (ii) a lack of investment to meet the growing demand for statistics or the need to make statistics available more quickly. The impact assessment recommended an amended programme including: (i) new statistical outputs in line with the 10 political priorities for the Commission’s agenda for jobs, growth, fairness and democratic change 8 ; (ii) investment in the statistical infrastructure; and (iii) investment in new sources of statistics (with an increased budget). The extension aimed at:

·providing high-quality statistical information and closing the statistical gaps that needed to be addressed most urgently, focusing on a number of priority areas that reflected the Commission’s 10 political priorities;

·building the permanent capacity needed to respond more quickly to emerging needs and to adapt the statistical infrastructure so as to harness the potential of new data sources; and

·strengthening the partnership within the ESS and beyond to further increase its productivity and secure its leading role in official statistics worldwide.

The amendment of the programme introduced the following three modifications, among others.

·It improved substantially the timeliness of data on inequality, poverty and material deprivation of people in Europe (for instance through flash estimates on the distribution of household income that are fit for purpose for the European Semester).

·It expanded the production of statistics on energy (efficiency, security, renewables, consumption, prices, etc.) to accompany the strategic framework for the energy union and improved the timeliness of these statistics (e.g. issuing flash estimates on energy balances).

·It improved the quality, coverage and timeliness of environmental data to support climate change policy and the circular economy package and to support the UN’s Sustainable Development Goals (SDGs).

In the extension of the ESP, the annual budget was increased for the years 2018 to 2020 to a total of around EUR 208 million for the 3 years compared to around EUR 281 million for the previous 5 years. The reason for this increase was to: (i) enable Member States to cope with new demands and to modernise their production systems, as recommended in the first mid-term evaluation of the programme; and (ii) invest in the statistical infrastructure and in new sources of statistics, as recommended in the impact assessment.

Programme progress report 9

A progress report on the implementation of the ESP, covering 2018 and the first half of 2019, was produced in 2019. The report indicated that the implementation of the programme was progressing well, producing significant results under the programme’s different objectives.

The two mid-term evaluations and the progress report demonstrated that: (i) the delivery mechanism had worked effectively; (ii) the programme was run efficiently; and (iii) the programme was reaching its objectives. They all showed that the ESP provided clear EU added value thanks to the harmonised provision of comparable and high-quality data for the EU. However, the evaluations also showed the need to ensure sufficient resources in the future to modernise the statistical production processes. In addition, the evaluations all argued that Eurostat needed to be able to: (i) respond to users’ growing needs, especially on the timeliness and coverage of new data for emerging policy needs; and (ii) become more agile and take advantage of new technologies.

3.Implementation/State of Play

3.1 Description of the current situation

The ESP lasted 8 years, from 2013 to 2020. The total programme budget was EUR 489 million (including EFTA funds and administrative expenditure in direct support of programme implementation). Around 41% was spent on grants and 59% on procurements. Almost another EUR 225 million supplemented the ESP’s own budget (i.e. EUR 225 million in addition to the EUR 489 million budget to make total support of EUR 714 million) in the form of credits sub-delegated by other policy DGs to cover data collections specifically requested by these same DGs.

The budget was spent in the programme’s three priority areas. The first priority area, ‘statistical outputs’, deals with the production of European statistics. In this area, the money was mostly spent on grants to provide financial support to Member States. Member States used this support to improve their national statistical systems and to implement measures to carry out new data collections. All Member States benefited from these grants, which ultimately allowed them to produce more and better quality data. As a result, the total number of disseminated datasets increased by 809 – or around 18% – from 2013 to 2020, as detailed in Section 5.3 on efficiency.

Figure 2: Number of datasets published by Eurostat, 2013-2020 10

Source: Eurostat’s dissemination database

The second priority area, ‘production methods of European statistics’, deals with improving the way statistics are produced, their quality, and the way they are disseminated. In this area, grants were given to Member States to strengthen the quality and efficiency of statistical production through innovative statistical methods and tools. Member States used this money to participate in the modernisation projects set out in the ESS Vision 2020 programme. In this same area, procurements were used either: (i) to partly finance ESS Vision 2020 projects; or (ii) to finance the modernisation of the statistical infrastructure used to regularly produce and disseminate the statistics. Examples of results under this priority area include: (i) the modernisation of the IT infrastructure to exchange data with Member States; (ii) the renovation of Eurostat’s website; and (iii) the introduction of a series of new visualisation tools and new ways to reach out to users 11 . More details on the ESS Vision 2020 programme can be found in the responses to various evaluation questions (EQs), and in particular under EQ 2 in the chapter on relevance.

The third priority area was ‘partnership’. This priority area aimed to support the production and quality of statistics by improving cooperation within the ESS and with other international organisations and countries outside the EU. In this area, the budget was mostly used to support the development – and, to a lesser degree, the production – of statistics in countries outside the EU and EFTA. The budget had a particular emphasis on enlargement and European neighbourhood policy. This priority area enabled Eurostat to regularly publish data for non-EU countries, especially to support the enlargement process and negotiations with those non-EU countries.

In spending the budget, there were no major problems and there were no infringements. Eurostat put in place an effective system of anti-fraud measures, which prevented cases of fraud.

Different monitoring arrangements were put in place for the ESP.

Twice a year, the status of all planned Eurostat activities and their outputs are checked. The percentage of activities and outputs which were on track or achieved has consistently been greater than 90%.

A set of key performance indicators (KPIs) was monitored each year and reported in the programme statements of the ESP, which are part of the annual performance and management reports of the European Commission. The values of the indicators, as extracted from the last programme statement of the ESP 12 , are shown in Table 2 .

Table 2: ESP KPIs

Source: ESP programme statement for 2020

In 2020, almost all indicators were on track or even surpassing their respective targets. Only the value of the last indicator, related to objective 3, saw a slight decrease in 2020, although it was still performing well. Some of the partnership activities experienced problems in 2020 caused by the COVID-19 pandemic. This was the case for seminars, missions, assessments of statistical systems, and statistics training, all of which were impacted by the restrictions due to the pandemic, especially in countries outside the EU.

More details on the indicators can be found under the different EQs, in particular in the chapter on effectiveness.

Eurostat wished to continue the activities of the ESP – while adapting them to both: (i) the changing environment, with new possibilities for producing statistics using more sources and innovative technologies; and (ii) changing user needs. To achieve this, Eurostat prepared the statistical annex of the SMP.

Drawing on the results of the two mid-term evaluations of the ESP, the 2018 impact assessment on the SMP 13 indicated some important issues to address in the new programme such as:

·the availability of high-quality statistical information to design and monitor new EU policies (e.g. on globalisation, digitalisation of the economy, and security);

·the need for agility to address new demands and provide both faster evidence on emerging topics and deeper analysis of: (i) the effects of globalisation; (ii) new technological developments; and (iii) socioeconomic trends;

·the need to reduce statistical reporting requirements on businesses as well as the public;

·the need: (i) to improve analysis of European statistics on today’s complex realities and interlinkages to better understand them; and (ii) for improved communication and engagement with users, especially EU citizens.

The SMP Regulation 14 is applicable from 1 January 2021. The SMP has as a general objective:

to develop, produce and disseminate high-quality, comparable, timely and reliable European statistics which underpin the design, monitoring and evaluation of all Union policies and help citizens, policymakers, authorities, businesses, academia and the media to make informed decisions and to actively participate in the democratic process.

The specific objective to realise the programme’s general objective is the following:

developing, producing, disseminating and communicating high-quality European statistics in line with the quality criteria laid down in Article 12(1) of Regulation (EC) No 223/2009, in a timely, impartial and cost-efficient manner, through a strengthened European Statistical System, referred to in Article 4 of Regulation (EC) No 223/2009, and enhanced partnerships within that system and with all relevant external parties, using multiple data sources, advanced data analytics methods, smart systems and digital technologies, and providing a national and, where possible, regional breakdown.

4.Method

4.1 Short description of methodology

The evaluation was performed as indicated in the road map.

A contractor, Tetra Tech International Development, was selected via an open call for tenders. Having an external contractor supporting the evaluation process was deemed very important to get an independent opinion on the work carried out by Eurostat. The contractor was charged with:

·designing the evaluation, and in particular the EQs for all criteria and the indicators to respond to them;

·preparing and executing a comprehensive stakeholder consultation;

·collecting and analysing the necessary information;

·compiling the stakeholder-consultation report;

·writing an evaluation report to support the preparation of the Commission evaluation staff working document (SWD).

The contract started on 10 September 2020 and the contractor delivered its final report on 23 July 2021.

The stakeholder consultation took place between December 2020 and March 2021, and it included a vast spectrum of stakeholders, who were consulted in several ways. The consultation comprised:

·scoping interviews with representatives of Eurostat and other DGs;

·a twelve-week public consultation (PC);

·targeted surveys of users and producers of statistics;

·a large number (50) of interviews of different sorts of stakeholders.

The contractor also carried out 4 thematic case studies and 5 country case studies. It collected and analysed many supporting documents.

The contractor triangulated information from the different sources to reach robust conclusions.

The synopsis report of the stakeholder consultation is included in Annex 2 of the present evaluation. Details of the sources for the contractor’s desk research and the case studies can be found in the company’s final report in Annexes 2, 3 and 4.

An interservice steering group (ISG) was set up for the project, including members from all directorates of Eurostat and from many other DGs. The first meeting of the ISG coincided with the kick-off meeting of the contract on 21 September 2020. The ISG then met on 10 November 2020, 6 May 2021 and 6 July 2021. The ISG was able to comment on the different deliverables of the contract, including: (i) the design of the project; (ii) the design of the stakeholder consultation. The ISG was also able to comment on the Commission evaluation report and the SWD.

More information on the ISG, including its composition, can be found in Annex 1.

4.2 Limitations and robustness of findings

The COVID-19 crisis prevented the contractor’s team from collecting data face-to-face. All activities took place remotely instead. This meant that the team made stronger and more targeted efforts to ensure good response rates for the interviews to be conducted for: (i) the overall programme assessment; (ii) the country case studies; and (iii) the thematic case studies. Overall the level of stakeholder engagement was high.

5.Analysis and answers to the EQs

This section presents the analysis and answers to the EQs for the five evaluation criteria. The contractor drew up evaluation sub-questions, judgement criteria (JC) and corresponding indicators to help respondents to reply to the questions, as shown in the evaluation matrix in Annex 3. Using the information collected on the five criteria, the evaluation then also answers the four questions explicitly asked in Article 15 of the ESP Regulation.

A more extensive analysis, that is more detailed level than what is possible in this document, can be found in the final report of the contractor (Annex 4).

5.1 Relevance

The ESP provided the framework for developing, producing, and disseminating European statistics, setting out the objectives of the activities envisaged for 2013-2020. It financed the development and maintenance of Eurostat’s statistical infrastructure and of the ESS. The objective of the ESP was therefore to both: (i) fulfil the needs of Eurostat, the NSIs, and other national authorities (ONAs) as producers of European statistics; and (ii) serve the needs of a wide range of users of these statistics.

The EQs are presented below. They explored the relation between: (i) the objectives of the programme; (ii) the activities of the programme; (iii) the needs of the ESS; and (iv) the needs of stakeholders more generally. The following sections provide an evidence-based judgement of both: (i) the extent to which the ESP was originally relevant; and (ii) the suitability of the activities to achieve the objectives of the programme. The first EQ focuses on ESS stakeholders, namely producers of statistics, while the second EQ also covers users.

The replies to the two questions indicate that the ESP was appropriately designed to satisfy stakeholders’ needs. The replies also indicate that, throughout its entire duration, the ESP’s activities were carried out to continue to satisfy these changing needs. At the same time, although the ESP made it possible to implement appropriate activities to meet its objectives, the replies showed that these activities were not enough to deliver all the statistics that users wished for. However, some of the thematic areas that users identified as missing go beyond the scope of the ESP – so non-delivery in these areas cannot be considered as failures in the implementation of the programme.

There was some tension between continuously increasing user needs and limitations from producers in keeping up with new demands under an already ambitious programme. This meant that a balance needed to be struck between producer capacity and user requests. In addition, needs related to adaptation to technological advances such as big data became more prominent during the programme, as reflected in the extension of the ESP. These increasingly prominent needs were translated into specific activities, mainly under the ESS Vision 2020 programme. Similar activities will be continued in the next programme, and their results have the potential to provide solutions to some of the problems the ESS still faces in terms of resources and costs.

EQ1: To what extent did ESP objectives reflect the needs of the ESS?

JC 1.1 ESP objectives and priorities corresponded to the identified needs and gaps of the European statistics sector and to their evolution (i) in 2013; (ii) in 2018 when the ESP was extended and revised; and (iii) at the time of the evaluation in 2020/2021

Stakeholder feedback for this evaluation generally confirmed that the specific objectives of the ESP addressed the needs of the ESS in 2013. Most producers who responded to the targeted survey assessed the four specific objectives of the ESP as relevant to either a great extent or a moderate extent. Most stakeholders interviewed highlighted that the objectives and priorities of the ESP were all relevant and well designed. They also confirmed that the ESP’s specific objectives were aligned with the needs of the ESS and that Member States’ statistical strategies were often based on the ESP’s objectives.

The stakeholder consultation did not highlight any mismatch between the objectives and priorities of the ESP and the needs and gaps of the European statistics sector in 2018 when the programme was extended and revised. Producers indicated that the objectives of the ESP had continued to meet their needs. The extension also made it possible to fully align the objectives of the ESP with the 10 priorities of the Juncker Commission, as detailed under JC 2.1.

More than half of the producers who responded to the targeted producer survey (55% of 43 overall responses) indicated that they did not see any gaps that the objectives of the ESP did not cover. Nevertheless, a considerable share of the producers who responded (31% of 43 overall responses) did indeed see such gaps. 46% of the 43 that responded also indicated they did not see gaps in terms of meeting future needs, compared to 27% who did. Perceived gaps included the provision of statistics in new or rapidly changing policy areas.

The targeted producer survey also revealed limitations in the ESP, including time constraints, challenges in keeping up with Eurostat’s demands, and requests for new statistics. The surveyed producers said these limitations came in addition to the already demanding nature of the ESP. Respondents to the targeted producer survey also felt there was a need to strengthen partnerships in the ESS, and said that Eurostat could coordinate NSIs’ work more closely, as it did during the COVID-19 crisis. Insufficient coverage of access to privately held data was another limitation identified in the targeted producer survey.

In addition, respondents to the targeted producer survey and producers interviewed said that an effective system for priority setting (especially for identifying statistics that have become less relevant over time) had been missing until the end of the programme. Eurostat had been conducting an annual priority-setting exercise, in consultation with users and producers of statistics since the start of the ESP in 2013. For the new programme 2021-2027, this priority-setting exercise included a review of statistical requirements in existing domains of European statistics with the intention of reducing costs for Member States. In February 2020, the annual priority-setting exercise was replaced by a new priority-setting mechanism adopted by the ESSC, and a pilot review for climate change related statistics was also put in place 15 . The new mechanism is based on four parts: a multiannual action plan, reviews, user dialogue, and resources/funding.

JC 1.2 Mechanisms for feedback and input from ESS stakeholders existed and were perceived as appropriate

The ESP relied on a ‘feedback loop’ and input from ESS stakeholders. This feedback loop consisted of the ‘partnership approach’, namely an ongoing discussion between the EU and Member States, which ensured the consideration of different perspectives. As a result, ESS stakeholders consulted by the contractor for this evaluation said that they found there was a good balance in the steering and development of the ESP. Consultation mechanisms were therefore part of the ESP and the ESS stakeholders that were consulted broadly perceived these mechanisms as appropriate.

EQ2: To what extent were the ESP’s activities appropriate to deliver the set objectives of the ESP?

JC 2.1 The ESP’s activities were the appropriate tools in 2013 to deliver the set objectives, and they continued to be the appropriate tools in both 2018, when the ESP was extended and revised, and again at the time of the evaluation in 2020/2021

80% of the producers who responded to the targeted survey (to which 43 responded) confirmed that the ESP’s activities were appropriate to achieve the four specific objectives – either to a great or to a moderate extent. Given that the objectives of the ESP were formulated at a general level, all NSI activities related to the production of European statistics fell within the scope of the programme. To support the implementation of the ESP, annual work programmes set detailed objectives and outputs for each year. Interviewees mentioned that the annual work programmes made it possible to adapt and adjust the ESP. However, the interviewees also said that the links between the activities of the programme, the ESP’s objectives, and the related indicators were hard to follow, even though Eurostat continuously registered and improved such links. The new ESP multiannual action plan 2021-2027 was designed to tackle this issue. It is intended as a bridge between the SMP, which covers European statistics for 2021-2027, and the annual work programmes.

In 2016, the impact assessment for the extension of the programme 16 concluded that there was a need at the time to address several statistical gaps in the ESP’s activities to support EU policies. In addition, the impact assessment said that the relevance of the ESP’s activities was being challenged by: (i) the need for statistics arising from the Juncker Commission’s 10 political priorities 17 ; and (ii) the increasing complexity of European society. The impact assessment said that the ESP should therefore be amended and that there also needed to be investment in the statistical infrastructure and new sources of statistics through an increased budget.

In addition, the extension of the programme was an opportunity to: (i) make adaptations and reflect the new policy directions, in line with the ESS Vision 2020; and (ii) complement the ongoing prioritisation in the ESP.

To assess whether the ESP’s activities were still the appropriate tools to deliver the set objectives at the time of the evaluation in 2020/2021, the contractor proposed in the evaluation’s methodology to look at their statistical outputs and see whether they served the needs of the wide range of users.

On statistical gaps at the time of the evaluation in 2020/2021, users who responded to the targeted survey (to which 45 responded) generally indicated that they would like Eurostat to collect and publish more statistics within its existing statistical themes (64% agreed with this statement while 20% of the respondents were unsure and 16% found the collection and publication of more statistics unnecessary). Respondents highlighted the need for a more detailed and granular breakdown of different types of statistics at regional level. They also highlighted the need for data and statistics subdivided into more detailed sociodemographic attributes, such as gender. Institutional users that were interviewed also identified similar needs that they said the ESP’s activities did not meet. However, producers (of which 43 responded to the survey), were less eager to extend the datasets. 43% of producers indicated that they would not like Eurostat to collect and publish more statistics, while 26% said they would. Producers pointed out the need for Eurostat to keep a balance between: (i) the demand for statistics; (ii) the producers’ (limited) capacity; and (iii) the increased burden new statistics could create for producers.

Nevertheless, all 45 users contributing to the targeted survey found that European statistics were appropriate for their purpose, either to a great extent (89%) or to a moderate extent (11%). This indirectly confirms the relevance of the ESP’s activities. The PC (to which 424 responded) also confirmed this, with more than 80% of respondents indicating that European statistics were relevant to their role or sector, either to a great extent (53% agreed) or to a moderate extent (31% agreed). In specific policy areas (assessed through the case studies on business statistics and managed-migration statistics), the evaluation team also confirmed this assessment, as both sets of statistics fitted within the objectives of the ESP and reflected the needs of the ESS.

In addition, interviewees from other European Commission DGs confirmed that the ESP’s original activities were appropriate to meet their needs, and said that these activities had continued to be appropriate to meet their needs until that day.

Nevertheless, some respondents from the European Commission were of the opinion that the ESP lacked flexibility. They said that any change to the scope of the programme required a change in EU legislation, which is a lengthy process. For example, stakeholder feedback, in the case study on the response to the COVID-19 outbreak, suggested that the outbreak did not activate any specific mechanisms embedded in the ESP to address the challenges the pandemic presented for European statistics. These stakeholders said that the ESP’s activities were neither referred to nor used to shape the response to COVID-19. They said that that the response to COVID-19 was based on informal solutions to unforeseen problems and needs, which formal ESP procedures did not support. However, these stakeholders also said that it was impressive how the ESS was kept running during the pandemic in Eurostat and the Member States, including by shifting to new ways of working. In addition, these stakeholders agreed that Eurostat was agile in leading the work on assessments of how COVID-19 impacted statistics 18 . Also, these stakeholders said that much work took place even if it did not lead to the production of new statistics, such as Eurostat advising colleagues responsible for policy developments on the statistical implications of anti-COVID-19 measures. The stakeholders agreed that this shows that the ESP was set up in a flexible enough way to allow for rapid innovations at a lower level than the ESSC.

The continuous and increasing relevance of the ESP is confirmed by the growing number of users accessing European statistics on the Eurostat website. The number of users is estimated via the number of distinct IP addresses that were used to access the database.

Figure 3: Total number of users (distinct IP addresses) accessing Eurostat’s website

Source: Eurostat’s website monitoring

Another phenomenon that shows the increasing relevance of the statistics produced is the number of data extractions made by users, which increased substantially during the ESP, as detailed under JC 3.2 in the chapter on effectiveness.

Yet another indication of the increasing relevance of the statistics produced under the ESP is the continuous growth in the numbers of people following Eurostat on social media. This can be seen in the figures from social-media monitoring, which are presented in Table 3 below. Eurostat opened its Facebook account on 10 January 2017 and its Instagram account on 4 May 2020.

Table 3: Eurostat’s followers on social media at the end of the year

Source: Eurostat’s social-media monitoring

Following the demands of the scientific community, Eurostat has increased the numbers of microdata datasets that are available for research purposes. As these datasets contain confidential data, researchers must follow a specific procedure to access them, according to Commission Regulation (EU) No 557/2013 on access to confidential data for scientific purposes. Table 4 below shows the list of datasets and the number of requests for the years 2014-2020. Three new datasets have been published over this period and the number of requests has increased every year until 2019, diminishing only in 2020 due to the COVID-19 pandemic. This is a sign of growing interest in the data contained within these datasets. A single request can be made for more than one dataset.

Table 4: Requests for access to microdata, 2014-2020

Source: Eurostat

JC 2.2 Mechanisms for feedback and input by users existed and were perceived as appropriate

Bilateral hearings between Eurostat and European Commission policy DGs were organised regularly. These bilateral hearings gave DGs the possibility to give feedback on their needs, including new statistics requests, to Eurostat. They were also an opportunity for the different parties to discuss the overall balance within the programme. The large majority of the requests made by the DGs in these bilateral hearings could be taken into account by Eurostat, which ensured that the ESP’s activities remained relevant for the needs of the main users. These hearings also enabled Eurostat to maintain an inventory of the statistical data produced by other DGs and services, thus helping to avoid possible duplication of statistics. More details on the hearings can be found in JC 10.2 in the coherence chapter.

Throughout the period evaluated, Eurostat regularly carried out general user-satisfaction surveys (USSs) 19 . These surveys: (i) measured the degree to which Eurostat met its obligations towards its users; and (ii) helped Eurostat ensure that the design of the programme was appropriate, by implementing improvement actions as a follow-up on the feedback received. In addition to the USSs, Eurostat carried out specific usability testing and used feedback gathered through: (i) daily contacts; (ii) questions coming into their user support services; and (iii) comments left on social media.

In addition, as part of the ESS Vision 2020, Eurostat engaged in a regular dialogue with users to better understand their needs. Eurostat recognised that different user groups have different needs and planned to address this diversity by offering the right information in the most appropriate way.

Eurostat also co-organised the 2014, 2016 and 2018 conferences of European statistics stakeholders to bring together producers and users of statistics to discuss user needs.

Even though the documentary analysis showed that mechanisms for feedback and input by users existed, most of the users interviewed were not familiar with these mechanisms and therefore could not assess them. However, this is based on a limited sample of users with different relations to Eurostat, and is therefore not representative.

JC 2.3 New technologies and statistical methods have been considered in programme-planning exercises

The 43 producers of statistics who responded to the targeted survey confirmed the flexibility of the ESP in adapting to technological advances. Most of these producers indicated that the programme was effective in adapting to: (i) increasing data-security risks (55% said the ESP was effective or very effective in this); (ii) innovation in technology and methods for statistical production (77% agreed the ESP has been effective or very effective in doing so); (iii) statistic variances across countries (79% agreed); (iv) big data (58% agreed); and (v) advances in data visualisation (72% agreed). However, national producers that were interviewed pointed out the lack of a strategy to access and integrate privately held data.

The ESS Vision 2020 programme mentioned several challenges the ESS faced in addressing: (i) the new opportunities presented by new technologies and new statistical methods; and (ii) emerging needs (for instance to capture global phenomena and develop statistics with increasing geographical detail).

The extension of the programme in 2018 recognised several of these challenges. It also recognised that there were opportunities to improve the timeliness and relevance of official statistics and to reduce response burden.

To face these challenges and the continued constraints on resources, the ESS decided to gradually implement the strategic goals set out in the ESS Vision 2020, building upon a holistic approach to deliver quality and efficiency gains. It would implement these goals by:

·using both traditional surveys and other sources, including administrative data, geospatial data and, where possible, big data to deliver statistical products and services;

·getting access to new data sources, creating methods, and finding suitable technology to use such data sources to produce European statistics in a reliable way;

·improving the efficiency of statistical production by further intensifying the sharing of knowledge, experiences, and methodologies, but also by sharing tools, data, services, and resources where appropriate and duly justified;

·implementing a dissemination and communication strategy for European statistics that is flexible enough to adapt to emerging technologies, giving guidance in a world that is undergoing a data revolution, and serving as a reliable pillar of democracy.

The ESP made it possible to invest in projects launched in response to new challenges faced by European statistics, including through the ESS Vision 2020. This led to several new projects, including:

·proofs-of-concept for both: (i) the generation of outputs (based on new data sources) in response to user needs; and (ii) addressing horizontal topics related to the use of ‘big data’ for European statistics 20 ;

·preparing the IT infrastructure of the ESS so it can exchange large volumes of sensitive data 21 , which in turn supported developments towards the

·secure exchange of intra‑EU trade microdata between all ESS members 22 ; 23 , making it possible to both substantially reduce the response burden for businesses and increase quality.

Under the ESP, Eurostat developed a variety of data-visualisation tools to better meet user needs. These tools presented data from different statistical themes in a way that was attractive and as understandable as possible so everyone could explore them. Eurostat’s tools used for the dissemination of statistics included:

·visualisation tools, such as the European statistical recovery dashboard, themes in the spotlight, etc. 24 ;

·the My Region mobile app;

·extraction tools, such as the data browser and explorer;

·the ESS experimental statistics hub 25 , which provided a way to disseminate experimental statistics on different topics, including COVID-19 (examples of these experimental statistics include: (i) COVID-19 labour effects across the income distribution 26 specifically; or (ii) quarterly data on registrations and bankruptcies, which can provide critical information on the economic effects of the COVID-19 pandemic 27 ).

Mechanisms to adapt to new technologies and statistical methods existed under the ESP in the form of pilot projects for example. This clearly shows that the ESP’s programme-planning exercises considered new technologies and statistical methods.

5.2 Effectiveness

The analysis of effectiveness considers how successful the ESP’s activities have been in achieving or progressing towards its objectives. This analysis gives an opinion on the progress made to date and the role of the ESP in delivering the observed changes. It looks for evidence of whether or how these changes are linked to the programme.

The EQs, presented below, explore the relation between the objectives and activities of the programme. The answers to these questions provide an evidence-based judgement of: (i) the extent to which the ESP’s objectives were successfully achieved; or (ii) what factors influenced why an activity/objective was (partially) unsuccessful or not yet achieved.

The evaluation findings show that the ESP was effective in delivering on its objectives, despite remaining weaknesses in the timeliness and completeness of European statistics. Between 2013 and 2020, Eurostat provided high-quality statistics which were used by a wide range of users for a wide range of purposes. These purposes include supporting the development, monitoring and evaluation of policies at the EU and Member-State level. In that period, the ESP also increased the availability of its data and statistics. To a more limited extent, it also introduced efficiency gains in the production of its statistics.

In addition, Eurostat effectively developed and strengthened partnerships with its members within the ESS, as well as with partners beyond the ESS – in particular international organisations. However, more needs to be done, in particular to: (i) get access to privately held data; and (ii) create partnerships with private organisations. Work towards both of these goals started in 2020 and is being addressed in the next programme.

In addition to achieving its objectives, the ESP was also effective in making access to official statistics easier and more user-friendly between 2013 and 2020.

EQ3: To what extent were the objectives of the ESP 2013-2020 fulfilled?

JC 3.1 Users expressed satisfaction as to the timeliness and completeness of statistics

Objective 1 of the ESP was to provide statistical information in a timely manner. This reflected a particular concern about timely data. This concern was also highlighted in the ESS Vision 2020 programme and repeated in both the 2016 Commission impact assessment on the proposal to extend the ESP, and the 2019 Commission SWD on progress in implementing the ESP.

Despite improvement in a number of statistical domains, timeliness remained an area where users said that work was ongoing and needed. The continued issue of timeliness is also reflected in Eurostat’s USSs19 conducted between 2013 and 2020. In that time, around half of users (results fluctuated between 51% and 54%, and respondent numbers fluctuated between 1 009 and 4 839) rated the timeliness of European statistics as either ‘very good’ or ‘good’ for their purposes. A higher satisfaction rate (63%) was registered in the last USS in 2020 (to which 1 842 responded). Nonetheless, as in previous years, respondents to the 2020 USS continued to ask Eurostat to further improve the quality of statistics by improving timeliness. Similarly, interviews conducted with users and producers of statistics, including stakeholders at the Commission, highlighted timeliness as an issue needing to be addressed. However, interviewees also acknowledged that quality official statistics required more time to be produced, and confirmed that there was therefore a trade-off between timeliness and other aspects of quality. Lastly, the timeliness of statistics also fluctuated between domains/statistical themes.

However, while timeliness was identified as a challenge, improvements in that regard were also highlighted by users. For instance, 51% of respondents to the PC (to which 424 responded) agreed that ‘the ESP had been successful in improving the production of European statistics in regard to timeliness and punctuality’ to a great extent, while 28% said this was true to a moderate extent, and only 5% said it was true to a lesser extent. Similarly, 82% of respondents to the targeted user survey (to which there were 45 replies) thought that the ESP was successful in improving the timeliness and punctuality of European statistics. Progress reported against the ESP objectives confirmed these perceptions. Indeed, the percentage of outputs achieved or on target for strategic objective 1 has consistently increased since 2016, reaching a peak in 2019, as detailed under JC 3.6 (see Figure 5 ).

On the completeness of statistics, based on the USS, half of users between 2013 and 2019 (with results fluctuating between 49% and 52%, and respondent numbers fluctuating between 1 009 and 4 839) rated the overall completeness of European statistics as either ‘very good’ or ‘good’ for their purposes, with a peak at 62% in 2020. The issue of completeness was also raised during interviews with users who highlighted gaps, particularly when data for certain countries were not available. These users also noted inconsistencies in how data was collected for different countries. Respondents to the targeted producer survey judged the availability of statistical data as sufficient, pointing out limitations due to confidentiality. In contrast, the producers of statistics that were interviewed assessed the availability of data as very good, with the right balance between timeliness and availability of data. Similarly, users at the European Commission stressed that the statistics provided under the ESP were comprehensive, comparable, and reliable.

JC 3.2 European statistics and data were downloaded and used by a range of stakeholders for a range of purposes

Serving the needs of a wide range of users remained a specific objective (objective 1) of the programme throughout the period under evaluation.

According to the 2020 USS (to which 1 842 responded), 39% of respondents identified themselves as advanced users, 35% as intermediate users and 26% as light users. Respondents belonged to all categories, from ‘students, academics and private users’ to ‘business’, ‘government’ and ‘EU and international organisations’. This suggests that European statistics were indeed downloaded and used by a range of stakeholders.

Web analytics confirmed that European statistics are downloaded at an increasing rate. Between 2014 and 2020, the number of data extractions made by external users from Eurostat reference databases greatly increased from 6 813 000 extractions to 21 480 000 (+315%) as shown in Figure 4 below. Similarly, between 2014 and 2020, the number of distinct IP addresses consulting European statistics increased from 3 441 157 visits to 11 731 274 (+240%). The number of requests submitted for microdata also increased until 2019, while diminishing in 2020 due to the COVID-19 pandemic.

Figure 4: Number of data extractions from Eurostat’s reference databases 2014-2020

Source: Eurostat’s website monitoring

Users of European statistics were asked in the 2019 USS (to which 1 009 responded) to indicate the reason for their interest in using European statistics. Among the responses available, ‘monitoring or formulating policy’ (chosen by 32%) and ‘general background information’ (chosen by 23%) were the most commonly reported purposes. Looking at the importance of European statistics, more than three quarters of participants (77%) indicated them to be either ‘essential’ or ‘important’ for their work.

Building on the USS, a targeted user survey (to which there were 45 responses) was carried out, in which respondents were asked to indicate the usefulness of European statistics for nine areas of practice. Six of these areas (development, monitoring and evaluation of policies at the EU level; development, monitoring and evaluation of policies at the Member-State level; academic research; scientific research; media reporting; and giving information on Europe) received a positive assessment (more than 60% said European statistics in these areas were either ‘useful’ or ‘very useful’).

JC 3.3 Eurostat data were used in journalistic and research activity

European statistics were used in journalistic and research activities, even if these activities were not the main use of these statistics. According to the targeted user survey (to which there were 45 responses), 9% of respondents indicated that they used European statistics for ‘academic/research’ purposes. Similarly, 12% of respondents to the 2019 USS (to which there were 1 009 respondents) said they used European statistics for ‘research’ purposes, which was the fourth most commonly selected main purpose, while 1% indicated that they used the statistics for ‘media use’. As counted by the Commission’s ‘My news’ tool, Eurostat was mentioned in the media more than 6 000 times per month on average in the last months of 2020.

JC 3.4 Statistics were used by EU policy makers to inform policy making and decisions

The reformulation of objective 1 in the programme extension in 2017 stressed that decision-makers at EU level (but also in Member States, local governments and business) needed European statistics ‘to prepare, apply, monitor and assess all EU policies’.

The findings from the USS, the targeted user survey, and the PC suggested that European statistics were indeed used by policy makers to inform policy making and decisions. In the 2019 USS (to which there were 1 009 responses), the most common purpose identified by users of European statistics was ‘monitoring or formulating policy’ (selected by 32%). Additionally, ‘preparing legislation’ and ‘monitoring or formulating policy’ both got a combined share of 87% of participants who indicated them to be ‘essential’ or ‘important’ for their work. Similarly, 64% of respondents to the PC (to which there were 424 responses) considered European statistics useful to a great extent in informing and monitoring policy and decision-making in Europe.

Interviews with users from the Commission confirmed these findings. Commission users systematically mentioned the comprehensiveness, comparability and reliability of the data provided under the ESP and the value of this data for EU policy work. For instance, the European statistical recovery dashboard for tracking the recovery from COVID-19 was highlighted by policy makers in several DGs as important to ‘inform and support policy makers’ and to ‘respond quickly to emerging needs’. In the case studies, regional data were also reported as being heavily used for policy planning, especially in the policy area of regional development. Similarly, migration statistics were reported by DG Migration and Home Affairs as important for informing EU migration policy, policy briefs, and legislative proposals.

JC 3.5 Data were being used by Member State policy makers to inform policy making and decisions

The findings outlined in this JC suggested that European statistics were also used by policy makers at Member-State level, albeit: (i) to a lesser extent than by EU policy makers; and (ii) to a greater extent at the national level than at the regional and local level. Further breakdown of the data by contributors showed that among the respondents to the PC (to which there were 424 responses), 81% of respondents from public authorities with a national scope of work considered European statistics relevant to inform and monitor public policy.

When asked about the usefulness of European statistics in the targeted user survey (to which there were 45 responses), 30% of respondents found them to be ‘very effective’ and 52% ‘effective’ in the ‘development, monitoring and evaluation of policies at the Member State level’. European statistics were found to be less effective at the regional level (where only 31% found them effective) and at the local level (where only 10% found them effective). This could be linked to the insufficient disaggregation of statistics at local level.

JC 3.6 ESP activities resulted in a net increase of available data; users considered newly available data sources to be useful

When asked in the targeted producer survey (to which there were 43 responses) about the availability of European statistics, more than three quarters of respondents indicated there had been an increase in the availability of data to a great extent (37% agreed) or to a moderate extent (44% agreed). Similarly, in the targeted user survey (to which there were 45 responses), respondents found that there had been an increase in quantity for general and regional statistics (44% agreed) and for economy and finance (45% agreed). A noticeable share (43%) of respondents to the PC (to which there were 424 responses) also reported an increase in the availability of data on population and social conditions. This was further corroborated by stakeholders interviewed as part of the asylum and managed-migration case study. Interviews conducted with users of European statistics, producers of European statistics, and people from EFTA and candidate countries confirmed this point.

Table 5 below shows the quantity of statistics and long-time series 28 published by Eurostat. Statistical coverage is calculated as the number of statistical indicators, sub-indicators and all their breakdowns included in Eurobase (the Eurostat dissemination database) not taking into account those differing only because of the time dimension 29 . The table thus demonstrates the increasing amount of data made available by Eurostat over the years.

Table 5: Statistical coverage in millions of statistics

Source: Eurostat reference database

The ESP’s great effectiveness in producing statistics is further demonstrated by the percentage of planned outputs which were achieved under ESP objectives 1 and 4, which were constantly above 92% (and above 95% in the second half of the programme), with only 3% or less of outputs cancelled or not fully achieved.

Figure 5: Eurostat’s activities contributing to the management plan’s outputs by year (2014-2020) for objective 1 30

Source: Data from Eurostat Annual Activity Reports 2014, 2015, 2016, 2017, 2018, 2019 and 2020. (*) for the year 2014 and 2015, objective 1 and 4 are combined

Furthermore, the few limited problems encountered in three detailed objectives in the second mid-term evaluation of the programme were all solved by 2019 The actions implemented to solve these problems included: (i) a pilot to test the remote-access system connecting safe centres in national statistical institutes (NSIs) to the central node; (ii) an SWD covering progress on the European public-sector accounting-standards project; and (iii) carrying out a survey on international sourcing.

JC 3.7 ESS stakeholders and/or users considered that ESP activity increased the efficiency of European statistics production

Specific objective 2 of the ESP was to implement ‘new methods of production of European statistics aiming at efficiency gains’. As this objective relates to efficiency, a detailed answer to this JC has been provided under the chapter on efficiency. Overall, both the qualitative and quantitative evidence collected suggest that ESP activities have, to a limited extent, resulted in efficiency gains by increasing the use of new, more efficient methods of production of European statistics while reducing administrative burden. However, although the consulted stakeholders acknowledged efficiency gains implemented by the ESP, they also said that more innovative solutions and sources should be used to further increase efficiency.

Eurostat launched a set of projects to increase the efficiency and quality of the process for producing European statistics. These projects included: (i) implementing the ESS Vision 2020 programme; (ii) strengthening the European statistical infrastructure; and (iii) making better use of big data and shared infrastructure, as detailed under JC 2.3 in the chapter on relevance.

JC 3.8 ESS stakeholders considered that ESP activity had avoided or prevented duplication of effort on the part of stakeholders

As laid down in its 2016-2020 strategic plan, Eurostat sought to provide high-quality statistics in a ‘cost effective manner without unnecessary duplication of effort’. This was achieved through: (i) sharing knowledge and best practices across Member States; and (ii) developing new technologies, common tools and collaborative networks in the ESS for ‘taking advantage of possible synergies and avoiding duplication of effort’. Eurostat therefore produces European statistics in partnership with the Member States based on shared statistical standards, methods, procedures, practices, and tools.

Respondents to the targeted producer survey (to which there were 43 responses) indicated that to a moderate extent (51% agreed) and to a great extent (19% agreed) the ESP had been successful in reducing duplication of effort.

Additionally, stakeholders within the Commission highlighted the ongoing dialogue between Eurostat and policy DGs on the ‘inventory of other statistics’ as being effective at preventing overlap and duplication of efforts. Eurostat’s interviewees also highlighted the European Commission’s participation in discussions in international partnerships. Some of these discussions focused on efforts to limit the response burden for Member States that are statistics producers for different international organisations by avoiding the duplication of statistics production.

JC 3.9 ESS stakeholders and/or users considered that ESP activity had increased the quality of European statistics production

As mentioned above, specific objective 2 of the ESP was to implement ‘new methods of production of European statistics aiming at ... quality improvements’. The indicator on timeliness already covered this objective. Additionally, the percentage of users that rated as ‘very good’ or ‘good’ the overall quality of European statistics remained constant between 2013 and 2019 (fluctuating between 57% and 60%, with respondent numbers fluctuating between 1 009 and 4 839) and increased considerably in 2020 (when there were 1 842 responses to the USS) to 72%. The increase in 2020 will have to be checked in future surveys to confirm if the overall quality of statistics has improved. Nevertheless, it should be noted that overall quality is rated more favourably than individual quality dimensions.

Similar findings were reported in the targeted user survey (to which there were 45 responses) where respondents were asked to judge the overall quality of nine of Eurostat’s statistical themes. Overall, respondents said that all themes were of either high or moderate quality. However, when asked whether there had been changes to data that improved the quality of the statistics available, for most themes, respondents were unsure if there had been such changes. The only themes in which changes were noted were: (i) economy and finance where 58% of respondents reported having noticed changes to the data that improved the quality of the statistics available; and (ii) population and social conditions where 39% of respondents noted changes.

Producers of statistics, users of statistics, European Commission DGs, EU bodies, and stakeholders from EFTA and candidate countries all emphasised in their interview the high quality of the statistics produced by Eurostat. Additionally, and in contrast to the surveys’ findings, interviewees were positive about the progress which had been made in improving the quality of European statistics production.

Overall, the quality was assessed by all stakeholders as being high, which would mean that there was little room for improvement and would therefore suggest that the small increase witnessed still represents an improvement.

JC 3.10 ESP activity strengthened existing partnerships or developed new ones

The ESS is a partnership between Eurostat and the NSIs and ONAs responsible in each Member State for developing, producing and disseminating European statistics. This partnership also includes the EEA and EFTA countries. Specific objective 3 of the ESP was to ‘strengthen the partnership within the ESS and beyond in order to further enhance its productivity and its leading role in official statistics worldwide’.

Progress made against this objective has been reported on in Eurostat’s annual activity reports (AARs). An analysis of the AARs in recent years shows that the percentage of outputs achieved or on target for strategic objective 3 has been high since 2014, suggesting that ESP activities have been successful in strengthening existing partnerships.

Figure 6: Eurostat’s activities contributing to the management plan’s outputs by year (2014-2020) for objective 3

Source: Data from Eurostat AARs 2014, 2015, 2016, 2017, 2018, 2019 and 2020.

Additionally, the comparability of European statistics was one of the important expected results of the partnership within the ESS. According to the USSs, the percentage of respondents viewing the comparability of European statistics as either ‘very good’ or ‘good’ increased between 2014 (when it 50% assessed comparability as either ‘very good’ or ‘good’, from a survey to which 4 839 replied) and 2019 (when it was 53%, from a survey to which 1 009 replied) with a jump in 2020 (when 1 842 responded to the survey) to 58%. This was confirmed in the interviews with users, who highlighted the close and intense cooperation with Eurostat.

Producers of statistics in the Member States reported an increase in – and strengthening of – partnerships. They also reported a good overall level of cooperation across the ESS. On the COVID-19 crisis, these producers of statistics said that meetings and exchanges to share good practices and new methods had increased since the outbreak of the pandemic. Examples of aspects that worked well in the partnership and which were highlighted by interviewees were: (i) the working groups, taskforces and meetings between Eurostat’s Director and the directors of the NSIs; and (ii) the annual hearings organised bilaterally between Eurostat and the policy DGs. In interviews, EFTA and candidate-country stakeholders also noted the strengthening of partnerships taking place within the ESS. The evaluation found that Eurostat cooperated closely with international organisations to: (i) align their methodologies; (ii) agree on definitions for individual statistical indicators; and (iii) validate and exchange data. Eurostat currently has memoranda of understanding (MoUs) signed with four organisations: (i) the Organisation for Economic Cooperation and Development (OECD); (ii) the ECB; (iii) the UN International Civil Service Commission (UN ICSC); and (iv) the International Service for Remunerations and Pensions (ISRP). Eurostat has also signed administrative arrangements with four organisations: (i) EFTA; (ii) the UN Statistics Division (UNSD); (iii) the Food and Agriculture Organisation (FAO); and (iv) the OECD. Stakeholders interviewed from these organisations were overwhelmingly positive about their partnerships with Eurostat, highlighting a close and fruitful collaboration.

Additionally, according to the 2019 final audit report on the effectiveness of Eurostat’s cooperation with external stakeholders, this cooperation has had ‘a significant impact on the implementation of the ESP’. However, despite an overall positive assessment, this final audit report also highlighted some challenges. To address these challenges, corrective actions were undertaken in 2019 and 2020, including: (i) the development of a strategy for international cooperation; (ii) the revision of internal guidance on administrative arrangements; and (iii) the preparation of formal agreements to be signed with external stakeholders. The Internal Audit Service (IAS) concluded in its follow-up report in 2021 that the recommendations had been adequately and effectively implemented.

Eurostat also cooperates with non-EU countries based on regional and thematic priorities to exchange information and practices that help to address global challenges.

As part of its work to create a quality framework for big data, as first stipulated by ESGAB Recommendation 2017/4, Eurostat has, as part of the ESS Vision 2020 BIGD project, worked on developing a methodology and practices for dealing with privately held data and their quality aspects. However, Eurostat considered that access to privately held data was still insufficiently covered and more needed to be done in this area. In its 2020-2024 strategic plan, Eurostat is planning a stronger partnership with the private sector to improve data-production processes.

JC 3.11 Stakeholders identified factors that prevented or reduced the impact of the ESP’s activity

Producers of statistics stressed the need for more innovative solutions and sources to be used, such as big data based on privately held data. However, using such data requires specific skills traditionally not available in statistical offices. It also poses complex problems, such as:

·the lack of incentives and enablers, including in harmonised legal provisions, for private data holders to contribute to the production of official statistics for public benefit;

·the perceived risks for compliance with data-protection legislation and reputational risks linked to privacy protection on the part of private data holders;

·the high initial investment needed to re-purpose data and to develop methodological and quality frameworks related to the use of new data sources.

Eurostat is aware of these problems and will work together with the ESS to solve them. Work is already planned under the SMP and included in the 2021-2027 multiannual action plan.

Problems with timeliness reduced the impact and use of the statistics being produced by Eurostat. However, respondents and interviewees acknowledged that timely statistics were often dependent on external factors, namely Member States’ capacity to provide statistics in a timely manner. Interviewees also acknowledged that quality official statistics needed time to be produced and verified. Interviewees also agreed that an additional challenge to timeliness came from the fact that the EU’s legislative cycle is relatively long.

Stakeholders from Eurostat said that modernisation required an upfront investment which Member States may be reluctant to pay. They said that this had impacted the ESP’s ability to deliver efficiency gains to reduce burden in the medium-term. The stakeholders said that the impact of COVID-19 was likely to produce financial restraints on the EU and on Member State budgets dedicated to producing statistics. They said that, as a result, there was an increased risk of delays in achieving the objectives of modernising statistics production.

An additional factor that limited the impact of the ESP was the increasing risk of third-party dissemination of non-validated or fake information. In response, the ESGAB recommended in its 2019 report that Eurostat and the NSIs design appropriate actions of communication and outreach to highlight the trustworthiness of official statistics. To address this problem, ESAC also recommended promoting statistical literacy at all educational levels and improving the public’s statistical literacy to promote the value of official statistics.

EQ4: To what extent did the ESP make progress on rendering access to official statistics easier and more user-friendly?

JC 4.1 Effective processes were in place to monitor and receive feedback on data accessibility

User feedback is the best way to assess the accessibility and clarity of European statistics. To monitor feedback from its users, Eurostat carried out several general USSs throughout the period under evaluation. The objective of these surveys was to measure the degree to which Eurostat met its obligation towards its users. The surveys were designed to obtain better knowledge about users, their needs and their level of satisfaction with the services and statistics provided by Eurostat, including in the area of data accessibility. USSs were held in 2013, 2014, 2015, 2016, 2017, 2019 and 202019. In addition, input and feedback were also received through daily contacts from: (i) questions coming into the user-support service; (ii) comments left on social media; and (iii) a staff feedback mechanism. These forms of feedback fed into the revision and creation of new and existing communication materials. When asked in the PC (to which there were 424 responses) about Eurostat’s communication channels, more than half of the respondents (53%) said they found that these channels had been effective to a great extent in responding to their feedback and input, while 35% found them effective to a moderate extent. The same question was asked in the targeted user survey with similar results.

Both users and producers of statistics were satisfied with the processes in place to guarantee effectiveness. However, many users may not be familiar with the feedback mechanisms, as indicated in the interviews.

JC 4.2 Relevant information (e.g. new releases, data-quality standards) was available in a clear and accessible manner

The main way to access European statistics was through the Eurostat website, which has provided users with free access to its databases and all its communication and electronic publications.

The Eurostat website was updated daily throughout the period of the ESP, and gave visitors access to the latest and most comprehensive statistical information available on the EU, EU Member States, EFTA countries, and candidate countries. The Eurostat website also provided access to a narrower range of statistics covering non-members of the EU, including potential candidate countries, neighbouring countries, and other major economies, such as Japan and the United States. People who registered (free of charge) were able to receive tailor-made email alerts providing information on new publications as soon as they were online. They also had access to enhanced functionalities within databases (for example, the ability to save data queries and make bulk downloads). Journalists were able to obtain, on the day of publication, news releases or a weekly release calendar by email in German, English, or French. Data were made available for download from the website’s data explorer in various formats (XLS, CSV, HTML, PC AXIS, SPSS, TSV and PDF). Similarly, Eurostat’s dissemination unit also created different profiles for users to ensure that they receive tailored communication packages.

Eurostat’s website was completely redeveloped at the beginning of the programme. The new website was launched in December 2014 and it has been continuously improved ever since. Since the beginning of the ESP, Eurostat has developed and added many new: (i) electronic and interactive publications; (ii) visualisation tools; (iii) mobile apps; and (iv) tools for data extraction. These tools present data from different statistical themes in an attractive and easy-to-understand way for everyone to explore and can be easily accessed on Eurostat’s website . More details can be found under JC 2.3 in the chapter on relevance.

Adding modern ways of communication, Eurostat has also been active on social media, with its three corporate social-media accounts on Twitter , Facebook and Instagram .

Users that were interviewed provided positive feedback on Eurostat’s publications, with the publication ‘Statistics explained’ being referred to several times as useful and well written. Different aspects of the publications (such as detailed metadata with descriptions, sources and methods) were also deemed useful and easily accessible via links.

However, some limitations in accessibility and clarity were revealed in the USS. On average, in 2019 (when there were 1 009 responses to the USS), only 45% of respondents found metadata sufficient for their purposes (48% found them partially sufficient, and 7% not sufficient). Additionally, although some users reaffirmed in their comments that metadata were clear, complete, and better than those of other data providers, others found the metadata difficult to access, not clear enough, too long, or too technical. The main improvements suggested by users included: (i) providing some more basic metadata that was easy to understand and written in plain language for non-specialists; (ii) providing metadata at a more detailed level and for all indicators; and (iii) giving clearer and more complete definitions of all codes. Other respondents wished to get more information on the production of statistics and the methodology used, and to understand more easily the differences between countries. They also said that metadata should be consistent over time and among different statistics, and always updated if there were changes in the methodology.

In both the targeted producer surveys and the targeted user surveys, respondents were asked to rate the extent to which Eurostat was providing sufficient information about the sources of the data, the data-quality standards, and the methods of data collection. Answers were mostly positive in all cases, suggesting that both producers and users found relevant information on the data produced by Eurostat to be available in a clear and accessible manner, with producers being slightly more positive in their feedback than users.

Overall, and despite the weaknesses identified (notably with metadata), the findings suggested that relevant information was provided by Eurostat in a clear and accessible manner.

EQ5: Did ESP activity result in wider economic, social, or environmental impacts?

JC 5.1 Examples of wider social, economic, or environmental impact were identified by stakeholders and attributed in part or in full to ESP activity

Identifying the direct social, economic, or environmental impact of statistics is difficult. This is because statistics ordinarily only have an indirect impact through the policies that are based on them. However, stakeholders at Eurostat highlighted that the ESP had had a wider direct social impact, thanks to training, skills, and the diffusion of innovation. One example of this is the creation of the European Master’s in Official Statistics (EMOS) to strengthen collaboration among academics and producers of official statistics, and to help develop a professional network of statisticians. Similarly, interviewees mentioned the development of the ‘trusted smart statistics’ hub on web intelligence. At this hub, all NSIs are provided with the opportunity to explore the web as a data source, and to use algorithms and coding for free to illustrate the wider social impact of Eurostat activities. Additionally, Eurostat has an ‘ education corner ’ page on its website where teachers can find material to use in the classroom when teaching statistics, geography, social science, etc. The education corner can also be used directly by students to gain an easier understanding of statistics.

The case studies also showed the social impact of European statistics. For instance, asylum and managed-migration statistics were reported as having a social impact linked to the integration of refugees into their host communities. This integration affected issues beyond migration, ranging from health and housing to employment. Similarly, the case studies showed that the wide range of statistics and data published by Eurostat in its European statistical recovery dashboard as part of its response to the COVID-19 crisis had a wider impact by providing a baseline against which the impact of the crisis could be measured. The dashboard thus supports analysis of how the pandemic is developing, and of how EU recovery policies are decided on and coordinated. More broadly, European statistics have a crucial impact on the long-term economy of the EU as Eurostat is responsible for providing the data used by DG Economic and Financial Affairs for surveillance as part of the Stability and Growth Pact 31 .

European statistics also have an indirect environmental impact. For example, the European environmental economic accounts set out the share of the overall economy occupied by the environmental goods and services sector. It also detailed the environmental goods and services sector’s production and consumption of natural resources and energy.

5.3 Efficiency

Under the efficiency criterion, the evaluation focused on the costs of producing European statistics, trends in these costs, and the burden of producing these statistics faced by the NSIs and ONAs. Three general issues were assessed to find out whether the ESP was being run efficiently. The first issue related to whether the programme ensured the best use of available resources, both financial and human. The second issue concentrated on the costs and burden involved in producing European statistics in the ESS and how these costs and burden have evolved. The third issue was to assess how efficient the ESP process was for reporting and monitoring.

The analysis of the evidence suggests that the programme has been efficient. The ESP demonstrated efficient use of both financial and human resources while delivering high-quality European statistics. Moreover, the productivity of statistics production increased during the period evaluated. Factors underpinning these developments were good governance, good management, and effective monitoring mechanisms.

Stakeholders expressed concerns about the costs for Member States and the administrative burden placed on data providers. These concerns arose because of the need to produce an increasing volume of statistics, which made it necessary to increase the efficiency of statistics production. Several initiatives were undertaken to respond to this challenge including: (i) the activities within the ESS Vision 2020 programme; (ii) an initiative to modernise business statistics resulting in the European Business Statistics Regulation 32 ; (iii) an initiative to improve social statistics 33 with the Integrated European Social Statistics Regulation; (iv) an initiative to improve agriculture statistics with the Integrated Farm Statistics Regulation 34 ; and (iv) work to strengthen partnerships within the ESS.

EQ6: To what extent were ESP resources used efficiently to achieve the desired results?

JC 6.1 Sufficient information about costs and benefits across ESP activities was available and used to inform programme decisions and operations

Producers were generally positive in assessing the cost-benefit ratio of the ESP. In the targeted producer survey (to which there were 43 respondents), most respondents (55%) assessed the cost-benefit ratio as proportionate and a further 7% assessed it as very proportionate. However, 33% of respondents were not able to answer this question.

Producers were also asked to indicate the extent to which the ESP had been successful in introducing efficiency gains in the production of statistics by: (i) reducing duplication of effort; and (ii) reducing administrative burden. Most respondents indicated that the ESP was more successful in reducing duplication (70% of respondents agreed it was successful to a moderate or a great extent) than administrative burden (48% agreed to a moderate or a great extent).

The respondents to the targeted user survey (to which there were 45 respondents) were asked to assess the cost-benefit ratio of the ESP, but more than half of respondents (57%) said that they could not give an answer. In addition, 11% of respondents described the cost-benefit ratio as very proportionate and 29% as proportionate.

In the open-ended questions in the targeted producer survey, some respondents felt it was necessary to get rid of unused data and to stop collecting irrelevant data. However, when asked to indicate less important statistical fields that could be deleted from the programme, respondents did not consider any fields as less important.

JC 6.2 Measures and processes in place to monitor and optimise the efficient use of resources were effective

To measure costs, Eurostat launched three projects in the ESS: (i) a cost-assessment survey on producing official statistics in the ESS (including NSIs, regional offices, ONAs, and national central banks (NCBs)); (ii) a survey on the cost of European statistics (by product) in the ESS (including NSIs, regional offices, ONAs, and NCBs); and (iii) sharing best practices in the area of cost accounting within the ESS.

Information from the survey on the costs of producing official statistics in the ESS (project 1) is presented in Table 8 below (under JC 6.4). The cost fell in real terms between 2013 and 2020. A similar conclusion was derived from the analysis of the cost of European statistics (by product) in the ESS (project 2). The analysis covered 26 products in the first phase and 27 products in the second phase and it concentrated on costs related to the ESP only. Sharing best practices in cost accounting was aimed to apply common principles for the cost-collection surveys for the two projects. The initiative on sharing best practices in the area of cost accounting sought to both improve adherence to 23 common guiding principles for the cost-collection surveys and identify common items.

The evaluation also found that an important way to optimise the efficient use of resources in the production of statistics is to draw up framework regulations. The European Business Statistics Regulation (EBS) adopted in November 2019 brought statistics in the business sector under a common legal framework. The EBS Regulation showed that the principle of ‘collect once, use many times’ has been followed, and it therefore significantly reduced the risk of double counting or inconsistency. The Integrated Farm Statistics Regulation (IFS Regulation) adopted in 2018 modernised farm statistics by: (i) setting up the data-collection schedule for 2019-2029; (ii) introducing more flexibility and efficiency; and (iii) providing a harmonised basis not only for farm statistics but also for the whole agricultural statistics system. The IFS Regulation also strengthened the possibility of using administrative data sources, reducing the burden on respondents and data providers. The Integrated European Social Statistics Regulation (IESS Regulation) adopted in October 2019: (i) brought seven social surveys under a common legal framework; (ii) rationalised the planning of the different surveys; (iii) significantly reduced inconsistency between the different surveys by using common standardised variables; and (iv) improved overall relevance. In particular, the IESS Regulation makes possible a better use of resources and makes European Social Statistics more responsive to user needs by making the core and modules of the surveys more flexible and by making the data more timely and more precise.

JC 6.3 Anti-fraud measures and processes in place were effective

In 2013, Eurostat adopted an anti-fraud strategy valid for 2014-2017 and implemented an updated strategy for 2018-2020, following the updated methodology and guidance issued by the European Anti-Fraud Office (OLAF). The overall objective of Eurostat’s anti-fraud strategy was to provide assurance that risks related to managing financial transactions and the control environment were adequately managed, taking into account the cost-effectiveness of managing those risks. The updated strategy included an action plan with 10 actions, with accompanying key monitoring indicators for the actions. The strategy was evaluated in 2019, showing that all 10 actions had been effectively implemented in an efficient way. As a result of these good strategies, no investigations were opened by OLAF, nor was any potential fraud‑related case reported to OLAF during the ESP.

JC 6.4 ESP operations spending was efficient

To assess the efficiency of spending by the ESP operations, the evaluation examined the following elements:

·the ESP budget for 2013-2020 and its execution rate;

·Eurostat’s staff productivity;

·expenditure on other contributions;

·the costs faced by Member States in producing official statistics.

The first element examined was the budget for the ESP. The budget consisted of: (i) the regular budget made available by European Commission decisions; and (ii) additional sources, mainly a contribution from EFTA. The execution rate of the budget showed whether available financial resources were spent as planned to provide an indication of whether the ESP achieved the desired results. The execution rate was extremely high during the entire programme, at more than 99% since 2015 and reaching 100% in 2020.

Table 6: ESP budget execution rates 2013-2020

Sources: Eurostat Unit A.4

Staff numbers at Eurostat fell by 7% (52 people) in full-time equivalents (FTEs) during the implementation of the ESP. In the same period, the number of annual datasets published by Eurostat increased by 18%. This means an increase in statistics productivity of Eurostat staff by more than 20% during this time.

Table 7: Number of Eurostat staff and published datasets at the end of years 2013-2020

Source: Eurostat

The staffing policy meant that there was pressure on existing staff and no flexibility to manage new features of the programme (the only possibility was to stop dealing with other parts of the programme so as to be able to reallocate staff to any new feature). Eurostat also faced recruitment challenges to attract and retain skilled staff. These staffing challenges also existed at Member-State level, with interviewees emphasising that some NSIs were critically understaffed.

The efficiency of the ESP was strengthened by access to additional money. There was – and there still is – very good cooperation with thematic DGs in the European Commission, and this led to the mobilisation of specific budget lines. For instance, a DG was sometimes able to fund a particular dataset through ‘sub-delegated credits’ to the ESP (however, this raised the question of how this dataset would be managed in the absence of additional staff). The funding of the partnerships with candidate countries and non-EU countries provided another example. These partnerships fell under EU development cooperation, and were covered by a contribution from the IPA (Instrument for Pre-accession Assistance). Partnerships with the European neighbourhood-policy east countries were funded from the European Neighbourhood Instrument, and partnerships with the Pan-African programme were funded under the Development Cooperation Instrument.

Eurostat launched a survey to assess the cost of producing official statistics in the ESS including NSIs, regional offices, ONAs and NCBs. The information collected showed that costs fell by around 4% in 2012-2016. The decline in FTEs was reported at around 3% over the same period. Cost estimations for 2017 and 2020 showed a slight increase over the period but remaining practically stable in real terms. At the same time, the numbers of staff working in official statistics across the ESS decreased by 6 700 FTEs between 2013 and 2020. These estimated figures show that the burden on the resources of European statistics fell at an annual pace of 3-4%.

Table 8: Costs estimates and FTEs in the ESS

Source: Surveys on the cost of statistics in the ESS

At Member-State level, interviewees commented on the appropriateness of the budget available for producing statistics. This has remained a concern since the adoption of the ESP in 2013. Back then, the EU had just gone through the 2008 economic and financial crisis which put a lot of pressure on national budgets and even more pressure on funding for statistical production. Interviewees all agreed that a similar situation of reduced funding would likely happen soon because of the COVID-19 crisis. Interviewees also said that Member States often lacked capacity and staff numbers to deal with the increasing number of statistics requested.

Overall, and even if it was difficult to assess production costs, these costs remained a concern for the ESP. They also remained a variable that should be monitored to ensure the continued acceptability of the programme. The continued pressure on costs was another reason why the work done on harmonising methods and modernisation under the programme was critical for the ESP. This work helped make the ESP more efficient.

JC 6.5 The ESP was successful in introducing efficiency gains in statistics production

Specific objective 2 of the ESP was to implement ‘new methods of production of European statistics aiming at efficiency gains’. Progress made towards this objective was reported on in several Eurostat AARs. An analysis of these AARs over the lifetime of the ESP shows that the percentage of outputs achieved or on target for strategic objective 2 has significantly increased since 2014. This suggests that the ESP activities dedicated to finding more efficient ways to produce European statistics have increasingly been met. As stated in Annex 2 of the 2020 Eurostat AAR, the relatively lower percentage of outputs achieved or on target registered in 2020 (89%) was because a few of the new projects encountered some delays, mainly due to COVID-19.

Figure 7: Eurostat’s activities contributing to the management plan’s outputs by year (2014-2020) for objective 2

Source: Data from Eurostat AARs 2014, 2015, 2016, 2017, 2018, 2019 and 2020

To further assess the extent to which this objective was achieved, respondents to the targeted producer survey (to which there were 43 responses) were asked to rate the extent to which the ESP had been successful in introducing efficiency gains in the production of statistics by reducing administrative burden. 48% of respondents indicated that the ESP had been successful to a moderate or great extent, while 43% said it had been successful to a small extent or not at all. This suggests that while ESP activities may have resulted in efficiency gains in the production of European statistics by reducing administrative burdens, stakeholders had a mixed view of the effect of these measures.

Efficiency gains in the production of statistics were also mentioned during the interviews with producers of statistics. In particular, these efficiency gains were mentioned with regard to: (i) the increased use of administrative and register data; (ii) the increased use of estimation methods; and (iii) the application of microdata exchanges. Stakeholders also noted significant improvements in the provision of European statistics, for example due to the adoption of framework regulations. These efficiency gains were closely related to the strengthening of partnerships within and beyond the ESS. However, some interviewees said that these efficiency gains in recent years had tended to focus on the provision of European statistics, whereas data collection tended to be the costliest part of producing statistics for Member States. Additionally, interviewees highlighted that more innovative solutions and sources should be used, such as big data based on privately produced data.

Finally, in 2019 the IAS completed an audit of Eurostat’s quality management of statistical processes 36 . The IAS concluded that although the primary quality controls embedded in the production processes were adequately implemented, the quality review function was not yet effective. Therefore, IAS made two recommendations, one to improve the design and implementation of quality reviews and one to strengthen the quality controls by production units. Eurostat implemented an action plan to address these recommendations and the IAS concluded in its follow-up on 18 January 2021 that the first recommendation had been adequately and effectively implemented and was therefore closed. The second recommendation will be implemented as set out in the action plan by 31 December 2021.

EQ7: To what extent were ESP activities successful in limiting the administrative burdens for ESS stakeholders, including Member States and data providers (respondents)?

JC 7.1 Measures and processes in place to analyse administrative burdens for NSIs, statistics producers and data providers were effective

There was a concern expressed by statistics producers that the pressure for better quality, consistency, timeliness and availability often led to a higher burden of statistical production. Statistics producers said that these increasing requirements may exceed producers’ resources.

Interviewees highlighted that the Member States varied in size, and therefore the NSIs varied significantly in capacity. However, all NSIs were obliged to produce the same mandatory statistics. Interviewees said that treating all countries equally created a greater burden on smaller Member States. Nevertheless, they acknowledged that smaller countries may receive derogations and sometimes do not need to produce data with the same breakdowns or level of detail.

Progress made with the framework regulations and the ESS Vision 2020 programme only partly helped to reduce costs. Interviewees said that a really effective system for priority setting was needed, especially for identifying statistics that had become less relevant over time. Although Eurostat had been conducting an annual priority-setting exercise, interviewees said this exercise was not very effective in reducing costs for Member States. In February 2020, the annual priority-setting exercise was replaced by a new priority-setting mechanism adopted by the ESSC, and a pilot review for climate change-related statistics was also put in place. The new mechanism is based on four elements: a multiannual action plan, reviews, user dialogue, and resources/funding.

Interviewees from candidate countries also said that they felt a somewhat heavy administrative burden. They mentioned as valuable the support received through the financial support of the IPA multi-beneficiary programme and the European statistical training programme .

In conclusion, smaller Member States experienced more challenges in dealing with administrative burden than larger countries. Funds available for pilot statistics partly contributed to limiting the burden. Processes to analyse administrative burdens for NSIs were partly effective from the stakeholders’ point of view.

JC 7.2 Measures undertaken to reduce administrative burdens for NSIs, statistics producers and data providers resulted in an improved cost-benefit balance

Most statistics producers consulted within the evaluation tended to provide much more input on production costs than on administrative burden, which is the focus of JC 7.2. Nonetheless, the stakeholders consulted stressed that there was a constantly increasing demand for timely and high-quality statistics, including at EU level. They also said that the mechanisms within the ESP did not help to address these problems. Some interviewees highlighted that institutional users, both national and at EU level, should be made more aware of the production costs of statistics.

Stakeholders referred to several approaches that had been implemented to reduce this burden. The most effective approaches were the introduction of new methods and tools of data collection such as: (i) more intensive use of administrative data and registers; (ii) more sample surveys; (iii) online data transmission or web scraping. Another approach taken by Eurostat was the development of experimental statistics, which they believed held great promise – for example the use of big data as an additional data source. Eurostat also made efforts to increase the interoperability of data and metadata exchange via: (i) the use of standards (SDMX, SIMS); and (ii) improvements to the IT infrastructure for data exchange. Stakeholders also highlighted several solutions that had been developed to reduce the burden of producing business statistics, such as microdata exchange in intra-EU trade statistics and in the European system of interoperable business registers.

The management and governance mechanisms of the ESP were generally considered efficient. Statistics producers considered that there was good coordination with – and good communication from – Eurostat.

Several interviewees mentioned that the ESS should adopt a ‘one in, one out’ approach to statistical surveys, because there had been no critical review of new tasks and the impact of new tasks on additional resources were not sufficiently explored. However, some interviewees also acknowledged the difficulty of introducing the ‘one in, one out’ approach in practice, saying that this approach had also been considered at country level but never successfully implemented.

JC 7.3 NSIs, Member States and other data providers considered the benefits of the ESP to outweigh the administrative burden they face

All interviewees appreciated the benefits of participation in the programme. Closer partnership and the exchange of experience and best practices between NSIs helped the members of the ESS to find better ways to produce statistics. Therefore, the benefits of participating in the ESP outweighed the costs and burdens. The benefits and added value of the ESP are presented and discussed in Section 5.5.

EQ8: Was the management/organisation of the ESP as a whole conducive to supporting efficient delivery?

JC 8.1 Effective systems were in place to review the ESP’s performance and identify risks to the efficient use of resources

Eurostat prepared the annual work programmes to ensure that European statistics could be produced with the resources made available at the national and EU level. By prioritising, Eurostat aimed to help reduce costs and burdens for new statistical requirements by reducing statistical requirements in existing domains of European statistics.

The ESS Vision 2020 programme was founded on a clear need – agreed by all the members of the ESS – to modernise the production of European statistics to ensure that the ESS remains competitive in the future. One of the five outlined priorities was the promotion of efficient and robust statistical processes.

Staffing challenges existed at Eurostat and at Member-State level, with interviewees emphasising that some NSIs were critically understaffed (saying that this in turn might have consequences for their capacity to deliver against the set objectives). The efficiency of the ESP was also strengthened by the financial leverage existing at different levels. There was very good cooperation with thematic DGs of the European Commission, which had led to the mobilisation of specific budget lines.

The financial management of the ESP was led confidently and did not create risks.

JC 8.2 Stakeholders involved in the governance of the ESP and Eurostat had sufficient access to information to monitor the efficiency of resource use

The information included in the AARs proved that Eurostat’s operational budget implemented the ESP in line with the multiannual framework and in accordance with the Financial Regulation. An appropriate set of checks, both internal and external, were applied to Eurostat’s financial transactions, and this resulted in a positive assessment on the efficiency of resource use. Audit observations on budget implementation were also positive.

Table 9: Overview of costs and benefits identified in the evaluation

5.4 Coherence

This section focuses on the internal and external coherence of the ESP. The internal coherence of a programme is determined by looking at how the various components of that programme operate together to achieve the intended objectives. The external coherence of a programme relates to the extent to which the programme’s activities are aligned with other EU and international partner activities.

The answers to the EQs analyse where and how the programme’s objectives and activities worked internally and externally in a coherent manner, and points to areas where there are tensions.

No overlaps or inconsistencies were identified at the programme level between the objectives and among the activities listed within the regulations and planning documents of the ESP (internal coherence). The evaluation identified various governance bodies and advisory boards that had a coherent mandate. The work of these bodies and boards contributed to the internal coherence of the ESP by ensuring that its activities were in line with its objectives. Furthermore, interviewees involved in the production of European statistics highlighted the internal coherence of the ESP.

Eurostat effectively cooperated with EU bodies and agencies, as well as international organisations. This cooperation resulted in the external coherence of the ESP by ensuring synergies with: (i) wider EU objectives and needs; and (ii) international statistical activities. This cooperation also ensured the development of comparable and harmonised European statistics at the regional and international level. However, some weaknesses which could potentially affect the external coherence of the ESP were identified, namely: (i) the lack of flexibility in European statistics to respond to emerging needs (because this lack of flexibility might cause a misalignment with other EU strategies should the needs not be covered); (ii) the burden on Member States when replying to requests from individual DGs for statistics; and (iii) the necessity of getting access to privately held data and creating partnerships with private organisations (for which activities began in 2020). Eurostat has begun to tackle these weaknesses and will continue to address them in the next programme.

Overall, the findings suggested that the ESP was internally and externally coherent.

EQ9: To what extent did ESP activities and objectives contribute to the internal coherence of the ESS?

JC 9.1 Activities and objectives set out in the Regulations and internal planning documents were internally coherent

The second mid-term evaluation of the ESP indicated that the ESP was internally coherent. Similarly, the desk review conducted as part of this final evaluation identified no coherence issues between the ESP’s objectives and its activities.

In 2013 and 2014, another spending programme was running in Eurostat: the programme for the modernisation of European enterprise and trade statistics 38 . The first mid-term evaluation concluded that the ESP was well coordinated with this programme. In the following years, the ESP was Eurostat’s only spending programme and the objectives of Eurostat were identical to the objectives of the ESP, and therefore automatically coherent with it.

The ESP had three priority areas set out in the regulations. The first priority area was statistical outputs, which dealt with statistics production. This priority area defined the European statistics that would be produced or developed by the ESS. The second priority area was production methods for European statistics, which supported the production of statistics by improving the way statistics were produced, improving their quality, and improving the way they were disseminated. The third priority area was partnership, which supported the production and quality of the statistics by improving cooperation within the ESS and with other international organisations and non-EU countries. The detailed structure of the ESP ensured that there was no overlap between the different activities undertaken under the programme. It therefore ensured coherence between the programme’s activities and related objectives.

Drawing up the catalogue of statistical products (priority area 1) was essential for the ESP to be relevant for EU policies and coherent with the Commission’s objectives.

Overall, the activities and objectives set out in the Regulations and internal planning documents were internally coherent, and this was further confirmed by interviewees during the stakeholder consultations.

JC 9.2 Effective processes were in place to monitor and enable the coherence of internal ESP activities

The cornerstone of the quality framework of European statistics was the code of practice 39 , which set the standards for developing, producing and disseminating European statistics. The implementation of the quality framework in the ESS was monitored through peer reviews which covered all the Member States, all the EFTA countries, and Eurostat itself.

In addition, a number of governance bodies provided strategic direction to the ESS and monitored its activities. These bodies included a wide range of relevant stakeholders, meeting in different formats to discuss various issues related to the activities of the ESS. In their meetings and discussions, and in the position papers and reports they published, these governance bodies were able to contribute to the internal coherence of the ESP by ensuring coherence between its activities and its objectives. These bodies were 40 :

·the European Statistical System Committee (ESSC);

·the Directors General of the National Statistical Institutes (DGINS) Conference;

·the Partnership Group;

·the European Statistical Governance Advisory Board (ESGAB);

·the European Statistical Advisory Committee (ESAC).

JC 9.3 Stakeholders involved in producing European statistics considered that the ESP’s activities promoted internal coherence and/or did not negatively impact internal coherence

To confirm the potential for internal coherence in the design of the ESP (see previous JC), stakeholders involved in producing European statistics were asked whether: (i) the objectives of the ESP as set out in the Regulations and programme planning were internally coherent; (ii) the activities of the ESP as set out in the Regulations and programme planning were internally coherent; and (iii) there were effective mechanisms in place to coordinate activities within the ESP. Overall, the largest share of respondents agreed with these statements to a great or moderate extent in relation to the programme’s objectives (85% agreed the objectives were coherent), the programme’s activities (86% agreed they were internally coherent), and the existence of coordination mechanisms (74% agreed they were effective). Noticeably, no respondents disagreed with any of the statements.

Respondents were also asked to provide examples of synergies between the objectives and activities of the ESP. For example, respondents said that strengthening the partnerships within the ESS had enabled exchanges of knowledge and experience that led to the joint development and implementation of new methods. In turn, new methods potentially led to improved timeliness and a wider scope of European statistics.

EQ10: To what extent did ESP activities complement/contradict/overlap with wider EU activity?

JC 10.1 ESP activities and data were actively aligned and/or not in contradiction with overarching EU strategies and objectives

Strong external coherence characterised the ESP throughout the period under evaluation. This was first reflected in the alignment between the ESP and overarching EU political objectives. Both mid-term evaluations of the ESP in 2015 and 2018 concluded that there was a high level of demand for official European statistics, and that the ESP’s objectives corresponded to the EU’s needs. Similarly, the Eurostat strategic plan for 2016-2020 highlighted the importance of Eurostat’s role in helping to achieve the overall objectives of the Commission by providing high-quality statistics. The strategic plan also provided extensive examples of relevant European statistics corresponding to the European Commission’s 10 general objectives/priorities. In addition, in 2015 an overview of the 14 priority areas informing the strategic objectives of the ESP highlighted the correspondence between the Juncker priority areas and the sub-priority areas of the ESP.

Stakeholders at European Commission DGs and EU bodies interviewed as part of the programme assessment supported this conclusion, as did the case studies. They said that the ESP was indeed aligned with the European Commission’s policy priorities. However, the stakeholders and case studies also argued that there was potential for improvement, as ESS’s flexibility and capacity to react to sudden changes and emerging priorities could be enhanced.

JC 10.2 Effective mechanisms were in place to monitor and respond to wider EU strategies and priorities

In addition to the question of alignment with wider EU objectives, external coherence also required a certain level of coordination between the ESP and the work done by other DGs to meet statistical needs. Since 2013, Eurostat has maintained an inventory of the statistical data produced by other DGs and services. Throughout the period evaluated, the updating and validation of this inventory was performed by statistical correspondents every year. This updating and validation: (i) included the statistical data collected by the different DGs; (ii) included plans for future collections of statistical data; and (iii) indicated whether the data could be reused by other DGs. Eurostat checked the completeness of the information provided by the DGs and, based on a conversation with the DGs (in bilateral hearings), it was decided what kind of coordination or cooperation was needed.

In 2017, the Commission’s IAS completed an audit 41 on the production process and the quality of statistics not produced by Eurostat. The overall objective of the audit was to assess whether the Commission had in place an effective process to ensure the quality of statistics not produced by Eurostat (i.e. produced or acquired and used by DGs and services) to support its key policies. The audit concluded that, within the limits of the current framework and the respective responsibilities of those involved, some steps had already been taken both in Eurostat and in other DGs to coordinate and manage the production process of statistics by DGs to improve their quality.

However, the audit also found that the current framework was not robust enough to ensure: (i) that the quality of the statistics produced by other DGs were of a satisfactory standard overall; and (ii) that the various processes currently used were sustainable in the longer term. Therefore, the IAS recommended that Eurostat and the other DGs strengthen the overall production process and improve the production and quality of other statistics. The recommendations for Eurostat concerned in particular: (i) the management of the inventory of statistics produced by other DGs; and (ii) guidance to DGs on methodological and quality issues. Eurostat implemented an action plan to address these recommendations, and the IAS concluded in its follow-up report in 2020 that the recommendations had been adequately and effectively implemented and were therefore closed. During the bilateral hearings between the DGs and Eurostat, DGs were able to ask for additional statistics needed for upcoming EU policy proposals. Eurostat then integrated the information from these hearings into its annual planning processes and was able to collect voluntary data from Member States to address the DGs’ needs. Thus, through its cooperative mechanisms with other DGs, the ESP was aligned with – and able to complement – upcoming and existing EU strategies and priorities. During 2018‑2020, 47 bilateral hearings took place between Eurostat and the other DGs. The ratio between requests approved, requests to investigate, and requests refused was approximately 3:1:1. Eurostat was not able to meet some requests due to confidentiality restrictions, a lack of legal basis, a lack of resources or the expected negative impact of a request on data quality.

Interviews conducted with stakeholders from DGs spoke positively about the usefulness of these hearings. However, as previously highlighted, these stakeholders also raised issues of timeliness and data gaps due to the lack of flexibility or long reaction time of the ESP in responding to emerging priority areas. This lack of flexibility and long reaction time was also raised by Member State stakeholders during interviews.

JC 10.3 EU stakeholders considered that Eurostat coordinated effectively with other EU bodies

Eurostat set up a network of statistical correspondents in the DGs. It also created and coordinated an inventory catalogue on the statistics being produced across the European Commission. Interviewees from all DGs that were sampled highlighted Eurostat’s important coordination work on statistics to ensure that, within the European Commission, there was no duplication of the data produced. Despite this overall positive assessment, respondents to the targeted producer survey felt that Eurostat should have carried out more exchanges with other EU bodies. These respondents felt that more exchanges with other EU bodies were particularly needed to coordinate the ESS when the other DGs prepared non-statistical regulations that required additional data and therefore increased the burden on NSIs. Similarly, producers of statistics interviewed at Member-State level also said that there had been increasing demand for statistics from different DGs, with DGs not always recognising the cost and burden of producing statistics. This would suggest there is potential for Eurostat to take on an increased role in coordinating data requests among EU bodies.

The producers of European statistics interviewed considered Eurostat’s coordination with Member States and other EU bodies to be very effective. Similarly, most respondents to the targeted producer survey (to which 43 responded) agreed – to a moderate extent (37%) and to a great extent (21%) – that there were effective mechanisms in place to coordinate with other EU bodies.

Beyond the European Commission, Eurostat also cooperated and coordinated with other EU bodies and agencies. For instance, in the area of asylum and managed-migration statistics, Eurostat cooperated with the European Asylum Support Office and the European Border and Coast Guard Agency (Frontex). This cooperation focused on developing a common methodology and classification methods to harmonise European statistics and reduce the burden of statistical production on Member States. Stakeholders interviewed were very positive about both the level and the nature of the cooperation. Similarly, Eurostat also cooperated with other EU institutions such as the European Committee of the Regions and the European Central Bank (ECB). The ESS and the European System of Central Banks (ESCB) also cooperated closely through the Committee on Monetary, Financial and Balance of Payments Statistics (CMFB) 42 , as well as through the European Statistical Forum (ESF).

Overall, the findings suggested that EU stakeholders considered that Eurostat coordinated effectively with other EU bodies. However, the issue of additional burden on Member States was raised as a weakness due to the DGs’ individual requests for statistics.

EQ11: To what extent were ESP activities coherent with the activities of international statistics organisations?

JC 11.1 Effective mechanisms were in place to coordinate with international partners

Eurostat participated in task forces, expert groups, and decision-making bodies of international organisations. It participated in these through both bilateral (e.g. the UNSD and United Nations Economic Commission for Europe (UNECE)) and multilateral settings (e.g. UN Statistical Commission). Eurostat also sponsored initiatives such as the initiative on statistical data and metadata exchange (SDMX) which develops standards for the exchange of statistical information. Eurostat’s cooperation strategy with international partners was outlined in its document on international cooperation strategy, which provided thematic and regional direction and outlined the mechanisms in place to coordinate with international partners. At working level, units within Eurostat also cooperated with units in relevant international organisations.

Eurostat and its international partners cooperated to align their methodologies, agree on definitions for individual statistical indicators, and validate and exchange data. The aim of this cooperation was to reduce the burden on Member States and to ensure the comparability of the statistics produced. Eurostat currently has MoUs signed with 4 organisations (the OECD, the ECB, the UN-ICSC and the ISRP) and administrative arrangements with 4 organisations (EFTA, the UNSD, the FAO and the OECD).

In 2019, the IAS issued an audit report 43 on the effectiveness of Eurostat’s cooperation with external stakeholders. In that report, the IAS concluded that Eurostat had set up effective cooperation arrangements with a number of external stakeholders. However, the IAS also said that the management and control systems in place lacked a clear overall policy at DG-level, and that Eurostat needed to more effectively coordinate activities and the exchange of information within Eurostat. The IAS also made three recommendations on: (i) further develop and formalise cooperation arrangements with Eurostat’s external stakeholders; (ii) clearly define internal roles, responsibilities and information sharing; and (iii) improve the performance-management framework. Eurostat implemented an action plan to address these recommendations and the IAS concluded in its follow-up that the recommendations had been adequately and effectively implemented and were therefore closed.

In addition to the desk review, Eurostat’s external coherence with international partners was also investigated through the targeted producer survey. In the targeted producer survey, participants were asked to rate: (i) the extent to which they agreed that effective mechanisms were in place to coordinate with international partners; and (ii) the extent to which Eurostat activities complemented the statistics-production and statistics-dissemination activity of international partners. Respondents (of which there were 43) agreed to a moderate (42%) or great (30%) extent that there were effective mechanisms in place to coordinate with international partners. A similar share agreed on the complementarity between Eurostat’s activities and international partners’ statistics production and dissemination activities (39% agreed to a moderate extent and 30% agreed to a great extent).

Similarly, producers of European statistics that were interviewed considered Eurostat’s coordination with international partners to be generally coherent and effective. They also said that this coordination had improved in recent years. Although there were still some overlaps between the requirements of Eurostat and international organisations, producers said that there were many initiatives aiming to prevent the duplication of work, in particular between Eurostat the OECD and the UNSD.

JC 11.2 International partners considered that Eurostat activities complemented and/or were not in contradiction with their own statistics-production and statistics-dissemination activities

International partners that were interviewed found the programme to be externally coherent but were generally unable to assess these questions in detail.

Interviews conducted with international organisations in case studies provided similar positive feedback. In the area of migration, the UNSD highlighted its very long-standing and good cooperation with Eurostat which: (i) involved the exchange of data and staff; and (ii) contributed to the overall external coherence of their activities by ensuring their complementarity.

Similarly, for statistics on education, health, and transport, Eurostat, together with the OECD, WHO and UNESCO developed a system for joint data collections for statistical indicators. This enabled Eurostat and its international partners to: (i) optimise the collection of data from the Member States; (ii) apply common concepts and definitions; and (iii) avoid duplication of efforts. Eurostat was also seen as a valued member by international organisations in international forums, where its collaborative attitude and activities received positive feedback from interviewees. In addition, the statistics produced by Eurostat were seen as reliable and were widely used to complement and even validate the statistics collected by other organisations. This is further evidence of the complementarity between the activities and statistics of Eurostat and its international partners.

5.5 EU added value

The EU added value criterion looks at the value resulting from the ESP that is additional to the value that would have resulted from similar activities carried out only at national level or by individual Member States. The EU added value criterion is linked to the principle of subsidiarity, which states that the EU should only act when it can achieve better results than the Member States acting alone. The analysis of the criterion focused on the added value of the ESP to both users and producers of European statistics.

The evidence collected confirmed the EU added value of the ESP. It suggested that the programme’s main added value was: (i) its contribution to harmonising European statistics so they deliver comparable statistics; (ii) providing these statistics in a single location; and (iii) supporting evidence-based policy making at EU and national level. The ESP produced high-quality, comprehensive, comparable and reliable statistics across the Member States.

In the absence of the ESP, evidence-based policy making at EU level and policy making at country level (based on comparative analysis of statistics across countries) would be very difficult (due to potential inconsistencies in the evidence).

Users from EU institutions, Member States, and candidate countries confirmed the EU added value of providing European statistics in a single online location (a ‘one-stop shop’).

The programme also drew up a clear roadmap for statistics production for producers of statistics, including in candidate countries. The ESP also helped to strengthen the international statistical community and contributed to efforts to ensure the quality of statistics at international level through cooperation between Eurostat and international organisations.

EQ12: What was the EU added value of the ESP?

JC 12.1 Users of statistics considered the ESP to be the preferred source of information when they compared indicators across Member States

The findings suggested that the comparability of statistics across Member States was one of the key reasons for using European statistics, and that no other source could provide these statistics in a similar manner and scale.

Users from European Commission DGs highlighted that the ESP made statistics available and contributed to the transparency objective on a wide range of issues. Without the ESP, users said that it would not have been possible to have comprehensive, high-quality and reliable statistics across Member States to support the European Commission’s policy making objectives. Even when users said that they also used other statistics, they emphasised that European statistics remained their preferred source of information because the scope of the statistics matched exactly the Commission’s (strategic) objectives and covered all Member States (as the primary constituency of EU policy making).

Individual users at Member-State level agreed with the statement of this JC. They indicated that there was clear EU added value in Eurostat’s work, and that this mainly related to ensuring the comparability of statistics from the Member States and providing these statistics in a single online location. Producers of statistics also shared this opinion. However, more advanced users pointed out that there was still more to do to further improve comparability, mainly by harmonising methods of collecting data across Member States.

Eurostat is also considered a trusted source of information, as confirmed by the USS. For instance, in 2020, as in previous years, responses were overwhelmingly positive, with 96% of users stating that they trusted European statistics greatly or tended to trust them.

JC 12.2 Individual and institutional users’ stated preference for European statistics over other international statistics/datasets

Users from the EU institutional level confirmed their preference for European statistics over other international statistics/datasets. There was a consensus that European statistics were the most relevant and of the highest quality for EU policy-making needs. Users from the Member States provided slightly different feedback. European statistics (and national statistics from their respective countries) were the main source of statistical information for these Member State users, but they indicated that they also used other international sources for other purposes, including comparisons with non-EU countries.

Most respondents to the targeted survey for users (77% of 45 responses) and to the PC (62% of 424 responses) indicated that they also used other international sources of statistics. Through the online consultation activities, respondents assessed the quality of European statistics in comparison to other international sources according to five principles. As shown in Figure 8 below, almost half of the respondents reported that the quality of European statistics and other international sources were similar. The proportion of respondents saying that the quality of European statistics was higher or much higher than that of other data sources ranged from 29% to 51% across the five principles.

Figure 8: In your opinion, how does the quality of European statistics compare to other international sources of data that you use

Source: Targeted user survey (45 responses received)

In the PC (to which there were 424 responses), respondents provided slightly more positive feedback, with the proportion judging the quality of European statistics higher or much higher than other international sources of data ranging from 34% to 50% across the five principles.

The analysis suggested that there was complementarity rather than competition between the ESP and other international sources.

JC 12.3 There was a recognised advantage of having a ‘one-stop shop’ for harmonised and validated European statistics

This JC complements the first JC related to the ESP as the preferred source of information for comparisons across Member States. Throughout the period evaluated, the main perceived advantage of the ESP was to enable cross-country comparative analyses, which the harmonisation and comparability of statistics made possible.

In the targeted producer survey (to which there were 43 responses), most producers agreed that the ESP contributed to a great (63% agreed) or moderate (26% agreed) extent to the increased comparability of national statistics.

Feedback from the European Commission DGs suggested that there was EU added value in having a ‘one-stop shop’ for harmonised and validated European statistics. The ESP was a central mechanism for all things related to statistics in the EU. It therefore decreased transaction costs for its stakeholders while making available all the necessary data and tools.

Finally, the existence of a ‘one-stop shop’ was also important for candidate countries. For this group of countries, increasing comparability and harmonisation with European statistics had been one of the main goals in their strategies. Collaboration through the ESP was instrumental in modernising the statistical systems of these candidate countries.

JC 12.4 What would happen in the absence of the ESP?

The EU added value of the programme is shown by the harmonised provision of comparable and high-quality data for EU countries. The ESP as a harmonised system with common quality standards for producing statistics is unparalleled in the world. A similar level of harmonisation, comparability and quality could not be achieved at Member-State level alone to make essential contributions to EU activities. Only a coordinated approach to developing, producing and disseminating European statistics, as ensured through the programme, could guarantee the required coherence and comparability of the statistics relevant for EU activities. At the same time, the programme’s financial support to Member States helped it to drive development, modernisation and innovation in the ESS. The programme has succeeded in implementing modernisation projects that are beneficial to all Member States, which could not have been achieved through uncoordinated national spending.

In the absence of the ESP, evidence-based policy making at EU level and country level, based on the comparative analysis of statistics across countries, would be very difficult (due to potential inconsistencies in the evidence). EU statistical production requires the implementation of a harmonised methodology and the definition of common outputs, which can only be fully achieved by EU action. Delivering the same results through Member States’ individual action, without the overarching and coordinating role of the ESP and Eurostat, would have been extremely challenging, if not impossible.

Producers also highlighted the usefulness of Eurostat’s grant scheme. In the absence of the ESP, access to these resources would not be possible. NSIs would not have access to methodological support or financial support for projects from Eurostat. Sharing best practices and mutual learning would be more challenging without the coordinating role of Eurostat.

The thematic case studies confirmed these findings. Recently, the actions at EU level supported the response of the statistical community to the COVID-19 pandemic by providing methodological support to statisticians and creating a European ‘dashboard’ with recovery indicators. New methodological guidelines were issued which aimed to ensure the consistency of data-collection processes related to reporting on the pandemic.

Lastly, and from a forward-looking perspective, new challenges have come with new data sources, like privately held data. These data sources must be correctly framed to ensure quality, reliability and the protection of personal data. This requires setting up the proper legal framework and addressing any possible legal barriers. Stakeholders expected that this would be more effectively done at EU level than at national level, especially for sources which were in essence pan-European.

5.6 Evaluation of the points raised in Article 15 of the ESP

The answers to the four specific points referred to in Article 15 of the ESP Regulation are based on the evidence collected and the analysis conducted for the five evaluation criteria. No additional EQs had to be drawn up for the first three specific topics. The fourth topic is based on an ad hoc sub-question, which can be linked to the effectiveness criterion. Therefore, it is treated separately in this section. All the answers, with a summary of the evidence supporting them, are given in this section.

5.6.1 The outcome of the reprioritisation and cost evaluation of statistical products

Despite improvements, prioritisation remained a challenge throughout the programme. This resulted in increased pressure on Member-State resources. This was due in large part to the increase in data needs expressed by users of statistics and the difficulties in identifying statistics that have become less relevant over time. To solve the issue, a review of statistical requirements in existing domains of European statistics and a new prioritisation mechanism are being implemented in the new statistical programme for 2021-2027. The new mechanism is based on four elements: a multiannual action plan, reviews, user dialogue, and resources/funding.

The assessment of costs for the ESS showed that the total costs of producing statistics decreased slightly between 2013 and 2020, as did the number of FTEs – although producers met an increasing number of statistics requests.

The detailed information leading to the conclusions above can be found in Section 5.3 on efficiency, in the analysis of EQs 6 and 7 and in particular in JC 6.1, 6.2, 6.4, 6.5, 7.1 and 7.2.

5.6.2 Actions taken by the ESS to reduce the production costs for Member States and to limit the overall burden stemming from the statistical projects and fields covered by the programme

The quantitative data show that both staff numbers and the cost of producing statistics fell while the number of datasets being produced increased. This is evidence that the actions taken by the ESS were effective in reducing production costs for Member States and limiting the overall burden stemming from the statistical projects and fields covered by the programme. Among the most important actions taken were: (i) the implementation of framework statistics regulations; (ii) increased use of administrative and register data; (iii) increased use of estimation methods; and (iv) the application of microdata exchanges.

However, qualitative feedback provided a more nuanced perspective, highlighting limitations in both priority setting and the identification of statistics which have become less relevant over time. In addition, concerns about shrinking budgets and staff numbers, coupled with the reported increase in requests for new statistics, suggested that issues of production cost and burden may emerge in the future.

The detailed information leading to the conclusions above can be found in Section 5.3 on efficiency, in the analysis of EQs 6 and 7, and in particular in JC 6.4, 6.5, 7.1 and 7.2.

5.6.3 Progress on rendering access to official statistics easier and more user-friendly, including the provision of data on the Eurostat website

The ESP helped make access to official statistics easier and more user-friendly, but there was still scope for further improvement.

Eurostat’s website was completely redeveloped at the beginning of the programme. The new website was launched in December 2014 and it has been continuously improved since, with another new update that will go live soon. Since the beginning of the ESP, Eurostat has developed and added many new electronic and interactive publications, visualisation tools, mobile apps and tools for data extraction. These tools present data from different statistical themes in an attractive and easy-to-understand way for everyone to explore and can be easily accessed on Eurostat’s website .

Adding modern ways of communication, Eurostat has also been active on social media, with its three corporate social media accounts on Twitter , Facebook and Instagram .

The provision of data on the Eurostat website was prioritised during the redevelopment of the website. However, opinions about the user-friendliness and ease of access to these data differed depending on the category of users. Although data and statistics were easily accessible to proficient users from the Commission, access was more challenging for less experienced users from the general public. Progress was made in making statistical data easy to retrieve and convert for practical use, including through graphs and maps. However, the main weakness of the Eurostat website was its complexity, which led to difficulties in finding the right data and statistics. Nevertheless, dedicated Eurostat publications provided data in a clear and accessible manner.

The detailed information leading to the conclusions above can be found in Section 5.2 on effectiveness, in the analysis of EQ 4 and in particular in JC 4.2.

5.6.4 Progress on improving data availability, including social-economy activities, and on the Europe 2020 indicators

From 2013 to 2020, the ESP included activities to develop new statistics and contributed to improving data availability, including on both social-economy activities and the Europe 2020 indicators. On social-economy activities, data and statistics became more available in the areas of population, social conditions and migration. Eurostat also increased the disaggregation of its data by migratory status. Despite these improvements, gaps remained in the territorial disaggregation of social statistics. On the Europe 2020 indicators, Eurostat continuously provided comparable, reliable, and timely statistical information. Eurostat also continued its work to develop and produce these indicators throughout the programme. Europe 2020 indicators are only available at the EU and country levels, because there is no territorial disaggregation of the targets that have been set. Nevertheless, some of the data used for these indicators are sometimes available at regional level. The set of Europe 2020 indicators – used to monitor the Europe 2020 strategy for growth and jobs for 2010-2020 – had been updated yearly and was available until 2021. The Europe 2020 strategy has now reached the end of its life cycle. Moreover, the EU aggregate (28 countries including the United Kingdom), for which the Europe 2020 targets were set, no longer exists in official statistics. For these two reasons, the Europe 2020 indicators were removed from the Eurostat reference database and archived. However, all five areas covered by the Europe 2020 indicators remain part of the EU indicator set on the SDGs and will continue to be updated under the heading of ‘ sustainable development indicators ’.

The findings suggest that ESP activities resulted in an improvement of data availability for all themes. Users and producers of European statistics that were interviewed confirmed this point. They assessed data availability positively and noticed an improvement in that respect.

Social-economy activities

43% of the respondents to the PC (to which there were 424 responses) specifically reported an improvement in data availability on population and social conditions. Stakeholders interviewed as part of the asylum and managed-migration case study further corroborated this. These stakeholders were satisfied with the improvement in data availability on social (migration-related) activities and the increased disaggregation of data by migratory status.

Nevertheless, stakeholders interviewed as part of the territorial case study said that, despite improvements in the availability of data that were disaggregated by territory, there remained gaps in social statistics.

Europe 2020 indicators

An audit was carried out in 2015 on the support given by Eurostat to the Europe 2020 strategy and the new Commission priorities. The audit aimed to assess whether Eurostat had put in place an efficient and effective process to provide up-to-date statistical data, including in the areas covered by the Europe 2020 strategy, to help to monitor progress towards the related targets 44 . The audit’s main recommendation was on other statistics (not Eurostat’s) used by the DGs to demonstrate progress made in achieving the Europe 2020 targets. This suggests that progress on improving data availability on the Europe 2020 indicators was not an issue.

The 2013-2019 AARs all indicated that Eurostat continuously provided comparable, reliable, and timely statistical information supporting the monitoring of progress towards an evidence-based implementation of the Europe 2020 strategy 45 . In the same period, these data were also available in the flagship publication Smarter, greener, more inclusive? Indicators to support the Europe 2020 strategy 46 . Throughout the ESP, Eurostat made progress on improving data availability as work continued on developing and producing high-quality indicators for Europe 2020. Some of these improvements in data availability are set out in the bullet points below.

·In 2014, major achievements concerned, among other things, the further development of the Europe 2020 indicators for education and training.

·In 2015, Eurostat accomplished major achievements on the Europe 2020 targets in the area of sectoral and regional statistics. Eurostat also made available timely Europe 2020 indicators on education and life-long learning.

·In 2016, Eurostat updated the headline target indicators for Europe 2020 on sectoral statistics with new data and disseminated them through its online database. The flexible dashboard and other interactive visual tools in the section on Eurostat’s website dedicated to the Europe 2020 targets also included these new data.

·In 2019, the provision of additional breakdowns and the revision of the methodology and compilation techniques further improved the quality of the Europe 2020 indicators: (i) a new child deprivation indicator was produced; (ii) revised inequality indicators were approved; (iii) the ‘low work intensity’ indicator was revised; (iv) the timeliness of social data was improved; and (v) a news release was published on the early results of indicators on material deprivation.

The case studies carried out as part of this evaluation confirmed the data availability of the Europe 2020 indicators. Stakeholders interviewed on business statistics stated that Eurostat provided extensive indicator sets to contribute to assessing the Europe 2020 targets. Asylum and managed-migration data also included migrant-integration statistics, which were part of the regional labour-market indicator. These statistics were broken down by region and degree of urbanisation and looked at activity rate, employment and unemployment rate. They supported monitoring of the headline target indicator on the employment rate for Europe 2020. Eurostat regularly published some of the data used for Europe 2020 indicators at regional level in its regional yearbook. Statistics with a territorial dimension supported a wide range of policy measures set out in the Europe 2020 strategy, such as urban renewal, increasing education opportunities, crime prevention, social inclusion, or environmental protection 47 . At the same time, the composition of some indicators made it difficult to make them available at regional level (for example, indicators focusing on narrow age groups or measuring the consumption of renewable energy at the regional or territorial level are not feasible). 

6.Conclusions

The European statistical programme 2013-2020 was created by Regulation (EU) 99/2013 48 for the years 2013-2017. It was extended by Regulation (EU) 2017/1951 49 to cover the remaining period of the previous EU multiannual financial framework, i.e. 2018-2020.

The general objective of the ESP was for the ESS to continue to be the leading provider of high-quality statistics on Europe. As a spending programme, the ESP constituted the overall framework for developing, producing and disseminating European statistics. The ESP therefore provided the financing for developing and maintaining Eurostat’s statistical infrastructure and the ESS as managed by Eurostat. The ESP also provided financial support to Member States.

The requirement to carry out a final evaluation and present a report on the ESP is enshrined in Article 15 of the amended Regulation, which also mentions four specific aspects to evaluate. This evaluation also serves to fulfil the requirements of the Commission’s push for better regulation and so it has to follow the better regulation guidelines for evaluations . The evaluation therefore covers the five compulsory evaluation criteria (effectiveness, efficiency, relevance, coherence and EU added value).

Two previous mid-term evaluations of the ESP4;5 covered the implementation of the programme in 2013-2017, while a progress report9 assessed the progress made in 2018 and the first half of 2019. The present evaluation takes into account the findings and conclusions of those other evaluations, but covers the implementation period of the programme in its entirety (2013-2020).

A contractor, Tetra Tech International Development, was selected (via an open call for tenders) to support the evaluation process. Having an independent opinion on the work carried out by Eurostat was deemed very important. The contractor’s work included: (i) designing the evaluation; (ii) preparing and executing the stakeholder consultation; and (iii) writing an evaluation report. The contractors’ report was used as a base to prepare the SWD.

The COVID-19 crisis prevented the contractor’s team from collecting data face-to-face. All activities took place remotely instead. However, this did not influence the quality of the work as the overall the level of stakeholder engagement achieved was high.

An ISG was set up for the project, including members from all directorates of Eurostat and from many other DGs. The ISG followed and steered the entire evaluation.

6.1.Relevance

The ESP provided the framework for developing, producing, and disseminating European statistics, setting out the objectives of the activities planned for 2013-2020. It financed the development and maintenance of both Eurostat’s statistical infrastructure and the ESS. The objective of the ESP was therefore to both: (i) fulfil the needs of Eurostat, NSIs and ONAs (as producers of European statistics); and (ii) serve the needs of a wide range of users of these statistics.

The replies to the EQs indicate that the ESP was appropriately designed to satisfy the stakeholders’ needs. The replies also indicate that, throughout its entire duration, the ESP’s activities were carried out to continue to satisfy these changing needs. At the same time, although the ESP made it possible to implement appropriate activities to meet its objectives, the analysis showed that these activities were not enough to deliver all the statistics that users would wish. However, some of the thematic areas that users identified as missing go beyond the scope of the ESP – so non-delivery in these areas cannot be considered as failures in the implementation of the programme.

The high relevance of the ESP is clearly demonstrated by the number of data extractions made by external users from Eurostat reference databases, which greatly increased since 2014 from 6 813 000 extractions to 21 480 000 in 2020 (+315%). That is further confirmed by the increase in the number of users accessing the Eurostat website, estimated by the number of distinct IP addresses visiting the website, which went from 3 441 157 visits in 2014 to 11 731 274 in 2020 (+240%).

During the lifetime of the programme, Eurostat increased its presence on social media and its numbers of followers grew constantly. Eurostat opened its Facebook account on 10 January 2017 and its Instagram account on 4 May 2020. Eurostat’s number of followers on Twitter grew from 25 000 in 2013 to 164 000 in 2020, by which time it had reached 62 000 followers on Facebook and 9 500 on Instagram.

There was some tension between continuously increasing user needs and limitations faced by producers in keeping up with new demands under an already ambitious programme. This meant that a balance needed to be struck between producer capacity and user requests. In addition, needs related to adaptation to technological advances such as big data became more prominent during the programme, as reflected in the extension of the ESP. These increasingly prominent needs were translated into specific activities, mainly as part of the ESS Vision 2020 implementation projects. Similar activities will be continued in the next programme, the results of which have the potential to provide solutions to some of the problems the ESS still faces in terms of resources and costs.

6.2.Effectiveness

The evaluation findings show that the ESP was effective in delivering on its objectives despite remaining weaknesses in the timeliness and completeness of European statistics. The percentage of planned outputs that were achieved under the ESP objectives 1 and 4 for the production of statistics was constantly above 92%. They were constantly above 95% in the second half of the programme, with only 3% or fewer of outputs cancelled or not fully achieved.

Between 2013 and 2020, Eurostat provided high-quality statistics which were used by a wide range of users for a wide range of purposes. These purposes include supporting the development, monitoring and evaluation of policies at the EU and Member-State level. During that period, the ESP also increased the availability of its data and statistics and, to some extent, introduced efficiency gains in the production of its statistics. The quantity of statistics and long-time series 50 published by Eurostat increased from 328 million and 35 million respectively to 446 million and 55 million.

Several projects were launched in response to new challenges faced by European statistics. These challenges included implementing the ESS Vision 2020, strengthening the European statistical infrastructure, and making better use of big data and shared infrastructure.

Eurostat also effectively developed and strengthened partnerships with its members within the ESS as well as with partners beyond the ESS – in particular international organisations. However, more needs to be done, in particular to: (i) get access to privately held data; and (ii) create partnerships with private organisations. Work towards both of these goals started in 2020 and is being addressed in the next programme.

Beyond the achievements of its objectives, the ESP was also effective in making access to official statistics easier and more user-friendly between 2013 and 2020.

6.3.Efficiency

Under the efficiency criterion, the evaluation focused on: (i) the costs of producing European statistics; (ii) the trends in these costs; and (iii) the burden of producing these statistics faced by the NSIs and ONAs. Three general issues were assessed to find out whether the ESP was being run efficiently. The first issue related to whether the programme ensured the best use of available financial and human resources. The second concentrated on the costs and burden involved in producing European statistics in the ESS and how these costs and burdens have developed. The third issue was to assess how efficient the ESP process was for reporting and monitoring.

The analysis of the evidence suggests that the programme has been efficient. The ESP demonstrated efficient use of both financial and human resources while delivering high‑quality European statistics. Moreover, the productivity of statistics production increased during the period evaluated. Staff numbers at Eurostat decreased by 7% (52 persons) in FTEs during the implementation of the ESP. In the same period, the number of annual datasets published by Eurostat increased by 18%. This means an increase in statistics productivity of Eurostat staff by more than 20% during this time. Staff numbers also fell across the ESS as a whole, going from 49 200 FTEs in 2013 to 42 500 in 2020. Factors underpinning these developments in productivity were good governance, good management, and effective monitoring mechanisms.

The IAS carried out an audit on Eurostat’s quality management of statistical processes. The audit found aspects to improve, and the IAS made two recommendations. Eurostat implemented an action plan to address these recommendations. The IAS concluded in its follow-up on 18 January 2021 that the first recommendation had been adequately and effectively implemented and was therefore closed. The second recommendation will be implemented by 31 December 2021as set out in the action plan.

There were concerns about the costs for Member States and the administrative burden placed on data providers. These concerns arose because of the need to produce an increasing volume of statistics, which made it necessary to increase the efficiency of statistics production. Several initiatives were undertaken to respond to this challenge, including: (i) the activities within the ESS Vision 2020; (ii) an initiative to modernise business statistics resulting in the European Business Statistics Regulation; (iii) an initiative to improve social statistics with the Integrated European Social Statistics Regulation; (iv) an initiative to improve agriculture statistics with the Integrated Farm Statistics Regulation; and (v) work to strengthen partnerships within the ESS.

6.4.Coherence

The evaluation examined both the internal and external coherence of the ESP and the findings suggested that the ESP was internally and externally coherent.

No overlaps or inconsistencies were identified at the programme level – either between the objectives or among the activities listed within the regulations and planning documents of the ESP (internal coherence). The evaluation identified various governance bodies and advisory boards that had a coherent mandate. The work of these bodies and boards contributed to the internal coherence of the ESP by ensuring that its activities were in line with its objectives. Furthermore, interviewees involved in producing European statistics highlighted the internal coherence of the ESP.

The work performed under the ESP was in agreement with the priorities of the Commission. Relevant European statistics corresponded to the European Commission’s 10 general objectives/priorities, and the sub-priority areas of the ESP were in line with the priority areas of the Juncker Commission.

Eurostat effectively cooperated with EU bodies and agencies, as well as international organisations. This cooperation resulted in the external coherence of the ESP by ensuring synergies with: (i) wider EU objectives and needs; and (ii) international statistical activities. This cooperation also ensured the development of comparable and harmonised European statistics at the regional and international level.

The IAS carried out two audits, one on the production process and the quality of statistics not produced by Eurostat, and one on the effectiveness of Eurostat’s cooperation with external stakeholders. Both audits found aspects to improve, and the IAS therefore made some recommendations. Eurostat implemented an action plan to address both sets of recommendations, and the IAS concluded in their follow-ups that the recommendations had been adequately and effectively implemented and were therefore closed.

However, some weaknesses which could potentially affect the external coherence of the ESP were identified in the evaluation, namely: (i) the lack of flexibility of European statistics to respond to emerging needs (because this lack of flexibility might cause a misalignment with other EU strategies should the needs not be covered ); (ii) the burden on Member States when replying to individual DG requests for statistics; and (iii) the necessity of getting access to privately held data and creating partnerships with private organisations (for which activities began in 2020). Eurostat has begun to tackle these weaknesses and they will continue to be addressed in the next programme.

6.5.EU added value

The evidence collected confirmed the EU added value of the ESP. It suggested that the programme’s main added value was: (i) its contribution to harmonising European statistics so Member States deliver comparable statistics; (ii) providing these statistics in a single location; and (iii) supporting evidence-based policy making at EU and national level. The ESP produced high-quality, comprehensive, comparable and reliable statistics across the Member States.

In the absence of the ESP, evidence-based policy making at EU level and policy making at country level (based on comparative analysis of statistics across countries) would be very difficult (due to potential inconsistencies in the evidence).

Users from EU institutions, Member States and candidate countries confirmed the EU added value of providing European statistics in a single online location (a ‘one-stop shop’).

The programme also drew up a clear roadmap for statistics production for producers of statistics, including in candidate countries. Through cooperation between Eurostat and international organisations, the ESP also helped to strengthen the international statistical community and helped efforts to ensure the quality of statistics at the international level.

6.6.Specific points of Article 15

The conclusions of the evaluation for the four specific aspects mentioned in Article 15 of the amended ESP Regulation are summarised here below.

a)The outcome of the reprioritisation and cost evaluation of statistical products

Despite improvements, prioritisation remained a challenge throughout the programme. This resulted in increased pressure on Member State resources. This was due in large part to the increase in data needs expressed by users of statistics and the difficulties in identifying statistics that have become less relevant over time. To solve the issue, a review of statistical requirements in existing domains of European statistics and a new prioritisation mechanism are being implemented in the statistical programme for 2021-2027 set out by the SMP. The new mechanism is based on four elements: a multiannual action plan, reviews, user dialogue, and resources/funding.

As detailed in b) below, the assessment of costs for the ESS showed that the total costs of producing statistics decreased slightly between 2013 and 2020 as did the number of FTEs – although producers faced an increasing number of requests for statistics.

b)Actions taken by the ESS to reduce implementation and production costs for Member States and to limit the overall burden stemming from the statistical projects and fields covered by the programme

Staff numbers and the cost of producing statistics fell while the number of datasets being produced increased. This is evidence that the actions taken by the ESS were effective in reducing production costs for Member States and limiting the overall burden stemming from the statistical projects and fields covered by the programme. Among the most important actions taken were: (i) the implementation of framework statistics regulations; (ii) increased use of administrative and register data; (iii) increased use of estimation methods; and (iv) the application of microdata exchanges.

However, qualitative feedback highlighted limitations in priority setting and the identification of statistics which have become less relevant over time. In addition, concerns about both falling staff numbers and the increase in data needs expressed by users seeking new statistics suggested that issues of production cost and burden may emerge in the future.

c)Progress on rendering access to official statistics easier and more user-friendly, including the provision of data on the Eurostat website

The ESP helped make progress on making access to official statistics easier and more user-friendly, but there was still scope for further improvement.

Eurostat’s website was completely redeveloped in 2014 and has been continuously improved since. Since the beginning of the ESP, Eurostat has developed and added many new electronic and interactive publications, visualisation tools, mobile apps and tools for data extraction. These tools present data from different statistical themes in an attractive and easy-to-understand way for everyone to explore and can be easily accessed on Eurostat’s website .

Adding modern ways of communication, Eurostat has also been active on social media, with its three corporate social media accounts on Twitter, Facebook and Instagram.

The provision of data on the Eurostat website was prioritised during the redevelopment of the website. However, different users gave differing views on how easy and user-friendly it was to access these data. Although data and statistics were easily accessible to proficient users from the Commission, access was more challenging for less experienced users from the general public, due to the volume and complexity of the website’s content.

d)Progress on the improvement of data availability, including on social-economy activities and on the Europe 2020 indicators

From 2013 to 2020, the ESP included activities to develop new statistics and contributed to improving data availability, including on social-economy activities and the Europe 2020 indicators. On social-economy activities, more data and statistics became available on population, social conditions, and migration. Eurostat also increased the disaggregation of its data by migratory status. Despite these improvements, gaps remained in the territorial disaggregation of social statistics. On the Europe 2020 indicators, Eurostat continuously provided comparable, reliable, and timely statistical information. Eurostat also continued its work to develop and produce these indicators throughout the programme. The Europe 2020 indicators are only available at the EU and country levels because there is no territorial disaggregation of the targets that have been set. Nevertheless, some of the data used for these indicators are available at regional level. The set of Europe 2020 indicators – used to monitor the Europe 2020 strategy for growth and jobs for 2010-2020 – was updated yearly and was available for reference periods until 2021. The Europe 2020 strategy has now reached the end of its life cycle. In addition, the EU aggregate (28 countries including the United Kingdom), for which the Europe 2020 targets were set, no longer exists in official statistics. For these two reasons, the Europe 2020 indicators were removed from the Eurostat reference database and archived. However, all five areas covered by the Europe 2020 indicators remain part of the EU indicator set on the SDGs and will continue to be updated under the heading of ‘ sustainable development indicators ’.

Annex 1: Procedural information

1.Lead DG, Decide Planning/CWP references

This evaluation staff working document was prepared by the evaluation team in Unit A2 ‘Legal affairs; Document management’ of Eurostat.

The purpose of the evaluation was to assess the implementation of the ESP (Decide planning reference: PLAN/2020/8378).

2.Organisation and timing

The work on the evaluation started in the 3rd quarter of 2020. As the main objective of the ESP is the production of high-quality statistics on Europe, which serve the needs of many DGs, the work was supervised by a large ISG. It included representatives from the following DGs:

·Eurostat

·Joint Research Centre

·Legal Service

·Secretary-General

·DG Agriculture and Rural Development

·DG Budget

·DG Climate Action

·DG Communication

·DG Communication Networks, Content and Technology

·DG Competition

·DG Economic and Financial Affairs

·DG Education and Culture

·DG Employment, Social Affairs and Inclusion

·DG Energy

·DG Environment

·DG Financial Stability, Financial Services and Capital Markets Union

·DG Health and Food Safety

·DG Internal Market, Industry, Entrepreneurship and SMEs

·DG International Partnerships

·DG Justice and Consumers

·DG Maritime Affairs and Fisheries

·DG Migration and Home Affairs

·DG Mobility and Transport

·DG Neighbourhood and Enlargement Negotiations

·DG Regional and Urban Policy

·DG Research and Innovation

·DG Taxation and Customs Union

·DG Trade

Meetings of the ISG were held on:

21/09/2020, 10/11/2020, 06/05/2021, 06/07/2021, 01/09/2021.

3.Exceptions to the better regulation guidelines

There were no exceptions to the better regulation Guidelines.

4.Evidence, sources and quality

The list of sources used to perform the evaluation can be found in the contractor’s final report (Annex 6 of the contractor’s final report; Annex 4 to this document) for the desktop review and in the contractor’s synopsis report (Annexes 3, 4 and 5 of the contractor’s synopsis report) for the stakeholder consultation.

5.External expertise

A contractor, Tetra Tech International Development, was charged with writing an independent evaluation report, to support the preparation of the Commission evaluation staff working document.

The contract started on 10 September 2020 and the consultant delivered its final report on 23 July 2021.

The final report is annexed to this document.

Annex 2: Stakeholder consultation

1.Introduction

This stakeholder consultation was conducted by an external contractor, Tetra Tech International Development, in the context of the final evaluation of the European Statistical Programme 2013-2020. The report has two main parts: one section, Section 2, describing the stakeholder consultation strategy, and one section, Section 3, presenting the overarching analysis of the different consultation activities. Further information can be found in the synopsis report of the contractor (contract deliverable D4).

2.Consultation strategy

2.1.Consultation objectives

The contractor prepared and carried out several consultation activities designed to support the ongoing evaluation of the European Statistical Programme (‘the ESP’, also ‘the programme’ thereafter). The objective was two-fold: to gather stakeholders’ views with respect to a range of evaluation criteria and then to compare and integrate the evidence collected through other data collection activities and to gather primary evidence to address any data gaps identified following the desk review.

2.2.Identification of stakeholders

As part of the consultation strategy, the contractor conducted a stakeholder mapping exercise to identify key ESP stakeholders, both producers and users of European statistics (as presented in the Inception Report and then revised in the D2 Stakeholder Consultation Report). In line with the Better Regulation guidelines, the mapping exercise involved two distinct activities:

1.Identification of stakeholder categories relevant for or interested in the ESP;

2.Prioritisation of stakeholder categories to engage with in line with their level of interest in or influence on the ESP.

Based on the mapping exercise, the contractor designed a stakeholder consultation plan which then formed the backbone of all stakeholder consultation activities. The activities included:

I.Stakeholder interviews, including interviews feeding into the overall programme evaluation, as well as interviews in the framework of Member States and country case studies

II.A public consultation (PC)

III.Targeted surveys of producers and users

2.3.Consultation methods and tools

Scoping Interviews

A series of eight scoping interviews were conducted early on in the evaluation with representatives of Eurostat and other European Commission Directorates General (DGs). These interviews helped the contractor to develop and validate the draft intervention logic and evaluation questions; establish the evaluation baseline; develop a comprehensive understanding of the programme context; and get suggestions for potential documents, interviewees and case studies that in turn informed the refinement of the evaluation methodology. The interviews took approximately one hour each and were semi-structured, allowing conversations to follow up on the interviewees’ background and expertise.

Stakeholder interviews

Stakeholder interviews allowed the contractor to collect specific information on the implementation of the ESP, as perceived by the participants in these activities (both users and producers of the statistics) but also to feed into the thematic and Member State case studies. The contractor carried out a general interview programme of 50 in-depth interviews with key stakeholders, including representatives from Eurostat and client DGs; a sample of other EU and international bodies, candidate and EFTA countries (European Free Trade Area), non-public institutional users, as well as producers and users in five Member States, namely Estonia, Germany, Poland, Spain and Sweden.

Due to the COVID-19 pandemic, the contractor conducted all interviews remotely (they took approximately one hour each), using a semi-structured format.

Surveys of producers and institutional users

The stakeholder consultation strategy included two separate, targeted, online surveys:

·A survey of producers: this online survey was primarily focused on NSIs (National Statistical Institutes) and ONAs (Other National Authorities) but also included other individuals and organisations who were able to share a perspective on the effectiveness of ESP implementation and operations. The survey generated 43 responses.

·A survey of users of European statistics: this online survey was primarily focused on institutional users, in particular EU institutions and umbrella organisations, but not exclusively. The contractor received 45 responses to this survey.

Implementing a two-pronged approach allowed the team to target questions specifically at the supply side (NSIs, ONAs and other ESS (European Statistical System) stakeholders involved in production of European statistics) and demand side (institutional and other users) for the statistics and consult a broader range of stakeholders than what would have been possible through in-depth interviews.

The questionnaires were administered through EU Survey and disseminated to stakeholders by e-mail with an accompanying cover text explaining their purpose and how the results were to be used. The producer survey was launched on 17 December 2020 and remained online until 5 February 2021. The user survey was launched on 17 December 2020 and remained online until 8 February 2021.

The public consultation

In order to facilitate a broad range of views, the questions posed in the public consultation were general questions about the statistics provided by Eurostat, in contrast to more specific questions about the ESP in the targeted surveys. This was done to make sure that the PC complemented, rather than substituted, the targeted consultations.

The contractor prepared the questionnaire used in the public consultation in English, with the translation into French and German undertaken by European Commission services. The three versions were implemented in EU Survey, and the PC was published on 16 December 2020 and remained open until 10 March 2021. It generated 424 responses. One response was discarded for not having respected the rules for feedback and suggestions.

The contributions received in the context of the public consultation published on the ‘Have your say’ web portal cannot be regarded as the official position of the European Commission and its services, and therefore do not bind the European Commission. The contributions cannot be considered as a representative sample of the EU’s population.

3.Results of stakeholder consultation

3.1.Stakeholders engagement

Table 1 below presents an overview of all stakeholders consulted in the framework of this evaluation. Besides demonstrating that both statistics users and producers were included, the table also highlights the different stakeholder categories represented in the consultation programme. The breakdown evidences that different perspectives, institutional and non-institutional, EU and non-EU, are taken into consideration. The choice of the most relevant consultation tool and the topics of the consultation reflects the level of knowledge, expertise and interest of each stakeholder category with the ESP.

The contractor secured the engagement of all stakeholder groups originally included in the consultation strategy. But some adjustments were made to further tailor the approach to the interview programme after the contractor observed a slow response rate to the interview requests sent out. A general consultation fatigue, often resulting from the COVID-19 crisis and the multiplication of online meetings, resulted in a slow response rate. To address this challenge, the contractor worked closely with Eurostat to identify relevant contacts (in the respect of the GDPR (General Data Protection Regulation) rules) to send out tailored interview requests. In addition, the contractor also agreed with Eurostat that it would be more efficient to carry out some interviews - with Eurostat thematic units, a sample of European Commission DGs, as well as other EU and international bodies - in the framework of the thematic case studies instead of the general interview programme. 51 This more focused approach allowed the contractor to carry out more interviews and avoiding multiple interview requests, while allowing to collect specific data relevant to the evaluation.

Table 1. Stakeholders engaged in consultation activities

3.2.Summary of the methodology and tools used to process the data

The contractor delivered a quantitative and qualitative analysis of the data collected through the different consultation activities. The quantitative analysis included a descriptive statistical analysis of the datasets, notably for the public consultation and the targeted surveys. The contractor cross-tabulated variables to check whether there were differences across different groups of respondents (e.g. between participants from a given country or stakeholder group). To analyse the data, the contractor used frequencies of satisfaction, revealing trends in the degree to which the European Statistical Programme has achieved its intended goals. Averages in terms of responses across respondents were also used as part of the descriptive statistical analysis. In order to present detailed and comprehensive statistical analysis, data were analysed using software such as SPSS and QResearch. All interview questions and open questions from the PC and targeted surveys were subjected to qualitative analysis. Where answers had been provided in different languages, the contractor translated all answers into English in order to code them together in a systematic way. The coding of the answers per question along several categories enabled the contractor to systematise the information provided and highlight the main trends emerging from open responses.

The analysis was first done at the level of individual data collection tools, before the contractor produced the overarching analysis to identify the conclusions across all consultation tools. These conclusions in turn fed into the preparation of the main evaluation report. The contractor analysed and integrated the data from different sources to elaborate the answers to the evaluation questions.

3.3.Key findings from consultation activities

Relevance of the ESP

Under the relevance criteria, the contractor collected stakeholder feedback about the relationship between the objectives of the ESP and the needs of the European Statistical System and users of European statistics to analyse whether the ESP was adequately designed to address the needs and challenges it intended to address.

The feedback from stakeholders generally confirmed that the European Statistical Programme was fit for purpose. Most of the respondents of the targeted producer survey assessed the four objectives of the ESP as relevant to a great or to a moderate extent, including:

·Objective 1: To provide statistical information in a timely and cost-effective manner to support the adequate development, monitoring and evaluation of EU policies (86%, namely 37 responses)

·Objective 2: To implement new methods of production to increase efficiency and improve quality (82%, namely 35 responses)

·Objective 3: To strengthen the partnership within the ESS (71%, namely 30 responses)

·Objective 4: To ensure statistics are delivered in a consistent manner and can quickly adapt to new priorities (74% namely 31 responses)

Four-fifths of the targeted producer survey respondents confirmed that the ESP activities were appropriate to deliver on the four objectives either to a great or to a moderate extent.

Users contributing to the targeted user survey found that Eurostat statistics were appropriate for their purpose of usage, either to a great extent (89%, namely 39 responses) or to a moderate extent (11% namely 5 responses). This group of respondents confirmed the importance of the first objective, with almost half of them indicating that they used Eurostat statistics in support of policy making at the EU level (42% namely 19 responses).

The public consultation also confirmed the overall relevance of the programme. More than four-fifths of the 424 respondents indicated that European statistics were relevant to the respondents’ role or sector, either to a great extent (53%) or to a moderate extent (31%).

Out of all the interviews carried out, a majority of the stakeholders also highlighted that the priorities and objectives of the ESP were all relevant and very well selected. Statistical strategies of the Member States were often based on ESP objectives, so there was a close link between them. Stakeholders also confirmed that the ESP individual objectives were aligned with the needs of the statistical system. Interviewees from the NSIs provided more detailed feedback on the four objectives:

·Objective 1: in some fields such as social statistics, the programme was adapted at the time of the extension and they saw a clear improvement.

·Objective 2: a lot of work went into the modernisation of production methods that allowed the ESS to improve the quality of statistics and deliver efficiency gains. The sample of NSIs interviewed considered the close links to the ESS Vision 2020 strategy to be very helpful. Interviewees noted however that more efforts were needed to modernise further production methods, which in turn was expected to enable the programme to meet increasing demands from users.

·Objective 3: the interviews with the sample of NSIs revealed the existence of a consensus on the importance of partnerships. They considered that these were very relevant for strengthening the relationships within the system and with external stakeholders and for delivering on other objectives of the programme towards the provision of timely, consistent and high-quality European statistics.

·Objective 4: The interviewees considered that this objective responded well to NSIs’ needs as they were facing numerous requirements and demands. They highlighted that the system for prioritisation had not worked that well through the programme, but that a new system would be piloted from 2021.

Stakeholders also assessed positively the introduction of multiannual planning and the collaborative approach to the planning process.

The representatives from EFTA and candidate countries interviewed confirmed that the ESP objectives in general were well chosen and very relevant, reflected the needs of the statistical organisations also outside the ESS and generally were relevant in 2013, as well as at the time of the extension, and had remained valid throughout the period evaluated. On objective 3, in particular, they highlighted the close and intense cooperation with Eurostat. One interviewee referred to Eurostat’s open and generous approach to their organisation since, as a non-ESS country, they were invited to and could participate in all meetings of the ESS.

Interviewees from the European Commission DGs also confirmed that the ESP had been and remained a useful tool to meet their statistical needs. The continued commitment to better and evidence-based policy making created a need for structured statistics production efforts and the production of high quality and comparable data across Member States. The ESP was set up to address this need and continues to do so. It also constituted an important instrument to support EU policy objectives.

Most respondents to the targeted producer survey (55%, namely 23 responses) indicated that they did not see any gaps that the ESP objectives failed to cover in terms of existing needs, but one third (31%, namely 13 responses) of them indicated that they did. 46% also indicated they did not see such gaps in terms of future needs compared to 27% who did see them (19 responses against 11 responses). Some of the issues related to the design of the programme that have been identified by respondents include: time constraints, challenges in keeping up with Eurostat’s requirements and the requests for new statistics, being an addition to an already demanding ESP (these issues are discussed more extensively under the efficiency criterion). They also mentioned they felt that partnerships in the ESS needed to be strengthened, that there could be a closer coordination from Eurostat of the work carried out by the NSIs, as it was done in the COVID-19 crisis, and insufficient coverage of the access to privately held data.

The respondents to the targeted producer survey assessed that the ESP was effective in adapting to new challenges, such as increasing data security risks (55% assessed it was very effective or effective, namely 23 responses), harmonisation of statistics across countries (79% or 34 responses), Big Data (58% or 25 responses) and advances in data visualisation (71% or 30 responses).

However, in the interviews, representatives of producers from the Member States pointed out some issues with the design of the programme such as:

·a lack of detail, which implied that the ESP was formulated at a very generic level, so in fact all NSIs’ activities fell within its scope and that there was a lack of prioritisation. However by some this was perceived as an advantage allowing for flexibility;

·a lack of flexibility in particular in reacting to emerging challenges, but also the ESP being a very ambitious, expensive and long programme for the NSIs;

·a lack of strategy to access and integrate privately held data.

The respondents from the European Commission also emphasised that the ESP lacked flexibility: any change to the scope of the programme required a change in the EU legislation, which was a lengthy process. This implied that ad hoc needs often could not be met with statistics which were reliable, timely and comprehensive at the same time. While flexibility mechanisms existed (e.g. pilot projects), Member States participated in them on a voluntary basis.

Users who responded to the targeted user survey generally indicated that they would like Eurostat to collect and publish more statistics within its existing statistical themes (64% or 29 responses agreed with the statement while 20% or 9 responses were unsure and even fewer (16% or 7 responses) found the collection and publication of more statistics unnecessary). Respondents highlighted the need for a more detailed and granular breakdown of different types of statistics at regional level, as well as the need for data subdivided by more detailed sociodemographic attributes. Statistical areas which were perceived as requiring more development by respondents were: circular economy, agricultural data relating to rural areas and urban and passenger mobility.

However, producers were less eager to extend the datasets. In the targeted survey, 43% of producers (18 responses) indicated that they would not like Eurostat to collect and publish more statistics (26% said they would). This was confirmed in the interviews. Producers pointed out the need to keep a balance between demand for statistics and their (limited) capacity.

Effectiveness of the ESP

Under the effectiveness criteria, the contractor collected stakeholders’ feedback about the progress towards achieving the four objectives of the ESP and discussed factors that drove or hindered that progress.

Overall, the feedback from stakeholders confirmed that the ESP had been effective in reaching its objectives, in particular improving access to statistics, however it also identified several areas that could still be improved.

Access to statistics

In the public consultation, nine out of ten respondents indicated that they were either satisfied (51% or 216 responses) or somewhat satisfied (39% or 165 responses) with the availability of Eurostat statistics. Interviews with users confirmed this positive assessment. All the users noticed a general improvement over the last years, in elements such as: data visualisation, filters, easiness of downloading data and overall user-friendliness. They noticed improvements in the access to the data through the website and its user-friendliness. They also indicated that statistics were better explained (the ‘Statistics explained’ articles were referred to several times as well written and useful). Statistics were well presented for different types of users and were accessible not only via tables, but also Application Programming Interface (API) 53 . Access to microdata, which could be downloaded, had also improved. Different elements, such as detailed metadata with descriptions, sources, methods were useful and easily accessible via links. Producers who contributed to the targeted survey agreed that the ESP increased the availability of Eurostat statistical data, either to a great (37% or 16 responses) or to a moderate extent (44% or 19 responses). The producers’ assessment of availability of statistics in the interviews was also positive. The interviewees assessed the situation as well as recent developments positively. They indicated that there was the right balance between timeliness and availability of statistics.

Despite this overall positive assessment, interviewees from the European Commission DGs still identified that some statistical needs were not met under the ESP. There was a need for more granularity of the statistics, with two examples mentioned more particularly (providing statistics at the NUTS3 level instead of at the NUTS2 level, and gender breakdown) and new topics to be covered, including those related to the current European Commission’s priorities (e.g. mainstreaming of environment in all statistical data to support the implementation of the EU’s Green Deal).

Quality and harmonisation of statistics

The stakeholder feedback confirmed the high quality of Eurostat statistics overall. It also confirmed that the comparability of statistics across the Member States was improving.

In the targeted user survey, 80% of users (36 responses) considered that the ESP has improved the accuracy and reliability of statistics. Moreover, 77% of users (34 responses in the targeted user survey) also assessed that the ESP was successful in ensuring the comparability of statistics among the Member States. The interviewees confirmed that the quality of statistics was good and, since 2018, there had been constant improvement of quality. Interviewees also noted that the statistics produced under the programme became more consistent across Member States through the use of new tools, guidance and methodologies implemented by Eurostat and that these were developed on the basis of robust methodologies, according to the principles of the European statistics Code of Practices. Examples provided included demographic and environmental statistics. The representatives of the European Commission DGs highlighted the importance of the ESP’s contribution to the definition and harmonisation of production methods and methodologies, which was considered a prerequisite for the production of reliable and comparable data. Nevertheless, interviews with some users noted some inconsistencies in how data were collected for different countries or some gaps in statistics when data for certain countries were not available, which in turn might affect the comparability of statistics. But overall, consistency and comparability were considered as key strengths of the ESP and the ESS. Producers shared the view that the ESP improved the accuracy and reliability of statistics (79% or 34 responses from the targeted producer survey). There was also a consensus within the representatives of EFTA and candidate countries interviewed that the ESP had delivered on its objectives, with a particular emphasis on the quality improvements achieved. On the challenge related to data being collected differently in different countries, producers had a rather positive perspective, highlighting it was introducing flexibility in the programme.

Timeliness of statistics and new sources of data

The timeliness of statistics was assessed to be good. In the targeted survey, 79% of producers (34 responses) and 60% of users (27 responses) indicated that the ESP had been successful in improving the availability of statistics based on the timeliness criterion. However, in the targeted user survey, the timeliness was rated lower than other criteria, such as relevance of statistics (80% or 36 responses), accuracy (80% or 36 responses), coherence (77% or 34 responses) and accessibility (73% or 32 responses). In the Public Consultation, respondents indicated that they were not using European statistics only but also other sources. The five alternative sources of statistics most frequently mentioned by contributors were country specific National Statistical Institutes (32%), the Organisation for Economic Cooperation and Development (OECD, 14%), other national institutes and organisations (11%), UN offices (9%) and the World bank (8%). Other sources such as National Banks, European organisations 54 , EU Offices, the World Health Organization, media and news, academic resources and local surveys were mentioned by 2% or less of those who gave an answer to this open-ended question. When respondents used other sources, they ranked European statistics compared to these other sources as higher on ‘coherence and compatibility’, ‘relevance’ and ‘accuracy and reliability’ but lower on ‘timeliness and punctuality’ as well as ‘accessibility and clarity’. In the interviews, producers from the EU as well as EFTA/candidate countries confirmed that timeliness improved in the last years, especially with regard to European economic indicators and that there was a strict calendar for the publication of statistics, which allowed the ESP to meet the needs of the European Commission’s DGs and the ECB. However, respondents pointed that not all countries were successful in this regard.

Delays in publishing statistics were also mentioned in the interviews with users from the Member States, who pointed out that statistics were not always provided in a timely manner, in particular for some countries. Some users also highlighted a time lag in their fields of interest (e.g. environmental statistics). Users from the European Commission DGs shared this assessment. While recognising the need for validation / quality review and that this could take time, interviewees also identified timeliness as an issue: certain policy needs were very pressing – including during a crisis situation such as COVID-19 – and timely statistics were not always available.

The interviews with producers indicated that statistics based on registers were timelier than those based on interviews, so this source should be used more when available – with the caveat added by Eurostat and users that this source should produce statistics of sufficient quality to be used more systematically. They also highlighted that more innovative solutions and sources should be used, such as Big Data based on privately produced data, for instance on electricity usage, credit cards, mobile data, etc. They noted that such sources had already been used in experimental statistics. It was noted that the use of this type of information depended at the time on the legal framework of each country and that there was a sense of a missed opportunity to develop this further under the ESP 2013-2020. They were of the opinion that more effort in this regard should be made in the future. At the same time, some interviewees expressed their concerns about the quality of such data. They pointed to the need for an in-depth analysis of any new sources before they could be used ‘with Eurostat’s quality stamp’.

Dissemination channels

The vast majority of respondents to the targeted user survey agreed that Eurostat dissemination channels (including the website) were easy to use and/or effective (40% to a great extent, 47% to a moderate extent namely 39 responses overall). The producer survey confirmed this positive assessment (42% of respondents agreed to a great extent that Eurostat dissemination channels were easy to use and/or effective and 42% to a moderate extent namely 36 responses overall).

However, one element that was often criticised by users in the interviews was the search engine on the Eurostat website (this issue was mentioned by all interviewees). Respondents indicated they often used Google search to find statistics on Eurostat website, which was more effective than Eurostat search option. According to them, the latter often provided misleading and unsatisfactory search results.

Metadata

In the survey, users assessed that Eurostat provided sufficient information about its statistics in terms of quality standards (83% or 35 respondents agreeing to a great or to a moderate extent), data collection methods (68% or 30 responses) and sources (86% or 38 responses), a result confirmed by the PC (83% of respondents agreed strongly or somewhat that Eurostat provided sufficient information about the statistics published). However, the interviews with users suggested that more advanced users noted substantial issues with metadata. Several respondents indicated that, while the content was still relevant, metadata were not detailed enough. The lack of details sometimes made cross-country comparability difficult.

Efficiency of the ESP

Under the efficiency criterion, the contractor collected stakeholders’ feedback about the costs and benefits of the ESP and the balance between them.

The statistics producers provided the most meaningful feedback about cost-efficiency of the ESP.

In the targeted producer survey, most of the respondents assessed the benefit to cost ratio of the ESP to be proportionate (55% namely 23 responses) or very proportionate (7% or 3 respondents). Fewer respondents (5% or 2 respondents) assessed the ratio as disproportionate and one third of respondents were not able to assess it.

In terms of the achievements in introducing efficiency gains in the production of statistics, most respondents considered that the ESP was successful in reducing the duplication of effort (70% namely 30 responses, including 19% agreeing with the statement to a great extent and 51% to a moderate extent). Slightly more respondents also supported a statement that the ESP was successful by reducing the administrative burden (17% or 7 respondents agreed with the statement to a great extent, 31% to a moderate extent versus 31% to a small extent and 12% did not agree at all).

For most of the producers interviewed, the topic of efficiency and NSIs’ and ONAs’ limited capacity to deal with statistical requirements was the most important one in the study. They highlighted that the demand for timely statistics was constantly increasing, including the demand at EU level. They also assessed that the ESP did not help to address these problems. They pointed out that the producers at national level already worked at their maximum capacity and therefore any new requests for statistics were challenging, in particular for the smaller Member States. Some interviewees highlighted that the institutional users of statistics should be more aware of the production costs of statistics.

Achievement of efficiency gains were mentioned in the increased use of administrative and register data, estimation methods, and the application of microdata exchanges. The management and governance mechanisms of the ESP were generally considered efficient.

Several interviewees mentioned a ‘one in, one out’ approach or a need to select negative priorities, i.e. less important statistical fields which could be deleted from the ESP. However, some interviewees also acknowledged the difficulty of introducing such approach in practice, which had also been considered at country level but never successfully implemented.

The respondents provided similar feedback in the open comments to the targeted survey, indicating that the elimination of unused data and irrelevant data was necessary. They suggested the implementation of a more automated data validation process and reckoned that gains could stem from exchanging good practices between countries.

Coherence of the ESP

Under the coherence criteria, the contractor collected stakeholder feedback about the internal and external coherence of the ESP. Internal coherence is the alignment between the different objectives of the programme. External coherence is the degree to which the objectives of the ESP are aligned with those of other statistical authorities including EU bodies, as well as international partners.

For all stakeholders engaged in the different consultation activities assessed that the ESP was both internally and externally coherent.

In the targeted producer survey, almost all respondents indicated that the ESP was internally coherent. Most agreed with the statement that ‘the objectives of the ESP as set out in the Regulations and programme planning are internally coherent’ to a great extent (52% or 22 responses) and one third (33% or 14 responses) did so to a moderate extent. No one disagreed (while 14% selected the ‘don’t know’ answer, namely 6 responses). There was slightly less agreement when the statement referred to activities of the ESP: 48% (20 respondents) agreed to a great extent and 38% (16 respondents) to a moderate extent. As with the statement related to objectives, no one disagreed and 14% selected the ‘don’t know’ answer, or 6 respondents). Moreover, 74% (31 respondents) found there to be effective mechanisms in place to coordinate activities within the ESP (50% to a great extent, 24% to a moderate extent).

These findings were confirmed by the interviews with producers. The objectives and activities under the programme were considered to be coherent between each other and in relation to the ESS. The general framework of the programme had allowed for it to be developed through the multiannual programming and the annual work programmes. Although the annual work programmes were developed by the European Commission, considering the needs of the DGs, Member States were also consulted and could provide feedback.

There was also an overall agreement about the external coherence of the programme. Respondents agreed to a great extent (30% or 13 respondents) or to a moderate extent (42% or 18 respondents) and outlined that there were effective mechanisms in place to coordinate with international partners. There was slightly less agreement in terms of the complementarity of Eurostat activities with statistics production and dissemination activities of international partners (39% or 16 respondents agreed with the statement to a moderate extent, and 30% or 12 respondents to a great extent). Most respondents also indicated there were mechanisms to coordinate with other EU bodies, 20% (9 responses) agreed with the statement to a great extent and 37% (16 responses) to a moderate extent.

In the interviews, the representatives of European Commission DGs (including Eurostat), highlighted the existence of coordination mechanisms within the EU and the cooperation with international partners, ensuring the complementarity between the ESP and other statistics produced by other (political) DGs / other international organisations. The ongoing coordination between Eurostat and thematic DGs was effective at preventing overlap/duplication.

In the interviews with producers of statistics, they assessed that the coordination between Eurostat, Member States and other EU bodies was very effective, but they also noted some issues:

·Interviewees highlighted the increasing number of requests from different DGs to produce statistics. In their opinion, the DGs did not always recognise the costs and burden of producing statistics.

·The ESP had adapted to new policy needs and the European Union’s strategies and objectives but was less able to address sudden changes or events or the pace of certain technological developments and digitalisation. It was noted that the European Commission could have better considered the views of Eurostat and Member States in the European Data Strategy, as statistics were only mentioned once or twice in the document.

Eurostat’s coordination with international partners was considered generally coherent and effective and producers of statistics and the representatives of the European Commission (EC) highlighted that it had improved over the last years. European Commission representatives agreed that there were complementarities between the ESP and international organisations, such as the OECD, the World Bank, the Food and Agriculture Organisation (FAO) and the WTO (World Trade Organisation). Complementarities also existed in certain policy areas, such as education, for instance when the European Commission used the OECD’s Pisa survey data to complement the ESP for particular indicators in this policy field. Under the ESP, Eurostat also negotiated bilateral agreements with international organisations to address data needs, joint data collections and data exchanges overall. Some producers noted that, although there were still some overlaps between Eurostat and international organisations requirements, there were many initiatives with the objective of getting more coherent concepts that had been effective at preventing the duplication of work, in particular between Eurostat and the OECD.

Representatives of EFTA/candidate countries interviewed agreed that the ESP was internally and externally coherent.

EU added value of the ESP

Under the EU added value criteria, the contractor collected stakeholder feedback about the value resulting from the ESP that is additional to the value that would have resulted from similar activities only carried out at national level.

The feedback collected clearly confirmed the EU added value of the ESP.

In the targeted producer survey, nine out of ten respondents confirmed that the EU added value of the ESP. They agreed that the ESP responded to particular needs that would not otherwise have been addressed by NSIs. 56% of respondents (namely 24) agreed with the statement to a moderate extent and 23% (10 responses) to a great extent.

When looking at two concrete elements of the EU added value, namely the harmonisation of statistics across Member States and enhancing the pool of resources available, the results were similar. Nine-tenths of the producers confirmed that the ESP contributed to the increased comparability of national statistics, including 63% (27 responses) indicating it contributed to a great extent and 26% (11 responses) to a moderate extent. A similar share of respondents agreed that the ESP enhanced the pool of resources available for the production and development of new statistics at Member State level (44% or 19 respondents agreed with the statement to a moderate extent and 28% (12 responses) to a great extent).

In the interviews, there was also a consensus among producers that the ESP provided clear added value in the harmonisation of definitions and in the compilation of comparable high-quality statistics for the EU, and that this could not have been done without Eurostat. Some interviewees also mentioned that the ESP provided strategic foundations for producers of statistics, with a clear direction over seven years giving the system continuity. The programme worked by setting priorities and offered a legal basis for identifying upcoming fields of action and for financing projects. Member States could successfully produce official statistics at national level, but Eurostat coordination was necessary for the production of European statistics. The interviewees also highlighted that, for many people, Eurostat was more trusted as an institution than NSIs. The respondents also highlighted the usefulness of Eurostat’s grant scheme, bringing up examples of grant topics such as visualisation, education activities, or fighting against fake news (these were only a few examples identified by the respondents).

The European Commission and other EU bodies’ officials mentioned in the interviews that the EU added value of the ESP through the support it provided to the delivery of the European Commission’s political and policy priorities. For them, the ESP constituted a central mechanism for all things related to statistical data in the EU. Thereby, it decreased the transaction costs for its stakeholders – within the EU as, for example, individual DGs do not have to go to individual Member States to request data and outside of the EU as the ESP was developing partnerships with international organisations to address data needs – while making available the necessary data and tools. As with producers, the EU officials highlighted that the ESP supported the harmonisation of statistics across the EU. They also highlighted that the programme delivered an institutional added value. It gave visibility both at the Member State level and at the international level to statistical data and their importance to support evidence-based decision making, as well as the strengthening of the international statistical community. In the future, the ESP could support a common approach to the challenges related to new data sources too.

The EU added value was also perceived by the sample of representatives from EFTA and candidate countries. The interviewees considered that the same progress and benefits, in particular with the modernisation of candidate countries statistical systems, would not have been achieved in the absence of the programme. However, it was noted that the programme included some heavy and lengthy procedures that were caused by the institutional set up, the different legal acts, and a large partnership between the EU Member States, the EC and the EFTA countries, making the implementation of the programme quite complex.

Individual users also confirmed the EU added value of the programme. They indicated that there was clear EU added value in Eurostat’s work, mainly related to ensuring comparability of data from the Member States and providing them in a single online location. Some more advanced users acknowledged the improvement in comparability of data and harmonisation of surveys across Member States. However, the users also pointed out that there was still a lot to do in terms of ensuring comparability, mainly in terms of harmonising methods of collecting data across Member States.

In terms of comparing the quality of Eurostat and other international sources of statistics, the most frequently selected answer in the targeted user survey was that quality was similar (between 47%-49% of responses across the range of quality criteria). Between one third and half of respondents assessed the quality of European statistics was higher, depending on the indicator (51% or 23 responses for coherence and comparability, 49% or 22 responses for accuracy and reliability, 40% or 18 responses for accessibility and clarity, as well as for relevance, 29% or 13 responses for timeliness). In the interviews however, users highlighted that it would be challenging to compare the different sources (e.g. OECD, the World Bank) given the differences in the scope of the datasets.

Annex 3: Methods and analytical models

3.1 Evaluation matrix

An evaluation matrix was developed when designing the evaluation and used to draw the conclusions for each criteria. The questions proposed for the five evaluation criteria could cover also the points mentioned in Article 15 of the ESP amended regulation.

Table 1: Evaluation matrix

3.2 Overview of the methodological approach

The evaluation team designed and implemented a phased approach to the evaluation.

The designing phase (task 1) resulted in the definition of the evaluation framework. Building on a round of scoping interviews and an initial review of relatively high-level documents, the team furthered its understanding of the ESP, as well as the evaluation and its objectives. These preliminary data collection activities led to the development of the evaluation baseline (as presented in the inception report, deliverable D1) and the preparation of the Evaluation Questions Matrix (EQM). The EQM structured all subsequent tasks under this evaluation, both in terms of the data collection and in terms of data analysis. Key statistical terms applied in the European Statistical System (ESS) are defined in the Glossary of the European Statistics Code of Practice. 55 In the report, the evaluation team often refers to the general terminology ‘data and statistics’, which is consistent with Eurostat’s mission statement. 56

The data collection phase (task 2) was dedicated to compiling all relevant data (both primary and secondary) to inform the evaluation. The evaluation team implemented a mixed-method approach. Data collection consisted of the following. First, the team carried out a comprehensive documentary review (see the reference list presented in Annex 2). The systematic review of all sources identified allowed to identify and collect all data with regard to the performance of the ESP between 2013 and 2020, to feed into the responses to the evaluation questions. Second, the evaluators designed and implemented a stakeholder consultation strategy. Consultation activities allowed to corroborate findings of the documentary review as well as to address any data gaps identified. The consultation strategy consisted of (1) targeted online surveys addressed to statistics users and producers, (2) a public consultation and (3) in-depth stakeholder interviews (all carried out remotely). The synopsis report (deliverable D4, submitted separately) provides a detailed description of the stakeholder consultation strategy, its implementation and its results. Overall, despite the challenges resulting from the COVID-19 pandemic and the need to consult all stakeholders remotely, the level of engagement achieved was high. The following table provides an overview of this engagement:

Table 2. Stakeholders engaged in consultation activities

Source: Evaluation team’s consultation strategy

The evaluation team also structured part of the data collection efforts into thematic and country case studies. Four thematic case studies explored the extent to which data made available under the ESP have added value in specific policy domains (see Annex 4): (1) European business statistics, (2) statistics on asylum and managed migration, (3) territorial statistics and (4) COVID-19. The case studies focused on the downstream impact of ESP activities by conducting a ‘deep dive’ of detailed outputs for institutional and non-institutional users. The sample of topics reflected several criteria, namely the balance between economic, social and territorial fields, as well as the opportunity to engage with a range of users. These case studies built on a documentary review of relevant sources and a programme of stakeholder interviews. In addition, the team carried out five country case studies. The sample covered Estonia, Germany, Poland, Spain and Sweden. These case studies built on interviews of officials responsible for the statistical work programme in the NSIs and in ONAs, and other stakeholders such as policy makers and regular professional users of the data at national level, and fed into the preparation of country factsheets (see Annex 3) providing an overview of the statistical systems implemented at Member State level and of the dynamics at play in statistics production.

The final two phases of the evaluation, analysis (task 3) and judging (task 4), involved the systematic analysis of quantitative and qualitative data, mapping the data to the evaluation topics and questions, comparing the evidence collected through different tools and drawing conclusions. They culminated in the formulation of practicable recommendations from the evaluation.

Figure 1: Three components of analysis

The conclusions and recommendations are strictly based on findings from the evaluation, providing founded judgements based on a solid and rigorous analysis using both quantitative and qualitative approaches as applicable. The answer to each EQ includes the following elements:

·interpretation and comprehension of the key terms of the EQs;

·indication of the judgement criteria allowing to answer the question as well as the quantitative analysis that is possible given the data available;

·validity of the quantitative and qualitative information used; and

·description of the evaluation methods used and an indication of their limitations.

In addition, each answer provides a detailed description of the reasoning followed in the analysis, also indicating the underlying hypotheses and validity limits of that reasoning.

Lastly, as the team delivered the evaluation, it faced a number of challenges to which it responded as follows:

1.The COVID-19 crisis prevented the team from undertaking face-to-face data collection activities. All activities took place remotely instead. This implied that the team deployed stronger and more targeted efforts to ensure a good response rates for the interviews to be conducted for the overall programme assessment, as well as the country and thematic case studies.

2.The scope of the evaluation means there have been numerous information and data sources processed. Several measures were implemented to avoid overload. These included: (i) a strict relevance test applied to data gathering to discard not strictly pertinent data; (ii) the accurate use of data already available in order to complement the primary data collection; and (iii) the use of a common template for gathering qualitative and quantitative information, following the evaluation questions and sub-questions, to allow easier processing. The preliminary mapping undertaken as part of the inception phase was instrumental for identifying relevant documents to map and the purpose they will serve. In addition, strong coordination with the evaluation manager at Eurostat was necessary to ensure that all relevant documentary sources, including those not publicly available yet, were made available to the evaluation team.

3.One of the main sources of data for the evaluation have been the perceptions and opinions of relevant stakeholders, including statistics users and producers. A consultation programme of this scale presents logistical and conceptual challenges. The list of proposed stakeholders and related tools has been designed to achieve the best possible balance between interest groups and topics. The contractor has designed different, tailored survey tools with skip logic that directed stakeholders to relevant questions based on their profiles and familiarity with key elements of the ESP. Dissemination was a particular challenge, in the absence of a comprehensive list of users of statistics. The contractor relied on the support provided by members of the Inter-service Steering Group on that aspect. In addition, stakeholder engagement with the consultation tools creates a self-selection bias: the sample of stakeholders who contributed to the evaluation is not statistically representative of the ESP stakeholders (for instance, the sample of users contributing to this evaluation did neither cover all users, nor a representative sample of users). Findings are only indicative of stakeholders’ opinions and need to be considered with caution. Lastly, the findings are also caveated by the fact that stakeholders’ feedback might have been subject to a bias, the interviewer effect (interviewees might give the responses they thought were desired by an interviewer) and conditioned by the self-selection of respondents contributing to online consultation tools, because the respondents and interviewees were aware that the study was carried out on behalf of Eurostat. The contractor mitigated this methodological challenge through the use and analysis of different sources of evidence, which fed into the answers to the evaluation questions, and caveated carefully the analysis.

Annex 4: Final Report of the Contractor

(1)      OJ L 87, 31.3.2009, p. 164.

(2)      OJ L 39, 9.2.2013, p. 12.

(3)      OJ L 284, 31.10.2017, p. 1.

(4)      First mid-term evaluation: https://europa.eu/!tcF9J4 .

(5)      Second mid-term evaluation: https://europa.eu/!QCyfpB .

(6)      Article 1 of Regulation (EC) 223/2009 states that ‘European statistics are relevant statistics necessary for the performance of the activities of the Community. European statistics are determined in the European statistical programme.’ This document therefore uses the term ‘European statistics’ to refer to all statistics developed, produced and disseminated by Eurostat under the ESP.

(7)       http://ec.europa.eu/eurostat/documents/10186/756730/ESS-Vision-2020.pdf (available in English only).

(8)       https://ec.europa.eu/commission/sites/beta-political/files/juncker-political-guidelines-speech_en.pdf

(9)       https://ec.europa.eu/eurostat/documents/64157/4375449/ESP_progress_report_2019.pdf

(10)      The decrease between 2017 and 2018 was due to the implementation of the new datasets in national accounts in 2017, following the new European system of accounts (version 2010) legislation, while keeping those following the previous European system of accounts (version 1995). The datasets following the previous legislation were removed at the beginning of 2018.

(11)      Further details can be found in the individual reports of the ESS Vision 2020 (see ESSC 2021/45/13/EN, Table 1).

(12)       https://ec.europa.eu/info/publications/annual-management-and-performance-report-2020_en

(13)       https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52018SC0320

(14)      OJ L 153, 3.5.2021, p. 1.

(15)      ESSC 2020/42/3/EN, Priority-setting in the ESS (42nd Meeting of the European Statistical System Committee, February 2020).

(16)      European Commission (2016), Commission Staff Working Document – Executive summary of the Impact Assessment Accompanying the document Proposal for a Regulation of the European Parliament and of the Council amending Regulation (EU) No 99/2013 on the European statistical programme 2013-2017, by extending it to 2018-2020, (SWD(2016) 287 final, 7.9.2016).

(17)      European Parliament, The Juncker Commission’s ten priorities – An end-of-term assessment (PE 637.943, 05.2019).

(18)       https://ec.europa.eu/eurostat/web/covid-19/overview

(19) The USS reports are available at Evaluation - Eurostat (europa.eu) .

(20)      See the final report of the project on big data for official statistics (BIGD); https://europa.eu/!bf94rQ .

(21)      See the final report of the project on the European Statistical Data Exchange Network (ESDEN); https://europa.eu/!CN43cG .

(22)      See the final report of the project on single market statistics (SIMSTAT); https://europa.eu/!hw88Qk .

(23)      See the final report of the project on the re-design of Intrastat (REDESIGN); https://europa.eu/!Wb73kK .

(24)      For the full list, see: https://ec.europa.eu/eurostat/help/first-visit/tools .

(25)      Eurostat, ESS – Experimental statistics hub:  https://ec.europa.eu/eurostat/web/experimental-statistics/overview/ess .

(26)      Eurostat, COVID-19 labour effects across the income distribution:  https://europa.eu/!Mn9wnq .

(27)      Eurostat, Experimental statistics > Quarterly registrations and bankruptcies:    
https://ec.europa.eu/eurostat/web/experimental-statistics/quarterly-registrations-and-bankruptcies .

(28)      Long time series are defined in this case as those covering 10 or more consecutive years.

(29)      The derived datasets are not counted, but the variables included in more than one primary dataset (e.g. GDP and population) are counted as different statistics.

(30)      Data from Eurostat AARs 2014, 2015, 2016, 2017, 2018, 2019 and 2020.

(31)   https://ec.europa.eu/info/business-economy-euro/economic-and-fiscal-policy-coordination/eu-economic-governance-monitoring-prevention-correction/stability-and-growth-pact_en

(32)      Regulation (EU) 2019/2152.

(33)      Regulation (EU) 2019/1700.

(34)      Regulation (EU) 2018/1091.

(35)      The decrease between 2017 and 2018 was due to the implementation of the new datasets in national accounts in 2017, following the new European system of accounts (version 2010) legislation, while keeping those following the previous European system of accounts (version 1995). The datasets following the previous legislation were removed at the beginning of 2018.

(36)   Final audit report on Eurostat’s quality management of statistical processes

(37)      From the impact assessment of the framework regulation integrating business statistics, now the Regulation on European Business Statistics.

(38)      Decision No 1297/2008/EC of the European Parliament and of the Council of 16 December 2008 on a Programme for the Modernisation of European Enterprise and Trade Statistics (MEETS) (OJ L 340, 19.12.2008, p. 76).

(39)       https://ec.europa.eu/eurostat/documents/4031688/8971242/KS-02-18-142-EN-N.pdf .

(40)      Please refer to https://ec.europa.eu/eurostat/web/european-statistical-system/overview for details on these bodies.

(41)       https://webgate.ec.testa.eu/Ares/documentInfoDetails.do?documentId=080166e5b824794f&_f=ext .

(42)      The CMFB was established by Council Decision in 1991. The original Council Decision 91/115/EEC was replaced on 13 November 2006 by Council Decision 2006/856/EC (OJ, L 332, 30.11.2006, p. 21).

(43)   https://webgate.ec.testa.eu/Ares/document/show.do?documentId=080166e5c09be320&timestamp=1631267315629 .

(44)      Commission Staff Working Document – Annual Internal Audit Report for 2015: summary of the engagements finalised by the IAS within the 2015 audit plan accompanying the document ‘Report from the Commission to the European Parliament and Council Annual Report on internal audits carried out in 2015’ (Article 99(5) of the Financial Regulation) (SWD/2016/0322 final, 30.9.2016).

(45)      European Commission, 2013/2014/2015/2016/2017/2018/2019 Annual Activity Report – Eurostat.

(46)      Eurostat, Smarter, greener, more inclusive? Indicators to support the Europe 2020 Strategy – 2013/20142015/2016/2017/2018/2019 edition.

(47)      Eurostat, Eurostat regional yearbook 2018 edition, available at:  https://ec.europa.eu/eurostat/documents/3217494/9210140/KS-HA-18-001-EN-N.pdf .

(48)      OJ L 39, 9.2.2013, p. 12.

(49)      OJ L 284, 31.10.2017, p. 1.

(50)      Long time series are defined in this case as those covering 10 or more consecutive years.

(51)      The analysis of the case-study specific interviews is presented in the case study reports.

(52)      A share of PC respondents indicated they were both users and producers. The contractor reports on them in one of the categories to avoid double counting. The PC also received responses from 22 stakeholders who did not identify themselves as users or producers. These are not included in this table.

(53)      API is a software intermediary allowing flow of information between two applications.

(54)      For instance, the EU-IDB network, the ECDC, the European Audiovisual Observatory and even EU agencies such as the European Environment Agency.

(55)      Available from: https://ec.europa.eu/eurostat/documents/4031688/9439112/Glossary/

(56)      ‘Our mission is to provide high-quality statistics and data on Europe.’ See: https://ec.europa.eu/eurostat/web/main/about/who-we-are

(57)      A share of PC respondents indicated they were both users and producers. The evaluation team reports on them in one of the categories to avoid double counting.