EUROPEAN COMMISSION

Brussels, 15.12.2021

COM(2021) 794 final

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

on the final evaluation of the implementation of the European statistical programme 2013-2020

{SWD(2021) 383 final}

1.Introduction

Under Article 13 of Regulation (EC) 223/2009 1 on European statistics, ‘the European statistical programme shall provide the framework for the development, production and dissemination of European statistics, setting out the main fields and the objectives of the actions envisaged for a period corresponding to that of the multiannual financial framework’.

The 2013-2020 European statistical programme was established by Regulation (EU) 99/2013 2 , initially for the years 2013-2017. It was then extended by Regulation (EU) 2017/1951 3 to cover the remaining period of the previous EU multiannual financial framework, i.e. 2018-2020. Throughout this document, it is referred to as ‘the programme’ or ‘the ESP’, while the Regulation establishing it is referred to as ‘the amended Regulation’.

The general objective of the ESP is for the European Statistical System (ESS) to continue to be the leading provider of high-quality statistics on Europe. As a spending programme, the ESP is the overall framework for developing, producing and disseminating European statistics. As such, it provides the finances for developing and maintaining Eurostat’s statistical infrastructure and the ESS as managed by Eurostat. The ESP also provides financial support to Member States.

The requirement to carry out a final evaluation and present a report on the ESP is enshrined in Article 15 of the amended Regulation, which also states that:

‘The report shall in particular evaluate:

a)the outcome of the reprioritisation and cost evaluation of statistical products;

b)the actions taken by the ESS to reduce the implementation and production costs for Member States and to limit the overall burden stemming from the statistical projects and fields covered by the programme; 

c)the progress on rendering access to official statistics easier and more user‑friendly, including the provision of data on the Eurostat website; and

d)the progress on the improvement of data availability, including on social economy activities and on the Europe 2020 indicators.’

The evaluation also serves to fulfil the requirements of the Commission’s better regulation agenda and thus follows the better regulation guidelines for evaluations . It therefore covers the five compulsory evaluation criteria: effectiveness, efficiency, relevance, coherence and EU added value.

The conclusions of this evaluation could also be useful when conducting the mid‑term evaluation of the single market programme (SMP), which is, with its objective on high-quality European statistics, the successor of the ESP. The SMP Regulation 4  is applicable from 1 January 2021. 

This evaluation covers the entire period of the programme (2013-2020), with the two previous mid-term evaluations covering 2013-2017 5 .

The report is based on an evaluation staff working document, produced by the European Commission, which accompanies the report. The staff working document gives more details on the evaluation process and results. It has in turn been supported by an evaluation study conducted by an external contractor, who gave an independent opinion on the work Eurostat carried out.

To get robust results, the contractor used different sources, starting with a large review of existing documents. Then, the contractor carried out an extensive consultation with users and producers of statistics 6 . This included i) scoping interviews with representatives of Eurostat and other Directorates-General of the Commission (DGs); ii) a public consultation; iii) targeted surveys of users and producers; and iv) 50 interviews with different types of stakeholders. The contractor also carried out four thematic case studies and five country case studies.

2.background

The ESP lasted for 8 years – 2013 to 2020. The programme’s total budget was EUR 489 million. Around 41% was spent on grants and 59% on procurements. Furthermore, almost EUR 225 million supplemented the ESP’s own budget in the form of credits sub-delegated by policy DGs to cover data collections they specifically requested.

The budget was spent on the programme’s three priority areas. The first ‘Statistical outputs’ dealt with producing European statistics. In this area, the money was mostly spent on grants used to provide financial support to Member States, who used the support to improve their national statistical systems and to implement measures enabling them carry out new data collections. All Member States benefited from grants, which ultimately allowed them to produce more and better quality data. As a result, the total number of disseminated datasets increased by 809 or around 18% from 2013 to 2020.

Figure 1: Number of datasets published by Eurostat, 2013-2020 7

Source: Eurostat’s reference databases

The second priority area ‘Production methods of European statistics’ dealt with improving the way statistics are produced, their quality and the way they are disseminated. Grants were given to Member States to strengthen the quality and efficiency of statistical production through innovative statistical methods and tools. The activities in the scope of the grants were co-financed by the European Commission and the Member States, which used the money to participate in the modernisation projects under the European Statistical System Vision 2020 (ESS Vision 2020). In addition, procurements were used either to partly finance ESS Vision 2020 projects or to finance the modernisation of the statistical infrastructure used to regularly produce and disseminate the statistics. Examples of results are:

·a modern the IT infrastructure used for exchanging data with Member States;

·a renovated website for Eurostat;

·a series of new visualisation tools; and

·new ways to reach out to users.

The third priority area ‘Partnership’ aimed at supporting the production and quality of statistics by improving the cooperation within the ESS and with other international organisations and countries outside the EU. The budget was mostly used to help develop and to a lesser degree produce statistics in countries outside of the EU and the European Free Trade Association, with particular emphasis on enlargement and the European neighbourhood policy. This has enabled Eurostat to regularly publish data on non-EU countries, especially to support the enlargement process and the negotiations with those non-EU countries.

In spending the budget, no major problems or infringements occurred. Eurostat put in place an effective system of anti-fraud measures, preventing fraud.

3.Key findings

3.1.Relevance

The ESP provided the framework for developing, producing, and disseminating European statistics, setting out the objectives of the activities envisaged for 2013‑2020. It financed the development and maintenance of Eurostat’s statistical infrastructure and the ESS. As such, the ESP’s objective was to fulfil the needs of Eurostat and the national statistics institutes, and other national authorities, as producers of European statistics, while also serving the needs of a wide range of users of these statistics.

The evaluation therefore explored the relationship between the programme’s objectives and activities and the needs of the ESS and stakeholders more generally, to provide an evidence-based judgement on the extent to which (i) the ESP was originally relevant and (ii) the activities were suitable enough to achieve the programme’s objectives. The evaluation focused on the needs of both the producers and the users of European statistics.

The findings indicate that the ESP was appropriately designed to satisfy the stakeholders’ needs, and it was found to have satisfied their needs which evolved over time. At the same time, while the ESP implemented appropriate activities to meet its objectives, the analysis showed that these activities were not enough to deliver all the statistics that users had wished for. However, some of the thematic areas that stakeholders identified as missing go beyond the scope of the ESP – so non-delivery in these areas cannot be considered as failures in the implementation of the programme.

The ESP has been confirmed as being highly relevant, clearly demonstrated by the number of data extractions made by external users from Eurostat reference databases, which greatly increased since 2014 from 6 813 000 to 21 480 000 in 2020 (+315%).

Figure 2: Number of data extractions from Eurostat’s reference databases 2014‑2020

Source: Eurostat’s website monitoring

That is further substantiated by the increase in the number of users accessing the Eurostat website, which is estimated based on the number of distinct IP addresses visiting the website, which rose from 3 441 157 visits in 2014 to 11 731 274 in 2020 (+240%).

Over the programme’s lifetime, Eurostat increased its presence on social media and the numbers of followers grew constantly. Eurostat opened its Facebook account on 10 January 2017 and its Instagram account on 4 May 2020. The number of followers on Twitter grew from 25 000 in 2013 to 164 000 in 2020, while also in 2020 the number of followers on Facebook reached 62 000 and on Instagram 9 500.

There was some tension between user needs which were continuously increasing and producers’ limitations in keeping up with new demands under an already ambitious programme. This meant that a balance had to be struck between producer capacity and user requests. In addition, needs related to the adaptation to technological advances such as big data became more prominent during the programme, as reflected when the ESP was extended. This resulted in specific activities being carried out, mainly in the context of the ESS Vision 2020 implementation projects. Such activities will continue in the next programme and their results have the potential to provide solutions to some of the problems the ESS still faces in terms of resources and costs.

3.2.Effectiveness

The analysis of effectiveness considers how successful the ESP’s activities have been in achieving or progressing towards fulfilling the programme’s objectives. The evaluation explored the relation between the programme’s objectives and its activities. It provided an evidence-based judgement on the progress made to date and the ESP’s role in delivering the observed changes, looking for evidence of whether or how these changes are linked to the programme or what factors have influenced why an activity/objective was either (partially) successful/unsuccessful or not yet achieved by the end of the programme.

The evaluation findings show that the ESP was effective in achieving its objectives despite remaining weaknesses with the timeliness and the completeness of European statistics. Between 2013 and 2020, Eurostat provided high-quality statistics, which have been used by a wide range of users for a wide range of purposes, e.g. helping to develop, monitor and evaluate policies at the EU and Member State level. In that period, the ESP also increased the availability of its data and statistics and, to a more limited extent, achieved efficiency gains in the production of its statistics.

Table 1 below shows the quantity of statistics and long time series 8 published by Eurostat. Statistical coverage is defined as the number of (statistical) indicators, sub-indicators and all their breakdowns included in Eurobase (the Eurostat dissemination database) not taking into account those differing only because of the time dimension 9 . The table thus demonstrates the increasing amount of data made available by Eurostat over the years.

Table 1: Statistical coverage in millions of statistics

Source: Eurobase

In addition, Eurostat effectively developed and strengthened partnerships with its members within the ESS as well as with those beyond the ESS, in particular international organisations. However, more needs to be done, in particular on gaining access to privately held data and creating partnerships with private organisations, something which started in 2020 and is being addressed in the next programme, the SMP.

Beyond achieving its objectives, the ESP was also effective in rendering access to official statistics easier and more user-friendly between 2013 and 2020.

3.3.Efficiency

Regarding the ESP’s efficiency, the evaluation focused on the costs of producing European statistics as well as the trends of these costs and the burden on the national statistics institutes and other national authorities. Three general issues were assessed to find out if the ESP was run efficiently. The first issue related to whether it ensured the best use of available financial and human resources. The second concentrated on the costs and burden involved in producing European statistics in the ESS and how they have evolved. The third was to assess how efficient the ESP process was for reporting and monitoring.

The analysis of the evidence suggests that the programme was efficient. The ESP demonstrated efficient use of both financial and human resources while delivering high-quality European statistics. Moreover, a higher level of statistics production was achieved during the period evaluated. Factors underpinning these developments were good governance, management and monitoring mechanisms.

Human resources at Eurostat decreased by 7% (52 people) while the ESP was being implemented. In the same period, the number of the annual datasets published by Eurostat increased by 18%. This means that Eurostat staff increased their productivity as regards providing statistics by over 20%.

Table 2: Number of Eurostat staff and published datasets at the end of years 2013‑2020

Source: Eurostat

The Internal Audit service (IAS) audited Eurostat’s quality management of statistical processes. The audit found aspects that needed to be improved and the IAS formulated two recommendations. Eurostat implemented an action plan to address them. The IAS concluded in its follow-up that Eurostat had adequately and effectively implemented the first recommendation. Eurostat will implement the second one by 31 December 2021, as scheduled in the action plan.

There were concerns over the costs for Member States and the administrative burden placed on data providers, due to the need to produce an increasing volume of statistics, which required steadily raising the efficiency of statistics production. Several initiatives were undertaken to respond to this challenge, such as:

·carrying out the activities within the ESS Vision 2020;

·modernising business statistics resulting in the European Business Statistics Regulation 10 , social statistics 11  resulting in the Integrated European Social Statistics Regulation and agriculture statistics resulting in the Integrated Farm Statistics Regulation 12 ; and

·strengthening partnerships within the ESS.

3.4.Coherence

This section focuses on the internal and external coherence of the ESP. The internal coherence of a programme is determined by looking at how the various components of that programme operate together to achieve the intended objectives. The external coherence of a programme relates to the extent to which the programme activities align with other activities of the EU and its international partners.

The evaluation analysed where and how the programme objectives and activities worked internally and externally in a coherent manner and points to areas where there were tensions.

No overlaps or inconsistencies were identified at the programme level between the objectives and among the activities listed within the ESP’s regulations and planning documents (internal coherence). The evaluation identified various governance bodies and advisory boards with a coherent mandate. Their work contributed to the internal coherence of the ESP by ensuring that its activities were in line with its objectives. Furthermore, interviewees involved in producing European statistics highlighted the internal coherence of the ESP.

The work carried out under the ESP was in line with the Commission’s priorities. Relevant European statistics corresponded to the European Commission’s 10 general objectives/priorities and the ESP’s sub-priority areas were in line with the Juncker Commission priority areas.

Eurostat cooperated effectively with EU bodies and agencies, as well as international organisations. This cooperation resulted in the external coherence of the ESP by ensuring it was aligned with the wider EU objectives and needs as well as with international statistical activities. It also ensured the development of comparable and harmonised European statistics at the regional and international level.

The IAS carried out two audits, one on the production process and the quality of statistics not produced by Eurostat and the other on the effectiveness of Eurostat’s cooperation with external stakeholders. Both audits found aspects that needed to be improved and the IAS formulated some recommendations. Eurostat implemented an action plan to address both sets of recommendations and the IAS concluded in their follow-ups that Eurostat had adequately and effectively implemented them.

However, the evaluation identified some weaknesses that could potentially affect the ESP’s external coherence:

·the lack of flexibility of European statistics to respond to emerging needs (which might cause a misalignment with other EU strategies should these needs not be covered);

·the burden on Member States when replying to individual DG requests for statistics; and

·the need to gain access to privately held data and create partnerships with private organisations (for which activities began in 2020).

They are now being tackled and will continue to be addressed in the next programme.

Overall, the findings suggested that the ESP was internally and externally coherent.

3.5.EU added value

The EU added value criterion looks at the value resulting from the ESP that is additional to the value that would have resulted from similar activities only carried out at national level or by individual Member States. This criterion is linked to the principle of subsidiarity: the EU should only act when it can achieve better results than the Member States acting alone. Here, the analysis focused on the added value of the ESP to both users and producers of European statistics.

The evidence collected confirmed the EU added value of the ESP. It suggested that the programme’s main added value was the way it contributed to harmonising European statistics to deliver comparable statistics, provide them in a single location and support evidence-based policy making at EU and national level. The ESP produced high-quality, comprehensive, comparable and reliable statistics across all Member States.

Without the ESP, evidence-based policymaking at EU level and policymaking at country level, based on comparative analysis of statistics across countries, would be very difficult (due to potential inconsistencies in the evidence).

Users from EU institutions, Member States and candidate countries confirmed the EU added value of providing European statistics in a single online location (‘one-stop shop’).

The programme also established a clear roadmap for statistics production for producers, including candidate countries. Through cooperation between Eurostat and international organisations, the ESP also helped strengthen the international statistical community and increase efforts to ensure the quality of statistics at the international level.

3.6.Specific points of Article 15

The answers to the four specific points mentioned in Article 15 of the ESP Regulation are based on the evidence collected and the analysis carried out for the five evaluation criteria. The conclusions of the evaluation for those four specific aspects are summarised below:

a)‘The outcome of the reprioritisation and cost evaluation of statistical products’

Despite improvements, prioritisation remained a challenge throughout the programme resulting in increased pressure on Member State resources. This was due in large part to the increased data needs expressed by users of statistics and the lack of identification of statistics that would have become less relevant over time. To solve the issue, a review of statistical requirements in existing domains of European statistics and a new prioritisation mechanism are being implemented in the statistical programme for 2021-2027 set out by the SMP4. The new mechanism is based on four components: a multiannual action plan, reviews, user dialogue, and resources/funding.

As detailed in section b below, the assessment of costs for the ESS showed that the total costs for producing statistics decreased slightly between 2013 and 2020, as did the number of staff – despite producers meeting an increasing number of statistics requests.

b)‘The actions taken by the ESS to reduce the implementation and production costs for Member States and to limit the overall burden stemming from the statistical projects and fields covered by the programme’

Production costs and the number of staff went down while the number of datasets produced increased. This shows that the actions taken by the ESS to reduce the production costs for Member States and to limit the overall burden stemming from the statistical projects and fields covered by the programme have been effective. The most important actions included: (i) the implementation of framework statistics regulations, (ii) increased use of administrative and register data, (iii) new estimation methods, and (iv) the application of microdata exchanges.

Table 3: Costs estimates and staff as full time equivalents (FTEs) in the ESS

Source: Surveys on cost of statistics in the ESS

At the same time, qualitative feedback highlighted limitations in terms of priority setting and in identifying statistics that would have become less relevant over time. In addition, concerns over the diminishing number of staff, coupled with the increase in the data needs expressed by users for new statistics, suggested that issues with production costs and burden may emerge in the future.

c)‘The progress on rendering access to official statistics easier and more user‑friendly, including the provision of data on the Eurostat website’

The ESP helped render access to official statistics easier and more user-friendly, but there is still scope for further improvement.

In 2014, Eurostat’s website was completely redeveloped and has been continuously enhanced ever since. Many new electronic and interactive publications, visualisation tools, mobile apps and tools offered for data extraction have been developed and added since the ESP began. These tools present data from different statistical themes in an attractive and easy-to-understand way for everyone to explore and can be easily accessed on Eurostat’s website.

Eurostat has also been active on social media, with its three corporate accounts on Twitter, Facebook and Instagram, helping it to communicate in a more modern way.

Providing data on the Eurostat website was key but the ease and user-friendliness of the access to these data differed depending on the category of users. While data and statistics were easily accessible to proficient users from the Commission, access was more challenging for less experienced users from the general public, due to the complexity of the website’s content.

d)‘The progress on the improvement of data availability, including on social economy activities and on the Europe 2020 indicator

From 2013 to 2020, the ESP included activities to develop new statistics and helped improve data availability, including on social economy activities and on the Europe 2020 indicators. On social economy activities, data and statistics on population and social conditions, as well as on migration, became more widely available, including an increased disaggregation by migratory status. Despite these improvements, gaps remained in terms of territorial disaggregation of social statistics. On the Europe 2020 indicators, Eurostat continuously provided comparable, reliable, and timely statistical information, and work continued on developing and producing these indicators throughout the programme. While the Europe 2020 indicators are only available at the EU and country levels because there is no territorial disaggregation of the targets set, some of the data used for these indicators are available at regional level. The Europe 2020 indicators set – used to monitor the Europe 2020 strategy for growth and jobs between 2010 and 2020 – was updated yearly and is available for reference periods until 2021. As the Europe 2020 strategy has reached the end of its life cycle and since the EU aggregate (28 countries including the United Kingdom), for which the Europe 2020 targets were set, no longer exists in official statistics, the Europe 2020 indicators were removed from the Eurostat reference database and archived. However, all five areas covered by the Europe 2020 indicators remain part of the EU Sustainable Development Goals indicator set and will continue to be updated under the heading of ‘ Sustainable development indicators ’.

4.Recommendations

Focus on innovation, new methods and new data sources

Innovation is key to satisfying the increasing demands for new and timelier statistics while reducing production costs and administrative burden. Building on the lessons learned from the pilots implemented under the ESP, work must focus on sustained innovation in the most promising areas. Based on the final report of the expert group on facilitating the use of new data sources for official statistics and the activities of the European Commission, the work started in 2020 to gain access to privately held data should be completed.

Further improve access to European statistics via Eurostat’s website

The functionality of Eurostat’s website should be further developed by updating the search engine and improving its navigability, especially for non-expert users and considering the different user groups of European Statistics. The accessibility and quality of metadata should also be improved.

Implement the new approach to prioritisation

In February 2020, the ESS Committee underlined the importance of an effective priority-setting mechanism and endorsed the proposed priority-setting approach with its four components: a multiannual action plan (MAP), reviews, user dialogue, and resources/funding. The new approach should be thoroughly implemented to support the objective of ensuring the continued relevance of the statistics produced while minimising the burden on respondents and producers.

(1)

OJ L 87, 31.3.2009, p. 164.

(2)

OJ L 39, 9.2.2013, p. 12.

(3)

OJ L 284, 31.10.2017, p. 1.

(4)

OJ L 153, 3.5.2021, p. 1.

(5)

 First mid-term evaluation: https://europa.eu/!tcF9J4 ;

Second mid-term evaluation: https://europa.eu/!QCyfpB .

(6)

Users include different categories, such as decision-makers at EU, national and local level; international organisations; academia and researchers; businesses; NGOs; media and European citizens in general. Producers or are mainly national statistical institutes and other national authorities.

(7)

The decrease between 2017 and 2018 was due to the implementation of the new datasets in national accounts in 2017, following the new European System of Accounts (Version 2010) legislation, while keeping those following the previous system (Version 1995). The datasets following the previous legislation were removed in 2018.

(8)

Long time series are defined in this case as those covering 10 or more consecutive years.

(9)

The derived datasets are not counted, but the variables included in more than one primary dataset (e.g. GDP and population) are counted as different statistics.

(10)

Regulation (EU) 2019/2152.

(11)

Regulation (EU) 2019/1700.

(12)

Regulation (EU) 2018/1091.