EUROPEAN COMMISSION

Brussels, 27.9.2024

SWD(2024) 222 final

COMMISSION STAFF WORKING DOCUMENT

Evaluation of EU Agencies: Eurofound, Cedefop, ETF and EU-OSHA

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL AND THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE

Evaluation of EU Agencies: Eurofound, Cedefop, ETF and EU-OSHA

{COM(2024) 414 final}

Table of contents

1.INTRODUCTION

1.1.    Purpose and scope of the evaluation    

1.2.    Evaluation methodology and limitations    

2.WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION

2.1.    Description of the intervention and its objectives    

2.2.    Point(s) of comparison    

3.HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD?

3.1.    Financial and human resources    

3.2.    Delivering outputs and activities    

3.3.    Factors facilitating/hindering effectiveness    

4.EVALUATION FINDINGS

4.1.    To what extent was the intervention successful and why?    

4.2.    How did the EU intervention make a difference and to whom?    

4.3.    Is the intervention still relevant?    

5.WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED?

5.1.    Conclusions    

5.2.    Lessons-learned    

Annex I: Procedural Information

Annex II: Methodology and Analytical models used

Annex III: Evaluation matrix

Annex IV: Overview of benefits and costs

Annex V: Stakeholder consultation – Synopsis report

Annex VI: Agencies intervention logics

Annex VII: FIGURES AND TABLES

Annex VIII: TECHNICAL ANNEX

Annex IX: TECHNICAL ANNEX (II)

Annex X: List of indicators per agency

Annex XI: Cost-effectiveness of the Agencies’ key activities

Annex XII: Results achievement

Glossary

1.1.Purpose and scope of the evaluation

This document evaluates four of the five EU decentralised agencies that fall under the remit of DG EMPL: the European Foundation for the Improvement of Living and Working conditions (Eurofound); the European Centre for the Development of Vocational Training (Cedefop); the European Training Foundation (ETF); the European Agency for Safety and Health at Work (EU-OSHA) (‘the Agencies').

Since it was set up in 2019, the European Labour Authority (ELA) has also been under the remit of DG EMPL. However, due to its early stage of development and the on-going 2024 evaluation as required by its Founding Regulation, ELA does not fall within the scope of this evaluation. Nonetheless, ELA stakeholders have been consulted throughout this study, and the alignment of the mandates between the Agencies and ELA has been taken into account.

According to their Founding Regulations 1 , each of the Agencies should be evaluated at regular intervals. The previous joint evaluation of the Agencies took place in 2019 2 . The Commission evaluation report was published on 9 April 2019 3 along with the supporting study 4 . The Agencies also have their own evaluation programmes.

In line with the Commission’s ‘better regulation’ guidelines, this evaluation analyses the effectiveness, efficiency, relevance, coherence and EU added value of each Agency, assessing to what extent they have fulfilled their mandates, specific objectives and activities. It includes individual and cross-cutting assessments, with an emphasis on synergies, cooperation and how the Agencies complement one another. The evaluation seeks to provide background information and insights to feed into future discussions on the Agencies, and if necessary, to contribute to potential legislative proposals responding to evolving needs or circumstances, including changes in their mandates.

Scope of the evaluation

The evaluation covers 2017-2022 5 , and includes the entire thematic remit of the Agencies: employment, industrial relations and the improvement of living and working conditions (Eurofound); health and safety at work (EU-OSHA); vocational education and training, as well as skills and qualification policies (Cedefop and the ETF). The evaluation also addresses the role of each Agency in dealing with the COVID-19 pandemic, as well as Russia’s war of aggression against Ukraine, where relevant.

In terms of geographical scope, the evaluation covers the 27 EU Member States, as well as:

- for the ETF: activities in partner countries, including in Eastern Partnership countries, Central Asia, Southern and Eastern Mediterranean, Western Balkans and Türkiye 6 ;

- for Eurofound: surveys in non-EU Member States 7 ;

- for Cedefop: the contribution of Norway and Iceland to Cedefop’s budget and activities 8 ;

- for EU-OSHA: the participation of Iceland, Lichtenstein and Norway in EU-OSHA's work (European Commission and EFTA agreement); the projects run by EU-OSHA in the Western Balkans and Türkiye (under the Instrument for Pre-accession Assistance - IPA).

1.2.Evaluation methodology and limitations

The evaluation was supported by an external study (supporting study) 9 . The supporting study used a variety of primary and secondary data collection tools as well as triangulated analysis: mapping research based on the Agencies' management and monitoring documents, other studies and reports related to the Agencies, stakeholder feedback surveys (commissioned by the Agencies), a wide range of consultation activities including interviews with and surveys of key stakeholders and staff working in the Agencies.

The supporting study 10 included 20 Agency-specific and five cross-cutting in-depth case studies. It also made use of a European Commission public consultation published on ‘Have your say’ to collect feedback on the evaluation criteria covered by the study. A cost-effectiveness analysis focused on assessing Agency-level and activity-level cost-effectiveness. The cross-cutting analyses 11 assessed the mandates and cooperation between the Agencies and provided significant input to answer the coherence questions. Finally, a validation focus group with key stakeholders took place on 18 September 2023 to test preliminary evaluation findings.

Limitations to the evaluation research are associated with the scope, research coverage, data availability and data quality. Many of these limitations stem from the Agencies’ monitoring systems and key performance indicators (KPIs) which restrict the assessment of performance and achievement of objectives. These constraints were taken into account during the study’s design and implementation, with mitigation measures introduced accordingly.

1.2.2.1. Agencies monitoring systems and KPIs limitations:

Overall, the Agencies have numerous and similar mechanisms to ensure accountability and assess performance, including: monitoring: internal performance monitoring/measurement systems, mechanisms to track risks; reporting: single programming documents and multiannual strategies, annual reports and consolidated annual activity reports, annual budgets and accounts, establishment plans, management board and executive board meeting minutes; evaluation: reports by ECA, EP, and IAS, internal and external evaluations and controls, staff surveys and internal user/stakeholder surveys (except ETF).

Despite growing cooperation between the Agencies to increase the alignment and quality of their monitoring systems, notable differences and limitations persist:

Limited comparability of KPIs among the Agencies: Although each Agency collects a large amount of data, these data are not always comparable due to variations in methodology when defining and quantifying KPIs. To address this, priority was given to KPIs that are common to most Agencies; proxy indicators were used where possible. Qualitative information from other data collection methods was also used to triangulate the findings (e.g. Cedefop does not use a quantitative indicator for work programme delivery so alternative qualitative information was used, including European Parliament discharge reports).

Result indicators and benchmarking limitations to assess success:

(i) The Agencies did not consistently set targets for performance indicators, which made it difficult to analyse whether the objectives had been achieved. While targets were set for some operational KPIs (e.g. budget implementation and output delivery), this was not the case for most result indicators (see Annex X for a list of Agencies’ KPIs with and without targets). In such instances, the study analysed trends over time, comparing achievements with 2016 baseline figures or with data from the entire 2011-2016 period when available.

(ii) ETF and Cedefop changed their approach to monitoring indicators during the evaluation period making it difficult to analyse some trends 12 . In 2021, the ETF changed its approach to monitoring indicators with the introduction of the 2027 ETF strategy, which reduced data comparability. In those cases, indicators are used to a limited extent and explanations are given based on triangulated evaluation sources. 

(iii) In some cases, the ETF set targets, but the ETF consolidated annual activity reports do not provide sufficient information for assessing the achievement of those quantitative targets. As a result, there is a disconnect between the specified targets and the description of what was achieved 13 .

(iv) Some of the proposed indicators are no longer relevant for assessing success (e.g. number of press releases).

Lack of KPIs on country-level impacts: The data on results and impacts at national level is limited and is not systematically monitored by the Agencies. The responses to the public consultation and stakeholder survey were insufficient to analyse the results by country. To address this, qualitative information collected through interviews, case studies and mapping tasks was used to identify examples of uptake, use, and impact of Agencies activities at national level. Evaluations of specific programmes and activity areas commissioned by the Agencies were also used (e.g. the evaluation of the EU-OSHA OiRA tool was used to identify uptake and use at national level alongside information from the OSH-wiki website).

1.2.2.2. Other limitations

Mapping methodology: The mapping task, based mainly on the Agencies’ monitoring and evaluation systems, had limited scope to incorporate outsider views. Interviews, a public consultation, a targeted survey and case studies were therefore needed to ensure that a variety of perspectives could be considered in the evaluation.

Limitations of the public consultation results: The public consultation received only 101 responses across all four Agencies, limiting the scope to analyse responses by stakeholder group. To address this, the supporting study included a targeted survey of Agency staff and stakeholders which generated 560 responses. This was sufficient to enable staff and stakeholder responses to be analysed separately, with the latter further broken down by replies from Management/Governing Board members and other stakeholders. This allowed a differentiation between those involved in the Agencies’ operations and those using the Agencies’ outputs. Public consultation and targeted survey data were also triangulated with all other data sources collected during the study.

Potential bias from evaluation tools based on perceptions: The evaluation collected staff and stakeholder views through semi-structured interviews, a targeted survey and a public consultation. To address any potential bias, the study considered that opinions were influenced by the stakeholders’ relationships with the Agencies. The perception-related data were therefore triangulated with other data sources. For instance, in the case of complex issues such as the functioning of the Management/Governing Boards, in-depth interviews with both direct participants and informed outsiders were used to inform the findings.

Limitations of the cost effectiveness analysis methodology: Typically, a cost-effectiveness analysis assesses if the cost of the activities match the achieved outputs and results. Yet, for many of the Agencies’ activities, it was not possible to identify a single effect or indicator that could be easily quantified. Unit cost analysis at the level of outputs was difficult as each Agency is unique, with its own remits, objectives and activities, making it difficult to apply a set of comparable performance indicators. Consequently, the performance of the Agencies had to be considered against multiple objectives rather than the achievement of a sole output by a given date. To address these challenges, a mixed-method approach was used to gather information through desk research, interviews, surveys and case studies. The goal was to evaluate production efficiency (the relationship between inputs and outputs), allocation efficiency (the relationship between inputs and higher-level results) and how efficiency could be improved.

Challenges linked to specific evaluation criteria: The effectiveness criterion assesses impacts which are affected by multiple factors that are often outside of the control of the Agencies. In-depth analyses of selected initiatives in case studies addressed these challenges. Moreover, the Agencies’ specific objectives, based on their programming documents, are not formulated in a SMART way, which makes it difficult to assess the extent of the success of the interventions. The efficiency criterion was affected by the limitations of the cost-effectiveness methodology presented above.

2.1.Description of the intervention and its objectives

This evaluation builds on the intervention logics (ILs) of the Agencies. Each IL includes a complex set of operational, specific and general objectives for each Agency. These objectives are mirrored, respectively, by activities/outputs, results and impacts, which are the core of the Agencies’ effectiveness and efficiency assessment (see Section 4.1.1.).

The objectives and tasks of the Agencies are set out in: (i) their Founding Regulations – which set the mandates/general objectives; and (ii) their programming documents for the relevant period (midterm perspectives and work programmes) - which set specific and operational objectives. Detailed intervention logics per Agency are presented in Annex VI, along with full references to the Agencies’ key strategic programming documents.

To fulfil their objectives19, the Agencies carry out several broad types of activity, depending on their specific mandate and intervention logic, namely: (i) research and monitoring; (ii) communication, dissemination and raising awareness; and (iii) capacity building.

During the current evaluation period several external factors need to be considered in the Agencies’ ILs, as they influenced the achievement of results and impacts. The most significant factor, common to all four Agencies, is the impact of the COVID-19 pandemic. Agency-specific external factors include the level of engagement of national focal points (for EU-OSHA), the Network of National Correspondents - NEC (for Eurofound), the geopolitical context and the level of engagement of partner countries (for the ETF).

The mandates of the Agencies, which correspond to the general objectives of the ILs, are presented below (in bold). Detailed specific objectives per Agency, are presented in Annex VI. Operational objectives are presented in Table 2.

EUROFOUND - Eurofound was established in 1975 by Regulation (EEC) No 1365/75 14 . Together with Cedefop and EU-OSHA, it was one of the first three EU Agencies to be created. Located in Dublin, Ireland, its main mission is to carry out research in the areas of employment, industrial relations, and living and working conditions, with the objective of shaping and implementing policies aimed at improving living and working conditions.

The 2021-2024 programming document of Eurofound sets out six strategic areas 15 on which to focus: (1) working conditions and sustainable work; (2) industrial relations and social dialogue; (3) employment and labour markets; (4) living conditions and quality of life; (5) anticipating and managing the impact of change; and (6) promoting social cohesion and convergence. Eurofound operates three observatories: (1) the European Observatory of Working Life (EurWORK - working conditions and industrial relations); (2) the European Monitoring Centre of Change (EMCC - monitoring of change for employment); and (3) the European Observatory on Quality of Life (EurLIFE - living conditions and quality of life).

Eurofound conducts regular surveys: the European working conditions survey (EWCS), the European company survey (ECS), the European quality of life survey (EQLS), and living and working in relation with COVID-19. The Agency also carries out individual research projects. In its field of competence, it contributes to the analytical and policy work of the European Commission and Member States, and of organisations representing both employers and employees. In 2023, Eurofound had around 100 staff members (own staff only) and a total annual budget of EUR 23.8 million, including an operational budget of EUR 6.1 million (programming document 2021-2024).

CEDEFOP - Cedefop was established in 1975 by Council Regulation (EEC) No 337/1975 16 . Located in Thessaloniki, Greece, its main mission is to support the promotion, development and implementation of evidence-based, high-quality EU policies on vocational education and training (VET). Cedefop provides information, research, analyses and evidence at the crossroads of VET, skills and qualifications, and the labour market. More specifically, Cedefop monitors and analyses developments in these three areas. It contributes to European tools in the area of education and training, such as the European Qualification Framework and Europass. It provides skills analysis and forecasting together with tools for validating non-formal and informal learning. As a platform for policymakers, social partners, researchers, experts and other VET and labour market actors, Cedefop promotes knowledge sharing and policy learning. It plays a key role in monitoring the implementation of the Council Recommendation on VET and the Osnabrück Declaration. To support its work, Cedefop manages several networks, including ReferNet, CareersNet, SkillsNet, and a community of apprenticeship experts. In 2023, Cedefop had 120 staff 17 and an annual budget of around EUR 19.6 million, including an operational budget of around EUR 4.9 million (programming document 2023-2025).

ETF – the ETF was established under Council Regulation (EC) No 1360/90, replaced in 2008 by Regulation (EC) No 1339/2008 18 . Located in Torino, Italy, it focuses on EU policy and cooperation instruments for external relations, which in 2021 were brought together under the umbrella of Neighbourhood, Development, and International Cooperation Instrument – Global Europe (NDICI - Global Europe). Its main mission is to contribute to EU external relations policies, notably on helping partner countries to develop human capital through improved VET and lifelong learning systems that are inclusive and innovative.

The ETF’s geographical scope includes the EU candidate countries, the potential candidate countries in the western Balkans, the southern Mediterranean, eastern European and southern Caucasus partner countries, and other countries, as decided by the ETF Governing Board. In 2023 it had around 129 staff and an annual budget of EUR 22.63 million, including an operational budget of EUR 4.33 million (ETF annual work programme 2023).

EU-OSHA – EU-OSHA was established in 1994 by Council Regulation (EC) No 2062/9431 19 . Located in Bilbao Spain its main mission is to develop, gather and provide reliable and relevant information, analyses and tools to advance knowledge, raise awareness and exchange occupational safety and health (OSH) information and good practices.

The Agency collects and disseminates information and develops tools on health and safety at work relevant for stakeholders and policymakers involved in OSH at EU and national level. It also carries out awareness raising campaigns and networking in the OSH field. EU-OSHA anticipates change as well as new and emerging risks. It develops tools for good OSH management. It has a network of focal points in Member States, which provides input into EU-OSHA’s work and disseminates products and information to national stakeholders. In 2023 it had around 65 staff and an annual budget of EUR 17.04 million, including an operational budget of EUR 7.065 million (programming document 2023-2025).

2.2.Point(s) of comparison 

Given that the evaluation period is 2017-2022, the baseline year is 2016. The supporting study has therefore taken into consideration the key findings of the previous evaluation (2011-2016). In addition, and where possible, data relating to indicators comparable with those assessed previously has been collected. For specific evaluation criteria (e.g. efficiency), other similar EU decentralised agencies have been used as benchmarks; for effectiveness, extensive use is made of the Agencies monitoring systems and KPIs (see Annex X - list of KPIs per Agency), comparing achievements to targets when the latter exist.

3.1. Financial and human resources

The Agencies, similarly to other EU decentralised agencies, are almost exclusively funded by the general EU budget, with annual contributions from the cohesion, resilience and values headings of the multiannual financial framework (annual subsidy to each Agency). 21 According to their Regulations, all four Agencies can also receive (i) fees charged for their services and publications, (ii) funding from the Member States and third countries participating in the Agencies’ activities 22 , and (iii) internal and external revenue 23 assigned for specific items of expenditure 24 .

According to their Founding Regulations, Eurofound, Cedefop and EU-OSHA can also receive (iv) other funding from the European Commission (through contribution agreements, grant agreements and financial framework partnerships), provided that specific conditions are met 25 . While the ETF’s Founding Regulation does not mention these agreements explicitly, it permits funding from other sources as well (e.g. contribution and grant agreements with the European Commission, if they are covered by mandates and are not already funded).

In practice, all four Agencies received most of their funding from the general EU budget (subsidy). The funding received from other sources 26 represented less than 5% of their total annual revenue 27 . The ETF is an exception as in 2022, it received additional funding totalling about EUR 3 million (approx. 13.8% of the ETF budget) through extra-subsidy projects, mainly from agreements between the European Commission and the Foundation (e.g. with EU Delegations). This additional funding enabled the ETF to increase operational expenditure, improve visibility and amplify the impact of its operations (see details of these additional projects delivered by the ETF in 2022 in Box 1 of Annex VII). In the case of Eurofound, Cedefop and EU OSHA, some extra-subsidy funding comes from EFTA countries (Iceland, Lichtenstein and Norway), which in exchange participate in some agencies’ activities 28 .  Eurofound has also received funding from a European Parliament Pilot Project dedicated to minimum wages 29 . 

The annual contribution to all four Agencies from the EU budget remained stable during the evaluation period, and similar to the baseline. In 2019, a slight (2%) increase was introduced to the MFF to partially compensate for inflation. However, this modest rise in annual revenues could not keep pace with the actual level of inflation and rising staff costs. Consequently, the proportion of staff costs increased during the evaluation period, leading to a decline in the share of operational expenditure 30 .

The Agencies faced significant pressure on their operational budgets, which limited their operations. For example, Cedefop reduced activities directly targeting Members States, and in 2022, EU-OSHA reduced activities related to the priority areas ‘Facts & figures’, ‘Raising awareness and communication’, and ‘Networking’ 31 . 

Between 2017 and 2022, the Agencies implemented about 100% of their annual budget, similar to the previous evaluation period (2011-2016), indicating a high operational efficiency (see Section 4.2 on efficiency). Budget implementation was slightly lower for EU-OSHA, but remained above the target of 95%. This can be explained by the large share of the operational budget which is implemented through procurement actions 32 .

Table 1: Annual budget implementation (%)

Source: Annual activity reports

The total number of staff 33 of the Agencies remained stable during the evaluation period (see Figure 1 and Table 1 in Annex VII). EU-OSHA is the smallest Agency with 65 staff in 2022. The ETF was the largest with 134 staff in 2022 down from 141 in the baseline year (2016). Cedefop staff numbers decreased from 121 in 2016 to 116 in 2022. Staff numbers at Eurofound remained largely the same at 110 in 2016 and 108 in 2022.

The Agencies completed in 2018 (EU-OSHA in 2017) the 10% staff cut – a policy introduced in 2013 for decentralised agencies, applicable to permanent staff (included in the establishment plans) 34 .

3.2.Delivering outputs and activities 

The Agencies largely achieved their annual programme output targets despite budgetary and staff limitations and, in the second half of the evaluation period, the COVID-19 pandemic 35 .

EUROFOUND - Eurofound exceeded 80% of its work programme delivery target throughout the evaluation period. Delivery was lower than the baseline between 2017-2019; this was linked to staff cuts of 10% in 2017, which resulted in delays of some outputs and a reallocation of priorities. Eurofound revised its work programme, adapted to remote working and completed the activities during the last months of the financial year, despite initial delays 36 .

EU-OSHA – EU-OSHA’s work programme completion rates were higher than the baseline, as well as its 90% target. Some outputs were cancelled in 2020 due to the pandemic, but this did not affect the overall completion rate, which was very high in 2020 (96%).

ETF – since 2016, the ETF’s work programme delivery remained above its 90% target, except in 2020 due to the COVID-19 pandemic. The pandemic severely impacted Agency operations due to an early lockdown in Italy and the Agency working in third countries. Most of the outputs were re-programmed for the last quarter of the year, with some of them being postponed, due to pressures on staff capacity.

CEDEFOP – Cedefop does not have a specific quantitative indicator to monitor the completion rate of its work programme. However, based on the supporting study’s assessment of the outputs, the Agency has fully implemented the annual work programmes. This includes 2020, when Cedefop even exceeded its 2020 work plan targets, as noted by the European Parliament’s discharge decision 37 .

The Agencies have carried out their activities in line with their operational objectives (see Table 2). Each Agency collects and monitors the data on the outputs produced per year but the level of detail, type of outputs and definitions vary between the Agencies (see list of each Agency’s indicators in Annex X). For a detailed analysis on the delivery of each operational objective, see Annex VIII and Section 4.1.1.2 of the supporting study as well as Annexes 1, 2, 3 and 4 of the supporting study on the specific situation in each Agency.

Figure 1: Work programme delivery: Eurofound, ETF and EU-OSHA  38

Source: Consolidated annual activity reports for Eurofound, ETF and EU-OSHA 2016-2022

3.3.Factors facilitating/hindering effectiveness

Facilitating factors

At a general level, the most significant internal facilitating factors contributing to the effectiveness of the Agencies’ operations were: (1) flexibility to adapt to changing circumstances, as a result of proactive internal measures 39 ; and (2) tripartite representation in the three tripartite Agencies, which increased the reliability and credibility of the Agencies   vis-à-vis stakeholders. 

The COVID-19 pandemic could be considered as an external facilitating factor. The shift to online work not only generated efficiency savings, but also allowed the Agencies to enhance their visibility by offering information and guidance on COVID-19 related matters. As a result, audiences also became aware of the broader work that the Agencies carry out (e.g. previous Eurofound work on youth and gender) 40 .

For individual Agencies, internal facilitating factors include Eurofound’s Network of Expert Correspondents, the level of engagement of ETF’s partner countries, as well as of EU-OSHA’s focal points in Member States.

Hindering factors

In general, the most significant internal hindering factor was the lack of financial resources. Staff costs increased for all Agencies due to inflation adjustments, as did administrative expenditure (except for ETF). As a result, and given that the budget remained stable, the operational expenditure decreased. Confronted with the need to address new EU policy priorities, Agencies had in some cases to revise their activities in order to deliver on their work programmes.

Other internal hindering factors included lack of human resources, lack of time to prepare data and finalise outputs, and accessibility issues on outputs linked to the quality of translations.

The COVID-19 pandemic was another external hindering factor. It led to live events having to be cancelled and disrupted work, including surveys.

For individual Agencies, hindering factors included insufficient tailoring of outputs for specific target audiences (ETF, EU-OSHA), unforeseen delays with external contractors (EU-OSHA), translation issues (EU-OSHA 41 , ETF 42 ) and a need to improve engagement with training (Eurofound). For the ETF, insecurity, political instability and economic volatility were also among the external factors that limited the engagement of partner countries.

4.1.To what extent was the intervention successful and why?

The evaluation has assessed the effectiveness, efficiency and coherence of the work of the Agencies during the current 2017-2022 period, confirming for the three criteria the overall positive assessment given in the previous evaluation (2011-2016), though with room for improvement in some areas (see in particular Section 5.2. on lessons learnt).

The evaluation has assessed the effectiveness of the Agencies in achieving the objectives outlined in their Founding Regulations and programming documents and as reflected in their intervention logics (see Annex VI). The evaluation assessed to the extent possible whether the Agencies successfully met their operational and specific objectives in terms of outputs and results (see Section 4.1.1.1) and produced impacts for EU policymaking or policy implementation at national level (see Section 4.1.1.2). In addition, it assessed the Agencies ability to adapt to changes and, notably, their responses to the challenges posed by the COVID-19 pandemic and Russia’s war of aggression.

4.1.1.1 Achievement of objectives

Overall, and based on the available evidence, it can be concluded that the Agencies have achieved many of their operational and specific objectives to a high degree 43 . There are however opportunities for improvement in certain areas.

A) outputs (operational objectives - activities)

Overall, the Agencies largely delivered the outputs planned in their annual programming documents, as shown in Section 3.2. They managed to do this in spite of staff and budget constraints, and in the difficult context of the COVID-19 pandemic. 

Some delays occurred for Eurofound (mainly due to staffing issues) and EU-OSHA (due to the editing of final outputs taking longer than expected and delays in getting the deliverables from contractors), as well as for Cedefop and the ETF (mainly due to the COVID-19 pandemic).

Table 2: Agencies operational objectives

Source: Intervention logics of the Agencies, see Annex VI

EUROFOUND - Eurofound successfully delivered the expected outputs for all its objectives. The Agency performed well in carrying out research and collecting data (desk research, KPIs 44 and confirmed by consultations -see below stakeholders’ views on quality of outputs). The core of the Agency’s work relies on its pan-European surveys, 45 which were delivered as planned. For instance, the 2020 European working conditions survey (EWCS) covered 37 countries, two more than the 2015 edition. Additionally, a follow-up telephone survey in 2021 assessed the impact of COVID-19 on job quality. The 2019-2020 European company survey (ECS), carried out in cooperation with Cedefop, was the first large-scale, cross-national survey introducing a ‘push-to-web’ approach 46 . Due to COVID-19, Eurofound adopted computer-assisted telephone interviewing (CATI) instead of face-to-face interviews for surveys. This shift prompted the Agency to explore alternative data collection methods to save costs in the long run, without compromising the quality 47 . 

CEDEFOP - Cedefop successfully delivered the expected outputs for all its objectives in VET, skills and qualifications. Cedefop performed particularly strongly on its operational objective of ‘carrying out research on topics relevant to the policy agenda’ – especially on ‘Skills anticipation and matching’- as well as on its objective of ‘Monitoring and analysing VET policy developments. Cedefop publications were frequently referenced in EU policy documents (with 524 and 449 references respectively) 48 . Although the overall number of publications slightly decreased during the evaluation period (compared to 2016), this was a deliberate shift from traditional to online content, enhancing accessibility through interactive tools and databases.

In terms of compiling relevant data, Cedefop made strides by developing online tools and databases 49 , such as Skills Forecast, the European database on apprenticeship schemes, and the European VET policy dashboard. This move aimed to enhance user interaction and accessibility. Cedefop also contributed to capacity building through events and policy learning activities, albeit to a lesser extent than in 2016, with a reduction in events starting before 2020 due to the COVID-19 pandemic 50 .

Finally, Cedefop successfully supported the development and implementation of European tools to support VET, skills and qualifications policies.  It produced the expected outputs in its support of EQF, ECVET and EQAVET, and played a crucial role in the transition to the new Europass. 

ETF – the ETF successfully achieved its five operational objectives, focusing on partner countries in the area of EU external relations. To provide information and policy analysis, the ETF produced various informative materials, including reports, policy briefings, guides and toolkits, offering valuable insights into human capital development in partner countries. Stakeholders assessed this ‘information and policy analysis’ objective positively (see Figure 8 in Annex VII).

ETF’s support on capacity building and networking grew significantly with an increased number of events held compared to the baseline 51 (a notable rise from 24 events in 2017 to 143 events in 2021). ETF stakeholders viewed its support to partner countries as largely or partially successfully, according to the 2018 Policy Delphi (partner country governments, social partners, EU Member States, institutions, international organisations) 52 . 

The fifth round of the Torino Process (2018-2020) covered 100% of partner countries, while the sixth round (2022-2024) adopted a new approach based on two separate stands. The first strand focuses    on the collection of statistical data and indicators, covering all partner countries. The second strand consists of an in-depth review (known as ‘Level 2’) which became voluntary in 2022 53 .  Stakeholders, especially partner countries, highly valued the Torino Process for its role in facilitating policy dialogue, guiding education and training strategies, as well as evaluating ongoing reforms (see ETF's Case Study 5 in Annex 8 to the supporting study). As participating in the ‘in-depth country policy review’ has become voluntary, the ETF now focuses efforts on those countries that are motivated to participate, ensuring that ETF’s resources are used where they can have the greatest impact. However, this means that some countries may have opted out of the process, even though they would need support.

On the objective of supporting the EU project and programming cycle, the ETF has offered guidance and expertise to EU stakeholders. This includes providing input for policy dialogues with EU institutions and agencies 54 . Although the number of requests from EU departments has declined since 2020, the ETF has observed that the requests it receives are more substantial 55 . 

EU-OSHA – EU-OSHA successfully met all six operational objectives, consistently achieving timely delivery, with rates exceeding 90% of its outputs delivered on time for most activities. The number of outputs related to facts and figures nearly tripled compared to the baseline, showcasing a significant increase in outputs related to networking knowledge. Outputs in other areas remained similar to the 2016 baseline (see Table 1 in Annex VIII).

The most relevant outputs falling under facts and figures were the European survey of enterprises on new and emerging risks (ESENER), and support with compliance for SMEs (source: KPIs from the 2021 annual activity report ). The ‘tools for OSH management’ were used more widely during the current evaluation period, to enable workplaces to manage their occupational safety and health risks. Over 72 000 new risk assessments were conducted in 2022, surpassing both the target of 30 000 and the baseline) 56 .

To raise awareness on OSH issues, the Agency actively developed the healthy workplaces campaigns (HWC). Through these campaigns, EU-OSHA engaged with national stakeholders ranging from SMEs to larger corporations and national bodies. Awareness was raised about the risks of dangerous substances in the workplace and preventing work-related musculoskeletal disorders 57 (see EU-OSHA Case Study 4 in Annex 8 to the supporting study).  

Finally, on its objective of maintaining and developing strategic and operational networks, EU-OSHA successfully delivered 91% (21 out of 23) of its outputs on time. However, in 2020, the COVID-19 pandemic led to the cancellation of four actions and one event 58 . 

Common operational objective: conducting communication and dissemination activities

In addition to the Agency-specific objectives, the Agencies shared the operational objective of conducting communication and dissemination activities. The degree of achievement varied between the Agencies:

EU-OSHA’s healthy workplaces campaigns (HWC) were successful in raising awareness and communicating on OSH issues 59 . 

CEDEFOP increased its social media reach and the number of visits to its website compared to baseline (as shown in Annex VIII).

EUROFOUND’s communication and dissemination activities were generally appreciated (and improvements were made throughout the evaluation period). However, the level of achievement of this objective could be improved. For instance, Eurofound monitored the uptake of knowledge through the media as of 2021 60 , but no monitoring data were available for social media activities.

The ETF increased its social media use and presence throughout the evaluation period. However, despite these improvements, ETF Case Study 2 showed that the effectiveness of the ETF’s efforts was hindered by an imbalance between delivering outputs and carrying out dissemination activities 61 . The case study highlighted that prioritising the production of research publications over communication and communication initiatives posed a challenge to effectiveness 62 . 

B) Specific objectives (Results)

Given the Agencies’ focus on knowledge generation, results are achieved when stakeholders at both EU and national level utilise the knowledge outputs, which are of high quality. As outlined in the intervention logics of the Agencies (see Annex VI), one common expected result is that policymakers receive valuable and pertinent data from the Agencies’ outputs, supporting relevant policies. Another result is that stakeholders, in general, receive relevant, timely, and high-quality information (see full list of the Agencies expected results in Table 1 of Annex XII).

Consequently, the achievement of results (meeting the Agencies’ specific objectives) has been assessed based on the use of the Agencies’ outputs and services, as well as their quality 63 . This perspective aligns with the results indicators used by the Agencies, primarily focusing on the extent of stakeholders’ adoption of the Agencies’ outputs. The Agencies also assess these results through regular user satisfaction surveys.

B.1) Use of the Agencies’ outputs

Based on the Agencies’ KPIs 64  (e.g. references in EU policy documents, academic literature, social media following and download numbers), along with consultations with stakeholders and staff 65 , the evaluation concludes that each of the Agencies’ services and outputs have been used by their key stakeholders. There is an overall trend of increased usage compared with the previous evaluation period. Usage by EU institutions was greater and more widespread than at national or international level (as shown in Annex VIII).

EUROFOUND experienced a significant increase in the usage of its outputs for EU-level policymaking, evidenced by a higher number of references in key European-level policy documents, particularly from the European Commission and the European Parliament 66 . This trend has been consistent since 2011-2016. The number of downloads of Eurofound publications from its website was higher than the 2016 baseline figure of 155 943 throughout the evaluation period, with a peak in 2020 of 188 556 pdf downloads, supported by an increasing number of social media users. However, a stakeholder survey suggests that there is still room to increase the use of and participation in Eurofound's services and activities. Only 49% of stakeholders (N=65) frequently used Eurofound's information, resources or services, and only 37% (N=65) participated often in the Agency's activities or events.

The CEDEFOP stakeholder survey reported that 43% (N=174) of respondents ‘often’ consult or use the Agency’s information. However, one fifth of respondents (22% - 38 out of 174) declared that they seldom or never use the Agency's information resources. Since 2020, Cedefop downloads have exceeded the 2016 baseline (383 600). Downloads peaked at 465 000 in 2020 linked to the shift to only disseminating publications digitally.

Eurofound and Cedefop have also substantially increased usage by researchers and academia compared to the previous period (see Table 3). Despite this positive trend, the supporting study concluded there is room to increase the outreach of Cedefop's evidence through academia and VET providers to ensure wider dissemination of its outputs 67 .

Table 3: Number of citations of each Agency's work in academic literature

Source: Annual activity reports, 2016-2022

ETF - EU-level stakeholders have continued to use ETF’s services and advice, although requests from EU departments have decreased since 2020. Regarding beneficiaries from partner countries, feedback from staff, stakeholders’ surveys and public consultation suggests that the ETF has room for improvement in disseminating its outputs 68 .  The ETF stakeholder survey revealed that 44% (N=91) of stakeholders use the Agency’s outputs ‘often’ while 43% reported using them ‘sometimes’. Downloads of ETF publications saw a dramatic decrease from 1 181 462 in 2016 to 351 219 in 2018 69 (see Table 4, Annex VIII).

EU-OSHA - Stakeholders actively engaged with EU-OSHA activities and related outputs, as evidenced by various user surveys and indicators (e.g. huge increase in downloads on corporate and campaign websites). The Agency saw a huge leap in downloads in 2019 with a peak of 2 259 137 in 2020 70 (compared to a low baseline of 82 558 downloads in 2016). This exceeded the target of a 5% increase in downloads per year. The evaluation survey showed that a majority of stakeholders often or sometimes participated in the Agency’s activities or events (89% of respondents N=75) and consulted or used its information, resources or services (95% of respondents N=75).

The use of the Agencies’ services at Member State level is less well documented than their use by European level stakeholders 71 . Nevertheless, the evaluation provides evidence of effective use by national stakeholders depending on the Agencies’ different remits and activities. Eurofound services used by national stakeholders include its work on minimum wages. This work is continuously referred to by national wage-setting institutes, including governments, trade unions and low pay commissions 72 , as highlighted in the Eurofound case study on minimum wages 73 . Cedefop support to national stakeholders decreased during the reporting period. This is because direct support to Member States (e.g. by way of thematic country reviews and policy learning forums) decreased following the Agency’s decision to prioritise EU level policy support in the face of intense budgetary constraints 74 . EU-OSHA outputs and tools have been used intensively by national stakeholders, with outputs such as OiRA 75 , OSH-wiki and the healthy workplaces campaigns (HWCs) seeing increasing use and stakeholder participation compared to the 2016 baseline.

The importance of social media channels for the visibility, uptake and use of the Agencies’ outputs increased during the evaluation period. Cedefop and the ETF showed a year-on-year growth of social media followers across all the channels used by both Agencies 76 (with the number of followers at least doubling across most channels during the evaluation period). The use of social media channels is particularly important for the ETF as many stakeholders in partner countries rely on information from these channels. Comparable data are not published by EU-OSHA 77 and Eurofound.

B.2) Quality of outputs and other results achievements 78

One specific objective / expected result common to all the Agencies is to provide stakeholders with relevant, timely and high-quality information 79 . The evaluation, based on various Agencies’ indicators, consultation tools and case studies, concludes that during the evaluation period, the Agencies achieved this objective to a great extent. Most of the Agencies’ outputs and services have received very positive feedback and high user satisfaction, even surpassing scores from the previous evaluation period. Table 4 highlights the most used and highest quality outputs, as explained in the below Agency assessment.

EUROFOUND - Eurofound received increased recognition for the scientific quality of its research, peaking at 1 220 references in peer-reviewed journals in 2021 (see KPI in Table 3). 90% (N=65) of stakeholder survey respondents rated the Eurofound research outputs as ‘very or rather high quality’ while 89% of the public consultation respondents (N=31) gave similar ratings. The Agency's COVID-19-related research was widely acknowledged and contributed to EU policy discussions (e.g. on telework and the right to disconnect). Stakeholder feedback highlighted Eurofound's significant contribution to understanding working and living conditions 80  and emphasised the consistently high quality of its surveys. A 2022 external evaluation 81  highlighted the success of the new monitoring convergence activity, contributing to timely and policy-relevant knowledge on EU convergence. Eurofound developed in 2018 the convergeEU tool, which permitted the analysis of convergence for the full set of indicators of the Social Scoreboard 82  accompanying the European Pillar of Social Rights.

However, stakeholders have been requesting more visually accessible digital content since 2020 83 . While the 2022 user satisfaction survey showed high satisfaction levels (91% of Eurofound stakeholders were satisfied or very satisfied with research reports and 86% with survey information 84 ), a majority (79%) expressed the desire for more digital content, especially data visualisation. Furthermore, consultations highlighted the need for more in-depth outputs, accompanied by additional methodological guidance, considering the extensive amount of primary data Eurofound reports on (Annex 1 to the supporting study, p.140).

The outputs that have contributed most to Eurofound’s results are: (i) the three pan-European surveys; (ii) the living, working and COVID-19 e-survey; and (iii) the outputs on minimum wages (see Table 4).

The three surveys mentioned in point (i) are key for achieving results, because they underpin most of Eurofound research work, and therefore contribute to several specific objectives and results. They have, for example, contributed to the improvement of working conditions and sustainability of work, labour market functioning and inclusiveness, as well as to promoting social cohesion and convergence, e.g. social convergence indicators are drawn from the European quality of life survey (EQLS) and the European working conditions survey (EWCS), (see also the added value of Eurofound surveys in Section 4.2).

The ‘Living, working and COVID-19’ e-survey mentioned in point (ii) was conducted in five rounds between April 2020 and May 2022. It contributed to the result ‘Key stakeholders have increased understanding of ways in which to improve quality of life, public services, and society’. The timeliness and relevance of this research has been crucial for understanding the impact of the COVID-19 pandemic; the case study on Eurofound’s contribution to COVID-19 policy showed that Eurofound more than doubled its media impact thanks to the knowledge gathered from this e-survey.

Finally, Eurofound’s work on minimum wage mentioned in point (iii) helped provide knowledge to improve working conditions (e.g. the annual review reports blogs ‘How to ensure adequate minimum wages in an age of inflation’ 85 ) and supported the 2022 Directive on adequate minimum wages in the EU (see details in Section 4.1.1.2, and in Eurofound Case Study 3) 86 .

CEDEFOP - The users’ satisfaction and appreciation of Cedefop’s outputs is high and has significantly increased compared to the previous evaluation period. According to the user satisfaction surveys conducted in 2019 (N=453) and 2022 (N=486), 99% of respondents in both years found information provided by Cedefop to be reliable. In 2019, 95% (N=463) of the respondents and 96% in 2022 (N=507) indicated that Cedefop addressed their needs. This is a notable improvement from the stakeholder survey conducted for 2011-2016, where only 70-75% of respondents (N=213 in total) felt that their needs had been met by the Agency's outputs.

Furthermore, Cedefop received the highest rating among the Agencies for introducing new concepts or ideas in public policy 87 . Both staff (89% N=45) 88 and stakeholders (79% N=174) rated it as either ‘very good’ or ‘rather good’.

Cedefop’s work on monitoring and analysing VET policy developments, skills anticipation and matching, and promoting access to and attractiveness of VET, were the three most referenced and downloaded areas throughout the evaluation period (see Table 4). Its work on monitoring and analysing VET contributed to shaping the common priorities set out in the Council Recommendation on VET and the Osnabrück Declaration 89 . VET policy monitoring is highly rated by stakeholders, with 72% of survey respondents rating them as very or rather high quality. The Agency’s work on skills anticipation and matching was also among the most widely used and appreciated, informing EU employment strategies and wider EU policies (e.g. the EU migration network, and the testing of different scenarios in the work of DG GROW, DG ENER, and DG ENV). Finally, the outputs on promoting access to and attractiveness of VET were among the most cited (especially by international organisations) and downloaded, though less than the above-mentioned areas. These outputs contributed to the strategic area of ‘Valuing VET and skills’. 

Cedefop has also achieved positive results on the policy learning objective, facilitating valuable exchanges for EU and national actors. Although the overall number of meetings and events decreased over the evaluation period, the number of participants has increased each year since 2017 (from 1 159 in 2017 to 1 961 in 2021) 90 . In 2021, participants gave a 98% rating for the ‘quality and expected impact’ of events organised by Cedefop, the highest satisfaction level during the evaluation period.

EU-OSHA - Similar to the previous evaluation period, most of the EU-OSHA’s outputs received high user satisfaction ratings 91 . The EU-OSHA’s online risk assessment tools (OiRA) were particularly valued and widely used, indicating a successful achievement of the specific objective to provide small workplaces with relevant tools for health and safety management. More than 300 OiRA tools were available in 17 EU languages by the end of the reporting period, roughly 200 more than in 2016, covering over 30 sectors from manufacturing to education. These tools were widely used, especially by SMEs, helping them to initiate risk assessments. The healthy work campaigns (HWCs) were also widely used and valued (see Annex XII, Figure 4 on stakeholders’ assessment of HWCs), and have been key to achieving the result of making beneficiaries and intermediaries aware of workplace risks and how to prevent them (see also Section 4.1.1.2. on generating impacts).

Other Agency outputs, including the foresight reports on the circular economy and digitalisation have effectively contributed to the expected result of raising awareness among policymakers on new and emerging risks, therefore helping them to inform their decisions. In particular, EU-OSHA’s foresight reports on digitalisation brought added value even in countries that were familiar with the issue, by exploring aspects that were under-researched at national level. Management Board members and focal points also found foresight exercises to have been very relevant for EU and national employers’ organisations. This is a considerable improvement compared to the baseline when the foresight studies were used less than the other outputs.

However, focal point members emphasised the need to customise EU-OSHA outputs to the specific needs of target groups, particularly for workplaces, which may involve creating concise infographics with key information.

ETF - Similar to the previous evaluation period, the Torino Process was highly appreciated by stakeholders, especially partner countries. The latter consider the Torino Process reports as one of the most valuable ETF outputs (see Table 5), as they are key to monitor the performance of lifelong learning policies and systems and to identify partner countries’ needs, contributing to policy dialogue. During 2019-2021 the ETF substantially exceeded its target of 50% of partner countries using the Torino Process and achieved a figure of 78% of partner countries in 2019 (see also Section A above).

The expected result of providing partner countries with relevant and timely support to modernise national qualification frameworks was also mostly achieved. The ETF consistently monitored and supported the development of national qualification frameworks (NQFs), playing a crucial role in supporting the development of the African Continental Qualifications Framework. The Agency built on its positive work in this area during the previous evaluation period.

The ETF’s work on facilitating collaboration through the creation of networks was often highlighted by stakeholders as a key area in which the Agency has achieved significant results, including three of the most used and valued ETF outputs during the evaluation period (see Table 4 below), namely: (i) the ETF’s Network of Excellence, cited by both survey respondents and interviewees as being a high-quality output and a success in terms of the number of participants involved and its contribution to vocational excellence provision; (ii) the Skills Lab Network, set up during the current period, which brings together labour market experts and researchers from different institutions and countries to share knowledge on skills anticipation and skills matching (see supporting study case study on Skills Lab); and (iii) the New Learning Network, which contributed to innovative teaching and learning.

Table 4: Most used and highest quality outputs / themes per Agency

Source: supporting study, based on triangulated assessment, using several sources/indicators, including outputs that have helped to achieve results, most downloaded outputs and most highly rated outputs in the stakeholder survey / public consultation.

The ETF also achieved positive results in other areas. For example, the ETF’s enterprise skills development support activities were specifically highlighted in the European Parliament’s discharge report for 2020 92 . The report welcomed the Agency’s ‘Skills for enterprise development’ initiative, which helps enterprises respond to challenges related to COVID-19 and the digital and green transitions. In addition, the ETF’s EU external policy support met KPI targets (the ETF provided support each year to 58-62% of partner countries, against a target of 50%), and the rapid education diagnostics reports (RED) enhanced dialogue and supported EU policy for Lebanon and Kosovo (see further evidence in Section 4.1.1.2 and Annex XII).

However, the ETF fell short of meeting its targets in two areas: developing and implementing mechanisms to facilitate the transition to work in 50% of partner countries; and labour market and skills intelligence to inform VET and skills development policies and to support entrepreneurial learning (see Annex XII and Section 4.1.1.3. of the supporting study). The evaluation of ETF’s activities also suggested room for improvement on the relevance of the Agency’s support for capacity building among teachers and trainers, and for work-based learning (supporting study, p. 144). It also achieved mixed results on supporting governance of human capital systems in partner countries, with good results in Moldova, but insufficient indicators and evidence for concluding an overall assessment (see Annex XII).

One reason for the underachievement of some objectives was the limited willingness of partner countries to adopt the advice and support offered by the ETF. This challenge was identified in the previous evaluation, highlighting the need to better tailor the ETF support to the needs and policy cycle of partner countries.

4.1.1.2. Generating impacts

During the evaluation period, the Agencies directly contributed to EU policymaking in the areas relevant to their mandates, confirming the positive assessment on impacts achievement from the previous evaluation. 

The Agencies also supported the implementation of EU policies at national level (e.g. EU-OSHA on OSH policies - see Box 4 - CEDEFOP on upskilling and recognition of qualifications, EUROFOUND on minimum wage and ETF on supporting the implementation of EU external policies in partner countries).

The Agencies’ work has also led to specific impacts, as outlined in their respective intervention logics 93 , along with unexpected impacts, primarily driven by the COVID-19 pandemic. Moreover, the Agencies contributed to several Sustainable Development Goals (SDG) during the evaluation period by supporting EU policymaking in employment, social inclusion and education. In particular, Eurofound contributed to SDG8 (decent work and economic growth) and SDG10 (reduced inequalities); CEDEFOP and the ETF to SDG4 (quality education) and SDG10 (reduced inequalities); and EU-OSHA to SDG3 (good health and well-being) and SDG8 (decent work and economic growth).

EUROFOUND - Similar to the previous evaluation period, Eurofound continued to make a direct impact, enhancing policymaking capacity in various EU policy areas. Notably, Eurofound played a significant role in areas such as minimum wage (see Box 1), and emerging forms of work. Through its research on digitalisation and new, non-standard employment forms, including the platform economy, Eurofound strengthened the ability of the European Commission and other EU bodies to draw up effective employment policies. A concrete example is Eurofound’s contribution to platform work, which has underpinned the proposed Directive on improving working conditions in platform work 94 . 

Eurofound’s activities also contributed to the lasting impact of improving dialogue between management and labour 95 . Notably, the 2022 Tripartite Exchange Seminar 96 positively influenced social partners’ capacity and improved social dialogue. Eurofound’s work in this area has indirectly impacted areas such as the gender pay gap and the production of EU-wide comparative information on COVID-19. The impact was substantial, fostering frequent communication with trade union stakeholders. However, the case study found that these impacts could be further enhanced if Eurofound’s resources were available in more languages.

Finally, Eurofound has significantly contributed to improving the capacity to address current and emerging challenges for enhancing living and working conditions. The Agency’s noteworthy contributions include an increased understanding of policy topics related to COVID-19 (see Eurofound Case Study 1 in Annex 8). This understanding encompasses the effects on remote working, platform work, the disproportionate impact of the COVID-19 crisis on lower paid and minimum wage workers, and the identification of groups significantly affected (see below on overall unexpected impacts and on adaptation to COVID-19 challenges). As it has done every year since 2015, Eurofound published its ‘Minimum wages in 2020: Annual review’, which was mentioned in 15 Commission key policy documents related to the initiative on fair minimum wages for workers in the EU.

Box 1: Eurofound’s contribution to minimum wage policy (case study) 97

CEDEFOP - Cedefop successfully achieved its expected direct impacts by providing knowledge, evidence and data, and facilitating peer exchanges. This support was instrumental in helping both EU and national policymakers to develop or enhance policies and systems on VET, skills and qualifications. At EU level, Cedefop actively participated in shaping EU VET policies by giving presentations at various EU-level forums (see Box 2). Notably, Cedefop significantly contributed to enhancing transparency and recognition of qualifications (see Cedefop Case Study 3 in Annex 8 to the supporting study). Its work in the area contributed to advancing the use of the European Qualification Framework (EQF), by continuously comparing and analysing its implementation across the EU at national and sectoral level.

Cedefop achieved significant impact by aligning with the priorities of the 2019-2024 Commission, notably the 2020 renewed Skills Agenda 98 and the first-ever Council Recommendation on vocational education and training for sustainable competitiveness, social fairness and resilience. This contribution was acknowledged by the European Parliament in its 2022 discharge report 99 . Cedefop also provided valuable input to the Osnabrück Declaration.

The Agency’s analytical framework on apprenticeships informed the 2018 Council Recommendation on a European framework for quality and effective apprenticeships. Cedefop co-organised policy learning forums on upskilling and reskilling of low-skilled adults, together with the European Economic and Social Committee (EESC). The European Parliament recognised Cedefop's expertise and potential, suggesting that further reference be made to it when preparing guidelines for foresight-based policy analysis 100 .

At national level, Cedefop made an impact by conducting country-focused thematic reviews and contributing to national events. For instance, in 2021, thematic reviews and peer learning activities were organised on skills governance in Slovenia and on upskilling pathways in France and Italy. Cedefop also provided support to Slovenia’s Labour Ministry in preparing a national skills forecast and developing a career platform 101 .

Box 2: Cedefop’s impact on VET policy at EU level (case study) 102

ETF - Evidence shows that the ETF’s work on supporting human capital development reforms in partner countries has had a positive impact in at least three areas listed in its intervention logic: (i) adapting VET governance to changing skills demands; (ii) enhancing labour market policies and practices; and (iii) contributing to EU external relations policy through advice. In particular:

·ETF actions effectively contributed, with concrete inputs, to the development of national qualifications frameworks (NQFs) / regional qualifications frameworks (RQFs) in at least nine countries and two regions of Africa.

·The ETF designed new/upgraded tools for analysing skills needs, benefiting stakeholders in partner countries (see Box 3).

·Since 2013, the ETF collaborated with Türkiye on a key priority for the country: a reform to set up centres for recognition of prior learning. The ETF partnered with EU delegations, the national qualifications agency and relevant sectoral committees. As a result, 25 centres for VET recognition have been set up in the country.

Box 3: ETF support on improving active labour market policies and practices (case study) 103  

EU-OSHA – Overall, EU-OSHA has achieved its expected direct impacts, contributing to EU OSH legislation, Member State policies, as well as initiatives by social partner and other OSH stakeholders. This contribution has helped these entities anticipate and manage change, prevent workplace accidents and increases readiness for future health crises.

·On contributing to OSH legislation and strategies, EU-OSHA significantly contributed to the EU’s strategic frameworks on health and safety at work, collaborating extensively with the Commission. The Agency’s work is cited in the 2014-2020 strategic framework, particularly its work in supporting SMEs, including through the OiRA tool 104 . EU-OSHA’s healthy workplace campaigns influenced legislation and strategies, such as the European Commission’s proposals on carcinogens (see Box 4 below).

·EU-OSHA supported stakeholders in preventing workplace accidents and illnesses, and increased readiness for potential future health crises. For instance, the OiRA tool for COVID-19, introduced in June 2020, helped with COVID-19 risk management. The tool was actively promoted in Belgium 105 and France 106 among other Member States. In France it provided 3 797 risk assessments in just over 2 months. It is, however, difficult to assess the actual use of tools such as OiRA in companies, and its longer-term impacts (e.g. whether companies acted on the information from OiRA and applied changes to prevent workplace accidents and illnesses), as the use of the tools is anonymous 107 .

·In terms of enabling anticipation and management of change, EU-OSHA successfully provided foresight information on OSH. Most respondents across the three consultations (staff, stakeholders and the broader public) responded that the Agency was successful in delivering foresight information either to a large extent or to a moderate extent during the evaluation period 108 .

Box 4: EU-OSHA’s impact on contributing to preventing workplace accidents (case study) 109

4.1.1.3. Unexpected impacts

The COVID-19 pandemic had a significant impact on the Agencies’ outputs and activities, with the specific impact depending on the mandate and way of working of each Agency. The COVID-19 pandemic had a significant impact on the Agencies’ outputs and activities – impacts differentiated according to their mandates–, and their way of working. For all agencies, COVID-19 was a highly disruptive factor, e.g. cancellation of some events and activities in 2020 involving physical presence. Eurofound’s survey fieldwork was profoundly affected, as it could not be carried out face-to-face, and the EU OSHA ESENER outputs, as no dissemination events could be held. The number of events organised by Cedefop reduced by around half in 2020 and 2021. The pandemic also affected the ETF’s Partner Countries ability to engage in some activities that were planned in 2020 with some of them having to be postponed to the following year.

However, these changes also led to several positive outcomes and unexpected effects. The transition to online work not only resulted in cost-savings but also allowed the Agencies to increase their visibility by providing information and guidance on COVID-related issues, expanding their audience.

For example, Eurofound’s work on remote and platform work fed into policy discussions on COVID-19, particularly its findings on youth and gender 111 . This unexpected event led to higher engagement, measured through policy references, media coverage, academic citations and downloads, showing the increased interest in their expertise on gender and youth 112 .

Other broader achievements and unexpected impacts included EU-OSHA’s publications reaching a wider audience, positive impacts from the ETF’s work related to Russia’s war of aggression against Ukraine, and heightened visibility for Cedefop’s work due to its activities related to Ukraine (see below).

Agencies’ adaptation to changes, COVID-19 pandemic and Russia’s war of aggression

During the 2017-2022 evaluation period, the EU was faced with new challenges, alongside its ongoing long-term issues 113 . Overall, the Agencies adjusted their activities and outputs responding to: (a) changes in EU policies, including the political priorities of the 2019-2024 Commission, focusing more on green and digital transitions policies (see Tables 4 and 5 in Annex VII); and (b) the impacts of the COVID-19 pandemic and Russia’s war of aggression against Ukraine. 

A) Regarding the adjustment to EU policy priorities, such as digitalisation and the European Green Deal, the Agencies adapted their activities, within their mandates, and worked on addressing these megatrends during the evaluation period. This involved realigning work programmes and creating specific outputs on these topics such as publications, studies, surveys and collaborative projects (see Table 6 in Annex VII) 114 . The Agencies also revised their environmental policies to align with EU priorities around climate change. The evaluation identified a need to improve dissemination of outputs related to these topics, to ensure that stakeholders are well-informed of the Agencies’ efforts in these areas.

On the European Green Deal in particular, Eurofound published a range of reports on sustainability and related social challenges 115 in addition to a pilot project proposed by the European Parliament on the Future of Manufacturing in Europe . Eurofound and EU-OSHA are active partners in the EU Climate and Health Observatory. This collaboration enhances their ability to address the intersecting challenges of climate change and health, providing comprehensive data, analysis, and tools to stakeholders across Europe. Their work within the Observatory focuses on assessing and mitigating the impacts of climate change on working and living conditions, thereby contributing to more resilient and adaptive occupational and public health policies, which monitors and evaluates the impacts of climate change on public health.  Cedefop produced outputs analysing the impact of the green transition on jobs and skills across sectors and occupations, united under the Cedefop Green Observatory initiative. Publications 116 included the sectoral skills foresight , which focused on occupational profiles for the green transition of smart and green cities, waste management, agri-food and circular economy, and the green employment and skills transformation report (2021), which assessed the effects of the European Green Deal on different sectors and identified opportunities for upskilling and reskilling.

The ETF carried out a significant amount of work on green skills, including: (i) pilot reviews in eight countries 117 in sectors such as agri-tech, agri-food, automotive and energy; (ii) the GRETA initiative 118 to support the green transition through peer learning; and (iii) the annual ETF Green Skills Award that recognises successful efforts in creating circular and carbon-neutral economies (see ETF Case Study 4). Finally, EU-OSHA produced a growing number of thematic publications on green jobs, including foresight studies aimed at anticipating OSH challenges relating to green jobs and the circular economy, and the ‘heat at work’ guidance for workplaces.

B) The COVID-19 pandemic significantly influenced the latter part of the evaluation period. All Agencies swiftly adapted to the challenges posed by COVID-19, both in terms of internal processes and thematic focus. They produced knowledge and dissemination tools to address COVID-19 challenges in line with their mandates. For instance, EU-OSHA and Eurofound responded promply to provide advice on occupational health and safety, and support for living and working conditions, respectively. Cedefop and the ETF also quickly adapted their activities to assist stakeholders in facing the challenges brought about by the pandemic.

EUROFOUND adjusted its work on ‘living and working conditions’, by introducing two new tools for monitoring and data collection: the COVID-19 EU PolicyWatch database and the ‘living, working and COVID-19’ e-survey 119 . The e-survey was particularly helpful in assessing the impact of the pandemic on living and working conditions.

The EU PolicyWatch tool provided valuable information to the European Commission and European Parliament by showcasing initiatives responding to crises. This resulted in Eurofound being one of the first organisations to provide insights into the pandemic’s effects 120 . Since then, Eurofound has produced 145 publications related to COVID-19 121 , which were highly effective and widely downloaded 122 .

Table 4: Agencies’ responses to COVID-19

The evaluation study and Case Study 1 confirmed that Eurofound responded rapidly and effectively to the challenge posed by COVID-19. Stakeholder interviews and staff surveys 127 indicated that Eurofound’s response to the crisis was effective, timely and impactful. Eurofound has largely aligned its research areas and key outputs with the European Commission’s six strategic priorities for 2019-2024 128 , making significant contributions.

EU-OSHA - As detailed in EU-OSHA Case Study 1 129 , the pandemic provided an opportunity for the Agency to significantly contribute to an increased focus on OSH issues, both in terms of practical needs and in policy debates. The Agency responded promptly by offering various tools to manage the pandemic in European workplaces 130 . These tools included guidance for workplaces, a dedicated webpage (‘COVID-19: Resources for the workplace’), and 20 resources developed in 2021 131 , such as EU guidance documents, OSH-Wiki articles, and awareness-raising materials 132 . In addition, the Agency developed the OiRA risk assessment tool for COVID-19 and made it available to national partners in June 2020 133 . 

The ETF carried out several activities to address the urgent needs arising from the COVID-19 pandemic. Initially, it set up a monitoring system to track the pandemic’s impact on education and training systems in partner countries, along with measures to mitigate negative effects 134 . The findings were shared through reports for partner countries and Commission departments 135 as well as public webinars. The #LearningConnects campaign provided further support by organising webinars and other activities, and publishing articles and reports on good practice in partner countries. The ETF stepped up its collaboration with other Agencies and international organisations, focusing on skills, youth and career guidance 136 .

CEDEFOP collaborated with Eurofound on a survey to gather information from organisations on how they were coping with the pandemic and organised a seminar with Members of the European Parliament on post-pandemic employment, skills and social policies. Cedefop also led an international survey on career guidance policy and practice during the pandemic, in collaboration with an inter-Agency working group including representatives of UNESCO, ILO, OECD, the European Commission and the ETF. It also produced products and resources informing stakeholders about COVID-19's impact on the labour market, skills and VET, and specifically, the tourism sector 137 . A user satisfaction survey run by Cedefop in 2022 revealed that approximatively 90% of users were highly satisfied with the Agency's COVID-19 outputs, finding them clear, sound, attractive, and very useful for understanding the pandemic's impact on VET, skills and qualifications 138 . 

Joint outputs - In addition to their individual actions, the Agencies cooperated on joint outputs and activities to address the cross-cutting challenge of COVID-19 139 . These included: (i) a joint European company survey run by Eurofound with Cedefop, exploring  the impact of the pandemic on companies in the EU-27; (ii) an e-survey by Eurofound on ‘Living, working and COVID-19’, distributed by the ETF to its partner countries; and (iii) collaboration between Eurofound and EU-OSHA, such as on organising discussions at the European Parliament’s EMPL Committee on the COVID-19 impacts, and the participation of Eurofound in EU-OSHA’s psychosocial risk assessment.

Russia’s war of aggression against Ukraine also meant that the Agencies needed to respond and adapt their activities to varying degrees, depending on their mandates, over the course of 2022. The situation in Ukraine was most relevant for the ETF; it was of intermediate relevance for Eurofound and Cedefop and of least relevance for EU-OSHA. This is reflected in the varying intensity of the Agencies’ reactions, and in the responses to the staff and stakeholder surveys.

ETF - For the ETF, Ukraine has long been a partner country, allowing the Agency to provide tailored support. The ETF’s range of activities in this area is therefore much greater than that of the other three Agencies. Stakholders largely perceived the ETF as responsive 140 . The Agency’s initiative in monitoring and responding to the Ukrainian crisis was also highlighted in the 2021 European Parliament discharge report on the ETF’s activities 141 . 

The ETF’s initiatives included work on the recognition of qualifications to help Ukrainian refugees enter the workforce, through the report ‘Comparing Qualifications Frameworks for Inclusion: a Ukrainian case study’ 142 . In addition, in collaboration with Eurofound, the ETF published a report in 2022 that provides an assessment of the impact of Russia’s invasion on the economy and education 143 . In addition to other outputs and resources 144 , the ETF carried out two key initiatives to support the country in setting up a new education paradigm: the UA Re-Emerge(ncy) programme, focused on vocational e-learning, reskilling and upskilling courses 145 , and the Ukrainian Resource Hub: to provide wide education and work information for both Ukraine and EU countries hosting Ukrainian refugees, including schools, universities and employers 146 .

CEDEFOP – For Cedefop, the Ukrainian crisis has been of more moderate relevance. The Agency focused on supporting displaced Ukrainian students in VET institutions of the host countries, providing guidance to teachers and trainers involved in their reception and learning pathways 147 (e.g. Using ReferNet 2022 national outputs, Cedefop analysed Member States' national policies to support the integration of Ukrainian refugees into VET) 148 . 

EUROFOUND, for which the Ukrainian crisis was less relevant than the COVID-19 pandemic, had fewer research outputs on Ukraine. Survey responses indicated a lower perception of Eurofound's effectiveness in responding to the Ukraine crisis compared to COVID-19 (see Annex 13 to the supporting study).

The available evidence indicates that the four Agencies were cost-effective during the evaluation period. The four Agencies have fully implemented their budgets and delivered to a large extent the planned outputs set out in their annual programming documents, indicating high operational efficiency (see Sections 3.1.1 and 3.2). They managed to lower costs through various measures while maintaining the quality of their outputs. The cost-effectiveness analysis of the Agencies’ key activities concluded that, for most of them, there was no scope to reduce costs without compromising quality (see Annex XI) 149 .

There are differences in the annual expenditure among the four Agencies as well as between the Agencies and those decentralised agencies used as benchmarks. This includes differences in the share of expenditure allocated to administrative and operational activities. However, these differences are based on reasonable factors and do not indicate any specific or obvious inefficiencies.

The Agencies have taken action to improve their cost-effectiveness over the evaluation period. Key examples include:

-cost-saving measures, e.g. joint procurement with other agencies (the agencies share information on upcoming procurements through the EU Agencies Network (EUAN));

-sharing services (e.g. the ETF and EU-OSHA now share accounting services);

-sharing costs related to research activities (e.g. Eurofound’s preparation of the fourth European company survey was completed in collaboration with Cedefop);

-investing in actions that, while increasing short-term costs, should ensure long-term savings (e.g. increasing the energy efficiency of offices, disseminating outputs in digital formats rather than physical formats).

These examples are described in detail below. See also Table 3 in Annex VII.

The evaluation also examined how the Agencies responded to audit reports. It found that the Agencies were highly responsive in putting audit recommendations (both by the Internal Audit Service (IAS) and the European Court of Auditors (ECA)) into practice and transparently publishing follow-up to audit recommendations in consolidated annual activity reports of audit recommendation implementation.

Stakeholders perceived the four Agencies’ operations to be cost-effective although the evaluation stakeholder survey showed differences between the main stakeholder groups. Members of the Agencies' management or governing boards (MBs) were far more positive than other stakeholders. Across all four Agencies, 83% or more of MB members rated the cost-effectiveness of the Agencies’ operations as either high or medium, compared to 61% of other Eurofound stakeholders, 57% of other EU-OSHA stakeholders, 49% of other ETF stakeholders and 46% of other Cedefop stakeholders. The percentage of survey respondents rating the cost-effectiveness as low was very low across all four Agencies, at 6% or less (see Figure 12 in Annex VII).

However, the interviews, survey responses, consultations and analysis of expenditure data identified some key areas where the Agencies’ cost-effectiveness could be further improved, in particular staff resources, inter-agency cooperation, procurement and contracting procedures (see below).

The evaluation analysed the efficiency of the Agencies in terms of: (i) balance of operational and administrative expenditure (and staff); (ii) staff resources and workload; (iii) cost-effectiveness of key activities; (iv) internal monitoring and reporting mechanisms and potential for simplification and administrative burden reduction; and (v) the efficiency of governance structures. The following sections provide an overview of the main evidence by area.

4.1.2.1 Balance of operational and administrative expenditure

A key factor contributing to cost-effectiveness is the balance between the Agencies’ ‘front-line’ operational activities and their staff and administrative costs. Although administrative roles are essential to an agency’s performance, these should be streamlined to maximise the proportion of expenditure on operational activities. To assess the balance of administrative, staff and operational expenditure 150 , the four Agencies were benchmarked against other decentralised agencies working in related policy areas: the European Institute for Gender Equality (EIGE), the Fundamental Rights Agency (FRA) and the European Labour Authority (ELA) 151 .

Compared to the previous evaluation period, the four Agencies reported increases in staff costs (and their share in total budget) between 2016 and 2022 (see Figure 2). Rather than being driven by increases in staff numbers (see Section 3.1 and Annex VII), these increases were due to high inflation rates and corresponding wage adjustments through the correction of the country coefficients 152 (e.g. Eurofound, headquartered in Dublin, saw its country coefficient rise from 118.3 in 2016 to 136.9 in 2022). Among the four Agencies, the ETF had the smallest increase in the share of staff costs to total expenditure (4 pps.) and EU-OSHA had the largest (9 pps.), although it began with the lowest share in 2016.

The increase in staff costs, coupled with relatively stable budgets (which are both factors out of the Agencies’ control), led to decreases in the share of expenditure devoted to operational activities when comparing 2022 to 2016, the baseline year. This suggests that higher staff costs, mostly driven by the country correction coefficients, limited the Agencies’ ability to allocate financial resources to the operational activities that are necessary to fulfil their mandates. Eurofound saw the largest decrease in the share of operational expenditure to the total budget (9 pps.), while the ETF saw the lowest (3 pps.).

The shares of expenditure devoted to administrative costs saw smaller fluctuations. When benchmarked with similar agencies, the four Agencies showed generally similar or lower shares of administrative expenditure, which was already the case in the previous evaluation period. Three of the four Agencies experienced slight increases from 2016 to 2022: Eurofound (following a large decrease in the previous period), EU-OSHA 153 (remaining substantially below the 2013 value when the share of administrative costs peaked at 12%) and Cedefop (mainly due to increased investment in its building’s digital infrastructure and energy efficiency). The ETF’s administrative expenditure share decreased, continuing the downward trend since 2013 154 .

Although the share of administrative expenditure has only slightly increased in the current period, it is expected to grow in the coming years as the Agencies invest in energy efficiency to achieve the net-zero target. This will put even more pressure on the Agencies’ operational budgets (if total budgets remain stable) but does not relate to any specific or obvious inefficiencies on their part.

Figure 2: Type of expenditure (% of total expenditure 2016 and 2022)

Source: Consolidated Annual Activity Reports, 2016 & 2022. Data not available for ELA in 2016.

4.1.2.2 Staff resources and workload

Skilled staff are essential for the Agencies to carry out their work plans to a high standard. However, the four Agencies have faced staffing challenges to varying degrees 155 . These challenges are: (i) cuts to and subsequent caps on staff numbers (see Section 3.1); (ii) an unexpected increase in workload due to the response to COVID-19 and Russia’s war of aggression against Ukraine without a corresponding increase in staff; and (iii) the Agencies expanding certain areas of their work. These difficulties have undermined the Agencies' ability to keep achieving their targets and maintaining a manageable workload for their staff.

According to the evaluation’s staff surveys, most staff in the four Agencies considered their workload to be well balanced or mostly balanced. This ranged from 90% in EU-OSHA to 56% in Eurofound. However, high percentages of staff reported that their workload was either mostly unbalanced or unbalanced, especially in Eurofound (44%), followed by the ETF (29%), Cedefop (27%), and EU-OSHA (10%).

The supporting study found that all four Agencies have coped with the demands by increasing overtime and the use of contract staff, which is unlikely to be sustainable 156 . Interviews complementing staff surveys corroborated this conclusion 157 . In particular, Cedefop and EU-OSHA have relied on external contractors to support both administrative and operational tasks 158 . Although this has reduced pressure on statutory staff (especially for carrying out administrative tasks), it comes with significant budget implications. EU-OSHA staff and executive board members pointed to concerns about the quality of contractors, difficulties in finding contractors specialised in OSH, and a time-consuming process to review deliverables. Based on these elements, bringing tasks in-house by hiring more staff would enable the agency to have more control over work processes and the quality of outputs. For the ETF, the Court of Auditors recently highlighted a need to avoid overreliance on external consultants and maintain expertise in-house 159 . Eurofound had also relied on external contractors although, over the evaluation period, it increased the amount of work done in-house, one of the drivers being to have more control over research quality. On the gender balance of ETF staff, the European Parliament discharge reports between 2018 and 2021 noted that further efforts were needed to achieve a better staff gender balance (67.7% women and 32.3% men in 2020).

Regarding their headquarters, Eurofound, the ETF and EU-OSHA staff reported high satisfaction about the fulfilment of their respective host Member States’ obligations. However, Cedefop staff had mixed views (45% satisfaction), primarily due to delays in signing a new headquarters agreement. Moreover, EU-OSHA, the ETF and Cedefop staff had mixed views on schooling (around a 50% satisfaction rate), while Eurofound, the ETF and Cedefop staff identified transport connections as an area for improvement (see Figure 6 in Annex VII).

4.1.2.3 Efficiency and adequacy of internal monitoring and reporting mechanisms and the potential for reducing administrative burdens

Overall, the Agencies have complied well with the measures necessary for ensuring accountability and assessing performance. The existing mechanisms cover different areas of the Agencies’ operations (including financial, human resources and core business performance), allowing for an assessment of how resources are spent and how they convert into the planned output.

In the current period, the four Agencies collaboratively implemented an internal control framework between 2018 and 2019. The framework allows the Agencies to assess how they mitigate problems and maximise their effectiveness in five control areas156. It also enables the four Agencies to better align their reporting and evaluation output. The Agencies made additional efforts to streamline their indicator systems 160 and reduce the administrative burden created by the existing mechanisms.

However, the current evaluation has identified two main areas for improvement.

(i) While the monitoring and reporting systems are extensive, a number of limitations were identified. In particular, some monitoring data were not fully complete or missing (particularly for the ETF). Moreover, for some performance indicators there is an absence of target indicator values (across all four Agencies). Another major gap is the absence of indicators for monitoring agencies results at national level. Some agency-specific issues were identified: for Eurofound, although the indicators had no problems, reporting could be further improved by using more user-friendly software and by streamlining project-reporting procedures; Cedefop, unlike other Agencies, does not have a separate indicator for monitoring the work programme completion rate; for the ETF, some indicators are not available across partner countries, and some performance indicators are missing data for certain years (e.g. the number of downloads from the ETF’s website).

(ii) There is potential to reduce the administrative burden associated with accountability mechanisms. Internal mechanisms are generally sufficient for ensuring accountability and assessing performance (and the Agencies’ staff and Parliament discharge reports corroborated this overall positive assessment). However, these mechanisms do require extensive administrative resources. Most staff survey respondents still experience some administrative burden 161 , which is perceived as higher by Cedefop and EU-OSHA staff (see Figure 3 in Annex VII). These results are in line with the previous evaluation, indicating only slight shifts 162 . In particular, Cedefop staff see accountability mechanisms as very burdensome, viewing them as a source of additional workload, despite simplification measures piloted by the agency’s administration 163 . As for EU-OSHA, which is the smallest of the four Agencies and has the highest administrative costs per staff member 164 , its staff and focal points members (FOPs) spoke of the increasing administrative burden associated with reporting and monitoring mechanisms 165 as multiple online tools are used. The associated workload was disproportionate to the size of the agency.

Moreover, interviews with staff of all Agencies revealed that administrative staff perceived their workload as less balanced than the workload of operational staff. This suggests that the efforts needed to meet the reporting, monitoring and evaluation requirements may be disproportionate to the relative sizes of the Agencies and are hindering effectiveness.

4.1.2.4 Agencies’ actions to improve cost-effectiveness and potential areas for cost reduction

During the evaluation period, each agency looked to adapt to the budget and human resources challenges and decrease the administrative burden with the aim to achieve efficiency gains. These have been mostly achieved through: a) cooperation and sharing costs with partners; b) digitisation/automation; and c) streamlining of work processes.

First, the Agencies looked for ways to share services, including recruitment (e.g. sharing reserve lists) and joint public procurement contracts (see agency-specific details in Table 3 in Annex VII). These efforts were facilitated by coordination through the EU Agencies Network. Moreover, the digitalisation of processes, which was strengthened during the COVID-19 pandemic, enabled some savings. As regards work processes, the four Agencies have implemented an activity based management (ABM) approach, where budgeting is organised under key activity headings, ensuring that sufficient budget is allocated to different activities.

The Agencies have made efficiency gains and implemented simplification measures. However, there is room for further action to improve cost effectiveness and reduce the administrative burden in the following areas.

1)Strengthening inter-agency cooperation: although there is already evidence of cooperation between the Agencies, there is room to deepen it further and create synergies (e.g. more information sharing and joint events to improve external stakeholders’ access to information across the four Agencies; identifying further areas for collaboration on research/survey development and implementation) 166 .

2)Expanding joint procurement and contracting procedures to cope with the increasing number of regulatory requirements and to improve procurement procedures, particularly in EU-OSHA 167 . 

3)Streamlining performance indicators and reporting activities (see section above). 

4)Reducing administrative burdens of the tripartite boards of Eurofound, Cedefop and EU-OSHA (see section below).

5)Balancing operational and administrative staffing: explore possibilities to increase the ratio of operational staff to administrative staff and reduce the reliance on external contractors. This could be based on: (i) potential gains in efficiency (as highlighted in this section); and (ii) prioritising activities and output without compromising quality.

4.1.2.5 Efficiency of governance structures

To provide further insights into the efficiency assessment, the evaluation considered a specific aspect of the Agencies' administration process, namely the efficiency of their governance arrangements.

Eurofound, Cedefop and EU-OSHA’s managing boards have a tripartite structure. Each Member State is represented in each board by a government, an employer and a trade union member (81 representatives in total). The European Commission has three representatives, and there is an independent expert (without voting rights) appointed by the European Parliament 168 .

The smaller size of the ETF’s governing board includes 27 representatives from Member States (without social partners), three Commission representatives (who share one vote in the Board), three experts appointed by the European Parliament (compared to one per tripartite Agency), and three partner country representatives appointed by the Commission.

The key benefits of the tripartite governance composition are the representation and strategic direction provided by social partners, in addition to the government representatives. Respondents across these three groups unanimously recognise the value of being heard and represented. In addition, the management boards, with such an extended membership, constitute a significant pool of experts contributing to the Agencies’ annual programmes with their knowledge and experience and serving as multipliers for disseminating evidence at national level.

However, governance efficiency faces challenge from the size and the number of voting members in this tripartite model in terms of: (i) the costs incurred by the Agencies to run the boards, including the organisation of meetings, information support, logistics; and (ii) the efficiency of the decision-making process and the way in which stakeholders are involved.

This tripartite structure is also not in line with the Common Approach 169 , which suggests that the management board should be limited in size 170 so that it can operate as a true supervisory body rather than a consultative assembly 171 . This matter has been flagged repeatedly by the European Parliament in its discharge reports. It was also raised in the study requested by the BUDG Committee on the management boards of the decentralised agencies (published October 2021) and in the ECA Special Report on the Future of EU Agencies – Potential for more flexibility and cooperation (2020) 172 .

However, the previous evaluation showed that the size and composition of the governing boards were not a significant concern for the Agencies themselves or for most of their stakeholders. Most respondents to targeted surveys positively evaluated the size and composition, and this assessment remains unchanged in the current evaluation (see Figures 4 and 5 in Annex VII). In contrast, as in the previous evaluation, a significant share of respondents to the Eurofound staff survey 173 , corroborated by staff interviewed in Eurofound and Cedefop, consider that, compared to the small number of internal staff, particularly research staff, the management board is disproportionally large, bringing with it financial, administrative and logistical challenges.

In assessing efficiency and the potential for streamlining, the evaluation explored alternative governance models with different ways of involving social partners. It compared the advantages and disadvantages of the tripartite management boards of Cedefop, Eurofound, and EU-OSHA with the ETF’s Governing Board and other models (ELA, EIGE, ILO) 174 . This comparison took into account the particular features of these agencies, including which countries are covered by the Agencies’ analysis. The comparative analysis of the different models led to the following conclusions.

·Meeting costs for larger boards are higher, especially if they are held in person, although this can be moderated to some extent by adopting online or hybrid meetings formats 175 . Since 2020, there has been a shift to more online meetings, which is reflected in the reduced overall costs  (see Table 5) 176 .

Table 5 – Amount spent on tripartite board meetings as a percentage of operational expenditure (Cedefop, Eurofound) (%)* 

* Data for EU-OSHA is not available.

·The productivity of management board meetings is dependent on the level of its members’ involvement. The engagement and level of knowledge will vary among members in such large governing bodies. In particular, social partner representatives (especially national ones) often lack background/experience in (EU-specific) administrative, financial, institutional and other governance/ matters. Consequently, in some Agencies, time is sometimes lost on explaining issues or discussing problems raised as a result of a lack of knowledge or a misunderstanding of the institutional context.

·While applicable in varying degrees to the Agencies concerned and in line with the Founding Regulations, a portion of the topics covered by management board meetings concern administrative matters, sometimes overshadowing core business topics. In Agencies where such discussions represent a significant part of the discussions, this raises the question whether this is the best use of the time of people who are mainly policy experts. Although stakeholder expertise in the Agencies’ work is an important asset that cannot be compromised for the sake of financial efficiency, there are other models that might allow involving stakeholders in an equally effective but more efficient manner.

·Even though the mission of each agency is laid down in their respective founding acts, groups will tend to represent their specific interests; this can result in a careful balancing act for the Agencies’ management and a potential loss of focus in their objectives and activities 177 .

The ETF’s governing model is considered to be financially and procedurally efficient. It is also in line with the Common Approach mentioned above. A smaller board facilitates smooth logistics for in-person meetings and information support from the agency. Along with effective and efficient decision-making, this allows for clearly defined priorities and focused activities. 178 On the other hand, as there is no involvement of social partners, the main concern is lack of ownership and less precise targeting of policies and tools.

Smaller sized executive boards mitigate to some extent some of the tripartite model’s inefficiencies described above. However, the analysis of other governance models in this evaluation’s supporting study points to possible alternatives that could keep the current model’s strengths (notably the involvement of stakeholders, in particular social partners) and address its weaknesses.

In this context, the evaluation considered the broader EU governance structure in the domains concerned, with the Advisory Committee on Vocational Training (ACVT) 179 and the CEDEFOP management board both covering the same policies (VET), even if their roles are different. The same applies to the Advisory Committee on Safety and Health at Work 180 and the EU-OSHA management board. A clearer distinction between the advisory and governance roles in the management boards could allow stakeholders to focus on their domain of expertise and offer significant efficiency gains.

A case in point is the fact that the composition of existing advisory committees and the corresponding Management Board overlaps significantly (ACVT/Cedefop, and OSH Advisory Committee/EU-OSHA 181 ). In this context, the ELA model could be explored (while the ELA management board is not tripartite, social partners are involved through a separate stakeholder group, which gives thematic expertise to the management board) 182 .

4.1.3.1 Coherence between the Agencies mandates and activities

During the evaluation period, the mandates and activities of the four Agencies were coherent. The four Agencies’ mandates identify broad policy themes in which they operate (see Section 2.1). While the four Agencies worked on related policy areas (labour markets, vocational education and training, skills, qualifications, health and safety at work, and the working environment), each Agency approached the broad policy themes from different perspectives. Where two or more Agencies’ mandates covered the same policy, the Agencies cooperated. Cooperation between the Agencies was closer and deeper than in the previous evaluation period, and the evaluation found no instances of activities being duplicated.

Considering the Agencies’ mandates and strategic objectives, there are shared themes addressed by two or more Agencies. However, evidence shows that the Agencies approached these common policy themes from different perspectives or a different geographical focus. The most significant cases concerned three pairs of agencies.

(i) Cedefop and the ETF both worked on issues such as VET 183 . However, the actual focus of the two Agencies’ work in this area was clearly distinguished: the ETF focused on capacity building and policy implementation in partner countries outside the EU, while Cedefop focused on research and policymaking in the EU.

(ii) Cedefop and Eurofound worked on the intersection between VET, skills mismatches, working and living conditions and labour market policies 184 . The two Agencies approached these topics from different angles: Cedefop focused on skills mismatches and VET providers from the supply side, while Eurofound focused on the demand side.

(iii) Eurofound and EU-OSHA worked on occupational health and safety as this is connected to Eurofound’s wider ‘working conditions’ field. It is noteworthy that EU-OSHA was established after Eurofound. The Founding Regulation of EU-OSHA established a specific focus on providing highly technical and specialised support for protecting the safety and health of workers. This mandate requires a high level of expertise, especially in light of the rapid societal, technological and scientific developments in occupational safety and health. The Agencies’ focus and ways of working are also different. Eurofound focuses on broad research (e.g. European working conditions survey), whereas EU-OSHA focuses on providing workplaces with practical knowledge and tools for implementing OSH rules and raising awareness, using their network of national focal points.

According to the open public consultation, most stakeholders familiar with the Agencies’ mandates and activities agree that they are coherent or very coherent (53% of respondents, N=101). Only 3% of respondents believed that the mandates and activities were incoherent or very incoherent. 46% did not know if the mandates and activities were coherent and were not able to answer this question.

The current evaluation has found no instances of duplicated work across the Agencies 185 . While the Agencies’ mandates covered broad, complex and interconnected themes, the common thematic areas were well understood by most policymakers and agency staff. Thematic overlaps in the mandates did not necessarily mean duplicating work. In fact, common thematic areas were both accepted and used as a basis for inter-agency cooperation, which increased consistency. For examples of joint output and activities delivered by the agency pairs related to common thematic areas, see the dedicated section below and Table 4 in Annex VII. Moreover, staff in all agencies were much more aware of each other’s work than in the previous evaluation and were therefore able to complement each other’s work rather than duplicate it.

The evaluation also assessed whether the creation of the European Labour Authority (ELA) could prompt a reassessment of the previous evaluation’s conclusions on merging the Agencies. In this regard, the risk of the Agencies duplicating the work of ELA was limited. EU-OSHA, Cedefop, and Eurofound were predominantly research-focused and did not have an operational and cross-border focus as the ELA did. The potential for merging the ELA with the ETF was also limited as the ETF only dealt with non-EU countries, mainly in the EU’s neighbourhood.

Cooperation between the four Agencies and the ELA was at an early stage due to the recent establishment of the ELA. Nevertheless, the three tripartite agencies had begun to cooperate with ELA on certain issues, for example on seasonal workers campaigns (EU-OSHA) and skills intelligence and job shortages (Eurofound and Cedefop). All five agencies cooperated when the ELA participated in the September 2022 event ‘Employment, skills and social policies that work for young Europeans in times of uncertainly’.

Nevertheless, there is scope for further cooperation with the ELA. For instance, the four Agencies could support the ELA by sharing their existing research in their respective areas of competence; this could also lead to pooling resources or producing joint reports on issues of common concern, e.g. on restructuring or respecting occupational health and safety rules in mobility situations, including posting, in the context of risk assessment. The evaluation identified two further areas for cooperation with ELA: (i) digital labour and labour mobility with ETF; and (ii) OSH implementation and workplace working conditions with EU-OSHA.

4.1.3.2 Cooperation between the four Agencies

Joint work and cooperation increased during the evaluation period. The 2011-2016 evaluation showed some cooperation between the agencies, particularly between Cedefop and ETF, Eurofound and EU-OSHA, and Eurofound and Cedefop. Formal agreements already existed between them, but cooperation was mostly limited to the mere exchange of information and knowledge, without leading to joint outputs and lacking strategic planning. Therefore, it identified a need for strengthening cooperation, engaging in joint-value creation, developing mechanisms for joint delivery, and sharing support services.

From 2017 to 2022, cooperation and synergies among the Agencies intensified, with a rise in the number of initiatives and projects where Agencies have cooperated. This includes the development of joint research, tools, events, and publications, demonstrating genuine cooperation with clear shared input and output from two or more agencies. This increased cooperation was at both the strategic (board) level and project/initiative level.

Cooperation to share administrative functions, such as HR and finance, also increased during the evaluation period. The Agencies were more likely to procure services together than in the previous evaluation period. They launched joint framework contracts and cooperated with other agencies and the Commission. As discussed below, opportunities could be explored for sharing more staff and office space in Brussels.

The benefits of cooperation between the Agencies are in three main areas. First, improved coordination and complementarity arising from interaction at different levels on different activities: there are over 20 formal agreements in place, and over 30 joint projects (events, research, procurement) have been delivered as a result of cooperation. The benefits of the Agencies complementing each other during this evaluation period are best illustrated through the joint work of the Agencies on EU cross-cutting priorities (the green transition, digital economy and migration). Second, improved relevance and quality of joint Agencies outputs. Joint outputs benefit from co-production, through broader evidence or expertise and the possibility to reflect multiple perspectives (e.g. Eurofound-Cedefop joint working on skills forecast). Moreover, joint dissemination has a wider reach due to the use of the Agencies’ multiple communication channels. This is a marked advance compared to the previous evaluation period where separate reports were produced on similar issues for the same audience. Third, improved efficiency of individual Agency operations thanks to co-financing. A case in point was the joint Eurofound / Cedefop European Company Survey (2019).

The extent of cooperation between the four Agencies is analysed below under four headings: (i) joint cooperation agreements, activities, events and data collection; (ii) cooperation between management boards; (iii) cooperation on administrative and organisational functions; and (iv) joint communication activities.

(i) Joint cooperation agreements, activities, events and data collection

Overall, the number of joint action plans and cooperation agreements between the Agencies increased (in particular, between Cedefop and Eurofound, EU-OSHA and Eurofound, Cedefop and the ETF, and the ETF and Eurofound) 186 . Staff surveys show that formal cooperation agreements increased staff awareness and clarified each agency’s role in projects. A number of joint activities and events were organised, and shared data collections were created.

·Cooperation between Cedefop and Eurofound became more extensive. The two Agencies launched an EU-wide joint European company survey. Moreover, joint actions and publications focused on skills forecasting and manufacturing. These included the ‘Skills forecast: trends and challenges to 2030’ report 187  and the joint report on ‘Workplace practices unlocking employee potential’ 188 . A recently revised memorandum of understanding includes contact points for all areas of joint interest.

·Cooperation between Eurofound and EU-OSHA is not new, and formal agreements followed by action plans date back to 2007. The two agencies have focused their cooperation on a range of themes, including psychological risks, mental health, self-employment, working-time patterns, burn-out, older workers, and foresight 189 . During the COVID-19 pandemic (May 2020), the European Parliament’s Employment and Social Affairs Committee (EP EMPL) held an exchange of views with the two agencies on the social and employment impact of the pandemic 190 . For details, see Annex IX, Section 4(i).

·Cooperation between Cedefop and the ETF, specifically on VET policy monitoring, is envisaged in their respective mandates. The agencies have a 3-year framework agreement 191 , which is implemented through a joint annual work programme annexed to each agency’s programming documents. The two agencies report on progress to the European Parliament every 2 years. The following joint actions were implemented during the evaluation period:

ojoint support for monitoring the Riga Conclusions deliverables in the VET field; 

oregular joint contributions to work on national and regional qualifications frameworks in cooperation with UNESCO (2017-2018, 2020);

ojoint implementation of the European Skills and Jobs Survey (ESJS) and joint monitoring of the implementation of the Osnabrück Declaration, supporting digital skills and inclusion in VET (from 2020); 

oCedefop and the ETF shared expertise on validating refugee work skills in Türkiye.

·Cooperation between the ETF and Eurofound increased, especially on social partnership and dialogue, public-private partnerships in VET, skills shortages and skills mismatches. The two agencies launched a joint online survey comparing the impact of the COVID-19 pandemic on EU Member States and selected ETF partner countries. The agencies also cooperated on themes related to platform work, NEETs and capacity building of social partners through sharing data, contributing to events and knowledge-sharing activities. In particular, they participated in or jointly organised seminars on the digital economy, platform work (2019, 2020, 2021) and the impact of COVID-19 on businesses (2021). Moreover, Eurofound participated in the advisory board of the ETF’s initiative on skills demand in partner countries (2020).

·Although the ETF and EU-OSHA do not have a joint action plan or framework agreement (due to limited possibilities for thematic cooperation), they do have a service level agreement in place to share an accounting officer (see point (iv) below).

·Cooperation between the Agencies increased on pan-European surveys. This cooperation led to wider benefits, such as cost sharing (particularly for the European company survey), aligning definitions and tools, facilitating sharing data and best practices, and overcoming methodological difficulties. For more details, see Annex IX, Section 4(ii).

·The Agencies were also involved in several multilateral (more than two agencies), in particular on digitalisation, the COVID-19 pandemic, Russia’s war of aggression against Ukraine and readiness for future challenges/foresight. For more details see Annex IX, Section 4(iii) 192 .

·Overall, various stakeholder groups across both the public consultation and the stakeholder survey gave positive ratings to the joint publications, joint data collection and joint information or expertise shared. However, when analysing these surveys by specific stakeholder groups, there were no conclusive patterns on the perceived quality of these joint activities. 

(ii) Cooperation between management boards

Each agency has a management board to provide leadership and strategic direction. All the boards strengthened inter-agency cooperation by exchanging information at their level on activities, priorities and operational issues. Most boards now include representatives from the other agencies, and there is a general agreement on cross-participation among boards. This leads to better understanding the ongoing work of other agencies, sharing programming documents earlier, developing joint action plans and generally making the Agencies’ activities more consistent. Previously, cooperation between the Agencies was primarily bottom-up rather than based on joint strategic planning 193 . Although board-level cooperation is positive, the Agencies’ staff 194 did point out that participation in other board meetings is time-consuming.

(iii) Cooperation on administrative and organisational functions

Beyond operational matters, the four Agencies cooperated on various administrative and organisational functions; however, there could be room to improve the scope of this cooperation. In terms of sharing and co-funding staffing, there are a few instances of collaboration in HR, IT, data protection and legal services. One particular example is the formal ETF–EU-OSHA agreement to share an accounting officer through a service level agreement 195 . Where feasible, opportunities could be further explored for sharing roles that require technical or specialist skills, for example, in cybersecurity, employment statistics and labour market economics.

On office space, close cooperation takes place between the staff of the three existing Brussels Liaison Offices (Brussels Liaison Offices were set up by all four Agencies, although the ETF closed its office in 2019). These offices support partnership with stakeholders, monitor changes in EU policymaking and contribute to the Agencies’ strategic management and communication. The recommendation from the previous evaluation to merge agency offices was not implemented. The possibility to share office space in Brussels could be considered. Locating all liaison office staff in one Brussels could stimulate ongoing and ad hoc collaboration and allow agency staff visiting from their headquarters to work together and save costs across all four Agencies.

On joint procurement, in contrast to the previous evaluation period, the Agencies were increasingly likely to procure work together. Between 2017 and 2022, Cedefop participated in 18 joint procurements led by other EU agencies and launched one joint procedure for its ICT helpdesk service with ENISA (the EU Agency for Cybersecurity) in 2019. It also participated in 40 Commission-led procurements for HR, ICT, audits and communications activities and launched one joint procedure with Eurostat. Eurofound launched two multiple framework contracts for evaluation and feedback services. The 2017-2020 contract was shared among eight EU agencies. The 2021-2025 contract is shared among 15 EU agencies. Eurofound also launched a joint procurement procedure in cooperation with Cedefop to align its performance measurement system with other agencies. The ETF has participated in five joint tenders since 2017 (although not all were with other agencies), and 16% of its contracts were concluded following joint procurement tenders with other agencies or with Commission departments. Lastly, EU-OSHA also participated in joint procurement activities with other agencies and the Commission to save costs, improve quality and increase efficiency. The agency also chaired the EUAN Shared Services Task Force. 

(iv) Joint communication activities

Joint communication activities were largely focused on outputs and events. The Agencies’ staff emphasised that joint dissemination ‘reach’ was greater when two or more agencies used their dedicated networks, databases and other means to engage with stakeholders.

For example, the report titled ‘Towards age-friendly work in Europe: a life-course perspective on work and ageing from EU Agencies’ 196 and another report, ‘Health and safety risks at the workplace: a joint analysis of three major surveys’ 197 , were shared across multiple channels (membership databases, agency websites, newsletters, mentioned at events).

Staff highly rated the cooperation and joint activities with other agencies under the remit of DG EMPL (Figure 11 in Annex VII) 198 . However, cooperation with the ELA was assessed slightly less positively compared to other agencies. This may be attributed to the ELA being a newly established agency, which has allowed for less time for develop joint activities.

4.1.3.3 Coherence with the European Commission’s strategic priorities for 2019-2024

The mandates, activities and outputs of the four Agencies were coherent with the Commission’s strategic priorities for 2019-2024. In particular, the Agencies produced many tailored outputs related to the priorities 199  ‘A Europe fit for the digital age’ and ‘An economy that works for people’. The Agencies’ work was aligned with new strategic priorities tailored to initiatives such as the European Green Deal (see Table 5 in Annex VII for more details about the coherence of the four Agencies’ mandates and activities with the Commission’s strategic priorities for 2019–2024).

Figure 3: Coherence of the Agencies’ mandates and activities with other EU programmes/initiatives

Source: Public consultation, question: To what extent do you think that the Agencies’ mandates and activities are coherent with other relevant EU programmes/initiatives?

Most respondents to the public consultation believed that the Agencies’ mandates and activities were very coherent or coherent with those of the relevant EU strategic priorities (see Figure 3).

4.1.3.4 Coherence with DG EMPL and other Commission departments

The Agencies’ mandates and activities were coherent with those of DG EMPL: all four Agencies worked in thematic areas that were relevant to and coherent with DG EMPL policies, in particular the implementation of the European Pillar of Social Rights (see Section 4.1.1 on effectiveness for detailed evidence).

The ETF’s mandate is a special case as it focuses exclusively on the partner countries outside the EU, while DG EMPL focuses largely on EU Member States. Consequently, the ETF also worked with other DGs with an international profile (e.g. DG NEAR, DG INTPA).

DG EMPL was represented in the management boards of each agency, providing input to the programming cycle and decision-making at board level. In particular, DG EMPL participated in the executive and management boards of Eurofound, Cedefop and EU-OSHA, with director-level representation in each. The ETF Governing Board is chaired by the Director-General of DG EMPL.

Overall, DG EMPL and the Agencies worked collaboratively during the evaluation period, and the Agencies’ outputs were used to inform EU policies, particularly those of DG EMPL.

During the evaluation period, progress was made on having more regular formal and informal communication between DG EMPL and the Agencies. Each agency has set up cooperation channels with DG EMPL, ranging from day-to-day cooperation at project level to interaction between senior managers in the Agencies and DG EMPL.

On thematic contributions to DG EMPL policies, events, working groups and advisory committees, each agency worked with DG EMPL based on its specific role. Most often, the Agencies provide research, expertise and input into policy documents or proposals and also participate in a variety of formal committees and other meetings. For further details, see Annex IX (iv) and Section 4.1.1 on effectiveness.

Coherence with other Commission departments

The relationship between the ETF and DG EMPL is distinctive due to each body’s different geographical coverage. This requires coordination among several DGs. The structured dialogue process 200 between DG EMPL, DG NEAR and DG INTPA is well-established 201 . However, despite coordination, the different DGs have competing expectations of where the ETF should focus its work (reflecting the different DGs’ objectives and geographic responsibilities). An agreed joint vision between the different DGs and the ETF could benefit the agency’s work.

For EU-OSHA, the evaluation found that cooperation with other Commission departments could be strengthened for small and medium-sized companies, research dissemination and mental health. This could involve closer cooperation with several Commission departments (DG JUST, DG FISMA, DG GROW and DG SANTE) and EASME, the executive agency for SMEs (supporting study Annex 4, pp. 104-5).

4.1.3.5 Coherence with other EU decentralised agencies

The mandates and activities of the four Agencies were consistent with those of other relevant decentralised agencies. Most cooperation between the four Agencies and other decentralised agencies took place through the EU Agencies Network (EUAN) and the EUAN sub-network, EU-ANSA (EU Agencies Network on Scientific Advice) 202 . For further details, see Annex IX, 4(v).

Via the EUAN, the four Agencies and other EU agencies pooled existing expertise to address new cross-cutting themes, such as the socio-economic impact of the COVID-19 pandemic, digitalisation and the green transition. Sub-networks enabled agencies working with primary research to hold regular in-depth methodological exchanges, for example, on sustainability and strategic foresight. Another benefit of cooperation within EUAN was staff cooperation on identifying solutions to common challenges and cost-saving measures (e.g. on risk management, the sustainable management of buildings and teleworking arrangements).

Within the Performance Development Network (PDN), a sub-network of the EUAN 203 , Eurofound played a leading role in developing an evaluation handbook and catalogue of key performance indicators across all four Agencies. The Agencies also took part in PDN exchanges on experiences with user satisfaction.

EU-OSHA intensified cooperation with the European Centre for Disease Prevention and Control (ECDC), particularly in response to the COVID-19 pandemic. Notably, this was the first time EU-OSHA cooperated with other agencies on public health issues.

In 2021, the ETF began cooperation with EIGE on the gender equality index. Given the overlapping geographical mandates of ETF and EIGE, which include EU candidate countries and potential candidates, there is an opportunity to improve the consistency of the two agencies’ activities. This would give ETF the opportunity to mainstream the gender equality part of the European Social Pillar in EU partner countries.

4.1.3.6 Coherence with other international organisations 

All four Agencies cooperated with international organisations active in the same policy areas through joint publications, events and surveys. This cooperation intensified, in particular, with the ILO and the OECD through cooperation on new thematic areas.

All four Agencies had already cooperated with the ILO before 2017 leading to joint outputs produced between 2017 and 2022. Cooperation has increased, in particular as a result of the COVID-19 pandemic, as has the use of joint online surveys. For further details, see Annex IX, 4(vi).

Eurofound, Cedefop and the ETF cooperated with the OECD. Eurofound and the OECD cooperated on job quality, collective bargaining, minimum and living wages and platform work. Cedefop and the OECD have cooperated on the role of VET in employing refugee and migrant workers since 2017, and in 2019, a 2-yearly joint event on skills was launched. The ETF cooperated with the OECD on skills-demand analysis and foresight, qualification systems and working with companies on skills development.

EU-OSHA cooperated with the World Health Organization, including via data sharing on healthcare. The two organisations attended each other’s events and organised joint workshops 204 .

The ETF collaborated with the ILO, the OECD, UNICEF and UNESCO, and this cooperation has intensified since the start of the COVID-19 pandemic (see supporting study, Annex 4, Section 6.2.3.1). However, as the international landscape for human capital development becomes more complex, involving a growing number of non-traditional players, there is an opportunity for the ETF to strengthen cooperation with international donors and financial institutions (within the scope of its mandate).

The four Agencies also continued to cooperate multilaterally with international organisations and regularly conducted joint activities with them. These included the ILO, the OECD, UNICEF and UNESCO.

4.1.3.7 Coherence with the national level

Consistency with national-level work is supported through the representation of Member States in all the Agencies’ management/governing boards. National social partners are also represented in the management boards of the three tripartite Agencies. Member States and social partners have two roles: (i) they provide input to the Agencies’ programming documents; and (ii) act as multipliers for disseminating the Agencies’ output. The extent to which the four Agencies provide direct support to Member States and partner countries varies according to their mandates and their operations. A number of examples can be given for each agency.

EU-OSHA conducts OSH communication campaigns in Member States that would not otherwise be possible in those that have less funding available for OSH systems. The OiRA tool is well regarded in many Member States as the tool is referred to in national OSH strategies and legislation. Broader references to risk assessment tools suggest that EU-OSHA awareness-raising has influenced national policies.

Cedefop prepared country-focused thematic reviews and attended events organised at national level. In 2021, thematic reviews and peer-learning activities were organised on skills governance in Slovenia and on upskilling pathways in France and Italy. Cedefop supported Slovenia’s Labour Ministry in developing national skills forecast and a career platform. The agency contributed to a total of 18 events organised at national level.

Eurofound’s outputs are not directly aimed at specific Member States. Nevertheless, Belgium, France, Slovenia and Spain provided additional funding to increase the sample size of the European working conditions survey (EWCS) for their countries for a more in-depth analysis. Eurofound also participated in the research days of the Irish Low Pay Commission, and an Irish government speech from April 2022 acknowledged Eurofound research.

The ETF works to support human capital development reforms in partner countries. In Moldova, support for VET reforms contributed to sectoral committees being set up to address labour market needs. In Türkiye, the support provided for setting up centres for the recognition of prior learning contributed to establishing 25 centres for VET recognition. National ministries in Ukraine and Azerbaijan have used SELFIE (Self-reflection on Effective Learning by Fostering the use of Innovative Educational technologies) as a source for policy discussions.

4.1.3.8 Potential for mergers: previous evaluation findings and current evaluation assessment

Based on the assessment of the Agencies’ mandates and common thematic areas, the previous cross-cutting evaluation in 2019 identified which Agencies might be more suitable for a merger and assessed the benefits and risks of each merger scenario.

The previous evaluation concluded that there was no straightforward scenario for mergers and that, to improve efficiency, there was rather room for strengthening cooperation. Although Cedefop/ETF and Cedefop/Eurofound merger options could be considered, they were high-risk scenarios, and there were difficulties in balancing the positive and negative effects. In both merger options, potential synergies could be exploited through developing methodological expertise and the pooling of knowledge (e.g., in skills anticipation and matching). Cedefop/ETF merger cost savings were possible for research roles (estimated at up to EUR 2 million) and for back-office roles (up to EUR 1.5 million), whereas for Cedefop/Eurofound, they were estimated to amount to up to EUR 1.5 million for research roles and up to EUR 1.5 million for back-office roles (see more details on costs and benefits of these two merger options in Annex IX, Table 3: Overview of benefits and risks of agency merger scenarios).

A Eurofound/EU-OSHA merger was also considered. Cost savings for research roles are estimated at up to EUR 1 million and up to EUR 0.5-1 million for back-office roles). Potential synergies could be exploited by pooling methodological expertise in researching working conditions and OSH issues. However, it was concluded that a possible merger would face significant ‘stumbling blocks’, especially because the work of EU-OSHA is highly specialised. Moreover, there is a clear risk that Eurofound, as the larger Agency, would become the dominant partner and OSH policy issues are given insufficient priority relative to other issues. This would reduce their visibility and impact at the EU level.

In all merger scenarios, closure costs are high, around EUR  2-3 million for each agency. Overall, as individual agencies were shown to be overall efficient and effective, the previous evaluation noted that one of the key reasons for a merger – joining a ‘weaker’ organisation to a ‘stronger’ one – was absent.

The support study for this evaluation confirmed the previous conclusions on the feasibility of mergers as there were no substantial changes to the mandates and activities during the current evaluation period, and the potential for duplication with the ELA is limited. The 2019 revision of the tripartite Agencies’ Founding Regulations did not change or expand the scope of the Agencies’ mandates, and the Agencies are putting in place cooperation mechanisms with the ELA to create synergies where possible. For a detailed analysis of the benefits and risks of the different agency merger scenarios, see Table 3 in Annex IX 205 .

4.2.How did the EU intervention make a difference and to whom?

This chapter examines the added value of the Agencies’ work and how they made a difference to their stakeholders. It analyses the extent to which the Agencies’ contributions are unique when compared to those of other agencies and organisations and the extent to which their activities could be substituted by other EU, international or national organisations.

Like the previous evaluation 206 , this evaluation found that the four Agencies continued to bring unique value through: (i) specialised thematic knowledge; (ii) high-quality data, tools and methodologies; (iii) extensive European coverage in their respective fields, ensuring consistent data availability; and (iv) maintaining a tailored focus on EU policy needs 207 . This is confirmed by the results of the stakeholder survey (see Figure 7 in Annex VII).

Overall, the Agencies’ stakeholders considered that other organisations could either partially or not at all replace the Agencies’ activities, emphasising the Agencies’ unique contribution. Around 75-80% of Cedefop and the ETF’s stakeholders believed that ceasing the Agencies’ activities would affect their work; for Eurofound and EU-OSHA 208 , over 90% of their stakeholders shared this view. This feedback was comparable to the previous evaluation’s findings. For the ETF, some stakeholders (35 % of respondents to the stakeholders survey, N=91) indicated that international organisations, such as UNESCO, UNICEF, the OECD, and the ILO could carry out similar work to ETF, matching its level of expertise and organisational capacity 209 .

The Agencies generate significant value for various stakeholders, albeit to different degrees.

The Commission remained the primary beneficiary of all four Agencies, and the evaluation found extensive evidence of the added-value and benefits that the Commission gained from their support. Examples include Eurofound’s knowledge and input in preparing one of the six political priorities of the Von der Leyen Commission in 2019 (An economy that works for people). Cedefop supported the Commission through its initiatives and support for empowering adults through upskilling and reskilling 210 , among others. EU-OSHA played a key role in shaping EU OSH policy, particularly the EU’s strategic frameworks on health and safety at work (see EU-OSHA Case Study 5). Lastly, the ETF provided effective support to EU development cooperation and external relations. (See Section 4.1.1.3).

The evaluation underscored the added value provided by the Agencies to Member State stakeholders (and EU partner countries for ETF), offering quality data, extensive European coverage and thematic expertise. Cedefop provided support to education and training providers, academic/research institutions and social partners. EU-OSHA provided insights into national policymaking, benefiting social partners in particular, especially during the COVID-19 pandemic. EU-OSHA and Eurofound supported employers and trade union associations. The ETF supported education and training providers 211 . The evaluation suggested that the ETF could strengthen its collaboration with businesses and social partners.

Other stakeholders benefited from the Agencies’ support less often: the EU public, businesses, EEA/ETFA national public authorities, other EU decentralised agencies and non-governmental organisations (NGOs). The public and NGOs do not directly benefit from the Agencies. However, considering the expending thematic scope and current budget constraints, the evaluation identified opportunities to involve more NGOs. NGOs could serve as multipliers to increase the reach and dissemination of the Agencies’ outputs with a minimal impact on costs.

Main sources of added value across the Agencies are described below.

EUROFOUND. Eurofound provided strong added value through its pan-European surveys 212 . The surveys offer a unique and established source of comparative information on the quality of living and working conditions across the EU, providing EU-wide data that is consistent, reliable and covers a wide range of themes. Eurofound’s surveys offer much more detailed harmonised information that complements the yearly data that Eurostat produces in the Labour Force Survey. The EU-level data helped to understand dynamics that were not apparent at individual Member State level. The surveys inform and guide a substantial part of Eurofound’s analytical work. The methodology and questionnaires used in Eurofound’s pan-European surveys are freely available for use by other researchers, subject to certain copyright conditions. Moreover, they serve as an inspiration for other surveys worldwide, such as the American Working Conditions Survey, the South Korea Working Conditions Survey, the Israel Social Survey and the China Urban Labour Survey 213 . The United Nations Development Programme (UNDP) has used Eurofound’s European quality of life survey (EQLS) questionnaire to carry out surveys on the quality of life in Croatia and North Macedonia.

Eurofound’s representativeness studies were important for social dialogue and EU lawmaking 214 . As highlighted in the validation workshop for this study, the representativeness studies were crucial for ensuring proper social dialogue at EU level and facilitating social dialogue at national level (thus also having a national impact). Eurofound’s methodology is reliable and extensive (having conducted these studies since 2006), and this has strengthened these studies’ credibility. While these studies were beneficial for social partners, providing visibility and influencing national agendas, some interviewees, including members of the executive and management boards, expressed concerns about the high cost of these studies and questioned their overall added value to stakeholders, especially at national level. In the final years of the current evaluation period, Eurofound increased its national focus as outlined in its 2021-2024 work programme. To improve national adoption of its research, Eurofound built on its Network of Eurofound Correspondents and contributed to national events on working conditions and pay. Eurofound’s work has been used more extensively at national level in recent years, particularly from 2020, demonstrating its use beyond EU level.

CEDEFOP. Cedefop brought EU added value by providing evidence with unique thematic and European coverage. In the current evaluation period, Cedefop’s increased focus on supporting EU policymaking and implementation was recognised by the Commission as a significant source of added value, in particular to support the 2020 European Skills Agenda and the 2020 Council Recommendation on VET (see detailed evidence in Section 4.1.1.3). It has also added value by providing consistent definitions and concepts across Member States to facilitate the implementation of agreed EU policies (e.g. apprenticeships, transparency tools, upskilling pathways, individual learning accounts and micro-credentials).

Cedefop adds value to Eurostat’s statistical data, complementing and contextualising it with quantitative and qualitative information obtained from surveys and ReferNet partners 215 . Cedefop’s data have also provided added value for the EU migration network, and, beyond the employment domain, have been used to test different scenarios in the work of DG GROW, DG ENER and DG ENV.

At national level, Cedefop’s high added value also stemmed from producing comparable EU-level data, such as the VET opinion survey and skills forecast, which national organisations often lack the capacity to produce. Additionally, during the current evaluation period, Cedefop developed tools and repositories accessible across Europe, including the Financing adult learning database, the European Skill Index (ESI) and the Skills Online Vacancy Analysis Tool for Europe (Skills OVATE). The ex-post evaluation of Cedefop’s projects and tools 216 confirmed how unique they were, with no national or regional equivalents. The agency provided EU comparable data on VET, skills and qualifications to researchers and academics, which they could not have accessed otherwise. Cedefop’s 2022 user satisfaction survey highlighted the frequent use of Cedefop output to strengthen knowledge, understand policy issues, trends or challenges, and support research/policy analysis. 217  

ETF. The ETF’s added value comes from its distinctive expertise in both EU and partner countries’ policies, a breadth of knowledge not found in other EU institutions or partner countries. The agency served as a bridge, transferring experience and expertise from Member States and EU institutions to partner countries, thereby strengthening the capacity of those countries’ administrations. The ETF also added value to EU external policy through the advice it provided to Commission departments drawing on its expertise in neighbourhood and enlargement countries. The ETF’s role in providing both EU and non-EU expertise, producing reports and sharing methodologies has proven beneficial for VET and TVET providers in partner countries. The Agency’s networks of VET and skills policy experts (see Box 5) provided readily available thematic and synthetised information that informed VET policy. This information would not have been available without the ETF 218 . The ETF works closely with other data organisations, including Eurostat, for example for the Western Balkans, Türkiye and the neighbourhood countries, to avoid any duplication. All data produced by the ETF on country intelligence are not available anywhere else.

Box 5: Supporting education and training providers – the ETF’s networking activities

Additionally, the ETF’s unique work on Russia’s war of aggression against Ukraine (including its manuals to include refugees in the job market) has become a primary source of information. An example of uptake of the ETF work by the Commission and Member States is the Ukrainian Resource Hub , which has had more than 27 000 visitors since its launch 220 . 

EU-OSHA was particularly valued for its support to Member States, extensive thematic coverage, and effective awareness-raising efforts, confirming the added value of EU-OSHA at national level. In addition, the agency brings added value to the Commission by supporting OSH policymaking and implementation. The agency’s stakeholders emphasised the added value of EU-OSHA’s OSH assessment tools, awareness campaigns and networking activities involving focal points, including the ESENER survey, the OiRA tool and the healthy workplaces campaigns (e.g. the 2019 campaign, which supported EU policymaking in OSH, see Box 4 in Section 4.1.1.3). The European Parliament study, ‘Cost of Non-EU Agencies Focusing on the Health and Safety Cluster of the EU Decentralised Agencies’ 221 , highlighted that EU-OSHA provided significant support to Member States by bringing the EU perspective to the work of national health and safety agencies. Additionally, the evaluation demonstrated EU-OSHA’s essential role in conducting awareness-raising campaigns on OSH at the workplace, addressing gaps at the national level 222 (case study on EU-OSHA’s healthy workplaces campaigns). In particular, EU-OSHA brought significant added value to countries with less advanced OSH systems 223 . There is potential to increase EU-OSHA’s impact in those countries if the required resources are available.

Box 6: EU-OSHA European coverage of publications – ESENER survey

4.3.Is the intervention still relevant?

The Agencies’ mandates, objectives and activities are highly relevant. The Agencies contribute very significantly to the EU policy priorities. They also address their stakeholders’ needs, although there is still room for improvement.

The four Agencies developed their work programmes in response to EU policies and priorities. There was a clear link between EU priorities and the Agencies’ key programming and reporting documents – the Agencies’ work was often referred to in EU policy documents. The expertise of the Agencies’ staff was leveraged to inform policy and programme development. Table 8 in Annex VII shows Agencies’ outputs corresponding to the Commission’s 2019-2024 strategic priorities. See also Annex IX for examples of the Agencies’ work that contributes to EU policy needs.

EUROFOUND closely aligned its activities with DG EMPL’s three main areas of action: supporting the European Semester, fostering stronger social dialogue and promoting decent and safe working conditions for all 225 .

EU-OSHA successfully contributed to the EU’s OSH legislation and strategies. Tools for OSH management, especially OiRA, have contributed to EU policy needs by promoting a culture of prevention, particularly in small and medium-sized companies.

CEDEFOP closely aligned its work with EU upskilling and reskilling policies, in particular the European Skills Agenda, the Council recommendations on VET, the Osnabrück Declaration and the Digital Education action plan 226 . 

The ETF closely aligned its work with EU policy priorities. The agency also significantly contributed to the EU’s external relations policies and programmes. While the ETF’s work is highly relevant, its close relations with different Commission departments led to competing expectations about the geographical scope and focus of its activities (e.g. focusing on INTPA countries versus neighbourhood/candidate countries). Consequently, there is a need to clarify the agency’s long-term vision and which of its activities should be prioritised.

The Agencies’ priorities, mandates and activities addressed their stakeholders’ needs. The Agencies were able to monitor and adjust to them thanks to their respective monitoring and feedback systems (dedicated user surveys 227 or participatory activities 228 ).

Over 90% of EU-OSHA stakeholders and over 75% of ETF, Cedefop and Eurofound stakeholders considered that the Agencies met at least partially their needs (evaluation feedback survey – see Figure 3). Analysis by stakeholder group (to be treated with caution as the sample is relatively small) shows that over 80% of trade unions and consultancy organisations found Eurofound’s activities and output very relevant to their work. This figure was around 45% for the EU institutions and national governments.

On Cedefop, 75% of trade union respondents considered the Agency’s activities and outputs to be very relevant, compared to 42% of employer institutions and VET providers. A 2022 user satisfaction survey suggests that Cedefop output related to forecasts and future trends are the most valuable 229 . Nonetheless, a limited number of trade union representatives argued that the agency should not focus as much on micro-credentials and individual learning outcomes. Other stakeholders indicated that the agency could respond more quickly to emerging trends and have a greater focus at regional level.

Figure 4: Stakeholder survey responses on the extent to which the Agencies' services responded to their needs

Source: Stakeholder survey.

Regarding the ETF, 75% of research or consultancy organisations considered the agency’s outputs and activities most relevant, compared to about 50% of VET providers and EU institutions. A Policy Delphi review from 2018 found that partner country stakeholders found the ETF’s outputs related to VET governance, the modernisation of VET systems, the facilitation of labour market reforms, and support for more effective skills and employability policies to be highly useful 230 . Moreover, the agency received 100% positive feedback on its support to EU services from 2017 to 2020.

Taken together, the survey, public consultation and other evaluation feedback suggests that greater efforts are needed to increase the visibility and uptake of ETF resources at national level  231 .

Figure 4 above shows that EU-OSHA had the lowest proportion of stakeholder survey respondents who indicated that the agency’s activities and outputs were only slightly or not at all relevant to their work. Stakeholders highlighted the agency’s outputs such as ESENER-2 data (which was used by research organisations for studies 232 ), the healthy workplaces campaign (which enriched national policy debate on OSH 233 ) and activities on strategic and operational networking (which was rated very highly in a recent user feedback survey 234 ). While the OiRA tool for OSH management has been widely used by EU-OSHA’s stakeholders, there is room to improve the tool by making the layout more appealing 235 .

Stakeholders and staff from Cedefop, EU-OSHA and Eurofound pointed out that different stakeholder groups may perceive relevance differently; therefore, meeting all stakeholder needs can be a challenge. Nevertheless, Cedefop, EU-OSHA and Eurofound all balanced different stakeholders’ needs effectively. In particular, for EU-OSHA, the tripartite structure of the management board was seen as fostering a strong level of ownership among stakeholders that contributed to the effective delivery of the agency’s work programme 236 . 

Figure 5: Stakeholder survey responses on the extent to which they were able to provide input on the Agencies’ priorities, activities and/or outputs

Source: Stakeholder survey.

While the Agencies’ activities and outputs were largely aligned with stakeholder needs, more opportunities could be given to stakeholders to provide feedback and shape the Agencies’ approaches. Figure 5 shows that, for both EU-OSHA and Eurofound, over two thirds of stakeholders indicated they were able to shape the Agencies’ priorities to a large or moderate extent. However, a lower proportion of ETF and Cedefop stakeholders indicated the same level of influence. The ETF is the only one of the four Agencies that does not regularly conduct user satisfaction surveys.

Finally, the COVID-19 pandemic and the war of aggression against Ukraine have not changed the relevance of the Agencies’ mandates or work. Each agency adapted its work programme where relevant and feasible to respond to the two crises by producing additional targeted output. Looking ahead, the Agencies are already working on highly relevant future trends, such as climate change and digitalisation, which stakeholders agree as important areas of focus. However, stakeholders also highlighted demographic change as a fundamental area of work, given Europe’s ageing population (see Figure 13, Annex VII). The Agencies’ mandates are already broad enough to accommodate these megatrends and the work programmes already include them to stay aligned with EU policy needs (see Section 4.1.1.4). Therefore, there is no need to modify the Agencies’ mandates or intervention logics.

5.1.Conclusions 

Based on the available evidence, the four Agencies operated effectively during the 2017-2022 evaluation period. Overall, the Agencies achieved many of their general, specific and operational objectives to a high degree, as was the case during the previous 2011-2016 evaluation period. There is however room for improvement in certain areas (see Section 5.2. on lessons-learned)

Operational objectives (activities)

The four Agencies completed the activities and outputs set out in their work programmes to a high level, despite budget and staff limitations and the COVID-19 pandemic. Delivery against key performance targets was good.

While the Agencies have improved their communication and dissemination activities (including increased social media use and presence) the evaluation highlighted scope for improvement in that field.

Specific objectives (results)

The four Agencies achieved to a great extent the expected result of providing EU (and ETF Partner Countries’) policymakers and stakeholders with high-quality information to support relevant policies. The evidence suggests that the quality of the Agencies’ outputs, as well as the use of these outputs by stakeholders, was high.

 Use of outputs - The Agencies’ KPI as well as staff and stakeholder consultations indicate that each of the Agencies’ services and outputs have been used by their key stakeholders. Use has increased compared to the previous evaluation period and is more widespread across the EU institutions compared to international or national stakeholders.

Quality of outputs – The user satisfaction of the Agencies’ outputs was very high (and has even increased for Cedefop compared with the previous period).

For each Agency, the outputs that were most used and of the highest quality were:

Eurofound - research related to COVID-19 and on working and living conditions, the three pan-European surveys (as in the previous period), and work on minimum wages;

Cedefop - skills anticipation and matching, monitoring and analysing VET policy developments and promoting access to and the VET’s appeal;

ETF – predicting skills demand, monitoring and diagnostics reports, innovative teaching and learning and vocational excellence provision models;

EU-OSHA – the OiRA tool and the healthy workplaces campaigns.

However, other results were achieved to a lesser extent, particularly for the ETF. Unlike agencies operating under EU boundaries, support to partner countries is naturally limited by external factors beyond the ETF’s control, such as insecurity, political instability, and economic volatility. The ETF’s achievement of results depends, to some extent, on the willingness of partner countries to engage with the support offered by the ETF.

Impacts: achievement of the general objectives

Each Agency achieved its general objectives, by positively contributing to EU policy making and implementation in the policy areas established in its mandate. Overall, the activities and outputs of the Agencies were aligned to the Commission’s priorities and adapted well to changes in EU policies (as in the previous evaluation period).

Examples of the most significant impacts include:

Eurofound: the work on digitalisation and new and non-standard forms of work (including the platform economy) helped the Commission and other EU bodies to underpin the EU legislative initiative on platform work.

Cedefop: the evidence and data gathered by Cedefop contributed to the renewed European Skills Agenda, the first Council Recommendation on VET and the Osnabrück Declaration. Cedefop has therefore contributed to improving policies on VET, skills and qualifications.

EU-OSHA made a significant contribution to the EU’s strategic frameworks on health and safety at work. It strongly supported policymaking with its work on COVID-19 and two healthy workplace campaigns.

ETF supported policymaking by improving the adaptation of VET governance to evolving skills demand and labour market transitions. Its work also helped to improve labour market policies and practices and informed EU external relations policy.

In addition, the Agencies generated wider impacts and demonstrated adaptability by responding, within the scope of their mandates, to unexpected events such as the COVID-19 pandemic and Russia's war of aggression against Ukraine.

Eurofound leveraged its longstanding work on structural changes in policy debates on how the COVID-19 pandemic was accelerating change in the workplace.

Cedefop supported the integration of Ukrainian refugees into VET.

The ETF played a key role in facilitating Commission support to Ukraine in response to Russia’s war of aggression. The ETF’s longstanding presence in and expertise on Eastern Partnership countries prepared the Agency for this unexpected role.

EU-OSHA’s healthy workplace campaigns contributed to improved social dialogue and cooperation among the national governments and social partners.

The available evidence shows that overall, the four Agencies were cost-effective. This was the case also in the previous evaluation. They lowered costs through a variety of measures while maintaining the quality of their outputs. The cost-effectiveness analysis of the key activities concluded that, for most of them, there was no scope to reduce costs without compromising quality. Stakeholders continued to highly rate the Agencies' expertise and outputs and had a generally positive view of the Agencies’ cost-effectiveness.

During the evaluation period, the Agencies took a variety of cost-saving measures to boost cost-effectiveness. For instance, joint procurement and sharing accounting services, identifying deliverables where costs could be shared (e.g. the joint Eurofound-Cedefop European company survey), and investing in actions that lead to long-term savings (e.g. increasing office energy efficiency, dissemination of digital rather than print outputs). The rise in digital usage during the COVID-19 pandemic also contributed to cost savings.

However, the assessment of administrative costs, staff allocation, and monitoring and reporting systems shows that the four Agencies have the potential to reduce the administrative burden and simplify processes even further. In this context, it should be noted that, during the reporting period, a higher proportion of administrative staff than two other Agencies – the EU Agency for Fundamental Rights (FRA) and the European Institute for Gender Equality (EIGE).

Monitoring and reporting systems

During the evaluation period, the four Agencies have developed internal control frameworks to monitor, report and evaluate their activities. These frameworks cover financial, human resources, and core business performance. They allow for a broad assessment of how resources are spent and translated into the planned outputs.

Despite these positive developments, the evaluation highlighted that the Agencies should continue to work on their monitoring and reporting systems. There is room to: (i) improve their monitoring systems and KPIs to better define targets and assess the scale of success/achievement of results; and (ii) simplify reporting. Such simplifications would help to reduce further the workload of administrative staff while maintaining the Agencies’ capacity to ensure accountability and assess performance.

Agencies’ governance

The tripartite governance of Eurofound, Cedefop, and EU-OSHA offers significant advantages, such as representation, strategic direction, and knowledge-sharing facilitated by government representatives and social partners. However, a primary challenge comes from the size of management boards and the number of voting members who must navigate compromises not only on the Agencies’ core businesses but also on administrative decisions, budgets, and infrastructure.

The evaluation explored alternative governance models with different ways to involve the social partners. Although expenditures on the larger boards are higher, they can be reduced to some extent through hybrid meetings. However, the size of tripartite management boards and the diversity of participating stakeholders raises questions about how to ensure that the discussions remain focused, and how to efficiently use stakeholder expertise in board meetings given that operations, budget, and administration can overshadow core business topics. While stakeholder expertise is an asset that contributes to the Agencies’ work and cannot be compromised for the sake of financial efficiency, other models may allow involving stakeholders in an equally effective, but more efficient manner.

Coherence among the Agencies

The four Agencies cooperated more actively and intensively than before 2017. Cooperation among the Agencies moved beyond passive information sharing and took place: (i) in a variety of forms (joint research, tool development, events and publications); (ii) on a variety of topics (labour markets, vocational education and training, skills, qualifications, health and safety at work, and the working environment); and (iii) at several levels (board level, project / initiative level, administrative functions, such as HR/finance). This cooperation led to a higher degree of coherence among the Agencies.

The assessment of the coherence across the four Agencies’ mandates shows that there are some overlaps in the thematic areas of the mandates (e.g. Cedefop and ETF, both dealing with VET; Eurofound, Cedefop and ETF on skills mismatches). However, these overlaps do not materialise in activity or output duplications because the Agencies approach these topics from different angles based on their expertise. As a result, the Agencies have complemented rather that duplicated their work. The Agencies have intensified inter-agency cooperation, producing more joint outputs based on the common thematic areas. In fact, the current evaluation has not found any example of duplication of activities.

Regarding the potential for mergers between the four Agencies, and based on the cross-cutting analysis of the four Agencies’ mandates and activities, as well as the fact that the current Agencies mandates have remained unchanged, the evaluation has not found evidence to change the conclusions from the previous 2019 evaluation. The 2019 evaluation concluded that: (i) merger options show difficulties in balancing positive and negative effects; and (ii) efficiency can be improved through better cooperation.

Concerning a potential merger of the European Labour Authority (ELA) with any of the other four EMPL Agencies, the ELA’s highly specialised mission and its very distinct types of activities do not suggest significant overlaps with the mandates of other agencies.

Coherence between the Agencies and other stakeholders

The four Agencies’ mandates and activities continued to be coherent with DG EMPL and EU policies. The mandates and work of the Agencies were in line with the Commission’s strategic priorities for 2019 to 24. The thematic areas covered by the Agencies were coherent with DG EMPL’s policies and the policies of other relevant Commission departments.

Cooperation between the Agencies and DG EMPL was good even though there remains scope for even stronger cooperation. DG EMPL participated in each Agency’s Management / Governing Board and commented on the Agencies work programmes. Day-to-day coordination also took place at operational level.

The four Agencies continued to cooperate with other relevant EU decentralised agencies, in particular, through the EU Agencies Network (EUAN). The four Agencies also intensified cooperation with international organisations, such as the ILO, the OECD, UNESCO. Cooperation with the recently established ELA has begun.

The four Agencies generated EU added value across a range of areas, as was the case in the previous evaluation. Stakeholders particularly appreciated the EU added value in relation to the “quality of information produced, specific thematic knowledge not available elsewhere and European coverage of the publications, surveys and data, and a focus on EU policy needs”.

The degree of EU added value varied according to each Agency’s specific mandate and area of work. Eurofound ranked highly in terms of European coverage / EU level comparability. Cedefop performed well in terms of quality of information produced. The ETF ranked highest in terms of representing the Commission’s external dimension in partner countries. EU-OSHA ranked particularly highly in terms of awareness raising.

Other examples of EU added value provided by the Agencies are:

- for Eurofound, a key area of EU added value are its surveys, providing unique insights into key living and working topics on a comparable basis. Eurofound scored highly in terms of its EU-wide comparable data which is reliable and has continuity over the years, and its thematic coverage.

- for Cedefop, added value arose from publications, surveys, and data with a unique thematic and European coverage, including the work on skills anticipation and forecasting. Also, for the tools and repositories shared across the EU, such as the financing adult learning database, the European Skill Index (ESI) and the Skills Online Vacancy Analysis Tool for Europe (Skills OVATE).

- for the ETF, its EU added value arose from its specific expertise in policies of both the EU and partner countries, which does not exist at such a scale within other EU institutions or partner countries. Stakeholders also praised the added value of the ETF’s work in respect of Russia’s war of aggression against Ukraine.

- EU-OSHA was valued particularly highly for its support to Member States and thematic coverage that adds a European dimension to the work of national health and safety agencies. This added value was particularly high in countries with less developed OSH systems. Key tools that provided EU added value include the ESENER survey, the healthy workplaces campaigns and the OiRA.

Overall, most stakeholders indicated that other organisations could only substitute to a small extent or not at all the Agencies’ services, and that ending the Agencies would affect their work. This suggests the Agencies generate strong EU added value as they make a distinct contribution that cannot be replaced.

The activities, objectives and mandates of the Agencies were highly relevant and responded to both EU policy and stakeholders’ needs. EU policies and priority areas were reflected in key programming documents. The work of the Agencies was often referred to in EU and other policy documents. Feedback from stakeholders also suggests that the Agencies responded to their needs.

No issues were identified which would put into question the future relevance of the mandates or work of the Agencies. The Agencies already work on highly relevant future trends, such as climate change and digitalisation. The Agencies’ stakeholders agree with the policy focus of the Agencies, but also pointed out that the ‘mega-trend’ of an ageing population could receive greater attention from the Agencies.

The Agencies demonstrated adaptability by responding to the COVID-19 pandemic and Russia’s war of aggression against Ukraine with additional targeted deliverables where relevant and feasible.

The Agencies responded promptly and transparently to audit recommendations.

5.2.Lessons-learned

Effectiveness

1. Overall, the use of and the quality of the four Agencies’ outputs is high. However, communication and dissemination could be further improved to better reach target audiences. The agencies could: (i) produce more ‘tailored’ outputs; and (ii) consider dedicating an appropriate level of budget and focus to the dissemination and communication activities (see Section 4.1.1.1.).

2. The four Agencies could explore how to better engage with the national level, developing a more targeted approach to awareness-raising and dissemination to better reach target audiences in Member States and the ETF partner countries (see Section 4.1.1.1. and Annex VIII).

3. The four Agencies contribute well to relevant EU political priorities and have incorporated the key EU themes into their work. The Agencies could continue to ensure that: (i) they address key themes such as green and digital transitions; and (ii) outputs related to key themes are properly disseminated so that stakeholders are aware of the Agencies’ efforts in these areas (see Section 4.1.1.5.).

Efficiency

4. There may be scope for the four Agencies to further reduce their administrative burden and simplify processes. The Agencies could: (i) examine how to further reduce their administrative burden on staff to maintain sustainable workload (e.g. further scope for cooperation and synergies between the Agencies); and (ii) reduce the share of staff engaged in administrative activities and increase the share of staff focused on the Agencies’ operations (see Section 4.1.2.).

5. The four Agencies could consider cutting costs and increasing cost-effectiveness by organising more hybrid or online meetings where possible, and without compromising the Agencies' outreach/presence among stakeholders. The Agencies could also further prioritise activities and outputs, using transparent criteria agreed with their governance bodies (see Section 4.1.2.4.).

6. Eurofound, Cedefop and EU-OSHA could consider how to address the balance between research needs and budgetary restrictions. They could explore additional funding possibilities, as was successfully by the ETF while remaining within the scope of their mandates (see Section 3.1.1.).

7. The governance of the tripartite Agencies offers significant advantages, such as representation, strategic direction, and knowledge-sharing facilitated by government and social partner representatives from all Member States. At the same time, due to their larger size, the Agencies’ management boards have relatively high operating costs, meetings tend to be long and complex to manage, decision-making can be challenging, and the overall management can be burdensome.

Where deemed necessary, a case-by-case exploration of the governance arrangements could be carried out while giving due consideration to the importance of social dialogue and the role of social partners in tripartite agencies. In this context, the future results from the ongoing ELA evaluation could be considered, as relevant, taking also into account the different nature of the respective governance arrangements (see Section 4.1.2.5.).

8. While the four Agencies’ control mechanisms are effective and overall efficient, the Agencies could review and significantly improve their monitoring systems, including KPIs, to properly assess the scale of success of their interventions, and to make the evaluations more useful in improving the Agencies’ performance. This could be achieved by: (i) defining SMART objectives; (ii) identifying meaningful KPIs to measure result achievement; (iii) systematically setting targets for all KPIs; (iv) filling data gaps (v) further harmonising and aligning indicators across the Agencies; and (vi) better monitoring of services / outputs use at national level. Moreover, the Agencies could further streamline monitoring processes and tools (streamlining of reporting) (see Section 1.1.2. and 4.1.2.3.).

Coherence

9. The Agencies could continue to explore shared or central services, where this adds value. This covers functions such as HR, legal and IT, and more specialist and technical roles (such as cyber security, employment statistics, and labour market economics) (see Section 4.1.2.4. and 4.1.3.2.).

10. The Agencies could explore shared office space, using the Brussels Liaison Office, to contribute to both coherence and efficiency (see Section 4.1.3.2.).

11. The Agencies could continue to explore with the Commission those areas in which working together brings most added value to quality of outputs and stakeholders’ use. These include cooperation between Eurofound and Cedefop on the European company survey, Cedefop and the ETF on VET policy monitoring, and Eurofound and EU-OSHA at the intersection of working environment, working conditions and occupational health and safety, including psychosocial risks and a focus on climate and environmental health issues. (Section 4.1.2.4. and 4.1.3.2.).