Table of contents

1.Introduction

2.Background description and programme objectives

2.1.Baseline and points of comparison

3.Implementation / state of play

3.1.Programming and budget

3.2.Type of operations funded

3.3.Financial and operational implementation

4.Method

4.1.Short description of the methodology

4.2.Limitations and robustness of findings

5.Analysis and answers to the evaluation questions

5.1.Effectiveness: How effective is the YEI, and other ESF-funded youth employment operations, in achieving their objectives?

5.2.Efficiency: How efficient has the YEI, and other youth employment operations funded by the ESF, been in the achievement of their objectives?

5.3.Relevance: How relevant is the YEI and the other ESF-funded youth employment operations?

5.4.Coherence: How coherent are YEI and the other ESF-funded youth employment operations among themselves, and with other actions in the same field?

5.5.EU added value: What is the EU added value of the YEI and other ESF-funded youth employment operations?

5.6.Sustainability: How sustainable are the YEI and the other ESF-funded youth employment operations?

6.Conclusions and lessons learned

6.1.OVERALL CONCLUSIONS

6.2.Lessons learned

Annex 1: Procedural information

Annex 2: Stakeholder consultation

Annex 3: Methods and analytical models

Figures, Tables and Boxes 

Figure 1    Youth employment intervention logic    

Figure 2    Youth unemployment rate (15-24) (% of active population), 2014    

Figure 3    The rate of young people (aged 15-24) not in employment, education or training, by labour market status (% of population), 2014    

Figure 4    Young people in the EU28 (aged 18-24) having left education early, by country and labour status, in 2014 and 2018    

Figure 5    Change in unemployment rates for young people aged 15-24 by level of education, 2014-2018 (in percentage points)    

Figure 6    Share of costs eligible for ESF funding by type of operation    

Figure 7    Implementation rates of YEI/ESF funding by Member State (expenditure declared to the managing authority vs funding allocated)    

Figure 8    Progress towards the milestones set for 2018 – YEI    

Figure 9    Share of NEETs (aged 15-24) receiving ESF/YEI funding as a proportion of the NEET population in 2014 by cluster of regions (%)    

Figure 10    How successful were the youth employment operations funded by the European Social Fund and the Youth Employment Initiative in providing support to the following target groups?    

Figure 11    Share of job offers with permanent contracts according to YEI evaluations    

Figure 12    To what extent do you agree/disagree that the following measures have been actually effective in helping young people enter quality and sustainable employment? (population: 399 organisations)    

Figure 13    How would you qualify the following administrative arrangements for the implementation of youth employment operations under the ESF and YEI?    

Figure 14    Overview of budget allocation to clusters of regions    

Figure 15    To what extent are the YEI and ESF coherent with other youth and youth employment EU national/regional schemes? (n=399)    

Figure 16    In your opinion, how useful are the following actions in helping young people find a job, improve their professional skills or enter the labour market? (Group A1, A2 and B, n=976)    

Figure 17    The support you received helped you in… (Group A1 and A2, n=431, multiple answers allowed)    

Figure 18    To what extent do you agree or disagree that the following activities implemented under the Youth Employment Initiative were cost-effective? (Group C, n=206)    

Figure 19    To what extent are the YEI and ESF coherent with other youth and youth employment EU national/regional schemes? (Group C and D, n=399)    

Figure 20    Has the EU provided support that goes beyond what is addressed by national or regional programmes or support schemes aimed at youth employment? (Group C and D, n=282)    

Table 1    Funding for Youth Employment ESF + YEI, including amendments until 2018    

Table 2    Comparative table of unit costs    

Table 3    Expenditure per entry: expenditure minimum-rate threshold of 75.1% - breakdown by region and type of intervention    

Table 4    Use of ESF and YEI funds in implementing national Youth Guarantee schemes, 2016-2019    

Table 5    Employment rates by type of operation, upon completion and after 6 months    

Table 6    Labour market status of individuals who took part in ESF/YEI initiatives    

Table 7    Number of interviewees by Member State and type of organisation    

Box 1    Cooperation and partnerships in Germany and Portugal    

Box 2    Special support for hard-to-reach young people in Belgium    

Box 3    Common standards and tools developed in YEI programmes    

Box 4    ‘From training to employment’ – the Voluntary Labour Corps project in Poland    

Box 5    Common standards and tools developed in the YEI programme    

Box 6    The impact of training on the employment rate in Italy    

Glossary

1.Introduction

This evaluation takes stock of the activities and results achieved by providing EU support for youth employment under the Youth Employment Initiative and the European Social Fund. It builds on the evaluations carried out by the Member States of the Youth Employment Initiative, delivered by December 2018 1 . It also assesses the extent to which the Youth Employment Initiative contributed to meeting the objectives of the Youth Guarantee.

The evaluation assessed the effectiveness, efficiency, relevance, EU added value and the sustainability of youth employment activities funded by Youth Employment Initiative and other actions under the European Social Fund over the period 2014-2018. It covers all Member States, particularly those who funded operations under investment priority 8.ii of the European Social Fund 2 and the dedicated investment priority for programming the Youth Employment Initiative resources. It also looked at whether the initiatives were complementary and coherent with other initiatives in this policy area over the same period (training, vocational education, lifelong learning, skills, mobility of young workers). 

The conclusions of the evaluation are relevant to the ongoing work in the final stages of the Youth Employment Initiative and the 2014-2020 European Social Fund investments in youth employment. The findings can also feed into the process to programme additional resources under the REACT-EU instrument 3  for cohesion policy with the aim of crafting an effective response to the detrimental impacts of the COVID-19 crisis on employment and on society in general. In particular, support for youth employment and protecting vulnerable groups such as young people not in employment, education or training have been identified as some of the key areas for ESF support with additional resources allocated for 2020-2022. Member States will have flexibility in organising the recovery measures for young people, such as to provide incentives for job hiring.

The results of the evaluation should feed into the next programming period, in the form of lessons learned (notably on cost effectiveness, outreach and main target groups). This will underpin the negotiation of operational programmes under the European Social Fund Plus and the design of operations by the Member States targeting youth employment. The findings can provide guidance for EU funding on how to further reform youth employment policies and vocational education and training and apprenticeships systems, co-funded by the European Social Fund+ and by the Resilience and Recovery Facility as part of the reinforced Youth Guarantee and the upcoming policy proposals on vocational education and training. The results will also feed into the Commission’s ex-post evaluation of the European Social Fund.

2.Background description and programme objectives

Following the financial and economic crisis of the late 2000s, the employment conditions for young people in the EU deteriorated significantly. Following the Commission’s recommendations, the Council issued in April 2013 a Recommendation on establishing a Youth Guarantee 4 . The Youth Guarantee is a political commitment made by the Member States to give every person under the age of 25 (or under 30, if the Member State so decides) 5 a good-quality offer of employment, continued education, an apprenticeship or a traineeship within four months of becoming unemployed or leaving formal education.

The Youth Employment Initiative is one of the main EU funding resources to support the Youth Guarantee schemes. However, the aim and scope of the programme are more focused than Youth Guarantee policy. The Initiative provides direct support exclusively to young people who are not in education, employment or training, including the long-term unemployed or those not registered as jobseekers living in regions 6  where youth unemployment was over 25% in 2012. It thus aims to ensure that young people in parts of Europe with the most acute employment challenges can receive targeted support.  

The Youth Employment Initiative was integrated into the European Social Fund (chapter IV of Regulation (EU) No 1304/2013 of the European Parliament and of the Council on the European Social Fund and repealing Council Regulation (EC) No 1081/2006, the ‘ESF Regulation’), in order to provide support for action under the investment priority in Article 3(1)(a)(ii) of the ESF Regulation (the “IP8.ii”):

‘Sustainable integration into the labour market of young people, in particular those not in employment, education or training, including young people at risk of social exclusion and young people from marginalised communities, including through the implementation of the Youth Guarantee.’

Action under the Youth Employment Initiative often takes the form of pathways or packages of measures with the long-term objective of integrating young people into the active labour market. Under the Initiative, each individual participant receives an offer of employment and/or continued education and training, a traineeship or an apprenticeship, and therefore contributes to the implementation of national Youth Guarantee schemes. A single Youth Employment Initiative operation can cover both the offer itself and the range of support services (such as job search assistance, counselling, psychological support) leading to the job/education/training offer. However, the Initiative cannot directly fund reforms of systems and structures carried out under the Youth Guarantee.

The European Social Fund provides additional assistance to youth employment policy, and has a broader scope than the YEI. It includes support for system-related activities and support for regions not eligible for Youth Employment Initiative funding.

Figure 1    Youth employment intervention logic

2.1.Baseline and points of comparison

The situation of youth employment at the start of the programming period for EU28

The economic and financial crisis of the late 2000s had severe effects on young people. Young people trying to make the transition from education or training into work faced a market with fewer jobs and unparalleled levels of competition, not only from their peers but also from adult workers with extensive work experience who had recently been laid off. As a result, they increasingly found themselves unable to get a job without prior experience and yet little chance of getting a job to gain that initial experience.

At the beginning of the programming period, the unemployment rate of young people was about 30% higher than it was for older adults in 2014 7 .

In 2014, on average across the EU, more than one in five young people was unemployed, with wide variations between the Member States.

Figure 2    Youth unemployment rate (15-24) (% of active population), 2014

Source: Eurostat, Labour Force Survey (lfsa_pganws), data extracted on 21 June 2018.

The unemployment rate was particularly high for low-skilled young people, where one in three of the population did not have a job, twice the unemployment rate of high-skilled young people 8 .

In 2014, 12.5% of young people across the EU aged 15-24 were not in employment, education or training, with an equal share of inactive and unemployed, up from 10.9% in 2008. Again, there were huge variations between Member States (Figure 3).

Figure 3    The rate of young people (aged 15-24) not in employment, education or training, by labour market status (% of population), 2014

Source: Eurostat, Labour Force Survey (yth_empl_150), date of extraction 21 June 2019.

In 2014, more than one in ten (11.2%) young people aged 18-24 left school early, meaning that they had completed at most lower secondary education and were not currently enrolled in further education or training (Figure 4). The rate of early leavers from education and training leaving varied widely between countries. At EU level, six out of ten (59.8%) early leavers from education and training were not employed.

Figure 4    Young people in the EU28 (aged 18-24) having left education early, by country and labour status, in 2014 and 2018

Source: Eurostat, Labour Force Survey (edat_lfse_14), data extracted on 1 July 2019.

At EU level, there was only a small gender gap in the rate of young people not in employment, education or training in 2014 (12.3% for men aged 15-24 v 12.7% for women), though some countries recorded more substantial differences. The gender gap is wider in the labour market status of these young people. At EU level, out of all young people not in employment, education or training, 60.2% of young men were unemployed and 39.8% were inactive, compared to 43.3% of young women unemployed and 56.7% inactive 9 .

In 2014, almost a third (31.4%) of young people aged 15-24 lived in households at risk of poverty or social exclusion. Poverty was therefore more prevalent amongst this age group than in the population as a whole (24.4%). In 2014, young people aged 15-24 were more likely (compared to the general population) to be at risk of poverty or social exclusion in all EU countries but Estonia 10 .

It was especially hard for young people with a migrant background (i.e. people born in a country other than the one in which they seek work) to find employment, compared to nationals. In 2014, the difference in unemployment rates at EU28 level between the two groups was 7.6 percentage points on average.

Labour market trends for young people since 2014

Between 2014 and 2018, as the economic situation improved, youth unemployment rates fell in all Member States and by an average of 7 percentage points at EU28 level. However, not all Member States saw the same degree of improvement.

There are also differences in the trend of unemployment rate based on education level. Although the overall improvement in the employment rate was higher for the low-skilled (primary/lower secondary education) than for medium (upper secondary education) - and highly (tertiary education) educated young people, the unemployment rate for low-skilled people (21.9%) remains double the rate of unemployed highly educated young people (11.0%).

Figure 5    Change in unemployment rates for young people aged 15-24 by level of education, 2014-2018 (in percentage points)

Source: Eurostat, Labour Force Survey (yth_empl_100), data extracted on 21 June 2018.

Over that period, the EU28 average rate of young people not in employment, education or training fell from 12.5% in 2014 to 10.5% in 2018. More importantly, it fell in all countries but Germany and Denmark, where it had only marginally increased. The improvement derives entirely from the fall in the number of unemployed young people (down 3.0 percentage points) as the number of inactive young people slightly increased (up 1.2 percentage points). The composition of the target population is thus shifting, with a higher share of inactive young people (58% in 2018) 11 .

By 2018, the rate of early school leaving fell to 10.6% while the proportion of early school leavers that were not in employment fell from six in ten to five in ten (52.8%).

By 2018, the rate of young people at risk of poverty or social exclusion had fallen from 31.5% in 2014 to 28.1%, though it remained higher than the rate for the overall population (21.8%).

For young migrants (15-24 years), the unemployment rate fell from 29.2% in 2014 to 20.9% in 2018.

Note that, given the inevitable time lags associated with the quantitative cross-EU data, all figures in this report predate the COVID-19 crisis.

3.Implementation / state of play

3.1.Programming and budget

Originally, the resources for the Youth Employment Initiative consisted of:

(1)a dedicated budget line (specific allocation) of €3.2 billion; 

(2)a matching contribution from the European Social Fund of €3.2 billion; 

(3)national co-financing for the European Social Fund.

The specific allocation for the Youth Employment Initiative is not complemented with national co-financing 12 . At the start of the programming period, the total EU budget allocated to the Initiative (under both the Youth Employment Initiative (YEI)  and the European Social Fund (ESF)) was €6.4 billion (€7.67 billion including national co-financing of the European Social Fund).

Given the persistently high level of youth unemployment, in June 2017, the European Parliament and the Council agreed to increase funding for the Initiative by another €1.2 billion, matched by an equivalent amount of ESF funding (€2.4 billion in total). The increase in ESF funding was also topped up by eligible funding from Member State own resources. All in all (combining EU funding on the dedicated YEI budget line, the matching funding under ESF and national co-financing), the total budget for YEI was €10.4 billion, as shown in Table 1 below. The total ESF budget allocated to Investment Priority 8.ii (non-YEI) is €8.2 billion, bringing the total investment in youth employment operations to around €18.6 billion.

Table 1    Funding for Youth Employment ESF + YEI, including amendments until 2018

Source: SFC2014, based on data on the operational programmes reported by the Member States in AIR2018 (data extracted on 6 September 2019)

Youth employment operations under both funds are programmed and implemented under the shared management system with the Member States. Member States identify relevant operations to support the objectives set in the operational programmes, in agreement with the European Commission. Therefore the range of measures supported by the European Social Fund and the Youth Employment Initiative varies significantly across Member States (also in view of the funding available to each Member State).

There are 37 operational programmes running in 20 Member States including YEI (out of a total of 187 programmes under the European Social Fund). Only three Member States include YEI in more than one programme (Belgium and the UK with two each, and France with several regional programmes implementing YEI and a dedicated national operational programme for YEI).

All Member States, including those not eligible for the YEI, can programme ESF resources to support youth employment operations. There are 58 operational programmes running in 20 Member States that include ESF investments under IP8.ii (access to youth employment) and that not covered by the YEI.

Altogether only five countries (Austria, Denmark, Estonia, Finland and the Netherlands) did not programme ESF and YEI investment under the youth employment investment priority. This echoes the fact that these Member States did not receive a country-specific recommendation on youth employment under the European Semester process over the years 15 , which means that these Member States did not present major policy challenges in this policy area. However, even though these Member States did not allocate specific budget to the youth employment investment priority, several operations are running in Austria, Denmark and Finland that are mainly focused on young people.

During the negotiations of the 2014-2020 cycle of operational programmes, the Commission called for all action on youth employment to be programmed under the dedicated investment priority on youth employment (IP 8.ii), to ensure visibility and to concentrate action. Nevertheless, a map of all operations reported by the managing authorities under Thematic Objective 8 related to employment-related actions shows that ESF support for employment programmed under other investment priorities may also be accounted as youth employment. The costs of these operations are estimated at €2.1 billion, which can be extrapolated to an estimated €3.5 billion (assuming similar levels of implementation as the Youth Employment operations under IP8.ii: 66%).

Based on these additional calculations, the total estimated funding for Youth Employment schemes under the European Social Fund and the Youth Employment Initiative is €22 billion, representing 18% of the total budget for the European Social Fund (including national co-financing).

3.2.Type of operations funded

A total of 596 different ESF and YEI operations 16 across all 28 Member States were identified under IP 8ii.

For ESF-funded operations, the budget was allocated as follows:

·24% (the largest share) to operations that support work-based learning,

·22% to guidance and support measures,

·13% to measures that support education and training,

·10% to financial incentives, 

·13% to operations with multiple categories of operations that run under a single operation, with no dominant type. For example, a single operation could encompass a pathway of training and education combined with guidance and counselling for individuals.

Given the specific scope and objectives of the Youth Employment Initiative, the share of different types of operations funded is different. This is confirmed by the high share of the budget (20%) allocated to multiple types of operations in a single approach. In particular, the higher share of financial incentives (26% compared to 10% for ESF-funded operations), such as wage subsidies, reflects the slightly different set-up of the Initiative. However, the provision of guidance and support accounts for a considerably smaller share in YEI operations (11% against 22% under the ESF). The share of funding allocated to work-based learning and education and training are relatively similar for both YEI and ESF-funded operations.

Figure 6    Share of costs eligible for ESF funding by type of operation

Source: Mapping of operations eligible costs declared in AIR 2015 – 2018

3.3.Financial and operational implementation

By the end of 2018, there had been about 3.8 million participations 17 under ESF/YEI operations. Some 1.4 million participations operations resulted in employment immediately after participation, with further results including employment-focused education and training, and self-employment.

To direct resources to youth employment measures at the start of the programming period, YEI investments were frontloaded in the first two years of the programming period, while ESF investments could be allocated over the full programming cycle 18 . Some Member States planned to continue YEI schemes after 2018 from the start, with ESF funding. This initial frontloading of YEI funding explains the difference in implementation rates between YEI and ESF, with YEI financial execution overall more advanced in the Member States that receive YEI funding, in particular in Member States that receive a substantial share. The start of the actual implementation was delayed due to the time needed to set up control and monitoring systems that satisfy the Regulations and national legislation. The socio-economic context, absorption and administrative capacities of Managing Authorities and beneficiaries appear to be decisive factors for the progress in implementation which varies a lot between the different Member States.”

  Figure 7    Implementation rates of YEI/ESF funding by Member State (expenditure declared to the managing authority vs funding allocated)

Source: SFC2014, based on AIR 2018, data extracted on 6 September 2019 MD=more developed region, LD=less developed region, TR=transition region. No regional data available for YEI funding.

4.Method

4.1.Short description of the methodology

The evaluation questions, derived from the Better Regulation evaluation criteria, were broken down into a number of sub-questions.

The evaluation was based on three main sources of data:

·a study 19 carried out by an external contractor;

·the twelve-week public consultation, which was carried out by and analysed with the assistance of the external contractor;

·an estimate of macroeconomic indirect effects of youth employment operations using the Dynamic Computable General Equilibrium model ‘Rhomolo’ developed by the European Commission’s Joint Research Centre. 

There were 1,376 responses to the public consultation. Over half (57.6%) were responses by citizens up to 33 years of age, almost a quarter of who had received funding under ESF or YEI operations. Some 21.8% of respondents were from organisations involved in managing and implementing of YEI and/or ESF-funded operations. Where data on the gender of respondents was provided, 58.7% of respondents were female.

The study combined all information on the other two instruments in a single report, which provided the main materials and analysis for this evaluation. The study applied a mix of evaluation methods and tools, which are explained in further detail in Annex 3 to this Staff Working Document. In summary, the study involved:

·mapping and carrying out an in-depth analysis of youth-related thematic objectives, investment priorities, target populations and type of operations, based on the SFC2014 database, operational programmes, AIRs and related documentation, including the Commission’s country-specific recommendations. The study used data up to 31 December 2018, as reported by Member States by September 2019. For the mapping, the regions were categorised into four clusters in terms of their socioeconomic context, and used in the analysis and conclusions of this evaluation:

oCluster A – regions with a strong initial socioeconomic context that made substantial progress over the period; 

oCluster B – regions with a strong initial socioeconomic context that made limited progress over the period;

oCluster C – regions with a weak initial socioeconomic context that made visible progress;

oCluster D – regions with a very weak initial socioeconomic context that made limited progress. 

·reviewing available literature (policy documents, regulations, national and EU-wide evaluations, ESF websites and publications);

·a synthesis of the 2018 compulsory national YEI evaluation reports and screening of other ESF evaluations of youth employment programmes using the database of evaluations compiled by the Evaluation Helpdesk;

·the results of the public consultation;

·a comparative analysis of labour market trends and the role of EU-funded operations;

·a cost-effectiveness analysis; 

·econometric analysis and provision of data to the Joint Research Centre (JRC) for the Rhomolo modelling simulations;

·case studies in 10 Member States 20 , covering 20 operational programmes and a mix of YEI and ESF youth employment programmes;

·62 interviews with desk officers, managing authorities and other stakeholders, in particular for the case studies;

·five focus groups in the Member States and one at EU level;

·a set of fact sheets for all countries with YEI and/or ESF-funded youth employment programmes; 

·a compilation of lessons learned and good practices.

4.2.Limitations and robustness of findings

The findings and conclusions are reasonably robust, since they are based on a series of measurements from the multiple sources and tools used in the evaluation. However, the evaluation also noted several limitations.

Data quality and timeliness. There is a significant time lag between reporting the outputs, results and financial implementation, as a result of work on data checking and reporting. The costs are declared only after checks by relevant authorities. Often, outputs are reported once operations are completed and declared, and the results reported later still. Lastly, since the data is based on participations in operations and not on individual participants, the calculation of certain ratios, such as success rate, is challenging.

Detailed programme information. Detailed information on individual operations is usually not available, despite the obligation to record and store minimum data. Moreover, across the EU, the rules and standards for reporting operations are uneven. This means it is difficult to make a systematic comparison. To address this challenge, the evaluators focused on the cost of operations and made a number of assumptions such as estimated values by type of operation where no financial allocation was available, removing implausible values or the assumption that operations not covered by Article IP 8.ii reached similar implementation levels as those covered by Article IP 8.ii.

Clearly defining youth employment operations. Regarding ESF support, a number of Member States decided to run youth employment schemes also across other investment priorities. As a result, the EU-level mapping exercise required clear definitions based on robust monitoring data, and estimates of the costs of operations not covered by the investment priority on youth employment (8.ii). The report makes it clear throughout what the underlying data is based on and where reliable comparisons can be drawn. For YEI support, the rules require that all YEI funding (including the matching ESF support) be programmed under IP 8.ii.

Public consultation. The public consultation was in the form of an online survey 21  run to gauge the views of a wide range of stakeholders, on a voluntary basis. This means that the information is not representative. Almost three quarters of the responses came from individuals and organisations in four Member States: Italy, Bulgaria, Spain and Slovakia, despite extensive efforts to publicise the consultation. This was highlighted in the report. Nonetheless, the public consultation provided a substantial amount of useful information and insights into the experiences and perceptions of youth employment operations.

Comparative analysis of labour market trends. At the time of writing the report, data in the labour market policy database 22  were only available up to 2017; excluding 2018.

Regional clusters. YEI monitoring data are not broken down by type of region, so data on expenditure and participants may cover more than one type of region.

Cost effectiveness. Due to issues of data timeliness and completeness, the lack of full information on the outputs and results made it difficult to draw overall conclusions on cost effectiveness. Thus the cost per result are not yet meaningful and the cost per participation (number of individual participants registered against the total eligible expenditure declared) are but a snapshot at the cut-off date of 31 December 2018. The approach taken was to combine partial findings from a number of sources (counterfactual impact evaluations available, micro-data from case studies in the Member States, econometric analyses, Rhomolo modelling and the public consultations) in order to draw valid, if only partial, conclusions.

Econometric analysis. The explanatory power of the regressions – especially for unit costs – is low, due to the high degree of variability in the observations. This is the result of the complexity of youth employment schemes, mixed target groups and the fact that the data is still in the process of being consolidated (as mentioned above).

Rhomolo analysis. General equilibrium models need to be based on a number of simplifying assumptions. Given the overall scale of EU youth employment operations in comparison with GDP figures and national programmes, the macro impacts identified by the model should be viewed with due perspective. Nonetheless, the work has been useful and informative.

COVID-19 pandemic. Note that the field work for this staff working document was carried out before COVID-19 pandemic came to Europe. The Commission proposed the Recovery Plan for Europe on 27 May 2020, as part of the current and future multiannual financial frameworks. One of the aims of the Recovery Plan is to invest significant resources in measures to support youth employment and related reforms for the remainder of the current implementation period and over the next programming period, with a view to mitigating the consequences of this pandemic on young people and turning around the situation for youth employment for the better.

5.Analysis and answers to the evaluation questions

5.1.Effectiveness: How effective is the YEI, and other ESF-funded youth employment operations, in achieving their objectives?

The section on effectiveness describes the extent to which YEI and ESF-funded operations have progressed towards the set objectives, broken down into operational (outputs), specific (results) and general (longer-term results/impacts) objectives. It covers both quantitative (success rates) and qualitative aspects (e.g. the quality of employment offers). This section also analyses the factors that influenced performance (socioeconomic context, implementation arrangements, type of action and outreach strategies).

Financial implementation and expected outputs are progressing, albeit slower than anticipated

For the Youth Employment Initiative, progress on implementation has been below the targets set in the operational programmes, both in terms of financial implementation and number of participations. By the end of 2018, only France, Hungary, Lithuania, Latvia, Poland and Slovenia had met the 2018 milestone targets set at the start of the programming period. Despite action taken by the European Commission to facilitate the implementation of the Initiative in the early stages 23 , Member States experienced delays in implementing the Initiative in 2014 and even in 2015, as they decided to first ensure compliance with ex ante conditionalities 24  including set up of monitoring systems that complied with the Regulations 25  and national legislation, before initiating the work.

In addition, a common challenge across Youth Employment Initiative programmes was reaching young people not in employment, education or training. Since many of these young people were often not registered as unemployed, identifying the target group often required active canvassing and encouraging individuals to register. Member States and implementing organisations (beneficiaries of YEI support) often reported gathering the evidence on NEET status as possibly the most challenging legal requirement in relation to implementing the Initiative.

Over time, most Member States implementing the Initiative broadened the eligibility of investments to include 25-29 year olds. This required refocusing the support to help young people overcome barriers to work and to fund outreach, rather than on developing their qualifications and skills.

Faced with these considerable challenges to meet the initial targets for 2018, the 2017 revision of the multiannual financial framework provided resources for 2017-2020, which led to an extension of YEI funding until the end of the programming period. This extension was prompted by the need to ensure that operations already started and considered crucial by the Member States could continue. As a consequence, the operational programmes were amended and the implementation targets were revised 26 . The EU average financial implementation rate reached 52% by the end of 2018.

Figure 8    Progress towards the milestones set for 2018 – YEI

Source: SFC2014, based on AIR 2018, data extracted on 6 September 2019 (Cluster A – Strong start/substantial progress, Cluster B – Strong start/limited progress, Cluster C – Weak start/visible progress, Cluster D –Weak start/limited progress)

According to the econometric analysis, the country specific context affects how the Initiative is implemented. In addition, the countries in most need of support often face the greatest implementation challenges. For example, countries with an unfavourable context at the start but where conditions are improving saw a positive correlation with progress in the project selection rate. By contrast, in general there was no correlation between the change in the youth unemployment rate and financial progress.

The impact of socioeconomic factors on achieving the output targets, such as difficulty to reach target audience, lack of jobs or low education of participants, is more shaded.

Other factors that hindered implementation were related to the administrative capacity of the managing authorities and the absorption capacity of beneficiaries.

EU-funded youth employment operations have made a positive contribution to the integration of young people in the labour market

Over 1.5 million completed participations in YEI were recorded in the monitoring system for unemployed young people under 25 years of age. This is slightly over half the total participations (2.7 million total participations). However, it is important to note that many participations continued after 2018 so they were not recorded in the study as completed.

The 1.5 million represents on average 65% of the final target set for this population. Out of this total, 0.7 million unemployed people received an offer of training or employment immediately after completing the operation (with an average target achievement of 58%). Moreover, 0.9 million people unemployed at the start left the operation either in education or training, having gained a qualification or having found employment (60% of the target). This is encouraging progress, considering that YEI implementation is currently in its most active phase and will continue at least until the end of 2022, under cohesion policy rules. Therefore, it is likely that the overall YEI targets will be achieved.

There were positive results for the long-term unemployed as result of participating in YEI 27 :

The results for economically inactive 28 participants have steadily improved, but the target for this group is relatively low (compared with implementation rates, for example, but also with the results for the long-term unemployed), as people are further from the labour market and less responsive to active labour market schemes than people who are unemployed. They are also often more difficult to reach out to and may require more comprehensive support, including motivation, counselling and social inclusion measures.

Most participants who responded consider that their participation in EU youth employment operations has been beneficial

The opinions expressed by participants during the public consultation were consistently positive regarding the effects of the EU support they received in helping them integrate into the labour market.

The most frequent types of support the participants received were support to develop general skills (33.6% of respondents), help in looking for a job (33.4%), and gaining professional skills and qualifications (32.3%). Only 10.9% of respondents said that the support received did not help.

The benefits of youth employment support go beyond access to employment

The views expressed by the organisations responding to the public consultation echoed those expressed by individual participants. According to the organisations, the benefits of ESF/YEI funding are mostly linked to the development of skills and qualifications and to help young people find a job, including young people not in employment, education or training and other disadvantaged individuals.

Respondents from the organisations involved in running ESF/YEI also expressed broad agreement on the following positive results of the Initiative:

·promotion of new partnerships and exchanges; 

·inclusion of marginalised people; 

·development of soft skills and group work; 

·youth empowerment; 

·increased connection with labour market requirements; 

·increased motivation and

·integration into society of young people not in employment, education or training.

Overall, respondents from organisations are also more sceptical than individual respondents concerning the ESF/YEI’s effectiveness in helping young people improve their situation on the labour market in a sustainable manner.

Identifying and reaching out to NEETs was often a challenge

Despite the high levels of young people not in employment, education or training in a number of countries, actually identifying them to provide support proved difficult, in particular in Member States that had not yet set up policy channels and mechanisms to do so.

Several Member States developed strategies to overcome this challenge. For example, they used social networks, media, newsletters or more innovative channels. In Bulgaria, concerts were organised and adverts were placed on fast food trays to reach young people. They also used street work to meet young people not in employment, education or training in public places where young people gather, e.g. in parks or shopping centres.

Member States also invested in developing the experience and skills required to reach the ‘hardest to reach’ young people, working with professional youth workers 29 who represent local organisations, promoting cooperation between the various bodies involved, especially ‘vertically’ (e.g. from local level to the public employment service), going beyond traditional outreach channels. One example is in Germany, where ESF funding successfully supported tailored activities (support chains and dedicated preventative systems of assisted transition from school to work through systematic multi-party cooperation).

Resources were channelled to the population most in need

The figures overleaf show that ESF/YEI funding reached a much higher proportion of young people not in employment, education or training in regions where the labour market situation of young people was worse at the start 30 of the programming period than in regions with better initial favourable conditions 31 , particularly when the situation continued to be unfavourable, e.g. when there has been limited progress 32 . The higher coverage in regions where the labour market situation of young people was worse confirms that resources were channelled to the areas (and people) most in need.

Figure 9    Share of NEETs (aged 15-24) receiving ESF/YEI funding as a proportion of the NEET population in 2014 by cluster of regions (%)

Source: SFC2014, based on AIR 2018, data extracted on 6 September 2019 and Eurostat, Labour Force Survey ( edat_lfse_22 ), data extracted on 21 June 2019.

These findings concur with the Rhomolo modelling analysis, which found that EU youth employment funding went mostly to the regions most in need. The Rhomolo analysis also found that, in terms of indirect employment, the results are stronger for low-skilled participants, which was the main target group. At regional level too, in regions located in Southern European Member States where youth unemployment is high, EU funding has been the most beneficial for participants, considering also the high relative size of the support provided in those regions.

Overall participation was well balanced from a gender perspective

Across the EU, participation is well balanced from a gender perspective, with a broadly equal share of men and women (51% against 49%) reached by YEI operations, although participation in ESF-funded operations is higher for men (54% against 46%).

There are considerable differences among the Member States. YEI operations in Croatia and Greece reached more young women (66% for the YEI and 63% for youth employment operations under the ESF), while youth unemployment initiatives in Belgium reached more young men. The participation rate of young men in ESF-funded operations in Belgium, Germany, France and Sweden was over 60%. Cyprus, Croatia, Poland and Slovenia, by contrast, had above-average female participation rates. In some countries (Belgium and Spain), the national evaluations concluded that women benefited less from the operations under the ESF/YEI than men.

Reaching out to specific groups of disadvantaged young people proved particularly challenging, requiring multiple methods

According to the 2018 YEI evaluations, the most vulnerable young people are often under-represented, and potentially under-reported. This is in line with the replies submitted to the public consultation, which noted difficulties in reaching out to and supporting vulnerable young people in rural areas, those with disabilities, ex-offenders, those at risk of poverty and the homeless, and young people with multiple disadvantages.

Figure 10    How successful were the youth employment operations funded by the European Social Fund and the Youth Employment Initiative in providing support to the following target groups?

Source: Final Report, Public Consultation on the Evaluation of the EU Support to Youth Employment by the Youth Employment Initiative and the European Social Fund. Study supporting the 2019 evaluation of youth employment (VC/2018/0175)

On the basis of the 2018 annual implementation reports, and bearing in mind that the categories below are not mutually exclusive, note that at least at EU(28) level:

·11.3% of YEI and 13.2% of the ESF participations reported were young migrants, participants with a foreign background or minorities (including marginalised communities such as Roma); 

·4.5% of YEI and 5.2% of ESF participations were young people with disabilities; 

·13.7% of YEI and 14.2% of ESF participations were people with other disadvantages;

·0.8% of YEI and 0.3% of ESF participations were homeless people or people affected by housing exclusion;

·19.3% of YEI and 18.7% of ESF participations were young people in rural areas.

Under-reporting is quite likely for these indicators, since information on disadvantage is not always systematically collected by organisations or volunteered by participants.

It is also important to note that not all EU countries have the same definition of the disadvantaged, or the same definition of the NEET category. In some regions particularly hit by the crisis, a substantial share of the young population could be considered disadvantaged at the beginning of the operation, while in other parts of Europe the definition covered only certain groups. Because of this, there are notable differences between the Member States. For example, Sweden recorded a significant share of migrants (38.7% of participations in YEI and 38.6% in ESF operations against an average of 11.3% of YEI and 13.2% of ESF) and a significant share of people with disabilities (69% of ESF compared to an average of 5.2%).

The case studies illustrate the different methods used to reach out to disadvantaged groups.

In Portugal, for example, the national operating programme did not define sub-groups of young people not in employment, education or training, but referred only to young unemployed and young inactive people. The indicators did not differentiate participants further. Nevertheless, some operations included an increase in the budget for vulnerable groups such as the long-term unemployed, ex-prisoners, people with disabilities, etc. The regional operating programme run in the Azores in Portugal reached all target groups of disadvantaged young people in all nine islands, except for young people with disabilities.

Evidence from the case studies shows that some of the differences in target groups reached between the regions could be due to the type of organisations providing support rather than to the type of region.

In Poland, for instance, there were differences in the targeting methods used by the local labour offices and the Voluntary Labour Corps. The local labour offices operated on a first come, first served basis with people registered with the public employment service. The Voluntary Labour Corps targeted NEETs including those not registered with the public employment service. Public employment services use additional eligibility criteria, including a lack of professional qualifications or qualifications not adapted to the needs of the labour market, disabilities, gender, particularly difficult social situations (e.g. people from jobless households, families using social assistance services, single parents and people from rural areas).

Participants were mostly satisfied with the offers received but had contrasting views on the quality of the offer

One of the concerns of participants receiving YEI/ESF funding from the start was the quality and timeliness of the employment or training offers received.

In practice, the criteria most used to assess the quality of job offers were whether the offer was for a permanent or full-time job, and the wage level.

For most operational programmes for which data on the quality of job offers is available, over a third of the offers provided with EU support were for ‘permanent’ employment contracts (in the sense that they were for an indefinite period of time).

Figure 11    Share of job offers with permanent contracts according to YEI evaluations

Source: 2018 YEI Evaluation reports

Data for FR relate to three national measures (‘Accompagnement APEC’, FR-GJ: ‘Garantie Jeunes’, FR-PA: ‘Parcours Autonomie’).

Data for CY relate to ‘Acquisition of work experience for young unemployed graduates’.

Data for ES refer to the situation 12 months after exit, and 6 months for UK-England.

The results of the public consultation indicate that ESF/YEI participants who took part in the consultation and who started the operation unemployed were more likely to have transitioned to full-time employment (39.7%) than to part-time. However, among the long-term unemployed, only 19.5% found full-time employment after support, 7.3% found part-time work and half remained unemployed. When looking at responses from all types of participants, more respondents reported that they found a temporary job (20.9% of all ESF/YEI participants) than a permanent one (16.8%).

Several YEI evaluation reports highlighted that for most participants, the wage level offered was an issue, as it was often at minimum wage level. This was a particular concern in (though not only in) Bulgaria, Croatia, Cyprus and Lithuania.

In addition to the statistical data, the YEI evaluations carried out by the Member States revealed a high level of satisfaction with the support received by participants. In Croatia and Slovenia, for instance, the satisfaction rates are close to 70%, and in Italy, the participants surveyed are on average more satisfied with the support received through the YEI than participants in other employment support measures. There is, however, a general consensus among stakeholders in Italy that the quality of the offers should and could be further improved and that it should provide a form of certification of the skills acquired.

The effectiveness of operations was influenced by the type of operations and, to a certain extent, socioeconomic factors and organisational arrangements

The results of the public consultation indicate that paid apprenticeships are considered to be the most effective type of intervention by organisations involved in ESF/YEI, together with vocational education and training and basic skills training, which are considered useful for most target groups, including the most disadvantaged. This is generally supported by the evaluations carried out by Member States and the case studies providing specific examples.

Community or voluntary work, as well as non-paid apprenticeships/traineeships, are considered the least effective types of operations in terms of access to employment, but not necessarily in terms of employability.

Figure 12    To what extent do you agree/disagree that the following measures have been actually effective in helping young people enter quality and sustainable employment? (population: 399 organisations)

Source: Final Report, Public Consultation on the Evaluation of the EU Support to Youth Employment by the Youth Employment Initiative and the European Social Fund. Study supporting the 2019 evaluation of youth employment (VC/2018/0175)

Socioeconomic and geographic factors clearly influenced the effectiveness of operations

Although the econometric analysis indicated that the socioeconomic context does not seem to play a major role in achieving the targets for output indicators for the YEI, regions in less favourable economic situations recorded comparatively lower achievement rates for ESF operations. ESF operations in regions in a weak socioeconomic situation at the start and that made limited progress (Cluster D) 33 had more difficulties in reaching the target number of individuals, whereas ESF investments in regions where the socioeconomic situation was improved made steadier progress towards their targets. The socioeconomic situation also had a greater bearing on the achievement of targets for result indicators than for outputs, independently of delayed reporting.

A high share (47.4%) of organisations responding to the public consultation reported structural problems (such as the lack of jobs and the low education level of participants) as the factors hindering the achievement of ESF and YEI objectives, in addition to a lack of flexibility to adapt operations to changing socioeconomic contexts. This is understandable, given the difficult economic environment in which YEI and ESF youth employment operations were implemented in the aftermath of the late 2000s economic and financial crisis, in particular in southern Europe.

The main geographical factor to emerge as having an influence on effectiveness of operations was the difference between urban and rural areas. This is based on information from the public consultation and the case studies, which provide some examples. For instance, in Spain actions taking place in large cities were considered more effective than those in rural areas, in particular as those run in rural areas typically tend to be more expensive due to the larger distances. They also suffer from having less infrastructure available and a smaller overall local job market.

The administrative capacity, experience and organisation of managing authorities and beneficiaries also had an impact on effectiveness

Structural challenges relate not only to the socioeconomic context, but also to the capacity of managing authorities and beneficiaries to deal with complex programming requirements, including eligibility criteria for young people not in employment, education or training under the YEI, and engaging with stakeholders. This last point was particularly relevant in regional programmes where the successful implementation of projects depends on cooperation between multiple governance levels and bodies.

The 2018 national YEI evaluations revealed that prior experience and continuity are factors contributing to effective management. In the case studies, experience in managing similar measures was reported as helpful in implementing YEI. In Ireland, for instance, the long history of some operations (such as BTWEA, which has existed since 1993 or ‘Community Training Centres’) is considered to contribute to the success of YEI.

Good governance was widely seen as essential to the success of the YEI evaluations. This is illustrated in Sweden, where the high quality implementation of the operations (the competence of staff, and low staff turnover) was regarded as a major success factor.

Partnership and cooperation were also mentioned in several YEI evaluations and in the public consultation as crucial aspects to support implementation. For instance, in Brussels-Capital Region, cooperation between Actiris and other partners improved access to support for young people with a low level of qualifications or those living in jobless households.

The managing authorities participating in the EU-level focus group also confirmed that cooperation among the several bodies involved, in particular among employers and employment services, is key to the efficient implementation of youth employment operations. This can only work well if the training offer is in line with companies’ need for skills or when there is a financial incentive for companies.

The ability to reach the target population seems to have been the single biggest factor in effectiveness

All in all, there seems to be a consensus among stakeholders that the factor with the single greatest impact on effectiveness was the ability to reach the target groups. Approximately six out of ten organisations responded to the public consultation that the main hindrance was the difficulty in reaching target groups, followed by structural problems such as the lack of jobs and the low level of education of participants.

This was also confirmed in the EU-level focus group where the managing authorities agreed that reaching those furthest from the labour market was key to the success of the programmes. They identified three main essential aspects: local outreach work, local staff with the right qualifications and skills, and the use of innovative channels to reach young people.

The composition of the target group also had an impact on the success rate of operations

The composition of the target group also has an impact on effectiveness. The more ambitious the interventions were in terms of outreach to target groups requiring intensive and comprehensive support (for example people with disabilities or other disadvantaged people) the more difficult it was to achieve the financial targets and employment targets. This was also confirmed by the econometric analysis. For some disadvantaged participants, access to employment is not an immediate objective or even an option; instead the key outcomes were confidence building, gaining qualifications and some experience (not just employment).

Although not an explicit objective of YEI, youth employment operations led to structural changes in systems

According to over half of the respondents from organisations (59%) to the public consultation, the employment operations run under the European Social Fund or the Youth Employment Initiative successfully contributed to the promotion of structural reforms in employment, education and training systems. A number of examples identified in the case studies confirmed this finding.

The main impact of YEI and ESF funding for young people across operational programmes was in changes to the way public employment services and other bodies involved in employment policy work approach youth employment issues.

Box 1    Cooperation and partnerships in Germany and Portugal

Macroeconomic effects are small, but positive, especially in the medium to long term

The results from the modelling work carried out by the JRC’s Rhomolo analysis conclude that the ESF/YEI investments in human capital and education for young people produce macroeconomic returns on top of the direct positive results for participants, though they need some time to materialise. In the long term, youth employment operations are expected to provide a boost to overall EU GDP and employment levels, although at a modest rate.

At EU level, the modelling found that, in addition to the jobs covered in direct relation to the funded operations, 11,000 jobs are expected to have been generated by 2023 thanks to the ESF and YEI investments in youth employment, with long-lasting effects generated by the structural policies and the change in productivity 34 . The increase shows persistence over the long term (by 2030), indicating that GDP may be 0.06% higher than the baseline and over 35,000 jobs are expected to have been created.

These positive but comparatively small effects should be read in the light of four key facts:

·the overall investment is modest compared to GDP (the whole Youth Employment package accounts for just 0.05% of the EU’s GDP) and uneven across regions;

·in addition to the overall macroeconomic effects, the operations provide direct benefits to those receiving support;

·these are just partial estimates given the programmes are still ongoing and reporting inevitably lags actual performance;

·EU cohesion policy supports investment in physical capital along with human capital, under the European Regional Development Fund. Such investments may have synergies with ESF/YEI investments in human capital.

The key point here is that productivity-enhancing human capital investments ensure that jobs are created over the medium to long term. Although the effects may seem modest at EU level, they are greater in some regions, with a number of regions located in southern European states standing to reap most of the benefits, which can be quite substantial given the scale of EU support.

5.2.Efficiency: How efficient has the YEI, and other youth employment operations funded by the ESF, been in the achievement of their objectives?

Three aspects of efficiency of EU youth employment operations are analysed below:

·First, the cost effectiveness of operations in delivering outputs and results, and the factors involved;

·Secondly, the impact of administrative arrangements on the effectiveness of support;

·Lastly, how visibility of the funding and the operations contributed to outreach and hence to successful delivery.

Stakeholders viewed vocational education and training, as well as paid apprenticeships, traineeships and internships, as the most cost-effective type of operations

The public consultation elicited broadly similar responses for the ESF and the YEI. Most respondents from organisations involved in implementing the ESF/YEI agreed on the cost effectiveness of vocational education and training activities (87.6% ESF, 83% YEI) followed by apprenticeships, traineeships and internships (83.3% ESF, 82.5% YEI), basic skills training (80.9% ESF, 79.1% YEI) and guidance and career support (80.4% ESF, 76.7%). In both cases, and in line with the public consultation responses on effectiveness, voluntary and community work were ranked the least cost-effective.

Unit costs per output are comparable with the YEI and other ESF youth employment operations, with significant variations between Member States

Mindful of the above limitations in terms of data availability, this evaluation resulted in an average cost per participation in ESF 8.ii (excluding YEI) investments of €1,854, which is higher than the average unit cost for Thematic Objective 8 (Promoting sustainable and quality employment and supporting local mobility) overall (at €1,390). These figures are also higher than the unit costs for youth operations given in the update of the ex-post evaluation of ESF 2007-2013 for Access to Employment (€1,215).

Investments in YEI have similar average unit costs at €2,035 (EU average). The econometric analysis confirmed that there was no statistically significant difference between the unit costs for YEI and ESF youth employment operations outside YEI.

A Commission study 35  also concluded that ‘considering the rigorous data verification process that the data in the sample has undergone, […] unit costs for the unemployed established at EU level (Simplified Cost Options that Member States can use to reduce the administrative burden of providing evidence for eligibility of expenditure) reflect the specific training practices in those Member States’.

The calculations of both costs per participation and simplified cost operations vary significantly between the Member States. There are some reasons for the substantially higher than average values for Germany, Ireland and Sweden. For Germany, the likely explanation is that typical operations in Germany last longer (12 months and over). In Ireland, operations target specific groups, such as the socially excluded, long-term unemployed, and youth, which increases the cost of the intervention. For Sweden, factors are both the type and the duration of operations.

Note that the experience in running YEI and ESF youth employment support operations has become a solid basis for the Commission and Member States to work on developing simplified cost operations for the integration of young people not in employment or education 36 . Developing EU-level simplified cost operations for this target group and such measures could be particularly relevant for transnational cooperation activities involving several Member States, as evidenced in the current period by the ESF Transnational cooperation network on youth mobility.

Table 2    Comparative table of unit costs

Sources: SFC2014, based on AIR 2018, data extracted on 6 September 2019. The figures for BG are subject to the same caveats as in the previous table, YEI 2018 evaluations. LMP database

Data compiled by PPMI based on data extracted from the LMP database and provided by Member States for training of all unemployed

The intensity of support is likely to be a major factor in determining the unit cost and the cost effectiveness of operations

The focus on outreach to vulnerable and hard-to-reach groups means that effective operations require a significant investment in identifying, recruiting and supporting – often with a high level of intensity – those from vulnerable and hard-to-reach groups, which implies additional costs. This may further increase if inactive NEETs account for an increasing share of NEETs supported.

This can also be related to the fact that training activities are also considered the most costly type of operations ( Table 3 ). In this context, it should be underscored that high costs are not in themselves indicators of low efficiency. Cost effectiveness is also positive when costly operations are linked to high effectiveness for specific target groups.

Table 3    Expenditure per entry: expenditure minimum-rate threshold of 75.1% - breakdown by region and type of intervention

Source: Micro-data and own calculations, based on operations completed to 75% or more

The influence of the socioeconomic context on the cost per participant is not significant for outputs, but it does have some impact on results

According to the econometric analysis carried out, the socioeconomic context tends does not make a statistically significant difference to the average cost per participant by country. Additional findings from the econometric analysis are that:

·costs tend to be higher in programmes run in the cluster of countries with a strong starting position and that made good progress (Luxembourg and Ireland, for example). Costs are comparatively lower in areas with greater need of support;

·changes in the unemployment rates are not correlated with cost per participant.

The costs of measures are not directly related to the characteristics of the target population

According to the econometric analysis, overall, the composition of the target population (different shares of inactive, low-skilled, disadvantaged participants, and shares of participants from minorities and with disabilities) is not strongly correlated with unit costs. The exception is for programmes that target a high share of inactive participants, which show statistically significantly lower unit costs. This finding would appear to contradict 37 other sources that highlight more intense levels of provision for those young people from more disadvantaged backgrounds. For instance, most case studies highlight higher unit costs for hard-to-reach/disadvantaged groups (especially the long-term unemployed) for comparable forms of support.

Administrative requirements have facilitated implementation, with some exceptions

Overall, respondents had a positive perception of the administrative requirements and organisational arrangements to implement youth operations. Most respondents to the public consultation (300 responses from organisations) found the administrative arrangements for the implementation of projects to be appropriate (see Figure 13 ).

The types of administrative requirements more often rated as appropriate are those related to project implementation (65.3%), project selection (64%), communication (62.7%), evaluation (61%), reporting and monitoring (55.7%), setting up standard cost options (51.3%) and audits (50.3%). A lower score was given to the arrangements for setting up the management and control systems, with 46.3% rating it as appropriate.

Figure 13    How would you qualify the following administrative arrangements for the implementation of youth employment operations under the ESF and YEI?

Source: Draft Final Report, Public Consultation on the Evaluation of the EU Support to Youth Employment by the Youth Employment Initiative and the European Social Fund

However, the case studies identified a number of areas for improvement, notably as regards the requirements to provide evidence to support the eligibility of costs regarding the support provided to young people not in employment, education or training and the problems related to the initial set-up and functioning of the IT systems for the operational programmes. This was found to be a general weakness in all cohesion policy programmes.

Respondents to the public consultation were asked in an open question to provide any examples of gold plating or excessive administrative burden that they had experienced. However, the examples given do not seem to be deliberate on the part of public administrations or ESF/YEI operations, but more due to inefficiencies resulting from either work to implement EU regulations, or from incompatibility in national systems. Essentially, both the public consultation and the case studies noted improvements since the previous programming period and support for the EU’s simplification agenda, which will take time to deliver all of the anticipated benefits.

Reporting and monitoring requirements are complied with, but delays are common

The respondents had a generally positive perception on this aspect, after the initial transition period. Beneficiaries comply with the Commission’s reporting and monitoring requirements and systematically use the system for fund management for reporting purposes, in line with the guidance provided and the regulation.

Compared to the previous period, the current ESF Regulation 2014-2020 sets out common output and result indicators (including those setting a common approach to monitoring/reporting on the sustainability of results through systematic measurement of longer-term results for participants) and the Commission has issued detailed definitions, guidance and support for setting up monitoring systems. Member States and the Commission use this framework to report progress, notably on the support provided to young and disadvantaged people.

The main issues are under-reporting, difficulties in measuring soft outcomes and in capturing the long-term benefits of YEI/ESF youth employment operations after the projects end. In addition, there is a significant time lag between reporting outputs, results and financial implementation, as a result of data checking and reporting requirements. Costs are declared only after the authorities carry out checks. Outputs are reported often only once operations are completed and declared, with results reported later still.

The reporting of indicators focusing on employment and on the formal qualifications gained typically underestimates the total benefits to society resulting from active labour market operations, for example the benefits to health and levels of crime. The managing authorities recognise the value of soft outcomes, especially for disadvantaged groups with multiple problems and they are said to be highly valued by participants in the context of youth employment operations. The value is generally in the social skills built up (self-esteem/confidence, cultural and community interaction and interaction) and in the non-vocational skills sets that employers look for (including reliability, honesty/discipline, time keeping and personal organisation). However, Member States seldom monitor soft outcomes. This may be due to difficulties in devising a consistent measurement method and in verifying recorded data. Moreover, soft outcomes are specific to each operation, which makes them unsuited for tracking by common indicators.

Managing authorities are aware of the value of capturing results after a period of time (six, 12 or more months after participation) and the value of tools such as ad hoc surveys, but practice varies.

From an evaluation perspective, and for the purpose of making a pan-EU comparative analysis, managing authorities should be able to use national data sets for econometric analysis or to track individuals (e.g. through tax and national insurances numbers or surveys, especially for tracking results after a participant has left an operation). However, this raises data privacy issues, in particular since the entry into force of the GDPR 38 .

Visibility is an issue

The visibility of EU funding is not only a legal requirement but also an important aspect of youth employment operations, since increasing awareness contributes to reaching the target population. Therefore, good visibility is key to boosting the efficiency of operations.

Visibility covers both general awareness-raising and specific communication on activities. On both counts, there is room for improvement for the ESF and even more for the YEI. The public consultation, for instance, revealed that some respondents who had received EU funding for youth employment operations were not aware of the ESF or the YEI.

Communication activities are essential in raising awareness. More work seems warranted to find effective channels to reach the target population. This appears clearly in the difference between the demand shown in responses to the public consultation for ‘new’ forms of communication and ‘old-style’ forms.

Most respondents viewed social media campaigns to be the best communication channel (76.9%), followed at a distance by youth networks, clubs and online groups (42.7% overall) and word of mouth (36.5% overall). However, most participants had found out about the support measures they had received via employment services and employment info centres (52.9%), and much fewer by word of mouth, family or friends. Only 26.2% had found out about these measures via social media.

5.3.Relevance: How relevant is the YEI and the other ESF-funded youth employment operations?

The following section refers to the extent to which the YEI and ESF-funded youth employment operations address the needs of young people, in particular young people not in employment, education or training. Given the changing socioeconomic context, especially the fall in the youth unemployment rate, this section also provides an analysis of how the implementation solutions have adapted to the changing needs.

Youth employment operations funded by the ESF and the YEI addressed the most relevant needs and groups

In line with the purposes of the Youth Guarantee, the YEI was created to provide support for young people not in employment, education and training to help them access the labour market. These needs were not only formulated in terms of providing jobs, but also in terms of developing their professional and personal skills to help young people make the transition to employment. The country-specific recommendations issued by the Commission to the Member States between 2014 and 2018, though not exhaustive, provide a good overview of the main needs at the beginning of the programming period:

·Improving basic training and skills;

·Reaching out to inactive young people and helping them find the way back to employment;

·Providing counselling and mentoring; 

·Facilitating the transition from school to work;

·Offering quality traineeships and apprenticeships as a first step towards employment;

·Providing incentives to employers for hiring young people.

The bulk of expenditure for ESF/YEI-funded operations is channelled to work-based learning, education and training in professional skills, guidance and support for individuals and combined operations as pathways to employment. For the YEI, a substantial share (26%) has been allocated to financial incentives to employers. The range of operations funded confirm that programmes were aligned with the identified needs of young people.

Box 2    Special support for hard-to-reach young people in Belgium

The Wallonia-Brussels AIR for 2018 in Belgium identified the difficulties experienced by young people not in employment, education or training and participating in YEI operations. They include their low skills and competence levels, lack of motivation, instability, problems linked to multiple disadvantages (e.g. access to housing, health, mental health problems and family problems). These issues require special support, such as individual and constant guidance, adaptation and close collaboration with specialist institutions (e.g. medical centres, mental health centres, diagnoses of learning difficulties, emergency accommodation) to prevent the young people from dropping out of operations. There is often a problem with attendance (absences and late arrivals).

The Wallonian ‘Sac a dos’ project finances several activities carried out by the beneficiary. It includes individual guidance and counselling, workshops to help participants improve their self-confidence and self-esteem and developing personal skills (such as autonomy, living in a group, expressing oneself) and short projects (of about four days) where young people can give back to the community and discover or improve specific techniques in a range of sectors (such as in construction, or in the ‘green sector’). It also funds short training sessions (minimum two weeks) in several businesses (such as in a flower shop or a fitness centre) to give the young people work experience. In 2018, the project provided assistance to 50 young people, with the number rising each year (especially among homeless young people). There are no official statistics tracking the situation of young people once they have left the project, but it would appear that most go back to school or enrol in a vocational training course.

Stakeholders and participants rated the youth employment operations as relevant and useful

During the public consultation, participants reported that training to build general skills, support in overcoming barriers to work/training, and training to obtain qualifications were the most useful types of operations.

Most organisations (over 90%) that responded to the public consultation consider the following actions as relevant or very relevant to help young people find quality and sustainable jobs: vocational-focused education and training, guidance and career support, paid apprenticeships/traineeships/internships and basic skills training.

By contrast, they ranked support to set up a business (71%), find a job or opportunity abroad (70.3%) and support to get back to school (64.8%) as relatively less useful. The contributing organisations considered community or voluntary work and non-remunerated apprenticeships to be the least relevant (47.5%).

Gender issues are taken into consideration initially, but there is a lack of active monitoring and targeting

In general, programme documents contain detailed strategies on how attention to gender equality is mainstreamed. The monitoring systems meet the obligation to disaggregate data collected by gender but often did not define gender-specific indicators that extend beyond levels of participation and provide results from a gender perspective. The same applies to the targets. Although all Member States report outputs and results by gender, only 41% of the output indicators focusing on youth employment with targets are gender-specific targets and measure whether implementation meets with the programme’s ambitions for gender equality. In terms of results, this is slightly lower as only 28% of all result indicators are gender-specific targets, corresponding to fewer than half of the operating programmes.

Only a small number of operating programmes (in Austria, Spain and Italy) included dedicated youth employment investments in gender equality (IP 8.iv), accounting for a total amount of €18 million. Other ESF/YEI operations (IP 8.ii) generally tend to focus on gender equality as a cross-cutting issue.

The operations did not always reach the most vulnerable groups

The strict eligibility criteria for young people not in employment, education or training mean that some vulnerable (young) people groups were not directly targeted by YEI operations, and sometimes not reached by ESF operations either. In focus group discussions, participants noted that YEI currently does not allow youth employment operations to focus on prevention work targeting groups at risk, for example potential early school leavers. Some countries were in favour of extending the definition of the target population (beyond those still at school) such as Italy where, due to severe unemployment in less developed regions, there is a need to extend the support to people up to the age of 34 years. In addition, some countries underlined the complexity of the NEET concept, as it requires verifying negative conditions.

Programmes sought to address the most relevant and vulnerable groups by taking steps such as:

·defining specific output targets (e.g. Spain, Luxembourg, Romania);

·creating mechanisms to identify the most disadvantaged groups (e.g. Malta);

·putting a specific focus on disadvantaged young people (e.g. Germany, Spain and Greece).

There are various ways in which operational programmes under the ESF address the most relevant target groups and tackle disadvantage. There are no strict eligibility requirements under the ESF for participation, so funding can be allocated to any such target groups, either under the dedicated investment priority for youth employment, or under broader types of objectives and investment priorities.

Support for youth employment focused on regions most in need of support

The budget allocation to different clusters of regions, as presented in Figure 14 overleaf, clearly shows how YEI focuses predominantly on regions where youth unemployment is most problematic (72% of the budget was allocated to regions in a very weak situation at the start of the process and that made limited economic progress).

The Rhomolo modelling study showed that some regions (particularly in southern Europe) that depend greatly on youth employment reap much of the benefits of YEI. It also highlighted that the allocated YEI amounts are proportionately higher in regions that have much higher youth unemployment rates, NEET rates or social exclusion and at-risk-of-poverty rates.

ESF funds are spread across different clusters of regions, but the fact that almost no youth employment operations are allocated to regions that had favourable youth employment figures in 2014 and that had improved further over recent years shows how ESF funding has been targeted to regions where youth employment is most problematic.

When combining ESF and YEI funding, the regions with the highest youth unemployment figures also received over half of all the youth employment budget (56%). Regions that also had high youth unemployment at the start of the programming period but that improved over time were the second largest beneficiary of funding (28%).

Figure 14 Overview of budget allocation to clusters of regions

Source: SFC2014, based on AIR 2018, data extracted on 6 September 2019 (Cluster A – Strong start/substantial progress, Cluster B – Strong start/limited progress, Cluster C – Weak start/visible progress, Cluster D – Very weak start/limited progress).

Operational programmes were flexible and able to adapt to changes

Operational programmes were able to adapt to changes stemming from the business cycle such as the fall in youth unemployment and the rising share of inactive young people in the target group, but also to unexpected developments, such as the immigration crisis.

Most Member States set specific objectives and target groups in a relatively broad way. As a result, minor annual adjustments to the programming did not have to go through formal amendment procedures.

The additional budgets made available to YEI did, however, require formal amendments to the programmes. They are also evidence of how the procedures were flexible enough to provide for additional budget to be programmed where necessary and facilitated the adjustment of formal targets set in programmes.

The budgetary changes made were mainly to the volume of operations supported, without leading to significant changes in the programme strategy.

Most adaptations to the programme were to accentuate the focus of regional targeting, to broaden target groups and the type of actions. Only in a limited number of programmes was the underlying programme strategy revised (e.g. North Rhine-Westphalia, Saxony-Anhalt or the operational programme for knowledge education development to adapt to the changing socioeconomic context).

Flexibility was thus a key factor in the adaptation of programmes and cooperation between programme stakeholders was an essential success factor in this regard.

5.4.Coherence: How coherent are YEI and the other ESF-funded youth employment operations among themselves, and with other actions in the same field?

The ESF Regulation (Article 4) requires that the strategy and actions set out in the operational programmes are consistent with, and respond to:

·the challenges identified in the national reform programmes,

·other national strategies (where relevant) that aim to combat unemployment, poverty and social exclusion, and

·the relevant Council recommendations, in order to contribute to achieving the headline targets of the Europe 2020 strategy on employment, education and poverty reduction. 

This section looks at coherence between YEI/ESF-funded youth employment operations among themselves and with the national policies.

The design and programming of the ESF and the YEI contributed to their mutual complementarity

On the one hand, YEI was designed to support similar operations to the ESF (such as employment and training, apprenticeship, hiring incentives, self-employment programmes), though it focuses exclusively on young people not in employment, education or training in eligible regions. YEI is therefore technically complementary to the ESF, available to the regions most affected by youth unemployment and providing support for particularly disadvantaged young people.

On the other hand, ESF also offered system support or was often budgeted as a follow-up measure to operations run under the YEI.

YEI and ESF youth employment operations are complementary and coherent with other EU schemes

The issue of complementarity and coherence with other EU funds in the same areas was barely covered in the national YEI evaluations. The public consultation showed that most organisations involved in running youth employment operations considered that the YEI and ESF were coherent with each other and with other EU schemes in similar or related areas.

Figure 15    To what extent are the YEI and ESF coherent with other youth and youth employment EU national/regional schemes? (n=399)

Source: Final Report, Public Consultation on the Evaluation of the EU Support to Youth Employment by the Youth Employment Initiative and the European Social Fund

This point was confirmed by the case studies and the analysis of the policy framework for the ESF and other EU funding schemes.

The YEI and youth employment measures under the ESF are specifically focused on access to the labour market for young people and helping them become active on the labour market. None of the other EU schemes replicate or unnecessarily duplicate YEI and youth employment measures under the ESF. Some funds (ERDF, EAFRD, EMFF) are geared towards creating sustainable jobs open to young people. The EAFRD and EMFF target young people by providing support for farming and fisheries ventures, including helping them to develop skills for these sectors.

The EURES and Erasmus+ programmes have a strong complementarity with YEI and ESF youth employment measures. EURES facilitates information sharing on job opportunities while Erasmus+ facilitates mobility and learning for young people.

Institutional coordination is essential element to ensure coherence

Coordination is key to ensuring complementarity with other EU funds or programmes. Conversely, poor coordination can be an impediment to success. Although it did not find evidence of extensive overlaps, a study commissioned by DG REGIO found evidence of poor coordination only in a limited number of cases 39 .

There is overall complementarity between EU-funded youth employment operations and national schemes, but some cases of overlapping and replacement were reported

According to the results of the public consultation, most respondents (43%) considered that YEI/ESF youth employment programmes are complementary to national/regional programmes. This was confirmed by the case studies run in Italy, France and Germany.

However, the analysis of YEI evaluations showed that in a few Member States, the YEI programmes sometimes replaced measures that used to be financed by the national budget, raising a question as to what extent EU resources have been used to complement national action and budgets.

In general, there are clear lines of demarcation. In some countries and regions, this is facilitated by coordinating partnerships that take a holistic approach to youth employment operations across EU and national programmes. Several EU programmes have a focus on young people, but none have the clear employment focus or the breadth of the YEI or ESF youth employment operations.

5.5.EU added value: What is the EU added value of the YEI and other ESF-funded youth employment operations?

To analyse EU added value, the evaluation used a wider framework extending beyond what could be achieved by Member States at national and/or regional levels. This framework helps assess the extent to which the YEI/ESF-funded youth employment operations was additional to national funding and programmes, and the ways in which the support helped reform and improve national systems.

Without YEI and ESF funding, some operations would not have been implemented or would have been more limited

At a time of economic crisis, EU youth employment operations enabled the running of operations that would have been difficult to achieve otherwise. According to the replies from the public consultation, the main effects of YEI were to strengthen approaches by providing extra funds, and by providing individualised support to participants.

According to the information available to the Commission on national labour market policies, in several Member States, some of the labour market measures specifically designed to help disadvantaged young people (i.e. unemployed, employed but at risk and inactive young people) were fully funded by the ESF/YEI, i.e. there are no equivalent measures funded only through national/regional resources. This reflects the significant volume effect of YEI and ESF-funded operations. For instance, all apprenticeships were co-funded by ESF/YEI in Ireland, Spain, Portugal and Finland. This was also the case for traineeships in Bulgaria, Greece, Latvia, Malta, Portugal and Slovenia, for institutional training in Estonia, Ireland, Latvia, Malta, Portugal and Slovenia, and for employment incentives in Estonia, Cyprus, Lithuania, Spain and Slovakia.

The case studies provided further examples where YEI and other ESF-funded operations produced effects at national and regional level that would not have been achieved without EU support.

Box 3    Common standards and tools developed in YEI programmes 

YEI and ESF-funded operations covered target groups that were not covered by other national or regional interventions

National evaluations, case studies and the public consultation provide evidence of the fact that the YEI enabled an expansion and refining of the scope of public employment operations. The YEI and the ESF brought to the forefront the needs of the target group (the NEETS in high youth unemployment regions) which would not have been a specific focus under other funding schemes.

Box 4    ‘From training to employment’ – the Voluntary Labour Corps project in Poland

The project led to the development of intensive, and sometimes tailor-made, approaches to target groups (vulnerable young people in different contexts), who might not otherwise have received such specific services. It also raised the level of ambition of support objectives.

Though rather limited information is available on the possible effects of the operations, in some countries it is recognised that the YEI/ESF helped to raise awareness about young people not in employment, education or training to pay specific attention to helping this target group.

The positive trends in NEET rates up until early 2020 are likely to be reversed as a result of the ongoing COVID-19 pandemic. Support to young people not in employment, education or training will continue to be at the core of the anti-crisis measures currently running in the Member States with funding from the current programmes, notably the ESF, as well as the future ESF+. As those instruments invest in both people and systems, they provide comprehensive support to respond to the challenges.

Outreach to and specific measures for this target group have been identified as a key policy areas where Member States will step up their action. They will continue to learn from each other in the context of the Council Recommendation on a Bridge to Jobs – reinforcing the Youth Guarantee, which the Commission proposed on 1 July 2020 40 . 

Box 5    Common standards and tools developed in the YEI programme 

 EU-funded youth employment operations have supported the implementation of national Youth Guarantee schemes

The YEI was created with the political will and aim ‘to complement other ESF-funded operations and national actions targeting NEET, including through the implementation of the Youth Guarantee’ 41 .

YEI and ESF-funded operations run in the context of the Youth Guarantee appear to have had a positive impact on employment results. According to information collected by the European network of Public Employment Services, from 2016-2019, ESF funding was used in at least 23 Member States to support the implementation of the national Youth Guarantee schemes, and YEI funding was used in at least 19 Member States. In 11 Member States, ESF was the main source of funding of the Youth Guarantee. The YEI was identified as the main source of funding of the Youth Guarantee in two countries.

Table 4    Use of ESF and YEI funds in implementing national Youth Guarantee schemes, 2016-2019 

✓ ESF/YEI funds are used to fund the national Youth Guarantee scheme.
✓✓ ESF/YEI are identified as the main source of funding of the national Youth Guarantee scheme.
N/A: information not available
Notes:
BE: between 2017-2019, BE-Actiris identified ESF as the main source of funding for the YG.
DK: No centralised national data on the specific amount of funds allocated to the Youth Guarantee.
FR: ESF funds were available from 2014 to 2020 and funded 50% of the scheme. As of the end of 2016, more regions were covered by the ESF. The YEI funds the scheme at a 92% funding level, but only in a few regions and certain provinces in regions already covered by the ESF.
Source: European Commission, Report on PES Implementation of the Youth Guarantee. The report of September 2017 covers the period from spring 2016 to spring 2017, while the report of September 2019 covers the period from spring 2017 to spring 2019.

Aggregate monitoring data on the Youth Guarantee and YEI for the period 2014-2018 indicate that one in six (16%) participating young people aged 15-29 who received a Youth Guarantee offer (of employment, education, an apprenticeship or a traineeship) had received support under the YEI. There is only a small difference between age groups: 15.5% of offers were taken up by young people aged 15-24 and 17.5% of offers were taken up by those aged 25-29 (includes only the countries that expanded the Youth Guarantee to also cover this age group). In Spain, Cyprus, Latvia, Lithuania, YEI outputs account for more than one in four Youth Guarantee offers, and in Italy more than half. By contrast, YEI outputs account for fewer than one in ten Youth Guarantee offers in Czechia, Romania and Slovenia. In total, the cumulated YEI outputs reported between 2014 and 2018 account for over half (55.1%) of all subsidised Youth Guarantee offers (i.e. those provided using public funds targeted either directly at young people or more broadly to the unemployed).

The YEI seems to have been particularly heavily used to (co)fund Youth Guarantee offers. It was used to cover at least 80% of all subsidised Youth Guarantee offers in Belgium, Greece, France, Italy, Latvia and Lithuania, though only marginally used (<10% of subsidised Youth Guarantee offers) in Czechia, Romania and Slovenia. This does not mean that the Youth Guarantee was underfunded in these countries, rather that the subsidised offers were mainly funded from other sources (ESF/national).

5.6.Sustainability: How sustainable are the YEI and the other ESF-funded youth employment operations?

According to the information available, the rate of participants making the transition to employment in YEI/ESF operations generally improved over time. The impact on employment is sustainable for low-skilled participants, who were the main focus of the operations, and for medium and high-skilled participants.

Continuity of EU-supported operations after the funding stops is largely dependent on the availability of alternative funding. Nevertheless, the systemic changes achieved through the ESF/YEI operations are likely to remain independently of the funding. From a macroeconomic perspective, the impact of the operations on GDP and particularly on employment are expected not only to last but even to increase over the medium to long-term, peaking in 2026-2030.

The effects of the operations for participants seem to last and increase over time

Participation in YEI and ESF youth employment operations would appear to have a positive impact on the employability of participants, becoming more visible over time.

The national YEI evaluations indicated that in most cases, the employment rate increased over this period.

Table 5    Employment rates by type of operation, upon completion and after 6 months 

Source: 2018 YEI evaluation reports
Note: The shares given for Lithuania vary according to the target groups (young people who are economically inactive, close to the labour market or far from the labour market)

The counterfactual impact evaluations carried out by the Member States back up this finding. In Italy for example, the counterfactual impact evaluation concludes that 54.7% of young people who participated in YEI are in employment 18 months after completing the programme (compared to 42.4% of their peers who did not participate). The Counterfactual Impact Evaluation for France highlight the value of the Youth Guarantee, estimating that at 18 months the employment rate for those on the Youth Guarantee would be 33.4%, against 24.3% for those who were not. In Spain, the proportion of participants in ‘Training and apprenticeship contracts’ who were in full-time temporary contracts after 18 months, was larger among those supported by the YEI than for those not supported (i.e. the control group - 68% against 52%. The Counterfactual Impact Evaluations for YEI in Portugal conducted by JRC (Competence Centre on Microeconomic Evaluation) 42 shows that individuals participating to YEI interventions have a higher probability of being employed 36 months after the intervention starts. The effect increases with the duration of the intervention and is stronger for hiring support schemes relative to internships. More specifically, the average effect on employment ranges between 7.7 and 31.7 percentage points. On average, individuals participating in YEI activities earn between 145 and 313 euros more, respectively for the shortest internship and for internship plus hiring support, 3 years after the program start.

The case studies indicate that the young participants gained a better knowledge of the labour market and improved their skills and employability.

Box 6    The impact of training on the employment rate in Italy

The national operational programme ‘Youth Employment 2014-2020’ ran in all regions (except the Province of Bolzano). It targeted young people not in employment, education or training aged 15-29 years and since end-2017, it also covers young unemployed people in less developed and transition regions. The programme involved providing extra-curricular traineeships, training, community service and accompanying paths to support self-employment.

Some 56% of participants had an upper secondary education, 24% had a low level of secondary education and 19% had completed higher education. In the programme as a whole, the average effect of the treatment increases over time since the participants started receiving support. Participation in the programme means that in the first semester lock-in effects are dominant and the employment rate of participants is lower than the control group. 18 months after the start of the support (approximately one year after the end of the support), the average impact on the employment rate of young people under 30 years is positive, ranging from an increase of 4 percentage points (in northern regions) to 17.1 percentage points (in central regions).

Changes to employment policies seem likely to last if the funding continues

The change of scope and the more targeted approach to the needs of young people, especially those in a vulnerable and disadvantaged situation, seems to have had positive effects and contributed substantial added value. However, programme continuity can only be guaranteed if sufficient funding is made available in the future.

The analysis revealed that many activities funded under the ESF would not continue without the funding, particularly where national funding is scarce (as it is in southern Europe). However, Spain has examples of programmes continued even without EU funding where stakeholders have integrated operations into their core activities, or in Italy where the new partnership approach was implemented at national level/ regional level, involving private and public employment services, and most importantly creating an integrated approach between employers and employment services and local organisations. In addition, where ESF/YEI funds strengthened and extended the scope of existing measures, this tended to ensure sustainability (in Germany, France and Malta).

6.Conclusions and lessons learned 

The analysis carried out for the evaluation leads to the conclusion that YEI and other ESF youth employment operations have been relevant in terms of meeting the needs of the target population and effective in providing solutions that addressed those needs, in particular for young people not in employment, education or training. It also indicates that, overall, the operations boosted the employability of participants, which may help them enter the labour market at a later stage.

This is in itself a positive outcome. It links with the objective, set in the Youth Guarantee, of helping young people integrate into and remain in the labour market, especially those further from the labour market.

A key challenge for the evaluation, being carried out at an intermediate stage of implementation, is that the programmes evaluated are still ongoing and a series of operations are incomplete (therefore the data on outputs and results are also incomplete).

6.1.OVERALL CONCLUSIONS

As a general conclusion, the study finds that YEI/ESF youth employment operations have helped improve the employability of young people across Europe. Some aspects of the operations were particularly relevant, such as innovative methods to reach the target population, coordination between partners, and holistic approaches to employment.

Based on the evidence available, it seems justified to conclude that, without EU support, fewer young people not in employment, education or training would have received support, and the overall NEET population would have been higher. The rate of YEI/ESF participants making the transition to employment generally improved over time.

In addition to achieving good results in terms of access to employment, employment-focused education and training, and self-employment, the operations boosted the overall employability of participants, notably by increasing their self-esteem, building positive attitudes to employment, and building their competencies, ensuring that the young people receiving support do not return to the NEET group. YEI also helped meet the information needs of participants, especially where operations worked closely with the public employment service, helping to ensure that young people have better access to the opportunities available.

In the vast majority of cases, the operations reached the target groups they aimed to reach, which expanded over the period to cover the 25-30 age group. Despite the challenge of reaching the target groups furthest away from the labour market, several outreach approaches were successful, many of them based on using social media.

Due to its initial frontloading, YEI performed better than the ESF-programmed funding for youth employment (e.g. 52% financial implementation against 27% for ESF youth employment operations at the end of 2018). Taking into account the learning curve in implementing targeted support to NEETs, the eligibility of YEI was extended until the end of the multiannual budget period and additional funds were budgeted for 2017-2020 to allow support to be maintained in the regions with high youth unemployment and to reach the targets set.

The organisations involved in ESF/YEI considered paid apprenticeships to be the most effective type of measure, together with vocational education and training and basic skills training. The latter are considered useful for most target groups, including the most disadvantaged, such as young people affected by poverty, young people who are at risk of social exclusion, young migrants, young homeless people and young people leaving education without a qualification. Several Member States provided support in the form of support pathways. This proved to be effective approach, in particular for participants further away from the labour market, who need preparatory support before they can make the transition to work.

The average unit cost per participation is around €2,000 with significant variations in costs and unit costs between types of measures, and between Member States. The cost per participant does not vary significantly between YEI and ESF youth operations. However, cost effectiveness is not determined by costs alone. Vocation education and training can involve high costs, but these measures have shown to be efficient in different contexts, when linked to work experience, thereby justifying the higher cost. By contrast, guidance is provided at a relatively low cost, but if it is not tailored to the individual, it is generally less effective in terms of generating employment results.

In some cases, the support provided by YEI and the ESF contributed (indirectly, for the YEI, as it targeted individuals and not systems), to structural changes in national systems of education and training and public employment systems. The influence on youth policies is perhaps more evident as YEI and ESF helped raise awareness of the challenges that young people face in accessing training and the labour market, especially those further away from the labour market, and to create interventions to support them. The ESF/YEI has brought about some significant changes in the strategic approach of public employment services and other bodies to youth unemployment, including cooperation between the multiple bodies involved and adjustments to the services they provide.

The administrative arrangements facilitated implementation, with few examples of gold plating. Bringing in the simplified costs options system helped reduce administrative burden, after initial delays and after building capacity in setting up the system.

Managing authorities needed time to take on new monitoring and reporting requirements and faced delays in setting up online monitoring systems and databases. Monitoring and reporting arrangements are now operational. There are, however, information gaps and delays in reporting, notably as regards outputs and results. This is due in part to checks carried out before declaring and reporting costs and performance indicators.

YEI and the other ESF-funded youth operations are relevant to meet the needs of young people. The action taken by the Member States to meet these needs, combined with a general economic uplift following the crisis, have yielded positive results. YEI and ESF-supported actions contributed to this positive development, some of which were found to have helped young people improve their skills and enter the labour market.

In addition to specific YEI/ESF actions, the way in which they were designed and implemented (the YEI/ESF approach or strategy) is also relevant, for instance, they started with a needs analysis and focused on targeting or step-by-step implementation.

As a consequence of the design of YEI operations, support for youth employment focused on the regions with the highest youth unemployment. The relevance of the youth employment operations is also evident in the capacity of programmes to adapt to changing conditions, such as the drop in the number of young people not in employment, education or training, by changing budget allocations or shifting operations to focus on certain regions or target groups that were most in need. In addition, the design of programmes proved to be flexible enough, in most cases, to accommodate any required changes without major reprogramming.

There is, however, scope for improvement. Some of the actions were less useful, due to limited work to identify the needs or insufficient targeting. Some needs were difficult to address due to limited capacity (e.g. lack of specialised social workers), eligibility conditions or the situation of extreme vulnerability of target groups. There is also scope to improve the employment offers provided, in some cases (in terms of duration and wage levels).

Evidence shows that YEI and other ESF-funded youth operations were coherent amongst themselves, notably due to the way they were designed and programmed. The key factors that contributed to this complementarity include the specific types of operations supported, capitalising on past experience with similar operations and institutional cooperation amongst stakeholders. This is also a key factor in achieving complementarity of YEI and ESF-funded youth operations with other related actions, i.e. actions supported by other programmes and funds at EU, national or regional level. Coordinating committees play an important role in ensuring complementarity and there are interesting examples of such set-ups at regional level.

The YEI and ESF have demonstrated considerable European added value. This includes volume effects by supporting interventions that were not funded by other national or regional programmes and enabling additional actions to take place. It also had important scope effects by widening the range of existing action and expanding target groups or including groups not covered by other programmes.

Although role effects were less evident so far, YEI and ESF-funded youth operations were instrumental in raising awareness of the situation of NEETs in the Member States. In fewer cases, the YEI and ESF had role effects in terms of bringing in innovative actions that were later mainstreamed into national youth employment policies.

YEI has made a positive impact on sustainable employment. The impact on employment is sustainable for the low-skilled, the main focus group of the operations, and for medium and high-skilled young people.

Continuity of the EU-supported operations after the funding stops is largely dependent on availability of alternative funding. Nevertheless, the systemic changes achieved by running ESF/YEI operations are likely to be lasting, independently of the funding.

The YEI Youth Guarantee-funded operations appear to have had a positive impact on employment. According to information collected by the European network of Public Employment Services, between 2016 and 2019, ESF funds were used in at least 23 Member States to help rollout the national Youth Guarantee schemes, and YEI funds were used in at least 19 Member States. In 11 Member States, ESF was the main source of funding of the Youth Guarantee. The YEI was the main source of funding of the Youth Guarantee in two countries. Aggregate monitoring data for the Youth Guarantee and YEI for the period 2014-2018 indicate that one in six (16%) participating young people aged 15-29 that received a Youth Guarantee offer (of employment, education, an apprenticeship or a traineeship) was supported by the YEI.

6.2.Lessons learned

Effective outreach is one of the main challenges in youth employment policy

Identifying and recruiting young people not in employment, education or training to join youth employment operations requires a range of innovative approaches, especially when the share of economically inactive young people in the target group rises.

Giving a very precise definition of that target population has resulted both in opportunities and challenges. Focusing the support of the NEET population has had a number of positive outcomes in terms of the public employment system and the assistance provided. It has put the spotlight on a specific part of the population particularly affected by the crisis, but had fallen somehow off the radar of mainstream employment policies. It also helped create the mechanisms and processes that may help reach out to these young people and provide the solutions they need.

Effective outreach to young people was identified as one of the main challenges in youth employment policy. Despite the high levels of young people not in employment, education or training in a number of countries, it was difficult to identify and reach potential candidates for youth employment operations. Moreover, the strict eligibility criteria for young people not in employment, education or training mean that some vulnerable (young) people groups were not directly targeted by YEI operations, and sometimes not reached by ESF operations either.

In addition, the most vulnerable young people are often under-represented, and potentially under-reported. This is in line with the replies submitted to the public consultation, which noted difficulties in reaching out to and supporting vulnerable young people in rural areas, those with disabilities, ex-offenders, those at risk of poverty and the homeless, and young people with multiple disadvantages.

Concerning the gender issue, the programme documents contain detailed strategies on how attention to gender equality is mainstreamed. The monitoring systems meet the obligation to disaggregate data collected by gender but often did not define gender-specific indicators that extend beyond levels of participation and provide results from a gender perspective. The same applies to the targets.

The EU-level focus group highlighted three principles for outreach, generally agreed by the participating Member State representatives, namely:

·local round work to assess the nature and extent of the challenge and potential solutions geared to the local context;

·using qualified outreach staff; and

·identifying innovative communication, or ‘interception’ channels to identify and reach potential participants.

Although the current overall positive outlook for youth unemployment in the EU might be significantly affected by the ongoing COVID-19 pandemic, the lesson remains valid as even in times of crisis, the most in-need young people in the target group must not be forgotten.

This is reflected in the Commission’s proposal for a regulation on the European Social Fund Plus by maintaining the provision that Member States support access to employment for all jobseekers, in particular young people.

Good cooperation and partnerships is key to success Stakeholders involved in youth employment policies emphasise the importance of cooperation at local and regional level, to both identify young people not in employment, education or training and to run programmes efficiently. Efficiency could be further improved by building up the capacity of programme authorities and social partners, as provided for in the proposed regulation on European Social Fund Plus.

Both the case studies and the replies to the public consultation highlighted the positive effect of aligning EU and national youth employment policies in programming. In the same vein, the public consultation highlighted the value of cooperation between different types of organisations, including trade unions, schools, research centres and youth organisations.

Regarding outreach, cooperation with youth organisations helped to identify NEETs who are not registered with the public employment service or in education. Cooperation during the operation ensures that it provided support for the range of services this population of young people needed. It was also very helpful to involve the local authorities in driving the process. Involving local companies helped raise awareness of the Youth Guarantee and generate employment offers. It also proved useful to ensure and formalise the participation of young people in planning activities in order to ensure that the operation provides the right services.

Evidence from the case studies illustrates the value gained from working in partnership and conversely, the missed opportunities, when cooperation and partnership is not strong – especially in outreach work, as shown above. At a policy level, aligning the ESF/YEI with national and other youth employment policies and programmes is critical in maximising the added value of the EU youth employment interventions. The replies to the public consultation also highlighted the importance of having functioning partnerships as one of the key factors for effective programmes.

Efficiency could be further improved by providing better information and further boosting the capacity of programme authorities and social partners, as provided for in the proposal for a regulation on European Social Fund Plus.

Tailored offers are most beneficial in the long term 

Individuals furthest away from the labour market benefit greatly from tailored and intensive guidance and support.

A significant proportion of young people targeted by youth employment actions are not job ready, and require preparatory and ongoing support to help them make the transition to successful employment, qualification or labour market inclusion. It has proved essential to involve trained youth workers to help disadvantaged young people make the transition from inactivity to participating in training or employment schemes.

Very often this group has low educational attainment levels and requires a blend of basic skills training and vocational education, in combination with employment interventions. During the EU-level focus group work, a number of representatives emphasised the importance of working with schools as most of the problems start before young people reach the age to qualify as NEET, i.e. at 15, though there are limits to using YEI and ESF in working with young people at an earlier age. Belgium, Latvia and Slovakia emphasised the need to cooperate with schools to reach these young people as early as possible. In this respect, the definition of NEET can be restrictive since it only includes young people already not in employment, education or training, not those that may soon join the category.

Many ESF/YEI operations focus on work experience to give an opportunity to young people who would otherwise find it very hard to find a placement or traineeship. Often work experience is more attractive to young people who have experienced failure at school. It gives them a chance to discover what they are interested in and where their strengths lie. It also helps them develop social skills that are essential to the workplace. To conclude, work experience has proved to be a vital route into employment and helps young people develop their social skills.

Many Member States have adopted a ‘pathway approach’ (or combined approach) to tackling youth employment. This is particularly the case with YEI support. Many young people not in employment, education or training need to participate in a series of interventions in order to make the transition into work or education and therefore they benefit from a comprehensive approach. In Germany and Poland, the individualised approach to assisting and supporting young people (in particular those most in need) is working well. Organisations responding to the public consultation highlighted the importance of tailored approaches for programmes to be effective, ensuring that they focused on individual needs.

Measuring the effectiveness of youth employment programmes can be improved, as can the visibility of EU funding

Measuring the effectiveness of youth employment operations improved significantly over the current programming period but there is still scope for improvement. The problem of under-reporting would be aided by more regular reporting by the managing authorities instead of the current yearly practice. Achieving a greater degree of standardisation of soft outcomes and longer-term results in addition to the compulsory indicators, at least at national level, would also aid future evaluations.

To reduce the administrative burden, the next regulation should provide the legal basis for using administrative data in the future.

Further efforts should be made to make microdata more available and evaluation practices would benefit from a further increase of the number of counterfactual impact evaluations and more stable information on unit costs, as the operations come to an end.

There seems to be an issue of the visibility of EU support provided for youth employment. Broader communication strategies can be used to address this issue, by using social media and forging partnerships to work with schools, community organisations and frontline services. Increased visibility would also aid better outreach.

Annex 1: Procedural information

1.Lead DG, Decide Planning/CWP references

This evaluation was carried out by DG EMPL as an initiative published in Decide with the reference number PLAN/2018/2966. It was published in July 2018 ( https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2018-3876690_en ).

2.Organisation and timing

An interservice steering group (ISSG) was set up in July 2018, which included the following DGs: EMPL, AGRI, BUDG, EAC, GROW, HOME, JRC, REGIO, RTD, SG and the Legal Service.

The timeframe for the evaluation was as follows:

3.Exceptions to the better regulation guidelines

N/A

4.Consultation of the Regulatory Scrutiny Board (if applicable)

N/A

5.Evidence, sources and quality

The evaluation was based on three main sources of information:

-the twelve-week public consultation, carried out and analysed with the assistance of the external team of experts;

-an estimate of the macroeconomic effects of youth employment operations using the dynamic computable general equilibrium model (Rhomolo) run by the Joint Research Centre under an administrative agreement between DG EMPL and JRC;

-a study carried out by an external team experts, under contract (VC/2018/0715) through DG EMPL Multiple Framework Contract VT/2016-027 for the provision of services related to the implementation of the Better Regulation Guidelines. The external study combined the results of the two other sources with additional work.

The quality of the final report of the external contractors was assessed as good by the interservice steering group.

The conclusions and findings of the evaluation are considered to be robust, mindful of the limitations and mitigating measures described in section 4.2 of the evaluation.

Annex 2: Stakeholder consultation

1.The consultation strategy

1.1.Objectives

This synopsis report outlines the consultation organised to evaluate ESF support for Youth Employment from 2014 till 2018 and presents the main findings.

To ensure transparency and involve the stakeholders, the process followed the standards and methods set out in the Better Regulation guidelines. The various consultations have followed the roadmap and consultation strategy. The roadmap 43 of the evaluation itself was published on the Better Regulation website and open for public feedback between 20 July 2018 and 17 August 2018 44 .

1.2.Consultation stakeholders

The stakeholders targeted by the consultation were organisations or individuals that:

·had an interest in youth employment operations funded under the ESF/YEI;

·had or might have participated in the operations;

·had expertise in the subject;

·and had or might have run or been involved in running the operations.

Therefore, the following groups were formed for the consultation:

1.Participants: people who have received support under the Youth Employment Initiative or other youth employment operations, with the aim of gaining an insight into the extent to which the objectives of the funding have been achieved in terms of their integration in the labour market, and which factors played a role in this that weren’t directly identified by the monitoring mechanisms;

2.Young people not in education, employment and training not reached by the operations (NEETs): the consultation tools collected the views of members of this group (the main target for the Youth Employment Initiative) who had not participated in the operations, with the aim of understanding why they had not;

3.Bodies involved in running the operational programmes, such as managing authorities and other Member State representatives, social and economic partners represented in the monitoring committees;

4.Organisations involved in delivering youth employment operations as beneficiaries or project partners and their EU-level representatives: public administrations at the national, regional or local levels, public and private training and education providers, workers’ and employers’ organisations, youth organisations, NGOs, charities and companies. Their feedback was relevant to all evaluation questions, particularly to identify any issues of efficiency and delivery in running the youth employment operations;

5.Organisations and individuals not directly involved in running the operations, but who have a stake in youth employment issues, notably with regards to young people and, specifically, young people at risk of social exclusion, who contributed their views on how the operations met the specific needs of young people;

6.Academic and research bodies with expertise and knowledge of youth employment policies and issues, who provided insights into the relevance and coherence of measures;

7.Organisations representing employers, who should contribute feedback on the factors that played a role in the successful integration of participants, and young people in general, in the labour market;

8.Young people in general, who contributed their opinions on the relevance of the action taken;

9.The general public, i.e. any individual or organisation outside the previous groups who wanted to provide their views on EU support for youth employment.

1.3.Consultation methods and tools

2.Synopsis Report

2.1.Stakeholders involved in managing the ESF

Stakeholders involved in managing the funds were consulted regularly during the ESF evaluation partnership, which comprises Member State representatives of ESF monitoring and evaluation capacity. They carried out numerous tasks, ranging from preparing the evaluation questions to presenting the findings.

Draft reports were circulated to the partnership members, who were given the opportunity to comment on the outputs of the evaluation. The comments made by the managing authorities were mostly to flag inconsistencies in data related to individual Member States, which were then corrected.

2.2.The public consultation

2.2.1.Description of the activity

The online public consultation was launched on 24 May 2019 and ran until 16 August 2019 using the EU Survey tool. The questionnaire comprised 26 questions (excluding profiling and closing sections), structured around the main evaluation questions and including both open-ended and closed questions.

2.2.2.Stakeholders

The public consultation was structured around five groups of respondents, defined as follows:

·Group A.1: Individual citizens 33 years old or under who had taken part in the ESF/YEI – i.e. ‘ESF/YEI participants’;

·Group A.2: Individual citizens 33 years old or under who either did had not taken part in the ESF/YEI, had taken part in other support schemes not funded by the ESF/YEI, or had taken part and didn’t know if it was ESF/YEI-funded – i.e. ‘other young people’;

·Group B: Individual citizens over 33 years old – i.e. the ‘general public’;

·Group C: Organisations involved in managing the ESF/YEI (such as managing authorities, intermediate bodies, beneficiaries, and social partners involved in running or monitoring the ESF/YEI) likely to have a direct and detailed knowledge of YEI/ESF – i.e. ‘organisations involved’;

·Group D: Organisations not involved in managing, monitoring and running the ESF/YEI, i.e. entities or organisations that presumably do not have a direct stake in the ESF/YEI – i.e. ‘other organisations’.

The number of questions for the respondents varied by profile of respondent.

2.2.3.Results

The public consultation received 1,376 responses.

It achieved a satisfactory level of overall representation of different respondents’ profiles, in particular for young people, who were a key target of the consultation. Most respondents were under 33 years of age (57.6%) who either took part in ESF/YEI (24.7%) or did not take part in ESF/YEI (32.9%). Just over one fifth of respondents (21.8%) belonged to organisations involved in managing the ESF/YEI. The remaining respondents were either from the general public or belonged to organisations not involved in the schemes.

Most of the respondents who participated in some form of youth employment support (339 ESF/YEI participants and 92 other participants) did so through information on job opportunities, guidance and tutoring, support to find work experience and training for general skills. The least used type was support for going back to school or setting up a business. There were no significant differences in the type of support received by ESF/YEI participants and other participants.

Almost one third of respondents replied on behalf of an organisation. Overall, the most responses from organisations were from public authorities (38.8%), followed by companies and business organisations (22.3%) and NGOs (18.5%). Beneficiary organisations (applying for ESF/YEI funds and running the projects) are the most represented among organisations involved (Group C).

Over half of respondents are familiar with or have an idea of the goal and scope and know at least one activity funded by the ESF or YEI, or by both.

There is an important caveat regarding representativeness. The public consultation run as a voluntary, online survey may not have reached all target groups in the desired way. The responses to the public consultation show a very unbalanced geographical distribution, with four countries covering almost three-quarters of responses: Italy (20.4% of responses, considering only those referring to one single country), Bulgaria (18.8%), Spain (16.4%) and Slovakia (16.4%). All other countries elicited much fewer responses and two countries no responses at all (Denmark and Luxembourg). Almost 60% of individual respondents were women; 40% were men.

Relevance

The questions on relevance explore the extent to which the different types of actions and support are in line with the needs of young people, in particular young people not in education, employment or training, in terms of helping them find a job, improve their professional skills or get closer to the labour market.

When asked to rank the usefulness of youth employment support actions, individual respondents (as opposed to those answering on behalf of an organisation) ranked information on job opportunities, guidance and tutoring followed by support to find work experience; training for general skills; support in overcoming barriers to work and training; and training for qualifications as the most useful forms of support (between 84.8% and 82.3% say they were mostly or very useful).

Fewer respondents ranked support in setting up a business, in finding a job or opportunity abroad and support to get back to school to be most useful (between 71% and 64.8%). For more details see Figure 16 . Respondents suggested additional support that could be useful, such as training in social skills and psychological support. 

Figure 16    In your opinion, how useful are the following actions in helping young people find a job, improve their professional skills or enter the labour market? (Group A1, A2 and B, n=976)

The ESF/YEI action considered most relevant by responding organisations was vocational education and training, followed by guidance and career support; paid apprenticeships, traineeships and internships; and basic skills training. Community or voluntary work and non-paid apprenticeships were ranked the least relevant (47.5% find it relevant or very relevant vs over 90% for other types of support).

In the open questions, a few respondents suggested that social skills training and emotional support (such as counselling or psychological support) are also relevant types of support that are less often provided by ESF/YEI operations.

Overall, most responding organisations reported that EU support was flexible enough to adapt to emerging needs. Respondents highlighted the capacity of the ESF/YEI to adapt to the needs of young people and to the socioeconomic context in the different regions and Member States. Conversely, some others sent negative replies citing a lack of capacity of the programmes to adapt to the specific needs of young people and to tailor the actions to specific national and regional needs.

Effectiveness

Responses to the questions on effectiveness describe the extent to which ESF/YEI measures achieved the expected results.

A key factor in the effectiveness of ESF, and especially YEI, is the programmes’ capacity to reach out to and involve those farthest from the labour market. Overall, respondents to the public consultation identified social media campaigns as the best channels to inform young people of support initiatives. In distant second place were youth networks, clubs and online groups, and word of mouth. However, when young participants were asked how they actually learned about the support measures they took part in, most said through employment services and employment info centres, followed at a distance by word of mouth, family or friends and only a smaller share learned about them via social media.

The main results that ESF/YEI participants achieved through the programmes were in improvements to their skills and qualifications, whether to develop general skills (38.3% of respondents), professional skills and qualifications (35.7%), or to start or resume education and training courses (29.2%). A smaller but still significant share acknowledged that ESF/YEI support helped them join the job market, either by finding a temporary or a permanent job (20.9% and 16.8% respectively) or by improving their employment situation (18.3%). One in ten respondents reported that the support received did not help.

Figure 17    The support you received helped you in… (Group A1 and A2, n=431, multiple answers allowed)

Compared to other participants, the ESF/YEI participants are more likely to receive support to improve their education and qualifications and to develop their skills in general, although the employment outcomes are similar (in terms of finding either a permanent or temporary job). The other participants reported in higher numbers that the support had helped them find a job.

Respondents were also asked about their current and past labour market situation (before they received support). Of the ESF/YEI participants, 46.7% are currently unemployed and 29.5% employed full time. Before receiving support, 88.1% were unemployed and 1.6% were employed full time, showing a consistent increase in the employment rate. Of the other participants, 51.1% are currently unemployed and 28.4% are employed full time. Before receiving support, 77.2% were unemployed and 0.6% were employed full time.

Table 6    Labour market status of individuals who took part in ESF/YEI initiatives

According to the organisations that responded, the main results of the ESF/YEI programmes were to develop the skills and qualifications of young people and to help them find a job, including for young NEETs and other disadvantaged individuals, such as the disabled. This echoes the results reported by individual recipients. Overall, organisations were more sceptical concerning the effectiveness of ESF/YEI in helping young people improve the quality of their employment. Respondents also reported that ESF/YEI generated ‘soft outcomes’ in terms of increased partnerships and collaboration among stakeholders, of integrating marginalised people, developing soft skills and group work, empowering young people, increasing their fit with job market requirements, increasing their motivation and integrating young people not in employment, education and training into society.

In terms of target groups, the organisations that responded reported that the ESF/YEI is relatively more successful in helping young NEETs and, to a lesser extent, young people leaving education without a qualification and those at risk of social exclusion or marginalisation than in helping young people in rural or hard-to-reach areas, or those at risk of poverty.

Individual respondents gave a positive assessment on all facilitating factors mentioned, in particular concerning the provision of financial benefits to participants and measures that are aligned with labour market needs. Respondents from organisations, especially from organisations involved in ESF/YEI, focus on the importance of having operations that are well integrated with general youth policies and on providing accompanying service or financial and non-financial incentives to support young people during their participation. They also mentioned flexibility in implementation relatively more frequently as an important factor. Additional factors are cooperation among stakeholders (e.g. enhancing the capacity of measures to adapt to local specificities and improve dialogue between public and private institutions), focus on individual needs and follow-up activities.

Young respondents who did not participate in ESF/YEI mostly mentioned the lack of information as the reason for not participating.

Organisations indicated the difficulty in reaching out to target groups as the most frequently selected factor hindering effectiveness (60.9%). Other factors mentioned, although less frequently, are structural problems such as the lack of jobs and the low level of education of participants, the administrative burden for beneficiaries, and the lack of involvement of stakeholders. Of the countries with the highest response rate, Portugal cited difficulties in reaching out to target groups relatively more frequently (81.3%), followed by Spain (67.6%).

Efficiency

The questions on efficiency focus on the ‘value for money’ of measures and whether resources invested by the ESF/YEI are proportionate to the results achieved.

For the ESF, respondents from organisations involved agreed mostly on the cost effectiveness of vocational education and training activities, followed by apprenticeships, traineeships and internships, basic skills training (80.9%) and guidance and career support (between 83.3% and 80.4%). Agreement seems to be quite low regarding community and voluntary work (41.6%). The assessment of the YEI is similar.

Figure 18    To what extent do you agree or disagree that the following activities implemented under the Youth Employment Initiative were cost-effective? (Group C, n=206)

Overall, most respondents from organisations involved in the programmes judge the administrative arrangements for project implementation, project selection, communication and evaluation to be appropriate (between 65% and 61%). Fewer than half of respondents judged the management and control system to be appropriate.

Coherence

The questions on coherence assess the extent to which ESF/YEI activities are aligned and complementary with other youth and youth employment national or regional schemes.

Most organisations could not rate the coherence of ESF or YEI with other youth and youth employment national/regional schemes for all of the suggested schemes (Erasmus+, EURES, ERDF and European Solidarity Corps). The only exception regards coherence with existing national or regional schemes, with 43.1% of respondents saying that overall they are coherent, with a higher rating from organisations involved than from organisations not involved (45.3% vs 36.4%). However, overlapping or non-alignment of schemes does not seem to be an issue.

Figure 19    To what extent are the YEI and ESF coherent with other youth and youth employment EU national/regional schemes? (Group C and D, n=399)

EU added value

The EU added value questions assess the additional value resulting from ESF or YEI support, compared to what could be achieved by Member States at national, regional and local levels with own funds. This question was only put to organisations.

Organisations gave an overall positive perception of an EU added value resulting from ESF or YEI support, mostly because it expanded coverage of the assistance provided (63.5%) and increased the assistance provided (47.2%). Very few respondents (3.9%) said that it did not make a real difference. Overall, respondents from the organisations involved gave a more positive response while more group D respondents reported that it did not make a real difference (11.6%, v 1.4%).

Figure 20    Has the EU provided support that goes beyond what is addressed by national or regional programmes or support schemes aimed at youth employment? (Group C and D, n=282)    

Closing questions

A high share of young respondents would be interested in taking part in EU youth employment activities in the future (66.8%). The type of activities that appear to interest them most are training courses to improve job skills, work experience such as traineeships and internships, basic skills training and support to find a job.

2.3.Member State-level interviews and focus groups

2.3.1.Stakeholders involved

Interviews and focus groups with stakeholders at Member State-level were carried out during the evaluation in 10 Member States and for 20 operational programmes. The types of stakeholders consulted were managing authorities, beneficiaries, socioeconomic partners, evaluators/researchers and representatives of participants – e.g. youth organisations.

Table 7    Number of interviewees by Member State and type of organisation

In addition to interviews, focus groups were carried out in in Spain, France, Poland, Greece and Portugal in order to obtain detailed information from key stakeholders on the evaluation questions. Stakeholders involved included national and regional managing authorities, PES and other beneficiaries and other private or public organisations such as research institutes. Where focus groups could not be held, additional interviews were organised as well as wrap-up meetings with interviewees.

2.3.2.Results

The case studies were used as the main source of information to answer the evaluation questions and feed into case study reports.

The interviews for the case studies highlighted an increase in the time required to locate and recruit from harder-to-reach groups. Member States also use different targeting policies, reflecting national priorities and national assessments of needs, with some countries using the youth programmes to target young people closer to employment, including graduates, and with the results generally showing higher levels of effectiveness. ESF/YEI operations showed a high level of flexibility and innovation in tackling the challenge of accessing hard-to-reach and vulnerable groups.

2.4.EU-level focus group

2.4.1.Stakeholders involved

An EU-level focus group was held on 2 October 2019 in Brussels. It was organised back-to-back with the Technical Working Group and the aim was to discuss some of the evaluation questions with a smaller group of participants, focusing on key issues and gaps. The focus group comprised representatives from managing authorities and European Commission officials. There were 15 participants from 11 Member States.

2.4.2.Results

Effectiveness/outreach

The discussion confirmed the preliminary findings of the study and highlighted that managing authorities face difficulties with outreach, especially to reach the most disadvantaged young people in the broader target group of young people not in employment, education or training (NEETs). Member States discussed the country-specific obstacles they face and the remedial actions taken to overcome them.

Efficiency/partnerships

The discussion confirmed, in line with the preliminary findings of the evaluation, that cooperation among the multiple bodies involved, in particular among employers and employment services, is key to the efficient implementation of youth employment measures. It also confirmed that this is best facilitated when employers have a clear interest to work with the employment services, which can only happen if the training offer is in line with the companies’ needs for skills or when they have a financial incentive.

Sustainability

The focus group discussion revealed that managing authorities are aware of the need to measure and monitor the sustainability of results of youth employment operations, particularly with regard to making employment outcomes sustainable over time. It also revealed that Member States have difficulties in doing so, even though the result indicators (short and longer term), administrative data sets and ad hoc surveys are useful support tools. Sustainability can also be interpreted in terms of systemic changes such as an improved governance of youth employment policies or increased institutional capacity.

Annex 3: Methods and analytical models

1.Overall approach to the evaluation work

In line with the Better Regulation guidelines, DG EMPL decided to base this evaluation on the work carried out by external evaluators and took the following approach:

·collect and analyse the relevant evidence;

·provide answers to all evaluation questions;

·present evidence-based conclusions.

2.Rationale of the evaluation

The main objective of this evaluation is to assess the youth employment-focused operations and their relevance, effectiveness, efficiency, coherence and EU value added. The evaluation includes both an individual assessment of each country and a cross-cutting and comparative assessment. An external contractor collected and examined data covering 2014-2018 and prepared a study, including a forward-looking analysis of future implementation.

3.Evaluation questions and structure of the report

The evaluation was based on the following evaluation questions:

Data and info sources to reply evaluation questions: (1) – external study; (2) – SFC2014; (3) – national evaluations; (4) – EUROSTAT data

The evaluation work and the report were structured around the conclusions reached on each of the questions.

4.Methodology and data sources

The evaluation was based on a complex methodology aimed at collecting solid evidence and providing well-informed answers to the evaluation questions.

It consisted of:

·extensive desk research;

·an open public consultation;

·interviews;

·case studies;

·synthesis of 2nd YEI evaluations;

·a cost-effectiveness analysis; and

·focus groups.

To better understand and evaluate ESF and YEI-funded operational programmes, it was important to look not only at the national situation, but also the situation in the specific regions in which the operational programmes are run, since there can be significant differences between different parts of the country. This was recognised in the categorisation of regions for implementing the Structural Funds (into more developed, less developed and transition regions). The aim here was not, however, to re-describe the context with additional detail by type of region, but to identify clusters of regions with similar characteristics that can be used together with the ESF/YEI monitoring data to see if the underlying socioeconomic situation and how it has developed have an impact on the effectiveness and efficiency of implementation.

To this end, data on selected indicators (based on the availability of data at regional level) for regions (NUTS 2 level) were aggregated to produce data by type of region in each country (i.e. maximum three types of region per country, total of 50 regions) and then used to develop two composite indicators: one describing the situation at the beginning of the programming period in 2014 and one describing how it evolved between 2014 and 2018. By combining the data for the two composite indicators, four broad clusters of regions were identified:

·Cluster A: Regions with a strong starting point that made significant progress;

·Cluster B: Regions with a strong starting point that made little progress;

·Cluster C: Regions with a low starting point that made significant progress;

Cluster D: Regions with a low starting point that made little progress.

The following steps were taken to create composite indicators and to cluster regions by typology:

1.Indicators were selected based on the availability of data at regional level (i.e. only indicators with more or less complete data by NUTS 2 regions could be used). The indicators had to be youth-related and cover the general economic context since the underlying economic situation generally has a significant impact on potential labour market and programme outputs and outcomes;

2.The indicators were calculated by type of region for each country. For indicators for which low values suggest a positive situation (i.e. the rate of young people not in employment, education or training and the early school leaving rate) reciprocal values were used so that when the composite indicator is calculated, higher values indicate a more favourable situation or greater progress.

3.The change between 2014-18 was calculated (or other dates depending on data availability) for each indicator in absolute numbers (percentage points used for % indicators, PPS for GDP per capita);

4.The indicator values for 2014 and for 2014-18 change were standardised using z-scores (or standard scores) to indicate how many  standard deviations  an element is from the mean 45 ;

5.The value of the two composite indicators was calculated (describing the starting point and change) based on the average standardised values of each contributing indicator;

6.Countries and regions were clustered by typology using the k-Means method 46 . .

4.1.Synthesis of 2018 YEI evaluation reports

The synthesis of 2018 YEI evaluation reports was also formally considered part of Task 1. It was carried out based on 21 evaluation reports submitted by the Member States up to mid-September 2019 (see annex 7 for the full list).

By that time, evaluation reports were available for all Operational Programmes except for Operational Programme Wallonie-Bruxelles (the final evaluation report of the Operational Programme will include a specific section on the evaluation of YEI) 47[1].

The approach used to prepare the synthesis of the evaluation reports was as follows:

·Review of the executive summary in English if available

·Translation of the evaluation report for languages not covered by the research team (including the executive summary if not available in English)

·Detailed analysis of the evaluation report

·Summary of evaluation findings (in Word) using the template elaborated by the Evaluation Helpdesk (including the quality assessment)

·Editing of each summary by Terry Ward, the quality manager (but also directing work on the Evaluation Helpdesk)

·Categorisation of the reported findings in an Excel database

·Preparation of the synthesis

Final editing of the synthesis by the quality manager.

In addition to the YEI evaluation reports, other evaluations of youth employment programmes were screened using the database of evaluations compiled by the Evaluation Helpdesk. The relevant findings for these are not included in the Synthesis of the 2018 YEI evaluation reports (since these only deal with YEI evaluations) but are reflected in the replies to the evaluation questions.

4.2.Comparative analysis of the evolution of the labour market and the role of the EU funded operations (see also section 2.1. of the main report)

The aim of the comparative analysis is to analyse the context and evolution of the youth labour situation across the EU in relation to the YEI and ESF activities and to identify, as far as possible, factors that have contributed to the effectiveness and efficiency of YEI and ESF youth-related operations. The outcomes are used to put the results of the ESF/YEI operations into context when answering the evaluation questions and prepare the background information included in the case study fact sheets. The analysis focuses on the following aspects:

·    Socioeconomic context at national and regional level;

·    Administrative and organisational procedures;

·    Actions to support young people; and

   Efficiency and effectiveness of ESF/YEI operations.

The work carried out for each of these areas is briefly presented below.

To complete the comparative analysis for the aspects mentioned above the following activities were carried out:

·Selection and analysis of indicators at national level relevant to youth employment derived from readily available statistics published by Eurostat. The analysis looks at changes over time (2014 – 2018 or latest year with available data) and includes comparisons between countries and different age groups.

·Clustering of regions based on the general socioeconomic context in four broad clusters:

·Regions with low starting point and little progress,

·Regions with low starting point and significant progress,

·Regions with strong starting point and little progress,

·Regions with strong starting point and significant progress,    

·Comparative analysis of administrative and organisational procedures in terms of the delivery of youth employment activities in the EU Member States. The aim is to briefly describe the main actors providing youth employment activities – YEI/ESF Managing Authorities and Youth Guarantee implementing authorities – and to provide some insight on the role of Public Employment Services (PES) in implementing the national Youth Guarantee schemes. To collect information on major actors, three different data sources have been used: ESF/YEI Operational Programmes to identify the ESF/YEI Managing Authorities; the national Youth Guarantee implementation plans to identify the Youth Guarantee implementation authorities; and the Labour Market Policies (LMP) database to identify the Public Employment Services (PES). Additionally, to conduct the analysis on the role of national PES in implementing the national Youth Guarantee schemes and the progress made through time, the annual PES Network reports on the implementation of the Youth Guarantee were utilised. The findings of the reports are based on responses provided by national PES to an email questionnaire distributed by the European Commission.

·Comparative analysis of interventions specifically targeting young people implemented at the national level and main target groups. The analysis is based on information obtained from the EU Labour Market Policies (LMP) database (managed by DG EMPL) and focuses on three different aspects: types of interventions; target population (i.e. specific groups of young people); and level of funding. The aim of the analysis is to assess the degree to which ESF and YEI funding has contributed to existing interventions or whether it has been used to introduce new interventions providing more/different options to young people or to target groups that are not targeted by existing nationally funded interventions.

·Assessment of the effectiveness and efficiency of the ESF/YEI Operational Programmes by combining ESF/YEI monitoring data with context indicators at national and regional level.

4.3.Cost-effectiveness analysis 

The analysis involves two main sub-tasks. The first is to provide an indicative estimation of efficiency across the Member States starting from the mapping of the costs per outputs and results for Investment Priority 8.ii. Our analysis highlights the range, nature and overall extent of costs per person or operation and the costs per type of operation based on monitoring data and econometric analysis. The second sub-task involves a detailed cost-effectiveness analysis for a sample of operations. We also relate costs to evidenced causal effects, i.e. effects of the operation beyond its outputs and results (net results) drawing mainly on the analysis of YEI and counterfactual impact evaluations in which we identify evidenced examples of effective operations and their costs. We also explore whether lump sums, standard scales of unit costs or flat rates used as a means for simplification for beneficiaries and lowering administrative burden for partners, led to cost-effectiveness.

5.Overall assessment of strengths and weaknesses of the methodology and data

The various sources were triangulated, and where the evidence was insufficient or inconclusive, the approaches were combined: data-based, documentary and perception-based sources as well as quantitative and qualitative techniques, depending on the nature of the evaluation question and the strengths of the relevant data and approaches.

The evaluation drew on a number of sources providing opinions and perceptions, including surveys, interviews and the public consultation. Data on opinions and perceptions are important evidence where evaluation questions elicit the views of stakeholders. In some cases, respondents or interviewees may be the only source of knowledge or witness accounts of events when no other sources are available.

Under the Better Regulation guidelines, the open public consultation is an important tool for collecting stakeholder input and views on EU policy initiatives. It cannot be expected to provide a fully representative view of EU public opinion, but it does offer a channel for the people who care about a given issue to voice their opinion. As expected, respondents’ knowledge and involvement in running youth employment operations differed, but the design of the public consultation made it possible to distinguish between the respondents who were well-informed and others. This distinction fed into the analysis of the responses, and triangulated with other sources of evidence.

In conclusion, while acknowledging that there are some data gaps and methodical limitations (see above), the evaluation presents well-informed, evidence-based and reliable answers to the questions, to the extent possible.

(1) Article 19(6) of Regulation (EU) No 1304/2013 of the European Parliament and of the Council of 17 December 2013 on the European Social Fund and repealing Council Regulation (EC) No 1081/2006.

(2)  ‘Sustainable integration into the labour market of young people, in particular those not in employment, education or training, including young people at risk of social exclusion and young people from marginalised communities, including through the implementation of the Youth Guarantee.’ Article 3(1) of Regulation (EU) No 1304/2013 of the European Parliament and of the Council of 17 December 2013 on the European Social Fund and repealing Council Regulation (EC) No 1081/2006.

(3) https://ec.europa.eu/info/sites/info/files/com_2020_451_act_v8.pdf.

(4)   https://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:C:2013:120:0001:0006:EN:PDF

(5)  Currently, the Youth Guarantee is limited to those aged 15-24 only in Belgium, Denmark, Ireland, France, Luxembourg, Hungary, the Netherlands, Austria, Romania and Sweden. In the other 17 Member States, it is open to the 15-29 cohort.

(6)  NUTS level 2 regions. The NUTS classification (Nomenclature of territorial units for statistics) is a hierarchical system for dividing up the economic territory of the EU for the purpose of, among others, socio-economic analyses of the regions where NUTS 2 are basic regions for the application of regional policies.

(7) The unemployment ratio is the youth unemployment ratio divided by the adult unemployment ratio. Thus, all values>1 imply a disadvantage for young people. Note that we are using ratios and not rates (ratio is the share of unemployed among all those aged 15-24; unemployment rates refer to the share of unemployed among the active population of the same age).

(8) Source: Eurostat, Labour Force Survey (yth_empl_090)

(9) Source: Eurostat, Labour Force Survey ( yth_empl_150 )

(10)  Source: Eurostat, EU-SILC ( ilc_peps01 )

(11) Source: Eurostat, Labour Force Survey (yth_empl_150)

(12) Article 22(3) of the ESF Regulation.

(13) Excludes ESF allocations to the Youth Employment Initiative.

(14) Includes ESF allocations to the Youth Employment Initiative.

(15)  One exception is Austria that received a recommendation on improving the education outcomes / achievements / basic skills, in particular of young people with a migrant background, for four years in a row.

(16)      An ‘operation’ is defined in the Common Provisions Regulation (Article 2, Regulation (EU) No 1303/2013) as ‘a project, contract, action or group of projects selected by the managing authorities of the programmes concerned, or under their responsibility, that contributes to the objectives of a priority or priorities; in the context of financial instruments, an operation is constituted by the financial contributions from a programme to financial instruments and the subsequent financial support provided by those financial instruments’.

(17)  This number denotes the number of participations in all operations, which is not the same as individual participants, as an individual can participate in several operations over the course of the programming period.

(18) Taking into account the N+3 rule for spending and declaring investments, the last eligible declarations for YEI support were initially planned for 2018.

(19)   https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8307&preview=cHJldkVtcGxQb3J0YWwhMjAxMjAyMTVwcmV2aWV3

(20) Belgium, Germany, Greece, Malta, Portugal, Spain. France, Italy, Poland, Slovakia.

(21) As indicated in the Commission Better Regulation guidelines , the data gathered through public consultations does not provide a representative view of the EU population.

(22)

 Labour Market Policy statistics are one of the data sources for monitoring the Employment Guidelines. The scope of LMP statistics is limited to public interventions that explicitly target groups of people with difficulties in the labour market. Data on public expenditure and participants (stock and flows) are collected annually from administrative sources. Quantitative data on expenditure and participants are complemented by a set of qualitative reports that describe each intervention.

(23) There are two main factors to consider. First, Member States could begin implementing the YEI as early as September 2013. Second, in 2015, an amendment to the ESF Regulation substantially raised the initial pre-financing amount paid to operational programmes (from 1% to 30%), to help countries facing budget constraints to start implementing the Initiative.

(24)  Ex ante conditionality means a specific and precisely pre-defined critical factor, which is a prerequisite for and has a direct and genuine link to, and direct impact on, the effective and efficient achievement of a specific objective for an investment priority or an EU priority.

(25) Regulation (EU) No 1303/2013 of the European Parliament and of the Council and Regulation (EU) No 1304/2013 of the European Parliament and of the Council.

(26) Due to the extension of the implementation period, most Member States substantially lowered their targets for 2018. After the adjusted target following the 2017 reprogramming exercise, the EU average milestone for 2018 was set at 61% of the revised YEI budgets (or 82% of the allocated YEI budget before the budget increase), instead of 100% as set at the start of the programming period.

(27) The figures for success rate are the share of each type of result compared with their target population (i.e. first three columns are the share of unemployed that achieved each specified outcome, the next three columns give the share of long-term unemployed that achieved each specified outcome).

(28)  According to Definitions for ESF common indicators (Annex C1 to Commission Guidance document on monitoring and evaluation of ESF) "inactive" are persons currently not part of the labour force (in the sense that they are not employed or unemployed according to the definitions provided).

(29) 'Youth worker' means a professional or a volunteer involved in non-formal learning who supports young people in their personal socio-educational and professional development

(30) Cluster C and D regions – see section 4.1.

(31) Cluster A and B regions – see section 4.1.

(32) The data relate to flows of funding to the initial population of young people not in employment, education or training. Since people joining this category after the initial stock was recorded could still receive support in this funding period, a perfect coverage of the whole NEET population would produce results above 100%.

(33) This cluster comprises six less developed regions (in Bulgaria, Spain, France, Italy, Hungary and Romania) and five transitional regions (in Belgium, Spain, Greece, France - overseas, and Italy).

(34)  The modelling work was carried out before COVID-19 pandemic.

(35) Developing ‘off-the-shelf’ Simplified Cost Options (SCO) under Article 14.1 if the European Social Fund (ESF) Regulation, PPMI for DG EMPL, April 2018.

(36) A key task of the abovementioned Commission study is to develop EU-level off-the-shelf solutions for funding transnational mobility of young people not in employment, education or training (NEETs). The development of EU-level SCOs will be based on the current funding arrangements under TLN Mobility network and on the collection and analysis of administrative and monitoring data from countries and regions that have funded such mobility programmes under ESF operational programmes.

(37) Students are by definition inactive, so when we find high shares of inactive young people in a programme it may indicate a high share of students. More broadly, by definition YEI supports people with some degree of distance from the labour market. Being low skilled or being inactive does not necessarily mean these participants are at a greater distance from the labour market. In addition, basic skill training and job guidance is arguably cheaper than specialised training for the high skilled. It is not uncommon to offer relatively low-cost guidance to inactive participants, to get them ‘in the system’.

(38) General Data Protection Regulation.

(39) European Commission (2018) study on the coordination and harmonisation of ESI funds and other EU instruments.

(40)  https://ec.europa.eu/social/main.jsp?langId=en&catId=89&newsId=9719&furtherNews=yes

(41) Recital (11) of Regulation 1304/2013.

(42) See Duarte, N., Geraci, A., Granato, S., Mazzarella, G, Mortágua, M. “The evaluation of the Youth Employment Initiative in Portugal using Counterfactual Impact Evaluation methods”, JRC Technical Report nr. 120942.

(43) https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/1863-Evaluation-of-the-support-to-youth-employment-by-the-Youth-Employment-Initiative-and-the-European-Social-Fund.

(44) Feedback on the roadmap: https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/1863-Evaluation-of-the-support-to-youth-employment-by-the-Youth-Employment-Initiative-and-the-European-Social-Fund/feedback?p_id=273121.

(45)

 A z-score can be calculated from the following formula: z = (X - μ) / σ, where z is the z-score, X is the value of the element, μ is the population mean, and σ is the standard deviation. A z-score less than 0 represents an element less than the mean; A z-score greater than 0 represents an element greater than the mean; A z-score equal to 0 represents an element equal to the mean; A z-score equal to 1 represents an element that is 1 standard deviation greater than the mean; a z-score equal to 2, 2 standard deviations greater than the mean; etc. A z-score equal to -1 represents an element that is 1 standard deviation less than the mean; a z-score equal to -2, 2 standard deviations less than the mean; etc.

(46)  K-means provides a simple way to organise the values of a given dataset into a predefined number of clusters (k clusters). It is based on defining a centroid for each cluster and regularly adjusting their positions. Starting with k centroids spaced as far away from each other as possible, each point in the dataset is associated to the nearest centroid. The centroid is then moved to the average of the points assigned to it and the process is repeated until no further changes in the position of the centroids is possible

(47) [1] The final version of the reports for Hungary and England (expected in the course of 2019) were not available yet by mid-September 2019.