EUROPEAN COMMISSION

Brussels, 31.1.2018

SWD(2018) 40 final

COMMISSION STAFF WORKING DOCUMENT

Mid-term evaluation of the Erasmus+ programme (2014-2020)

Accompanying the document

REPORT FROM THE COMMISION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF REGIONS

Mid-term evaluation of the Erasmus+ programme (2014-2020)

{COM(2018) 50 final}

Table of Contents

1.Introduction

2.Background to Erasmus+ and its predecessor Programmes

3.Implementation of Erasmus+ until 2016

4.Method of the evaluation

5.Findings

5.1.Effectiveness

5.1.1.Outcomes for learners and practinioners………………………………........20

5.1.2.Outcomes for organisations and systems……………………………….......31

5.1.3.    Transversal questions about effectiveness………………………………….40

5.2.Relevance

5.3.Coherence

5.4.Efficiency and simplification

5.5.European added value72

6.Conclusions

Annex 1: procedural information 78

Annex 2: stakeholder consultation 82

Annex 3: methods and analytical models91

Annex 4: evaluation: criteria and questions103

Annex 5a: intervention logic of Erasmus+ and its predecessor programmes107

Annex 5b: factsheet on Erasmus+113

Annex 5c: specific objectives of Erasmus+115

Annex 5d: factsheets on predecessor programmes117

Annex 5e: legal basis indicators127

Annex 6: glossary 135

1.Introduction

The Union Programme for Education, Training, Youth and Sport for the period 2014-2020 'Erasmus+', and its predecessor programmes for the years 2007-2013, support learning mobility of individuals worldwide, transnational cooperation between organisations and promote Member States' reforms in the education, training, youth and sport fields. This is how the EU invests in people with a view to unlocking individuals' potential regardless of age or background, in support of Member States' efforts to develop human resources in Europe and beyond.

Erasmus+ contributes to the objectives of the Europe 2020 Strategy 1 and more specifically to the Strategic Framework for European Cooperation in Education and Training (ET 2020) 2 , the European Youth Strategy 3 and the EU policy in the field of sport 4 . Erasmus+ also contributes to the EU's more recent overall political objectives, such as the European Pillar of Social Rights and the EU Global Strategy on Foreign and Security Policy.

The Erasmus+ programme (2014-2020) 5  has integrated all previously existing EU programmes in the domains of Education, Training, Youth and Sport, and includes an international dimension that is funded by different external action instruments. In the fields of Education, Training and Youth, the programme pursues its objectives through three types of actions: learning mobility abroad, transnational cooperation projects and policy support. Separately, the Jean Monnet activities promote teaching and research on the European integration 6 , while the Erasmus+ supports transnational cooperation activities in the field of sport, focusing in particular on grassroots level sport.

The Regulation establishing the Erasmus+ Programme stipulates that a mid-term evaluation report accompanied, if appropriate, by a legislative proposal to amend the Regulation, shall be submitted by the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions by 31 December 2017. Furthermore, the mid-term evaluation shall include the ex-post evaluation of the Erasmus+ predecessor programmes over the period 2007-2013 i.e. Lifelong Learning, Youth in Action, Erasmus Mundus, ALFA, Tempus, Edulink and sport preparatory actions, taking into account their long-term results and impact. Therefore, this report covers actions for the period 2007-2016 in all programme countries (Member States of the European Union, the Former Yugoslav Republic of Macedonia, Iceland, Liechtenstein, Norway, Turkey) and partner countries (neighbouring the European Union and other partner countries). The baseline for this evaluation is the period 2007-2013 unless otherwise specified. In total, the period under evaluation corresponds to a total budget of over EUR 15 billion.

The objectives of the mid-term evaluation are to assess five evaluation criteria a) the effectiveness of the measures taken to achieve the Erasmus+ programme's objectives, including the contribution made to the realisation of the Europe 2020 strategy; b) the continued relevance of all of its objectives; c) the Programme's internal and external coherence; d) the efficiency of the Programme and the scope for simplification; and lastly e) its European added value. The results of the evaluation will be used to improve the implementation of the current programme until 2020 where possible. They will also be used to feed into the impact assessment for its successor programme.

The mid-term evaluation assesses the performance of the different (Key) actions 7  included in Erasmus+ and evaluates to which extent findings differ across fields and sectors 8 included in the programme and across the different target levels (individuals, organisations and systems). It also assesses the extent to which the programmes have contributed to policy development and implementation in the participating countries. The evaluation does not cover actions that were discontinued during the previous programming period (2007-2013) 9 as these were not to be evaluated, under the Terms of Reference. However, feedback from stakeholders on the interruption of actions discontinued with Erasmus+ was collected and analysed.

This Staff Working Document on the mid-term evaluation of Erasmus+ and predecessor programmes has been carried out according to a Roadmap published in January 2016 10 . It draws mainly, among other sources 11 , on the National Reports submitted by the programme countries in accordance with Article 21(4) of Regulation (EU) No 1288/2013, previous evaluations and other studies 12 and the final deliverables prepared by ICF Consulting Services Ltd, (hereinafter ICF), under contract to European Commission (DG Education, Youth, Sport and Culture) 13 . The final report delivered by ICF provides answers to all evaluation questions 14 defined in the Terms of Reference and related to the aforementioned five evaluation criteria. The final report of the contractor contains also recommendations addressed to the Commission.

2.Background to Erasmus+ and its predecessor Programmes

Erasmus+ is the EU Programme in the field of education, training, youth and sport, with a budget of EUR 16.45 billion for the period 2014-2020 15 . It provides opportunities for people of all ages (university students but also school pupils, trainees, apprentices, etc.) to study, be trained, volunteer and learn in other countries. It also fosters the professional development of practitioners and supports cooperation on tangible results, networking and share of knowledge among organisation and institutions in the fields covered by the programme. 

Through cooperation in formal, informal and non-formal learning, the Programme aims to address the following challenges: economic recovery and high youth unemployment; skills' mismatches, low employability and education poverty; global competition for talents; Information and Communication Technology potential and digital divide; social exclusion and intolerance; lack of trust in the EU and low participation in democratic life; threats to the integrity of sport and, more generally, to common European values.

The general objectives 16 of Erasmus+ are to contribute to the Europe 2020 strategy for growth and jobs 17 , including the headline education targets 18 , as well as the strategic framework for European cooperation in education and training ('ET 2020'), including related benchmarks. Erasmus+ also aims to contribute to achieving the objectives of the EU Youth Strategy and the EU Work Plan for Sport, to promote the sustainable development of partner countries in the field of higher education and youth, as well as to foster European values 19 .

Specific objectives 20 tackled by the programme include the improvement of the level of key competences and skills, with particular regard to their relevance for the labour market and their contribution to a cohesive society; the promotion of solidarity and participation in democratic life in Europe and the labour market; the improvement of quality, innovation, excellence (including in European studies) and internationalisation at the level of organisations and practitioners; support to the modernisation of education and training systems, in particular through evidence-based policy cooperation; the enhancement of the European/international dimension of its sectors, including with partner countries in complementarity with the Union's external action; the promotion of the Union's linguistic diversity and intercultural awareness; cross-border threats to the integrity of sport; support to good governance in sport and dual careers of athletes; and the promotion of voluntary activities in sport.

Erasmus+ results from the integration of the following predecessor European interventions 21 implemented during the period 2007-2013: Lifelong Learning (LLP) 22 , Youth in Action (YiA), Preparatory Actions in sport (as of 2009), as well as in the sector of international higher education: Erasmus Mundus II (as of 2009, with the rest of the world), Edulink (as of 2008, with African, Caribbean and Pacific Group of States), Tempus IV (with neighbouring countries), Alfa III (with Latin America) and programmes of cooperation with certain industrialised countries.

The current Erasmus+ programme has the following architecture:

Key Action 1: Learning mobility of individuals (opportunities for students, trainees, apprentices, young people and volunteers, as well as for professors, teachers, trainers, youth workers, staff of educational institutions and civil society organisations to undertake a learning and/or professional experience in another country)

Key Action 2: Cooperation for innovation and the exchange of good practices (transnational or international projects promoting cooperation, innovation, exchange of experience and know-how between different types of organisations and institutions involved in education, training and youth or in other relevant fields)

Key Action 3: Support for policy reform (actions supporting national authorities and stakeholders in defining and implementing new and better coordinated policies in the field of education, training and youth).

Jean Monnet activities (actions aimed at improving the quality of teaching on European integration studies, as well as projects and operating grants aimed at promoting discussion, reflection on EU issues and enhancing knowledge about the EU and its functioning)

Sport (cooperation projects, events, studies and other initiatives aimed at implementing EU strategies and priorities in the field of sport)

Erasmus+ relies on management modes inherited from predecessor programmes.

The European Commission bears the overall responsibility for the supervision and coordination of the agencies in charge of implementing the Programme at national level. The European Commission manages the budget and sets priorities, targets and criteria for the Programme. Furthermore, it guides and monitors the overall implementation, and evaluates the Programme at European level after having received the National Reports from participating countries, as described under "method". It also manages directly few actions of the programme 23 .

At European level, the European Commission's Education, Audiovisual and Culture Executive Agency (EACEA) 24 is responsible for the implementation of centralised actions of the programmes 25 (direct management) which account for a small share of the total budget.

The largest share of the budget is implemented through indirect management. The European Commission entrusts implementation and promotion tasks to National Agencies (NAs) 26 established in each of the 33 countries participating to the Erasmus+ programme (named hereafter "programme countries" 27 ) which implement those actions of the programme with the highest volume 28  so as to bring the programme as close as possible to its beneficiaries and to adapt to the diversity of national education, training and youth systems.

No direct support is given to individual beneficiaries 29 ; all support is channelled through participating organisations, which distribute it to individual learners or practitioners.

2.1.Main changes in the programmes over the period under evaluation

In line with the subsidiarity principle and Articles 165 and 166 of the Treaty on the Functioning of the European Union, the EU intervention aims at supporting and supplementing the actions of Member States, while fully respecting their responsibility for the content of education, training, youth and sport policies. Meanwhile, more than its predecessors, Erasmus+ has been designed with a view to better aligning spending actions with the priorities set at EU level in each policy area.

The integrated structure of the programme is the most noticeable change in scope brought by Erasmus+. Other important developments between the two programming periods have taken the form of an increased budget allocation (+40%); a simplified structure in 3 Key Actions; streamlined ways of implementation and systematic monitoring, as evaluated under "effectiveness", "coherence" and "efficiency".

a) Erasmus+ does not only draw on the legacy of its predecessors but also brings them altogether into a single integrated programme, including its international dimension funded by the EU external action instruments (2% of all Erasmus+ student mobility activities have involved partner countries) 30 . The suppression of (sub-) programmes per field (sector) contributes to the "effectiveness", "efficiency" and "internal coherence" of the programme, as evaluated below.

b) Erasmus+ benefits from increased resources with over EUR 16.4 billion spread over seven years as opposed to a total budget of slightly more than a total of EUR 9 billion allocated to its predecessors over 2007-2013.

c) For simplification purposes, following the conclusions of the Impact Assessment for Erasmus+ 32 , the current programme is based on three cross-cutting Key Actions 33 which apply to all education and training sectors and to the youth field: Key Action 1 (KA1) supports learning mobility of individuals; Key Action 2 (KA2) fosters cooperation for innovation and the exchange of good practices, including collaborative platforms, and Key Action 3 (KA3) supports policy reforms. This integrated approach does not apply to the two stand-alone strands for sport and for European integration studies ("Jean Monnet"). The fact that the nature of the funded activities has not fundamentally changed with Erasmus+, as reflected below, makes it possible, however, to draw conclusions about the likely impact of the current programme based on what can be observed ex post from the impact of its predecessors.

Main type of actions per Erasmus+ Key Actions

In bold: most significant novelties of Erasmus+

d) The integrated nature of the programme means that cross-field approaches or tools can be used for the purpose of grant application, monitoring, audit or dissemination.

This has materialised through on-line application, single audit system, generalisation of simplified grants (lump sums, unit costs), performance-based monitoring system, a single Project results platform, etc. In designing Erasmus+, specific attention has been paid to make it more result-oriented than its predecessors and more agile to adapt to policy evolutions notably in the light of the socio-economic challenges mentioned above.

2.2.Intervention logic

The EU mandate in the education, training, youth and sport areas sets the scope for the intervention logic 34 . An overarching intervention logic was designed to cater for all the programmes in scope of the evaluation, irrespective of the programming period. This approach was followed in order to assess and compare more swiftly results across programmes in order to make a more comprehensive assessment of the impacts achieved during the baseline period. As a matter of fact, although each evaluated programme had different architecture, most of the supported actions (mobility, cooperation and policy reform support) were transversally implemented – although with a different level of intensity - under all programmes. For that purpose, the programme's specific objectives and actions have been grouped according to the level at which the results are expected 35 .

Furthermore, considering the overwhelming dominance of the external factors of Member States' policy making and spending in these areas, it is not easy to attribute and quantify the specific effects of the EU intervention.

Spill-over between intervention levels is expected, as shown in the graphic below, explaining why most of the specific objectives are expected to deliver results at more than one level. For instance, the mobility of learners and practitioners (KA1) can – in addition to individual-level results – improve the performance of the organisations and have an impact on systems, especially in terms of outcome recognition. Also, the performance of individual organisations (KA2) is expected to benefit from European cooperation in the fields of the programme, including through its modernising effects on national systems and reforms prompted by the open method of coordination and KA3.

·Systems, to improve policies
in programme and partner countries;

·Organisations, to promote cooperation
and positive changes in work methods;

·Individuals, to increase
participants’ competences
as well as change their attitudes,
practices and perceptions;

Source: Roadmap for the evaluation, January 2016

3.Implementation of Erasmus+ and predecessor programmes until 2016

In the period 2007–2016, the programmes have funded mobility for more than 4.3 million learners and more than 880,000 practitioners. Many more have benefited from short-term blended mobility and/or other forms of transnational exchanges as part of cooperation projects. In the current programming period (2014–2016) alone, at the time of the mid-term evaluation (i.e. without taking into account fully 2016 data), the programme had already benefited over 1.4 million learners and 400,000 practitioners. Approximately 67 000 projects have been contracted in the same period, which represent a reduction in yearly number by around 50% compared to 2007-2013. Erasmus+ has indeed been designed to fund fewer, but more multifaceted projects with larger average numbers of participants. Comprehensive information is available from the Erasmus+ programme – Annual Report (taking into account fully 2016 data)  36 .

The Erasmus+ programme has been fully implemented as from 2015 without major difficulties 37 . The international higher education strand and the Student Loan Guarantee Facility 38 were launched only in the second year of the programme, when arrangements were ultimately made respectively with the External Relations instruments and the European Investment Fund. The implications of the newly created European Solidarity Corps on Erasmus+, more specifically on the implementation of the European Voluntary Service under the Youth Chapter, only materialised in 2017; it is, for that reason, out of the scope of this evaluation.

Erasmus+ was allocated EUR 16.4 billion for the 2014-2020 period 39 . Despite the overall significant budget increase compared to the previous programming period (+40%), Erasmus+ has only experienced a limited budget increase in 2014-2016 according to the programme's budgetary profile which concentrates the funding increase over 2017-2020 40 , as shown below. It is therefore too early at mid-term to evaluate the full impact of the additional resources allocated to the current programme.

Source: Erasmus+ activity report 2015

Distribution of Erasmus+ budget across sectors (2014-2016)

Source: ICF calculations based on EplusLink

The most substantial part of the Erasmus+ budget is allocated to education and training and, within this field, to higher education, as shown in the graph above 41 . In terms of actions, mobility has the largest share of the budget, as displayed below.

Key data, 2014 - 2016

Source: Erasmus+ Programme Annual Reports 2014-2016 (taking into account fully 2016 data)

*Amounts not only for project grants but including also operating grants

4.Method of the evaluation

Considering the high quality of data collected through several complementary techniques from various sources, the sophisticated analysis and triangulation of evidence, the positive opinion of the Regulatory Scrutiny Board and the improvement stemming from its comments, as well as the significant feedback received (with more than a million responses processed in total), this evaluation can be considered very reliable and valid.

4.1.Evaluation framework

To undertake the combined mid-term evaluation of Erasmus+ and the final evaluation of its predecessor programmes, as outlined above in section 1, the Commission has drawn upon a set of evaluation questions relating to five main criteria: effectiveness, relevance, coherence, efficiency and EU added value, as per the evaluation Roadmap 42 .

To ensure comparable approaches, the evaluation framework developed by the external evaluator 43 classifies actions across the three levels of intervention of the programmes and uses that as the basis of comparison over 10 years, across all sectors and various target groups, namely:

·at System level

·at Organisation level

·at Individual level, divided into actions for learners and practitioners across all different areas (education and training, youth and sport) and – within education and training – different sectors (school education (SE), vocational education and training (VET), higher education (HE) and adult education (AE)). For the purpose of the evaluation, the overarching terms of Learners and Practitioners are used 44 .

In addition, the analysis of effectiveness has been based on the intervention logic represented graphically in annex 5a accompanied by a detailed elaboration of expected outcomes (results directly for individual beneficiaries; impact indirectly for a wider population, participating organisations or education, training, youth and sport systems) 45 . All specific objectives of the programme (annex 5c) are reflected in the outcomes or impacts of the intervention logic.

The purpose of the evaluation is to evaluate the outcome of the programme on target groups, not to audit each type of activities. Therefore this intervention logic is a powerful simplified model to develop questionnaires based on the most relevant levels of outcomes. It is worth noting that these three levels of intervention are not equivalent to the three Key Actions, as explained above 46 . A given Key Action can indeed have an effect on more than one level of intervention (see 2.2). Key Actions have been set within Erasmus+ but these three levels of interventions apply to both programming periods 47 .

4.2.Data collection and analysis

Evaluation findings have been based on a wide range of data sources systematically triangulated i.e. cross-checked against each other: programme documentation, programme data and views of the managing bodies, views of direct beneficiaries as well as non-beneficiaries, EU and national stakeholders, policy makers, programme countries, bodies in charge of implementation of other comparable programmes and the general public 49 .

The evaluation combined a number of techniques for data collection and analysis. Most results are reached by a mix of evidence combining a quasi-counterfactual assessment based on beneficiaries’ surveys (i.e. comparisons between beneficiaries and a control group), self-reported feedback of beneficiaries (i.e. monitoring surveys), as well as qualitative interviews and case studies. While monitoring surveys could only give findings as from 2014, evaluation surveys covered 2007-2016 and enabled the evaluators to compare the two programme generations. Monitoring data have been fully used and supplemented with surveys carried out by the external evaluator, providing more objective findings (with questions not based on self-perception), more comprehensive data (covering as well the 2007-2013 period and KA2 participants) and more comparable responses (as questions were harmonised across sectors and periods for comparison purpose).

Collection tools presented below covered all programme countries, sectors and main types of actions (except where mentioned otherwise). The quality of the one million responses processed was in most cases exceptionally good. All quantitative data was statistically processed and presented using visualisation tools, while all qualitative data was systematically coded.

As required by the Erasmus+ Regulation, programme countries have submitted their own evaluation reports on the implementation and impact of decentralised actions of Erasmus+ in their respective territories. Their findings are summarised in the National reports synthesis (NRS) annexed to the external evaluator's report 50 .

In line with Better Regulation rules, an Open Public Consultation (OPC) gathered the opinions of the general public and all interested groups on all evaluation criteria. 51 Different questions were asked depending on the level and scope of knowledge of the programmes declared by each respondent. Consultation findings are presented separately in the synopsis drawn by the external evaluator 52 . More widely, responses from all sources are presented in Annex 2.

Box 1 provides an overview of the data sources from which the evidence was drawn. A detailed description of the individual methods is provided in Annex 3.

4.3.Method strengths and limitations

Overall, the quality of available data for the predecessor programmes was good, while for Erasmus+ it was very good. This reflects improvements made in terms of monitoring since 2014. The evaluation used the wealth of the programme data available from the Erasmus+ management or monitoring tools, including some data on results (e.g. satisfaction-rate) since 2014. Although the period within the scope of the evaluation was 2007-2016, programme data was not fully available for 2007 53 .

The main issues that affected the analysis of participation patterns include the following:

i) the absence of a harmonised definition, across sectors, of people with fewer opportunities and from disadvantaged backgrounds (programme data were completed with other findings) 54 ;

ii) the lack of a complete picture on the type of participating organisations (some 40% of organisations were classified as "other" in the programme dataset limiting the analysis of participation per type of organisation). 

It should also be noted that the analysis over time of networking among participating organisations or repeated participation of individuals in the programme could not be fully pursued, as prior to 2014 the same organisations could be recorded under multiple names, which made such an analysis difficult to execute. These limitations have however remained marginal or have been overcome through the use of other sources.

The monitoring surveys of beneficiaries carried out by the Commission concern all learners or practitioners taking part in mobility under KA1 (since 2014 only) 55 . All are requested to fill in an online survey on completion of their mobility. Therefore the reliability of the data is strong. Given the sample sizes (first two years of the programme) the data is considered to be strongly reliable even though it was too early to include 2016 data. This being said the link between the monitoring questions and the indicators they are supposed to measure can be weak in some cases. Above all, most questions are based on the self-perception of beneficiaries in contrast to more objective measurements such as the assessment of language proficiency after mobility (OLS 56 ). This is why monitoring data were supplemented with primary data collection, such as surveys carried out by the external evaluator, providing more objective and comparable responses.

Series of surveys of beneficiary learners and staff and related control groups covered all programme target groups and predecessor programmes as well (2008-2016 in total), both for mobility and cooperation, over 2007-2016. Overall, findings could be generalised to the whole programme with a sufficient degree of confidence because of the sample sizes of all surveys and the distribution of various background variables within the survey samples. Given the limited data available on the background of beneficiaries, it nevertheless remains impossible to quantify the degree of comparability of the samples to the programme population 57 .

A partially counterfactual approach ("quasi-experimental") was also used to quantify the contribution of the programme to the results measured. This was done by assessing the difference in results between Erasmus+ beneficiaries and their "peers" who did not take part. The challenge is to attribute the difference in outcomes between both groups to the programme isolating any other underlying factor (attribution). Control groups of non-beneficiaries were therefore set up. In an ideal world, these should be selected on random basis. However, a random allocation is neither feasible in practical terms nor desirable for ethical reasons. By default, a ‘matched sample’ was sought where individuals are similar in some background characteristics such as gender, age, etc. There were no significant differences between the control group and the treatment group for most variables, and where there were, they concerned variables that were not likely to strongly influence the findings. Most control groups reached a significant size allowing comparison with beneficiaries, except in sport and Adult Education sectors where the control groups were too small. As a consequence, the control group of VET staff was also used for Adult Education and the control group of Youth staff was also used for sport, as these were found to be sufficiently comparable against a range of background characteristics. However this counterfactual approach cannot exclude the possibility that the difference in a given variable between beneficiaries and non-beneficiaries could be partially due to the selection into the programme rather than the effect of the programme.

In addition to the above ‘post’ surveys (i.e. beneficiaries were surveyed after they took part in the programme), the external evaluator carried out pre-surveys in two sectors where short term mobility exchanges apply (i.e. school pupils and Youth exchanges). This was done as far as possible within the time frame of the evaluation assignment for the purpose of pre/post comparison. In spite of repeated recruitment efforts, the sample sizes suffered from a high attrition from the pre to the post survey, especially for the Youth sector where comparison of profiles prior to Erasmus+ mobility could not be done. Pre-post results for the Youth sector have therefore not been interpreted considering the high likelihood of sample bias. This allowed nevertheless for the school sector to draw conclusions on selection into the programme and to observe changes during the time of the mobility experience.

To get a better understanding of their quality and dissemination potential, a review of selected project outputs, mainly collected from National Agencies, was carried out. Given that only relatively few Erasmus+ funded projects were completed at the time of this evaluation, the sample remained inevitably somewhat biased towards predecessor programmes.

As for the Open Public Consultation (OPC), a large number of partial responses only contained very basic background information on respondents and had therefore to be marked as unsuitable for analysis. In any case, in contrast to surveys, public consultation findings can never be representative due to the selection bias inherent to any open recruitment.

Lastly, some programme countries have delivered their national reports with a four month delay. This, in turn, has delayed the adoption of the Commission report to take into account all programme countries' views. The period of time between the deadlines set by the legislator respectively for the delivery of the national reports and the Commission report was reduced from 9 months (LLP) to 6 months (Erasmus+), which has not allowed the external evaluator to make a full synthesis of all findings 58 .

4.4.Judgement on the validity and reliability of the findings

The external evaluator (ICF) contracted for this assignment 59 has carried out since May 2016 all tasks as required under the scrutiny of an inter-service group (ISG) 60 and the daily steer of DG Education, Youth, Sport and Culture (EAC). The only significant change compared to the initial work plan was a two-month delay of the (sub)contractor in launching the beneficiary surveys 61 . The ISG was consulted at each stage of the evaluation process and reviewed each deliverable produced by the contractor as well as this Staff Working Document 62 .

The Regulatory Scrutiny Board, responsible for the independent quality control of this evaluation, acknowledged in its positive opinion with comments the significant efforts of data and evidence collection and noted the good methodology used 63 . This document has been improved following its comments.

Based on the above elements of the evaluation method and as further described in Annex 3, the reliability and validity of the evaluation can be regarded as strong 64 .

Source: ICF, 3.5; see section above for assessment grounds

5.Findings

5.1.Effectiveness

The assessment of the effectiveness 65 of Erasmus+ is overall very positive, especially when compared to the situation before the integration of predecessor programmes 66 . It has been informed by 15 evaluation questions that looked into the extent to which:

·outcomes (i.e. outputs, results and impacts) were delivered with particular attention to disadvantaged groups

·spill-over, sustainable or unintended effects took place

·certain fields or actions were more effective than others (including discontinued actions)

·results were disseminated and exploited.

The programme reaches or exceeds the vast majority of its indicator targets set in the legal basis 67 , except in the case of the Student Loan Guarantee Facility (SLGF) which is missing the target by a high margin, as detailed below 68 . However, some output targets might have been set at a low level initially (sometimes below predecessor programme levels). The quality of the data on participation of disadvantaged groups although on target is unclear due to various definitions and practices (see below). Lastly, for Jean Monnet activities there is no exact data on numbers of students reached and this indicator is likely to be very overestimated. These limitations concern only a few indicators and the vast majority are without doubt on target.

Incoming mobility of students from higher education international strand is also below target but the action started in 2015 only and the overall trend over ten years is positive. Evaluated programmes have been important for the EU's global outreach, as elaborated hereafter 69 .

The Open Public Consultation confirms that the programme is achieving its objectives to a (very) large extent. OPC respondents highlighted two elements as particularly helpful in terms of effectiveness: the new integrated structure of the programme 70 and the overall increase in funding 71 . One sustainable result of the programme and its predecessors is the cultural shift in the perception of mobility and its positive image, at least in the higher education sector and progressively in VET. These long-term changes are yet to fully materialise in the other sectors.

5.1.1.Outcomes for learners and practitioners 

Compared to the last three years of predecessor programmes, Erasmus+ has supported during the first two years 72 a much higher volume of mobility within and outside the EU in higher education (the sector that has most beneficiaries overall), but also a higher number of practitioners in VET and schools 73 . The repeated participation of individuals is kept reasonably low, indicating a continuous impact on renewed cohorts. In contrast, as intended with the design of Erasmus+ 74 there has been a significant decrease in mobility for pupils - due to the fact that individual mobility of pupils was not embedded in the programme as such any more - and a slight decrease for VET learners 75 .

5.1.1.1. Outcomes for learners

Beyond contrasted evolutions in outputs across sectors, drawing on a variety of sources 76 , the evaluation found positive and sustainable results for all individual learners taking part in mobility 77 . Firstly, monitoring surveys outline that most learners themselves are satisfied with their mobility experience 78 . The rate of formal recognition of participation in Erasmus+ is high but, more importantly for those concerned; the rate of recognition of learning outcomes is also rather high 79 . The vast majority agree that the programme helped them improve more than 6 competences out of a list of 9 (see fig.) 80 . To be noted, higher education students are comparatively least positive in their self-assessment 81 .

Share of learners who (strongly) agree that they developed transversal competence

Source: ICF calculations based on Erasmus+ participant monitoring surveys. Multiple answer question. N = 730,254

In line with the intervention logic 82 , all programmes under review are expected to contribute to changes in learners’ skills and competences as well as attitudes. In the longer term, they are expected to result in better completion rates and more positive employment outcomes. Both have materialised as detailed below. As far as the general indicators of Erasmus+ are concerned, trends are also positive and the Education and Training Monitor 2017 highlights a further decrease in the rate of early school leavers, and a tertiary attainment target that is within reach 83 .

Most result indicators set in the legal basis of Erasmus+ are on or above target 84 . The evaluation found, using a variety of sources, clear evidence of results for mobile learners in: foreign language skills development, especially for those with low entry levels; feeling of belonging to the EU; willingness to be mobile and to work abroad; shorter transition from education to work 85 ; positive perception of the value of education and positive feeling towards the sending organisation which are key precursors for retention and attainment in education and training. As evidenced for those benefiting from KA1 mobility, the programme is associated with stronger rate of completion of studies 86 .

However the result indicators set in the programme legal basis and related targets set in DG EAC's strategic plan have some limitations in that they are mostly based on the self-perceived contribution of KA1 reported by beneficiaries. Some of the targets were exceeded and others not reached (e.g. participants declaring that they have increased their language skills; young participants declaring being better prepared to participate in social and political life). However, there are concerns about the plausibility of the target values in both cases 87 .

The evaluation does not permit isolating how many of these results are the direct effect of the selection of the most motivated into the programme. For instance, Jean Monnet students do show better results than other higher education students when it comes to knowledge of the EU. This however can be expected given that the programme reaches mainly those who have shown an interest in the EU topics 88 .

The most successful objectives of the Programme, which are also the most relevant, according to OPC respondents, have been a) developing the skills and competences of learners; and b) promoting the European dimension of education, training and youth activities 89 . Other studies also confirm this in the relevant sectors 90 . Higher education who have participated in the programme are not only more likely to be employed, but also more likely to secure management positions. On average, 64% of Erasmus students, compared to 55% of their non-mobile peers hold such positions within 5-10 years from graduation. This holds even more true for Erasmus students from Central and Eastern Europe, where around 70% of them end up in managerial jobs 91 . The same can be said for the international dimension of higher education, with clear evidence of strengthened capacities and employability of individual students and scholars who have participated in mobility 92 .

Albeit mostly based on the self-reported perceptions of participants, most National Reports refer also to improvement of the level of key competences and skills of learners with a high level of similarity with the ones listed above. On top of technical/professional knowledge and skills 93 , the impact of Erasmus+ on employability is underlined by several countries mostly in the case of HE and VET 94 , transversal skills 95 or soft skills are also frequently identified. These include communication skills 96 , social and civic skills 97 , and more widely personal development 98 .

The case studies confirmed the strongly positive influence the programme plays on personal development and maturity of young people. Both sources frequently outline that learners gain in confidence, independence, ability to cope with new environments and open-mindedness through their mobility. Participation in the programme also leads to development of learners’ social capital 99 . In contrast to very positive effects of all forms of mobility, there is less clear evidence of impact of transnational partnerships on learners (excluding their mobility component) except in the school sector 100 . For example, a survey conducted as part of the eTwinning 101 evaluation, indicated that 55% of teachers believe that the platform is helpful in motivating their pupils 102 .

At programme level, the areas where learner surveys have identified strongest positive difference between the results of participants and non-participants 103 include: willingness to move abroad permanently (+31% likeliness compared to non-participants); stronger feeling of being an EU citizen (+19%); shorter transition to employment (+13%) 104 ; value attached to education (+8%) and also openness to immigration and minorities (+2.6%). The figure below gives an overview of all areas evaluated according to the degree of difference found 105 .

Comparison of results for learner beneficiaries and control group

Source: ICF Beneficiary surveys: only areas where the difference is statistically significant i.e. unlikely due to a sampling interference

The share of beneficiaries who took less than three months to find a job stood at 68.5%, while the percentage of the overall control group was at 59.2%. This trend mirrors the positive expectation of most beneficiaries towards Erasmus+ 106 . This is particularly strong for the higher education 107 and VET beneficiaries, in contrast with the youth sector 108 . Erasmus+ also leaves a positive effect on entrepreneurship 109 . Overall, this confirms the findings of several previous studies about the effects of learners’ mobility on employability 110 .

The pre/post analysis for pupils confirms the positive contribution of Erasmus+ in five similar result areas, the order of which appears nevertheless specific to the school sector. There are statistically significant differences in results across pre- and post-survey of school pupils in EU citizenship (+7 score points), digital competence (+4), racism and xenophobia (+3), self-confidence in education (+4), civic participation and volunteering (negative difference).

However, there is likely selection of already more performant pupils into the programme at the level of schools, since the pre-post survey also shows already a significant difference between participants and the control group at entry into the programme. Pupils participating in short-term mobility speak a foreign language more often than the control group (+13 percentage points or p.p.) and feel more often as EU citizens already before going on mobility (+9 p.p.). Cultural awareness (+7 points in index score or p.i.s.), positive attitudes towards immigration and minorities (+6 p.i.s.), digital competence (+5 p.i.s.) and perceptions of the value of their education (+4 p.i.s.) were also higher. This is confirmed by the case studies where some respondents claim that participation is seen as a certain form of reward.

Foreign language skills development is more important in those sectors where learners tend to have lower entry levels (i.e. in VET and youth more than in higher education). The analysis of OLS foreign language tests (pre/post mobility) proved that the weaker the proficiency entry level, the stronger the linguistic improvement due to mobility 111 .

Comparison of levels of proficiency before and after mobility, by sector

Legend: share of learners in each sector at a given level of proficiency; blue: before mobility (first language assessment), red: at the end of the mobility (final language assessment)

Source: ICF calculations based on OLS data as of 31/08/2016. Higher education: n= 500,410; VET: n=19,441; EVS: n=3,387

Beneficiaries were asked in case studies to describe the most important contribution of the programme. The word-cloud below visualises the most cited testimonies.

What do you see as the most important contribution of the programme for yourself?

Source: ICF case studies –interviews with learners: 200 mentions or expressions were collected; the size of the font captures the frequency with which a given result was mentioned.

When asked about reasons hindering the achievement of programme objectives 112 , respondents identified the following barriers during the public consultation, in descending order of frequency: linguistic barriers; financial difficulties and complex application process 113 . The factors that enable positive results at the level of individual learners are related to their motivation to participate as well as the quality of the learning experience including foreign language learning support, the support received in the host organisation and their integration among other students and local communities 114 . Lastly, levels of recognition of learning outcomes vary according to types of actions 115 , but signing a learning agreement remains a strong predictive indicator for later recognition. Only a small share of learners does not sign one 116 .

Although financial barriers are a major obstacle to mobility 117 , the Student Loan Guarantee Facility, which was an innovation in Erasmus+, has so far failed to attract financial intermediaries in sufficient numbers 118 – especially for the incoming student segment 119 . First beneficiaries are, however, satisfied. The Student Loan Guarantee Facility already shows signs of social fairness 120 , but its visibility at this early stage is far from being sufficient throughout the supply chain 121 .

5.1.1.2. Outcomes for disadvantaged learners 

The current programme is rightly perceived as paying more attention to the participation of disadvantaged people 122 . To respond to the criticisms regarding the predecessor programmes – notably in the field of higher education – of being elite programmes, Erasmus+ has put in place actions to strengthen the participation of disadvantaged groups compared to predecessor programmes 123 . In the period under review, 11.5% of participants under KA1 had special needs or came from a disadvantaged background. This share rose to 30% for learners in the Youth field between 2014 and 2016 124 . The number of participants with special needs or fewer opportunities has more than doubled since the predecessor programmes, i.e. more than proportionally compared to the overall budget increase. The indicator targets have even been exceeded 125 , though data reliability is weak 126 . This is why some countries call for more consistent definitions and rules for disadvantaged participants 127 .

According to case studies, the programme reaches out to those who are easier to reach among the disadvantaged, and not to those who are disengaged or at risk of marginalisation. This is not specific to Erasmus+ as other benchmarked programmes struggle with the same challenge. This is partly due to the fact that the participating organisations, even under an action as inclusive as the European Voluntary Service (EVS) 128 , tend to select the most motivated and performing individuals. In spite of the specific measures mentioned above to foster social inclusiveness, the evaluation found that more specific support and adequate funding would be needed 129 . The lack of spending flexibility limits the capacity to face unforeseen situations inherent to vulnerable groups (see Efficiency) 130 72% of respondents to the consultation indicated that there has not been sufficient progress in this area under the current Programme and that this aspect deserves more attention and funding 131 . This was also flagged by the majority of countries in their National Reports 132 .

To mitigate the limitation of the related programme indicator mentioned above, beneficiaries were personally asked about their social background 133 . Evaluation survey responses indicate that the self-reported participation of people from a minority background or with learning difficulties is rather small. Yet, responses highlight that disadvantaged learners show more positive results than the others, in particular in completing formal learning and in boosting their self-confidence in their education capacities. However, no significant difference was noted for them in terms of benefiting from shorter transition periods to employment following their mobility experience.

These survey results are statistically significant 134 , but the pre/post surveys of school pupils and the case studies indicate that they are likely due to selection into the programme, at least for a part 135 and cannot be confirmed with strong evidence from data collected or the literature reviewed 136 .

Most interviewees in case studies believe in the potential of the programme to provide powerful results for disadvantaged groups. Several of them were clear that more than other groups, disadvantaged learners would not have gone abroad to study had it not been for the programme. Meanwhile, the participation of organisations with high shares of disadvantaged learners remains low. The case studies suggest that specific top-up points for organisations with high share of disadvantaged people should not have been discontinued in the current programme and that accompanying disadvantaged learners (e.g. in European Voluntary Service) requires a particular care and specific measures.

On the external dimension, the picture is somewhat similar. All the predecessor programmes promoted inclusiveness in higher education from a gender perspective. Some promoted other under-represented, disadvantaged groups. However their specific evaluation noted that the mobility programmes only partly succeeded in achieving equitable participation 137 .

Beyond mobility, the other types of action have also inspired new ways of working with people from a disadvantaged background, especially in sectors other than higher education 138 . This was reported by 44% of practitioner respondents 139 , although only 19% stated that the activity they took part focused on integration of disadvantaged groups. This implies that there are significant spill over effects even where it is not the main purpose to target these groups (i.e. impacts beyond results).

Lastly, certain actions perform better in terms of social inclusiveness. Studies in the Youth sector found that international youth projects or mobility 140 have had significantly more impact on young people with fewer opportunities, especially in learning to learn, developing cultural awareness and building self-confidence (+3.6%), likely due to the lower starting points 141 . The collaborative platform eTwinning seem slightly more able to reach disadvantaged groups than the rest of the programme 142 . Some of the adult education actions are succeeding in engaging with hard to reach groups, but given the low budget share the scale of these actions have only been small.

5.1.1.3. Outcomes for practitioners

Within Erasmus+, practitioner mobility has overall seen a significant increase compared to the past 143 . It remains more common in higher education than in all other education and training sectors put together, although higher education practitioners have comparatively more opportunities abroad through other schemes. It should be noted that an important share of practitioners in the adult education sector teach primarily young people (aged below 24) 144 . This could also be related to the limited effectiveness of the programme in reaching out to practitioners who are not employed in large organisations.

In addition, collaborative platforms expand participation in the programme (eTwinning, School Education Gateway 145 , EPALE 146 ) and are associated with positive results for beneficiary practitioners 147 . In particular eTwinning, the platform for schools, attracts a very large number of teachers and pupils, in the EU and beyond, as mentioned previously. In 11 countries, eTwinning seems to be reaching to more than 10% of all school teachers 148 . As a stepping stone into the programme, as noted by some interviewees, it does reach to a broader audience than other types of action 149 . After stagnation in 2011-2013, its audience has been sharply increasing under Erasmus+, pointing at a strong potential of this kind of intervention for the future.

Practitioners are even more satisfied with their mobility experience (98% compared with 93% for learners) and very positive about the contribution to their competence development according to the monitoring surveys. For most of the competences the differences across sector are minor 150 .

Practitioners who (strongly) agree that they developed a given key competence or skill

Source: ICF calculations based on Erasmus+ participant monitoring surveys. This is a multiple answer question therefore the data does not add to 100%. N = 227,319

Most National Reports refer to results for practitioners with a high level of similarity with the benefits listed above, in particular for education and training staff: language skills 151 , new teaching methods 152 and job satisfaction 153 ; as well as for youth organisations’ staff: professional development 154 and new professional methods 155 .

Evaluation surveys found in addition, when comparing with a control group, that participation in the programme is associated with wider networking and cooperation, stronger attachment to Europe, and greater use of digital resources, as shown below 156 :

Main counterfactual results for practitioners

Source: ICF beneficiary survey for practitioners

From virtual collaboration on Erasmus+ funded platforms, beneficiary practitioners also observe positive results, though admittedly with smaller intensity than in the case of mobility or partnerships. Even though it is difficult to differentiate its contribution from other actions 157 , in certain areas practitioners associate eTwinning with more positive results than the control group, such as international networking, but also digital skills (the collaborative platform might attract practitioners that are already digitally skilled, but can also enhance such skills). Findings of other sources support the positive results from ICF’s general surveys: another distinct evaluation of eTwinning revealed that a vast majority of teachers find the platform helpful to network across Europe (64%) or to improve relationships between teachers and learners (62%) 158 .

More generally, all practitioners interviewed cited positive results for themselves as well as their learners and organisations 159 . They found their Erasmus+ experience as an opportunity to go ‘outside of the ordinary’ which can be important for retention in the teaching profession, a crucial issue in many countries 160 . As practitioners appear mostly motivated by individual improvement, it is not surprising to find a clearer contribution to development of staff than hard evidence of evolution of pedagogical or institutional practices. Other studies point to positive impact on teaching practices, social competences and motivation for professional cooperation 161 .

Recognition is meanwhile less formalised for practitioners than for learners by nature. Only around half of the practitioner survey respondents recorded having received some form of recognition 162 . While a majority of practitioners noted informal recognition by peers, effective recognition by the hierarchy is less common. At the same time, nearly half of respondents state that participation in the programme helped them achieve new roles or positions, which is very positive.

It should also be noted that several of the results discussed in this section also have a spill-over effect from individual level to organisational level, as elaborated below.

5.1.2.Outcomes for organisations and systems

As for mobility (KA1), the evaluated programmes reached or exceeded the vast majority of the targets when it comes to the number of projects funded (KA2 and KA3). Compared to predecessor programmes, this represents a strong increase in the number of higher education projects 163 , but also a strong decrease in all other sectors. This is due to the lower budget available at the start of the programme but also to the fact that Erasmus+ aims to focus on funding fewer large-scale projects 164 . Erasmus+ is nonetheless perceived as providing more opportunities for cooperation than it was previously possible 165 .

However, the fact that Erasmus+ and its predecessor programmes have produced a high volume of intellectual outputs is not necessarily an indicator of project quality 166 . The panel experts have pointed out several issues with the selection process. They consider that the quality of applications selected is in general quite average. According to monitoring data, applications are frequently scored higher than the final outputs. Many projects (other than KA2 Alliances) lack a continuous quality review, external support, or solid needs analysis. National Reports refer to some enhancing-effectiveness factors (e.g. experienced staff on the applicants' side - a condition more often met by large organisations and therefore in higher education, and good cooperation between partners); they also outline challenges which hinder effectiveness (e.g. high-quality projects rejected due to lack of funding).

On the contrary, the evaluators find that fewer outputs with clearer added value might possibly meet more clearly the innovation objective of the programme. According to the qualitative assessment of outputs carried out by the expert panel, an area for further improvement of Erasmus+ is the quality of KA2 (and to a more limited extent KA3 which cannot be fully evaluated at mid-term) outputs and especially their effective potential for mainstreaming beyond participating organisations. Although the programme often produces something new for participating organisations 167 , this innovation does not enough follow the state of the art developments in a given sector. While the programme does fund good and innovative activities 168 , these are relatively hidden among many standard projects. The vast majority of case studies identified some form of innovation or change in the organisations reviewed (e.g. use of materials or methods shared during the projects) 169 . Yet often these innovations remain small scale or light touch and limited to a specific part of the organisation. For many funded projects, it appears that the main benefits arise from the participation process rather than the output itself. However, where there are instances of high quality outputs, these could be better supported upstream and thereafter mainstreamed building upon the appreciated precedent of the 2016 specific call in the area of social inclusion 170 .

5.1.2.1. Outcomes for organisations 171

Over 2007-2016, more than 940,000 organisations were contracted, at an average of approximately 115,000 organisations per year for the predecessor programmes but 80,000 under Erasmus+ 172 . During the public consultation, 74% of respondents thought that the Programme improves the quality, innovation and internationalisation in education, training and youth organisations 173 . When asked in the monitoring survey about expected changes to their institutions, more than 55% (up to 85%) of practitioner respondents stated that most such changes (10 out of 11) had indeed taken or were taking place 174 . Later, when asked in the evaluation survey about changes that had happened after their participation 175 , a vast majority of practitioners once again agreed with a high number of types of changes listed, such as the introduction of new teaching materials 176 or new assessment methods 177 . In the youth sector, 75% of practitioners agreed with most statements about organisational results. Lastly, most of other socio-economic actors (companies, public authorities, civil society bodies) stated that their objectives were (fully) met 178 . For them, the most commonly cited barriers concern the complexity of the application procedure and the low success rates in applications, which is due to budget availabilities 179 .

Furthermore, the level of cross-sectoral cooperation increased under the current programme by 23 percentage points 180 . Paradoxically, interviewees are often sceptical about the reality of this key evolution. This contrasts with OPC or agency respondents, whom, to a very high extent, recognise the effectiveness of cross-sector cooperation 181 . In the same vein, the network analysis shows promising results in the form of an increase in the number of sectors participating per project. There are notably many more higher education institutions taking part in other strands of the programme than in the past. Cross-sector participation seems more challenging in other education and training sectors where organisations are smaller in average. Reciprocally, youth organisations strongly contribute to the international dimension of the programme and sport organisations take part in other sectors of the programme even more than in the their own strand of Erasmus+. This contrast between perception and observation suggests that there is a need for better communication to potential applicants about cross-sectoral opportunities.

In terms of improved geographical balance, the centrality of countries in the programme network is changing. Small countries and countries from central and eastern Europe are better integrated 182 . The main exception remains the country coverage of Jean Monnet activities, with about 25% of its grants going to two countries only 183 .

The surveys also demonstrate a clear internationalisation of organisations, in and outside the EU 184 . In the monitoring survey, practitioners in general 185 most strongly believe that their mobility will lead (85%) or has already led (74%) to internationalisation of their sending institution, as shown in the figures below. For instance, 59% of concerned respondents believe that the Jean Monnet grant helped them to allocate additional funding for teaching or research about the EU. The fact that practitioners have developed strong networks, as demonstrated above, corroborates other findings on the internationalisation of their organisations. Participant organisations are significantly more likely than non-participants to be engaged in transnational cooperation, except in the sport sector 186 . This holds true to the largest extent for higher education institutions 187 and to the least for schools 188 , as confirmed in case studies. Specific findings for the international dimension of the programme underline the strong contribution of the predecessor programmes to the internationalisation of higher education institutions in partner countries through curriculum development and the improvement of management practices of universities 189 . But even in the case of school education, a vast majority of respondents believe, for example, that Comenius transnational partnerships strengthened the European dimension of schools 190 and the survey conducted as part of the eTwinning platform evaluation has observed, as a result, a certain internationalisation of schools. It is particularly important for organisations in the youth and sport sectors which work on specific issues for which they have few counterparts in their countries 191 . Given their scale and the number of organisations reached, it can be concluded that the programmes strongly contribute to strengthening the internationalisation of education, training, and youth sectors.

Whether mobility has led or will lead to changes in the sending institution

Source: ICF calculations based on Erasmus+ participant monitoring surveys.

In contrast, the internationalisation effect is less clear-cut on sport organisations. Competitive sport is already much internationalised independently from any EU intervention, whereas grassroots sport organisations have fewer international opportunities. In the latter case Erasmus+ has not reached the required critical mass. Yet, sport organisations' knowledge and usage of the sport guidelines promoted by the EU (on dual careers 192 or health enhancing physical activity 193 ) suggest some alignment of sport actions with EU policies (see fig. below) 194 . A total of 45% of organisations strongly comply 195 with the good governance in sport principles 196 .

Usage of the EU guidelines in the field of sport

Source: ICF beneficiary surveys (among practitioner respondents aware of these guidelines))

Only 4 out of 10 participant organisations have strong measures in place to combat doping and violence, while almost half have measures to combat discrimination 197 .

Share of respondents who comply with the anti-doping, violence and discrimination in sport principles (% of organisations)

Note: Index (1 (strongly agree)-4 (strongly disagree), 1-1.5=strong measures, 1.51-2=medium measures, less is low measures

More generally, another very positive monitoring finding is that overall 91% of practitioners state that they have shared their own knowledge with students and/or other persons after their participation in Erasmus+ 198 . The programme is particularly seen as transformational for youth organisations, where members of staff are significantly more optimistic about the contribution of the programme to their organisations than in other sectors 199 . Interestingly, 71% of practitioners in the sport sector state that the programme contributed to new cooperation with civil society 200 . The literature review points as well to strong positive impact, particularly on: internationalisation (all sectors), teaching capacity (school education, VET), and staff development (VET, youth) 201 , although evidence of benefits for organisations on sport, adult education and Jean Monnet activities is somewhat scarce.

Other results are softer and there is less evidence of deep changes of institutional or pedagogical practices 202 . The qualitative findings, however, suggest that a key contribution that was not measured quantitatively is the self-reflection and soft-benchmarking that happens through the opportunities for practitioners to see how other organisations tackle a given issue. In all case studies, concrete changes at organisational level were indeed cited 203 . In many cases these are light adjustments rather than major structural evolutions; however an accumulation of light changes over time can result in a step change. Frequently noted mentions include not only internationalisation (most commonly cited) 204 , but also soft benchmarking, implementation of new tools, materials, increased positive image and with that associated empowerment of practitioners and learners. Other effects for organisations include the development of national partnerships, cooperation with companies 205 or quality frameworks specifically for youth work. On average, 58% of practitioners reported as an outcome the creation of quality assurance approaches 206 .

The National Reports confirm that Erasmus+ has three main impacts on participating organisations, namely their internationalisation 207 , the quality of education and training they provide 208 and the professionalisation of youth work 209 . Both mobility actions and Strategic Partnerships allow for the development of new methods, often innovative at least for participating organisations, which can have a positive impact on quality. However, the evidence gathered at EU level in that respect is not sufficiently strong to assess whether the programme is transforming the practices of practitioners and organisations. Though practitioners are positive about the fact that they implement the lessons learnt and results from cooperation projects and mobility exchanges, there is only anecdotal evidence that the programme is changing institutions’ approaches to pedagogy and their target groups. Deeper qualitative work of a larger scale would be needed to see these types of results at final evaluation stage.

5.1.2.2. Outcomes for systems and policies 210

The general literature shows little evidence of system level impacts of preceding interventions, in contrast to the abundant findings about individual impacts. Where such evidence exists, it is positive. But it is very fragmented in all sectors, although stronger in higher education, VET and youth.

This is why, since 2014, Erasmus+ focuses through KA3 on funding actions that are specifically designed to have system level effects 211 . Comparable actions existed under LLP and Youth in Action, but some new features were introduced under the current programme 212 , enabling the Commission to address more rapidly emerging needs (e.g. the Paris Declaration, the New Skills Agenda for Europe). The systemic impact of Erasmus+ is only expected "by the end of the Programme" 213 and therefore can only be evaluated at final stage 214 . In particular, most KA3 projects were not finalised at mid-term. Nonetheless some National Reports outline KA3's policy effects although limited by the relatively small funding allocated to KA3 215 . Not all KA3 case studies could identify system level effects. Although KA3 has the highest potential to produce systemic impact 216 , further involvement of decision-makers is required. It seems therefore too early to conclude on the additional systemic impact brought specifically by Erasmus+.

However, even if effectiveness varies across instruments and countries (some use them much more strategically than others 217 ), some policy networks financed through Erasmus+ have proven influential since 2007: the national contact points for EQF 218 or EQAVET 219 , the Bologna experts 220 or the partnership between the Commission and the European Youth Forum 221 . Studies and cooperation with OECD 222 funded by Erasmus+ and LLP have provided Member States with stronger evidence, according to the evaluation of ET2020 223 .

On the international dimension, the general finding is that the projects funded by previous programmes have contributed to policy and institutional reforms in partner countries, through improved quality assurance, standardisation of higher education and a rapid extension of EU-supported Bologna principles. This has had an evidenced impact on the harmonisation of programmes and curricula in partner countries as well as on mobility and the recognition of studies and qualifications within and between non-European regions and Europe 224 .

Furthermore, the effectiveness of the programmes evaluated should also be assessed through their contribution to policies via the Open Method of Cooperation (OMC), which they have substantially supported over the period 2007-2016 225 . The evaluation of the OMC in education and training (ET2020) noted that according to national stakeholders the outputs of OMC activities have been used to influence national policies and practices. Therefore the programme plays a major role in supporting financially the OMC process, which in turn influences national education and training policies, including those priorities of ET2020 which are not easy to tackle through mobility and cooperation actions (e.g. policies on qualifications design or early school leaving policies). The programmes had also a certain systemic effect in the field of youth. The most recent evaluation of the European Youth Strategy 226 concluded that the OMC was rather positive about the influence on national policies and structures in the youth field, while the results in the youth field might not always be measurable in a way that is easily comparable with the area of education and training. At the same time, the activities funded by Youth in Action and Erasmus+ were identified as strong pillars of the OMC, regarding the structural dialogue with young people and the increasing use of the Youthpass to promote learning outcomes of volunteers. Sport is too recent a programme sector to be evaluated at interim stage in relation to systemic impact 227 .

Overall, 88% of respondents to the consultation agreed that Erasmus+ has increased systemic impact compared to the predecessor programmes 228 , whilst 46% estimated that the programmes had been effective in modernising education, training and youth systems 229 . As shown in the figure below, around 80% of the key informant interviewees saw some positive influence of the programme on policy, of which half were able to cite concrete examples. This can be regarded as a good performance 230 . Interviewees tend to be more positive about the systemic influence of the programme in smaller and less wealthy countries. Generally, most frequent examples cited dealt with EU tools related to qualifications. It seems that interviewees no longer think of systemic changes as resulting from the scale of funded mobility. Lastly, the examples remained particularly rare in the sport sector 231 .

Share of key informants interviewed who were able to cite concrete policy/ system changes linked to Erasmus+ or its predecessor programmes

Legend: As positive were coded responses which named a concrete and influential project or a clear area of influence supported via one of the programmes. As neutral were coded interviews that thought there was some influence but did not give a concrete example. As negative were coded interviews which said they are not aware of any such influence.

The interviewees mainly cited four types of changes at system level 232 . Occasionally, even a single funded project can be used as a strategic input at national policy level (e.g. no alternative resource to design work-based learning legislation would have been available in Latvia). More frequently, examples were found where an effort has been made to pool together the findings of several projects to accompany a systemic development inspired by European priorities (e.g. the Polish reform based on ECVET). Thirdly, the scale of mobility actions can trigger a change in national policy to remove barriers (e.g. Erasmus+ beneficiaries are no longer charged a visa fee when going to Turkey). Lastly, programme funded activities can raise awareness of an issue at policy level (e.g. early development of Lifelong Learning Platforms at regional level in Spain).

System level influence via individual cooperation projects remains an exception. This does not so much depend on the type of action but rather on the own merits of a given project. Effects at system level are more likely to happen if the project is aligned with country priorities 233 . Even when small scale projects prove to be innovative, unless they had a high level influential stakeholder involved from the beginning 234 they typically would not have had the capacity to reach out effectively to decision makers. It should nonetheless be noted that Knowledge Alliances and Sector Skills Alliances, two actions of a new kind and more ambitious within KA2, could not be evaluated at mid-term but might have a certain impact on systems at the time of the final evaluation of Erasmus+.

The evaluation shows that a better way for the programmes to achieve system level impact is by reaching out to a critical mass of practitioners and organisations. This has been the case in higher education where mobility actions, by their volume, led to system level changes e.g. implementation at national level of European credit systems (ECTS) and joint degrees (Erasmus Mundus). Erasmus increased the employability advantage for Erasmus alumni over their non-mobile peers by 45% and half of all European graduates who studied or trained abroad benefited from it according to ex-post impact research 235 . In several of the other sectors, the programme is also close to reaching a critical mass. KA1 alone supported in 2014 1.6% of higher education students, close to 1% of VET students and 0.36% of school staff 236 . However the impact on VET and adult education systems is more difficult to detect because of the scale and the heterogeneous nature of these sectors, which would require more targeted actions considering available funding. In KA2, eTwinning is also instrumental in reaching out to a critical mass of teachers, as described above. Other types of activity do not reach the same volume, but can be influential in areas where system level changes are expected 237 . For the youth sector, although an impact on systems is observed through other channels, it is difficult to evaluate to what extent the programme succeeds in reaching a critical mass 238 . Other sectors such as adult education or sport remain too small to see system level results through this channel.

Last but not least, Erasmus+ is expected, in line with its legal basis, to contribute to the Europe 2020 headline target in education 239 , namely (a) higher education attainment and (b) tackling early school leaving. Though the EU countries seem well on target to deliver on these two dimensions, the contribution of the programme and its predecessor to their evolution is mostly indirect. The choice of these overarching indicators for the programme can indeed be questioned in terms of testability and plausibility. Though the programme recitals cover both targets, this is done together with many other priorities. It is true that, as seen above, some countries did make policy changes linked to these two benchmarks. The programme also reaches a high number of learners via KA1 and KA2, whose attitude towards education/training is more positive when compared to control groups. Such a positive attitude can be analysed as a precursor of education retention and as reducing the risk of drop-outs. However early school leavers or people not attracted by higher education are more likely to be found among disadvantaged people. As seen above, both the share of organisations targeting hard-to-reach groups and the share of individuals particularly at risk of exclusion are low within the programme. The programme does improve attractiveness of participating higher education institutions but it cannot be deduced that it attracts people who would not have otherwise studied at higher education level. Lastly, the review of selected projects reveals a weak alignment between the transnational partnerships funded and these two key priorities. Though some potential does exist, the causality link is only indirect.

As regards other ET2020 European benchmarks mentioned in the general objectives of the programme, employability of young people has also improved in recent past (77% are employed i.e. +1 percentage point from 2014 to 2015). In contrast, other impact indicators more related to social inclusiveness have registered deterioration over the last years, with worse levels of basic skills and with lower rates of participation in youth out-of-school or physical activities 240 , which could justify a strengthened focus on social inclusion.

5.1.3.Transversal questions about effectiveness

When considering all objectives together, 71% of respondents to the consultation thought that the programme is achieving its objectives to a (very) large extent. Education and training providers who responded to the consultation claimed that lack of funding and insufficient language skills are among the barriers hindering the achievement of Erasmus+ objectives.

5.1.3.1. Most effective actions and fields 

The majority of respondents and countries 241 believe that mobility is essential in ensuring the effectiveness of Erasmus+ 242 . Actions cited as most effective by agency respondents are mobility ones 243 and strategic partnerships, with the latter mentioned with more mixed views 244 . Moreover, longer term mobility actions are associated with stronger results for learners in beneficiary surveys and by a few countries 245 . For practitioners, what matters most seems to be participation itself, irrespective of the type of action 246 . Lastly, KA2 Alliances are perceived by agencies as being less effective for organisations 247 . It can meanwhile be noted that respondents tend to consider the actions closest to their interest as more effective.

Comparing the relative effectiveness of Erasmus+ fields cannot be done without keeping in mind the diversity of needs they address and funding available at EU level. Nevertheless, 30 years of Erasmus with a larger and targeted budget allocation makes of higher education 248 the strongest field in effectiveness. In contrast, less funded or more recent fields, such as adult education or sport, not only miss the critical mass but also a similar strong focus to guarantee an equivalent level of effectiveness. School education and VET fields appear in an intermediary position, indicating at this stage the highest potential for improving the overall effectiveness of an EU programme in the future 249 .

More generally, the respondents to the consultation have considered factors which could maximize the effectiveness of the programme, as follows:

To what extent do you think the following topics need to be addressed to maximize the impact of any successor to the Erasmus+ Programme?

Source: Erasmus+ OPC survey, 28 February – 31 May 2017, ICF analysis. Base indicated next to each topic

According to the public consulted, the programme should be better funded, more inclusive and user-friendly while its opportunities should be better known and its outcomes more widely exploited and publicised.

5.1.3.2. Visibility and dissemination of results

Erasmus+ is very well-known across Europe and visible beyond. A majority of citizens have heard about Erasmus+ and student exchange programmes come third in the list of the EU's positive results 250 .

The dissemination and exploitation of project results are much more effective and systematic with Erasmus+ compared to the predecessor programmes, albeit with nuances according to the geographical level considered 251 . The monitoring was much less performant in 2007-2013, when it was scattered over different tools covering only parts of previous programmes. Good practice examples are used to raise awareness of the general public; to guide potential applicants; to inform the work of practitioners, researchers, social partners, organisations and policy makers. This new approach has been recognised in the public consultation 252 and by the experts 253 . It addresses the objectives of transparency, accountability and visibility 254 , drawing on the lessons learnt from the interim evaluations of predecessor programmes.

At EU level, the establishment for the first time of an overarching dissemination and exploitation strategy 255 with regularly reviewed indicators and a single project results platform 256 represent clear improvements in comparison to the past 257 . Specific steps have been taken both by the Commission and the Executive Agency to ensure that the exploitation of projects’ results effectively serves to inform policy making 258 .

The project results platform offers a common and accessible space for disseminating most project results whilst also highlighting good practices and success stories (including from LLP and Youth in Action). Its value in comparison to the past lies in structured and selected material. Insights gained from EU interviews suggest that the platform is actually used to inform the work of policy makers at EU level. Content-wise, the main limitation is the lack of complementary external reference materials that could inspire users and a lack of success stories in certain sectors (e.g. adult learning) due to the insufficient quality of reports.

At national level, the assessment is also positive about the strategy, setting clear rules for the selection of good practice examples. However, the evidence of the exploitation of project results by policy makers 259 and the effective engagement of the latter when they are not included in the project itself is not clear. The vast majority of National Agencies mainly disseminate examples of good practices to potential applicants or beneficiaries, and only rarely to inform policy making bodies. Moreover, though valued by over half of respondents as shown in the graph below, 37% of agency respondents do not agree that the dissemination platform is user-friendly. This sentiment, more mixed than at EU level, concurs with insights gained from national interviews and reports 260 .

Source: ICF experts’ survey

Furthermore National Reports refer not only to the dissemination platform, but also to other dissemination channels (e.g. events, websites, social media). Despite all tangible dissemination efforts, some observe that the results of projects are seldom further exploited 261 . While this can be due to factors that are external to the programme itself (e.g. professionals’ lack of time) some regret a lack of concentration of resources on the most interesting projects 262 and an insufficient involvement of national authorities and/or municipalities in their dissemination 263 .

At project level, the expert panel confirmed that dissemination is effective in 90% of the cases they assessed 264 . The platform is also of use for people looking for partners. Interviewees and participants were aware of the dissemination and exploitation strategy requirements, but often perceived these as burdensome. Despite the standardised processes in place (except in the case of Jean Monnet activities 265 ), project leaders often do not have the capacity to effectively exploit results. This might explain why, according to OPC respondents, the dissemination of results is one of the aspects of Erasmus+ that should be improved in the future.

5.2.Relevance

The assessment of the relevance 266 of Erasmus+ is generally positive 267 in regard to both its alignment with policy priorities and its adaptability to socio-economic needs. The relevance of the programme, compared where appropriate with its predecessors, has been assessed against 8 evaluation questions that overall looked into the extent to which Erasmus+ is:

·aligned with EU and national policy priorities

·responsive to the needs of stakeholders, sectors and society at large

·visible

·suitable for attracting different target audiences, including hard-to-reach groups

The vast majority of interviewees and most of OPC respondents 268 consider that the programme shows a high degree of alignment with key EU priorities namely as set in the Europe 2020 Strategy, ET2020 Joint Report, the Youth Strategy, and to some lesser extent, the Union's external action, including its development objectives, and the EU Work Plan for sport 269 .

Erasmus+ is strongly relevant to the rationale of the European project: it fosters the intra-European integration through learning. For instance, Erasmus+ and its predecessors have played an important role in supporting the Member States and beyond in implementing the Bologna process with the introduction of the ECTS. Had it not been for Erasmus+, much fewer students would have benefited from learning mobility or would have been much more oriented towards non-EU English-speaking countries, according to interviews 270 . The consequences of a scenario where Erasmus+ would not exist are elaborated further in the section on its European added value below.

The international higher education dimension of the evaluated programmes has contributed to achieving socio-economic development objectives. The EU programmes for international cooperation in higher education have contributed to fostering regional integration as a key objective in the EU’s relations with partner regions and supporting EU policies in the field of legal migration and mobility 271 .

Other evidence 272 and the consultation 273 reveal in addition that the alignment with European policy priorities is stronger with Erasmus+ than with its predecessors, as shown below.

Share of projects that focus on issues that are high on the European policy agenda

Source: ICF, experts’ survey

On the one hand, the review of topic coverage of funded projects undertaken for this evaluation and the expert survey confirm that these align to a high extent with EU-level general priorities, such as promoting lifelong learning, equity or social cohesion. For instance, 73% of expert respondents as opposed to 52% in the case of predecessor programmes, estimate that a high share of the funded projects which they assessed align with EU-level priorities. This is also consistent with the data analysis of cooperation projects, albeit with variations in topic prioritised before and after 2014, as shown in the table below:

First 10 topics of cooperation projects under the Erasmus+ and LLP

Source: ICF calculations based on the Epluslink and LLPLink 274 (% of total decentralised projects)
In red: significant decrease under Erasmus+. NB: most projects of the sport strand are centralised, so do not appear here.

On the other hand - when analysing projects more in-depth, beyond statistical data - some key challenges are only addressed marginally in practice. The extent to which the projects reviewed align with more specific priorities (e.g. early school leaving, higher education attainment, adults’ participation in adult learning, etc.) is not clear enough. For instance, across the sampled projects in higher education, none focused on the attainment headline target 275 and in school education, only a very small number dealt with early school leaving 276 . There is still room for improvement in the number of activities in topics on key competences and basic skills, even though these have been a priority in many calls for project applications. Lastly, in the sport field, respondents at national level often felt a gap between broad EU priorities and the need for more customised ones 277 .

Most interviewees also consider that Erasmus+ has proven to be flexible in adjusting to new emerging EU-level challenges 278 (e.g. refugee crisis, intolerance or violent radicalisation were often mentioned), particularly through its annual work programming 279 . Several National Reports value as well the flexibility of Erasmus+ objectives to address emerging challenges 280 .

Meanwhile, 79% of OPC respondents indicated that in their view there are no other priorities or actions that Erasmus+ should address. For the remaining share of respondents, the main suggestions 282 included in descending order of frequency: i) promoting European citizenship and civic education; ii) fostering inclusion and diversity, especially the integration of migrants; iii) linking education more strongly with enterprise and promoting international internships.

In contrast, the context has changed radically since the Jean Monnet programme was founded. There is now substantial volume of teaching and research about the EU, particularly in EU countries, irrespective of Erasmus+ support. Because of their focus on higher education, the evaluation shows that the relevance of Jean Monnet activities could be enhanced by focussing more on other segments of the learning continuum such as schools and VET. Given the low turnout in the European Parliament elections 283 and the lack of trust in the EU institutions 284 , there is a need to strengthen Europeans’ understanding of the EU. This is the underpinning challenge that Jean Monnet aims to address. Overall, only 56% of Europeans state that they understand how the EU works - a number that has nevertheless improved by 9 percentage points since 2004. However, Jean Monnet activities target a population that has been tested as already more knowledgeable about the EU than non-beneficiaries. One third of staff respondents dedicate substantial share of their teaching to students outside EU-related fields of study. However, most of those reached remain students who would still have studied European integration 285 .

The extent to which all the ambitious objectives of the programme are achievable over time is also sometimes questioned. Some stakeholders call for greater emphasis on the promotion of social inclusion. In the aftermath of the 2008-2009 economic crisis, Erasmus+ was primarily focused on economy-related priorities. Since 2016, a shift towards social inclusion is observed in its implementation. Responses seem to indicate that the social inclusion objective could be explicitly defined and go together with a higher and more visible budget to achieve a more tangible impact 286 . However, views remain mixed on this issue. Many stakeholders, not only from the youth sector but also from education and training, believe that too much emphasis is put on employability to the detriment of the cultural value of education or the youth sector; whereas some other stakeholders insist on the continued relevance of employability. Some National Reports refer to needs that could be further addressed, including i) the objectives connected to the Paris Declaration and prevention of violent radicalisation 287 ; ii) traineeships to enhance labour-market inclusion 288 ; iii) transversal, social and personal skills – hence aligning well with data collated by the external evaluation at national level.

The alignment of Erasmus+ objectives with national policies is almost equally recognised, by most of respondents 289 . Overall, a vast majority of experts surveyed estimate that a high share of the projects funded align very highly to fairly with national level priorities 290 . Experts felt that there has been an improvement on this matter between Erasmus+ and its predecessor programmes, as illustrated below.

Share of projects that focus on issues that are high on the policy agenda of my country

Source: ICF experts’ survey

Nevertheless views from national interviewees are more mixed 291 , with organisations being more complimentary than policy makers 292 or agencies’ respondents 293 . The most noticeable difference is found in the sport field where no stakeholders and only one out of 4 policy makers see Erasmus+ as strongly relevant to solving national challenges.

Most of the national authorities 294 consider as well that Erasmus+ decentralised actions 295 are relevant to country needs, although fewer countries refer to national priorities in the adult education, sport and Jean Monnet sectors. Furthermore some countries ask for more flexibility to allow the programme to adapt better to local needs 296 .

Examples of relevance to national needs or need for more flexibility

Source: ICF 4.1, NRS 3.1

Most position papers collected during the consultation highlight that Erasmus+ is one of the EU’s most successful programmes and that it remains highly relevant. The majority of the consultation respondents (59%) thought that the current Erasmus+ objectives are extremely relevant to the current challenges and needs, especially as regards the development of skills and competences, the European dimension of education and youth activities, languages and intercultural awareness. In contrast, fewer respondents saw the relevance of the objectives regarding the international strand, Jean Monnet, policy support and sports 297 .

Relevance of the programme can also be assessed in relation to its capacity to fund innovative projects, considering the needs for innovation in a context of global competition between economies and particularly between education systems. While evidence shows that a wide range of applicants are well targeted, the innovative capacities are rather limited 298 . The experts’ survey indicates that the projects are most frequently (42%) moderately innovative with a relatively small share that was considered as highly or very highly innovative (15% and 2%, respectively). At the same time, similarly to the increased quality of applications 299 , an improvement can be seen in terms of innovation when comparing Erasmus+ with its predecessor programmes. This could further improve at the final evaluation stage when KA2 Alliance-type projects are sufficiently advanced or have been finalised to be evaluated.

More generally, key evidence suggests that the programme is well suited to attract a wide range of target audiences across the fields it covers. It is particularly suitable for reaching out to learners with different profiles 300 , with one noticeable exception: disadvantaged groups, as elaborated below 301 .

A deeper analysis shows that the needs of different stakeholders are generally well addressed, albeit with variations according to sector, organisation size and level of intervention. The majority of the interviewees estimate that the programme strongly contributes to addressing the socio-economic needs of the learners 302 . Monitoring survey highlights that the motivations of learners are fairly well aligned with programme objectives and related to key competences. Foreign language skills is the most mentioned need in all sectors. In VET, learners’ motivations for participation are particularly well aligned with programme objectives on key competences, including technical skills (80%). At least 50% of agencies’ respondents believe that Erasmus+ meets the needs of target groups including practitioners 303 better than its predecessor programmes. They all agree that learners’ needs in higher education and VET are met by both KA1 and KA2, but the needs of school pupils are partially or weakly met by KA1, despite the fact that teachers take part in the action. On a qualitative note, many examples in interviews suggest strong relevance of the programme to learners’ needs 304 . Lastly, as detailed above in the section on effectiveness, participants’ satisfaction is very high (81% to 100% claiming being very or rather satisfied with their mobility experience). In contrast, interviewees have more mixed views on the programme’s ability to meet the needs of organisations. The needs of small organisations especially are not met sufficiently 305 . Lastly, agencies’ staff consider that the needs of schools, VET and youth systems are addressed by KA3 (highly relevant), while the relevance for higher education and adult learning systems would be more moderate.

The discontinuation of a small number of (Lifelong Learning Programme and Youth in Action) actions had only a limited impact on the extent to which stakeholders’ needs were addressed, except in two areas: 306

·many interviewees claimed that pupil mobility (and to a smaller extent adult learner mobility) should return to a KA1-type of funding, instead of under the more complex coverage of KA2 307 .

·discontinued local youth initiatives previously funded by Youth in Action were considered more accessible than current Erasmus+ transnational youth initiatives 308 . The former enabled young people to initiate their own projects, whereas the latter are seen as too complex for informal groups of young people, especially NEETs (young people not in Education, Employment, or Training), to apply 309 .

Knowledge about Erasmus(+) 2009 vs 2016

Source: Standard Eurobarometer 86 and Special Eurobarometer survey 316

All findings reveal that the programme is perceived as increasingly visible and attractive 310 . Erasmus+ is well known to the audience in general (see fig. above) 311 and especially to the young generations. Its large visibility also, including outside the EU, stands out when compared with that of other EU programmes such as Horizon2020, Europe for Citizens or EU Aid Volunteers 312 . It can also be positively noted that the degree of repeated participation appears to be low for learners (less than 10%), whilst it is expected for the more permanent population of staff 313 . It is nevertheless found that the extent to which Erasmus+ reaches out to potential organisations varies across the different fields. The visibility of the Jean Monnet activities could, in particular, be stronger 314 . A few National Reports 315 link the lower participation of primary education and early childhood education and care in Erasmus+ to a lower awareness of the programme in these sectors. Some National Reports make suggestions on how to increase visibility 316 .

Furthermore, a number of barriers to participation are identified limiting the extent to which the programme addresses the needs of stakeholders. At the individual level, financial barriers 317 and administrative burdens 318 are more often reported than are geographical disparities 319 . At the organisation level, the programme offers less room for small scale projects than in the past, reducing the extent to which it reaches grassroots level organisations in all their diversity 320 . Evidence suggests that smaller sized organisations often have neither the experience, nor the resource capacity that a successful application requires 321 . Many stakeholders believe that funding has become too highly competitive, making it only accessible to the most experienced/largest organisations, preventing access to potential newcomers, hence reducing the number of those who may benefit from a European experience 322 .

Despite the specific attention paid to social inclusion 323 with varied results across sectors 324  and to widening the participation of disadvantaged target groups 325 , recognised by a majority of interviewees 326 and 30% of agencies, evidence suggests that there is room for improvement 327 . The share of disadvantaged groups represents almost 10% of KA1 participants (almost 25% in the youth field as well), as detailed in the section on effectiveness above, due to the programme’s specific efforts in the area (see fig below). Erasmus+ has offered new experiences abroad to many people who would have otherwise not been able to afford it. In countries where there is no mobility programme targeting disadvantaged group, Erasmus+ is particularly appreciated. Where programmes supporting mobility do exist, they do not have such a priority or – if they do – they do not promote it to such an extent as Erasmus+ does 328 .

Perceived level of participation of disadvantaged groups (Erasmus+ compared to other comparable national/international programmes/initiatives

       Higher level        Equivalent    Lower level    No opinion/data/n.a.

Source: ICF survey with National Agencies (n = 117)

However, according to a vast majority of National Reports 329  and a shared perception among all groups of respondents, young people from disadvantaged backgrounds and with special needs are not sufficiently addressed. The view prevails that Erasmus+ has even limited potential to reach those organisations that address specifically this group of people. Even the European Volunteer Service (EVS) remains strongly competitive. Most National Reports and findings from the public consultation welcome the additional funding available for disadvantaged participants 330 but do not consider it sufficient on its own. Some countries 331 call for clearer definitions and streamlined rules applying to disadvantaged participants in Erasmus+. According to the expert panel, while a good share of projects viewed considered disadvantaged groups, this was often not substantiated with outputs suitable to them. Moreover, the Student Loan Guarantee Facility (SLGF) is commonly found to be insufficiently tailored to address the needs of the disadvantaged who are risk averse to go abroad for a full Master programme or to take up loans even if repayments are not income-contingent 332 , although the first evidence from the beneficiaries surveys rather shows many beneficiaries are first-generation higher education students or coming from families with financial difficulties 333 . Despite some progress made in the programme design, it can be concluded that there is a certain dissonance between its ambition and actual project outputs specifically addressing the disadvantaged target population. The views whether more can be done at European level to address these challenges were nevertheless mixed among national policy-makers.

Source: NRS, 3.4 (PL, TR)

5.3.Coherence

The assessment of the coherence of Erasmus+ is overall very positive, both internally (compared to its predecessors) and externally (compared to other existing interventions). It has been informed by 10 evaluation questions that looked into the extent to which:

·the programme’s internal coherence improved compared to its predecessors

·the Erasmus+ objectives were consistent, mutually supportive across fields

·synergies or duplications between fields and actions existed and were dealt with

·the centralised and decentralised actions complemented each other

·Erasmus+ design provided appropriate support to the EU policy agenda

·Erasmus+ was coherent with relevant policies and programmes and complementary to other interventions in the same fields at EU or national/international levels respectively.

5.3.1.Internal coherence 334

The merger of seven predecessor programmes into Erasmus+ has improved the coherence of the EU intervention according to the vast majority of respondents 335 . As mentioned before, Erasmus+ now covers learning in all its contexts – whether formal or non-formal, including youth work– and at all levels of the lifelong learning continuum: from early childhood education and schools, vocational education and training (VET) through to higher education and adult learning. The evaluation highlights the positive effects of this integrated approach underpinned by the lifelong learning logic. It underlines the necessity of the integrated approach in today's education and training environment, where the borders between the traditional education and training sectors have become less clear-cut 336 .

Different sources agreed on the added value of the integration of several programmes within a single structure. The objectives are seen as mutually supportive across education/training, youth and sports, as well as across actions 337 . These fields are seen as naturally fitting in a programme allowing cross-sector fertilisation, which - according to 90% agency respondents - was not always possible in the past. Around 3/4 of these respondents feel that the integrated programme enables increased synergies, while 2/3 estimate it avoids internal overlaps.

All these improvements are confirmed by desk research, national reports and a vast portion of the interviewees who appreciate that the programme has been designed with better logical linkages between objectives and actions 338 . It is very often valued for its clearer and simplified architecture with only three Key Actions (KAs) 339 , cross-cutting priorities (e.g. social inclusion, prevention of violent radicalisation, etc.) and a much lower fragmentation of actions than denoted in the past (e.g. Grundtvig, international cooperation 340 ), with only a few exceptions 341 . Almost all country reports call for stability of the structure and scope of the programme in the future. The stronger attention given in the design and implementation of Erasmus+ to budget based on performance 342 , monitoring of individual outcomes 343 and dissemination of project results 344 is also valued. As detailed under "effectiveness" above, actions are delivering at more than one level, with spill-over effects (e.g. individual staff mobility having impact on sending organisations) and common support tools (e.g. widening scope of transparency and recognition tools).

The advantages of Erasmus+ programme listed by the national respondents include its strong branding and its greater visibility 345 , especially among policy-makers. This is particularly true for VET, adult education and even for the youth sector in contrast to stakeholders' initial fears. European challenges (such as the integration of refugees or the attention to disadvantaged groups in higher or adult education) are now addressed at least in a more flexible way 346 . Moreover, lifelong learning is approached more as a lifecycle (including non-formal and informal learning) rather than in a segmented way. Even if there remains residual concern 347 that the brand “Erasmus” is traditionally linked to higher education, the fact that the Erasmus+ provides one place to look for funding opportunities in the related sectors with a standardised and more transparent administration is appreciated.

Meanwhile, some concerns have been reported at national level. In two-thirds of the countries, different NAs continue to deal with distinct sectors, replicating the way they functioned under the previous programme generation 348 . Some National Reports confirm that there is no systematic exploitation of potential synergies between different types of actions 349 . The disadvantages inherent to any one-size-fits-all approach are naturally pointed out 350 . Respondents from various organisations and sectors noted that the adult education strand has shrunk in size and lost its specificity. The youth chapter would be less integrated, less visible for instance within KA2 (or would now link to more abstract EU policy targets than in the past). Whilst the objectives of the sport chapter align well with EU priorities, some potential applicants do not always see how to make them fit in their local perspective. More generally, an integrated programme is likely to offer less autonomy to sectors and less room for country-specific priorities compared to its predecessors. There is for instance the sentiment among national authorities that the integration of the programme committees has led to discussions becoming more abstract for each policy area integrated in Erasmus+.

When looking at specific examples, further nuances emerge. The increased decentralisation was positively viewed by the vast majority of respondents because it brings the programme closer to its beneficiaries and helps engage local partners, while centralised actions (e.g. in KA3) complement well other actions of the programme. Meanwhile, policy makers and NAs were clearly more positive than EU level organisations which look for a more direct access to funding. The main critical perceptions relate to KA2 suggesting that too broad scope of the action and the decentralisation of strategic partnerships may result in differences in implementation or a prevalence of nationally-driven agenda 351 . Some argue that this could even push partnerships or cross-country organisations to apply rather in larger countries. Besides to attract talent to Europe, some suggest it could make more sense to centralise international credit mobility. However, providing concrete examples of synergies or overlaps has proven more difficult than anticipated for most interviewees at national level or agency respondents. The only area where some overlaps have been noticed concerns those projects in the sport sector which focus on social inclusion and could be youth projects as well. However, overlaps might be more often due to misunderstanding at application stage than to the design of the programme as such 352 .

Examples of internal synergies or overlaps

Source (where not mentioned otherwise): ICF, 5.2.2

Another area where findings are more contrasted concerns cross-sector opportunities. Whilst a significant number of respondents 353 see this as a clear improvement under the new programme (e.g. for youth), many others 354 remain more uncertain (e.g. unclear remit of adult education 355 ) or have concerns about its actual materialisation (e.g. for sports 356 ). A majority of respondents asked are either not interested or not convinced to take part in cross-sectoral projects (e.g. finding different partners require more efforts). However, as detailed under 5.1.2, the level of cross-sectoral cooperation increased under the current programme by 23 percentage points

5.3.2.External coherence 357

At programme level, complementarities are mainly found with the European Social Fund 358 (ESF) and the Research framework programme Horizon 2020, and to a lesser extent with INTERREG, Erasmus for Young Entrepreneurs programme (EYE) and in the field of sport with the Third Health programme. For instance, in Greece, the participation of pupils in eTwinning was boosted nation-wide by the combined support of European Investment and Structural Funds to teachers' training (ICT skills) and classroom equipment (digital tablets) 359 . Potential synergies that were noted by interviewees include: Erasmus+ and the European Training Foundation (ETF); Europe for Citizens and Culture/Creative Europe programmes; the EU Programme for Employment and Social Innovation (EaSI) which has a component of entrepreneurship education, etc. Few potential overlaps were alleged by some interviewees (e.g. in the case of strategic partnerships) but without any concrete examples 360 .

The integration of the programme has also enhanced the external coherence in removing certain overlaps 361 . Insights on the coherence of the programme against other EU level programmes were gained from both a majority of EU level interviews and the vast majority of agency and OPC 362 respondents. Those provided examples of (potential) synergies or overlaps, confirming desk research and benchmarking.

Examples of external complementarities, synergies or overlaps

Source (where not mentioned otherwise): ICF, 5.3

Despite this promising picture, most of the examples supplied remained rather general. In particular, few examples were provided where ESF would have funded follow-up of Erasmus+ projects (KA2). More synergies should be encouraged between ESF and policy actions (KA3) for designated bodies. Considering the call for action set in the Erasmus+ legal basis 365 , some respondents suggest that communication between services in charge of given programmes could be strengthened at EU level. In particular, no clear example of dedicated coordination tools between EC services was reported.

At policy level, the topical coverage of the selected projects reviewed reveals a clear and satisfactory alignment with different EU-level policies 366 . Most interviewees (particularly at EU level) perceive Erasmus+ as supporting key EU policy agendas 367 , especially in education and training, similarly to its predecessors but in a more streamlined manner. Most of EU interviewees could confirm it with examples of complementarities. Out of the 80% of reviewed projects 368 aimed at contributing to the main EU policies/strategic documents analysed as part of desk research 369 , two-thirds focused on ET2020 topics, mainly social inclusion 370 . However, across the sampled projects, only a very small number focused on the Europe 2020 headline targets, despite these being the two first key indicators of Erasmus+ 371 . Lastly, the Jean Monnet projects reviewed appeared to be the least aligned, being perceived as too focussed on experts' needs 372 .

Source: ICF, 5.3

Erasmus+ programme is also coherent, to a high extent, with interventions pursued at national and international level which have similar objectives. A vast majority of interviewees consider it to be overall well aligned with national priorities in all sectors. Benchmarking of comparable programmes 373 but also National Reports 374 , interviews, agency survey and desk research did not detect any notable overlaps 375 and provided interesting examples of complementarities at both levels. The area most often mentioned by VET interviewees in that respect is school and work-based learning. The higher education interviewees offered several examples of national-level programmes that show significant coherence with Erasmus+ 376 .

The benchmarking confirmed a fair alignment between Erasmus+ and national comparable programmes in terms of objectives, target groups 377 and duration of actions 378 . Although few overlaps were reported 379 and efforts have already been made to clear them 380 , the existence of parallel funding opportunities 381 could likely be better communicated to the public at national level, considering the aforementioned call for action set in the legal basis. Indeed, despite the fact that several NAs run some schemes in parallel to Erasmus+, many beneficiary organisations and NA respondents 382 claimed they were not aware of any similar national support.

Countries themselves did not identify any specific overlaps between Erasmus+ and other national and international programmes, but rather more synergies. Erasmus+ is different from (inter)national initiatives in many respects 383 , e.g. wider number of beneficiaries and projects; wider geographical scope; different target groups or sectors than national initiatives; mobility for longer periods; no other initiatives in some smaller countries to support certain sectors (e.g. youth) or policy reform similar to Key Action 3. A few national reports even mention cases of synergies 384 .

Although aiming to achieve comparable goals in most cases 385 , none of the national schemes reviewed can match the pan-European scope of Erasmus+, not to mention its worldwide dimension, making it the best-placed programme to develop internationalisation. In most cases it is also more generous than other similar schemes. Furthermore Erasmus+ offers some unique actions that none of its comparators does (or, at least, not to the same extent), such as electronic support platforms (eTwinning, EPALE etc.), stakeholder dialogue or knowledge building for evidence-based policymaking. Over half of agency respondents strongly or rather agree that Erasmus+ is doing better than the latter in all five areas considered 386 . This can be regarded as a real added value of Erasmus+, as developed under the related section below.

5.4.Efficiency and simplification

When it comes to assessing the efficiency of the programme 387 , including in comparison to its predecessors, the overall picture is positive 388 , stemming from the analysis of 24 evaluation questions that cover the following areas:

·the cost-effectiveness of the main types of actions

·the efficiency of implementation and management modes

·the efficiency gains through changes in the integrated programme

·the efficiency of monitoring arrangements and measures to identify and prevent fraud and irregularities.

Overall the costs of management for the EU of the entire programme are reasonable (6% of the Erasmus+ administrative and operational budget) 389 . This is particularly clear when compared to other, much smaller, comparable national actions, which appear more costly (on average, 14% of their respective budget) 390 .

Most of respondents to the consultation believe that the user-friendliness of the programme has improved over time 391 . However, work on simplification (e.g. online application forms) has already started in 2017. The level of dissatisfaction has decreased as of 2016 after a steep learning curve.

Nonetheless, there is still room for improvement in several features of the programme 392 , especially concerning further simplification of procedures and tools (e.g. on-going development of e-forms), more flexibility on budget allocations, adjustment of the level of simplified grants as already proposed in 2017, as well as the reversal of the low success rates 393 in some particular fields and sectors.

5.4.1.Cost-effectiveness

With regard to learning mobility of individuals under KA1, the relationship between the costs (inputs) and effects (results and impacts), indicates a positive cost-effectiveness 394 especially for learners but also for staff 395 . Considering the positive outcomes identified above under "effectiveness", the costs per individual appear clearly reasonable. This is further supported below by the evidence of European added value of these types of actions, also compared to similar programmes reviewed at national level. The average cost of a mobility activity is 1,500€ per learner (15€ per day/learner) 396 and between 700€ and 900€ per staff member (200€ per day/staff) 397 . An increase in costs per learner has been measured compared to the predecessor programme as regards higher education (+9%) and VET (+1%) 398 . An increase in costs per staff member is similarly observed in higher education (+22%), adult education (+19%) and VET (+2%), while a decrease is noted in the case of school staff (-6%) 399 . Although the impact of a mobility activity is more significant by nature for learners than for staff, the effects on staff also create positive spill-over effects at the level of the organisations/institutions as well as on the learners who are not necessarily taking part in mobility activities. Hence, the overall ratio of costs versus effects can be considered positive in all cases. There is also an European added value of the programme actions for staff as for learners, since the comparison with national programmes is favourable for Erasmus+, as demonstrated below under "added value".

As the grant amounts mentioned above may appear low, the degree to which these grants cover expenses incurred has been examined. On average for 42% of KA1 learners the Erasmus+ grant covered most of expenses (76-100%), but as the graph shows below, wide differences are made between learner categories 400 . Erasmus+ is successful in leveraging complementary funding from national or regional budgets 401 .

Share of expenses covered by the grant – by type of learner

Source: ICF calculations based on Erasmus+ participant monitoring surveys 402 .

On the other hand, the conclusion is more nuanced as regards the cost-effectiveness of cooperation projects. This is because there are a variety of multilayer effects having a significant impact on the solid assessment of cost-effectiveness 403 . These include differentiated budget items, as well as variations between their types and sizes across and within sectors (around 180,000€ per project on average) 404 . In spite of efforts made for this evaluation, the conclusion on cost-effectiveness for cooperation projects cannot be generalised as for mobility activities. Although quantitatively, the cost-effectiveness ratio can seem positive at first sight (widened participation, multiple layers of effects), on a more qualitative note, there are elements that can be improved in terms of efficiency of implementation (application process) and effectiveness (quality of project outputs) 405 . As a general trend, cooperation projects seem to be very well designed and conceived at application stage although they are not able to keep the same level of excellence at the implementation stage. The majority of projects receive indeed lower quality scores on completion than at application stage. In contrast, given the relatively low costs of collaborative platforms for the EU, these types of actions appear as particularly cost-effective.

5.4.2.Size of budget

Budget envelopes for most of the sectors were regarded as insufficient 406 and strongly correlated to low success rates 407 . In some actions or countries, this had led to a discouragement of applicants over the first years of implementation of Erasmus+. The potential of the programme for broader organisational and system level effects, as discussed under "effectiveness", could be enhanced by reaching a critical mass across all the fields of education, training and youth.

The demand largely exceeds the funding available. In KA2 the competition and the scores for successful projects are high 408 , which makes it more difficult for organisations with no or little experience to access the programme. Given the increased demand for the cooperation projects, the budget allocated to KA2 and KA3 did not appear to be sufficient yet in absolute terms. Even in KA1, success rates can be low depending on countries and actions, meaning that more learners could benefit from the programme if there were more funds available 409 . Without prejudice to negotiations on the next Multi Financial Framework, the evaluation, including the public consultation, points to the need for reconsidering the budget envelope and the need for reviewing the budgetary distribution between programme sectors to have a better alignment to their relative effectiveness. However, there is no evidence that seriously questions the current overall balance between Key Actions. The vast majority of respondents to the consultation agreed with the current budget distribution between the three Key Actions set in the Erasmus+ Regulation 410 .

5.4.3.Implementation modes and user-friendliness

The division of responsibilities, as inherited from predecessor programmes, between the Commission, National Authorities, National Agencies and EACEA, is overall clear and fit for purpose 411 . It has not significantly changed over the two programming periods, if it was not for the decentralisation of the management of the higher education international (non-EU) programmes and some changes in the level of management of certain types of cooperation projects. The only outstanding challenge stems from the rigidity inherent to the addition of the development-related requirements of the EU external cooperation funds which are also used to promote the international dimension of higher education 412 . This requires a complicated management of multiple small budget envelopes with different rules at decentralised level and a disproportionate effort compared to the numbers of beneficiaries 413 .

The guiding role of the Commission with regard to the programme management has been acknowledged by 63% of agencies, due to efforts made in terms of coordination and communication. However, many National Agencies expressed a need for EACEA to enhance its cooperation with the national level.

Above all, there is a general call for further simplification of the management IT tools. Lowering the burden of the application procedure through redefining the type as well as the volume of information required depending on the actions of the programme and making better use of the reporting system (see below) would contribute to continued improvement of the programme's implementation 414 .

5.4.4.Efficiency gains and simplification through changes in the integrated programme

Overall, efficiency gains were acknowledged by a majority of countries 415 , sectors 416 and most of National Agencies 417 . In general, countries recognised a certain simplification of administrative procedures, while participants expressed concerns about application and reporting.

Due to the lack of time for adjustment between the adoption and the entry into force of the legal basis (i.e. less than 3 months) and for business continuity reason (e.g. the constraints of the academic year), all players (the European Commission, the implementing bodies of the programme and a large spectrum of programme stakeholders) had to go through a challenging inception phase to adapt to the many novelties of Erasmus+ over the first two years of its implementation. This was mainly due to the late adoption of the Erasmus+ Regulation (end of 2013), coupled with the necessity to implement the majority of the actions already by the first trimester of 2014 418 , which significantly reduced the time for the Commission, the National agencies and the Executive Agency to put in place the necessary normative framework and technical infrastructure to implement the programme smoothly. The new architecture of the programme, although it is now positively valued, led to an overhaul of previous administrative rules, definition of new criteria and new ways of operating. A new generation of IT tools has replaced the previous one and new automated support has been introduced for a number of management tasks.

National agencies and authorities recognised that initial challenges were gradually overcome and that there has been a steep learning curve with substantial efficiency improvement and simplification over the period 2014-2016. The challenge of digitalisation that was put in practice through the online management of applications and reports is a case in point, and so is the generalisation of simplified grants 419 . The programme management has now reached cruising speed.

On the positive side, different stakeholders and countries have acknowledged the difference made with the introduction of: a) simplified forms of grants, mainly lump sums and unit costs 420 , b) uniform application forms, c) digitalised application procedure 421 , d) possibility to apply as an institution instead of as a participant, and e) a single website to get easier access to all programme information

On the negative side some other stakeholders and countries did not consider that the integrated programme makes it easier for potential applicants to understand the funding opportunities and even claimed that the standardised approaches (e.g. applications, reporting) made work more time consuming than in the past 422 . Although acknowledging some administrative simplification introduced with Erasmus+, according to National Reports 423 , participants express concerns about too complex application and reporting procedures. For instance, beneficiaries responded that forms tend to be repetitive and some questions are quite long and difficult to interpret.

Overall there are a number of areas that need further improvement and simplification, such as: a) better field-customisation of the unified procedures, b) further simplification of the application and reporting procedures, c) making the programme more accessible to smaller organisations 424 , d) decreasing the time required for project management, especially in KA2 425 , and e) making the terminology used more user-friendly 426 .

In comparison with the previous programmes, the main changes which agencies associate with efficiency gains are online applications and reporting as well as the fact that there is a smaller diversity of types of actions to manage. However, the agencies are not very positive about the evolution of their own administrative workload. There is also room for improvement concerning the lowering of the administrative burden, as shown in the graph below 427 .

To what extent do you agree with the following statements about efficiency gains
for your agency under Erasmus + compared to predecessor programmes?

Source: Survey of programme agencies 

The rationale of the economies of scale expected from the consolidation of several programmes into one was that an integrated programme with fewer but bigger actions/projects would be simpler to administer, while the use of unit costs would further drive the management costs down. If the model proposed in 2011 by the Commission to have a single National Agency per programme country had been adopted in 2013, it would have allowed according to estimates "to reach a cumulative effect of these simplifications to a productivity increase of around 40%" 428 . This objective however could not be a target for Erasmus+, as most countries continued to manage the programme through several National Agencies.

Management fees for NAs – LLP, Youth in Action and Erasmus+ (% of overall funding)

Source: ICF calculations based on a review of annual work programmes (% of overall funding)

The management costs of National Agencies compared to the programme value are smaller than under predecessor programmes, when looking at LLP and Youth in Action. The comparison looks inevitably less positive when taking into account the overall budgets of predecessor international higher education programmes (Erasmus Mundus, Tempus, etc.), as their complexity reduces the average efficiency of Erasmus+. This is because the decentralised actions with partner countries have specific criteria and 12 different budget envelopes within the EU external relations funds. In any case, the comparison is not a simple one. There are important limitations in assessing the change of management fees: a) the base is not comparable between 2007-2013 and 2014-2016, as some actions that used to be grants managed by National Agencies have been moved to agencies' core budget and b) the comparison with management costs of the Executive Agency is not available since EACEA used to manage all international higher education mobility actions, of which an important share has been decentralised to National Agencies.

Nonetheless more efficiency gains are still expected to materialise, as the Erasmus+ budget will increase in the second half of the programming period whereas the management costs will proportionally decrease. It is therefore too early to conclude at mid-term and the actual cumulative effect on efficiency will have to be measured after 2020.

5.4.5.Monitoring mechanisms

The programme monitoring has seen major improvements with Erasmus+ compared to the predecessor programmes. Much more qualitative (e.g. beneficiary surveys showing results beyond outputs) and comprehensive data is now available (e.g. international HE mobility being now covered as well) 429 . Data is also more systematically used and disseminated through new tools dedicated to Erasmus+, namely an on-line Dashboard for internal management purpose and an Annual Report for public accountability purpose 430 .

Erasmus+ was expected to have a much clearer performance-based management than predecessor programmes and indeed arrangements have been put in place to allow regular monitoring: key performance indicators have been defined in the legal basis as well as in DG EAC's Strategic Plan 431 , data is collected on ongoing basis in relation to most of these indicators and targets have been set 432 .

However, as partly discussed in the section on effectiveness, to lower the reporting burden on beneficiaries 433 , there is room for improvement when it comes to: a) clarity and relevance of some output indicators, as well as of the quality of the data 434 , b) robustness of the self-reported results indicators, c) proportionality between the related burden on beneficiaries and the actual use of the data, d) balance of monitoring efforts according to types of action, e) user-friendliness and further inter-operability of IT tools 435 , f) and promotion of truly performance-based management at the level of programme agencies. At Commission's level, the reporting is used primarily for accountability; greater use of the information should be made for programme monitoring and continuous improvement purposes 436 .

When looking at comparable national programmes, the monitoring arrangements of Erasmus+ are nevertheless in line with existing practices and perform well, given the size of the programme 437 . Availability of objective data for results (e.g. based on actual pre-post language proficiency assessment through the OLS), as well as availability of large volume of self-reported feedback collected through systematic beneficiary surveys are two areas in which Erasmus+ monitoring practice can even be considered better than that of other national comparable programmes 438 .

Overall, the monitoring mechanisms are effective when it comes to providing an up-to-date view of programme implementation to the Commission, the agencies and national authorities but also to the general public, in particular via the Erasmus+ Project results platform 439 , as discussed under "dissemination". However, there is less evidence of the use of the data for management and planning purposes, which may be due to the fact that the monitoring arrangements have only been enhanced since a couple of years 440 .

5.4.6.Anti-fraud measures

Overall there is a limited scope for fraud in the types of actions carried out under the programme. The main concerns are project coordinators failing to honour obligations to other partners, and possible multiple submissions of project applications.

According to the programme legal basis, a system of controls has been in place to prevent fraud and irregularities at the EU and national levels 441 . The number of fraud cases reported is very low compared to the scale of the programme, with agencies proceeding to recover the necessary amounts 442 . The vast majority of National Agencies (71%) consider that the measures are fully sufficient 443 . Therefore the anti-fraud strategy in vigour is considered appropriate.

A few agency respondents made suggestions, such as using IT-tools to verify plagiarism 444 . As the programme moves to its later phases, more checks could be performed on the accumulated population of grants. In any case, the ratio of administrative burden compared to the low number of cases reported and the money recovered is to be carefully considered.

5.5.European added value

The evaluation highlights the strong European added value 445 of EU programmes in the fields of education, training and youth 446 .

The additional benefit of Erasmus+ was compared to what could be achieved at regional, national or other international level (see 5.5.1), to what was achieved by EU predecessor programmes (see 5.5.2) as well as to what would be the most likely consequences of discontinuing the programme (see 5.5.3).

The European added value of Erasmus+ has been informed by 6 evaluation questions and assessed against the following criteria that examined to what extent Erasmus+ supports:

·a scale of actions between countries which would not otherwise be achieved (volume effects);

·target groups or sectors that would not be otherwise covered (scope effects);

·processes which participating countries translate further into their own practice (process effects)

·cooperation between programme countries (integration effects)

·innovation which would not be otherwise mainstreamed so widely (innovation effects);

5.5.1.European added value as compared to what could be achieved at regional, national or other international level

Firstly, two evidence sources show clear added value of the programme due to its strong volume effects 447 , except in the cases of the Student Loan Guarantee Facility 448 , sport 449 and Jean Monnet activities 450 . The programme manages to reach out to substantially more learners, staff and organisations than all comparable programmes reviewed taken together, as well as groups that are much more rarely covered otherwise. The benchmarking analysis shows in particular that the scale of Erasmus+ is much bigger than that of other comparable schemes reviewed in the fields of education, training, youth and sport. There are only few other actions that support credit mobility and they are of a much smaller scale 451 . In terms of "market share", it is likely that Erasmus+ represents about half of credit mobility in EU higher education 452 . It is also likely that the share of Erasmus+ in VET mobility is significant, although available data is not of sufficient quality to calculate it 453 . Most respondents from National Agencies consider that Erasmus+ funds more than 75% of actions of a given type, as illustrated below 454 .

Perceived market share of Erasmus+ (number of responses)

Source: Survey of programme agencies

Secondly, three evidence sources point to the conclusion that Erasmus+ scope effects are significant 455 . Comparable national and international actions do exist (e.g. bilateral programmes, philanthropic actions, etc.) but these are more focused on mobility than on cooperation actions 456 . The Erasmus+ programme covers fields rarely covered by other international interventions except in higher education. Only in the latter sector was it possible for two-thirds of National Agencies to identify a number of comparable although much smaller programmes 457 . For all other fields or types of actions less than half of respondents could identify comparators 458 .

Erasmus+ has its own and unique boosting effect on mobility and cooperation in every field it targets. The findings from programme benchmarking 459 and agencies' survey (see fig. below) reinforce each other to provide a solid case in that respect. None of the comparable schemes combine as much as Erasmus+, in terms of country, sector as well as action coverage. Moreover with a dedicated strand (KA3) since 2014, the EU programme encompasses system level cooperation actions that are very rarely supported by other programmes.

In addition, the evaluated EU programmes show specific added value to the benefit of disadvantaged groups whose opportunities for mobility largely result from the EU intervention, more than from national initiatives. This is all the more true in the youth and sport fields where there are fewer comparable programmes 460 . Over two thirds of the agency respondents agreed with the notion that Erasmus+ offered more opportunities for disadvantaged groups in comparison to other similar schemes. In all comparable fields, Erasmus+ stands out as having the largest and most inclusive outreach 461 .

Comparison between Erasmus+ and other comparable initiatives

Source: Survey of agencies

Both volume and scope effects magnify the impacts of the programme as detailed above.

Thirdly, there are clear examples that the evaluated EU programmes have influenced other comparable programmes both in terms of process or types of actions supported.

At national level, Erasmus+ and its predecessors have contributed to shaping the framework for student and staff mobility or youth exchanges across Europe 462 . For instance, quality frameworks - in particular the concept of a European Quality Charter for Mobility introduced first for higher education and extended to VET due to its success - are also used for national schemes 463 . Best practices of EU programme management are often reported to be transferred to national programmes 464 . Other comparable actions sometimes follow the same rules 465 as Erasmus+. Generally speaking, among National Agencies which manage programmes other than Erasmus+, 71% of respondents acknowledge a spill over effect from the EU programmes under review to their own interventions 466 . These are good examples of unintended positive effects of the evaluated programmes. Furthermore, 91% of respondents to the public consultation strongly agreed or agreed that lessons learnt from the Erasmus+ actions (which they were most aware of) are being applied elsewhere. In particular, processes established to manage mobility have been mainstreamed into other national and European interventions.

Several spill-over effects on process have also been identified at EU level. Erasmus for young entrepreneurs shares comparable approaches with Erasmus+ mobility. Schemes launched recently, such as the European Solidarity Corps 467 or the EU AID scheme for volunteers in humanitarian aid 468 , follow similar selection principles as the European Voluntary Service (EVS).

Fourthly, as discussed in the effectiveness section, the network analysis showed that there are overall increasingly good levels of integration and cooperation between programme countries with some that were initially more peripheral moving to more central positions. Trends in interconnections show that the programme is not necessarily dominated by large countries. From a systemic perspective, the greatest added value comes from cooperation at the level of staff and organisations and that of policy makers and stakeholders. As regards the latter, many Open Method of Coordination activities organised in the context of ET2020, the EU Youth strategy and the EU Work Plan for sport are funded through Erasmus+. The EU programmes reviewed definitely enable cross-country cooperation and integration at a scale that is incomparable to other actions.

While its volume, scope, process and integration effects are substantial, the innovation effects of Erasmus+ appear to be more modest. This is the only type of added value analysed where the results are less positive. The evaluation of effectiveness showed above that while there are some examples of innovations that emanate from funded projects, these are rather ad-hoc, soft and of modest scale compared to the volume of projects funded rather than significant and mainstreamed. Although Erasmus+ has potential to enhance innovation (collaborative approaches, specific KA3, brand attractiveness), its added value cannot be said as emanating significantly so far from a role model in that respect.

5.5.2.European added value of Erasmus+ as compared to what was achieved by its predecessor programmes

Compared to its predecessors, Erasmus+ manages to reach out to more learners, practitioners and organisations. This scale potential results from the integration of several programmes into one and from a 40% increase in budget in comparison to 2007-2013. This increase has not yet fully materialised as most of it will be observed as of 2017. Meanwhile the integration has generated efficiency gains 469 .

Through its well-known branding, the current programme benefits from a better visibility in media/social media 470 , but also in policy making accompanied with a strengthened alignment with policy priorities 471 . The fragmented nature of its predecessors did not give such a visibility. Increased attention has been continuously paid to synergies between the programme and policy priorities in education, training, youth and sport. In particular, the introduction of a Key Action devoted to system level support (KA3) is a novelty valued by most respondents concerned. It is nevertheless too early to evaluate whether this programme will have achieved stronger system level changes.

Thanks to the integration of several programmes, the degree of cross-sectoral cooperation has increased compared to the period 2007-2013. This is mainly due to the rising participation of higher education organisations in other sectors and to a lesser extent to some increase in participation of companies and public authorities.

The Erasmus+ programme has also been designed to pay more attention than previously to information about performance, which helps to allocate the budget more appropriately. The monitoring of Erasmus+ is much more developed than in the past. The dissemination and exploitation of project results is underpinned by a clearer strategy and more systematic publication on line. This represents actual progress, even though in both cases newly set-up tools are not all as user-friendly as expected and though data remains insufficiently analysed 472 .

The international dimension, where it exists (i.e. in higher education and youth), has been made clearer with Erasmus+. The integrated programme has addressed the fragmentation issues raised in the interim evaluations of its predecessors (Erasmus Mundus, Tempus, Alfa). There remains room for improvement notably in the management, visibility and communication with regard to international actions.

From one programme generation to the next, the question of repetitive actions has to be assessed 473 . Here again added value is reflected in the differentiated level of repetitions. Each year, EU programmes reach out to a new cohort of learners (i.e. though mobility is repetitive it addresses different audiences each year), whereas the repeated participation of practitioners and organisations in the funded activities over time enables the respective actions to achieve deeper effect.

Overall, most of respondents to the Open Public Consultation value Erasmus+ for its undisputed benefits 474 . More than 95% of respondents agreed or strongly agreed that it is funding activities which would not have been made possible otherwise, and that it contributes to improving what is already available at other geographical levels for the education, training, youth and sports sectors 475 .

Most of the National Authority reports confirm the above findings by stating that Erasmus+ and its predecessor programmes produce effects that could not have been attained through actions at national level alone. Country reports confirmed by the public consultation responses highlight the impacts of Erasmus+ on mobility within the EU and beyond, cooperation opportunities and internationalisation of organisations, quality improvements based on learning from others, as well as the promotion of common European values and intercultural awareness. Erasmus+ added value is more strongly praised in the youth sector, while in the sports field, the European added value is rather acknowledged where activities go beyond professional sport, such as cross-border cooperation between grassroots-level actors  476 .

5.5.2 European added value as compared to what would be the most likely consequences of discontinuing the programme

The absence of Erasmus+ would result in a steep decrease in the volume, but also in the quality, equity and efficiency of both mobility and cooperation among the programme countries and worldwide 477 . This would entail in turn a negative impact on education, training, youth and sport sectors, as detailed under effectiveness above.

As shown in the efficiency section, Erasmus+ is managed particularly efficiently due to economies of scale. A decline in efficiency would result from the inevitable downscaling of coordinating structures. According to National Reports, collaboration between institutions and organisations at European level would be more complex to realise without Erasmus+. If National Agencies 478 , European networks with central support function and European collaborative platforms would not exist, the efficiency of comparable schemes, not to say their mere existence, would probably be put at risk in many countries.

Both for mobility and cooperation actions, the absence of Erasmus+ might be compensated in certain countries and sectors (mainly higher education) only in scaling up other comparable schemes. However, when asked in a Delphi survey about plausible alternatives to Erasmus+, the experts consider that these partial and uncoordinated compensatory initiatives would be highly unlikely to reach the scale of Erasmus+ 479 . To achieve the full range of its results, a greater variety of actions than the three main types of actions under Erasmus+ would be needed. No single alternative would result in comparable results.

Unequal access to mobility and cooperation abroad would once again become a reality, as was the case when these EU programmes did not exist. National Reports note that participation in alternative activities would probably be linked more directly to organisational or local resources. Therefore, differences in international opportunities between countries as well as between sectors would increase as some would not be able to afford funding international activities from national funds. As a consequence of the above, differences would also increase between wealthy and less wealthy organisations or individuals, as confirmed by the monitoring survey among higher education learners 480 and the Delphi survey among experts 481 .

This in turn would lead to a decline in positive adherence to the European project, especially among young people, and a much weaker international outlook of staff. It would also marginally negatively affect their transversal skills, especially foreign language acquisition, though other channels to develop these skills would continue being used. Over time the connections between organisations and staff would fade out, which would eventually have a knock-on effect on learners and outcome recognition. Together this would lead to much smaller international outlook of education, training, youth and sport sectors, including beyond the EU, which would generate negative externalities given the globalised nature of modern economies. In other words, bringing Erasmus+ to an end would leave negative effects way beyond its direct beneficiaries.

In a nutshell, it might be worth quoting the High Level Group on Education and Training concluding under the Estonian Presidency of the EU Council that "The Erasmus+ programme and its preceding programmes have 'created the first generation of young Europeans and therefore have the highest possible added value for Europe as well as the potential to promote common European values, empower young people, tackle socio-economic challenges and foster competitiveness."  482

6.Conclusions

Overall, the programmes evaluated have proved to be highly effective. Erasmus+ appears more coherent, relevant and only partly more efficient than its predecessors.

The external evaluator made 11 recommendations 483 to the Commission to address weaknesses of the current Erasmus+ programme. These recommendations will be followed-up as stated in the Commission report on the mid-term evaluation of the Erasmus+ programme (2014-2020).

·Five recommendations are cross-cutting (with a view to boosting respectively: inclusiveness for more vulnerable groups; strategic investment in the sectors with highest potential performance (schools, VET, youth); policy prioritisation; relevance of innovation potential at sector level; systemic impact of projects through further involvement of policy makers);

·Four recommendations concern specific areas (aiming at addressing the relative weaknesses of the Student Loan Guarantee Facility; Jean Monnet activities; adult education; sport);

·Two recommendations are more process-oriented (aiming at simplification and better monitoring).

Effectiveness

The current programme and its predecessors deliver a unique package of outcomes in the field of education, training, youth and sport. When concerns are expressed, these relate to specific parts of the programme or to certain modalities of disbursement.

At mid-term, Erasmus+ is on track to achieve or to exceed most of its performance indicators, as set in the legal basis. Beneficiaries report above 90% satisfaction rates for learners and even higher for staff. In the period 2007–2016, all programmes reviewed have supported together more than 4.3 million learners and 880,000 practitioners as well as 940,000 organisations. In terms of volume of outputs, the main exception concerns the Student Loan Guarantee Facility. Even if the feedback received from its first users has been positive, it has not yet lived up to volume expectations due to delays of its launch, low take-up among financial institutions and lack of awareness among students.

The evaluation went beyond these indicators, which are not all equally relevant. A counterfactual approach (quantifying the contribution of the programme in measuring the difference in results between beneficiaries and non-beneficiaries) was used for the first time across the programme, making this evaluation very reliable and more valid than previous ones. However such an approach cannot exclude the possibility that the difference in outcomes between beneficiaries and non-beneficiaries could be partially due to the selection into the programme (in attracting the most motived individuals for instance) rather than purely the effect of the programme. Based on this very sound methodology, the evaluation identified a broad range of results and impacts at the level of learners (with particularly outstanding effects of mobility), practitioners, organisations (in particular their internationalisation, including beyond the EU) and also, to a certain extent, systems and policies (fostering the recognition of learning outcomes across borders).

At individual level

For most individuals, including when compared with the situation of non-beneficiaries, their experience abroad has contributed not only to their skills' development and future career prospects but also to deep changes to their personal development. The programmes evaluated have stimulated networking among both learners and staff. Finally, they have contributed to the development of openness to other cultures and positive attitude towards the European project. Yet the evaluation points to the need to do more to reach out to the more vulnerable in society and to facilitate the participation of smaller-size organisations.

At organisational level

The programmes reviewed have demonstrated a range of soft effects on organisations, particularly a clear internationalisation of organisations, in and outside the EU. The changes at organisational level are however progressive and small scale and continued participation is needed for deeper transformations.

At system level

In the long run, the programmes have led to a cultural shift in Europe in the way learning mobility abroad is perceived and its learning outcomes are validated and recognised. The programmes evaluated have also been important for increasing the EU's global outreach.

Though less visible, the evaluation also confirmed that the programmes reviewed have had a direct impact on education, training, youth and sport policies where the related budget was large enough or had an indirect effect by funding policy cooperation between authorities. The system level changes through critical mass are much clearer in the higher education sector than in other sectors which receive comparatively much less funding.

Systemic effectiveness is less in evidence for Jean Monnet, adult learning and sport, where funded EU actions lack the critical mass and/or a sufficiently relevant focus. The evaluation underlines in particular that the impact in the adult learning sector is diluted due to the fragmented and diverse nature of the sector.

More generally, the resources for mainstreaming project results are spread across too many topics at EU level and are insufficient at national level. Although local innovation (producing something new for participating organisations) is achieved by the programme, this innovation does not enough follow the state of the art developments in a given sector. Lastly, the evidence of the exploitation of project results by policy makers is not always clear, especially when the latter are not engaged in the project from the beginning.

Coherence

The evaluation found a high external coherence between Erasmus+ and other relevant EU policies and programmes (e.g. European Social Fund, Horizon 2020). Although the level of synergies differs, it is notable that the evaluation detected very few overlaps.

The internal coherence of the programme resulting from the lifelong learning coverage is strong as well. The further integration of fields often kept separate at domestic level into a single EU programme is increasingly perceived as facilitating complementarities and international outlook. This consistent scope sends the message that the learning opportunities offered are equally important for the development of people, be it through formal or non-formal and informal learning.

There is a strong degree of cross-sectoral cooperation within Erasmus+, which has increased sharply compared to predecessor programmes. The majority of cooperation projects in all sectors include at least one organisation which can be considered as coming from another sector. There is an overall consensus that an integrated programme boosted the promotion of the actions and the visibility of the different sectors.

Coherence can still be improved in relation to sport in reducing overlaps with non formal education activities.

Relevance

The evaluation showed that Erasmus+ is better aligned with EU policies than its predecessor programmes and entails sufficient flexibility to adapt to EU-level emerging needs. However, to maximise the programme's impact, the evaluation recommends that priorities at selection stage be reduced and better focused.

The contribution of Erasmus+ to a more cohesive Union is also clearly evidenced. The programme reaches out to disadvantaged people more than previous EU programmes or comparable national schemes, although more still needs to be done to reach out to the more vulnerable in society as outlined above. The geographical balance has also improved with small countries and countries from Central and Eastern Europe being better integrated than in the predecessor programmes.

The evaluation also noted that there is potential for introducing better targeted actions to maximise the relevance of Jean Monnet activities. Teaching and research on EU matters is now relatively wide-spread in higher education. In contrast, there is a need to strengthen the understanding of European integration and sense of belonging to Europe amongst the youngest generations.

Efficiency

The economies of scale linked to the integration of several programmes did not deliver all efficiency gains that had initially been anticipated in 2011 based on a "one agency-one country" model which was not adopted by the legislator in 2013. Nevertheless, the costs of programme management appear reasonable (6% of Erasmus+ administrative and operational budget) and, in any case, lower than for similar programmes at national level (14% in average). More efficiency gains are expected to materialise during the growing phase foreseen by the budget profile of Erasmus+ until 2020.

The hybrid combination of different programme management modes and bodies is fit for purpose with a good overall coordination and no major inefficiencies identified. Through decentralised actions, the programmes get close to their target audience and offer the possibility to align with national priorities. Very positive feedback was received on the support provided by and to National Agencies. Centralised actions align more with EU level priorities and help to respond to urgent political priorities. Some National Agencies report a need for more communication with the Executive Agency (EACEA) to maximise synergies between centralised and decentralised actions.

The first two years of Erasmus+' implementation were, however, very challenging for all players. This was mainly due to the late adoption of the Erasmus+ Regulation (end of 2013), coupled with the necessity to implement the majority of the actions under the new architecture and with a new generation of IT tools already by the first trimester of 2014, which significantly reduced the time for the Commission, the National agencies and the Executive Agency to put in place the necessary normative framework and technical infrastructure to implement the programme smoothly. National agencies and authorities have recognised that initial challenges have been gradually overcome and that there has been a steep learning curve with substantial efficiency improvement over the period 2014-2016. Despite this difficult start, the programme has now reached its cruising speed.

Simplification

The integration has led to greater simplicity of programme architecture, notwithstanding the separate chapters for Jean Monnet and sport, which is beneficial for both beneficiaries and those in charge of management. It also makes programme monitoring more comprehensive and clearer, although there is room for a smarter collection and a better exploitation of data, enhancing transparency and accountability. The counterfactual approach (results compared between beneficiaries and non-beneficiaries) should be pursued and developed for evaluating the next programme.

The procedural burden on beneficiaries (application, reporting, etc.) is the main recurrent issue. Many agree that Erasmus+ has considerably simplified a number of processes (simplified grants, digitalisation, VET Charter, linguistic support, etc.). However, procedures and IT management tools impose a burden on beneficiaries which is not always proportionate to the grant they receive.

By contrast, the evaluation notes that the simplification process may have gone too far as regards cooperation projects. Applicants for small strategic partnerships are too often expected to meet the same requirements as applicants for large ones. Furthermore, it is hard to clearly assess the cost-effectiveness of these actions considering their relatively average level of innovation. There is not enough differentiation made in the way of handling smaller (mainly collaborative) and larger (innovative) projects. The evaluation recommends simplifying the application forms, reviewing the award criteria to better reflect key success factors for effectiveness and strengthening the review at mid-term in particular for bigger projects.

Another challenge hampering efficiency is linked to the funding for international higher education, which remains managed separately for each global region in a too rigid manner.

European added value

30 years after its beginnings in the field of higher education, Erasmus+ has become a flagship programme of the EU. The programme is highly valued by the general public as well as by all stakeholders. It is consistently identified by citizens as one of the three most positive results of European integration.

Erasmus+ remains unique. Whereas there are other schemes funding comparable actions at national level, these are much smaller both in volume and scope. It is unlikely that alternative measures would be able to compensate for Erasmus+ funding, demonstrating its undisputable European added value.

For the future, without prejudice to negotiations on the next Multi Financial Framework, the evaluation concludes that it would be relevant to reconsider the overall budget. It also suggests modifying the share-out between the programme sectors, whereby potential increases could be directed to sectors showing the highest performance, but which have, until now, received relatively less funding. School education and vocational education and training (VET), where the impact is not yet as widespread as in higher education, were identified as having the most promising potential for an expanded participation in Erasmus+ activities in the coming years. The unmet demand and the potential for maximising effectiveness call for a stronger investment for the future development of the programme.

Annex 1: Procedural information

1.Lead DG, Decide Planning/CWP references

·DG Education, Youth, Sport and Culture (DG EAC)

·EAC/2015/014

2.Organisation and timing

This evaluation has been steered by DG Education, Youth, Sport and Culture since July 2017 484 under the scrutiny of an inter-service group (ISG) comprising of representatives of DG BUDG, DEVCO, EAC, EEAS, EMPL, FPI, HOME, JRC, NEAR, SG, as well as EESC.

The ISG was consulted at each stage of the evaluation process and reviewed each deliverable produced by the contractor as well as this Staff Working Document.

ISG meetings took place on:

·9 September 2015 (Kick-off),

·18 July 2016 (Inception report);

·21 November 2016 (Interim report 1);

·24 March 2017 (Interim report 2);

·10 July 2017 (Draft final report);

·11 September 2017 (Final report)

The ISG was consulted on the draft Staff Working Document on 29 September and informed about the Regulatory Scrutiny Board's opinion on 27 November.

3.Exceptions to the better regulation guidelines

None

4.Consultation of the RSB (if applicable)

The Regulatory Scrutiny Board was consulted on 16 October and met on 15 November 2017.

The Board acknowledged in its opinion with comments 485 the significant efforts of data and evidence collection. It also noted the good methodology. The Board gave a positive opinion, with comments to improve the document. All have been addressed.

The following changes have been made to this Staff Working Document, in response to the Board’s main considerations:

5.Evidence, sources and quality

Sources of evidence

The following box provides an overview of the data sources from which the evidence was drawn. A detailed description of the individual methods is provided in Annex 3.

Expert advice

National Reports

Programme countries have submitted their own evaluation reports on the implementation and impact of decentralised actions of Erasmus+ in their respective territories, as required by Erasmus+ legal basis. Their findings are summarised in the National reports synthesis (NRS) drawn by the external evaluator 486 .

Expert Panel

Erasmus+ and predecessor programmes fund a high number of cooperation projects which result in outputs (e.g. handbook, toolkits, methodologies, etc.) that are aimed to be used within the organisations that took part in the project but also beyond. To get a better understanding of their quality and dissemination potential, an expert panel reviewed collectively the actual outputs of a selection of 100 cooperation projects.

External consultant

The external evaluator (ICF) contracted for this assignment 488 has carried out since May 2016 all tasks as required under the scrutiny of an inter-service group (ISG) and the daily steer of DG Education, Youth, Sport and Culture.

Primary data were mainly collected from November 2016 to May 2017.

The only significant change compared to the initial work plan was limited to a two-month delay of the (sub) contractor in launching the beneficiary surveys 489 .

Annex 2: Stakeholder consultation synopsis report

This evaluation has drawn on data from direct consultations with various stakeholder groups involved in the implementation of the Erasmus+ and predecessor programmes.

A4.1 Overview of main stakeholders consulted

The table below provides an overview of the stakeholders consulted over the lifetime of the evaluation through: semi-structured interviews (for the purpose of the key informant interviews (KIIs) and the case studies), online surveys and an open public consultation. Other consultation activities are described at the end of the section.

Table 0.Overview of stakeholder consultation 

Source: ICF

Based on the above, 68,675 stakeholders’ views were collated and analysed among which:

·66,383 through the seven above-mentioned parallel online surveys 491 among which 47,815 through the sole beneficiary and control group surveys

·492 through interviews

·1,800 through the OPC

As the table also reflects, the range of stakeholders consulted was broad, encompassing both direct and less direct beneficiaries of the programmes at the individual (learners and practitioners), organisation and system level comprising (not in specific order):

·general public including a variety of key stakeholders active in education and training, youth and sport with different levels of knowledge and experience with Erasmus+ and/or predecessor programmes: addressed through the OPC 492

·all current programme target groups, both for KA1 (mobility) and KA2 (organisational cooperation): addressed though the surveys of beneficiary learners and staff and related control groups

·socioeconomic actors (i.e. companies, public authorities, civic organisations, sectoral bodies, etc.): addressed through the socioeconomic actors survey

·project assessors (i.e. experts contracted by the EACEA and/or the NAs to assess project applications and reports): addressed through the experts survey

·NAs and EACEA staff members: addressed through the programme agencies survey

·Selected EU and national level 493 key informants involved with programme(s) implementation (EU level) or drawing on it and/or its predecessors: key stakeholders/ key stakeholder organisations representatives, EC officials (EC and agencies (EACEA and ETF), ministries’ representatives (policy-making) 494 . Those were addressed through the Key informant interviews (KIIs)

·Selected national level 495 informants benefitting from the programme(s) in the three fields above comprising: beneficiary organisations’ leadership, practitioners and learners, targeted funded project leaders, other key stakeholders (where applicable). Those were addressed through the case studies

Next to the consultation methods above that were designed and implemented to collate stakeholders’ opinions on the current programme’s (and/or predecessors where appropriate) performance against all or several of the main evaluation criteria, the evaluation team also undertook ad hoc stakeholders consultation. This served to inform the evaluation of specific part or activities of the programme namely: Jean Monnet programme 496 and the Student Loan Guarantee Facility (SLGF) 497 . In each case, the consultation methods comprised tailored surveys and interviews. For an overview of the types of stakeholders reached in both cases, see table above.

Besides, to gain complementary insights and to ultimately assess the difference in results between Erasmus+ beneficiaries and their counterparts who did not take part, control groups of non-beneficiaries (targeted at practitioners 498 and learners 499 ) were set-up.

Overall, the mix of consultation activities described above enabled the evaluation to effectively address a relevant breadth of stakeholders. This in turn enabled the evaluation team to gain insights from a range of relevant key players on different and meanwhile complementary dimensions of the programme(s)’s performance and/or on suggestions towards the future programming period.

With regard to the representativeness of the data, sample sizes (more particularly applicable to the main surveys) and targets had been agreed upon by the ISG during the inception stage of the evaluation study. In practice, the number of responses received and processed have been on or above targets in most cases. In the case of the beneficiary and control group surveys the satisfactory response rate offered to the evaluation a solid and reliable basis for data comparison amongst beneficiaries and sectors. A noticeable exception though was in the sport and adult education fields where the control groups were too small. For details on the actual response rates against minimum sample sizes defined for the above-mentioned main surveys’ target and control groups see evaluation final report, volume 1 (section 3).

Concerning the other surveys, the minimum sample sizes 500 were attained. In the case of the programme agencies, no ad hoc target had been fixed, but all programme countries were covered.

In the case of the OPC, the number of total responses received and this of exploitable ones (i.e. complete responses) was considered statistically reliable.

With regard to the key informant interviews (KIIs) undertaken to inform the main evaluation report as well as accompanying standalone reports, the number of interviews foreseen at the start of the evaluation process was attained in all cases. The main types of pre-identified stakeholders were also consulted.

The same applied to the case studies (i.e. at organisational level (spread across all education and training, youth and sport fields), at system level and focusing on Jean Monnet). With the exception of one case study initially foreseen that could be not carried out, all the others were effectively conducted either on-site (organisation level case studies) or over the phone (system level and Jean Monnet ones).

In addition to the above, other consultation related activities are worth being mentioned. These have not been listed in table 1:

·Expert panel assessment of project outputs: experts in education and training, youth and sport fields were sub-contracted by ICF to assess a number of selected project outputs and take part in a virtual expert panel assessment

·Delphi survey of experts: experts in the fields above were sub-contracted by ICF to take part in the survey that whose main purpose was to inform the analysis of the EU added value and cost-effectiveness of the current programme

·Informal workshop organised by two EU-level organisations (EUCIS-LLLP and the European Youth Forum -EYF) in May 2017, prior to the closure of the OPC. It was agreed with DG EAC that the event could be an opportunity for ICF to grasp additional insights from key stakeholders ideally before OPC closure.

A4.2 Consistency of results across consultation activities

The table below presents key results per consultation activity, organised by evaluation criteria, as well as the level of (1) consistency of results across consultation activities and (2) complementarity of results across consultation activities.

Overall, there was a large degree of convergence in results from the different consultation activities.

Table 1.Main results of the consultation processes and level of their consistency and complementarity

Annex 3: Methods and analytical models

As the differences in the architecture, objectives and activities of the current and predecessor programmes 501 made it difficult to make a straightforward comparison between their respective actions, the current evaluation did not look at programmes along their own types of actions but through a customised matrix. To ensure comparable approaches, the evaluation framework developed by the external evaluator 502 classifies actions across the three levels of intervention of the current programme (see fig. below) and uses that as the basis of comparison, namely:

■System level

■Organisation level

■Individual level divided into actions for learners and practitioners across all different areas (education and training, youth and sport) and sectors (school (SE), vocational (VET), higher (HE) and adult education (AE)). For the purpose of the evaluation, the overarching terms of Learners and Practitioners are used 503 .

Summary of analytical techniques used

Desk research and interviews

An extensive literature review was carried out across varied sources. In addition to the contextualisation of other data, the main purpose was to review research notably on programme outcomes, including success factors and obstacles to learning mobility and transnational cooperation in the fields of education, training, youth and sport, mainly encountered through predecessor programmes. This helped to inform whether and where the programmes have had impacts at the individual, organisational and system levels and was central to the analytical framework underpinning this evaluation. A full account of the literature reviewed is annexed to the contractor's report. External statistics were also used to put the programme in perspective 504 .

Key informant interviews (KIIs) have been conducted at both national and European level targeting EC/agencies officials, EU key stakeholder organisation representatives, national policy makers and key stakeholders in 15 selected countries, as well as managers of comparable programmes for benchmarking purpose. All were selected using category based purposeful sampling  505 .

National reports

Programme countries have submitted their own evaluation reports on the implementation and impact of decentralised actions of Erasmus+ in their respective territories, as required by Erasmus+ legal basis. Their findings are summarised in the National reports synthesis (NRS) drawn by the external evaluator 507 . These reports involved the review of documentation and statistics concerning the programme and related policies and ad hoc consultations with relevant stakeholders. The most often used data collection methods were interviews and surveys (both used by 25 out of 34 countries). In addition 11 countries used focus groups.

Open Public Consultation

The objective of the Open Public Consultation (OPC) was to gather the opinions, including forward-looking perspectives, of the general public and various stakeholders, on all evaluation criteria 509 . Different (closed and open-ended) questions were asked depending on the level and scope of knowledge of the programmes declared by each respondent. Most respondents were either organisations or practitioners. In any case, consultation findings can never be representative due to the selection bias inherent to any open recruitment, but they can be triangulated with other sources to inform the evaluation. Alongside their responses, the OPC respondents submitted 24 position papers, which are included as well in the synopsis drawn by the external evaluator 510 .

Social media analysis

Social media analysis was used to collect evidence about the awareness on the programme and its public image, globally and by sector. All relevant posts identified 512 were analysed for trends in the volume of conversation about Erasmus+, audience profile, and content of discussions both in terms of topic and sentiment. The volume of information shared was compared with the volume for other EU programmes (e.g. Marie Curie, MEDIA, Citizens programme).

Benchmarking

Erasmus+ was benchmarked across the different evaluation criteria against 18 selected comparator national and transnational mobility and cooperation programmes 513 . Three different types of benchmarking were undertaken: Strategic benchmarking helped to compare the relevance, coherence and added value of the programmes. Process benchmarking compared efficiency and effectiveness of their management. Financial benchmarking assessed the respective financial efficiency and effectiveness of the programmes compared. The mapping was not expected to be comprehensive but to inform about the scale and profiles of comparable initiatives 514 . The large difference in scope and size made comparison between programmes fairly complex. The contractor only had access to information available in the public domain, which means that in a few cases, it was not possible to collect data for some indicators.

Programme data

The evaluation used the wealth of the programme data available as regards inputs (funding) and outputs (numbers of projects, numbers of beneficiaries, etc.). Moreover data on certain results (e.g. satisfaction-rate) is available since 2014 directly from the management or monitoring tools set up for Erasmus+.

Overall the quality of data was good for the predecessor programmes and even very good for Erasmus+. This shows improvement made in terms of monitoring since 2014 (see table of monitoring tools below) 515 , albeit with certain limits 516 . The quality and completeness of the data 517 was best for the Erasmus+ programme, followed by the LLP and Youth in Action 518 . Fewer data was available for Erasmus Mundus and Tempus and even less for the remaining higher education international programmes 519 and sport pilot actions 520 . The main issues affected the analysis of participation patterns due to an incomplete coverage of data on i) organisations 521 ; ii) participants with few opportunities and disadvantaged backgrounds 522 .

Overview of Monitoring arrangements

The monitoring surveys of beneficiaries carried out by DG EAC concern all learners or practioners who take part in mobility under KA1 over 2014-2016 523 . The reliability of the data is strong. Given the sample sizes (first two years of the programme) the data is considered to be strongly reliable even though it was too early to include 2016 data. This being said most questions are based on the self-perception of beneficiaries 524 in contrast to the more objective OLS 525 assessment of progress in language proficiency for instance; even for very similar questions the responses offered vary across sectors affecting the comparability across target groups.

In addition to descriptive and multivariate statistical analysis of programme data, a social network analysis showed more specifically how organisations cooperate and participate in Erasmus+ and assessed in particular cross-sector and trans-national cooperation. It is to be noted that the analysis of repeated participation in the programme could not be pursued, as it would have required substantial data cleaning across previous programmes' data sets (due to multiple names used for the same organisations prior to 2014).

The analysis of programme data has been enriched with other secondary data: contextual indicators from the Education and training monitor, the Youth indicators, relevant Eurobarometer studies or the Sport indicators available on Eurostat. Furthermore, external data sources on education and training institutions with high share of disadvantaged students have been compared across a sample of countries with the programme data to identify the schools that have (or have not) engaged with Erasmus+.

Most result indicators are based on self-perception and the link between the survey questions and the indicator measured can sometimes be weaker. That is why monitoring data were complemented through primary data collection such as surveys.

Surveys of beneficiaries and control groups

To collect more comprehensive, objective and comparable quantitative data on programme results, the external evaluator has carried out ten specific surveys of beneficiary learners and staff and related control groups covering all programme target groups, both for KA1 (mobility) and KA2 (organisational cooperation) 528 . The Erasmus+ monitoring survey shows a subjective valuation by beneficiaries themselves of their experience, whereas the external evaluation surveys show more objectively, through factual questions whether the beneficiaries demonstrate different attitudes, competences or beliefs than a control group that was asked the same questions. As often as possible the questions used in the survey had been previously tested in large scale surveys such as PISA, TALIS or Eurobarometer and in any case were as often as possible factual, so as to enable a comparison with the control group of non-beneficiaries. This means that to strength the validity of surveys statements about self-perceived contribution of the programme were as often as possible avoided. The latter surveys were disseminated to all contactable beneficiaries of Erasmus+ as well as predecessor programmes 529 . Overall findings could be generalised to the whole programme with sufficient confidence because the sample sizes for all surveys were large enough in all cases to make judgements and the distribution of various background variables within the survey samples was assessed as acceptable 530 .

In addition to the standard tests of significance and regressions, a quasi-experimental approach was used to assess the contribution of the programme to the results measured 531 . As a proxy of counterfactual assessment 532 , control groups of non-beneficiaries were set up 533 with similar profiles to compare their responses with those made by respondents who participated in the Erasmus+ programme. This has been achieved in matching respondents in the treatment group (i.e. Erasmus+ beneficiaries) with ‘similar’ individuals in the control group, to come up with a ‘matched sample’ where subjects are alike in some background characteristics, called covariates, such as gender, age, etc. 534 . Overall the comparability was considered satisfactory even if there are some minor though statistically significant differences on certain surveys and certain variables. For most variables there were no significant differences between the control group and the treatment group and where they were these concerned variables that were not likely to strongly influence the findings. In other words, this might not be considered as genuine counterfactual assessment, as the design cannot exclude that the differences in result variables measured between the beneficiaries and the control group would be related to the selection into the programme rather than the programme. Nevertheless it does control for various characteristics of both beneficiaries and non-beneficiaries when judging the differences in results. Most control groups reached a significant size allowing comparison with beneficiaries, except in sport and Adult education sectors where the control groups were too small 535 .

In addition to the above ‘post’ surveys (i.e. beneficiaries were surveyed after they took part in the programme), the external evaluator carried out pre-surveys in two sectors where short term mobility exchanges apply (i.e. school pupils and Youth exchanges) for the purpose of pre/post comparison. This allowed - for the school sector only 536 - to draw conclusions on selection into the programme and to observe changes during the time of the mobility experience. The sample sizes suffered from a high attrition from the pre to the post survey, especially for the youth sector where comparison of profiles prior to Erasmus+ mobility could not been done. Pre-post results for the youth sector have therefore not been interpreted considering the high likelihood of sample bias.

The surveys of learners and staff were complemented with a survey of socio-economic organisations taking part in the programme: companies, public authorities, civic organisations, sectoral bodies, etc.

Other surveys and interviews targeting beneficiaries of specific activities were carried out in the case of Jean Monnet actions and the Student Loan Guarantee Facility (SLGF) 537 .

Surveys of staff involved in the programme implementation

National agencies and EACEA, as key managing bodies in the implementation of the programme, were subject to a dedicated programme agency survey, especially useful to inform efficiency of the programme management.

To complement data on projects, in particular about the quality of applications and final reports but also the efficiency of the selection process, a survey of project assessors was also carried out. They are experts assessing project applications and reports.

Lastly, a so-called Delphi survey was disseminated among these experts to collect project assessors views about the added value of the programme and thus to provide inputs into the analysis of cost effectiveness. They were asked for each sector to elaborate about hypothetical impact of dismissing Erasmus+ and about alternatives 539 .

Case based and qualitative data collection

On a qualitative note, the case studies served to complement insights gained from key informant interviews and surveys, in particular about how the programme generates organisational and system level changes including spill over effects from individuals who are direct beneficiaries to other individuals and to organisations. Case studies were selected using a combination of criterion-based selection and random selection.

The most significant change approach was applied to case study interviews. It is based on open ended questions which ask beneficiaries to give account in their own words of the difference the programme makes to them. Indeed, the surveys measure always predefined results from the perspective of the programme design. However the participants themselves may judge how the programme has benefited to them differently.

Erasmus+ and predecessor programmes fund a high number of cooperation projects which result in outputs (e.g. handbook, toolkits, methodologies, etc.) that are aimed to be used within the organisations that took part in the project but also beyond. To get a better understanding of their quality and dissemination potential, a review of selected project outputs, mainly collected from national agencies, was carried out. Given that only relatively few Erasmus+ funded projects were completed at the time of this evaluation, the sample is somewhat biased towards predecessor programmes. All qualitative fieldwork was based on a standard protocol: the outputs were captured in standardised grids, reviewed and where appropriate enriched with feedback asked by interviewers. Lastly, an expert panel reviewed collectively the actual outputs of cooperation projects.

Annex 4: Evaluation criteria and evaluation questions

Source: Terms of Reference for Mid-term evaluation of Erasmus+ (January 2016)

Effectiveness

·To what extent have the various programme fields (both within Erasmus+ and its predecessor programmes) delivered the expected outputs, results and impacts? What negative and positive factors seem to be influencing outputs, results and impacts?

·What are the long-term impacts of the predecessor programmes? We are interested in the impact of all actions/elements of the predecessor's programmes, in particular those actions/elements that are continued in new Erasmus+ programme. We are also interested in impact of actions/elements that have discontinued to the extent that it might help to design the future programme.

·Please identify, describe and quantify (if possible) the spill over effects between various actions (clusters of actions) of the Programme.

·What are the differences in impact of programme actions on specific disadvantaged groups of the population who traditionally do not engage in transnational activities as compared to other groups that benefit from the programme?

·Are there positive /negative effects that existed in the previous programmes, but that no longer exist with the new programme?

·What conclusions can be drawn on the likely impact of Erasmus+ programme given the fact that significant parts of their actions are continuation of predecessor's programmes?

·What have been the unintended effects and their magnitude of the Erasmus+ and the predecessor programmes if any?

·To what extent are the effects likely to last after the intervention ends?

·How effective are the forms of cooperation and the types of actions under the Programme for the purpose of supporting the political priorities? Which fields and actions of the Programme are the most effective considering the needs?

·How and to what extent is the existence of one integrated Programme a help or a hindrance in promoting greater effectiveness and visibility for the EU's activities in the field of education and training, youth and sport, compared with a sector-specific or target group-specific approach to Programme design, using sector-specific action names (e.g. Comenius, Leonardo, Grundtvig)?

·To what extent are the Programme results adequately disseminated and exploited?

Relevance

·To what extent are current and emerging key socio-economic needs and challenges that Europe is facing reflected in the policy priorities, objectives and actions/activities of the Programme?

·To what extent are needs of different stakeholders and sectors addressed by Programme objectives? How successful is the Programme in attracting and reaching target audiences and groups within different fields of the programme's scope? How well is the Erasmus+ programme known to the education and training, youth and sport community? In case some target groups are not sufficiently reached, what factors are limiting their access and what actions could be taken to remedy this?

·To what extent is the design of the programme oriented and focused towards the hard-to-reach groups or specific disadvantaged groups of the population who traditionally do not engage in transnational activities? In case some target groups are not sufficiently reached, what factors are limiting their access and what actions could be taken to remedy this?

·Based on the analysis of the impact of predecessor programmes, are there any elements that have been discontinued and could have a possible value added in future generation of programmes?

Coherence

·To what extent are the objectives of different programme fields within Erasmus+ programme consistent and mutually supportive? What evidence exists of synergies between the different programme fields and actions? How well do different actions work together? To what extent there exist duplications, overlaps, or other disadvantageous issues between the programme fields and how are they dealt with?

·To what extent are the centralised and decentralised actions coherent? How do they interact / complement each other?

·To what extent is the Erasmus+ programme coherent with relevant EU policies and programmes with similar objectives such as Creative Europe Programme, European Structural and Investment Funds (ESIF) or other EU's funding programmes such as Horizon 2020, financial instruments relating to justice and citizenship, health, external cooperation or pre-accession assistance programmes? To what extent has the Erasmus+ programme proved complementary to other EU interventions/initiatives in the field of education and training, youth and sport?

·To what extent does the Erasmus+ programme design (including needs, objectives, activities, outputs and effects) provide appropriate links and support to the EU policy agendas, in particular policy initiatives and political priorities within the EU 2020 Strategy, the Education and Training Strategic Framework ET 2020, the European youth strategy or the EU policy priorities in the field of sport?

·To what extent is the Erasmus+ programme coherent with various interventions pursued at national and international level which have similar objectives? To what extent has the Erasmus+ programme proved complementary to other Member States' interventions/initiatives in the field of education, training, youth and sport?

Efficiency

·What is the cost-effectiveness of various actions (clusters of actions) of Erasmus+ programme and its predecessor programmes? How the relative costs and outcomes (effects) of various programme actions compare within and across the programme fields? Provide quantifications.

·To what extent is/was the size of budget appropriate and proportionate to what the Programme and its predecessors set out to achieve? To what extent is/was the distribution of funds across the programme fields and Key Actions appropriate in relation to their level of effectiveness and utility?

·What is the prospect of other policy instruments or mechanisms in providing better cost-effectiveness ratio?

·To what extent is the implementation and management structure of centralised and decentralised actions appropriate, efficient, and well-functioning? How efficient is the cooperation between the different management bodies (Commission –Executive Agency –National Agencies –European Investment Fund – National Authorities – Independent Audit Bodies – Erasmus+ Committee), and to what extent does the Commission fulfil its guiding role in the process? How has this changed between the two programming period? What are the areas for improvements?

·Are there differences in efficiency of Programme management and implementation between National Agencies, the Commission Executive Agency, the European Investment Fund or between different programming periods? If so, what are the differences and what are the underlying reasons for them? Compare the strategies, approaches and outcomes of the different National Agencies.

·To what extent are the monitoring mechanisms applied by the Commission, the Executive Agency, the National Agencies and the European Investment Fund efficient/cost effective? To what extent are the monitoring mechanisms of the beneficiaries and participants by National Agencies and the Executive Agency effective and proportionate? To what extent are internal monitoring mechanisms of activities of the National Agencies and the implementation of the Programme at national level effective and proportionate? What are the areas for improvement, considering the need for a smooth and effective implementation of the Programme?

·To what extent do the indicators identified for the Programme in the legal base correspond to the monitoring purposes? How could the overall management and monitoring system be improved?

·To what extent are the management support tools (e.g. Epluslink, Mobility tool) adequate and sufficient to support a sound management of the Programme?

·What are the efficiency effects of the integration of previous programmes and actions/activities into Erasmus+ programme? What are the advantages and disadvantages of the integration of previous programmes and actions into Erasmus+ programme?

·To what extent has the system of simplified grants resulted in a reduction of the administrative burden for National Agencies, Executive Agency and programme beneficiaries and participants? Are there differences across actions or fields? What elements of the programme could be changed to further reduce the administrative burden and simplify the programme's implementation, without unduly compromising its results and impact?

·To what extent have the anti-fraud measures allowed for the prevention and timely detection of fraud?

European added value

·What is the additional value and benefit resulting from EU activities, compared to what could be achieved by MS at national and/or regional levels? What does the Erasmus+ programme offer in addition to other education & training support schemes available at both international and national levels?

·What is the benefit and added value of the Erasmus+ programme compared to the benefit of the predecessor programmes? What is the added value of repetitive actions/activities of the Erasmus+ programme and the predecessor programmes?

·What would be the most likely consequences of stopping the Erasmus+ programme?

·To what extent does the Erasmus+ programme promote cooperation between participating countries?

Annex 5a: The intervention logic of Erasmus+ and its predecessor programmes

Programmes’ aims

Drawing on the legacy of its predecessors (Lifelong Learning, Youth in Action, Erasmus Mundus, Tempus, Alfa, Edulink and the Preparatory Actions in Sport), Erasmus+ aims to contribute to similar general objectives in the fields of education and training, youth and sport. As for its predecessors, these objectives have been defined based on EU level priorities set out in key policy documents (e.g. Europe 2020, ET 2020, Youth Strategy, EU Sport Action Plan and ‘Increasing the impact of EU development policy: an Agenda for Change’ 542 and since then confirmed through various European Commission Communications such as: ‘New Skills Agenda for Europe’ 543 ; ‘Renewed EU agenda for higher education’ 544 , etc.). These objectives jointly with the programme’s rationale are specified in the legal basis 545 and the programme guide 546 .

Next to these overarching objectives, Erasmus+ is aimed to help achieve a number of specific objectives applying to the respective fields it covers. These notably relate to:

■improving the level of key competences;

■improving quality, innovation, excellence and internationalisation of education, training and youth organisations;

■promoting the emergence of European lifelong learning area;

■enhancing the international dimension of education, training and youth;

■promoting teaching and research about the EU;

■promoting good governance in sport, volunteering in sport and tackle threats to integrity in sport.

The predecessor programmes and Erasmus+ share a number of specific objectives that remained common to both periods. Over both programming periods specific emphasis has been put for instance on:

■Competence development of participating learners;

■Professional development of staff; and

■Foreign language learning;

However, the predecessor programmes and the current programme were designed in quite different contexts. Hence, a number of differences in objectives can be noted:

■The Erasmus+ programme has stronger emphasis on high level policy objectives (and result-oriented approaches);

■The emphasis on employability is also clearer in the current programme which was designed in a period when young people were facing high unemployment, unlike the period of design of the predecessor programmes;

■The predecessor programmes on the other hand had specific objectives about the quality and volume or mobility exchanges and quality and volume of organisational cooperation. This ‘Europeanisation’ was perceived as an objective in its own right in the predecessor programmes while in the current programme it seems to become a means to achieve other ends rather than a goal on its own;

■The predecessor programmes also emphasised quite strongly the use of ICT in education and training and the introduction of ICT-based pedagogies was an objective of the programme while it does not figure in the Erasmus+ legal basis;

■The youth in action programme put much more emphasis on the objectives of youth participation and the citizenship dimension of the youth programme than the current programme.

There were also some issues that were topical at a given point in time for which specific actions were implemented temporarily. This is for instance the case of the situation of Roma population across the EU which was rather high on the policy agenda in the period 2009-2010. Regarding Erasmus+, the issue of social inclusion has become for instance prominent during the refugee crisis since 2015 and the issue of violent radicalisation became urgency after the terrorist attacks that started in 2015.

Overall, despite progressive adjustments as above, the aims of Erasmus+ have not radically changed in comparison to those of its predecessors. A noticeable difference between the current and the previous programmes is the integrated architecture of Erasmus+. Bringing together the education and training (including Jean Monnet programme), youth and sport fields into a single integrated programme is in particular expected to foster synergies, cross-fertilisation and to stimulate new forms of cooperation that did not or failed to materialise in the past.

Inputs

To operate the main types of actions and achieve the expected outcomes and long-last impacts discussed below, three main types of inputs underpin the Erasmus+ programme:

■funding range of actions;

■system and management structures, and;

■support measures for dissemination and knowledge management.

Whilst these do not differ much in theory from those offered under the predecessor programmes, Erasmus+ inputs are in practice rather different. A number of novelties have been indeed brought into Erasmus+. Besides the integrated structure mentioned above, the most noticeable changes have taken the form of: an increased budget allocation; a renewed internal structure of the programme; the introduction of new implementation and monitoring approaches and support measures for dissemination and knowledge management.

Types of actions

A major evolution compared to the predecessor programmes has been the change in programme structure. Rather than being structured by sectors with each sector having embedded a variety of types of actions each specific to a given sector, the programme was restructured according to main categories of types of actions (Key Actions - KA) which are common to the education and training and youth fields. The sport field and the Jean Monnet programme have made the object of separate chapters. While there are still some specificities in the fields and subfields, the main types of actions are shared. These are:

■Mobility of individuals: through KA1 (in education and training and youth);

■Cooperation partnerships: through KA2 (in education and training and youth) and other cooperation actions in sport and Jean Monnet) ; and

■System level projects: through KA3 and ad hoc actions in sport and Jean Monnet

This logic of these three main types of actions corresponds to those levels at which the programme aims to trigger change: individual, organisation and system. This is an improvement in the logic of the programme compared to the myriad of actions with different names under the predecessor programmes.

These broad categories of actions are further subdivided into a small number of types of actions which as often as possible share a common name if they are common to several sectors. This enables to cater for a variety of needs within a broad category of activities funded.

Expected outcomes and long-lasting impact

Erasmus+ aims to deliver outcomes and long-lasting impact at its three levels of intervention as the figure above illustrates. This can be further summarised as follows:

■At individual level: the programme is aimed to bring positive changes at both learners (students, trainees, apprentices, young people and volunteers) and practitioners (teachers, trainers, youth workers) in the form of (not exhaustive): improved skills and competences (including soft skills), self-empowerment and self-esteem, better awareness of the EU values, etc. For practitioners, additional outcomes are expected such as: enhanced motivation, opportunities to test and implement new practices, ability to address the needs of the disadvantaged, etc. The achievement of these outcomes is in turn expected to generate long-lasting impact at individual level (e.g. enhanced employability, entrepreneurship, active participation in society, participation in formal/non-formal education or training, etc.) but also at organisation and system levels (e.g. improved education attainment and completion rates, employability, transition to further levels of education, solidarity and career progression of staff).

■At organisation level: the transnational cooperation project opportunities offered by the programme are expected to generate the following types of outcomes (not exhaustive): development and/or implementation of new pedagogies or curricula, implementation of new organisational practices, enhanced networking with foreign partners (including outside Europe and from other fields), improving the dialogue between the academic research arena and policy makers. etc. The achievement of these outcomes is in turn expected to generate long-lasting impact at system level notably in the form of better quality of teaching, youth work and sport activities, sustainable partnerships, increased levels of participation in sport, physical activity and voluntary activity, etc.

■At system level: much greater systemic impact than in the past (e.g. KA3 clearly sets a framework for system level-oriented actions) is expected overall. Anticipated outcomes at both EU and national levels relate to achieving: stronger awareness about key policy challenges in education and training, youth and sport; enhanced mutual learning and good practice exchanges among policy makers and key stakeholders; better understanding of key EU tools and policies; supporting research and training about the EU, etc. This is in turn aimed to help achieve long-lasting impact in the form of:

■Enhanced quality, efficiency and equity of education and training systems and youth policies through the OMC;

■Effective implementation of reforms converging with the OMC at national level

■Effective implementation of EU tools for assessment, transparency and recognition of skills and qualifications acquired through formal, non-formal and informal learning at national level;

■Increased visibility of the external dimension of the programme (both within and outside Europe) and credibility to support structural reforms in partner countries, etc.

The different levels of intervention and related types of actions are not to be seen in isolation but on the contrary as aiming to contribute to commonly shared objectives and to generate mutually reinforcing outcomes and impacts. More than in the past, spill-over effects are expected to materialise across Erasmus+. Overall, the logic of the programme is that the simplification it offers should help reach greater and long-lasting impact at the individual, organisation and system levels and contribute to the achievement of the key EU strategic documents mentioned above.

Annex 5b: Erasmus+ Factsheet

Annex 5c: Specific objectives of Erasmus+ by level of intervention