TABLE OF CONTENTS

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS

Report on the Interim Evaluation of the Erasmus Mundus II Programme (2009-2013)

1........... INTRODUCTION........................................................................................................ 3

2........... BACKGROUND TO THE EXTERNAL EVALUATION............................................. 3

3........... THE EXTERNAL EVALUATION................................................................................ 4

3.1........ The terms of the evaluation.............................................................................................. 4

3.2........ Methodology.................................................................................................................. 4

3.3........ The evaluator’s findings................................................................................................... 4

3.3.1..... The EM II novelties:....................................................................................................... 4

3.3.2..... Other general findings..................................................................................................... 5

3.3.3..... Relevance....................................................................................................................... 6

3.3.4..... Effectiveness................................................................................................................... 6

3.3.5..... Sustainability................................................................................................................... 7

3.3.6..... Efficiency........................................................................................................................ 7

3.3.7..... Programme management: design and structure................................................................. 7

4........... MAIN RECOMMENDATIONS OF THE EXTERNAL EVALUATION..................... 8

4.1........ Relevance general recommendations:............................................................................... 8

4.2........ Effectiveness general recommendations:........................................................................... 8

4.3........ Sustainability general recommendations:........................................................................... 8

4.4........ Efficiency general recommendations:................................................................................ 8

5........... THE COMMISSION’S CONCLUSIONS................................................................... 9

1.           INTRODUCTION

The interim evaluation of the Erasmus Mundus II (EM) programme was launched by the European Commission following the requirements of the Erasmus Mundus Decision n° 1298/2008/EC of the European Parliament and of the Council of 16 December 2008[1]. The purpose of this evaluation was to assess the relevance, effectiveness, sustainability and efficiency of the programme, paying particular attention to the novelties introduced in phase II of the programme. This evaluation covered all three actions (Action 1, Action 2 and Action 3) over the implementation period of 2009-2011.

The evaluation was steered by the Steering Group drawn from Directorate-Generals of the European Commission (Education and Culture; Development and Cooperation - Europe Aid and Enlargement), the European External Action Service and the Education, Audio-visual and Culture Executive Agency, all of them involved in the EM programme. The full text of the evaluation can be obtained via the link: http://ec.europa.eu/dgs/education_culture/evalreports/index_en.htm

This Report on the Interim Evaluation is presented under Article 13 of the EM Decision. It draws on the findings of the Interim Evaluation of EM II regarding the results achieved and on the qualitative aspects of the implementation of the programme. It puts forward the Commission’s position on the main conclusions and recommendations of the Interim Evaluation.These conclusions and recommendations are based on extensive surveys and interviews of EM participants and key stakeholders.

2.           BACKGROUND TO THE EXTERNAL EVALUATION

The EM II (2009 – 2013) programme aims to enhance the quality of European higher education, to help improve the career prospects of students and to promote dialogue and understanding between peoples and cultures through international cooperation and to contribute to the sustainable development of third countries in the field of higher education, in accordance with EU external policy objectives..

EM has a budget of over €950 million, with around €494 million allocated to Actions 1 and 3, taken from the EU’s education budget, and €460 million allotted to Action 2, taken from a number of different funding instruments.

Structure of the programme: EM II 2009-2013 was implemented through the following actions:

– Action 1: EM joint programmes of outstanding quality at masters and doctoral levels, including scholarships/fellowships to participate in these programmes;

– Action 2: EM Partnerships between European and Third Country higher education institutions, including scholarships and fellowships for mobility at all academic levels. This action replaces the former “External Cooperation Window” scheme;

– Action 3: Promotion of European higher education through projects to enhance the attractiveness of Europe as an educational destination and a centre of excellence at world level.

Novelties of EM II: Under Phase II the scope of the EM programme was extended by incorporating the following key new dimensions:

– Extending Joint Programmes to the doctoral level;

– Offering scholarships for European students;

– Integrating the “External Cooperation Window” scheme into the EM programme as Action 2 and widening its scope;

– Allowing third country higher education institutions to participate in the EM Joint Programmes.

3.           THE EXTERNAL EVALUATION

3.1.        The terms of the evaluation

Following a call for tenders[2], the Public Policy and Management Institute (PPMI) was selected to carry out the evaluation. The scope of the interim evaluation was the period 2009 – 2011, during which a number of the call for proposals to implement the programme took place.

The interim evaluation focused on providing answers to the evaluation questions related to the relevance and utility, efficiency, effectiveness, sustainability and European Union added value of the programme.

With respect to each of the evaluation questions, the interim evaluation has provided concrete recommendations on how the logic, the objectives, the design, the implementation and the results can be further improved from the perspective of the Commission, the beneficiaries and potential applicants.

3.2.        Methodology

The applied methodology firstly begins with a set of 45 evaluation questions and sub-questions/operational questions. To answer the questions, explicit indicators and judgement criteria were defined.

3.3.        The evaluator’s findings

3.3.1.     The EM II novelties:

- The EM Joint Doctoral Programmes:

The Joint Masters and Doctoral programmes funded by Action 1 had considerable added value by facilitating the success of graduates when looking for work and / or further research positions. International experience and intercultural competence could be regarded as the most important assets that distinguished EM students from other graduates. Inclusion of doctoral and post-doctoral co-operation within EM II has been one of the most successful innovations of phase II, attracting the participation of many prestigious higher education institutions.

- The new EM Action 2

The inclusion of Action 2 with specific objectives related to third country cooperation was made within a programme design which addressed the potential conflict between the sustainable development of third countries (Action 2) and the promotion of excellence (Action 1). In this way the EM II was designed to prevent "brain drain". This new action includes two strands: one for developing countries (strand 1) and one for industrialised countries (strand 2).

The interviews with institutional and individual beneficiaries as well as the case studies suggest that the cooperation and excellence objectives were more complementary than contradictory. Beneficiaries from third countries appreciate the large positive impact of cooperation on the capacities of HEIs in their countries. However, the stakeholders from third countries emphasised the need for more reciprocal relationships between HEIs from EU and third countries.

- Offering scholarships for European Union students

Overall, this new phase of the EM Programme has offered added value for EU students when looking for work or further research positions (as it was only for third country students before). International experience and intercultural competencies are important EM assets now also offered to EU students.

However, within Action 1, the grants to European students can be considered the least successful of the programme novelties (category B grants for EU students were not valuable enough to attract students). Furthermore, it is suggested that adequate information is crucial to attract European students.

Attracting European students to Action 2 mobility was also a challenge. However, interviewed representatives of partner institutions in Action 2 strand 2 (industrialised countries) strongly believed that mobility to and from industrialised third countries should remain in future EM programmes because it supports excellence and offers great opportunities to build relationships with HEIs in major economies across the world.

- Allowing third country HEIs to participate in Action 1 of the Programme as full partners

This novelty offered opportunities to HEIs in third countries to share within consortia a cross-European design of joint programmes.

Cooperation between EU and non-European HEIs is hindered by the diversity of national higher education systems. The participation of non-European HEIs in the EM programme is made difficult by regulations, governance issues, limited resources and specific features relating to various subject areas. Non-EU HEIs and EU-partners have addressed obstacles linked to the diversity of national systems, administrative issues, consortium coordination, award or joint diplomas and the diversity of costs of education across the consortium. In general terms, the challenges were overcome on an ad-hoc basis, but more systemic measures to address them will be needed in the future.

In EM II, institutional beneficiaries were enthusiastic about the impact the programme had in strengthening the international ties between European and third country institutions. Examples have clearly demonstrated the capacity of the programme to be useful in internationalising the higher education systems of the participating countries (increasing the international inter-institutional cooperation opportunities).

3.3.2.     Other general findings

This Interim Evaluation was useful in learning more about the ultimate objectives of EM II beneficiaries:

– Action 1 beneficiaries argued that excellence in teaching and research was the main objective of their selected projects.

– The beneficiaries of Action 2 – Strand 1 emphasised co-operation, mobility and capacity building.

– The stakeholders participating in Action 2 – Strand 2 suggested that academic excellence was the central motivation of the involved institutions.

The EM programme contributed to strengthen Europe's competitive advantage in higher education by helping higher education systems to offer a more homogeneous image under the joint programmes.

The objectives of EM II were in line with EU Policy initiatives and political priorities. However in the future EM II could further strengthen its focus on the employability of young people which is emphasised in Europe 2020. The programme provides valuable career skills for graduates but actual employability remained uneven across regions and subject areas.

3.3.3.     Relevance

The objectives of EM II remained highly relevant even if the needs faced by the target groups and stakeholders were quite diverse. Both joint programmes (action 1) and mobility (Action 2) contribute to promoting excellence, capacity building and developing international co-operation.

The beneficiaries from third countries emphasised the potential of the programme to contribute to the capacity building of their HEIs. The risk of “brain drain” from third countries remained, but stakeholders from third countries also suggested that the developmental element could be strengthened by ensuring more balanced and reciprocal relationships between participants from the EU and third countries.

EM II was closely linked with and complemented the other EU Programmes, namely: LLP, Tempus, Alfa, Edulink and Marie Curie Actions.

3.3.4.     Effectiveness

Although the programme contributed to EU strategic policies, its impact on the Bologna process was mixed and varied across different countries. The contribution was very significant in some third countries, notably in Neighbourhood countries, and particularly in the field of legislation necessary for the recognition of joint degrees and credit recognition mechanisms. However, it is recognised that additional effort should be made on these issues within the EU and in third countries.

Overall, the programme contributes significantly to strengthening the international orientation of participating institutions, but the process should be seen in the context of a wider international collaboration and should take into account different situations within HEIs. The programme has been understood as a "soft power" initiative to change attitudes and views of policy-makers and stakeholders. In some cases, EM II participants already had longstanding commitments to international cooperation.

Overall, on the topic of difficulties encountered, the evaluation identified three sets of obstacles related to mobility: 1) obstacles relating to the diversity of national higher education systems; 2) obstacles relating to the design of the programme; 3) obstacles relating to the administrative burden and co-financing (EM consortia normally entrust centralised administration and management to their coordinating institutions. These institutions try to ensure administrative and financial effectiveness, which includes the identification of possible co-funding sources).

3.3.5.     Sustainability

The participating institutions were typically highly internationalised and had staff members responsible for international project management, but nearly half of the beneficiaries reported lacking human resources to cope with the joint EM project workload. The participating institutions were generally prepared for the phasing out of EU funding, but they hoped it would be gradual.

The EM label allowed institutions, which are outstanding in their field but not internationally prestigious, to access external funding and gain easier recognition of their courses. Allowing previous beneficiaries to retain the label after the end of the funding period would increase the sustainability of their courses.

Programme promotion and results exploitation could improve the sustainability of EM projects. The EMA (EM Alumni Association) is particularly motivated and active in promoting the programme and developing the EM graduate identity. Their dissemination activities mostly took the form of programme promotion rather than promoting Europe as a study destination.

3.3.6.     Efficiency

The programme was efficiently implemented. The evaluation shows the following programme efficiency details:

- Most of the planned outputs of the programme are set to be achieved by 2013 with lower costs than initially anticipated.

- Outputs of the programme were being produced with analogous or even lower costs than those of similar scholarships schemes. Overall, the analysis suggests that achieving the same results with less funding was not problematic because the programme was implemented efficiently in terms of budget and outputs produced.

- In the second phase of EM, relevant administrative tools, such as the calculation of incurred costs based on lump sums, have been put in place. Although the number of activity reports was reduced during the programme implementation, the beneficiaries assessed their participation in EM as burdensome.

3.3.7.     Programme management: design and structure

The programme beneficiaries assessed positively the preparation and implementation of EM projects, except for the extensive administrative workload. Beneficiaries suggested that further simplifications were possible (e.g. activity reports), including replacement of the annual re-application approach.

Project monitoring and evaluation was primarily quantitative. For the future, the potential of the EM Quality Assessment Project (EMQA) may also be applied to better assess the quality of joint programmes with the involvement of field experts.

The beneficiaries were satisfied with the services of the Executive Agency (EACEA), and information and guidance provided by the National Structures, the National Tempus Offices and the EU Delegations. Although the mechanisms for project selection were efficient, low success rates for some Actions and strands reduced the trust of the target groups in the transparency of project selection.

The division of responsibilities between DGs in the implementation of actions 1 and 2 as well as the absence of a single Committee in charge of the whole programme does not contribute to the development of synergies between the Actions of the programme. However, weaknesses in strategic coordination are to some extent offset by good coordination at the operational level as the administration of all actions is under the responsibility of a single unit within the Executive Agency (EACEA – Education, Audio-visual and Culture Executive Agency).

There is a need to appoint a single committee (including representatives from DG EAC, DG DEVCO and other Directorates-General and Services of the European Commission) for steering the post-2013 Programme,

4.           MAIN RECOMMENDATIONS OF THE EXTERNAL EVALUATION

General recommendations

4.1.        Relevance: general recommendations:

According to the general evaluation recommendations, relevance may be reinforced through more balanced programme objectives and strengthened links between EM and other EU programmes in the field of higher education.

4.2.        Effectiveness: general recommendations:

In order to improve effectiveness, the evaluation recommendations are addressed to the following main issues: employability; programme balance between objectives for excellence, capacity building and geographical participation; involving employers; helping the visa process; and mapping best practices.

In relation to employability, the programme activities could be more open to international mobility and cooperation in the field of vocational education provided by higher education institutions for training highly qualified professionals.

Retain and strengthen the balance between Programme objectives for excellence, development of capacity and geographical representation. Incentives should be provided to include a wider range of institutions from candidate and potential candidate countries in the Programme and strengthen their capacities, in order to ensure balanced mobility.

Good practices for involving employers should be fostered, and outreach activities in candidate and potential candidate countries are needed.

Since the main barriers are related to administrative issues, it is important that DG EAC continues to support the European Commission's action to facilitate the visa process for the beneficiaries of European mobility programmes, currently enforced through the issue of directives.

Good practices should be mainstreamed to applicants and beneficiaries.

4.3.        Sustainability: general recommendations:

Ensuring sustainability is a big concern shared by the EM beneficiaries. Recommendations on sustainability are related to a better use of the limited EU funding. Sharing good practices and a more integrated programme may be useful as well.

While there is a risk of dependence on EU funding, it is also true that applications for funding from other instruments requires deepening and expanding academic networks and thus helps excellent courses and partnerships to mature.

It is important to share good practices to help partners strengthen their recognition mechanisms and save resources spent on their development.

4.4.        Efficiency: general recommendations:

In accordance with the Interim Evaluation's recommendations, programme efficiency could be enhanced by linking the financial allocations to each action with clearly identified policy objectives, which take into due consideration the specific needs of different world regions in the definition of the subject priorities, level of intervention – undergraduate, masters, PhD - degree vs. credit mobility, etc.; improving the monitoring and evaluation of future projects; and streamlining programme promotion.

Reconsider the balance among various actions and their strands in the post-2013 programme in line with clearly identified policy objectives and with a view to maximise success opportunities.

Improve the monitoring and evaluation of future projects by better balancing quantitative and qualitative assessment and by involving field experts in the monitoring visits and project evaluation. Continue the EM Quality Assessment Project and better link it to project monitoring.

Streamline programme promotion across different Actions under the post-2013 programme, while maintaining the current institutional framework that involves the Executive Agency, the National Structures and the EU Delegations.

5.           THE COMMISSION’S CONCLUSIONS

The Commission shares the overall assessment of the evaluator that the programme EM II has provided strong support and made an important contribution to the internationalisation process of the European Higher Education Area.

The programme remains highly relevant in promoting excellence, international cooperation support, mobility and academic capacity building. During this first half of the EM programme (2009 – 2011), the programme has shown effectiveness in strengthening the international orientation strategy of participating institutions, providing as well sustainability to strategic networking activities. The Interim Evaluation of EM reveals a programme which has been efficiently implemented with a very good cost/benefit ratio.

The 3 programme Actions have provided outstanding outcomes for HEIs, students and scholars from EU MS and third countries. Action 1 has contributed to improving excellence in teaching and learning and to supporting institutional networking.

The EM II Action 2 Partnerships have been relevant in supporting international cooperation in the field of higher education and quite efficient in including third country institutions as members of international partnerships supporting academic mobility.

EM II Action 3 has been effective in giving coherence and sustainability to the whole programme.

EM II was designed to include 4 novelties to overcome several deficiencies, shortcomings and weakness observed in the previous phase (2004 – 2008). Even if the novelties need further improvements, the overall result is highly positive, notably in the integration of the new Action 2, the more active participation of third country partners in consortia and in the scholarships offered for European Union students.

The Commission notes the findings of this evaluation in favour of further improvements which could be made in the second half of the programme and beyond.

In particular, the Commission welcomes recommendation 4.1 to reinforce relevance by strengthening links between EM and other EU programmes in the field of higher education. This recommendation very much reflects the preferred option identified in the Impact Assessment on the International Dimension of the Higher Education Actions of the future Integrated Programme for the period 2014-2020. That option (based on which the Commission drafted its proposal for the future integrated Erasmus for All programme for Education, Youth and Sport) aims to strengthening objectives and impact through concentration and streamlined architecture. The use of this option in the design of the future integrated programme should create the necessary links within internal programmes (between Erasmus and Erasmus Mundus) and between external and internal policies and programmes in the higher education field (Erasmus Mundus, Tempus, EDULINK, Alfa, Atlantis, etc.). It is also expected to reinforce links between mobility and partnerships (including capacity building and policy support measures) and provide support to increase the overall quality and relevance of higher education, enhancing the links between policies and programmes, accompanying universities in their internationalisation strategy and the modernisation of higher education in non-EU countries together with the development of their human capital.

The Commission also agrees on recommendation 4.2 to improve the effectiveness of the programme by allowing for a better balance between excellence and capacity building (notably linking the financial allocations to each action with clearly identified policy objectives, reflecting different world regions' needs); further involving employers; and helping to improve – as far as possible – the visa process.

On recommendation 4.3, the Commission has already been supporting (and will continue to do so) sustainability through different initiatives, such as the establishment of the Erasmus Mundus brand name and the implementation of a cluster regrouping all the best sustainability practices under Erasmus Mundus.

Regarding recommendation 4.4, the Commission agrees on the need to improve the monitoring and evaluation of projects and streamline the programme's promotion (with a view to enhancing the current institutional framework that involves the Executive Agency, National Structures and the EU Delegations). This will be done in the framework of the future integrated programme, using the tools produced in its current phase, notably under the EM Quality Assessment Project.

[1]               OJ 340 19.12.2008 p.83

[2]               Restricted call for tender under the Multiple Framework Contract EAC/50/2009 for evaluation, evaluation related support and impact assessment.