52004DC0157

REPORT FROM THE COMMISSION TO THE COUNCIL on the interim evaluation of the third phase of the Tempus programme

1. Introduction

This report is presented pursuant to article 12 of Council decision 1999/311/EC of 29 April 1999. It puts forward the Commission position on the main conclusions and recommendations of the external evaluation of the third phase of the Tempus programme (Tempus III).

The complete interim evaluation report on Tempus III [1] can be found at the following Internet address: .

[1] The final report on the second phase of the Tempus programme can be found at the same Internet address as well.

2. Background

The Tempus programme was first proposed at the meeting of the Council of Ministers of Education in December 1989 as an instrument of co-operation between higher education institutions in the Member States and in the Partner Countries. Its first phase (Tempus I) was adopted in 1990 [2].

[2] Council decision 90/233/EEC of 7 May 1990.

The programme initially covered the Central and Eastern European Countries (CEEC); between 1992 and 1993, it was extended to the Russian Federation, Ukraine, Belarus and Albania. For a short period (1991) it operated in Yugoslavia as well.

Tempus sought to contribute to socio-economic reform through co-operation in higher education. More specifically, the programme was designed to promote the joint development of new curricula, the acquisition of new management skills by academic and administrative staff and the opening up of Partner Countries' educational systems to civil society.

The second phase of the programme (Tempus II) was adopted in 1993 [3] for the period 1994-1998 and, in 1996, it was extended until 2000 [4]. The second phase provided an extension of the programme to new eligible countries covered by the PHARE [5] and Tacis [6] programmes, but also increased the ambition and expectation levels for the programme. In particular Tempus II introduced specific national priorities which complemented the original «bottom-up» approach whereby initiative rested exclusively with Universities.

[3] Council decision 93/246/EEC of 29 April 1993.

[4] Council decision 96/663/EC of 21 November 1996.

[5] Council Regulation (EEC) No 3906/89 of 18 December 1989 on economic aid to certain countries of central and eastern Europe programmes and subsequent amendments.

[6] The current legal base for this programme is the Council regulation (EC,EURATOM) 99/2000 of 29 December 1999 concerning the provision of assistance to the partner States in Eastern Europe and Central Asia.

Although its logic of intervention has, to a large extent, remained unchanged, the Tempus programme has evolved significantly, accompanying the changes in the political context and contributing to the process of accession of the CEEC to the European Union.

The third and current phase of Tempus (Tempus III) was adopted in 1999 for a period of six years from 1 July 2000 [7]. The logic of intervention of the programme was not fundamentally altered. However, two innovative aspects were introduced. First, in addition to the tried-and-tested country-specific approaches, particular emphasis was placed on the programme's capacity to encourage regional co-operation. Second, more explicit reference was made not only to the need for the programme to ensure consistency and, where necessary, complementarity with other Community programmes, but also to create synergies with other forms of assistance to the partner countries.

[7] Council decision 1999/311/EC of 29 April 1999.

Council regulation 2666/2000/EC of 5 December 2000, setting out the framework for Community Assistance for Reconstruction, Development and Stabilisation (CARDS) to the Western Balkans amended the Tempus III decision to include the participation of Croatia and the Federal Republic of Yugoslavia.

Tempus III was further amended in 2002 [8] extending the programme to the Southern and Eastern Mediterranean countries covered by the MEDA programme, in the framework of the Euro-Mediterranean Partnership which emerged following the 1995 Barcelona Declaration.

[8] Council decision 2002/601/EC of 27 June 2002.

The extension seeks to respond to the need for closer co-operation in higher education between the Union and its partners in the region with a view to contributing to their socio-economic development. Additionally it aims to promote inter-cultural dialogue and understanding among them as a means to secure sustainable growth, peace and stability in the region and serves to reinforce the intercultural and civil society dimension of the programme.

Tempus III comes to an end in December 2006 [9], along with the other mainstream programmes in education and training (Socrates and Leonardo da Vinci).

[9] The decision regarding the extension of the Tempus programme to the MEDA countries amended the duration of the programme so that the end date became the same for Tempus, Socrates and Leonardo da Vinci.

3. Brief description of the programme

Tempus is primarily designed to contribute to the reform and upgrading of partner countries' higher education systems. Through co-operation at higher education level, the programme aims also at reinforcing civil society, promoting democracy in these countries as well as enhancing mutual understanding and intercultural dialogue between the EU and its partners.

The programme combines a bottom-up approach, whereby initiative is left mainly to the universities, with a top-down approach, whereby national priorities are established for each partner countries with a view to maximise the impact of the programme on reform processes.

To achieve these objectives, the programme supports three types of projects:

- Joint European Projects, which are multilateral projects bringing together higher education institutions from the EU and from partner countries with a view to support institutions in partner countries efforts to develop and upgrade curriculum development or university management. These projects seek also to contribute to build up the institutional tissue of partner countries. They last normally two or three years and they are the main type of project within Tempus.

- Structural and Complementary Measures, which are also multilateral projects designed to support short term interventions aimed at system reform and linked closely to partner countries' priorities.

- Individual Mobility Grants, which are awarded to individuals - professors, lecturers, members of staff or ministry officials - in order to help them to travel to other countries for work related to a particular reform process.

Beneficiary partner countries under Tempus are:

- Albania, Bosnia and Herzegovina, Croatia, the former Yugoslav Republic of Macedonia and the Federal Republic of Yugoslavia (these are referred to as the 'CARDS' countries).

- Armenia, Azerbaijan, Belarus, Georgia, Kazakhstan, Kyrgyzstan, Moldova, Mongolia, the Russian Federation, Tajikistan, Turkmenistan, Ukraine and Uzbekistan (these are referred to as the 'Tacis' countries).

- Algeria, Egypt, Israel [10], Jordan, Lebanon, Morocco, the Palestinian Authority, Syria and Tunisia (these are referred to as the 'MEDA' countries).

[10] Israel's participation is possible on a self-funding basis only.

The Tempus programme is funded from the CARDS, MEDA and Tacis budget for bilateral co-operation with each one of the partner countries.

4. The evaluation work

The Commission launched an open call for tender in 2002 as a result of which the firm ECORYS-NEI was selected to carry out the evaluation of the third phase of Tempus III. This evaluation was undertaken together with and taking into account the findings of the final evaluation of the second phase of the programme.

The terms of reference requested the evaluator to examine the intervention logic of the Tempus programme, addressing in particular the following issues:

(1) Do the socio-economic needs that gave rise to the first and second phase of the Tempus programme persist today? Do the objectives of the programme correspond to the existing education conditions in eligible countries? To what extent has the programme contributed to changing the socio-economic conditions in these countries?

(2) Are the call for proposals and selection process, based on national priorities, a valid approach and how appropriate is it to encourage co-operation between neighbouring eligible countries in a programme driven by national priorities?

(3) Is the multilateral model involving EC and eligible countries' institutions a valid one to achieve the objectives of the programme and in particular, what is the value added in relation to structural investment in educational reform?

(4) Are the type of projects (outputs) supported by the programme (Joint European Projects and Individual Mobility Grants in particular) relevant in the light of expected results, outcomes and impact on higher education systems?

(5) Is the level of funding sufficient to meet the objectives outlined? Does the present approach generate a critical mass capable of producing a durable impact?

(6) Are current implementation tools, management approaches and in particular the modalities of technical support appropriate and sufficient to ensure project quality and proper project implementation? Are monitoring practices sufficient? Are the existing mechanisms for feedback and result dissemination adequate to exploit the experience acquired through the programme?

The evaluation team designed an evaluation methodology with a strong emphasis on interactive participation by the various stakeholders in Tempus III. The main evaluation tools used in the period October 2002- June 2003 were:

- Three interactive workshops (Moscow, Almati and Skopje) with stakeholders and a lessons learned workshop (Brussels) where all key evaluation questions were discussed. This approach was supplemented with an intervention logic analysis; a literature review; interviews with stakeholders; and country case studies.

- Data on Tempus II achievements were used and questions relevant for this Mid-term evaluation were included in three survey instruments: an on-line questionnaire on Tempus impact on higher education reforms and sustainable partnerships (599 JEP co-ordinators); a survey amongst (80) JEP partners in the eligible countries; and (26) interviews with higher education authorities in the eligible countries.

It is important to note that the terms of reference were elaborated before Tempus III was extended to the Southern and Eastern Mediterranean. However conclusions and recommendations apply also to the implementation of the programme in this region.

5. The evaluation conclusions

5.1. The relevance of the programme in view of the socio-economic needs

Prevailing socio-economic conditions and need to support higher education

The evaluator carried out a comparative analysis which shows that the CARDS and the Tacis countries are now in a situation which is similar to that prevailing during previous phases of the Tempus programme. These countries still cannot be considered as stable democracies. In both CARDS and Tacis countries, transition is still only very partially completed and the process suffers from sluggish economies, limited capacity for reform implementation, weak welfare measures, disruptions in public services and high public finance deficits. Public expenditure in education suffers from this. Across the Tacis and CARDS regions more students than ever before continue on to higher education, whilst funding in real terms remains around a third of levels 10 years ago.

In many of the MEDA countries, although set in an entirely different historical context, higher education has also suffered from severe under-funding. In the poorer MEDA states the chronic lack of available funds has been compounded by the prevailing view amongst the international donor community that aid for education should concentrate on primary and secondary school. This would be more cost-effective and provide higher short-term social returns than investment in higher education. This view has changed in the recent past and support to higher education is now considered key to ensure sustainable development.

The evaluator argues that the challenges faced by the eligible countries of the third phase of the Tempus programme are just as formidable than those addressed by Tempus I and II and they provide a strong case for giving priority to training and higher education reforms.

"The transition process from communist society to democracies and market economies created the socio-economic needs for belief system change, institutional change and skills change/training. These needs gave rise to first phase of Tempus after the fall of the Berlin wall and to the second phase of Tempus in the early nineties... In eligible countries for the third phase of Tempus these needs are still very much felt and this makes the programme highly relevant."

The Commission shares the evaluator's view that "in the eligible countries the needs for further innovation and investment in the higher education sector are much higher than they can afford at present, and the needs can be expected to increase. The rationale for higher investment levels in the higher education and research sector is strong."

Relevance of Tempus to support higher education reform and development

The evaluation (of both Tempus II and Tempus III) proves the value of Tempus to change socio-economic conditions in the eligible countries. The staff and mobility programmes have reached hundreds of thousands of students, teachers and lecturers from CEEC higher education institutions. This exposed the participants of Tempus I and Tempus II to new ideas, ideals, concepts and best practices on virtually every aspect of EU societies, and has shown the variety EU Member States adopt in dealing with the challenges of societal reform and to education. This exposure has involved nearly all participants in extensive and wide ranging dialogue, and has contributed to awareness raising, comparison of national values and cultures, attitude changes and increased commitment to support the social and economic reform process in their own countries and in their own working environment.

The establishment of partnership relations between individuals and institutions from the partner countries and the EU has provided higher education institutions with enormous access to knowledge networks. This has made an important indirect contribution to legal, administrative and institutional changes facilitating transition towards a market-based economy and more democracy.

The evaluation report suggests that Tempus has done more than any other external programme in the eligible countries to help university departments and faculties to modernise courses and to introduce new ones. It has thus helped the transition of existing education systems to better educate the next generation in view of the evolving labour market needs.

The Commission agrees with the evaluator that the third phase of Tempus is relevant given the prevailing socio-economic conditions in partner countries and that, without question, Tempus continues to be a relevant and efficient tool to address current needs as far as higher education is concerned.

5.2. The validity of national and regional priorities

The evaluator demonstrates that there is a broad consensus among the programme stakeholders as to the importance of national priorities to maximise impact as well as on the need to reinforce the dialogue between the Commission and the local authorities to better define these priorities.

The evaluator argues that the perception regarding the validity of the national priority approach mainly depends on the appropriateness of the process established to formulate them. "In countries where the main stakeholders in the Tempus programme were consulted, the national priority approach was considered useful for guiding applicants. In countries were the priorities were clearly set without consultation, the selection was considered arbitrary and an unnecessary form of irrelevant top-down meddling."

The Commission points out that there is no significant difference in the way in which the national priorities have been established for different countries. However it acknowledges that there are indeed differences as to the nature and presentation of national priorities. In some cases national priorities can be easily translated into operational proposals. In other cases priorities are too broad or ambiguous to be operational.

The consensus proved more difficult to reach on the actual weight to be given to national priorities in orienting the programme. On one hand it was observed that in a bottom-up programme, innovative ideas should in principle prevail over prescriptive guidance from the top. On the other hand, a more resolute strategic planning may enhance the impact of the programme and avoid dispersion of resources. Similarly, on the issue of the weight to be given to the compliance with national priorities in the selection process, programme stakeholders did not have a single view.

The Commission takes the view that national priorities are essential if we are to maximise the impact of the programme on reform and development process in higher education, and through higher education on the development of the whole education and training systems. However, as the evaluation suggests, it is important to maintain the bottom-up approach in order to stimulate innovation and creativity at institution-level. The bottom-up approach can also serve to guarantee (where necessary) that the programme is not held hostage to bureaucracy and inappropriate political or economic interests.

As regards regional co-operation, the evaluator concludes that the needs for, inter alia, intensified cultural co-operation, security co-operation, EU accession prospects and growing interest among partners to follow the Bologna process provide a strong case for intensifying university co-operation at regional level.

Although the Commission has been encouraging applicants to submit regional projects (involving more than one partner country), it is true that, within Tempus, there is no mechanism in place for joint formulation of region-wide priorities by the higher education authorities in the respective regions.

5.3. The value of the multilateral model of partnership

This model is broadly perceived by the overwhelming majority of Tempus stakeholders as the key to the success of the Tempus intervention logic.

At the heart of the Tempus approach there has always been a reliance on the ability of universities in the EU and in the eligible countries to identify needs and formulate and implement innovative and targeted projects. This approach appears to have worked well not only in reforming university curricula, but also in introducing new management practices. It has also provided an incentive to higher education institutions to make their knowledge available in response to new needs arising from processes of societal change.

The evaluator concludes that the multilateral model has proven successful in promoting that Tempus achieves its twin objectives of cultural rapprochement and adaptation of higher education systems. This model has a direct major impact on the emergence of sustainable personal networks and institutional partnerships between higher education institutions in the eligible countries and in EU Member States - which is a main indicator of success for the 'cultural rapprochement' objective of Tempus III.

The validity of the multilateral model to promote the reform and upgrade of higher education has been confirmed by the final evaluation of Tempus II.

The evaluator compared the Tempus programme to other interventions and points out that the strength of the Tempus multilateral, bottom-up approach is its networking, people-to-people dimension. Tempus puts a human face to co-operation aid since as all Tempus projects are based on close collaboration between willing institutions and individuals.

As the evaluator puts it "the value added of Tempus compared with structural investment in education lies in its promotion of international co-operation, which generates new insights, ideas and new networks of personal and professional contacts. In addition, the strength of multilateral mobility model is that it facilitates entrepreneurship at faculty level, which provides a bottom-up motor for innovation and reforms. As a result the diversity of the innovation projects is very large."

The Commission notes the evaluator's argument that Tempus brings "good value for money" in that the cost of Tempus interventions is comparatively lower than, for example, structural interventions involving international expert input.

5.4. Relevance of the type of projects (outputs)

Tempus types of projects are described in section 2. In the light of the programme objectives, the evaluator concludes that there is a very good match between the programme objectives as stated in the Council Decision and the programme instruments:

- Joint European Projects and Structural and Complementary Measures contribute to the adaptation and development of higher education systems since they provide the partner countries access to the expertise on modernisation and reform available in the EU Institutions.

- Joint European Projects on curriculum development address development at institution level through the introduction of innovative study programmes; and projects on the improvement of university management address the management reforms in the higher education institutions.

- Joint European Projects and Individual Mobility Grants promote mobility and cultural dialogue as well as institution building, and are therefore relevant for reaching the Tempus III objective of rapprochement of cultures and civil society development.

The evaluation concludes that the different types of project have been effective in achieving the expected outcomes of the programme and the overall Tempus impact on higher education reforms. In particular, Joint European Projects have been proved to contribute to legislative/regulatory change in the field of higher education. They also contribute to openness and preparedness for international co-operation among higher education institutions; institutions management reform; curriculum development; training of decision makers; and to sustainable partnerships between higher education institution in the EU and in the eligible countries.

However, the evaluator does identify some scope for enhancing the programme's impact, suggesting in particular: additional emphasis on information exchange and dissemination of results; and enhanced university-industry relations with an explicit focus on employability. The evaluator suggests also integration of Tempus Joint European Projects with vocational training programmes for teacher-training, and the dissemination of innovative results to training centres, and secondary and primary schools. Joint European Projects should also serve to educate top students for research careers.

The Commission agrees largely with this analysis, and underlines that there is a case to consider the mechanisms to reinforce synergies between higher education and vocational education and training within the present programme, and to extend the programme beyond higher education in its next incarnation.

5.5. Level of funding and critical mass

The evaluator observed that the programme objectives were not translated into a number of performance targets verifiable through quantitative indicators. He argues, however, that the analysis of the selection process reveals that the number of quality applications has always exceeded the available funding. Once the programme was well established, the absorption capacity has never been exceeded. This suggests that Tempus has been under-funded.

As for the critical mass, the conclusion of the evaluation is that such critical mass can be achieved with current level of funding at project level. However, at country level, achieving critical mass depends on the size of the country and on the allocation of sustained and increased funding.

The Commission agrees with the evaluator's analysis and acknowledges that variations in funding levels that the programme has experienced in the past, though often justified for political and programming reasons, do jeopardise the effectiveness of the programme.

5.6. The management of the programme

The evaluation report states that the current implementation tools, management approaches and modalities of technical support are appropriate and have in the past ensured a satisfactory level of project quality and proper project implementation.

While the report acknowledges the Commission's recent efforts to streamline procedures, it points out that the rules for contracting and financial administration are more rigid and time-consuming than necessary. The Commission underlines that, although there is an element of truth in this conclusion, there is a very difficult balance between the need to ensure appropriate accountability for the use of public funds and administrative simplicity. The introduction of the new Financial Regulation has not made such work any simpler.

As far as selection procedures are concerned, the evaluator underlines the transparency of the system, but points out that there is insufficient feedback provided to assessors on the performance of past projects, and no horizontal comparison of the relative merits of projects in the wider context of the higher education reform needs of the eligible countries. The Commission has already taken steps to improve the selection process, in particular by reinforcing assessor training.

Regarding monitoring, the evaluator concludes that Tempus III monitoring practices are limited but can be considered adequate. The Commission acknowledges that field monitoring has not taken place in a systematic manner during Tempus III, and this has limited the information flow on Tempus achievements in the eligible countries. The Commission agrees also that it would be desirable to improve the dissemination of results.

6. The recommendations of the external evaluator

The evaluation report makes a series of recommendations, which are summarised below (in italics) together with the Commission comments:

(1) The successful multilateral model of Tempus should be extended to other regions of the world.

The Commission has drawn the evaluators' attention to the fact that some existing programmes (such as ALFA or Asia-Link) have adopted the Tempus partnership model.

(2) The annual national budgets for the remaining years of Tempus III should be increased.

The Commission agrees with the evaluators arguments regarding the fact that 'critical mass' can only be achieved through increased levels of funding and is committed to keep the funding of the programme at a sufficiently high level to ensure an efficient use of resources.

(3) The process of formulation of the country-specific priorities should be strengthened by means of a more structured dialogue with the educational authorities of the partner countries.

The Commission recognises that dialogue with the authorities of the partner countries is key to the success of the programme, and will take immediate measures towards a more structured and coherent approach.

(4) Increase the usefulness of the National Priorities further by specifying the priorities as horizontal education issues, rather than as list of vertical academic specialisations.

The Commission recognises that the nature and presentation of national priorities should be improved.

(5) The Call for Proposal approach and the selection mechanism for regional projects should be improved.

The Commission recognises that the current system based on national priorities and national budgets does not allow the realisation of the full potential of regional co-operation, even if this is explicitly encouraged in the current phase of the programme. The Commission will examine how best to reinforce the regional dimension of the programme.

(6) Tempus funds should be used for what the programme does best, i.e. promoting mobility, exchanges and innovation of study programmes.

The Commission is pleased that the Tempus programme has gained a reputation for promoting the mobility and the innovation of study programmes and teaching methods, but considers that the ambition of the programme to extend its impact to reach system level is the logical consequence of the growing importance attached to national priorities. Therefore while activities focusing on curriculum development at institutional level will continue to be central to the programme, other activities, particularly those seeking to involve actors at system level, will be further developed.

(7) Increase the relevance and effectiveness of the Individual Mobility Grants further and limit the room for misuse of the grants.

The Commission will revise the criteria and funding conditions for IMGs and plans to involve the partner countries' national authorities in defining priorities for this type of activity. The Commission considers that the perception that IMGs are or can be misused is not supported by evidence. It is worth noting that IMGs do allow beneficiaries to carry out legitimate activities that go beyond the original scope of the grant; this may have given raise to the misperception.

(8) The co-ordination between DG EAC and EuropeAid should be strengthened to maximise the impact of the programme on higher education reform.

The Commission will ensure co-ordination at all levels between services with responsibility over co-operation in education in the partner countries concerned. This co-ordination should take place both as regards the identification of overall priorities, programming as well as co-ordination, monitoring and evaluation.

(9) Investigate the feasibility of reducing the maximum JEP grant amounts to say EUR 200,000 for a two-year project with the aim of selecting more projects for funding.

While for some countries smaller grants and higher number of projects may be desirable, experience has shown that the current maximum funding level (EUR 500,000) is necessary to ensure critical mass at project level.

(10) Introduce a new project type for conference organisation in support of regional co-operation/networking and dissemination of project result. (11) Reserve funding for actions that will promote information exchange and more dissemination of the intangible and tangible results of Tempus.

The Commission considers that the launching of the Structural and Complementary Measures, which include support for information and dissemination projects, and the organisation of Tempus regional conferences addresses these recommendations.

(12) Focus the Tempus Guidelines for Applicants more firmly on the promotion of: top talent, employability, university-industry-civil society relations and the trickle-down of JEP innovation to other training establishments including secondary and primary education programmes.

The Guide for Applicants already encourages university-industry-civil society relations. The Commission will seek to stimulate further this collaboration through targeted information activities in partner countries.

The Commission will also seek to exploit, within the current phase of the programme, the potential of higher education to stimulate development of vocational and adult education and training as well as school education. As regards the renewal of the Tempus programme, the Commission will consider the extending the Tempus model so that it covers the whole spectrum of lifelong learning.

(13) Statistics on good quality submitted proposals should be published.

The Commission agrees that such statistics serve to prove the additional absorption capacity of partner countries and therefore serve as a reference for a potential increase of programme funding. They also relate to the requirements of the new Financial regulation as regards the publication of results.

(14) An effort should be made to raise the average quality level of funded proposals.

The Commission will address this issue in two ways: (1) it will encourage National Tempus Offices [11] to organise training session for prospective Tempus applicants; (2) it will improve the information material for the applicants and the training of the experts involved in the project selection.

[11] National Tempus Offices are designated by national authorities in partner countries and assist the Commission in the implementation of the programme providing information and advise to prospective applicants and feedback to the Commission on project implementation.

(15) The relevance and the impact of the programme should be actively promoted at the level of the local authorities.

The Commission recognises the importance of closer involvement of national authorities in defining priorities for the implementation of the programme and in stimulation participation of higher education institutions as well as other organisations. To this end, the Commission has taken steps to set in place a mechanism for structured dialogue with national authorities.

(16) Contracting procedures should be simplified.

Simplifications of the contracting procedures have been introduced in 2003, reducing the number of payments and the number of reports required from grant holders. However, further efforts in the direction must take account of the changed (and more complex) regulatory environment.

(17) The selection process should focus on project relevance.

The Commission is of the view that the relevance of project to higher education reform will be improved through the reinforcement of national priorities, better information up-stream the selection process and appropriate training of experts involved in the selection process so that project relevance to local needs can be better assessed. Steps are being taken in all these respects.

(18) Reintroduce field monitoring on a selective basis.

The Commission plans to reinforce the current system for field monitoring, subject to the availability of additional human and financial resources, both within the Commission and within its technical assistance.

7. Conclusion

The interim evaluation confirms the relevance of the Tempus programme and fully endorses its intervention logic as effective and efficient. While no fundamental reorientation of the programme appears necessary, the Commission has already taken steps to improve its implementation and will adopt further measures as indicated above.

In particular, the Commission will seek to reinforce national priorities, improve result dissemination and strengthen monitoring. The Commission will also seek to ensure a sustained level of funding in order to ensure an efficient operation of the programme.

Through Tempus, the Commission will also stimulate reform in school, vocational and adult education and training in order to fully exploit the programme's value as an instrument for socio-economic change in partner countries.