52004DC0109

REPORT FROM THE COMMISSION TO THE COUNCIL - Final report on the second phase of the Tempus programme (1994-2000)

1. Introduction

This report is being presented pursuant to the provisions of Council Decision 96/663/EC of 21/11/1996 (OJ L 306, 28/11/1996), amending Decision 93/246/EEC and extending the second phase of the Tempus programme until 2000. The Commission is thereby required to present a final report on the second phase of the programme by 30 June 2004 at the latest. This report examines the impact of the Tempus programme in the countries eligible to take part and its role in stimulating the reforms that have taken place in those countries. It follows the Tempus II Programme Interim Report [1] presented by the Commission in March 1998 and it is based on the findings and conclusions of the independent evaluator's report.

[1] COM(1998) 379

The report has the following structure:

1. Introduction

2. Tempus: A programme evolving over time

2.1. The launching of the programme

2.2. From Tempus I to Tempus II

3. Overview of the external evaluation exercise

3.1. The Terms of Reference

3.2. The evaluation approach

4. The Main Conclusions of the external evaluation

4.1. The programme outcomes

4.1.1. Impact on legislation

4.1.2. Contribution to HE reform

4.1.3. Contribution to human resource development

4.1.4. Towards a sustainable partnership?

4.2. Programme management

5. Summary of conclusions

List of documents

Annexes

2. Tempus: A programme evolving over time

2.1. The launching of the programme

Over more than a decade, the TEMPUS Programme has provided assistance to partner countries reforming or restructuring their higher education systems by supporting co-operation with the Member States. The programme is based on the assumption that higher education institutions are of particular importance for the social and economic transition process as well as for cultural development; they are also pools of expertise and of human resources and provide for the training of new generations of political leaders.

It is perhaps worth emphasizing that, while the Tempus programme is fundamentally geared toward assisting partner countries, the programme also gives EC member states better access to areas of knowledge where some of these countries have reached at high level of development and creates enduring partnerships, thus ensuring mutual benefit.

The Tempus programme was first proposed at the meeting of the Council of Ministers of Education in December 1989, the very year of the fall of the Berlin wall, an event which gave rise to the process that would eventually completely re-draw the map of Europe.

The European Community seized this historic opportunity and launched two programmes (Phare [2] and later Tacis [3]) to assist the Central and Eastern European countries (CEEC) [4] and the countries of the former Soviet Union (plus Mongolia) in their transition towards democracy and market economies and to ensure political stability in the region.

[2] Council Regulation EEC No 3096/89 of 18/12/1989, OJ L 375, 23/12/1989, p. 11-12

[3] Council Regulation (Euratom, EC) No 1279/96 of 25/06/1996, OJ L 165, 04/07/1996, p. 1-11

[4] i.e. Bulgaria, the Czech Republic, Estonia, Hungary, Latvia, Lithuania, Poland, Romania, Slovakia and Slovenia

Education was perceived as one of the pillars of the transition process. The reform and overhaul of partner countries' education structures and systems was necessary to prepare citizens for the challenges of a competitive economy and for a multiparty system where civil society was to play an ever greater role.

As part of Phare, the European Community adopted the first phase of the Tempus programme (Tempus I) in May 1990.

The programme operated initially in the CEEC and between 1992 and 1993 it was extended to the Russian Federation, Ukraine, Belarus and to Albania. For a short period (1991) it operated in Yugoslavia as well. The budgets allocated to the programme were decided on a national basis within the overall budgets of the Phare and Tacis programmes.

The programme was designed to promote multilateral co-operation between higher education institutions in the Member States and in the partner countries thereby encouraging the transfer of know-how with a view to stimulating:

- Academic development and improvement, including the development of new curricula, pedagogic methods, didactic material, etc.

- The opening-up of institutions to international co-operation (including greater inter-regional co-operation) in general and, in particular, to co-operation within the emerging European education area.

- The acquisition of management skills by academic and administrative staff thus ensuring balance between institutions' newly granted autonomy, the growing demand for accountability and the need for greater financial sustainability.

- The opening up of partner countries' education systems to civil society.

From the outset the programme has supported partnerships involving networks of higher education institutions from both the EC and partner countries. These partnerships (Joint European Projects - 'JEPs') have constituted the main type of project throughout the programme lifetime. They provide the framework for a wide range of specific outputs such as:

- New curricula, teaching methodologies and didactic materials;

- Training for university management and staff;

- Policy papers and recommendations;

- New structures within institutions, such as services for international relations;

- Computerisation of university facilities, and libraries in particular.

Next to the JEPs, Individual Mobility Grants (IMG) are a fundamental type of project within Tempus. IMGs have provided more than 15,000 staff with internships and periods of study abroad.

2.2. From Tempus I to Tempus II

Although its architecture and intervention logic have remained to a very large extent unchanged, the Tempus programme has significantly evolved, accompanying the changes in the political context and contributing to the process of accession of the CEEC to the European Union. The experience accumulated by the programme stakeholders also offered significant pointers to the way ahead for Tempus in other geopolitical areas (see the recent extension of the programme to the Mediterranean countries).

A key characteristic of the programme from the outset was its «bottom-up» approach within Universities. The programme initially allowed working teams operating at Faculty or Department level to submit project proposals without direction from the top, generating a high motivation to engage with change, a motivation which was matched by a similar desire on the side of the Western partners to contribute to the reform process. A dynamic process was launched that at first was very positive in terms of getting the Programme off the ground. Later however, it became necessary to introduce into the programme a greater degree of planning at institutional level.

As a result the Council decisions of 93/246/EEC and 96/663/EC on Tempus II and Tempus II (bis) provided not only for continuation of the basic approach and an extension of the programme to new eligible countries in the Tacis and Phare area, but also increased the expectation levels for the programme.

In particular Tempus II introduced specific national priorities and this brought a top-down (or strategic) component into the programme. The national priorities gave the higher education authorities in the eligible countries and the Commission an instrument for guiding the Tempus applicants and the Tempus activities (funded from their national Phare and Tacis budgets) more firmly towards the longer-term systemic reform needs of higher education as they perceived them. Tempus II aims thus went beyond the mobility objective and bottom-up innovation of curriculum development and university management that characterised Tempus I.

Tempus II bis (see Annex I) introduced institution building as a new field of work. Under Tempus Phare, "institution building" was linked to the accession process of the eligible countries. Institution building projects were seen as instruments for using university co-operation and EU university know-how in supporting the preparation of national administrative, economic and legislative bodies for operating within the European Union by training their staff for the adoption of the 'acquis communautaire', the accumulated mass of European legislation. With this instrument, the Tempus programme opened up for involving non-academic institutions in the projects.

Other new features of the second phase that were introduced to help the programme meet high ambitions included the introduction of compact measures (Phare) or compact projects (Tacis). These were two types of activity (next to JEPs and IMGs) designed as new instruments for more targeted actions in the field of university management reforms and institution building.

Despite the changes introduced during the second phase of the programme and its ambitious reorientation towards a more strategic approach and the opening to the civil society, the bottom-up image of the programme remained largely unchanged and the overall majority of the projects continued to address the reform of curricula. The reorientation might probably have been more visible, had it been accompanied by an enhanced financial contribution. As a matter of fact, during the six years of the second phase of Tempus (1994-1999), the annual budget was reduced to less than 40% of the initial figure (see Annex II).

It is useful to emphasise at this point the different level of funding in the years 1994-1999 in proportion to the size of the higher education sector in the respective partner countries. In the Phare countries the vast majority of the higher education institutions have participated in one or more Tempus projects. In Russia, on the contrary, only 100 out of some 800 higher education institutions have participated in Tempus. Out of a total of 1386 funded JEPs, 1120 were in the candidate countries of the Phare region, 184 in Tacis countries and 82 in the countries of former Yugoslavia (plus Albania).

In terms of financing, Tempus Phare equalled some 30% of national spending on higher education in the partner countries. In Russia this percentage was less than 1 %.

These crude financial figures should always be kept in mind in what follows and make us expect a more pervasive impact in the Phare countries, whereas in Russia, for instance, one might expect a more patchy effect.

3. Overview of the external evaluation exercise

3.1. The Terms of Reference

Since the adoption of Tempus I, the programme has undergone three evaluations:

- The evaluation of the first phase 1990-1994;

- Mid-term evaluation of the second phase 1994-1996;

- Evaluation of the Tempus programme in the countries eligible under Tacis [5].

[5] Council Regulation (Euratom, EC) No 1279/96 of 25/06/1996, OJ L 165, 04/07/1996, p. 1-11

Pursuant to the provisions in the Council decisions on Tempus II bis, the Commission, (DG EAC) launched an open call for tender for the provision of an evaluation report on the second phase of the Tempus programme (1994-1999).

The tender was won by ECORYS Nederland BV and the contract for this assignment was signed in October 2002. The evaluation was carried out between October 2002 and July 2003 culminating in a 'lessons learned' workshop in Brussels involving a significant representation of programme stakeholders. The final report was submitted to the Commission in October 2003 and finally approved in December 2003.

This report sets out the findings of the external evaluation.

The Terms of Reference for the assignment specified that the evaluation aimed to serve the Commission in formulating a well founded judgement on the medium-term impact of the programme and on the extent to which the second phase of the Tempus programme had produced the outcomes expected.

In the light of the provisions of the relevant Council decisions indicated above, the evaluator was requested to judge the extent to which the Tempus programme produced the following outcomes:

1. The extent to which the programme has contributed to changes in legislation in eligible countries leading to:

- (where appropriate) greater convergence of their education systems with those in the EC,

- education systems better adapted to the changing socio-economic reality, in particular market economies and multiparty political systems,

- (specifically for pre-accession countries) the transposition into national law of the acquis communautaire, particularly in so far as the regulated professions are concerned.

2. The extent to which the programme prompted reforms in higher education institutions (including polytechnics and technical schools) leading to:

- institutions' openness and preparedness for international co-operation,

- new management approaches and structures in line with more independent, accountable higher institutions.

3. The extent to which the Tempus programme has brought about curriculum development and reforms leading to the acquisition by staff and students, of skills and qualifications necessary for the transition process and for the new socio-economic reality.

4. The extent to which the Tempus programme has contributed to the training of decision makers in eligible countries and the extent to which staff with experience in Tempus played and continue to play a role in universities in partner countries.

5. The extent to which the Tempus programme has encouraged sustainable partnerships in higher education involving EC and partner countries.

The evaluator summarised the above in the following eight expected outcomes:

1. Contribution to legislative changes leading to greater convergence of education systems in the eligible countries with those in the EC

2. Contribution to legislative changes leading to better adaptation of education systems to the changing socio-economic reality of market economies and multi-party political systems (curriculum development, mobility and ECTS)

3. Contribution to the transposition into national law of the acquis communautaire, particular in so far as the regulated professions are concerned

4. Reforms prompted in Higher Education Institutions (HEIs) leading to institutions' openness and readiness for international co-operation

5. Reforms prompted in HEIs leading to new management approaches and structures in line with more independent, accountable higher institutions

6. Curriculum development and reforms leading to the acquisition by staff and students of skills and qualifications necessary for the transition process and for the new socio-economic reality (curriculum development and employability)

7. Contributions to the training of decision-makers (institution building) and retention of Tempus experienced staff in universities (IMGs)

8. The development of sustainable partnerships between HEIs in eligible countries and EU countries

3.2. The evaluation approach

The evaluators proposed a review of the relevant literature on the programme. This included the previous evaluation reports, the Tempus Impact Studies produced at the national level within the EC and the eligible countries, and the report produced by the European Training Foundation Tempus Department on the Tempus contribution to Accession (a report limited to Hungary, Poland, Bulgaria and Romania).

The information so gathered, however, would have been clearly insufficient for the purposes of the evaluation. In order to complement it with more structured information of both quantitative and qualitative nature, the evaluators built three questionnaires, which focused on ex-post assessment of project outcomes and programme impact:

- an online questionnaire for JEP co-ordinators;

- a questionnaire for structured interviews of JEP participants in the eligible countries;

- a questionnaire for structured interviews with higher education authorities in the eligible countries

The questionnaires were designed to provide the answers to the expected outcomes and the elements to support the findings. The response rate to the online questionnaire was 43% and was considered quite satisfactory by the evaluator. Due to the use of national experts engaged by ECORYS for this assignment the structured interviews were successfully completed in 100 percent of the cases. The contractor organised the meetings and conducted the interviews; likewise higher education authorities were interviewed in 20 countries.

The approach chosen by the evaluator was certainly determined by the limits imposed in order to keep the study within the proposed budget. There are two comments on the selected approach. The first is about the choice of the evaluator of making limited use of the JEPs Final Reports. These provide first hand information on the results achieved by the single projects and it is believed that an in-depth analysis might have usefully complemented the findings of the questionnaires and of the interviews. This deficiency is partially remedied by the use the evaluator made of the ETF report, which was precisely based on the Final Reports but focussed on accession issues only and was limited to four countries. The second consideration is that the respondents were not necessarily always best positioned to give answers to all questions. Several examples of this will be given in the following.

4. The Main Conclusions of the external evaluation

4.1. The programme outcomes

This section outlines the expected outcomes and the corresponding impact found by this evaluation. The Commission's comments are presented as well.

4.1.1. Impact on legislation

The request contained in the Terms of Reference to judge the extent to which the Tempus Programme contributed to changes in legislation (Outcome 1) may appear surprising, as legislative reform is not and has never been the primary objective of the Tempus programme. As a matter of fact the lack of direct influence of the Programme on the higher education reform has often been flagged up. This criticism has been partly echoed in some previous programme evaluation reports. For example the 1998 'Evaluation of the Tempus Programme in the NIS and Mongolia', by FTP International stated: "So far, the Tempus programme has had no visible impact on the national higher education policies of the individual partner countries, nor on education legislation". This lack of impact at the political level was generally considered as the inevitable downside of the bottom-up philosophy of the programme. With faculties and departments as main actors -so the idea was- the effects of Tempus on system reforms, if any, could only be long-term and indirect. Such opinion was shared by many programme stakeholders and taken for granted. In fact some even believed that, after all, this was the maximum an EU-funded co-operation programme could possibly achieve. In fact it was rather doubtful that top-down measures would have had a strong and immediate effect, given the widespread scepticism towards authority and top-down initiatives in the aftermath of the fall of communism.

However, by giving the evaluator the mandate to assess this outcome it was the Commission's intention to investigate whether, in the medium term, a more explicit link than was previously acknowledged, might be established between the Tempus activities and the reform process ongoing in most of the programme partner countries. The findings of the final evaluation report seem partly to change the picture described above. Of course it is tempting to ascribe this to the reorientation of the programme towards a more strategic approach. The report gives some interesting clues that seem to confirm this view. Tempus appears to have been especially important for sensitising policy makers to the need for legislative and regulatory reforms.

The Russian test case presented in the report deserves a closer look. Evidence is provided here that Tempus results were made available to the so called 'Teaching and Teaching Methods Associations', special bodies responsible for the development of education standards based on a bottom-up approach. With the help of further supporting evidence gathered through structured interviews with the competent authorities, the evaluator reached a conclusion apparently at odds with the one expressed above. The evaluation found that Russian academics who participated in Tempus, have subsequently been able (in particular through the academic support groups), to influence the new strategy papers and eventually orient the recent legislation characterised by a strong European flavour. This striking contradiction between the current and previous findings may have several explanations. The most obvious but probably also the most plausible is that in 1998 when the FTP report was issued, it was not yet possible fully to assess the extent of a 'trickle-up' effect, which is intrinsically slow and takes place under the surface.

Nevertheless some other elements should also be taken into account. First, the conclusion of the positive impact of the programme on legislative changes was reached not only on the basis of structured interviews but also on the basis of the online questionnaire. As the evaluator points out, all the respondents were project co-ordinators, that is persons directly involved in the projects and responsible for the academic content. Some positive bias in the answers is therefore to be expected.

Second, despite the efforts of the evaluator to avoid misunderstandings, respondents interpreted this particular question regarding the programme impact in rather different ways. Respondents gave a range of examples including changes in rules and regulations at faculty, university and national level. This also explains an ancillary statement made by the evaluator that is that in the countries of former Yugoslavia (plus Albania) no influence beyond faculty level was reported. In countries where the rector is basically a figure head and deans traditionally negotiate their budgets directly with the responsible ministries, this comes as no surprise. Moreover it must not be forgotten that in all countries of former Yugoslavia the programme was introduced well into the Tempus II phase and a fair impact assessment is probably not yet possible (in very much the same way as it was premature to do it for Russia in 1998).

All of the above allow us to conclude that the programme reorientation towards a more strategic approach has made the impact of the programme on legislation a reality, but (consistently with the other findings hereafter) it is fair to say that the image of the programme characterised by mainly a bottom-up approach has remained unchanged to a large extent.

The findings related to the influence of the programme on the adaptation of the higher education systems to the changing socio-economic needs (Outcome 2) are somewhat variegated. The influence is said to have been 'major direct' by the majority of the higher education authorities in the Phare area, in Russia the influence was found to be 'minor indirect'. In the smaller Tacis countries both 'minor direct' and 'major indirect' were popular answers in the survey. It is difficult to give a unique and straightforward explanation for these differences. A few hypotheses are suggested here. First, the distinction between direct and indirect influence, and between major and minor influence might simply be inappropriate for this particular question. Second, one should not forget that, as it was already mentioned, project co-ordinators played an important role in the survey. In current Phare projects, according to Tempus rules, project co-ordinators may belong to both the partner countries and the EU, whereas in Tacis project co-ordinators need to belong to the EU. It is hardly surprising that such a qualitative judgement turns out to be highly dependent on the cultural and political background and also on the role and the perspective of the respondents.

The Commission agrees with the evaluator's conclusion (reached by averaging out all respondents from all regions) that Tempus II had a major indirect impact on adaptation to socio-economic circumstances.

The contribution of the programme to the harmonisation of curricula for the seven regulated professions covered by the EC directives was a less controversial issue (Outcome 3). The survey shows that almost half of the Phare projects dealt somehow with regulated professions, mainly introducing best practices and curriculum reforms on the related subjects (in medicine and veterinary studies more than others), even if an in-depth analysis of the Tempus contribution to the implementation of the EC directives was not performed by the evaluator. It is maybe worth mentioning that even if the ToR explicitly limited the question to pre-accession countries, the contribution of Tempus to veterinary studies (one of the most expensive faculties to run!) in the countries of former Yugoslavia can hardly be underestimated. For example, in the former Yugoslav Republic of Macedonia there was no faculty of veterinary previous to the break up of the Yugoslav Republic and the Faculty of Veterinary in Sarajevo was totally destroyed by the war. In both cases, the Tempus contribution in the set up and in the reconstruction of the Faculty has been broadly acknowledged.

The Commission agrees with the evaluator's acknowledgement of the Tempus contribution to the reform of the curricula leading to the regulated professions.

4.1.2. Contribution to HE reform

The substantial contribution of the Tempus programme to the openness and readiness for international co-operation (Outcome 4) is one of the undisputed findings of the external evaluation report. The project co-ordinators' survey clearly spells out the importance of the second phase of the programme for the computerisation, increased access to and use of information and communication technologies in classrooms; for the exchange of visits of students and staff; for the access to international scientific literature; for the adoption of textbooks and teaching material in an EU language; for the introduction of teaching and training in an EU language; and finally, for the establishment of international offices in higher education institutions. The impact has been defined as 'major direct' by the majority of the respondents in all geographical areas. A non-negligible quota of respondents assessed the impact as minor only in the Phare countries and in Russia. This can easily be explained in Phare countries due to the process of internalisation starting early and with the second phase of Tempus offering less. In Russia on the other hand, the perception of a minor impact is certainly due to the lower level of penetration of the programme due to a proportionally lower budget in comparison with the size and needs of the higher education sector.

The Commission considers the conclusions of the evaluator's report on this particular issue as yet another indication of the validity of the overall programme design.

The impact of the programme on university management (Outcome 5) turned out to be a relatively controversial issue and the feedback provided by the evaluator may appear as rather inconclusive. UM projects represented approximately 10% of all Joint European Projects and therefore the evaluator perceived the clear risk of statistical insufficiency, if evidence were gathered only from the few UM projects included in the sample. Therefore the option was taken to extend the survey to all JEPs claiming to have had some impact on university management (even if their primary focus was different, notably curriculum development). The Commission tends to agree with the evaluator's conclusion that the impact was not negligible but that a systematic focus on university management might have multiplied the risks of clashes between old-fashioned structures and pressure for reform and that the gentle approach chosen was probably the best to achieve the maximum relative impact.

One has to take into account however, that the UM type of projects were unevenly distributed across the different geographic areas. Almost absent from the Tacis countries, they were particularly popular in the last two years of Tempus II in some Phare countries. For example in Poland 51 % of the total number of proposals submitted in 1998 was of the University Management type, whereas in Slovenia the percentage was 15 % only. This might be an indication of a different level of readiness to carry out reform in the various countries. One can also easily ascertain that rather different types of projects are in fact grouped under the UM label: from equipment provision to faculty or central libraries and the establishment of management information systems to the development of strategic planning and the introduction of democratic principles in university governance. There was a general consensus in acknowledging the impact on the more material domains (computerisation, update of services and facilities, provision of equipment, etc.). On the other aspects opinions tended to diverge. The higher education authorities emphasised in their answers the Tempus impact in their own field of competence, that is the decision making process, whereas the JEP participants underlined the internal reform aspects. In some cases the impact was perceived to have been a side effect of Tempus activities.

In conclusion, the findings confirm a mixed response partly due to the sensitivity of the issue for the people involved in the survey, and partly for the objectively heterogeneous situations in which the programme operated.

Outcome 6 focussed on the programme impact on curriculum development. Curriculum development projects have been the core elements to Tempus and therefore the findings of the evaluation report can be considered of the highest interest. A significant component to these projects is staff and student exchanges. The report clearly states the significance of this Tempus component: the mere exposure of thousands of students and staff to new ideas and social environments is recognized to have tremendously contributed to raising the awareness of national and societal values, stimulating at the same time the dialogue between cultures, critical thinking and changes of attitude towards study, work and life in general.

More specifically curriculum development projects had the objective of updating and restructuring the contents of existing courses and introducing entirely new curricula. In Tacis countries modernisation and restructuring of courses and curricula occurred most frequently. In Phare countries focus was more on the introduction of new subjects, with a marked preference for interdisciplinary subjects. Regarding the relevance of degree programmes to the needs of the labour markets, Tempus contributed by responding to the emerging demand for new competencies, even with limited involvement of the different economic actors.

The perceived importance of Tempus in the different groups of eligible countries was highlighted by a specific question put by the evaluator. Asked whether the curriculum development changes introduced thanks to Tempus would or would not have occurred without Tempus, responses significantly varied depending on the areas where the programme operated. For example, in Tacis countries a large proportion of respondents indicated that many of the changes would not have taken place at all, while in the candidate countries of Phare the prevailing view was that reforms would have occurred sooner or later, but that Tempus speeded up the process. Finally, in the countries of the former Yugoslavia and in Albania, people thought that without Tempus reforms would probably have occurred much later, with many adding that 'later' would probably have meant 'too late'.

The Commission is particularly pleased that the programme is broadly perceived as having had a positive impact on the employability of students and on the reputation of the universities and their ability to attract students.

4.1.3. Contribution to human resource development

The contribution of the programme to the training of decision-makers was to be analysed under Outcome 7. The 2001 impact study of Tempus in Latvia - mentioned in the evaluation report - sheds light on the career development of a few participants in the programme. A list is provided that includes the head of the higher education council, the head of the Latvian Rectors' Conference, several rectors, the head of department of higher education and research in the Ministry of Education and several other senior Ministry officials. Although an extended cohort study to track careers of former Tempus grant holders was never undertaken, the evaluator believes (with some good reasons) that Latvia is no exception and that especially in small countries the programme had a tremendous impact on human resource development. From the survey it clearly emerges that the experience made in Tempus strengthened the academic knowledge of the participants and/or their interpersonal skills, contributing to their move to senior positions both inside and outside their academic institutions. This conclusion is important and it is a striking example of how the bottom-up approach can be effective in the medium term.

The ability of universities to retain trained people was also investigated. Staff and student mobility has often been accused of increasing the risk of brain drain. No such evidence emerged from the survey. It must be observed however, that in this specific case the sample used (project co-ordinators and higher education authorities) was probably not very appropriate. A survey among former IMG grant holders could certainly have given more significant results.

4.1.4. Towards a sustainable partnership?

The last outcome (Outcome 8) in which the evaluator was requested to provide a response was about sustainable partnership, a key-issue for each funded co-operation programme. The report clearly indicates that enthusiasm for academic co-operation is the greatest resource of the programme. The overwhelming majority of Tempus participants are prepared to stay in touch with their Tempus partners. The results of the survey are a strong indication that more than half of the JEP consortia have indeed succeeded in building up a sustainable partnership. This is an impressive outcome by any standard.

A caveat should perhaps be raised. In assistance programmes sustainability is defined as the ability to maintain an adequate level of results over a longer period of time after termination of the donor's financial support. This definition needs to be adapted to Tempus specific features. As a partnership programme Tempus, to a large extent, can be described as a co-operation programme, contributing more than just technical assistance. A Tempus project can therefore be considered sustainable if the partnerships (valuable networks of academic contacts within and outside the EU built during the project) remain active and relevant following the end of project funding. Among the forms that sustainable partnerships have taken after completion of the projects, the most frequently mentioned in the survey were: academic networking and participation in conferences; exchange of academic staff; joint research and publication; and regular exchange of students. Of course there is no sustainable partnership at zero cost, but the evaluators emphasised that even if nominally funded by the EC, Tempus projects have always been largely based on personal commitment and good will.

A remarkable fact is that partnerships seem to have been most important in Tacis, where over 80% of respondents indicated to have maintained a form of sustainable partnership. A factor here may be the fact that in this area, Tempus was and is the only programme of this nature, whereas in the Phare countries, universities have access to other programmes through which to continue international co-operation. Many in fact noted the Socrates programme as the most natural umbrella under which to continue the co-operation. University budgets were also indicated as a main source of funding to continue co-operation (alongside bilateral programmes, multilateral organisations, private foundations and new Tempus projects). The only exception in this rosy scenario is perhaps to be found in the countries of the former Yugoslavia (plus Albania) where higher education authorities do not seem very optimistic about sustainability of partnerships.

An interesting complement to the above is the issue of incentives for EU universities to take part in Tempus, which was also investigated by the evaluator. The desire to assist the transition process was surprisingly ranked as high as the professional interest in joint curriculum development. Of course the interest for the institutions to receive EU funding was also quoted, suggesting that Tempus JEPs are apparently still considered a net source of income for EU universities.

4.2. Programme management

An assessment of programme management was not foreseen in the "Terms of Reference," but with the Commission's agreement, the evaluators decided to introduce a few questions on this aspect in their survey.

With the online questionnaire, the following aspects were looked at:

1. The information material provided to the applicants;

2. The programme technical assistance and the project management reporting mechanisms;

3. The support provided by the National Tempus Offices (NTOs) or Tempus Information Points (TIPs).

The information material was, in general, well appreciated. On the other hand, there were some criticisms on the procedures for administrative reporting and invoicing; both described as time consuming and excessively inclined towards control. Representatives from EU universities mentioned that this high administrative burden was perceived as one of the main barriers for new universities participating in the programme.

Appreciation for the support received from the National Tempus Offices varied significantly between the regions. High appreciation was marked in the ten acceding countries of the Phare area and in the countries of the former Yugoslavia (plus Albania). In the Tacis area, the answers show a lower level of satisfaction with 30% being dissatisfied with the support available. This result reflects the situation of the Russian Federation, where for some time, Tempus did not succeed in establishing an office. Having noted this however, it did not seem to have prevented an excellent response to the calls from the higher education institutions.

In general these results are in line with expectations. In recent years there has been a significant effort to improve the administrative procedures and simplify the project cycle, and the effects of these improvements are anticipated to emerge gradually.

5. Summary of conclusions

The Commission agrees with the evaluators' findings that the second phase of Tempus has demonstrated the flexibility of the programme and its ability to evolve in response to the different socio-economic conditions and political perspectives in the various regions where it operates. It was also shown that dialogue with national authorities and the mechanism of setting national priorities is a valid instrument for orienting the programme towards the most urgent local needs and making sure that Tempus activities have a higher chance of embedding to sustain longer term reform.

The external evaluators' report clearly shows that Tempus II has significantly contributed to removing barriers to international mobility of students. The core element of the programme remains curriculum development. Tempus's most durable impact has probably been on training people who will shape the higher education environment for decades to come, and on providing students with skills and qualifications needed for employment and career development.

The Commission is particularly pleased that the programme instruments have been proved to be equally appropriate and adaptable to different environments. In Phare countries Tempus II has been instrumental in opening the higher education sector to dialogue and co-operation with their EU partners. It also addressed the needs of new skills linked to the process of economic reform and EU accession. Finally, it has contributed to positioning them in the best situation in order to more fully reap the benefits of participating in other EU education programmes, notably Socrates.

In the countries of former Yugoslavia (plus Albania) the participation in Tempus has offered a safety belt to education and science, during a period of sharply declining national resources for the sector. In former Yugoslavia in particular, outcomes of regional co-operation has helped to mitigate the devastating effect of the break-up of the previous state-based higher education system.

Given the modest financial contribution in Tacis countries, the Tempus impact on curriculum development and reform is broadly recognised by project participants and national authorities to have gone beyond expectations in triggering momentum for change.

This report is meant to be a contribution to the analysis of the medium term impact of the second phase of the Tempus programme. The Commission recognizes, however, that other aspects might be further investigated, for example the following:

- the process of embedding the newly developed courses into the regular curricula;

- the barriers opposing effective dissemination of Tempus results;

- the employability issue and the relations of Tempus with industry and the private sector;

- the impact of Tempus on the phenomenon of the 'brain drain'.

List of documents

Final Report of the second phase of the Tempus Programme, by Ecorys

Evaluation of the first two years of Tempus II, by Kehm, Mauworm, Over, Reisz, Steube, Teichler

Evaluation of the Tempus Programme in the NIS and Mongolia, by FTP International

Tempus @ 10, by Ard Jongsma

From Association towards Accession, by Ewa Kolanowska, ETF (Turin)

Annex I

Legislation

>TABLE POSITION>

Annex II

Chronology and overall country allocations during 1994-1999

>REFERENCE TO A GRAPHIC>