[pic] | COMMISSION OF THE EUROPEAN COMMUNITIES |

Brussels, 13.6.2008

COM(2008) 356 final

REPORT FROM THE COMMISSION TO THE COUNCIL, THE EUROPEAN PARLIAMENT, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS

on the external evaluation of the European Centre for the Development of Vocational Training

TABLE OF CONTENTS

1. Introduction 3

2. The European Centre for the Development of Vocational Education and Training 3

2.1. Background to Cedefop 3

2.2. Priorities and activities 3

2.3. Previous evaluations, studies and reviews 4

2.4. Funding 4

3. Overview of external evaluation exercise 5

4. Findings to Main evaluation Questions 5

4.1. Relevance and Complementarity 5

4.2. Effectiveness 6

4.3. Efficiency 6

4.4. Added value and impact 7

5. MAIN RECOMMENDATIONS OF THE EXTERNAL EVALUATION 7

5.1. Strategic issues 7

5.2. Operational issues 8

5.3. Internal management issues 8

6. Conclusion 9

Introduction

This report concerns the external evaluation of the European Centre for the Development of Vocational Training (Cedefop), a Community Agency established under Council Regulation 337/75 of 10 February 1975[1]. Article 27(4) of the Financial Regulation applicable to the general budget of the European Communities[2] and Article 21.3b of the implementing rules[3] specify that the Commission evaluates such activities on a regular basis and disseminates the evaluation results to spending, legislative and budgetary authorities.

The external evaluation took place during 2006-2007 covering the period 2001-2006.

This report is based on the findings, conclusions and recommendations of the final report, submitted by the evaluator which, is available in full, including annexes, at http://ec.europa.eu/dgs/education_culture/evalreports/index_en.htm.

THE EUROPEAN CENTRE FOR THE DEVELOPMENT OF VOCATIONAL EDUCATION AND TRAINING

Background to Cedefop

Cedefop [4] is the European Union’s agency for Vocational Education and Training (VET); it is located in Thessaloniki[5], Greece. Cedefop supports the European Commission, Member States and social partners in developing and implementing European VET policy.

It is administered by a quadripartite Governing Board with representatives of Member State governments, employer and employee organisations and the European Commission. The Centre is managed by a directorate, comprising the director and deputy director and has around 130 staff (professional and support staff). Cedefop’s overall budget is around € 17 mil. per year. Detailed information about Cedefop is available on its website (footnote 4).

Priorities and activities

In line with its Founding Regulations, Cedefop’s priorities and activities are defined in medium-term priorities (valid for 3 years) and annual work programmes[6].

Since 2000, the Lisbon strategy has stimulated a rapidly evolving and comprehensive European agenda in vocational education and training (VET). Over this period Cedefop has realigned its strategy to focus on supporting the European Commission, Member States and the social partners in developing European VET policy.

The strategic realignment of Cedefop is reflected in its priorities:

1. providing evidence from research, statistical data and policy analysis to support VET policy-making;

2. increasing knowledge and mutual learning on VET policy and practice in Member States by reporting on developments and policy options;

3. strengthening European cooperation in VET policy development by providing expertise to help design and support common European VET initiatives and tools;

4. increasing the visibility and understanding of VET issues through effective communication.

To achieve these goals Cedefop uses its internal expertise, research and policy analysis skills, networking experience, data resources and dissemination tools to:

5. support exchanges of information and experience and the sharing of good policies through study visits, networks, conferences and seminars;

6. provide stakeholders and citizens with relevant information on key issues via electronic and printed publications;

7. involve candidate countries in the EU VET policy framework, working closely with the European Training Foundation (ETF).

Cedefop’s activities are specified in detail in its annual work programmes.

Previous evaluations, studies and reviews

An external evaluation was finalised in 2001[7] and led to a Commission position paper[8] and a Cedefop action plan[9].

Funding

Cedefop’s overall budget and allocation between different titles:

Year | Overall budget in € | Title 1: Staff expenditures In € | Title 2: Administrative expenditures in € | Title 3: Operating expenditures in € |

2008 | 17.604.322 | 10.603.000 | 1.409.000 | 5.592.322 |

2007 | 17.374.402 | 9.881.740 | 1.928.260 | 5.564.402 |

An overview of Cedefop's budget for the 2001 – 2006 evaluation period is provided in Annex 2; detailed information can be found in the Centre's annual accounts, which are available on Cedefop's website (footnote 4).

Overview of external evaluation exercise

DG Education and Culture (DG EAC) contracted Ecotec Research and Consulting Ltd[10], to carry out the external evaluation of Cedefop. A Steering Committee, chaired by DG EAC, with one representative of each of the different groups of Cedefop’s Governing Board and two Cedefop staff members was set up to support and follow the work of the contractor.

The overall objectives of the evaluation were agreed as:

- an assessment of the relevance, complementarity, effectiveness, efficiency, added-value and impact of Cedefop’s activities and organisation in achieving the key objectives, priorities and tasks defined in its guiding policy documents for 2001-2006, such as the medium-term priorities and annual work programmes etc);

- provision of useful lessons and recommendations to enable Cedefop to face the challenges of coming years. Particularly important are recommendations that help improve Cedefop’s programming, management, performance and impact of its products and services and its accountability to the Budgetary Authority and the public at large. The evaluation should have a formative character.

The external evaluators followed the prescribed method and terminology for evaluation under the DG Budget Guidelines[11], and used several methodological tools. They gathered comprehensive evidence from literature, policy documents, web searches, a survey of Cedefop activities, and four in-depth case studies. To support this analysis, the views of over 100 key actors were captured through interviews and focus groups, and more than 650 respondents in two web surveys (details provided in Annex 3 of the report).

FINDINGS TO MAIN EVALUATION QUESTIONS

This section summarises the principal evaluators’ analysis and findings. Overall they conclude that Cedefop is a relevant and effective organisation with a clear distinct added value. Providing a wide range of activities and considering the ratio between inputs to outputs, Cedefop can also claim to be efficient.

Relevance and Complementarity

The evaluation concludes very positively on Cedefop’s relevance and complementarity. It notes that Cedefop is not only responding to the emerging EU VET policy agenda but is also helping influence its development. Cedefop has become an active promoter of EU VET policy at the highest levels. It is also the evaluators’ assessment that Cedefop has performed a critical role as an 'open source' of information for the VET community at large since its foundation, acquiring a strong brand reputation and visibility in European VET.

The evaluation points out that although Cedefop’s 'open source' role serving a wide VET community was successful, now may be the time for Cedefop to focus more on the interests of its immediate stakeholders, particularly in policy development. The evaluation correctly states that these two activities ('open source' – supporting policy development) are not incompatible. Although Cedefop now provides more support for policy development, many activities serve the wider VET community. The issue is not one of incompatibility, but of balance that needs to be captured in strategy.

The evaluators also perceive Cedefop to be playing a complementary role with other agencies, and found evidence of Cedefop being aware of the need to guard against duplication and taking active steps to be clear as to respective functions. Looking to the future, and given the growing importance of Cedefop’s dual role (supporting policy development; 'open source'), this boundary terrain with other agencies will continue to require active management.

Effectiveness

Cedefop is doing a good job, satisfying the majority of its users, having a very good reputation and strong brand image in European VET. It is a valued source for reporting, collecting and analysing VET information at European level and recognised for its role in VET research and disseminating VET information. The evaluators found that Cedefop’s activities evoke good feedback across the board.

The evaluation points to some of Cedefop’s strengths and weaknesses. Strengths include networking, encouraging peer learning and providing comparable information. Cedefop adds most value through bringing together policy and research, in particular, by applying research findings to policy issues. In this way Cedefop made an effective contribution to the follow up to the Copenhagen process for European cooperation in VET[12], a task of great breadth and complexity completed successfully with few resources. Further, Cedefop skills forecasting activities fill a knowledge gap at European level.

Weaknesses lie in Cedefop’s communication strategy, in particular through its websites, and the absence of a well developed evaluation culture. Further aspects of the work with ReferNet, - Cedefop’s main information providing network – need attention, in particular ReferNet’s ability to provide information validated by national authorities that accurately reflects VET developments and the network's visibility in the Member States.

Efficiency

The task of assessing the efficiency of the agency proved to be more difficult. Measuring efficiency requires looking at how resource input (staff and money) are transformed into activity outputs, to estimate value for money and comparative costs. The financial and project reporting data available did not enable the evaluators to measure efficiency scientifically. The evaluators consider that activity based budgeting started to be implemented towards the end of the evaluation period and that measured performance indicators were only comprehensively available for the first time in 2005, which allowed making partial comparability observations between 2005 and 2006.

To address the absence of scientific data, the evaluators used a number of proxy measures to assess the degree of efficiency of Cedefop. They conclude that the scale of activity has been simply enormous and considering the ratio of inputs to outputs (wide range of activities and products), Cedefop can lay good claim to be efficient.

Evaluators reported that some of these issues were already taken on board by the current management team, however too late for this evaluation. The full implementation of activity-based budgeting in Cedefop in 2008 is considered an important step to facilitate future assessment exercises.

Added value and impact

According to the evaluators' report, Cedefop is an organisation with a very distinct added value. There simply are no valid alternatives for what it does and for many of its activities it has no obvious peer. No other organisation has a dedicated focus on and a Europe-wide pool of experience and competency applied to VET.

The evaluation noted several areas where Cedefop clearly has a positive impact and brings added value. These activities represent key strengths of the organisation, namely:

- analysing progress in the Copenhagen process to enhance European cooperation in VET on an informed basis and producing reports for ministerial meetings. This led to Cedefop being given a stronger mandate to monitor and report on progress in the Member States in implementing European VET policy;

- bringing together relevant VET research to interpret current trends and encourage a European approach to tackling VET issues;

- filling knowledge gaps by providing much needed analysis of current and future skill needs in Europe;

- helping strengthen European cooperation by providing expertise to help design, develop and implement European tools such as Europass, European qualification framework (EQF) and European credit system (ECVET);

- promoting understanding and peer learning, by bringing people together – practitioners, researchers and, notably, social partners. In 2006, the European Commission invited Cedefop to coordinate the new consolidated study visits programme in the lifelong learning programme[13] from 2008-13.

MAIN RECOMMENDATIONS OF THE EXTERNAL EVALUATION

This section provides an overview of the evaluation’s 16 recommendations. They are grouped under three headings: strategic, operational and internal management issues.

Strategic issues

- Cedefop should continue to search for ways to reduce the breadth of its activity portfolio. It should focus its resources on fewer core activities and consolidate its management to maximise strong leadership across its operational areas. The medium-term priorities for 2009-11, which Cedefop is currently formulating, are a good opportunity to address the strategic balance of its operations between 'open source' provision and contributing and supporting implementation of EU VET policy priorities. An improved communication policy should enable Cedefop to get the backing of its stakeholders and users for possible strategic readjustments. (recommendations 2, 3, 4 and 5);

- Cedefop should use better its clear and coherent strategic objectives to make its management-by-objectives culture a day-to-day practice, cascaded throughout the entire staff (recommendation 1)

- Cedefop should take steps to assemble an investment/restructuring fund (within the allowable rules) to allocate development resources to new ventures and to break out of the zero-sum cycle of paring down (recommendation 6).

Under this heading, the evaluator also addresses two specific recommendations to the Commission, namely to:

- assist Cedefop to explore ways to respond more flexibly to the strategic demands placed upon it (recommendation 7);

- explore with the relevant European services and institutions how it might be possible within the regulations to give Cedefop a longer-term planning horizon for its actions (recommendation 8).

Operational issues

- Research, advice and policy support: more attention should be given to producing concise, timely and focused outputs for policy makers. Cedefop is at the crossroads of policy and research. This must be communicated better to its wider community of users and stakeholders to emphasise its unique role in applying the latest research to EU VET problems and in filling knowledge gaps. Cedefop should develop a 'foresight' process exploring emerging VET related issues beyond 2010 and feed into development of the EU VET agenda (recommendations 12, 13 and 14).

- Communication, information and dissemination: Cedefop’s communication strategy should be reviewed to take better account of stakeholder and target group needs. Action should be taken to realise ReferNet’s full potential and use to the full the qualities embedded in the model (recommendations 9 and 10);

- Exchange and cooperation: the study visits programme should be extended and developed into a platform to identify, disseminate and exchange good practice in VET across the EU (recommendation 11);

Internal management issues

- Activity-based budgeting should be complemented by activity-based financial reporting. This will enable Cedefop to assess accurately the actual resources used for its activities (as opposed to those budgeted for). It will also assist when making strategic choices knowing the opportunity cost of each activity (recommendation 15);

- To strengthen Cedefop’s evaluation culture, measures for value for money and effectiveness (unit costs, comparative cost analysis) need to be installed to periodically take stock of performance. This should include being able to report impact as well as visibility (recommendation 16).

The Commission and Cedefop accept overall the recommendations made by the evaluators. Cedefop has limited resources and it is important to use them in the best way. Given the diversity of the main stakeholders (governments; employer organisations; trade unions; European Commission) and of the user community (from policy makers to practitioners), it is important to find the right mix and balance between the activities related to supporting policy development and implementation and those related to the provision of support as an 'open source' for VET issues. The Commission and Cedefop will draw up detailed action plans for the follow-up of the findings and recommendations of the external evaluation.

CONCLUSION

The Commission shares the evaluators’ overall positive assessment of Cedefop's work. It considers Cedefop has made a valuable contribution to the Community's activities in VET. Since 2000, the Lisbon strategy, Education and training 2010, the Copenhagen process and EU enlargement have dramatically changed the European context for VET. The evaluation acknowledges Cedefop’s increasingly complex and demanding environment over the last six years. It concludes that Cedefop responded well, delivering greater flexibility, impact and value.

In particular, Cedefop supported the policy development process by helping implement the Education and training 2010 work programme and monitoring and reporting on Member States’ progress in implementing European VET policy priorities agreed in the Copenhagen process. Cedefop contributed to strengthening European cooperation by helping develop European tools such as Europass and the European qualifications framework and by stimulating exchanges and peer learning through the study visits programme.

The evaluation concludes that Cedefop has a strong brand image, brings a very distinct added-value and for many activities has no obvious peer. " No other organisation has a dedicated focus on VET and a Europe-wide pool of experience and competency applied to the field. Indeed, there is a strong argument that if Cedefop did not exist an organisation that looks something like it would probably have to be invented."

During the evaluation period, 2001-06, Cedefop moved away from its more traditional role as an ‘open source’ at the disposal of the wider VET community, to become more proactive in supporting the development of EU VET policy. This strategic shift started in 2002 with the Copenhagen declaration, accelerated with the Maastricht and Helsinki communiqués and is expected to continue.

The evaluation found that the Governing Board largely supports Cedefop’s strategic realignment. However, to make such a shift successful requires prioritising and resourcing activities effectively, yet Cedefop continues as an ‘open source’ for VET in Europe, pursuing its support to the policy mission in tandem. The evaluators consider that ‘ there is no problem with this, of course, so long as the level of resources can sustain it and where the pool of competency in the organisation can span both .’

However, as Cedefop is expected to provide even greater support for policy development and implementation and resources are limited, it is important to focus the available resources on fewer core activities. Cedefop must move away – to some degree – from its ‘open source’ role. That means painful choices about stopping activities that have a value of their own, but do not add substantial value to core stakeholders.

Cedefop needs to review its information and communication strategy and services to target information to specific groups of stakeholders who require different types and formats. It is important that Cedefop involves and explains to the wider VET community its strategic realignment to policy-related support activities.

Full implementation of activity-based budgeting in 2008 by Cedefop is an important step to make visible and understand better the relationship between investment and outcome for different activities, and so their strategic value to the organisation and its stakeholders. As recommended by the evaluators, Cedefop should investigate how this system can be used for financial monitoring and reporting to assess accurately the actual resources used by activities (as opposed to the resources budgeted for). This should be complemented by relevant measurable indicators to make follow-up of objectives and goals easier and to help assess Cedefop's impact in its areas of intervention.

Cedefop's management must continue its effort to establish a clear vision, cascading its strategic mission throughout the entire staff.

The external evaluation report provides useful findings and recommendations for further development of Cedefop as the EU’s agency for VET. The concrete follow-up of the evaluation’s findings and recommendations will be ensured on two levels:

- an action plan submitted by Cedefop to its Governing Board outlining its analysis and proposed actions on the evaluation’s recommendations. Implementation of the action plan will be monitored by the Governing Board.

- an action plan drawn up by DG EAC, taking into account Cedefop’s action plan, but addressing the findings and recommendations of the evaluation beyond Cedefop's competencies, in particular those addressed to the Commission.

[pic]

Annex 2

BUDGET EVOLUTION, 2001-2006

Community subsidy | Revenue (entered in the final budget and effectively collected) | Expenditure appropriations (entered in the final budget) per title | Expenditure appropriations (committed and paid) | Human resources |

Year | |Revenue entered

in 1000 € |Revenue collected

in 1000 € |Staff

(Title 1)

in 1000 € |Administration and infrastructure

(Title 2)

in 1000 € |Operations

(Title 3)

in 1000 € |Commitments done

in 1000 € |Payments done

in 1000 € |Establishment Plan

| Total nb of staff (at date of 31.12 of year) | | 2001 | B3-1030: 8,5 m€

B3-1031: 4,7 m€

Total: 13,2 m€ | 13.600 |13.500 |7.500 |1.000 |5.000 |13.500 |10.600 |81 posts |127 | | 2002 | B3-1030: 9,2m€

B3-1031: 4,5 m€

Total: 13,7 m€ |14.200 |12.600 |8.100 |1.100 |5.000 |13.900 |10.900 |83 posts |123 | | 2003 | B3-1030: 9,4 m€

B3-1031:5,1 m€

Total: 14,5 m€ |14.700 |15.300 |8.000 |1.200 |5.500 |15.400 |11.300 |83 posts |125 | | 2004 | 15 03 01 03:

10,638 m€

15 03 01 04:

5,162 m€

Total: 15,8 m€ | 16.546 |14.466 |9.243 |1.395 |5.958 |16.313 |12.270 |88 posts |137 | | 2005 | 15 03 01 03:

10,662 m€

15 03 01 04:

5,438 m€

Total: 16,1 m€ | 17.093 |16.989 |9.468 |1.372 |5.578 |15.290 |14.381 |91 posts |123 | | 2006 | 15 03 01 03:

10,962 m€

15 03 01 04:

5,438 m€

Total: 16,4 m€ | 17.563 |15.313 |9.443 |1.419 |5.978 |15.872 |13.475 |95 posts |123 | |Since move to Activity Based Budgeting in 2004: 15 03 01 03 replaces B3-1030; 15 03 01 04 replaces B3-1031

[1] OJ L 39, 13.2.1975, p. 1–4

[2] Council Regulation (EC, EURATOM) No 1525/2007 of 17 December 2007 amending Regulation (EC, Euratom) No 1605/2002 on the Financial Regulation applicable to the general budget of the European Communities

[3] Commission Regulation (EC, Euratom) N° 478/2007 of 23 April 2007, amending Regulation No 2342/2002 of 23 December 2002

[4] Cedefop's website at http://www.cedefop.europa.eu/

[5] [6].7ABEEstablished in 1975 and originally based in Berlin, in 1995 Cedefop’s head office was transferred to Thessaloniki

[7] The mid-term priorities and the annual work programmes are accessible at Cedefop's website (see footnote 4).

[8] http://www.ec.europa.eu/education/programmes/evaluation/cedefop/final_report_pl_ramboll_en.pdf

[9] http://www.ec.europa.eu/education/programmes/evaluation/cedefop/annex_position_paper_en.pdf

[10] http://www.ec.europa.eu/education/programmes/evaluation/cedefop/actionpl_en.pdf

[11] Following a competitive tendering procedure, Ecotec Research and Consulting Ltd has been awarded a framework contract for evaluation of related activities of DG EAC

[12] ‘Evaluating EU Activities: A practical guide for the Commission services’, European Commission, Directorate General for the Budget, July 2004

[13] Information on the Copenhagen process at http://ec.europa.eu/education/policies/2010/vocational_en.html

[14] Council of the European Union. Council Decision establishing an action programme in the field of lifelong learning (1720/2006/EC). Official Journal of the European Union L 327/45, 24.11. 2006.