12.12.2006   

EN

Official Journal of the European Union

C 302/1

I-XI

1-9

6-9

7-9

10-14

15-80

16-66

16

17-20

21-28

29-40

41-44

45-53

54-66

67-75

67-70

71-75

76-80

81-113

82-113

82-87

88

89

90-93

94-96

97-102

103-110

111-113

114-118

119-125

ADAPT

Adaptation of the Workforce to Industrial Change (a Community initiative to promote employment and the adaptation of the workforce to industrial change)

ALMP

Active Labour Market Policies

CSF

Community Support Framework

Deadweight

Change observed among direct addressees following the public intervention, or reported by direct addressees as a consequence of the public intervention, that would have occurred, even without the intervention.

Displacement effect

Effect obtained in an eligible area at the cost of another area.

ERDF

European Regional Development Fund

ESF

European Social Fund

FIFG

Financial Instrument for Fisheries Guidance

HERMIN model

Macroeconomic model used to simulate the impact of SF interventions

Inception report

Report containing methodological questions such as the evaluation objectives, the methodological framework, the questionnaires and the organisations to be contacted

Monitoring Committee

Committee to oversee the strategic implementation of the funding programme

OP

Operational programme

RTD

Research and technical development

SFs

Structural Funds

SME

Small and medium-sized enterprises

Soft outcome

Result difficult to measure quantitatively

SPD

Single programming document

Synthesis Report

Final reports of DG REGIO (concerning Objective 1) and of DG EMPL (concerning Objective 3) about the ex post evaluation of operations financed by the ERDF and the ESF

ToR

Terms of reference

I.

The audit concerned the Commission's ex post evaluations of Structural Funds interventions in Objective 1 and 3 regions over the 1994 to 1999 programming period totalling in excess of 245 billion euro, including matching private and public sector investments.

II.

Ex post evaluation seeks to compare actual expenditure outcomes against the original objectives. It is the mechanism whereby insights gained during one programming period are exploited to make future SF expenditure more effective. The ex post evaluations were carried out by the Commission which relied on contracted external consultants.

III.

The audit objectives were to examine if, at the conclusion of the ex post evaluation exercise, the Commission was able to estimate the effects of the interventions relative to the objectives and if the results and recommendations have been helpful for the revision of the programmes from 2000 onwards.

IV.

The starting point for the audit was the final Synthesis Report for each fund. These reports summarise the overall assessment of the impact of the funds and capture the elements that should be of relevance when taking decisions concerning subsequent planning periods. The next step in the audit was to assess the extent to which any identified deficiencies could be traced back to weaknesses in the ToR or problems during implementation of the evaluation process.

V.

Evaluating Structural Fund expenditure is an intrinsically complex process. However, the audit identified weaknesses in the evaluation process that cannot be attributed merely to such inherent constraints.

VI.

The Court identified the following weaknesses in the evaluation of SF interventions in Objective 1 regions:

VII.

The Court found that the evaluation of ESF operations under Objectives 1 and 3:

VIII.

The Commission does not appear to have anticipated some of the difficulties involved in the evaluations in the terms of reference which it formulated for them. Nor did the Commission fully address the shortcomings in the ex post evaluation exercise. The Court’s findings indicate that there were weaknesses in the oversight of the evaluations.

IX.

On the other hand the evaluations yielded useful insights concerning the management systems for ERDF and ESF-funded interventions.

X.

However, while some worthwhile and useful recommendations were made as input for future programming periods, the Commission could not benefit fully from the experiences of the 1994 to 1999 programmes.

XI.

The Court recommends that

1.

Since 1996, the Commission has been developing the evaluation of Structural Funds within a single framework. The term ‘evaluation’ is defined by the Commission as the ‘judgement of interventions according to their results, impacts and needs they aim to satisfy’ (1). The main purpose of the evaluation activity is to support decision making by providing information and judgements about the relevance, economy, efficiency, effectiveness, utility and sustainability of the impacts of EU interventions. Such information is crucial for improving existing, or designing new, interventions and for setting political priorities and allocating resources.

2.

Evaluations are carried out at various levels and stages in the programming cycle (six years) the most important of which are:

3.

The Court’s audit concerned ex post evaluations which the Commission carried out in connection with SF interventions in Objective 1 and 3 regions over the 1994 to 1999 programming period. For the purpose of this audit, the scope of ex post evaluation is taken as that defined in the MEANS collection (2), namely: ‘Ex post evaluation recapitulates and judges the entire programme, particularly its impacts. Its aim is to account for the use of resources and to report on the effectiveness and efficiency of interventions and the extent to which expected effects were achieved. It focuses on factors of success or failure, and on the sustainability of results and impacts. It tries to draw conclusions that can be generalised and applied to other programmes and regions’ (3). It is against this background that the ex post evaluation exercise has been assessed.

4.

The ex post evaluation task is part of a ‘process’ that also draws on the results of ex ante assessment and mid-term review. Furthermore, evaluations are undertaken at various levels both national and thematic and culminate in a Synthesis Report. Together they constitute an important element in the management of the Structural Funds and, in particular, the programming and review of Community assistance. Responsibility for the evaluation process rests with the Commission (in partnership with the Member States). It is up to the Commission to decide which aspects of the evaluation to carry out in-house and which it needs to subcontract. In either case, overall and final responsibility still rests with the Commission.

5.

Over the 1994 to 1999 period, the aid scheduled under the Structural Funds (4) amounted to 136,4 billion euro for ‘objectives’ and to 14,5 billion euro for ‘Community initiatives’. Annex I provides a breakdown of the former amount by Member State and objective and of the latter by Community initiative. These funds were supplemented by at least a further 95 billion euro of matching private or public sector investment (5) bringing the total investment to over 245 billion euro.

6.

The ex post evaluations were carried out on behalf of and under the direction of the Commission. Following a public tendering procedure, a principal consultant was selected for each specified set of objectives and Community initiatives. In most cases, the consultant was supported by a team of evaluators in the individual Member States (6) whose role was to undertake the necessary evaluations of the respective programmes on the basis of a common methodological framework. Moreover, the evaluation process was monitored by a steering group. The evaluations resulted in national reports prepared by the national evaluators and a Synthesis Report prepared by the contracted consultants for each specified set of objectives. These reports were completed between early 2003 and mid 2004.

7.

The DGs involved opted for a wide-ranging approach covering a broad list of issues. The overall objectives of the evaluations were to:

8.

To meet these overall objectives, the terms of reference (ToRs) established a number of key questions around the themes of appropriateness of strategy, effectiveness, efficiency, impact, management and implementation systems, Community added value and lessons from current and for future programming. To structure the work, the terms also established a number of specified tasks and identified a certain number of techniques and sources of information.

9.

The ToR for each objective, as compiled by the Commission, thus provided the basis for the work of the evaluators. These were complemented by the inception report and the methodology guide for Objective 1, in January 2002. For ESF operations, they were complemented by the methodological progress report, in December 2002, and the evaluation questions, included in each national evaluation report. For this purpose, the evaluators relied on literature reviews, questionnaires, stakeholder interviews, the analysis of case studies and other field research as deemed necessary.

10.

The general audit objectives were to examine if the evaluations provided the Commission with adequate information to estimate the effects of the interventions relative to the objectives and if the results and recommendations have been helpful for the revision of the programmes for subsequent programming periods.

11.

The audit focused on the ex post evaluations concerning ERDF interventions under Objective 1 and ESF interventions under Objectives 1 and 3, undertaken respectively by the Directorate-General for Regional Policy (DG REGIO) and the Directorate-General for Employment, Social Affairs and Equal Opportunities (DG EMPL). The relevant Synthesis Reports and a sample of national evaluations were examined corresponding to these selected objectives and instruments (7) which, together, accounted for 124 billion euro, that is 82 % of the total scheduled aid under the SFs for the 1994 to 1999 programming period (8).

12.

For each evaluation, the audit sought to answer three questions:

13.

In particular, the audit assessed the quality and completeness of the reports in terms of their evaluation of the appropriateness of the policies, and the effectiveness, efficiency and impact of the measures, as well as how the measures impacted on the management systems. It also assessed the adequacy of the Commission's role in the prepartion, monitoring and supervision of the evaluations.

14.

The first part of this report is dedicated to the ex post evaluation of ERDF interventions in Objective 1 regions (paragraphs 15 to 80) while the second part covers the ex post evaluation of ESF operations under Objectives 1 and 3 (paragraphs 81 to 113).

15.

Before undertaking the detailed assessment of the quality of the evaluation, it is useful to provide a brief overview of the conclusions of the Commission's evaluation. These are summarised hereunder (see Text box 1).

Text box 1The overall conclusions of the evaluation by DG REGIOThe Synthesis report recognised that the Structural Funds have had a positive impact on the GDP of the Objective 1 regions and that their overall performance, relative to the EU as a whole, has improved. It considered that the extent of the beneficial impact has been heavily dependent on both institutional capacities and factors such as the structure and openness of the economy and that ‘these aspects are clearly crucial in particular to the capacity of the economy to respond to the challenges and opportunities of the process of economic integration’. It noted, however, that given the disparity of countries and contexts and because of limitations of the available monitoring data and lack of clearly defined objectives, it was not possible to aggregate the results to give a reliable picture of the overall Objective 1 achievements.As an overall assessment of the appropriateness of the strategies adopted, the report concludes that the strategic approach to Objective 1 taken in 1994 must be ‘viewed as broadly appropriate in each of the Member States’. In its view: ‘The most significant criticism of the strategies was that they did not constitute much more than the sum of their parts. Whilst each was an adequate response to the identified needs in particular fields, there was less indication as to how these different elements contributed towards an overarching development strategy.’In terms of effectiveness, the report indicated that the analysis relied to a substantial extent on ‘qualitative judgements rather than quantified comparisons of results with targets’ due to shortcomings in the available data. As a general comment, the report states that: ‘Many of the apparently most effective actions have been those of a largely “mainstream character.” Funding has largely tended to reinforce existing patterns of economic development related activity, rather than providing a basis for developing new activities to meet identified objectives or stimulating more innovative approaches.’In terms of impact, the report relied, for most regions, on a macro-modelling approach to assess the impact of the SFs on economic and social cohesion. It recognised the fact that: ‘The emerging results inevitably flow to some extent from assumptions made within the modelling process.’ The report noted that the scale of their impact, in different cases, reflected the relative importance and allocation of the CSF resources and factors such as initial factor endowments, economic structures, levels of competitiveness and the degree of ‘openness’ of the economy concerned. It drew general conclusions concerning the impact of the structural funds relative to other external factors and indicated that, while job creation has been substantial, deadweight and displacement was also likely to be substantial.The Synthesis Report also included a series of observations concerning management and implementation. Its overall conclusion concerning these aspects was that: ‘Management and implementation arrangements, though often characterised by weaknesses, were generally fit for the purpose, and significant benefits were achieved through the partnership approach. Decision-making structures typically failed to establish strong strategic direction. However, important benefits in terms of wider “added value” were achieved through the programming approach and the development of public sector management capacity and practices.’

16.

It is important to acknowledge that the broad approach adopted faced some intrinsic constraints, the principal ones being: time limitations, a limited range of quantified objectives and a lack of detailed result/impact indicators. Furthermore, the impact that can be attributed directly to the programmes and related measures was often difficult to isolate since the target regions frequently benefitted from other operations that are instrumental in achieving similar objectives. It was also difficult to identify and measure the indirect effects of the Structural Funds expenditure. However, the Court considers that certain weaknesses in the evaluation process cannot be attributed merely to inherent constraints but were due to deficiencies in the evaluation process which need to be addressed, particularly so before the ex post assessment of the 2000 to 2006 programming period.

17.

The Synthesis Report starts with a brief analysis of developments in GDP per capita, unemployment and employment rates in the various regions concerned (9). The report concludes that: ‘Increased levels of GDP per head have generally not been the result of increased employment rates resulting from job creation’ (10). Such a conclusion would be a potentially important one for policy makers, suggesting that the main goal of cohesion policy of closing the GDP gap has not been significantly helped by improving labour participation rates (11), a conclusion of major relevance to the SFs.

18.

From the information presented in the Synthesis Report, it is difficult to judge the validity of the statement quoted above. There are many regions where GDP per capita went up, as did the participation rates. However, there are some regions where either GDP per capita or the participation rate declined or increased only marginally. The Court has sought to assess the claim quoted in paragraph 17 by applying statistical techniques to the data published in the Synthesis Report. The technical details of this analysis are given in Annex II and the essential points are briefly summarised hereunder.

19.

The results obtained indicate that changes in employment had a statistically significant impact on real GDP per capita. In particular, a one percentage point increase in the employment rate was associated with a 0,24 percentage point increase in GDP per capita over the period under assessment. This result is inconsistent with the conclusion of the Synthesis Report quoted above. It also suggests that there could indeed be some potential to close the GDP gap by improving labour participation rates — a relevant result from the policy perspective.

20.

To explore further the robustness of the above result, the Court carried out an identical analysis of 145 regions for which data was consistently available (i.e. not just Objective 1 regions). The results, also reported in Annex II, show very similar results across the Union. These results show that information from basic data was not sufficiently analysed and indicate an important relationship between increases in GDP and increases in the employment rate which the Synthesis Report did not highlight.

21.

The second chapter of the Synthesis Report examines the appropriateness of strategies adopted in terms of geographical and programme balance, as well as the balance between the funds. The planned and actual programme expenditures (12) of the Structural Funds are analysed by Member State and by type of fund but not by the relevant regions — in contrast to what is done in the report on GDP per capita and employment and unemployment rates in the various Objective 1 regions. This limits the analysis that can be performed, as is noted by the report itself in giving the results of a regression analysis which attempts to relate EU funds per capita by Member State to the GDP per head of population (13). Accordingly, the results given in the report are not statistically significant and, consequently, are of limited value.

22.

However, the Court applied slightly different statistical techniques, using the same data as in the report. The technical details of this analysis are given in Annex III and the essential points are briefly summarised hereunder.

23.

The results obtained support the intuitive conclusion that the higher the EU funds provided per capita, the higher is the GDP growth per capita in Member States. This result shows that with a broader approach, relevant assessments can be made even with the unnecessarily limited data base used in this important area under consideration.

24.

A central part of the assessment in this section of the Synthesis Report is based on a comparison of various categories of planned expenditure in 1994 and the estimated actual expenditure in 1999. However, the data presented suffers from a number of serious errors or discrepancies which limits the validity of the conclusions which can be drawn. In particular, in the case of five Member States, the figure for actual expenditure given under the ‘total’ column (in the relevant table published in the Synthesis Report) is different from the sum of the figures which appear under the individual columns that denote EU, public and private sector expenditure respectively (as they appear in the very same table). These discrepancies are summarised in Table 1. The percentage discrepancy, which is shown in the last column of this table, is significant and implies an inadequate quality control in the formulation of the report content and results.

25.

Notwithstanding these discrepancies, transforming the data given on a per capita basis (14) provides some useful observations which in some contexts differ from what was concluded in the Synthesis Report. Table 2 gives the actual per capita expenditure ranked from highest to lowest for each category.

26.

The table shows that the total actual funds average was 2 104 euro per person. The EU Structural Funds contributed approximately half of this, while three of the four cohesion Member States managed around 50 % more than the average of 1 033 euro per person. Of particular importance is the contribution of the private sector, especially in the light of the emphasis on long-run supply side impacts on the regions. It appears that some Objective 1 regions in Member States, such as Austria, Germany and the Netherlands, have been particularly successful in combining private investment with the Structural Funds, even surpassing, significantly in the case of Austria and Germany, the average contribution from the EU Structural Funds.

27.

However, the factors contributing to a higher participation by the private sector remain unexplored in the report. For example, was this a result of the nature of the projects chosen, the skills of the human resources available or indeed a different kind of assistance that is given to the private sector from the funds? In the Court’s view, these aspects merited further examination.

28.

The assessment of appropriateness was seldom based on formalised techniques (17) and the practices followed by the national evaluators were quite different and were generally largely descriptive. The matter of appropriateness concerning the degree to which SF interventions address ‘market failures’ was seldom the explicit subject of ex post evaluation. Similarly, the changes in strategy during the programming period have not been examined thoroughly in terms of their appropriateness.

29.

Specific chapters in the national evaluations and in Chapter 3 of the Synthesis Report tackle the effectiveness of Structural Funds, based on the extent to which expected effects have been obtained and objectives have been achieved. Five themes are explored, namely Transport, SMEs, R&D, Education and Training, and Rural Development. Two horizontal themes dealing with the environment and equal opportunities are also examined.

30.

The analysis is based on a sample of 70 % of the value of Objective 1 programmes. However, a great deal of emphasis was placed on qualitative appraisals, for example rating community added value on a ‘yes’ or ‘no’ basis. Such appraisals and opinions are often classified too broadly and are not always objectively verifiable. There was no attempt to set more concrete objectives by identifying criteria which could have been useful for this purpose and no additional data was gathered for the purpose of micro analysis. Important considerations for a fuller assessment of effectiveness, such as deadweight and displacement effects, were often not considered, and when these aspects were raised, they were not analysed in sufficient detail.

31.

Although there are a number of sections which have valid and useful observations, in general the Court finds this crucial chapter (18) in the Synthesis Report lacking robustness, with missing data and often weak conclusions and an insufficient or missing assessment of the results of other thematic studies commissioned by DG REGIO for the SFs for the 1994 to 1999 programme.

32.

For example, in the ex post assessment section on transport, in the tables dealing with actual and target kilometres of roads improved by country, about half the target information is either missing or incomplete, making it difficult to get a robust conclusion on achievements as compared to targets. Yet, the Thematic Evaluation (19) of the Impact of Structural Funds on Transport Infrastructures for the same programme period, which is quoted frequently in the report, provides a more complete and rigorous assessment of the results by Member State.

33.

Of particular relevance to the ex post assessment is the estimated employment impact of transport programmes that is given in the Thematic Evaluation Report on Transport. The total direct and indirect employment impact is estimated at 2,3 million person years (20). Furthermore the economic benefits of the transport programmes given in the study are judged to be significant. In Ireland, the reported economic rate of return for road projects is given as 13 %, while that of Spain is given as 23 % (21).

34.

These two relevant conclusions of the Thematic Evaluation Report on Transport are not taken into consideration in the Synthesis Report. Furthermore, the employment impact estimate in the thematic evaluation is inconsistent with that of the ex post study. Based on calculations made by the Court, the total employment impact of Structural Funds over the period 1994 to 1999, implied by the study (22) (using the HERMIN model and the assessment of the other regions) is 1,62 million jobs, which is inconsistent with the estimate of 2,3 million jobs in the thematic report, especially since transport only accounts for approximately 20 % of all Structural Fund expenditure.

35.

In contrast, the results of the Thematic Evaluation of the Impact of the Structural Funds on SMEs are used in the Synthesis Report which states: ‘The Thematic Evaluation of the Impact of Structural Funds’ on SMEs recommends a shift from grant expenditure to financial engineering measures such as seed and venture capital funds, loans, interest rate subsidies in the future, arguing that these methods are more sustainable in the long term (23). This is an important conclusion within the thematic study which the ex post assessment does well to bring out (24). However, the report does not develop this issue further despite the important implications that could be drawn for future programmes.

36.

The Synthesis Report quotes the Thematic Evaluation on the Impact of Structural Funds on SMEs as having estimated that ‘in the absence of Community support, 70 % of investment projects would have either not taken place at all, or been smaller in scale or postponed’. The thematic evaluation ‘further estimates that such assistance contributed to creating more than 300 000 additional jobs, even after taking account of “deadweight” and substitution effects. The ex post evaluation of Objective 1 is not able to replicate these figures owing to the lack of comparable and robust data on which to build such estimates’ (25). Given the significance of the different results and that the two studies were prepared for the same contractor, treated the same subject, dealt with the same period and were published only two or three years apart, a fuller treatment of their differences was warranted.

37.

Following the above statement, in a section dealing with an ‘overall analysis’, the Synthesis Report does give an estimate of the number of jobs created in SMEs as a result of the Structural Funds. ‘From the evidence available it appears that Objective 1 supported the development of some 800 000 jobs. This is in terms of gross jobs and does not take into account substitution, deadweight or multiplier effects. The overall effectiveness in terms of net effects is unknowable’ (26). This conclusion is not given sufficient attention, especially in the light of the estimate provided in the thematic study on SMEs. Furthermore, the table in the ex post study, giving a breakdown of employment created by Member State, provides no details for Spain, so that the estimate given is clearly incomplete (27). This aspect merited a more in-depth examination in the ex post evaluation.

38.

In the ‘other topics’ which the Synthesis Report covers under Effectiveness, there is a useful if brief summary of various pertinent issues under a number of headings. However, one underlying limitation is ‘weak comparative data which has forced a qualitative assessment rather than a detailed project-based assessment’ (28). Some of this qualitative assessment could have been better linked to other thematic studies undertaken, such as the thematic evaluation on RTD. In the ToR, it is stated that: ‘A broad scoped literature review will be undertaken in order to identify key issues in the implementation of the Structural Funds’ (29). However, there are shortcomings in the way in which such implications are reported, and are linked to subsequent findings. In the few cases where there is such integration with the assessment, the benefits on the quality of the report are clearly evident.

39.

For example, the ex post assessment reports that the thematic evaluation on RTD notes that the focus of activities would be better directed at developing region specific innovation strategies to exploit existing RTD capacity, rather than based on perceived ‘Technology Gaps’. The ex post study reports that this was so in a number of cases, with results being much more successful when focused on local strengths (30). This example clearly highlights the relevance of building on what other reports and studies have demonstrated, as emphasised in the criteria developed in the MEANS programme and its update, which was included as part of the ToR of this evaluation exercise (31).

40.

The national evaluations were seldom based on a formalised or quantitative analysis of effects. Furthermore, the national evaluators rarely used the results of quantitative analysis carried out by other parties.

41.

The aim of the Synthesis Report under this heading was the examination of the efficiency in the implementation of large projects supported in Objective 1 regions. The analysis was based on a geographically and thematically representative sample of approximately 60 large projects (32). Efficiency is understood to refer to the effects being obtained at reasonable cost. ‘The main objective is to establish unit costs, i.e. the cost per output achieved. The analysis should also address the quality of the management of large projects by the final beneficiaries. At EU level, unit costs should be analysed across Objective 1 regions. Significant differences in unit costs across regions should be explained where possible and a range of appropriate costs should be established, which may be used as the basis for benchmarking exercises in the future’ (33). These objectives are relevant and form a critical element in the efficient management of projects across the EU.

42.

However these objectives were largely not met by the ex post assessment. This is admitted in the Synthesis Report, with the explanation that project managers found it virtually impossible to determine the individual component parts of the project and subsequently could not calculate unit costs (34). Such a broad statement implies that there is such a poor state of knowledge on major projects funded by the SFs as to make this objective in the ToR unattainable. As a minimum, it should have been possible to arrive at comparable costs for typical outputs, e.g the cost per square metre of building a factory, or per kilometre of road built.

43.

Some partial results are presented in three tables dealing with road construction in Member States, environment projects in Spain and costs per job created in industrial projects (35). However, there is little evaluation or analysis of these results, for example to factor in the impact of different labour costs and particular geographic characteristics in say road construction.

44.

The Synthesis Report reached firm conclusions about the timing of projects and respect for budget limits. These were, first, that only one third of the 60 projects reviewed were completed within the originally planned timescale, with over a third a year late. Secondly, approximately two thirds of projects examined ran over budget, with 20 % costing over 30 % more than originally planned. However, the analysis is limited to a few general observations, providing little or no assessment of the quality of management by the final beneficiaries as required in the ToR (36). Furthermore, the relative efficiency of public private partnerships and of the tendering procedures employed were not evaluated.

45.

The Synthesis Report claims that the primary aim of Structural Funds is to influence the long-run supply potential of the economy (37). These supply side effects work through a number of channels that are designed to: (a) increase investment in order to improve physical infrastructure as an input to private sector productive activity; (b) increase human capital, due to investment in training, as an input to private sector productive activity; and (c) channel public funding assistance to the private sector to stimulate investment, thus increasing factor productivity and reducing sectoral costs of production and of capital (38).

46.

The report goes on to discuss the importance of what are called positive externalities — ‘conditions where private firms enjoy the use of additional productive factors at no cost to themselves’, thereby improving the productivity and cost competitiveness of the economy. These considerations are relevant to an ex post assessment. However, private sector co-financing was excluded from the analysis because of uncertainties about its ‘driving mechanisms’ (39). This limited approach leads to a severe understatement of the impact of certain expenditures in the Structural Funds by excluding one of the major long-term supply side effects. One clear demonstration of this is the example of a development company presented in the Synthesis Report (40) whose co financed expenditures have attracted significant additional private sector investment. Instead of focusing on the more directly discernible impact of SF expenditure on private sector investment, the study concentrates on attempting to assess the so-called externalities, with rather limited techniques of estimation as a result of a lack of studies providing measures of elasticities which are relevant to the regions under consideration.

47.

To assess the impact of the Structural Funds on economic and social cohesion, a macroeconomic modelling approach based on the HERMIN model was applied to four Member States (Greece, Ireland, Portugal and Spain) (41) as well as for the regions of the new German Länder and Northern Ireland (42). This approach was intended to measure the impact of Objective 1 progammes in terms of the effects on GDP and employment.

48.

An alternative assessment scenario would be to use microeconomic data from projects to build up to the total or aggregate impact of the Structural Funds on a region or country basis. However, despite the use of standardised data requirements across Objective 1 regions, there are difficulties in collecting microeconomic data. This results in attempts to measure impacts solely from macroeconometric models, giving rise to many restrictions and assumptions which, for the reasons set out below, can restrict the assessment significantly, thereby limiting the policy implications that can be derived.

49.

The principal difficulties identified with the specific application of the HERMIN macromodel used in the ex post assessment under review are:

50.

The HERMIN model, in the manner in which it has been applied, thus appears unsuited to the task of estimating the effects of SF interventions in Objective 1 regions.

51.

The Court analysed the results of HERMIN simuilations concerning changes in the rate of unemployment, against a set of pre-established hypotheses (46). The outcome is shown in Annex IV. The results show significant variations regarding unemployment from those available from other sources.

52.

Finally, in many cases, the degree to which the intermediate objectives were attained was not evaluated.

53.

As regards the strategic objective of convergence, the only mention in the Synthesis Report is ‘that the overall convergence between Member States is progressing but that this is not occurring to the same extent between regions across Europe. Whether the convergence is due to the Structural Funds is a debatable point’ (47).

54.

The management and implementation arrangements adopted for the Objective 1 programmes have a crucial role to play in contributing to the effectiveness, efficiency and impact of the Structural Funds. This aspect was addressed in both the ToR and the Methodology Guide and was taken up in the national evaluations (48). The national evaluation reports described the impact on management systems but also identified situations where the effects on management systems could have been more pronounced.

55.

The importance of this topic is also recognised in the Synthesis Report and, in the recommendations section (49), no less than 13 out of 19 recommendations deal with management issues.

56.

The Synthesis Report provides a number of relevant insights into the management of EU Structural Funds and refers to several conclusions of the Thematic Evaluation of the Partnership Principle that was conducted in 1999. This provides for continuity in assessment, and some evidence in support or otherwise of observations made in the earlier study. However, in a number of areas, the conclusions reached are in conflict with the thematic evaluation.

57.

The Synthesis Report provides a summary of the average size of membership of Monitoring Committees(MCs) by Member State. The overall average was 44, but there was a wide variation from a low of 15 to a high of 85. Indeed, the report concludes that the high membership level was regarded as a key constraint on the efficiency of the Monitoring Committees, and that this was common across all Member States (50). In contrast, the thematic study concluded that there was no evidence to indicate a loss in efficiency through extending inclusiveness. Given the significant difference in the conclusions of the different reports on this aspect, it would have been appropriate to analyse this aspect more thoroughly in the Synthesis Report, and to draw more concrete recommendations on the way forward.

58.

Another area which was linked to the thematic evaluation concerns the management of different funds. Both evaluations concluded that the funds operated according to different principles and with different financial requirements and that this restricted synergy between the funds. This important observation echoes the views already expressed by the Court (51) and underlines the need to strive for greater synergy between the funds.

59.

The assessment that is made about the effectiveness of the overall management systems also demonstrates some weaknesses. The Synthesis Report states that: ‘The management systems were thought to be adequate by the national evaluations in just under half of the eligible member states’. This is an important result and a detailed assessment of both the success cases and the inadequacies identified could have been expected. Although some points raised are developed, there is an overall lack of clarity in the Report about what was actually found.

60.

The Thematic Evaluation of the Partnership Principle in relation to the Structural Funds (1999) found that this principle provided a number of positive advantages to Structural Funds implementation (52). The Synthesis Report states that ‘Broadly, these advantages were also recognised by the Member States involved in the implementation of the 1994 to 1999 Objective 1 programme, and horizontal partnership working was welcomed as one of the “added value” aspects of European programming’ (53). This is a relevant result, and although some highlights of various cases are presented, including brief comments on negative experiences or difficulties encountered, the discussion could have been more directed at highlighting improvements that could be made, on the basis of the experiences mentioned, so as to make the partnership principle more effective.

61.

The project application/selection process is identified as one of the two most frequently cited processes that caused administrative problems. A number of examples are given, indicating a lack of clarity in the procedures and an urgent need for the Commission to issue a best-practice benchmark. A table is provided giving the duration of the project selection process by country, although no data is provided for three Member States. The duration of the process for the remaining Member States ranges from two to nine months with an average of five months. The evidence provided implies that there is still work to be done in order to streamline the application process. In this context, no thorough analysis has been conducted concerning the respective merits and/or effectiveness in the choice between the competitive process and the queuing process.

62.

Due consideration must also be given to the monitoring and control process. It is claimed, in the Synthesis Report, that the ‘Financial control systems appear to have functioned relatively well within the Member States under the Objective 1 1994 to 1999 although some adjustments needed to be made to accommodate the new Financial Regulations in 1997. In the majority of Member States financial control systems were felt to be fit for the purpose and operated relatively efficiently. Financial control appears to have been generally reliable and respected the 5 % rule’ (54). This finding does not coincide with the various reports of the Court (55) which indicate that the 1994 to1999 Structural Funds programme suffered from significant financial control deficiencies.

63.

The other element of the monitoring process that gives rise to administrative problems relates to the non-financial monitoring and evaluation systems. Monitoring comes in for heavy criticism. The Synthesis Report concludes that ‘monitoring remained a major weakness despite the emphasis on developing its role since the first programming period. The available data is fragmented, inconsistent, overly focused on physical outputs and often of doubtful reliability’ (56). It also states: ‘Across all of the Member States there was a lack of quantification to the objectives defined at the start of the programme’ (57). Of the 10 Member States listed, seven are classified as having a poor level of quantification, one as variable and two as adequate only at the start of the programme. Brief summaries are given for each Member State. However, no further insight is given.

64.

It is also stated that the computerised systems set up for monitoring the 1994 to 1999 programme ‘were frequently criticised due to their lack of integration across programmes and between funds. This had an impact on the ongoing ability of managing authorities to achieve an oversight on progress within the programmes, as well as posing a problem for later evaluations’ (58). These findings must also cast doubt on the validity of the earlier conclusion in the report that the financial management of the programme performed well if monitoring information, which is such an important input into the management process, was deemed to have been ‘fragmented, inconsistent and of doubtful reliability’ (56).

65.

The Synthesis Report also provides some brief comments on so-called ‘internal evaluations’ (59). The following observation is made: ‘These internal evaluations were carried out by the ministries or regional authorities, with assistance from universities and external agencies. The degree to which evaluation was carried out may, perhaps, be surprising in the light of the weakness of the monitoring systems. This perhaps is a reflection of the perceived requirement to undertake evaluation activity, although the results did often feed through into programme management’ (60). In view of this statement, it would seem that it would have been useful to focus more on these internal evaluations, and to build on the results achieved by the relevant ministries or regional authorities. In contrast, the Synthesis Report only gives some very general observations with limited relevance to the overall ex post assessment.

66.

With regard to the 2000 to 2006 programming period, the Synthesis Report notes that attempts have been made to address the monitoring problem experienced in the 1994 to 1999 period (61).

67.

In the light of the various problems identified in the analysis of the Synthesis Report it is necessary to focus specifically on the original terms of reference. These, together with the inception report and the Methodology Guide, suffered from certain weaknesses. In particular:

68.

In addition, the strengthening of economic and social cohesion in the EU and the convergence of the less-developed Member States were not accorded sufficient priority in the ToR even though they are amongst the main objectives of structural measures. The inception report did include some questions about the development models used, the constraints on growth which are to be addressed and the channels through which an impact on the regional economy can be expected (68). However, the ex post evaluation should ideally have included empirical studies on the convergence between regions and the reduction of disparities as a result of such factors as infrastructure investment, developments in the education system and in R&D capacity.

69.

Furthermore, as an evaluation objective, it would have been appropriate to identify and analyse specific case-studies representing benchmarks for regional development policies, for instance, in the area of clustering or growth hubs.

70.

At the launch of the evaluation exercise, the question of appropriateness of the strategy was addressed by means of a list of issues and questions. However, the evaluation guidelines provided by the ToR (69) and the Methodology Guide were not sufficiently precise or concrete.

71.

The (general) contract between the Commission and the consultants selected to carry out the evaluation stipulated that technical assistance with the evaluation was to be provided under the responsibility and supervision of Commission officials. The quality of the evaluation was to be assessed by the Commission on the basis of the criteria developed under the MEANS programme.

72.

The steering group chaired by DG REGIO which included representatives of other DGs was involved in drafting the ToR, approving the methodology and detailed work plan (in the Inception Report) and examining the national reports and the Synthesis Report. A panel of four experts was appointed to assist with their principal objective being that of ensuring that high-quality evaluations were produced. However, their intervention came so late so that the process was unable to deliver significant improvements in evaluation quality.

73.

The contractually agreed schedule was not adhered to. The submission of the different reports was delayed, with the result that the final Synthesis Report and the national reports were not submitted to the Commission within the planned 12-month period, i.e. by 12 November 2002, but only in April 2003, and were published in May 2003. It would appear that the timetable was excessively tight from the outset and did not include sufficient time reserves for treating the relatively large number of changes/amendments to the reports requested by experts and representatives in the steering group. This has important consequences for future ex post evaluations.

74.

The ToR (67) required the examination of two specific themes by Member State and a list of such themes was proposed. However none of the national evaluation reports included a specific focus on this aspect. Although individual chapters sometimes provide statistics or a partial exploration of these themes, they have not been subject, as required by the ToR, to specific research.

75.

As a general conclusion, the oversight of the evaluation process, by the Commission, needs to be improved. In particular, given that the lack of reliable data and performance indicators and of precise objectives was foreseeable from the outset, the Commission should have ensured that appropriate measures were taken to address this critical constraint. Furthermore, given the tight timetable and the inherent constraints, the evaluation should have been concentrated on essential elements rather than attempting to undertake such a wide overall assessment of the programming period.

76.

Given the problems identified in the preceding sections of this report, the ability of the ex post evaluation process to yield worthwhile recommendations was limited. While the process yielded few recommendations about programme content, it delivered a number of relevant and sensible suggestions on how to improve the management of the SFs and of their implementation (70).

77.

The report recommends that: ‘The role of the Monitoring Committees needs to be shifted towards one of strategic direction rather than issues of financial management’ (71). However, the Monitoring Committees should be able to do both functions, as required by the regulations.

78.

The evaluations have led to recommendations for the programming period 2000 to 2006 which mainly address the questions of management systems. A wide range of themes is covered and these are generally well suited to the specific situation of the country concerned.

79.

With respect to the 2000 to 2006 programmes, the key lessons to be drawn from the ex post evaluation of the preceding period relate to implementation, in particular with reference to the case study work undertaken to evaluate the effectiveness and efficiency of Structural Fund operations.

80.

As regards post 2006, the recommendations proposed within the Synthesis Report are rather limited and of a general nature. However, they are still relevant and, as such, are useful. For example, it is stated that ‘More use could be made of independent, ongoing evaluations, providing an objective external input’ (72). While this is a sensible recommendation, it would have been of benefit if it had been adopted before the launch of the ex post evaluation exercise under review.

81.

Together with the evaluation of Objective 1 regions, under the direction of DG REGIO, another evaluation exercise was carried out, in this case under the direction of DG EMPL, to cover ESF operations under Objectives 1, 3 and 4 and the Community Initiatives, Employment and Adapt. The ToR for the latter ex post evaluation anticipated the utilisation of the results obtained from the evaluation of the Objective 1 regions. However, despite organisational arrangements intended to facilitate an effective coordination between the two evaluations, there is little evidence that this was achieved in practice.

Text box 2The overall conclusions of the evaluation by DG EmploymentThe evaluation aimed at assessing the impact of ESF operations in the Member States, at the level of individuals and of systems. It also sought to analyse the impact and effectiveness of Community action on specific structural problems and to ascertain the Community value added. In the methodology adopted to address these objectives, evaluators were asked inter alia to indicate whether the cases which they were asked to assess provided support for or contradicted a number of hypotheses.The report noted a number of strengths which can be summarised as follows: Much of the ESF funding was targeted at the long-term unemployed which was deemed appropriate given the persistence of the latter. The improvements in the labour market position for the beneficiaries were commensurate with the resources involved. There were major improvements in employment services and educational and training provisions in several national contexts. The resources used enabled a strengthening of certain policy priorities, such as equal opportunities, adaptation of the workforce and targeting those with disabilities. Through the ESF, there was an increased involvement of social partners and regional authorities. The funds contributed to improvements in capacities to manage and implement labour market interventions at all levels.However, the report also noted a number of weaknesses. Deployment was largely driven by availability of resources rather than policy and took place in parallel and separate from the relevant policy debates. Interventions were largely concentrated on training and mainly allocated to service providers which is not, in isolation, the most effective means of helping the long-term unemployed. Consequently, it is likely that there was a high degree of deadweight associated with the intervention. Finally, administrative arrangements were perceived as complex while evaluation arrangements were weak and unstructured.

82.

The influence of ESF interventions often takes the form of a contribution or of a potential effect that is intangible and therefore not easily quantifiable (73). Notwithstanding this intrinsic problem, it must still be a prime objective of an evaluation exercise to assess the impact and, in conjunction with it, also the question of effectiveness and efficiency. This is what the ex post evaluation exercise was required to do.

83.

In seeking to analyse the impact of ESF interventions, the Synthesis Report provides estimations of the number of ESF beneficiaries per country and per objective (74). As a complement to the national evaluation reports, the ex post evaluation also attempted to provide evidence for a number of pre-established hypotheses (75). These considerations are discussed in the following paragraphs.

84.

One of the most basic requirements of the ex post assessment of ESF operations relates to the number of beneficiaries of the programme over the period 1994 to 1999. The report provides a table which gives the estimated number of beneficiaries as 52 million. However, the additional notes to this table cast serious doubts on the figures presented (76). The uncertainty about the accuracy of the data detracts from some of the useful and relevant observations resulting in other parts of the assessment.

85.

The Synthesis Report (75) provides estimates of impacts on individuals from ESF interventions. This information is potentially important in a number of aspects, including the calculation of unit costs of programmes and estimates of beneficiaries still in employment after 12 months. However, there are factors which seriously limit the usefulness of the information, notably the lack of explanation as to how the data has been derived. The report simply states that: ‘These ratios applied to generate these estimates have been informed by the individual cases examined and the review of the literature’ (74). Such a statement cannot be considered as being sufficient and more detailed information on the derivation of the various estimates is necessary to understand better the applicability of these numbers, such as benchmarks of unit costs and success rate of programmes.

86.

A second unsatisfactory aspect is the analysis accompanying the data. There is a mismatch between what is stated in the assessment and what is found in the accompanying table. For example, the text states that ‘in broad terms unit costs were lower for the unemployed and young unemployed, around 6 000 euro per worker in employment, and considerably higher and wide-ranging for those with disabilities and long-term unemployed’ (74). Yet the table gives the unit costs for training the unemployed as the highest (at 6 000 euro) and the integration of the disabled as one of the lowest (at 4 000 euro). Furthermore, it is common practice to give the range of variation when there is what is claimed to be ‘wide variation’ between and within programmes. The analysis in this section is of limited value because it relies overly on reducing the assessment to a single number rather than defining a relevant range of values and providing a more comprehensive analysis of these ranges found in the various cases examined.

87.

This mismatch between the commentary and the figures given occurs in other parts of Chapter 3 (see Text box 3 for some examples).

Text box 3For instance, it is stated that: ‘The ESF per capita expenditure was highest in Ireland, Portugal and Greece (fully Objective 1 countries) and in Spain (1)’. Yet Table 3.1 gives an ESF expenditure per capita per year for Portugal of 2 086 euro and for Germany of 405 euro, while Ireland is given as 12 euro and Greece as 17 euro. Clearly, the figures in the table do not match the discussion in the text. Even more significant is the statement: ‘The ESF expenditure on education relative to national expenditure was in all countries lower than the relative expenditure to ALMP. It was highest in Greece and Portugal (2)’. Yet in Table 3.1, Greece is classified as having no ESF expenditure in the relevant column giving ESF expenditure on education relative to national expenditure.For example in paragraph 3.2.3 (3) it is stated that: ‘The national expenditure on labour market training as a proportion of GDP ranged from 0,2 % in Luxembourg to 0,58 % in Sweden (4)’. Yet in Table 3.3, the expenditure on training in Denmark is shown as 0,94 % of GDP, almost twice that of Sweden. As another example, consider paragraph 3.2.6 (5), dealing with expenditure on the integration of the disabled. In the text, one finds the following statement: ‘The ESF was particularly important in Portugal, Spain, Greece and Austria.’ Yet in Table 3.6, Spain and Greece are listed as having no ESF expenditure on the disabled. A further example is found in paragraph 3.2.7 dealing with expenditure on education. It is stated that: ‘The ESF contribution was most significant in Greece, Ireland and Portugal (6)’. But Greece is listed in Table 3.7 as having no ESF expenditure on education and training.Synthesis Report: (1) p. 78; (2) p. 79; (3) p. 81; (4) p. 81; (5) p. 84; (6) p. 85.

88.

The evaluators who took part in the study were asked to indicate whether the cases assessed provided support for or contradicted a number of hypotheses that had been formulated on the basis of an initial review of the literature and the findings of the inception phase of the evaluation. The study claims ‘strong supporting evidence for four hypotheses’ (see Table 3  (77)). However, the data presented does not support the claim of strong supporting evidence. For example, only seven of the 30 evaluators said there was ‘strong supporting evidence’ for the first hypothesis. A more objective evaluation of these results would have been warranted.

89.

The terms of reference concerning the impact of ESF 1994 to 1999 also required an evaluation of the extent to which ESF operations provide support to the ‘equal opportunities (78)’ objective. This aspect is tackled in Chapter 7 of the report. However, it is claimed that: ‘A major finding is that gender disaggregated data were not collected systematically for each of the ESF Objectives and Community Initiatives across the EU’ (79). This has limited significantly the ability of the study to evaluate this topic. It is difficult to accept this result as a major finding of this report since a preliminary analysis would have revealed this shortcoming, necessitating field work and sample surveys to overcome this restriction, rather than simply concluding that disaggregated data is not available.

90.

As regards the impact of ESF operations on individuals, the interventions ‘combined measures’ (80) have not been specifically examined in spite of the substantial percentage of ESF resources having been allocated (18 %) to them. As a result, the conclusions of the Synthesis Report are rather modest: ‘Most of the evaluation work points to positive outcomes. However, the evaluation methods tend to focus on identifying the components of the programmes that are most effective rather than judging the performance of combined measures compared with single measure interventions’ (81).

91.

As regards the impact of Active Labour Market Policies (ALMP), especially with regard to Objective 1 countries, the Synthesis Report draws some conclusions without adequately explaining how they were reached.

92.

As regards the relative financial contribution of the ESF, the Synthesis Report (82) concludes: ‘that the combined allocation of national expenditure and ESF resources does not appear to have led to a convergence of the patterns of expenditure between the countries, nor the adoption of norms linking specific economic and labour market conditions to resource allocations’ (83). It is difficult to see how this conclusion can be sustained given the level of errors in the relevant tables and/or mistakes in the discussion. Similarly, another important conclusion in this part of the report is that: ‘These circumstances suggest that not all of the ESF interventions are likely to have been additional. Additionality, in the sense of the ESF co-financed interventions being unlikely to otherwise have taken place, is more likely to have occurred where ESF resources were focused and concentrated on particular types of interventions and where the related national expenditure and programmes were relatively small and less developed compared with countries with similar labour market conditions.’ Here again, it is difficult to see how this conclusion is reached from the evidence presented in this chapter, especially given the mismatch that there is between the analysis and the data given in the tables.

93.

With regard to the hypotheses advanced on the impact of ESF on systems, a problem arises similar to that described earlier. Strong supporting evidence is claimed this time for seven hypotheses (84). However, an analysis of the data presented also indicates that the conclusion is not warranted.

94.

Giving due weight to the constraints faced by the evaluators, the following critical comments must nevertheless be made with regard to the Commission's own assessment of the ex post evaluation which concluded with a satisfactory overall appraisal.

95.

The evaluation of the report by DG EMPL is based on eight performance criteria and four possible quality levels, namely: excellent, good, acceptable and unacceptable. None of the applied criteria are rated as excellent or unacceptable. Four criteria are rated as good, namely; meeting needs, relevant scope, defensible design and clear report. The four other criteria considered are ranked as acceptable. These relate to reliable data, sound analysis, credible results and impartial conclusions.

96.

It is significant that DG EMPL assessed the latter four criteria as being less satisfactory than the previous four. However given the problems encountered as regards reliabilty of the data and the inconsistencies of the commentary thereon, as highlighted in this report, the Court considers that the evaluation report was unsatisfactory, especially with regard to reliable data, sound analysis, credible results and impartial conclusions.

97.

Some pertinent comments can also be made with reference to the national evaluation reports, with regard to the evaluation objectives and results:

98.

The sampled national reports do not always present a thorough analysis of the impact of ESF interventions on national policies. However, they generally do suggest that the ESF had an impact on the development of ALMPs and, in some cases, a concrete, direct effect was stated. However, in some other instances, a different assessment was made, namely that it is difficult to show that the ESF has led to changes in the ALMPs.

99.

Each national evaluation report and the Synthesis Report describe in a more or less extended way the impacts on systems as regards employment services; initial education and training; continuing education and training; accreditation systems and mechanisms for anticipating needs. The impacts on systems often concerned a more specific policy context. It was generally difficult to identify the net impacts of ESF operations (86); only in exceptional cases, was the net impact effectively delimited (87).

100.

Concerning the European Employment Strategy, the evaluation reports highlighted the positive influence of the ESF in some Member States such as Greece, France, Italy, Portugal and Spain. However, it was noted in the case of Germany that the top-down approach associated with the European Employment Strategy had the potential to conflict with the ESF bottom-up approach (88). At the more general level, the Synthesis Report concludes that the EES has provided a stronger strategic framework within which to programme ESF interventions (89).

101.

As regards the Community value added, the national evaluation reports have contributed some relevant results (90), although some of the specified questions have not been dealt with in some national reports.

102.

The financial implication of policy choices has rarely been addressed in detail. The Synthesis Report includes a calculation of unit costs concerning some measures but this is based on unsatisfactory data, Where cost issues have been addressed, they would seem to highlight potentially serious findings that merit further analysis. For example, for Spain, it is claimed that ‘This roughly doubling of resources due to the ESF has led, according to some experts, to a substantial amount of wasteful spending. The larger source of funds has been used to expand operations and not so much to introduce new measures.’ Similarly for Italy, it is claimed that there have been relevant transaction costs associated with the ESF, referring to implementation issues and the costs associated with partnership building and new management practices (91).

103.

The following main evaluation topics were specified in the ToR: the principal characteristics and the accomplishments of the 1994 to 1999 programming; the impact of the ESF vis-à-vis national policies and the effectiveness and impact of ESF operations on individuals and on systems.

104.

A methods report was compiled by the Commission and published a year later to provide a critical review of the methodologies adopted for the assessment of results and of their impacts on individuals and systems. This report was intended to prepare the ground for future evaluations but it is questionable if this objective has been effectively achieved. This report suffers from various weaknesses. Its preparation was not based on a sufficiently thorough enquiry as regards to the approaches applied in assessing the net impact on individuals. As concerns the evaluation of the impact on systems, it does propose a model for the classification of ESF expenditures and of their anticipated associated impacts (capacity, capability and responsiveness) (92) though some issues remain to be addressed (see page 35 of the report).

105.

In the Methods Report, within the context of the ‘Methods for policy and strategic evaluation in the ESF field’ (93), no concrete recommendations have been presented as regards policy analysis, with the report merely referring to MEANS. Similarly, there was no reference to the approach to be followed for the selection of appropriate indicators at different strategic levels.

106.

It should be remarked that the relevant elements for some evaluation tasks are not always clearly specified in the ToR. For example: the description of the tasks as concerns effectiveness is rather abstract, especially with regard to the underlying concepts, the selection of mechanisms for impact measurement and the evaluation techniques to be used. In particular, no concrete indication has been given as regards the methods of generalising, at the EU level, the direct impact on individuals from the results of the activities in the different Member States. The formulation of the tasks has not given due regard to the possible mismatch between quantitative objectives and the data available. Furthermore, no concrete action has been foreseen to evaluate the impact on individuals when the indicators are not concretely established, or when specific statistical evaluations have not been carried out recently. The other methodological documents, compiled subsequent to the ToR, do not contain the desired explanations either.

107.

Changes in programming have not always been adequately examined. For example, this subject was not addressed in the studies concerning Northern Ireland, Merseyside and Spain. Even where changes in programming have been indicated, their treatment is far from being optimal. In some cases the evaluator has not provided a clear assessment of the relevant changes and of their appropriateness (94).

108.

The Commission did not carry out a pre-evaluation survey or reconnaissance exercise and this has made it more difficult for it to anticipate the problems that have been encountered by the evaluators.

109.

There is no clear evidence that the steering group was sufficiently aware of the lack of data available to the national evaluators, nor the impact that this would have on the quality of the final evaluations or the problems regarding delays. No remedial action was taken by the steering group to address this issue.

110.

As was the case for the Objective 1 evaluation, the Commission's oversight of the evaluation process needs to be improved (see paragraph 75).

111.

For most Member States, the recommendations derived from the 1994 to 1999 ex post assessment have mainly dealt with the management systems and procedures. They relate to a variety of issues and are generally well adapted to the specific situations.

112.

Fewer recommendations for 2000 to 2006 have concerned the content of the co-financed policies. Where such recommendations have been made, they appear to be of relevance (95).

113.

The Synthesis Report brought up a list of strengths and weaknesses for each Objective (96) and main recommendations for the 2000 to 2006 period and post 2006 (97). These statements are rather general although they do appear to be valid in certain cases, e.g. in highlighting potential differencies between EU-15 and EU-10 and in the policy response to territorial differencies.

114.

The evaluation process is of critical importance in terms of providing information and judgements about the appropriateness, relevance, economy, efficiency, effectiveness and the impacts of EU interventions. By doing so, it can support the decision-making process for setting political priorities and allocating resources. In order for the process to be successful, however, there are certain prerequisites to be respected, namely, clear objectives for policies and programmes, and performance indicators which are produced on a regular basis and which are complete, reliable and relative to the objectives set. In addition, it is necessary to ensure that the resources available to carry out the process are adequate for the purpose, both in terms of time and technical competencies.

115.

The Court’s audit of the ex post evaluations under review identified a number of significant shortcomings in the overall approach adopted and in the quality of the assessments made:

116.

With regard to the various themes of the evaluations, the following conclusions can be drawn concerning the Objective 1 evaluation:

As regards the Objective 3 evaluation, it was in part based on unreliable or incomplete data while there were significant mismatches between the data provided and the commentary thereon. Furthermore, there was a lack of convincing support for some of the hypotheses advanced thereby shedding doubts on the methodology employed to reach them.

117.

Many of the shortcomings of the evaluations can be traced to the circumstances in which they were carried out, notably the limited information available and the tight time constraints set. It should have been possible to anticipate these constraints and the evaluation exercise should either have been given the necessary resources or the scale of the exercise should have been limited to what was more feasible. Furthermore, the Commission's oversight of the evaluation process needs to be improved.

118.

The ex post evaluations have still provided some worthwhile insights, notably into the management systems underlying the implementation of SFs. However, the Commission encountered serious difficulties in delivering the desired in-depth analysis of the effectiveness, efficiency and impact of the funds, largely due to limitations in the approach adopted and restrictions in the availability of data. The latter constraint had not been adequately anticipated at the start of the exercise. These problems were compounded by certain weaknesses in the Commision's oversight of the process. Consequently, while some worthwile and useful recommendations were made as input for future programming periods, the Commission could not benefit fully from the experiences of the 1994 to 1999 programming.

119.

A reappraisal of the scope, procedures and approach used in ex post assessments needs to be done with some urgency, before the next batch of contracts for ex post evaluations are issued at the end of the current programming period 2000 to 2006.

120.

To improve the evaluation process, better quality control procedures need to be introduced and effectively applied by the Commission so that the problems encountered in the evaluations under review do not recur in future ex post assessments. Such procedures should ensure:

121.

Secondly, particular attention needs to be reserved to the choice of appropriate techniques for assessing and measuring the economic impact. If a macroeconomic modelling approach is used to analyse the impact of the structural funds, it must be adapted to take due account of the characteristics of the economies under review. This consideration is particularly relevant in the situation where the evaluation will need to be extended to cover the 10 new Member States. In this light, the Court considers that there should be a re-appraisal of the appropriateness of an aggregate macroeconomic modelling approach (e.g. such as one based on the HERMIN model). The Commission should examine whether greater emphasis could be placed on models that rely on the use of micro-data derived from projects. It is also likely that a more in-depth evaluation of certain, specific themes could better lead to the identification of best practices and provide additional insight into particular issues such as for instance the impact of clustering, the significance of growth hubs in regional development vis-à-vis the overall objective of convergence.

122.

Thirdly, greater emphasis needs to be placed on establishing linkages between the ex post assessments and thematic studies, and to ensure that results are consistent with each other. If results appear to be incompatible, then this aspect needs to be specifically tackled. In the evaluations under review, where linkages were developed by the evaluators, the quality of the assessment improved markedly.

123.

Fourthly, a number of recommendations for future assessments more particularly for Objective 1 Regions may be made, namely:

124.

Finally, it is important to strengthen the capacity of the Commission to oversee the evaluation process, including the application of complex macroeconometric models. One possibility to consider, in this context, would be for the relevant DGs to seek a closer collaboration with research institutes and universities specialised in this area of assessment. Appropriate and timely supervision of contracted consultants is crucial in ensuring that objectives are met and that value for money is derived from such ex post assessments.

125.

In conclusion, the ex post evaluation exercise must go beyond the compilation of distinct assessments, carried out at given intervals of time, but should be envisaged and managed as an integral part of an overall process. This is another aspect where a closer, and long term collaboration with research institutes or universities could be of significant benefit in providing a more effective framework within which to integrate the specific reports and recommendations generated at each stage of the evaluation process.