19.5.2006   

EN

Official Journal of the European Union

C 119/1

I-VII

1-16

1-8

9

10-11

12-16

17-53

18-19

20-22

23-34

23-24

25

26-28

29-31

32-34

35-38

35-36

37-38

39-49

39-40

41

42

43-46

47

48-49

50-53

50-51

52

53

54-56

54-55

56

AP

Action Programme

CBC

Cross-Border Cooperation

CIS

Commonwealth of Independent States

DG

Directorate-General

EuropeAid

Europe Aid Cooperation Office

MIP

Multi-annual Indicative Programme

PCA

Partnership and Cooperation Agreement

PCM

Project Cycle Management

RCoA

Russian Chamber of Accounts

RELEX

DG External Relations

SME

Small and Medium Enterprise

SP

Strategy Paper

TACIS

Technical Assistance for the Commonwealth of Independent States

ToR

Terms of Reference

I.

From 1991 to 2006 over 7 000 million euro have been allocated by the European Union for the TACIS (1) programme devoted to assist the newly created Commonwealth of Independent States (CIS) after the break-up of the Soviet Union. The objective of this programme was and still is to promote the transition to market economy and to reinforce democracy and the rule of law in the recipient countries. The European Commission services are in the process of devising a new concept and legal basis for assistance from 2007 onwards (see paragraphs 1 to 8).

II.

The Russian Federation is the largest beneficiary of the TACIS programme, having received about 40 % of all funding under the programme. In total the TACIS funding allocated to the Russian Federation has been about 200 million euro annually. TACIS aims to help create the conditions for sound economic growth and develop a true partnership between the Russian Federation and the EU (see paragraphs 9 to 11).

III.

The objective of the Court's audit was to assess whether the Commission had managed the TACIS projects in the Russian Federation in such a way that these projects had been effective. The Court examined 29 contracts (projects) out of a total of 275 from the contracting year 2000 or later and with an expiry date no later than the end of 2003, i.e. the most recent projects for which the sustainability could be assessed. These projects were financed under the Actions Programmes 1997 to 2000, and implemented mainly in 2002 and 2003. The total value of the contracts audited was over 56 million euro out of a total population of 109 million euro (see paragraphs 12 to 16).

IV.

Overall the Court concluded that nine of the audited projects achieved their objectives. In eight cases the objectives were partially met and in twelve cases they were not achieved. The results of five of the projects were sustainable. Considering the above audit results, which show that the effectiveness of the use of TACIS funds has been low, the Court cannot assess the performance of TACIS projects in the Russian Federation positively. However, the audit revealed that the performance of contractors and monitors met the requirements of their contracts (see paragraph 55).

V.

The Court observed that the Commission had exhaustive programming systems. However, the programming process took long and, despite the Commission's efforts, there was a lack of a real dialogue between the Commission and the Russian authorities and beneficiaries. As a consequence there were some cases where the Russian authorities accepted assistance, which the final beneficiaries did not actually want. This had a negative impact on the effectiveness of the projects (see paragraphs 20 to 22).

VI.

Other observations were that projects’ objectives were sometimes imprecise and not measurable, or sometimes missing, and in some cases underlying assumptions were unrealistic (see paragraphs 26 to 31). The selection and involvement of beneficiaries on some occasions was not successful (see paragraphs 32 to 34). There were problems related to the determination of the project size. In some cases individual projects were incorporated into one set of ToR with loose connection. Co-financing was hardly used (see paragraphs 37 and 38). In twelve cases (of 29) there were major delays in implementation, but the devolution of responsibilities to the Delegation resulted in more effective project management procedures (see paragraphs 39 and 40). Steering committees did not work well. Equipment was often not used for the purpose of the project (see paragraphs 43 to 46). Ex post assessment (evaluations) of the impacts of the projects did not take place (see paragraphs 48 and 53).

VII.

The main recommendations of the Court are the following (see paragraph 56):

—

All planning of external action programmes should be based on a real dialogue between the Commission and the beneficiaries. Only such projects should be financed where there are clear objectives which are shared by the Commission and the beneficiaries.

—

To improve the working of its systems the Commission should apply the Project Cycle Management (PCM) including the logical framework whenever appropriate, and it should avoid consolidating small projects with different objectives and beneficiaries.

—

The Commission should appraise realistically the potential for sustainability during the planning phase of the projects; ensure the ex post assessment, or evaluation, of the completed projects in order to learn the good and bad practices; and provide information on the projects and their results for dissemination purposes.

1.

After the break-up of the Soviet Union, the European Union devised a technical assistance programme for the countries of the newly created Commonwealth of Independent States (CIS). The objective of this programme, named TACIS, was to promote the transition to market economics and to reinforce democracy and the rule of law in the recipient countries.

2.

Two Council Regulations (2) covering the period from 1991-1999 made available approximately 4 221 million euro for assistance to the partner countries of Armenia, Azerbaijan, Belarus, Georgia, Kazakhstan, Kyrgyzstan, Moldova, Mongolia, the Russian Federation, Tajikistan, Turkmenistan, Ukraine and Uzbekistan. The current Council Regulation ((EC, Euratom) No 99/2000 of 29 December 1999 (3)) earmarks 3 138 million euro for TACIS assistance during the period from 2000 to 2006. The Commission services are in the process of devising a new concept and legal basis for assistance to these countries from 2007 onwards.

3.

The aim of TACIS is to transfer know-how through technical assistance, including undertaking studies, preparation of plans, and giving advice and training. Consultants from the EU countries work with the beneficiaries with the aim of transferring know-how and skills, which are intended to be replicated by other projects and organisations in the recipient country. The aid is financed through payments from the Community budget being made directly to the European contractors providing the technical assistance.

4.

In addition to the Council Regulation (see paragraph 2), relations between the European Union and the TACIS countries are based on Partnership and Cooperation Agreements (PCAs) signed between the EU, Member States and individual partner countries. The External Relations Directorate-General (RELEX) is responsible for the programming of TACIS in cooperation with the partner countries. The implementation of the TACIS programme is subsequently defined in multi-annual Country and/or Regional Strategy Papers (SPs) and Multi-annual Indicative Programmes (for the national programme as well as the regional or multi-country programmes) (MIPs). Before establishing the MIPs the Commission has to discuss the priorities with the Committee for assistance to the New Independent States (TACIS Committee) (4).

5.

The Europe Aid Cooperation Office (EuropeAid) currently transforms the MIPs into annual or bi-annual Action Programmes (APs) in dialogue with the TACIS national coordinators, relevant ministries and other organisations in the beneficiary countries and after internal consultation within the Commission. The TACIS Committee is required to give its opinion on the contents of APs, including the list of projects. These documents are completed by financing memoranda (agreements between the EU and the beneficiary countries on the projects to be financed) and planning documents pertaining to the individual projects (Terms of Reference, ToR). The TACIS Cross-Border Cooperation (CBC) programme and the TACIS nuclear safety activities have their own strategy papers and indicative and action programmes.

6.

The planning of the TACIS intervention is intended to be dialogue-driven. This means that at all levels of the programming and project identification stages the Commission and the beneficiary countries should agree on the priorities and the projects. Therefore all documents mentioned above from the MIPs down to the ToRs have to be discussed and signed by representatives of the Commission and of the beneficiary countries.

7.

EuropeAid has overall responsibility for the management of the project cycle, from project identification to project appraisal, financing, project implementation and evaluation.

8.

In 2000, the Commission embarked on a major reform of the management of its external aid programmes (5). The main objective was to make significant improvements to the speed and quality of EC external aid. A key component of this reform was the extensive devolution of aid management tasks and responsibilities to the Commission's Delegations. As a result of this devolution, the Delegations became responsible for project preparation, contracting, and financial and technical implementation. The role of the central services in Brussels has developed from direct management of projects towards monitoring and supporting Delegations.

9.

The Russian Federation is the largest beneficiary of the TACIS programme, having received about 40 % of all funding under the programme. In total the TACIS funding allocated to the Russian Federation has been about 200 million euro annually.

10.

The European Union's policy towards the Russian Federation is geared towards: contributing to strengthening the rule of law through the development of efficient institutions as well as effective legislative, executive and judicial systems; improving the investment climate; enhancing legislative harmonisation with the EU; and cooperating in the fields of justice and home affairs, environment and nuclear safety. These measures should help create the conditions for sound economic growth in the Russian Federation, and developing a true partnership with the EU.

11.

The PCA between the European Union and the Russian Federation regulates the political, economic and cultural relations between the EU and the Russian Federation. It was signed in 1994 and entered into force on 1 December 1997 for an initial period of ten years. The Commission and the Russian authorities are in the process of devising a new instrument for the period after the expiry of the present PCA.

12.

The audit objective was to assess whether the Commission had managed the TACIS projects in the Russian Federation in such a way that these projects had been effective. To achieve this the Court audited a sample of projects in order to determine (i) the extent to which the projects' objectives had been achieved and (ii) how far projects were sustainable. In its audit the Court followed the stages of the Commission's system of Project Cycle Management (PCM): programming; project identification and planning; project financing; project implementation and evaluation.

13.

As the audit concentrated on the project level, the assessment of the programming process was limited to aspects which had a direct impact on the planning of the projects audited. The audit did not cover other general non-measurable political objectives of the projects.

14.

In the framework of this audit, the Court was cooperating with the Russian Chamber of Accounts (RCoA). The RCoA performed a parallel audit on the side of relevant Russian authorities on TACIS projects. The report of RCoA will be published at the beginning of 2006.

15.

The Court examined 29 contracts (projects) out of a total of 275 from the contracting year 2000 or later and with an expiry date no later than the end of 2003 which were the most recent projects for which the sustainability could be assessed. These projects were financed under the Action Programmes 1997-2000, and implemented mainly in 2002 and 2003. The total value of the contracts audited was over 56 million euro out of a total population of 109 million euro (6). The Nuclear Safety projects as well as TACIS CBC projects were excluded from the sample, as the Court has published Special Reports based on audits of these areas in 1998 (7) (with a follow up in the Annual Report 2000) and 2001 (8).

16.

The sample covered different kinds of actions and areas of intervention. In addition to large TACIS projects with a budget of at least 2 million euro, there were three smaller Bistro (9) projects, three smaller extensions of earlier projects and one feasibility study contract.

17.

Most of the projects faced problems during one or several phases of the project cycle. These problems are elaborated below. Table 1 shows the number of projects affected by the main problems revealed by the audit.

Table 1

Number of projects affected by the main problems revealed by the audit

18.

Nine projects fully achieved their objectives. In eight cases the objectives were partially reached and in twelve cases the objectives were not achieved.

Box 1 One project had several specific objectives. One of them was to ensure harmonisation of legal road standards between the EU and the Russian Federation. This objective could not be reached because there is no such EU standard. The second one was to enable comparative tests of road materials by providing a Russian agency with testing equipment. The equipment was delivered but it was still unused almost two years after the project's end. The third objective, training of the beneficiary's staff, was cancelled in the middle of the project.

19.

In general, the most successful projects in terms of meeting their objectives were the small ones, where the objectives were well and clearly identified. Also projects implemented in the regions generally performed better than projects targeting ministries at central level because of a greater level of commitment and feeling of ownership by the beneficiary, and more staff motivation.

20.

Over the years the Commission has developed a system of programming external actions. In theory the system is comprehensive and logical. It has been stable already for many years. In line with other external action programmes many documents are drafted at various levels of TACIS programming and implementation (see paragraphs 4 and 5).

21.

However, the programming process including the internal and external consultations is long. This put pressure on the project identification and preparation of the planning documents for the individual projects and finally when the deadlines for submission of projects were approaching, resulted in ToRs of insufficient quality (see paragraph 26).

22.

The results of the projects audited, and the analysis of the problems faced by them, show that, despite the apparent efforts of the Commission during the formal consultations, the programming and subsequent project identification including preparation of financing memoranda and individual ToRs, were carried out in the absence of an adequate dialogue (see paragraph 6) with the various levels, i.e. Russian authorities administering the TACIS programme and beneficiaries (ministries and other organisations). For some areas of intervention the reform process in the Russian Federation had not advanced as hoped or the political environment on the beneficiary side had not matured enough for a constructive dialogue with the Commission. As a consequence there were some cases where the Russian authorities accepted assistance which the final beneficiaries did not actually want. This lack of dialogue and the length of the planning process (see paragraph 21) had a negative impact on the effectiveness of the projects.

23.

In 1992 the Commission adopted the PCM methodology setting out the format and contents of Financing Proposals and including a requirement to draw up a logical framework (10) in order to improve its project management. However, the use of the PCM methodology and logical frameworks was made obligatory by the Commission only from 2001.

24.

In 2001 the Court published Special Report No 21/2000 on the management of the Commission's external aid programmes. One of the findings in the report was that the logical frameworks as part of the PCM were often not well prepared (11). This TACIS audit showed that the Commission still did not use the PCM, including the logical frameworks, systematically. The audit also showed that in most of the cases where the logical frameworks were used effectively, the projects achieved better results. The use of the PCM framework forced the planning to be done more thoroughly.

25.

The time-span from the original idea or request for project funding to the inception of the actual project was very long. It may take up to three years to launch a consultancy project, which is too long, as the needs and situations of the beneficiaries change. Due to the delays, the projects, especially those related to policy development or crisis management, were already outdated when they eventually started. Consequently, as time had solved the original problems the contractors had to change the projects' approaches which further delayed their implementation.

Box 2 Prior to providing assistance in the area of anti-money laundering activities the Commission insisted on a feasibility study be produced. Mainly due to internal coordination difficulties between Commission services but also because it took the beneficiary organisation unduly long to approve its ToR, the start of this preparatory project itself was delayed by two years. Eventually, this delayed the technical assistance project — which was politically and economically important and also urgent for the Russian Federation — by five years.

26.

The long planning period did not always result in ToRs of good quality. In nine cases insufficient background investigations in the project design phase led to unrealistic objectives. Often, the project designs did not take into account the legal constraints in implementing the project, or the objectives were outside the competence of the beneficiary, or they were just too extensive and abstract to be reached by a single project. For example, most of the projects with objectives or tasks related to policy formulation, including enacting of legislation, were unsuccessful.

Box 3 A pilot project was established to test a new model for facilitating the start of new businesses. However, a similar model had already been established in the Russian Federation with the help of Canadian technical assistance in the middle of the nineties.

27.

The participation of the Russian counterparts, as well as the line directorates-general at the Commission, in the processes of drafting and approval of the planning documents varied greatly and can be deemed to have been insufficient to secure realistic planning in some cases.

Box 4 In one case a city was involved in a district heating and power project but not nominated as a primary beneficiary. It did not endorse the ToR and as a result, the project was delayed and faced other managerial problems caused by a lack of commitment by the city authorities.

28.

Examples of unrealistic assumptions and objectives include: the institutional framework of a beneficiary was not adequate or was non-existent, the target group had no interest in the services provided, the cooperation potential of the local administration was over-estimated or the beneficiary was not, by legislation, allowed to provide the services or information foreseen by the project.

29.

Thirteen projects audited had clear objectives. However, in sixteen of the projects audited, the initial project objectives in the ToR either described only the intended activities or input instead of setting clear and measurable objectives; or they were formulated in the same general way as in the political programming documents, forcing the contractors to define the actual purpose of the projects during implementation. The projects did not have objectives that would have identified the problems to be solved nor did they have such objectives against which the achievements of the projects could have been measured.

30.

For most of the projects audited objectively verifiable indicators had not been defined. In other cases, where the objectives were clear and intelligible, there were differences between the actual project activities and the project's initial objectives as described in the ToR. In one case the project was executed without specific objectives or objectively verifiable indicators. Instead it was directed towards providing ad hoc services to a ministry without being clear about the exact objective of the work. In the case of another project the beneficiaries in the private sector received assistance to develop their commercial activities which in fact was outside the scope of the overall objective of the project.

31.

For some projects the design was overly complex, including too many separate and not interlinked tasks and objectives.

Box 6 One project had 23 different major outputs or objectives integrated into 8 different components. These covered themes like environmental protection in oil production, energy saving measures and studies of fish hatcheries. As a consequence of this complexity several of the foreseen objectives, including the two most significant ones, were not fully achieved.

32.

Selecting suitable beneficiaries as project partners was very problematic because the project objectives were sometimes outside the competence of the project partner, or the beneficiary was not ready to receive or able to benefit from the support. In one case the Commission wanted to support a federal agency for political reasons, in view of the forthcoming cooperation negotiations between the EU and the Russian Federation. However, this agency was not in the end involved in the actual negotiations.

33.

When a project had several beneficiaries, implementation faced a number of problems. Lack of cooperation between the federal and regional authorities or lack of cooperation between different federal and between different regional authorities was usual. This failing, although apparent, had not been taken into consideration during the planning.

Box 7 In one case the initiating beneficiary was operating only at the level of a region. To meet the minimum size requirements of the project budget it had to be expanded to cover several regions. To solve the problem another beneficiary was created but only on paper. It did not have legal identity, nor did it otherwise exist outside the project planning documents.

34.

In five cases audited the beneficiaries were not involved in the project planning, or not early enough, to have an influence on the project design.

35.

According to the programming guidelines, the minimum budget size of TACIS projects in the Russian Federation should be 2 million euro in order to achieve sufficient impact and visibility. However, the programme may finance a number of small-scale initiatives, including Bistro projects. Most projects in the audited sample had a budget of between 2 and 3 million euro (see paragraph 16). In eight cases it was not clear from the project documentation how the project budgets had been arrived at. When the projects’ objectives are not clear, defining a well-founded budget is difficult.

36.

In some cases this minimum size led to problems. Instead of launching several, well-defined and focused projects, multiple tasks were incorporated into one set of ToR with loose connection or no synergy at all (see paragraph 31). The audit results show that such a budget-driven approach, which artificially creates larger projects, is unlikely to enhance the projects' impact and visibility.

Box 8 In the case of an industrial restructuring project 30 subprojects were planned. The budget was set in order to reach the minimum size rather than to satisfy the need of the subprojects.

37.

According to the TACIS Regulation, national co-financing of projects was to be actively encouraged by the Commission. However, co-financing requirements are rare in the TACIS projects in the Russian Federation, even though they are intended to have a positive impact on the beneficiary's and contractor's commitment to achieve the expected results. There were only two cases among the audited projects where the beneficiary was co-financing the activities. The co-financing had a positive impact on the ownership of the projects by the beneficiaries which led to achievement of the projects' objectives.

38.

Most of the projects include in-kind contributions from the beneficiaries such as commitments to make local counterpart staff available or to provide the contractor with premises. In eight cases there were problems regarding this indirect co-financing because the commitments were not honoured, particularly for projects in Moscow. The ministries in the Russian Federation generally are not legally allowed to provide the contractors with office accommodation within their premises. Although the Commission was aware of the problem, it continued to insert such conditions into the ToRs.

39.

Seventeen projects were implemented within the time set. In the remaining twelve cases there were major delays in implementation. In this respect the most critical part was the inception phase, which usually took more time than planned. Those delays were mostly linked to an inadequate quality of the preparation of the projects. Due to the poor ToR, the projects needed many approvals for changes during inception and implementation. The procedure to change the project approach through addenda is time-consuming. Even a change of personnel requires the same procedure as modifications to the project's actual substance.

40.

The devolution of project management responsibilities from Commission headquarters to the Delegation resulted in more effective management procedures through more direct contacts between the project managers and project stakeholders; in fact, more proactive management by the Delegation even salvaged several of the projects from serious difficulties. These observations are in line with those of the Court's audit of the devolution of EC external aid management to the Commission Delegations published in Special Report No 10/2004 (12). However, the implementation of projects which had been programmed and prepared centrally in Brussels and which in 2002/2003 were transferred to the EC Delegation in Moscow (about half of all the projects audited) encountered one-off delays and obvious managerial deficiencies because of this transfer process.

41.

Although the objectives were often not achieved or achieved only partially (see paragraph 18), in most of the cases the contractors had delivered the output (deliverables) as foreseen either in the original ToR or in the ToR amended through addenda. A typical problem for the contractors was to find and maintain suitable staff in more remote project locations when the staff originally allocated was not available. Only one small project was a failure due to a lack of output by the contractor. Another contractor faced financial problems during project implementation, which led to significant delays in implementation.

42.

Where steering committees had been established they usually did not work effectively or efficiently. This hampered the implementation of the projects. The powers of the committees were limited by the Commission so the Russian members did not see what kind of an impact the steering committees could have and were, thus, not actively participating in the meetings. The contractors' project managers had to take over (part of) the tasks of the steering committees.

43.

According to the TACIS Regulation (Euratom, EC) No 1279/96 the programme shall cover reasonable costs of supplies required in support of the implementation of the technical assistance (13). TACIS Regulation (EC, Euratom) No 99/2000 states that a maximum of 20 % of the annual budget can be allocated to investment (14). In practice the service contracts included a supply component which obliged the technical assistance contractors, in cooperation with the beneficiaries, to define the technical specifications of the supplies, and to manage the procurement process in accordance with the TACIS procurement rules.

44.

There was not always clear synergy between the technical assistance and equipment provided under the projects. Nor was there any evidence that the equipment component had addressed the specific areas described (15) in the Regulation or Guidelines. The beneficiaries treated the equipment component (mostly office and IT equipment) more as an additional bonus given for accepting the consultancy.

45.

In many cases the equipment was delivered at the end of the project, due to the long procurement procedure, and therefore could not be used for the project activities.

46.

The equipment needs maintenance (including the supply of consumables) after the project has ended. However, this maintenance did not always take place.

Box 9 In one case, the customs offices did not have any budget to buy consumables for the laboratory. Another laboratory remained mainly unused, as the beneficiary did not have adequate premises in which to use the equipment. One of the federal beneficiaries refused to accept IT equipment for its own bureaucratic reasons (as it did not pay for the equipment it did not know how to book it into the balance sheet). In another case some of the equipment was sold by the beneficiary because it did not know how to make use of it. Part of the equipment could not be used because the needed internet connection was not available. Some fitness equipment was not suited for the target group, children, but was used by soldiers from a local military academy.

47.

Contractors indicated problems in domestic procurement of equipment and services as well. According to the TACIS Regulation (16), taxes and duties are not eligible expenditure. Purchases free of VAT from domestic suppliers in the Russian Federation were impossible because the Russian secondary legislation does not allow it. This is not in line with the agreement between the EU and the Russian Federation (17).

48.

An external consulting company was charged with the monitoring of the projects, during their lifetime. The consultants (monitors) form an opinion about design, efficiency, effectiveness, impact and sustainability of the projects. The financial side of the project is excluded. Also, assessment of the impact of the projects after the expiry of the projects' activities was and is not part of the contract between the Commission and the monitors. The lack of this assessment was not compensated for by evaluations.

49.

In general the monitors performed according to the requirements of their contracts. However, in a few cases the monitors' reports repeated statements expressing future intentions as if they were merits of the projects. Such reports did not reveal clearly enough the implementation problems, which may partly explain the lack of corrective actions.

50.

Typical dissemination activities were seminars, a web-page and brochures. Even though some projects had some dissemination activities during project implementation, those ceased when the project activities finished. For example, the web pages created as part of the activities of six projects were not maintained by the beneficiary after the expiry of the project. To some degree, this also demonstrates the perceived value of the results achieved, as insufficient funds were devoted by the Commission and Russian side to the dissemination of the results.

51.

Nor had the Commission proper systems and instructions for effectively disseminating the results and lessons learned from the TACIS projects. For example, there is no consolidated, public database on TACIS projects. Nor are the Russian authorities fully aware of the results of TACIS projects.

52.

As noted above, there were several problems in achieving the results of the projects and disseminating information about the results achieved. Furthermore, the beneficiaries have only rarely used or further developed the outputs such as reports and models. The results of only five of the projects audited were sustainable.

53.

The Commission carries out sector or countrywide evaluations, but targets rather seldomly individual projects. Only two of the 29 projects audited were evaluated. This is inadequate considering the intended pilot nature of many of the projects as well as the amount of money involved, and it is not at the level required to improve the design of future projects.

54.

The Russian Federation is an important trading and political partner of the EU. Effective EU assistance aiming at creating conditions for democracy, rule of law and sound economic growth in the Russian Federation and to develop a true partnership between the EU and the Russian Federation can be an important and meaningful instrument.

55.

About one third of the audited projects fully achieved their objectives. In eight cases the objectives were partially reached and in twelve cases they were not reached (see paragraph 18). The results of five of the projects audited were sustainable (see paragraph 52). Considering the above audit results, which show that the effectiveness of the use of TACIS funds has been low, the Court cannot assess the performance of TACIS projects in the Russian Federation positively. However, the audit revealed that the performance of contractors and monitors met the requirements of their contracts. The main failings are the following:

56.

To facilitate improved effectiveness of the future assistance to the Russian Federation the Court makes the following recommendations.