EUROPEAN COMMISSION

Brussels, 16.12.2021

SWD(2021) 386 final

COMMISSION STAFF WORKING DOCUMENT

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

Final evaluation of the Regulation (EU) No 250/2014 of the European Parliament and of the Council of 26 February 2014 establishing a programme to promote activities in the field of the protection of the financial interests of the European Union (Hercule III programme) and repealing Decision No 804/2004/EC

{COM(2021) 809 final}

Table of contents

1.Introduction

2.Background

3.Implementation / State of play

4.Method

5.Analysis and answers to the evaluation questions

6.Conclusions and recommendations

Annex 1 - Procedural information

Annex 2 - Stakeholder consultation – synopsis report

Annex 3 - Methods and analytical models

Annex 4 - List of acronyms and abbreviations

Annex 5 - Glossary

Annex 6 - External evaluation support study

1.    Introduction

The Hercule III programme was one of the operational programmes implemented by the Commission under the 2014-2020 multiannual budget. It was a key component of the EU’s two-pronged action to protect its financial interests: at EU level, internal control practices, audits and the investigations from the European Anti-Fraud Office (OLAF) seek to ensure value for money and combat fraudulent practice and irregularities. In parallel, the EU supports Member States in multiple ways in their work to protect the EU budget at national level, including with material support. The Hercule III programme was the vehicle to provide that support for anti-fraud work over the last financing period.

The EU budget for the subsequent multiannual period 2021-2027 has increased substantially. An evaluation of the Hercule III programme can feed in important lessons to the successor programme, the EU’s new anti-fraud programme (UAFP). It mirrors the overall approach of Hercule III, but adapts it to the ever-changing fraud landscape.

Article 13(2)(b) of Regulation (EU) No 250/2014 1 mandates the Commission (with OLAF as lead department) to evaluate the Hercule III programme. This Commission staff working document accompanies the Commission report on the achievement of the objectives of the Hercule III programme.

1.1.    Purpose and scope of the evaluation

The purpose of the final evaluation is to analyse the performance of the now finalised Hercule III programme through a retrospective, evidence-based qualitative and quantitative assessment. The aim is to establish to what extent the programme and the projects it funded can be considered successful, and what lessons can be learnt for the future implementation of its successor programmes. Article 34 of the 2018 Financial Regulation 2  states that programmes and activities that entail significant spending must be subject to both prior and post evaluations, which must be proportionate to the objectives and expenditure.

The aim of this evaluation is also to comply with internal control framework principle 12 (previously internal control standard 14 3 ): ‘the organisation deploys control activities through policies that establish what is expected and in procedures that put policies into action.’

The scope, as set out in the evaluation roadmap 4 , is to evaluate the performance of the programme and to what extent its results will sustain in the longer term. It looks at five evaluation criteria: relevance of all the programme objectives; (internal and external) coherence of the programme; effectiveness in attaining its objectives and expected results, including the sustainability of certain types of actions; the efficiency of the use of resources; the programme's added value to the EU and to society.

The evaluation takes into account the findings of the 2017-2018 Hercule III mid-term evaluation 5 , and data from the annual overviews on implementation of the programme, annexed to the Commission annual reports on the protection of the EU’s financial interests (PIF Reports) based on Article 325 of the Treaty on the Functioning of the European Union (TFEU). The timeframe for the evaluation is the seven-year period between 1 January 2014 and 31 December 2020, including preparatory legislative and implementing activities.

The main stakeholders for this evaluation are the programme beneficiaries and applicants, in particular national and regional administrations in the Member States; participants to conferences, seminars and training events organised with financial support from the programme; partners that could have an overview of action taken under the programme: the Advisory Committee for the Coordination of Fraud Prevention (COCOLAF 6 ), the Anti-Fraud Coordination Service (AFCOS, a national service designated by each Member State to facilitate cooperation and information exchange with OLAF, including exchange of information of an operational nature), the Network of Associations of European criminal law, academic institutions and relevant Commission departments.

Certain projects that started in 2020 – when possible, given that the COVID-19 pandemic prevented physical meetings at certain moments – are still ongoing. The Commission will report on grant agreements and most procurement contracts signed in 2020 and on their effects at a later stage (2021 and onwards).

This evaluation covers all current and previous EU Member States, including unsuccessful applications submitted by potential beneficiaries.

The Commission will draw on the results of this evaluation when preparing future annual work programmes, and when monitoring the EU’s new anti-fraud programme. It adopted the first annual work programme under the UAFP for 2021 by means of a Commission implementing decision on 23 July 2021 7 .

1.2.    Policy context

The protection of the EU’s financial interests (PIF) has to be seen in the broader political and policy context, which includes the institutional architecture governing anti-fraud action that is currently undergoing a significant change.

The expansion of the European Union in 2004 and 2007 led to a strengthened coordination of national anti-fraud activities. The objective of the Hercule I programme was to ensure an equivalent and efficient level of protection of the EU’s financial interests and to support the exchange of best practice. 8 In line with this objective, national anti-fraud coordination services (AFCOS) were set up in all Member States.

In 2011, the Commission adopted the anti-fraud strategy 9 , which was revised in 2019. The strategy applies to all Commission departments and executive agencies. It sets out the general objectives of policy action to combat fraud. Its main goals are to adapt anti-fraud measures to address today’s challenges; to increase the amount of recovered funds affected by fraudulent activities; and to enhance cooperation between EU bodies in the fight against fraud and corruption.

Since 2012, the Commission encourages Member States to adopt national anti-fraud strategies, which should contain anti-fraud measures and policies and thereby contribute to combating fraud both on national and EU levels. 10 According to the latest PIF report 11 on anti-fraud measures taken by the Member States in 2020, the situation on national anti-fraud strategies has improved since by the end of 2020, 14 Member States 12  had adopted national strategies.

In June 2021, the European Public Prosecutor’s Office (EPPO) became operational. Based on Council Regulation (EU) 2017/1939 (EPPO Regulation) 13 , the EPPO was set up to conduct criminal investigations in respect of crimes affecting the EU’s financial interests. The crimes falling within the material competence of the EPPO are defined in the Directive on protection of the European Union’s financial interests by means of criminal law 14 (PIF Directive). Not all Member States have decided to join the EPPO. Therefore, different forms of cooperation between the EPPO, OLAF, and national judicial and administrative authorities will have to be established.

Although tangible progress has been achieved over the last two decades, the level of protection of the EU’s financial interests among Member States remains uneven. This is due to factors such as differing enforcement priorities, technical capacity, the level of PIF awareness, the geographical location (especially on the revenue side), the wealth of the Member State and related resources they allocate to police and justice administrations, as well as differing levels of fraud and corruption.

Hercule III, and now the EU’s anti-fraud programme (UAFP), support the transition towards the new institutional landscape by funding, for instance, training and conferences. By integrating the main IT tools, the anti-fraud information system (AFIS) and the irregularities management system (IMS), into its scope, the UAFP is even better placed to support the multiple bodies involved in this work at national level.

2.    Background 

2.1.    Hercule III - description and intervention logic

The concept of protecting the EU’s financial interests covers prevention, detection and investigation of fraud against the EU’s budget and can extend to measures to address other serious irregularities. Action to protect the financial interests covers the whole of the expenditure side of the EU budget. On the revenue side, action mainly covers ‘traditional own resources’ 15 .

The Hercule programme was specifically designed to protect the EU’s financial interests by providing financial support to Member States and other relevant bodies in combating irregularities, fraud and corruption that are a risk to the EU budget. It was administered by the European Anti-Fraud Office (OLAF), part of the European Commission. The first Hercule programme 16 was established in 2004 by Decision No 804/2004/EC. The Hercule II programme was established in 2007 by Decision No 878/2007/EC 17 , which extended the scope to cover cigarette smuggling and counterfeiting. In 2014, Regulation (EU) No 250/2014 18 established the Hercule III programme for 2014-2020.

Following calls for proposals, the eligible beneficiaries for technical assistance and training under the programme were national or regional administrations of Member States, educational institutes and non-profit-making entities established and operating in a Member State for at least one year. The participation of administrations from non-EU countries was subject to an agreement between the non-EU country concerned and the European Union on participation in EU programmes. For the duration of the Hercule III programme, no such agreement specifically for this programme had yet come into force.

2.2.    The 2011 Hercule III impact assessment – identified drivers

The December 2011 impact assessment accompanying the Commission proposal for a regulation establishing the Hercule III programme identified five key drivers to address. Linked to the five drivers, the impact assessment identified nine specific problems 19 .

-Key driver 1: fraudsters adapt quickly to new circumstances at EU level, operate across borders and exploit the weakest points. Linked with problems 1, 2, 3, 6 and 7.

-Key driver 2: the Commission and Member States have committed themselves to stepping up the fight against cigarette smuggling. Linked with problems 1, 2, 5 and 9.

-Key driver 3: pressure on public finances require increasingly cost-efficient ways to protect the EU’s financial interests through improved detection and prevention. Linked with problems 1, 4 and 5.

-Key driver 4: the vast majority of anti-fraud resources are at national level, in many and varied competent authorities. Linked with problems 6, 7 and 8.

-Key driver 5: differences in incentives and capacity between Member States impede equivalence in the protection of financial interests. Linked with problems 6 and 9.

-Problem 1: how to develop specialist knowledge and deploy state-of-the-art technical equipment to modernise prevention, detection and investigation work to protect the EU budget?

-Problem 2: how to respond to the rapid development in organised crime in key sectors such as cigarette smuggling and counterfeiting?

-Problem 3: how to maintain public confidence in the EU given the rise in challenges to good governance and public finance ethics from past and prospective expansions and the continued increase in EU expenditure in high-risk non-EU countries?

-Problem 4: lack of sufficient awareness and expertise to prevent and detect fraud on an EU scale.

-Problem 5: insufficient and/or ineffective use of risk analysis and information sharing about fraud patterns at EU level.

-Problem 6: shortcomings in the ability and/or willingness of competent authorities to cooperate among themselves and/or with OLAF.

-Problem 7: difficulties in setting up information exchange mechanisms and tools that enable standardised, interconnected and efficient transnational anti-fraud cooperation.

-Problem 8: lack of a uniform administrative and judicial environment to investigate and prosecute EU budget fraud.

-Problem 9: gaps in the skills, experience and sharing of best practice among national competent authorities.

Based on these identified problems, the general objective of the Hercule III programme was ‘to protect the financial interests of the Union thus enhancing the competitiveness of the Union's economy and ensuring the protection of taxpayers' money’ 20 . The programme’s specific objective was ‘to prevent and combat fraud, corruption and any other illegal activities affecting the financial interests of the Union, including cigarette smuggling and counterfeiting’ 21 .

In addition, to provide solutions to the problems identified, the programme had five operational objectives as listed in Article 5 of Regulation (EU) No 250/2014:

a)to improve the prevention and investigation of fraud and other illegal activities beyond current levels by enhancing transnational and multidisciplinary cooperation;

b)to increase the protection of the EU’s financial interests against fraud by facilitating exchanges of information, experiences and best practice, including staff exchanges;

c)to step up the fight against fraud and other illegal activities by providing technical and operational support to national investigations, in particular to customs and law enforcement authorities;

d)to limit the currently known exposure of the EU’s financial interests to fraud, corruption and other illegal activities with a view to reducing the development of an illegal economy in key risk areas such as organised fraud, including cigarette smuggling and counterfeiting;

e)to step up the development of specific legal and judicial protection of the EU’s financial interests against fraud by promoting comparative law analysis.

Lastly, reporting by Member States on irregularities and suspected fraud cases detected during the implementation of the budget, and reported on the basis of Article 325 TFEU, showed that the detrimental impact on the EU budget over the years remained at a level that required the EU to maintain this work to protect the EU budget. Figure 1 overleaf illustrates the programme’s overall intervention logic, as described above and with a breakdown by type of action, including the planned outputs, results and impacts.

Figure 1 – General intervention logic of the Hercule III programme

2.3.    Intervention logic per type of Hercule III action

2.3.1.    Technical assistance (see Figure 2 below)

Financial support from the programme under technical assistance enabled the purchase of:

-technical equipment, investigation tools and methods with training provided for staff to operate such tools; devices and animals to carry out inspections; training courses; automated systems for the recognition of vehicle number plates and container codes; training to operate such tools;

-access to databases with information on trade flows, ship-manifest data, container traffic and company information, as well as services to support Member States in storing and destroying seized cigarettes and other counterfeit goods; and the chemical analysis of samples from tobacco and cigarette seizures.

2.3.2.    Training and conferences, seminars and digital forensics (Fig. 3)

The programme funded the organisation of conferences, seminars, courses, workshops, ad hoc training and staff exchanges to exchange experience and good practice between competent authorities. The aim was to disseminate knowledge on how to identify risk factors and increase knowledge of the PIF Directive 22 .

The programme provided support for digital forensics training courses, including launching a quality assurance process and certification procedure for digital forensics experts (‘DFAT’: digital forensic analysis training), and creating a network of experts.

2.3.3.    Legal training and studies (Figure 4)

The programme funded the organisation of regular high-profile research activities, such as conferences, seminars and workshops to improve cooperation between academics and practitioners; scientific publications and dissemination of knowledge among the judiciary (prosecutors, judges and court staff) and other legal professions (lawyers, barristers, solicitors, notaries, bailiffs). For example, it funded an external study 23  to identify an approach to measure the illicit market for tobacco products (April 2020).

Figure 2 – Intervention logic – Technical assistance: financial support to purchase equipment and services needed by national authorities for investigations

Figure 3 – Intervention logic - Training for law enforcement staff to boost operational and investigative capacity to run PIF investigations

Figure 4 – Intervention logic – Support for conferences and legal studies to develop academic research into PIF at EU and national levels

2.4.    Input of resources, expected results and impact

Over the period 2014-2020, the Commission managed a budget of EUR 104.9 million for the Hercule III programme, with an annual average of commitments between EUR 14 and 17 million, disbursed in the form of grants and procured services.

The programme provided financial support to several types of activities that come under three broader areas of action 24 :

1.Technical assistance (at least 70% of the programme's budget – through grants), providing national authorities with financial support to acquire specific knowledge, equipment and information technology tools and to acquire access to specific databases and IT tools that facilitate data collection and analysis. 

The aim of the technical assistance was to obtain the following results:

-successful seizures, confiscations and recoveries following joint action and cross-border operations;

-technical equipment used properly to generate sustainable added value (at least for the normal – variable – life span of the equipment); 

-information exchanged among Member States on the results achieved with the technical material.

2.Training (maximum 25% of the programme's budget – through grants and procurement), to help fund the organisation of targeted specialised training, risk analysis workshops, conferences and legal studies.

The expected results of the training initiatives were to increase awareness among the judiciary (prosecutors, judges and court staff) and the legal profession (lawyers, solicitors, barristers, bailiffs) about the protection of the EU’s financial interests, to increase digital expertise and to train participants on evidence-based policies and legislation to protect the EU budget.

The expected results of the legal training were also to create and increase knowledge about the fight against fraud and to continue the exchange of good practice, as well as to improve as far as possible the coordination of exchanges between practitioners and academics.

3.Any other action (maximum 5% of the budget): other types of action provided for in the annual work programmes, as necessary to achieve the programme’s general, specific and operational objectives.

Under this category, the programme funded action such as a 2015 ‘Eurobarometer’ survey (updated July 2019) 25 on public opinions, attitudes and behaviour in the EU on cigarette smuggling and the consumption of smuggled cigarettes. It also funded the development of evidence-based policies and legislation to contribute to better policy-making.

The overall expected impact of the programme is a significant and lasting contribution to:

-an effective protection of the EU’s financial interests;

-ensuring the competitiveness of the EU’s economy (for example by strengthening the competitiveness of European businesses by tackling and eradicating illegal economic circuits wherever possible) and protecting public finances;

-ensuring a continued and reliable protection of taxpayers’ money, maintaining public confidence that fraud and corruption are being tackled.

2.5.    External factors that affected the programme’s achievement

2.5.1.    Positive external factors that affected the delivery of results and achievement of objectives

Technical assistance projects were found to have a number of positive external effects. The evaluation showed that, based on the assessment of achievements, the acquisition of modern equipment produced a number of positive effects that are not among the programme’s primary objectives, but nevertheless benefit the national authorities.

One of the main positive external effects identified was that the equipment purchased to perform border checks had proven useful to detect, for example, illegal drugs or illegal weapons. Evidence from the case studies also indicates the usefulness of equipment in seizing illegal substances or artefacts that do not primarily jeopardise the EU’s financial interests but pose a more general security threat to EU Member States.

Evidence from the assessment of achievements and from the 16 case studies suggests that technical equipment increased the health and safety of customs and police officers. The purchase of more modern scanners minimised the risk to radiation exposure. Modern communication and surveillance equipment helped officers monitor criminal activities while staying undetected and at a safe distance.

Figure 5 – Unexpected or unintended knock-on effects of the programme - example

2.5.2.     Negative external factors that affected the delivery of results and achievement of objectives

The evaluation identified two possible negative effects that could have affected the delivery of results and achievements.

-Administrative capacity. The lack of national resources to manage the administrative burden can hinder the uptake of applications from Member States that may not have the same organisational capacity.

This can be linked to the lack of clarity regarding decisional responsibilities in the Member States.

-Lack of legislative support. One EU-level interviewee noted that customs units uncovered an increase in the illegal traffic of raw tobacco. But, as raw tobacco is not covered by the EU excise system, it was often not possible for the enforcement services to analyse collected or seized samples of illegal raw tobacco, which made (criminal) investigations into the origins of seized raw tobacco difficult.

This legal gap may relate to issues that are being identified in the monitoring and control system applied to the raw tobacco supply chain. The Commission is currently carrying out an impact assessment related to excise duties legislation 26 .

2.6.    Baseline and points of comparison

The baseline provides a point of comparison against which the Commission will evaluate the implementation of the Hercule III programme. It takes account of the impact assessment of the Hercule III Regulation and the evaluation of Hercule II (the preceding programme).

2.6.1.    The 2011 impact assessment of the Hercule III Regulation

The 2011 Commission proposal for a regulation establishing the Hercule III programme was based on an impact assessment. 27  The assessment found that the positive impact of the Hercule II programme was not fully reached at the time (2011), in part because some Member States were unable to co-finance Hercule II projects, especially in terms of procuring technical equipment.

Notably on the eastern and southern fringes of the EU, the affected areas were those where there was the greatest common interest to strengthen the control of the EU’s borders against smuggling or transnational fraud. In response, the Commission increased the maximum funding rate in co-financed activities from 50% to 80% in the Hercule III programme.

The maximum co-financing rate can be exceptionally raised from 80% to 90% for very specific projects that meet at least two of the following criteria: i) they take place at an external EU border; ii) they take place at the most vulnerable locations; iii) they reflect the results of the Eurobarometer survey of public attitudes to counterfeited, smuggled cigarettes and ‘cheap whites’; and iv) they reflect the findings of the 2014 report on the protection of the EU’s financial interests.

2.6.2.    Final evaluation of the 2015 Hercule II programme

The 2015 ex post evaluation of Hercule II found that the support granted to Member States was largely positive and led to better transnational and multidisciplinary cooperation between authorities in their work to protect the EU’s financial interests. The programme contributed to strengthening the operational and investigative capacity of its main beneficiaries, such as police forces, customs authorities and other law enforcement agencies.

The evaluation also found that the effectiveness of the programme could be even greater by facilitating more personal contacts between stakeholders, for example, through staff exchanges. It also highlighted the need to strengthen the reporting on the programme’s activities, and monitoring of results. These findings were echoed in the mid-term evaluation of Hercule III.

2.6.3.    The January 2018 Hercule III mid-term evaluation report

In January 2018, the Commission published its mid-term evaluation report of Hercule III 28 covering the period 2014-2017. The evaluation identified similar problems to those listed in the 2011 impact assessment, in particular:

-gaps in skills, expertise and the sharing of best practices among national competent authorities;

-lack of standardised and interconnected information exchange mechanisms; cooperation among authorities as well as a lack of awareness and expertise to prevent and detect fraud;

-gaps in developing specialist knowledge and technologies to fight fraud, in improving the use of risk analysis and information sharing, and in responding to developments in organised crime;

-differences among Member State systems to investigate and prosecute fraud.

The mid-term evaluation also demonstrated that, at that stage (end 2017), the programme had fulfilled its mission. It had proven relevant, with mechanisms in place to ensure coherence, and it had been effective and efficient. There was a clear added value in continuing to run a programme at EU level, and its activities were found to be broadly sustainable. The evaluation report also contained suggestions for improvements, in particular in terms of cross-border cooperation between Member States, cooperation with non-EU partners and new technological developments.

With regard to training, the mid-term evaluation concluded that only a few projects had involved staff exchanges between national administrations and few involved international participation.

2.6.4.    The July 2018 UAFP inception impact assessment

When comparing the situation between 2011 (impact assessment for the Hercule III programme) and 2018 (the impact assessment 29 for the EU’s new anti-fraud programme (UAFP) 30 ), the main issues that the initiatives aimed to tackle were largely similar. Fraud undermining the EU’s financial interests remained a prevalent issue.

In 2016, 19 080 fraudulent and non-fraudulent irregularities were reported to the Commission, involving a total of approximately EUR 2.97 billion. 1 410 irregularities were reported as fraudulent, involving EUR 391 million, covering both expenditure and revenue. Although this was an improvement compared to 2015, the sheer volume of identified fraud and irregularities remained high.

Most of the key drivers of fraud identified in the inception phase of the Hercule III programme remained relevant when the impact assessment was carried out for the UAFP. In particular, the capacity of fraudsters to adapt quickly and operate across borders to exploit the weakest points had not noticeably diminished. Pressure on public finances is even higher today, requiring the Commission and the Member States to protect the EU’s financial interests by stepping up detection and prevention work.

Lastly, the differences in incentives and capacity between Member States persist and result in an uneven protection of the financial interests within the EU.

3.    Implementation / State of play

3.1.    Administrative implementation

Following the final evaluation of the Hercule II programme, some changes were made. Hercule III integrated IT support into the scope for technical assistance and brought in stricter reporting requirements for beneficiaries. It also brought in an annual overview with information on the results of the Hercule III programme, i.e. an annual report by the Commission on actions and results of the Hercule III programme.

Another major change was the increase of the co-financing rate under technical assistance actions, from 50% (Hercule II) to 80% (Hercule III) of all eligible costs. Financial support was available in the form of grants, public procurement or reimbursement of the cost of participation in activities under the programme.

The Commission financing decisions with the annexed annual work programmes ensured that the general, specific and operational objectives of the Hercule III programme were implemented in a consistent manner. They outlined the outputs and the expected results, the methods of implementation and total amount (input, available budget per sector, list of the calls for proposals to be published and the contracts to be concluded).

Once the Commission had adopted the financing decision and made available the financial resources, it published calls for proposals (e.g. for technical assistance, conferences, legal training and studies). The proposals submitted by Member State authorities or other eligible applicants were then evaluated according to the selection and award criteria.

Every year, from 2014 to 2020, the Commission launched three calls for proposals: one for technical assistance, one for legal training and studies and one for training, conferences and staff exchanges. In 2020, following requests from applicants, in view of their organisational challenges during the COVID-19 crisis, the Commission extended the deadline for the three calls for proposals by 3-4 weeks, depending on the type of call.

The Commission implemented the Hercule III programme by issuing grants following the calls for proposals. It entered into public procurement contracts (for purchasing (access to) databases, organising conferences and (specialised) training, plus the development of specific IT tools) following calls for tender and administrative arrangements concluded with the European Commission’s Joint Research Centre (JRC) for the development of specific tools.

As of November 2021, OLAF registered 280 finalised projects out of 336 granted projects (83.3%). For a project to be considered finalised, the technical and financial reports have to be submitted and the final payment made, which is some time after the project ends.

3.2.    Monitoring implementation of the programme

The programme is monitored through annual implementation reports that the European Commission submits to Parliament and to the Council. The reports present input and output indicators on the use of the budget, the number of actions carried out, the main results achieved, information on cooperation and coordination between the Commission and the Member States, and insights in terms of consistency and complementarity with other programmes.

The annual implementation report is a Commission staff working document annexed to the annual PIF report. The European Parliament’s Committee on Budgetary Control (CONT Committee) issues its opinion on the PIF report by the end of each year. The Commission carefully takes account of the observations for the following years.

Four key performance indicators (KPI) are used to measure the achievement of the programme’s specific objective 31 , ‘to prevent and combat fraud, corruption and any other illegal activities affecting the financial interests of the Union’:

a)information on seizures carried out by joint actions and cross-border operations;

b)added value of technical equipment funded under the programme;

c)(the number of) information exchanges among Member States; and,

d)the number and type of (specialised) training.

Beneficiaries of the actions must report on the results achieved. The method for reporting differs according to the type of action (technical assistance and training). For technical assistance grants, the results are first reported in a ‘final technical report’ issued after the request for final payment of the grant. Results are also reported in a ‘final implementation report’ submitted one year after the final date of the grant agreement. The reporting condition is set out in the annual work programme, specified in the calls for proposals and required by the grant agreement.

Beneficiaries of training grants must submit a ‘final technical report’ with the main results of surveys held among the participants to assess the relevance of the event and user satisfaction, as set out in the calls for proposals and required by the grant agreement. As of 2017, grant beneficiaries must organise a ‘post-event survey’ six months after the event took place to measure the mid-term impact, in particular in relation to the use of the skills acquired during the training course and the effect of networking.

For procurement-based activities, conferences and training initiatives, the beneficiaries must run user satisfaction surveys following these events. For contracts for the purchase of access to databases, monitoring the results of the actions takes a different form: the Commission receives detailed information on how the different users make use of the databases.

The findings based on the beneficiaries' reports in both the technical assistance and the training reports are included in the annual overviews on implementation of the programme and are used as input to estimate the KPI 32 . The Commission also monitored the identified and reported impacts of the programme through information contained in grant beneficiaries or contractors’ final technical reports. These covered success stories, the results achieved with the activities funded and the contribution they made to protecting the EU’s financial interests.

3.3.    Implementation figures 2014-2020

The budget available for Hercule III was only slightly bigger than for the Hercule II programme (EUR 98.5 million for Hercule II for 2007-2013, compared to approximately EUR 104.9 million for Hercule III for 2014-2020).

Figure 6 - Annual commitments Hercule III - 2014-2020 33 (actual figures)

Applications for Hercule III grants were consistently over the level of grants available to award over the period 2014–2019. In particular, applications for technical assistance grants are very competitive, with a peak in 2014, when only 25% of all applications were successful. The average success rate of applications for technical assistance is 33%, while on average 43% of all applications for anti-fraud training are successful.

On average 51% of applications for legal training and studies are awarded a grant. Thus, in all three areas, only half or fewer than half of all applicants can proceed with their suggested projects, showing that the Hercule III programme is successful and relevant for applicants, with the high co-financing rate of grants being an incentive to apply. No more than 5% of the budget can be spent on other types of action. They include funding studies that are not directly related to anti-fraud or legal issues, or in some circumstances, support for nationals from non-EU countries to participate in workshops or conferences. The special Eurobarometer survey from July 2016 (updated, July 2019 34 ) on public perception of illicit tobacco trade was also funded under ‘other types of action’.

Figures 7-8-9 - Number of grant applications and grants awarded per type of action

Source: OLAF own data on Hercule III applications.

The number of applications received fluctuated over the programme’s lifespan. Hercule III, compared to the previous Hercule II programme, increased the maximum co-financing rate in technical assistance (from 50% in the previous programme to up to 90% of the total cost). This triggered a high interest from Member State authorities in applying for funding in the first three years. The increased interest combined with the limited amount of funding available under the programme led to the Commission having to reject a high number of proposals due to insufficient funds. This is likely to have been the reason for a gradual decrease in the applications received for the following years, stabilising at around 40-50 technical assistance applications per year.

For applications for funding training, where the maximum co-financing rate remained as in the previous Hercule II programme (80%, with an exceptional rate of 90%), the number of applications varied between 17 and 42 for anti-fraud training and between 6 and 22 for legal training, with no obvious explanation for these fluctuations.

The Hercule III calls were publicised in a similar way each year, through existing networks and channels, on OLAF’s website and through Official Journal announcements. Starting with 2017, the calls were also published on the Commission’s ‘Funding and tenders’ portal, increasing visibility for the targeted stakeholders.

All Member States are eligible for funding under Hercule III. The evaluation found – based on publicly available aggregated data under ‘technical assistance’ and ‘anti-fraud training & legal studies’ during the past funding period of 2014-2020 – that some Member States obtained relatively more funding under Hercule III than others.

Technical assistance actions are (more) concentrated in Member States that have an external border to the east, as well as some coastal states such as Spain and Ireland.

The cause of this possible inequity between the 28/27 Member States is likely to be due to multiple reasons. In some cases it may be linked to the geographical location of the Member State; in others it is not immediately identifiable, such as for example the low participation rate of Denmark, Austria or the United Kingdom.

In some circumstances, projects run in non-EU countries may also receive funding from Hercule III. Costs incurred for persons from non-EU countries to participate in an event funded under the programme, such as conferences, seminars or training sessions, can be considered eligible for funding under certain conditions. In such instances, the funds are mainly used to organise conferences, workshops, or training in order to strengthen cooperation with the EU and support capacity building of tax authorities in those non-EU countries. This includes coordination of activities of participating countries and representatives of international organisations with the aim of exchanging experience and best practice between authorities, developing high-profile research and conducting studies. Figure 10 shows all non-EU countries in which projects were run.

Figure 10 - Non-EU countries in which (training and conferences) projects were run that received funding under Hercule III

Source: OLAF - Annual overviews with information on the results of the Hercule III programme as part of the annual reporting by the Commission to the European Parliament and to the Council on the protection of the European Union's financial interests - Fight against fraud. 2014-2019 (PIF Reports).

3.4.    Costs and benefits of the programme for the stakeholders

3.1.1.Direct and in-direct benefits experienced by stakeholders

The main direct benefit of the programme related to technical assistance actions was the purchase of technical assistance equipment and the provision of innovative tools. As a result, 14 out of the 80 respondents to the survey noted that the new equipment had led to increased detection rates and, by default, an increase in the number of seizures. This finding was similar to the finding under training, where the development of research and knowledge building produced the greatest benefit.

The programme also generated indirect benefits in the form of strengthening collaboration and cooperation between and within Member State authorities and stakeholders, and the exchange of best practice.

3.1.2.Relationship between the costs and benefits

Considering the costs and benefits incurred/accrued by stakeholders, most applicants stated in the survey that the benefits of the programme outweighed the costs. In addition, several stakeholders highlighted that, for technical assistance projects, the value of goods seized often exceeded the initial value of the contract awarded by the programme. This was also seen in the case studies in Poland, where the value of a single seizure made possible through Hercule-funded tools covered the entire investment cost of the project.

A few interviewees highlighted several contextual factors that can affect the balance between costs and benefits. In particular, one partner organisation highlighted that the programme budget had not taken into account the rise in staff costs over the funding period, thus in comparison with other funding opportunities, the programme can appear relatively more restrictive.

The next figure shows the difference between the average maximum amounts to be awarded and the average amounts finally requested and received per type of action.

Figure 11 - Average amounts awarded and requested by type of action 35

On average, the beneficiaries of technical assistance actions request payment to a level of around 88% of the maximum awarded grant amounts. The beneficiaries of training actions requested on average only 73% of the maximum amount awarded. The reason for the difference between the maximum amount awarded and the amounts requested are corrections or cost adjustments that are proposed either by OLAF or by the beneficiaries when submitting the final financial reports.

4.    Method

4.1.    Short description of methodology

To complement its work, the European Commission commissioned a study to underpin this evaluation. The external study was carried out between October 2020 and October 2021 by RAMBOLL Management Consulting S.A./N.V., in a consortium with London Economics (UK) and Università Cattolica (Milano, IT). The conclusions of the study are set out in the final report 36 .

An inter-service steering group of Commission departments 37 was set up to steer the evaluation process. The group reviewed the roadmap 38 , the terms of reference for the external contract, and the inception, interim and final reports.

In June 2020, the European Commission launched an online public consultation on the evaluation roadmap. It did not attract any responses, probably because the programme targets a specific and specialised audience. The Commission did not run an open public consultation on this stage of the evaluation as the primary beneficiaries are national administrations that were contacted via targeted surveys.

The evaluation is based on information from a selection of 98 projects funded by the programme. The information was obtained through desk research and targeted consultations of internal and external stakeholders through interviews and three specific surveys (to beneficiaries and unsuccessful applicants of technical assistance; to participants of funded events; and to users of the databases procured). Further detailed information on the external study methodology can be found in Annex 3 to this staff working document and in Appendix 5 to the study’s final report (the synopsis report).

Data were collected through desk research and literature review, online interviews for 16 case studies in eight selected Member States, and during online meetings with stakeholders from the EU institutions and external partners associated to the programme.

The replies to the evaluation questions are based on two data sources:

Primary data. This includes data collected specifically for the evaluation via:

-online interviews with Commission staff; national institutions active in fraud prevention and the protection of the EU's financial interests, as well as beneficiaries of actions funded under the programme;

-targeted online surveys with beneficiaries of projects funded during the programme; unsuccessful applicants; participants in events; and users of services procured under the programme, in particular database users.

Secondary data. This includes data collected by examining:

-public sources, such as the text of the Regulation and supporting documents (the Commission proposal as adopted in 2011, together with the impact assessment); Hercule III annual work programmes; Hercule III annual implementation reports; Article 325 reports; the final evaluation report and interim reports of the Hercule II programme, legal acts underpinning other EU programmes 39 ;

-documentary evidence on applications for grants (application forms), contracts and grants awarded (final technical reports, final financial reports and final implementation reports);

-quantitative information on implementation of the annual work programmes, such as the budget and the number of grants/contracts, broken down by type of action and Member State as collected from the annual implementation reports.

4.2.    Analysis of data and results

The primary and secondary data gathered by RAMBOLL was analysed using several methods, including descriptive statistical analysis (closed questions in the online surveys), quantitative data analysis techniques (case studies, economic analysis), and qualitative data analysis (desk research, interviews and open questions in online surveys).

The evaluation question matrix 40  was a key tool for the analysis, which follows the structure of the evaluation questions and related indicators to facilitate cross-checking and matching of results. Checked and confirmed data were used to respond to the evaluation questions as listed in the evaluation question matrix.

The collected primary and secondary data were verified and confirmed at the level of the evaluation criteria, comparing and contrasting the results of different data sources. The data were used to identify and analyse the overall achievement of the programme’s objectives, their continued relevance, programme efficiency and the sustainability (effect) and added value of the programme at EU level.

In using triangulation to provide answers to the evaluation questions and sub-questions put forward in the evaluation matrix, the principle is that a hypothesis set out in the intervention logic can only be confirmed if different types of stakeholder support it and different data sources provide relevant evidence. The synthesis was built on the findings and fed into the development of robust conclusions and recommendations, and the final study report. The results of this analysis are presented in the appendices 41  to the final study report.

4.3.    Limitations to the methodology, mitigating measures and robustness of findings

The lifecycle of a programme like Hercule III is such that it takes some time, at the end of the programme, to identify and obtain results for the most recent projects. This limits the input available for a final evaluation, considering that in 2020 many (physical) activities had to be suspended due to the pandemic.

Another limitation is that, although most data are fully available for most projects, for other (types of) projects, the information has been difficult to collect in practice. Due to restricted physical access to offices in the OLAF premises during the COVID-19 pandemic, the final project reports were not available for all 100 initially selected projects. The incomplete project files were replaced by other projects according to the selection criteria set out in the inception report. The final sample totalled 98 instead of 100 projects. This did not affect the findings of the study.

The number of respondents to three targeted online survey consultations appeared unbalanced both in the number of addressees contacted (partly lacking private data such as email addresses) and in the number of (completed) surveys returned to the consultant.

Lessons learnt from past evaluations. The study on the mid-term evaluation in 2017-2018 found that beneficiaries had difficulties providing the data to measure the key performance indicators (‘KPI’) laid down in Article 4 of the Regulation (information on seizures carried out by joint actions and cross-border operations; added value of technical equipment; and, information exchanges among Member States).

The collection of data to measure these indicators is reportedly the most burdensome aspect of the reporting phase. The reports to feed the KPIs therefore provided only relatively limited input into the mid-term evaluation of the programme.

Figure 12 – Alleviating the issue of data collection issue under the new UAFP

5.    Analysis and answers to the evaluation questions

The analysis was conducted against the evaluation criteria of relevance, effectiveness (including sustainability), efficiency, coherence (both internal and external), and EU added value.

5.1.    Relevance of the programme

The assessment of relevance explores the degree to which the programme proves relevant to protect the financial interests of the EU, and the extent to which it is able to tackle current and future problems and needs faced by stakeholders.

3.1.3.Key contextual factors

In understanding the programme’s overall relevance, it is important to examine the context 42 in which it operates and the current and future challenges it aims to tackle. For the present purposes, that is notably the level of fraud to the detriment of the EU budget on both the expenditure and the revenue side.

Every year, the Commission reports to the European Parliament on irregularities related to the EU’s financial interests reported by the Member States. The Commission prepares the Report on the Protection of the Union’s Financial Interests (the ‘PIF report’).

Protecting the EU’s financial interests is a shared responsibility of the European institutions and the Member States.

Member States contribute to the largest share of the EU budget, collect the traditional own resources on behalf of the EU, and also directly manage about two thirds of its expenditure (mainly cohesion policy funding, agricultural and fisheries-related expenditure). Under Article 325 TFEU, Member States take the same measures to counter fraud affecting the EU’s financial interests as they take to counter fraud affecting their own financial interests.

3.1.3.1. Reported irregularities and fraud with regard to revenue

In 2020, the number of reported cases in comparison with the five-year average decreased for both fraudulent and non-fraudulent irregularities related to EU revenue. The related amount increased for fraudulent irregularities but decreased for non-fraudulent irregularities 43 .

Figure 13 - Revenue: irregularities detected by Member States - traditional own resources (reported in 2020)

Figure 14 – Trends and variations in reported cases

As concerns revenue, the overall situation in 2020 46 appears to be less affected by the COVID-19 pandemic than could have been expected, with the number of cases reported as fraudulent or non-fraudulent and of the related amounts within the usual range of annual fluctuations. However, the impact of COVID-19 on the Member States varies.

Inspections by anti-fraud services was the most successful method of detecting fraudulent cases and related amounts in 2020.

Most cases reported in 2020 as fraudulent or non-fraudulent affecting EU revenue relate to undervaluation, incorrect classification or wrong description of goods or smuggling. 

Footwear, textiles, vehicles, electrical machinery and equipment were the types of goods most affected by fraud and irregularities in terms of number of cases and monetary value.

Compared to the number of cases detected and reported in previous years, the number of cases of fraudulent irregularities reported in 2020 (451) is comparable to 2019 but the financial impact is higher, with EUR 107.8 million involved, compared to EUR 80 million in 2019 (+35%). The number of detected cases has steadily decreased from 600+ in 2014 to 451 in 2020. These figures indicate the need to maintain the programme.

3.1.3.2. Reported irregularities and fraud as concerns expenditure

Between 2015 and 2020, the number of reported irregularities (fraudulent and non-fraudulent) related to EU expenditure for the programming period 2007-2013 decreased. The number of cases linked to the next period 2014-2020 have been increasing, consistently with each implementation cycle. Reported irregularities related to annual spending (direct aid to farmers and market support measures) remained stable.

The EU and its Member States share the responsibility for protecting the EU’s financial interests and for fighting fraud. Member State authorities manage three quarters of EU expenditure and collect the EU’s traditional own resources. The Commission oversees both of these areas, sets standards and verifies compliance.

The European Commission’s annual reports on the protection of the EU’s financial interests detail the measures taken at European and national level to counter fraud affecting the EU budget. These reports are based on information from the Member States, including data on detected irregularities and fraud.

The number of irregularities reported as fraudulent (including cases of suspected or established fraud) and the associated amounts are not a direct indicator of the level of fraud affecting the EU budget. They primarily indicate the level of detection and reporting of potential fraud by Member States and EU bodies. However, analysing this information helps to assess the overall level of fraud, identify the areas most at risk, and thereby help target action at both EU and national level.

In 2019, a total 939 irregularities (514 on the expenditure side, 425 on the revenue side of the budget) were reported as fraudulent (i.e. 8% of all irregularities detected and reported), involving about EUR 461 million (28% of all reported financial amounts were affected by irregularities) and covering both expenditure and revenue.

The numbers reported in 2019 and their monetary value decreased significantly compared to 2018.

Looking at a five-year period (2015-2019), the number of reported fraudulent irregularities was 40% less than in 2015, and 25% below the five-year average. The financial impact fluctuates substantially, as it is skewed by individual cases involving large sums.

In 2019, the Commission was notified of 10 787 irregularities (6 550 on the expenditure side, and, 4 237 on the revenue side for traditional own resources) reported as non-fraudulent (similar to 2018). The financial amounts involved decreased by 8% to approximately EUR 1.2 billion.

In 2020, the financial value of detected fraud against the EU budget continued to fall according to the 2020 PIF report, adopted by the European Commission on 20 September 2021. The 1 056 fraudulent irregularities (451 on traditional own resources and 605 on expenditure) reported in 2020 had a combined financial impact of EUR 374 million (EUR 108 million – own resources; EUR 266 million - expenditure), around 20% fewer than in 2019 and continuing the steady decrease seen over the last five years. The number of reported non-fraudulent irregularities remained stable, but fell in value by 6%, according to the report 47 .

Over the last five years, the number of reported irregularities (fraudulent and non-fraudulent) related to EU expenditure for the programming period 2007-2013 decreased, while those linked to the 2014-2020 EU budget have increased, consistently with the implementation cycles. Reported irregularities related to annual spending (direct aid to farmers and market support measures) remained stable.

Organised crime is a significant threat to the European public, to business and to state institutions, as well as to the economy as a whole. Organised crime groups operate across all EU Member States and use their large illegal profits to infiltrate the legal economy and public institutions, including via corruption. They erode the rule of law and fundamental rights, and undermine people’s right to safety as well as their trust in public authorities. Fraud is becoming increasingly appealing for organised crime.

3.1.3.3. Fight against illicit trade in tobacco products

Combating the illicit tobacco trade is an important component of the EU policy to protect the EU’s financial interests. Its effects go beyond the finances though, since the illegal tobacco trade also affects the EU security and health. Throughout 2020, the Commission continued implementing the second action plan to combat illegal tobacco trade 48 . 

The work to combat the illicit tobacco trade, which had featured prominently in activities supported by Hercule III, is also illustrative of the difficulties in measuring the prevalence of fraud, even in a single sector. Several studies have attempted to measure the true scale of illicit tobacco markets, including a 2020 external study financed by the Hercule III programme 49 . This study discussed the methodological difficulties in detail, and only with those caveats suggested that counterfeit and contraband consumption could account for 8% of total consumption in 2018 and that there were approximately 43.6 billion counterfeit and contraband cigarettes in Europe in 2019, all of which evaded taxes and customs duties.

3.1.4.The relevance of the Programme’s objectives

The 2018 mid-term evaluation 50  of the Hercule III programme gives a similar picture in terms of the context. The evaluation demonstrated that the programme’s specific and operational objectives and activities remained relevant. It noted that the rationale underlying the programme was still appropriate and no additional adjustments were required to account for current problems affecting the financial interests of the EU.

The findings of the current evaluation are in line with the 2018 findings, with a large majority of stakeholders stating that both the specific objective and operational objectives were highly relevant in terms of protecting the EU’s financial interests. The programme’s specific objective was found to be the most relevant by an average of 83% respondents across all three surveys.

In particular, operational objective 3 on stepping up the fight against fraud by providing technical and operational support to national authorities, ranked the highest (80%) in terms of relevance. Crucially, the interviews also confirmed that the operational objectives will remain relevant in the future. Respondents found operational objective 2 on the exchange of information and operational objective 3 on providing technical and operational support to national authorities to be particularly relevant.

3.1.5.The relevance of the Programme in meeting stakeholders’ needs

The programme was viewed as highly relevant in addressing the main problems of stakeholders related to the protection of the EU’s financial interests. In the applicants’ survey, 101 out of 126 (80%) answered that the programme was relevant to address the main problems they faced in combating fraud over the period 2014-2020, and they noted that Hercule III was the only programme specifically designed to tackle fraud and illicit activity across the EU. Respondents considered the programme relevant because it complemented the work carried out at Member State level to combat fraud.

Figure 15 – Example of the Programme’s relevance from the case study findings

Evidence from the case studies and interviews found that, while most Member States have their own national strategy to fight fraud, the programme provided additional support that they could access quickly, allowing national authorities to reach their national objectives faster. These findings correspond with the 2011 impact assessment and the 2018 mid-term evaluation. The following needs remain relevant since the 2011 impact assessment:

-the need to tackle the increased technological capacity of criminals involved in illicit activities (outlined in pink in the overview chart overleaf, Figure 16); 

-the need for up-to-date technical support (outlined in blue); 

-gaps in knowledge and the sharing of best practices (outlined in green); and, 

-problems in creating a harmonised approach across all Member States in the fight against fraud (outlined in yellow).

Figure 16 - Overview of the main needs and drivers - areas of continued relevance

The outline colours shown in the legend highlight for each evaluation where the theme was mentioned. Source: RAMBOLL Final Report on the evaluation of the Hercule III Programme.

Each of the top five most salient needs identified by stakeholders (shown in green) can be linked to past issues uncovered in the 2011 impact assessment and previous evaluations, thus highlighting the recurrent needs and problems (2011-2020).

Data collection and analysis tools - example of good practice 51

Two projects for the analysis of ‘big data’ that were initiated under the Hercule II programme continued beyond 2014 under Hercule III.

The automated monitoring tool (AMT) and container traffic (CONTRAFFIC, financed by the Commission through AFIS) are projects carried out by the European Commission’s Joint Research Centre under administrative arrangements with the Commission (OLAF). These two projects provide tools to support investigations that need data and analysis on container movements (CONTRAFFIC: current and past vessel/container movements) and EU import and export flows (AMT), providing more realistic and effective scenarios for joint customs operations. 52  

In 2020, the AMT tool was used to monitor and examine imports of medical supplies needed for the COVID-19 pandemic. It was also used to assess the findings from joint customs operations carried out in previous years, such as the operation to identify the issue of undervaluation in imports of textiles and footwear goods. 53

In 2020, OLAF and the JRC also continued with the administrative arrangement for the data analysis for customs anti-fraud project (INTEL4CUSTAF). Funded by the Hercule III programme, the administrative arrangement was initially set up at the end of 2017 54 between OLAF and the JRC to provide scientific and technical support in advancing the EU’s customs anti-fraud data analytics capacity 55 .

OLAF initiated the project following requests from Member State customs representatives, who recognised the potential of working together with new datasets and analytical methods to address the ever-changing world of customs fraud. The project scope is defined by activities under Council Regulation (EC) No 515/97 56 on mutual administrative assistance to ensure the correct application of customs legislation.

The project is part of a wider effort to provide knowledge and facilitate the work of customs authorities in Member States for the effective and efficient use of data and analytical approaches in the area of customs anti-fraud. In line with the objectives of the Hercule III programme, the main beneficiaries of this project are Member State customs authorities. The aim is to improve the overall effectiveness and efficiency of customs anti-fraud data analysis across the EU.

3.1.6.The future needs of stakeholders

The stakeholders interviewed provided several examples of future needs where the (new) programme could increase its relevance.

Figure 17 – Specific needs voiced by the stakeholders

The Regulation establishing the EU’s anti-fraud programme (UAFP) for the 2021-2027 period takes these needs into account by stressing that ‘the protection of the financial interests of the Union should address all aspects of the Union budget, on both the revenue and expenditure side’ (Regulation (EU) 2021/785, Recital 11). Expenditure fraud is also a special focus of the first implementing decision 57 adopted in 2021.

The evaluation also found that the programme should be more reactive and responsive (on a yearly basis) to key risk areas as they develop. In particular, interviews with Member State AFCOS, applicants and beneficiaries (from the case studies) suggested that the programme could play a rule in implementing or potentially developing action plans, including on emerging threats such as food fraud, (criminal or) unlawful use of drones and developments in cybercrime.

The need to adjust to new technological developments and challenges is prominent in the EU’s anti-fraud programme (Regulation (EU) 2021/785, Recital 20), and in the 2021 UAFP annual work programme.

Figure 18 – Key findings in terms of relevance

5.2.    Effectiveness

Assessing the programme’s effectiveness looks at the progress made towards achieving the set objectives, whether the intended outcomes have been achieved (or are likely to be achieved in the future) and to what extent the achievements are sustainable. At a broader level, the analysis assessed the achievements of the programme, in relation to the key activities of technical assistance, (legal) training, conferences, research activities and studies. To that effect, dividing the criterion of effectiveness into technical assistance and training facilitated the assessment.

3.1.7.Technical assistance

Article 8(a) of Regulation (EU) No 250/2014 provides that technical assistance consists of the provision of specialised technical assistance for Member State authorities. This includes providing specific knowledge, specialised and technically advanced equipment and effective information technology tools, providing the necessary support and facilitating investigations conducted by Member State authorities.

3.1.7.1. Support to law enforcement agencies

The evidence collected during the evaluation indicates that the programme continues to be successful in supporting national law enforcement agencies, in line with the results of the final evaluation of the previous Hercule II programme and the mid-term evaluation of the Hercule III programme.

Overall, there was broad consensus among the different stakeholders that technical assistance actions contributed to providing support to the law enforcement authorities of the Member States in their work to combat illegal cross-border activities. 

In the applicants’ survey, 52 out of 60 respondents agreed to a great or to a certain extent that the support provided by the programme contributed to supporting law enforcement agencies in their work to combat illegal cross-border activities. The interviewees confirmed that the technical assistance was effective in supporting law enforcement agencies.

The results of the desk-based assessment of achievements suggest that the programme helped increase both the operational and analytical capacity of law enforcement agencies. 

According to the analysis of the project reports, purchased equipment helped national law enforcement agencies carry out more operations, for instance by increasing the number of checks at border crossing points through scanners or by enabling them to operate in difficult or dangerous situations due to better and safer communication equipment (such as surveillance equipment or IMSI-catchers 58 ).

The technical assistance project reports also show that the projects enabled national law enforcement agencies to improve the quality of the evidence collected using more modern technologies. IT systems and software enabled law enforcement agencies to analyse a greater volume of data and thus reveal complex criminal activities. 

On the programme’s contribution to stimulate more staff exchanges for specific projects, the programme was perceived as making a limited contribution, with only 38% of beneficiaries answering that the programme had contributed towards this aspect. In-depth interviews conducted at EU level confirmed this finding and acknowledged that staff exchanges as part of technical assistance projects were not an area of focus.

However, respondents suggested that staff exchanges could have a greater impact as part of training rather than technical assistance projects. This is in line with evidence from the Hercule III mid-term evaluation and the final evaluation of the Hercule II programme, which uncovered limited evidence showing that joint operations and staff exchanges had increased and/or were effective.

Minimising the risks in the key areas of cigarette smuggling and counterfeiting

Evidence collected from an assessment of the achievements and case studies shows that technical assistance projects were successful in reducing the risk of cigarette smuggling and counterfeiting. The programme thus continued to focus on the key risk areas of cigarette smuggling and counterfeiting. 

In total, 19 technical assistance projects out of the 98 selected projects assessed were able to report on the quantities of seized cigarettes with the purchased equipment.

The project reports show that the following factors helped national law enforcement agencies reduce the risk of cigarette smuggling:

-modern fixed scanners enabled operators to scan more objects at border crossing points and produce higher quality imagery;

-mobile scanners and sniffer dogs enabled customs authorities to search vehicle parts that could not have been searched without the equipment;

-surveillance equipment helped law enforcement agencies collect evidence on large smuggling operations and illegal cigarette production facilities.

Interviewees from the JRC also highlighted the importance of the TOBLAB project 59  at the JRC in helping national law enforcement agencies collect additional evidence of cigarette smuggling patterns within Europe and worldwide. 

Overall, there is comparatively less evidence available on the effectiveness of technical assistance in reducing the risk of counterfeiting than of cigarette smuggling.

Only four out of 19 projects 60  reported on the quantities of counterfeited products seized. One reason could be the nature of the technical equipment purchased. Interviewees for one of the case studies for example pointed out that scanners are not able to detect counterfeit products in containers per se, as this would require a manual inspection by customs officers.

However, respondents agreed to a certain or great extent that technical assistance projects contributed both to reducing the risk of cigarette smuggling (45 out of 60) and reducing the risk of counterfeiting (40 out of 60). Ultimately, law enforcement officials may also not always see the need to establish whether the smuggled product they have seized is genuine or counterfeit, unless there are specific reasons to do so. 

3.1.7.2. Support to enhance internal and transnational cooperation

Technical assistance projects helped national law enforcement agencies cooperate better with other organisations and departments within their countries, but also with law enforcement agencies from other Member States.

According to evidence collected from the assessment of achievements, the purchase of forensic IT systems and communication technologies enabled law enforcement authorities to exchange intelligence information and evidence with other organisations within their countries and outside their countries.

Figure 19 – Case study example - Effectiveness of technical assistance projects in contributing to the programme’s specific and operational objectives

3.1.7.3. Positive external effects of technical assistance actions

Technical assistance projects were found to produce a number of positive external effects. Based on the assessment of achievements, the purchase of modern equipment generated a number of positive effects that are not among the programme’s primary objectives, but nevertheless benefit the national law enforcement agencies.

Equipment purchased and used to carry out border checks has proven useful to detect illegal drugs or illegal weapons. 10 out of the 19 final implementation reports analysed as part of the assessment of achievements reported seizures of drugs or illegal weapons.

Evidence from the case studies also indicates the usefulness of equipment in seizing illegal substances or artefacts that do not primarily undermine the EU’s financial interests but pose a more general security threat to EU Member States.

Evidence from the assessment of achievements and from case studies suggest that technical equipment increased the health and safety of customs and police officers.

The modernisation of scanners minimised the risk to radiation exposure. Modern communication and surveillance equipment helped officers monitor criminal activities while staying undetected at a safe distance.

Figure 20 – Case study example - external factors affecting the delivery of results and achievements

Figure 21 - Key findings regarding the effectiveness of technical assistance projects

3.1.8.Effectiveness of anti-fraud training and conferences

Contribution to improving prevention and investigation of fraud – Training

Under Article 8(b) of Regulation (EU) No 250/2014, a training project is described as the organisation of targeted specialised training, risk analysis training workshops, and where appropriate, conferences. This includes coordinating activities of participating countries and representatives of international organisations with the aim of exchanging experience and best practice between authorities and developing high-profile research activities and studies.

This section provides an overview of the main results achieved by beneficiaries implementing awarded grants for training purposes, throughout the programme duration, including projects ending in 2020 61 , which may have started in 2018 or 2019.

Training activities finalised in 2020 concerned a wide range of disciplines and topics such as: fighting transit fraud in the EU; advanced X-ray image interpretation training; methodologies of detecting and investigating crimes detrimental to EU financial interests; and enhancing the cooperation and control models in the field of prevention of fraud affecting the EU’s own resources. Between 2014 and 2020, about 10 000 participants followed specialised training and conference activities, co-financed by the programme through grants. The received feedback upon the end of the event resulted in a 93% participants’ satisfaction rate. About 1 340 law enforcement officials from all Member States participated in the specialised digital forensic and analyst training funded under the programme (2014-2020). A valuable network of forensic and analysis specialists was created in each Member State as an effect of such training possibilities.

As for the procured events, 45 high-level events organised by OLAF focused on the protection of the EU’s financial interests, including high-level meetings between customs and other regulatory or enforcement authorities, conferences with prosecutors dealing with fraud cases, meetings of the dedicated task group on combating illicit cigarette production and smuggling, seminars with the AFCOS members, etc.

The effects of training and conferences funded by the programme were largely found to be of a less tangible nature, as they rarely result in the reduction of fraud or increased seizures of illicit cigarettes, which can be directly attributed to a training event. Respondents that had participated in events funded under the programme highlighted the importance of these events for networking and meeting colleagues from other countries.

In the framework of the study supporting this evaluation, the stakeholders consulted by the external contractor found that training and conferences were a mixed success in enhancing multidisciplinary cooperation and cooperation with practitioners.

The majority of respondents to the survey of event participants did not agree, or agreed only to a limited extent, that the events improved cooperation between practitioners and academia. This may to a certain extent be explained by the participation rates. The vast majority of participants in events came from law enforcement agencies (inspection services, customs, police), the judiciary (prosecutors, judges and court staff) and national authorities (audit services, EU-related administrative services). Only a minority of participants (5% on average) came from academia (universities, study centres). Note, however, that events open only to the law enforcement community (sometimes due to the sensitivity of the matters discussed), cannot promote cooperation with academia.

Moreover, during the interviews of the study, representatives of academic institutions highlighted the programme’s success in enhancing cooperation between practitioners and academics, while interviewees from the EU institutions or Member State AFCOS were more critical of this aspect.

There is evidence that projects under the programme were effective at producing high-level research. 

A majority of respondents to the applicants’ survey agreed to a great or to a certain extent that the event they participated in or the support they received contributed to the development of high-profile research. Interviewees from academic institutions also reported that the programme funds were successful in creating a community of scholars that work in the field of EU criminal law and on protecting the EU’s financial interests.

One example of the success of the programme in developing high-profile research activities is the EUCRIM journal, which received a periodic grant during the funding period under review. For 2014–2020, EUCRIM published 27 special issues in total, each of which contained multiple academic articles and commentaries written by renowned scholars in the area of European criminal law.

Figure 22 – Key findings on the effectiveness of anti-fraud training and conferences

3.1.9.Sustainability or short-, mid- or long-term effects of the Programme

Sustainability of the programme’s interventions

The criterion of sustainability examines the extent to which the effects or changes caused by the programme could be permanent and to what degree their effects would last after project completion. Thus, the evaluation looks at the extent to which the long-term effects of the projects are viable and the degree to which they (may) have triggered any additional initiatives at national level.

In the case of technical assistance, evidence from the stakeholder consultations suggests that the sustainability of the programme is highly dependent on the nature and cost of the equipment purchased. Thus, as long as the equipment used in the fight against fraud is in use, effects can be prolonged. However, potential costs concerning software updates and general maintenance were raised in the interviews as factors, which hinder the sustainability of the programme’s effects, as in most cases these costs are not covered as part of the funding provided for one particular project.

Beneficiaries of technical assistance grants must mention the Hercule III programme support in their reporting and in any other document or press release that reports on the results. For example, press releases issued following seizures of counterfeit or smuggled cigarettes through funded equipment must mention the EU funding. Hence, the EU-funded investment by the beneficiary maintains its continued impact beyond the date of the end of the project, depending on the lifetime and use of the purchased material(s).

Technical assistance grants are awarded for the purchase and installation of technical equipment that, at the time of final reporting, were operational only for a short period. The first tangible results of the equipment, such as seizures, arrests of suspected persons or estimates of prevented losses to the national or EU budget, can only be reported months after the closing date of the action. For this reason, beneficiaries are required to submit a final implementation report one year after the closing date of the action to report on the results from the use of equipment.

Figure 23 – Case study example - long-term sustainability of projects

In terms of the provision of technical assistance, evidence from the stakeholder consultations suggests that the sustainability of the programme is highly dependent on the nature and cost of the equipment purchased. If the equipment used to combat fraud is in use, the effects can be sustained. However, potential costs of software updates and general maintenance were raised in the interviews as factors that could make the effects of the programme less sustainable, as in most cases the costs are not covered as part of the funding provided for that particular project.

The mid- to long-term effects of training and conferences funded by the programme were found to be largely of a less tangible nature, as they rarely result in reduction of fraud or increased seizures of illicit cigarettes, which can be directly attributed to such an event. Respondents, participating in events funded under the programme, highlighted the importance of such events as opportunities for networking and meeting colleagues from other countries.

According to the respondents, the informality of the events enabled participants to build a trustworthy working relationship, which is key for successful transnational cooperation. Participants of events echoed this in the event feedback forms, while the survey with participants of procured events provides further evidence that training and conferences have been successful in enabling participants to exchange experience and best practice.

While anti-fraud events and conferences were found to contribute to the development of professional networks, evidence on the sustainability of these effects over time is inconclusive. Evidence seems to suggest that in the aftermath of events, participants have gone on to share the acquired competences and knowledge with their peers at their workplace. This thus adds to the sustainability of the Programme through its ability to have an effect not just at the EU level but also at the national and regional levels.

Beneficiaries of grants for training must mention the support received from the Hercule III programme in every publication (title page) or related material (studies, booklets, newsletters, leaflets, etc.), in online information (websites, audio-visual material, videos, software, other), and at events (conferences, workshops and seminars). The mentions must include a link from their website to the Hercule III programme site.

In this way, the benefits and impact of the programme are magnified via communication channels, providing digital access to Hercule and enhancing recognition of the programme. The study found that the long-term effects of the programme are sustainable to a certain extent, given the limited durability of technical assistance projects.

Although the survey of the study found that events and conferences on anti-fraud contributed to the development of professional networks, evidence on how sustainable these effects are over time is inconclusive. Evidence seems to suggest that after the events, participants have gone on to share the acquired competences and knowledge with their peers at their workplace. This makes the programme more sustainable by generating an effect not just at EU level but also at the national and regional levels.

As for the extent to which the programme triggered additional initiatives at national level, the evidence is inconclusive as to whether the programme triggered similar initiatives at national level. The number of respondents who do not know whether the programme triggered initiatives at national level was significant in all three surveys. The lack of awareness or knowledge of comparable national initiatives was also reflected in the interviews where only one respondent mentioned a research project implemented at national level as a result of a discussion at a conference funded by the programme.

The above findings may be due to a combination of several reasons, for example the doubt that a beneficiary would no longer receive financial support from the programme once a similar initiative is set up with national funding or other sources of funding.

5.3.    Efficiency

The aim of assessing efficiency is to measure the relationship between the results and the resources invested in attaining the results. The assessment focused on identifying the benefits (cost savings, fraud reduction, etc.) related to the application process and implementation of funding projects, as well as the programme overall.

3.1.10.Main costs borne by stakeholders 

Evidence from the stakeholder consultations suggests that the main direct cost in both the application and implementation phases is staffing costs. Interestingly, the degree to which the costs were due to administrative costs and/or burden in both the application procedures and the reporting required to implement the contracts differed. This is consistent with the finding from the mid-term evaluation, which found that, on average, beneficiaries spent more time completing applications than on reporting results. To understand these differences, the following subsections will explore each part in turn.

Figure 24 – Case study example – Efficiency and cost effectiveness of projects

3.1.11.Application procedure

Evidence from the applicants’ survey found that, on average, applicants to the programme spend 15.3 full-time person days preparing and submitting an application. This is on average 2.7 days less than the time to prepare the mid-term evaluation, suggesting that the programme may have made some efficiency gains over 2014-2020. 

This decrease could be explained by the switch to the online platform ‘e‘-grants’ in 2017. However, the respondents and sample sizes vary between the mid-term evaluation and the applicants’ survey conducted for the purpose of this evaluation. Therefore, efficiency gains could purely reflect a different set of respondents.

Although there may have been a reduction in the number of full-time person days needed to complete the application procedure, 43% (53 out of 123) of respondents to the applicants’ survey answered that the administrative costs had increased. This compares to 30% (37 out of 123) who answered that the costs had not changed.

Evidence from the interviews suggests that a lack of dedicated staff in the given organisations that are knowledgeable of EU funds could in part explain the administrative costs incurred, with more time being spent understanding the application process and what was required for an application to be successful.

Most stakeholders were of the view that the application procedure could be further simplified. Across the interviews, survey and case studies, they suggested that the programme could provide more support during the application phase, particularly by fostering a better understanding of the award criteria and what constitutes best practice in preparing proposals. Furthermore, clearer instructions on the application procedure could help minimise common mistakes that are made in all applications, therefore helping to reduce delays and the overall time spent completing applications. Despite the costs incurred and scope to simplify the application procedure, most applicants were of the view that additional administrative and financial burdens did not discourage them from applying to a call for proposals under the programme.

There were mixed views, however, on the extent to which the procedures in the implementation phase could be simplified, with 42% of beneficiaries answering that it could be simplified, and 38% answering that it could not. The interviews also revealed these differences of opinion, where only a few interviewees could provide examples of ways to simplify the implementation procedure. Suggestions included minimising the need for final reports from beneficiaries and a specific example of using the multiannual administrative arrangements between OLAF and the JRC to better implement projects.

3.1.12.Direct and in-direct benefits experienced by stakeholders

The majority of the programme beneficiaries acknowledged that they experienced both direct and indirect benefits as a result of implementing the projects.

The main direct benefit highlighted relating to technical assistance projects was the simple benefit of acquiring technical assistance equipment and innovative tools and methods. Respondents noted that the equipment had resulted in higher detection rates and, by default, an increase in the number of seizures. This was similar to the findings under training, where the greatest benefit was in developing research and building knowledge among stakeholders.

Concerning the indirect benefits of the programme, the most salient answer was in strengthening collaboration and cooperation between and within Member State authorities, and stakeholders, and in exchanging best practice. This was reiterated in the interviews where beneficiaries reported that the main indirect benefits were in improving the daily work of law enforcement authorities through better knowledge and skills.

3.1.13.Relationship between the costs and benefits

Considering the costs and benefits incurred/accrued by stakeholders, most were of the view that the programme benefits outweighed the costs. For example, evidence from the applicants’ survey highlighted that 92% of beneficiaries agreed that the benefits generated outweigh the costs incurred during the contract.

In addition, several stakeholders highlighted that, for technical assistance projects, the value of goods seized often exceeded the initial value of the contract awarded by the programme. The case studies in Poland confirmed that the value of a single seizure covered the whole project investment cost. A few interviewees highlighted several contextual factors that can affect the relationship between costs and benefits.

A partner organisation highlighted that there can be differences in the use of funding under the programme to organise events and conferences between different stakeholders. For example, some organisations may not always have the experience of organising large-scale events compared to other organisations who may have more trained and experienced staff. Thus, in this situation it may not be fair to compare the use of budget between organisations in assessing the individual efficiency of each project.

Figure 25 – Main findings under efficiency

5.4.    Coherence

The assessment of internal and external coherence examines the extent to which aspects of the Hercule III programme worked well together and with other EU programmes and areas of action.

3.1.14.Internal coherence

The assessment of internal coherence examines whether the different projects funded under the programme are complementary, whether there are synergies between them or whether they overlap. The findings on internal coherence drew on three sources: desk research, feedback received in interviews and responses from the applicants’ survey.

The activities eligible for funding under the programme appear to be complementary. The evaluation did not identify any critical overlaps between the different activities funded by Hercule III. 

The review of the regulation establishing the Hercule III programme and its annual work programmes did not uncover any inconsistencies that would generate significant overlaps. Desk research was supplemented by opinions provided by multiple stakeholders to assess how the programme was implemented in practice. Although interviewees from EU institutions did not mention any current overlaps between programme activities, a quarter of participants to the applicants’ survey (27 out of 99) did. A few of the respondents who provided written feedback (6 out of 18) mentioned that conferences, seminars, and legal studies are somehow related. In fact, after a conference is organised, stakeholders often work together to prepare a study on the topics covered.

Therefore, survey respondents found that the distinction between these types of activities was not clear. However, it can be argued that rather than overlapping, these activities appear to be strongly complementary, in that one precedes and feeds into the other. Similarly, several respondents who provided written feedback highlighted that the purchase of equipment often requires training. This may result in an overlap between technical assistance and training activities.

The assessment did not reveal any significant contradictions between the types of activities funded. The desk-based assessment of coherence and the feedback collected through interviews did not highlight any contradictions.

Internal coherence between the types of activities of the Hercule III programme

Figure 26 provides an overview of the activities eligible for funding under each pillar of the Hercule III programme. Activities listed in row (i) under technical assistance and under training appear similar but they target different audiences and have different goals.

Training under the technical assistance pillar specifically focuses on providing specific information on the use of equipment funded under technical assistance as well as on the use and sharing of data collected by the equipment in transnational cooperation or with the Commission. The aim of the ‘general’ training is to raise awareness of the mechanisms available at national and EU level that help Member States combat fraud.

Overall, the activities eligible for funding under the Hercule III programme appear to be strongly complementary. Activities funded under training may help stakeholders implement technical assistance projects effectively in future. In turn, projects eligible under the third pillar may help fill any specific gaps arising from the first two pillars.

Figure 26 - Activities eligible for funding under the Hercule III programme

The actions eligible for funding appear to be coherent with one another. They have been implemented in line with the allocation of funds provided in the Annex to the Hercule III Regulation. Whilst respecting the set financial values, the number of actions funded were also implemented in a well-balanced manner: approximately 50% of the actions were technical assistance opportunities (about 80% financial value), while 48% of them were training activities (about 19% financial value).

Figure 27 – Finding on the coherence of eligible projects and their implementation

3.1.15.External coherence

The assessment of external coherence examines whether there are any complementarities, synergies or overlaps between the Hercule III programme and other EU legislation or programmes. Findings on external coherence are based on two sources: desk research and feedback received in the interviews with the external contractor during the study.

3.1.15.1.Regulation setting up the European Public Prosecutor’s Office

The desk-based assessment and EU interviews uncovered significant complementarities between Regulation (EU) 2017/1939 implementing enhanced cooperation on the establishment of the European Public Prosecutor’s Office 62  (‘EPPO Regulation’) and the Hercule III programme. The objective of the EPPO Regulation is to ‘enhance the fight against offences affecting the financial interests of the Union by setting up the EPPO’ (Recital 12 of Regulation (EU) No 2017/1939).

Regulation (EU) 2017/1939 specifies that the EPPO shall establish and maintain a close relationship with OLAF based on mutual cooperation. They act according to their respective mandates and avoid duplicating investigative activities.

In relation to the above, four interviewees mentioned that, while the core activities of the EPPO are related to the investigations and prosecutions of crimes affecting the Union’s financial interests, some specific EPPO’s initiatives, such as the training of national authorities on features and activities of the EPPO, are complementary to other schemes funded by the programme. Conferences funded by Hercule III were also organised to support the debates on the establishment of the EPPO. The desk-based review of the EPPO and Hercule III legal texts also highlights aspects of complementarity:

-both pursue the objective to protect the EU’s financial interests;

-both promote, albeit in very different contexts, collaboration between EU and non-EU bodies to facilitate and step up the fight against fraudulent activities.

There is a small potential risk that the training activities organised by the EPPO and the training activities funded by the programme (especially on legal training) may overlap, despite being seen as complementary. Since the EPPO started its operational activities on 1 June 2021, this potential overlap is limited to training projects, funded through the Hercule Programme under the 2020 budget, and to those, which would be implemented in 2021 and 2022. The training activities funded by the Hercule programme focus on protection of the EU’s financial interests by means of administrative law whereas training organised by the EPPO is dedicated to criminal investigations within their remit. The EPPO Regulation and the Hercule III Regulation have similar objectives but they contribute to their achievement in different ways.

3.1.15.2.Directive on the protection of fight against fraud to the EU’s financial interests by means of criminal law

The PIF Directive sets minimum rules concerning the definition of criminal offences and sanctions in the area of combating fraud and other illegal activities affecting the EU's financial interests. The PIF Directive seeks to strengthen the fight against criminal offences that undermine the EU’s financial interests.

There appears to be some complementarity between the PIF Directive and the Hercule III Regulation. The aim of the PIF Directive is to strengthen the fight against fraud and protect the EU’s financial interests, in line with the general objectives of the Hercule III programme. Both desk research and interviews with stakeholders from the EU institutions indicate that the two pieces of legislation differ quite substantially in terms of how they envisage achieving their goals.

The PIF Directive aims to harmonise the rules on the definitions and penalties in the area of crimes affecting the Union’s financial interests, which national authorities and the EPPO must apply when investigating and prosecuting such crimes. The PIF Directive also defines the EPPO’s material competence. The Hercule III Regulation, by contrast, combats fraud by providing funding for activities to conduct investigations on fraudulent activities, to seize and destroy fraudulent products and to learn about the modus operandi of criminal networks. To sum up, the PIF Directive, the EPPO and the Hercule III programme activities have similar objectives but pursue them in very different ways.

3.1.15.3.The Anti-fraud Information system (AFIS)

There is strong complementarity between action under the Hercule III programme and AFIS.

The anti-fraud information system (AFIS) is a web-based platform that was set up on the basis of Council Regulation 515/97 63 . AFIS offers several IT applications to users in organisations tasked with preventing, investigating, and prosecuting infringements of EU customs or agricultural legislation. OLAF operates the AFIS anti-fraud applications.

The general objective of AFIS is to protect the EU’s financial interest and taxpayers’ money. Its specific objective is to support mutual assistance in customs matters by providing secure information exchange tools for joint operations and specific customs anti-fraud information exchange modules and databases such as the Customs Information System. AFIS allows for a safe and efficient exchange of information. The Hercule III Regulation encourages such exchanges between different competent bodies. Interviewed stakeholders confirmed that AFIS is complementary to Hercule III, as it provides a useful IT tool to implement technical assistance projects funded by Hercule III.

One interviewee mentioned that AFIS (and Council Regulation (EC) No 515/97) are complementary to the Hercule III programme. Both legal bases have in the past allowed for funding of similar conferences run under both programmes.

The Commission decided to make better use of this complementarity and integrated in the EU’s new anti-fraud programme the Hercule component (spending part), the AFIS part and the Irregularity Management System (IMS); the latter two based on Regulation (EC) No 515/97. This will further increase complementarity and avoid any risk of overlap.

3.1.15.4.Regulation 1286/2013 establishing the Fiscalis 2020 Programme

Desk research also found complementarities between the Hercule III programme and the Fiscalis 2020 programme. Under Regulation 1286/2013 64 establishing the Fiscalis 2020 programme, the general objective of Fiscalis 2020 is to improve the proper functioning of taxation systems in the internal market by stepping up cooperation between participating countries, their tax authorities and their officials.

Several areas for action eligible for funding under the Fiscalis programme are similar in nature to those eligible under the Hercule III programme. This may be a source of potential, though limited, overlap. Two interviewees mentioned that training/workshops, working visits, administrative cooperation and expert skills development could have been funded under either the Hercule or the Fiscalis programmes.

Nevertheless, an in-depth analysis and comparison of the Hercule III and Fiscalis’ work programmes established that there was no overlap. The Hercule III programme did not fund activities primarily focusing on tax fraud, tax evasion, and aggressive tax planning. On the contrary, it can be argued that Hercule III was complementary to Fiscalis in that action to combat fraud using Hercule-funded equipment also contributed to combating tax fraud and evasion.

3.1.15.5.Regulation 1294/2013 establishing the Customs 2020 Programme

Desk research and interviews also found an area of potential overlap between the Hercule III programme and the Customs 2020 programme (2014-2020, repealed and replaced by the new Customs programme 65 ). However, both programmes are to be considered complementary to each other and their output should not mean duplication of the efforts and financial support. The participating Commission departments are preparing a future coordination to improve cooperation across the departments with a view to regularly review joint implementation issues, likely supported by an arrangement to avoid any potential double funding.

Under Regulation 1294/2013 66 , the general objective of the Customs 2020 Programme is to support the functioning and the modernisation of the customs union in order to strengthen the internal market. One of its specific objectives is to support customs authorities in protecting the financial and economic interests of the EU and of the Member States. In that sense, the joint actions pillar of Customs 2020 set up a specific grant for the customs eastern and south-eastern land border expert team (CELBET III 67 ), which supported the cooperation and sharing of expertise of the partner country customs experts, the coordination of data analysis and risk assessment efforts to increase the quality and efficiency of customs controls at the borders.

The technical assistance pillar of the Hercule III programme provides for the purchase and upgrading of tools to carry out inspections of containers, trucks, railway wagons and vehicles at the EU’s external borders and within the EU to detect smuggled and counterfeit goods.

Therefore, Hercule III provided support to applicants for grants from the CELBET customs authorities concerned to purchase equipment to combat smuggling or other criminal cross-border activities.

As such, there was complementarity between the two programmes reinforcing each other’s impact in the eastern and south-eastern EU Member States having access to both sources of funding.

The Hercule III programme funded the purchase of customs control equipment whereas the Customs programme did not. To avoid any potential future risk of overlap between the two programmes, Recital 17 of the Regulation establishing the EU anti-fraud programme (UAFP) clarifies that such potential overlapping of funding should be avoided.

In addition, the UAFP’s 2021 annual work programme clarifies that the UAFP will support the acquisition of types of equipment that do not fall under the scope of the new instrument for financial support for customs control equipment, as part of the Integrated Border Management Fund 68 .

Finally, as specified in the UAFP annual work programme, the Commission will look at the potential overlap while evaluating the coherence and added value of a proposal for the protection of the EU’s financial interests.

Figure 28 – Case study example (Malta) – External coherence

3.1.15.6.Regulation 1382/2013 establishing the Justice Programme

The objectives of the Hercule III and the Justice programme are inherently complementary. Under Article 3 of Regulation 1382/2013 69 establishing the Justice programme, the general objective of the programme is to contribute to the further development of a European area of justice based on mutual recognition and mutual trust, in particular by promoting judicial cooperation in civil and criminal matters. Article 12 of Regulation 1382/2013 states that the Commission must ensure overall consistency, complementarity and synergies with the work of EU bodies, offices and agencies operating in areas covered by the objectives of the programme. The Hercule III programme has the objective to prevent and combat fraud, corruption and any other illegal activities affecting the EU’s financial interests. The achievement of Hercule objectives strongly depends on a fully functioning EU judicial system.

Four stakeholders interviewed mentioned that training for the judiciary (prosecutors, judges and court staff) could be implemented and financed under either programme. To avoid such overlaps, the target groups of both programmes should be better defined. The Hercule III programme should (mainly) target public authorities and law enforcement agencies (inspection services, customs, police) and the Justice programme should (mainly) target training for public prosecutors, magistrates and court staff. In any event, it should be mentioned that the same costs should not be double or multiple funded, but training, conference or study activities can be implemented and funded by two or more (complementary) programmes. The Pericles programme and the structural reform support programme finance similar activities to the Hercule III programme. More synergies between both programmes could be achieved on counterfeiting.

3.1.15.7.Other EU Programmes

Figure 29 - Main findings under coherence (internal and external coherence)

5.5.    Added value

3.1.16.EU added value as identified through the studyEU added value as identified through the external study

The evaluation looked at evidence of EU added value at national level and the extent to which similar results could be achieved without EU action.

Figure 30 – Case study example – EU added value

Stakeholders consider that EU action taken under the programme provided significant added value, compared to what could have been achieved at national and regional level. 

The evaluation of the Hercule II programme and the mid-term evaluation of Hercule III confirmed that the programme did indeed provide added value. This evaluation supports the same finding. The majority (70/99) of respondents to the applicants’ survey disagreed that the same results could have been achieved at national or regional level. In particular, the programme was seen to provide two main aspects of additional value.

Firstly, it acted as a key driver and enabler for stakeholders to counter and reduce the risks of fraud harming the EU’s financial interests. 

Across each of the stakeholder consultation activities, many stakeholders (particularly Member State authorities) mentioned the role the programme had in providing the means to tackle the key areas of fraud in their countries. This was particularly the case for technical assistance where beneficiaries noted that it is often not feasible to make large investments in different types of technologies or equipment due to limits in the national budget available to them.

Interviewees from central and eastern European countries mainly cited a lack of available budget/ability for Member States to finance new equipment. This was also the case in the applicants’ survey where a greater number of respondents who answered that the same results could not have been achieved at national level were authorities in countries such as Poland, Bulgaria, Latvia, and also Italy.

The grants funded by the programme therefore made the prospects of such investments more probable, and in some cases, aided Member State authorities in making a case to their ministries to make the investments. This increased probability was also linked in some of the interviews to the speed at which the programme can provide support, compared to national level funding opportunities.

Several beneficiaries, as well as unsuccessful applicants, noted the additional credibility and reliability generated by being associated with the programme. In particular, being connected to the programme, and by default to activities carried out with OLAF, added increased legitimacy to the work carried out at national level.

Secondly, beneficiaries viewed the programme as a facilitator for EU-level cooperation between Member States, specifically by funding training, conferences, databases and tools.

More specifically, the main additional value found in terms of training (i.e. training activities, conferences and funded databases and tools) was the programme’s ability to enable EU-level groups and events to be created and to function. In particular, anti-fraud training and conferences were found to provide significant value in bringing countries together that may not have worked together previously.

3.1.17.Added value identified throughout the Programme’s implementation

For each of the three annual calls (for projects on technical assistance, legal training and training and conferences) under the Hercule annual work programmes, the applications received are assessed by an evaluation committee on the basis of four criteria (eligibility, exclusion, selection and award) and ranked with a score and award criteria. The latter include (ex ante identifiable) added value of the application in protecting the EU’s financial interests; conformity with the operational objectives of the programme; quality of the project; and value for money.

Evidence from the 2020 Hercule III annual work programme found that for technical assistance, the Commission encouraged projects that are carried out jointly by the programme's beneficiaries in multiple Member States in order to strengthen cross-border cooperation to combat activities that are detrimental to the EU's financial interests, such as smuggling tobacco, cigarettes, counterfeit goods or VAT-carousels.

While the evaluation found evidence that these joint operations took place, it is unclear whether there has been an increase since 2016. The EU’s new anti-fraud programme puts greater emphasis on joint action between Member States for technical assistance projects. Its 2021 annual work programme reflects that.

3.1.18. The Programme’s contribution to Commission’s prioritiesThe programme’s contribution to the Commission’s priorities

This evaluation also looked into the programme’s contribution to the Commission’s overarching priorities, in particular to its ‘Europe 2020’ strategy. In March 2010, the European Commission adopted the Europe 2020 strategy for smart, sustainable and inclusive growth 70 . Its objective was to enhance competitiveness within the EU between businesses and to protect their competitiveness from external unlawful competition.

Figure 31 – The Hercule III programme’s contribution to the Commission’s strategies

Though the Hercule III programme did not make a direct contribution to this strategy, it can be seen to have indirectly contributed. For example, following the logic that preventing and combating fraud, corruption and other illegal activities against EU financial interests helps secure EU resources that can be invested into other EU initiatives to promote smart, sustainable and inclusive growth, the programme can be seen as a crucial gatekeeper for action under the Europe 2020 strategy.

Figure 32 - Summary of key findings in terms of EU added value

6.    Conclusions and recommendations

The following section provides a summary of the conclusions with a breakdown by evaluation criterion (relevance, effectiveness (including sustainability), efficiency, coherence and EU added value). It also makes certain recommendations.

This section focuses only on the Hercule III programme, as it was set up for the previous multiannual budget period. It does not take into account further developments in EU policy, notably with the proposal and subsequent adoption in 2021 of the EU anti-fraud programme. Indeed, many of the conclusions and recommendations presented in this section and similarly in the consultant’s report are to a good extent already reflected in the UAFP. In that sense, this evaluation lends good support to the Commission’s current practice. That, however, is outside the scope of this technical evaluation of a past programme. It is to be developed further in the Commission’s Report to the European Parliament and the Council, which the present evaluation is accompanying.

Conclusion 1 – The programme is considered highly relevant, and must be continually adjusted to new developments in the fraud landscape, in anti-fraud tools and in the regulatory framework

The programme remains highly relevant in meeting the needs of stakeholders working on the front line to protect the EU’s financial interests. Evidence collected by the contractor through consultations with stakeholders showed that their needs are highly aligned with the objectives of the programme. This is particularly the case for operational objective 3 on strengthening the fight against fraud by providing technical and operational support to national authorities.

However, new, more recent developments in the field of expenditure fraud and cybercrime affect the continued relevance of the programme, drafted in 2011. This applies in particular to new developments in expenditure fraud, cybercrime, and the need for the programme itself to be(come) more reactive to emerging threats.

Stakeholders also highlighted the continuous need for greater awareness and publicity of EU funding opportunities under this and similar programmes.

-Recommendation 1.1: To maintain the programme’s relevance, the Commission should regularly review and adjust the focus of the programme where needed.

For an anti-fraud programme to remain relevant, it must continuously adapt to a fast-changing anti-fraud landscape. New trends in criminal activity arise, and fraudsters will always invent new techniques and use the opportunities of operating seamlessly across borders, and in cyberspace.

In seeking to address this ever-changing panorama, the Commission has drawn on multiple sources to develop an anti-fraud programme. Two main sources stand out: on the one hand, continuous dialogue with the stakeholders, notably those on the frontline of protecting the EU’s financial interests at European and national level. Several of these groups are already consulted during the preparatory phase to draw up each new work programme.

On the other hand, several Commission action plans (in areas such as customs 71 , work to combat tobacco smuggling 72 , or the Commission’s anti-fraud strategy 73 or the priorities agreed with the Member States in the framework of the European multidisciplinary platform against criminal threats (EMPACT) and similar policy documents), provide authoritative guidance on the evolving priorities of EU law enforcement. A continuous close alignment with these guidance documents will also increase the coherence of the anti-fraud programme with the EU’s wider action on the expenditure, revenue, home and justice domains.

-Recommendation 1.2: the Commission should continuously strive to promote to stakeholder groups the opportunities available under the programme, including its evolving focus.

The Commission (OLAF), implementing the programme, has already made good use of several channels to publicise opportunities for funding under the Hercule III programme. These range from the OLAF website via the AFCOS network to the Council and expert groups where all Member States are represented. In addition, all relevant information is always published on the Commission (OLAF) website and on the web portal that collects all Commission funding and tender opportunities.

Conclusion 2 – stakeholders perceive the programme’s effectiveness as very high, both in terms of technical assistance and training, though actual effectiveness has proved difficult to assess.

The stakeholders perceived the programme as highly effective across all three sectors of action. The programme was influential in supporting national authorities in investigations, creating national and international networks and working towards a strengthened EU level approach to combating fraud.

Technical assistance generated more direct impacts than training activities. This was particularly the case for the provision of technical and advanced equipment used by national authorities in the prevention and investigation of fraud or other illegal activities. Most notably, technical assistance grants generated specific knowledge facilitating transnational cooperation and cooperation with the Commission, as well as increasing both the operational and analytical capacity of law enforcement agencies across the EU.

Training activities were also perceived to be effective, but their effects were more indirect. The indirect effects were principally created through the development of networks facilitated through participation in training events and conferences.

Member States made comparatively little use of the scope for staff exchanges. Any administrative bottlenecks in this area would warrant analysis in more detail. A low level of take-up of such initiatives may also be indicative of low demand for such initiatives. It is possible that such initiatives, as welcome as they are in principle, in practice have a suboptimal cost-benefit ratio for participating administrations.

Notwithstanding these broadly positive findings, assessing the programme’s effectiveness through other (documentary) sources of evidence proved to be challenging. In particular, data collection via final technical and implementation reports, especially for technical assistance, was found to produce disparate types of data that could not be analysed easily. This was underlined in the economic analysis carried out as part of the 16 case studies, where it was not possible to fully determine the extent to which the programme had a measurable effect on reducing illicit trade and fraudulent activities. However, it is necessary to give a caveat, which is, that assessing the impact on illicit trade through any EU initiative is fraught with challenges and complexities due to the dynamic and unknown aspects of illicit activities that operate across the EU.

Conclusion 3 - the programme produces several positive external side effects.

The programme generated a number of positive external side effects, which further enhanced its effectiveness in protecting the EU’s financial interests. First and foremost, the acquisition of modern equipment generated a number of positive effects including the detection of additional illegal activities that were not originally covered under the programme. This included the detection of illegal drugs, weapons and artefacts. The detection of these substances or objects provides increased security to Member States and the EU, and has an impact that goes beyond the protection of financial interests (i.e. increased public safety, increased safety for users of the equipment and further hindering organised crime groups).

Conclusion 4 - the programme’s benefits outweigh the costs incurred, but more guidance and simplified processes could improve efficiency in the future, and help address challenges in terms of uneven administrative capacity in the Member States.

The stakeholders found the programme’s benefits outweigh the costs incurred. Though only indicative, the time taken to prepare applications was found to have decreased compared to the results of the 2018 mid-term evaluation, but the time taken to fulfil the reporting requirements increased slightly. Despite these conclusions, stakeholders confirmed that they were not deterred from applying to the programme, and did not face any significant administrative costs or red tape due to the reporting requirements. Thus, despite any costs incurred through the programme, the burden was largely found to be appropriate given the benefits it generated.

Overall, the main direct costs during both the application and implementation phases are staffing costs. On average, applicants to the programme spent 15.3 full-time person days preparing and applying, compared to an average of 14.4 full-time person days fulfilling the reporting requirements.

The programme was found to generate both direct and indirect benefits. Direct benefits included the possibility to acquire technical assistance equipment where Member States do not have available funding to invest in technologically advanced and expensive equipment. It therefore acts as an enabler for Member State authorities.

By contrast, training projects were found to generate more indirect benefits for stakeholders. These benefits can be seen to contribute to an EU level network of stakeholders, collectively working together to combat fraud, though measuring the impact of these network effects is challenging.

However, there are also indications that different levels of administrative capacity in some entities at national or regional level in the Member States may hamper the programme’s ability to deliver results and achieve its objectives. As a result, the argument was made by the consultant in the study that not all Member States were able to benefit equally from the programme due to national organisational circumstances.

-Recommendation 4: the application and reporting procedures in the programme should take into account the administrative capacity of Member States, and to this effect, the Commission should issue suitable guidance material. 

The Commission (OLAF) already makes available a number of documents to guide potential applicants on what is expected from them. This includes the annual work plan and the special calls for proposals. In addition, OLAF conducts a wide range of awareness raising initiatives by presenting the funding opportunities to networks, stakeholder groups and forums such as the GAF 74 meetings with the Council, the COCOLAF committee meetings, meetings among the AFCOS members, Customs 2020 expert teams, and reaching out to networks of lawyers and PIF organisations. All relevant information is always published on the Commission websites and on the electronic portal that collects all Commission funding and tender opportunities, so that it also reaches NGOs. The call notices are published in all EU languages in the EU Official Journal.

Lastly, and importantly, the Commission operates a dedicated email address (functional mailbox) at OLAF for questions or observations on the Hercule III programme, 24/7 accessible. Applicants can also contact OLAF via its website.

It appears feasible to go a step further in making more detailed guidance material available to all potential applicants. Making such information available upfront and to all potential applicants also levels the playing field to those who decide to apply.

The Commission could issue this guidance in various formats, including in the form of hypothetical questions and answers (Q&A). Another option would be to include several guidance questions directly in the corresponding fields of the application. The questions could help applicants understand the type of information requested, thus simplifying the drafting process for applicants and increasing the quality of the applications. The contractor provided suggestions for questions that could be included in the application for technical assistance.

At the same time, any further initiative to disseminate information about the programme and provide guidance to applicants should be proportionate to the Commission’s available resources. The Hercule III programme, like its successor, the UAFP, are among the smallest programmes managed by the Commission, and thus do not offer an economy of scale comparable to other programmes.

Conclusion 5 - the programme is both internally and externally coherent.

The analysis of the programme’s internal coherence did not uncover any major overlaps between the different activities identified.

With regard to external coherence, the analysis of the programme against other relevant EU policies and pieces of legislation (i.e. the PIF Directive, the EPPO Regulation, the Fiscalis and the Customs 2020 programmes, the AFIS platform and the Justice programme) uncovered no major overlaps or inconsistencies.

-Recommendation 5: the programme should maintain alignment and coherence with other relevant pieces of legislation. 

The Commission already systematically runs a series of consultations to ensure full alignment and coherence between the Hercule III programme (and likewise the EU’s new anti-fraud programme) and other EU instruments. This applies notably to links with other programmes in the customs or justice policy areas. These consultations should continue unabated, particularly since new programmes are currently being implemented at the beginning of a new financing period.

Conclusion 6 - the programme was found to have strong short to medium-term effects; evidence of long-term impacts was less tangible.

The sustainability of the programme’s effect was mainly achieved through the short to medium-term impacts generated by beneficiaries running projects or events funded by the programme. For technical assistance, whether the project produces long-term impacts is highly dependent on the nature and additional costs linked to the equipment or tools purchased. Thus, as long as the equipment is in use, the impact extends beyond the project lifetime. However, the potential costs of software updates and general maintenance were raised as factors that could hinder the sustainability/durability of the programme’s effects, as in most cases these costs are not covered by the project funding.

Training, however, provided greater indications of the programme achieving a long-term impact. Participants actively sharing the competences and knowledge gained during (training and other) events with their peers and colleagues in other Member States seemed to produce more long-term effects. However, measuring the exact extent of these effects over the longer term is difficult.

-Recommendation 6: for technical assistance, the programme could include specific costs under future calls for proposals for technical upgrades or software updates to secure the long-term impact of the investment. 

The Hercule III Regulation had no specific provision to fund equipment maintenance or updates of software, although in certain projects, the cost of maintenance in technical assistance was accepted for the duration of the grant agreement, as included in the applicant’s bid for funding.

Conclusion 7 – there is clear added value in taking EU action under the programme.

The programme is widely seen as providing EU added value. It does so by acting as an enabler and a driver for Member States to use a higher standard of equipment to detect and reduce the risk of fraud. It also acts as a facilitator for EU-level cooperation between Member States, specifically by funding training, conferences, databases and tools. The programme was also seen to provide greater legitimacy to the work carried out at national level through the additional credence by being associated with the programme and by default, with OLAF.

There is also a widespread belief that withdrawing the programme would have a negative impact on the fight against fraud at EU level, as it is likely to increase fragmentation in the action taken across the Member States.

Conclusion 8 – It was challenging to fully assess the true impact of the programme under the current monitoring and evaluation framework applicable to Hercule III.

One of the main difficulties in evaluating the programme is accurately assessing the direct impacts (i.e. the magnitude of impacts on supressing illicit and fraudulent activities). It remains crucial to improve the current monitoring and evaluation framework for future evaluations of the programme. This was emphasised not only in this study but also in the mid-term evaluation, which highlighted that the outcomes of technical assistance projects were limited and it was not possible to measure the key performance indicators listed in Article 4(a)-(c) of the Hercule III Regulation.

In the study, the contractor found that there is scope for the programme to develop and implement other indicators that could monitor the programme more closely. As the current performance framework provides top-level indicators, it would benefit from having indicators and judgement criteria set at the level of eligible actions. This is particularly important given the subtle variations found in the performance of the programme between different eligible actions (i.e. the direct and indirect effect differences between technical assistance and training) as part of this evaluation.

It is important to outline, however, that even with better data collection and monitoring, it would still be a challenge to gauge the true impact of the programme. Differences between national contexts and levels of illicit activity have a direct impact on the type of data that beneficiaries can provide. In addition, variations in the type of projects funded under the programme can create challenges in assessing the programme’s overall direct impact.

-Recommendation 8: the programme should try to improve the current monitoring and evaluation framework by developing more specific indicators set at the level of eligible actions.

It should be noted that for the EU’s new anti-fraud programme, like for all EU programmes, the Commission is fine-tuning an enhanced monitoring and evaluation framework developed for programmes run under the multiannual budget. Similarly, Article 12(4) of the UAFP Regulation (EU) 2021/785 provides a performance reporting system to ensure that data for monitoring implementation and evaluating the results of the programme are collected efficiently, effectively, and in a timely manner.

Conclusion 9 – creating a centralised database of projects implemented under the programme would provide a readily accessible evidence base for future evaluations.

The analysis of qualitative and quantitative data enabled this evaluation study to assess (in part) the effectiveness of the programme both at the level of individual projects and for the programme as a whole. The contractor did so by creating their own Excel database of a sample of projects and compiling data available in the consulted final technical, financial and implementation reports, as well as event feedback surveys and post-event surveys for training submitted by beneficiaries. The consultant’s database contained information on both training and technical assistance projects and actions. It enabled the contractor to formulate statements on key indicators of the effectiveness of the different actions and to uncover trends and changes to the achievements of funded projects.

However, given the scope of this study, only the data on a sample of 98 analysed projects were included in their database. It would aid future monitoring and evaluation work to have readily accessible key indicators on implemented projects, and uniform data formats that enable users to compare different projects.

-Recommendation 9: OLAF should centralise data collection and storage on implemented projects in a single database to facilitate access to project documents and overviews of results.

The Commission will look at appropriate and efficient ways to implement these recommendations.

------

Annex 1 - Procedural information

1.Lead DG, Decide planning reference

·Lead DG: OLAF, which has managed and implemented the Hercule III programme

·DeCIDE Planning reference: PLAN/2020/8105

2.Organisation and timing

In line with Article 13(2) of Regulation (EU) No 250/2014, which mandates an independent final evaluation, the European Commission (OLAF) used a DG TAXUD framework contract to take out a specific contract 75  with an external contractor to conduct a study to underpin this Commission evaluation. The external study was carried out by the Belgian subsidiary RAMBOLL Management S.A./N.V. 76 of the Danish mother company RAMBOLL Management Consulting A/S 77 , in a consortium with London Economics (UK) and Università Cattolica (IT) 78 . The study ran between October 2020 and October 2021.

·An inter-service steering group (ISG) of Commission departments 79 was set up to steer this evaluation process. The ISG reviewed the roadmap 80 , the terms of reference for the external contract, the inception phase of the study and the end-of-fieldwork report (quality assessment).

·ISG meetings were held with the contractor Ramboll:

-on the inception report – 7 December 2020

-on the interim evaluation report – 18 June 2021

-on the quality assessment of the draft final report – 13 September 2021.

3.Exceptions to the better regulation guidelines

The Commission (OLAF) obtained an exception that it did not have to run an open public consultation on the evaluation. Instead, it ran an open public consultation on the roadmap in June 2020 and targeted consultations of stakeholders in 2021.

4.Consultation of the RSB

Not applicable.

5.Evidence, sources and quality

The Commission (OLAF) made the following analysis of the assessment carried out by the contractor of the quality of the 49 sources of evidence consulted, for each evaluation (sub)questions to be answered per evaluation criterion:

Table 1 – Overview of the number of sources consulted and their quality assessment

Just under half of the sources consulted were noted as indicative but not conclusive evidence (23/49), followed by the more than satisfactory evidence (18/49). The effectiveness of the programme was substantiated by most sources (16). The consulted sources are previous evaluations and studies, key policy documents, legislative documents, annual reporting, academic literature and (other) studies.

However, both the criteria on internal coherence and sustainability (long-term effects of the programme) had only two sources of reference, presenting indicative but not conclusive evidence (the 2018 Hercule III mid-term evaluation and the 2014 PIF report for internal coherence; the Hercule II final evaluation and the Hercule III mid-term evaluation for the sustainability criterion).

The contractor RAMBOLL carried out a detailed and systematic review of Hercule III legal texts and policy documents to screen for any potential inconsistencies, overlaps, contradictions, or duplications across the programme’s action areas.

The findings on internal coherence were based on three sources: desk research, feedback received during interviews and responses to the applicants’ survey.

Annex 2 - Stakeholder consultation – synopsis report

This synopsis report on consultations is structured as follows:

1.Introduction (and exemption to the scope)

2.Stakeholder consultation activities – methodology and limitations

a.interviews - exploratory and in-depth interviews

b.case study interviews

c.targeted online surveys

3.Comparison of results from different consultation activities

4.Integration of consultation results in the answers to the evaluation questions

1. Introduction (and exemption to the scope)

This synopsis report presents the stakeholder consultation activities carried out as part of the final evaluation of the Hercule III programme. The purpose is to explain the stakeholder groups consulted as part of the study and the consultation methods, timings, questions asked, and the limitations faced in the process.

The aim of the report is to compare and contrast the results of the different consultation activities, including interdependencies, synergies and/or contradictions. This includes explaining how far the results were taken into consideration in the assessment of the evaluation questions provided in the final report.

OLAF requested an exemption to not conduct an open public consultation as part of this evaluation process, which was granted by the Commission’s Secretariat-General. Instead, it ran a consultation on the roadmap in June 2020 and targeted consultations of stakeholders in 2021 by the external contractor RAMBOLL in the course of the study. The motivation for that approach is that the primary beneficiaries of the Hercule III programme are national administrations and public servants. The programme’s main goal is to facilitate and enhance cooperation in tackling financial irregularities, fraud, corruption or other offences detrimental to the EU budget between national authorities within the European Union and with participating countries. What defines this enhanced cooperation are aspects such as financial support to facilitate and secure the exchange of information, training, exchange of staff, cooperation in the field, purchase of dedicated material and tools, availability of information, working visits, exchange of good practices, networking and many more. All of these affect almost exclusively the national administrations of the Member States. Consulting the national administrations and beneficiaries (participants to – training and other – events) on a variety of issues through targeted consultations is therefore the most important feature of the methodology of the current final evaluation.

Through the external contractor, performing an in-depth final evaluation of Hercule III, OLAF is seeking targeted feedback from three groups of stakeholders through dedicated online surveys to request and assess their opinion on the impact of the programme between 2014 and today (beneficiaries of grants, unsuccessful applicants, participants to procured events, users of procured services and databases financed by the programme).

2. Stakeholder consultation activities – methodology and approach

The stakeholder consultation process for this study included the following activities:

-interviews

oexploratory interviews and in-depth (semi-structured) interviews

ocase study interviews

-three targeted online surveys.

The following section summarises the approach taken at each stage of the stakeholder consultation.

a)Interviews - exploratory interviews and in-depth interviews

At the design phase of the study, four exploratory interviews with six stakeholders in OLAF were carried out, as displayed in Table 1. The aim of the exploratory interviews was to gather insights into the functioning of the Hercule III programme and gain a better understanding of the funded projects. The findings from these exploratory interviews fed directly into the revised evaluation questions matrix, the refined intervention logic and into the development of the method for the remaining stages of the project.

Table 1 - Stakeholders consulted in exploratory interviews

During the evidence collection phase, targeted semi-structured interviews were conducted to collect more details on why stakeholders hold certain opinions, and to collect additional evidence on evaluation questions for which qualitative data were judged to be an important source.

Overall, 25 semi-structured interviews were conducted with stakeholders in EU institutions and bodies, with partner organisations and with the AFCOS in the Member State authorities.  Table 2  gives a full overview of the stakeholders interviewed as part of this study.

Table 2 - Stakeholders consulted as part of the in-depth semi-structured interviews

The data collected through the interviews were summarised in individual interview reports after each interview. These were clearly labelled, and their structure was based on a template following the same format. The data collected was then coded and analysed using the QDA software NVivo®. This meant that each response from interviewees was assigned to the evaluation sub-questions and categorised by topic, providing insights into the questions.

This enabled the study team to give detailed answers to each evaluation sub-question for which the interviews provided evidence and report the results per stakeholder group. The following analysis provides answers to each evaluation sub-question. Where possible and relevant, the results are broken down by stakeholder group. Throughout the course of the interviews, several challenges were encountered, as listed in Table 3 overleaf.

Table 3 - Problems and solutions for the interviews

b)Case study interviews

Each case study included the collection of documentary evidence and interviews with stakeholders involved in operating the purchased equipment.

The case study interviews were targeted at the staff of the organisation that received Hercule III funding for projects. The study team intended to conduct two to three interviews per project in the organisations concerned. The study team was able in all cases to interview the project coordinator as the person in charge of managing the Hercule III-funded project, and they also conducted interviews with staff responsible for managing and operating the equipment purchased and trained on using it. Table 4 gives the exact number of interviews conducted per case study.

Table 4 - Stakeholders interviewed as part of the case studies

c)Online surveys

The evidence collection phase included sending three targeted survey questionnaires to three core stakeholder groups:

-applicants’ survey – to complement the assessment of the programme by gathering the views of representatives of authorities or departments who have applied to calls for proposals for technical assistance or training activities under the programme over the period 2014-2020 and whose applications were either successful or unsuccessful. OLAF provided contact details of applicants, in compliance with GDPR requirements.  

-participants’ survey – to target beneficiaries/people who attended events funded by the Hercule III programme over the period 2014-2020. This was used primarily to gather further insights on the programme, specifically on the training side;

-users’ survey – to target users of services procured by the programme (i.e. databases, statistics and IT tools and services to carry out chemical analysis of samples from tobacco and/or cigarette seizures) during the same period.

Both the applicants and the participants surveys were launched on 26 February 2021 and remained opened until 26 March 2021. The user survey was also launched on 26 February but remained open until 6 April 2021.

Table 5 provides an overview of the responses per survey type, as well as the number of respondents the different surveys aimed to reach.

Table 5 - Overview of the status of responses per survey type

The table below provides an overview of the number of responses per stakeholder type. It is important to note that the number of respondents is higher than that displayed in Table 5 as the design of the survey allowed respondents to select more than one option when choosing their organisation type.

Table 6 - Online survey respondents – Type of stakeholder

It was not possible to provide an accurate response rate for either the participants or the user survey as a snowball approach was used to distribute the surveys. For the participants’ survey, OLAF initially provided the contact details of organisers of events who distributed the survey among participants. For the user survey, OLAF contact points distributed the survey among beneficiaries of Hercule III services. In this case, it was also directly distributed among actual users of databases whose contact details were initially provided by OLAF. Table 7 notes the type of limitations encountered and the corresponding strategies taken to mitigate these issues.

Table 7 - Problems and limitations associated with the online surveys

3. Comparison of results from different consultation activities

The consultation activities targeted a wide range of stakeholders across multiple governance levels and geographic coverage. Within these groups, OLAF consulted a wide range of stakeholders via the different consultation activities. The following table gives an overview of the stakeholders reached during the different consultation methods.

Table 8 - Stakeholders reached through different consultation methods 84

Although the number of stakeholders consulted differs significantly across the different levels of governance, all the stakeholder groups identified at the design stage of the study were consulted.

The main stakeholder group targeted by the interviews was national law enforcement authorities, as they remain the main beneficiaries and final users of the funded equipment (case study interviews). It also included representatives of EU institutions such as OLAF and partner organisations who could provide insights into the functioning of Hercule III and its funded actions. For the surveys, the overview table shows a bias towards greater representation of national law enforcement authorities in the sample size compared to other groups. In terms of geographical coverage, overall, there is an even geographical distribution of respondents, although there was less representation from northern and central European countries in the applicants’ survey.

While it is impossible to consult a statistically representative sample of stakeholders for the EU as a whole, the study was designed to be as inclusive as possible. In addition, efforts were made to ensure geographical balance between interviewed stakeholders. To facilitate this, the study team conducted interviews in the language of the interviewees to further encourage participation.

To ensure the representativeness of all stakeholder groups, the stakeholder consultations were designed to be complimentary in terms of the topic areas covered. The topics and questions were tailored to each of the main stakeholder groups while ensuring a sufficient degree of overlap in terms of the evaluation questions. In particular, for the interviews, interview guides were developed per stakeholder group and the online surveys were sent to three different target groups (applicants, users and participants). Each of these areas included both sub-evaluation questions and top-level questions to give respondents the opportunity to express more general views when they do not have full knowledge of the topic area.

Triangulation of the data on the above activities revealed that the stakeholder views were largely consistent with similar trends emerging throughout and within stakeholder types. While some differing views were found in each of the consultation activities, they were taken into account and adequately represented in the analysis, factoring in any potential bias.

4. Integrating the consultation results in the answers to evaluation questions

Table 10 shows the sources of evidence for each evaluation sub-question. In most cases, the evidence was collected via at least two stakeholder consultation activities.

Table 9 - Ranking

Table 10 - Coverage of evaluation (sub)questions by different stakeholder consultation activities