EUROPEAN COMMISSION

Brussels, 14.5.2019

SWD(2019) 174 final

COMMISSION STAFF WORKING DOCUMENT

Evaluation of the European Monitoring Centre for Drugs and Drug Addiction (EMCDDA) 2018

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

Evaluation of the European Monitoring Centre for Drugs and Drug Addiction (EMCDDA) 2018

{COM(2019) 228 final}

TABLE OF CONTENTS

GLOSSARY    

I.    INTRODUCTION    

A.    PURPOSE OF THE EVALUATION    

B.    SCOPE OF THE EVALUATION    

II.    BACKGROUND TO THE INITIATIVE    

A.    DESCRIPTION OF THE INTERVENTION AND ITS OBJECTIVES    

B.    BASELINE AND INTERVENTION LOGIC    

III.    IMPLEMENTATION / STATE OF PLAY    

IV.    EVALUATION QUESTIONS    

V.    METHOD    

A.    SHORT DESCRIPTION OF METHODOLOGY    

B.    LIMITATIONS AND ROBUSTNESS OF FINDINGS    

VI.    ANSWERS TO THE EVALUATION QUESTIONS    

A.    RELEVANCE    

B.    EFFECTIVENESS    

C.    EFFICIENCY    

D.    COHERENCE    

E.    EU ADDED VALUE    

VII.    CONCLUSIONS AND LESSONS LEARNED    

VIII.    ANNEXES    

ANNEX I – PROCEDURAL INFORMATION    

ANNEX II – METHODS, SOURCES AND LIMITATIONS    

ANNEX III – STAKEHOLDER CONSULTATION    

ANNEX IV – INTERVENTION LOGIC    

ANNEX V – SUMMARY OF THE MAIN AREAS AND GOALS IN THE 2013-2015 AND 2016-2018 STRATEGIES AND WORK PROGRAMMES    

GLOSSARY

I.    INTRODUCTION

A.    PURPOSE OF THE EVALUATION

The European Monitoring Centre for Drugs and Drugs Addiction (EMCDDA; referred to in this document as “the Agency”) was established in 1993 with the mission to provide the EU and its Member States with factual, objective, reliable and comparable information at European level on drugs and drug addiction and their consequences. Its core tasks are to collect and analyse existing data, improve data comparison methods, disseminate data and to cooperate with European and international bodies and organisations as well as with third countries.

Regulation (EC) 1920/2006 1 (referred to as “founding Regulation”), together with the Financial Framework Regulation 2 , the Common Approach on Decentralised Agencies 3 and the principles of sound and efficient management require that the Agency be evaluated on a regular basis. Article 23 of the founding Regulation specifically stipulates that the Commission should initiate an external evaluation every six years to coincide with the completion of two of the Agency’s three-year work programmes. The evaluation has to include the Reitox system (the European Information Network on Drugs and Drug Addiction). 4 The evaluation report has to be transmitted to the European Parliament, the Council and the Agency’s Management Board.

The previous external evaluation covered the period 2007-2012 5 and was completed in mid-2012 so that its findings would be available ahead of the following multiannual work programme of the Agency. The current six-year programming phase covers the last two multiannual work programmes, namely 2013-2015 and 2016-2018, and the findings of the evaluation will feed the Agency's future multiannual and annual work programmes and will contribute to the shaping of the future EU Drugs Strategy (the current one ends in 2020).

The evaluation has two main objectives:

-to assess the relevance, effectiveness, efficiency, coherence and EU added value of the Agency's performance since 2013 in terms of the implementation of the EU Drugs Strategy (2013-2020) 6 and its Action Plans 7 , of the European Agenda on Security 8 and of its tasks laid down in its founding Regulation (Article 2); and

-to propose concrete and useful recommendations to the Agency to better respond to the challenges posed by the constantly changing environment against the background of the current authorised financial and human resources for the Agency, bearing in mind the current and future EU budgetary constraints.

The evaluation assesses also the compliance of the Agency with the objectives and provisions of the Common Approach on Decentralised Agencies. 9

B.    SCOPE OF THE EVALUATION

As foreseen in Article 23 of the founding Regulation and based on the Terms of Reference for the external evaluation, the contractors assessed in the evaluation study:

-the implementation of the last two three-year work programmes, more precisely from January 2013 until as late as possible in 2018; in this context, the evaluation examined in particular the way and extent to which the Agency effectively contributed to the implementation of the strategic documents referred to above and fulfilled its tasks laid down in its founding Regulation;

-the European Information Network on Drugs and Drug Addiction (Reitox), whose members are national institutions or agencies responsible for data collection and reporting on drugs and drug addiction, in particular to what extent the network contributed to the overall performance of the Agency; and

-whether the provisions and the scope of the founding Regulation are still adapted to current needs given the evolved context in which the Agency now operates and the new challenges it has to face.

The evaluation also had to verify to what extent the provisions establishing the Agency are aligned with the Common Approach on Decentralised Agencies, looking in particular into the Agency's governance, internal structures and procedures and applicable administrative and financial rules. Another intention of the evaluation was to provide solid grounds for any possible decision concerning the Agency’s future mandate.

Based on these principles, the external evaluation study assessed the five evaluation criteria (relevance, effectiveness, efficiency, coherence and EU added value) by taking into account the following scope:

-Material scope: the evaluation covers the two pillars of the Agency’s work, i.e. health and security. It looks into its governance and administration, the organisational structure, operations, funding and resourcing, its information management and the work of the Reitox network, cooperation with other relevant EU agencies (such as the Justice and Home Affairs agencies 10 , the European Medicines Agency (EMA), the European Centre for Disease Prevention and Control (ECDC)) and international organisations (such as the World Health Organisation (WHO), the United Nations Office on Drugs and Crime (UNODC), etc.), partnerships with third countries, and communication and dissemination of research;

-Geographical scope: the evaluation covers the countries reporting data to the Agency, i.e. the EU Member States as well as Norway and Turkey, and third countries with which the Agency has closer relations due to the conclusion of working arrangements, cooperation agreements or similar;

-Temporal scope: the evaluation covers the activities carried out from 1 January 2013 until 30 June 2018 11 to coincide with the completion of two consecutive 3-year strategies and work programmes of the Agency.

The evaluation was the fourth evaluation of the Agency. The current evaluation built on the results of the previous external evaluation.

II.    BACKGROUND TO THE INITIATIVE

A.    DESCRIPTION OF THE INTERVENTION AND ITS OBJECTIVES

The market for illicit drugs is the most dynamic criminal market in the EU, which is able to adapt rapidly in response to drug control measures. Its socio-economic, demographic and international context has evolved considerably over the last years. The human and social costs of drug addiction are very high and it generates costs for public health (drug prevention, healthcare and treatment), public safety, the environment and labour productivity.

The drugs market value is estimated at EUR 24 billion 12 and is facilitated by the emergence of new internet technologies. Online marketplaces have emerged and are spreading. The number of New Psychoactive Substances is on the rise. An estimate of at least 7929 overdose deaths happened in 2016. There has been a shift from drug use to poly-drug use (where people use two or more psychoactive drugs or alcohol in combination) and on-going debates and shifting policies across the world regarding medical and/or recreational use of cannabis. The issue of misuse of prescription drugs is becoming more and more important also in Europe. Purity and availability on the market of conventional drugs (such as cocaine and heroin) are increasing. 13 In addition, national drug policies developed further over time and many Member States' new strategies go beyond only covering illicit drugs to cover also other addictions, such as alcohol, tobacco, gambling, etc. 14

In order to be able to understand better the nature of Europe’s drug problem and to respond better to them, the European Monitoring Centre for Drugs and Drug Addiction (EMCDDA) was established in 1993 15 as a decentralised agency to provide independent, science-based information on these issues.

Its objective, pursuant to Article 1 of the founding Regulation, is "to provide in the areas referred to in Article 3, the Community and its Member States with factual, objective, reliable and comparable information at European level concerning drugs and drug addiction and their consequences". Annex I, to which Article 3 of the founding Regulation refers, reads as follows:

"A.    The work of the Centre shall be carried out with due regard to the respective powers of the Community and its Member States in the area of drugs, as those powers are defined by the Treaty. It shall cover the various facets of the drugs and drug addiction phenomenon, and the solutions applied. In doing so, the Centre shall be guided by the Drugs Strategies and Action Plans adopted by the European Union.

The Centre shall focus on the following priority areas:

1)    monitoring the state of the drugs problem, in particular using epidemiological or other indicators, and monitoring emerging trends, in particular those involving poly-drug use;

2)    monitoring the solutions applied to drug-related problems; providing information on best practices in the Member States and facilitating the exchange of such practices among them;

3)    assessing the risks of new psychoactive substances and maintaining a rapid information system with regard to their use and also regarding new methods of using existing psychoactive substances;

4)    developing tools and instruments to help Member States to monitor and evaluate their national policies and the Commission to monitor and evaluate European Union policies.

B.    The Commission shall make available to the Centre, for dissemination, the information and statistical data which it possesses pursuant to its powers."

Based on the objectives, tasks and priority areas of work set out in the founding Regulation, the multi-annual strategies and work programmes detail the main areas, goals and expected results of the work of the Agency further. For the period relevant for this evaluation, these are set out in particular in the “EMCDDA 2013-15 strategy” 16 , the “Strategy and work programme 2016-18” 17 and the “EMCDDA Strategy 2025” 18 , with further details included in the annual work programmes 19 . These documents formed and form the basis of the Agency’s operating framework over the evaluation period.

B.    BASELINE AND INTERVENTION LOGIC 

When the Agency was established, no detailed intervention logic was prepared. It was developed for this evaluation by the contractor, also taking into account the intervention logic prepared for the previous external evaluation. A high-level overview of the intervention logic for this evaluation is included as Annex IV in this Staff Working Document.

The high-level principle of the Agency is the aim to contribute to a healthier and more secure Europe and to meet the information needs of policy-makers and practitioners at EU, national and international level (general objective). As the Agency is subject to a complex hierarchy of specific objectives, based on the founding Regulation and the different strategic documents, the intervention logic illustrates the core objectives and priorities stemming from these documents.

The Agency’s inputs are its financial and human resources, while its core tasks are set out in the founding Regulation.

The following three levels of expected results can be identified:

-Outputs are the immediate products and services. These include the different periodic reports, i.e. annual and multiannual reports, such as the European Drug Report (EDR), the EU Drugs Market Report (EDMR) and the European Drug Responses Report (EDRR); strategic and situational analysis and threat assessment analysis; statistics, Country Drug Reports, thematic and technical reports and other written outputs; training and capacity building, etc.

-Outcomes are second-level intermediate results stemming from the outputs. These include facilitated information exchange, better informed policy-makers and practitioners, transferring know-how and improved cooperation and coherence with partners.

-Impacts are third-level long-term results, which are influenced by factors outside the control of the Agency. Its activities ultimately aim to support the objectives of the EU drugs policy, including the EU Drugs Strategy and its Action Plans, in reducing both demand and supply of drugs and thus contribute to a reduction in drug use and minimising the threat of this phenomenon.

The general baseline for the establishment of the Agency was that no factual, objective, reliable and comparable information at European level on drugs and drug addiction and their consequences was available. No concrete or detailed baseline was established at that time. The contractors tried to re-establish baselines by using – as much as possible – the results of the previous external evaluation. However, this was possible only to a limited extent as the baselines were lacking for most elements. 20  

III.    IMPLEMENTATION / STATE OF PLAY

The European Monitoring Centre for Drugs and Drug Addiction (EMCDDA) was set up in 1993 as one of the EU’s decentralised agencies. It started working fully in 1995 and is located in Lisbon, Portugal.

Structure and management

The Agency’s organisation consists of a Directorate, two statutory bodies (the Management Board and the Scientific Committee) to advise and assist in the decision making process, and various working units (which comprises the majority of the staff) to deliver its output.

The Agency has approximately 100 staff members (state of play June 2018) and is managed by a Director who is appointed for a 5-year period by the Management Board based on a proposal by the European Commission. 21 The Director is ultimately responsible for meeting the strategic objectives and implementing the Work Programmes, budget frameworks and Management Board decisions. He is accountable for his activities to the Management Board.

The internal organisation looks currently as follows 22 :

The Management Board is the main decision-making body. 23 It meets at least once a year 24 and consists of one representative from each EU Member State, Norway and Turkey, two representatives from the European Commission and two independent experts particularly knowledgeable in the field of drugs designated by the European Parliament. 25 Several observers are invited to participate in the Management Board meetings. The Chairperson and the Vice-Chairperson of the Management Board are elected for a (once renewable) three-year period.

An Executive Committee assists the Management Board. 26 The Executive Committee consists of the Chairperson and the Vice-Chairperson of the Management Board, two other members of the Management Board representing the Member States and appointed by the Management Board and two Commission representatives. The Executive Committee prepares the Management Board meetings and decides on those matters provided for in the Financial Framework Regulation which are not reserved to the Management Board by the founding Regulation. Although not provided for in the founding Regulation, the Agency also set up a Budget Committee to ensure better monitoring and follow-up of financial issues.

The Scientific Committee 27 advises the Board and the Director on the scientific aspects of the work and contributes to the risk assessment of new psychoactive substances (NPS). The Scientific Committee consists of at most fifteen well-known scientists appointed in view of their scientific excellence and their independence by the Management Board, following the publication of a call for expression of interest in the Official Journal of the European Union. The members of the Scientific Committee are appointed in their personal capacity.

Reitox 28 is the European information network on drugs and drug addiction. Members of the Reitox network are designated national institutions or agencies responsible for data collection and reporting on drugs and drug addiction (‘national focal points’ or ‘national drug observatories’). The Reitox network links national drug information systems and is the main way in which the Agency exchanges data and methodological information on drugs and drug addiction in Europe and thereby directly contributes to the core task of collecting and reporting consistent, harmonised and standardised information. The three core functions of a national focal point are: data collection and monitoring; analysis and interpretation of data collected; and reporting and dissemination of the results at national level.

Budget

The annual budget over the evaluation period was around EUR 16 Mio. The main part of the budget comes from the European Union subsidy. Norway and Turkey, both members without voting rights 29 , also contribute to its budget.

Main areas of work

The key areas of activity of the Agency are in accordance with Article 2 of its founding Regulation: collection and analysis of existing data; improvement of data comparison methods; dissemination of data; cooperation with European and international bodies and organisations as well as with third countries; information obligations; and – as added by the 2017 amendment to the founding Regulation 30 – exchange of information on, early warning system for, and risk assessment of, new psychoactive substances.

During the evaluation period, the Agency published two 3-year strategies and work programmes: the “EMCDDA 2013-15 strategy” and the “Strategy and work programme 2016-18”. For the first time, a 10-year strategy was also published, the “EMCDDA Strategy 2025”. These strategic goals were transformed into more concreted (planned) outputs in the annual work programmes. They provided detailed information about the activities and outputs of the Agency. The coherence of the main areas of work and goals set out in these documents is presented below 31 .

The 2013-2015 Strategy identified 12 main areas, each with a goal and further specific objectives and ‘primary interventions’. The 2016-2018 Strategy and Work Programme was built around six strategic action areas, composed of three key action areas: communicating evidence and knowledge exchange; early warning and threat assessment; and situation, responses and trend analysis; three cross-cutting action areas: information collection and management; quality assurance; and cooperation with partners; and two corporate areas: governance, and administration and ICT. The main areas and goals are presented in more detail in a table in Annex V.

The “EMCDDA Strategy 2025” attributed the activities and the tasks of the Agency to two main pillars, health and security. In addition, it defined four business drivers, i.e. institutional, partnership, scientific capacity and management. The strategic objectives of the two pillars reflect the priority areas set out by the founding Regulation.

While the Agency is primarily European in focus, it also works with partners in other world regions, exchanging information and expertise. The Agency cooperates with candidate and potential candidate countries as part of their accession process to the EU. When it comes to neighbouring countries of the European Neighbourhood Policy and with other non-EU countries, cooperation takes place based on bilateral agreements 32 or in the context of EU-funded projects 33 . Cooperation ranges from implementation of technical assistance projects to ad-hoc training or consultative support. Collaboration with European and international organisations in the drugs field is central to its work as a means of enhancing understanding of the global drugs phenomenon.

Main achievements

The achievements of the Agency are set out in detail and on an annual basis in the General Report of Activities. 34 Some key achievements of the Agency in relation to the priority areas set out in the founding Regulation are summarised below. This is only a small snapshot of the work of the Agency and more detailed outputs and achievements are presented throughout this Staff Working Document.

The main achievement of the establishment of the Agency is the availability of factual, objective, reliable and comparable information at European level concerning drugs and drug addiction. This is crucial to allow the development of an integrated, balanced and evidence-based approach to drugs policy.

Some examples of main achievements: When it comes to the monitoring of the state of the drugs problem, the solutions applied to drug-related problems as well as of European and national policies, the Agency has published almost 400 scientific and institutional reports from 2013 to 2017. Among its flagship publications are the annual European Drug Report, including the country reports, the EU Drug Markets Report, published in 2013 and 2016, and “Health and social responses to drug problems – A European Guide”, which was published for the first time in 2017 but will be updated every 3 years. 35 The Agency facilitated the exchange of best practices and drug-related information through the participation of its staff in more than 1500 events (contributions to conferences, policy forums, etc.) and authoring more than 150 scientific articles during the same period. The Agency organised numerous meetings of its stakeholders, mainly in Lisbon, which also allowed for the sharing of best practice. When it comes to new psychoactive substances, the maintenance of the Early Warning System, which is available 24/7, is crucial for the assessment of their risks. The Agency carried out 26 joint reports and risk assessments from 2013 to 2017.

IV.    EVALUATION QUESTIONS

In accordance with the Commission's Better Regulation Guidelines, the evaluation looked at the effectiveness, efficiency, relevance, coherence and the EU added value of the work of the European Monitoring Centre for Drugs and Drug Addiction.

The following questions were addressed during the evaluation: 36

Relevance

1.To what extent have the EMCDDA work programmes covering the 2013-2018 period addressed the objectives, tasks and priorities set out in the EMCDDA recast regulation as well as those of the EU Drugs Strategy and its Action Plans and those stemming from the European Agenda on Security?

2.To what extent have the outcomes of the EMCDDA work programmes covering the 2013-2018 period responded to the needs of its multiple stakeholders (policy-makers, scientific community, practitioners and the general public)?

3.To what extent are the EMCDDA activities contributing to the EU's priorities?

4.How well adapted is the EMCDDA to subsequent economic, technological, scientific, social, political or environmental advances?

Effectiveness

5.To what extent has the EMCDDA met its core objective as required in its regulatory framework to provide the EU with factual, objective, reliable and comparable information?

6.To what extent has the EMCDDA achieved the objectives of its two three year work programmes 2013-2015 and 2016-2018?

7.To what extent have the Reitox national focal points delivered the data and information required to meet the objectives of the aforementioned EMCDDA work programmes?

8.To what extent have the changes in the EMCDDA governance structure resulting from the implementation of the EMCDDA Strategy 2025 and the recent internal re-organisation impacted on the effectiveness of the EMCDDA?

9.To what extent are the EMCDDA tools to monitor and review its outputs and results adequate for ensuring accountability and an appropriate assessment of performance?

10.To what extent are the internal and external mechanisms for monitoring, reporting and evaluating the agency adequate for ensuring accountability and appropriate assessment of the overall performance?

11.To what extent and how have external factors influenced the effectiveness of the agency?

Efficiency

12.Which were the costs and the benefits of the EMCDDA activities?

13.To what extent has the EMCDDA efficiently spent its resources (human and financial) to achieve the objectives set out in its work programmes during the 2013-2018 period? To what extent are available resources adequate to these objectives?

14.Is the EMCDDA providing value for money?

15.To what extent have the EMCDDA governance, organisational set-up, management systems and working methods been conducive to the efficiency of its operations?

16.Is there scope for simplifying the administrative set-up and working methods in the context of the current administrative and financial regulations?

17.To what extent and how have external factors influenced the efficiency of the agency?

Coherence

18.To what extent are the objectives and activities of the EMCDDA for the 2013-2018 period coherent with the objectives set out in its regulatory framework?

19.To what extent are the objectives and activities of the EMCDDA coherent with the EU policy developments as defined in the documents presented in "EU policy documents" section?

20.To what extent are the objectives and activities of the EMCDDA coherent with and complementary to the drugs-related objectives and activities of the Commission, of other EU agencies, namely Europol, European Centre for the Prevention of Disease Control and the European Medicines Agency, and of the Member States? What are the overlaps and potential synergies and the option of merging with other EU agencies carrying out similar or identical activities?

EU Added Value

21.To what extent have the EMCDDA activities and outputs helped to improve the ability of Member States to monitor and respond to drug problems compared to what they could do at national level?

22.To what extent have the EMCDDA activities provided a European level information resource for informing the policy debate on drug issues?

23.To what extent has the EMCDDA been more effective in achieving its objectives in the 2013-2018 period compared to existing or alternative options of implementing drugs policy as set in the strategic documents referred to under point 2 (e.g. by the Member States, through the Commission services themselves, an executive agency, etc.)?

24.To what extent are the outcomes of the EMCDDA activities sustainable?

25.What would be the most likely consequences of the termination of the agency?

Lessons learnt 37

The evaluation also assessed how the EMCDDA could assist policymakers and practitioners even more in planning and delivering policies and programs that contribute to a healthier and more secure Europe, in an evolving political, socio-economic, legal, demographic international context. In particular, the following questions were addressed during the evaluation:

26.On the health aspects, to what extent can the scope of monitoring and identifying best practices be broadened in the field of prevention and care of addictive behaviours (illicit and licit substances including alcohol, but also addictions without substances, like gambling)?

27.In terms of synergies with other JHA agencies, to what extent and how can the agency be more to the forefront of the security issues and inform more on threats and trends concerning links between drugs trafficking and organised crime, drug-related crime, drugs markets, in particular the way drugs can be sold and bought through Internet, social networks and encryption software? To what extent and how synergies with other JHA agencies and in particular with Europol can be increased in this field?

28.How could the EMCDDA work better with International organisation, such as the World Health Organisation and the United Nations Office on Drugs and Crime (UNODC)? At UN level, as the EU strongly advocates for evidence-based policies and has the most performant regional data collection system, to what extent and how can EMCDDA promote its expertise in the revision process of the UN data collection system, including the possibility for the EMCDDA to respond for the EU to certain UN questionnaires and enquiries?

29.At international level, to what extent can the international activities of the EMCDDA be optimised in terms of impact? Are the activities carried out by EMCDDA outside the EU compatible with the EU priorities in the external action? Should EMCDDA do more or less in the international sphere?

30.In terms of communication/dissemination, to what extent and how can EMCDDA spread more widely its outputs, thus reaching out a higher number of policy-makers and professionals?

31.What conclusions and recommendations can be drawn from the evaluation of the EMCDDA and its work programmes relating to the 2013-2018 period, with a view to supporting the post-2020 EMCDDA programming cycles, future Commission's proposals for EU drugs policy initiatives and the possible revision of the EMCDDA mandate?

V.    METHOD

A.    SHORT DESCRIPTION OF METHODOLOGY

The external evaluation study was carried out by ICF Consulting Services Limited (ICF) in cooperation with the Centre for the Study of Democracy (CSD) and Optimity Advisors, following a call for services under a DG HOME framework contract. The evaluation was conducted through a mixed methods approach and was informed by the triangulation of a variety of sources. For more details on the methods and the stakeholder consultation, see Annexes II and III of this Staff Working Document.

The evaluation was informed by a thorough desk research of documentation being available online or provided directly (by e-mail and/or on a dedicated extranet site on the Agency’s website to which only the contractor and the responsible DG HOME desk officer had access). 38  

The external evaluators had interviews with 133 stakeholders 39 , which were accompanied – where appropriate – by questionnaires.

Targeted online surveys took place for the Agency staff, on the one hand, and the scientific community and civil society, on the other hand. 55 and 24 replies, respectively, were gathered through the two surveys.

A panel of experts undertook a peer review of selected publications; the results of the peer review were integrated in the final report and informed the replies to the evaluation questions.

Finally, the contractor untertook five case studies, which provided input to the evaluation and were presented as such in the final report.

The preliminary results and conclusions of the evaluation were discussed during a workshop on 6 September 2018 in Brussels.

In addition to the targeted consultations, the Commission organised an internet-based public consultation 40 . 147 contributions were submitted through the online questionnaire during the consultation period from 18 May to 30 August 2018. The contributions were published on the Commission website. 41  

An overview and analysis of the consultation activities are presented in Annex III.

B.    LIMITATIONS AND ROBUSTNESS OF FINDINGS

The following limitations to the robustness of the findings have been identified by the contactor: 42

-Available baseline: as already mentioned above, it was difficult for the contractor to define a baseline on all elements of the evaluation due to different approaches compared to and missing in the previous external evaluation. As far as possible, a baseline was identified nevertheless by the contractor.

-Short duration of the evaluation: this was identified from the outset, i.e. in the offer of the contractor, as a potential risk for the external evaluation. The contractor took the necessary measures by spreading the work over different subcontractors.

-Low response rate: this applies, on the one hand, to the targeted survey of the scientific community and the civil society, and, on the other hand, to certain questions in the questionnaires, in particular those that needed specific knowledge. For both issues the contractor took the necessary measures by carrying out additional desk research (e.g. as regards the efficiency criterion for which only the Agency staff could provide any information) and/or by using the outcomes of other methods to complement the missing information (e.g. civil society representatives also provided input through the public consultation).

-Quality of the received data: in some cases, open-ended questions were used in the interviews and in the public consultation. This might have led to some respondents providing substantive answers whereas others did not answer the question or did not think about the same particular issue. Therefore, the evidence presented in quantitative terms should be treated with caution in this regard. In addition, there were slight variations in the formulation of the question regarding the potential for an expansion of the Agency’s competences. However, these slight variations should not have any impact on the overall results.

-Performance measurement: due to the use of different key performance indicators (KPIs) over the evaluation period and the lack of an Activity Based Budget, the contractor faced some challenges in measuring the performance of the Agency. This was mitigated by using headline indicators instead.

-Stakeholder bias: there might be a certain positive bias of the results as most of the stakeholder groups work closely with and/or benefit from the outputs of the Agency. However, due to the variety of the stakeholders consulted and the different data collection methods employed, this should be minor.

In addition to these limitations, which apply to the results of the evaluation, the evaluation has to be seen in the context of the future multiannual financial framework. This provides a potential budgetary constraint for any considerations regarding any future development, being within its current mandate or through the deepening or expansion of its mandate.

Despite the limitations set out above, the evaluation findings are valid and reliable. Mitigation measures were taken and a large body of qualitative evidence, which provides a solid basis for drawing conclusions, underpins the evaluation.

VI.    ANSWERS TO THE EVALUATION QUESTIONS

The information provided in this section is a summary of the information included in the final report of the external evaluation, which will be made available online. More detailed information can be found, therefore, in the (main) final report as well as in Annex 3 of the final report of the external evaluation.

A.    RELEVANCE

The relevance criterion considers the adaptability of the Agency to its stakeholder needs as well as to scientific, economic, political, social and technological changes. In addition, the question related to the potential expansion of the mandate (listed under “Lessons learned” in the Terms of Reference) was also addressed under the relevance criterion.

The stakeholder needs of the Agency are very diverse. Overall, the analysis showed that the Agency responded very well to these diverse needs. 43 The Agency has an excellent international standing and is regarded as a credible and authoritative source. 44 The participants in the public consultation consider the information provided by the Agency useful across all areas. The information found most useful related to the emergence and use of new drugs (90% found it either “very useful” (63%) or somewhat useful (27%)) and EU drug policy (90% found it either “very useful” (61%) or “somewhat useful” (29%)). The least useful area of information related to links between drug trafficking and organised crime with 15% finding it not useful, 38% somewhat useful and 26% very useful. 45

The main stakeholders of the Agency are policy-makers on national and European level. Their primary need is information and evidence on which to base policy decisions. The evaluation showed that the Agency is more relevant to the needs of European than national policy-makers.

As regards the national policy-makers, the relevance of the information provided by the Agency for policy decisions depends partly on the size of the Member State and the related level of development of drug information systems. The better developed the national drug information systems are, the lower the impact of the information provided by the Agency. However, stakeholders, in particular the interviewed members of the Horizontal Working Party on Drugs, recognised cross-EU comparative analysis across the board as highly relevant to their needs. Another reason for the lower impact on national level might be that national drug policies of several Member States go beyond illicit drugs and cover other addictions.

As regards the scientific community and practitioners, their needs were partially addressed. The respondents from the scientific community indicated that they use the work of the Agency on a daily basis. However, there is scope for greater engagement of the Agency with the scientific community. The practitioners from the public health area appreciated the increased work on this pillar. The focus of the consulted stakeholders in this area is on harm reduction and prevention and the respondents from this field suggested increased involvement of the Agency. The Agency is increasingly engaged with law enforcement practitioners, through e.g. input into trainings of the European Union Agency for Law Enforcement Training (CEPOL) and closer cooperation with Europol. This is linked to the increased focus of the Agency on supply side issues.

Finally, when it comes to the general public, the evaluation concluded that the visibility is not very high. It should however be noted that the general public is not defined as a core customer. Information provided on national issues is usually targeted at the national focal points, which are then disseminating the information as appropriate in the Member State.

The evaluation confirmed that the Agency adapted overall well to the developments posed by the changes on the drug markets during the evaluation period. 46 Examples are the use of new monitoring tools and methods, such as wastewater analysis 47 , the use of open source data, trendspotting 48 and similar; the reorganisation of the Agency to better reflect the political priorities by putting more emphasis on security and public health as the two pillars of the Agency’s work; or the integration of new digital solutions. In addition, the Agency adapted to the changing economic environment, i.e. budget constraints, by redeploying resources to (new/different) priority areas without an increase of its overall budget. At the same time, stakeholders suggested that the Agency could produce more forward-looking products 49 and that the Agency could invest more in IT training for staff and upgrading their databases and data collection systems. Stakeholders from the scientific community underlined the need to improve the dialogue with the scientific and research communities to maintain and expand the scientific expertise of the staff.

One of the issues where the previous external evaluation stated that further work should be done is the drug supply side. The Agency stepped up work on supply side issues compared to the previous evaluation period. This led, for example, to an intensified cooperation with Europol and other relevant interlocutors, the publication of several topical reports 50 and the stronger involvement in the EU policy cycle 51 . In addition, the Agency is collecting more information on supply side issues through the Reitox network. However, the current evaluation showed that even more could be done in this context and that the Agency’s work to monitor supply side issues should be further enhanced. This could include further strengthening of cooperation with Europol and other bodies working in the area. Several stakeholders mentioned during the consultation the supply side as one element where further information from the Agency would be welcome. However, it is important to stress that this would have to be matched with increased data provision by the national focal points on drug supply related issues. In order to be able to do so, the national focal points would have to collaborate closely with the law enforcement and other relevant agencies in their Member State. It should be noted that full achievement of this objective is dependent on the mandate and financial resources available to national focal points.

Another element where some stakeholders suggested that the activities could be improved is harm reduction and public health. In this context, the Agency could consider focussing more on assessing the effectiveness of harm reduction and prevention interventions and social integration measures as well as on vulnerable populations. This is related closely to the fact that although outreach to stakeholders is well developed and appreciated, a stronger involvement of practitioners could help to make an even stronger impact on contributing to a healthier Europe.

In addition to the better regulation related questions on relevance, the evaluation also looked under this criterion at the question on the potential broadening of the scope of monitoring and identifying best practice in the field of prevention and care of addictive behaviours. This question addressed the issue of whether the Agency should continue to focus on illicit substances only or whether its mandate/competences should be revised to also cover other addictions, including e.g. alcohol, tobacco, prescription medicines and addictions without substances such as gambling.

The background to this question is that national drug strategies of several Member States already have a broader scope than illicit substances. The following table provides an overview of (national) drug strategies with a broader focus (state of play 2016). 52

Arguments put forward during the consultation included that addiction is an overarching health problem and should be addressed holistically; the trend of Member States to move to broader addiction strategies; the arbitrary distinction between what substances are licit or illicit; and the capacity for collection of good quality data. It should be noted that some data related to other substances are already collected by the Agency due to the involvement in the European School Survey Project on Alcohol and Other Drugs (ESPAD) 53 and the need to address poly-drug use as part of the existing mandate. Other stakeholders argued that such a broadening could lead to a loss of focus on current work and could decrease quality; a lack of expertise and resources in the national focal points to provide the additional data; and issues related to EU competences and potential creation of overlaps.

The stakeholder consultation was inconclusive as to the way forward. Whereas national stakeholders and Agency staff were largely in favour of broadening the mandate, other stakeholder groups (including EU-level stakeholders) were critical vis-à-vis such a development. Among those speaking in favour of a potential future broadening of the mandate, alcohol was mentioned most frequently. Other substance-related addictions followed. National stakeholders expressed less support to include addictive behaviours (not related to substance use) as it would be difficult to set the limits for such a mandate. 54 All stakeholders, whether in favour or against a broadening of the mandate, stressed that any change in the mandate would have to be accompanied by sufficient financial and human resources as there is no scope to do so within the currently available resources.

B.    EFFECTIVENESS

The effectiveness criterion considers the extent to which the Agency’s activities and outputs met its objectives set out in the founding Regulation, but also in the EU Drugs Strategy (and its Actions Plans) and the EU Agenda on Security. In addition, the question related to the international activities (listed under “Lessons learned” in the Terms of Reference) was addressed under the effectiveness criterion.

The main objective of the Agency is to provide factual, objective, reliable and comparable information on drugs, drug addictions and their consequences, as set out in Article 1(2) of the founding Regulation. The Agency has fully achieved this objective as it is highly regarded as a centre of excellence in providing such information, not only in Europe but internationally. The information provided is considered politically independent, is tailored to the particular stakeholder groups it wants to address and is based on an increasing use of different sources.

As regards “monitoring the state of the drugs problem, in particular using epidemiological or other indicators, and monitoring emerging trends, in particular those involving poly-drug use”, the Agency performed overall quite well. The Agency’s provision of demand and supply data contributed to informing relevant authorities and practitioners, improving their ability to respond to drug trends. The Agency established mechanisms to carry out regular and sustained monitoring of developments in the drug field, aimed at identifying emerging risks. The monitoring task is complemented by publishing regular and up-to-date information on drug supply and demand developments, in particular through the European Drug Report, but also through its many other publications. However, the evaluation also concluded that more could be done as regards drug-supply indicators and that work on poly-drug use is largely lacking. As regards the latter, the external study stated that the term “poly-drug use” is not mentioned once in the Agency’s 3-year strategies and work programmes.

As regards “monitoring the solutions applied to drug-related problems; providing information on best practices in the Member States and facilitating the exchange of such practices among them”, the conclusion of the external evaluation is similar. Stakeholders evaluated the performance of the Agency positively, in particular the civil society organisations participating in the public consultation as well as answering to the survey and to the targeted interviews. The Best Practice Portal 55 is central in sharing best practice in the areas of prevention, treatment, harm reduction and social reintegration. In addition, the work of the Reitox network as well as holding expert meetings is crucial for sharing best practices across Member States.

As regards “assessing the risks of new psychoactive substances and maintaining a rapid information system with regard to their use and also regarding new methods of using existing psychoactive substances”, the Agency contributes through the implementation of the Early Warning System (EWS), which is available 24/7, and the risk assessments of new psychoactive substances. Over the last years, the number of new psychoactive substances detected for the first time in the EU and therefore notified to the Early Warning System decreased from a peak in 2014 (see the figure below 56 ). At the end of 2018, the Early Warning System monitored more than 700 new psychoactive substances, of which more or less half are available in any given year on the European market. 57 The monitoring by the Agency enhances Member States’ capacity to tackle this growing phenomenon. However, the substances detected are becoming more dangerous and, therefore, the Agency finalised in 2017-18 eleven risk assessments, leading to Commission proposals for putting the substances under control. 58 The role of the Agency in this process has been reinforced by the new legislation on new psychoactive substances. 59  

Finally, as regards “developing tools and instruments to help Member States to monitor and evaluate their national policies and the Commission to monitor and evaluate European Union policies”, specific support was provided upon request to some Member States to help them design or monitor national drug strategies and policies. This was the case in 2015 for Germany, Ireland and Luxembourg. Tools and instruments for the evaluation of drugs policies and monitoring of drug markets have been developed. 60

The achievement of the objectives of the 3-year strategies and the related work programmes has further improved from an already good baseline in the previous external evaluation. This is partly due to a more focussed and realistic planning as consecutive strategies and work programmes reduced the number of planned outputs following a streamlining of the activities. Another element for improving the delivery of outputs is related to the three-level prioritisation system introduced. 61 The evaluation showed that the Agency achieved the goals and objectives set out in its two 3-year strategies and work programmes as shown by the table below. 62  

This conclusion is also confirmed by the outcome of the public consultation as shown in the figure below. 63

As regards the question to what extent the national focal points delivered the data and information required for the Agency to meet its objectives, the overall conclusion of the evaluation is that the data and information was delivered effectively during the evaluation period. The Reitox network and its national focal points are considered a cornerstone of the Agency’s work and activity. The “Reitox Development Framework” 64 , which was put in place in 2018, addresses several of the issues identified by the external evaluation and includes measures to build up further the capacity of the national focal points and to improve the quality of their data collection and analysis. New reporting packages, including detailed handbooks, guidelines and procedures, were adopted during the evaluation period in order to rationalise the data collection process. The provision of data by the national focal points by the deadline set by the Agency increased considerably. This is crucial for the Agency being able to provide the analysis of the data in due time for the publication of the annual European Drug Report.

Despite progress made, further improvements are considered possible and necessary. This relates in particular to the comparability of the data provided and in some cases the data quality. 65  

The problems identified by the external evaluation as regards the data provision by the national focal points are related to the varied mandates and resources available to national focal points. The appointment, setting up, functioning and maintenance of a national focal point is the responsibility of the national authorities and is therefore 100 % financed by the country concerned. However, the Agency co-finances national focal points in EU Member States to a certain extent by means of a grant agreement, which supports the implementation of specific activities in relation to national reporting functions and obligations in the Agency’s annual work programme. The mandate of the national focal points also depends on the national drug policy. For example, the mandate can go far beyond drugs to cover different tasks as regards all kinds of addiction or it can be very narrow and only cover certain data collection tasks.

Governance

The previous external evaluation already concluded that the governance structures and procedures worked well. The changes made to the structure and the internal reorganisation are quite recent as they happened in 2016 and 2017 following the adoption of the “EMCDDA Strategy 2025”. 66 Therefore, their impacts are not fully visible yet. However, the findings of the current evaluation, in particular the input received from interviews and the survey of the Agency’s staff, suggest that the changes have been generally positive. The internal reorganisation helped to focus resources better on the two pillars of the “EMCDDA Strategy 2025”, i.e. security and public health. This reorganisation improved the internal coordination within the Agency, led to providing more robust data and addressing new issues, and enabled the Agency to react better to challenges and contextual shifts in the field of drugs.

Monitoring

The evaluation report stresses that due to the use of different key performance indicators (KPIs) over the evaluation period and the lack of an Activity Based Budget, performance monitoring by the contractor could be done only based on headline indicators. Further efforts to streamline the indicators as well as to introduce an Activity Based Budget should be undertaken by the Agency to mitigate this for the future.

The tools 67 available to monitor and review the Agency’s outputs and results are considered adequate for ensuring accountability and an appropriate assessment of its performance. Starting with the 2014 work programme, a specific key performance indicator (KPI) was put in place for the monitoring of the work programme delivery. A detailed Monitoring and Evaluation Plan has been developed as internal tool to follow-up. In addition, a Management and Implementation System, a performance monitoring IT system, is in place; however, only limited training of the staff happened so far.

The evaluation concluded that the internal and external mechanisms 68 in place for monitoring, reporting and evaluating work well and are adequate to ensure accountability and appropriate assessment of the overall performance. Several regular review exercises are in place to review progress and monitor the outputs and achievements of results. These measurements are regularly discussed and followed up in the appropriate meetings. A mid-term review of the “EMCDDA Strategy 2025” is planned for 2020. As regards external mechanisms, the Agency is subject to the scrutiny of the annual discharge procedure, the European Parliament, the European Court of Auditors and the Internal Audit Service. The General Report of Activities provides an annual follow-up to these monitoring mechanisms.

External factors

The evaluation identified four main external factors, which had an influence on the effectiveness of the Agency: 69

-Changing European drug landscape: The significant increase of new psychoactive substances was considered the main challenge. The Agency stepped up the proactive monitoring of new psychoactive substances through its Early Warning System and developed innovative monitoring tools to identify the changes in drug consumption (e.g. trendspotting studies or wastewater analysis).

-Emergence of new marketplaces: New online marketplaces as well as changing trafficking routes are most notable. The Agency increased its collaboration with other EU agencies, such as Europol, to monitor better these new marketplaces and improved its online monitoring tools. This external factor will increase further over time and therefore adaptations of the work will be needed.

-New security challenges: These challenges stem from the involvement of organised crime groups in several criminal activities (“poly-criminality”), such as drug trafficking, human trafficking, migrant smuggling, or financing terrorist activities. The links between these activities need to be addressed in the future and this will have an impact on the information to be provided to policy-makers.

-Evolving national policies: With the change of national drug/addiction policies comes a change in the needs of national stakeholders. This had a significant impact on the Agency’s effectiveness, in particular with regard to the identification of best practices in health and social responses.

Cooperation with international organisations and third countries 70

The recast of the founding Regulation in 2006 gave the Agency a more explicit role as regards cooperation with international organisations and third countries. The previous external evaluation concluded that the “relationship with key partners at international level should be further developed to improve the capacity to monitor and analyse the drugs situation and responses to it”.

The Agency took on board this recommendation through continued work on the “EMCDDA Strategy for International Cooperation” 71 , which has largely been implemented, and consecutively, in 2018, the adoption of the “EMCDDA International Cooperation Framework 2018-25” 72 . Its activities at international level are compatible with the EU priorities in external action. The Agency brings the EU’s evidence-based policy-making experience in drug monitoring to third countries, thereby helping to improve the global understanding of the drug phenomenon, including the impacts global developments have on the European drug market.

The Agency is supporting the European Commission in several regional cooperation projects 73 and the analysis of progress of candidate countries as regards adherence with the EU acquis in the drugs field 74 . In addition, the Agency has concluded over time a number of working arrangements with third countries in order improve the capacity development in these third countries. 75  

Finally, the Agency cooperates closely with the United Nations Office on Drugs and Crime (UNODC) and the World Health Organisation (WHO), and to a lesser degree with other international organisations. As regards the cooperation with the United Nations Office on Drugs and Crime, several stakeholders raised the duplication of data collection as something that needs to be addressed, despite the need to overcome several legal and methodological difficulties.

It was also commented during the consultation that the cooperation with countries of the Western Balkan and immediate neighbourhood of the EU might be a too limited approach and that the Agency, in general, could do more on the international sphere, although within the limits of the general resource constraints. The Agency should clearly define the benefit to its objectives and the understanding of the EU drug situation of cooperation with third countries. This could also include the review of the priority countries/ regions. International cooperation activities should focus on facilitating the obtaining of information in a timely and regular manner from third countries or international organisations. Based on these considerations, the countries for future cooperation could be chosen. In this context, the funding opportunities for the work with third countries should be looked at, in particular the possibility of the Agency to receive ad hoc funding to implement its activities outside the EU.

C.    EFFICIENCY

The efficiency criterion considers whether the Agency achieved its objectives and delivered results in a cost-effective manner. More specifically, this section reviews the costs and benefits of activities; the adequacy of the human and financial resources to attain its objectives; value for money; the efficiency of its governance and organisational set-up and the scope for further simplification of its administrative structure and ways of working; and the external factors influencing its performance. In addition, the question related to the communication on and dissemination of outputs (listed under “Lessons learned” in the Terms of Reference) was addressed under the efficiency criterion.

The annual budget over the evaluation period was around EUR 16 million. The three main types of expenditure are staff costs (on average 59% of the budget); operational activities (on average 27% of the budget) and support activities (on average 14% of the budget). The budget remained relatively stable over the evaluation period with a decrease in 2014 and a one-off increase in 2015. The budget distribution is comparable with other agencies of similar size.

Distribution of budgets of selected EU agencies (2017) 76

The number of staff also remained stable with on average 105 full-time equivalents per year. The share of operational vs. administrative staff increased in favour of the operational staff whose share in the overall staff increased from 68% in 2012 to 71% in 2017. This lead to a consequential decrease in support staff, from 32% in 2012 to 29% in 2017.

The benefits of the Agency include clear and thorough understanding of the drug situation in the EU, better informed debates on drug policies and strategy, more effective exchange of information among professionals (including through the Reitox network), a proactive approach to new psychoactive substances and emerging trends, and the promotion of scientific excellence. These benefits are difficult to quantify, but overall the outputs of the Agency are valued highly by all stakeholders at EU, national and international level.

The evaluation concluded within the limits of the available data for the efficiency analysis, in particular that Activity Based Budgets were not available, that the Agency spent its resources, both human and financial ones, efficiently. Despite the budget staying relatively flat and the need to respond to new challenges and an increasing volume of tasks, the Agency managed to deliver without affecting the quality of the outputs by efficiently redeploying resources.

Another indicator for the efficiency of the performance is the implementation of the annual work programmes and delivery of its outputs. Level 1-targets were implemented by 88% (2014) to 97% (2017). 77 As the table below shows, the share of non-completed results/outputs declined from 21% in 2013 to 9% in 2017. At the same time, the number of specific objectives in the work programmes was streamlined without affecting the volume of its outputs and its outreach.

Status of results/outputs (2013-2017) 78

The Agency consistently used all allocated funds to achieve its work programmes. The budget execution rate for core activities (excluding the special projects) was very high, over 99.6% in each of the evaluation years, reaching 100% for 2016 and 2017. This indicates that the allocated funds were spent efficiently and no waste occurred.

Therefore, the evaluation concluded that the Agency provides value for money and that the resources available have been used efficiently. The evaluation also notes that two thirds of the stakeholders consulted considered the financial and human resources as adequate for the current priorities and tasks, but insufficient in case any new tasks would be added.

Despite the overall very positive conclusion as regards the use of human and financial resources, further improvements are possible. Further synergies, in particular on administrative issues, should be considered with other agencies, in particular with the European Maritime Safety Agency (EMSA), with which the Agency already shares some resources and uses common services. The implementation of change management programmes and the further development of targeted use of secondments or short-term contracts with scientists or interns should be considered.

The governance, organisational structures and procedures work in general well. The changes introduced following the adoption of the “EMCDDA Strategy 2025” were beneficial in terms of efficiency as it strengthened cross-unit working and the coordination of scientific activities. The increased efficiency is shown by the fact that the Agency produced more outputs while investing the same resources in absolute terms.

Overall, the evaluation concluded that the structure and organisation is efficient and no major changes are needed.

The following issues were mentioned as influencing efficiency: the decrease of the subsidy in 2014; the increase in numbers and in dangerousness of new psychoactive substances; the increase in poly-drug use; the decrease in national funding for national focal points; and the shift in policy focus to other (security) areas, such as terrorism. Overall, it should be noted that no additional financial (or human) resources were allocated to the Agency for additional work, except for the work related to the implementation of the new legislation on new psychoactive substances. As mentioned before, the Agency still managed to deliver results and outputs on these issues through the redeploying of resources.

In terms of communication and dissemination, the feedback from stakeholders was very positive and it was considered as efficient. The evaluation concluded that the online outreach 79 , in particular the social media outreach increased considerably 80 , in particular taking into account the limited dedicated financial and human resources compared to other agencies. Improvements were made, following the recommendations of the previous external evaluation, as regards printed communication material, making in particular the European Drug Report timelier and including more visual content. However, these are still areas where there is potential for further improvement, i.e. making reports and information more interactive, providing more and better translation, and more generally improve the website. In addition, the outreach to the general public and national stakeholders (policy-makers and practitioners) should be further improved. Outreach at national level is done currently through the national focal points, which act as multipliers and disseminators of the Agency’s products. However, it could be worth considering addressing these target group also more directly.

D.    COHERENCE

The coherence criterion considers the extent to which the activities and objectives of the Agency support the key EU policy developments, as well as are complementary to actions undertaken by other EU agencies and Member States, including being consistent with its regulatory framework. In addition, the questions related to synergies with other Justice and Home Affairs agencies on security issues and to cooperation with the United Nations level, in particular with the United Nations Office on Drugs and Crime, (listed under “Lessons learned” in the Terms of Reference) were also addressed under the coherence criterion.

The external evaluation mapped the objectives, tasks and priority areas set out in the founding Regulation as the well as the EU priorities with the two 3-year strategies covering the evaluation period 81 . The same was done for checking coherence more generally with EU policy developments. 82 The desk research of the contractor and the mapping exercise of the founding Regulation and the EU strategies with the Agency’s 3-year strategies and work programmes and other relevant strategic documents of the Agency is the basis for the information presented below.

The table below presents the number of specific objectives in the 3-year strategies and work programmes for each of the priority areas set out in Annex I of the founding Regulation. 83

The external evaluation shows that all priority areas set out in the founding Regulation have been addressed in both programming periods and that all objectives, tasks and priority areas are covered well by the work programmes. This confirms the conclusions of the previous external evaluation. One element of the founding Regulation where some gaps were identified during the evaluation is the monitoring of data on emerging trends in poly-drug use. There are only very few outputs in relation to the poly-drug use and the issue itself is as such not even mentioned in the Agency’s strategic objectives. Some work is done in the context of the European School Survey Project on Alcohol and Other Drugs (ESPAD), but the evaluation concluded that more could be done. Other areas where the evaluation concluded that more could be done are work on drug supply, in particular the related indicators 84 , or further cooperation with the local level and with the scientific community.

When it comes to the EU Drugs Strategy 2013-2020 and its Action Plans, the mapping showed a strong alignment between the Agency’s outputs and its results with key and overarching EU objectives. This is evidenced by the fact that the Agency is mentioned in 22 out of 55 actions of the EU Action Plan on Drugs 2017-2020 as responsible party. In addition, the Agency is required to report on 14 of the 15 indicators for both EU Action Plans. The mid-term assessment of the EU Drugs Strategy highlighted the significant contribution by the Agency and its Reitox network to the implementation of the EU Drugs Strategy and to a better understanding of all aspects of the drugs situation in the EU and trends in drug markets. 85  

The 3-year strategies and work programmes address and contribute to the objectives, principles and priorities of the European Agenda on Security. Given the limited remit of the Agency on the wider security issues, it cannot be expected that it comprehensively covers all aspects of the European Agenda on Security. However, through the growing focus on supply-side issues and the cooperation with other agencies working in the area (such as Europol, the Maritime Analysis and Operations Centre (MAOC(N)), or the European Union Agency for Law Enforcement Training (CEPOL)) on, for example, joint trainings for law enforcement agencies, joint publications or the stronger involvement of the Agency in the EU Policy Cycle, the Agency increasingly contributes to this priority.

Therefore, it can be concluded that the objectives and the activities set out in the two 3-year strategies and work programmes are highly coherent with the regulatory framework. All of the objectives set out in these documents align with and can be traced back to the founding Regulation.

The impact case study 86 indicated a moderate to strong causal link between the Agency’s activities and the impact of providing an evidence base for policy-making. Several activities, in particular in cooperation with other European and international agencies, provide evidence for the coherence of their work with EU policy developments. Coherence with the work of the most relevant EU agencies is facilitated by mutual review of the respective work programmes.

Overall, the Agency is well aligned with the Common Approach for decentralised agencies. 87 The tools and mechanisms in place work quite well and are adequate to ensure accountability and appropriate assessment of the overall performance. Monitoring of the implementation of the 3-year strategies and work programmes and the Management Plan takes place via a series of monitoring exercises. The development of clear key performance indicators is positive; however, these have changed during the evaluation period and therefore comparisons over time and with the previous evaluation were in parts not possible. In addition, the Agency has not yet implemented some Common Approach actions, in particular those relating to Activity Based Management and Activity Based Budgets. Other elements where improvements would be needed to achieve full compliance with the Common Approach are related to making the website as multilingual as possible, the Management Board considering the need for ex-ante evaluations of activities and programmes that have budget implications, and looking into the potential joint procurement with contracting authorities of the host Member States. All these actions are on-going within the Agency, but have not been completed for different reasons. 88

When looking more closely into the coherence of the objectives and activities with the drug-related objectives of the European Commission and other EU agencies, in particular Europol, the European Centre for Disease Prevention and Control (ECDC) and the European Medicines Agency (EMA), the evaluation concluded that the objectives set out by the Agency in its 3-year strategies and work programmes and the “EMCDDA Strategy 2025” are complementing those of its partners on EU level. The analysis 89 shows that there are hardly any overlaps with other EU agencies, but there could be considerable synergies. Many of these synergies are already exploited by the agencies. Examples include the sharing of information on new psychoactive substances among relevant agencies, in particular the European Medicines Agency; cooperation with Europol regarding the role of the internet and the darknet in selling drugs; the cooperation with the European Union Agency for Law Enforcement Training (CEPOL) on training activities; or the cooperation with Eurojust on issues related to judicial challenges in transnational prosecution. More concrete examples are provided in the following paragraphs.

The merging with another agency was considered detrimental to the quality of the scientific output. In addition, the fact that the Agency’s activities are very specific is seen by consulted stakeholders as there not being a potential for a merge with another agency. 90

The conclusion as regards coherence with Member States’ national drug policies is slightly less positive as some Member States apply a pan-addiction approach whereas the mandate of the Agency is limited to illicit substances. As the Agency cannot provide data on other addictions to the Member States to support their policies, some interviewed stakeholders, in particular from Member States, saw this as limiting the coherence of the work of the Agency with their work on national level.

Synergies with the Commission exist for example in the work on drugs precursors and on health-related issues. As regards precursors, the Agency cooperates with DGs TAXUD and GROW. This cooperation takes place on a voluntary basis and therefore relies on the goodwill and dedication of the involved individuals. Therefore, there is a risk that these activities will be interrupted if staff or priorities change. This risk could be mitigated by a formal working arrangement with the Commission on drug precursors. Such a working arrangement already exists with DG SANTE. However, the evaluation concluded that a closer structural relationship could be developed. Cooperation also takes place with the Joint Research Centre (JRC), which is involved in the detection of drugs and precursors in Europe and provides data to the Agency. Cooperation with JRC is important, as the Agency does not have its own forensic laboratory facilities for scientific testing.

The cooperation with Europol 91 is very fruitful and is considered by Europol as optimal. Several joint publications have been published over the years, starting with the joint work on risk assessments of new psychoactive substances, to (up to now) two editions of the EU Drug Markets Report 92 to specialised publications such as the report on drugs and the darknet 93 . Both agencies provided their relevant expertise for the joint work, with Europol’s focus being on data on drug production, distribution, seizure and the role of organised crime.

The cooperation with the European Medicines Agency 94 is related mainly to work in the context of the Early Warning System and the assessment of new psychoactive substances. The two agencies also exchange and cross-reference each other’s data as part of pharmacovigilance efforts to prevent misuse of medicinal products and the use of new psychoactive substances. This leads to more accurate data for both agencies.

The collaboration with the European Centre for Disease Prevention and Control 95 focusses on areas where drug use constitutes a risk factor for the contraction of infectious diseases. The joint activities include gathering data from Member States and joint missions to specific Member States in the context of HIV. Again, the collaboration means that each agency can provide its scientific expertise to effectively conduct reviews and risk assessments. There is scope to update joint products more regularly and additional fields of work could be considered, assuming that resources are made available.

Other relevant Justice and Home Affairs agencies with which the Agency already collaborates are the European Union Agency for Law Enforcement Training (CEPOL) and the European Union Agency for judicial cooperation in criminal matters (Eurojust). As regards the European Union Agency for Law Enforcement Training 96 , the Agency contributes heavily to training activities aimed at enhancing awareness among EU law enforcement authorities working in the field of drugs. As regards Eurojust 97 , a publication 98 aimed at policy-makers and legal practitioners to find suitable responses to the phenomenon of new psychoactive substances was published. Given Eurojust’s increasing involvement in drug trafficking cases, the Agency is well placed to continue to inform and provide context on drug trafficking. No collaboration exists so far with the European Border and Coast Guard Agency. However, the evaluation recommended that joint activities with European Border and Coast Guard Agency should be explored, in particular as regards data exchange and the monitoring of potential links between drug trafficking, human trafficking and migration.

The Agency also cooperates with the Maritime Analysis and Operation Centre – Narcotics (MAOC(N)) 99 , an operational platform of several Member States, third countries and agencies set up to principally tackle the transatlantic flow of cocaine from South America to Europe. The Maritime Analysis and Operation Centre – Narcotics is dependent on information from the Agency for its operations. Closer cooperation might be beneficial for both, on the one hand, for the Maritime Analysis and Operation Centre – Narcotics to further enhance its effectiveness due to more direct data access, on the other hand, for the Agency to be more involved in operational activities and thereby having a stronger impact on creating a more secure Europe.

As regards the international level, the objectives were set out in the “2007 Strategy for international cooperation” which was replaced by the “EMCDDA International Cooperation Framework 2018-25” in 2017. When it comes to international organisations, the focus is the cooperation with the United Nations Office on Drugs and Crime 100 and the World Health Organisation 101 and in particular the exchange of data with these organisations. The Agency also participated as experts in several of their meetings and provided input to their publications. In addition, cooperation also takes place with the Pompidou Group 102 and the American Drug Abuse Control Commission (CICAD) 103 .

The external study shows that data exchange between the Agency and international organisations is largely unidirectional. An example is the work of the Expert Committee on Drug Dependence of the World Health Organisation on new psychoactive substances. The Agency is providing all data available in its database on the substances, which are critically reviewed by the Expert Committee. On the other hand, there is no or very limited provision of data from the United Nations Office on Drugs and Crime to the Agency, which then could be used by the Agency in its reports. Getting more information from international organisations would help the Agency to develop better its understanding of how the global drug markets affect the European drug market.

The organisation with which most overlaps were identified was the United Nations Office on Drugs and Crime (UNODC). Several stakeholders, in particular the Reitox network, identified duplications in the data provision to the two agencies. The parallel reporting streams partly lead to discrepancies in data due to occasionally different individuals or departments reporting to the two organisations. It was suggested by some consulted stakeholders that the Agency could support the Member States in their reporting to the United Nations Office on Drugs and Crime; however, some practical and legal issues would have to be clarified beforehand. In any case, the two agencies should work towards further aligning the different reporting requirements and questionnaires. The Agency’s stakeholders also identified further scope for increased cooperation with international organisations on issues such as sharing best practices and adoption of common guidelines on data collection.

E.    EU ADDED VALUE

The EU added value criterion considers the benefits that the Agency brings as compared to what could have been done at national level. More specifically, it seeks to assess the Agency’s capacity to improve the ability of Member States to monitor and respond to drug-related problems, and the extent to which it is a valuable source of information for its main ‘customers’. This section also explores the sustainability of the activities and identifies the advantage of attributing these tasks to an agency like the European Monitoring Centre for Drugs and Drug Addiction as compared to other alternatives. It concludes by reflecting on the consequences of a potential termination of the Agency.

Its stakeholders regard the Agency as the main EU-level information source in the field of drugs. The provision of factual, objective, reliable and comparable information is the main objective of the Agency based on its founding Regulation. The evaluation concluded that the Agency has an excellent reputation as a source of scientific and comprehensive quality data 104 , not only in Europe but also on an international level. The information, which is considered objective and not politically influenced, contributes highly to the policy-making on European and to a lesser extent on national level.

The Agency contributes to informing the policy debate at EU level through the consolidation and dissemination of scientific and timely information, as well as through its participation in meetings and debates where drug policies are discussed (for example the meetings of the Council’s Horizontal Working Party on Drugs). The references made to the Agency and its outputs in various legislative and policy documents at EU level show its tangible impacts. This is especially visible with respect to new psychoactive substances, where the Agency has a central role in the risk assessment of new substances, and the action plans adopted in the framework of the EU Drugs Strategy, where the Agency has been assigned the role of responsible party in relation to several actions.

The information provided by the Agency is heavily relying on the provision of data from the national focal points. For this to happen, the Agency implemented a drug-related data collection system that requires the national focal points to collect and subsequently report to the Agency the same set of data using a common methodology for every reporting country. The data provided by the national focal points is then analysed by the Agency and provided in different forms, including reports, to its stakeholders. The EU-level overviews are used for the evidence-based EU-level policy-making, but are also valuable for national stakeholders. It allows Member States not only to design evidence-based policies but also to monitor the effectiveness of their strategies, policies and interventions as well as to increase their capacity to understand and to respond to drug-related threats. For the Agency to be in a position to provide sufficient information on emerging threats and trends in drug demand and supply, it would have to develop additional state-of-the-art methodologies to get the underlying data. This would make the Agency more responsive to emerging threats and the dynamic nature of the drug situation in the EU.

One example of such additional state-of-the-art methodologies is wastewater analysis. This was funded originally by the Commission through project-based funding to research institutions with the involvement of the Agency. If such analysis should be continued, financial resources have to be made available to the Agency. Such new and innovative (state-of-the-art) data collection methods are needed to ensure that a complete picture of the drug phenomenon in Europe is available to policy-makers and that the Agency remains on top of the policy developments in the drugs field

An important tool in this context is the Early Warning System (EWS) on new psychoactive substances. The Agency collects data provided by Member States, analyses the data and then shares the information with the Early Warning System network. This information is a key component of many national early warning systems.

Other forms of support to the national level include specific support upon request of a Member State to help them design or monitor national drug strategies and policies 105 ; the identification and sharing of best practice 106 as well as training which is related to capacity-building exercises; and financial support 107 . When it comes to input towards national drug policies, the evaluation recommended that the Agency could further develop its tools to support Member States in evaluating their national drug policies and strategies and in assessing the effectiveness of drug-related measures.

The feedback received from consulted stakeholders, in particular from the reporting countries, showed that in some countries, in particular those with a shorter tradition in doing so, the collection of drug-related data and the related monitoring might not survive without the Agency in place. Consulted stakeholders expressed clearly that without the common standards, guidance and methodological support provided by the Agency, it would not be possible to collect drug-related data with the same quality, accuracy and timeliness. A clear outcome of the evaluation was that without the help and support of the Agency such a comprehensive and comparable data collection would not be possible. 108

Although national policy-makers often rely on the work of the Agency, e.g. as regards identifying best practices, evidence-based information on hot topics or input to their national drug policies, the evaluation concluded that the Agency did not always manage to inform the national debates to the same extent as at EU level. In particular those Member States with a robust national drug monitoring system do not necessarily use the Agency’s data and information as the primary source of information but might rely more on their national standalone reports. Several Member State representatives also claimed that the Agency did not succeed in triggering political shifts on a national level. Therefore, the Agency could enhance its engagement with national stakeholders and more proactively involve practitioners, including through training and exchange of best practices.

As regards the sustainability of the activities, the main contributors to its sustainability are the nature of the data collected, i.e. the comprehensive nature and regular updating of the information provided together with the use of new analytical methods, which allow for long series of statistical data; the political impact of the outputs through the provision of input for evidence-based policy-making; the forward-looking vision of the EMCCDA; and the development of a cooperation network, in particular with the national focal points. Several other elements further support the sustainability such as the expertise available in the Agency, the development of methodological tools and indicators or the use of modern technologies. The stakeholders identified as the main challenge for sustainability going forward the availability of resources as limited resources will lead to the stopping of some of its activities. Another challenge is that keeping information relevant for future challenges requires significant efforts by the Agency to ensure that the data are up-to-date and that new information sources can be used.

The question whether other existing or alternative options of implementing drug policy would have been more efficient in achieving the objectives set out in the EU strategic documents was assessed. The overall conclusion on this question from the external evaluation was that the Agency is the most effective option in achieving the objectives set out in the EU Drugs Strategy and its Action Plans.

One alternative option would be to allocate the tasks to the Commission services in cooperation with other EU agencies already active in the area (such as Europol, the European Medicines Agency or the European Centre for Disease Prevention and Control). While this option would still ensure a regular EU-level data collection and analysis, it has two main disadvantages compared to the current set-up. Firstly, the Commission would have to outsource some of the activities, which would lead to a loss of institutional capacity and expertise in the longer term. Secondly, this loss of expertise might compromise the quality of the data analysis and thereby might lead to a loss of trust in the data and information provided.

The second alternative option would see the responsibility for monitoring the drug phenomenon reverting to the Member States. The Member States would need to take steps to coordinate themselves and to ensure that the collection of existing data and descriptive analysis continues. However, an agreement on common methodologies, indicators and quality standards would be more difficult to reach, compromising the comparability of data used to produce EU-level overviews. In addition, the reliability and impartiality of the information might suffer as Member States with more expertise and resources would be able to influence the analysis and formulation of policies more than others. Finally, this option would also lead to losing the Agency as a major actor in implementing the EU Drugs Strategy and subsequently would make it more difficult for the EU to speak with one voice on the international level.

The third alternative option would be to attribute the activities (or at least some of them) to international organisations active in the field, such as the United Nations Office on Drugs and Crime, the World Health Organisation, the Council of Europe or others. Although the collection of data would be maintained with a central organisation to coordinate, it would probably mean that the picture of the drugs phenomenon in Europe would be incomplete as less data on a regional level would be available. This might also lead to a loss of expertise and understanding of the influencing factors in Europe and to a diminished role of the EU on the international level.

This links with the question about the most likely consequences of a potential termination of the Agency. 109 All stakeholders agreed that the termination of the Agency would have negative impacts and should be avoided. While at the first instance it would be policy-makers who would be affected by such a termination, ultimately, the general public would also feel the impacts, albeit more indirectly.

The most likely consequence would be a loss of EU-level information because of the interruption of data collection and analysis. As there would be no obligation to collect the relevant data and no more centralised support, some reporting countries would cease to collect the data. Even those who would continue collecting drug-related data would likely do so to a more limited extent and would agree on the least common denominator. This would result in incomplete and fragmented data. The exchange of best practice would be more difficult without a central organisation bringing the information and people together. The cessation of the Agency would undermine the possibility to design and implement drug policies and strategies based on objective and scientific evidence, both at EU and at national level. Finally, the termination would have negative consequences on the role of the EU in the international sphere. It would make it more difficult to have a common understanding of the drug

phenomenon and therefore to speak with one voice in international fora. To conclude, the termination of the Agency would constitute an undesirable step backwards.

VII.    CONCLUSIONS AND LESSONS LEARNED

The overall conclusion of the evaluation, supported by an external study carried out between March and November 2018 by ICF Consulting Services Limited (ICF) in collaboration with the Centre for the Study of Democracy (CSD) and Optimity Advisors, was that the European Monitoring Centre for Drugs and Drug Addiction (EMCDDA) is working well. The evaluation was positive as regards all five evaluation criteria, but further improvements are possible.

The evaluation was carried out in accordance with the requirement set out in Article 23 of the founding Regulation, which obliges the Commission to initiate an external evaluation every six years to coincide with the completion of two of the Agency’s 3-year strategies and work programmes.

The evaluation concluded that both pillars of the Agency, i.e. public health and security, are well covered by the work programmes and strategies. The Agency’s work programmes and strategies, but also the outputs and outcomes produced, are relevant in relation to the overall objectives of EU drugs policy. The Agency meets the needs of its many stakeholders and was able to adapt to changes in the drug markets as needed. The Agency’s work is coherent with the EU drug policy objectives. Most synergies available with the work of the EU institutions, other EU agencies and international organisations are already used. In general, only few overlaps of work exist due to the very specific work area of the Agency. As regards efficiency, within the limitations of the available data, the evaluation concluded that the Agency used the available human and financial resources efficiently to deliver the outputs, outcomes and impacts set out in its work programmes.

The Agency is well recognised as a centre of excellence in providing information on the drugs phenomenon, not only in Europe but also internationally. The work of the Agency also has important impacts on the international level, including bringing the EU’s evidence-based policy-making in drug monitoring to third countries. The information produced is considered factual, objective, reliable and robust. The Reitox network is seen a cornerstone of the Agency’s work and activities as its members, the national focal points, are the providers of the relevant data.

There is scope for greater engagement with some other stakeholder groups, in particular the scientific community and practitioners, in order to better cover their needs in the future. As regards the general public, more direct engagement should be considered to improve the visibility, but also to ensure equal information across Europe, e.g. on new psychoactive substances. Outreach and communication should be improved in particular with these stakeholder groups. Synergies with the Commission could be reaped further by formalising the working arrangements regarding the work on precursors, developing a closer structural relationship when it comes to health-related issues and cooperating close with the Joint Research Centre.

Despite the very positive recognition of the work of the Agency as data provider, the evaluation concluded that there is room for further improvement. During the interviews, stakeholders commented that more forward-looking information and products should be made available in order to better help EU preparedness and response in this fast changing policy area. Although many efforts have been made following the previous external evaluation on the comparability of data and considerable improvements have been implemented 110 , further efforts are needed to ensure better comparability of data. This is true similarly for supply side information, where considerable efforts have been made during the evaluation period, but the Agency’s work on this should be further enhanced. However, more work on drug supply issues, including the further development of relevant indicators, would have to be matched by the availability of additional data, including through the provision of such data from the national focal points. Other areas, which could be addressed better and where the Agency’s input would provide added value, are poly-drug use and the support to Member States in evaluating their national drug policies.

More targeted support, e.g. through Reitox Academies, is considered useful to develop further the knowledge among the staff of the national focal points. Several consulted stakeholders suggested the enhancement of the IT systems for data collection, in particular the FONTE-system.

The use of visual aids in reports was acknowledged positively, but a few stakeholders suggested to further streamline the provision of information and to make it more user-friendly and interactive. The quality of the translation was put forward as a concern. It has to be acknowledged that translations are very costly and therefore a good balance in selecting the products for translation has to be found. However, whenever translations are made, they should be of good quality.

The evaluation was inconclusive on the potential future broadening of the scope of the Agency to other licit and illicit substance and addictive behaviours.

The Agency provides good value for money; however, there is room for improvement when it comes to simplification of the administrative set-up and the internal working methods. Despite the available budget remaining relatively flat over the evaluation period and the increased demands during the same time, the Agency continued to deliver excellent outputs due to a reprioritisation of its work and the related redeployment of resources. The key bottlenecks as regards the organisation were identified as the administrative burden which is considered (by the staff) to be too heavy for an agency of the size of the Agency; the need to review and modernise information and communication technology (ICT) tools; the need to introduce Activity Based Management and related budgeting; and better internal planning. The key performance indicators should be streamlined and developed further.

The EU added value of the work of the Agency is very high. The evaluation clearly concluded that the Agency is instrumental for continued collection of good scientific, comprehensive and (largely) comparable quality data. No other institution, on international, EU or national level, or other EU agency could take over the work of the Agency with the same results. All stakeholders considered the possibility of a termination of the Agency as very negative.

As regards international cooperation, overlaps have been identified with the data provision to the Agency and the United Nations Office on Drugs and Crime, which should be addressed, together with the questions regarding some legal and methodological difficulties. The Agency should also address the fact that data provision to international organisations is currently mainly unidirectional. Overall, the consultation suggested that more could be done on the international sphere. Therefore, the Agency should (re-)consider the priority setting as regards its cooperation with third countries, taking into account the EU policies in the area and the understanding the Agency might gain from working with a third country on the EU drug markets.

Some of the issues set out above are already addressed by the “EMCDDA Strategy 2025” or are considered for implementation in the Agency’s most recent work programme as they have been acknowledged by the Agency already as areas for improvement. However, as the budget of the Agency is constraint, it would not be possible for the Agency to do more with the currently attributed human and financial resources 111 . It should be clearly noted that this applies to activities and tasks covered by the current mandate.

VIII.    ANNEXES

ANNEX I – PROCEDURAL INFORMATION

The external evaluation of the European Monitoring Centre for Drugs and Drug Addiction (EMCDDA) has been carried out in the period April to November 2018. Preparatory steps, such as the roadmap of the initiative 112 and the Terms of Reference for hiring a contractor 113 , were initiated in summer 2017. The Management Board was informed about the upcoming evaluation at its meeting of December 2017. The agenda planning reference for this evaluation is PLAN/2017/1680.

An inter-service steering group was set up for the evaluation. The following Directorates-General (DGs) and services followed the invitation to nominate a participant in the inter-service steering group: the Secretariat-General of the Commission (SG), the Legal Service (LS), DG Human Resources and Security (HR), DG Budget (BUDG), DG Taxation and Customs Union (TAXUD), DG Neighbourhood and Enlargement Negotiations (NEAR), DG Health and Food Safety (SANTE), DG Internal Market, Industry, Entrepreneurship and SMEs (GROW), DG Mobility and Transport (MOVE), DG Research and Innovation (RTD), the Joint Research Centre (JRC), Eurostat and the European External Action Service (EEAS).

In addition, an ad hoc group was set up to allow an inclusive evaluation process. The ad hoc group consisted of the Chair and the Vice-Chair of the Agency’s Management Board and two staff members of the Agency.

DG Migration and Home Affairs (DG HOME) chaired the meetings of the inter-service steering group and of the ad hoc group. Both groups were systemically consulted during the evaluation process taking into account the crosscutting nature of drug policies in the EU.

The inter-service steering group was consulted by e-mail on the Terms of Reference in December 2017 with all comments having been taken into account. The members of the inter-service steering group were invited to the kick-off meeting with the contractor, which took place on 28 March 2018 (DGs NEAR, RTD and the EEAS participated in the meeting). The questionnaire for the public consultation, for which the contractor prepared a first draft, was sent to the inter-service steering group for input in April 2018.

Following the submission of the inception report, back-to-back meetings with the contractors and the ad hoc group as well as the inter-service steering group took place on 25 April 2018. The draft interim report was discussed with the ad hoc group on 6 July and with the inter-service group on 10 July 2018. The meetings on the draft final report took place back-to-back on 28 September 2018. All meetings took place in the presence of the contractor.

Both, the ad hoc group as well as the inter-service steering group, had the opportunity to provide their comments on the different draft reports, ask for clarifications and submit comments on the developments of the external evaluation. They received the draft and revised inception report, the draft and revised interim report as well as the draft final report for comments and review, before the reports could be deemed approved.

The final report of the external evaluation was submitted to the Commission in the beginning of November 2018.

Further details on the evidence used in the evaluation as well as the discussion of its limitations are detailed in Annex II on "Methods and Sources" and in Annex III on "Stakeholder consultation".

ANNEX II – METHODS, SOURCES AND LIMITATIONS

(1)    Methods and sources

In order to gather the necessary information for the external evaluation, the contractor undertook an extensive desk research, carried out interviews, undertook two targeted surveys, peer-reviewed several Agency reports and undertook case studies. The external evaluation was informed also by the results of the public consultation and a workshop.

(a)Desk research

The contractor reviewed all programming and reporting documents available on the Agency’s website, together with those uploaded on the online portal specifically set up for the evaluation. The reviewed documents are listed in Annex 2 of the final report of the external evaluation. Annex 3 of the final report of the external evaluation provides in detail the results of the desk research, in particular the mapping tables.

(b)Interviews, surveys and public consultation

See below in Annex III for details on the stakeholder consultation process.

(c)Peer review

Experts reviewed the effectiveness, relevance and EU added value of selected publications 114 , based on the agreed methodology outlined in the inception report. For each output, the experts completed a template to assess each criterion. More specifically, they provided a score and short written assessment of the key questions under each criterion. The contractor reviewed the templates and the outcome of the peer review exercise was incorporated into the final report.

The peer review of the selected publication of the Agency resulted in an average score across the selected publications of 4.2 (1 being lowest/5 highest). 115 It showed that – across the documents reviewed – there was a clear orientation of the information to the respective target group. The experts rated the scientific quality of the information provided by EMCDDA very highly for robustness, traceability (referencing) and evidence base. For some documents, the experts concluded that they were particularly important to introduce policy-makers to general drug policy principles or to close the gaps in terms of reliable data at national level. 116

(d)Case studies

The contractor developed five case studies (thematic, organisational, benchmarking, counterfactual and impact) by conducting additional interviews, carrying out desk research, and taking into account the outcomes of the peer review exercise and the conclusions from the workshop. The case studies are included as Annex 4 in the final report of the external evaluation.

Thematic case study 117

The objective of the thematic case study was to explore the themes that could be relevant for the Agency if any broadening of its scope was to be considered in the future to enhance its preventative action and achieve its strategic goals. The case study summarised the method used to produce the report and discussed the factors influencing any expansion of scope. It also presented four main themes identified as potential areas for expansion (alcohol consumption, tobacco consumption, prescription medicines and gambling).

The thematic case study concluded that given its existing remit and responsibilities, the Agency might be able to play a role in assessment, monitoring and intervention of other addictive behaviours. The complexity of the drug phenomenon is illustrated by the fact that many of the factors that are linked to drug use and/or exacerbate drug problems are broader social issues, including those associated with other addictions (i.e. alcohol or prescription medicines) or addictive behaviours such as gambling. 118  

As the “EMCDDA Strategy 2025” highlights, the Agency should have a more strategic, situational and holistic understanding of the European drug situation and its implications if it is to achieve its vision of a healthier and more secure Europe. The EU Drugs Strategy 0 sets out the need for prevention measures, early detection and intervention, promotion of healthy lifestyles, and targeted prevention directed at families and communities. Poly-drug use is already mentioned as a key priority for the Agency in its founding Regulation and this provides an appropriate framework to address some of the four themes discussed in the thematic case study. 119  

If a proposal to extend the mandate would be made in the future, further considerations are needed on what would be of most benefit for the Agency and its stakeholders, and what would be possible within the resources available. Part of this will include understanding possible synergies with partners working in the same area and ways in which the Agency can contribute to existing work, e.g. through exchange of information compared to full participation or improving data collection systems.

Organisational case study

The objective of the organisational case study was to assess how cooperation with international stakeholders has contributed to the core activities and the implementation of various relevant strategies where international cooperation is an explicit objective. The case study specifically assessed how the Agency’s organisation can be adapted to provide a comprehensive and up-to-date understanding of the drugs market in Europe, by analysing the impact of developments in the international drugs markets on the European drugs market.

The organisational case study concluded that the “EMCDDA Strategy for International Cooperation” has been largely implemented. The Agency has worked towards the achievement of all three objectives, i.e. (1) better assessment of the global drug situation, (2) improving the understanding of the drug situation in third countries, and (3) supporting EU policies and initiatives in the drugs field. The Agency has become internationally recognised for the quality and scientific rigour of its work and is recognised as a key partner for international organisations, non-governmental organisations and third country government agencies working in the field of drugs. In many cases, Member States continue to work directly with international organisations, whose activities often overlap with those of the Agency.

The case study also found that the “EMCDDA Strategy for International Cooperation” did not prioritise the needs of the Agency with regard to the benefit of engaging in international cooperation activities, especially with third countries, nor did it clearly outline the principles on which to base the priorities for cooperation, in view of its limited resources. The objectives and key principles of the “EMCDDA International Cooperation Framework 2018-25” provide a much more aligned and focused approach to international cooperation.

The assessment of cooperation activities between 2013 and 2017 shows that:

-International cooperation remains a valuable means for the Agency to spread its methodological and scientific standards, as well as to understand better the global drug situation, which shapes the EU’s drug issues.

-    Cooperation with international organisations focused primarily on working with the United Nations Office on Drugs and Crime and the World Health Organisation, and to a much lesser degree with other organisations.

-International cooperation activities directly contributed to a significant extent to certain core activities, including: a) the improved understanding of global issues linked to third countries where the Agency does not have cooperation agreements (e.g. on heroin issues); and b) receiving support from international organisations in implementing activities in enlargement countries.

-All stakeholders noted the duplication of data collection with the United Nations Office on Drugs and Crime as something that needs to be addressed.

-The cooperation with third countries beyond candidate and neighbourhood countries was, and will remain, limited. There is scope for better alignment with the EU’s Global Strategy, increasing cooperation with priority regions (such as Central and South Asia, Middle East, and Africa) where issues of drugs, migration and terrorism intertwine.

The strategic decision, outlined in the “EMCDDA International Cooperation Framework 2018-25”, to rely primarily on additional or project-based funding limits the sustainability of its activities and the extent to which the Agency can build and maintain relationships with third countries and international organisations. Future international cooperation activities may need to demonstrate more direct evidence to satisfy the underpinning principles of the Agency’s cooperation with international stakeholders.

Benchmarking case study

The objective of the benchmarking case study was to draw conclusions on the efficiency and effectiveness of the Agency by comparison with another EU agency. The benchmarking provided a comparative understanding of key aspects of the Agency’s operations and structure. It also helped to identify best practices and recommendations on potential improvements to areas where inefficiencies are identified. The benchmarking case study compared the Agency with the Fundamental Rights Agency (FRA).

The benchmarking case study concluded that the two agencies have similar financial structures and are facing similar challenges: increased demand for their products and services while available financial resources remain relatively flat. Both agencies have been able to meet these challenges within available budgets. 120  

Both agencies appear to be facilitating the networks they support in an efficient way, considering their existing needs and working methods. The analysis shows certain room for improvement in mechanisms ensuring the quality of data and analysis the Agency receives from Reitox.

Despite appearing more burdensome and bureaucratic compared to the organisational set-up of the Fundamental Rights Agency, the Agency’s governance structure is more conducive to a system of checks and balances, where there are a number of bodies on its several organisational layers evaluating performance and assuring quality of outputs.

Both agencies have developed communication strategies and boast high satisfaction rates for their communication and outreach activities. 121 The Fundamental Rights Agency has a wider online and social media reach. 122 It is focused more on national stakeholders, whereas this category is less well reached by the Agency. The Fundamental Rights Agency facilitates actors at national level to consistently communicate and implement fundamental rights initiatives.

The content of the two agencies is highly valued by their stakeholders. The Fundamental Rights Agency produces many more diversified outputs, repurposing content to reach a wider array of stakeholders. 123 The two organisations have different strengths in this regard: the specialisation of the Agency is conducive to the production of highly specialised scientific content and larger, in-depth publications, while the Fundamental Rights Agency’s wider array of topics is conducive to the production of many shorter products, which are less technical and can be targeted towards a larger audience.

Impact case study

The objective of the impact case study assessed the contribution of the Agency’s activities to delivering intended impacts. It focused on two specific impacts based on the overarching objectives, which cover both the drug demand and drug supply aspects of the Agency’s activities, namely: development of national-level drug demand/ harm reduction strategies and policies; and improved EU capacity to act on drug supply reduction measures.

As regards the contribution of the Agency to demand side policy developments at national level, the impact case study concluded that national assessments, evaluations, general and thematic reporting and the Best Practice Portal are all viable routes for providing national level stakeholders with information and delivering results. The strength of this relationship appears to diminish as the number of harm reduction responses increases. There are many Member States with fewer harm reduction responses, which develop their strategies and policies based on information provided by the Agency, whereas other Member States rely more heavily on data collected and reports published at national level. Overall, there appears to be a causal link between the activities, outputs, results and impact on policy development, to which the Agency contributes moderately to strongly.

The contribution of the Agency to impact policy development depends greatly on the national political environment and the coherence of the attitudes and strategies across national institutions operating in the field of drug demand/ harm reduction. The Agency has limited scope to translate the evidence considered by policy makers directly into tangible policy.

As regards the capacity of the EU to take action on drug supply reduction, the impact case study concluded that the measures to enhance this capacity were the provision of an overview of drug precursors, the EU Drug Markets Report and the risk assessments on new psychoactive substances.

Combatting drug precursor diversion and trafficking is an objective of the EU Drugs Strategy and its Action Plans. The information on drug precursors is presented in the EU Drug Markets Report. The impact case study suggests that the Agency’s presentation of information on drug precursors in a wider EU context makes a moderate contribution to precursor scheduling decisions. The establishment of formal working arrangements with DG TAXUD and DG GROW, together with continued and comprehensive reporting of precursors in the context of the EU drug situation as supporting evidence in scheduling considerations, could provide further impact as supply reduction measures.

The EU Drug Markets Report provides a holistic overview of the drug market across Europe. The impact case study found that outputs provide key stakeholders with information on emerging trends, although this tends to be somewhat historical. There is evidence that the information from the EU Drug Markets Report fed directly into the development of the EU drugs policy and in Europol’s Serious and Organised Crime Threat Assessment (SOCTA). The collaboration with Europol and the direct input into the EU Policy Cycle means that the Agency is well placed to strongly contribute to EU-level drug supply reduction measures, including via the EU Drug Markets Report.

The Agency examines all available data to produce risk assessments of new psychoactive substances as part of the Early Warning System. There is a direct causal link between the production of the risk assessments and its consequent submission to control measures at EU-level. Thus, the Agency’s role in conducting risk assessments of new psychoactive substances has a significant impact on supply reduction measures.

Counterfactual case study

The objective of the counterfactual case study was to provide an in-depth response to the question ‘What would happen if the Agency did not exist?’ This case study aimed to contribute to the analysis of the EU added value evaluation criteria. It identified and described the possible consequences and impacts of the absence of the Agency’s functions and activities, at both EU and Member State level.

The analysis and data presented in the counterfactual case study on the Member States’ capacity to monitor the drug situation and produce and share early warning information on new psychoactive substances shows that the Agency adds value and without its support most Member States and the EU as whole would achieve less in drugs policy.

As regards the Member States’ capacity to monitor the drug situation, the sample of countries reviewed 124 suggests that:

-    Some Member States produce detailed and high-quality information on the drug situation, that meets their individual needs.

-EU-level information and the ability to compare data with other Member States is crucial in helping countries to understand how the domestic drugs situation compares to other Member State. This, in turn, helps Member States to determine whether or not their policies need to be further improved.

-The Agency has a significantly greater analytical and expert capacity and competence than many Member States and can thus more easily produce high-quality analysis or develop methods and standards for harmonisation.

With respect to Member States’ capacity to maintain a national early warning system, it can be concluded from the counterfactual case study that:

-National early warning systems in most Member States would be much less effective if EU-level data were absent.

-An EU-level mechanism would function far less effectively without the Agency’s role in collecting and analysing the information before sharing it with all Member States. The value of simply exchanging national signals or information between national early warning systems is much lower than the value of the current Early Warning System, which is based on EU-level analysis.

-Even Member States with pre-existing or well-developed national early warning system see value in the EU-level aggregation and analysis of information from other Member States.

(e)Workshop

A workshop took place on 6 September 2018, gathering the evaluation team of the contractor, the panel of four independent experts, who undertook the peer reviews, and the ad hoc group. The intention of the workshop was to have an open discussion on the preliminary results of the external evaluation and possible recommendations. The discussion took place based on a short background note.

(2)    Limitations and robustness of the findings

(a)    General limitations

The following limitations should be noted with respect to the data collection activities undertaken by the contractors:

-Quality of baseline in the previous external evaluation: Although efforts were made by the contractor to compare the results to the previous evaluation period (2007-2012), some aspects examined were omitted or only partially examined in the previous external evaluation. Given the length of time that has elapsed since the 2012 evaluation, it was difficult to re-establish the baseline where it was found to be lacking.

-Qualitative data interviews presented as quantitative (open-ended questions): While survey data shows the share of responses in quantitative terms, the interview evidence presented in quantitative terms (i.e. number of respondents) should be treated with caution. This is because interviewees were asked open-ended questions and the fact that some interviewees pointed to a specific issue does not mean that those respondents who did not give the same response disagree, but, rather, may not have thought about the same issue during the interview.

-Slight variations in the question on expanding the competencies: There were slight variations in the formulation of the question related to the potential expansion of the competences across various data collection templates (the two surveys as well as the interview questionnaires targeting different stakeholder groups). For example, the question was sometimes phrased as ‘Can the EMCDDA broaden ...’, and in other cases ‘Should the EMCDDA broaden ...’. Similarly, the addictive behaviour question was asked sometimes as a separate question and other times included in the overall question.

-Short duration of the evaluation: At the outset of the study, the short duration of the evaluation, including the summer holiday period, was highlighted as a potential risk. However, the contractor successfully completed the stakeholder consultation programme in due time.

-Low response rate to the external survey: Only 26 respondents provided answers to the external survey with the scientific sector and civil society. This was due to the public consultations running in parallel with the survey and being promoted more actively. However, as the same stakeholder group responded to the public consultation and a selected number of respondents were interviewed, the contractor collected sufficient evidence of the views of these stakeholder groups.

-Low response rate regarding some specific questions: Some questions that required greater familiarity with the processes of the Agency were difficult for certain stakeholders to answer. For instance, only the staff of the Agency could answer most of the questions related to efficiency, so there were limited stakeholder views as evidence on several questions. The contractor used desk research and literature review to mitigate this absence and close any gaps in evidence.

-Stakeholder bias: The risk of bias among any particular group of stakeholders was low because all relevant stakeholder groups were covered sufficiently through different data collection methods. However, there is a slight risk of positive bias, as many of the consulted stakeholder groups work closely with the Agency and/or benefit from its outputs.

(b)    Budget limitations and constraints

-Constraints set by the Multiannual Financial Framework (MFF): The Commission proposals for the next Multiannual Financial Framework 2021-2027 were adopted on 2 May 2018 125 , including an indicative envelope for the expenditure of the decentralised agencies. This sets a budgetary constraint on the involvement in future initiatives. However, it was agreed at the inception meeting that these considerations should not limit the contractor’s thinking, but that the budget limitations should be included as contextualisation in the evaluation.

-Cost-effectiveness ratios to measure the performance: Over the course of the evaluation period, the Agency used different key performance indicators (KPIs) and different indicators, making like-for-like comparisons difficult. The lack of Activity Based Budgets also created challenges to conduct a cost-effectiveness analysis and/or to compare the Agency’s efficiency with those of other agencies. To mitigate this, the contractor focused on headline indicators (staff per function, budgeted expenditure per core activity, etc.).

ANNEX III – STAKEHOLDER CONSULTATION

A broad stakeholder consultation accompanied the external evaluation. The aim of the process was to obtain views on the five evaluation criteria, on the functioning of the Agency and on possible future challenges and developments from a range of stakeholders working with the Agency or benefitting from the work of the Agency. The contractor carried out the targeted stakeholder consultation. The Commission's standards for Stakeholder Consultations were duly taken into account and met.

As part of the stakeholder consultation, the Commission conducted a public consultation, with the purpose of gathering views from private individuals, organisations, the industry and the public administration.

The replies to the stakeholder consultation addressed all evaluation questions. The outcomes of the interviews, the targeted surveys and the public consultation are reflected fully in the external evaluation study and in this Staff Working Document. Summarising them in this Annex would be a repetition of the conclusions of the external evaluation set out in the main text of this document.

(1)    Targeted stakeholder consultations (interviews and surveys)

Targeted consultations took place within the framework of the external evaluation and involved different types of stakeholders. The contractor consulted with representatives of Member States, the Agency staff, EU Institutions and agencies, EU funded projects, international organisations, third countries, the scientific community and civil society. These targeted consultations were conducted mainly via interviews and partly also through a targeted survey (the civil society and the scientific community, on the one hand, and the Agency staff, on the other hand). The interviews were accompanied, where necessary, by a written questionnaire to collect quantitative data.

133 stakeholders were interviewed. 126 In 16 instances the interview could not be completed as the envisaged interviewee either did not reply to the request for an interview, declined the interview or it was not possible to complete the interview for other reasons. A list of interviews carried out is available in Annex 1 of the final report of the external evaluation.

The following list indicates the main targeted stakeholder consultations:

(a)    Member States

At Member State level, three different types of interviewees were consulted: members of the Agency’s Management Board, Member State representatives in the Council’s Horizontal Working Party on Drugs (HDG) and the Reitox national focal points.

Interviews carried out:

1.Members of the Horizontal Working Party on Drugs: 23 interviews

2.Reitox national focal points: 30 interviews

3.Management Board members: 22 interviews

(b)    Staff

Interviews with a selected number of staff took place (as group interviews). 11 interviews were carried out in total with staff.

In addition, the staff was consulted through a targeted survey. This online survey was open from 25 June to 13 August 2018. It was circulated to all the staff. The survey received 55 responses, representing 55% of the target group.

(c)    European institutions and bodies and EU-funded projects

At EU level, different types of institutions and bodies were consulted. Interviews were carried out with the European Commission (DG HOME, DG NEAR, OLAF, DG TAXUD, DG GROW, JRC, DG SANTE and DG DEVCO), the European External Action Service (EEAS), the Council of the European Union (Secretariat), Members of the European Parliament, the relevant EU agencies (Europol, European Medicines Agency (EMA), the European Centre for Disease Prevention and Control (ECDC), the European Union Agency for Law Enforcement Training (CEPOL), Eurojust, and the Consumers, Health, Agriculture and Food Executive Agency (CHAFEA)). In addition, a representative of the EU-funded Cooperation Programme on Drugs Policies (COPOLAD) was interviewed.

Interviews carried out:

4.EU agencies: 6 interviews

5.EU institutions and European External Action Service (EEAS): 11 interviews

6.EU-funded programme: 1 interview

(d)    International organisations and third countries

Interviews were carried out with representatives of the United Nations Office on Drugs and Crime, the World Health Organisation, the International Narcotics Control Board (INCB), the Pompidou Group of the Council of Europe, the Inter-American Drug Abuse Control Commission (CICAD), and the Maritime Analysis and Operation Centre – Narcotics. As regards third countries, representatives of Colombia, Albania, Montenegro and Moldova were interviewed. 127

Interviews carried out:

7.International organisations: 6 interviews

8.Third countries: 4 interviews

(e)    Civil society and scientific community

The contractor consulted the Civil Society Forum on Drugs through a targeted survey and interviewed the members of the Core Group of the Civil Society Forum on Drugs.

As regards the scientific community, several members of the Scientific Committee were interviewed as well as some other representatives of universities, think tanks and research facilities. The scientific community was also the subject of a targeted survey.

The online survey for the civil society and the scientific community was open from 9 July to 15 August 2018. The survey received 24 responses.

(2)    Public consultation

In addition to the targeted consultations, the Commission organised an internet-based public consultation.

The public consultation was launched on 18 May 2018 on the European Commission's website 128 and was open for 14 weeks. 129 The public consultation was available in seven languages, i.e. in English, German, French, Spanish, Italian, Polish and Portuguese, but could be answered in all official languages of the EU.

The aim of this consultation was to gather views from private individuals, non-profit/private organisations, and national/regional/local public administrations on the Agency. The consultation covered the majority of the evaluation criteria: effectiveness, relevance, coherence and EU value added. 130 As such, the public consultation was intended to form part of the input for the external evaluation.

(a)    Profile of respondents

147 contributions 131 were submitted through the online questionnaire. Out of the total number of respondents, 68 consented to the publication of their full contribution, while 80 opted for publication in anonymous form.

Respondents were invited to identify themselves as one of the following categories: as "an individual in your personal capacity" or "on behalf of an organisation/ in your professional capacity". More than half of the submissions (55%) came from respondents answering as private individuals with the remainder answering on behalf of an organisation. Those answering on behalf of an organisation can be broken down as follows: non-governmental organisations (34% of respondents), Member State governments (28% of respondents) and academic institutions (17% of respondents). Only two respondents answered on behalf of private companies and one respondent answered on behalf of an EU-level institution. The last 17% of respondents represented other categories, for example political groups.

Only 11 out of 65 respondents answering on behalf of an organisation replied that their organisation is included in the Transparency Register. Over half of the respondents (52% of respondents) indicated that this option is not applicable to their organisation, while the remaining 31% of respondents answered negatively.

Regarding the geographical distribution of responses, respondents were asked to indicate which country (or the EU as a whole) their responses referred to. Most of the respondents indicated to come from Portugal (14% of respondents), France and Italy (10% of respondents each), followed by Germany (8% of respondents), Spain (7% of respondents), Belgium (6% of respondents), the United Kingdom (6% of respondents), Netherlands (5% of respondents) and the Czech Republic (4% of respondents). Respondents from Austria, Estonia, Finland, Greece, Hungary, Ireland, Luxembourg, Malta, Poland, Romania, Slovak Republic, Slovenia and Sweden also replied to the public consultation. Answers from non-EU countries came from Norway, Switzerland, United States, Serbia, Ivory Coast, Mali and Georgia. The generally low number of respondents per country does not allow for a meaningful disaggregation of responses by country in the responses.

82% of respondents stated that they had an affinity with the field of drug and addiction policies. Furthermore, 78% of respondents stated that they had a good to very good understanding of the field. Only 21% of total respondents stated that their knowledge and understanding of EU drug policies was limited to very limited. One respondent (representing 1% of responses) stated that he/she has no knowledge and understanding of EU drugs policies.

Most respondents (82% of responses) stated that they have good to very good knowledge and understanding of the activities of the Agency. Only 18% of respondents claimed that their knowledge is limited. Additionally, most respondents to the public consultation (78% of respondents) declared having cooperated with the Agency or having used their services.

Almost half of respondents (47% of respondents) had regular contact with the Agency, with most of them (37% of respondents) monthly, and 10% of respondents weekly. Only 14% of respondents did not have any contact with the Agency during the evaluation period.

(b)    Results of the Public Consultation

The analysis of the results of the public consultation as well as the individual replies are available online. 132 The input received through the public consultation was integrated in the evidence provided in Annex 3 of the final report of the external evaluation and was taken into account in the case studies (Annex 4 of the final report of the external evaluation).

The public consultation showed that the Agency is relevant in facilitating solutions for each of the challenges it was designed for, especially in providing reliable, objective, evidence-based and accessible information on drugs and drug policies. This has helped policy-makers at all levels, as well as practitioners and individuals working in the field of drugs to have a better understanding of the drug situation in Europe, which they found has been beneficial in their professional lives.

The public consultation further indicated that the Agency has been producing important outputs in the form of a range of publications, which respondents regularly consult and find useful and which respond to their needs. Respondents to the public consultation stated that the Agency was a leading source of information on new drugs, EU drug policy and developments on the EU drug market. The most consulted publications were the flagship publications, such as the European Drug Report, although respondents stated that they would consult regularly also publications on specific topics relevant for their work.

According to most of the respondents, the Agency has been contributing in a positive way to the EU Drugs Strategy and its Action Plans, in particular regarding its role in providing a better understanding of all aspects of the drugs phenomenon and of the impact of interventions to provide a sound and comprehensive evidence-base for policies and actions. The Agency was considered useful in strengthening further the cooperation between the EU and third countries and international organisations on drug issues, as well as in contributing to the reduction in the demand for drugs, drug dependence and drug-related health and social risks and harm. Over half of the respondents found that the Agency could have an even greater impact or improve its effectiveness if it broadened the scope of its monitoring, by including more criminal indicators or by focusing more on the link between licit and illicit substances.

Concerning coherence, most of the respondents found that Agency’s activities were coherent with other EU policies. They further stated that its activities are in synergy with the work of other agencies, especially with Europol or the United Nations Office on Drugs and Crime, as they focus on similar topics, but through different angles, which provides a broader view of the drug situation. Most of the respondents did not find that there were overlaps. For those that did, the stakeholders stated that the Agency’s work was complementary to the work of other agencies, even if it did sometimes lead to ‘double-work’ as in the case of data reporting to the Agency and the United Nations Office on Drugs and Crime.

The EU added value of the Agency was established by the respondents to the public consultation when most of them considered that the Agency’s activities had improved Member States’ ability to monitor the drug situation in their country and to respond to drug problems, as well as to inform the policy debate on drug issues. Indeed, the public consultation found that the Agency had contributed to the understanding of the drug situation in Europe in general by providing credible and reliable data or that the Agency helped practitioners to do their work more effectively. Should the Agency cease its activities, the public consultation stated that this would result in the loss of high quality, objective and reliable knowledge. For future development considerations, 18% of the total respondents provided a reply. These replies showed that the Agency could go into different directions, such as covering a wider range of addictive substances or behaviours. Finally, the public consultation established that the future evolution of the Agency would most likely be to continue providing quality information that is essential for policy-makers and practitioners in the field of drugs.

ANNEX IV – INTERVENTION LOGIC

ANNEX V – SUMMARY OF THE MAIN AREAS AND GOALS IN THE 2013-2015 AND 2016-2018 STRATEGIES AND WORK PROGRAMMES 133

(1)

     Regulation (EC) No 1920/2006 of the European Parliament and of the Council of 12 December 2006, OJ L 376, 27.12.2006, p. 1. This Regulation repealed Council Regulation (EEC) No 302/93 and was amended most recently by Regulation (EU) 2017/2101 of the European Parliament and of the Council of 15 November 2017 amending Regulation (EC) No 1920/2006 as regards information exchange on, and an early warning system and risk assessment procedure for, new psychoactive substances, OJ L 305, 21.11.2017, p.1.

(2)

     Commission Delegated Regulation (EU) No 1271/2013 of 30 September 2013 on the framework financial regulation for the bodies referred to in Article 208 of Regulation (EU, Euratom) No 966/2012 of the European Parliament and of the Council, OJ L 328, 7.12.2013, p.42.

(3)

     For further information on the "common approach", please see https://europa.eu/european-union/sites/europaeu/files/docs/body/joint_statement_and_common_approach_2012_en.pdf .

(4)

     See Article 5 of the founding Regulation. For more information see http://www.emcdda.europa.eu/about/partners/reitox-network .

(5)

     External Evaluation of the European Monitoring Centre for Drugs and Drugs Addiction, Centre for Strategy and Evaluation Services (CSES), 11 June 2012, final report, https://publications.europa.eu/en/publication-detail/-/publication/7d189e3b-f767-460b-b748-acf44d2daf9a/language-en/format-PDF/source-74343049 .

(6)

     OJ C 402, 29.12.2012, p.1.

(7)

     EU Action Plan on Drugs 2013-2016, OJ C 351, 30.11.2013, p.1; EU Action Plan on Drugs 2017-2020, OJ C 215, 5.7.2017, p.21.

(8)

     COM(2015) 185 final.

(9)

     The Common Approach sets out agreed principles on the following issues related to decentralised agencies: role and position of agencies in the EU’s institutional landscape; structure and governance of agencies; operation of agencies; programming of activities and resources; accountability, controls and transparency; and relations with stakeholders.

(10)

     The Justice and Home Affairs agencies are: CEPOL (European Union Agency for Law Enforcement Training), EASO (European Asylum Support Office), EIGE (European Institute for Gender Equality), European Monitoring Centre for Drugs and Drug Addiction (EMCDDA), eu-LISA (European Agency for the operational management of large-scale IT systems in the area of freedom, security and justice), Eurojust, Europol (European Union Agency for Law Enforcement Cooperation), FRA (Fundamental Rights Agency) and EBCGA (European Border and Coast Guard Agency (also known as Frontex)).

(11)

     The cut-off date for the work of the contractor was set for 30 June 2018, i.e. the date of submission of the Agency’s General Report on Activities 2017; http://www.emcdda.europa.eu/publications/gra/2017_en .

(12)

     EU Drug Markets Report 2016, http://www.emcdda.europa.eu/start/2016/drug-markets . An update of this estimate will be provided in the EU Drug Markets Report 2019, which will be published in 2019.

(13)

     For details on the evolving drug markets see in particular the annual European Drug Reports, which set out trends and developments and thereby present a top-level overview of the drug phenomenon in Europe, covering drug supply, use and public health problems as well as drug policy and responses. The most recent report is the one for 2018: http://www.emcdda.europa.eu/publications/edr/trends-developments/2018_en .

(14)

     For example, France's addiction strategy covers illicit drugs and NPS, alcohol, tobacco, medicines, behavioural addictions (e.g. gambling) and doping; Austria's addiction strategy covers all these addictions except doping; Portugal's addiction strategy covers illicit drugs and NPS, alcohol, medicines and behavioural addictions. An overview (state of play 2016) is available in the EMCDDA Papers: New developments in national drug strategies in Europe, http://www.emcdda.europa.eu/system/files/publications/6402/20175662_TDAU17002ENN_PDF.pdf ; see also Section VI.A. below.  

(15)

     Council Regulation (EEC) No 302/93 of 8 February 1993 on the establishment of a European Monitoring Centre for Drugs and Drug Addiction, OJ L 36, 12.2.1993, p. 1.

(16)

      http://www.emcdda.europa.eu/system/files/publications/676/wp2013-15_393821.pdf .

(17)

      http://www.emcdda.europa.eu/system/files/publications/2095/TDAX16001ENN_.pdf .

(18)

      http://www.emcdda.europa.eu/system/files/publications/4273/2017.1998_EMCDDA_STRATEGY_2025_web-1.pdf .

(19)

      http://www.emcdda.europa.eu/publications-seriestype/work-programmes-and-strategies_en .

(20)

     For the limitations of the finding of the external evaluation, see Section V.B. and Annex II(2) of this Staff Working Document.

(21)

     See Article 11 of the founding Regulation..

(22)

     Source: EMCDDA.

(23)

     See Article 9 of the founding Regulation.

(24)

     In practice, two meetings took place every year, one in June and one in December, during the evaluation period.

(25)

     For the constitution of the Management Board, see http://www.emcdda.europa.eu/about/mb .

(26)

     See Article 10 of the founding Regulation.

(27)

     See Article 13 of the founding Regulation.

(28)

     See Article 5 of the founding Regulation. The abbreviation ‘Reitox’ stands for the French ‘Réseau Européen d’Information sur les Drogues et les Toxicomanies’.

(29)

     See Article 21 of the founding Regulation.

(30)

     Regulation (EU) 2017/2101 of the European Parliament and of the Council of 15 November 2017 amending Regulation (EC) No 1920/2006 as regards information exchange on, and an early warning system and risk assessment procedure for, new psychoactive substances, OJ L 305, 21.11.2017, p.1

(31)

     See Section VI.D.

(32)

     Working arrangements are currently in place with Russian Federation, Ukraine, Moldova, Israel, Armenia, Georgia, Switzerland and Albania.

(33)

     See for example the participation of the Agency in the Cooperation Programme between Latin America, the Caribbean and the EU on Drugs Policies (COPOLAD II) or the Central Asia Drug Action Programme (CADAP).

(34)

      www.emcdda.europa.eu/publications-database_en?f%5B0%5D=field_series_type%3A599 .

(35)

     All publications of the Agency can be found here: www.emcdda.europa.eu/publications_en .

(36)

     The questions are copied from the Terms of Reference provided to the potential contractors.

(37)

     The questions listed in this section were attributed by the contractor to the 5 better regulation principles, i.e. to the criterion where the question suits best. Q26 was addressed under “Relevance”; Q29 under “Effectiveness”; Q30 under “Efficiency”; and Q27 and Q 28 under “Coherence”. Q31 was addressed by the key findings under each criterion and by the overall recommendations. It should also be noted that some of the questions were slightly modified, in agreement with the Commission, to better address some issues.

(38)

     The documents reviewed for the evaluation are listed in Annex 2 of the final report of the external evaluation.

(39)

     The actual number of interviews conducted differs as some of the Member States’ representatives belong to various groups of stakeholders (i.e. members of the Horizontal Working Party on Drugs, National Focal Points and Management Board members), meaning that in some cases various groups of stakeholders were covered by a single interview. For an overview of the interviews carried out, see Annex 1 of the final report of the external evaluation.

(40)

      https://ec.europa.eu/info/consultations/public-consultation-evaluation-european-monitoring-centre-drugs-and-drug-addiction-emcdda_en .

(41)

      https://ec.europa.eu/home-affairs/what-is-new/work-in-progress/initiatives/evaluation-emcdda .

(42)

     More details are available in Annex II (2) of this Staff Working Document.

(43)

     76% of the respondents in the public consultation stated that the activities of the Agency responded to their needs to a large or to some extent. This result was also mirrored in the interviews undertaken.

(44)

     This is evidenced by the overall impression from the stakeholder consultations, i.e. the answers and feedback received. The majority of the stakeholders from international organisations (5 out of 6 interviewed) and from third countries (4 out of 5 interviewed) confirmed this view.

(45)

     More detailed information is shown in the graph; see Figure 3 in Annex 3 of the final report of the external evaluation.

(46)

     84% of stakeholders interviewed.

(47)

      http://www.emcdda.europa.eu/topics/pods/waste-water-analysis_en .

(48)

     Trendspotter manual: www.emcdda.europa.eu/publications/manuals/trendspotter-manual_en ; the most recent trendspotter study concerned recent trends on the cocaine market: http://www.emcdda.europa.eu/system/files/publications/10225/2018-cocaine-trendspotter-rapid-communication.pdf .

(49)

     See below in Section VI.B. for more detail.

(50)

     See e.g. the report “Drugs and the darknet” (December 2017), http://www.emcdda.europa.eu/darknet_en ; or the latest version of the EU Drug Market Report (April 2016), http://www.emcdda.europa.eu/start/2016/drug-markets .

(51)

     The EU Policy Cycle for organised and serious international crime for the period 2018 – 2021 aims to tackle the most important threats posed by organised and serious international crime to the EU in a coherent and methodological manner through improving and strengthening co-operation between the relevant services of the Member States, EU institutions and EU agencies as well as third countries and organisations, including the private sector where relevant. Drug trafficking is one of the 10 priorities of the EU policy cycle. More information can be found at https://www.europol.europa.eu/empact .

(52)

     EMCDDA Papers: New developments in national drug strategies in Europe, http://www.emcdda.europa.eu/system/files/publications/6402/20175662_TDAU17002ENN_PDF.pdf .

(53)

      http://www.espad.org/ .

(54)

     For a detailed analysis of the arguments in favour and against and the stakeholder perceptions, see Table 7 in Annex 3 of the final report of the external evaluation as well as the thematic case study in Annex 4 of the final report of the external evaluation.

(55)

      http://www.emcdda.europa.eu/best-practice_en .

(56)

     European Drug Report 2018, Figure 1.10, p. 32.

(57)

     Source: EMCDDA.

(58)

     The substances covered by these risk assessments were acryloylfentanyl, furanylfentanyl, AB-CHMINACA, ADB-CHMINACA, 5F-MDMB-PINACA, CUMYL-4CN-BINACA, 4-fluoroisobutyrylfentanyl, tetrahydrofuranylfentanyl, carfentanils, methoxyacetylfentanyl and cyclopropylfentanyl.

(59)

     Regulation (EU) 2017/2101 of the European Parliament and of the Council of 15 November 2017 amending Regulation (EC) No 1920/2006 as regards information exchange on, and an early warning system and risk assessment procedure for, new psychoactive substances, OJ L 305, 21.11.2017, p.1; Directive (EU) 2017/2103 of the European Parliament and of the Council of 15 November 2017 amending Council Framework Decision 2004/757/JHA in order to include new psychoactive substances in the definition of ‘drug’ and repealing Council Decision 2005/387/JHA, OJ L 305, 21.11.2017, p. 12.

(60)

     See for example “Evaluating drug policy: a seven-step guide to support the commissioning and managing of evaluations”, www.emcdda.europa.eu/system/files/publications/4680/td0417390enn1.pdf , and the related website ( www.emcdda.europa.eu/publications/topic-overviews/policy-evaluation ); or “Trendspotter manual: a handbook for the rapid assessment of emerging drug-related trends”, http://www.emcdda.europa.eu/system/files/publications/10233/2018-trendspotter-manual.pdf .

(61)

     Level 1: must-do-tasks with a target of 100% achievement; Level 2: supporting tasks to key commitments with a target of 80% achievement; and Level 3: supplementary tasks that can be scaled down without significantly impacting the achievement of the Agency’s objectives.

(62)

     Annex 3 of the final report of the external evaluation, p. 75; based on the annual General Report of Activities of the Agency.

(63)

     Annex 3 of the final report of the external evaluation, p. 43.

(64)

      http://www.emcdda.europa.eu/system/files/publications/8137/TD0118374ENN.pdf .

(65)

     As regards drug supply data, see above in Section VI.A.

(66)

     See Section III above for the current organisational chart of the Agency.

(67)

     In the evaluation, ‘tool’ was defined as an IT system or platform and/or an indicator used to carry out a particular monitoring or performance measurement function (as opposed to ‘mechanism’, which is a process involving multiple stakeholders).

(68)

     In the evaluation, ‘mechanism’ was defined as a process involving multiple (internal or external) stakeholders, used to monitor, report and evaluate the Agency’s overall performance and or ensure accountability.

(69)

     The contractors noted in the final report of the external evaluation that most stakeholders misinterpreted this question. Most identified budgetary concerns, data collection through the Reitox network or the shifting European policies as external factors that could influence the Agency’s effectiveness, but the contractors considered those to be internal factors, as they fall within the control of the European Commission, which is also the case for the Agency.

(70)

     For more details on the work of the Agency on international level, see the organisational case study in Annex 4 of the final report of the external evaluation.

(71)

      http://www.emcdda.europa.eu/attachements.cfm/att_255775_EN_EMCDDA_30_07_International_cooperation_FINAL.doc .

(72)

      http://www.emcdda.europa.eu/system/files/publications/9886/International%20Cooperation%20Framework.pdf .

(73)

     For example, the Cooperation Programme between Latin America, the Caribbean and the EU on Drugs Policies (COPOLAD II) or the Central Asia Drug Action Programme (CADAP).

(74)

     For example, the Agency was part of assessment missions to Serbia and Montenegro to assess the advances in the Justice and Home Affairs area as part of the membership negotiations.

(75)

     Working arrangements are currently in place with Russian Federation, Ukraine, Moldova, Israel, Armenia, Georgia, Switzerland and Albania.

(76)

     Annex 3 of the final report of the external evaluation, Table 10, p. 74; based on the 2017 budgets of the respective agencies.

(77)

     Level 2-targets by 78% (2014) to 92% (2017) and Level 3-targets by 66% (2014) to 77% (2017).

(78)

     Annex 3 of the final report of the external evaluation, Table 12, p. 75; based on the General Reports of Activities for the years 2013 to 2017.

(79)

     Figure 21 in Annex 3 of the final report of the external evaluation, p.81, shows the number of downloads of the main products of the Agency. The main downloaded publication is by far the annual European Drug Report, which has between 66.000 and 118.000 downloads.

(80)

     At the end of 2017, the Agency had approximately 11450 Twitter followers (compared to 3400 at the end of 2013), almost 8000 Facebook followers (compared to below 2000 in 2013) and more than 190000 Youtube views (compared to around 12500 in 2013).

(81)

     Annex 3 of the final report of the external evaluation, p. 82.

(82)

     Annex 3 of the final report of the external evaluation, p. 84.

(83)

     Annex 3 of the final report of the external evaluation, p. 1.

(84)

     For a state of play, see the EMCDDA-Europol Joint Publication “Improved drug supply indicators for Europe: progress report”, www.emcdda.europa.eu/system/files/publications/10178/Improved%20drug%20supply%20indicators%20for%20Europe_Joint%20publication.pdf .

(85)

     Mid-Term Assessment of the EU Drugs Strategy 2013-2020 and final evaluation of the Action Plan on Drugs 2013-2016, Annex J; https://ec.europa.eu/home-affairs/sites/homeaffairs/files/what-we-do/policies/organized-crime-and-human-trafficking/drug-control/eu-response-to-drugs/20161215_final_report_annexes_en.pdf .

(86)

     See Annex 4 of the final report of the external evaluation.

(87)

     The analysis of the compliance with the Common Approach on Decentralised Agencies is included in Annex 6 of the final report of the external evaluation.

(88)

     For more details see Annex 6 of the final report of the external evaluation.

(89)

     See analysis of questions EQ23 and EQ24, Annex 3 of the final report of the external evaluation, as of p. 88.

(90)

     54% of all interviewed stakeholders expressed this view. In addition, almost all participants in the public consultation provided a clear answer to the potential closing or merging of the Agency with another body.

(91)

      http://www.emcdda.europa.eu/about/partners/europol .

(92)

     2013: http://www.emcdda.europa.eu/system/files/publications/741/TD3112366ENC_406102.pdf ; 2016: http://www.emcdda.europa.eu/start/2016/drug-markets .

(93)

      http://www.emcdda.europa.eu/darknet .

(94)

      http://www.emcdda.europa.eu/about/partners/ema .

(95)

      http://www.emcdda.europa.eu/about/partners/ecdc .

(96)

      http://www.emcdda.europa.eu/about/partners/cepol .

(97)

      http://www.emcdda.europa.eu/about/partners/eurojust .

(98)

      http://www.emcdda.europa.eu/system/files/publications/3353/TD0416736ENN.pdf .

(99)

      http://www.emcdda.europa.eu/about/partners/maoc .

(100)

      http://www.emcdda.europa.eu/about/partners/unodc .

(101)

      http://www.emcdda.europa.eu/about/partners/who .

(102)

      http://www.emcdda.europa.eu/about/partners/pompidou .

(103)

      http://www.emcdda.europa.eu/about/partners/cicad .

(104)

     The peer review of selected publication of the Agency resulted in an average score across the selected publications of 4.2 (1 being lowest/5 highest).

(105)

     In 2015, the Agency provided support to Germany, Ireland and Luxembourg.

(106)

     This is done in particular through the Best Practice Portal ( http://www.emcdda.europa.eu/best-practice_en ), but also through gathering the relevant experts in meetings and sharing information at such occasions.

(107)

     The national focal points are co-financed up to a certain amount by the Agency through grant agreements.

(108)

     See above in Section VI.B. (Effectiveness) where the need to make further efforts towards a better comparability of data was stressed.

(109)

     For more information on the impacts of a potential termination of the Agency, see also the counterfactual case study in Annex 4 of the final report of the external evaluation.

(110)

     The Agency provided handbooks, guidelines and procedures to Member States in an effort to harmonise data collection and reporting. All Member States use the same protocols for the collection of key indicators.

(111)

     For the EU contributions to the budget of the Agency, see the Financial Statements which are part of the EU budget; for the 2019 annual budget see page 335-342 of https://ec.europa.eu/info/sites/info/files/about_the_european_commission/eu_budget/draft-budget-2019-legal-personality-public-private-ownership-com-2018-600_2018_en.pdf .

(112)

      https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2017-5812828_en . The roadmap was published on 28 November 2017 for the 4-week feedback period. No feedback was provided by stakeholders.

(113)

     The services were requested within framework contract HOME/2015/EVAL/02. 2 possible contractors applied for carrying out the evaluation. The chosen contractor was ICF Consulting Services Limited (ICF) who collaborated with the Centre for the Study of Democracy (CSD) and Optimity Advisors in carrying out the external evaluation.

(114)

     1. Flagship products: EU Drug Market Report (jointly produced with Europol), Health and Social Responses Guide. 2. Routine reporting activities: Rapid communication on the MDMA market, Rapid Communication on Drug-related infectious diseases, Assessing illicit drugs in wastewater, The Internet and Drug Markets (2016), Drugs and the Darknet: perspectives for enforcement, research and policy (2017), Drug impaired driving, EMCDDA 7-step guide to aid drug policy evaluation, Rapid Policy review: Ireland, Take-home naloxone programmes. 3. Products from the work of the Early Warning and Risk Assessment Activities: Substance: Acryloylfentanyl a) EMCDDA Formal Notification to the EU EWS Network b) EMCDDA Risk Communication c) EMCDDA-Europol Joint Report d) EMCDDA Risk Assessment Report; Substance: Carfentanil a) EMCDDA Risk Communication, b) EMCDDA-Europol Joint Report, c) EMCDDA Risk Assessment Report.

(115)

     Other key scores across the peer-reviewed publications: user-friendliness and readability: 4.2; factual and objective information: 4.4; accurate and credible information based on robust evidence: 4.3; comparability: 3.6; traceability of information (e.g. referencing): 4; clearly outlined methodology: 4.3; accessibility of outputs: 4.3.

(116)

     The peer-reviewed publications scored on average 4.15 on relevance. The topics of the peer-reviewed documents reflect key stakeholder priorities with only two of 18 peer-reviewed outputs scoring below 4. The “Internet and the Drug Markets” report and the joint reports between the Agency and Europol on two new psychoactive substances were rated as most relevant (received the maximum score of 5 on all peer review questions pertaining to relevance). The lowest score on relevance was given to “Drug Impaired Driving” (3.5/5), mostly because it was not clear which target group the report is aimed at and whether it reflects key stakeholder priorities.

(117)

     As a summary of the arguments put forward, see Table 7 in Annex 3 of the final report of the external evaluation.

(118)

     See also Recital 3 of the founding Regulation.

(119)

     For example, it is very clear that alcohol consumption is intrinsically linked to many drug issues, which has led to efforts to encourage and undertake joint analysis within the European School Survey Project on Alcohol and Other Drugs (ESPAD).

(120)

     It should be noted that the staff numbers of the Fundamental Rights Agency have grown by 15% from 2013 to 2017, while the Agency’s remained unchanged. Cost for staff (per full-time equivalent) is higher at the Fundamental Rights Agency, but this may be due to the differences in the correction coefficients applied to remuneration of expatriate officials in the Member States hosting the two organisations.

(121)

     In terms of dedicated resources, the Fundamental Rights Agency has a much larger communications budget (two-three times higher than that of the Agency).

(122)

     However, when it comes to YouTube followers, the Agency has a better outreach than the Fundamental Rights Agency.

(123)

     Due to its larger amount of publications, the Fundamental Rights Agency has a lower cost per output.

(124)

     Belgium, Bulgaria, Germany, France, Ireland, Netherlands, Poland, Spain, Sweden and the United Kingdom.

(125)

     COM(2018) 321; see for more details and all relevant documents regarding the Multiannual Financial Framework 2021-27 http://ec.europa.eu/budget/mff/index_en.cfm .

(126)

     The actual number of interviews conducted differs as some of the Member State representatives belong to various groups of stakeholders (i.e. members of the Horizontal Working Party on Drugs, National Focal Points and Management Board members), meaning that in some cases various groups of stakeholders were covered by a single interview.

(127)

     The countries were chosen in relation to work being done with the Agency and/or involvement in projects.

(128)

      https://ec.europa.eu/info/consultations/public-consultation-evaluation-european-monitoring-centre-drugs-and-drug-addiction-emcdda_en .

(129)

     The consultation period was extended by two weeks due to the summer holiday period.

(130)

     No questions were included on efficiency as it was considered as too complicated to answer questions on this criterion in a public consultation.

(131)

     One additional reply was received after the end of the public consultation period. Therefore, some of the figures add up to 148 instead of 147.

(132)

      https://ec.europa.eu/home-affairs/what-is-new/work-in-progress/initiatives/evaluation-emcdda .

(133)

     The areas in bold type are common to both 3-year strategies and work programmes, i.e. data collection, communicating results, governance, administration and IT systems. Source: Final report of the external evaluation, p. 10.