52017SC0287

1) assess the performance of the Union Mechanism in order to determine progress towards the general and specific objectives set out in Article 1 and 3(1) of the Decision during the first half of its implementation period (i.e. 2014-2016);

2) identify any gaps or shortcomings in the current legislative framework 2 ;

3) put forward recommendations to increase the Mechanism's effectiveness and improve the implementation of the existing framework;

4) provide input for possible proposals to amend the Decision or implementing acts adopted thereof;

5) if appropriate, inform the review of the financial breakdown of the Union Mechanism as set out in Article 19(5) of the Decision 3 .

Furthermore, in accordance with point (a) of Article 34(2) of the Decision, the Commission is required to report to the European Parliament and the Council on the results obtained and the qualitative and quantitative aspects of the implementation of the Decision by no later than 30 June 2017, and to put forward, if appropriate, proposals for amendments to this Decision.

To this end, the Directorate-General for European Civil Protection and Humanitarian Aid Operations (DG ECHO) contracted an external consultant to carry out an independent evaluation of the activities implemented under the Union Mechanism 4 . As the reflections on the future of the Union Mechanism under the next Multi-annual Financial Framework (2020-2027) are starting, findings from the Evaluation could usefully contribute to developing a future impact assessment, as recommended by the EU Better Regulation Agenda 5 .

3Scope of the evaluation

As set out in the Interim Evaluation Roadmap 6 , the Evaluation covers all activities carried out under the Union Mechanism in the timeframe spanning from January 2014 to December 2016. Actions targeting the UCPM Participating States 7 , as well as those performed in relation to third countries, including in accordance with Article 28(2) 8 , have been covered by the Evaluation.

Article 3 of the Decision spells out four specific objectives against which the Union Mechanism should be evaluated and their respective results indicators. These indicators encompass the spectrum of activities of the Union Mechanism across the fields of: (i) disaster prevention, (ii) preparedness, (iii) and response to natural and man-made disasters. In addition, cross-cutting activities relating to prevention, preparedness and response are included in the scope of the Evaluation.

The Evaluation has addressed the five mandatory criteria set out in the European Commission's (“the Commission”) Better Regulations Guidelines: (i) Relevance; (ii) Effectiveness; (iii) Efficiency; (iv) Coherence; and (v) EU added value. In addition, one additional criteria, (vi) Sustainability, has been added to the scope of this evaluation.

4BACKGROUND TO THE INITIATIVE

5Baseline

Prior to the entry into force of the Lisbon Treaty in 2009, EU civil protection actions and legislation were based on the catch-all provision of Article 308 of the EC Treaty, authorising the Council of the European Union (“the Council”) to act (by unanimity) where necessary in order to attain a Treaty objective in areas where there the EC Treaty provided no other legal basis. In line with the need to ensure better protection in disasters occurring both inside and outside the EU, the Union Mechanism was established in 2001 by the Council 9 and subsequently recast in 2007 10 .

With the advent of the Treaty of Lisbon in 2009, civil protection became a self-standing policy with its own legal basis provided by Article 196 of the Treaty on the Functioning of the EU, with the aim to:

a.Support and complement Member States' action at national, regional and local level in risk prevention, in preparing their civil-protection personnel and in responding to natural or man-made disasters within the Union;

b.Promote swift, effective operational cooperation within the Union between national civil protection services;

c.Promote consistency in international civil protection work.

Several challenges needed to be addressed at the time. The Impact Assessment 11 of the civil protection regulatory framework carried out in the context of the preparation of the Decision had highlighted, among others, the following shortcomings.

·Prevention: Lack of coordination between sector specific prevention policies; under-developed organisation for the sharing of experience; insufficient links between prevention, preparedness and response; limited awareness, vision and strategic investments.

·Preparedness: Lack of consistency among national training programmes hindering the sharing of experience; limited scope of EU preparedness action and limited links with national programmes, leading to lower preparedness levels; uncertain level of preparedness for major emergencies.

·Response: Reactive and ad-hoc mandate of the Mechanism limiting the effectiveness, efficiency and coherence of the European disaster response; some needs not being met due to the lack of specific assets (capacity gap); limited financial and logistical transport solutions slowing down or hindering optimal response; burdensome administrative procedures jeopardising a swift and efficient response.

Based on a Commission proposal, the European Parliament and the Council repealed previous legislation and adopted Decision No 1313/2013/EU of the European Parliament and of the Council of 17 December 2013 on a "Union Civil Protection Mechanism". This Decision is currently in force and governs actions and activities in the field of civil protection.

6Description of the initiative and its objectives

Decision 1313/2013 added to the Mechanism's original focus on disaster response, a greater emphasis on cooperation for disaster prevention and preparedness with particular focus on national risk assessment and risk management planning. To enhance preparedness of European countries to respond to disasters, the Mechanism created a Voluntary Pool of response capacities pre-committed by the Participating States for use in European civil protection operations (see further the paragraph below on Disaster Preparedness).

Article 1(2) clarifies the scope of the UCPM, namely that protection ensured by the Union Mechanism shall cover primarily people, but also the environment and property, including cultural heritage, against all kinds of natural and man-made disasters, including the consequences of acts of terrorism, technological, radiological or environmental disasters, marine pollution, and acute health emergencies, occurring inside or outside the Union. In the case of consequences of acts of terrorism or radiological disasters, the UCPM covers only preparedness and response actions.

The Union Mechanism is open to the participation of the European Free Trade Association (EFTA) countries which are members of the European Economic Area (EEA) and other European countries (when agreements and procedures so provide), as well as to acceding countries, candidate countries and potential candidate in accordance with the general principles and terms laid down in framework agreements or similar arrangements 12 . In addition, candidate countries and potential candidates not participating in the Union Mechanism, and countries that are part of the European Neighbourhood Policy (ENP), may benefit from some of the activities financed under the Decision 13 .

Disaster prevention

Under the UCPM, Member States shall develop and make available to the Commission their national risk assessments and national risk management capability assessments. The Commission shall provide guidance to Member States in carrying out risk assessments and facilitate the dissemination and exchange of information of good practices. This includes, in particular, improving the knowledge base on disaster risks (e.g. by developing an overview of risks the EU may face which takes into account the impact of climate change) and developing an integrated approach to disaster management of natural and man-made hazards. The Commission shall facilitate the sharing of knowledge, best practices and information related to prevention, taking due account of existing international commitments (such as the Sendai Framework for Disaster Risk Reduction 2015-2030, and the Paris Agreement), including through voluntary peer reviews. The Commission shall also finance cooperation projects in the field of disaster prevention based on yearly calls for proposals, which are published on the DG ECHO website.

Disaster preparedness

The creation of a European Emergency Response Capacity (EERC) in the form of a "Voluntary Pool" is one of the main innovations of the UCPM. Participating States can pre-commit different types of response capacities for participation in EU Civil Protection missions. The European Medical Corps (EMC) is a sub-set of the EERC through which teams and equipment from EU Member States can be rapidly deployed to provide medical assistance and public health expertise in response to emergencies inside and outside the EU. Through the Common Emergency Communication and Information System (CECIS), a web-based alert and notification application enabling real time exchange of information, the UCPM Participating States and the Commission share a common information system for mobilizing and deploying these pre-committed assets. This increases predictability of assistance while significantly reducing deployment times. Participating States and the Commission work closely together to develop quality requirements and a certification process for different modules 14 and other response capacities 15 , thus ensuring that all teams meet high quality and inter-operability standards and can effectively work together in the field.

The UCPM also supports trainings and disaster preparedness exercises to test and further improve performance and coordination. Costs to upgrade existing national response capacities and certification costs, trainings and workshops can be financed by the EU. Once certified and part of the Voluntary Pool, the transport costs of teams in response to emergencies is eligible for EU co-financing of up to 85%. Additionally, the Union Mechanism works via its Emergency Response Coordination Centre (ERCC – see further paragraph below) to: (i) monitor hazards, collect, analyse and disseminate information on disasters; (ii) map available assets in close cooperation with the UCPM's Participating States; and (iii) prepare plans for the deployment of individual experts, teams and/or assets, providing emergency communications and monitoring tools.

Disaster Response

The operational heart of the UCPM is the 24/7 Emergency Response Coordination Centre (ERCC), which is located within the Commission's DG ECHO in Brussels. The ERCC supports a coordinated and rapid response to disasters both inside and outside Europe using resources from the countries participating in the UCPM. With a capacity to deal with several simultaneous emergencies on a 24/7 basis, the ERCC facilitates a coherent and efficient European response to emergencies, aiming to reduce unnecessary and inefficient duplication of efforts. Whenever the scale of an emergency overwhelms the response capabilities of a country, the UCPM may be activated upon official request of that country or the United Nations (UN) and its agencies, as well as the International Organization for Migration (IOM), International Federation of Red Cross and Red Crescent Societies (IFRC) and the Organisation for the Prohibition of Chemical Weapons (OPCW). When this occurs, the ERCC acts as an operational hub, facilitating the coordination of assistance made available by the UCPM's Participating States to the affected country. Requests for assistance through the UCPM can concern disasters of any type, both within and outside Europe 16 . The UCPM can also be activated in response to marine pollution and other maritime disasters, on which it coordinates closely with the European Maritime Safety Agency (EMSA).

Assistance made available via the UCPM may consist of operational assets, in-kind assistance (relief items) and experts 17 . The latter can be deployed for needs assessment and coordination missions as well as to provide advice to a requesting country on prevention and preparedness measures. The Commission can also co-finance the transport costs of assistance delivered via the UCPM. This enables a rapid distribution of assistance to the affected country and results in a reduced budgetary impact on the UCPM's Participating State(s) offering the assistance. Pooling and consolidating shipments of assistance for the affected country with the facilitation of the ERCC, allows to further boost the efficiency of the European response. The ERCC fosters coordination between civil protection and humanitarian aid operations. Maintaining direct links to the civil protection and humanitarian aid authorities in Member States it enables a smooth and real-time exchange of information. It ensures deployment of coordination and assessment teams composed of humanitarian aid and civil protection experts to conduct joint needs assessments.

Cross-cutting activities

Prevention and preparedness ("advisory") missions support Participating States or third countries with technical assistance on prevention and preparedness measures through the deployment of a team of experts. These missions can be requested by a Participating State, international organisation or third country. The exchange of expert programme allows for the secondment of civil protection experts from one Participating State to another, with some third countries also being eligible to participate. This exchange allows participants to gain direct knowledge and experience on all aspects of an emergency intervention and the different approaches of national systems. The peer review programme is addressed to all the Participating States (with their different systems and backgrounds) to identify – in a spirit of mutual support and peer-to-peer learning – areas for improvement. Finally, the Decision promotes the identification, analysis and implementation of lessons learnt distilled from the entire disaster cycle.

The intervention logic (Annex 1) presents all the activities 18 carried out by the Commission and the Participating States. Activities are linked to the respective objectives/indicators (as set out by Art. 3 of the Decision) they contribute to and categorised under the three thematic pillars of disaster prevention, preparedness, and response.

7EVALUATION QUESTIONS

In order to assess the implementation and performance of the Union Mechanism and to evaluate whether it is on course to achieve its objectives, the following general questions were used to guide the analysis. The full list of evaluation questions is provided in Annex 2.

	Criteria	Evaluation question
1.	Relevance	Are the objectives the UCPM is addressing still relevant to Europe's current problems and needs?
2.	Effectiveness	To what extent have the changes/effects generated by UCPM activities contributed to achieving general and specific objectives set out by Decision 1313/2013/EU?
3.	Efficiency	Could similar/greater benefits have been achieved at lower/similar costs?
4.	Coherence	(Internal) To what extent are the UCPM pillars and activities well-articulated, interconnected and mutually supportive? (External) To what extent has the UCPM developed synergies with other policy areas and initiatives?
5.	EU added value	What results of the intervention could not have been achieved (or would have been less effective/efficient) without coordinated effort at EU level?
6.	Sustainability	To what extent are results achieved by the UCPM likely to last beyond the timeframe of the intervention?

8METHOD

This Evaluation builds on the independent evaluation conducted by an external contractor, as well as certain recent inputs received during discussions with Participating States, including during Civil Protection Committee, Civil Protection Council Working Party, and Directors-General meetings.

Overall, the design of the external evaluation was satisfactory and used a sound methodological approach. The evaluation relied on a large amount of data collected from reliable sources (agreed with the Commission during the inception phase). The combination of evidence provided by different research tools (online surveys/questionnaire, semi-structured interviews, case studies, etc.) was balanced. In addition to civil protection authorities from Participating States, other stakeholders (e.g. other ministries from Participating States, international agencies, EU services, etc.) are well represented, which adds to the credibility of findings.

Generally, the external evaluation is in line with what was foreseen in the Roadmap. The evaluator has followed the guidelines provided in the Terms of Reference (ToR), which were adopted by the Inter-service Steering Group (ISG), as indicated in the Roadmap 19 .

The external evaluation was carried out between September 2016 and June 2017 and took account of the inputs received from the Open Public Consultation (OPC) 20 . The European Court of Auditors (ECA) special report on the UCPM 21 , and the Commission's Report on progress made and gaps remaining in the EERC 22 have also provided useful evidence to some areas of this Evaluation.

The external evaluation consisted of three phases (see figure 1):

(I)inception, including a desk review of UCPM activities, scoping interviews with desk officers and conception of research tools;

(II) research, during which a targeted consultation reached out to all the main stakeholders of the UCPM 23 through online surveys, face-to-face and telephone interviews; three case studies (Cyprus Forest Fire response; Estonia Module Exercise (ModEx); Balkan route migration response) were conducted.

(III) analysis and synthesis, including analysis of UCPM performance, data triangulation, elaboration of conclusions and recommendations, and drafting an external Evaluation Report.

The above-mentioned OPC was carried out online for three months (November 2016 – February 2017), with some 130 responders from all over Europe providing feedback on the UCPM. The report on the outcome of the OPC was published at the end of March 2017 24 .

Figure 1: Overview of the external evaluation process

However, two methodological limitations should be noted:

(i) The current UCPM monitoring system does not systematically capture quantitative data of performance. As already highlighted by the recent ECA audit, the lack of robust indicators (e.g. baseline and target values) systematically monitored by the UCPM partly hindered the collection of evidence needed to formulate more definitive findings.

(ii) Limited statistical base of the external evaluation. Most of the data gathered by the contractor during the Evaluation timeframe are of qualitative nature and they refer to "perceptions" of stakeholders on the UCPM performance. While consultations have been extensive across all the stakeholders, the external evaluation did not manage to introduce and monitor additional indicators to fill this gap.

To mitigate the shortage of quantitative data monitored by the UCPM, the contractor was encouraged to collect primary data and carry out quick analyses for a sample of activities (e.g. comparing timeframes of different response operations, cost analysis for forest fire fighting missions). While the statistic sample is too limited to draw any solid conclusion, these analyses point to a number of monitoring gaps which should be addressed in the second half of the UCPM implementation. Where the contractor failed to provide sufficient indicators/data, the Commission integrated the evidence obtained via alternative channels, such as bilateral meetings with Member States and operational lessons learned (e.g. feedback from module exercises, training programme, etc.).

Within the limits of data and analysis, the conclusions in this document are well balanced, follow logically from the findings and are linkable to facts and data.

9IMPLEMENTATION STATE OF PLAY

Financial data shows that nearly EUR 131 million was allocated to the UCPM over 2014-2016. The UCPM budget components are presented in table 1 below.

Table 1: Budget allocated for the UCPM 2014-2016 (million EUR)

Source: DG ECHO

Budget line	2014	2015	2016
EUR 23030101 - prevention and preparedness - Inside the EU	27.6	29.1	28.0
EUR 23030102 - prevention and preparedness - Third countries (outside the EU)	4.0	5.5	5.6
EUR 23030201 - response inside the EU	1.1	0.7	2.5
EUR 23030202 - response outside the EU	11.1	13.6	3.09
Total	43.8	49.09	38.9

The evaluators compared the average annual allocation by activity during the periods of 2007-2013 and 2014-2016. Whilst a direct comparison is difficult between the years due to differences in budget structures, this analysis gives a rough overview of the shift in priorities for the Mechanism over different periods.

Table 2: Planned budget for the periods 2014-2016 and 2007-2013 according to budget lines
(million EUR)

Prevention

All 28 EU Member States complied with Article 6 of Decision 1313/2013 and made available to the Commission a summary of their National or Sub-national level Risk Assessment by the deadline set 25 . Based on such summaries the Commission developed an EU-wide Overview of Risks 26 . The European Commission also published Risk Management Capability Assessment Guidelines 27 to facilitate the national capability assessments that are due by end of 2018.

The total funds committed for Prevention Projects amounted to over EUR 22 million for the period 2014-2016, compared to a budget of EUR 14 million for the period 2007-2013. The average project value was EUR 610 000 (with EU funding averaging EUR 470 000), which was 37% higher than for 2007-2013. The success rate of applicants for prevention projects during 2014-2016 was 39%, decreasing from approximately 50% in 2014 and 2015 to 27% in 2016. Financed projects include the following activities: knowledge sharing (meeting, workshops, conferences, exchange programmes), studies (e.g. new methodologies and tools for risk reduction), and awareness raising activities (websites, awareness campaigns, etc.).

Preparedness

Established in 2013 to provide a coordinated and quicker response to disasters both inside and outside Europe, the ERCC replaced the previous Monitoring and Information Centre (MIC). The period 2014-2016 marked its full maturation, coordinating European assistance to West African countries in the context of the Ebola outbreak (2014-2015), and serving as an operational hub for logistics and transport in the response to the 2014 Balkan floods as well as the devastating earthquake in Nepal in 2015. Furthermore, between 2014 and 2016 the ERCC’s supported the political decision-making in the Council, particularly in the context of Integrated Political Crisis Response (IPCR) arrangements that are triggered automatically by the invocation of the Solidarity Clause 28 or in cases of 'stand-alone' activations of the IPCR. In accordance with Article 9(7) of the Decision, Participating States designated national contact points to enable communication with the ERCC in the event of occurring or imminent disasters.

Between 2014 and 2016, CECIS registered 64 Requests for Assistance, 14 pre-alerts and 15 monitoring requests. The current platform, created in 2007, will undergo several improvements between 2017 and 2019 in order to improve user-friendliness and enhance its analysis and monitoring capabilities.

The European Emergency Response Capacity (EERC) or 'Voluntary Pool' was launched in October 2014 and, by the end of 2016, included a total of 77 response capacities from 16 Participating States.

The UCPM has taken advantage of the integration of the European Flood Alert System (EFAS) and the European Forest Fires Information System (EFFIS) under the umbrella of Copernicus' Emergency Management System (EMS) 29 in 2012. In close cooperation with the Joint Research Centre (JRC) (responsible for the design of the systems themselves) and DG GROW/Copernicus (managing the early warning systems) the UCPM has provided the link with the end-users, namely civil protection authorities from Participating States and the ERCC. Similarly, the rapid mapping services allow on-demand and quick provision of geospatial information (e.g. damage assessment maps) to inform emergency response and recovery. During the 2014-2016 period, more than 30 requests for maps have been received each year.

1 680 individual experts have undergone at least one course of the UCPM Training Programme between 2014 and 2016, whereas there were 2 255 trained experts over the entire 7-year period of the previous Mechanism. The yearly average has been 740 for the current financial framework, compared to 322 during the previous one. Around 15% of the total trained experts over the evaluation period are women. A European Disaster Management Training Network was set up at the end of 2016 to improve all phases of disaster management, create synergies and develop guidance on EU and international civil protection training.

A number of disaster simulation exercises are financed every year through a Call for Proposals for full-scale exercises 30 and for module exercises 31 (ModEx). Between 2014 and 2016 8 ModEx in 7 Participating States and 13 full scale exercises in 10 Participating States have been organised. Differently from the previous Mechanism (2007-2013), ModEx exercises under the UCPM have become an integral part of the process of certification for modules which will become part of the EERC. Overall, EUR 26.1 million were allocated to exercises.

During the first three years of the UCPM (2014-2016) the committed amount for Preparedness Projects reached nearly EUR 17 million, a significant increase from the budget of the previous Mechanism (nearly EUR 10 million for the entire framework 2007-2013). The average project value was EUR 691 000 (with EU funding averaging EUR 519 000), approximately 55% higher than the average figure during 2007-2013. The success rate of applicants for preparedness projects was 42% for the period spanning from 2014 to 2016. Financed projects included the following activities: training and exercises; knowledge management (best practices; study visits; workshops; etc.); new tools and procedures for preparedness.

Response

The UCPM was activated 19 times in 2014 and 2015, and 26 times in 2016. During the Evaluation timeframe (2014-2016), on average 33% (or 21 out of 62) of requests came from inside the EU and the remaining from third-countries. A total of EUR 28.4 million in transport grants were provided for operations inside (48 grants) and outside the EU (247 grants).

Figure 2: Number of activations inside and outside EU, 2008-2016.

Source: DG ECHO, last update 24 February 2017

Figure 3: Number of experts deployed per mission, 2008-2016

Source: DG ECHO, last update 24 February 2017

Although the number of disaster response missions has not changed significantly, in 2016 the total number of experts deployed 32 (86) increased when compared to first two years of the UCPM. On average, 4.8 experts were deployed per mission in 2016. Regarding modules, statistical data shows that over the evaluation period (and before) the majority of modules were deployed outside the EU (see figure 4).

Figure 4: Number of UCPM modules deployed inside and outside of EU, 2008-2016

Source: DG ECHO, last update 24 February 2017

Cross-cutting activities

Peer reviews: Two "pilot" peer reviews were carried out in the UK and Finland in 2013 and 2014 in cooperation with the United Nations Office for Disaster Risk Reduction (UNISDR) and the Organisation for Economic Cooperation and Development (OECD). While the two pilots focused on assessing progress in national implementation of the Hyogo Framework for Action, the subsequent six peer reviews focused on more specific issues of disaster risk management relevant to the selected countries. A total of EUR 1.3 million was allocated for peer reviews over 2014-2016.

Prevention and Preparedness ("advisory") missions: Out of thirteen of such missions undertaken between 2014 and 2016, six were carried out in a post-emergency context; the remainder focused on capacity building for recovery planning (two of them were embedded in two Post-Disaster Needs Assessments, i.e. in Albania and FYRoM); and ex ante prevention measures. A total of 13 missions were carried out over 2014-2016, with EUR 230 000 allocated.

Lessons learnt: DG ECHO implemented lessons learnt and technical workshops following the activation of the UCPM on a case-by-case basis, depending on the nature of the disaster. A total of EUR 1.85 million was allocated for lessons learned over 2014-2016 33 of which EUR 598 358 were disbursed as of June 15, 2017.

Exchange of experts: The programme was implemented by an external organisation for the 2014-2016 period. Overall, 375 experts participated in exchanges within the EU and 47 experts from 11 partner countries joined the programme. Budget allocated to this activity was: EUR 900 000 for experts from Participating States and EUR 196 025 for experts from third countries.

Monitoring

The external evaluation found that existing monitoring arrangements are rather limited and may not fully capture the UCPM performance nor the follow-up required. The Commission shares this view, which applies to all of the three pillars of the UCPM. Monitoring of results is based on the four indicators listed in Article 3 of the Decision. However for most of these indicators baseline/target values are not indicated in the Decision 1313/2013 nor in the 2014 Implementing Decision 34 , which makes it difficult to assess progress. For instance, assessing "speed of intervention under the Union Mechanism and the extent to which the assistance contributes to the needs on the ground" (indicator for Objective 3) is hindered by not recording systematically via CECIS time milestones for all response operations 35 . Similar considerations apply to the lack of a standardized format for needs assessments underpinning response operations, which in some cases can hinder the assessment of whether and how assistance (especially in-kind) matched needs on the ground. The preparedness indicator set by the Decision for Objective 2, ("quantity of response capacities included in the Voluntary Pool in relation to the capacity goals and the number of modules registered in CECIS") remains vague about including the type/number of individual experts registered in the system. At the moment this analysis cannot be fully carried out on CECIS (not all UCPM trained experts are in CECIS and its research filters are very limited, e.g. not allowing research by "tags" or keywords). This makes it challenging to comprehensively map Europe's response capacity in terms of experts. In addition, the Objective 4 (risk awareness) indicator ("progress in increasing public awareness and preparedness for disasters") has not been systematically measured by the UCPM. Although the Eurobarometer surveys do support DG ECHO in its ability to assess the general level of awareness of civil protection among European citizens, this cannot be fully attributable to the UCPM (furthermore it is only limited to EU Member States and several Participating States are not covered). The section below on effectiveness (6.2) elaborates further on the limitation of the current monitoring system. The external evaluation presents additional analytical limitations of CECIS, e.g. for producing automatic reports on response operations (e.g. speed, cost, etc.) 36 . Several of these shortcomings were already highlighted in the European Court of Auditors' report on the UCPM 37 .

10ANSWERS TO THE EVALUATION QUESTIONS

11Relevance

The general and specific objectives of Decision 1313/2013 continue to be relevant to Europe's current social and economic context.

Consultations with stakeholders in the context of the Evaluation have highlighted that, over the past few years, civil protection authorities have been confronted with a changing risk landscape, which calls for new response capacities and an increased level of coordination with a much broader array of stakeholders 38 . Within Europe, new response scenarios can range from supporting traditional police functions in the aftermath of a terrorist attack (e.g. victim identification) or dealing with the consequence of a cyber-attack affecting a lifeline infrastructure, to providing relief items to an exceptionally large number of migrants.

Outside Europe, the UCPM is increasingly being activated to respond to rapid onset disasters, including a significant number of complex emergencies (see figure 5) in which civil protection assets are being used to address acute health/shelter/water and sanitation needs. As the UCPM becomes increasingly in demand, the issue arises as to its core mandate. In particular, the challenge for the UCPM is to find the right balance between being a framework of mutual assistance between European countries in the aftermath of natural and man-made disasters and evolving towards a more complex instrument capable of addressing crises of a multiple nature and with a global reach 39 .

Figure 5: Hazards having led to UCPM activation: comparison between 2007-2013 and 2014-

2016 (annual average)

Source: Emergency Response Coordination Centre (ERCC), elaborated by ICF

The UCPM's flexibility in addressing emerging or unanticipated developments and needs in the civil protection/disaster risk management arena in Europe and beyond has also been recognized by many stakeholders 40 . The activation of the Mechanism in response to the Ebola outbreak (March 2014) and to support Participating States facing an exceptional influx of migrants (2015-2017) are probably the examples that best demonstrate the UCPM's flexibility in responding to non-traditional civil protection emergencies. However, stakeholders have voiced concerns about whether the UCPM is the right tool to address emergencies that are not fully covered by Decision 1313/2013 41 .

The relevance of the UCPM is also confirmed by:

·Several recent resolutions 42 of the European Parliament calling on the Commission and the Member States to foster cooperation among civil protection authorities across Europe and to further simplify the procedures for the activation of the Mechanism in order to make it rapidly available and effective in the immediate aftermath of a disaster.

·The 2017 Eurobarometer Report on Civil Protection 43 , according to which 90% of European citizens think that the EU should coordinate the response to disasters in any of the Member States and 87% believe there should be a common civil protection policy. There is also strong agreement (86%) that the EU should help any country worldwide hit by disasters in a coordinated manner, by sending experts and equipment to affected areas.

·The Overview of Risks in the EU report 44 stresses that disasters ‘know no borders’ and there is a need to further address cross-border and "cascading"/inter-connected risks.

As far as preparedness is concerned, the following challenges have emerged from consultations with national civil protection authorities and other key stakeholders (national training coordinators and international partners) 45 .

·The training and exercise programmes require a needs-based approach which focuses on the competences necessary to implement the specific activities of the UCPM, while incorporating relevant lessons learnt. It is therefore critical to thoroughly evaluate response missions, as well as prevention and preparedness activities, in order to distil specific training needs, with a view to develop a "competence curriculum" for European civil protection experts. The UCPM's Training Policy Group has recently (March 2017) developed a new strategic framework for training and exercises 46 .

Figure 6: Number of individuals attending at least one training and number of individuals deployed

2014-2016

Source: Data from ECHO, elaborated by ICF

·Further developing the cross-border early warning systems platforms would increase its relevance for the entire Europe. Forest fires continue to take a significant toll on Europe's society, economy and environment. Recent analyses from JRC have put the average annual losses caused by forest fires to EU Member States at nearly EUR 2.7 billion/year 47 . The case study carried out by the external evaluators (Cyprus forest fires of June 2016) underlined the importance of EFFIS as a tool to assess the likelihood of locating available response capacities in other Participating States during the forest fire season 48 . Currently the ERCC is working on the possibility of mapping such capacities. One important caveat about the relevance of the trans-boundary early warning systems for forest fires and floods concerns the different needs (and use) across Participating States. The countries that already manage comprehensive early warning systems at the national level consider the EU-level system as mainly relevant for informing disaster preparedness and response in other Participating States and third countries. Countries which are still in the process of consolidating their national early warning systems tend to see it as directly relevant for their national needs, in addition to response in third countries. Furthermore, interviews with Participating States and Commission officials have shown that the end-user side of early warning systems platforms such as EFFIS and EFAS should be further developed, so as to create a community of practitioners that reflects on operational evidence emerging from disaster events and which shares knowledge, with the objective of improving practice 49 . Such a development would increase relevance of UCPM early warning platforms for most countries within Europe, regardless of their level of preparedness.

Better linking prevention with preparedness and response activities was already recommended in 2014 by the ex-post evaluation of the previous Civil Protection Mechanism (2007-2013). This need remains relevant at the mid-term mark of the 2014-2020 Mechanism, based on what the main stakeholders have expressed 50 . The need to match response capacities with the major risks in Europe will represent the key challenge to ensure continued relevance of UCPM. In particular, an effort for forward looking assessments that take into account "emerging" threats (e.g. cyber-attacks, CBRN, critical infrastructure failure, etc.) is required.

12Effectiveness

The UCPM is making good progress towards achieving the objectives set out by the legislation, most notably in improving cooperation and coordination between stakeholders in the civil protection sector.

Article 3 of Decision 1313/2013 spells out four specific objectives which, in turn, are expected to contribute to achieving the two overarching general objectives (See Annex 1: Intervention Logic).

With regards to the specific objectives of the Decision:

Specific objective 1: To achieve a high level of protection against disasters by preventing or reducing their potential effects, by fostering a culture of prevention and by improving cooperation between the civil protection and other relevant services.

The Mechanism is on track to achieve this objective. The target for the first result indicator established by the legislation ("a National Risk Assessment submitted by all Member States by end 2015") was fully achieved 51 . As far as the second indicator is concerned ("an assessment of Member States' risk management capability"), the target is due by end 2018 and guidelines were issued by the Commission 52 . From the stakeholder consultation, a consensus emerged both among civil protection authorities and other national ministries from the Participating States consulted that the UCPM has contributed towards a higher level of protection against disasters 53 . For instance, the views of stakeholders consulted by the evaluators indicate that the national risk assessments were helpful to inform risk reduction policies and plans as well as improving the understanding of risks 54 . However, while the Decision's indicators capture outputs (National Risk Assessment and National Risk Management Capability reports), additional evidence would be needed to describe the UCPM progress in achieving outcomes (preventing or reducing potential effects and fostering a culture of prevention). For instance, this would require monitoring the uptake of outputs delivered by UCPM prevention projects (e.g. how do risk assessment studies or knowledge sharing activities result in more effective cross-border policies and investments for risk reduction once the project is closed?). The external evaluation did not conduct such an analysis nor pointed to possible measures to carry it out in future. As far as the second part of the objective (improving cooperation between civil protection and other services) is concerned, the Decision does not set any specific indicator. However, the survey carried out with national civil protection authorities in the context of the evaluation shows a significant number of agencies/ministries involved by civil protection authorities in the preparation of NRAs 55 . This information can be considered as a proxy of the good level of cooperation between civil protection and other services on disaster prevention, at least in sharing risk information.

Figure 7: Survey of national civil protection authorities. Question: Which ministries and/or public agencies were consulted during the development of your NRA?
Source: ICF, Survey of national civil protection authorities

Activities that stakeholders consider as having contributed the most to this specific objective are: (i) the introduction of the national disaster risk assessment provision in the legislation and (ii) the financing of prevention projects on a yearly basis. When compared with preparedness and response, the perception of the public 56 is that results for the prevention pillar of the UCPM are less evident. A factor that needs to be taken into account is that annual financial resources allocated to the prevention pillar are only 20% of the UCPM total budget. The impact generated by UCPM prevention activities should also include what these activities have managed to leverage (e.g. additional financing, technical resources, etc.) outside of the Mechanism. For instance, prevention (and preparedness) activities financed by the UCPM could work as 'seed funds' that national and sub-national authorities could use to mobilise and/or steer additional resources for disaster risk management from other EU (e.g. Cohesion/European Regional Development Fund, Horizon 2020, LIFE programme, etc.) and international (European Investment Bank; World Bank; UN agencies; etc.) programmes. Currently, from the reporting carried out by the UCPM-funded prevention activities, it is not possible to appreciate this type of impact.

Consultations with Participating States and Commission officials also highlighted a number of challenges relating to fully capturing and deepening the impacts of the prevention activities, which will be critical for deciding how to allocate future financing 57 . These include:

·Increasing the focus of the prevention (and preparedness) projects that the UCPM finances every year. The programme, which has been running since 2000, has financed a large number of projects without always capitalising on their results. Given the relatively small size of financing (between 2014 and 2016 average funding was EUR 470 000), projects need to be targeting the needs of the end-users and need to incorporate measures for scaling up their results.

·Integrate more systematically climate change adaptation into the National Risk Assessments across the Participating States. The 2010 Risk Assessment and Mapping (voluntary) Guidelines for Disaster Management 58 issued by the Commission has asked Participating States to adopt a longer term perspective to adequately capture the potential impacts of climate change on certain types of disasters such as floods and droughts.

·Better articulating the linkages between the UCPM prevention (and preparedness) projects and broader EU programmes (e.g. Cohesion/European Regional Development Fund, etc.). A more strategic link with such programmes could help maximize the uptake of projects results and leverage additional technical and financial resources for prevention.

Specific objective 2: To enhance preparedness at Member State and Union Level to respond to disasters.

The UCPM has been very effective in achieving this objective, based on the indicator set by the Decision ("quantity of response capacities included in the Voluntary Pool in relation to the capacity goals and the number of modules registered in the CECIS"). As highlighted by a recently published Commission report 59 , substantial progress was made with the establishment and the functioning of the EERC and, in particular, by exceeding the initial capacity goals 60 for most of the modules (see Figure 8) during its first three years. As far as the "other response capacities" set by the Implementing Decision, after a somewhat slow start (2 capacities registered by July 2015), significant progress was recorded with 14 other capacities registered in the EERC as of May 2017 61 . In addition to meeting these quantitative targets, the Decision has also resulted in higher quality standards for response capacities across Europe.

Figure 8: ERCC capacity goals compared with modules registered in the EERC 62 as of 31 May, 2017

Source: DG ECHO and the Commission Implementing Decision of 16 October 2014 (C(2014) 7489)

This assessment is also confirmed by the results of the stakeholder consultation 63 . Nearly all of the civil protection authorities and more than two thirds of the national ministries consulted agreed that the UCPM has been effective in enhancing preparedness at Participating State and Union levels to respond to disasters. In addition to the EERC or Voluntary Pool, this pillar, which received 50% of the total UCPM budget, included some of the flagship activities of the UCPM, such as the training programme; the exercise programme; the functioning of the ERCC and the CECIS. Although the Decision does not set specific results indicators for training and exercises, these were recognized among the activities that contributed the most to achieving the preparedness objective of the UCPM. National civil protection authorities consider training and exercises as being generally fit for purpose, improving the capacities of Participating States, establishing common approaches and facilitating cooperation. The stakeholder consultation also highlighted the need to assess the number of slots available to Participating States for training and exercises on operational criteria (e.g. number of pooled capacities and experts, level of national capacity, etc.) other than the population 64 .

The external evaluation and its stakeholders' consultation point to a number of challenges that may affect the full achievement of this objective by the end of the UCPM implementation period 65 . These include:

·A more evidence-based approach to preparedness is required. The capacity goals set by the Implementing Decision should be revised against the main/critical risks Europe is facing and prioritized accordingly. This is fully in line with Art. 14 (2) of the UCPM Implementing Decision 66 , which requires the Commission to regularly assess and revise the suitability of capacity goals "on the basis of risks identified in national risk assessments or other appropriate national or international source of information." A close dialogue with Participating States is critical in order to capture all national capacities, including those which are not registered in the EERC but may still be deployable via the UCPM. Similarly, the spatial distribution of (natural and man-made) risks across Europe should inform the geographical distribution of response capacities.

·The need to re-assess the system of incentives for pooling assets in the EERC based on lessons learnt and other knowledge emerging from operations. A significant part of these incentives concern the level of co-financing for the deployment of EERC capacities or the mobilisation of buffer capacities; providing health and third-party liability insurance coverage for experts deployed. The deployment of EERC capacities, particularly for long-term missions outside the EU, would certainly benefit from the availability of more ‘robust’ EU co-financing possibilities. The administrative burden involved in registering and certifying modules also ranks among the main concerns of Participating States, although this is being addressed by the Commission with a more streamlined and coherent planning process for exercises, certification and registration of assets in the EERC.

·The training and exercise activities of the UCPM require a strategic approach focusing on results. Some of the consulted Participating States have highlighted the need to move from a system focused on outputs (i.e. number of individuals trained) to addressing operational needs that stem from operations. This is currently being addressed with the Strategic Framework for Training and Exercises, recently launched by the Commission, which is focused on competences. Furthermore, a robust framework for evaluating the training courses administered as well as the individuals trained is considered as a priority in order to match the needs on the ground with the best technical resources available. Similar considerations were raised with regards to the exercises.

Specific objective 3: To facilitate rapid and efficient response in the event of disasters or imminent disasters

Overall, the UCPM has made significant progress towards achieving a rapid and efficient response to disasters 67 . Stakeholders across all categories, including those who participated in the Open Public Consultation, noted as one of the key achievements the improved coordination and flow of information at EU-level through the scaling-up of the ERCC and CECIS during the period 2014-2016. Civil protection authorities were generally satisfied with the ability of the UCPM to respond to disasters within the Union, a vast majority considering that information provided to countries for deploying assistance was comprehensive, sufficiently detailed, based on the needs on the ground and provided in a timely manner 68 . Three quarters of civil protection authorities consulted also considered response coordination to be smooth and timely.

Outside of the EU, UCPM response was considered "broadly effective" by the recent audit of the ECA 69 , which focused on three disasters (floods in Bosnia and Herzegovina, Ebola outbreak in West Africa and Nepal earthquake) to which the UCPM responded. According to the ECA, the UCPM activation was timely and, in each of the three cases observed, the EU Civil Protection Team facilitated coordination on the ground with the Participating States' response teams. The coordination among different Commission services, as well as with other EU and non-EU bodies, was inclusive (see also Section 6.4 on coherence). The Commission also respected the UN’s overall lead, and took steps to ensure a smooth transition into the recovery phase. Response missions outside the Union were deemed effective also by international actors (UN OCHA, WFP, IOM, UNHCR, etc.) participating in the stakeholders' consultation 70 . The importance of gaining a better/broader understanding of the context and partners on the ground was highlighted by some of the international stakeholders 71 . This goes in the same direction as the ECA's recommendation to involve more systematically DG ECHO personnel in the field during UCPM response missions.

In order to complement the perception of stakeholders with statistical evidence on the speed of UCPM response to comparable disasters inside Europe (e.g. forest fires), the external evaluation looked at response times for selected emergencies based on data recorded in CECIS 72 . While the analysis carried out in the external evaluation report is too limited to allow any solid conclusion on the speed of response, it pointed to a discrepancy between the indicators set by the UCPM Implementing Decision and the data that are currently recorded (in some cases manually entered) in CECIS. In particular, while the only speed indicator that the legislation (Implementing Decision 16 October 2014) refers to is the "availability" of each response module, the parameter monitored by the ERCC is the "departure time". Therefore, there is a need to harmonize indicators in order to allow comparisons and measure whether there is any progress in the speed of response. In addition to identifying and harmonizing the indicators, as already highlighted by a recommendation of the ECA, statistics and reports based on CECIS data should be automatically generated and analysed.

Moreover, the lack of baseline and target values (i.e. what can be considered as a "rapid response" to a forest fire/earthquake/flash flood/etc.?) is an obstacle to appreciating effective progress in making disaster response speedier. Similar challenges exist for the second dimension of this indicator ("efficient response"), as explained in section 6.3 on Efficiency.

In addition to adequately monitoring the speed of response, the external evaluation raised other challenges to fully measuring and, possibly, achieving the results under this specific objective:

·Needs assessments in the Requests for Assistance are often not available and/or not collected in a standardised format. Due to this, the second indicator set by the Decision for assessing disaster response, ("extent to which the assistance contributes to the needs on the ground") can only be measured based on perceptions and other qualitative input provided by the stakeholders. In this context the external evaluation noted a low level of awareness among Participating States of the common template for request for assistance, which was agreed in the context of the Host Nations Support Guidelines 73 .

·When facing imminent disasters, the pre-deployment of a small UCPM team ahead of the formal activation should be considered, as this could prepare the ground for the following deployment and make a significant difference to accelerate UCPM response.

·Further simplifying administrative and financial procedures for engaging response capacities under the UCPM which, according to some Participating States, in some cases is less rapid than bilateral channels. A number of operational 'quick fixes' have already been identified by the users (e.g. integrating in CECIS the transport co-financing request form; revising the minimum threshold for individual transport grant applications; etc.) and could be addressed rapidly.

·Ensuring the best match between operational needs and experts deployed in the EU Civil Protection Team. Consulted civil protection authorities found the Terms of Reference produced by the ERCC for selecting experts (their use became increasingly systematic since 2015) helpful but raised some concerns on the selection process of the experts, which could still be improved.

Specific Objective 4: "To increase public awareness and preparedness for disasters"

The UCPM has contributed to this objective mainly with the dissemination of good practices sharing and other knowledge management activities (e.g. Civil Protection Forum 2015, 3rd edition of the World Reconstruction Conference, establishment of the online-based Disaster Risk Management Knowledge Centre, Guidelines on Risk Communication, etc.). When compared to the other three, stakeholders consulted in the context of the external evaluation feel that progress under this objective has been less advanced. This seems also in line with the trend showed by the 2017 Eurobarometer survey, whereby over the past years a significant part of the surveyed population (42%) is not aware of EU civil protection work. Boosting the UCPM visibility (for instance during exercises, as observed by the evaluators during one of the case studies) to European citizens would also contribute to increasing the profile of the public awareness activities supported by the UCPM.

Finally, there is another factor related to the achievement of this specific objective which should be taken into account in the period leading up to the end of the UCPM implementation framework. The activities carried out by the UCPM (mainly complementing the work carried out at the national level by civil protection authorities) and the UCPM indicator for this objective (level of awareness of Union citizens of the risks in their region) are on two different levels. This mismatch can make it difficult to find a direct link between the awareness raising and knowledge management activities financed by the UCPM and the increased level of awareness among the population at the local level. No baseline or target has been set for this risk awareness indicator which, moreover, has not been surveyed by the 2017 Eurobarometer report on Civil Protection.

General Objectives:

·To strengthen cooperation between the Union and Member States and facilitate coordination in the field of civil protection in order to improve the effectiveness of systems for preventing, preparing for and responding to natural and man-made disasters; and

·To promote solidarity between the Member States and other participating countries through practical cooperation and coordination

Overall, based on the extent to which the specific objectives are being achieved, the UCPM has been successful in progressing towards the general objectives of the Decision. National civil protection authorities and other stakeholders agree that the UCPM contributed to improving cooperation in the civil protection sector within Europe, in particular with regards to disaster response and preparedness. The creation of a well-established system of civil protection authorities and a network of colleagues who know each other across all the Participating States was instrumental to enhance communication and information exchange. Survey results and interviews also suggest that the UCPM contributed to improving the effectiveness of systems for preventing, preparing for and responding to disasters 74 . The legislation does not set specific result indicators for the General Objectives and the external evaluation did not suggest potential indicators to use as proxies.

13Efficiency

Generally speaking, the stakeholders' perception of the UCPM's efficiency is very positive, particularly with regards to emergency response and, overall, administrative costs associated to implementing the UCPM decision are considered acceptable by most Participating States 75 .

While the efficiency question encompasses the entire spectrum of UCPM activities, it is particularly critical for the response pillar since one of the specific results indicators set by the Decision refers to "promoting rapid and efficient response". The external evaluators' analysis of efficiency focused on transport provision under the UCPM. In particular, the external evaluator attempted to carry out a cost analysis of transport grants requested by Participating States for comparable response operations, namely aerial forest fire fighting, water purification in the aftermath of floods and in-kind assistance in the context of the refugee crisis. Preliminary evidence 76 collected in the context of the evaluation points to some discrepancies in the net contribution requested by Participating States for comparable types of services (cost/mile ratio). This is the case for Canadair aircraft operations carried out by different Participating States, where the cost per mile ratio seems to vary significantly (even though the aircraft used is similar) 77 . Similar discrepancies also emerge from the net contribution granted to two Participating States providing a water purification module during a flood response operation inside Europe and from transport of in-kind resources by truck or plane. Although the analysis carried out by the external evaluators remains preliminary due to the limited data sample considered, they point to the need to systematically collect cost data and run analysis of comparability of transport reimbursement. Similarly, a robust analysis of transport costs for in-kind assistance goods and/or capacities via the EC broker or framework contractors could help in order to look for any significant variation and determine whether EU transport co-financing is always the most cost-efficient option. Similarly, the external evaluation highlights that useful data concerning costs remain locked in paper forms instead of being recorded in CECIS and analysed 78 . Such analysis could lead to determining benchmarks (lower/higher bounds or ranges) for cost of deploying certain assets. In some cases this information could lead to quicker approval of deployments and to faster reimbursement processes.

As far as efficiency of UCPM-financed activities, the following challenge emerged from the external evaluation:

·The UCPM needs to monitor more systematically and consistently cost and speed of response missions. This can help establish baselines/benchmarks (lower/higher bounds or a range) allowing comparisons between emergencies responses and produce analyses needed to improve efficiency of future performance. Moreover, rigorous monitoring arrangements could also help speed up the reimbursement process (for instance via pre-agreed ceilings).

14Coherence

The UCPM has achieved internal coherence between its different components and a good level of cooperation with other policy areas.

Internally, a number of cross-cutting activities (e.g. lessons learned, advisory missions, peer reviews, exchange of experts, etc.) within the UCPM provide for an integrated prevention-preparedness-response approach and therefore help different activities of the UCPM reinforce each other. This is confirmed by the views collected through the stakeholder consultations with Participating States (see figures 9-10), where a vast majority of the responding national civil protection authorities and ministries agreed that the UCPM promoted sufficient linkages and synergies across its three pillars 79 .

Figure 9: Stakeholder consultations.

Source: ICF, Survey of national civil protection authorities from 27 Participating States

Figure 10: Stakeholder consultations.

Source: ICF, Survey of National Ministries (other than civil protection) from 13 Participating States

Consultations with stakeholders in the context of the external evaluation also highlighted a number of significant challenges for strengthening the internal coherence of the UCPM through its cross-cutting activities 80 .

·There needs to be a continuous feedback loop between lessons learnt and the implementation of activities (including prevention and preparedness ones, in addition to response). Participating States have highlighted that some specific activities (e.g. training and exercises) would benefit from a stronger link with UCPM operations, for instance focusing on specific challenges arising from the deployment of modules in the field (e.g. custom clearance in fragile and conflict-affected states; cooperation with host countries; integration in broader international frameworks for disaster response and recovery planning; etc.). To this end, there is a shared feeling that the lessons learnt process of the UCPM needs to be more structured and systematic from identification and throughout implementation (on both the Commission and Participating States sides), with regular monitoring and transparent reporting on the progress.

·The UCPM training and exercise programme should promote a more coherent approach between training and deployment. Training should be focused on the competences needed during deployment (including for preparedness and prevention activities). In this respect, as confirmed by one of the case studies 81 , the link between training and exercises is critical and should be strengthened. Consultations with Participating States suggested potential synergies between training (particularly the most advanced courses) and ModEx or full-scale exercises, in order to enhance operational preparedness of experts 82 . Some of these recommendations are already being addressed by the Commission.

·Peer reviews should ensure a good match between reviewers and the needs of reviewed countries (i.e. between profile/competences of the reviewer and specificities of the country/sector to be reviewed). Additional linkages could be explored between training and the deployment of experts on peer review missions. This may include a process to certify experts' technical and operational competence in order to better address the needs of the reviewed countries.

·Prevention and preparedness ("advisory") missions should build on a well-formulated, result-oriented approach. A structured process needs to be put in place to prepare and implement these missions, be it in preparation for potential disasters (e.g. assessment of national emergency management capacity) or, in the aftermath of a disaster, to inform recovery and reconstruction strategies. Synergies with other EU-funded programmes (in particular the Prevention Preparedness and Response to Natural and man-made Disasters (PPRD) and the Instrument for Pre-Accession (IPA) programmes) and/or Participating States engaged in third countries should be strategically considered, with a view to provide additional resources (technical and financial) needed for implementing recommendations.

In general, the consultations highlighted the need for a more result-oriented focus for the cross-cutting activities of the Mechanism (e.g. lessons learned, peer reviews, advisory missions, exchange of experts) 83 . This includes focusing on measurable policy and/or operational results which should be achieved, in a given timeframe by the beneficiary. In some cases, activities conducted in response to a request from a Participating States or a third country (e.g. advisory missions) could be conditioned to the requesting country meeting some policy/capacity targets prior to the UCPM fielding a technical assistance mission.

As regards the external coherence, the external evaluation mapped the extent to which the UCPM has been pursuing synergies with other policy areas and instruments in the period 2014-2016. When compared to the final evaluation of the Civil Protection Mechanism (2007-2013), the evaluators noted that the current Union Mechanism is cutting across a larger number of policy areas. Although the degree of coordination could only be assessed on the basis of qualitative input provided during interviews and mostly at Commission level, a higher degree of coordination with other areas 84 was observed, compared with the previous Mechanism. The level of cooperation between the UCPM and other policy areas related to civil protection, based on the external evaluation (online survey of national civil protection authorities and interviews with several EC/EU services) is presented in figure 11 below.

Figure 11: Current status of coordination between UCPM and other policy areas. Red arrows show growing importance of the policy area and need for inter-institutional cooperation. Green arrows mean that inter-institutional cooperation has grown over the period 2014-2016.
Source: ICF External Evaluation (based on stakeholder consultation)

During the first half of its implementation, the UCPM strengthened its linkages with key policy areas. These include: research and innovation (including the Disaster Risk Management Knowledge Centre and the Copernicus rapid mapping services); regional policy (ex-ante conditionality to cohesion funds on disaster risk management and policy coordination with the cohesion policy instrument); environment (integration of hydrological and civil protection early warning systems for floods under the Flood Directive and, to a lesser extent, promoting the UCPM to support implementation of the Seveso Directive), climate adaptation (sharing of knowledge under the Disaster Risk Management Knowledge Centre and, to a lesser extent, integration of climate change in the national risk assessments); health (cooperation with DG SANTE and the European Centre for Disease Prevention and Control (ECDC) on health-related activities, including the European Medical Corps), neighbourhood and enlargement policy (in cooperation with DG NEAR) as well as coherence and complementarity between CP, humanitarian and development assistance.

As far as the neighbourhood policy is concerned, further coherence between UCPM and DG NEAR activities in Partner countries is needed. In particular, coordination between the yearly Prevention and Preparedness Calls for Proposals of the UCPM and the disaster risk management related activities of the PPRD was found to be insufficient by the 2017 final evaluation of the PPRD-East II 85 . The evaluation called for enhancing coordination between DG ECHO and DG NEAR on their respective programmes. Similarly the 2017 final evaluation of the PPRD-South Programme 86 underscored the need to provide to both decision- makers and practitioners in partner countries with more knowledge on the UCPM and the opportunities it offers them in prevention/preparedness/response. The IPA programme, would benefit from further integration in the UCPM 87 . This includes, among others, participation in UCPM exercises and increased awareness of UCPM procedures; integration of IPA-supported multi-countries response modules in the EERC; and joint efforts with UCPM to strengthen the governance of the prevention activities (e.g. on the Disaster Risk Assessment and Mapping initiative).

Recent developments affecting Europe's security (mainly related to terrorist attacks) have also prompted the security policy as a priority area for cooperation with civil protection. The "Security Union" initiative launched towards the end of 2016 aims at providing a platform for cooperation across several EU instruments, including the UCPM and is expected to take on a primary role in the second half of the UCPM implementation. The approach taken by the UCPM-Security Union coordination goes beyond the exchange of information and focuses on a common agenda with specific milestones. This result-oriented approach could possibly be replicated in other areas.

Finally, as far as the response outside of Europe is concerned, the comparative advantage of civil protection intervention vis-a-vis humanitarian assistance needs to be carefully considered as the UCPM becomes increasingly activated for dealing with humanitarian and complex emergencies (see section 6.1). In particular, the UCPM can fill important emergency gaps in cases where it is not possible to fully deploy humanitarian teams or where specific technical expertise and/or assets are needed. In the same vein, linkages with other prevention, preparedness or response activities outside of Europe should be ensured from early stages of intervention under the UCPM. For example, in the aftermath of the Nepal earthquake the support offered through the UCPM (e.g. structural engineering, water purification systems) was followed by the development of a Post Disaster Needs Assessment (PDNA), which became the recovery and rehabilitation framework for the country. In such situations, the assessments carried out by civil protection experts deployed by EU Member States or the Commission should feed into wider recovery frameworks conducted by partner countries governments, with support from the international community.

15EU Added Value

The UCPM provides Participating States with cooperation opportunities and increased capacity – and predictability – in terms of expertise and resources that would not be possible without coordinated effort at EU level. The vast majority of Participating States consulted see an added value in EU-level action under the UCPM as opposed to bilateral actions or to the action of international organisations or NGOs 88 .

Prevention: Consulted stakeholders noted the importance of the Decision's provisions on prevention (e.g. National Risk Assessments) as a "pull factor" to develop throughout all the Member States policies and strategies addressing disaster prevention. Furthermore, the introduction of European Commission guidelines for risk assessments had a very significant impact on this pillar of the UCPM, contributing to the first-ever overview of risks in the EU. Prevention and preparedness projects, which must necessarily involve three or more Participating States, deliver an inherent cross-border value by facilitating the establishment of informal channels between national authorities having to manage common risks. Most of the Participating States consulted by the external evaluators do not believe these projects would have been carried out in the absence of UCPM funding or, if they had been, they would have involved duplications and fragmentations 89 .

Figure 12: Stakeholder consultations.

Source: ICF, Survey of Project Leads of Prevention and Preparedness Projects from 12 Participating States.

A more robust UCPM monitoring framework (as explained in section 6.2 on Effectiveness) that, for instance, allowed to keep better track of outputs and outcomes of cross-border prevention projects (e.g. number of cross-border risk assessments developed by the UCPM, or number of macro-regional development plans that take into account findings of risk analysis developed by the UCPM) would also help to better appreciate and communicate to stakeholders and the public the added value of EU action in the area of civil protection and disaster risk management.

Preparedness: EU assistance to Participating States in organising resources through the creation of the EERC and the intervention modules is considered by stakeholders as the most tangible (and visible) added value generated by the UCPM 90 . The overall capacity and deployment capabilities (e.g. training and inter-operability) of the EERC is larger than what would have been possible through the sum of the national capacities. Voluntary pooling of Participating States' assets on standby has facilitated the planning of operations and added an element of predictability, thus enhancing EU-wide contingency planning. In addition to making response capacities more adaptable to the international context, most of the consulted countries agree that promoting common European standards for modules that join the EERC has contributed to enhance their quality and robustness 91 . Similar considerations apply to training and exercises, which have contributed to create a network of EU civil protection professionals sharing common operating procedures and standards throughout Europe. To this end, the exchange of expert program is also considered as being a valuable instrument to provide to a large number of civil protection experts opportunities to share knowledge and acquire new skills while belonging to an international network of civil protection professionals.

Response: The evolution of the MIC into a single "hub" for information sharing and operational coordination, the ERCC, was considered by stakeholders to be the most tangible added value for European and third countries. Managing multiple bilateral offers at the same time and without ERCC coordination would be more cumbersome/costly and would bring lower visibility. Another significant value brought by EU-level action in the area of civil protection is the possibility to leverage satellite systems (e.g. Copernicus) for making available trans-boundary hazard early warning systems and to inform disaster response operations (e.g. providing damage assessment maps).

Preliminary recommendations by the external contractor 92 to further increase the added value of the UCPM interventions echo those enshrined in the recent ECA report, i.e. shortening the process for selecting experts for deployment and better capture UCPM performance by systematically monitoring quantitative indicators for speed (e.g. time needed to respond to a request for assistance) of response. Tracking the cost-effectiveness of EU-level action (e.g. savings generated by well-coordinated response operations) could also demonstrate another significant added value.

16Sustainability

The sustainability of the UCPM's results is being ensured to a certain extent through the acquisition of knowledge and skills by experts participating in different UCPM activities. Consultations with Participating States highlighted the importance of training, exercises and exchange of experts and of fostering cross-border prevention and preparedness 93 . A critical mass of trained European civil protection experts and practitioners enables the uptake of knowledge and tools made available by the UCPM.

Sustainability of prevention and preparedness projects financed under the UCPM had been flagged as a potential issue by the ex-post evaluation of the Civil Protection Mechanism (2007-2013). Lack of trans-national coordination mechanisms and insufficient follow-up funding very often limited the uptake of project results at national level. The survey of stakeholders carried out for the external evaluation of the UCPM shows a better perception of sustainability of such projects which, according to the views of the project leads, are likely to be continued and/or scaled up with national, regional or other EU funding 94 . Compared to past years, progress was also helped by more systematic publication of project results via the UCPM website managed by DG ECHO. However statistical evidence is very limited and uncertainties remain on the effective use of results by end-users as well as on the public/private sector hand-over after project closure.

The following issues were highlighted by the stakeholders consulted in the context of the external evaluation and during discussions with the Commission as critical challenges that may affect the sustainability of UCPM activities and their results in the coming years 95 .

·A viable financial model to ensure availability of capacities in the EERC. Providing realistic economic incentives to Participating States for pooling their assets and developing new response capacities is critical to ensuring a durable pool of capacities in the EERC. At the same time, creating bigger incentives to pooling assets should not come at the detriment of Participating States ' investments in national response capacities.

·An efficient strategy to build the European civil protection human resource capacity. New ideas and approaches will be needed to ensure that UCPM foundation courses be rolled out extensively without creating duplications with the training program of Participating States, while more specialized/advanced training (e.g. training for deployment) be delivered to well-identified groups of experts. Matching competences with needs is critical to make sure the UCPM fulfil its mission of supporting its Participating States.

·The role of the private sector in the implementation of UCPM activities. Several Participating States have highlighted the potential risks arising from private companies carrying out needs assessments, advisory missions, peer reviews and other activities that provide commercial actors with access to potential market opportunities 96 . Measures should be introduced to guarantee that such companies are not in an advantageous position vis-a-vis other private competitors or the public sector itself when participating in a call for tenders. Furthermore, there may be a need to protect confidentiality and secrecy of sensitive information in some sectors (e.g. national security threats; critical infrastructure; critical capacity gaps; etc.).

·Bringing the UCPM closer to the European citizens while guaranteeing support to the national level. Subsidiarity is one of the core principles of the UCPM, whose primary function is to support Participating States' responsibility to protect their citizens and assets. Being relevant to the civil protection/disaster risk management actors in the front line (regions and other territorial authorities) while strengthening the coordinator/broker role of national civil protection authorities is a critical challenge for the sustainability of the UCPM. Similarly, the UCPM needs to promote the bi-lateral, tripartite dimension of cooperation between countries (e.g. neighbouring countries) while ensuring its pan-European approach and values.

·Maximizing the uptake of projects results (e.g. prevention and preparedness projects, exercises). UCPM-financed projects need to be designed and monitored with a view to focusing on the uptake of project results by end users. There is a need to shift the focus from the output (e.g. the delivery of a risk assessment, a workshop or a new training methodology) to the outcome (the actual change generated by that product, e.g. a new infrastructure project that takes into account the results of that risk assessment or workshop, or the use of that methodology by the trained experts in a real-life situation).

17CONCLUSIONS

Based on the findings of the external evaluation and taking into account the results of recent analytical work on the UCPM 97 and its own operational knowledge, the Commission's overall conclusion is that the implementation of UCPM activities is well underway and on track to achieve its objectives by the end of the legislative timeframe (December 2020). The performance of the UCPM during the first half of its timeframe has showed encouraging results in each of the three pillars (prevention/preparedness/response) and overall positive views from the Participating States and the main stakeholders. At the same time, the Evaluation has also highlighted critical challenges that, if addressed promptly, may further increase the impact of the UCPM. A general area for improvement is the strengthening of the UCPM results monitoring framework, through the possible introduction of quantitative indicators which capture more adequately progress achieved under the three pillars of the UCPM.

There is sufficient evidence to demonstrate the continued relevance of UCPM objectives to Europe's needs. The results of the 2017 Eurobarometer survey on Civil Protection show that a vast majority of European citizens support a common EU policy in the field of civil protection and agree that the EU should lead the coordination of disaster response to affected countries, both in Europe and in the rest of the world. Recent resolutions of the European Parliament have also called on the Commission to further support cooperation among European civil protection authorities. The stakeholders consulted also highlighted the flexibility of the UCPM to rapidly adapt and address newly emerging crises which were not "traditional" civil protection emergencies (e.g. the Ebola outbreak in West Africa and the migrant crisis in Europe). The increasing number and scope of UCPM activations outside of Europe pose, however, the challenge of finding the right balance between being a framework of mutual assistance between European countries in the aftermath of natural and man-made disasters and evolving towards a more complex instrument capable of addressing crises of a multiple nature and with a global reach.

An analysis of the UCPM effectiveness highlighted encouraging progress under each of the specific objectives. The UCPM is fully on track to achieve objectives pertaining to disaster prevention framework (objective 1) and readiness for disasters (objective 2), based on the indicators set by the legislation and on feedback received by key stakeholders consulted. The UCPM's response to disasters (objective 3) has been timely and effective according to the vast majority of stakeholders consulted. Confirming one of the findings of the ECA's review of the UCPM, the external evaluation struggled to find robust and consistent quantitative data (e.g. speed, needs assessments) to corroborate the stakeholders' very positive views on UCPM disaster response. Lastly, public awareness and preparedness to disasters (objective 4) have received slightly less satisfactory views from stakeholders when compared to the other objectives. In this respect, it is questionable whether the specific indicator ("level of awareness of Union citizens of risks in their regions") set by the Decision for this objective can be fully attributable to the level of activities carried out by the UCPM.

The external evaluation also underscored several challenges which, if promptly addressed, could further increase the effectiveness of the UCPM. These include, among others: establishing a baseline/benchmarks (e.g. speed, cost/effectiveness, etc.) for comparable types of emergencies; developing a more evidence-based process to set capacity goals for disaster preparedness; revisiting the system of incentives to pool response assets and develop new capacities in the EERC. Disaster prevention is considered a critical area that may require additional resources in future (currently only 20% of UCPM budget) although a more robust impact evaluation may be required to establish where the UCPM action to date has been most effective in assessing/reducing disaster and climate risks. Cooperation and coherence between the UCPM and the EU climate change adaptation agenda should be enhanced. Overall, closer linkages between UCPM activities in prevention and preparedness and broader EU-wide programmes (e.g. Cohesion/European Regional Development Fund, European Investment Bank, etc.) should be explored.

Generally speaking, the stakeholders' perception of the UCPM's efficiency is positive; particularly that emergency response and administrative costs associated to implementing the UCPM decision are considered acceptable by most Participating States. The external evaluation has attempted to carry out analysis of costs for comparable response missions (e.g. aerial forest fires missions; transport of modules and in-kind assistance; etc.). Although evidence cannot be considered sufficient to draw final conclusions, this confirmed the need to systematically carry out cost analyses and, possibly, determine cost benchmarks (or ranges) for certain type of missions/transport.

Internal coherence is deemed satisfactory by the external evaluation. Several cross-cutting activities of the Mechanism (e.g. lessons learned, advisory missions, peer reviews, exchange of experts, etc.) provide for an integrated prevention-preparedness-response approach. Externally, linkages between the UCPM and other policy areas have increased compared to the previous framework (2007-2013). This was confirmed by interviews with a large number of EU services. Focusing on a result-based agenda and milestones is seen as conducive to a more effective cooperation and it is felt that enhanced coordination mechanisms need to be considered in order to improve synergies between instruments funding prevention and preparedness activities in Neighbourhood and Pre-Accession countries 98 . As far as response outside of Europe is concerned, the comparative advantage of civil protection intervention àa-vis humanitarian assistance needs to be carefully considered as the UCPM becomes increasingly activated for dealing with humanitarian and complex emergencies.

The existence of a single European "hub" for information sharing and operational coordination, the introduction of common European standards for disaster response capacities and common guidelines on risk assessments, which have contributed to the first-ever overview of risks in the EU, are perceived as having delivered high EU added value in the areas of disaster response, preparedness and prevention, respectively. Increased capacity and predictability of emergency response capacity provided by the UCPM would not have been possible without coordinated effort at EU level. Over the next years, introducing indicators to monitor the effectiveness and efficiency of the UCPM more adequately may provide additional evidence to appreciate (and communicate) the added value of EU-level action in civil protection.

Finally, knowledge-sharing and training activities are critical to creating a community of European civil protection practitioners and, ultimately, to ensuring sustainability of the UCPM's results. However, at a system level, several issues are considered as critical for the sustainability of the UCPM going forward. A viable financial model needs to ensure availability of capacities in the Voluntary Pool; an efficient strategy to build the European civil protection human resource capacity needs to be rolled out; the role of the private sector in the implementation of UCPM activities needs to be clarified; end-users needs need to inform design and monitoring of prevention and preparedness projects financed by the UCPM, in order to maximize their impact beyond the project closure (impact should be evaluated one or two years after funding has ended). The UCPM needs to get closer to the territorial actors while supporting the role of the national civil protection authorities in cementing the risk management governance all along the European-national-regional-local chain.

The results of this Evaluation provide a good analytical basis to start reflection on a future framework. The Commission invites the European Parliament and the Council to take note of these evaluation findings.

ANNEX 1: INTERVENTION LOGIC OF THE UCPM

ANNEX 2: PROCEDURAL INFORMATION

A) Lead DG: European Civil Protection and Humanitarian Aid Operations (ECHO)

B) Reference in Decide: 2016/ECHO/007

C) Organisation: The external evaluation was supervised by an Inter-service Steering Group (ISG), composed of representatives of the EEAS and the following European Commission (EC) DGs: CLIMA, DEVCO, ECHO, ENER, ENV, HOME, JRC, MOVE, NEAR, RTD, SANTE and SG. The external evaluation contract had a length of 9 months, from October 2016 to June 2017. Four meetings were held:

·9 November 2016: Kick-off meeting

·8 December 2016: Inception report meeting

·3 March 2017: Interim report meeting

·20 April 2017: Draft final report meeting

D) Quality assurance:

·The ISG assessed the external draft Final Report against the following criteria: compliance with technical specifications; accuracy and completeness of data; relevance and soundness of analysis; robustness of conclusions; usefulness of recommendations; general clarity of the report.

·The ISG provided the following main comments to the external draft Final Report:

oCertain missing elements should be provided: a chapter explaining the intervention, a chapter on the evaluation methodology, the case study reports, a proposal on the dissemination plan…;

oThe soundness of the arguments should be improved, and triangulation should be more evident;

oConclusions on cost-effectiveness should be better supported on quantitative data;

oConclusions on effectiveness, coherence and EU added value should be better elaborated;

oSome links between conclusions and recommendations should be strengthened;

oRecommendations should be further tailored and detailed;

oThe general clarity of the report should be improved, and the report should be shortened.

These comments were fully taken into account in the final version of the report.

E) Evidence: Several information sources were used to reach triangulated conclusions, i.e. document review; analysis of the EC's project database (‘HOPE’); face to face and telephone interviews; four surveys (to national ministries, civil protection authorities, Trainers and national training coordinators, civil protection experts and project coordinators); an Open Public Consultation, and case studies.

F) External expertise: The evaluation was carried out by an external consultant, ICF Consulting Services Limited, who was selected through a competitive process under DG ECHO's evaluation framework contract. The current Staff Working Document aims at providing a self-reflected summary of all sources of evidence, including the evaluation carried out by the external consultant, about the first years of existence of the current Mechanism. The report is available at: http://bookshop.europa.eu .

LIST OF EVALUATION QUESTIONS

Effectiveness

ØQuestion 1. To what extent have the objectives of the Decision been achieved?

To what extent has the Union Mechanism contributed to achieving the general objective of the Decision? To what extent has progress been made in relation to the four specific objectives? Given the current timeframe, is the Union Mechanism on track to achieve the specific objectives of the Decision? To what extent have external factors influenced the observed progress towards the objectives?

Efficiency

ØQuestion 2. To what extent are the costs of the activities performed under the UCPM justified when compared to their benefits?

To what extent did ECHO achieve cost-effectiveness in its actions under the UCPM? To address this question the evaluator is invited to propose a dedicated, overall approach. What main factors influenced cost-effectiveness of the actions? Were there significant differences in costs and objective achievement between the Participating States? If so, what are the underlying causes? Which decisions could contribute to improve cost-effectiveness?

ØQuestion 3: To what extent have the monitoring arrangements contributed to an efficient and effective implementation of the intervention?

Are the indicators currently set by the Decision adequate and sufficient to capture to monitor successful implementation of the Decision? Which, if at all, additional indicators should be considered? Has data been properly collected and monitored? Is data generally available?

Relevance

ØQuestion 4: To what extent are the Decision's objectives still relevant to the needs identified and to the European Commission's priorities for 2014-2020?

To what extent do the general and specific objectives of Decision still correspond to current needs within the EU? To what extent have the general and specific objectives of the Decision proved to be appropriate to address the issues highlighted in the Impact Assessment (2011) Were the main findings and lessons learnt presented in the Ex-Post Evaluation of the Community Civil Protection Mechanism (2007-2013) taken into account for the implementation of the UCPM? Has the Mechanism been flexible enough to address emerging/unanticipated critical issues in the disaster risk management arena? To what extent has the UCPM integrated in its functioning as well as in its activities scientific and technological research and development that has become available since its creation?

Coherence

ØQuestion 5. To what extent is this intervention internally and externally coherent?

How well are the different activities of the UCPM articulated, interlinked, and mutually supportive? What examples of synergies exist between different activities and areas (i.e. prevention/preparedness/response) of the UCPM? Can further synergies be sought? Are there any missed opportunities? To what extent are the potential synergies between the UCPM and other EC policy areas as well as international frameworks/initiatives being exploited?

EU added value

ØQuestion 6. What is the EU Added Value of the UCPM, including the added value to other participating states?

What results of the intervention can be identified that could not have been achieved without a coordinated effort at the EU level? Which criteria could be considered to demonstrate EU added value of the UCPM-supported interventions in prevention, preparedness, and response?

Sustainability

ØQuestion 7. To what extent are the results achieved by the UCPM sustainable?

To what extent are the results achieved by the UCPM likely to last beyond the timeframe of the intervention? Are adequate arrangements in place to ensure (financial, capacity, technical, etc.) sustainability of current and future interventions?

ANNEX 3: OVERVIEW OF ACTIVITIES CARRIED OUT UNDER THE UCPM (2014-2016)

Prevention	Preparedness	Response
Improving the knowledge base for disaster prevention	Emergency Response Coordination Centre (ERCC)	Grants for transport and related logistics, inside the Union
EU overview of risks	Early warning and analysis	Grants for procurement, inside the Union: (1) Dispatching expertise (2) Transport and related logistics
Prevention projects	Preparedness projects	Grants for transport and related logistics, outside the Union
	Training	Grants for procurement, inside the Union: (1) Dispatching expertise (2) Transport and related logistics
	Training Network
	Civil Protection Mechanism Exercises
	Exercises on civil protection modules, technical assistance and support teams and EU Civil Protection Teams (EUCPT)
	The European Emergency Response Capacity (EERC) Voluntary Pool
	Maintenance and further development of the Common Emergency Communication and Information System (CECIS)
	Addressing temporary shortcomings
Exchange of experts Programme

Lessons Learned Programme

Advisory missions for prevention and preparedness in and outside the Union


Peer reviews

Awareness raising and dissemination of good practices

	CECIS hosting, ERCC security and resilience

ANNEX 4: SYNOPSIS REPORT ON STAKEHOLDER CONSULTATION

Table of Content

1. INTRODUCTION

2. SCOPE OF THE CONSULTATION

3. STAKEHOLDER GROUPS COVERED BY THE CONSULTATION ACTIVITIES

4. CONSULTATION METHODS AND TOOLS

4.1 Interviews

4.2 Online surveys

4.3 Open Public Consultation

4.4 Case Studies

5. RESULTS OF EACH OF THE CONSULTATION ACTIVITIES (TARGETED)

6. RESULTS OF THE CONSULTATION IN THE FRAMEWORK OF THE OPC

6.1 General Questions (for both questionnaires)

6.2 Relevance (only for in-depth knowledge)

6.3 Effectiveness (only for in-depth knowledge)

6.4 Efficiency (only for in-dept knowledge)

6.5 Coherence (only for in-depth knowledge)

6.6 EU added value (only for in-de pth knowledge)

7. CONCLUSIONS

1.INTRODUCTION

The general objective of the consultation activities 99 carried out in the context of the Interim Evaluation of the UCPM was to ensure that all concerned stakeholders had the opportunity to express their views on the performance of the UCPM during the first half of its implementation. In line with this, the following specific objectives were set out:

a)Collect views and opinions of the European citizens on the results achieved by the UCPM to date;

b)Determine the importance of different priorities for civil protection and disaster risk management as they are perceived across the EU;

c)Obtain targeted feedback and technical advice from the primary stakeholders of the UCPM;

d)Gather evidence to support the interim evaluation of the UCPM for each of the main evaluation criteria.

The Commission contracted an external consultant to carry out the consultations, prepare a report, and integrate the main findings into the final evaluation report.

2.SCOPE OF THE CONSULTATION

In line with the objectives, a consultation strategy was set up based on the EC Better Regulations Guidelines for conducting evaluations 100 . The consultation questions were based on the five criteria recommended by the Better Regulations guidelines: (1) Effectiveness, (2) Efficiency, (3) Relevance, (4) Coherence, and (5) EU added value. In addition, (6) Sustainability was also added to the evaluation criteria and assessed via the consultation.

The consultations covered all activities carried out under the UCPM in the timeframe of January 2014 – December 2016. The evaluation encompassed actions carried out under the framework of the UCPM and spanning across the three fields of prevention, preparedness and response to natural and man-made disasters.

3.STAKEHOLDER GROUPS COVERED BY THE CONSULTATION ACTIVITIES

The following stakeholders were identified for input collection:

-Entities implementing the UCPM: Member States (MS) Civil Protection (CP) authorities contributing with modules and/or experts to the Voluntary Pool; Other MS CP authorities; non-MS Participating States (PS) CP authorities.

-Entities affected by the UCPM: Host countries having received UCPM support (EU, Accession, Neighbourhood, and Third Countries), European Parliament, Committee of Regions, local authorities.

-Entities with a stated interest in the UCPM: EU MS (authorities other than CP 101 ), non-MS PS (authorities other than CP), UN agencies, World Bank, NGOs, private sector, foundations, etc.

-Entities having expertise relevant to the UCPM (i.e. disaster risk management): universities/centres of excellence and think tanks, mainly in PS.

The Open Public Consultation included views from all of the four groups above whereas the targeted consultation focused on the following ones: entities implementing the UCPM, entities affected by the UCPM, and entities with a stated interest in the UCPM.

4.CONSULTATION METHODS AND TOOLS

The overall consultation strategy aimed at gathering information and opinions on the implementation of the Decision as well as to test analyses and findings. Stakeholders were consulted through the following means:

(I)Interviews

(II)Online surveys

(III)Open Public Consultation

(IV)Case studies

4.1Interviews

Interviews were expected to be completed by end of February. In agreement with the ISG, it was decided to extend the consultation period until April to finalise two interviews with an EU Agency and an external evaluator that could not be undertaken beforehand.

A total of 40 interviews with stakeholders were originally planned however, in the end, 64 interviews were carried out. In particular, the external contractor conducted a higher number than planned interviews with: DG ECHO officers, other DGs, EU Agencies, among others.

Telephone and face-to-face interviews were conducted with relevant staff from a broad range of EC Directorates-General (DGs) and EU services, i.e.ECHO ENV, CLIMA, NEAR, HOME, SANTE, ENER, EU-Agencies (ECDC, EMSA, EASO, EEA), as well as with ECHO international partners including disaster management-focused organisations (e.g. UN agencies) and national authorities from Participating States. Interviews with professional organisations (e.g. exchange of experts programme manager, exercise provider, transport provider) and with external evaluators (External Evaluation of EMSA, External Evaluation of PPRD South II) were also carried out.

The main purposes of these interviews were to:

1.Collect evaluative evidence with regard to the effectiveness, efficiency, relevance, coherence, EU added value and sustainability of the mechanism; and

2.Outline the lessons learned, success factors, gaps and areas for improvement.

4.2Online surveys

In order to reach the large number of actors involved in the implementation of the Union Mechanism over the 2014-2016, five (rather than the initially planned four) online surveys were carried out targeting the following stakeholder (sub-)groups:

1.National CP authorities Contact Points;

2.Other relevant national authorities (e.g. other line ministries);

3.Trainers and National Training Coordinators;

4.CP experts participating in the exchange of experts programme

5.Leads of prevention and preparedness projects.

The five online surveys were launched through an online application and a separate web-link was distributed to each target group, focused along the lines of the evaluation questions.

The surveys were launched three weeks later than initially scheduled (mid-December 2016) but all remained open for six weeks (as initially envisaged) in order to generate the maximum number of replies for each of these five surveys. In view of response rates being in line or beyond expectations the three-week delay is not considered to have had an impact on the findings presented in the final report.

The response rate for the National civil protection authorities Contact Point survey was higher than expected, as well as for the other relevant national authorities and experts in the exchange of experts programme. Responses received from National Contact Points were followed up with phone interviews. The response rate from national training coordinators was initially lower than for civil protection authorities and project coordinators (initially due to missing contact details) but additional reminders were sent and the survey closure date was extended to the end of February 2017.

4.3Open Public Consultation

The Commission launched an Open Public Consultation (OPC) on the implementation and performance of the Union Mechanism for a period of three months, starting on the 24 November 2016 and ending on 23 February 2017. The OPC was divided in two parts - one designed for respondents who indicated to have limited/no knowledge of the Mechanism and one for those with in-depth knowledge. In total the OPC led to 67 answers for the survey from respondents without in-depth knowledge about the UCPM and 61 answers to the survey from respondents with in-depth knowledge about the UCPM, from 26 countries. 102 A brief summary of the main elements raised by respondents is included in section 7 below.

4.4Case Studies

Three case studies were also carried out as part of the stakeholder consultation for this evaluation. The information collected was analysed and used to triangulate information received from all consultation tools. The case studies selected were: (1) a case study on forest fires in Cyprus; (2) a module exercise (ModEx) in Estonia; and (3) the case study of the response to the migration emergency on the Balkan route. During these case studies a total of 23 interviews were conducted with relevant stakeholders (members of relevant ministries and national civil protection authorities, local experts, trainers, etc.). The case studies also served to test and confirm preliminary findings.

5.RESULTS OF EACH OF THE CONSULTATION ACTIVITIES (TARGETED)

This section presents a summary of the views of the main stakeholders responding to each of the evaluation questions (i.e. Relevance, Effectiveness, Efficiency, Coherence, Added Value and Sustainability). Stakeholders have been divided into three main categories, namely: a) Participating States; b) EC Services; and c) International Partners. This division corresponds to the way the data was presented by the external contractor in the Interim Evaluation.

For each main question, this section highlights the main elements where broad consensus was reached by each stakeholder group, as well as some diverging opinions.

5.1Relevance

-Participating States agreed on the need to ensure a strategic link between the UCPM mandate and its activities. This includes the need to allocate resources (e.g. for response) based on needs assessment and on the comparative advantage of the civil protection vs. other actors (e.g humanitarian aid, police, etc). There was also a consensus on building the training and exercise programme on a need-based approach which is informed by operational lessons learnt in the first place.

Different views emerged on what the new training programme should look like (number of people trained for each course, frequency, content, etc.). Views among PS diverged also with regards to providing new/increased incentives for Voluntary Pool assets, due to the possible risk of encouraging PS to be too reliant on EU assistance for events which should be addressed by national capacities.

-EC services noted the need to develop long-term strategies as opposed to ad hoc activities from the UCPM and called for further integration with EC programmes attaining similar/related objectives.

-International partners (including UN agencies) called for an increased use of risk assessment maps, more emphasis on international partners in training and exercises. Finally, some partners underscored that the UCPM response operations should be more used as an entry point to discuss disaster risk management with third countries.

5.2Effectiveness

-Consulted Participating States agreed, both in the online survey and in the interviews that the UCPM has been broadly effective during its first 3 years. They also agreed that the UCPM needs to better take into account existing bilateral agreements between PS (proximity factor) and clarify its cooperation/overlaps with the humanitarian assistance.

Diverging views were recorded with regards to the selection of experts for deployment (a number of PS believe the process can still be improved), the training programme (quotas allocated by country size vs. by number of assets contributed to the Voluntary Pool), and to the effectiveness of peer reviews and their format (e.g. experts selected).

-Several EC services highlighted that improvements (e.g. more strategic use) to advisory missions could increase their effectiveness. The integration of climate change adaptation into the UCPM (prevention and preparedness) as well as energy (e.g. risk assessments, preparedness plans) was also highlighted.

-International partners underscored the need for the UCPM to further build on the close linkages between the humanitarian aid actors and local communities/actors. They also raised the need to carefully look at how civil protection missions outside EU and in contexts of complex emergencies can be perceived as less neutral than humanitarian actors.

5.3Efficiency

-Participating States noted that the efficiency of some activities could be increased by better defining their objectives (e.g. training people for deployment vs. to simply increasing their knowledge) and in some cases, with the streamlining of administrative processes.

-EC services tended to question the efficiency of in-kind assistance, marking a difference with the view of Participating States.

-International partners did not express any views on this due to limited information available to them.

5.4Coherence

-Participating States agreed to a large extent that the UCPM cross-cutting activities (e.g. lessons learnt, peer reviews, advisory missions, exchange of experts) should be driven by a more result-oriented approach to reinforce the internal coherence of the mechanism. Participating States expressed diverging views when assessing the risk of duplicating training activities at EU and national level.

-At EC Services level, the main point of concern related to the relation of the UCPM with other policy areas. Generally, although an improvement of linkages with other areas was appreciated in relation to the previous mechanism (2007-2013), EC services concluded that the level of horizontal cooperation should be reinforced. These may include research and innovation, regional policy, environment, climate adaptation, health, the neighbourhood policy, as well as better integration of the new security landscape affecting Europe.

-The main point of concern for International Partners is the growing interlinkage between civil protection operations and humanitarian aid operations, as the mechanism is increasingly activated to deal with humanitarian and complex emergencies. Whereas the UCPM could positively contribute to humanitarian operations, the mandate and division of tasks need to be clearly set out to avoid overlapping and duplications.

5.5EU Added Value

-Participating States agreed to a large extent on the clear added value that the UCPM delivers in each of its three main pillars (i.e. prevention, preparedness and response). Participating States highlighted specially the creation of the ERCC as the most tangible and visible expression of the EU added value. In relation to the prevention pillar, generally Participating States agreed that the actions foreseen under the legislation, (e.g. requirement to produce National Risks Assessments, conduction of Prevention and Preparedness projects, etc.) represented a clear EU added value as they helped to establish a common understanding of risks across Europe and facilitated cross-border cooperation, inherent when responding to disasters. The main point of concern, though, which also generated broad consensus, was the lack of clear and quantifiable indicators that can help monitoring the results to better establish the EU added value of these actions.

-As far as EC Services are concerned, the main element emphasised by respondents is the clear added value that the UCPM can generate by becoming the 'coordination hub' of disaster response within the Union. As well as Participating States, EC Services also agreed that a better tracking system of cost-effectiveness of EU-level action could demonstrate further the significant added value.

-International Partners agreed that the creation of the ERCC represents a positive contribution towards the development of international networks of civil protection experts, as well as to facilitate the cooperation between EU and international actors.

5.6Sustainability

-Participating States generally agreed on the idea that better linkages between training and deployment need to be found, to ensure that knowledge acquired can be translated into actual action. On a similar note, Participating States also pointed out that a better follow-up on recommendations done to a beneficiary country would be essential to ensure sustainability of changes. Some Participating States also raised concerns about the participation of private companies in UCPM activities (e.g. needs assessment, advisory missions, peer reviews, etc.)

-EC Services agreed on a general level that a viable financial model to ensure availability of capacities in the EERC needs to be established to ensure the sustainability of the model.

-International partners did not express views on this due to the limited information available to them.

6.RESULTS OF THE CONSULTATION IN THE FRAMEWORK OF THE OPC

As stated above, the OPC was designed following a two-track approach: a questionnaire for respondents with limited/no knowledge of the UCPM, and a separate one for respondents with in-depth knowledge. A summary of the results is presented here below:

6.1General Questions (for both questionnaires)

Hazard/Emergencies

-Respondents without in-depth knowledge about the UCPM indicated that the top three emergencies that the UCPM should deal with are: earthquakes, floods and terrorist attacks.

-On the contrary, respondents with in-depth knowledge established earthquakes, floods and forest fires as the three main priorities.

Source of Information

-For respondents without in-depth knowledge the preferred source of information about the UCPM would be national civil protection authorities.

-Whereas for respondents with in-depth knowledge, the main source of information about the Mechanism would be by far DG ECHO.

Pillars

-Respondents without in-depth knowledge agreed to a large extent that an EU-coordinated action in the field of prevention, preparedness and response would be more effective than single actions undertaken by Member States.

-Similarly, respondents with in-depth knowledge also agreed that the effectives of an EU-coordinated action would be higher (particularly in the field of preparedness and response) than single undertakings.

6.2Relevance (only for in-depth knowledge)

-Respondents without in-depth knowledge about the UCPM agreed on the continuous relevance of the Mechanism as it addresses the critical challenges faced by the EU. Although still high, lesser consensus is reached when assessing the capability of the Mechanism to adapt to new risks in the emergency management.

6.3Effectiveness (only for in-depth knowledge)

-Respondents without in-depth knowledge stressed that coordination at EU level in the three pillars of the UCPM (prevention, preparedness and response) has improved the effectiveness of the disaster response mechanisms at national level. Some criticism was raised, though, regarding the response to the migration crisis, which was not considered positively handled.

6.4Efficiency (only for in-depth knowledge)

-Half of respondents without in-depth knowledge noted that the administrative procedures to prepare and submit an application for the prevention and preparedness projects were too cumbersome, therefore undermining the efficiency of the Mechanism.

6.5Coherence (only for in-depth knowledge)

-For respondents with in-depth knowledge, internal coherence within the Mechanisms (i.e. the three pillars) was achieved to a large extent. Regarding external coherence (i.e. coherence with other policy areas), respondents agreed that clear links were established with the humanitarian aid. However, a lack of synergies was mentioned regarding migration and refugees.

6.6EU added value (only for in-depth knowledge)

-By a large majority, respondents with in-depth knowledge noted that individual Member States benefit notably from a coordinated EU action through the Mechanism. Consensus was also reached on the fact that EU action also increased the international visibility of the EU as a global actor.

In the framework of the Open Public Consultation, and in addition to the information gathered through the questionnaires specifically designed, five other contributions (documents) were submitted via the online portal. These are of different nature and scope and therefore difficult to categorise. For transparency purposes, they are presented here below:

-National Red Cross Societies of EU Member States and the International Federation of Red Cross and Red Crescent Societies, Contribution to the Public Consultation on Evaluation of the Union Civil Protection Mechanism, February 2017.

-NN, The Securitisation of European Union Civil Protection. Investigating the logics of securitisation and the development of European Union Civil Protection, Thesis, School of Politics and International Relations – University College Dublin, August 2016.

-NN, Position Paper on the European Commission's 'Public Consultation on the Interim Evaluation of the Union Civil Protection Mechanism'.

-PLATAFORMA, Forced displacement and development. Plataforma's comments on the non-paper for expert consultations.

-International Amateur Radio Union, Response to Public Consultation on the Interim Evaluation of the Union Civil Protection Mechanism, February 2017.

7.CONCLUSIONS

The stakeholder consultation provided valuable feedback on the functioning of the UCPM during its first three years of implementation, highlighting its achievements as well as some of its shortcomings.

In general terms, stakeholders present a relatively high degree of satisfaction with the functioning of the Mechanism. Responses from the consultation exercise seem to indicate that the Mechanism is on its way to achieve its general and specific objectives. However, crossing all the input collected during the stakeholder consultation, there are certain recurring points of concern.

Stakeholders emphasised the need for a more strategic use of the UCPM activities already in place, that is, foster synergies within its three pillars and reinforce and systematise the integration of lessons learnt into the feedback cycle.

Overall, stakeholders also pointed out the need to enhance the synergies with other policy areas, actors and instruments with the aim to strengthen the external coherence of the Mechanism, and improve international cooperation.

Likewise, a recurrent point of concern among stakeholders seems to be certain administrative burdens that prevent the Mechanism from being speedier and fully effective, particularly for disaster response operations.

As a central piece of the Interim Evaluation of the UCPM, the results of the stakeholder consultation have highlighted the positive elements, as well as some shortcomings of the Mechanism. Several of the latter may be rectified by possible amendments to the implementing decision, i.e. Commission Implementing Decision 2014/762/EU. As for the rest, the results of the consultation will certainly be integrated in the reflection process for future possible modifications of the UCPM.

(1)

OJ L 347, 20.12.2013, p. 924

Available at: http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2013:347:0924:0947:EN:PDF

(2)

Namely, Decision 1313/2013/EU (see above) and Commission Implementing Decision 2014/762/EU of 16 October 2014 laying down the rules for the implementation of Decision 1313/2013/EU (herein 'implementing rules'), OJ L 320, 6.11.2014, p. 1

(3)

The financial envelope for the implementation of the Union Mechanism's actions in the period of 2014-2020 is EUR 368 428 000. Article 19(4) specifies that this financial envelope is allocated according to the following breakdown: 20% Prevention, 50% preparedness, 30% response, with the possibility to vary each by ± 8 percentage points.

(4)

Final report available at: https://ec.europa.eu/echo/sites/echo-site/files/ucpm_final_report.pdf

(5)

European Commission, Better Regulation for Better Results — An EU agenda, COM (2015)215, and Better Regulation Guidelines (Commission Staff Working Document), SWD (2015) 111.

(6)

The Roadmap of the Interim Evaluation can be consulted here: http://ec.europa.eu/smart-regulation/roadmaps/docs/2016_echo_007_evaluation_ucpm_en.pdf

(7)

As of June 2017, the Participating States of the Union Mechanism include all EU Member States, Iceland, Norway, Serbia, the Former Yugoslav Republic of Macedonia, Montenegro and Turkey.

(8)

"Financial assistance may also be granted to candidate countries and potential candidates not participating in the Union Mechanism as well as to countries that are part of the ENP (…)"

(9)

Council Decision 2001/729/EC, Euratom of 23 October 2001 establishing a Community mechanism to facilitate reinforced cooperation in civil protection assistance interventions (OJ L 297, 15.11.2001, p.7)

(10)

Council Decision 1007/779/EC, Euratom of 8 November 2007 establishing a Community Civil Protection Mechanism (OJ L 314, 1.12.2007, p.9)

(11)

Impact Assessment (2011) Review of the civil protection regulatory framework, SEC(2011) 1632 final

(12)

Article 28(1) of Decision No 1313/2013/EU

(13)

Article 28(2) of Decision No 1313/2013/EU

(14)

Defined in Annex II of the Commission Implementing Decision of 16 October 2014 (C(2014) 7489)

(15)

Listed in Annex III of the Commission Implementing Decision of 16 October 2014 (C(2014) 7489)

(16)

Recent examples include: the Ebola outbreak in West Africa (2014), the floods in the Western Balkans (2014), the conflict in Eastern Ukraine (2015), the voluntary evacuation of EU citizens from Yemen (2015), and the ongoing refugee crisis (2015-16).

(17)

When civil protection assistance is requested by third countries, it is often combined with humanitarian aid.

(18)

As per Commission Decision on the Union Mechanism Annual Work Programmes (2014, 2015, 2016) and Commission Decision on financing the UCPM emergency response actions (2014, 2015, 2016).

(19)

See footnote 6

(20)

The Report on the Open Public Consultation can be consulted here: http://ec.europa.eu/echo/sites/echo-site/files/ucpm_-_opc_report_0.pdf

(21)

Special report no 33/2016: Union Civil Protection Mechanism: the coordination of responses to disasters outside the EU has been broadly effective, January 2017

(22)

Report from the Commission to the European Parliament and the Council on progress made and gaps remaining in the European Emergency Response Capacity, 17.02.2017

(23)

Stakeholders that responded include: civil protection authorities from 26 Participating States (PS) (online surveys); civil protection authorities from 17 PS (interviews); National Training Coordinators from 21 PS (online surveys); 137 experts having participated in the exchange of expert programme and 23 leads of projects funded via the UCPM from 13 PS (online surveys); 14 international partners including UN agencies, OECD, World Bank, Red Cross, etc.(interviews); 15 Directorates-General and services of the EU (interviews)

(24)

See footnote 20

(25)

Council of the EU, Report of the Netherlands Presidency on the main achievements at EU level in the field of civil protection, Document 10019/16, Brussels, 17 June 2016, available at:

http://data.consilium.europa.eu/doc/document/ST-10019-2016-INIT/en/pdf

(26)

Staff Working Document, Overview of Natural and Man-made Disaster Risks the European Union may face, 23.05.2017 SWD(2017)176 final

(27)

Commission Notice, Risk Management Capability Assessment Guidelines, (2015/C 261/03). Available at: http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52015XC0808%2801%29

(28)

Article 222 TFEU

(29)

Copernicus EMS provides information for emergency response to different types of disasters, including meteorological hazards, geophysical hazards, deliberate and accidental man-made disasters and other humanitarian disasters as well as prevention, preparedness, response and recovery activities. For further information see http://emergency.copernicus.eu/.

(30)

Full Scale Exercises are designed to replicate one or several phases of an emergency with the commitment of all the public bodies and authorities that would be committed in a real situation.

(31)

Modules field exercises aim to provide an opportunity for testing specific response capacities, as well as the self-sufficiency, interoperability, coordination and procedures of response teams and equipment.

(32)

This includes: 46 experts deployed to 11 response missions and 38 experts deployed to 7 advisory missions.

(33)

This budget line of the Work Programme also include other meetings with Participating States, such as the bi-annual civil protection Directors-General meeting.

(34)

European Commission, Implementing Decision of 16 October 2014 (C(2014) 7489)

(35)

ICF, Interim Evaluation of the Union Civil Protection Mechanism, 2014-2016, Final Report, July 2017, (hereafter: External Evaluation Report), p. 51

(36)

External Evaluation Report, p. 18

(37)

European Court of Auditors, Special Report (2016), Union Civil Protection Mechanism: the coordination of responses to disasters outside the EU has been broadly effective. Available at:

http://www.eca.europa.eu/Lists/ECADocuments/SR16_33/SR_DISASTER_RESPONSE_EN.pdf

(38)

See Annex 4: "Synopsis of Stakeholder Consultation". Also, External Evaluation Report, p. 69

(39)

External Evaluation Report, p. 73

(40)

External Evaluation Report, p. 305

(41)

External Evaluation Report, p. 73

(42)

Among others, European Parliament resolution of 1 December 2016 on the situation in Italy after the earthquakes (2016/2988(RSP))

(43)

European Commission, Special Eurobarometer 454, Civil Protection, May 2017

(44)

See footnote 25

(45)

Based on conversations with PS representatives during meetings (CPC, lessons learned, experts' meetings, etc.).

(46)

European Commission, Towards a Strategic Framework for Training and Exercises within the Union Civil Protection Mechanism (UCPM) 2017-2020, not available online.

(47)

Source: European Forest Fires Information System (EFFIS). This figure is based on a conservative assessment of economic losses (does not include, for instance, the high cost of firefighting equipment).

(48)

External Evaluation Report, p. 153

(49)

External Evaluation Report, p. 97

(50)

External Evaluation Report, p. 114

(51)

Overview of Risks Report, p. 3

(52)

Commission Notice, Risk Management Capability Assessment Guidelines (2015/C 261/03), available at: http://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:52015XC0808(01)&from=EN

(53)

External Evaluation Report, p. 22

(54)

External Evaluation Report, p. 23

(55)

See footnote 54

(56)

This includes the 2017 Eurobarometer survey on civil protection, where about half of all respondents think that not enough is being done for disaster prevention by their region and country (49%) and at EU level (43%).

(57)

External Evaluation Report, p. 25, 93-94

(58)

Commission Staff Working Paper, Risk Assessment and Mapping Guidelines for Disaster Management, 2010

(59)

Report from the Commission to the European Parliament and the Council on progress made and gaps remaining in the European Emergency Response Capacity, 17.02.2017

(60)

Such goals were defined by the Commission Implementing Decision of 16 October 2014 (C(2014) 7489)

(61)

Source: DG ECHO

(62)

MUSAR - Medium urban search and rescue, GFFFV - Ground Forest Fire Fighting Using Vehicles, CBRNDET -Chemical, biological, radiological and nuclear detection and sampling (CBRN), HCP - High capacity pumping, HUSAR - Heavy urban search and rescue, AMP - Advanced medical post, GFF - Ground Forest Fire Fighting, USARCBRN - Search and rescue in CBRN conditions, WP - Water purification, FRB - Flood Rescue Using Boats, AMP-S - Advanced medical post with surgery, MEVAC - Medical aerial evacuation of disaster victims, FFFP - Aerial forest fire fighting module using airplanes, FFFH – Aerial forest firefighting module using helicopters, FHOS – Field hospital, FC - Flood Containment, ETS - Emergency temporary shelter, TAST – Technical Assistance and Support Teams

(63)

Externarl Evaluation Report, p. 26-28

(64)

External Evaluation Report, p. 32

(65)

See footnote 45

(66)

European Commission, Implementing Decision of 16 October 2014 (C(2014) 7489)

(67)

External Evaluation Report, p. 60

(68)

External Evaluation Report, p. 39

(69)

See footnote 21

(70)

External Evaluation Report, p. 41

(71)

External Evaluation Report, p. 41

(72)

External Evaluation Report, p. 39

(73)

External Evaluation Report, p. 43

(74)

External Evaluation Report, p. 55

(75)

External Evaluation Report, p. 60

(76)

These findings need to be considered as preliminary due to the short timeframe of the evaluation and to the fact in many cases final invoicing has not yet been processed.

(77)

External Evaluation Report, p. 63

(78)

External Evaluation Report, p. 65

(79)

External Evaluation Report, p. 305

(80)

See footnote 45

(81)

Estonia ModEx exercise, observed in the context of the external evaluation.

(82)

See footnote 45

(83)

External Evaluation Report, p. 119

(84)

Consultations included 15 DGs and other services of the EU. The complete list is provided in Annex 2.

(85)

European Commission, ROM Report, Prevention Preparedness & Response to natural and man-made Disasters in the EaP East (PPRD_East II), 03/05/2017

(86)

PARTICIP, Evaluation of the PPRD South II Regional Project (ENI South), January 2017

(87)

IPA Civil Protection Cooperation Programme II, Final Programme Report, December 2015

(88)

External Evaluation Report, p. 173

(89)

External Evaluation Report, p. 207

(90)

External Evaluation Report, p. 101-102

(91)

External Evaluation Report, p. 102

(92)

External Evaluation Report, p. 117-126

(93)

External Evaluation Report, p. 149

(94)

External Evaluation Report, p. 288

(95)

Based on dialogue with PS during several meetings occurred in 2016 and first half of 2017 (e.g. Civil Protection Committee, CP DG meeting, experts' meetings, etc.).

(96)

External Evaluation Report, p. 50

(97)

European Court of Auditors, Special Report no 33/2016; Report from the Commission to the European Parliament and the Council on progress made and gaps remaining in the European Emergency Response Capacity, 17.02.2017

(98)

This emerges as the main findings of recent evaluations, including: ROM Review of PPRD-East (2017); Evaluation of PPRD-South (2017); Evaluation of IPA II (2015).

(99)

The consultation strategy was published on 24 November 2017 and is available at: http://ec.europa.eu/echo/sites/echo-site/files/consultation_strategy.pdf

(100)

See: http://ec.europa.eu/smart-regulation/guidelines/toc_guide_en.htm

(101)

This includes other government counterparts whose actions are overlapping with activities carried out by the UPCM. Examples include: Ministries of Social Affairs (e.g. protection of people from the impact of disasters), Ministry of National Security/Interior (e.g. management of civil protection consequences of security threats), Ministry of Public Works (e.g. protection of assets), Ministry of Health (e.g. prevention and management of pandemics).

(102)

The report on the Open Public Consultation, summarising the main contributions by stakeholders can be found here: http://ec.europa.eu/echo/sites/echo-site/files/ucpm_-_opc_report_0.pdf