EUROPEAN COMMISSION

Brussels, 12.6.2018

SWD(2018) 331 final

COMMISSION STAFF WORKING DOCUMENT

on the ex post evaluation of the " Prevention, Preparedness and Consequence Management of Terrorism and other Security related risks" 2007-2013 Programme (CIPS)

Accompanying the document

Report from the European Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions

ex post evaluation for the period 2007 to 2013 of actions financed by the "Prevention and fight against crime" programme (ISEC) and the " Prevention, Preparedness and Consequence Management of Terrorism and other Security related risks" programme (CIPS)

{COM(2018) 455 final}
{SWD(2018) 332 final}

Contents

1    Introduction    

2    Background to the initiative    

2.1    Policy context    

2.2    Baseline    

2.3    The CIPS Programme as funding instrument    

3    Evaluation questions    

4    Method    

5    Limitations    

6    Status of the Implementation    

7    Answers to the evaluation questions    

7.1 Relevance    

7.2 Effectiveness    

7.3 Efficiency    

7.4 Coherence    

7.5 EU added value    

8    CONCLUSIONS    

ANNEX 1 — Procedural information    

ANNEX 2 — Stakeholders’ consultation    

ANNEX 3 — Methodology    

ANNEX 4 — List of evaluation questions    

ANNEX 5 — List of abbreviations and country codes    

ANNEX 6 — The role of the Joint Research Centre within CIPS    

ANNEX 7 — Sectoral analysis of CIPS projects    

ANNEX 8 — The different dimensions of CIPS    

ANNEX 9 — Intervention logic of the CIPS Programme    

1Introduction

The 2007-2013 ‘Prevention, Preparedness and Consequence Management of Terrorism and other Security-Related Risks’ programme (CIPS) was established by Decision No 2007/124/EC 1 and is one of the two financial programmes 2 falling under the framework programme on ‘Security and Safeguarding Liberties’. Its total budget amounted to EUR 126.8 million. The other component of this framework programme is the ‘Prevention and Fight against Crime,’ which is covered in a separate report.

The CIPS programme concerns critical infrastructure and other security issues, including operational issues in areas such as crisis management, environment, public health, transport, research and technological development. Its general objective was to support Member States’ efforts to prevent, prepare for and to protect people and critical infrastructure from terrorist attacks and other security-related incidents.

The Decision establishing the CIPS programme requires the Commission to submit to the European Parliament and the Council three evaluations reports:

-an annual presentation on the implementation of the programme 3 ;

-an interim evaluation report on the results obtained and the qualitative and quantitative aspects of the implementation of the programme by 31 March 2010 4 ;

-A communication on the continuation of the programme by 31 December 2010 5 ;

-An ex post evaluation report by 31 March 2015 6 .

This staff working document reports on the ex post evaluation for the evaluation of the CIPS programme 2007-2013 annual work programmes implemented in the participating Member States, based on the findings of an ex post evaluation study carried out by an external consultant and building on the previous evaluations mentioned above 7 .

The ex post evaluation assessed the CIPS programme on the basis of the following criteria:

·relevance (whether its objectives met policy needs);

·effectiveness (whether the objectives had been achieved);

·efficiency (whether the costs were proportionate to the benefits gained);

·coherence (between the actions financed by the instrument); and

·added value (of intervening at EU level).

The interim evaluation of the Internal Security Fund (‘the Fund’) 8 , its police component being the successor of the ISEC and CIPS programmes 2014-2020, takes into account the conclusions and findings of this evaluation, assessing what has already been implemented under the ISF-Police and what needs to be further addressed in the next Multiannual Financial Framework post-2020. The interim evaluation of the Fund is due in 2018 9 . The Commission will submit an interim evaluation report on the Fund by 30 June 2018 to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions. The results of the interim evaluation together with the results of this ex post evaluation will contribute to shaping future EU migration and home affairs policies, and in particular in preparing the new funding instruments for the Multiannual Financial Framework post-2020.

2Background to the initiative

2.1Policy context

This section introduces the EU policy context of the two main focus areas under the CIPS programme, namely: (i) protection of critical infrastructure and (ii) terrorism.

On the basis of the conclusions adopted in 1999 by the Tampere European Council in November 2004, the European Council reaffirmed its promotion of freedom, security and justice through the prevention of and fight against crime. This led to the Hague programme (2004-2009) 10 which focused on strengthening security in the EU’s fight against terrorism, recruitment, financing, protecting critical infrastructures and developing a sound consequence management framework. More recently, the 2010-2015 Stockholm programme 11 , complemented by an action plan, led to the development of an EU Internal Security strategy, underlining the need to fight against terrorism.

i) Protection of critical infrastructures

Infrastructures are considered ‘critical’ when their disruption could have an impact on the functioning of a society (in terms of economy, security and people’s wellbeing) 12 . Critical infrastructure protection is linked to terrorism as well as to chemical, biological, radiological, nuclear, and explosives (CBRN-E) threat and cyber security. However, critical infrastructures need to be protected also from man-made and natural disasters. Furthermore, these are highly interdependent: the disruption of one of them might cause what is known as the ‘domino effect’, with potentially broader consequences and multiplied impacts 13 .

The EU has been active in critical infrastructure protection since 2004, although protecting critical infrastructures is a Member State competence (while being highly interdependent).

On the critical infrastructure protection legal framework, the main pieces of legislation are:

i) the 2006 European Programme for Critical Infrastructure Protection (EPCIP) 14 ;

ii) the Council Directive 2008/114/EC of 8 December 2008 on the identification and designation of European critical infrastructures and the assessment of the need to improve their protection 15 ; and

iii) Commission Staff Working Document on a new approach to the European Programme for Critical Infrastructure Protection 16 , which is the result of a review of the Critical Infrastructure Protection approach presented in 2013 17 and heavily focuses on interdependencies of critical infrastructure protection across sectors.

The framework established by the EPCIP provides procedures to identify and designate European critical infrastructures, and to carry out a threat assessment, introduced by Directive 2008/114/EC. The Critical Infrastructure Warning Information Network 18 and the critical infrastructure protection expert groups at EU level 19 are also included. The general objective of this framework is to enable the implementation of EPCIP, information exchanges and to protect critical infrastructures.

Furthermore, the Council 20 identified two main sectors of critical infrastructure: energy and transport.

On the energy sector, European critical infrastructures include those linked to subsectors like electricity (infrastructures and facilities to generate and supply electricity), oil (oil production, refining, treatment, storage and transmission by pipelines) and gas (gas production, refining, treatment, storage and transmission by pipelines, LNG 21 terminals).

Within the meaning of transport sector, road, rail, air, waterways as well as ocean and short-sea shipping and ports are included.

The EU has also been active in the protection of CBRN materials, in particular those already present in the EU soil, e.g. stored in national plants although not included in the sectors regulated by Directive 2008/114/EC. For this reason, the EU adopted a CBRN action plan in 2009, as a main outcome of the CBRN Task Force that was set up in 2008. The main goal of the action plan is to minimise the threat and damage of CBRN incidents, by promoting a risk-based approach to security.

ii) Terrorism

Since 2007 Europol registered a decrease in the total number of completed, foiled and failed attacks. That said as from 2013 22 a slight increase has been observed. Over the evaluation period, the terrorism threat has significantly evolved and included, although in a smaller scale, also left-wing, separatist, anarchist and right wing extremism. More specifically, the EU has witnessed a number of religiously-inspired terrorist attacks as from 2011. Indeed, the last years of the period under evaluation and those immediately subsequent saw a significant change in the terrorism phenomenon, in particular with the continuation of the civil war in Syria and the rise of the Islamic State in Iraq and Syria (ISIS).

At the EU level, a Council Framework Decision on combating terrorism 23 was adopted in 2002, followed three years later by the EU Counter-Terrorism strategy 24 that was made up of four parts: (i) prevent, (ii) protect, (iii) pursue and (iv) respond.

Furthermore, the EU internal security strategy in action (EU ISS) was adopted for the 2011-2014 25 period. The EU internal security strategy identified five strategic objectives, including the objective of prevention of terrorism and prevention of radicalisation and recruitment. To prevent terrorism, the EU committed to cut off terrorists’ access to funding and non-conventional materials (CBRN) and strengthen the protection of aviation, land and maritime transport systems.

2.2Baseline

In order to fully identify the contributions the funding made to the different CIPS areas it would have been desirable to establish a baseline, reflected in a set of indicators, including data on national spending in the various areas. However, as explained in the limitations section, such baseline is lacking for the CIPS Programme and it has not been possible within the scope of this evaluation to develop a detailed baseline covering the situation before the start of the Programme.

2.3The CIPS Programme as funding instrument

As mentioned above, one of the general objectives of the CIPS programme was to support Member States’ effort to prevent, prepare for, and to protect people and critical infrastructure against terrorist attacks and other security-related incidents (prevention and preparedness, general objective 1) 26 .

Together with prevention and preparedness, the programme also sought to ensure consequence management (post-attack), namely protection in the areas such as the crisis management, environment, public health, transport, research and technological development and economic and social cohesion, in terrorism and other security-related risks in the area of freedom, security and justice (consequence management, general objective 2) 27 .

Specifically, the CIPS programme was particularly focused on critical infrastructures which is defined as: ‘those physical resources, services, information technology facilities, networks and infrastructure assets which, if disrupted or destroyed, would have a serious impact on the critical societal functions, including the supply chain, health, safety, security, economic or social wellbeing of people or of the functioning of the Community or its Member States’ 28 .

To achieve its overarching objectives, the CIPS programme was designed to contribute to achieving seven specific objectives (the first five specific objectives are linked to the general objective 1 whereas the last two are linked to the general objective 2) 29 :

3Evaluation questions

The evaluation was planned and tendered out before the Better Regulation guidelines were adopted on 19 May 2015, but still covered all the five key evaluation criteria required by the guidelines i.e. effectiveness, efficiency, relevance, coherence and EU added value. To assess these criteria, the evaluation reviewed the 10 questions that are in Annex 4.

4Method

The Commission’s evaluation relies on an external study carried out between August 2016 and August 2017 by a consultancy specialised in carrying out evaluations. The study’s methodology combined desk research, and both qualitative and quantitative analysis. It required a systemic synthesis of the evidence on the implementation of the CIPS programme. Information was drawn mainly from various primary and secondary sources, such as:

•    Primary sources: three online surveys (coordinating organisations, partner organisations and project participants); stakeholder consultations (37 face-to-face and telephone interviews were conducted with five stakeholders types; thematic case studies (2); analysis of public consultation responses (6 responses); expert workshops (2) and expert input.

•    Secondary sources: quantitative data on all CIPS projects provided by the Commission; in-depth analysis of a sample of projects (29 projects, representing 28 % of all 104 finalised projects); a literature review of EU policy baseline and context and a literature review of national baseline and context.

A more detailed overview of the methodology used for the evaluation study is in Annex 3.

Information was triangulated when possible to ensure validity and robustness. Findings from the study are presented in this report together with the financial data extracted from ABAC 37 .

5Limitations

The evaluation process encountered a number of difficulties that induced methodological limitations and hindered the extent to which findings and conclusions could have been formulated. A summary of these limitations is reported in this section and more details can be found in Annex 3.

The lack of a baseline

The first major obstacle encountered in the evaluation was that it has not been possible to establish a baseline (i.e. the situation before the intervention of the programme) for measuring the level of prevention, preparedness and coordination of crisis management and security actions in the aftermath of a terrorist attack or of security incidents across the EU due to a lack of ex ante targets to measure results against.

Furthermore, no methodologies were adopted for measuring in a comprehensive manner most of the activities covered by the actions funded through CIPS. Consequently, no robust conclusions could be drawn on the situation before the intervention started (i.e. the baseline) for the different thematic areas covered by the programme and the potential effect the funding has had on these compared to the situation after the programme had been implemented. In addition, differences in the definitions and content of critical infrastructures across the EU, including their interdependencies, made setting a common EU baseline even more problematic. Therefore, effectiveness has been mostly evaluated in terms of outputs or perceived impacts or results– which were more easily measured and compared across Member States — rather than real impacts or results.

However, to partly overcome these limitations, where this was relevant, the findings and conclusions in the ex post evaluation have been compared to the findings and conclusions formulated during the interim evaluation of ISEC and CIPS and where it concerned similar evaluation questions. Given the lack of an existing baseline, the evaluation team had developed a set of indicators during the inception phase in order to map out where possible the national and EU context for the different thematic areas covered by CIPS and to identify trends in those data before, during and after the programming period. However, the team encountered significant data issues when trying to populate the indicators and it was agreed with the Steering Group that due to the lack of (comparative) data, this information would only be used for illustrative purposes and to focus on presenting the EU context for the different policy areas (see Section 2).

Time lag between programme implementation and the ex post evaluation

There was a time lag between the implementation of the Programme (2007-2013) and the ex post evaluation (August 2016-August 2017). This meant that the relevant beneficiaries and national and EU authorities and experts were often no longer working in the same position and were therefore difficult to reach. This is reflected in a low response rate to the online survey for coordinating and partner organisations as well as to the public consultations (see Annex 3). To try to mitigate these difficulties, the external consultancy took some measures, such as sending several email reminders to the people surveyed, searching for alternative contacts when possible, raising the individual partner organisation interviews to two rather than one, and extending the deadline for the online survey carried out as part of the case studies in order to boost the number of responses. Overall, all evaluation questions have been answered to some extent, thanks to the use of triangulation of data where possible. However, where the evidence base has been deemed to be too low to be reliable, this has been indicated.

Gaps in financial data

The calculations on the absorption rate included all amounts in ABAC until 31 March 2017 although a number of projects were not fully completed by that date. For this reason, absorption rates and trends, in particular for the last programming year, should be interpreted with caution since final amounts may be subject to variations on the basis of final payment for the non-completed projects. To try to mitigate this limitation, the costs have been estimated as accurately as possible, but have to be interpreted with caution. However, the absorption rate was not expected to change significantly. As no estimates were available for JRC data, it has not been possible to analyse the absorption rate for the JRC part of the procurement sector.

Limitations on monitoring the results and impacts of the programme

Issues were noted when monitoring the progress of the project and the final results hindered the assessment of effectiveness, efficiency and EU added value. The absence of a central repository to store the CIPS findings led to a variety of definitions of results and impacts of the programme depending on the thematic area. Therefore the possibility to compare the results across different thematic areas was restricted. However, to mitigate this limitation to the extent possible, data has been compiled and triangulated from multiple sources (interviews, surveys, sampled projects).

6Status of the Implementation

As set out in the annual work programme 2007-2013, a total budget of EUR 126.8 million was allocated to the CIPS programme, of which EUR 74.6 million was committed.

The minimum EU co-financing increased from EUR 50 000 (2007-2009) to EUR 100 000 in 2010 and remained at EUR 100 000 from 2011 onwards. The maximum rate of the EU co-financing increased consistently with the raise in minimum EU grant, from 70 % (2007-2009) to 80 % in 2010 and stayed at 90 % from 2011 onwards for the remainder of the programme implementation period.

In total, 126 CIPS grants were implemented across the programming period, of which 121 were action grants to open calls for proposals and five were direct grants to European Committee for Standardisation. The number of grants implemented per year remained largely the same throughout 2007-2013 (between 17 and 20), with a rise registered in 2009 (27).

More action grants were funded via open calls (76.5 %), followed by 14.4 % allocated to administrative arrangements with the JRC. Less than 8.5 % was allocated to the remaining public procurement and less than 1 % to grants to European Committee for Standardisation.

The number of projects implemented varied across the programme years 38 . For instance, at the start of the programme, in 2007, it was quite low (17 projects) but increased in 2009 to 27 projects.

The demand-driven design of the annual work programme (see section 7.1) resulted in a significant geographical imbalance in terms of the number of projects funded per coordinating organisation in each Member State. While organisations from Italy and Spain coordinated 48 projects and 23 projects respectively, organisations from 15 Member States (UK, BE, NL, DE, AT, CZ, PO, HU, RO, BG, EL, CY, SE, EE, SI) coordinated less than eight projects, with Slovenia, Cyprus and Estonia at the bottom of the scale. These countries only coordinated one project each. Conversely, a slightly more balanced spread was achieved by the partner organisations, with five or more partner organisations located in 21 Member States.

The majority of grants were cross-border (73 % or 88) as national grants represented only 27 % (or 33). Italy coordinated the highest number of national projects (15), followed by Spain (5). From the evaluation analysis it emerged that, overall, only 11 Member States coordinated at least one national project. However, this irregular geographical spread of coordinating organisations was counterbalanced by the location of the partner organisations. Indeed, a total of 86 national partner organisations were involved across these projects; while Italy had the highest number of partner organisations involved overall (47 organisations across 15 projects), Romania had the highest number of partner organisations involved in one grant (9 organisations).

Projects were considered as cross-border if they included at least one partner organisation based in a different Member States than the one where the leading organisation was based. In terms of the geographical distribution of cross-border projects, coordinating organisations were based in 16 different Member States. The highest number of cross-border projects was hosted by Italy (33 projects or 36 %), which is consistent with the over-representation of coordinating organisations located in that Member State within the whole programme, followed by Spain (18 projects or 20 %). Only in Romania the number of national projects was higher than the one of cross-border projects. Some organisations in Slovenia and the UK coordinated an equal number of national and cross-border partnerships.

Overview of main characteristics of projects under the programme

This subsection presents the descriptive analysis carried out on the sample projects, namely projects which were selected for in-depth evaluation analysis 39 . The documents accompanying the sample projects have been analysed and included the grant agreements, the final narrative reports, the mid-term reports (as from 2010) and the final evaluation reports prepared by the Commission.

General description of the sample

The sample of 29 projects represents 28 % of all 104 finalised projects from the CIPS programme and was representative of all projects. Indeed, the list of finalised projects broadly matched the characteristics of the complete list. The characteristics of the sample project are further described below. However, given that the sample only contains 29 projects, this list should not be considered as exhaustive.

98 % of CIPS grants were funded through open calls while only 2 % were funded via grants to standardisation bodies. In the sample of projects compiled the proportion is 97 % to 3 %. The sample is representative of the projects awarded per annual work programme per year. It contains a lower proportion of projects from 2013 as at the time of the selection only three projects were eligible and thus included.

Partnership characteristics of the sample projects

An analysis of the partnership characteristics of the sample projects was carried out and included. For instance, the type of organisations and partnership funded, the size of the partnership and the diversity of partnership.

The findings on possible outcomes and results in light of the above-mentioned partnership characteristics are described in the EU Added value section.

In terms of types of coordinating organisations funded, the most common were universities or research organisations (10 projects) followed by private sector organisations. The latter were often partners with public organisations to deliver CIPS projects and vice versa.

On the size of partnerships, it varied across the sample. Overall, the average number of organisations participating in a CIPS project within the sample was four. On the diversity of partnership, a considerable variation in the number of different types of organisation included within a project partnership was shown by the sample. Among projects implemented by more than one organisation the average number of types of organisation involved was three.

Project activities and outputs

As explained in Section 2.3, financial support under the CIPS programme was awarded to five types of activities.

i) Operational cooperation and coordination;

ii) Analysis, monitoring and evaluation activities;

iii) Development: development and transfer of technology and methodology (including information sharing and inter-operability);

iv) training: exchange of staff and experts; and

v) Awareness: awareness and dissemination activities.

Activity V ‘Awareness’ was the only mandatory component for projects (as from 2009) which explains the high number of projects implementing this activity (86 %).

In terms of outputs, over a third (38 %) of all CIPS projects in the sample database involved an exchange of information or practice, and this was the most common output under activity 1.

For activity 2, over half (52 %) of all CIPS projects in the sample conducted a study or evaluation. Other outputs were created by four projects within the sample and included:

1.Creation of a database, classification of information;

2.Identification of relevant geographical information systems;

3.Analysis of technical standards and methods for resistance assessment and analysis of legislative and related documents; and

4.Policy analysis for creation of a handbook 40 .

For activity 3, over a third (38 %) of sample projects undertook the development of new equipment/software while one third (31 %) were related to the development of new methodologies. Only 3 % of sample projects concerned a transfer of equipment or software. Three projects in the sample produced outputs that did not fit into the categories provided, namely:

1.Development of a new model for security in critical infrastructures;

2.Multi-level framework for the alignment of regional CIP/R strategies with national and EU strategies; and

3.Development of security standards and elaboration of methodologies for assessment of objects’ security.

Only seven projects (representing 24 % of the sample projects) undertook training activities (activity 4).

As awareness activities (activity 5) became compulsory after 2009, the vast majority of sample projects reported to have undertaken this activity. The most common output of activity 5 was the organisation of dissemination events (83 % of projects) followed by dissemination materials (69 %).

Analysis of fund absorption 41

As stated in the introduction and the implementation section, a total budget of EUR 126.8 million was allocated to the CIPS programme of which EUR 74.6 million was committed and EUR 51.7 was spent. Indeed, not all available funding was awarded due to the fact that some project applications did not meet the eligibility/ quality criteria for being awarded funding. The absorption rates are calculated using the financial data included in the annual work programme, the modifying decisions applied on these amounts and the DG Budget ABAC system.

Absorption rate of grants

During the evaluation period, a total of EUR 97.8 million was allocated to grants, of which EUR 51.5 million was committed and EUR 43.0 million was spent by 31 March 2017. This translates into an absorption rate of 83 % 42 . The table below presents an overview of the overall spending for the programme.

Table 3: Overview of allocated, committed and spent funding, in millions of EUR (rounded) by 31 March 2017

Source: ICF analysis of data from the annual work programmes (allocated), and DG Budget ABAC data (committed and spent).

Over the programming period, CIPS had a higher amount of committed funding (83 %) compared to ISEC (74 %) with regard to overall grants. The reasons of the higher absorption rate under CIPS included the strongest knowledge of beneficiaries in order to properly manage the calls, and the changes adopted by the Commission to simplify the overall implementing process 43 . Furthermore, the enhanced capacity of CIPS stakeholders in planning project’s budget could also be considered as one of the reasons that contributed to this effect.

More specifically, as reported in the Figure 1 below, an increase in the fund’s absorption was registered for CIPS as from 2007 (68 %) to 2013 (88 %), with a pick in 2011 (96 %). The average absorption rate after 2011 was 90 %, and this can be explained by the fact that stakeholders became more familiar with the programme and as a consequence they were able to provide more realistic budget projections.

Figure 1 Committed and spent budget for grants, and absorption rate, by year by 31 March 2017

Source: ICF analysis of data on committed and spent budget.

Absorption rate in the procurement sector

A total of 106 procurement items were funded by the programme. The number of public procurement items in this sector experienced a significant increase during the evaluation period, from only 6 in 2007 to over 20 in 2012 and 2013.

On CIPS procurement amounts, EUR 10.80 million was set aside in line with the annual work programme, of which EUR 10.3 million was committed and EUR 8.7 million was spent by 31 March 2017. Figure 2 below shows the absorption rate per year, an average absorption rate of 85 % was registered 44 .

Figure 2 Initial procurement amount committed, paid and absorption rate across the period 2007-2013 by 31 March 2017

Source: ICF analysis of DG Budget ABAC data.

Absorption rate of JRC actions/ administrative arrangements

Seven administrative arrangements or actions carried out with JRC were identified by the evaluators based on the ABAC data received from the Commission. Originally, a budget of EUR 18.2 million was allocated to actions and administrative arrangements with the JRC of which EUR 12.8 million was committed across the programming period. Final data on project expenditure was not available at the time of writing the report.

7Answers to the evaluation questions

7.1 Relevance

The objectives of the CIPS programme were designed to be in line with the EU policy framework in the area of critical infrastructure protection. The specific objectives were linked to the general objectives, which are preparedness and protection (Art.4 (2)) and consequence management (Art.4 (3)). Further details on the architecture of the fund are reported in Section 2.3.

The European programme for critical infrastructures protection (EPCIP), whose implementation was supported by the CIPS programme, recognises that many critical infrastructures have a cross-border dimension, which would affect a number of EU countries 45 . The Commission considered that the objectives of the CIPS Programme corresponded to the needs related to prevention, preparedness and consequence management of terrorism and other security related risks. Furthermore, the Programme responded to a real need for cross-border cooperation and coordination in the areas of prevention, preparedness and consequence management of terrorism and other security-related risks given the increase of terrorist attacks in recent years. It should be pointed out that from the interviews with the stakeholders consulted it emerged that the CIPS programme was of continuous relevance to the prevention, preparedness and consequence management of terrorism and other security-related risks throughout the evaluation period.

The continuous relevance of the Programme was also ensured by the process of setting the CIPS priorities and sub-priorities, which were identified by Member States and competent Directorate-Generals of the Commission 46 on a yearly basis and included in the annual work programmes.

The majority of Member State authorities representatives interviewed did not mention any issues related to the process of priority-setting 47 . Most CIPS coordinating organisations and partner organisations responding to the online survey considered the priorities set by the Commission in the calls for proposals/annual work programmes as relevant to the specific needs of the target group (9 out of 15), to the specific strategic needs at EU level (9 out of 15) and to the specific strategic need at national level (10 out of 15) (conclusions).

Furthermore, around 50 % of coordinating organisation responding to the survey reported that they designed the project on the basis of a needs assessment whereas 20 % highlighted that such assessment was ‘sometimes’ carried out and another 20 % replied negatively to this question 48 . Needs assessments were often carried out through meetings and discussions with relevant stakeholders, including experts, critical infrastructure operators and relevant Ministries and government agencies.

The design of the programme relates to the main features of the CIPS programme, namely its scope, funding mechanisms, types of actions and stakeholders targeted under the programme. The evaluation study assessed to what extent this design corresponded to the needs related to the prevention, preparedness and consequence management of terrorism and other security related risks during the evaluation period.

The rationale behind the CIPS programme was to provide funds where demand was highest in line with the priorities in the annual work programmes. For this reason, the CIPs programme was broadly demand-driven (based on open calls for proposals and allowed actors to apply for funding based on a project proposal) rather than policy-driven. Indeed, if no project promoters applied for funding as part of a specific priority in a given year, that particular priority would not be implemented. However, the Commission could add remaining funding from targeted calls to a subsequent general call. The logic behind the annual work programme was to provide funds where demand was highest in line with the priorities outlined in the annual work programme.

The demand-driven design of the programme contributed to a significant geographical imbalance in its implementation, especially for the location of the coordinating organisations (see Section 7.1). For this reason, the Commission took measures to boost the geographical spread, particularly through the organisations of information days in the Member States. However, the programme’s implementation depended heavily on the pro-activeness of relevant government representatives and potential applicants from the Member States.

On the thematic coverage of the programme, as provided in Section 2.3, this was defined in broad terms and covered:

1.Risks: terrorism and other security-related risks (this could include other criminal activities, natural disasters etc.);

2.Protection of people and critical infrastructure;

3.Pre-attack (preparedness and prevention) and post-attack (consequence management);

4.Unlimited range of sectors: energy, cyber, food, health, space, etc.;

5.Activities: developing tools, methodologies and guidelines, training, exchange of experience, best practice and know-how, etc.

The evidence gathered confirmed that this extensive coverage was perceived as positive by consulted stakeholders as it allowed for flexibility throughout the funding period. Meanwhile, a finding emerging from the various consultation rounds and expert opinions was that the broad coverage of the programme might have contributed to the lack of clarity of its scope, which may have rendered the drafting of applications challenging for a number of organisations. For this reason, the difference between European and national critical infrastructure could be made clearer, as CIPS has a strong link to the EPCIP.

The funding mechanism was considered appropriate by the majority of stakeholders consulted 49 . Moreover, the close involvement of the JRC in the implementation of the programme was considered to be positive (further details on the role of the JRC are reported below). However, a minority of stakeholders considered that the programme should have been more flexible, as some of the eligibility criteria for obtaining CIPS funding discouraged some organisations from applying.

7.2 Effectiveness

To report on effectiveness of the Fund, the analysis presented below is structured around the two general objectives and the seven specific objectives set out in the legal basis. The assessment is based on nine evaluation questions (EQsB2 to B6).

General Objective 1- Support Member States’ effort to prevent, prepare for and to protect people and critical infrastructures

General Objective 2 — Crisis Management

Specific Objective 1 — Stimulating, promoting, and supporting risk assessments on critical infrastructures:

Specific objective 2: Stimulating, promoting, and supporting the development of methodologies for the protection of critical infrastructures (risk assessment methodologies)

Specific objective 3: Promoting and supporting shared operational measures to improve security in cross-border supply chains

Specific objectives 4: Promoting and supporting the development of security standards, and exchange of know-how and experience on protection of critical infrastructures:

Specific Objectives 5: Promoting and supporting Union-wide coordination and cooperation on protection of critical infrastructures;

Specific Objective 6: Promoting and supporting exchange of know-how and experience related to the consequence management, in order to establish best practices:

Specific Objective 7: Promoting joint exercises and practical scenarios in consequence management, including security and safety components, in order to enhance coordination and cooperation between relevant actors at the European level:

Overall, the evaluation findings suggest that the programme broadly achieved its general and specific objectives. This is supported by most coordinating organisations responded that the objectives were fully achieved (13 out of 14 respondents). Nevertheless, these findings should be seen in the light of the methodological limitations presented in Section 5, in particular the fact that the progress was assessed at output level mainly, on a limited sample of projects and based on qualitative evidence. Also, when looking to effectiveness, it should be taken into account that the programme’s first general objective was only to support Member States efforts in protecting people and critical infrastructure without substituting to Member States in this area. In addition, the relatively small allocated budget of EUR 126.8 million — in comparison to other EU programmes as well as national budgets in CIP area — should be also taken into account when assessing the results and impacts of the programme. Finally, the broad scope of the programme, resulting in a high number of projects of relatively small value, should also be taken into consideration when analysing its overall effects.

Regarding the results and outcomes of the sample of projects analysed (29), from the evaluation study, it emerged that the majority of these projects (62 %) have led to an improved understanding of methods, tools and practices, 59 % to an increased level of knowledge and skills of practitioners and 52 % to enhanced networking and forms of sharing information and best practice.

In terms of impacts, the majority of projects analysed (16) contributed to the increase of prevention capacity to protect people and critical infrastructure against terrorism and other security-related incidents. A high level of projects (15) contributed to increase the level of preparedness whereas only 7 projects contributed to increase risk assessment capacity on critical infrastructures. Conversely, no projects contributed to increase response capacity. Other impacts registered were the exchange of good practice, between Member States on technology and the improvement of participants’ skills and knowledge.

On the overall implementation of activities, the majority of coordinating organisations involved in CIPS projects who participated in the survey reported that all project activities (12 out of 14 respondents) or most of them (2 respondents) were delivered.

From the analysis of the sample of projects for in-depth analysis it also emerged that 90 % of projects achieved all of their objectives whereas only 10 % of projects partially achieved their objectives. The final assessment carried out by the Commission on the sample projects reported 7 % of projects as being excellent, 59 % as having had positive results and impacts and 21 % were considered acceptable. Furthermore, 26 projects from the sample (out of 29) reached their initial expectations. On the other hand, some of them also implemented activities not originally envisaged such as additional external events and/or higher number of participants to activities. However, the surveyed also reported on obstacles encountered during the project’s implementation and these included, for instance, (i) the low level of participation from project participants (6 projects) and the lack of cooperation from partners (1 project), (ii) technical difficulties, and (iii) language barriers. Moreover, based on the final evaluation carried out by the Commission it emerged that some outputs could have been better designed to gain a better outreach. Indeed, considering that CIP is technically complex, the results were perceived in some cases as being too technical to promote wide use among policymakers. In certain other instances, the Commission highlighted that some projects had a too strong national focus which prevented the broader use of their results.

In terms of actions funded under CIPS, a representative sample of 29 projects have been reviewed and showed that most of the projects were funded in the sub-area of protection of critical infrastructure and people (10 projects or 35 % of the sample), followed by risk assessment (eight projects or 28 % of the sample). Most of the examined projects (20 projects or 69 %) had the development of methodologies for the protection of critical infrastructures as an objective. More precisely, 38 % of the sampled projects aimed to contribute to the establishment of frameworks for the exchange of know-how and experience on the protection of critical infrastructures; 21 % aimed at establishing best practices and coordinating response measures; 17 % included coordination and cooperation on protection of critical infrastructures; 14 % focused on joint exercises and practical scenarios and 7 % on operational measures to improve security in cross-border supply chains.

Evidence collected through the stakeholder consultation showed that results based on the following two components were perceived as particularly valuable and contributed to the increase of the overall effectiveness of the programme:

a) potential for transferability to other sectors and other Member States; and

b) readiness to be used by critical infrastructure owners and operators in contrast to prototypes and models that are not operable or market-ready.

In consequence, for the future, similar programmes should be carried out in order to motivate coordinating organisations to work towards the development of demonstrators or prototypes (to be developed as market-ready products) which are based on the results of previous projects.

Furthermore, given the relative newness of the Programme and the lack of resources of the Commission at the time to closely follow all CIPS projects from start to end, the evaluation showed that a dedicated technical assistance budget could have been useful. Such a budget could have better supported the development of a state of the art monitoring system to collect and analyse data on financial progress, outputs and results of projects, as well as provided technical expertise throughout the lifecycle of projects and increased the dissemination of results. Therefore, the lack of a dedicated technical assistance budget has negatively affected the effectiveness of the programme.

Sectoral analysis of CIPS projects

Projects were categorised per specific sectors depending on their aims and purposes. The sectoral dimension has enabled the coverage of sectors with critical importance such as energy, transport and telecommunication. However, some sectors were covered to a much greater extent than others, and the more popular were the information, communication and technology (ICT, 23 %), transports (18 %) and energy (12 %). Most of the projects were ‘multi-sector’ which means that they covered more than one sector. Nearly a third of all projects were granted in 2009, half of them were led by a variety of bodies in Italy (including universities and research institutes, government ministries and regional/city councils). The remaining projects were funded to organisations in Greece (2), Spain (5), France (1), Hungary (1), the Netherlands (1), Romania (2) and UK (2). More details on the sectoral analysis of CIPS projects are reported in Annex 7.

Dissemination

Dissemination activities on the results were carried out by the coordinating organisations and, depending on the nature of results achieved, took place via dedicated websites, press announcements, printed material and conferences. DG HOME, with the technical support of the JRC, developed the Critical Infrastructure Warning Information Network (CIWIN), which aimed to provide a platform for exchanging CIP-related information, studies and/or good practices across all EU Member States and included also the results of CIPS-funded projects.

The CIWIN became operational only in January 2013 and the stakeholders involved appreciated the Commission’s efforts in providing information on the outcome of CIP national contact points meetings and the CIWIN platform.

It should be highlighted that despite the absence of a central repository for CIPS, the active involvement of a number of Directorate-Generals in the implementation of the CIPS programme proved that the monitoring and dissemination activities were shared between the Directorate-Generals involved.

The results produced by CIPS-funded projects implemented by JRC are stored in their publication repository: PUBSY 50 .

Overall, the Commission evaluated the results of the CIPS projects on protection methodologies as good on average, and the transferability of the outputs was particularly appreciated, stressing the importance of being able to adapt the outputs to other sectors and to implement them on a large scale.

More than 60 % of all CIPS-funded projects focused on developing frameworks, methodologies, tools and approaches for risk assessment at a sectoral level 51 . Some of the tools and methodologies that resulted from CIPS projects were the following:

i) A risk assessment toolbox for assessing the risks of terrorist attacks against specific critical infrastructures;

ii) Risk assessment conducted on online social media through the analysis of open source intelligence (SNAPSHOT project); and

iii) Methodologies dedicated exclusively to risk assessment of interdependencies and contingency plans of critical infrastructures.

As an example of projects focusing on risk assessments at sectoral level, the project Threat-Vulnerability Path Identification for Critical Infrastructures Compilation (THREVI2) was led by a private company (NIER INGEGNERIA S.p.A., Italy) and is a compilation of a comprehensive all-hazard catalogue for critical infrastructures. A literature review of data and historical events to develop a comprehensive catalogue of hazards and threats was produced and disseminated in conferences and workshops.

Furthermore, when analysing the risk assessment, a particular attention was given to the ‘cascading effect’ of common infrastructures. Indeed, to ensure the effectiveness of the risk assessment, the cascading effect needs to be taken into account in order to prevent and reduce consequences of security incidents. Critical infrastructures are interdependent within different sectors, not only for the proximity of their geographical location but also because of resource exchange or cyber dependencies. Such interdependencies, if not well assessed and considered, may lead to catastrophic cascading failures and domino effects. For this reason, approximately 20 projects funded by CIPS dealt with interdependencies of critical infrastructures, predominantly with a focus on risk assessment. The focus of a number of CIPS projects on interdependencies and ‘cascading effects’ in case of disruption and destruction of critical infrastructures during a terroristic attack or natural disaster was considered to have contributed to the programme’s objective related to critical infrastructure protection.

The JRC also contributed to the promotion and support of risk assessment capacity on critical infrastructures with different specific projects. For instance, one project on studies and support related to threat and vulnerability assessment (2008) and another one on Risk Assessment methodologies and tools for CIP entitled ‘Increasing resilience against all-hazards’ (2013).

It is not possible to draw clear conclusions on CIPS’s contribution in this area due to the small number of projects implemented in this sector. Only 6 out of 126 projects were dedicated to the security of cross-border supply chains, and focused on drinking water, food, electricity and ports. Only one project did not focus on a specific type of supply chain but was dedicated to the improvement of the management of cross-border disruptions. On the projects’ objectives, these varied and included: (i) identification and assessment of cascading effects in the global ports’ supply chain, and (ii) strengthening the resilience capacities required to manage cross-border critical infrastructure disruptions.

Some experts considered the wording ‘supply chain’ as misleading given that what needed to be protected was the cross-border infrastructure distributing the goods and not the ‘supply chain’, which refers to the whole distribution process.

On average, the results of CIPS projects on exchange of know-how and experience were considered as good by the Commission, and the transferability of the outputs as particularly valuable.

Overall, 17 projects focused on the exchange of know-how and experiences related to both protection and crisis management. For instance, the ‘MIRACLE’ project aimed to support the coherent development of critical infrastructures protection and/or resilience (CIP/R) strategies at regional level. Its final objective was to improve existing capacities of EU Member States to prevent, prepare and protect people against terrorist attacks and security-related risk by setting up an international network of professionals and institutions for the exchange of know-how in the resilience of critical infrastructures. Following selection and analysis of best practice, a manual dedicated to sharing good practice within Member States was produced. In term of results, it was considered as user-friendly and well adapted to the target audience, represented by policymakers.

The results of CIPS projects on EU-wide cooperation and coordination on CIP were evaluated by the Commission as not very effective, with a few peaks of excellence.

Several projects contributed to promoting and supporting EU-wide coordination and cooperation among Member States.

In prevention, EU-wide cooperation and coordination is linked to the exchange of information on threats among public authorities and stakeholders involved in the protection of critical infrastructures. CIPS funded for example the European Cooperation Network on Critical Infrastructure Protection (EUCONCIP) which brings together experts on CIP from across Europe and set the ground for the creation of a European partnership between critical infrastructures stakeholders, with the aim to strengthen public-private international cooperation, planning, networking and cross-border exchange of information.

As another example of project, the ‘security liaison officer’ had the objective to support the creation of the role of liaison officer in charge of security for all European and national critical infrastructures. Although every critical infrastructure has a security manager, the role of security liaison officers is to enable the protection of the European critical infrastructure through better information sharing and cooperation as security managers could also have other roles and responsibilities.

Overall, a significant number of projects (19 projects) contributed to promoting joint exercises and practical scenarios between different actors at European level.

An example of interesting CIPS project on consequence management is the ‘Disaster 2.0’ project. It aimed at establishing how successful the Web 2.0 platforms 52 were in mitigating damage to critical infrastructure, building resilience in the public, and what were the challenges encountered by their use. The project also explored the potential of next generation Semantic Technologies (Web 3.0) in disaster response. Furthermore, it collected substantial data on behaviour, uptake and barriers to the use of social media in emergency management and by emergency management agencies. As envisaged, it also developed a vision for the application of semantic technologies in this field by emergency management agencies, for which it created a prototype software.

7.3 Efficiency

As mentioned in Section 5, due to the lack of an initial baseline and to the difficulties encountered in comparing project costs, the assessment made of efficiency below relies largely on the qualitative data collected through the stakeholder consultations and interviews. Furthermore, the low number of respondents to the surveys (19 %) should be also taken into account.

In order to improve the inefficiencies related to accessibility, the interim evaluation of the programme’s implementation completed in 2010 identified certain areas for improvement. Consequently the Commission adopted the following measures:

i) Changes related to the application and award process

Overall, the changes introduced to the application and award process contributed to the decrease of ineligible project applications. However, some other factors such as the improved know-how of the application system and rules may have also contributed to this decrease.

Application and awarding/grant agreement procedure — guidance for applicants

The Commission organised some information days in 2009-2011 in Brussels which aimed to explain the application procedures to potential applicants. Some regional information days were also held in seven Member States (AT, IT, RO, HU, SE, SI and the UK) in 2010-2011. A significant increase in the total number of applications was registered from 2011.

The Commission also improved the guidance documents for the calls for proposals. However, there was still room for improvements as the volume of accompanying documents could have been seen as burdensome and the information could have been compiled in a single package.

Overall, 10 out of 14 respondents to the survey for CIPS coordinating organisations were satisfied with the management of the application process applied by the Commission. Furthermore, 9 out of 13 respondents also reported that they did not encounter any issue related to the signing of grant agreements (conclusions). However, a limited number of respondents (4) highlighted that they experienced certain problems, such as communication issues due to the numerous changes affecting the Commission’s staff and a lack of flexibility in the administrative process.

ii) Revision to the process of budget verification in 2010

Between 2007 and 2010, the Commission carried out a detailed analysis of the eligibility of the budgets outlined in the project applications. Consequently, a more efficient approach on the negotiations for the proposed budget with the beneficiaries was adopted by moving the negotiations before the award decision (whereas originally these took place between the award decision and the signing of the agreement). This change led to a sudden decrease of the number of days between the award decision and the grant agreement signature. Furthermore, from 2010 the concept of ‘value for money’ was considered as an award criterion hence cost-effectiveness of projects increased. The ‘value for money’ principle was overall considered as a key driver across all activities 53 by the majority of stakeholders therefore it is recommended to fund similar activities in future programmes.

iii) Strengthening the rules of cross-border partnerships

The Commission restricted the rules on the eligibility criteria on cross-border projects as, in the first years of the programming period, the Commission was not in a position to easily check the degree of cross-border partnerships. For this reason, for the 2007 call for proposals, applicants had to ensure that cross-border projects involved at least one partner organisation from another Member State or from an acceding/candidate country. However, stricter rules followed from 2008, such as the obligation to sign a partnership declaration.

iv) Maximum length of projects

In 2008, the maximum length of a project was decreased from three years to two years. Several reasons led to this decision, for instance, the fact that reducing the project’s length contributes to a better match between the implementation period and the period when the policy formulation was laid down in the annual work plan. Indeed, the two years duration was considered as the best time period reflecting overall the timeline set to achieve national policy priorities.

v) Minimum and maximum EU grant

From 2010, the minimum EU co-funding amount to be requested increased from EUR 50 000 (2007-2009) to EUR 100 000. The amount was increased mainly because higher amounts were needed for high technological investments.

Agreements on budget thresholds were discussed between Member States and the Commission. However, increasing the minimum EU grant was perceived as most problematic in EU-12 Member States.

The maximum rate of the EU grant share increased consistently with the increase in minimum EU grant, from 70 % (period 2007-2009) to 80 % in 2010 and remained at 90 % from 2011 onwards for the remainder of the programme implementation period.

Pre-financing rules

To improve cost-effectiveness, the CIPS programme had generous pre-financing rules 54 , i.e. coordinating organisations received 60 % (until 2010) to 80 % (in 2011) at pre-financing stage. The more generous advance proved to be particularly helpful for the smaller organisations to properly set up the partnership and undertake different actions.

Overall, coordinating organisations interviewed considered this as advantageous and no specific issues were mentioned.

vi) Changes to reporting requirements and monitoring of results

Financial reporting

Project partners were requested to provide information on the financial aspects of project implementation. The 2010 calls onwards, financial reporting became more detailed on cost claim requirements. Therefore, further guidance was provided in separate documents. A few grant beneficiaries stressed the fact that reporting project expenses in detail was too burdensome.

Monitoring of results and technical reporting

As stated above under the effectiveness section, one of the main weaknesses of the management of the programme by the Commission was the absence of a centralised coordinated approach for monitoring project results.

Furthermore, as other Directorate-Generals were involved in the programme’s implementation 55 , coordinating the monitoring of results across them was difficult to ensure.

For this reason, interim reporting was introduced to strengthen monitoring results through data collection and to allow coordinating organisations to take stock of progress made to date and/or possible needs for adjustments.

Although projects were required to consider EU added value as part of project implementation, there were no obligations to report on this. The reason behind this choice was to reduce the burdens on organisations, based on the fact that more detailed reporting alone does not ensure effective monitoring of results, which should consequently be accompanied by the formulation of indicators and quantification of results. Nevertheless, this still represents a shortcoming and it is recommended to include EU added value and a self-assessment of key results and overall impacts on the monitoring report.

Administrative burden

Despite the changes introduced during the CIPS programming period, some stakeholders raised concerns on the level of administrative burden. Five respondents to the survey for coordinating organisations considered the level of administrative burden as ‘significant’ whereas three of them as ‘very significant’. Similar concerns were raised by partner organisations.

Private companies involved in the CIPS implementation and who have implemented in the past projects under Framework Programme 7 reported a higher level of administrative burden under CIPS. From the interviews it emerged that the main obstacles were a lack of flexibility and excessive bureaucracy (e.g. high level of detail required conversely to some other Programmes such as Framework Programme 7 and H2020) in relation to administrative processes and the need for simplification of reporting in terms of length and detail (e.g. financial information requirements which is further explained below under ‘monitoring and reporting’).

A shift towards shared management as a management mode for the successor fund

Unlike CIPS, which was implemented 100 % under direct management, the successor fund, ISF-Police, is being implemented on the basis of a combination, with around 60 % of the available funding being allocated to national programmes under shared management and 40 % allocated to EU actions under direct management. This change is expected to have an impact on the efficiency and effectiveness of the Commission’s management of the ISF programme. However, it was not in the scope of this ex post evaluation to determine to what extent the shift to shared management has provided efficiency and effectiveness gains in the context of the current ISF-Police programme 56 .

On the cost-efficiency of the financial resources deployed a budget of EUR 126.8 million was allocated to the CIPS programme as laid down in the annual work programme 2007-2013, of which EUR 74.6 million was committed and EUR 51.7 million was spent. There was an increase in the fund’s absorption as from 2007 (68 %) to 2013 (88 %), with a peak in 2011 (96 %). The average absorption rate after 2011 was 90 %, and this can be explained by the fact that stakeholders became more familiar with the programme and in consequence they were able to provide more realistic budget projections. The EU funding provided was perceived by the vast majority of stakeholders consulted as having been sufficient and as having been highly cost-efficient for all activities implemented under CIPS.

Overall, the evaluation analysis showed that 88 % of the sample projects estimated budget was spent. However, some differences between the estimated budget included in the grant agreement and what was effectively spent (as declared in the final cost statement) were registered for certain costs categories. This led to a lower absorption of funding than what was expected. It should be highlighted that most underspend related to costs for travel and for conferences and seminars. Variations in expenses from the initially forecasted budget to the final declared project expenditure were due to different reasons. For instance, the fact that costs finally resulted higher or lower than expected or the fact that outputs were not delivered.

Sufficiency of EU funding

On average, 88 % of projects costs were funded by the EU contribution (for the 28 sample projects analysed). For these projects, on average 74 % of financing was made up by the EU contribution. The projects less dependent on the EU contribution registered 64 % of income made up by the EU contribution, and only 10 out of 28 projects had an EU contribution of 88 % or higher.

Overall, coordinating organisations reported that EU funding was sufficient to implement the planned activities. Only a limited number of stakeholders highlighted the need for additional funding and the fact that certain costs were underestimated, i.e. staff’s costs.

Value for money

Overall, for four out of five activities formulated in the survey 57 , at least half of all respondents to the coordinating organisations and partner organisations considered these to be high to very high value for money. On awareness and dissemination activities, just under half of the coordinating organisations responding to the survey considered these to be high to very high value for money (with the total number of respondents ranging from 13 to 14 per activity).

Efficiency of human resources deployed

On the cost-efficiency of the human resources deployed, a few key performance indicators were introduced by the evaluation team to measure the efficiency of programme management by the Commission. On human resources, they were considered sufficient for a fairly low number of projects across the whole programming period.

However, certain coordinating organisations also highlighted few weaknesses. For instance, the frequent staff changes and internal re-organisations (at their level and at the level of the Commission) led to more time-consuming communications, with valuable knowledge getting lost in the process. More specifically, five coordinating organisations also reported the absence of follow-up/feedbacks from the financial operational officers.

Several key performance indicators related to the Commission’s management of the implementation of the programme were stipulated in the Financial Regulation and are reported below.

1.With regards to the time to award (time from the call for proposals’ application deadline until the award decision), the evaluation showed an overall negative trend. However, it should be noted that it remained below the maximum of 6 months anyway.

2.On the time to contract (time from the award decision until the grant agreement signature) the average number of days was 122. A positive trend was registered during the evaluation period, with the average time reduced by more than half (from 181 days in 2007 to 53 days in 2013).

3.On the time to pay (time to pay the pre-financing and the final invoice), the evaluation showed that, overall, 73 % of payment requests were paid on time, with 27 % being late. Significant differences were identified between the pre-financing stage (87 % of requests were paid on time) and the final payment stage (only 59 % of invoices were paid on time).

Analysis of project applications

According to the award decisions, a total of 352 applications were received over the period 2007-2013 and 129 grants were awarded (37 %). The success rate increased from 53 % in 2007 to 69 % in 2008. However, from 2009, the success rate started to decrease (24 % in 2013). The two last years of the programme the number of applicants almost doubled, nevertheless, the number of projects awarded remained constant over the period.

Between 2007 and 2013, 224 applications were not awarded a grant. 91 were ineligible and 133 were unsuccessful. However, the ineligibility of projects started decreasing as from 2010. This was most probably due to the fact that eligibility criteria were better followed and understood in the later years of the programme and that the changes introduced by the Commission to the guidance and the application procedure helped applicants to produce conform proposals.

7.4 Coherence

Coherence with other EU programmes

To avoid overlaps between CIPS programme and other similar EU programmes, some internal mechanisms such as consultations and inter-service consultations were established. Moreover, a representative of DG Enterprise for the security research funding in Framework Programme 7 was involved in the evaluation of proposals for the first two calls for CIPS proposals 2008/2009 to the extent needed to avoid duplication of what was done under Security Research in Framework Programme 7.

The Framework Programme 7 and the Civil Protection Financial instrument were selected for analysis and comparison with the CIPS programme. The coherence between different funding instruments can be multi-dimensional in terms of: (i) objectives and themes, (ii) programme management and eligible actions, and (iii) target groups.

Coherence with Framework Programme 7

Overall, the evaluation study concluded that the activities implemented under Framework Programme 7 in security were coherent with the CIPS programme. Indeed, this latter focused on prevention and protection measures against terrorist attacks as well as crisis management whereas projects under Framework Programme 7 were more specifically oriented to conduct research on development of new technology solutions for civil protection, including risks arising from terrorist attacks.

The JRC also ensured synergies between the two programmes. More specifically, the actions carried out with the JRC under the CIPS programme and the activities of the JRC on CIP carried out under FP7/H2020 were merged into two JRC actions (first CIP action, and as of 2013 CIP and ERN-Sys actions). Under the administrative arrangements between DG HOME and JRS, some actions under CIPS were carried out by the JRC and included in the annual work programmes. However, given that the JRC also had its own annual work programmes (based on the Framework Programme 7); actions funded through CIPS were designed in coherence with the JRC’s own priorities.

No overlaps were identified in terms of target groups of CIPS and Framework Programme 7. Furthermore, the mapping of CIPS projects revealed that a number of other groups of beneficiaries were targeted, including the private sector, NGOs, the general public, researchers and universities.

In terms of eligible actions, the evaluation identified possible incoherencies and overlaps between the eligible actions of CIPS and Framework Programme 7. As stated above, actions funded under the Framework Programme 7 security area included, for instance, ensuring the security of citizens, infrastructure and utilities through technology solutions for civil protection and/or analysing and securing critical infrastructure. There was scope for complementarity with CIPS activities on: ‘development and transfer of technology and methodology’; ‘analytical, monitoring and evaluation activities’ and ‘operational cooperation and coordination’.

Coherence with the Civil Protection Financial Instrument

On the Civil Protection Financial instrument, its actions specifically related to research on prevention measures and disaster response and management, hence coherence between the two instruments was also ensured.

No overlaps were identified in terms of target groups of CIPS and the Civil Protection Financial Instrument. Furthermore, the mapping of CIPS projects revealed that a number of other groups of beneficiaries were targeted, including the private sector, NGOs, the general public, researchers and universities.

On actions funded under the Civil Protection Financial instrument, they were set out in Article 4 of Council Decision 2007/162/EC Euratom and included e.g. studies; training, exercises, workshops, exchange of staff and experts; public information, education and awareness raising and associated dissemination actions and monitoring, assessment and evaluation activities. The scope for complementarity between CIPS and the Civil Protection Financial instrument was ensured in all eligible actions under CIPS.

Overall, little to no evidence of overlap was observed between the three programmes compared, i.e. CIPS, Framework Programme 7 and the Civil Protection Financial instrument due to their differences in terms of thematic focus, eligible actions and eligible stakeholders and target groups. Furthermore, good coordination between different Directorate-Generals in the Commission (including DG CNECT, DG ECHO, DG MOVE, DG ENER, DG JRC, etc.) through CIPS inter-service group was ensured. Therefore links and complementary were enhanced.

Synergies between the results of actions funded under the CIPS programme and the results of similar actions supported under other EU programmes

The assessment of this evaluation question was limited by a lack of data collected. It is based on the stakeholders' responses to an online survey. A total number of 20 replies were received. Bearing this limitation in mind, 14 out of 20 respondents reported that they knew about other activities implemented outside the project and covering similar objectives only partially. Activities identified as covering similar objectives were for instance other Framework Programme 7 funded projects and/or other projects under ENISA 58 and were implemented at national, EU and international levels. Examples of links were provided only by six respondents who replied to this question and included, for instance, development of networks and exchange of best practice and links related to the exchange of results and mutual update.

Coherence with other related actions supported by other national resources

The assessment of this evaluation question was limited by a lack of available data due to the fact that, according to a number of Commission officials interviewed, national actions were not systematically reviewed on their potential coherence/similarities during the design and implementation of CIPS. From the interviews with some Member States where no coordinating organisations were presents, it emerged that the scope of the CIPS programme, maybe because of its broadness, was lacking clarity.

Furthermore, certain of these Member States reported that there was no need for extra funding since CIP programmes were already established at national level, hence no risks of duplication occurred. Although the data collected for this evaluation study showed little evidence of coherence between CIPS and the national frameworks, overall, it can be concluded that Member States were aware of the scope of the programme as they received a draft of the work programme and were invited to provide their comments. In a few cases, the evaluation identified connections between CIPS and national actions. For instance, the German federal cabinet in 2009 approved the national KRITIS-strategy, and this was in line with the CIPS programme 59 thus coherences were ensured.

7.5 EU added value

Overall, stakeholders consultations showed that the added value of the programme is closely linked to the ‘importance’ of EU funding for the organisations involved and to its ability to foster cross-border cooperation which would often not have taken place if projects were supported through national funding alone. Findings suggest that organisations struggled to identify and access national funding opportunities to implement the programme’s activities hence it can be assumed that a significant part of them would have not been developed in the absence of the CIPS funding.

Coordinating organisations confirmed that, during the programming period, only few national funding opportunities were available to develop the activities implemented under the various policy areas of the Programme. 9 out of 13 respondents to a dedicated survey stated that the project activities would not have been implemented without EU funding. Only 4 respondents reported that the project could have been fully financed through other funding sources. Nevertheless, these respondents stressed the fact that they preferred to use the CIPS funding anyway, as the latter provided them with the opportunity to work with partners in other countries and to have an impact at EU level. Out of the 9 respondents who reported that they could not have funded their projects through other sources, 8 indicated that it was due to the unavailability or insufficiency of other funds whereas 3 explained that other potential funding sources typically did not support their project’s aims.

The CIPS programme had a strong transnational dimension as reflected in its objectives which were in line with the EU objectives on the development of EU policies in the prevention, preparedness and consequence management of terrorism and other security-related risks and on enhancing coordination and cooperation between relevant actors at EU level in protection of critical infrastructure.

However, the EU added value may have been reduced by the fact that, as mentioned above 60 , CIPS projects were mostly led by coordinating organisations from a limited number of Member States (mostly Italy, Spain and to a lesser extent the United Kingdom and France). Despite the Commission's efforts in trying to involve applicants from other Member States through regional information days, the geographical spread remained unchanged. The evaluation found different reasons for the significantly varying degrees of uptake across the Member States and included, besides issues such as the ability of potential applicants to draft high-level quality proposals, also the pro-activeness of Member States' Committee representatives promoting CIPS amongst relevant organisations.

Conversely, a better geographical balance was reached with partner organisations where 30 or more of them located in 3 Member States (i.e. Italy, Poland and Spain) and 15 or more in 5 Member States (i.e. Czech Republic, France, Germany, Romania and the UK). Overall, partner organisations located in 21 of the 28 Member States participated to five or more CIPS projects, achieving a better geographical spread with regard to partner organisations than coordinating ones.

EU added value was enhanced by the fact that the vast majority of CIPS projects were cross-border, with 73 % of projects including at least one organisation from a Member State different than the Member State in which the coordinating organisation was located. From 2009 the eligibility criteria on cross-border projects became stricter to ensure that each cross-border partner had an active role in the activities implemented.

Overall, coordinating organisations interviewed assessed positively the partnerships established through projects. From the survey it emerged that the majority of these organisations considered the EU added value of the partnerships as positive a. For instance, it contributed to improved knowledge and expertise on the topic of the project (8 out of 14) or it helped to gain a better understanding on how neighbouring countries work (6 out of 14).

Furthermore, coordinating organisations interviewed assessed positively the partnerships established through projects and considered it brought EU added value. Examples of EU added value were the possibility to collaborate in real-life scenarios, in particular when it concerns cross-border cooperation (i.e. joint patrols to reduce terrorism and other security related risks) and the exchanges and training of staff from different countries engaged in protection of CI. The most important EU added value was found in the increase of knowledge and expertise related to a certain topic and the increased knowledge of EU policies related to this topic, according to 13 respondents (out of 14 responding to the online survey for coordinating organisations).

Similarly, the analysis of the sample projects showed that, overall, the activities implemented and the outputs provided by both cross-border and national CIPS projects contributed to the EU dimension of the programme.

69 % of the projects were dedicated to the development of common methodologies or tools, for instance on risk assessment. The activities, implemented by national and cross-border partnership, contributed to the EU added value as they produced outputs to be used in other Member States. As an example of EU added value projects, the Threat-Vulnerability Path Identification for Critical Infrastructures (THREVI2) was a national project which contributed to the implementation of Directive 2008/114/EC by promoting the development of risk assessment methodologies. Another national project which developed methodologies to be used at EU level was the ‘Interactive Risk Assessment in critical infrastructure based on the Integrated Geographic Information System’. It developed a practical tool for protection of critical infrastructure and identified interdependencies between sectors in CIP.

8CONCLUSIONS

This section summarises the evaluation findings and provides an assessment of the CIPS' overall role in the implementation of prevention, preparedness and consequence management of terrorism and other security-related risks.

Relevance

Overall, the evaluation showed that the CIPS programme was of continuous relevance to the prevention, preparedness and consequence management of terrorism and responded to a real need for cross-border cooperation and coordination in the above mentioned areas. The stakeholders that were consulted reported that such cooperation would not have been financed through alternative sources of funding, such as national budgets, due to the financial crisis that occurred during the programme period.

The relevance of the CIPS programme was also ensured by the process of priority-settings: these were identified by the Member States and competent Directorate-Generals on a yearly basis and included in the annual work programmes. Therefore an updated and more detailed analysis of needs was ensured. Most CIPS coordinating organisations and partner organisations responding to the online survey considered the priorities set by the Commission in the calls for proposals/annual work programmes as relevant to the specific needs of the target group, at EU level and national level. However, the demand-driven design of the annual work programme resulted in a significant geographical imbalance in terms of the number of projects funded per coordinating organisation in each Member State, although a slightly more balanced spread was achieved for the partner organisations.    

In order to allow Member States more equal access to funding and to improve their participation, a shift towards shared management mode was introduced for the successor fund, ISF-Police. Unlike CIPS, which was implemented only under direct management, ISF-Police is being implemented on the basis of a combination, with around 60% of the available funding being allocated to national programmes under shared management and 40% allocated to Union actions under direct management. This change is expected to have an impact on the efficiency and effectiveness of the Commission’s management of the ISF Programme.

The evaluation also identified a time lag between the formulation of priorities under AWPs and their actual implementation; this was due to the fact that the AWPs reflected the priorities set in the previous year, which meant that there was a one year difference between the setting of priorities and their implementation, which at times hindered the programme’s relevance. This was partly addressed in the successor fund, ISF-Police, through the inclusion of emergency assistance “to address urgent and specific needs”. 61

Effectiveness

Bearing in mind the limitations of the evaluation study, it can still be said that the CIPS programme broadly achieved its general and specific objectives. The in-depth analysis of the results, outcomes and impact (where this was evident) of the sample of CIPS projects and on the views gathered from the stakeholders consulted, showed that CIPS contributed positively on critical infrastructure protection. However, when assessing the impacts of the programme, its relatively small budget and wide scope (resulting in a high number of relatively small values of projects) should be taken into account. An important element of a number of CIPS projects was the focus on interdependencies and preventing ‘cascading effects’ in case of disruption and destruction of critical infrastructures during a terroristic attack or other security related risks, thus improving its overall contribution to critical infrastructure protection. However, the results of CIPS projects on EU-wide cooperation and coordination on CIP were evaluated by the Commission as not very effective, with a few peaks of excellence.

Certain issues which hindered the assessment of effectiveness (and consequently of efficiency and EU added value) were also identified. Given the relative newness of CIPS and the lack of capacity at the time within the Commission to closely follow all projects from start to end, the absence of a dedicated technical assistance budget that could have supported a monitoring system to collect and analyse data on (financial) progress, outputs and results, has negatively affected the effectiveness of the Programme. Key elements of this monitoring system could have been, for example, the use of monitoring visits and the use of an IT system to systematically record data on financial progress, outputs and results. This has been addressed to a limited extent under the successor fund, ISF-Police, through the possibility for the financial instrument to contribute annually to technical assistance activities, which could be used for the development of a monitoring system. However, under ISF-Police a problem remains in this area, given that the technical assistance available cannot be used for staff contracting and the development and implementation of a monitoring system requires also additional human resources on both the Member State and the Commission side. This remains to be addressed in the future MFF.

Furthermore, the evaluation found that the creation of a central repository containing detailed data on individual CIPS project results and outputs would have greatly enhanced the effectiveness of the programme. For example, methodologies developed to investigate critical infrastructures protection, best practices, training materials, etc. could have been compiled centrally and thus, their potential multiplier effect and continuous usage for other interested stakeholders in other Member States could have been maximized. In consequence, it is recommended that for projects conducted under the successor funding instrument, a repository of the results and outputs of such projects could be created. This has not yet been put into practice under ISF but should be taken into consideration for the next MFF.

Moreover, evidence gathered through the stakeholders' consultation showed that results based on potential for transferability to other sectors (and other Member States) and readiness to be used by critical infrastructure owners and operators, in contrast to prototypes and models, were perceived as particularly valuable and contributed to the increase of the overall effectiveness of the programme.

In consequence, for the future, similar programmes should be carried out in order to motivate coordinating organisations to work towards the development of demonstrators or prototypes (to be developed as market-ready products) which are based on the results of previous projects.

Efficiency

In general, CIPS was found to be efficient to a large extent. The vast majority of consulted stakeholders perceived EU funding to have been sufficient and the value for money was perceived as good to very high. No significant differences in the cost-effectiveness of individual projects were found following a comparison of similar financial inputs to their outputs and results. Some private stakeholders mentioned the higher level of administrative burden in comparison with the FP7 rules they were accounted to.

A significant upward trend in the absorption of funds spent on grants was observed, rising from 68% in 2007 to 88% in 2013, with 83% across the 2007-2013 period. Most probably, this was due on the one hand, to the changes introduced by the Commission during the CIPS programming period in order to increase the overall efficiency of the Programme and, on the other hand, thanks to the fact that stakeholders became more familiar with the Programme.

It should also be pointed out that, overall, the assessment of the data on KPIs showed a satisfactory performance with regard to the average time needed to award, to contract and to make pre-financing payments. However, in terms of timing for processing final payment requests, from the data collected it was evident that, in particular for the earlier years of the Programme, it resulted that the majority of these were not processed on time.

Pertaining to the monitoring of project outputs and results, the evaluation found that a more in depth reporting from project beneficiaries would have strengthened the overall efficiency of the Programme. Indeed, in terms of improving the monitoring of project results on the basis of the required reporting, it was concluded also that a qualitative assessment of the results of the Programme was missing due to the absence of a peer review on project results conducted by external experts. Although under the current Fund this has slightly improved as Member States have to submit an Annual Implementation Report under Shared Management and Interim and Final Reports under Direct Management, the absence of a peer review still represents a shortcoming that should be addressed under the next generation of Funds.

As already stated under relevance, with regards to dissemination of projects outputs and results, consulted experts and coordinating organisations recognised that the absence of a central repository bringing together the findings of CIPS projects informing stakeholders on available results has decreased the effectiveness and efficiency of the Programme to a certain extent. It is therefore recommended to set up such a central repository bringing together the outputs/results of the projects in future similar programmes.

Coherence

The CIPS investments made under the 2007-2013 annual work programme were coherent with activities funded under other EU Funds, namely Framework Programme 7 and the Civil Protection Financial instrument. Indeed, little to no evidence of overlap occurred between funding instruments due to the differences of these Programmes in terms of thematic focus, eligible actions and eligible stakeholders and target groups whilst demonstrating the good coordination between different Directorate-Generals in the Commission (including DG CNECT, DG ECHO, DG MOVE, DG ENER, DG JRC, etc.) through CIPS inter-service group. In this way, links and complementary were enhanced.

The JRC acted as a ‘synergy vehicle’ between CIPS and Framework Programme 7. Actions funded through CIPS were designed in coherence with the JRC’s own priorities.

On coherence with initiatives at national level, the data available was not sufficient to assess the extent to which this was achieved.

EU added value

The evaluation showed that due to a lack of national funding for cross-border cooperation in many Member States, a significant part of the activities developed under CIPS would not have been developed at all or would have not achieved the same results, in particular not at a cross-border level. Additionally, the high added value of CIPS was also linked to the high level of co-funding it provided (which could go up to 90 %).

As reflected in the objectives of the Programme, CIPS had a strong cross-border dimension. However, as mentioned above, a major weakness identified by the evaluators related to funding distribution that was uneven and concentrated in a few Members States with coordinating organisations that were mainly located in Italy and Spain. Although the Commission took measures to boost the geographical spread, particularly through the organisations of information days in the Member States, the submission of applications was still very dependent on factors such as the pro-activeness of Member States’ Committee representative’s knowledge and capacity of organisations to submit high-quality proposals and their compliance with the eligibility criteria. This geographical imbalance has been addressed through the shift towards shared management. Regarding the national/transnational characteristics of projects, the shift towards shared management has, however, meant that many projects implemented by Member States under ISF-P tend to have less of a transnational dimension than projects previously conducted under CIPS, which due to the Direct management mode naturally had a higher transnational dimension.

ANNEX 1 — Procedural information

ANNEX 2 — Stakeholders’ consultation

1. Overview

The ex post evaluation of the prevention, preparedness and consequence management of terrorism and other security related risks (CIPS) 2007-2013 programme included consultations with several stakeholders. The consultations were conducted by the Commission and by an external consultancy specialised in evaluation in the framework of their contract with the Commission to provide study on the ex post evaluation of the implementation of CIPS (‘the external evaluators’). This Annex provides an overview of the consultation processes and the type of stakeholders consulted. Moreover, it presents the results of these consultations. The evaluation assessed the relevance, effectiveness, efficiency, coherence and EU added value (‘evaluation criteria’) of the CIPS programme.

1.1 The public consultation conducted by the Commission

From 14 July 2016 to 15 November 2016, the Commission carried out an Internet-based public consultation on the CIPS 2007-2013 in the form of an online questionnaire. Contributions were sought from individuals (experts, beneficiaries), public authorities, intergovernmental and non-governmental organisations, social partners and civil society, academic institutions, international organisations, EU Institutions and Agencies, based in EU Member States and non-EU countries.

The aim of the consultation was to collect views and opinions on the results and impacts of actions co-financed by the CIPS programme in 2007-2013 and to assess their relevance, effectiveness, efficiency, coherence and EU added value. The final number of respondents was rather low: six participants. Two respondents responded on behalf of non-profit organisations (Italy), one responded as a beneficiary of the programme (United Kingdom) and three as EU citizens. The EU citizens were from Germany (two) and one from France. Given this low response rate, contributions cannot be considered as being representative of the targeted stakeholders. However, the results may provide additional insights that will be presented below together with the findings of the other consultations. The fact that the information available on the identities of the participants is based on self-reported values and cannot be verified should be considered as a further limitation.

1.2 The consultations conducted by the external evaluators

In addition to the consultation conducted by the Commission, the external evaluators conducted targeted consultations with range of stakeholders using four kinds of interviews — online surveys, stakeholder consultations, thematic case studies, expert workshops and expert input. This section provides an overview of each type of interview and consultation process that took place in each case. The unit responsible for the evaluation (DG HOME, Unit E2) supported the access to relevant EU level stakeholders. Stakeholders were contacted via email introducing the study and the importance of their contribution, including a letter of introduction from DG HOME.

1.2.1 Online surveys

The external evaluators launched three online surveys at the end of December 2016 — with coordinating organisations, with partner organisations and with project participants.

Online survey with coordinating organisations

The survey ran for four weeks — from 4 November 2016 to 5 December 2016 and it was made available in five languages: English, French, German, Italian and Spanish. The goal of this survey was to invite all coordinating organisations of CIPS projects to participate by answering questions on the evaluation criteria (relevance, effectiveness, efficiency, coherence, EU added value). The evaluation aimed to invite all coordinating organisations of CIPS projects to participate in the survey. Out of the 131 total CIPS contacts, 108 contacts were invited to complete the survey. The total number of respondents to the survey was 20 representing a response rate of 19 %. Out of these 20 responses, 15 were complete and five were incomplete responses.

Online survey with partner organisations

The coordinating organisations were asked to send this survey to the relevant partner organisations of CIPS projects. Similarly to the online survey with coordinating organisations, this one was open from 4 November 2016 until 5 December 2016 and it was available in the same five languages as mentioned above. A total number of 18 respondents replied to the survey of which 8 were complete responses and 10 incomplete responses.

Online survey with project participants

This online survey ran for 12 weeks — from 1 December 2016 until 28 February 2017. It was also available in English, French, German, Italian, and Spanish. The participants were identified with the assistance of the coordinating organisations which forwarded the survey to the participants of their CIPS projects. The total number of replies of this online survey was 15. However, 13 of the respondents were participants from the same project, so the results of this survey were treated with caution taking into account the strong bias to this particular project.

1.2.2 Stakeholder consultation

The external evaluators carried out 37 interviews with five types of stakeholders. The five types of stakeholders that were interviewed for the purposes of the evaluation were — scoping interviews with Commission officials (9), Commission officials responsible for other EU programmes (2), European and international organisations (1), Member State representatives (16), additional interviews with CIPS coordinators (9). The external evaluators carried out those interviews in a semi-structured way based on questionnaire developed for each stakeholder type. The interviews lasted one hour and were conducted via telephone or face-to-face.

1.2.3 Case studies

Following the request of DG HOME, the external evaluators carried out thematic case studies. The aim of those case studies was to examine more closely the result sand impacts in one specific thematic area, and where possible linking them to wider context at EU level. Two case studies were carried out in which three Member States participated:

-    Multi-country case study (Germany and Hungary): development of tools and protection;

-    Case study (Slovenia): risk assessment.

In total, six interviews were carried out as part of the case studies’ data collection, including four interviews with coordinating organisations of CIPS projects, and two with partner organisations.

1.2.4 Expert workshop and expert input

The external evaluators organised two expert workshops. Both were held in Brussels — one on 23 September 2016 and one on 6 March 2017. Both workshops were carried out with experts involved in the evaluation. The second workshop was followed by written contributions from the experts on the findings presented in the draft final report.

2. Results

The most relevant results of the consultations carried out by the Commission and the external evaluators are grouped in this section and presented according to the following evaluation criteria: relevance, effectiveness, efficiency, coherence, EU added value.

2.1 Relevance

CIPS’s relevance assessment aimed to find out the extent to which the objectives and the design of the programme corresponded to the needs problems and issues in the protection of people and critical infrastructure from terrorism and other security related risks in 2007-2013. On the relevance of the objectives, stakeholders consulted consider that the CIPS programme was of continuous relevance to the prevention, preparedness and consequence management of terrorism and other security related risks throughout 2007-2013. In addition, the Commission public consultation in Question 7 asked the participants to what extent the projects and the activities funded in their country of residence addressed the needs related to management of terrorism and other security related risks. Half of the respondents replied with ‘to a limited extent’, one EU citizen replied ‘significantly’ and the beneficiary of the programme mentioned that the projects and activities addressed the needs only ‘partially’.

On the relevance of the design of the programme, the evaluation explored whether the design sufficiently corresponded to the needs related to the prevention, preparedness and consequence management of terrorism and other security related risks over 2007-2013. Taking into account the design-driven design of the programme that led to a significant imbalance of geographical distribution of the programme, the results of the consultation showed that there are couple of reasons for the low level of participation: the administrative burden, the fact that CIPS was not relevant to some national context, unfamiliarity with the programme, lack of institutional capacity, lack of contacts and partners in other EU Member States.

Moreover, the broad thematic coverage of the programme was overall welcomed by the consulted stakeholders as it allowed for flexibility throughout the funding period. The funding mechanisms of CIPS were considered appropriate by the majority of the stakeholders. The involvement of JRC in the implementation of the programme was seen as positive. Those that expressed a desire for a more flexible programme design reported that some of the eligibility criteria for obtaining CIPS were a reason for some organisations not to apply.

2.2 Effectiveness

The evaluation questions under this criterion aim to assess the extent to which CIPS was effective in achieving its general and specific objectives. Question 4 of the Commission’s public consultations asked participant whether, based on their experience, the actions financed by the CIPS in their country were consistent with the objectives of the Fund. Four of the respondents replied that they ‘do not know’, one EU citizen replied with ‘to a limited extent’, while the beneficiary of the programme replied ‘partially’.

The evaluation of the effectiveness of the implementation of CIPS projects/programme suggested that the majority of the CIPS-funded projects achieved their planned objectives and results. Furthermore, the stakeholder consultation confirmed that JRC played an important role in implementing the EPCIP.

In terms of effectiveness in achieving the general and specific objectives of the programme, the evaluation showed that they were achieved and the programme contributed positively to the policy are of critical infrastructure protection. One of the limitations, however, is the fact that the programme had relatively small budget and a wide scope which led to a higher number of small value projects.

2.3 Efficiency

The efficiency of CIPS aimed at measuring the extent to which the costs of the actions funded under the programme were justified compared to the outputs. The evaluation focused on examining the Commission’s management of the programme, the financial efficiency of the programme (the extent to which the resources were used efficiently), and the human resource efficiency.

Overall, the majority of stakeholders consulted, reported perceiving some degree of administrative burden resulting from participation to the programme. However the number of respondents was very low. The majority of those that responded to the survey for coordinating organisations reported a significant (five) or very significant (three) administrative burden. A total of two respondents (out of 14) reported experiencing no administrative burden at all. On the Commission’s management of the application process the majority (10 out of 14) of respondents to the survey for CIPS coordinating organisations were satisfied.

The financial efficiency of the programme was based on analysis of the financial resources allocated, committed and spent under the CIPS programme and also the extent to which the financial resources were sufficient. Overall, EU funding was found to be sufficient by the coordinating organisations to implement the planned activities. In few cases, stakeholders highlighted the need for additional funding and the underestimation of certain costs, for example in relation to staff. A few coordinating organisations indicated that forecasting research and development costs was difficult and that these were ultimately higher than planned.

On human resource efficiency, the Commission management was overall considered positively. The results of the consultations noted the existence of some issues in the technical project follow-up, guidance and interpretation of rules. Overall there was sufficiency in the amount of human resources to deal with the number of projects, however frequent internal re-organisations taking place across the programming period have impacted on the efficiency of the management, especially with regard to the monitoring aspect of the programme.

2.4 Coherence

The evaluation criterion of coherence refers to the extent to which the intervention did not contradict or create duplications with other interventions with similar objectives. The assessment targeted coherence with other EU programmes in the same field, coherence with national programmes and initiatives and synergies between results of CIPS and results of similar action funder under other EU programmes.

The external evaluator’s analysis indicates that CIPS had ample opportunity for coherence with other EU programmes, especially with Framework Programme 7 and the Civil Protection Financial Instrument. The stakeholder consultation confirmed that there was little risk of overlap with either Framework Programme 7 or the Civil Protection Financial instrument due to the differences of these programmes with CIPS.

The results from the Commission’s public consultation did not provide for enough support to such finding. Question 10 of the consultation asked participants about the extent to which they consider that the EU funding for CIPS programme was coherent with other actions in the same area funded by other EU financial instruments. Three of the respondents replied with ‘I do not know’. Two respondents said that the EU funding for CIPS programme was coherent with other actions to a limited extent and one respondent believed that it was coherent.

The external evaluators found little evidence of coherence between CIPS and other national actions. The public authority representatives of some of the Member States commented that they already had working CIP programmes and frameworks in their countries and thus, there was no particular need for additional funding.

On the links between results of CIPS and results of similar action funded under other EU programmes, the CIPS coordinating organisations who responded to an online survey partially confirmed that they were aware of other activities implemented outside the project and covering similar objectives. Only six respondents to the online survey for coordinating organisations replied to the question relate to types of synergies that were established. As examples links in development of networks and exchange of best practice, knowledge-sharing and links related to the exchange of results and mutual update were given. In addition, the results from the survey conducted among partner organisations showed similar results.

2.5 EU added value

The external evaluators assessed the EU added value of CIPS based on three criteria- cross-border dimension, partnership aspects, and importance of the EU funding.

While examining the cross-border dimension of the EU added value of CIPS, the external evaluator concluded that eligible actions under CIPS also included, besides cross-border projects, national projects to prepare or complement cross-border projects and/or EU actions or contribute to developing innovative methods and/or technologies with a potential for transferability to actions at EU level or to another Member State. In addition, the evaluation shows that CIPS projects were mostly led by coordinating organisations from a limited number of Member States (mostly Italy and Spain). For the partner organisations, however, there was a better geographical spread. Thus, the evaluator concluded that the EU added value could have been higher if organisations in all Member States had participated more. The evidence gathered throughout the evaluation, show that some of the reasons for the significantly varying degrees of uptake across the Member States were various and included, besides issues such as awareness of the programme and the ability of potential applicants to draft high-level quality proposals, also the pro-activeness by the individual Member States’ Committee representative promoting CIPS among relevant organisations continued in part following completion of the project. Only three partnerships did not continue after the project was completed.

The EU added value was significantly strengthened by the partnerships established. During the interviews with the coordinating organisations the following examples of EU added value were provided — possibility to collaborate in real-life scenarios, in particular when it concerns cross-border cooperation (i.e. joint patrols to reduce terrorism and other security related risks) and the exchanges and training of staff from different countries engaged in critical infrastructure protection. In addition the online survey for partner organisations repeats the views of the coordinating organisations. The majority of stakeholders consulted confirmed that their partnership fully or at least partly had continued after the funding ended through for example continued collaboration of delivering training or continuing cooperation in general.

The importance of EU funding to carry out the activities is linked to the added value of the programme. The findings of the external evaluators suggest that organisations struggled to identify and access national or other funding opportunities to implement the envisaged activities. Thus, it could be presumed that without the CIPS funding a significant part of the activities developed under the CIPS programme would have not been developed. In addition, the results of the survey carried out with the CIPS coordinating organisations suggest that, during the programming period, there were few national funding opportunities to develop the activities implemented under the various policy areas of the programme. Nine out of 13 respondents indicated that the project activities would not have been implemented without EU funding.

Moreover, only four stated that the project could have been fully financed through other funding sources. Out of the nine respondents who indicated that they could not have funded their project through other sources, eight indicated that other funds were not available (or insufficient) and three explained that other potential funding sources typically did not support their project’s aims. Following the interviews with the coordinating organisations was concluded that one of the main benefits of the CIPS programme was indeed the possibility to implement specific activities which were unlikely to be funded by national or alternative funding sources (e.g. exploratory research activities, large scale projects). The four coordinating organisations which could have made use of other funding indicated to have chosen CIPS as the latter provided them with the opportunity to work with partners in other countries and to have an impact at EU level.

ANNEX 3 — Methodology

Study conducted by external evaluators

The preparatory ex post evaluation study conducted by the external evaluators was regarded as providing the most robust and impartial overview possible of the Fund because it was highly structured:

•A regular and transparent dialogue took place between the Commission departments and the contractors;

•The parameters of the contract were clearly set out and respected;

•All data sources were assessed and presented data are clearly labelled.

Communication between the Commission and the external evaluator

The study's progress was followed by an Inter-service Steering Group (ISG) comprised of officials from the following services: DG for Migration and Home Affairs, the DG for Environment, the DG for Development and Cooperation, the DG for Humanitarian Aid and Civil Protection, the DG for Energy, the DG for Justice and Consumers, the DG for Internal Market, Industry, Entrepreneurship and Small and Medium Enterprises, the DG for Budget, the Secretariat-General, the Joint Research Centre, the European Anti-Fraud Office, as well as the external evaluators with their partners and experts. Regular meetings took place between the contractors and the ISG and structured feedback (in both directions) was provided throughout the contract. This two-way dialogue was enriched by the active participation in the ISG of policy and implementation units and shadowed by horizontal units and the Secretariat-General.

Evaluation stages

A structuring feature of the external evaluation was the segmentation of the tasks into clearly defined stages which were closely observed by all parties. These stages had been determined in the Terms of Reference. Complying with the Terms of Reference made the study itself more efficient and transparent.

1.Inception and refinement of the methodological approach: The overall purpose of the Inception Phase was to lay the groundwork and further elaborate and validate the method of approach to this evaluation.

2.Data collection and analysis.

3.Formulation of conclusions and recommendations: This phase aimed at bringing together the data collected and to produce findings, conclusions and recommendations.

4.Revision and finalisation: The aim of the final phase was to quality assure the deliverables through revision and finalisation based on the comments of the Steering Group.

Data sources

1.Primary data

·Three online surveys, with (i) coordinating organisations; (ii) partner organisations; and (iii) project participants.

·Stakeholder consultations — 37 face-to-face and telephone interviews.

·Two thematic case studies.

- Development of tools and protection in Germany and Hungary, and;

- Risk assessment in Slovenia,

·Open public consultation carried out by the Commission.

·Expert workshops and expert input.

2.Secondary Data

·A quantitative analysis of all CIPS projects based on a monitoring table provided by the Commission was undertaken.

·In-depth analysis of a sample of CIPS projects (29 projects representing 28 % of all 104 finalised projects).

·A literature review of EU policy baseline and context.

·A literature review of the national baseline and context.

Methodological limitations

A number of methodological obstacles were encountered throughout the evaluation which affected, for a number of evaluation questions, the extent to which findings and conclusions could be formulated.

Lack of a baseline

The first key obstacle encountered in the evaluation was the fact that establishing a baseline for measuring the level of prevention, preparedness and coordination of crisis management and security actions in the aftermath of a terrorist attack or of security incidents across the EU is difficult. Indeed, no methodologies were established for measuring in a comprehensive manner most of the activities covered by actions funded via CIPS. It has not been possible to establish direct causal links between the CIPS programme’s intervention or identified needs on the one hand and observed impacts or results on the other hand; therefore, effectiveness has been mostly evaluated in terms of outputs or perceived impacts or results– which were more easily measured and compared across MSs — rather than actual impacts or results. By the time the official baseline was adopted with the Better Regulation Guidelines, the CIPS programme was already under implementation.

Time lag between programme implementation and the ex post evaluation

There was a time lag between the implementation of the programme (2007-2013) and the ex post evaluation (August 2016-August 2017). This meant that the relevant beneficiaries and national and EU authorities and experts were often no longer working in the same position and were thus difficult to reach, which is reflected in a low response rate to some of the consultations (see Annex 3). The external evaluator took some initial measures to mitigate these difficulties, such as sending several email reminders to the persons surveyed, searching for alternative contacts when possible, raising the individual partner organisation interviews to two rather than one, and extending the deadline for the online survey carried out as part of the case studied. Overall, all evaluation questions have been answered to some extent, thanks to the use of triangulation of data where possible. Where the evidence base has been deemed to be too low to be reliable, this has been indicated.

Monitoring, accessibility and consistency of project data

Issues have been noted on the monitoring of project progress and final results which hindered the assessment of effectiveness, efficiency and EU added value. Especially given the relative newness of the programme and the lack of capacity at the time within the Commission to closely follow all projects from start to end, a dedicated technical assistance budget could have supported a monitoring system to collect and analyse data on (financial) progress, outputs and results. The effect of these limitations on effectiveness, efficiency and EU added value is further explained in Sections 7.2 and 7.5.

While relevant information could be collected for all evaluation questions, a full overview of final financial data and project data, especially on final absorption rate was not consistently available across all projects, given that data on final payments was not available for projects which were not yet complete. In addition, data provided by the Commission and the data drawn from other sources were not always consistent. This has limited the overall findings of the evaluation to some extent. Where possible, estimates have been used to calculate absorption rates and implementation trends. However, the figures on the last programming year should be used with caution. Given that no estimates were available for the JRC data, no absorption rate has been calculated for the JRC element of the procurement sector.

Another limitation which affects the assessment of effectiveness is that the outputs/results of the projects were not centrally collected and the final assessments of projects by the Commission were not carried out comprehensively, often consisting only of a brief paragraph on the overall project, without elaborating on particular elements or activities of the project and their potential future use. In addition, no peer review of the products/outputs was undertaken. In addition, results could not be measured due to the lack of ex ante targets. It was not feasible to compare the ‘cost of projects’’ under CIPS to other similar projects due to the absence of comparable projects. This hindered the assessment of efficiency and effectiveness to a large extent, as it was difficult to compare project costs and to measure the extent to which the same objectives were met by the different types of projects in the various areas addressed by CIPS.

To mitigate these limitations, the assessment of effectiveness and efficiency had to rely largely on the qualitative data collected through the stakeholder consultations. This, combined with the absence of a baseline, proved to be a limitation in terms of methodology.

ANNEX 4 — List of evaluation questions

Relevance

EQB1.a: To what extent did the objectives of the CIPS programme correspond to the needs related to the prevention, preparedness and consequence management of terrorism and other security related risks?

EQB1.b: To what extent did the design of the CIPS programme correspond to the needs related to the prevention, preparedness and consequence management of terrorism and other security related risks?

Effectiveness

EQB2: To what extent did the CIPS programme contribute to supporting Member State efforts to prevent, prepare for and protect people and critical infrastructure against terrorist attacks and other security related incidents?

EQB3: To what extent did the CIPS programme contribute to protection in areas such as crisis management, environment, public health, transport, research and technological development in the field of terrorism and other security related risks?

EQB4.a: To what extent did the CIPS programme contribute to the protection of people and critical infrastructure […] by promoting and supporting risk assessments on critical infrastructure, in order to upgrade security?

EQB4.b: […] by promoting and supporting the development of methodologies for the protection of critical infrastructure, in particular risk assessment methodologies?

EQB4.c: […] by supporting shared operational measures to improve security in cross-border supply chains, in compliance with EU regulations on competition within the single market?

EQB4.d: […] promoting and supporting the development of security standards, and exchange of know-how and experience on protection of people and critical infrastructure?

EQB4.e: […] promoting and supporting EU-wide coordination and cooperation on protection of critical infrastructure?

EQB5: To what extent did the CIPS programme contribute to promoting and supporting exchange of know-how and experience related to the consequence management, in order to establish best practices so as to coordinate the response measures and to achieve cooperation between various actors in crisis management and security actions?

EQB6: To what extent did the CIPS programme contribute to promoting joint exercises and practical scenarios in consequence management, including security and safety components, in order to enhance coordination and cooperation between relevant actors at the EU level?

Efficiency

EQB7.a: To what extent were the results of the CIPS programme achieved at a reasonable cost in terms of financial resources deployed?

EQB7.b: To what extent were the results of the CIPS programme achieved at a reasonable cost in terms of human resources deployed?

EQB7.c: What kind of initiatives or approaches were adopted to simplify access to and implementation of the actions funded by the programme (e.g. changed to shared management in ISF-P)?

Coherence

EQB8.a: To what extent was the CIPS programme coherent with other related actions supported by other EU programmes?

B8.b: To what extent was the CIPS programme coherent with other related actions supported by other national resources?

EQ B8.c: To what extent is it possible to identify synergies between the results of actions funded under the CIPS programme and the results of similar actions supported under other EU programmes?

EU added value

EQB9: To what extent would the beneficiaries under the CIPS programme (Member States, universities, institutes, associations, etc.) be able to carry out the activities necessary for the implementation of the EU policies in the prevention, preparedness and consequence management of terrorism and other security related risks without the support of the CIPS programme?

ANNEX 5 — List of abbreviations and country codes

CIP/R        Critical Infrastructures protection and/or Resilience

SLO        Security Liaison Officer

EMAs        Emergency Management Agencies

ISF        Internal Security Fund

KPIs        Key Performance Indicators

ENISA        European network and Information Security Agency

List of country codes

ANNEX 6 — The role of the Joint Research Centre within CIPS

The JRC, the Commission’s science and knowledge service, played an active role in the Programme’s implementation and actively participated in the CIPS’s inter-service consultations and groups. Furthermore, the JRC influenced the implementation process of EU policy on critical infrastructure protection. On the other hand, CIPS funding supported the JRC’s implementation of EU initiatives on critical infrastructures, including the development of the Critical Infrastructure Warning Information Network (CIWIN) and the European Reference Network for Critical Infrastructure Protection (ERN-CIP) as well as specific research activities for instance in the CBRNE and cybersecurity sectors. Additionally, through CIPS funding, the JRC provided support to DG HOME and the Member States on the preparation and review of Directive 2008/114/EC.

Due to the JRC’s expertise in security and CIP 62 , it contributed to the support and implementation of CIPS actions throughout the programming period. Actions carried out by JRC were indicated in the annual work programme as administrative arrangements with JRC 63 . The JRC has its own annual work programme (based on Framework Programme 7) which meant that actions funded through CIPS had to be in line with JRC’s priorities. To avoid gaps, an analysis on what actions could have been financed by Framework Programme 7 and what could have been financed by the CIPS programme was carried out by the JRC. Moreover, under Framework Programme 7 and H2020 64 , the JRC started a scientific initiative in support of the implementation of the CIP Directive in 2010. This included organising and coordinating the meetings of all CIP national contact points.

Two of the key actions implemented by the JRC under CIPS included ERN-CIP and CIWIN. The box below presents additional information on ERN-CIP and CIWIN.

ANNEX 7 — Sectoral analysis of CIPS projects

Transport sector

24 projects were funded under this sector during the evaluation period. Over two-fifths of them were led by organisations in Italy. However, projects were also funded to organisations located in Germany (3), Greece (2), Spain (2), Poland (2), Belgium (1), Bulgaria (1), Romania (1), Slovenia (1) and the UK (1).

As an example, the Risk management of Transport of Hazardous Substances (RMTHS) project was awarded funding in 2012 and coordinated by the Urban Planning Institute of Ljubljana. Its main aim was to develop an application for an Integrated Geographic Information System which could model the safest route for trucks carrying hazardous materials. The activities of the project varied and included different domains such as legislation, definition of the main road network and preparation of final documentation. However, the final result was the development of an Integrated Geographic Information System application to support road authorities in real-time management of traffic with a particular focus on hazardous substances. The project delivered the following outputs:

• Integrated Geographic Information System application for managing data on transport of hazardous substances;

• Integrated Geographic Information System application for identification safest route for transport of hazardous substances;

• Methodology for the monitoring of the transport of hazardous substances;

• Integrated Geographic Information System expert model; and

• A webpage, workshops, articles and final publication.

Energy sector

A total of 16 projects across organisations in different Member States 66 were granted CIPS funding in this sector. The organisations leading the projects included police bodies, universities and research institutes, private companies (i.e. Deloitte) and state-sponsored companies (i.e. the Italian National Agency for New Technology, Energy and Environment).

The project’s topics included:

i) identifying risks and improving the security of energy smart grids (5 projects);

ii) The coordination of EU-wide and Member States stakeholders (3 projects); and

iii) prevention of malfunctions in energy infrastructures (3 projects).

Three energy projects focused on the development of tools for protection or assessing risks. They were rated as ‘good’ by the Commission, who highlighted the importance of creating techniques, methods and tools to prevent threats to energy smart grids.

The JRC also implemented actions in the energy section such as actions aiming to identify the key vulnerabilities of the European Gas Transmission Network in order to determine needs and actions and reduce its vulnerability.

As an example of implemented project, the Italian National agency for new technologies, energy and sustainable economic development (ENEA) led the MIA project 67 that aimed to develop a methodological framework to identify and measure interdependencies between ICT infrastructures and electricity generation infrastructures.

Chemical sector

In 2008, 2009 and 2013, four projects focusing on the chemical sector were implemented. Two focused on developing workshops agreement for chemical, biological, radiological and nuclear defence and were led by the European Committee for Standardisation (CEN) in Belgium. The third one focused on building protection against adversaries using high-risk chemicals and was led by a consulting company in Czech Republic. Finally, the fourth one was led by the European Chemical Industry Council which aimed at improving knowledge on effective critical infrastructure protection.

Finance sector

Two organisations (based in Italy and Spain) coordinated a project each in this sector. Both projects aimed at improving coordination between different sectors hence had a cross-border focus. More specifically, one project focused on the analysis of the domino effects due to critical infrastructure failures whereas the other one aimed at preventing the electronic payment fraud.

Water sector

Four projects were implemented under CIPS in this sector. Two were led by police headquarters in Poland while the other two by the European Committee for Standardisation in Belgium. However, their common aim was the protection of drinking water supplies in crisis situations (terrorist attacks included).

Food sector

One project was implemented under the CIPS programme and was led by a university in Italy. It focused on identifying risks to prevent terrorist attacks on the food supply chain.

Health sector

One project focused on improving Member States response to terrorist attacks on hospitals.

Space sector

Three projects focused on this sector, two in 2007 and one in 2011. All were led by the same European spaceflight service company (Telespazio Spa) based in Italy.

Research sector

Four projects focusing on this sector were funded under CIPS and were led by different bodies in the UK, IT and ES. Their aims varied significantly between Member State. One project focused on changing the higher education curriculum to help students with the design and development of new or improved products and services aimed at the prevention, detection or response to terrorism activities, whereas another one focused on the development of a methodology and tool to permit the mathematical calculation of the resilience of an infrastructure.

Nuclear sector

Two projects were granted CIPS funding under this sector. One was led by a private engineering consultancy firm in Germany in 2010 and aimed at identifying critical infrastructures in nuclear power in order to strengthen its protection capacity. The second one was implemented by an institute in Bulgaria and focused on the development of tools needed to coordinate inter-sectoral power and transport critical infrastructures protection activities.

ANNEX 8 — The different dimensions of CIPS

ANNEX 9 — Intervention logic of the CIPS Programme

(1)      Council Decision of 12 February 2007 establishing for the period 2007 to 2013, as part of the Framework Programme on Security and Safeguarding Liberties, the Specific Programme ‘Prevention, Preparedness and Consequence Management of Terrorism and other Security-related risks’.

(2)      Decision 2007/125/JHA established the Specific Programme ‘Prevention and Fight against Crime’ (ISEC) for the same period, namely 2007 to 2013. The ex-post evaluation of this programme is covered in a separate report.

(3)      The state of play of the annual work programmes was presented at CIPS Committee meetings twice a year.

(4)      See European Commission (2010) Evaluation of ‘Prevention and Fight against Crime’ and ‘Prevention, Preparedness and Consequence Management of Terrorism and other Security Related Risks’ Programs — JLS/2010/ISEC-CIPS/001-F4.

(5)      Communication from the Commission to the European Parliament and the Council on the mid-term evaluation of the framework programme ‘Security and Safeguarding Liberties’ (2007-2013), COM(2011) 318 final.

(6)      Due to the internal reorganisation of DG HOME, the reporting on this ex post evaluation was postponed to 2018.

(7)      Austria, Belgium, Bulgaria, Cyprus, Croatia, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Sweden, United Kingdom, and Croatia as of 2013.

(8)    Regulation (EU) No 513/2014 of the European Parliament and of the Council of 16 April 2014 establishing, as part of the Internal Security Fund, the instrument for financial support for police cooperation, preventing and combating crime, and crisis management and repealing Council Decision No 2007/125/JHA.

(9)      Regulation (EU) No 514/2014 of the European Parliament and of the Council of 16 April 2014 laying down general provisions on the Asylum, Migration and Integration Fund and on the instrument for financial support for police cooperation, preventing and combating crime, and crisis management, OJ L 150, 20.5.2014, p. 112-142, Article 57. Part of the interim evaluation is the mid-term review of the ISF national programmes of the Member States (Regulation (EU) No 515/2014, Article 8) which will take place in 2017 and 2018. The purpose of the mid-term review exercise is for the Commission and the Member States to review their national programmes and assess the need for a possible revision of the programme, in the light of developments in Union and national policies through a questionnaire and bilateral dialogues with the Member States. In addition, if the need arises, the results of the mid-term review of the national programmes may support requests for additional funding made by the Commission to the budgetary authorities for the remaining implementation period.

(10)      See https://ec.europa.eu/home-affairs/what-we-do/policies/crisis-and-terrorism/critical-infrastructure_en

(11)      See Liquefied natural gas — LNG terminal is a facility for regasifyng of LNG (definition from https://www.elengy.com/en/lng/what-is-an-lng-terminal.html ).

(12)      Council Directive 2008/114/EC of 8 December 2008 on the identification and designation of European critical infrastructures and the assessment of the need to improve their protection.

(13)      Anne van Aaken, Isabelle Wildhaber, Symposium on Critical Infrastructures: risk, responsibility and liability. Available at http://ejrr.lexxion.eu/data/article/7933/pdf/ejrr_2015_02-004.pdf

(14)       http://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=URISERV:l33260&from=EN

(15)       http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32008L0114&from=EN

(16)    See    
http://ec.europa.eu/energy/sites/ener/files/documents/20130828_epcip_commission_staff_working_document.pdf

(17)       https://ec.europa.eu/home-affairs/sites/homeaffairs/files/what-we-do/policies/crisis-and-terrorism/critical-infrastructure/docs/swd_2013_318_on_epcip_en.pdf

(18)      The CIWIN serves both as internet portal to exchange ideas, studies and good practices in the field and as a repository for CIP information. Indeed, one of the main challenges in the field of critical infrastructure protection is the non-optimal partnership between public and private sectors in a field where private companies play a key role. Please see https://ec.europa.eu/home-affairs/what-we-do/policies/crisis-and-terrorism/critical-infrastructure_en

(19)       http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2006:0786:FIN:EN:PDF

(20)      Council Directive No 2008/114/EC.

(21)      Liquefied natural gas — LNG terminal is a facility for regasifyng of LNG (definition from https://www.elengy.com/en/lng/what-is-an-lng-terminal.html).

(22)      Europol, EU Terrorism Situation & Trend Report (Te-Sat), 2014, https://www.europol.europa.eu/activities-services/main-reports/te-sat-2014-eu-terrorism-situation-and-trend-report

(23)       http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32002F0475&from=en

(24)       http://register.consilium.europa.eu/doc/srv?l=EN&f=ST%2014469 %202005 %20REV%204

(25)       http://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=URISERV:jl0050&from=EN

(26)      Council Decision 2007/124/EC, Euratom, February 2007, Art. 3.1.

(27)      Council Decision 2007/124/EC, Euratom, February 2007, Art. 3.2.

(28)      Council Decision 2007/124/EC, Euratom, February 2007, Art.2 (c).

(29)      Council Decision 2007/124/EC, Euratom, February 2007, Art.4.

(30)      Political priorities and objectives for projects are defined in the Commission’s annual work programmes (AWPs) adopted each year and calls for proposals are launched following the approval of each AWP.

(31)       https://ec.europa.eu/home-affairs/financing/fundings/security-and-safeguarding-liberties/terrorism-and-other-risks_en

(32)      More details on the role of the JRC within CIPS are reported in Annex 7.

(33)      Article 6 of Decision No 2007/ 124/ EC, Euratom.

(34)      Council Decision 2007/124/EC, Euratom, February 2007, Art. 5.

(35)      On the basis of the scoping interview with a representative of the JRC, it was noted that ‘development’ does not always constitute a proper development of something new but rather it might be a re-deployment of existing tools to assess their feasibility and quality into the domain of the critical infrastructure protection. Therefore, projects in this category might refer to activities of ‘deployment and assessment’ and/or include a reference to ‘pilot projects’.

(36)      Ibidem.

(37)      ABAC is the Commission’s accrual-based online accounting system.

(38)      Year related to the year of the AWP under which a certain call for proposals was issued and included in the project’s reference indicated in the Commission monitoring table.

(39)      The sample of individual projects included only ‘finalised’ project since the implementation phase was ended and the necessary documentation was available.

(40)      More specifically this concerned analysing the status of related Policy documents to provide a methodology to be applied to the selected use cases for the envisaged development of a European Smart Grid Operation Handbook and cyber-security aware Grid Codes for the Member States.

(41)      The analysis of fund absorption takes into account budgetary information on grants as received up to 31/03/2017.

(42)      With a total of 19 projects not finalised by 31/03/2017 of which 16 for 2013 and 3 for 2012. For these projects, the final invoice amount, as recorded in ABAC, has been used for the calculation of the absorption rate.

(43)      For more information, see the answer to the evaluation question B7.c.

(44)      Please note that the absorption rate for procurement items is based on ABAC data up to 31/03/2017. For all procurement entries used for the procurement sector as presented in this report there was both a committed and spent value which have been used to calculate the absorption rate.

(45)      See also the EPCIP, p 8 on     
http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2006:0786:FIN:EN:PDF

(46)      The process mentioned for the identification of priorities and sub-priorities was carried out in regular consultation with the Member States, through a comitology procedure and an inter service group with relevant DGs competent in the area of protection of critical infrastructure in the Commission, including DG CNECT, DG ECHO, DG SANCO, DG MOVE, DG ENER and the JRC.

(47)      Only one interviewee commented that the process was not very open and that there were not real possibilities to influence the content of the programmes.

(48)      One coordinating organisation responded ‘I don’t know’.

(49)      See more details on the financial mechanism of the programme in Section 2.3.

(50)       http://publications.jrc.ec.europa.eu/repository/

(51)      For more information, see Annex 7 ‘Sectoral analysis of CIPS projects’.

(52)      The Web 2.0 platforms are applications enabling live chatting and are already used in some cases by practitioners and the public in response to disasters.

(53)      Whether the desired effects were achieved at reasonable costs.

(54)      Compared to FP7 rules for instance.

(55)      For more information see the Section on relevance.

(56)      Terms of Reference, Art. 2.3.

(57)      The five activities were formulated in the survey as follows: Operational cooperation and coordination; Analytical, monitoring, evaluation and audit activities; Development and transfer of technology and methodology; Training, exchange of staff and experts and Development and creation of network.

(58)      ENISA is the European Network and Information Security Agency: https://www.enisa.europa.eu/

(59)       https://www.bmi.bund.de/SharedDocs/Downloads/DE/Broschueren/2009/kritis.pdf?__blob=publicationFile

(60)      See Section 7.1.

(61)     Article 10, Regulation No 513/2014.

(62)       https://ec.europa.eu/jrc/en/research-topic/cybersecurity

(63)      A detailed description is provided in the technical annexes.

(64)       https://ec.europa.eu/programmes/horizon2020/

(65)

     This prototype was developed by an external service provider under the direct and solely control of DG HOME (ex- DG JLS). The JRC supported only the hosting of CIWIN in its Data Centre.

(66)      ES, IT, NL, HU, CZ, UK.

(67)      Definition of a methodology for the assessment of mutual interdependencies between ICT and electricity generation / transmission infrastructures (MIA) JLS/2007/CIPS/019.

(68)      CIPS aimed to protect the societal impact of the disruption/destruction of CI.