EUROPEAN COMMISSION

Brussels, 29.4.2020

SWD(2020) 74 final

COMMISSION STAFF WORKING DOCUMENT

Evaluation of the European Research Council Executive Agency (ERCEA)

Accompanying the document

Report from the Commission to the European Parliament, the Council and the Court of Auditors

Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Executive Agency for Small and Medium-Size Enterprises, the Innovation and Networks Executive Agency, the Education, Audiovisual and Culture Executive Agency, the Research Executive Agency and the European Research Council Executive Agency

{COM(2020) 184 final} - {SWD(2020) 73 final} - {SWD(2020) 75 final} - {SWD(2020) 76 final} - {SWD(2020) 77 final} - {SWD(2020) 78 final}

TABLE OF CONTENTS

1.    INTRODUCTION    

2.    BACKGROUND    

2.1 Description of the intervention and its objectives    

2.2 Baseline and points of comparison    

3.    IMPLEMENTATION / STATE OF PLAY    

4.    METHOD    

4.1 Short description of methodology    

4.2 Limitations and robustness of findings    

5.    ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS    

5.1 Effectiveness    

5.2 Efficiency    

5.3 Coherence    

5.4 Retrospective Cost-benefit analysis    

6.    CONCLUSIONS    

ANNEX : EVALUATION QUESTIONS    

1.

1.Introduction

The purpose of this periodical (every three years) evaluation of the operation of the European Research Council Executive Agency (ERCEA) is to assess the ERCEA’s implementation of the parts of the EU funding programmes which have been entrusted to it. The ERCEA is governed by the Framework Regulation for executive agencies 1 , its Act of Establishment 2 , which sets out its mandate; the Act of Delegation 3 which specifies the tasks to be carried out by, and the powers delegated to the Agency in order to perform its mandate; and the decision establishing the ERCEA Steering Committee.

In line with the Commission’s Better Regulation principles 4 , the evaluation applies several standard evaluation criteria. The evaluation assesses whether the Agency has fulfilled its tasks in an effective and efficient way, whether there are overlaps / gaps / inconsistencies in the management of the programme portfolio by the Agency, and whether there is a clear delineation of tasks between the ERCEA and the parent DG or other executive agencies (coherence) 5 .

The evaluation also assesses whether the functioning of the Agency has yielded the expected positive results as estimated in the ex-ante Cost-Benefit Analysis (CBA) for the delegation of tasks to the executive agencies 6 and identify potential areas of improvement. To this end the estimations of the Cost-Benefit Analysis of 2013 have been tested to provide evidence on the validity of the assumptions made in the ex-ante scenario by considering the actual costs and benefits of programme implementation by the Agency in a structured way. The aspects covered by the Cost-Benefit Analysis are specified in Article 3(1) of the Framework Regulation 7  and the Guidelines on establishing and operating executive agencies 8 . 

The evaluation does not cover the achievements of the programmes managed by the ERCEA, which are subject to mid-term and ex-post evaluations themselves. The evaluation of the ERCEA nevertheless provides useful input for these programme evaluations, considering that the performance of the Agency affects the efficiency and effectiveness of the programmes it manages.

The evaluation examines the efficient use of resources and the effective achievement of the tasks entrusted to it. In particular, it looks at whether the alignment of more coherent programme portfolios with the Agency’s core competences and its brand identity delivered the estimated qualitative benefits; whether the assembly of the management of different EU programmes and provision of support services delivered the estimated synergies, simplification and economies of scale; whether the pooling of instruments guaranteed consistent service delivery and whether there is scope for simplification and further efficiency gains.

The evaluation covers all the tasks carried out by the Agency during the period 16 July 2015 to 15 July 2018. The evaluation is supported by a study carried out by external contractors 9 .

The results of this study are summarised hereafter and this evaluation will be presented to the European Parliament, to the Council and to the Court of Auditors in accordance with Article 25(1) of the Framework Regulation.

The results will feed into the reflection to assess the opportunity of expanding/ modifying the Agency’s mandate in view of the delegation of the implementation of 2021-2027 EU programmes.

2.Background

2.1 Description of the intervention and its objectives

The outsourcing of certain management tasks to the Agency according to the Framework Regulation and the ERCEA’s Act of Establishment intends:

·To allow the Commission to focus on its institutional tasks, i.e. tasks assigned to the institutions by the Treaty which require discretionary powers in translating political choices into action. Such institutional tasks should not be outsourced.

·To enable the Commission to achieve the objectives of the delegated EU programmes more effectively and efficiently. According to the Cost-Benefit Analysis carried out in 2013, delegating tasks to the ERCEA was estimated to be more cost-efficient than an in-house scenario. In addition the CBA showed that the Agency has high quality programme management and service delivery, visibility and existing communication outreach channels which have proved effective so far. Making use of the accumulated experience and expertise of the Agency would lead to efficiency gains.

The original objective of the intervention is entrusting the Agency with the implementation of the following EU programmes in parts or fully:

·Within the Horizon 2020 Framework Programme for Research & Innovation ( Horizon 2020 ), the specific objective ‘Strengthening frontier research through the activities of the European Research Council’ of Part I ‘Excellence Science’,

·Within the 7th Framework Programme ( FP7 ), the implementation of the legacy of the Specific Programme ‘Ideas’.

The ERCEA has one parent-DG, namely Directorate-General for Research and Innovation (DG RTD).

The Agency works closely with the Common Support Centre (CSC), now named Common Implementation Centre (CIC), administratively part of DG RTD, which provides the framework for harmonised implementation of the EU Research and Innovation (R&I) programmes for all R&I DGs family of implementing bodies, including Executive Agencies and Joint Undertakings. The CIC is responsible for defining common business processes for grant management, and providing IT, legal, audit and information and data services to the Horizon 2020 implementing bodies.

With regards to the tasks delegated in the framework of Horizon 2020 and FP7 legacy, the ERCEA performs the following programme implementation tasks:

·managing programme implementation and specific projects on the basis of the relevant work programmes (WP) established by the Scientific Council of the European Research Council (ERC Scientific Council) and adopted by the Commission;

·adopting the instruments of budget execution for revenue and expenditure and carrying out all the operations necessary for the management of the programme;

·providing support in programme implementation where the Commission has empowered it to do so in the instrument of delegation;

·providing support to the ERC Scientific Council in the conduct of all of its tasks.

2.2 Baseline and points of comparison

The current evaluation of the ERCEA operations during 2015-2018 assesses the actual costs and benefits of programme implementation by the ERCEA (executive agency scenario) when compared with the alternative scenario of management by the Commission services (in-house scenario).

Accordingly, the reference point for the present ERCEA evaluation is the 2013 ex ante Cost-Benefit Analysis, and the Specific Financial Statement (SFS) of the ERCEA.

Wherever possible, the analysis of the ERCEA performance during the period of reference assesses the progress compared to the two previous evaluations, covering the Agency’s operations during 2009-2012 and 2012-2015. The current evaluation covers the period between the beginning of the second half of 2015 and the end of the first half of 2018. However, wherever possible and relevant, and provided that all the necessary data was available, the analysis was extended to include the whole of 2015 and, at times, 2014.

Over the period 2009-2012 the savings resulting from the delegation of tasks to the ERCEA have been evaluated at around EUR 45 million compared to the in-house scenario, and at EUR 46.5 million over the period 2012-2015. The 2013 CBA estimated the efficiency gains over the period 2014-2024 at EUR 79 million and at EUR 41 million over the period 2015-2018. The total number of full time equivalents (FTEs) required to manage the relevant programmes in year 2020, the peak programming year in terms of workload, was estimated at 529 for ERCEA. The total estimated operational budget entrusted to ERCEA in 2020 is around EUR 2 322.956 million in commitment appropriations and EUR 1 787.180 million in payment appropriations. Considerable efficiency gains were expected as compared to the in-house scenario over the period, along with non-quantifiable benefits such as improved quality of programme management and service delivery, improved visibility of the EU programmes and proximity to beneficiaries.

3.Implementation/State of play

Established in 2007 and autonomous since July 2009, the ERCEA was entrusted with the management of EU actions in the field of frontier research to perform the task of implementing the Specific Programme Ideas. The Agency’s mandate was extended by the Commission Implementing Decision No 2013/779/EU, which came into effect on 1 January 2014. The ERCEA’s renewed mandate under Horizon 2020 concentrates on the implementation of the ERC programme within the “Excellent Science” pillar, thus bringing continuity in relation to the FP7 period.

The new mandate did not bring any changes in the governance of the Agency. A new written agreement (WA) between ERCEA and DG RTD was signed in April 2014 and was further amended in December 2015 10 . The new WA defined the modalities and procedures of interaction between ERCEA and its parent DG. The distribution of tasks and responsibilities, as presented in the WA, is included in sections 5.1 and 5.3 of this document. Compared to this reference, this evaluation assesses whether this distribution of roles has been respected for the 2015-2018 period, and the effects of the intervention on the coherence and the efficiency of ERCEA’s action.

What is specific to the ERCEA compared with other executive agencies is that the scientific funding strategy focuses solely on bottom-up, frontier research and scientific excellence, and methodologies are defined by the European Research Council (ERC) via its independent Scientific Council 11 (ScC) and then adopted by the Commission, while the executive agency serves as a dedicated implementation structure and is in charge of the day to day evaluation and grant management. Another specificity relates to the process of selection of experts for the evaluation process: made by the ERC, it aims at ensuring the excellence of the evaluation and attracting top-class experts who may not otherwise be reached under procedures used for other research family programmes 12 .

The ERC provides funding via five different types of grants. Four of them were launched under FP7 (Starting Grant (StG), Consolidator grant (CoG), Advanced grant (AdG) and Proof of Concept grant (PoC)), and have been continued under Horizon 2020. The Synergy grant (SyG) was launched in FP7 as a pilot, and relaunched as a fully-fledged grant scheme in 2017 13 under Horizon 2020. Whereas the StG, CoG, AdG and PoC are all single grants awarded to a principal investigator (PI) who can then recruit a team of researchers to assist with the research project, the SyG supports small teams of scientists who wish to jointly address research problems through frontier research. PoC funding is made available only to those who already have an ERC award to establish proof of concept of an idea that was generated in the course of their ERC-funded projects.

Overview of the ERC grants scheme 14  

Overall, the actual workload of the ERCEA during the evaluation period generally corresponded to the initial 2013 CBA estimates. The ERCEA’s workload is determined by the allocated operational budget, the number of proposals received and the number of grants concluded/projects managed. The operational budget managed by the ERCEA was significant in scale and totalled EUR 9,172 million during the 2014–2018 period. It ranged from EUR 1,731 million in 2014 (19% more than estimated in the CBA and SFS) to EUR 1,995 million in 2018 (5% more than estimated). The actual number of new grants managed by the ERCEA under 2014-2018 calls was 13% higher than 2013 CBA forecast (5 450 versus 4 738 grants), with a substantial increase especially in 2014 and 2015 (30% and 23% more than CBA estimates). On the other hand, the actual number of proposals received under the 2014-2018 calls was substantially (32%) lower than the initial CBA calculations (40 532 versus 59 500 proposals). This is a result of the restriction rules on re-submission introduced by the ERC Scientific Council in the ERC Work Programmes to contain the experts’ workload at a reasonable level and hence avoid compromising evaluation quality but also to keep the overall success rate at an appropriate level 15 of around 15%.

The ERCEA gradually grew in size in accordance with the multi-annual planning provided in the SFS, from 417 at the end of 2015 to 494 staff members 16 at the end of 2018 17 . The Agency is based in Brussels.

A series of organisational and procedural changes took place during the implementation of the ERC programme under the multiannual financial framework (MFF) of 2014-2020:

·Adoption of new business processes and use of the Commissions Horizon 2020 IT tools and systems for the grant management process and the electronic submission of proposals (i.e. Compass and SyGMa);

·The adoption of the multiannual human resource (HR) strategy for 2017-2020 to address the findings of the 2016 staff opinion survey. Based on this new strategy and in combination with a dedicated action plan addressing attention points identified through the survey, the Agency implemented specific measures related to staff engagement, internal communication, career development opportunities, work and private life balance and training, meant to improve effective HR management and increase the effectiveness and flexibility of the Agency’s operations.

4.Method

4.1 Short description of methodology

The evaluation was supported by a study carried out by external contractors. The methodology of the ERCEA evaluation was consistent with the approaches employed for similar parallel evaluations of REA, EACEA, INEA, EASME and CHAFEA and built on the previous evaluation of the ERCEA operations during 2012-2015. Whenever possible, data was collected and the analysis extended to cover 2014 as well as the whole of 2015 and 2018, even if formally outside the evaluation timeframe. Specific methodological approaches were used for each of the evaluation tasks (Regulatory framework, the ERCEA mission and governance; the ERCEA performance between 2015-2018, including effectiveness, efficiency and coherence; the Cost-Benefit Analysis and Conclusions and Policy Recommendations) to ensure data triangulation. The study was structured around a series of evaluation questions available in Annex.

The evaluation relied on the evidence gathered and analysed using qualitative and quantitative methods and included:

·an extensive documentary review and desk research of the ERCEA and Commission documents, including Annual Work Programmes and Annual Activity Reports, audit reports and related documents, CBA studies and other financial documents, the Written Agreement 18 , previous evaluations and action plans addressing the ensuing recommendations; the evaluation also relied on the results of the 2016 staff opinion survey;

·38 semi-structured in-depth interviews with Commission staff and the ERCEA staff, including: Directors and selected Commission staff in the parent DG – DG RTD; heads of units and senior staff from the Common Support Centre; the Director of the ERCEA, heads of departments, heads of units and heads of sectors, other employees in the ERCEA;

·6 interviews with senior members of the Scientific Council (including the President and all the Vice Presidents);

·13 interviews with the ERCEA beneficiaries, unsuccessful applicants and independent experts;

·1 survey addressing the ERCEA’s independent experts (745 answers or 25,4% response rate) and analysis of 2 surveys carried out by the ERCEA earlier in 2018: a survey of grantees (successful applicants) (964 answers or 31% response rate) and host institutions (319 answers or 20% response rate) and a survey of unsuccessful applicants (693 answers or 22% response rate);

·a retrospective cost–benefit analysis;

·benchmarking of the ERCEA’s performance in the 2012-2015 and 2015-2018 periods and a comparative analysis and benchmarking of the various executive agencies was also carried out against a set of (qualitative and quantitative) indicators. This enabled the evaluation team to compare the performance of ERCEA effectively with that of other executive agencies.

4.2 Limitations and robustness of findings

All the relevant stakeholders within the Commission, the ERC Scientific Council and the ERCEA were duly consulted via the interviews programme. An extensive number of the ERCEA’s independent experts were covered by the dedicated survey carried out during the evaluation. The evaluation used the analysed and confirmed results of two surveys carried out in early 2018 by the ERCEA (of beneficiaries and unsuccessful applicants) complemented by a number of additional interviews. However, compared with the overall population reached, unsuccessful applicants participated less actively in the survey than programme beneficiaries.

No sampling bias was observed as the profile of the respondents to the surveys was very similar to the overall population, guaranteeing statistical representativeness. The non-response bias (not all characteristics of the group that did not reply had been captured in full) was mitigated through triangulation with the results of follow up interviews. The triangulation approach, using multi-level and multi-stakeholder dimension in the data collection, ensured the robustness and reliability of the data and information used to draw up conclusions in the supporting study.

Some views from Commission and Agency staff gathered from the interviews and mentioned in the study illustrate personal experience, although they may not be representative of overall relations between the Commission and the Agency.

5.Analysis and answers to the evaluation questions

Analysis and answers to the evaluation questions are grouped below as per the three evaluation criteria: effectiveness, efficiency and coherence. The quantitative results of the ex-post cost benefit analysis are presented separately in section 5.4 below, while the qualitative aspects are embedded into the answers to the evaluation questions (sections 5.1-5.3). The detailed evaluation questions and sub-questions are listed in annex.

5.1 Effectiveness

For the purpose of this evaluation, effectiveness relates to how successful the Agency has been in achieving or progressing towards its objectives.

During 2015-2018, the ERCEA operated according to the 2013 legal framework that set out its revised mandate and its delegated responsibilities. The ERCEA’s activities, as planned in its annual work programmes and reported in the Annual Activity Reports, corresponded to the tasks set out in its Acts of Establishment and Delegation. The ERCEA was also found to be fulfilling its delegated management responsibilities to provide technical support and the secretariat function to the ERC’s Scientific Council. From the other side and as acknowledged by the ERC ScC Members, the Commission duly fulfilled its obligations in guaranteeing the European Research Council’s autonomy and integrity, including in what regards the development by the Scientific Council of the ERC work programme setting out the key scientific priorities for funding.

The ERCEA’s governance framework is characterised by a dual leadership, unique among executive agencies. The ERCEA is accountable both to the ScC and to its parent DG (DG RTD), which has supervisory responsibilities for monitoring the ERC programme implementation, and in overseeing the execution of the Agency’s administrative and operational budget. Working arrangements and procedures for interaction between the Agency and DG RTD, are set out in the Written Agreement (WA) of 2015. The delimitation of responsibilities between them was found to be generally clear, with a Written Agreement between the two providing operational guidance.

In 2015-2018, the ERCEA achieved very good results in terms of most key performance indicators (KPIs) targets as set out in its Annual Work Programme 19 . Compared to the previous years the Agency’s performance remained stable or even improved during the period of reference, particularly in regarding to the timely evaluation of proposals and the signing of grant agreements.

The ERCEA’s indicator on Time-to-Inform (TTI) 20 had various targets within the 2015-2018 period also depending on the type of grant. During 2015-2018 the ERCEA was largely in line with the TTI targets with some deviations observed in the 2015 AdG and 2017 StG calls. It should be noted that the average TTI of successful applicants is longer compared to the average TTI for unsuccessful applicants due to a difference in the procedures involved for successful and unsuccessful applicants.

Despite longer TTI than those of Horizon 2020, the surveys of the ERC Horizon 2020 principal investigators (grantees) and host institutions (HIs) revealed that 92% and 89% of the respective respondents were “satisfied” or “very satisfied” with the overall application and submission process. The satisfaction level was lower, however, among unsuccessful applicants (67% satisfied and 20% neutral).

During the evaluation period, the ERCEA was largely in line with the Time-to-Sign (TTS) 21 target; more significant deviations from the target concerned the 2015 CoG call and a few PoC calls (mainly in 2015). The ERCEA’s performance in terms of TTS significantly improved in 2017 thanks to further administrative simplifications in the granting phase. The ERCEA was largely in line with the overall Time-to-Grant (TTG) 22 target or close to it for all 2014-2017 calls.

The number of redress requests received by the ERCEA has been decreasing since 2014. The number of these requests that lead to a re-evaluation of proposals is very low and dropped significantly after 2013. It is considerably below 23 the Annual Work Programme target of 0.1%.

For payments to grants, the average ERCEA’s Time-to-Pay (TTP) stood significantly below the target for all types of payments (pre-financing, interim and final payments) in 2015-2018 both for FP7 and Horizon 2020. The TTP for 2015 and 2016 for FP7 was close and within the target. 99% of payments were executed on time in 2016, 2017 and in the 1st semester of 2018.

In payment to experts, average TTP for experts decreased during 2016-2017 to 13.3 and 10.7 days respectively, or 60% below the TTP target of 30 days. Accordingly, the ratio of experts “strongly” or “rather” agreeing that the time to make payments was appropriate grew to 94% - compared to 84% in the previous evaluation period (2012-2015). A considerable number of experts (46%) expressed dissatisfaction with the amounts received for hotel/accommodation and subsistence allowance being insufficient, however these are set at corporate level.

Similar to the previous years, the ERCEA managed to achieve full execution of its operational budget both in commitment and payment appropriations during 2015-2018.

The ERCEA ethics review and monitoring procedures generated a low level of satisfaction among both grantees and Host Institutions, with only 29% of grantees and 35% of Host Institutions “very satisfied” or “satisfied” with the overall ERC ethics review process 24 . The process encompassed a two-stage approach – (1) an ethics review which takes place between the evaluation phase and grant signature and (2) ethics monitoring carried out during project implementation. The ERCEA took a number of steps to simplify the ethics procedure by streamlining the two stages approach and adopted a new procedure in 2017. Accordingly, an initial ethics review procedure between the end of the evaluation stage and contract commencement stage scrutinises the ethical aspects of projects before the research begins; a further ethical assessment or ethics closure procedure signs the project off once the research has been completed. Also, a simplified risk-based approach to both ethics review and ethics monitoring was introduced by classifying projects into categories of normal or high ethics sensitivity where projects with normal ethics sensitivity will no longer be required to produce ethics deliverables. While the progress achieved is duly noted, it is concluded that the ERCEA should further review and adjust its ethics supporting guidance document. In parallel, it should increase awareness among grantees and Host Institutions on the need for particular ethics-related information for the ERC grants.

With regard to the legality and regularity of transactions, the multiannual residual error rate supporting the ERCEA’s declaration of assurance decreased during evaluation period from 1.31% in 2015 to 0.82% in 2017 and remained well below the 2% materiality threshold. For Horizon 2020, there were not enough audits to draw statistically representative conclusions for the ERCEA specific error rate. However, since the ERCEA's beneficiary base remained the same as under FP7 25 and the specificities of the programme have not been changed in Horizon 2020, it was expected that the error rate for Horizon 2020 remains below the 2% materiality threshold.

Overall satisfaction with the performance of the ERCEA is one of the highest among executive agencies. 89% of grantees responding to the ERCEA survey of 2018 were satisfied (33% of them – very satisfied) with the services provided by the Agency. This result is slightly below that of the previous evaluation of the ERCEA (95% in 2014), however it is higher than the overall level of satisfaction reported by the beneficiaries of CHAFEA (74%) and similar to what was reported by the beneficiaries of REA (86%) and EACEA (89%). A considerable part of the respondents to the surveys wished to work with the Agency again in the future (85% of beneficiaries, 94% of the ERCEA’s independent experts and around 39% of unsuccessful applicants).

The survey of beneficiaries indicates a high degree of satisfaction with the way the Agency communicated with them. 81% of the respondents to the beneficiaries’ survey strongly or rather agreed that the ERCEA staff assigned to their project were easily available and responsive during the implementation. Most favourable results were received in relation to the ERCEA’s availability and responsiveness during the grant amendment (82% strongly or rather agreed) and in the grant finalisation phases (85% strongly or rather agreed). The areas with reported lower levels of satisfaction were related to the user-friendliness of the IT tools employed throughout the project management life-cycle. The Commission, which is leading the development of grant management IT tools at corporate level, is currently taking measures to address beneficiary satisfaction by improving user-friendliness of the IT tools.

According to the survey of the ERCEA’s independent experts, 93% of respondents agreed that the process of appointment and contracting of experts was smooth and clear, and were satisfied with various aspects related to the appointment and contracting process; 98% of surveyed experts stated that the invitation was issued in sufficient time to organise their schedule or work. These results improved compared to the previous survey of independent experts, where 87% of respondents agreed that the process of appointment and contracting of experts was smooth and clear.

The evaluation identified as an area for further improvement the communication with the ERC independent experts serving in the evaluation panels. Some panellists surveyed indicated a possible mismatch between their area of expertise and their allocation to evaluation panels focusing on particular scientific disciplines. However, the selection procedure is expressly designed so that panellists evaluate the ERC project applications outside their own narrow subject specialisation. It was found that this is more a communications issue with members of the evaluation panels, and this could be further improved in the future.

In terms of proximity to the addressees, in the reference period, the ERCEA dedicated significant efforts in engaging with potential applicants but also beneficiaries, both with Principal Investigators (PIs) that hold the ERC grants and with Host Institutions (HIs) that accommodate or host the ERC grant holders, through information days, workshops and conferences organised by the ERCEA in Brussels and across the Member States. In addition, the ERCEA launched the “Top 20 Host Institutions initiative” (which account for ca. 70% of the total ERC grant holders), in order to promote a closer relationship between the ERCEA and leading HIs aiming to provide a more consistent and uniform approach to administrative and financial queries. The ERCEA is also active on social media – its website attracts more than half a million visitors yearly, and the Agency organises information stands at three to five selected scientific conferences every year.

Concerning the visibility of the EU and the particular role of ERCEA as a promoter of the programmes entrusted to the Agency, the feedback from beneficiaries (via interviews) indicate that they are to a very large extent aware that the ERCEA grants were funded from the EU budget. The redesigned logo of the ERC now specifically indicates that the ERC was established by the Commission. Moreover, the ERC grant label has gained in recognition over time, commensurate with the strong bibliometric track record associated with the ERC grantees (publications, performance in terms of citations and in the top-cited publications). Visibility has been further enhanced through the awarding of prestigious scientific prizes, such as the Nobel and Wolf Prizes and the Fields Medal, to ERC grantees. In 2014, an analysis of over 7000 leading researchers in Europe found that 30% had applied to the ERC’s calls and around one in six were ERC grant holders 26 .

5.2 Efficiency

This section considers the relationship between the resources used by the Agency and the output. The analysis, among other factors, also includes analysis of administrative and regulatory burden and looks at aspects of simplification.

Similar to the previous evaluation period, during 2015-2017 the ERCEA proved to be an efficient and cost-effective structure for the management of the delegated programmes. Its administrative budget was within 2.2-2.4% of the operational budget based on commitment appropriations and within the range of 2.9-3.2% based on payment appropriations.

Ratio between the ERCEA’s administrative and operational budget

The ERCEA’s actual operational budget during 2014-2018 constituted EUR 9.172 billion, of which EUR 8.460 billion from EU budget and EUR 711.7 million related to EEA/EFTA 27 and third countries’ contributions (not considered in the ERCEA’s SFS). The ERCEA’s actual administrative budget increased from EUR 39.34 million in 2015 to EUR 48.60 million in 2018 (in commitments, including EEA/EFTA and third countries’ contributions).

A detailed analysis of the ERCEA cost-efficiency and results of the cost-benefit analysis (CBA) are presented under the point 5.4 below.

As previously mentioned, the actual workload of the ERCEA during the evaluation period generally corresponded to the initial 2013 CBA estimates. The actual number of new grants managed by the ERCEA, which is the main workload driver for the Agency, under 2014-2018 calls was 15% higher than CBA forecast. It can be explained by the fact that the increased number of actual grants was primarily related to the additional contributions from EFTA/EEA and third countries that were not included in the CBA forecast.

The actual ERCEA’s operational budget was higher than the SFS estimates for StG (148% of the initial estimates), CoG (123%) and PoC (131%). At the same time, it was lower for AdG (89%) and Synergy Grants (38%), which were re-launched only in 2018, following the implementation of a 2012-2013 pilot Synergy Grants scheme.

CBA estimated vs actual ERCEA’s operational budget 2014-2018 by grant scheme, EUR million

The actual number of proposals (40 532) received under the 2014-2018 calls was substantially lower than the initial CBA calculations (59 500). This could be explained in part by the restriction rules on re-submission 28 . The lower number of applications determined that the success rate for 2014-2018 the ERC calls remained at a reasonable range 29 of 12%-15%, whereas the CBA estimated that it would drop to 7.6%-8.7%.

As the ERC grant schemes under Horizon 2020 remained very similar to FP7, the average grant size estimates for specific grant schemes in CBA were close to the actual average grant size in 2014-2018 calls: StG: EUR 1.427 million (CBA) vs EUR 1.482 million actual; CoG: EUR 1.871 million vs EUR 1.919 million; AdG: EUR 2.271 million vs EUR 2.367 million; PoC: EUR 0.150 million vs EUR 0.149 million and Synergy EUR 10.000 million vs EUR 9.656 million.

Given the increased operational budget to grant schemes from one size, and the lower average grant size from another side, the actual number of grants managed by the ERCEA under 2014-2018 calls was 15% higher than the CBA estimates.

CBA estimated vs actual No of grants, by grant schemes, 2014-2018 calls

The number of running projects ‘per head’ increased from 10.49 in 2014 to 12.52 in 2017. Furthermore, the ratio of budget ‘per head’ decreased from EUR 4.45 million to EUR 4.04 million (in terms of commitments) but increased from EUR 2.60 million to EUR 3.36 million (in terms of payments) in 2014-2017.

Budget, staff and workload indicators, ERCEA, 2014-2017, € MILLION OR %

One of the key challenges for the ERCEA during the reference period related to workload variation across the Agency. Between 2015 and 2018, ERCEA’s workload increased in respect to ethics reviews and number of grants under management. The Agency was able to cope with this additional workload thanks to appropriate allocation of staff (forward resource planning and adoption of workload planning models 30 ), administrative simplifications and efficiency gains.

The ERCEA took a series of actions to cope with the varying levels of workload across its different units. The ERCEA was also effective in filling vacancies and maintaining a relatively low vacancy rate (3%) between 2015 and 2018. Staff turnover was relatively stable throughout the period of 2015-2018 constituting 7% on average. By the end of 2017 the number of the ERCEA staff had risen to 477 (from 417 in 2015).

The staff opinion survey carried out in 2016 revealed that the ERCEA demonstrated consistently high results compared to the averages of the Commission and the Executive Agencies for the three key indicators, i.e. staff engagement index 31 , overall job satisfaction and wellbeing. The ERCEA also ranked second among all the Commission services and the executive agencies in terms of the staff valuing the Commission/executive agency as a modern and attractive workplace. The Agency demonstrated particularly positive results for staff’s understanding of the Commission/executive agency purpose (95%), their willingness to make an extra effort when required (94%) and job clarity (92%).

As was the case with the other executive agencies, the ERCEA demonstrated lower satisfaction rates in the overall mobility and career development, the links between performance and career progression and the available opportunities to move to another job matching their skills and management of career choices. Those results were still somewhat higher than or equal to the levels of agreement for the Commission and other executive agencies.

Responding to these staff opinion survey results, the Agency adopted in 2016 a revised Human Resources (HR) Action Plan and established a new comprehensive multi-annual HR strategy. This was complemented by actions at corporate level, notably by adopting, in 2017, new implementing provisions for promoting the inter-agency mobility of contract and temporary agent staff.

The evolution of key indicators for staff satisfaction – after the evaluated period – is encouraging. In the latest staff opinion survey (4th quarter of 2018, i.e. outside of the evaluation period), the ERCEA managed to increase its result for staff feeling that the Agency cares about their well-being to 69%, up from 58% in 2016. Staff engagement rose to 73% (from 70% in 2016).

Regarding ERCEA’s organisational structure, the Agency is composed of the office of the Director, the office of the Chief accountant, units A1 and A2 of support to the ScC and external communications, respectively, and three departments, Department B (scientific management), Department C (grants management) and Department D (supporting business functions e.g. IT, HR and legal).

Overall, the size and structure of the ERCEA, and the level of resourcing in each department, were appropriate to its mandate and delegated tasks during the evaluation period.

During 2015-2018, a number of simplifications were implemented by the ERCEA which led to efficiency savings, some stemming from changes across the whole of Horizon 2020 (e.g. electronic grants management and reporting), whereas others were instituted internally (e.g. simplification of ethics and granting procedures etc.)

5.3 Coherence

Coherence looks at any overlaps and complementarities within the programme portfolio managed by the ERCEA or delimitation of responsibilities between the ERCEA and its parent DG.

For the period 2015-2018 the evaluation did not find any evidence of duplication, overlaps, gaps or inconsistencies between the ERC programme and other Horizon 2020 programmes, which was attributed to the ERC programme being the sole bottom-up research programme focused on promoting scientific excellence by supporting Principal Investigators. There is a strong continuity in the ERCEA mandate between the FP7 and Horizon 2020 programming periods – the ERCEA is operating already for a decade and its institutional maturity has grown progressively. A strong factor of coherence derives from the fact that ERCEA is responsible for implementing the ERC programme only, i.e. a part of the Research and Innovation framework programmes (Horizon 2020 and legacy of FP7), with common rules and procedures.

The transition to common business support functions provided by the CSC as of 2014 and later CIC required new ways of working by the Agency to ensure the proper balance between the specificities of the ERC programme and its associated grant instruments, and the harmonisation of procedures and IT tools across the DGs and EA within the research family. While some difficulties were encountered in ensuring that the customisation of IT tools meets the ERCEA’s specific needs and on a sufficiently timely basis, this is being addressed and increasingly demonstrates a co-development approach.

As explained in section 5.1, the ERCEA has a specific governance framework compared to other executive agencies. It has only one parent DG, DG RTD, while it serves as a dedicated implementing structure of the independent European Research Council. This is also reflected in the composition of its Steering Committee, which comprises five members, including the Director-General of DG RTD, Director-level representatives from DG RTD-A and DG HR, and two members of the ScC. The parent DG has supervisory responsibilities for monitoring the ERC programme implementation, and in overseeing the execution of the Agency’s administrative and operational budget. Delimitations of responsibilities and tasks between the ERCEA and its parent DG, as well as the working arrangements and procedures for interaction, are set out in a clear and appropriate manner in the Written Agreement of 2015.

Overall, there is an effective mechanism for monitoring, reporting and supervision enabling the Commission to benefit from the know-how created within the Agency. The monitoring and reporting data and information on the ERC programme implementation through regular reporting mechanisms (e.g. Annual Activity Report, quarterly and monthly financial reporting, data analysis of programme implementation by country, theme, grant scheme etc.) are of high-quality, and sufficient for governance and scrutiny purposes.

Also, there is an effective, both formal and informal, information flow between the ERCEA and the Commission services, and between the ERCEA and the Scientific Council to ensure that both the ERC and the parent DG are kept informed about the Agency’s performance and implementation of the ERC programme.

In the reference period the ERCEA stepped up its efforts in providing policy feedback to the ERC’s ScC to facilitate scientific policy-making on frontier research. Measures included policy briefings and data analyses of the ERC programme implementation data; presentation materials for external presentations (e.g. to national policy makers and scientific audiences, but also initiatives like ‘Science behind the Project’, the ‘Qualitative Evaluation of Completed Projects’, and the ERIS database, that focus on the identification, categorisation, monitoring and assessment of ERC calls for proposals and projects.

However, the evaluation identified the domain of policy feedback on frontier research as an area where the ERCEA’s role should be further defined and clarified. The parent DG should further define its needs in terms of policy feedback, including the format and frequency, and a corresponding business process should be agreed between the Agency and its parent DG. Currently the CIC is leading the efforts of strengthening the processes and procedures for feedback to policy at the R&I family level as part of the Strategy for an Effective Dissemination and Exploitation of Research Results in Horizon 2020.

5.4 Retrospective Cost-benefit analysis

The retrospective cost–benefit analysis for the 2015-2018 period was carried out based on the results of the 2013 ex-ante CBA, the assumptions laid down in the Specific Financial Statements (SFS) and the actual costs of the ERCEA.

As mentioned in section 2.2, the 2013 ex ante CBA estimated the savings resulting from the delegation of tasks to the ERCEA at around EUR 79.4 million over the whole period 2014-2024 when compared to the management by the Commission services (in-house scenario), while for the period of 2015-2018 these were estimated at around EUR 40.77 million. The ERCEA SFS estimated the savings of the executive agency scenario at EUR 72.65 million for the period 2015-2018. The ERCEA SFS differs from the CBA estimations for the administrative budget as a result of adjustment of the ERCEA staff numbers (reduction by around 12%) and calculation of expenses without any indexation.

The retrospective CBA revealed that the executive agency scenario actual costs in 2015-2018 were much lower than the costs of the in-house scenario. The savings constituted EUR 76.97 million 32  and were 89% higher compared to the initial CBA estimates and 6% higher compared to the initial SFS estimates.

Estimated costs and savings of the executive agency scenario in 2015-2018, EUR

As forecasted in the SFS and the ex-ante CBA, savings of the executive agency scenario primarily resulted from a higher share of lower-cost contract staff employed within the Agency and the lower overall number of staff.

Significant cost savings occurred in the ERCEA’s Title II ‘Infrastructure and operating expenditure’. The costs in Title I ‘Staff-related expenditure’ were higher than estimated in the SFS, as the latter did not take into account increases to staff costs due to salary indexation, promotions and increasing staff seniority.

As to the assumptions laid down in the CBA, it should be noted that the ERCEA actual workload during the evaluation period generally corresponded to the initial estimates, with variations across different grant schemes and years. The Agency duly took measures to ensure appropriate staffing levels across all departments, an appropriate balance between operational staff and administrative support staff and adequate flexibility to move resources around on the basis of workload fluctuations.

The actual number of new grants managed by the ERCEA under 2014-2018 calls increased by 13% compared to the 2013 CBA forecast. This is a result of the increased actual operational budget of the ERCEA by 9% given the additional contributions from EEA/EFTA and third countries. On the other hand, the average grant size of the ERC grant schemes under Horizon 2020 remained similar to FP7.

6.Conclusions

The results of the evaluation of the ERCEA operations during 2015-2018 period confirm that delegation of the ERC programmes to the Agency was fully justified in terms of the cost-savings achieved and value added generated. Higher than expected savings for the EU budget were made and these were further enhanced through administrative simplifications (e.g. from electronic grants management and reporting). The ERCEA demonstrated strong value for money based on the assessment of its performance. The continued delegation of the ERC programme implementation remained considerably more cost-effective than the in-house scenario, generating substantial savings for the EU budget.

The ex-post cost benefit analysis found that the actual savings of the executive agency scenario were of around EUR 77 million during the period 2015-2018 compared with the in-house scenario.

Given that the ERC was set up as an autonomous, scientifically-driven entity, the ERCEA has provided an effective dedicated implementation structure to the ERC. This would have been difficult to achieve were the programme implementation functions to be housed within the Commission. The ERCEA continued to contribute to the strong image and reputation of the ERC in the scientific world while the experts selected as the ERC Evaluation panel members were high-calibre.

During this period, the ERCEA was effective in achieving its core objectives relating to both programme implementation and supporting the ERC’s ScC, and its beneficiaries were highly satisfied with the performance of the Agency. The ERCEA has consistently achieved its KPI targets. Beneficiaries’ satisfaction with the ERCEA performance increased to 89% compared with the previous evaluation (82% in 2015). The ratio of budget ‘per operational head’ increased from EUR 3.46 million to EUR 3.54 million (in terms of commitments) or from EUR 2.44 million to EUR 2.99 million (in terms of payments) in 2014-2017. The Agency’s administrative budget was below 3% of the operational budget based both on commitment and payment appropriations between 2015 and 2017.

The coordination and communication mechanisms that are in place facilitate information flows between the ERCEA and the Commission services, in particular with regard to the content of projects supported and their results. No evidence of overlaps, duplication, gaps and inconsistencies within the programme portfolio were identified.

The current evaluation identifies only three areas where further improvements are necessary from either the ERC, the ERCEA, Commission or the ERCEA and Commission together. These are in regard to:

·The communications to ERC panel members on the rationale for their selection.

·The ERCEA’s ethics supporting guidance document and the beneficiaries’ awareness on the need for particular ethics-related information for the ERC grants.

·The ERCEA’s role in extracting and providing policy-relevant feedback to the Commission.

The parent DG and the Agency are preparing an action plan to address the shortcomings identified in this evaluation. This action plan aims to increase the Agency’s performance and contribution to an efficient implementation of the delegated programmes.

ANNEX: EVALUATION QUESTIONS

Effectiveness

·To what extent has ERCEA been operating according to the legal framework establishing it?

·To what extent has ERCEA achieved its objectives with special focus to (a) the implementation of the delegated programmes Horizon 2020 and FP7, (b) support to the Scientific Council and (c) the implementation of the internal control principles notably sound financial and human resource management? What, if anything, could be done to render ERCEA more effective in achieving these objectives?

·To what extent has ERCEA contributed to an improved management of the delegated programmes and better services to the stakeholders and addressees9 in terms of the elements assessed in the 2013 CBA and as compared to the alternative options mentioned in that CBA?

·To what extent has ERCEA contributed to improved management of the programmes in terms of:

− Proximity to addressees (e.g. role played by the Agency as focal point for applicants and beneficiaries of the programmes; instruments and mechanisms put in place for communicating with the applicants and beneficiaries).

− Effective implementation of the programmes, taking into account the interests of the addressees and those of the EU:

ürate of execution of commitment appropriations;

ürate of execution of payment appropriations;

ütime to grant;

ünet time to pay;

üresidual multi-annual error rate identified at ex-post control.

− Visibility of the EU as promoter of the programmes entrusted to the Agency (e.g. compliance with the Commission’s guidelines on information and visibility of programmes10, instruments put in place to ensure the visibility of the EU as promoter of the programmes).

Efficiency

·To what extent have the actual costs (including cost of coordination and monitoring) of ERCEA corresponded to the estimates of the 2013 CBA? If not, what are the reasons behind?

·To what extent have the actual benefits corresponded to the estimates of the 2013 CBA? If not, what are the reasons behind?

·To what extent has the management and execution of the programmes by ERCEA been cost-effective as compared to the alternative options11?

·To what extent has ERCEA contributed to improved management of the programmes in terms of:

oSimplification of the procedures and flexibility in the implementation of delegated tasks (e.g. capacity to adapt to periods of high workload)?

oWhich further scope for simplification exists?

·Which aspects/means/actors or processes render ERCEA more or less efficient? What could be improved? What is the quality of the services/advice provided by ERCEA to stakeholders and addressees?

·To what extent has the establishment of ERCEA resulted in savings to the EU budget as compared to the alternative options (e.g. difference in costs between the Commission option and the Agency option)?

·To what extent have ERCEA’s internal organisation and procedures been conducive to its efficiency?

·To what extent is ERCEA’s internal organisation capable and flexible to rapidly respond to resource needs due to uncertainties related to volumes of work?

·To what extent does ERCEA’s human resources management contribute to the achievement of the Agency’s objectives?

oIs the size and structure of the organisation appropriate?

oIs the balance between operational staff and administrative support staff appropriate?

oAre the staff turnover and vacancy rate well managed?

oHow does the Agency follow up on the findings of the latest staff survey (2016)?

Coherence

·To what extent have there been overlaps/ gaps/ inconsistencies/ complementarities within the programme portfolio managed by ERCEA and how are these addressed?

·Is there a clear and appropriate delimitation of responsibilities and tasks between ERCEA and the Parent DG? Are there overlaps or gaps? Are the different responsibilities adequately communicated to the beneficiaries?

·To what extent has ERCEA enabled the Commission to better focus on its policy related tasks?

oAre there any governance (financial and policy) issues in relation to the implementation of the tasks that are delegated to the executive agencies/ issues as regards the Commission’s services being able to steer EU policy or budget implementation?

oAre appropriate mechanisms and instruments in place, and at which level, to ensure an adequate coordination and information flow between ERCEA and the Commission services, notably on the content of the projects supported and their results?

oDoes ERCEA provide useful information on the implementation of the delegated programmes and their progress (in terms of management and content) in support of the policy process (e.g. information required for the annual Management Plan of the Parent DG)?

·To what extent have the activities of ERCEA resulted in unintended effects (both desirable and undesirable)?

·To what extent has the Commission, in the presence of ERCEA, been able to maintain an adequate level of know-how in relation to the programmes entrusted to the Agency? How has this been achieved? What are the feedback channels, means and methods used for this purpose? What are areas for improvement, if any?

·Have the monitoring, reporting and supervision arrangements in place enabled the Commission to benefit, in the short and medium term, from the know-how created within ERCEA?

·How effective is the flow of information and communication between ERCEA and the Commission services (in particular Parent DG), and between ERCEA and the Scientific Council?

·To what extent would the closing down of ERCEA result in losing significant know-how in relation to the management of the programmes entrusted to ERCEA?

(1)   http://data.europa.eu/eli/reg/2003/58/oj

(2) http://data.europa.eu/eli/dec_impl/2013/779/oj

(3) http://ec.europa.eu/transparency/regdoc/index.cfm?fuseaction=list&coteId=3&year=2013&number=9428&language=EN

(4)   Commission’s Better Regulation Communication COM (2015) 215 final and its accompanying SWD (2015) 111 final and SWD (2015) 110 final .

(5)  The assessment of ‘EU added value’, why the EU should act, is not perceived to be a relevant criterion, as ERCEA executes tasks which the European Commission has transferred to it. The EU added value of the programmes that ERCEA manages is assessed as part of the programme evaluation. The previous needs which ERCEA were meant to address and whether they still are pertinent today (relevance) are presented in the context of the evaluation.

(6)  Cost-Benefit analysis for the delegation of certain tasks regarding the implementation of Union Programmes 2014-2020 to the executive agencies - Final report for the Commission of 19 August 2013.

(7)

 Identification of the tasks justifying outsourcing, a cost-benefit analysis which includes the costs of coordination and checks, the impact on human resources, possible savings within the general budgetary framework of the European Union, efficiency and flexibility in the implementation of outsourced tasks, simplification of the procedures used, proximity of outsourced activities to final beneficiaries, visibility of the EU as promoter of the EU programme concerned and the need to maintain an adequate level of know-how inside the Commission.

(8)

Appendix II of the Guidelines for the establishment and operation of executive agencies financed from the Union budget (C (2014) 9109 from 2 December 2014; pp. 64-72).

(9) Study supporting the evaluation of the European Research Council Executive Agency (2015-2018), final report prepared by CSES and PPMI, March 2019.

(10)

 Written Agreement between the European Research Council Executive Agency and the Directorate General for Research and Innovation – Modalities and Procedures of Interaction, as last amended on 17 December 2015.

(11) The Scientific Council is the decision making body of the ERC and sets the ERC’s scientific funding strategy. The chair of the Scientific Council is also the President of the ERC.

(12) Under other research family programmes, candidates have to register in the Commission’s central database of experts via the Participant Portal. The experts are then selected and afterwards contracted and managed centrally by REA. The ERCEA’s independent experts do not have to register via the Participant Portal. The ERC selects experts for their peer review tasks on the basis of a proposal from the ERC’s ScC.

(13) Under the 2018 ERC WP.

(14) Under ‘Funding’, up to an additional EUR 500 000, 750 000, 1 000 000 and 4 000 000 for StG, CoG, AdG and SyG respectively, could be requested in the proposal to cover the following additional extraordinary costs: (a) eligible “start-up” costs for Principal Investigators moving to the EU or an Associated Country from elsewhere as a consequence of receiving the ERC grant and/or (b) the purchase of major equipment and/or (c) access to large facilities.

(15) The CBA estimated that it would drop to 7.6%-8.7%.

(16) Total number of staff, financed from EU Budget and contributions from EFTA/EEA and/or third countries.

(17) For comparison, REA, the largest executive agency, had 735 staff members in 2018 while CHAFEA had 59 staff members in 2016.

(18) Written Agreement between the European Research Council Executive Agency and the Directorate General for Research and Innovation – Modalities and Procedures of Interaction, 17 December 2015.

(19)  Due to their specificity, the ERC grants have a derogation from the common Horizon 2020 targets as per Art. 20(3) of the Horizon 2020 Rules for Participation and the TTI, TTS and TTG targets are set in the corresponding Annual Work Programmes of the ERCEA.

(20)  The time to inform all/ successful applicants on the outcome of the evaluation of their application from the final date for submission of completed proposals.

(21) The time to sign grant agreements from the date of informing successful applicants.

(22) The time from the final date for submission of complete proposals to signature of grant agreements with applicants.

(23)  Between 0.02 and 0.04% in 2015-2017.

(24) ERCEA ethics review has an increased complexity as the frontier research is often carried out at the boundaries of knowledge. Also, for some areas of research (such as human embryonic stem cell research) the ethics review had to involve the Commission and the Member States (comitology procedure), which prolonged the ethics review process.

(25)

i.e. mostly individual researchers, while most of the Horizon 2020 themes support collaborative research projects where at least three organisations from different countries form a consortium.

(26) Interim evaluation of Horizon 2020, 16 August 2017, https://publications.europa.eu/en/publication-detail/-/publication/fad8c173-7e42-11e7-b5c6-01aa75ed71a1/language-en/format-PDF/source-77918455 .

(27) EEA stands for ‘The European Economic Area’; EFTA stands for ‘The European Free Trade Association’.

(28) Due to the past increasing trend in resubmissions, these restrictions were introduced by the ERC Scientific Council not only to contain the experts’ workload at a reasonable level and hence avoid compromising evaluation quality but also to keep the overall success rate at an appropriate level around 15%.

(29) The average success rate for all calls under respective Annual Work Programme.

(30) These models are still being used to estimate resource needs, and were viewed by interviewees at the ERCEA as having been an effective planning tool, one which has fed directly into resource and during annual budgetary and Annual Work Programme implementation.

(31) The staff engagement index is composed of 7 individual survey questions:•    I have the appropriate and timely information to do my work well.•    My colleagues are committed to doing quality work.•    I have a clear understanding of what is expected from me at work.•    I have recently received recognition or praise for good work.•    I feel that my opinion is valued.•    My manager seems to care about me as a person.•    My line manager helps me to identify my training and development needs.

(32) These costs do not take into account the additional contribution to the ERCEA’s administrative budget of EUR 7.65 million over 2015-2018 coming from EFTA/EEA and third country contributions to manage the additional operational budget of EUR 712 million coming from these countries.