EUROPEAN COMMISSION
Brussels, 29.4.2020
SWD(2020) 77 final
COMMISSION STAFF WORKING DOCUMENT
Evaluation of the Research Executive Agency (REA)
Accompanying the document
Report from the Commission to the European Parliament, the Council and the Court of Auditors
Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Executive Agency for Small and Medium-Size Enterprises, the Innovation and Networks Executive Agency, the Education, Audiovisual and Culture Executive Agency, the Research Executive Agency and the European Research Council Executive Agency
{COM(2020) 184 final} - {SWD(2020) 73 final} - {SWD(2020) 74 final} - {SWD(2020) 75 final} - {SWD(2020) 76 final} - {SWD(2020) 78 final}
TABLE OF CONTENTS
1.
INTRODUCTION
2.
BACKGROUND
2.1 Description of the intervention and its objectives
2.2 Baseline and points of comparison
3.
IMPLEMENTATION / STATE OF PLAY
4.
METHOD
4.1 Short description of methodology
4.2 Limitations and robustness of findings
5.
ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS
5.1 Effectiveness
5.2 Efficiency
5.3 Coherence
5.4 Retrospective Cost-Benefit Analysis
6.
CONCLUSIONS
ANNEX : EVALUATION QUESTIONS
1.
1.Introduction
The purpose of this periodical (every three years) evaluation of the operation of the Research Executive Agency (REA) is to assess REA’s implementation of the parts of the EU funding programmes which have been entrusted to it and the provision of administrative and logistical support services. REA is governed by the
Framework Regulation for Executive Agencies
, its
Act
of Establishment
, which sets out its mandate, the
Act of Delegation
which specifies the tasks to be carried out by, and the powers delegated to the Agency in order to perform its mandate; and the decision establishing the REA Steering Committee.
In line with the Commission’s Better Regulation principles, the evaluation applies several standard evaluation criteria. The evaluation assesses whether the Agency has fulfilled its tasks in an effective and efficient way, whether there are overlaps / gaps / inconsistencies in the management of the programme portfolio by the Agency, and whether there is a clear delineation of tasks between REA and the parent DGs or other executive agencies (coherence).
The evaluation also assesses whether the functioning of the Agency has yielded the expected positive results as estimated in the ex ante Cost-Benefit Analysis (CBA) for the delegation of tasks to the executive agencies and to identify potential areas of improvement. To this end the estimations of the Cost-Benefit Analysis of 2013 have been tested in order to provide evidence on the validity of the assumptions made in the ex ante scenario by considering the actual costs and benefits of programme implementation by the Agency in a structured way. The aspects covered by the Cost-Benefit Analysis are specified in Article 3(1) of the Framework Regulation
and the Guidelines on establishing and operating executive agencies
.
The evaluation does not cover the achievements of the programmes managed by REA, which are subject to mid-term and ex post evaluations themselves. The evaluation of REA nevertheless provides useful input for these programme evaluations, considering that the performance of the Agency affects the efficiency and effectiveness of the programmes it manages.
The evaluation examines the efficient use of resources and the effective achievement of the tasks entrusted to the Agency. In particular, it looks at whether the alignment of more coherent programme portfolios within the Agency’s core competences and its brand identity delivered the estimated qualitative benefits; whether the assembly of the management of different EU programmes and provision of support services delivered the estimated synergies, simplification and economies of scale; whether the pooling of instruments guaranteed consistent service delivery and whether there is scope for simplification and further efficiency gains.
The evaluation covers all the tasks carried out by the Agency during the period 16 July 2015 to 15 July 2018. The evaluation is supported by a study carried out by an external contractor.
The results of this study are summarised hereafter and this evaluation will be presented to the European Parliament, to the Council and to the Court of Auditors in accordance with Article 25(1) of the Framework Regulation.
The results will feed into the reflection to assess the opportunity of expanding/ modifying the Agency’s mandate in view of the delegation of the implementation of 2021-2027 EU programmes.
2.Background
2.1 Description of the intervention and its objectives
The outsourcing of certain management tasks to the Agency according to the Framework Regulation and REA’s Act of Establishment intends:
-To allow the Commission to focus on its institutional tasks, i.e. tasks assigned to the institutions by the Treaty, which require discretionary powers in translating political choices into action. Such institutional tasks should not be outsourced.
-To enable the Commission to achieve the objectives of the delegated EU programmes more effectively and efficiently. According to the Cost-Benefit Analysis carried out in 2013, delegating tasks to REA was estimated to be more cost-efficient than an in-house scenario. The Act of Establishment estimated that the alignment of more coherent programme portfolios with the Agency’s core competences and its brand identity would bring qualitative benefits. In addition, it projected that assembling the management of different EU programmes would bring synergies, simplification and economies of scale.
The original objective of the intervention is entrusting the Agency with the implementation of several EU programmes in parts or fully:
-Grants to Horizon 2020 framework programme for research and innovation projects (
Horizon 2020
). REA manages scientific projects funded under the following parts of the Horizon 2020 framework programme (grouped by pillar):
oExcellent Science:
§Marie Skłodowska-Curie Actions (MSCA);
§Future and Emerging Technologies.
oIndustrial leadership:
§Space Research.
oSocietal challenges:
§Food Security, Sustainable Agriculture and Forestry, Marine, Maritime and Inland Water Research and the Bioeconomy;
§Innovative, Inclusive and Reflective Societies;
§Secure Societies (partially).
oSpecific objectives:
§Spreading Excellence and Widening Participation;
§Science with and for Society.
As of 2018, REA’s mandate was extended to the processing of grants to projects that may generate EU classified information (EUCI), including under the specific objective Secure Societies.
-Research projects of the 7th framework programme for research and innovation (
FP7
). The Agency continues to manage ongoing FP7 research projects in the following areas:
oCertain programme areas of the specific programme PEOPLE of FP7;
oThe actions research for SMEs and research for SME associations of the specific programme capacities of FP7;
oThe theme space of the specific programme cooperation of FP7;
oThe theme security of the specific programme cooperation of FP7 (partially).
-In addition, REA was entrusted with administrative and logistical support services:
oThe Agency runs the support for independent experts evaluating project proposals, including the management of on-site evaluation facilities for most of Horizon 2020 activities and provides services for the legal and financial validation of participants in Horizon 2020 and certain other programmes;
oAs of 2018, in the context of the Commission’s initiatives for synergies and efficiencies between programmes, REA plays a major role in the development and roll-out of the Single Electronic Data Interchange Area (SEDIA) project, building on REA’s participant validation activities supported by the Participant Data Management tool. This entails the legal validation of third parties and the preparation of financial capacity assessments for all the Commission services and executive agencies (which implement grants and procurements under direct management) and the first level of transactions under indirect management.
The Agency has six parent Directorates-General:
-Research and Innovation (RTD) – Lead parent DG;
-Internal Market, Industry, Entrepreneurship and SMEs (GROW);
-Education and Culture (EAC);
-Communications Networks, Content and Technology (CNECT);
-Agriculture and Rural Development (AGRI);
-Migration and Home Affairs (HOME).
The Agency also works closely with the Common Support Centre (CSC), now named the Common Implementation Centre (CIC), administratively part of DG RTD, which provides the framework for harmonised implementation of the EU Research and Innovation (R&I) programmes for entire DGs in the R&I family of implementing bodies, and executive agencies and Joint Undertakings. The CSC is responsible for defining common business processes for grant management, and providing IT, legal, audit and information and data services to the Horizon 2020 implementing bodies.
With regard to the tasks delegated in the framework of Horizon 2020 and FP7 legacy, REA performs the following programme implementation tasks:
a.Manage some or all of the stages in the lifetime of projects of the delegated programmes. In this regard, it is responsible for launching calls for proposals, evaluating proposals and signing grant agreements, monitoring the projects, making the necessary checks and recovery procedures, and for performing budget implementation tasks covering revenue and expenditure within the meaning of Regulation (EU, Euratom) 966/2012;
b.Perform all the operations required to launch contests and award prizes in accordance with the Regulation (EU, Euratom) 966/2012;
c.Conclude public procurement procedures and manage the ensuing contracts, including the operations required to launch and conclude public procurement procedures;
d.Provide support in programme implementation.
2.2 Baseline and points of comparison
The current evaluation of REA operations during 2015-2018 assesses the actual costs and benefits of programme implementation by REA (executive agency scenario) when compared with the alternative scenario of management by the Commission services (in-house scenario).
Accordingly, the reference point for the present REA evaluation is the 2013 ex ante Cost-benefit analysis, and the specific financial statement (SFS) of REA.
Wherever possible, the analysis of the performance of REA during the period of reference assesses the progress compared to the two previous evaluations, covering the Agency’s operations during 2009-2012 and 2012-2015. The current evaluation covers the period between the beginning of the second half of 2015 and the end of the first half of 2018. However, wherever possible and relevant, and provided that all the necessary data was available, the analysis was extended to include the whole of 2015 and, at times, 2014.
Over the period 2009-2013 the savings resulting from the delegation of tasks to REA have been evaluated at approximately EUR 106 million compared to the in-house scenario.
According to the SFS of REA prepared in 2013 the estimated efficiency gains were of EUR 80 million over the period 2015-2018. The total number of full time equivalents (FTEs) required to manage the relevant programmes in year 2020, the peak programming year in terms of workload, was estimated at 793 for REA. The total estimated operational budget entrusted to REA in 2020 is around EUR 2 511,382 million in commitment appropriations and EUR 2 188,754 million in payment appropriations. Considerable efficiency gains were expected as compared to the in-house scenario over the period, along with non-quantifiable benefits such as improved quality of programme management and service delivery, improved visibility of the EU programmes and proximity to beneficiaries.
3.Implementation/State of play
The Agency’s mandate was extended by the Commission Implementing Decision No 2013/778/EU, which came into effect on 1 January 2014. Based on its renewed mandate, REA implemented large parts of the Horizon 2020 programme and the FP7 actions delegated to the Agency under its first mandate. In addition, REA was also mandated to provide administrative and logistical support services to all entities involved in Horizon 2020 management and the provision of participant validation services for a variety of programmes. As of 1 January 2018, REA plays an important role in the SEDIA project as detailed under point 2.1.
The governance of the Agency remained the same at the start of the new mandate and was adjusted in 2016 when DG HOME joined the REA Steering Committee as a new parent DG. A new Memorandum of Understanding (MoU)
between REA and its parent DGs was concluded in 2014 and further adjusted in 2016. The MoU defined the modalities and procedures of interaction between REA and its parent DGs in view of the implementation of Horizon 2020 (and legacy of FP7) by the Agency. The distribution of tasks and responsibilities, as presented in the MoU, is included in sections 5.1 and 5.3 of this document. Compared to this reference, this evaluation assessed whether this distribution of roles has been respected for the 2015-2018 period, and the effects of the intervention on the coherence and the efficiency of REA’s action.
The operational budget managed by REA ranged between EUR 1 520 - 2 010 million (in terms of commitments) and between around EUR 1 420 - 1 700 million (in terms of payments) per year during the period 2015-2017. On average, REA’s staff managed an increasing volume of the operational budget during this period, primarily because the growing volume of the operational budget outpaced the increase in the number of the Agency’s operational staff (i.e. REA’s staff minus staff working in the central support services).
The annual number of proposals received at REA fluctuated around 15 000 between 2015 and 2017 and the total number of running projects decreased from 7 020 in 2015 to 6 420 in 2017. On the other hand, the budget per operational head (in commitments) increased from EUR 3.03 million in 2015 to EUR 3.54 million in 2017. During the reference period, the actual workload of REA was higher than estimated in the 2013 CBA due to a higher than expected number of new Horizon 2020 grants. This also increased the workload of monitoring project implementation.
REA gradually grew in size in accordance with the multi-annual planning stated in the SFS and was the largest Executive Agency in 2018 with 735 actual staff members. The Agency is based in Brussels.
A series of organisational and procedural changes took place during the implementation of the Horizon 2020 programme under the multiannual financial framework (MFF) of 2014-2020:
-Delegation of the management of additional activities and objectives, as detailed under point 2.1 above, and as a consequence new tasks for the Agency;
-Further simplification and improvement of grant management processes and procedures, notably electronic submission of proposals, due to the adoption of new business processes and use of the Commissions Horizon 2020 IT tools and systems (i.e. Compass and SyGMa);
-The adoption of the multiannual human resource (HR) strategy for 2017-2020 to address the findings of the 2016 staff opinion survey. Based on this new strategy and in combination with a dedicated action plan addressing attention points identified through the survey, the Agency implemented specific measures related to staff engagement, internal communication, career development opportunities, work and private life balance and training, meant to improve effective HR management and increase the effectiveness and flexibility of the Agency’s operations.
4.Method
4.1 Short description of methodology
The evaluation was supported by a study carried out by external contractors. The methodology of the REA evaluation was consistent with the approaches employed for similar parallel evaluations of ERCEA, EACEA, INEA, EASME and CHAFEA, and built on the previous evaluation of REA operations during 2012-2015. Whenever possible, data was collected and the analysis extended to also cover 2014, the whole of 2015 and the whole of 2018, even if formally outside the evaluation timeframe. Specific methodological approaches were used for each of the evaluation tasks (Regulatory framework, REA mission and governance; REA performance between 2015-2018, including effectiveness, efficiency and coherence; the Cost Benefit Analysis and Conclusions and Policy Recommendations) to ensure data triangulation (confirmation from different sources). The study was structured around a series of evaluation questions available in Annex.
The evaluation relied on the evidence gathered and analysed using qualitative and quantitative methods and included:
-an extensive documentary review and desk research of REA and Commission documents, including Annual Work Programmes and Annual Activity Reports, audit reports and related documents, CBA studies and other financial documents, the Memorandum of Understanding (MoU), previous evaluations and action plans addressing the ensuing recommendations; the evaluation relied also on the results of the 2016 staff opinion survey;
-31 semi-structured in-depth interviews with Commission staff and REA staff, including: members of the REA Steering Committee; selected Commission staff in the parent DGs/mirror units; the Director of REA & heads of department; heads of unit/representatives of the operational units; heads of unit/representatives of units of the Administrative Department; Representatives of the REA Staff Committee, other employees;
-15 interviews with beneficiaries of different programmes (including MSCA, Societal Challenges, Industrial Leadership, and Specific Objectives), funding instruments, and organisation types (i.e. higher education institutions, public research organisations, small and medium enterprises);
-3 surveys addressing the REA’s beneficiaries (652 answers or 10.5% response rate), unsuccessful applicants (306 answers or 6.1% response rate), experts (2470 answers or 39.7% response rate) and of Commission staff (50 answers or 59.5% response rate);
-a retrospective cost–benefit analysis;
-a focus group meeting with representatives of REA and Commission staff (including the Common Support Centre), to validate the draft conclusions and recommendations of the evaluation;
-a comparative analysis and benchmarking of the various executive agencies was also carried out against a set of (qualitative and quantitative) indicators. This enabled the evaluation team to compare the performance of REA effectively with that of other executive agencies.
4.2 Limitations and robustness of findings
All the relevant stakeholders within the Commission and REA were duly consulted via interviews and surveys. An extensive number of REA applicants, beneficiaries and experts were covered by the surveys and the response rates are solid and comparable with similar evaluations. However, compared with the overall population reached, unsuccessful applicants participated less actively in the survey than programme beneficiaries.
No sampling bias was observed as the profile of the respondents to the surveys was very similar to the overall population, guaranteeing statistical representativeness. The non-response bias (not all characteristics of the group that did not reply had been captured in full) was mitigated through triangulation with the results of follow up interviews. The triangulation approach, using multi-level and multi-stakeholder dimension in the data collection, ensured the robustness and reliability of the data and information used to draw up conclusions in the supporting study.
Some views from Commission and Agency staff gathered from the interviews and mentioned in the study illustrate personal experience, although they may not be representative of overall relations between the Commission and the Agency.
5.Analysis and answers to the evaluation questions
Analysis and answers to the evaluation questions are grouped below as per the three evaluation criteria: effectiveness, efficiency and coherence. The quantitative results of the ex post cost benefit analysis are presented separately in section 5.4 below, while the qualitative aspects are embedded in the answers to the evaluation questions (sections 5.1-5.3). The detailed evaluation questions and sub-questions are listed in annex.
5.1 Effectiveness
For the purpose of this evaluation, effectiveness relates to how successful the Agency has been in achieving or progressing towards its objectives.
During 2015-2018, REA operated according to the legal framework that established it. REA’s activities, as planned in the annual work programmes and reported in the annual activity reports, corresponded to the tasks set out in the Commission Decisions and the Act of Delegation.
The modalities and procedures of interaction between the Agency and its parent DGs are set out in the Memorandum of Understanding (MoU) between REA and its parent DGs of 2016. The MoU provides for a supervision strategy aimed at avoiding gaps and duplication of efforts resulting from crossover between monitoring and supervision tasks of the parent DGs and the execution tasks of the Agency, while ensuring that no micro-management was present. In 2018, three separate legal documents were adopted in the context of REA’s extended mandate. These provisions have generally worked well and all parent DGs and REA appreciate the effectiveness of the collaboration. However, several processes related to the selection of independent experts, validation of expert lists, and the participation of Commission staff in project monitoring activities require further consolidation. In these areas, the interviews and the survey revealed that a number of Commission staff have different expectations as regards REA’s role, possibly due to varying levels of awareness of the MoU provisions. Accordingly, the Commission and REA should work together to ensure that these provisions are clearly communicated to Commission staff.
The Agency was flexible and effective in addressing the key changes that occurred during the evaluation period (delegation of a specific role in SEDIA; implementing projects generating EU classified information for Societal Challenge 7), and continuously improving its operations, even though it was already achieving a high level of overall effectiveness. REA began the delivery of corporate validation services on 1 January 2018. During 2018, the participant validation activity grew progressively with the integration of new clients from both grant and procurement domains.
In 2015-2018 REA achieved very good results in terms of most key performance indicators (KPI). Compared to previous years the Agency’s performance improved during the period of reference. The prompt conclusion of grant agreements (measured in ‘Time-To-Grant’ – TTG) improved, on average, from 222 days in 2014 to 203 in 2015 and became stable at 193 days during 2016-2018, 52 days (over 20%) below the Horizon 2020 target (245 days). The share of grants concluded within TTG targets reached nearly 100% for the 2016-2017 calls. Regarding payments to grants, the average Time-to-Pay (TTP) stood well below the contractual thresholds for all types of payments (pre-financing, interim and final payments) in 2015-2018 both for FP7 and Horizon 2020. Concerning the share of payments within contractual limits, nearly 100% of all pre-financing payments in 2015-2018 were executed on time. With regard to interim and final payments, REA maintained a similar performance level compared to the previous evaluation period for FP7 (94% of payments executed on time) and during 2017-2018 improved its performance for Horizon 2020 (98% of payments executed on time).
As was the case in previous years, REA achieved full execution of its operational budget both in commitment and payment appropriations during 2015-2018.
The REA target with regards to the share of projects that achieved all or most of their objectives, the REA Annual Activity Reports showed that over 95% of FP7 projects that closed in 2015-2018 achieved all or most of their objectives (against a target of 90%).
With regard to the legality and regularity of transactions, the estimated residual multi-annual error rate remained in a similar range over 2014-2018 compared with those in the previous years for the FP7 programmes implemented by REA. The residual multi-annual error rate for the People Programme remained below 2% while for the SME actions, Space and Security it was above the materiality threshold of 2% that lead REA to enter reservations in its Annual Activity Reports. In addition, it should be noted that the estimated error rate can be attributed, to a large extent, to the complexity of the funding schemes, which is generic to FP7 as a whole; therefore, these error rates remained in line with the general FP7 trend. For Horizon 2020 financial transactions, REA reports the residual error rate common to the Research family as a whole (2.24% for 2017) but also on REA specific residual error rate (1.46% for 2017). Concerning MSCA, there were not enough audits to draw any conclusions for Horizon 2020 but it is expected that the error rate will remain below 2%.
REA’s efforts to improve the quality of the evaluation process and the Evaluation Summary Reports (ESRs) were reflected in a reduced share of evaluation review/redress cases filed and (fully or partially) upheld compared to the number of proposals evaluated. The share of evaluation review/redress cases filed decreased from 3.0% for the 2011 FP7 calls to 1.2% for the 2017 Horizon 2020 calls. Similarly, the share of evaluation review/redress cases upheld decreased from 0.8% for the 2011 calls to 0.3% for the 2017 calls and for the 2014-2017 Horizon 2020 calls it stayed below the maximum target of 0.5%.
As regards the provision of administrative and logistic support services, and in particular the management of the Research Enquiry Service (RES), Financial Capacity Checks/Assessments and the participant Validation Services, the overall performance of the Agency was favourable and target deadlines for validations were fully respected between 2015 and 2018. In 2017, 7 900 enquiries were received of which 99.4% were answered or forwarded within 8 days, with an average of 3 working days. The REA target to perform 95% of validations within the 90 days from the “raise priority” date – for all direct management operations at the Commission – was fully met as of 30 June 2018. For the period 2014 to 2016 REA handled from 1 600 to 2 000 Financial Capacity Assessments (FCA) requests per year. From 2017, the reporting on assessments provided to clients changed – only those cases in which financial data had been provided to REA clients were taken into account.
Overall satisfaction with the performance of REA is one of the highest among executive agencies. 86% of beneficiaries responding to the survey were satisfied (42% ‘very satisfied’) with the services provided by the Agency. This result is a further improvement compared to the previous evaluation of REA (82% in 2015) and is higher than the overall level of satisfaction reported by the beneficiaries of CHAFEA (74%) and comparable to what was reported by the beneficiaries of EACEA (89%) and ERCEA (89%). Almost all respondents to the surveys would like to work with the Agency again in the future (98% of beneficiaries, 99% of REA’s independent experts and around 90% of unsuccessful applicants).
The independent experts were highly satisfied with the overall processes of registration, selection and contracting and their level of satisfaction improved as compared to the previous 2015 experts’ survey. However, the experts remained less positive about the online evaluation system used, the quality of the templates used for reports, and the time available for drafting them.
In terms of proximity to the addressees, REA continued to serve as a direct ‘contact point’ between the applicants/beneficiaries of EU funding, as well as other stakeholders and the Commission during the evaluation period. Building on the results of the 2015 beneficiary satisfaction survey, REA made sustained efforts to boost awareness on new funding opportunities and opportunities for experts, to broaden the participants’ group, to consolidate a service-oriented communication, and to support parent DGs by giving visibility to EU research via success stories and providing input to the policy-feedback loop.
The survey of beneficiaries indicates a broad satisfaction with the way in which the Agency communicated with them during various phases of the project life-cycle. As many as 81% of the respondents to the beneficiaries’ survey strongly or rather agreed that REA staff assigned to their project were easily available and responsive during the implementation. Most favourable results were received in relation to REA’s availability and responsiveness during the grant amendment (82% strongly or rather agreed) and in the grant finalisation phases (85% strongly or rather agreed). About 74% of the respondents claimed that the feedback received on the scientific and technological progress in their project was useful. The areas with reported lower levels of satisfaction were related to the user-friendliness of the IT tools employed throughout the project management life-cycle. The Commission, which is leading the development of grant management IT tools at corporate level, is currently taking measures to address the beneficiaries’ lower satisfaction in this area.
Concerning the visibility of the EU as a promoter of the programmes entrusted to REA, the survey shows that a large majority of beneficiaries were aware that REA grants were funded from the EU budget (95%) and that the European Commission entrusted the grant management to REA (84%).
With regards to the lessons learned during the 2015-2018 period, the evaluation identified that the growing prevalence of actions with policy interest emerges as an area where REA and its parent DGs need to formalise the specific modalities and procedures for their implementation. Examples are the Coordination and Support Actions, including grants to identified beneficiary, which are managed by the Agency but are of particular interest for Commission policy work. Since it is important to ensure that these actions can respond to continuous developments in policy needs, a closer dialogue between REA and Commission services should be maintained.
5.2 Efficiency
This section considers the relationship between the resources used by the Agency and the output. The analysis, among other factors, also includes analysis of administrative and regulatory burden and looks at aspects of simplification.
During 2015-2017 REA proved to be an efficient and cost-effective structure for the management of the delegated programmes. Its administrative budget (excluding the share of costs of the central administrative and logistical support services) was below 3% of the operational budget based both on commitment and payment appropriations for the period 2015-2017, while generally the cost-effectiveness of REA grew over 2010-2017.
Ratio between REA’s administrative and operational budget
REA’s actual operational budget during 2014-2018 constituted EUR 9.109 billion, of which EUR 8.508 billion from EU budget and EUR 602 million related to EEA/EFTA and third countries’ contributions (not considered in REA’s SFS). REA’s administrative budget increased from EUR 39.88 million in 2015 to EUR 43.92 million in 2017 (in commitments, without the central support services).
A detailed analysis of the REA cost-efficiency and results of the Cost-benefit analysis are presented under the point 5.4.
The number of proposals managed by REA fluctuated at around 15 000 from 2015 to 2017. The number of total running projects decreased from 7 020 in 2015 to 6 420 in 2017. During the evaluation period, the actual workload of REA was higher than estimated in the 2013 CBA due to factors beyond the Agency’s control that influenced its workload, such as the average grant size and the number of proposals and grants. The actual number of new grants managed by REA during 2014-2018 was 32% higher than the 2013 CBA forecast (9 646 versus 7 306). However, the increase in the number of grants managed by REA (20%), mostly relates to the smallest grants – MSCA Individual Fellowships, which are characterised by the lowest workload level per grant (number of person days required to manage a typical project life-cycle). The increased number of new projects also added to the workload of monitoring project implementation.
Due to an increase in the number of operational staff during this period and the closure of many small FP7 projects (MSCA individual fellowships and grants), the number of running projects ‘per operational head’ decreased from 15.42 in 2014 to 11.32 in 2017. However, the ratio of budget ‘per operational head’ increased from EUR 3.46 million to EUR 3.54 million (in terms of commitments) or from EUR 2.44 million to EUR 2.99 million (in terms of payments) in 2014-2017.
Budget, staff and workload indicators, REA, 2014-2017, € million or %
|
|
2014
|
2015
|
2016
|
2017
|
|
Operational budget, commitments
|
1 553.51
|
1 521.74
|
1 666.36
|
2 009.15
|
|
Operational budget, payments
|
1 096.88
|
1 418.48
|
1 642.94
|
1 697.18
|
|
Administrative budget, commitments
|
51.24
|
54.19
|
59.74
|
66.87
|
|
Administrative budget, payments
|
49.68
|
54.10
|
59.23
|
66.12
|
|
Administrative budget without the central support services, commitments
|
33.39
|
39.88
|
42.42
|
43.92
|
|
Actual number of staff (at the end of the year)
|
548
|
608
|
628
|
690
|
|
Actual number of operational staff (without the central support services)
|
449
|
503
|
516
|
567
|
|
Programme management cost at REA (excluding the central support services), commitments
|
2.15%
|
2.62%
|
2.55%
|
2.19%
|
|
Programme management cost at REA (excluding the central support services), payments
|
3.04%
|
2.81%
|
2.58%
|
2.59%
|
|
Budget ‘per operational head’, commitments
|
3.46
|
3.03
|
3.23
|
3.54
|
|
Budget ‘per operational head’, payments
|
2.44
|
2.82
|
3.18
|
2.99
|
|
Proposals received
|
11 473
|
15 415
|
13 005
|
14 927
|
|
Total running projects
|
6 925
|
7 020
|
6 658
|
6 420
|
|
Running projects ‘per operational head’
|
15.42
|
13.96
|
12.90
|
11.32
|
REA took a series of actions to cope with the varying levels of workload across its different units. The Agency introduced an annual workload measurement, which helped to re-balance its staffing levels across all of its units and the appropriate ratio of administrative and operational staff. REA was also effective in filling vacancies and maintaining a relatively low vacancy rate (2%) between 2015 and 2018. In accordance with the multi-annual planning provided in the SFS, by the end of 2017 the number of REA staff had risen to 690 (from 608 in 2015), of which 123 are dedicated for the support services.
The staff opinion survey carried out in 2016 revealed that REA’s key strengths were related to overall job satisfaction and staff’s satisfaction with the Agency as a modern and attractive workplace. In addition, REA achieved positive results with respect to the willingness of its staff to give extra effort when required, the level of understanding of the Agency’s purpose as well as job clarity.
At the same time, REA staff perceived less positively their career development and mobility opportunities, work recognition as well as equal opportunities and internal communication. Those results were however still higher than or equal to the levels of agreement for the Commission and other executive agencies.
The Agency has stepped up its HR management by the adoption of its 2016 HR Action Plan to follow up these staff opinion survey results, and through a new comprehensive multi-annual HR strategy. The evolution of key indicators for staff satisfaction – after the evaluated period – is encouraging: In the latest staff opinion survey (4th quarter of 2018), REA managed to increase its result for staff feeling that the Agency cares about their well-being to 63%, up from 49% in 2016. Staff engagement rose to 69% (from 67% in 2016).
Overall, the size and structure of REA, and the level of resources in each department, were appropriate to its mandate and delegated tasks during the evaluation period. The organisational structure of REA was based on separate organisational units for different programmes and actions grouped into Department A ‘Excellent Science’
and Department B ‘Industrial Leadership and Societal Challenges’
according to the respective pillars of the Horizon 2020. The units, responsible for the Agency’s horizontal tasks (Administration and Finance) and central support services (Participant Validation and Expert Management and Support unit) were organised into the Administration, Finance and Support Services Department. REA’s organisational structure was reviewed and effectively aligned to cope with changes, such as the evolution of the REA mandate.
During 2015-2018, REA, in cooperation with the Commission, continued the optimisation of its procedures and programme management functions and introduced a number of simplifications. The improvements and simplifications concerned wider use of remote evaluation of proposals, remodelling of the evaluation building, improved procedures for allocating proposals to the most suitable experts, measures to automate and improve the detection of possible conflict of interest for experts, electronic workflows and wider use of IT tools, etc. REA’s internal staff networks played an important role in the optimisation and simplification process. They assumed a key role in the development of improvements to numerous REA operations such as management of calls (from call publication to grant signature) and grants in Horizon 2020. In addition, they contributed to the development of guidance, the sharing of good practices and to ensuring a harmonised implementation of the Horizon 2020 control system provided by the Common Support Centre.
5.3 Coherence
Coherence looks at any overlaps and complementarities within the programme portfolio managed by the Agency or delimitation of responsibilities between the Agency and its parent DGs.
The evaluation did not find any evidence of overlaps, duplications, gaps or inconsistencies within the REA programme portfolio or the support services managed by REA. In the 2015-2018 period, REA demonstrated flexibility in implementing a diverse thematic programming portfolio and being answerable to six different parent DGs. There is a strong coherence between the FP7 and Horizon 2020 programmes delegated to REA, because of strong continuity in a number of programmes (for example, implementation of MSCA under the Excellent Science pillar in both framework programmes and most activities within FP7 Security and the successor Secure Societies programme in Horizon 2020).
Another factor of coherence derives from the fact that REA is responsible for implementing several different parts of a specific programme (Horizon 2020 and legacy of FP7) for multiple DGs and across a broad thematic spectrum. Accordingly, REA’s responsibilities are broadly coherent in that they are all part of a single programme, with common rules and procedures. Moreover, the business processes, as designed by the CSC, are harmonised for different sub-programmes, which means that the overall grant management process is coherent, even if the thematic areas of intervention differ (with some specificities within the MSCA).
The extension of REA’s mandate to provide administrative and logistical support services to all entities involved in Horizon 2020 management and also to a wider range of Commission DGs has strengthened coherence, not least due to the introduction of a common approach across Horizon 2020. Overall, these developments indicate that REA is increasingly becoming a central provider of support services in the Commission.
The Memorandum of Understanding of 2016 and the additional working arrangements of 2018 reflecting REA’s extended mandate provide for clear and appropriate delimitation of responsibilities and tasks between REA and its parent DGs.
REA has continued to generate and retain know-how and knowledge, capitalising on the fact that it has been managing some of the programmes for a decade. While programme implementation know-how is embedded within the Agency itself, significant progress has been made in strengthening REA’s input to policymaking, and in addressing some of the weaknesses identified in the 2012-2015 period of operations, through the adoption of strategic and operational measures. For instance, REA organised thematic project cluster meetings and set up the AGILE Rapid Reaction Network, participated in coordination meetings and in thematic events organised by the Commission, contributed to the preparation of policy reports, provided important input for the Commission’s communication and dissemination events, etc.
However, policy feedback is still a domain where further improvements are considered necessary, with actions required from both the Commission and the REA side to strengthen the efficiency and effectiveness of this function. REA’s parent DGs need to formulate better their policy needs as these differ from one DG to another, due to the nature of ‘top-down’ or ‘bottom-up’ research programmes for which they are responsible. Parent DGs should also specify in which form, and at which frequency, they would prefer policy inputs to be produced by REA to allow it for a better customised policy feedback. The Catalogue of Policy Options developed by REA in 2018 serves as a basis for establishing specific policy feedback mechanisms and business processes most relevant to parent DGs. Currently the CIC is leading the efforts of developing procedures at the R&I family level for feedback onto policymaking as part of the strategy for an effective dissemination and exploitation of research results in Horizon 2020.
5.4 Retrospective Cost-Benefit Analysis
The retrospective Cost–Benefit Analysis for the 2015-2018 period was carried out based on the assumptions laid down in the latest version of the Specific Financial Statements (SFS) and the actual costs of REA. Since REA’s mandate was substantially changed compared to the 2013 ex ante CBA, any comparison of the ex ante CBA estimates with the SFS estimates and the actual figures must be considered carefully, given the change in the tasks and workload level. REA’s SFS was updated in 2017 following the update of the REA’s mandate, linked primarily to delegation of classified projects from Societal Challenge 7 to REA and the expansion of centralised services for registration and validation of all beneficiaries and service providers (SEDIA). Consequently, the retrospective CBA is based on the latest SFS version of April 2017.
The REA SFS estimated the savings resulting from the delegation of tasks to REA at around EUR 178 million over the whole period 2014-2024 when compared to the management by the Commission services (in-house scenario), while for the period of 2015-2018 these were estimated at around EUR 80 million.
The retrospective CBA revealed that the executive agency scenario actual costs in 2015-2018 were much lower than the estimated costs of the in-house scenario and constituted savings of EUR 105 million (or 29% of the estimated costs under the in-house scenario).
Estimated costs and savings of the executive agency scenario in 2015-2018, EUR
As forecasted in the SFS, savings of the executive agency scenario primarily resulted from a higher share of lower-cost contract staff (CAs) employed within the Agency and the lower overall number of staff.
Significant cost savings occurred in REA’s Title II ‘Infrastructure and operating expenditure’. The costs in Title I ‘Staff-related expenditure’ were higher than estimated in the SFS as the latter did not take into account increases to staff costs due to salary indexation, promotions and increasing staff seniority.
Finally, it should be noted that the assumptions laid down in the SFS forecasted a workload level significantly lower than the actual one, across different programmes. Although many parameters influencing the workload (operational budget of delegated Horizon 2020 programmes and actions, average grant size and corresponding number of grants) were beyond the control of the Agency, proper organisation and management helped allocate human resources across REA in the most appropriate way and avoid shortcomings.
6.Conclusions
The results of the evaluation of REA operations during 2015-2018 period confirm that the delegation of programme implementation to REA remained considerably more cost effective than the in-house scenario. During this period REA continued to implement the delegated programmes in an effective, efficient and coherent manner, generating important cost-savings to the EU budget and high value for money. The ex-post cost benefit analysis found that the actual savings of the executive agency scenario were EUR 105 million during the period 2015-2018 compared with the in-house scenario and higher than the EUR 80 million estimated in the SFS of 2013.
During this period REA was effective in achieving its objectives, and its beneficiaries were highly satisfied with the performance of the Agency. REA consistently achieved its key performance indicators. Beneficiaries’ satisfaction with the REA performance increased to 86% compared with the previous evaluation (82% in 2015). The ratio of budget ‘per operational head’ increased from EUR 3.46 million to EUR 3.54 million (in terms of commitments) or from EUR 2.44 million to EUR 2.99 million (in terms of payments) in 2014-2017. The Agency’s administrative budget was below 3% of the operational budget based both on commitment and payment appropriations between 2015 and 2017 (excluding the costs of the administrative and logistical support services). REA delivered high quality and effective administrative and logistical support services to its clients and other stakeholders, achieving and exceeding the KPIs set.
The coordination and communication mechanisms that are in place facilitate information flows between REA and the Commission services, in particular with regard to the content of projects supported and their results. No evidence of overlaps, duplication, gaps and inconsistencies within the programme portfolio and support services managed by REA were identified; on the contrary, the extension of REA’s remit to include the provision of common administrative and logistical support services for the research and innovation family has strengthened coherence, particularly through the introduction of a common approach across Horizon 2020.
The current evaluation identifies only three areas where further improvements are necessary from both the Commission and the REA side and which aim to further streamline or define the working arrangements between REA and the Commission. These relate to:
-Increasing awareness on business processes related to the selection of independent experts for evaluations, validation of expert lists, as well as the participation of Commission staff in project monitoring activities;
-formalising modalities and procedures for the implementation of actions with policy interest;
-establishing a set of specific policy feedback mechanisms & business processes most relevant to parent DGs.
The parent DGs and the Agency are preparing an action plan to address the shortcomings identified in this evaluation. This action plan aims to further improve the Agency’s performance, as well as the interaction and collaboration between Commission services and REA and to contribute to an efficient implementation of the delegated programmes.
ANNEX: EVALUATION QUESTIONS
Effectiveness
-To what extent has REA been operating according to the legal framework establishing it?
-To what extent has REA achieved its objectives with special focus to (a) the implementation of the delegated programmes Horizon 2020 and FP7, (b) the implementation of the support services and (c) the implementation of the internal control principles
, notably sound financial and human resource management? What, if anything, could be done to render REA more effective in achieving these objectives?
-To what extent and how has REA contributed to an improved management and visibility of the delegated programmes and better services and satisfaction to the stakeholders and addressees
in terms of the elements assessed in the 2013 CBA and as compared to the alternative options mentioned in that CBA?
-To what extent does REA’s communication function support the mission of the Agency and does REA ensure an effective feedback loop with the policy making DGs?
-To what extent has REA contributed to improved management of the programmes in terms of:
−Proximity to addressees (e.g. role played by the Agency as focal point for applicants and beneficiaries of the programmes; instruments and mechanisms put in place for communicating with the applicants beneficiaries and experts);
−Effective implementation of the programmes, taking into account the interests of the addressees and those of the EU:
ürate of execution of commitment appropriations;
ürate of execution of payment appropriations;
ütime to grant;
ünet time to pay;
üresidual multi-annual error rate identified at ex-post control.
−Visibility of the EU as promoter of the programmes entrusted to the Agency (e.g. compliance with the Commission’s guidelines on information and visibility of programmes
, instruments put in place to ensure the visibility of the EU as promoter of the programmes).
Efficiency?
-To what extent have the actual costs (including cost of coordination and monitoring) of REA corresponded to the estimates of the 2013 CBA? If not, what are the reasons behind?
-To what extent have the actual benefits corresponded to the estimates of the 2013 CBA? If not, what are the reasons behind?
-To what extent has the management and execution of the programmes by REA been cost-effective as compared to the alternative options
?
-To what extent has REA contributed to improved management of the programmes in terms of:
−Simplification of the procedures and flexibility in the implementation of delegated tasks (e.g. capacity to adapt to periods of high workload)?
−Which further scope for simplification exists?
-Which aspects/means/actors or processes render REA more or less efficient? What could be improved? What is the quality of the services/advice provided by REA to stakeholders and addressees?
-To what extent has the establishment of REA resulted in savings to the EU budget as compared to the alternative options (e.g. difference in costs between the Commission option and the Agency option)?
-To what extent have REA’s internal organisation and procedures been conducive to its efficiency?
-To what extent is REA’s internal organisation capable and flexible to rapidly respond to resource needs due to uncertainties related to volumes of work?
-To what extent does REA’s human resources management contribute to the achievement of the Agency’s objectives?
−Is the size and structure of the organisation appropriate?
−Is the balance between operational staff and administrative support staff appropriate?
−Are the staff turnover and vacancy rate well managed?
−How are provisions on overtime and teleworking implemented in the Agency?
−How does the Agency follow up on the findings of the latest staff survey (2016)?
Coherence
-To what extent have there been overlaps/ gaps/ inconsistencies/ complementarities within the programme portfolio and support services managed by REA and how are these addressed?
-Is there a clear and appropriate delimitation of responsibilities and tasks between REA and the Parent DGs? Are there overlaps or gaps? Are the different responsibilities adequately communicated to the beneficiaries?
-To what extent has REA enabled the Commission to better focus on its policy related tasks?
−Are there any governance (financial and policy) issues in relation to the implementation of the tasks that are delegated to the executive agencies/ issues as regards the Commission’s services being able to steer EU policy or budget implementation?
−Are appropriate mechanisms and instruments in place, and at which level, to ensure an adequate coordination and information flow between REA and the Commission services, notably on the content of the projects supported and their results?
−Does REA provide useful information on the implementation of the delegated programmes and their progress (in terms of management and content) in support of the policy process (e.g. information required for the annual Management Plan of the Parent DGs)?
-To what extent have the activities of REA resulted in unintended effects (both desirable and undesirable)?
-To what extent has the Commission, in the presence of REA, been able to maintain an adequate level of know-how in relation to the programmes entrusted to the Agency? How has this been achieved? What are the feedback channels, means and methods used for this purpose? What are areas for improvement, if any?
-Have the monitoring, reporting and supervision arrangements in place enabled the Commission to benefit, in the short and medium term, from the know-how created within REA?
-How effective is the flow of information and communication between REA and the Commission services (in particular Parent DGs)?
-To what extent would the closing down of REA result in losing significant know-how in relation to the management of the programmes entrusted to REA?
