EUROPEAN COMMISSION
Brussels, 29.4.2020
SWD(2020) 73 final
COMMISSION STAFF WORKING DOCUMENT
Evaluation of the Innovation and Networks Executive Agency (INEA)
Accompanying the document
Report from the Commission to the European Parliament, the Council and the Court of Auditors
Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Executive Agency for Small and Medium-Size Enterprises, the Innovation and Networks Executive Agency, the Education, Audiovisual and Culture Executive Agency, the Research Executive Agency and the European Research Council Executive Agency
{COM(2020) 184 final} - {SWD(2020) 74 final} - {SWD(2020) 75 final} - {SWD(2020) 76 final} - {SWD(2020) 77 final} - {SWD(2020) 78 final}
TABLE OF CONTENTS
1.
Introduction
2.
Background
2.1 Description of the intervention and its objectives
2.2 Baseline and points of comparison
3.
Implementation/State of play
4.
Method
4.1 Short description of methodology
4.2 Limitations and robustness of findings
5.
Analysis and answers to the evaluation questions
5.1
Effectiveness
5.2
Efficiency
5.3
Coherence
5.4
Retrospective Cost-Benefit Analysis
6.
Conclusions
7.
Annex: Evaluation questions
1.Introduction
The purpose of this periodical (every three years) evaluation of the operation of the Innovation and Networks Executive Agency (INEA) is to assess INEA’s implementation of the parts of the EU funding programmes (Connecting Europe Facility, Horizon 2020 and the Trans-European Transport Network (TEN-T) and Marco Polo legacy programmes) which have been entrusted to it.
INEA is governed by: (1) the
Framework Regulation for executive agencies
; (2) the INEA
‘Act of Establishment’
,
which creates the Agency and sets out its mandate; (3) the
INEA Act
of Delegation
which specifies the tasks to be carried out by, and the powers delegated to, the Agency in order to perform its mandate; and (4) the decision establishing the
INEA Steering Committee
.
In line with the Commission’s better regulation principles
, the evaluation applies several standard evaluation criteria. The evaluation assesses whether the Agency has fulfilled its tasks in an effective and efficient way, whether there are overlaps / gaps / inconsistencies in the management of the programme portfolio by the Agency, and whether there is a clear delineation of tasks between INEA and the parent DGs or other executive agencies (coherence)
.
The evaluation also assesses whether the functioning of the Agency has yielded the expected positive results as estimated in the Cost-Benefit Analysis for delegating task to the executive Agency
and identify potential areas of improvement. To this end, the estimations of the Cost-Benefit Analysis of 2013 have been tested to provide evidence on the validity of the assumptions made in the ex-ante scenario by considering the actual costs and benefits of programme implementation by the Executive Agency in a structured way. The aspects to be covered by the Cost-Benefit Analysis are specified in Article 3(1) of the Framework Regulation and the Guidelines on establishing and operating executive agencies.
The evaluation does not cover the achievements of the programmes managed by INEA, which are subject to mid-term and ex-post evaluations themselves. The evaluation of INEA nevertheless provides useful input for these programme evaluations, considering that the performance of the Agency affects the efficiency and effectiveness of the programmes it manages.
The evaluation examines the efficient use of resources and the effective achievement of the tasks entrusted to the Agency. It looks in particular at whether the alignment of more coherent programme portfolios with the Agency’s core competences and its brand identity delivered the estimated qualitative benefits; whether the assembly of the management of different EU programmes delivered the estimated synergies, simplification and economies of scale; whether the pooling of instruments guaranteed consistent service delivery and whether there is scope for simplification and further efficiency gains.
The evaluation covers all the tasks carried out by the Agency during the period from January 2014 to December 2016. The evaluation is supported by a study carried out by an external contractor.
The results of this study are summarised hereafter and this evaluation will be presented to the European Parliament, to the Council and to the Court of Auditors in accordance with Article 25(1) of the Framework Regulation.
The results will feed into the reflection to assess the opportunity of expanding/ modifying the Agency’s mandate in view of the delegation of the implementation of 2021-2027 EU programmes.
2.Background
2.1 Description of the intervention and its objectives
The outsourcing of certain management tasks to the Agency, as outlined in the Framework Regulation and in INEA’s Act of Establishment, intends to:
-Allow the Commission to focus on its institutional tasks, i.e. tasks assigned to the institutions by the Treaty which require discretionary powers in translating political choices into action. Such institutional tasks should not be outsourced.
-Enable the Commission to achieve the objectives of the delegated EU programmes more effectively. According to the Cost-Benefit Analysis carried out in 2013, the delegation of certain programme tasks to the Agency was estimated to be more cost-efficient than an in-house scenario. The Act of Establishment stated that the alignment of more coherent programme portfolios with the Agency’s core competences and its brand identity would bring qualitative benefits. In addition, it projected that assembling the management of different EU programmes would bring synergies, simplification and economies of scale.
The original objective of the intervention is entrusting the Agency with the implementation of several EU programmes in part or fully:
-Connecting Europe Facility: transport, energy and telecommunications,
-Horizon 2020: transport and energy research (Part III Societal challenges of the Specific Programme),
-Legacy of the Trans-European Transport Network (TEN-T) programme,
-Legacy of the Marco Polo II programme (transferred from the Executive Agency for Competitiveness and Innovation (EACI), now Executive Agency for Small and Medium-Sized Enterprises (EASME).
As of 1 May 2018, the mandate of the Agency was extended to include implementation of the new Wifi4EU initiative as part of CEF Telecoms.
The Agency works under the supervision of four parent Directorates-General:
-DG Mobility and Transport (MOVE) – Lead parent DG;
-DG Energy (ENER);
-DG Communications Networks, Content and Technology (CNECT); and
-DG Research and Innovation (RTD).
In implementing the programmes delegated to it, INEA is responsible for monitoring the projects, making the necessary checks and recovery procedures, and performing budget implementation tasks covering revenue and expenditure within the meaning of the EU Financial Regulation, and in particular:
-Manage the operations and procedures leading to the adoption of Commission award decisions and to the conclusion of grant agreements and manage the ensuing decisions and agreements;
-Provide support in programme implementation;
-Perform all the operations required to launch contests and award prizes in accordance with the EU Financial Regulation;
-Conclude public procurement procedures and manages the ensuing contracts, including the operations required to launch and conclude public procurement procedures.
2.2 Baseline and points of comparison
The current evaluation of INEA operations during 2014-2016 assesses the actual costs and benefits of programme implementation by INEA (executive agency scenario) when compared with the alternative scenario of management by the Commission services (in-house scenario).
Accordingly, the reference point for the present INEA evaluation is the 2013 ex ante Cost-benefit analysis, and the Specific Financial Statement (SFS) of INEA.
Wherever possible, the analysis of the INEA performance during the reference period assesses the progress achieved since the two previous evaluations (of the precedent TEN-T EA), covering the Agency’s operations during 2008-2011 and 2011-2013.
It should be noted that over the period 2008-2011 the savings resulting from the delegation of tasks to TEN-T EA have been evaluated at around EUR 8.6 million compared to the in-house scenario. The estimated efficiency gains over the period 2011-2013 were of EUR 8.8 million and of EUR 24.4 million over the period 2014-2016.
According to the SFS of INEA prepared in 2013 the estimated efficiency gains resulting from programme delegation to INEA were of EUR 22.8 million over the period 2014-2016. The total number of full time equivalents (FTEs) required to manage the relevant programmes in year 2020, the peak programming year in terms of workload, was estimated at 318 for INEA. The total initial operational budget entrusted to INEA in 2020 was estimated at around EUR 37 billion in both commitment and payment appropriations. Considerable efficiency gains were expected as compared to the in-house scenario over the period, along with non-quantifiable benefits such as improved quality of programme management and service delivery, improved visibility of the EU programmes and proximity to beneficiaries.
3.Implementation/State of play
In 2006, by Commission Decision 2007/60/EC
, the Commission established the Trans-European Transport Network Executive Agency (TEN-T EA) and entrusted it with the management of Community actions in the field of the trans-European transport network. The initial duration of the Agency was limited to 31 December 2008. Then the main task of the Agency was to manage the remaining open TEN-T projects and financing decisions under the Multi-annual Financial Framework (MFF) 2000 – 2006.
Subsequently, in 2008, by Decision 2008/593/EC
, the Commission extended the period of operation of the TEN-T EA to 31 December 2015 and redefined its objectives and tasks so that it became responsible also for the implementation of financial aid from the trans-European transport network budget under the MFF 2007-2013. Both the establishment and extension of the TEN-T EA were preceded by dedicated cost-benefit analyses.
TEN-T EA managed a TEN-T programme budget of EUR 0.8 billion during the 2000-2006 MFF and EUR 7.8 billion during the 2007-2013 programming period. In the latter period the Agency’s project portfolio consisted of 411 projects and by end 2013 its staff was around 100.
The Innovation and Networks Executive Agency (INEA) was set up on 23 December 2013 and is an autonomous legal entity since 1 January 2014. It is responsible for the management of parts of the Connecting Europe Facility (CEF) and of Horizon 2020 as well as of the TEN-T and the Marco Polo legacy programmes.
The new mandate also brought changes in the governance of the Agency as three new parent DGs joined the Steering Committee. As stipulated in the Act of Delegation, a Memorandum of Understanding was signed on 1 October 2014, which defines the modalities and procedures of interaction between INEA and its parent DGs
. In addition, it brought a clearer delimitation of the administrative activities between the Agency and its parent DGs. The present evaluation assesses whether this distribution of roles has been complied with for the 2014-2016 period, as well as the effects of the intervention on the coherence and the efficiency of INEA’s action.
The total operational budget earmarked for INEA for the MFF 2014-2020 is EUR 33.8 billion, of which EUR 28.5 billion is accounted for by the CEF and EUR 5.3 billion by Horizon 2020. This means that INEA handles the largest budget of all the executive agencies in the current financial period of 2014-2020.
The operational budget managed by INEA in 2014-2016 was approximately EUR 11.9 billion in terms of commitments (EUR 3.6 billion in 2014, EUR 3.2 billion in 2015 and EUR 5.1 billion in 2016) and EUR 5.1 billion in terms of payments (EUR 0.6 billion in 2014, EUR 2 billion in 2015 and EUR 2.5 billion in 2016). The Agency’s administrative budget evolved from EUR 13.1 million in 2014 to EUR 18.2 million in 2015 and EUR 21.5 million in 2016.
The total budget per head in terms of payments increased from EUR 3.97 million in 2014 to 10.75 million in 2015 and to EUR 11.11 million in 2016. The increasing volume of INEA’s operational budget during this period outpaced the increase in the number of the Agency’s staff.
The number of proposals received and evaluated in 2014-2016 was in total 3 490, i.e. very close to the CBA estimations. The number of proposals managed per head varied between 3.1 in 2014, 8.3 in 2015 and 6.5 in 2016, which results mainly from the heavy frontloading of the CEF Transport budget (2014 and 2015 calls, evaluated respectively in 2015 and 2016).
The number of INEA running projects increased from 496 in 2014 to 1026 in 2015 and 1383 in 2016. While the number of projects in 2014 was below the CBA estimations (which related to the later than initially forecasted launch of the 2014 calls and later conclusion of grant agreements), in 2015-2016 the actual number of running projects generally reached the CBA estimations. Programme management costs (defined as the ratio between the administrative and operational budget in payments) decreased from roughly 2% in 2014 to 0.9% in 2015 and 0.9% in 2016.
A series of organisational and procedural changes were introduced to render more efficient the implementation of the delegated programmes under the MFF 2014-2020, notably in the evaluation period:
-Further simplification and improvement of grant management processes and procedures, notably concerning the widespread use of the TEN-Tec portal functionalities concerning submission and evaluation of proposals, preparation and signature of Grant Agreements and reporting (in 2014, the Agency merged the submission of the Strategic Action Plan (SAP) with the Action Status Report (ASR) – previously two separate documents);
-The adoption of new business processes, Horizon 2020 IT tools and modes of service delivery;
-The adoption in April 2017 of a multi-annual human resource (HR) strategy, covering the years 2016 – 2020, including an action plan for following up on the key findings from the 2016 staff opinion survey. Based on this strategy, the Agency is implementing specific measures related to staff engagement, training, internal communication, career development opportunities, well-being and the balance between work and private life, aimed at improving HR management and increasing the effectiveness of the Agency’s operations. A staff retention policy has since been approved in September 2018 to increase staff satisfaction levels and help reducing staff turnover.
4.Method
4.1 Short description of methodology
The evaluation was supported by a study carried out by an external contractor. The methodology of the INEA evaluation was consistent with the approaches employed for similar parallel evaluations of EACEA, EASME, ERCEA, CHAFEA and REA. The study was structured around a series of evaluation questions available in Annex.
A combination of evaluation methods providing for the collection of qualitative and quantitative information and evidence included:
-Documentary review and desk research concerning all relevant documentation relating to INEA, including the legal basis of the executive agencies and of INEA, the Memorandum of Understanding between the parent DGs and INEA (MoU), Annual Work Programmes, Annual Activity Reports, European Court of Auditors (ECA) and Internal Audit Service (IAS) reports and related documents, previous evaluations and CBA studies, financial documents, and action plans; the evaluation relied also on the results of the 2016 staff opinion survey;
-Semi-structured or in-depth interviews with EU staff and stakeholders of the delegated programmes. 59 interviews were carried out with: staff of the parent DGs, of other DGs represented in the INEA Steering Committee, and of INEA; project applicants and beneficiaries; external experts contracted to carry out the external phase of the CEF and Horizon 2020 evaluations;
-Survey addressed to stakeholders, experts and Member States’ officials involved in the implementation of programmes managed by INEA. 1878 responses were received: 1404 from project applicants and beneficiaries; 382 from external experts; 79 from programme committee members; and 13 from members of INEA’s CEF Transport Advisory Group;
-A retrospective cost–benefit analysis;
-A comparative analysis and benchmarking of the various executive agencies was also carried out against a set of (qualitative and quantitative) indicators. This enabled the evaluation team to compare the performance of INEA effectively with that of other executive agencies.
Case studies were used to complement the desk research and interview / survey programme and to provide insights into the performance of the Agency on issues such as coherence and maintenance of the know-how within the Commission, efficiency of simplification measures etc. Those case studies have fed the different assessment questions.
4.2 Limitations and robustness of findings
A representative sample of Commission and INEA staff and stakeholders (both applicants of rejected proposals and beneficiaries) and other officials / experts (external experts, programme committee members and advisers) were consulted in the course of the evaluation. Compared to the overall population reached, unsuccessful applicants participated less actively in the survey than programme beneficiaries.
No sampling bias was observed as the profile of the respondents to the surveys was very similar to the overall population, guaranteeing statistical representativeness. The non-response bias (not all characteristics of the group that did not reply had been captured in full) was mitigated through triangulation with the results of follow up interviews. The triangulation approach, using multi-level and multi-stakeholder dimension in the data collection, ensured the robustness and reliability of the data and information used to draw up conclusions in the supporting study.
A mix of parent DGs and INEA managers and policy / project officers were interviewed extensively. Some answers to evaluation questions, notably in the ‘Coherence’ section, rely largely on interview and survey data. Some views from Commission and Agency staff gathered from the interviews and mentioned in the study illustrate personal experience, although they may not be representative of overall relations between the Commission and the Agency.
Also, it should be noted that, in parallel to this evaluation, the European Court of Auditors (ECA) carried out a performance audit to ascertain if INEA’s management of the EU delegated programmes optimise their implementation. The audit focused on the 2014-2020 programming period and covered INEA’s set-up and organisation, and its implementation of the CEF programme. It involved inter alia the examination of the regulatory framework, the responsibilities and the control systems in place as well as INEA’s performance in the management of the CEF programme and projects. In its report, the ECA inferred that INEA is a well-organised Agency and that it is implementing the programmes delegated to it, in particular the CEF, efficiently. However, the report also made a number of recommendations to enhance the implementation of the Connecting Europe Facility and the overall operational performance of the Agency, such as: a multi-annual programming of calls for proposals; increased synergies between CEF and Horizon 2020 (Horizon Europe); a performance framework to measure programme and project achievements; an upgraded and harmonised procedure for the evaluation and selection of project proposals; to enhance the link between funding and the achievement of project milestones; and to ensure more flexibility in the allocation of staff among programmes within the Agency. The Commission takes the view that the European Parliament and Council common understanding on the new CEF Regulation is consistent with the ECA’s recommendations. The Commission and INEA are already addressing the ECA’s recommendations with a view to improving programme management under the present MFF and in view of the CEF 2 programme.
5.Analysis and answers to the evaluation questions
5.1 Effectiveness
For the purpose of this evaluation, effectiveness relates to how successful the Agency has been in achieving or progressing towards its objectives.
The evaluation inferred that during 2014-2016 INEA operated according to the legal framework that established it. INEA’s activities, as planned in the annual work programmes and reported in the annual activity reports, corresponded to the tasks set out in the Commission’s establishment and delegation Decisions.
As provided in the delegation Decision, the modalities and procedures of interaction between the Agency and its parent DGs are set out in the Memorandum of Understanding (MoU) of 1 October 2014. The MoU provides for a supervision strategy aimed at avoiding gaps or duplication of efforts resulting from crossover between the policy-making, monitoring and supervision tasks of the parent DGs and the execution tasks of the Agency. These provisions have generally worked well and all parent DGs and INEA appreciate the effectiveness of the collaboration.
In the evaluation period, INEA continuously improved its operations and achieved a high level of overall effectiveness. The Agency was flexible and effective in addressing the key challenges that occurred during the evaluation period, in particular:
-the continuous phasing-out of its two legacy programmes TEN-T and Marco Polo (the latter transferred from EACI, now EASME);
-the evaluation and grant award process for the CEF Transport calls for proposals, especially the first and largest ever CEF Transport call in 2014/2015 involving the evaluation of almost 700 proposals for a final selection decision for funding of EUR 13.1 billion;
-the gradual assumption of responsibilities on the implementation of the CEF Energy and Telecoms programmes and the Horizon 2020 sub-programmes concerning transport and energy;
-the launch of a CEF synergies (transport and energy) call in 2016; and
-the preparation as of 2016 of an innovative CEF Transport Blending call for EUR 1,35 billion, launched in 2017.
INEA has developed strong internal control standards, including on the management of financial and human resources. The Agency established overarching control strategy with a large number of control and reporting mechanisms that allow progress against objectives to be closely monitored and risks to its operations to be prevented and mitigated in a timely and effective way. INEA has in particular established a coherent anti-fraud strategy which it pursues effectively. This is reflected in assessments undertaken by the Court of Auditors and by the Commission’s Internal Audit Service.
In 2014-2016 INEA achieved almost all its targets in terms of key performance indicators (KPI). Only the target for the net time-to-pay for the TEN-T programme 2007-2013 was slightly exceeded in 2015 by 1 day (31 days instead of 30). The prompt conclusion of grant agreements (measured in ‘Time-To-Grant’ – TTG) varied, for Horizon 2020, between a minimum of 216 days in 2015 and a maximum of 225 days in 2014 for Energy, and between 232 days in 2015 and 237 days in 2014 for Transport (still below the Horizon 2020 target of 245 days). Concerning the CEF programme, the longest sectorial TTG in 2015-2016 (no grants were awarded and signed in 2014) were 264 days for CEF Transport, 249 days for CEF Energy and 253 days for CEF Telecoms, all below the ceiling of 276 days. The share of grants concluded within TTG targets, for both Horizon 2020 and CEF, reached nearly 100% for the 2014-2016 calls.
Regarding payments of grants, the average Time-to-Pay (TTP) stood well below the contractual thresholds for all types of payments (pre-financing, interim and final payments) in 2014-2016 both for Horizon 2020 and CEF. Concerning the share of payments within contractual limits, nearly 100% of all pre-financing, interim and final payments in 2014-2016 were executed on time.
INEA achieved full execution of its operational budget during 2014-2016, both in commitment and payment appropriations; with the exception of the execution of commitment appropriations for the TEN-T
and Marco Polo II programmes in 2014, both of which were well below the target of 100% , although this target was subsequently met in 2015 and 2016.
With regard to the legality and regularity of transactions, the estimated residual multi-annual error rate remained well below the threshold of 2% for the TEN-T and Marco Polo II programmes in 2014-2016. Concerning CEF and Horizon 2020, this KPI was not yet applicable and there were not enough audits in this period to draw any conclusions; however, it is estimated that, by the end of the programme implementation, the error rate will be of 2% cumulative for CEF while for Horizon 2020 it is established within the range of 2% - 5%.
Summary of the KPIs over the period 2014-16
Source: INEA activity reports 2014, 2015, 2016
Despite some shortcomings in call coordination between the parent DGs concerning the evaluation period, INEA adapted well to peak periods of very high work-load. Such heavy work-load was especially felt in 2015/2016 on the occasion of the 2014 and 2015 CEF Transport calls for proposals, when the Agency was not yet fully staffed. Regular meetings between the parent DGs and INEA resulted overtime in more efficient planning and coordination of the calls, though continuous efforts on call coordination are necessary in the run up to the last calls of the present MFF.
While providing recommendations for addressing minor shortcomings, the reports of the independent observers accompanying the evaluations, for both CEF and Horizon 2020, were consistently complimentary on the transparency and efficiency of the evaluation process. Concerning Horizon 2020, INEA’s efforts to improve the quality of the evaluation process were reflected in a very small share of evaluation review/redress cases filed and (fully or partially) upheld compared to the number of proposals evaluated.
The survey of INEA’s applicants and beneficiaries indicates satisfaction in their dealings with the Agency, with a great majority of them considering that INEA has contributed to a significant improvement of the management of the delegated programmes and helped to develop better services. 90% of respondents stated that INEA managed the procedures (applications, evaluation, contract negotiations, project monitoring, etc.) in relation to their project very or quite efficiently. Only 8% of respondents indicated that during the application process INEA was unresponsive to the queries or problems applicants raised and only 9% indicated that the decision on the application was not taken within the acceptable timeframe.
It should be noted however, that applicants’ recommendations included a number of areas where improvement may be necessary, for example:
-establishing a “hotline” for National Contact Points/Committee members to obtain instant information and advice on application procedures;
-improving the feedback to applicants on the rejected proposals;
-organising local information meetings on calls for proposals and other implementation matters and making them accessible online;
-more flexibility in reporting on project progress, both regarding how to submit and save documentation (paper vs digital) and more use of electronic signatures;
-improving the participant portal and more workshops to update on progress of the programmes; and
-guidance on how to better manage project risks.
Overall, external experts showed high levels of satisfaction with INEA’s selection and contracting process, whether they worked on Horizon 2020, on CEF, or both. 58.8% of all experts were very satisfied and an additional 32.4% were quite satisfied. Overall less than 10% were quite or very dissatisfied, or had no response. Experts working on Horizon 2020 alone had the highest level of satisfaction, with only 4.1% dissatisfied in any way (an additional 4.5% did not respond or were unable to comment). By contrast, 9.4% of those working on both CEF and Horizon 2020 were dissatisfied, as were 11.8% of those working on CEF alone. Experts working on both projects were also least likely to be very satisfied (46.9%) compared to those working on Horizon 2020 (61.5%) or CEF (60.3%) alone.
Experts overwhelmingly stated that evaluation / review consensus meetings are run efficiently by INEA. While only 4.5% of the experts who replied to the survey said they were dissatisfied, the proportion of experts who were very satisfied was 51.6%. Concerning perceptions on the quality of evaluation/review activities, just 2% felt that they were low quality, with 56.3% believing they were of high quality across the board.
The extension of the Agency’s mandate and change of name (from TEN-T EA to INEA) in 2014 did not have an impact in respect of proximity to stakeholders and the visibility of the EU as a promoter of the programmes delegated to it. In terms of proximity to the addressees, INEA acted as a direct contact point between programme applicants and beneficiaries of EU funding, as well as other stakeholders, and the Commission. The most regular contact takes place between project beneficiaries and the respective project officers at INEA. Separation of policy making and implementation benefits the beneficiaries insofar as they obtain specialised advice and service. During the evaluation period, INEA made sustained efforts to boost awareness of new funding opportunities for project promoters (both under CEF and Horizon 2020) and to consolidate a service-oriented communication. It also supported parent DGs by giving visibility to the delegated programmes via success stories.
Concerning visibility of the EU as a promoter of the programmes entrusted to INEA, the survey suggests that there is awareness amongst stakeholders that the Agency is acting under powers delegated by the Commission. The Agency’s contributions to enhanced visibility of the programmes included Info-days, project management workshops, pro-active use of social media tools and good quality graphic and communication materials. Additionally, the website of the Agency includes a significant amount of material related to the delegated programmes: project information, brochures, maps and statistics. The many channels of communication used by the Agency have resulted in the enhanced visibility of the programmes.
5.2 Efficiency
This section considers the relationship between the resources used by the Agency and the output. The analysis, among other factors, also includes analysis of administrative and regulatory burden and looks at aspects of simplification.
During 2014-2016 INEA proved to be an efficient and cost-effective structure for the management of the delegated programmes. Its administrative budget decreased from 2% of the operational budget based on payment appropriations in 2014 to 0.9% in 2016.
INEA’s actual operational budget during 2014-2016 (in commitment appropriations) constituted EUR 11,788 billion, of which EUR 11,679 billion from EU budget and EUR 109,2 million related to EEA/EFTA and third countries’ contributions (not considered in INEA’s SFS). INEA’s administrative budget (including such contributions) increased from EUR 13.1 million in 2014 to EUR 18.2 million in 2015 and EUR 21.5 million in 2016.
A detailed analysis of the INEA cost-efficiency and results of the Cost-benefit analysis is presented under point 5.4.
Programme implementation cost (ratio between the administrative and operational budget, percentage), payment appropriations
Source: CSES, PPMI and CSIL, based on INEA Annual Activity Reports.
The number of proposals managed by INEA in 2014-2016 was 3490, very close to the CBA forecast of 3576. The number of total running projects increased from 496 in 2014 to 1383 in 2016. During the evaluation period, the actual workload of INEA was higher than estimated in the 2013 CBA due to factors beyond the Agency’s control that influenced its workload, such as the average grant size, the number of proposals and grants, and the re-injection of budget amounts. The number of running projects ‘per operational head’ increased from 4.4 in 2014 to 7.6 in 2016 as a result of the first Calls of the 2014-2020 MFF. The ratio of budget ‘per operational head’ did not fluctuate much in terms of commitments (from EUR 31.8 million to EUR 27.8 million) but increased significantly in terms of payments (from EUR 5.4 million to EUR 13.5 million) in 2014-2016.
INEA Budget, staff and workload indicators, 2014-2016 (in € million, n° or %)
|
|
2014
|
2015
|
2016
|
|
Operational budget, commitments
|
3590
|
3251
|
5057
|
|
Operational budget, payments
|
606
|
2021
|
2448
|
|
Administrative budget, commitments
|
13.1
|
18.2
|
21.5
|
|
Administrative budget, payments
|
12.4
|
17.9
|
21.9
|
|
Actual number of staff (at the end of the year)
|
151
|
186
|
225
|
|
Actual number of operational staff
|
113
|
145
|
182
|
|
Programme management cost* (commitments)
|
0.4%
|
0.6%
|
0.4%
|
|
Programme management cost (payments)
|
2.0%
|
0.9%
|
0.9%
|
|
Budget ‘per operational head’ (commitments)
|
31.8
|
22.4
|
27.8
|
|
Budget ‘per operational head’ (payments)
|
5.4
|
13.9
|
13.5
|
|
Proposals received
|
479
|
1548
|
1463
|
|
Total running projects
|
496
|
1026
|
1383
|
|
Running projects ‘per operational head’
|
4.4
|
7.1
|
7.6
|
* Defined as the ratio between the administrative and operational budget.
Human Resource Management
In order to optimise allocation of human resources across its delegated Programmes, INEA has developed a methodology to calculate its operational workload and staffing needs which was approved by its Steering Committee in February 2018 and which has been kept continuously updated. The methodology is based on three main workload drivers: proposals to be evaluated, grant agreements to be signed, ongoing projects to be managed. While current budgetary rules do not allow shifting Agency staff between Programmes, the work-load assessment methodology proved useful to allocate staff more efficiently within Programme sectors, in agreement with the parent DGs.
INEA had to cope with an increase of its staff turnover rate from 7.5% in 2015 to close to 10% in 2016 (the target level being 3%). INEA has actively addressed the underlying issues and has elaborated a new staff retention policy with elements such as encouraging internal staff mobility, announcing vacant positions internally and organising job shadowing. As a consequence, INEA is now being more effective in retaining staff and filling vacancies, and at the end of 2018 the vacancy rate had dropped to 5.4%.
The staff opinion survey carried out in 2016 revealed an increase in INEA’s staff satisfaction compared to 2014. The staff engagement index is a composite indicator that helps measure how connected the staff is to the Agency and how committed to help it achieve its goals. In 2016, INEA had a staff engagement index result of 68%, above average of the other executive agencies (66%) and the Commission (64%). With an improvement of 5% from 2014, it clearly shows that good progress has been done in this area. The overall INEA staff job satisfaction in 2016 was within the average of the Commission departments (71% in INEA compared to 70% in the Commission) and the other executive agencies.
Several other indicators are evidences of the good results achieved by INEA in terms of HR management. 62% of INEA staff felt the Agency was a modern and attractive workplace, 6% higher than the Commission average (56%). 95% of employees at INEA said they had a clear understanding of the Agency’s purpose in 2016, a 9% increase compared with 2014. Other examples are: the managers’ approach to the staff (staff has a much better perception of management, especially middle management, as compared to the 2014 survey, while it is important to note that a new management structure was just evolving in 2014) and training and opportunities for staff to advance their professional development.
At the same time, INEA staff perceived less positively the offer of training and development activities and the opportunities for their mobility and career development.
To follow up these staff survey results, as mentioned under section 3, the Agency adopted in April 2017 a comprehensive multi-annual HR strategy followed by a staff retention action plan in September 2018. The evolution of key indicators for staff satisfaction – after the evaluated period – is encouraging.
In the latest staff survey, which was carried out in the 4th quarter of 2018 (thus outside of the evaluation period), INEA managed to increase for the third consecutive time the staff engagement index - from 61% in 2014 to 68% in 2016 and 70% in 2018. Also, the overall job satisfaction, even if it is remains lower than the Commission average (77%), has also increased to 75%. The results of this new staff opinion survey also show points for attention: decreasing staff satisfaction with workload levels, work recognition and performance management as well as lower satisfaction rates concerning the Agency’s middle management. These matters will have to be addressed by the Agency together with its parent DGs in the most appropriate way and will feed into a new follow-up action plan.
Overall, the size and structure of INEA, and the level of resources in each of its departments, were appropriate to its mandate and delegated tasks during the evaluation period. The organisational structure of INEA was, as of the start of its mandate in 2014, based on separate departments for each of the delegated programmes – CEF (Department C) and Horizon 2020 (Department H) – as well as for Programme support and resources (Department R). Compared with the previous TEN-T EA, a new hierarchical level was therefore introduced in the form of heads of department function, which led to the need for staff to adapt to new internal reporting and approval circuits. Department C has 3 units for CEF Transport, reflecting the substantial respective grant budget delegated to INEA, split by funding priority and geographical coverage, as well as 1 unit each for CEF energy and CEF telecoms (formerly one single unit). Department H has 2 units respectively for transport and energy research. Department R comprises 4 units responsible for the Agency’s horizontal tasks such as planning and coordination of the calls and communication, administration and finance, HR and IT and logistics, as well as the internal control function. INEA’s organisational structure was reviewed and effectively aligned to cope with changes, such as the evolution of its mandate, notably with the creation of a dedicated unit for CEF Telecoms on the occasion of the delegation of the WiFi4EU programme to INEA.
During 2014-2016, INEA, in cooperation with the Commission, continued the optimisation of its procedures and programme management functions and introduced a number of simplifications. Concerning Horizon 2020, the improvements and simplifications concerned wider use of remote evaluation of proposals, the introduction of a single set of rules, electronic signature of grant agreements, the Participant Portal as the one-stop-shop for interactions with participants, single reimbursement rate, flat rate for indirect costs, etc. With regard to CEF, these concern the streamlining of submission of progress reports by the beneficiaries (Action Status Reports), e-submission and evaluation modules in the TEN-tec portal, harmonisation of Grant Agreements etc. Following its successful testing end 2015, Speedwell has become as of 1st January 2016, the standard tool to process administrative payment paperless.
In July 2014 the Agency revised its Manual of Procedures. Furthermore, an exercise for streamlining and updating the internal procedures and guidelines for the new programmes was initiated. For document management, the use of paperless workflows was extended. A specific Agency Task Force was created which led to a concrete action plan to increase paperless procedures.
5.3 Coherence
Coherence looks at any overlaps and complementarities within the programme portfolio managed by the Agency or delimitation of responsibilities between the Agency and its parent DGs.
In the 2014-2016 period, INEA demonstrated flexibility in implementing a diverse thematic programming portfolio and being answerable to four different parent DGs. The evaluation did not find any evidence of overlaps, duplications, gaps or inconsistencies within the INEA programme portfolio. The Memorandum of Understanding as last amended in July 2018 provides for a clearer and appropriate delimitation of responsibilities and tasks between INEA and its parent DGs. However, due consideration should be given to the fact that supervision tasks (e.g. financial, auditing, planning, reporting, coordination etc) have increased significantly as of the start of the present MFF. Hence, the necessary supervision capacity in the parent DGs (with benchmarking among DGs) needs to be ensured.
Since INEA was established around the concept of network industries and related research and innovation, there is a strong coherence between the CEF and Horizon 2020 programmes delegated to INEA notably in respect of the energy and transport sectors. Concerning the transport sector in particular, the Commission benefitted from the continuity of support to TEN-T infrastructure actions within the same Agency since 2006, as well as the taking over by INEA of the Marco Polo programme from EACI (now EASME).
The coherence in the programmes managed by INEA also helped to exploit synergies to the extent possible. The mid-term evaluation of the CEF programme noted that, on the one hand, CEF synergies have been effective at programme level, notably through common implementation by INEA with the same business processes, 3-sector coordination, common CEF coordination committee of Member States, and common Work Programmes for CEF Financial Instruments, all of which generate economies of scale and simplification. On the other hand, the legal basis of the CEF programme is not sufficiently flexible to allow funding of large-scale infrastructure projects combining 2 or 3 sectors; hence the limited success of the CEF synergies (Transport and Energy) call for proposals of 2016. The Common Understanding on CEF II provides for facilitation of synergies within the programme which would benefit from a coherent set-up of the programmes managed by INEA.
With regard to synergies between Horizon 2020 and CEF, despite some intrinsic difficulties linked with the legal bases of both programmes, different business processes and distinct type of support projects (research vs deployment and / or infrastructure building), INEA and the parent DGs endeavoured to take initiatives in this field. INEA launched a reflection exercise concerning the transport sector which led to a report being presented to the Steering Committee in 2017. Several actions were taken at project level including technical meetings with the parent DGs, TRIMIS collaboration, and a synergies webpage in INEA website. As a result of this exercise INEA Horizon 2020 projects had a greater presence at the TEN-T days organised by DG MOVE (as of the 2017 edition). In addition, there is a higher degree of coordination between Horizon 2020 and CEF Work Programmes. Several proposals applying for CEF Transport funding have referred to results from Horizon 2020 projects. A similar exercise is currently being carried out in the Energy sector.
The evaluation however stressed that INEA would gain from a closer alignment of procedures, as appropriate, between the infrastructure programme (CEF) and the research programme (Horizon Europe) in terms of call launching, submission of proposals, evaluation and selection process and grant awarding for the next MFF. Work is being developed in this respect in the context of the corporate model grant agreement and e-grants process.
INEA has a very good working relationship with parent DGs which are in general satisfied with the levels of formal and informal communication taking place in a regular and structured manner. The fact that some of the Agency’s personnel, mainly at management level, is either seconded from one of the parent DGs (in 2018, 19 of 277 staff) or has previously worked therein is an important asset in this respect.
Despite the inherent difficulties linked with reporting on programme implementation (grant funding) on the variety of delegated programmes and sub-programmes, INEA has been proactive and constructive in providing policy feedback to parent DGs. This is the case notably on lessons-learned from project management and project funding. However, the evaluation found that, as the projects supported at the start of the programmes are unfolding in the second half of the present MFF, INEA could use its knowledge of programme implementation to further help Commission policy-making notably on the preparation of the Work Programmes. INEA has also been proactive at proposing a framework for the provision of KPIs on CEF implementation, as well as preparing Country and Corridor reports (notably on CEF Transport) and brochures on achievements and successful stories. While improvements were found necessary on the provision of real-time information on programme and project progress, this is being addressed and improved constantly through the close cooperation between the parent DGs and INEA.
5.4 Retrospective Cost-Benefit Analysis
The retrospective Cost–Benefit Analysis for the period 2014-2016 was carried out based on the results of the 2013 ex-ante CBA, the assumptions laid down in the Specific Financial Statement (SFS) and the actual costs of INEA.
The 2013 ex-ante CBA estimated that the delegation of tasks to INEA would result in savings of around EUR 54 million over the 2014-2020 Multi-Annual Financial Framework when compared to the management by the Commission services (in-house scenario). For the period 2014-2016 such savings were estimated at around EUR 16.1 million. The savings estimate in the INEA SFS for the period 2014-2016 was in the order of EUR 22.8 million.
The retrospective CBA revealed that the actual costs of the executive agency approach in 2014-2016 were much lower than the costs of the in-house scenario estimated both in the ex-ante CBA and in the SFS. The actual cost savings constituted EUR 24.4 million (or 29% of the estimated costs under the in-house scenario).
Estimated costs and savings of the executive agency scenario (INEA) in 2014-2016, EUR
As forecasted in the SFS and the ex-ante CBA, savings of the executive agency scenario primarily resulted from a higher share of lower-cost contract staff (CAs) employed within the Agency and the lower overall number of staff.
Significant cost savings occurred in Title II “Infrastructure and operating expenditure” and Title III “Programme Support Expenditure” of the administrative budget. Title I “Staff related expenditure” was higher than estimated in the SFS, which related to higher average staff costs. It should be noted in this respect that while in the SFS the average staff cost estimations remain constant during 2014-2020 period, the actual average staff costs might raise further in the run up to the end of the period due to salary indexation, promotions and/or increasing staff seniority.
The workload analysis in the retrospective CBA revealed, that while the operational budget actually executed by INEA was lower than initially estimated in the ex-ante CBA/SFS (92% of the estimations), the actual number of projects managed by INEA, which constitutes the main workload driver for the Agency, in 2015-2016 largely corresponded to the CBA estimates (number of running projects at the end of 2016: 94%). The number of proposals evaluated was about 98% of the estimations.
At the same time, many parameters of the delegated programmes significantly deviated from the initial CBA/SFS estimations, e.g. higher than estimated operational budget of the CEF-Telecoms programme and much lower average project size, reflux of funds for CEF Transport following cancellation of projects and cost savings into new calls, lower than anticipated number of projects for studies under CEF Energy, reallocation of CEF/Horizon 2020 funds to EFSI, as well as operational budget changes resulting from the MFF review exercise and the annual budget review exercise. As a consequence of the different changes, the number of authorised staff in INEA was reduced in 2016 (from 262 estimated in the SFS to 247). The actual execution of payment appropriations was slower than anticipated in the SFS and CBA, which indicates that increase in the projects’ management workload will unfold later than estimated.
6.Conclusions
The results of the evaluation of INEA operations in 2014-2016 confirmed that INEA achieved effective management of the delegated programmes while operating according to the legal framework governing it. For each part of the CEF and Horizon 2020 programmes delegated to INEA, as well as the legacy programmes, the Agency has proven to be responsive and flexible to all challenges and evolving policy context and has timely and efficiently accommodated these. Its objectives in terms of key performance indicators (Time-to-grant, Time-to-inform, Time-to-pay, residual error rate etc.) were almost continuously attained if not out-performed.
Moreover, applicants, beneficiaries and experts showed a large satisfaction with the operation of INEA. In particular, the Agency achieved its objectives in terms of proximity to the addressees and of visibility of EU programmes. During the 2014-2016 period, INEA improved its processes by adopting various simplification measures and IT tools.
The retrospective CBA confirmed that the delegation of programme implementation to INEA remained considerably more cost effective than the in-house scenario. During this period INEA continued to implement the delegated programmes in an effective, efficient and coherent manner, generating important cost-savings to the EU budget and high value for money. The actual cost savings constituted EUR 24.4 million, or 29% of the estimated costs under the in-house scenario.
The staff survey of 2016 revealed a very reasonable level of staff engagement and staff satisfaction for INEA. The Agency has actively addressed identified shortcomings in its 2017-2020 multi-annual HR strategy and dedicated action plan, including through its staff retention policy. In particular the turn-over levels and vacancy rate have been significantly reduced. The new staff survey results of 2018 show that staff perception of key areas of work has considerably improved.
The Memorandum of understanding between INEA and its parent DGs set a clear definition of tasks between the Agency and the Commission, which was observed over the evaluation period.
The coordination and communication mechanisms that are in place facilitate information flows between INEA and the Commission services, in particular with regard to the content of projects supported and their results. No evidence of overlaps, duplication, gaps and inconsistencies within the programme portfolio managed by INEA were identified.
The current evaluation identifies areas where further improvements are necessary from both the Commission and INEA and which aim to further streamline or define the working arrangements between both sides.
These relate to:
-Coordination of the calendar for the calls for proposals;
-Improve synergies between the delegated programmes and between sectors within the same programme;
-Further improve policy feedback from INEA through appropriate mechanisms;
-Align to the extent possible programme implementation business processes between the infrastructure (CEF) and the research and innovation (Horizon) programmes for the new MFF, notably in terms of model grant agreement and e-grants.
The parent DGs and the Agency are preparing one action plan that will address the shortcomings identified in this evaluation. This action plan will also address the recommendations outlined in the performance audit report of the European Court of Auditors (Special Report No 19/2019 “INEA: benefits delivered but CEF shortcomings to be addressed”) (mentioned in section 4.2 above). This action plan aims to further increase the Agency’s performance and contributes to an efficient implementation of the delegated programmes.
7.Annex: Evaluation questions
Effectiveness: to what extent have the objectives of delegating programme management tasks to INEA been achieved?
-To what extent has INEA been operating according to the legal framework establishing it?
-To what extent has INEA achieved its objectives with special focus to (a) the implementation of the delegated programmes CEF, Horizon 2020 and the legacy programmes (TEN-T and Marco Polo II), (b) the implementation of the internal control principles, notably sound financial and human resource management? What, if anything, could be done to render INEA more effective in achieving these objectives?
-To what extent has INEA contributed to an improved management of the delegated programmes and better services to the stakeholders and addressees in terms of the elements assessed in the 2013 CBA and as compared to the alternative options mentioned in that CBA?
-To what extent has INEA contributed to improved management of the programmes in terms of:
−Proximity to addressees (e.g. role played by the Agency as focal point for applicants and beneficiaries of the programmes; instruments and mechanisms put in place for communicating with the applicants and beneficiaries).
−Effective implementation of the programmes, taking into account the interests of the addressees and those of the EU:
ürate of execution of commitment appropriations
ürate of execution of payment appropriations
ütime to grant
ünet time to pay
üresidual multi-annual error rate identified at ex-post control.
−Visibility of the EU as promoter of the programmes entrusted to the Agency (e.g. compliance with the Commission’s guidelines on information and visibility of programmes
, instruments put in place to ensure the visibility of the EU as promoter of the programmes).
Efficiency - to what extent has INEA carried out its tasks efficiently in line with estimations?
-To what extent have the actual costs (including cost of coordination and monitoring) of INEA corresponded to the estimates of the 2013 CBA? If not, what are the reasons behind?
-To what extent have the actual benefits corresponded to the estimates of the 2013 CBA? If not, what are the reasons behind?
-To what extent has the management and execution of the programmes by INEA been cost-effective as compared to the alternative options
?
-To what extent has INEA contributed to improved management of the programmes in terms of simplification of the procedures and flexibility in the implementation of delegated tasks (e.g. capacity to adapt to periods of high workload)?
-Which further scope for simplification exists?
-Which aspects/means/actors or processes render INEA more or less efficient? What could be improved?
-To what extent has the establishment of INEA resulted in savings to the EU budget as compared to the alternative options (e.g. difference in costs between the Commission option and the Agency option)?
-To what extent have INEA’s internal organisation and procedures been conducive to its efficiency?
-To what extent is INEA’s internal organisation capable and flexible to rapidly respond to resource needs due to uncertainties related to volumes of work?
-To what extent does INEA’s human resources management contribute to the achievement of the Agency’s objectives?
To this end, and as a minimum, the contractor shall analyse the following:
-Is the size and structure of the organisation and its separate entities appropriate?
-Is the balance between operational staff and administrative support staff appropriate?
-Is the staff turnover and vacancy rate well managed?
-How does the Agency follow up on the findings of the two latest staff surveys (2014; 2016)?
The Commission will provide the relevant statistical data to the successful tenderer after the signature of the contract.
The management performance indicators have to be compared, where applicable, with the indicators used in the previous evaluation study (2011-2013).
Coherence - to what extent there is consistency between the tasks managed by INEA and those managed by the parent DGs and other executive agencies, and has the Commission been able to maintain an adequate level of know-how in relation to the programmes entrusted to the Agency?
The establishment of INEA has enabled the Commission (within the parent-DG's and across Commission services via the central redeployment pool) to allocate staff to institutional tasks as compared to the alternative options. As a minimum, the following questions shall be addressed:
-To what extent have there been overlaps/ gaps/ inconsistencies within the programme portfolio managed by INEA?
-Is there a clear and appropriate delimitation of responsibilities and tasks between INEA and the Parent DGs? Are there overlaps or gaps?
-To what extent has INEA enabled the Commission to better focus on its policy related tasks?
−Are there any governance (financial and policy) issues in relation to the implementation of the tasks that are delegated to the executive agencies/ issues as regards the Commission’s services being able to steer EU policy or budget implementation?
−Are appropriate mechanisms and instruments in place to ensure an adequate coordination and information flow between INEA and the Commission services?
−Does INEA provide useful information in support of the policy process (e.g. information required for the annual Management Plan of the Parent DGs)?
-To what extent have the activities of INEA resulted in unintended effects (both desirable and undesirable)?
-To what extent has the Commission, in the presence of INEA, been able to maintain an adequate level of know-how in relation to the programmes entrusted to the Agency? How has this been achieved? What are the feedback channels, means and methods used for this purpose? What are areas for improvement, if any?
-Have the monitoring, reporting and supervision arrangements in place enabled the Commission to benefit, in the short and medium term, from the know-how created within INEA?
-How effective is the flow of information and communication between INEA and the Commission services (in particular Parent DGs)?
-To what extent would the closing down of INEA result in losing significant know-how in relation to the management of the programmes entrusted to INEA?
