EUROPEAN COMMISSION
Brussels, 29.4.2020
SWD(2020) 75 final
COMMISSION STAFF WORKING DOCUMENT
Evaluation of the Consumers, Health, Agriculture and Food Executive Agency (CHAFEA)
Accompanying the document
Report from the Commission to the European Parliament, the Council and the Court of Auditors
Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Executive Agency for Small and Medium-Size Enterprises, the Innovation and Networks Executive Agency, the Education, Audiovisual and Culture Executive Agency, the Research Executive Agency and the European Research Council Executive Agency
{COM(2020) 184 final} - {SWD(2020) 73 final} - {SWD(2020) 74 final} - {SWD(2020) 76 final} - {SWD(2020) 77 final} - {SWD(2020) 78 final}
TABLE OF CONTENTS
1.Introduction
2.Background to the intervention
2.1.
Description of the intervention and its objectives
2.2.
Baseline and points of comparison
3.Implementation / state of play
4.Method
4.1.
Short description of methodology
4.2.
Limitations and robustness of findings
5.Analysis and answers to the evaluation questions
5.1.
Effectiveness
5.2.
Efficiency
5.3.
Coherence
5.4.
Retrospective Cost-benefit analysis
6.Conclusions
Annex : Evaluation Questions
1.Introduction
The purpose of this evaluation is the periodical (triennial) evaluation of the operation of the Consumers, Health, Agriculture and Food Executive Agency (CHAFEA). CHAFEA is governed by its Act of Establishment
, which sets out the mandate of the Agency and the Act of Delegation
, which specifies the tasks to be carried out by, and the powers delegated to the Agency to perform its mandate. These tasks relate to the implementation of dedicated parts of certain EU financial programmes in the fields of health, consumer rights and agriculture. Furthermore, the Agency is governed by the
Framework Regulation for Executive Agencies
, which lays down general rules, notably in terms of governance and budget.
In line with the Commission’s Better Regulation principles
, the evaluation assessed whether the Agency has fulfilled its tasks in an efficient and effective way, whether there are overlaps, gaps or inconsistencies within the management of the programme portfolio executed by the Agency, and whether there is a clear delineation of tasks between CHAFEA and the parent DGs
. The results of this evaluation are summarised hereafter.
The evaluation assesses whether the functioning of the Agency has yielded the expected positive results as estimated in the cost-benefit analysis for delegating task to the Executive Agency
and identifies potential areas of improvement. To this end the estimations of the cost-benefit analysis of 2013 have been tested to provide evidence on the validity of the assumptions made in the ex ante scenario by considering the actual costs and benefits of programme implementation by the Executive Agency in a structured way. The aspects covered by the cost-benefit analysis are specified in Article 3(1) of the Framework Regulation
and the Guidelines on establishing and operating Executive Agencies
.
The evaluation does not cover the achievements of the programmes managed by CHAFEA, which themselves are subject to mid-term and ex post evaluations. The evaluation of CHAFEA nevertheless provides useful input for these programme evaluations, considering that the performance of the Agency affects the efficiency and effectiveness of the programmes it manages.
The evaluation examines the efficient use of resources and the effective achievement of the tasks entrusted to it. In particular, it looks at whether the alignment of more coherent programme portfolios with the Agency’s core competences and its brand identity delivered the estimated qualitative benefits; whether the assembly of the management of different EU programmes delivered the estimated synergies, simplification and economies of scale; whether the pooling of instruments guaranteed consistent service delivery and whether there is scope for simplification and further efficiency gains.
The evaluation is supported by a study carried out by an external contractor and it covers all the tasks carried out by the Agency during the years 2014-2016. The results of the study are summarised hereafter and this evaluation will be presented to the European Parliament, to the Council and to the Court of Auditors in accordance with Article 25(1) of the Framework Regulation.
The results of the evaluation will feed into the reflection to assess the opportunity of expanding/modifying the Executive Agency’s mandate in view of the delegation of the implementation of 2021-2027 EU programmes.
2.Background to the intervention
2.1.Description of the intervention and its objectives
According to the Framework Regulation and CHAFEA’s Act of Establishment, the outsourcing to CHAFEA intended:
To allow the Commission to focus on its institutional tasks assigned by the Treaty which require discretionary powers in translating political choices into action. Such institutional tasks should not be outsourced.
To enable achieving the goals of EU programmes more effectively. According to the cost-benefit analysis carried out in 2013, delegating tasks to the Executive Agency was estimated to be more cost-effective than an in-house scenario. The Act of Establishment estimated that the alignment of more coherent programme portfolios with the Agency’s core competences and its brand identity would bring qualitative benefits. In addition, it projected that assembling the management of different EU programmes would bring synergies, simplification and economies of scale.
The original objective of the intervention is entrusting the Agency with the implementation of several EU programmes in parts or fully:
the consumer programme 2014-2020 (DG JUST);
the public health programme 2014-2020 (DG SANTE and DG GROW);
the food safety training measures (Better Training for Safer Food - BTSF) under the Common Financial Framework for Feed and Food (CFF Regulation 652/2014
) (DG SANTE);
the information provision and promotion measures concerning agricultural products implemented in the internal market and in third countries (Regulation 1144/2014
) (DG AGRI).
The Agency also implements the legacy of the predecessor programmes and actions
.
CHAFEA carried out the following tasks:
(a) (i) monitoring the projects, making the necessary checks and recovery procedures; performing budget implementation tasks covering revenue and expenditure:
awarding grants and managing the ensuing agreement or decision, including the operations required to launch and conclude grant award procedures;
concluding public procurement procedures and managing the ensuing contracts, including the operations required to launch and conclude public procurement procedures;
performing all the operations required to launch contests and award prizes in accordance with the relevant rules;
(a) (ii) regarding simple programmes, CHAFEA is entrusted with the publication of the calls for proposals and with the selection of the programmes, both “simple” and “multi”;
(b) providing support in programme implementation, in particular:
collecting, processing and distributing data, and in particular compiling, analysing and transmitting to the Commission all information required to guide implementation of the programme, promote coordination with other Union programmes, the Member States or international organisations;
contributing to evaluation of the impact of the programme and to monitoring the actual effect of the measures;
managing and directing a network, in particular concerning the target public (beneficiaries, recipients, projects, actors);
organising meetings, seminars or talks; organising training; organising Commission's own information and promotion campaigns for agricultural products;
contributing to studies and evaluations, in particular the annual and/or mid-term evaluation of implementation of the programme, and contributing to preparation and implementation of follow-up action on evaluations;
preparing recommendations for the Commission on implementation of the programme and its future development;
planning and implementing information operations;
ensuring overall control and supervision data;
participating in preparatory work on work programmes and financing decisions;
managing technical support services.
2.2.Baseline and points of comparison
The current evaluation of CHAFEA operations during 2014-2016 assesses the actual costs and benefits of programme implementation by CHAFEA (executive agency scenario) when compared with the alternative scenario of management by the Commission services (in-house scenario).
Accordingly, the reference point for the present CHAFEA evaluation is the 2013 ex ante cost-benefit analysis, and the specific financial statement (SFS) of CHAFEA.
According to the SFS of CHAFEA prepared in 2013, the estimated efficiency gains amounted to EUR 2.3 million over the period analysed. The total number of full time equivalents (FTEs) required to manage the relevant programmes in year 2020, the peak programming year in terms of workload, was estimated at 79 for CHAFEA. The total initial operational budget entrusted to CHAFEA in 2020 was estimated at around EUR 198.868 million in commitment appropriations and EUR 176.759 million in payment appropriations. Considerable efficiency gains were expected as compared to the in-house scenario over the period, along with non-quantifiable benefits such as improved quality of programme management and service delivery, improved visibility of the EU programmes and proximity to beneficiaries.
3.Implementation / state of play
Initially founded as an executive agency for the management of the EU programme in the field of public health (PHEA)
, CHAFEA has been progressively entrusted with the implementation of the public health programme, consumer programme and food safety training measures. The Agency’s mandate was extended by the Commission Implementing Decision No 2014/927/EU, amending Commission Implementing Decision 2013/770/EU, which came into effect on 1 January 2014. Through this new Decision, CHAFEA was entrusted with certain implementation tasks of the information provision and promotion measures concerning agricultural products implemented in the internal market and in third countries as well as Commission own initiatives.
The new mandate also brought changes in the governance of the Agency as new parent DGs joined the Steering Committee, and subsequently signed a Memorandum of Understanding (MoU) with CHAFEA
. The new MoU defined the modalities and procedures of interaction between CHAFEA and its parent DGs. In addition, it brought a clearer delimitation of the administrative activities between the Agency and its parent DGs. The distribution of tasks and responsibilities, as presented in the MoU, is included in section 5.3 of this document. Compared to this reference, this evaluation assesses whether this distribution of roles has been respected for the 2014-2016 period, and the effects of the intervention on the coherence and the efficiency of CHAFEA’s action.
At the beginning of 2014, CHAFEA was already a well-established organisation based in Luxembourg with a stable level of staffing. The Agency had 49 staff (without intérimaires) in 2014 and 2015, and 59 staff
(without intérimaires) in 2016. The number of projects only slightly increased over the period and the number of total running projects ‘per head’ thus decreased from 4.88 in 2014 to 4.31 in 2016. The total number of funded projects increased in particular in the public health programme, from 27 in 2014 to 57 in 2016 and in the Consumer protection programme, from 92 in 2014 to 124 in 2016. The Agency began implementing activities under the promotion of agricultural products programme in 2016, receiving 226 proposals (of which 199 ‘simple’ and 27 ‘multi’ proposals
) and funding 6 ‘multi’ programmes.
The operational budget of CHAFEA (according to payments) increased from EUR 61.27 million in 2014 to EUR 66.26 million in 2015 and to EUR 80.73 million in 2016. The Agency’s administrative budget increased
as well from EUR 6.62 million in 2014, to EUR 6.73 million in 2015 and to EUR 8.16 million in 2016. The total budget ‘per head’ (in terms of payments) increased from EUR 1.25 million to 1.65 million between 2014 and 2016. This relates mostly to the growing volume of the operational budget, which outpaced an increase in the number of the Agency’s staff between 2014 and 2016.
The number of proposals (both for grants and projects) managed ‘per head’ shows a greater year-to-year variability, with an increase of 50% between 2014 and 2016. Programme management costs (defined as the ratio between the administrative and operational budget) only slightly decreased, from 8.97% to 8.70%. The costs of auditing the grant management process during the year fell from 5.3% in 2014 to 3% in 2016.
A series of organisational and procedural changes started with the implementation of the programmes under the multiannual financial framework (MFF) of 2014-2020:
Delegation of the management of additional programmes, new funding instruments and as a consequence new tasks and activities for the Agency
.
Further simplification and improvement of grant management processes and procedures, notably through the electronic submission of proposals and the adoption of the Commissions IT tools and systems (i.e. Compass-SyGMa).
The adoption of new business processes, Horizon 2020 IT tools and modes of service delivery
.
The adoption of the multiannual human resource (HR) strategy for 2017-2020, mainly as a response to the relatively low staff satisfaction rates revealed by the staff opinion survey 2016. Based on this new strategy and in combination with a dedicated action plan addressing attention points identified through the survey, the Agency implemented specific measures related to staff engagement, internal communication, career development opportunities, work and private life balance and training, meant to improve effective HR management and increase the effectiveness and flexibility of the Agency’s operations.
4.Method
4.1.Short description of methodology
The evaluation was supported by a study carried out by an external contractor
. The methodology of the CHAFEA evaluation was consistent with the approaches employed for similar parallel evalautions of EACEA, ERCEA, EASME, REA and INEA. The evaluation methods used included: desk research and analysis of business processes, statistical analysis of administrative and monitoring data, interview programme, case studies, surveys, cost-benefit analysis, focus group and benchmarking. Specific methodological approaches were used to ensure data triangulation (confirmation from different sources).
The study was structured around a series of evaluation questions available in Annex.
The evaluation relied on the following data and methodology:
An extensive documentary review and desk research of CHAFEA and Commission documents, including Annual Work Programmes and Annual Activity Reports, audit reports and related documents, CBA studies and other financial documents, the Memorandum of Understanding (MoU), previous evaluations and action plans addressing the ensuing recommendations. The evaluation also relied on the results of the 2016 staff opinion survey.
Exploratory interviews were conducted with Commission and CHAFEA staff members (around 60 persons), in particular managerial staff of CHAFEA and of the parent DGs (33 interviews in total) and with the Agency’s beneficiaries and intermediaries (33 interviews in total), with a focus on the case studies.
Two interlinked surveys addressed various target stakeholder groups. The first was addressed to private bodies acting as intermediaries or facilitators for projects and beneficiaries of the ‘standard’ calls for proposals (for all four programmes managed by CHAFEA). 92 intermediaries and beneficiaries responded to this survey. The second survey targeted external experts contracted by CHAFEA. 87 experts responded to this survey.
Case studies were used to complement the interview programme and desk research to provide in-depth insights into the performance of the Agency on issues such as coherence and maintenance of the know-how within the Commission, efficiency of introduced simplification measures, etc. Those case studies have fed the different assessment questions.
A retrospective cost-benefit analysis was performed.
A focus group meeting composed by representatives of CHAFEA, members of the Steering Committee, Commission staff and stakeholders was organised to present and validate the findings and the recommendations of the evaluation
.
A comparative analysis and benchmarking of the various Executive Agencies was also carried out against a set of (qualitative and quantitative) indicators. This enabled the evaluation team to compare the performance of CHAFEA effectively with that of other Executive Agencies.
4.2.Limitations and robustness of findings
Most relevant stakeholder groups were consulted (beneficiaries, intermediaries, external experts). However, compared with the overall population reached, unsuccessful applicants participated less actively in the survey than programme beneficiaries.
No sampling bias was observed as the profile of the respondents to the surveys was very similar to the overall population, guaranteeing statistical representativeness. The non-response bias (not all characteristics of the group that did not reply had been captured in full) was mitigated through triangulation with the results of follow up interviews. The triangulation approach, using multi-level and multi-stakeholder dimension in the data collection, ensured the robustness and reliability of the data and information used to draw up conclusions in the supporting study.
Some views from Commission and Agency staff gathered from the interviews and mentioned in the study illustrate personal experience, although they may not be representative of overall relations between the Commission and the Agency.
The availability of comparable data with other Agencies was also a limit for the study. For example, the satisfaction of experts and beneficiaries of CHAFEA could not be compared with that of other Executive Agencies. This is mainly due to the fact that final studies supporting the evaluations of the other Executive Agencies were not finalized at the date when the final report concerning CHAFEA was submitted and presented to the Steering Committee of the Agency
.
5.Analysis and answers to the evaluation questions
5.1.Effectiveness
For the purpose of this evaluation, effectiveness relates to how successful the Agency has been in achieving or progressing towards its objectives.
The evaluation inferred that during 2014-2016 CHAFEA operated according to the legal framework that established it. CHAFEA’s activities, as planned in the annual work programmes and reported in the annual activity reports, corresponded to the tasks set out in the Commission’s establishment and delegation Decisions.
As provided in the delegation Decision, the modalities and procedures of interaction between the Agency and its parent DGs are set out in the Memorandum of Understanding (MoU). The MoU provides for a supervision strategy aimed at avoiding gaps or duplication of efforts resulting from crossover between the policy-making, monitoring and supervision tasks of the parent DGs and the execution tasks of the Agency. These provisions have generally worked well but further improvements are needed (see section 5.3).
In the evaluation period, CHAFEA continuously improved its operations and achieved a sound level of overall effectiveness. The Agency demonstrated a high degree of flexibility to integrate all the key changes encountered, especially those induced by the extended mandate
. During the reference period, CHAFEA has taken the necessary measures to reach the objectives of the new programmes and adjust its internal procedures to the emerging challenges and needs. However, some of these changes are highlighted in the support study as having raised systematic challenges (i.e. the new instruments, such as operating grants and framework partnership agreements, were initially not fully integrated into the corporate IT tools
). In this respect, new modules were adapted to the needs of CHAFEA’s programmes. The Agency dedicated continuous efforts to integrate these needs and to ensure an optimal service to its applicants and beneficiaries.
Other challenges encountered, related to the late adoption of the 2014 annual work programme, had direct relevance for the Agency’s functioning in the public health area. The support study identified that around 20-25% of planned tenders in the Health programme were not implemented following the formal adoption of the 2014 annual work programme and financing Decision. Furthermore, there is an increasing need for a strengthened cooperation with the parent DGs (see the “Coherence” section).
Evidence collected from the annual activity reports combined with interviews and surveys concludes that the Agency has achieved to a large degree its objectives set out in its annual work programmes between 2014 and 2016. For each delegated programme the Agency has managed to be responsive to requirements and has accommodated effectively all the key related changes.
The Agency has effectively produced the planned outputs (i.e. issuing calls, evaluating proposals and administering grants and tenders) and achieved good results in terms of key performance indicators (KPI)
, which were not affected by the extension of its mandate. Over the evaluation period CHAFEA has efficiently managed the delegated programmes and achieved all its targets with respect to the timely evaluation and grant finalisation (Time-to-Inform (TTI), Time-to-Contract (TTC)
and Time-to-Grant (TTG) and timely execution of payments (TTP).
Regarding the Time-to-Inform, the Agency was successful in reaching TTI targets
for all programmes (e.g. for the consumer programme, the average TTI decreased by half, from 94 days in 2014 to 47 days in 2016)
. For the Time-to-Contract, CHAFEA also reached the TTC targets
for the programmes managed (e.g. for the health programme, TTC even decreased from 82 days in 2014 to 68 in 2016). The Agency’s average Time-to-Grant for all programmes was well below the TTG target
(e.g. the shortest average TTG in 2016 was for the consumer programme – 100 days, whereas the longest TTG was for the new AGRI programme – 235 days)
. With respect to the Time-to-pay indicator, CHAFEA’s average TTP remained significantly below the contractual thresholds for all types of payments
.
The Agency was also able to achieve an almost full execution of its operational budget (which ranged from 97% to 100%) during the evaluation period, depending on the year.
Operational commitment appropriations from the EU general budget were almost fully executed (around 99%), while the execution rate for the payment appropriations
constituted 100% in 2014-2015 and 97% in 2016.
The multi-annual residual error rate was below the 2% materiality threshold for all programmes delegated to the Agency (e.g. the estimated residual error rate for the consumer programme decreased from 1.18% in 2014 to 0% in 2016, while for the Health programme was near the 2% threshold in 2014 and 2015, but it decreased to 1.37% in 2016)
.
Over the evaluation period, significant improvements have been achieved in the management of the programmes delegated to the Agency. Through, among others, the simplification of the procedures
, the implementation of new IT tools and modes of service delivery, the new measures introduced to improve the quality of deliverables
, the Agency has effectively contributed to the overall improvement of the management of the programmes entrusted to it (see “Efficiency” section).
Overall satisfaction with the performance of the Agency is high. Around 74% of beneficiaries responding to the survey were satisfied with the services provided by the Agency. In particular, 32% of respondents declared to be very satisfied
. Most beneficiaries and intermediaries of the Agency (95%) further indicated that they would certainly or possibly consider applying for CHAFEA’s calls or tenders again in the future
. The same response was provided by a similar share of CHAFEA’s independent experts: 88% stated that they would certainly consider working with CHAFEA again in the future
. In general, the surveys show that CHAFEA had a positive performance with respect to all the project’s lifecycle stages, including the application, grant finalisation/contracting, project/contract implementation, grant amendment, reporting and payment phases.
In terms of proximity to the addressees, the Agency worked effectively with its beneficiaries and served as a direct contact point between the applicants/beneficiaries of EU co-financing. CHAFEA maintained effective communication arrangements with potential beneficiaries of the programmes in the form of organised stakeholders sessions where relevant information was provided in relation to each programme’s objectives. Furthermore, the Horizon 2020 Participant Portal became the single-entry point for any information on a call, proposal submission and evaluation, grant management, eligibility of costs and other data. The Agency also ensured the relevant professional support to addressees on grant preparation and the use of the new IT tools in the management of Joint Actions under the Health Programme. This has had a practical positive development translated in the sharp decrease of the number of complaints
about technical difficulties received from applicants in 2014 and 2015.
This proximity to addressees is confirmed by survey data. 91% of experts
found information delivered by CHAFEA to be clear and sufficient. 90% of beneficiaries thought that CHAFEA staff was easily available and responsive
. Results regarding the tendering process are less satisfactory. The evaluation process was clear and transparent for only 55% of service providers
. The relatively low satisfaction among service providers was related to certain aspects of the services delivered by the Agency (e.g. the results of the tender evaluation should have been better explained and presented to the tenderers). With respect to service contracts, 66% of service providers thought that CHAFEA gave them useful feedback on the progress of their contract
. 13% of beneficiaries believed that funding opportunities for the programmes are well advertised, showing room for progress in the proximity to all potential addressees
. According to the survey conducted for the evaluation, around 88% of external experts who worked with CHAFEA during the evaluation period mentioned their commitment to working with the Agency again in the future
, being at 94% satisfied with CHAFEA’s responsiveness and at 92% satisfied with CHAFEA’s competence.
Concerning the visibility of the EU as a promoter of the programmes entrusted to CHAFEA, the survey shows that a large majority of beneficiaries (82%)
were aware that CHAFEA was entrusted by the Commission to manage EU projects funded by the EU budget.
5.2.Efficiency
This section considers the relationship between the resources used by the Agency and the output. The analysis, among other factors, also includes analysis of administrative and regulatory burden and looks at aspects of simplification.
During the 2014-2015 period, the programme management cost of the Agency (i.e. the ratio between the administrative budget of the Agency and the operational budget managed) was above 9% and decreased to 8.7% in 2016 and 7.2% in 2017. In other words, the Agency became more cost-efficient in budgetary terms. Also, it is important to note that some programmes’ management tasks delegated to CHAFEA are not related to the operational budget (such as launching calls and evaluation of ‘simple’ programmes under AGRI Promotion Programme).
The new mandate almost doubled the ratio of budget ‘per head’ in the case of CHAFEA
, from EUR 1.36 million in 2013 to an expected EUR 2.49 million in 2020. This indicates that each employee of the Agency managed an increasing volume of the operational budget, which outpaced an increase in the number of the Agency’s staff between 2014 and 2016. However, these efficiency gains remain limited compared to other Executive Agencies. The programme management cost of EASME is only 2,6%, that of REA 3,6% and that of INEA 0,4%. This difference can be explained by the small size of CHAFEA (with an operational budget eight times smaller than the closest other Executive Agency).
Projects managed by CHAFEA are also of a different nature and on average smaller than for other Agencies. The number of running projects ‘per head’ is still 4.31 for CHAFEA, compared to an average of 8.67 for all other Agencies (10.60 for REA but 2.94 for INEA for example). However, the Agency started to implement parts of the promotion of agricultural programme only in 2016, hence the findings of the evaluation regarding the number of projects managed under this programme is limited to this period. CHAFEA also faces particular complexity linked to the diversity of programmes managed as well as the nature of the grant attribution and follow-up procedures
.
A detailed analysis of the CHAFEA cost-efficiency and results of the Cost-benefit analysis are presented under the point 5.4.
Over the evaluation period, the Agency has improved its internal mechanism for project management and implementation and introduced several simplification measures, which were generally positively perceived by beneficiaries, through:
the implementation of IT tools for electronic data management and electronic data exchange between the administration and beneficiaries;
use of Framework Partnership Agreements for operating grants in Health Programme and European Consumer Centres’ grants in the consumer programme;
simplification of financial management using standard cost option;
organisational efficiency & human resource management.
The implementation of Horizon 2020 IT systems by CHAFEA has led to the simplification of procedures related to grants. In addition, IT systems resulted in savings in human resources at all programme management stages: evaluation, award and conclusion of grant agreements, and grant management. Beneficiaries of Joint Actions were able to sign multilateral agreements with 20 or more countries in a much more efficient way. Nonetheless, a certain level of dissatisfaction was reported regarding the administrative burden in cases of small grants and the user-friendliness of the electronic systems used for submitting applications (only 35% users satisfied). The Participant Portal and other Horizon 2020 tools
were also only partly considered sufficiently customised for the programmes administered by CHAFEA
.
For expert management, the survey showed that CHAFEA’s external experts assisting the evaluation process of the proposals were satisfied with CHAFEA’s performance and working with the Agency. The majority of the respondents agreed that the proposal evaluation process was smooth and were in general satisfied with most aspects related to the evaluation process (quality of information and assistance provided by CHAFEA, clarity of the evaluation process, user-friendliness of the IT systems, etc.). Lower satisfaction was reported only in relation to the amount of time allocated for drafting evaluation reports.
For the conclusion of grant agreements, almost all respondents agreed that the CHAFEA staff assigned to the grant finalisation phase were easily available and responsive, while the majority considered that requests from CHAFEA for proposal modification, providing missing information and other issues were clear. However, the contract negotiation process was identified as a potential point for improvement by the beneficiaries interviewed, in line with the perceived rigidity of the Horizon 2020 IT tools in that regard
. Moreover, the implementation of Framework Partnership Agreements for operating grants in Health Programme and European Consumer Centres’ grants in the consumer programme allowed for easier grant management (no need for annual calls for proposals and subsequent procedures) and facilitated long-term cooperation. Finally, the change in processes for project supported under the consumer programme (use of standard cost option for the Exchange of Officials) also reduced the administrative burden both for the beneficiaries and for the Agency. The simplification of procedures led to higher number of processed applications in 2016, from 60 in 2014, to a total of 92 applications processed in 2016
.
The Agency’s workload distribution among units in 2014-2016 deviated from the initial estimates. The 2016 Annual Activity Report further indicates that the workload of some units further intensified in 2016 (ex. for the ‘Promotion of agricultural products’ unit following the effective handover early 2016). With the Agency’s relatively small size
, unforeseen staff mobility created significant workload variations and business continuity challenges. Even though the number of projects ‘per head’ can be considered low in CHAFEA as compared to other Executive Agencies, only 30% of CHAFEA’s perceived their workload as acceptable, against 59% on average for other Executive Agencies. In order to address these challenges, CHAFEA implemented new IT tools
for improving the working processes for grant management and monitoring. However, the Agency should take a more proactive role by working with DG Research and Innovation, notably the Common support centre (CSC), which is responsible for the Horizon 2020 IT tools, to enhance the user-friendliness and functionality of its IT tools and further simplifying the administrative provisions of the delegated programmes.
Overall, the Agency’s consistent good performance on key performance indicators suggest CHAFEA managed to cope with fluctuating workload levels (good results in time-to-grant indicators, high satisfaction of beneficiaries consistently over the period), thus demonstrating a high flexibility, both at operational and administrative levels. Through the services provided to its final beneficiaries, the Agency contributed to increase the efficiency in the implementation of EU programmes, and thus to contribute, in general, to the positive EU image through a good visibility of the Commission’s action.
CHAFEA’s internal organisation and procedures were instrumental for the performance of the tasks delegated to the Agency during the reference period. CHAFEA’s HR policy reached some notable results: around 72% of CHAFEA staff reported that their skills match their current job
.
The 2016 Commission staff opinion survey however reported an overall lower staff wellbeing compared to other Executive Agencies, and a decrease in staff engagement
. CHAFEA staff also perceived as less good its access to learning and development opportunities compared to other Agencies.
Following the results of the survey, the Agency implemented several measures to improve its performance in critical areas. It adopted its first multi-annual HR strategy for 2017-2020 as well as a specific action plan to follow up the survey results, all meant to create and maintain a high level of expertise for an optimal and efficient performance of the tasks entrusted to it. CHAFEA’s efforts to address its HR management challenges were notable and have positively contributed to the management of the delegated tasks and the objectives of the Agency. Constant efforts will be needed to continue addressing the Agency’s HR requirements, in particular the under-performing areas when benchmarking against the other Executive Agencies and the Commission. The evolution of key indicators for staff satisfaction is however encouraging. In the latest staff opinion survey (i.e. from the 4th quarter of 2018)
, CHAFEA managed to increase the results of the indicator on well-being to 53%, compared to 38% in 2016. Along the same lines, the staff engagement index for 2018 was 63% (against 55% in 2016).
Staff turnover was relatively stable and constituted 6% on average during the evaluation period. The occupation rate maintained the target above in 2015 (96% compared to the target of 95%).
The management of the programmes delegated to CHAFEA is covered by its three operational units, each with its corresponding programme: Health unit, Consumers and Food Safety unit, and Promotion of Agricultural Products unit. The Agency has also a fourth horizontal administration unit. The working methods of each operational unit were adapted to the specific needs of each programme. In the public health unit, the work was organised according to different policy areas, where it was particularly efficient, as it allowed for increased specialisation. In other units the work was grouped together by funding instruments, which proved especially useful for consumer policy as it afforded greater operational flexibility. Moreover, several internal networks were set up to facilitate the exchange of information between the units with different levels of experience and workload.
The Agency’s relatively small size has obvious consequences on the Agency’s ability to take on additional responsibilities, or to maintain knowledge and business continuity when key staff unexpectedly leaves the Agency or changes assignment internally. Along the same lines, it also has implications for horizontal functions (e.g. HR, finance, IT, administrative and legal support). Several internal working groups have been set up within the Agency to ensure the exchange of information and overcome this risk to some extent.
5.3.Coherence
Coherence looks at any overlaps and complementarities within the programme portfolio managed by CHAFEA or delimitation of responsibilities between CHAFEA and its parent DGs.
The Agency manages a diverse portfolio of programmes, which are supervised by different DGs. Each requires a certain degree of subject-specific knowledge, but also makes use of different forms of financial instruments: Better Training for Safer Food, for example, relies heavily on procurement; while the public health and consumer programmes rely more on grants and Joint Actions. The units are also organised differently, with some units organised according to knowledge of specific instruments (such as the Agricultural Products Unit) and others according to thematic knowledge of specific programmes (such as the Health Unit).
There seems to be relatively little overlap between the programmes managed by CHAFEA, both in terms of the area covered and the types of financial instruments used for implementation. This triggered altogether a rather heterogeneous/non uniform organisation of the different units within the Agency. Therefore, more efforts should be made towards the achievement of a more systematic and harmonised management of the programmes.
In 2014, CHAFEA was entrusted with the promotion of agricultural products programme, which included the incorporation of a new unit. This had an impact on the overall coherence of CHAFEA’s programmes, with the introduction of tasks that require a certain degree of subject-specific knowledge. However, interviews suggest that this process was in general well managed, with the inclusion of Agency staff from other units in the new unit.
In general, the Agency’s reorganisation procedures and arrangements brought by the 2014-2020 multiannual financial framework and the extended mandate worked well. Some inconsistencies appeared in relation to the overall communication and collaboration procedures between the Agency and the parent DGs, which are different from one unit to the other. Other aspects, such as staff changes within the Commission and the Agency, including the additional structural changes, affected the overall coherence of certain programmes
.
Overall, the Agency has been able to provide timely information when requested and it has also made efforts to maintain good communication channels with its parent DGs. A clear distribution of tasks and responsibilities between CHAFEA and its parent DGs, as presented in the MoU, is summarized in in the table available below
.
|
|
|
Implementation level, e.g.
- Implementing programmes/initiatives
- Establishing and implementing the control strategy
- Programme/project implementation monitoring
|
Political and policy level, e.g.
- Strategy setting
- Programming, financing decision
- Programme policy monitoring/evaluation
|
Communication at programme/project implementation level, e.g.
- Publicity of the Calls
- Dissemination of application forms and guide for proposers/tenderers
- Dissemination of action’s results
|
Communication and coordination at policy/programme setting level, e.g.
- Intra-DG and Cabinet communication
- Inter-institutional communication
- Inter-service consultations
- Dissemination of Programme policy results
- Communication with/to Member States, Third Countries
|
In general, there seems to be a clear division between the administrative responsibilities of CHAFEA and the policymaking responsibilities of the Commission. Some uncertainties existed concerning the role of the Agency in the implementation of administrative tasks related to potentially highly relevant policy issues (ex. high-level missions of the promotion of agricultural products programme). The MoU signed between CHAFEA and its parent DGs, does not provide a definition of what could be classified as highly relevant policy topics. This matter could be addressed by adopting clearer provisions and concrete actions in the MoU.
Other challenges referred to the monitoring of deliverables in Joint Actions. Parent DGs duplicate work by controlling the quality of deliverables produced by CHAFEA
. Looking forward there is potential for improving effectiveness in the operation of CHAFEA by further simplifying the management of the programmes and strengthening collaboration with its parent DGs. Interviews with parent DGs highlighted the need for the Agency to be more involved and dedicated to activities related to the development of terms of reference, the evaluation of proposals and monitoring of project outputs. For example, the Terms of Reference for procurement contracts are often drafted by the parent DGs, which should simply provide a “technical input” according to the MoU. In some cases, there is also direct contact between contractors and Commission officials (ex. questions on technical content for the BTSF programme), where CHAFEA should have been the contact point.
As indicated in CHAFEA’s 2016 Annual Work Programme, the Agency’s results in the form of implementing the delegated programmes and carrying out studies are contributing to achieving the policy objectives set by the Commission at political level
.
With respect to grants, “[…]
the Agency is requested to consult the Parent DG before taking appropriate action in cases where possible changes could significantly impact on the programme objectives and have an important policy impact.” Concerning procurement, the Agency is responsible for implementing all procurement procedures delegated to it
. The management of and the knowledge about a range of financial instruments within CHAFEA provides flexibility in choosing the right instrument to meet the specific needs of the Commission. This is particularly important when there are urgent requests which were not foreseen within the annual work programme
.
The Agency and its parent DGs used various procedures to ensure a good interaction, coordination and information flows include contacts at different levels, through: the Steering Committee meetings with all parent DGs, participation in the parent DGs’ management meetings, regular programme-specific meetings, and further contacts at unit and working level
.
Cooperation mechanisms worked well in general. The division of roles and responsibilities for ongoing coordination was set up in the MoU. However, more systematic communication could have avoided some overlaps such as the Agency’s intervention in the implementation of certain tasks that are related to highly relevant policy topics.
Formal coordination and exchange of information between the Agency and its parent DGs takes place through its Steering Committee, at least four times a year, ensuring in this way a broad coherence of the overall activities. CHAFEA's activities are monitored by the means of participation of the parent DGs in the Steering Committee
.
At a more informal level, the study found that communication between the Agency and Commission colleagues working on the same portfolio is generally going well. A couple of differences were noted in respect to the organisation of the methods preferred for communication and collaboration. In DG SANTE, there are 11 units working with CHAFEA. Some of these units require meetings every week, others every two months, others when a problem arises. Agency staff suggested meeting on a more regular basis, even when (other more) formal meetings were not foreseen. The perceived problem of the location of the Agency in Luxembourg, seen as an impediment for day-to-day informal communication, has been partially solved by ensuring regular visits to Brussels (from some Agency’s senior staff members) in order to achieve a more efficient coordination of activities.
On top of this, parent DGs were granted access to CHAFEA’s internal project database, in order to maintain adequate know-how sharing with the Commission and monitoring of ongoing projects. This provided summaries of all projects CHAFEA is managing, as well as the tracking of KPIs for each project.
The Commission found that the information provided is not entirely fit for purpose as it is too granular and does not give clear and comprehensive overview of progress towards policy objectives. Information made available to the Commission was rather related to KPIs at project level and did not necessarily contribute to policy-related tasks (see assessment question on information and communication). Moreover, the large quantity of information from the internal project database is not processed and analysed by CHAFEA to provide the parent DGs with policy results. In terms of dissemination of the results of programmes, interviews conducted in parent DGs suggested CHAFEA only provides limited feedback to inform policymaking.
Any parent DG may draw up or require the Agency to draw up reports based on data and structured information available in these IT systems
. Finally, if provided for in the Agency’s annual work programme, CHAFEA also disseminates the results of projects/initiatives in accordance with parent DG’s communication strategies
. The Agency implemented new ways of service delivery to share information and knowledge more easily, especially in relation to the BTSF initiative
. However, interviewed officials in the parent DGs still point out the limited dissemination of results by CHAFEA, which could have a larger outreach.
Seconded officials from the Commission play a key role in ensuring the continuous exchange of institutional knowledge between the Executive Agencies and the Commission. But in this particular case, new secondments were very limited (i.e. only two cases during the evaluation period, with just one relevant case given the previous time of professional experience in DG AGRI). This situation was the result of the legal limitations on the ratio of seconded personnel
.
5.4.Retrospective Cost-benefit analysis
CHAFEA’s costs were higher than initially anticipated. The actual costs of the executive agency scenario were 23% higher compared to the initial cost-benefit analysis estimations
and 15% higher compared to the specific financial statement estimations. This primarily resulted from staff related expenditure as CHAFEA’s average staff costs were underestimated in the 2013 cost-benefit analysis: contrary to what was forecasted, CHAFEA had higher staff costs compared to other Executive Agencies. The location of CHAFEA in Luxembourg means that very few local staff were recruited, and that the number of employees receiving expatriation benefits is thus higher than for other Agencies
.
While the ex ante 2013 cost-benefit analysis necessarily relied on incomplete data and extrapolation, the 2014 specific financial statement estimation failed to update CHAFEA’s costs using new data that had become available. If updated costs would have been used in the specific financial statement for 2015-2016, the actual costs of the executive agency scenario would have been 2.5% lower than the specific financial statement estimations. The specific financial statement indeed used CHAFEA’s 2014 draft budget data to update the (increased) average staff costs for that year, but kept average cost estimations provided from the 2013 cost-benefit analysis for 2015-2020, where the 2014 data showed that these hypotheses could not be sustained anymore.
Compared to a situation where the Commission would have performed those tasks within its departments, CHAFEA’s costs were slightly lower. Over 2014-2016 the actual cost savings of the executive agency scenario were only of 1.8% compared to the in-house scenario while for other Executive Agencies savings can get close to 30%. Comparing the savings initially estimated in the ex ante cost-benefit analysis with the actual savings from the delegation of tasks to CHAFEA, the study found that the actual savings during 2014-2016 period were substantially lower than initial estimations (EUR 0.4 million compared to EUR 4.4 million for the 2013 cost-benefit analysis and EUR 2.3 million for the Specific financial statement estimates)
.
Figure 1. Estimated savings of the executive agency scenario in 2014-2016 (Title I and Title II expenditure), EUR
Source: compiled by PPMI.
As forecasted in the ex ante cost-benefit analysis and the Specific financial statement, savings resulting from the creation of CHAFEA came from a higher share of lower cost external personnel (contract agents) employed within the Executive Agency. However, the actual staff costs of CHAFEA’s temporary and contract agents were higher than what had been initially estimated, based on estimated Commission’s average staff costs (see Figure 2). This limited actual savings. CHAFEA’s infrastructure and operating expenditure was also higher than expected in 2015-2016 due to the cost of rent being higher in Luxembourg compared to Brussels. On the contrary, the costs of coordination and monitoring (Programme support expenditure) were in fact 9% lower than the ex ante cost-benefit analysis estimates
.
fIGURE 2. CHAFEA’s average staff and overhead costs 2014-2016, EUR
|
|
Initial cost estimations
|
Actual cost
|
|
|
CBA, SFS for 2015-2016
|
SFS for 2014
|
2014
|
2015
|
2016
|
|
Average staff costs: TA
|
106 294
|
129 650
|
135 975
|
134 089
|
126 203
|
|
Average staff costs: CA
|
49 672
|
71 700
|
62 024
|
62 691
|
64 467
|
|
Overheads
|
20 344
|
19 150
|
19 642
|
27 751
|
23 065
|
Source: compiled by PPMI.
6.Conclusions
CHAFEA achieved effective management of the delegated programmes while operating according to the legal framework governing it. For each programme the Agency has proven to be responsive and flexible to requirements and has timely accommodated most of the key related changes. Its objectives in terms of key performance indicators (Time-to-grant, Time-to-contract, etc.) were attained. Moreover, beneficiaries and experts show a large satisfaction with the operation of CHAFEA. In particular, the Agency achieved its objectives in terms of proximity to the addressees and of visibility of EU programmes. During the 2014-2016 period, CHAFEA improved its processes by adopting various simplification measures and IT tools.
In terms of efficiency, CHAFEA manages lower amounts of grants and budget per capita than other Agencies, implying higher programme management costs. The Agency nonetheless managed to reduce its programme management costs from 9% to 7% of the total operational budget. The 2013 cost benefit analysis estimated that the creation of CHAFEA would reduce costs by 4.4 million euros over the 2014-2016 period. The actual cost of the Agency was higher than initially anticipated and real cost savings through the creation of CHAFEA amounted to 0.4 million euros.
The staff opinion survey of 2016 had revealed lower than average staff satisfaction results for CHAFEA in areas such as staff engagement, staff wellbeing, professional future and middle management. The Agency has actively addressed these shortcomings in its 2017-2020 multi-annual HR strategy and dedicated survey follow-up action plan. However, the new staff opinion survey results of 2018 show that the perception of key areas of work has significantly improved.
The Memorandum of understanding between CHAFEA and its parent DGs set a clear definition of tasks between the Agency and the Commission, which was overall observed over the evaluation period. However, a lack of clarity persists in relation to the Agency’s role in the implementation of certain tasks that are notably related to highly relevant policy topics. In spite of the location of the Agency in Luxembourg, which limits informal contacts, communication channels between the Agency and the Commission are well established and performing well.
Finally, a number of ongoing work-streams are already contributing to address the above findings of the evaluation. More precisely, an action plan was prepared in order to enable the Agency to work more efficiently across its objectives and programmes. In collaboration with the parent DGs, the actions are focused on the simplification of the management of the programme
, the further improvement of its HR management, the timely adoption of the annual work programme
and the proper planning of the activities, the exchange of information and the feedback to the parent DGs on the programme implementation
, as well as the revision of the MoU to better define roles and responsibilities between the Commission and CHAFEA
.
Annex: Evaluation Questions
EFFECTIVENESS – to what extent has the objective of delegating programme management tasks to CHAFEA been achieved?
To what extent was CHAFEA operating according to the legal framework establishing it?
To what extent has CHAFEA achieved its objectives? What, if anything, could be done to render CHAFEA more effective in achieving these objectives?
To what extent has CHAFEA contributed to improved management of the programme(s) in terms of the elements assessed in the 2013 Cost-Benefit Analysis
EFFICIENCY – to what extent has CHAFEA carried out its tasks efficiently in line with estimations?
Did the actual costs (including cost of coordination and monitoring) of CHAFEA correspond to the estimates of the 2013 Cost-Benefit Analysis which was carried out in view of its establishment? If not, what are the reasons behind?
Did the actual benefits correspond to the estimates of the 2013 Cost-Benefit Analysis? If not, what are the reasons behind?
Was the management and execution of the programme(s) by the Agency cost-effective as compared to the alternative options assessed?
To what extent has CHAFEA contributed to improved management of the programme(s) in terms of simplification of the procedures and flexibility in the implementation of outsourced tasks (e.g. capacity to adapt to periods of high workload)?
Which further scope for simplification exists?
What aspects/means/actors or processes render CHAFEA more or less efficient? What could be improved?
To what extent has CHAFEA’s internal organisation and procedures been conducive to its efficiency?
Is the size of the Agency adequate? Is the balance between operational staff and administrative support appropriate?
COHERENCE
Are there overlaps/ gaps/ inconsistencies within the programme portfolio managed by the Agency?
Is there a clear and appropriate delimitation of responsibilities and tasks between CHAFEA and the parent-DGs? Are there overlaps or gaps?
To what extent has CHAFEA enabled the Commission to better focus on its policy related tasks?
Are there any governance (financial and policy) issues in relation to the implementation of the tasks that are delegated to the executive agencies/ issues as regards the Commission’s services able to steer EU policy or budget implementation?
Are appropriate mechanisms and instruments put in place to ensure an adequate coordination and information flow between CHAFEA and the Commission services?
Does CHAFEA provide useful information in support of the policy process?
