EUROPEAN COMMISSION

Brussels, 29.4.2020

SWD(2020) 78 final

COMMISSION STAFF WORKING DOCUMENT

Evaluation of the Education, Audiovisual and Culture Executive Agency (EACEA)

Accompanying the document

Report from the Commission to the European Parliament, the Council and the Court of Auditors

Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Executive Agency for Small and Medium-Size Enterprises, the Innovation and Networks Executive Agency, the Education, Audiovisual and Culture Executive Agency, the Research Executive Agency and the European Research Council Executive Agency

TABLE OF CONTENTS

1.    Introduction    

2.    Background    

2.1.    Description of the intervention and its objectives    

2.2.    Baseline and points of comparison    

3.    Implementation / State of play    

4.    Method    

4.1.    Short description of methodology    

4.2.    Limitations and robustness of findings    

5.    Analysis and answers to the evaluation questions    

5.1.    Effectiveness    

5.2.    Efficiency    

5.3.    Coherence    

5.4.    Retrospective Cost-benefit analysis    

6.    Conclusions    

ANNEX - Evaluation questions    

1.Introduction

The purpose of this evaluation is the periodical (triennial) evaluation of the operation of the Education, Audiovisual and Culture Executive Agency (hereinafter EACEA). The EACEA is governed by:

·The Framework Regulation for executive agencies 1 ;

·EACEA Act of Establishment 2 , which sets out its mandate;

·EACEA Act of Delegation 3  and its successive amendments, specifying the tasks to be carried out by, and the powers delegated to, the Agency in order to perform its mandate;

·The decision appointing the members of the EACEA Steering Committee 4 .

In line with the Commission’s better regulation principles 5 , the evaluation applies several standard evaluation criteria. The evaluation assesses whether the Agency has fulfilled its tasks in an effective and efficient way, whether there are overlaps / gaps / inconsistencies in the management of the programme portfolio by the Agency, and whether there is a clear delineation of tasks between EACEA and the parent DGs (coherence).

The evaluation assesses whether the functioning of the Agency has yielded the expected positive results as estimated in the cost-benefit analysis for delegating task to the Agency 6 and identifies potential areas of improvement. To this end, the estimations of the cost-benefit analysis of 2013 have been tested to provide evidence on the validity of the assumptions made in the ex ante scenario by considering the actual costs and benefits of programme implementation by the Agency in a structured way. The aspects to be covered by the cost-benefit analysis are specified in Article 3(1) of the Framework Regulation 7  and the Guidelines on establishing and operating executive agencies 8 .

The evaluation does not cover the achievements of the programmes managed by EACEA, which are subject to mid-term and ex post evaluations themselves. The evaluation of EACEA nevertheless provides useful input for these programme evaluations, considering that the performance of the Agency has an impact on the efficiency and effectiveness of the programmes it manages.

The evaluation examines the efficient use of resources and the effective achievement of the tasks entrusted to it. In particular, it looks at whether:

·The alignment of programme portfolios with the Agency’s core competences and its brand identity delivered the estimated qualitative benefits;

·The assembly of the management of different EU programmes delivered the estimated synergies, simplification and economies of scale;

·The pooling of instruments guaranteed consistent service delivery;

·There is scope for simplification and further efficiency gains.

This evaluation – the fourth one since the establishment of EACEA – covers the period 2015-2017. The evaluation is supported by a study carried out by an external contractor 9 .

The results of this study are summarised hereafter and this evaluation will be presented to the European Parliament, to the Council and to the Court of Auditors in accordance with Article 25(1) of the Framework Regulation.

The results will feed into the reflection to assess the opportunity of expanding/ modifying the Agency's mandate in view of the delegation of the implementation of 2021-2027 EU programmes.

2.Background 

2.1.Description of the intervention and its objectives

The outsourcing of certain management tasks to the executive agency according to the Framework Regulation and EACEA’s Establishing Act aims at:

·Allowing the Commission to focus on its institutional tasks, i.e. tasks assigned to the institutions by the Treaty, which require discretionary powers in translating political choices into actions and should not be outsourced.

·Enabling to achieve the goals of EU programmes more effectively. According to the cost-benefit analysis carried out in 2013, delegating tasks to the Agency was estimated to be more cost-efficient than an in-house scenario. The Act of Establishment estimated that the alignment of more coherent programme portfolios with the Agency’s core competences and its brand identity would bring qualitative benefits. In addition, it projected that assembling the management of different EU programmes would bring synergies, simplification and economies of scale. 

The original objective of the intervention is entrusting the Agency with the implementation of several EU programmes in parts or fully:

·Erasmus+, including activities under external policies,(DG EAC) 10

·Creative Europe, (DG EAC, DG CNECT) 11

·Europe for Citizens, (DG HOME) 12

·EU Aid Volunteers, (DG ECHO) 13

·European Solidarity Corps (DG EAC) – Programme created (and delegated to EACEA) in 2018 14

·Actions under direct management via grants and other forms of funding.

EACEA also implements the legacy of the predecessor programmes and actions.

Over the evaluation period, and in relation to these programmes, EACEA was entrusted with the following tasks:

·Monitoring projects, carrying out the necessary checks and recovery procedures, performing budget implementation tasks covering revenue and expenditure:

–Awarding grants and managing the ensuing agreement or decision, including the operations required to launch and conclude grant award procedures;

·Providing support in programme implementation, in particular:

–Collecting, processing and distributing data, in particular compiling, analysing and transmitting to the Commission all information required to guide the implementation of the programme, promote coordination with other Union programmes, the Member States or international organisations;

–Contributing to assessing the impact of the programme and to monitoring the actual effect of measures taken;

–Keeping a direct link with the target public of beneficiaries and recipients;

–Cooperating in the Commission’s own information and promotion campaigns;

–Contributing to studies and evaluations, in particular the annual and/or mid-term evaluations of programmes implementation; contributing to the preparation and the implementation of follow-up actions to evaluations;

–Preparing recommendations for the Commission on the implementation of the programme;

–Ensuring control and producing supervision data;

–Participating in preparatory work on work programmes and financing decisions;

–Managing technical support services. 

2.2.Baseline and points of comparison 

The current evaluation of EACEA operations during 2015-2017 assesses the actual costs and benefits of programme implementation by EACEA (executive agency scenario) when compared with the alternative scenario of management by the Commission services (in-house scenario).

Accordingly, the reference point for the present EACEA evaluation is the 2013 ex ante Cost-benefit analysis, and the specific financial statement 15  (SFS) of EACEA.

Wherever possible, the analysis of the performance of EACEA during the period of reference assesses the progress compared to the previous evaluation, covering the Agency’s operations during 2012-2014.

According to the SFS of EACEA prepared in 2013:

·The estimated efficiency gains of EUR 57,1 million over the period analysed;

·The total number of full time equivalents (FTEs) required to manage the relevant programmes in year 2020, the peak programming year in terms of workload, was estimated at 442 for EACEA 16 ;

·The total initial operational budget 17  entrusted to EACEA in 2020 was estimated at around EUR 870 million in commitment appropriations and EUR 778 million in payment appropriations;

·Considerable efficiency gains were expected as compared to the in-house scenario over the period, along with non-quantifiable benefits such as improved quality of programme management and service delivery, improved visibility of the EU programmes and proximity to beneficiaries.

3.Implementation / State of play 

The Agency’s mandate was extended by the Commission Implementing Decision No 2013/776/EU, which came into effect on 1 January 2014. Activities delegated to the Agency were expanded to include the following:

·Creative Europe, which replaced the MEDIA, MEDIA Mundus and Culture programmes;

·Erasmus+, which replaced seven programmes of the previous period (the Lifelong Learning Programme, Youth in Action, Erasmus Mundus, Tempus, Alfa, Edulink and a programme for cooperation with industrialised countries);

·Europe for Citizens, which is a continuation of the previous programme;

·EU Aid Volunteers (an initiative for the cooperation of volunteers and organisations from different countries that was delegated to the EACEA for the first time in 2014);

·Projects in the field of higher education under the multiannual financial framework regarding the financing of EU cooperation for African, Caribbean and Pacific States and Overseas Countries and Territories for the 2014-2020 period (10th European Development Fund).

As regards the activities delegated to EACEA, over the 2015-2017 period the Agency had four extensions of its mandate 18 . Comparison with other executive agencies reveals that the EACEA had the largest number of changes in its mandate and had to demonstrate its flexibility in accommodating them.

The new mandate also brought changes in the governance of the Agency as new parent DGs joined the Steering Committee (currently DG EAC, DG CNECT, DG HOME 19 , DG ECHO), and subsequently signed a Memorandum of Understanding (MoU) with EACEA 20 . The new MoU defined the modalities and procedures of interaction between EACEA and its parent DGs. In addition, it brought a clearer delimitation of the administrative activities between the Agency and its parent DGs. The distribution of tasks and responsibilities, as presented in the MoU, is included in section 5.3 of this document. Compared to this reference, this evaluation assesses whether this distribution of roles has been respected for the 2015-2017 period, and the effects of the intervention on the coherence and the efficiency of EACEA’s action.

The operational budget of the EACEA allocated to grants under 2014–2017 calls 21 amounted to EUR 2.35 billion (of which EUR 71.3 million related to EFTA, third countries’ and European Development Fund contributions) for the programmes listed under chapter 2 above.

The Agency’s workload is closely linked with the allocated operational budget, the number of proposals received and the number of grants concluded/projects managed. During 2014-2017, the actual number of proposals received by the EACEA was 22% higher compared to the CBA estimates.

According to the study supporting the evaluation and based on a thorough review of activities carried out during the reference period, the overall number of grants concluded under the 2014-2017 calls 22  was 12% higher compared to the CBA estimates. In detail:

·19% higher for Erasmus+;

·17% higher for Creative Europe;

·20% lower for Europe for Citizens 23 .

On the other hand, the budget per operational head (in commitments) increased from EUR 1.35 million in 2015 to EUR 1.61 million in 2017. On average, EACEA’s staff managed an increasing volume of the operational budget during this period.

EACEA remained substantially stable in size in accordance with the multi-annual planning stated in the SFS with 436 actual staff members in 2017 24 . The Agency is based in Brussels.

A series of organisational and procedural changes started with the implementation of the programmes under the multiannual financial framework (MFF) of 2014-2020:

·Delegation of the management of additional programmes, new funding instruments and as a consequence new tasks and activities for the Agency 25 .

·Further simplification and improvement of grant management processes and procedures, including through new business processes and IT tools. 

4.Method

4.1.Short description of methodology

The evaluation was supported by a study carried out by external contractors 26 . The methodology of the EACEA evaluation was consistent with the approaches employed for similar parallel evaluations of ERCEA, EASME, REA, INEA and CHAFEA, and built on the previous evaluation of EACEA operations during 2012-2014. Specific methodological approaches were used to ensure data triangulation (confirmation from different sources). The study was structured around a series of evaluation questions available in Annex.

Methods used for the analysis includes:

·Desk research (in the form of a review of relevant documents and administrative data);

·Statistical analysis of the EACEA’s administrative and monitoring data;

·Survey programme;

·Interview programme;

·Retrospective cost-benefit analysis;

·Case studies/focus areas;

·SWOT analysis;

·Comparative analysis and benchmarking.

Desk research carried out for the study consisted of two interrelated components – a review of relevant documents and an analysis of statistical and monitoring data collected by the EACEA. The evaluation relied also on the results of the 2016 staff opinion survey.

As regards the survey and interview programmes, an extensive consultation was carried out to collect the stakeholders’ views on the functioning and outputs delivered by the Agency. This consisted of an extensive interview programme and two surveys:

·30 semi-structured in-depth interviews (44 persons interviewed in total) with Commission and EACEA staff, including: members of the EACEA Steering Committee; selected Commission staff in the parent DGs and in other DGs; Management of the EACEA; Representatives of the EACEA Staff Committee, other employees;

·37 interviews with unsuccessful applicants and beneficiaries of different programmes managed by the EACEA who, while answering the survey, provided their agreement to be further contacted;

·3 surveys targeting EACEA’s beneficiaries and unsuccessful applicants (overall 4879 answers or 26% response rate), and experts (700 responses or 63% response rate). 

Comparative analysis and benchmarking of the Commission’s various executive agencies was also carried out against a set of (qualitative and quantitative) indicators. This enabled the evaluation team to compare the performance of the EACEA effectively with that of other executive agencies.

4.2.Limitations and robustness of findings

The range of stakeholders consulted and the size of the sample within each group make the consultation statistically significant. However, compared with the overall population reached, unsuccessful applicants participated less actively in the survey than programme beneficiaries. Furthermore, unsuccessful applicants and beneficiaries of the Erasmus+ programme were the most active group in answering the survey as compared to other applicants and beneficiaries contacted by the Agency.

No sampling bias was observed as the profile of the respondents to the surveys was very similar to the overall population, guaranteeing statistical representativeness. The non-response bias (not all characteristics of the group that did not reply had been captured in full) was mitigated through triangulation with the results of follow up interviews. The triangulation approach, using multi-level and multi-stakeholder dimension in the data collection, ensured the robustness and reliability of the data and information used to draw up conclusions in the supporting study.

Survey results were used extensively to answer the evaluation questions on effectiveness and efficiency of the Agency. Results of the interviews with the Commission and the EACEA representatives were especially relevant for answering coherence questions. Whereas interviews with applicants and beneficiaries mostly informed efficiency questions, interviews provided both first-hand factual information and opinions, and allowed to contextualise and explain the data collected through surveys and desk research. Interviews also informed preparation of some of the case studies/focus areas. Some views from Commission and Agency staff gathered from the interviews and mentioned in the study illustrate personal experience, although they may not be representative of overall relations between the Commission and the Agency.

5.Analysis and answers to the evaluation questions

This section is structured around the evaluation criteria, effectiveness, efficiency and coherence. Findings are presented by evaluation criteria and, within each criterion, by the individual questions that form the basis for this evaluation.

5.1.Effectiveness 

For the purpose of this evaluation, effectiveness relates to how successful the Agency has been in achieving or progressing towards its objectives.

Legal framework

Mandate of the Agency

The key documents defining EACEA’s current legal framework are its acts of Establishment 27 and Delegation 28 , as well as guidelines on executive agencies 29 . During the evaluation period, the Agency implemented its delegated tasks as defined by the legal framework.

The core responsibility of the EACEA is to manage most (or all, depending on the programme delegated) of the phases of programme management and stages in the lifetime of projects and operating grants, including:

·Calls for proposals;

·Project evaluation and selection;

·Conclusion of grant agreements;

·Financial management;

·Information and support to potential applicants, applicants and beneficiaries;

·Monitoring of projects;

·Feedback to parent DGs and dissemination;

·Calls for tender (e.g. training, insurance);

·Management of framework and service contracts.

Interviews carried out during this evaluation indicate that stakeholders are generally satisfied with the overall effectiveness of the Agency and how the tasks delegated to it are implemented.

During the evaluation period, there were no changes in the legal framework itself, except for the updated mandate of the Agency. Updates occurred to accommodate the new policy needs of the Commission, with EACEA taking on board new actions delegated through extensions of its mandate, with particular regards to new activities delegated to the Agency under the Erasmus+ programme. During the interviews with the Commission and the EACEA some concerns were raised that the current legal framework is quite burdensome and not flexible.

For example, the introduction of even minor changes in the mandate of the Agency creates the need for a lengthy legal process. Given that in case of the EACEA these changes in its mandate are quite frequent (due to large number of actions managed / number of parent DGs), dealing with this administrative procedure requires a considerable amount of time and resources.

Financial Regulation

An important issue has been however identified concerning the compliance to the Financial Regulation in relation to the management of programmes. This non-compliance was identified by the Internal Audit Service (IAS) in its 2017 audit on Grant Management - Phase I (from call publication to contract signature) for Erasmus+ and the Creative Europe programmes. The IAS recommendations identified the need for actions on the control environment for grants, the evaluation process, evaluation committee and the role of experts, as well as the contracting phase. As a consequence, the Agency introduced a reservation in its Annual Activity Reports for 2017 and 2018. All the necessary corrective actions were taken in the course of 2018, so that at the beginning of 2019 all recommendations were closed 30 and the Agency has lifted up its reservation in the 2019 Annual Activity Report. 

Implementation of delegated programmes and support services

EACEA’s tasks and objectives were set out in its Commission Decisions and Acts of Delegation. Priorities of the year, objectives, description of activities and the expected results are outlined in the work programmes and plans of the delegated programmes. Each year the annual work programme of the EACEA defines several operational priorities with the specific KPIs. Operational priorities set in the annual work programmes of the EACEA between 2014 and 2017 are linked to their parent DGs’ objectives and were quite stable.

At the beginning of the MFF 2014-2020, an emphasis was placed on the successful launch of new programmes and timely selection and contracting processes, including comprehensive guidance for beneficiaries of new programmes, while continuing to ensure the financial management and monitoring of projects of the previous generation of programmes under the respective rules.

Since 2016, when the implementation of new programmes was streamlined, the emphasis has been placed on the added value of the Agency. The need for enhancing knowledge sharing and good practice has started to be underlined since 2015. Providing policy support to the Commission remained a stable priority from 2014 to 2017.

Overall, four operational priorities of the EACEA guided its operations in 2015-2017:

·Adapt to the new structure of the Commission (relevant in 2015);

·Improve performance in programme management;

·Reinforce policy support to the Commission;

·Enhance knowledge sharing and good practice.

EACEA’s Annual Activity Reports confirm that the Agency has been effective in implementation of its operational priorities. The Agency was establishing communication and setting certain practices in place to adapt to the new Commission structure and accommodate the needs of its new parent DGs. It was constantly introducing various measures to improve performance in programme management. It emphasised the importance of reinforcing policy support to the Commission and enhancing knowledge and good practice sharing and looked for the ways to respond to these needs.

The evaluation of the EACEA’s performance is primarily based on the analysis and interpretation of the key performance indicators related to:

·Timely execution of the delegated functions;

·Cost-effectiveness of the management and control arrangements;

·Effectiveness of the established supervisory and control systems in ensuring the legality and regularity of the programmes;

·Budget execution of commitment and payment appropriations.

Evaluation and selection of proposals, conclusion of grant agreements

The main indicators describing the timeliness of the evaluation and the contracting process are the following:

During the evaluation period, the EACEA successfully met all targets. The increase for TTG in 2017 compared to the previous period was related to the revision of the TTG calculation methodology, which was aligned to the methodology specified in the Vademecum for Grant Management.

EACEA’s performance in terms of TTI and TTC

Source: PPMI based on EACEA’s Annual Activity Reports.

The increased TTI and TTC in 2017 results from a revised calculation methodology of these indicators.

EACEA’s performance in terms of TTG by programme

Note: TTG for 2017 is based on the revised calculation methodology

Source: PPMI based on EACEA’s Annual Activity Reports.

Even though EACEA met the KPIs related to timely evaluation and selection of proposals and conclusion of grant agreements, in surveys and interviews applicants and beneficiaries expressed their desire for an even quicker evaluation and selection process.

EACEA’s performance can also be measured in terms of the quality and timeliness of information and guidance. The survey revealed that applicants and beneficiaries generally understood well the application and selection process (85% of respondents), the principles of the selection procedure (79%) and the priorities and objectives of the Programme Guide (81%). The respondents were generally positive on the clarity and ease of finding of information material and agreed that the provided information enabled them to understand the application procedures. A somewhat lower satisfaction level (73%) was recorded in relation to the user-friendliness of the Programme Guide for applicants 31 .

On the IT side, the survey revealed that the perception of beneficiaries and applicants as regards the proportionality of administrative requirements and user-friendliness of the IT tools remained similar to the previous evaluation. A total of 70% of respondents agreed that the requirements for the application process are reasonable and proportionate, though a lower share of respondents (66%) was satisfied with the user-friendliness of the Participant Portal 32 . Interviews with beneficiaries and unsuccessful applicants also revealed that certain parts of the application procedure are seen as overly complex: while the process itself was called ‘straightforward’ and ‘logically coherent’, some of the applicants complained about the application forms, which are said to be labour- and time-consuming.

Lastly, interviews revealed that applicants are generally grateful for feedback on their application and find the comments and suggestions helpful for their future applications, even if they do not necessarily agree with the evaluators’ opinion and would, in some cases, appreciate even more in-depth comments.

The evaluation and selection of proposals was impacted by the conclusions of the 2017 IAS audit on Grant Management referred to above. Following this report, changes were introduced in the evaluation procedures for the 2018 calls for all the programmes managed by EACEA, which led to improved controls over potential cases of conflicts of interests among external experts. The revised evaluation procedures also brought about a redefinition of the role of evaluation committees as evaluators of all proposals, which implied a more active role of staff from the EACEA and the Commission in the evaluation process. All these changes were introduced rapidly as a reaction to the audit conclusions, in parallel with the evaluation process of 2018 calls. This experience is being taken into account in the definition of the implementing provisions for the 2021-2027 MFF programmes 33 .

Follow-up, monitoring and control of grant implementation, payments

During 2015 – 2017, the EACEA maintained a good performance in terms of timely execution of payments to beneficiaries (measured as Time-to-Pay – TTP) with over 94% of payments made within contractual time limits. The Agency’s performance hence remained stable compared to the previous evaluation period.

Good performance of the EACEA in terms of timely processing of payments was also reflected in the results of the beneficiaries’ survey – 93% of beneficiaries were satisfied with the time period it took the Agency to make a pre-financing payment, 91% and 88% of them with the time period it took the EACEA to process interim and final payments, respectively.

During 2015-2017, the EACEA further improved project monitoring activities. Like in the previous evaluation period, EACEA’s project monitoring framework included desk-based monitoring (verification of various reports submitted by project beneficiaries and other administrative data) and monitoring visits.

The survey of EACEA’s beneficiaries revealed that beneficiaries were generally satisfied with most aspects related to the project implementation stage, and the satisfaction level improved compared to the results of the previous evaluation. Compared with the results of the staff opinion survey carried out in 2016, more respondents agreed that the project reporting requirements were reasonable taking into account the size of the grant and the scale of the project.

As regards budget execution, in the period 2015- 2017, the EACEA managed to fully execute its operational budget, both in terms of commitment and payment appropriations, hence meeting both targets: 98% for 2015 – 2016 (raised to 99% in 2017) for commitment appropriations and 95% for 2015 – 2016 (raised to 98% in 2017) for payment appropriations.

Execution of the eacea’s operational budget

Note: Based on C1 and C5 credits. Executed credits compared to the available budget at the end of the year (%).

Source: PPMI, based on the EACEA’s Annual Activity Reports and Annual Work Programmes.

Concerning the legality and regularity of the programme expenditure, executive agencies have to set up internal control processes ensuring adequate management of the risks related to the legality and regularity of the underlying transactions, taking into account the multiannual character of the programmes and the nature of the payments concerned. The related control objective is to ensure that the residual error rate does not exceed 2% on a cumulative basis by the end of each programme implementation. The residual risk of error is estimated by the residual error rate obtained from an examination of a sample of randomly selected transactions, less any corrections made as a result of the control systems that are in place.

During the evaluation period, the estimated residual error rate was above 2% for some of the “legacy” programmes, i.e. programmes from the 2007 – 2013 MFF. Therefore, EACEA introduced reservations in its Annual Activity Reports. However, due to the increasingly smaller proportion of the financial impact on the overall financial portfolio of the Agency, these reservations were lifted by the Agency as of its 2019 Annual Activity Reports 34 . 

The residual error rates were estimated for the 2014-2020 MFF programmes starting from 2016. The first results for 2016 and 2017 were very good with an estimated residual error for all programmes being substantially below the 2% materiality threshold, as visible in the last three columns of the chart below.

Residual error rate

Source: PPMI, based on the EACEA’s Annual Activity Reports. 

Overall satisfaction of EACEA’s beneficiaries

Results from surveys indicate that the satisfaction with the overall performance of the EACEA remained stable since 2012, with a comparable degree of satisfaction across the programmes managed by EACEA.

In particular, the level of satisfaction expressed in terms of the share of beneficiaries agreeing with the statement ‘We are satisfied with the overall quality of the programme management services provided by the EACEA during the whole project period’ increased slightly as compared to the previous evaluation and reached 88.8%.

EACEA beneficiaries rated most positively the courtesy (1.84), commitment (1.9) and competence (2.01) of the Agency’s personnel (where 1 is the highest possible score, and 9 the lowest).

Satisfaction of applicants and beneficiaries with the overall quality and timeliness of EACEA’s services

Source: PPMI, Survey of EACEA’s unsuccessful applicants and beneficiaries, 2018 (average response on scale of 1-9 – 1=strongly agree and 9=strongly disagree).

Overall satisfaction with the EACEA among beneficiaries and unsuccessful applicants by programme

Note: on the scale from 1 to 9, with one being the most positive rating possible

Source: PPMI, Survey of unsuccessful applicants and beneficiaries of the EACEA, 2018.

Proximity to addressees and visibility of the EU

The nature of the programmes and actions managed makes EACEA one of the closest agencies to citizens, and results of the surveys carried out during the evaluation confirm that EACEA managed to remain close to its applicants and beneficiaries.

Proximity of the EACEA and its staff to beneficiaires and unsuccessful applicants

Note: the first two statements represent a combined opinion of both beneficiaries and unsuccessful applicants; the remaining statements were addressed to and answered only by beneficiaries.

PPMI, Survey of beneficiaries and unsuccessful applicants 2018.Questions 12, 15, 19 and 23.

Thematic cluster meetings help discover synergies among projects and create additional opportunities. For instance, beneficiaries can interact and present their projects in an informal setting. By taking part in these events, project representatives get the necessary clarifications and learn from more experienced participants.

External experts also confirm their very strong satisfaction with EACEA’s staff ability to provide useful answers to their questions and their responsiveness. It was also clear for the external experts whom to contact or where to get help regarding any questions they had when working on their tasks. 90% of the external experts interviewed were satisfied with their overall experience with the EACEA, with another 4% giving a neutral assessment.

Proximity of the EACEA and its staff to external experts

Source: PPMI, Survey of external experts of the EACEA, 2018. Question 20: To what extent do you agree or disagree with the following statements concerning your work as an expert for the EACEA? During the evaluation/monitoring:

Applicants express, however, also several concerns. First, over the years it has become more difficult to get a prompt response from EACEA. Applicants and beneficiaries would also appreciate more overall stability in staff managing a project, as well as more on-site project visits.

As regards the visibility of the EU as promoter of the programmes entrusted to the Agency:

·Interviews with beneficiaries and unsuccessful applicants revealed that, when talking about the EACEA, they usually refer to the European Commission, which proves their awareness of EACEA managing programmes on behalf of the Commission;

·When asked explicitly, the large majority of respondents states being aware that EACEA is entrusted with the management of programmes by the European Commission;

·Over two in three respondents to the surveys agreed that funding opportunities under programmes managed by the EACEA are well promoted.

·

It should also be noted that the EACEA is using the same visual identity as the Commission.

External communication

Through its external communication activities, the EACEA implements one of its tasks, namely to support the Commission in communication and promotion of the programmes it manages.

Certain positive developments in external communication materialised during the evaluation period. In 2016, the culture of feedback was a priority for the EACEA, therefore collection, analysis and reporting of feedback from events has been streamlined and a systematic approach has been developed and tested for all infodays as well as several smaller events. In 2017 alone, a total of 40 such surveys were carried out with a view to identifying points for further improvement in the communication with stakeholders (both beneficiaries and experts). Advancements have also been achieved on the IT side, with continuous improvements to the website, which in 2017 ranked 7th on the list of the most visited sites on europa.eu portal.

During the evaluation period, the Agency acted as a focal point for applicants and beneficiaries of the programmes it manages. Its external communication helped to ensure that the potential applicants are informed about the funding possibilities and the beneficiaries received timely, helpful and relevant information for their queries. This is proved by the feedback received from unsuccessful applicants and beneficiaries through the survey and interview programmes, undertaken during the current evaluation.

Effectiveness of communication channels in communicating with beneficiaries

Source: PPMI, Survey of beneficiaries and unsuccessful applicants 2018. Question 24: Please indicate on a scale from 1 to 9 (1 = to a large extent, 9 = not at all) to what extent the following communication channels used by the EACEA have provided you, if at all, with relevant and helpful information when you needed it.

Interviews suggest that both unsuccessful applicants and beneficiaries evaluated the Agency’s communication activities as being effective, especially Agency’s efforts in promoting funding opportunities. Although some of the interviewees recommended to further improve the external communication by employing a more direct approach in disseminating the information to interested parties. In addition, some feedback received during interviews with applicants and beneficiaries and via the open-ended survey questions indicated that the website should be further improved.

Satisfaction with various external communication aspects by beneficiaries and unsuccessful applicants

Source: PPMI, survey of the EACEA’s beneficiaries and unsuccessful applicants 2018.

·Question 9: Please indicate on a scale from 1 to 9 (1 = strongly agree, 9 = strongly disagree) to what extent you agree or disagree with the following statements regarding the awareness of programmes managed by the EACEA;

·Question 12: Please indicate on a scale from 1 to 9 (1 = strongly agree, 9 = strongly disagree) to what extent you agree or disagree with the following statements concerning the application process;

·Question 29: Based on your experience, please indicate on a scale from 1 to 9 (1 = strongly agree, 9 = strongly disagree) to what extent you agree or disagree with the following statements concerning the EACEA’s actual performance and the services it provides

Opinions on the effectiveness in disseminating project results, identified during the interviews, were more divergent, with quite some respondents being unaware where to find such information or indicating that it should be made more user friendly.

5.2.Efficiency 

This section considers the relationship between the resources used by the Agency and the output. The analysis, among other factors, also includes analysis of administrative and regulatory burden and looks at aspects of simplification.

The ratio between the administrative budget of the Agency and the operational budget managed by it, was during 2015-2017 in the range of 7.0 - 7.7% based on commitment appropriations of the operational budget and 7.4% - 8.1% based on payment appropriations, i.e. similar to the values of the previous evaluation period.

The relatively high administrative costs of the EACEA and the programmes managed by the Agency were related primarily to the specific character of the funding schemes. Their complexity and the large number of rather small grants (around half of projects managed by EACEA are below EUR 50.000) awarded to a large population of grant beneficiaries (which in many cases are small and possess limited administrative capacities) make implementation of the programmes delegated to the EACEA on average more expensive, though with significant variations across programmes:

Ratio between the EACEA’s administrative and operational budget by programme

Note: the Agency’s administrative budget is based on executed commitment appropriations

Source: PPMI, based on the EACEA’s Annual Activity Reports and data provided by the Agency.

Performance indicators of the EACEA, 2015-2017

Source: PPMI, based on data provided in the EACEA’s documents (annual activity reports for 2015, 2016 and 2017).

A detailed analysis of the EACEA cost-efficiency and results of the Cost-benefit analysis is presented under point 5.4.

Human resource management

The number of staff at the EACEA remained quite stable during the evaluation period. In 2016, each EACEA staff member managed on average 28.28 proposals (compared to the average of 18.55 in all six executive agencies) and 13.64 ongoing projects (compared to the average of 8.67 in all six executive agencies). For the purpose of these calculations, the whole staff of executive agencies is considered (i.e. not limited to staff members with project management functions).

Interviewees indicated that overall the Agency had sufficient resources to implement the tasks delegated to it and, despite the relatively high amount of files handled by the Agency’s staff, in the 2016 Commission staff opinion survey 67% of EACEA’s staff agreed that their workload was acceptable compared to 59% of the Commission’s and executive agencies’ average. The actual workload of the Agency was generally in line with the estimates from the initial CBA estimates.

In terms of the ratio between administrative vs. operational staff in the EACEA, the Commission’s screening exercise in 2017 indicated that the share of administrative support and coordination staff in the executive agencies was 9.5% on average. The EACEA employed slightly more staff dedicated to general coordination, human resources, information technologies and other administrative job categories, with a share 35 of 10.7%.

Regarding the staff turnover and vacancy rate, during the period 2015-2017 the average job occupancy rate within the EACEA was around 95%. The Agency was therefore nearly fully staffed throughout the evaluation period. The rates of staff turnover in the years 2015, 2016 and 2017 were 5.4%, 6% and 4.1%, respectively.

The extent of internal mobility declined during the evaluation period, from 48 moves in 2015 to 20 in 2017, although the Agency undertook some efforts to improve this towards the end of the evaluation period. According to the evaluation evidence, the processes of staff selection and recruitment at the Agency ran smoothly.

The results of the 2016 European Commission staff opinion survey made it possible to assess and compare the HR performance of the different Commission’s executive agencies. EACEA
s results for staff engagement index 36 , staff wellbeing and overall job satisfaction were close to the average for the European Commission and its executive agencies:

Staff Engagement Index, Overall Job Satisfaction and Wellbeing in the EACEA, the Executive Agencies and all Commission Services, 2016

Source: PPMI, based on the 2016 European Commission Staff Opinion Survey: Analysis of the findings, September 2016.

The EACEA’s staff engagement score increased from 60 in 2014 to 65 in 2016, but this index has not yet returned to the level achieved at the Agency in 2013, when it scored 72. However, the EACEA saw an increase in its scores for overall job satisfaction between 2014 and 2016 from 68% to 70%, and from 33% to 47% for staff wellbeing in the same period.

However, staff’s perception of the Agency as a modern and attractive workplace was below the average for the Commission’s executive agencies (56% in the EACEA, compared with 65% in all executive agencies). Another weak point in EACEA’s staff opinion survey results concerns career opportunities.

These results provided useful guidance when the Agency defined its strategic HR priorities and developed an action plan to further increase staff engagement and to strengthen the Agency as an attractive workplace, primarily by looking into how the career opportunities for its contract agents can be improved under the existing rules. Other points from the action plan that were implemented during the evaluation period included a staff day focusing on staff engagement mobility and knowledge sharing, as well as a job-shadowing pilot.

It needs however to be mentioned that the new staff opinion survey results of 2018, released after the evaluation period, have shown significant drops of EACEA staff satisfaction in a variety of surveyed areas. The Agency with the support of its parent DGs is in the process of analysing more in depth those results and their origins in order to actively address them through appropriate action in the near future.

The findings of the staff opinion survey of 2018 were addressed mostly by the EACEA Transformation Project (ETP). The purpose of the ETP was in particular to transform the Agency into a modern organisation ready to respond to the future challenges of the current and new programmes, and thus to continue delivering high-quality results and keep staff engaged.

The Transformation Project was designed to be fully participatory. Staff were able to contribute directly to shaping the future of the Agency by having their say on matters impacting their working life. The project was carried out in full transparency and was designed to include all staff.

In addition, on 10 July 2019, the Agency presented to the Steering Committee the Draft Development Plan .

Specific staff engagement actions started in the second half of 2019 are “Connecting EACEA” (info sessions and conferences targeting topics related to the Agency’s portfolio and connecting to the parent DGs) and “EACEA Empowered” (bottom-up initiatives on various topics of common interest).

Organisational structure

Regarding the organisational structure of the Agency, the internal structure of the EACEA had been adjusted to the new programmes under its administration after the start of the 2014-2020 programming period (creation within Erasmus+, hence under the responsibility of the Head of Department, of the Unit responsible for Youth and Sport) and remained substantially unchanged during the evaluation period.

Up to March 2020, the Agency consisted of 13 units: seven under the Department of Erasmus+, and the remaining six reporting directly to the Director – two units for the Creative Europe programme, one for the Europe for Citizens programme, and three support units.

Within its organisational structure, the EACEA has centralised its financial circuits somewhat by merging the financial resource management of Units A7 (Education and Youth Policy Analysis) and A1, as well as merging the financial sector of A5 with A1, which increased the economy of resources.

Despite appointing a new Head of Department for Erasmus+ units during the previous evaluation period, the organisational structure of the EACEA has remained flatter than in most other executive agencies of the Commission.

This could be reviewed taking into account the organisational characteristics of the Agency, the ongoing changes in the organisation and the various options for its organisational structure (including the possibility of introducing new departments). A new organisational structure could improve coordination of the implementation of programmes.

The organisational chart of the EACEA is integrated in the organigrams of the parent DGs by linking various units of the Agency to the relevant directorates and units of the parent DG’s organigrams. This integration increased transparency and facilitated the collaboration between EACEA’s units and their mirror units in the parent DGs.

Aspects of simplification

Programmes managed by the EACEA benefit from the streamlining of various processes and practices. For instance, the implementation of the Erasmus+ programme benefits from the widespread use of simplified cost options covering travel, subsistence, staff salaries and other costs. The use of standard cost options in the Erasmus+ programme was expanded during the evaluation period. For example, simplified grants were introduced for the Collaborative Partnerships actions in Erasmus+ Sport programme’s 2017 calls.

The 2007-2013 Europe for Citizens programme is another example. The programme was often used as a testing ground to pilot simplified cost options and other simplifications. Considering the positive effects of the simplification measures put in place for the 2007-2013 generation programme, a financing system based on simplified costs was applied to all action grants of the new generation of the Europe for Citizens Programme 2014-2020.

During the evaluation period, the EACEA also implemented a number of other administrative measures aimed at simplification and optimisation of programmes and grants management.

As regards simplification of financial management, in cooperation with the EACEA, the Commission put a lot of effort into widening the use of standard cost options and simplifying financial management of 2014-2020 MFF programmes delegated to the Agency. The development of a system of lump-sums and unit costs simplified the financial management of the programmes both for the beneficiaries and for the Agency.

The use of eForms (electronic tool for the submission of proposals) was significantly expanded during the reporting period, with over 94% of applicants using them. Automatic checks were integrated into the eForms to facilitate the application process and to reduce the share of ineligible proposals. Such developments allowed the Agency to achieve an almost entirely paperless application and evaluation process.

During the evaluation period, the Agency worked to replicate the success of eForms by developing a generalised use of electronic interim and final reports (eReports) for the 2014-2020 generation of programmes. Its aim was to simplify the grant administration process for beneficiaries (by automating the preparation and submission of reports, facilitating collaboration between project partners, etc.), to improve productivity within the Agency (by avoiding the re-encoding of information already submitted), and to facilitate communication between the beneficiaries and the EACEA (faster receipt of reports, etc.) and the transition to a future corporate grant management system.

The use of eForms, eReports and paperless processes produced significant efficiency gains. The resources freed up by the simplification process were re-invested in many other processes of the Agency, in order to improve the quality of the service provided by the EACEA to its beneficiaries and stakeholders, in order to support policy objectives.

EACEA also pursued the development of its IT systems, i.e. through the integration of Pegasus (grant management system) with APPFIN (financial system). It is expected that such integration would be a significant simplification for the Agency, allowing it to use a single tool for grant management. In 2015 the EACEA also introduced a new tool (Speedwell) for the payments of its administrative budget, which enabled a paperless workflow. ICT tools that support online interaction with experts were used in the majority of project selection procedures.

During 2017, the EACEA prepared for the 2018 adoption of the Commission’s Single Electronic Data Interchange Area (SEDIA). This provides applicants and tenderers (third parties) with a single entry point to communicate and exchange information relating to the legal and financial verification process that the EACEA needs to perform, prior to issuing contracts and making pre-financing payments.

The survey shows that simplification is appreciated by applicants and beneficiaries, who, at the same time, would appreciate that further efforts are put to simplify and streamline the administrative arrangements of the programmes managed by the EACEA.

5.3.Coherence 

Coherence looks at any overlaps and complementarities within the programme portfolio managed by the EACEA or delimitation of responsibilities between the EACEA and its parent DGs.

Coherence of portfolio

As noted in the previous evaluation of the Agency, the EACEA was very successful at exploiting synergies resulting from the optimisation of different programmes and was extending its experience gained across the programmes.

In terms of complementarities, a finding emerging interviews concerned links between Intra-Africa, the Erasmus+ programme, and other EU initiatives involving collaboration with Africa. The programmes were said to complement each other very well and encourage systemic reforms. The EACEA in particular was praised for its ability to understand these links and reflect upon them. The Agency, DG DEVCO and DG EAC engaged in dynamic, triangular cooperation involving joint events on Intra-Africa mobility and other types of meeting involving multiple stakeholders including the EACEA. This helps the Agency understand the context of systemic changes resulting from the mobility scheme. In return, the EACEA can promote the complementary activities of DG EAC and DG DEVCO, when they visit African countries on a monitoring mission.

An additional instance where synergies were exploited is eReports (see above). After carrying out an initial eReports pilot project on the Europe for Citizens programme, the EACEA extended the use of eReports to several actions of the Erasmus+ and Creative Europe programmes in 2015.

The broadness of activities delegated to EACEA means that the Agency was confronted with quite diverse actions in terms of scope and scale. In the case of the Eurydice network 37 , for instance, EACEA successfully ensured the management of a network characterised by a relatively small budget compared to the average of the programmes managed by the Agency.

Relations with parent DGs and supervision

The evaluation confirmed that responsibilities of the Agency are clearly defined by the existing legal framework governing the interaction and roles of the EACEA and its parent DGs. In particular, the tasks delegated to the Agency and tasks reserved for the Commission are clearly defined in the Act of Delegation and the General Memorandum of Understanding (MoU) between EACEA and its parent DGs, updated in March 2015.

In addition to general principles, the MoU defines how information exchange, administrative matters, cooperation in relation to other actors and institutions and other modalities and procedures of interaction are supposed to work. The MoU reduces the risk of overlaps or gaps which might originate from an inadequate formal delimitation of responsibilities borne by the EACEA and its parent DGs to a minimum.

Based on the evidence drawn from interviews, there is a clear understanding of the general principle that defining the policy is in the hands of the Commission, whereas the EACEA is supposed to implement the defined policy goals.

In terms of the Agency’s contribution to the preparation of the Work Programmes of its parent DGs, EACEA is closely consulted both formally and informally, with both sides expressing their satisfaction with how the cooperation takes place.

Lines drawn between roles of the Agency and its parent DGs are not completely static, but adapt to circumstances: in times of need, the operational units in the Commission have increased expectations and need more help from their colleagues in the Agency. Having close collaboration ties between the mirror units in the DG(s) and the Agency and some flexibility in the definition of EACEA’s responsibilities are necessary for ensuring the Agency’ versatility and capacities to respond to expectations of its multiple parent DGs.

As regards supervision, according to the Regulation 58/2003, the Commission sets the limits to the actions of each executive agency and exercises real control over its operation.

Overall, the existing supervision arrangements proved to be effective and were exercised thoroughly during the evaluation period.

Yet, even though the arrangements concerning supervision are working as defined in the existing legal framework, it has been found that clarifications in the definition of a supervisory function are needed. For instance, the boundaries of supervision should be defined better, as it was not always obvious for the parent DGs how far the supervision should go in order not to breach the autonomy of the Agency and avoid micro-management. To address this issue, corporate discussions on the supervision of executive agencies have begun in 2018 in the Commission.

It should be mentioned that the relocation of the Agency close to the premises of its parent DGs in 2016 helped ensure better coordination.

Policy Feedback

Concerning policy feedback, EACEA is faced with the challenge of being able to address the policy-making needs of its parent DGs and participating DGs. Indeed, while EACEA is praised for its capacity to manage the programmes delegated to it, this recognition is accompanied by increasing expectations and pressure to improve the execution of its other key task – gathering, analysis and transmitting information required by the Commission to guide the implementation of the programmes.

The need for the EACEA to improve in this aspect was already emphasised by the previous external evaluation. The current evaluation finds that support to policy-making was defined as a priority for the Agency in the EACEA’s Annual Work Programme for 2016.

Policy support was provided, among other means, by organising kick-off and thematic cluster meetings, placing greater emphasis on policy aspects and engaging all relevant stakeholders. EACEA staff provided support to several policy-related projects in DG EAC. The Agency has also provided support and data relating to adult education for DG EMPL, and responded to questions of the European Parliament. In addition, the Agency supported DG NEAR in the evaluation of the Fellowship for Youth Action, deepening the relationship between the Agency and this DG.

At the level of actions, the EACEA has also implemented an analysis of the results produced the projects via the carrying out of various studies, e.g. Europe for Citizens Programme Results 2017, and by disseminating success stories not only via the Project Results Websites (VALOR), but also through dedicated publications such as Europe for Citizens.

However, the evaluation observed that the Agency’s efforts in relation to reporting the results and impacts of the programmes it manages lack either wider recognition, or better awareness among EACEA’s parent DGs. Parent DGs expected a stronger emphasis in EACEA’s reporting on the topics and outputs of awarded projects.

Slight dissatisfaction was expressed in relation to the declining resources allocated to monitoring missions and/or the changing approach of the EACEA to organising missions. Some DGs also expected information from the EACEA on interim results, rather than just the final results of projects – particularly for newly delegated actions.

The reasons behind these comments by parent DGs may lay in the shift of attention of the Agency and its parent DGs towards responding to the outcomes of the IAS audit, and the significantly increased workload associated with implementing selection procedures that complied with its formal requirements.

5.4.Retrospective Cost-benefit analysis

When looking at the ex ante cost-benefit analysis, the overall costs of the executive agency scenario was estimated at EUR 146.0 million in 2015-2017, i.e. lower than the estimated costs of the in-house scenario. The actual savings during 2015 -2017 period were found to be 6.5% higher compared to the initial SFS estimates (EUR 57.1 million compared to EUR 53.6 million) and 37.0% higher compared to the initial CBA estimates (EUR 57.1 million compared to EUR 41.7 million). As forecasted in the SFS and the ex ante CBA, savings of the executive agency scenario primarily resulted from a higher share of lower cost external personnel (CAs) employed within the Agency and lower overall number of staff:

Estimated savings of the executive agency scenario in 2015-2017 (Title I and Title II expenditure), EUR 

Sources: PPMI, based on CBA, SFS, final annual accounts, the EACEA’s AWPs.

The workload initially estimated during the period 2015-2017 was correct as overall the EACEA managed to cope with the foreseen human resources while implementing the programmes delegated to it. To ensure the optimal allocation of resources, and to respond to new activity requests, the Agency revised its methodology for the workload indicators and screened the allocation of human resources within the organisation. In addition to its authorised staff, the Agency employed interim staff to temporarily replace staff on long-term leave (such as maternity/parental leave).

It should be noted that costs in the CBA were calculated at constant 2013 prices (i.e. neutralising the effect of inflation). However, in the SFS these estimations were used as current prices without any further indexation. In real terms, this constituted a reduction of the administrative budget, with an impact on staff costs (structural under-estimation of staff expenditure).

6.Conclusions

The results of the evaluation of EACEA operations during 2015-2017 period confirm that the delegation of programme implementation to EACEA proved considerably more cost effective than the in-house scenario.

While successfully implementing its annual work programmes, EACEA managed to further improve the efficiency in the use of its administrative and financial resources, to develop a more resilient and adaptive organisation. The retrospective cost benefit analysis found that the actual savings of the executive agency scenario were EUR 57.1 million during the period 2015-2017 compared with the in-house scenario.

During the evaluation period, the EACEA achieved very good results in terms of almost all KPIs. In detail, the Agency performed well in ensuring timely execution of the delegated functions and met the main indicators describing timeliness of evaluation and contracting process (Time-to-Award, Time-to-Contract, overall Time-to-Grant) and timeliness of processing payments to beneficiaries (Time-to-Pay).

The overall satisfaction with the quality of services among its applicants and beneficiaries was rather high (79.2%), though somewhat uneven between unsuccessful applicants and beneficiaries. Applicants and beneficiaries gave a very positive opinion about the simplifications already introduced in the programmes managed by the EACEA.

The coordination and communication mechanisms in place enabled information flows between EACEA and the Commission services, in particular with regard to the content of projects supported and their results. No evidence of overlaps, duplication, gaps and inconsistencies within the programme portfolio and support services managed by EACEA were identified.

The current evaluation identifies a few strategic areas where further improvements would be desirable: 

1.Continue adapting EACEA’s strategy, policies and procedures, particularly in light of the 2017 audit, so as to strengthen the Agency and its functioning. Ensure that the experience gained and lessons learnt contribute to making EACEA even more resilient.

2.Review the experience of the 2018 evaluation and selection processes and use the lessons learnt from this experience to optimise future evaluation and selection procedures. Take into account this experience while defining implementation acts and procedures for the management of the 2021-2027 MFF programmes.

3.Take action to update the existing supervision arrangements between EACEA and its parent DGs.

4.Explore further opportunities to improve the EACEA’s reporting of programme results and policy feedback to its parent DGs.

5.Review the organisational structure of the EACEA, taking into account the organisational characteristics of the Agency, the recommendations of the 2017 IAS audit and the possibility of introducing new departments.

Extensive actions have already been undertaken and are currently being (or have already been) implemented to address the issues identified under the points 1 to 3. Actions to respond to point 4 will be explored in more depth. As regards the organigramme of EACEA (point 5), a reorganisation of the structure has been implemented by the new Director with effect on 16 March 2020.

The parent DGs and the Agency prepared an action plan to address the shortcomings identified in this evaluation. This action plan aims at increasing the Agency’s performance and contribution to an efficient implementation of the delegated programmes.

The Agency has also taken extensive corrective action to respond to the findings of the IAS audits on Grant Management - Phase I (from call publication to contract signature – all recommendations now closed) for Erasmus+ and the Creative Europe programmes, and Phase II (from project monitoring to payment) where the IAS is in the process to perform the follow-up engagement.

Actions were also taken to address the outcome of the Commission-wide staff opinion survey carried out in the course 2018, which showed the need for corrective measures to be implemented in order to solve the identified issues and to increase the satisfaction of EACEA staff.

More globally and to address the various issues outlined above, the agency has been undertaking since 2019 a participatory ‘Executive Agency Transformation Project’ to equip the agency with an improved organisational structure, processes and internal control framework for the challenges of the coming years. This resulted in the reorganisation mentioned above.

ANNEX - Evaluation questions

Effectiveness

·To what extent has EACEA been operating according to the legal framework establishing it?

·To what extent has EACEA achieved its objectives with special focus to (a) the implementation of the delegated programmes4 and (b) the implementation of the support services? What, if anything, could be done to render EACEA more effective in achieving these objectives?

·To what extent and how has EACEA contributed to an improved management and visibility of the delegated programmes and better services and satisfaction to the stakeholders and addressees in terms of the elements assessed in the 2013 CBA and as compared to the alternative options mentioned in that CBA?

·To what extent does EACEA’s external communication function support the mission of the Agency?

·To what extent has EACEA contributed to improved management of the programmes in terms of:

oProximity to addressees (e.g. role played by the Agency as focal point for applicants and beneficiaries of the programmes; instruments and mechanisms put in place for communicating with the applicants and beneficiaries).

oEffective implementation of the programmes, taking into account the interests of the addressees and those of the EU:

§Rate of execution of commitment appropriations;

§Rate of execution of payment appropriations;

§Time to grant;

§Net time to pay;

§Residual multi-annual error rate identified at ex post control;

§Time to assess submitted reports

oVisibility of the EU as promoter of the programmes entrusted to the Agency (e.g. compliance with the Commission’s guidelines on information and visibility of programmes5, instruments put in place to ensure the visibility of the EU as promoter of the programmes).

·To what extent is the management of the Eurydice network by EACEA effective?

Efficiency

·To what extent has the EACEA carried out its work efficiently.?

·To what extent have the actual costs (including cost of coordination and monitoring) of EACEA corresponded to the estimates of the 2013 CBA? If not, what are the reasons behind?

·To what extent have the actual benefits corresponded to the estimates of the 2013 CBA? If not, what are the reasons behind?

·To what extent has the management and execution of the programmes by EACEA been cost-effective as compared to the alternative options6?

·Examples of alternatives to the Executive Agency: (i) management of the programme(s) by the Commission, (ii) mixed Agency - Commission management, (iii) partial management by the Commission while outsourcing some activities to the extent legally possible.

·To what extent has EACEA contributed to improved management of the programmes in terms of:

oSimplification of the procedures and flexibility in the implementation of delegated tasks (e.g. capacity to adapt to periods of high workload)?

oWhich further scope for simplification exists?

·To what extent is the management of the Eurydice network by EACEA cost-effective?

·Which aspects/means/actors or processes render EACEA more or less efficient? What could be improved?

·To what extent has the establishment of EACEA resulted in savings to the EU budget as compared to the alternative options (e.g. difference in costs between the Commission option and the Agency option)?

·To what extent have EACEA’s internal organisation and procedures been conducive to its efficiency? Is the size of the Agency adequate?

·To what extent is EACEA’s internal organisation capable and flexible to rapidly respond to resource needs due to uncertainties related to volumes of work?

·To what extent does EACEA’s human resources management contribute to the achievement of the Agency’s objectives?

oIs the size and structure of the organisation appropriate?

oIs the balance between operational staff and administrative support staff appropriate?

oAre the staff turnover and vacancy rate well managed?

oHow are provisions on overtime and teleworking implemented in the Agency?

oHow does the Agency follow up on the findings of staff surveys?

Coherence

·To what extent have there been overlaps / gaps / inconsistencies / complementarities within the programme portfolio and support services managed by EACEA and how are these addressed?

·Is there a clear and appropriate delimitation of responsibilities and tasks between EACEA and the Parent DGs? Are there overlaps or gaps? Are the different responsibilities adequately communicated to the beneficiaries?

·To what extent has EACEA enabled the Commission to better focus on its policy related tasks?

oAre there any governance (financial and policy) issues in relation to the implementation of the tasks that are delegated to the executive agencies / issues as regards the Commission’s services being able to steer EU policy or budget implementation?

oAre appropriate mechanisms and instruments in place, and at which level, to ensure an adequate coordination and information flow between EACEA and the Commission services, notably on the content of the projects supported and their results?

oDoes EACEA provide useful information on the implementation of the delegated programmes and their progress (in terms of management and content) in support of the policy process (e.g. information required for the annual Management Plan of the Parent DGs)?

·To what extent have the activities of EACEA resulted in unintended effects (both desirable and undesirable)?

·To what extent has the Commission, in the presence of EACEA, been able to maintain an adequate level of know-how in relation to the programmes entrusted to the Agency? How has this been achieved? What are the feedback channels, means and methods used for this purpose? What are areas for improvement, if any?

·Have the monitoring, reporting and supervision arrangements in place enabled the Commission to benefit, in the short and medium term, from the know-how created within EACEA?

·How effective is the flow of information and communication between EACEA and the Commission services (in particular Parent DGs)?

·To what extent would the closing down of EACEA result in losing significant know-how in relation to the management of the programmes entrusted to EACEA?

(1)  Council Regulation 58/2003 of 19 December 2003 laying down the statute for executive agencies to be entrusted with certain tasks in the management of Community programmes, OJ L 11 of 16 January 2003, p. 1.

(2)  Commission Implementing Decision 2013/776/EU of 18 December 2013 establishing the ‘Education, Audiovisual and Culture Executive Agency’ and repealing Decision 2009/336/EC.

(3)  Commission decision C (2013) 9189 of 18 December 2013 delegating powers to the Education, Audiovisual and Culture Executive Agency with a view to performance of tasks linked to the implementation of Union programmes in the field of education, audiovisual and culture comprising, in particular, implementation of appropriations entered in the general budget of the Union and of the EDF allocations.

(4)  Commission Decision C(2018) 4421 appointing members and observers to the Steering Committee of the Education, Audiovisual and Culture Executive Agency and repealing Decision C(2015) 758.

(5)   Commission’s Better Regulation Communication COM (2015) 215 final and its accompanying Staff Working Documents SWD (2015) 111 final and SWD (2015) 110 final .

(6) Cost-benefit analysis for the delegation of certain tasks regarding the implementation of Union Programmes 2014-2020 to the executive agencies - Final report for the Commission of 19 August 2013.

(7)

Identification of the tasks justifying outsourcing, a cost-benefit analysis, which includes the costs of coordination and checks, the impact on human resources, possible savings within the general budgetary framework of the European Union, efficiency and flexibility in the implementation of outsourced tasks, simplification of the procedures used, proximity of outsourced activities to final beneficiaries, visibility of the EU as promoter of the EU programme concerned and the need to maintain an adequate level of know-how inside the Commission.

(8)

Appendix II of the Guidelines for the establishment and operation of executive agencies financed from the Union budget (C (2014) 9109 from 2 December 2014; pp. 64-72).

(9) PPMI Group, Study supporting the evaluation of the Education, Audiovisual and Culture Executive Agency (EACEA) (2015 – 2017).

(10)   https://ec.europa.eu/programmes/erasmus-plus/node_en  

(11)   https://ec.europa.eu/programmes/creative-europe/  

(12)   http://ec.europa.eu/citizenship/europe-for-citizens-programme/index_en.htm . As from 1 January 2020, the Europe for Citizens programme is managed by DG JUST.

(13)   https://ec.europa.eu/echo/what/humanitarian-aid/eu-aid-volunteers_en  

(14)   https://europa.eu/youth/solidarity_en  

(15)  Original specific financial statement annexed to Commission Decision C(2013)9189 of 18 December 2013 and successively amended (including outside the evaluation period) by C(2014)4084 of 25 June 2014, C(2015)658 of 12 February 2015, C(2016)401 of 1 February 2016, C(2016)1851 of 31 March 2016, C(2017)3049 of 12 May 2017, C(2018)5011 of 1 August 2018, C(2018)7435 of 13 November 2018, C(2019)1299 of 19 February 2019. 

(16) The responsibility over certain competences (i.e. centralisation of validation tasks) originally under the remit of individual Commission services including EACEA was reattributed to REA). The transfer led to a compensation that resulted, for EACEA, in a decrease of 8 FTEs.

(17) This initial budget was estimated prior to the delegation of other spending programmes and activities launched in the course of the current MFF 2014-20 (e.g. European Solidarity Corps).

(18) Commission Decision C(2013)9189 delegating powers to the Education, Audiovisual and Culture Executive Agency with a view to performance of tasks linked to the implementation of Union programmes in the field of education, audiovisual and culture comprising, in particular, implementation of appropriations entered in the general budget of the Union and of the EDF allocations was amended by the following four Commission Decisions:

(19) As of 1 January 2020 Directorate-General for Justice and Consumers (JUST) replaced DG HOME.

(20)

Memorandum of Understanding between the EACEA and its parent DGs on 26 March 2015.

(21) Includes 2014 commitments and selected project grants as these elements have a direct impact on the 2015-2017 EACEA’s workload.

(22) Excluding grants under EU Aid Volunteers and Pan Africa programmes, as no estimates of the expected number of grants was included in the respective cost benefit analysis or Specific Financial Statement.

(23) The number of Europe for Citizens grants was 20% lower than initially estimated in the CBA and Commission annual Work Programme, which related to a higher than anticipated average grant size.

(24) Total number of staff, financed from EU Budget and contributions from EFTA/EEA and/or third countries.

(25) The Agency’s mandate was extended several times to have additional activities include in the remit of the Agency (see above). 

(26)   Study supporting the evaluation of the EACEA, final report prepared by PPMI, April 2019.

(27)  Commission implementing decision 2013/776/EU of 19 December 2013 establishing the ‘Education, Audiovisual and Culture Executive Agency’ and repealing Decision 2009/336/EC.

(28)  Commission Decision of 18 December 2013 delegating powers to the Education, Audiovisual and Culture Executive Agency with a view to performance of tasks linked to the implementation of the Union programmes in the field of education, audiovisual and culture comprising, in particular, implementation of appropriations entered in the general budget of the Union and of the EDF allocations, C(2013) 9189 final.

(29)  Commission, Commission Decision of 2.12.2014 establishing guidelines for the establishment and operation of executive agencies financed by the general budget of the Union. Brussels, 2.12.2014, C(2014) 9109 final. // Annexes to the Commission Decision.

(30)  The IAS also identified other serious shortcomings affecting the effectiveness of the overall control system in its audit on Grant Management - Phase II (from project monitoring to payment), following which the Agency took a set of necessary corrective actions. The kick-off meeting of the third and final phase of the IAS audit on Grant Management, focussing on ex-post controls, took place on 9 December 2019.

(31) The Programme Guide is published by DG EAC and co-shared by EACEA and DG EAC.

(32) The Participant Portal is not managed by EACEA and DG EAC.

(33) In 2018, in its audit on Grant Management - Phase II (from project monitoring to payment), the IAS identified shortcomings affecting also the effectiveness of the overall control system, following which the Agency took a set of necessary corrective actions. The full impact of the changes in the procedures implemented by EACEA in response to the findings of Phases I, II and III (started December 2019) of the audit will be assessed in full detail in the next triennial evaluation report.

(34)  In 2017 and 2018, the Agency also introduced a reservation in its Annual Activity Reports concerning its internal control system assessed as partially functioning following the findings of the IAS audits on grant management – Phase I and Phase II. Due to the corrective actions put in place and the improvement of the situation, this reservation was lifted by the Agency in its 2019 Annual Activity Report.

(35)  The share of staff working in the Agency’s horizontal functions has gradually decreased since, to 9,6% in 2017 and 8,5% in 2018.

(36) This index is composed of the following seven questions: •    I have the appropriate and timely information to do my work well; •    My colleagues are committed to doing quality work;•    I have a clear understanding of what is expected from me at work;•    I have recently received recognition or praise for good work;•    I feel that my opinion is valued; •    My manager seems to care about me as a person;•    My line manager helps me to identify my training and development needs.

(37)  In 2009, the EACEA gained responsibility for the implementation of Eurydice, the European network of information on education and youth. Eurydice’s work includes the collection, analysis and dissemination of information, as well as the production of studies and other publications.