EUROPEAN COMMISSION

Brussels, 29.4.2020

SWD(2020) 76 final

COMMISSION STAFF WORKING DOCUMENT

Evaluation of the the Executive Agency for Small and Medium-Size Enterprises (EASME)

Accompanying the document

Report from the Commission to the European Parliament, the Council and the Court of Auditors

Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Executive Agency for Small and Medium-Size Enterprises, the Innovation and Networks Executive Agency, the Education, Audiovisual and Culture Executive Agency, the Research Executive Agency and the European Research Council Executive Agency

{COM(2020) 184 final} - {SWD(2020) 73 final} - {SWD(2020) 74 final} - {SWD(2020) 75 final} - {SWD(2020) 77 final} - {SWD(2020) 78 final}

TABLE OF CONTENTS

1.    Introduction    

2.    Background    

2.1.    Description of the intervention and its objectives    

2.2.    Baseline and points of comparison    

3.    Implementation/State of play    

4.    Method    

4.1.    Short description of methodology    

4.2.    Limitations and robustness of findings    

5.    Analysis and answers to the evaluation questions    

5.1.    Effectiveness    

5.2.    Efficiency    

5.3.    Coherence    

5.4.    Relevance    

5.5.    Retrospective Cost-benefit analysis    

6.    Conclusions    

ANNEX - Evaluation Questions    

1.Introduction

The purpose of this evaluation is the periodical (triennial) evaluation of the operation of the Executive Agency for Small and Medium-Size Enterprises (EASME). EASME is governed by its Act of Establishment 1 , which sets out the mandate of the Agency and the Act of Delegation 2 , which specifies the tasks to be carried out by, and the powers delegated to the Agency to perform its mandate. These tasks relate to the implementation of dedicated parts of certain EU financial programmes. Furthermore, the Agency is governed by the Framework Regulation for executive agencies 3 and the decision establishing the EASME Steering Committee .

In line with the Commission’s Better Regulation principles 4 , the evaluation applies different evaluation criteria to guide the analysis of the performance of the Agency. It assesses whether the Agency has fulfilled its tasks in an efficient and effective way, whether there are overlaps/ gaps/ inconsistencies or synergies within the management of the programme portfolio executed by the Agency; and whether there is a clear delineation of tasks between EASME and the parent DGs or other executive agencies (coherence) and whether objectives and tasks responded to past needs and still respond to current needs (relevance).

The evaluation assesses whether the functioning of the Agency has yielded the expected positive results as estimated in the cost-benefit analysis for delegating task to the Agency 5 and identifies potential areas of improvement. To this end the estimations of the cost-benefit analysis of 2013 have been tested to provide evidence on the validity of the assumptions made in the ex ante scenario by considering the actual costs and benefits of programme implementation by the Agency in a structured way. The aspects covered by the cost-benefit analysis are specified in Article 3(1) of the Framework Regulation 6 and the Guidelines on establishing and operating executive agencies 7 .

The evaluation does not cover the achievements of the programmes managed by EASME, which are subject to separate mid-term and ex post evaluations. The evaluation of EASME is, nevertheless, providing useful input for these programme evaluations, as the performance of the Agency affects the efficiency and effectiveness of the programmes it manages.

The evaluation examines the efficient use of resources and the effective achievement of the tasks entrusted to the Agency. In particular it looks at whether the alignment of more coherent programme portfolios with the Agency’s core competences and its brand identity delivered the estimated qualitative benefits; whether the assembly of the management of different EU programmes delivered the estimated synergies, simplification and economies of scale and whether there is scope for simplification and further efficiency gains. Also the continued relevance of all objectives and tasks entrusted to the Agency were assessed as well as the coherence of its programme portfolio and the delineation of tasks between the Agency and other executive agencies. Furthermore, the evaluation examined whether the outsourcing allowed the parent DGs to focus on core tasks.

The evaluation covers all the tasks carried out by the Agency during the years 2014-2016. The evaluation is supported by a study carried out by an external contractor. The results of this study are summarised hereafter and this evaluation will be presented to the European Parliament, to the Council and to the Court of Auditors in accordance with Article 25(1) of the Framework Regulation.

The results of the evaluation will feed into the reflection to assess the opportunity of expanding/modifying the Agency’s mandate in view of the delegation of the implementation of 2021-2027 EU programmes.

2.Background 

2.1.Description of the intervention and its objectives

According to the Framework Regulation and EASME’s Act of Establishment 8 , the outsourcing of certain management tasks to the Agency intends:

·To allow the Commission to focus on its institutional tasks; i.e. tasks assigned to the institutions by the Treaty which require discretionary powers in translating political choices into action. Such institutional tasks should not be outsourced.

·To enable achieving the goals of EU programmes more effectively.

According to a Cost-Benefit Analysis carried out in 2013 delegating tasks to the Agency was estimated to be more cost-efficient than an in-house scenario. The Act of Establishment estimated that the alignment of more coherent programme portfolios with the Agency’s core competences and its brand identity would bring qualitative benefits. In addition, it projected that assembling the management of different EU programmes would bring synergies, simplification and economies of scale. Furthermore, the pooling of all aspects of the Horizon 2020 ‘SME instrument’ would provide a single point of access to potential beneficiaries and guarantee consistent service delivery.

The Agency implements delegated tasks in close cooperation with its seven parent Directorates General: (1) DG for Internal Market, Industry, Entrepreneurship and SMEs (DG GROW), (2) DG for Research and Innovation (DG RTD), (3) DG for Communications Networks, Content and Technology (DG CNECT), (4) DG for Climate Action (DG CLIMA), (5) DG for Energy (DG ENER), (6) DG for Environment (DG ENV) and (7) DG for Maritime Affaires and Fisheries (DG MARE).

The Agency is entrusted to implement SME-related parts of several EU programmes under the 2014-2020 multiannual financial framework (MFF): the Programme for the Competitiveness of Enterprises and small and medium-sized Enterprises (COSME), the Framework Programme for Research and Innovation (Horizon 2020), the Programme for the Environment and Climate Action (LIFE), the European Maritime and Fisheries Fund (EMFF), the legacy of certain actions under the Competitiveness and Innovation Programme 2007-2013 (CIP).

The Agency implements the COSME programme excluding the financial instruments and some other actions which are managed by DG GROW. Under the Horizon 2020 programme the following parts are delegated: innovation in SMEs of “Part II Industrial Leadership”, SME Instrument, fast track to innovation pilot scheme, societal challenge ‘Secure and efficient energy’, societal challenge ‘Climate action, environment, resource efficiency and raw materials’. Under the LIFE programme the Agency manages the calls for action grants under the Environment sub-programme 9 and the Climate Action sub-programme, as well as the call for operating grants to NGOs that help to implement and shape EU policies on the environment and climate change. Concerning the European Maritime and Fisheries Fund, a large part of the direct management of policy support actions in the area of Integrated Maritime Policy, Control and Scientific Advice and Knowledge is delegated. The Agency works closely with other partners such as the Common Support Centre for Horizon 2020 (now named the Common Implementation Centre), the Research Executive Agency (REA) for some logistic and administrative support services and an external contractor for certain tasks under the LIFE programme.

The main tasks of the Agency are to:

·Manage some or all stages of programme implementation and some or all phases of specific projects;

·Adopt the instruments of budget execution and carry out all the relevant operations; and

·Provide support in programme implementation.

Specific tasks delegated to the Agency:

2.2.Baseline and points of comparison 

The current evaluation of EASME operations during 2014-2016 assesses the actual costs and benefits of programme implementation by EASME (executive agency scenario) when compared with the alternative scenario of management by the Commission services (in-house scenario).

Accordingly, the reference point for the present EASME evaluation is the 2013 ex ante Cost-benefit analysis, and the specific financial statement 10 (SFS) of EASME.

Whenever possible, the analysis of the performance of EASME during the period of reference assesses the progress compared to the previous evaluations, covering the Agency’s operations during 2006-2013.

According to the SFS of EASME prepared in 2013 estimated savings are EUR 9.9 million over the period 2014-2016 11 . The total number of full time equivalents (FTEs) required to manage the relevant programmes in year 2020, the peak programming year in terms of workload, was estimated at 502 FTEs for EASME. The total initial operational budget entrusted to EASME in 2020 was estimated at around EUR 11.100 million in commitment appropriations and EUR 11.316 million in payment appropriations. Considerable efficiency gains were expected compared to the in-house scenario over the period, along with non-quantifiable benefits such as improved quality of programme management and service delivery, improved visibility of the EU programmes and proximity to beneficiaries.

3.Implementation/State of play

Initially founded as the Intelligent Energy Executive Agency (IEEA) 12 to implement renewable energies and energy efficiency programmes, the Agency was transformed in December 2007 13 into the Executive Agency for Competitiveness and Innovation (EACI) extending its mandate to the Competitiveness and Innovation Framework Programme (CIP) and the second Marco Polo programme. EASME was created in 2013 14 by expanding competences previously entrusted to EACI with the delegation of operational tasks regarding the implementation of parts of the Framework Programme for Research and Innovation (Horizon 2020), the Programme for the Competitiveness of Enterprises and small medium size enterprises (COSME), the Programme for Environment and Climate Action (LIFE) and the European Maritime and Fisheries Fund (EMFF). The Agency is based in Brussels.

The new mandate also brought changes in the governance of the Agency as new parent DGs joined the Steering Committee, and subsequently signed a Memorandum of Understanding (MoU) with EASME 15 . The new MoU defined the modalities and procedures of interaction between EASME and its parent DGs. In addition, it brought a clearer delimitation of the administrative activities between the Agency and its parent DGs. This evaluation assesses whether the distribution of tasks has been respected for the 2014-2016 period, and the effects of the intervention on the coherence and the efficiency of EASME’s action.

EACI counted 161 staff in December 2013. In 2014, to cope with the increase of delegated activities, 140 candidates were recruited (18 left the Agency), corresponding to a 76% staff growth. The staff progressively grew 16 to 283 in 2014, 373 in 2015, and 417 in 2016. The operational budget for commitments delegated to EASME increased from EUR 1056.5 million in 2014 to EUR 1208.5 million in 2015 and to EUR 1321.65 million in 2016.

The Agency’s administrative budget increased as well from EUR 20.26 million in 2014, to EUR 30.17 million in 2015 and to EUR 31.71 million in 2016. The Agency faced teething problems in 2014 with the massive recruitment and the substantial increase in delegated activities. Some delays occurred in implementation and activities were carried over to 2015. In 2015, not only did the Agency catch up with the backlog but it reached cruising speed. The budget executed by the Agency grew from EUR 123.1 million to EUR 618.23 million in 2015 and to EUR 1013.41 million in 2016.

A series of organisational and procedural changes started with the implementation of the programmes under the multiannual financial framework (MFF) of 2014-2020:

4.Method

4.1.Short description of methodology

The supporting study was performed by using the following data collection methods and specific methodological approaches were used to ensure data triangulation (confirmation from different sources):

·Document review and desk research;

·Semi-structured interviews;

·Online survey and statistical analysis of survey data;

·Retrospective cost-benefit analysis; and

·Open public consultation.

The document review and desk research covered EU-level reports and documents, previous evaluations and audit reports of EASME and its programmes, other relevant action plans or reports. Through the document review, the study team gathered contextual information, indicators for assessment and data for comparison of yearly developments and other executive agencies. The evaluation also relied on the results of the 2016 staff opinion survey.

The consultation strategy relies principally on targeted consultations and interviews to collect data on the implementation of the programmes and on the perception of EASME by the main stakeholders of the Agency. The public consultation was organised as an exercise of transparency and to give the opportunity to a wider public to express opinions.

124 interviews were conducted with Commission staff (DG GROW, DG ENER, DG ENV, DG RTD, DG HR, DG MARE, DG CLIMA, and DG CNECT), EASME staff, Member States National Contact Points, experts, beneficiaries, tenderers and other stakeholders (e.g. companies). The interviews were semi-structured and adapted to the relevant stakeholder group. Most of the interviews were conducted face to face.

The online targeted survey ran for four weeks (15 November 2018 to 15 December 2018). 1,864 stakeholders involved in the operation of EASME at different levels participated in the targeted survey.

Stakeholder participation in the targeted survey

Source: targeted survey

A cost-benefit analysis (CBA) was performed providing a comparative analysis of various actions and possible alternatives to the existing organisational structure of EASME in terms of its costs and benefits.

The online Public Consultation (OPC) was published on the website of the European Commission 19 for 12 weeks between 14 December 2018 and 8 March 2019. 69 respondents expressed their opinion. More than 26% were companies/business associations and around 20% EU citizens. Respondents are mostly from UK (around 19%) followed by Spain (around 16%), Netherlands (7%) and Germany (7%). The majority of respondents were projects coordinators (around 48%) or project applicants (19%) and most of them (67%) participated to Commission actions managed by EASME.

4.2.Limitations and robustness of findings

Most relevant stakeholder groups were consulted (beneficiaries, intermediaries, external experts). However, compared to the overall population reached, unsuccessful applicants participated less actively in the survey than programme beneficiaries.

No sampling bias was observed as the profile of the respondents to the surveys was very similar to the overall population, guaranteeing statistical representativeness. The non-response bias (not all characteristics of the group that did not reply had been captured in full) was mitigated through triangulation with the results of follow up interviews. The triangulation approach, using multi-level and multi-stakeholder dimension in the data collection, ensured the robustness and reliability of the data and information used to draw up conclusions in the supporting study.

Some views from Commission and Agency staff gathered from the interviews and mentioned in the study illustrate personal experience, although they may not be representative of overall relations between the Commission and the Agency.

5.Analysis and answers to the evaluation questions

5.1.Effectiveness

For the purpose of this evaluation, effectiveness relates to how successful the Agency has been in achieving or progressing towards its objectives.

Legal Framework

During 2014-2016, EASME operated according to the legal framework that established it. EASME’s activities, as planned in the annual work programmes and reported in the annual activity reports, corresponded to the tasks set out in the Commission Decisions and the Act of Delegation.

The modalities and procedures of interaction between the Agency and its parent DGs are set out in the Memorandum of Understanding (MoU) between EASME and its parent DGs. The modalities and procedures of interactions prescribed by the MoU with the parent DGs clearly define the spheres of responsibilities between the various entities.

How did the Agency perform in 2014-2016?

The key performance indicators related to the operational efficiency of the project’s lifecycle were positive in the period 2014 - 2016 and the beneficiaries’ satisfaction ranked high (as per results of the online survey). Nevertheless, there is scope for improvement.

During the time period of the evaluation, the Agency fully implemented the operational budget. The implementation of the administrative budget was committed for 99% and the payment implementation increased from 76 to 87% over the three years. The payment implementation was affected in 2014 by the slower than forecast recruitment rate (thus lower salaries and allowances), and also the cancellation of the forecasted office move.  Corrective measures were taken leading to an overall improvement in administrative payment implementation by the end of 2016.

Budget execution, 2014-2016

Source: EASME

In the period 2014 - 2016 the multiannual residual error rate was below the target of 2% for the CIP EEN legacy programme. On the other hand, the multiannual residual error rate for the IEE II (as of 2015) and CIP Eco Innovation (as of 2016) programmes were above 2% and the Agency had to issue reservations in its Annual Activity Report 20 . In 2017, for IEE II it was in a range of 2.7 to 2.8% and for the CIP Eco Innovation in a range of 2.5 to 7.5%. In 2017, the first year available, the residual error rate for Horizon 2020 was 2.2% with a target as close as possible to 2% (within the range of 2 to 5%) (as per Horizon 2020 audit strategy). Given the fact that the majority of projects of the newly delegated programmes to EASME had not reached the appropriate level of maturity (interim/ final payment), the ex-post control audits focused exclusively on the legacy programmes of the Agency.

However, in 2018, EASME had also to issue a temporary reservation for COSME as the residual error rate was temporarily above 2%. This error rate was a preliminary indication calculated on a small and non-representative sample of COSME grants (representing 1.12% of the COSME payments). The sample did not include EEN grants that represent more than half of the COSME grants and have an error rate below 2% in the 2007-2013 period. The Agency has put in place corrective measures that improved the situation and as of 2019 the reservation was lifted.

There are two main factors influencing the error rates and the results of the ex-ante/ex-post controls:

1.Preventive and educative measures; and

2.Beneficiaries’ willingness to comply with the grant agreement rules.

By analysing the open questions in the online survey, some project beneficiaries commented that the criteria for the eligibility of the costs have changed and become more flexible. In some cases, this has proved to be problematic, as “less clear” rules made it difficult to assess the right eligible costs.

The time to grant targets were mostly met during the evaluation period. The time to grant for Horizon 2020 Challenges 3 and 5, COSME and EMFF was clearly within the target of 274 days in both 2015 and 2016. In 2016 the average time to grant for all programmes managed by the Agency (except the Fast Track Innovation - FTI) was within target. The average time-to-inform results demonstrate a faster response to the applicants than the originally scheduled target 21 .

Time to grant, 2014-2016

The Agency received a very low number of evaluation review requests ranging from 0 to 2.4% of the number of proposals evaluated, all below the target of 3%, which is a good indication of the robustness of the grant award process and assurance on the effectiveness of the internal control system 22 .

The survey also shows that EASME’s beneficiaries were largely satisfied with the evaluation process. However, other applicants mentioned their wish for more detailed feedback.

The vast majority of the number of payments (pre-financing, interim and final payments) were executed within contractual limits in 2014 – 2016. As regards the time to pay, in the period 2014-2016 the percentage of payments respecting the legal time limits set up by the Financial Regulation was close to the target of 100%. In 2016, 96% of all payments respect the legal time limits, despite a 32% increase of the total number of financial transactions executed in 2016 compared to 2015.

Average time to pay - % of payments within the legal deadlines

Source: EASME AAR 2014, 2015 & 2016

EASME was also very successful in timely evaluation and selection of proposals. The relevant time-to-inform (TTI) targets were reached and the satisfaction of project applicants and contractors about the time taken to evaluate the proposals is high.

75% of beneficiaries consulted 23 were positive about how supportive EASME’s staff was during the grant application process. The evaluation of applications is generally well received (65,7% positive evaluation). During targeted interviews, some applicants suggested that they would appreciate a more transparent grading system and more constructive feedback in order to improve future applications. EASME’s beneficiaries, are largely satisfied with the way the Agency responded to their needs during the contractual phase. Across programmes, over 75% of respondents had a positive opinion about EASME’s service delivery.

To what extent has EASME considered the beneficiaries’ needs in the services it offers?

Source: targeted online survey

In the period 2014 – 2016 the Agency has improved the management of the programmes in terms of proximity to beneficiaries (through the organisation of European information days, events in the Member States or kick-off meetings). The majority of grant applicants and tenderers responding to the survey were satisfied with the EASME staff’s availability (72.6%;) and its promptness in responding to the applicants’ requests (grant applicants: 73.3%; tenderers: 82.4%). Through concerted actions and making use of synergies between programmes, EASME succeeded in raising the visibility of the EU as promoter of the programmes concerned.

5.2.Efficiency

This section considers the relationship between the resources used by the Agency and the output. The analysis, among other factors, also includes analysis of administrative and regulatory burden and looks at aspects of simplification.

Cost-effectiveness

In the period 2014-2016, EASME was delegated a lower amount of operational budget for Horizon 2020 and EMFF compared to the estimates of the SFS. Whereas, the budget allocated for COSME and LIFE is higher than the SFS estimates.

Operational budget delegated to the Agency, actual vs foreseen (in the 2013 CBA)

Source: VVA based on 2013 CBA, EASME AAR for the years 2014, 2015, 2016 and the SFS 2014

The actual administrative costs were lower than forecasted in the SFS and 2013 CBA for COSME and the EMFF. For Horizon 2020, the administrative costs of managing the programme were (approx.) aligned with the ex-ante estimations, whereas the administrative costs of LIFE were substantially higher.

By expressing the cost-efficiency as the ratio between the administrative budget and the operational budget, the predictions of the 2013 CBA and of the SFS differ from the “actual” results achieved by the Agency in the period 2014-2016.

Administrative over Operational budget (%) in 2014-2016, per programme delegated

Source: VVA based on 2013 CBA, SFS, EASME Annual Activity Report for the years 2014, 2015, 2016 and the SFS 2014

This resulted from the fact that both variables changed: overall, less operational budget has been delegated to EASME, (mainly in relation to Horizon 2020) and at the same time, the actual administrative costs of the Agency were lower. The CBA and the SFS underestimated the difficulty of the massive recruitment that had to take place in 2014. There were delays in the recruitment, but at the same time staff costs were lower. The cancellation of the initially envisaged move of the Agency to new premises generated also savings.

The cost of monitoring the Agency were reviewed together with the parent DGs. They were not deemed to differ from the CBA and SFS estimates.

More information on the EASME cost-efficiency and results of the cost-benefit analysis are presented under section 5.4.

Workload

The Agency’s performance was assessed as effective overall. But the fact that EASME manages a diverse portfolio of programmes and coordinates its activities with a large number of parent DGs made it more complex to manage the launch of new calls and the subsequent activities in 2014.

The actual workload related to the management of the programmes significantly deviated from the initial CBA estimates. Staffing levels between the different programmes was imbalanced, in particular for the LIFE programme. Since 2014 the staff number increased and EASME counted 417 staff by the end of 2016.

Although it is recognised that EASME managed to cope with the available resources and took actions to deal with unexpected workload, the Agency’s resource allocation could be more flexible. As reported by the supporting study, many factors have to be considered when assessing the implementation workload. The programmes closely linked to policy (e.g. Horizon 2020 energy efficiency) require more involvement of the staff of the Agency in the policy related issues and extensive work on policy feedback.

In 2016, the Agency performed a detailed workload assessment. The focus of this exercise was to compare the assumptions made in the CBA in 2013 with the actual situation 24 . According to the data collected in this exercise, most of the workload is situated in operational activities, which account for 63% of all FTEs in the Agency. The second largest category of activities (25%) is finance and control activities. Finally, local activities (e.g. HRM, IT and logistics) represent 12% of all FTEs in the Agency 25 .

In addition, the study makes three general observations on the differences between programmes:

·The stability in delegated actions is an important factor determining workload. Both the COSME and EMFF units were confronted with frequent revisions of the Work Programme leading to changes in the delegated actions. Contrary to the Horizon 2020 units, both programmes have a mix of calls for proposals and calls for tender of which the proportion also changes during the years, leading to extra instability in workload. Half of the budget delegated for COSME is implemented through the EEN, the remaining is allocated to a number of small actions. While smaller actions are necessary to test new ideas, a high number reduces the potential for cost efficiency in the programme implementation.

·Centrally provided support services, and in particular IT-tools for the management of the programme, are vital in moderating the workload.

While acknowledging challenges in terms of workload, further flexibility to redeploy staff had been allowed by the Steering Committee on a temporary and limited basis as of 2017.

Human Resources Management

Staffing and administrative budget are in line with the establishment plan for the planned number of posts. The key challenges of human resource management are identified each year in the Agency’s work programme. During the evaluation period, the main challenges for the Agency concerned recruitment, integration of newcomers, progressive building of core teams and skills development 26 . Over the period 2014 - 2016, EASME’s staff numbers increased in line with the 2013 Communication on delegation of programmes to executive agencies 27 .

It was challenging during the start-up phase of EASME in 2014 to recruit the high number of additional staff scheduled in the budgets. The largest recruitment occurred in 2014, when 140 candidates were recruited, corresponding to a 76% staff growth in that year alone. This massive recruitment operation allowed EASME to cover the technical and financial skills required for delivering effective programme management. Some limited delays and budget carried forward from 2014 were successfully absorbed in 2015.

EASME has been able to recruit suitable profiles, skills and qualifications and has received positive feedback. Most of the interviewees (EASME employees, parent DGs, experts) perceive EASME’s staff as dedicated and competent. The general observation among the parent DGs is that EASME staff have a high level of expertise in managing the projects and a good understanding of the policy context.

During the assessed period recruitment process faced three other major challenges: understaffing for certain programmes, delays in filling the posts, and challenges in filling certain posts (notably Heads of Sector).  The Agency proposed a flexibility mechanism approved by the Steering Committee, allowing a temporary and limited reallocation of human resources. Despite the difficulties to fill the vacancies, by 31 December 2016 EASME managed to reach its recruitment target as foreseen in the Agency’s Work Programme (415 staff members).

The staff turnover rate recorded in EASME remained stable around 7%, lower than the benchmark rate of 10% for contract agent staff. According to the 2016 Staff Opinion Survey, EASME staff positively evaluated the initiatives put in place by the EASME management to improve the turnover rate, such as the adoption of a new management charter and the introduction of trainings. According to the same survey, 66% of EASME staff consider the Agency as a modern and attractive place to work. This corresponds to an increase of 6% compared to the 2014 Staff Satisfaction Survey, although still below ERCEA (78%) and REA (76%).

EASME managed to improve all indicators under the section of wellbeing of the Staff Opinion Survey in 2016. Compared to other executive agencies, EASME scored highest in the work-life dimension. The increase in the wellbeing dimension can be seen as a success of the Agency’s actions for promoting staff’s work-life balance through concrete initiatives during the assessed period.

Only 21% of the 2016 staff survey respondents said that there were reasonable opportunities to move to another job that matches their current skills and competences. In terms of current job, 76% of EASME staff members reported to be satisfied with their job content and tasks, which corresponds to an increase of 6% compared to 2014 staff survey results.

The evolution of key indicators for staff satisfaction – after the evaluated period – is encouraging: In the latest staff survey (4th quarter of 2018), EASME managed to increase its result for staff feeling that the Agency cares about their well-being to 72%, up from 54% in 2016. Staff engagement rose to 71% (from 70% in 2016). A less positive issue is related to the limited career opportunities within the Agency 28 and the professional future developments. This trend is also confirmed by the 2016 staff survey and it is common to the other executive agencies. The main issues identified in the survey will have to be addressed by the Agency together with its parent DGs in the most appropriate way and will feed into a follow-up action plan.

Organisational structure

The organisational structure of EASME has been assessed as functional:

·Under the 2014-2020 MFF, the number of parent DGs increased from four to seven (GROW, ENER, ENV, CLIMA, RTD, CNECT and MARE), which added more complexity to the Agency’s governance mechanisms.

·The Agency was delegated new programmes 29 and as a consequence expanded from six units (in the predecessor agency) to three departments in EASME and eight units. Two departments deal with the various new programmes (one is in charge of COSME, Horizon 2020 SME and EMFF, the other LIFE, Horizon 2020 Energy, Environment and Resources), the third one deals with finances and administration 30 .

This allocation of tasks in the different departments allows synergies (for example between COSME and Horizon 2020) 31 .

The structure of the Agency was also streamlined through the regrouping of the financial tasks into one unit to increase economies of scale and the effectiveness and efficiency of financial control 32 . In addition, a procurement cell was created to centralise all administrative tasks related to the launch and evaluation of calls for tenders 33 .

Interviewees (parent DGs and EASME staff) mentioned that a further streamlining of EASME’s organisational structure could contribute to generate more synergies in the future between the different programmes. In this regard, there are specific actions to create further links, such as the initiative PROBE that aimed at linking different results of operational units by analysing the diversity of data from all programmes.

Simplification

To increase its efficiency, the Agency took measures to simplify procedures. For example, in 2014, following an internal audit, the Agency took several cost-saving initiatives in relation to missions. In 2015, following a thorough screening of business processes, a more generalised introduction of e-signatories was introduced and approval workflows were rationalised.

The feedback gathered from stakeholders indicate there is room for further improvement. In particular applicants and beneficiaries would appreciate better planning of calls. This would help them better manage the timing of submission of proposals, preparation and conclusion of grants. Some parent DGs also considered that the preparation of procurement and grant procedures and evaluation of ensuing proposals and tenders could better take into account the situations of low risk to improve efficiency without reducing compliance.

The IT infrastructure is considered satisfactory and effective by the interviewees (in particular EASME staff). EASME receives the IT services and tools from the Commission and is in charge of maintaining the infrastructure. In 2014, new IT tools for the management of Horizon 2020 programmes were introduced in EASME and a sector for horizontal IT and logistic topics established. COSME started using adapted Horizon 2020 IT tools for the project management. For the LIFE programme, the IT tools available in DG ENV and DG CLIMA were used. The EMFF programme was implemented without any specific tools.

However, the implementation of IT tools differs across programmes: the tools for Horizon 2020 are more developed and have been used as a basis for other programmes with adaptation. The introduction of these IT tools initially brought difficulties both for staff and applicants but did not impact the projects negatively. Furthermore, the different programmes have very specific needs.

5.3.Coherence

The evaluation did not find any evidence of overlaps, duplications, gaps or inconsistencies within the EASME programme portfolio. In the 2014-2016 period EASME demonstrated flexibility in implementing a diverse thematic programming portfolio and being answerable to seven different parent DGs.

Portfolio coherence

There are no particular inconsistencies between the tasks performed by EASME in implementing the different programmes. The supporting study mentions that “the delegation of operational tasks regarding the implementation of the Framework Programme for Research and Innovation (Horizon 2020), the Programme for the Competitiveness of Enterprises and small medium size enterprises (COSME) and the Programme for Environment and Climate Action (LIFE) had generated synergies, simplification and economies of scale.” However, the study also mentions that there is potential to further exploiting synergies between different programmes.

The main difficulty identified is the tendency to work in silos given that the programmes have different objectives and procedures. Synergies could have been created by grouping some activities (e.g. workshops, presentations, promotional activities) based on the content (e.g. energy or environment, support to SMEs). This could promote knowledge spill-overs and learning curve effects.

According to the interim evaluation of Horizon 2020 34 , there is a good level of coherence between the SME Instrument and the COSME programme, with synergies created by making use of the COSME Enterprise Europe Network to ensure access to the Horizon 2020 SME Instrument business innovation coaching and mentoring services. EASME staff also reported benefits from the collaboration between the units working on the Horizon 2020 SME instrument and COSME, such as using the same structure to identify coaches, dialogue with investors and offer support during proposal preparation. In addition, some COSME EEN members are involved in a selection of INNOSUP calls of Horizon 2020 35 , which offers highly specialised support services that aim to complement existing national and regional support initiatives 36 . Synergies were also created between LIFE, Horizon 2020 Energy, Environment and Resources actions.

Synergies can be exploited also in external communication. The networks and multipliers of the Agency could be used across programmes. For example, during the EU Sustainable Energy Week, SME Instrument companies in the field of Energy Efficiency pitched their projects and products 37 .

Relations with parent DGs and policy feedback

The rationale behind the creation of executive agencies is partly to allow the Commission to focus on institutional tasks, mainly on its core activities and functions that cannot be delegated, such as policy-making, implementation and monitoring of the application of the EU law, and strategic management. Almost all interviewees from the parent DGs expressed the opinion that the delegation of the implementation of their programmes to EASME has enabled them to better focus on other tasks, mainly their policy-related tasks.

When examining any potential overlaps and inconsistencies between the tasks and responsibilities of the Agency and the parent DGs, the supporting study found that there is a clear and appropriate delimitation of responsibilities and tasks between the Agency and the parent DGs within the current legal framework. This is particularly important given that the Agency has seven parent DGs and implements a diverse set of programmes.

The division of tasks is clearly outlined in the legal framework. A list of the tasks outsourced to EASME under each programme is provided in the Annexes to the Act of Delegation. The general principles from the Act of Delegation are further set out in a Memorandum of Understanding between EASME and all the parent DGs. The detailed modalities and procedures of interaction between the Agency and its parent DGs are defined in four programme-specific Memoranda of Understanding between the Agency and the parent DGs ultimately responsible for the programme. The sharing of responsibilities for the execution of tasks between the Commission and EASME varies according to the scope and the level of political sensitivity of each action and is detailed in each MoU. These tasks are even further concretised in the Annual Work Programme of the Agency and the Multiannual and Annual Work Programmes of the different programmes. These documents present the key actions, targets, Key Performance Indicators (KPIs) and budget lines of all the programmes managed by the Agency, as well as the responsibility for each action at the level of units within the Agency.

However, it appears from the interviews that this legal framework might not be clear or known to all Commission and EASME staff. For example, in the area of policy feedback, DGs have different views of the level of the Agency’s involvement and the level of information to be shared with the DGs.

The creation and further reinforcement of the Agency has allowed a retention of talent. While the Commission was reducing its resources, a number of contractual agents moved from the DGs to the Agency. In this way, the Agency has incorporated a good understanding of how the Commission works, facilitating the collaboration between the two entities. On the other hand, for Commission officers seconded to the Agency, working in EASME is a great opportunity to acquire management skills.

The exchange of information between the Agency and the parent DGs works well on administrative and financial issues, however according to some staff interviewed from a few parent DGs, the information sharing related to the implementation of projects or useful for policy developments could be improved. EASME staff thinks that a better definition of the needs of the parent DGs would help them in better addressing those needs.

The evaluation shows a mixed result for the collaboration and know-how sharing. There is a fruitful collaboration between units in parent DGs and units/sectors in EASME but some policy officers in the parent DGs feel a loss of know-how and contact with stakeholders.

Policy feedback on project implementation is an essential element for the Commission to play its role of policy-maker. EASME provides regular and detailed factual reports to the DGs containing information and data on the implementation of projects under the various programmes. The supporting study reported on positive examples, in particular as regards the policy-feedback mechanisms set up between EASME and DG ENER. The supporting study identified as a key factor the high degree of involvement of the Agency in policy and programme shaping issues. Nevertheless, in other cases and according to some interviews, the policy feedback produced by the Agency do not always respond to the DGs’ needs which in turns must be clearly defined.

Relations with other executive agencies

There is no overlap between the tasks of EASME and of other executive agencies. Most collaboration takes place with REA and INEA in the implementation of Horizon 2020. The evaluation reports smooth coordination and communication with INEA and REA. Some stakeholders mentioned that collaboration with other executive agencies could be further strengthened on cross-cutting focus themes to learn from each other and increase synergies on work procedures.

5.4.Relevance

The performance of the Agency has been relevant. It has responded to the needs for which it has been created. It has allowed the Commission to focus on policy-making and has allowed the creation of synergies, has contributed to simplification and streamlining as well as economies of scale.

Overall, it is perceived by all parent DGs, experts, applicants and beneficiaries alike, that EASME has delivered good services. The client-oriented approach has benefited beneficiaries and applicants in the first place. The expertise of EASME is highly valued by some parent DGs. The economies of scale the Agency benefits from in handling project applications (e.g. for LIFE) could not have been matched by the Commission. It is particularly noted that EASME succeeded in effectively creating a community of SMEs, which results in synergies across all programmes.

5.5.Retrospective Cost-benefit analysis

The 2013 CBA examined four possible scenarios (full internalisation, full externalisation and two alternative scenarios with different options for delegation). The results of the ex-ante CBA were then adapted in the Specific Financial Statement (SFS).

The supporting study compared two scenarios, full externalisation and in-house scenario (or full internalisation), by using actual data. The actual costs of the executive agency scenario were lower than forecast in the SFS and in the 2013 CBA. The actual savings were higher than the initial SFS estimations (EUR 33.9 million compared to EUR 9.89 million). This is due to the assumptions used in the calculations of the “in-house scenario” and the lower number of FTEs who worked in EASME. Savings of the executive agency scenario (in the 2013 CBA, in the SFS and in the “actual” scenario) primarily resulted from the higher share of CAs employed within the Agency.

Planned versus actual total costs under the in-house and externalisation scenarios in 2014-2016 

Source: Own elaboration based SFS, EASME financial accounts for the years 2014, 2016 & 2016

To perform the delegated tasks, the Agency’s resources were calculated on the basis of a cost-benefit analysis (CBA) and fixed at the end of 2013 in the Communication chapeau and the Specific Financial Statement accompanying the Agency’s establishment decision. In December 2015, the Agency assessed the workload with an internal study. The focus of this study was to compare the assumptions made in the CBA in 2013 with the actual situation 38 . In 2016, the Agency had 363 FTEs at its disposal (EASME, 2018). This number is clearly lower than the number of posts (“heads”) foreseen for the Agency in that same year, i.e. 437 posts (of which 417 were filled at the end of 2016). The reason is that not all staff worked full-time for the whole year (e.g. newcomers whose contract started later in the year, staff with part time working arrangements, sick leaves, long-term leaves or time credits).

6.Conclusions

Effectiveness

EASME has been effective in allowing the Commission to focus on institutional tasks by implementing the programmes that have been delegated. EASME manages substantial parts of four different delegated programmes covering a diverse range of activities and funding instruments responding to seven parent DGs.

Efficiency

In terms of cost-efficiency, the actual figures for the period 2014-2016 show that both administrative costs and total budget delegated were lower than envisaged in the 2013 CBA or in the Specific Financial Statement.

The costs of the options set out in 2013 were recalculated in the light of the actual costs of the implementation. This shows that the savings were more important than envisaged. This is mainly due to the staff structure of the Agency that is in majority composed of contractual agents.

The level of staff well-being in EASME is good, and constant efforts are made to improve it. Several initiatives have been taken by EASME management and are still continuing after the time scope for this evaluation. Nevertheless, according to the staff survey report, staff has the feeling to be limited in career development and this is mainly due to the contract type used. This remains a challenge for the future of executive agencies.

Difficulties in recruiting new staff during the first year generated delays in the programmes’ implementation. The Agency managed to overcome the difficulties and succeeded in building up an organisational structure that delivered, based on the diversity of the delegated programmes.

Coherence

EASME activities have been found to be coherent with the Commission activities and with the portfolio delegated to the other executive agencies. EASME has been able to produce synergies at different levels by organising events that were beneficial to stakeholders of various programmes, by exploiting the synergies that exists between the Enterprise Europe Network and Horizon 2020 actions or by pooling expertise (in the field of financial resource for example) in specialised teams or units to the service of the various DGs.

Nevertheless, further synergies could be exploited. The structural organisation of EASME around thematic departments is convenient from the point of view of everyday work but it can also contribute to creating silos. It is clear from interviews and surveys with Commission staff and economic operators that participated in the different programmes that the IT tools and procedures could be further simplified.

The legal framework and the division of tasks between the Agency and the Commission are clear on paper. However, in practice the boundaries between policy development and project implementation are difficult to draw. Policy feedback is key for the Commission to play its role of policy-maker. In some areas (energy) EASME provides relevant feedback and tangible contribution to policy implementation, while in other areas there is scope for improvement. To this end, the Commission services should provide a more precise definition of their needs. Coordination and cooperation between Commission staff and EASME staff are reported to work and to produce results.

Relevance and future role of the Agency

The action of the Agency is still relevant, it has allowed the Commission to pursue its political objectives, while allowing economies of scale and savings, producing synergies and improving its performances as the measurement of key performance indicators demonstrate.

Margins for improvement can be found in the fields of policy feedback, faster procedures, synergies between programmes, simplification and career development for the staff of the Agency. To address the issue of policy feedback, the Agency has already organised several workshops to share the good practices of each delegated programme with all parent DGs.

The parent DGs and the Agency will prepare an action plan to address the shortcomings identified in this evaluation. This action plan will aim to increase the Agency’s performance and contribution to an efficient implementation of the delegated programmes.

ANNEX - Evaluation Questions

Efficiency

Q1: To what extent was the Agency operating according to the legal framework establishing it?

Q2: To what extent has the Agency achieved its objectives? What could be done to render EASME more effective in achieving these objectives?

Q3: To what extent has the Agency contributed to improved management of the programme(s) in terms of the elements assessed in the 2013 Cost-Benefit Analysis?

Q4: To what extent have the activities of the Agency resulted in unintended effects (both desirable and undesirable)?

Effectiveness

Q5: Did the actual costs (including cost of coordination and monitoring) of the Agency correspond to the estimates of the 2013 Cost-Benefit Analysis which was carried out in view of its establishment? If not, what are the reasons behind it?

Q6: Did the actual benefits correspond to the estimates of the 2013 Cost-Benefit Analysis? If not, what are the reasons behind it?

Q7: Was the management and execution of the programme(s) by the Agency cost-efficient?

Q8: To what extent has the Agency contributed to improved management of the programmes in terms of simplification of the procedures and flexibility in the implementation of outsourced tasks (e.g. capacity to adapt to periods of high workload)? Where is there further scope for simplification?

Q9: What aspects/means/actors or processes render the Agency more or less efficient? What could be improved?

Q10: To what extent does EASME’s human resources management contribute to the achievement of the Agency’s objectives?

Coherence

Q11: Are there overlaps/gaps/inconsistencies within the programme portfolio managed by the Agency?

Q12: Is there a clear and appropriate delimitation of responsibilities and tasks between the Agency and the parent DGs (no overlaps or gaps)?

Q13: To what extent are there overlaps/inconsistencies between the tasks carried out by EASME and other EU executive agencies?

Relevance

Q14: Have objectives and tasks responded to past needs and still respond to current needs?

(1)  2013/771/EU: Commission Implementing Decision of 17 December 2013 establishing the ‘Executive Agency for Small and Medium-sized Enterprises’ and repealing Decisions 2004/20/EC and 2007/372/EC.

(2)  There is an act of delegation for each of the programmes delegated to EASME.

(3)  Commission Regulation (EC) No 1653/2004 of 21 September 2004 on a standard financial regulation for the executive agencies pursuant to Council Regulation (EC) No 58/2003 laying down the statute for executive agencies to be entrusted with certain tasks in the management of Community programmes and modification by the Commission Regulation (EC) No 651/2008 of 9 July 2008.

(4)  Commission’s Better Regulation Communication COM (2015) 215 final and its accompanying SWD (2015) 111 final and SWD (2015) 110 final.

(5)  Cost-benefit analysis for the delegation of certain tasks regarding the implementation of Union Programmes 2014-2020 to the executive agencies - Final report for the Commission of 19 August 2013.

(6)

 The cost-benefit analysis shall take into account a number of factors such as identification of the tasks justifying outsourcing, a cost-benefit analysis which includes the costs of coordination and checks, the impact on human resources, possible savings within the general budgetary framework of the European Union, efficiency and flexibility in the implementation of outsourced tasks, simplification of the procedures used, proximity of outsourced activities to final beneficiaries, visibility of the EU as promoter of the EU programme concerned and the need to maintain an adequate level of know-how inside the Commission.

(7)

 Appendix II of the Guidelines for the establishment and operation of executive agencies financed from the Union budget (C (2014) 9109 from 2 December 2014; pp. 64-72).

(8)  2013/771/EU: Commission Implementing Decision of 17 December 2013 establishing the ‘Executive Agency for Small and Medium-sized Enterprises’ and repealing Decisions 2004/20/EC and 2007/372/EC.

(9)  Except for integrated projects and technical assistance projects, which have been managed by DG ENV until 2017.

(10)  SFS C(2014)6944 and C(2014)4636.

(11)  The estimated efficiency gains over the period 2014-2024, covering full completion of the projects according to the SFS of 2013, were of EUR 158 million.

(12)  Commission Decision (EC) 2004/20 setting up the Intelligent Energy Executive Agency (IEEA).

(13)   Commission Decision 2007/372/EC of 31 May 2007 amending Decision 2004/20/EC in order to transform the Intelligent Energy Executive Agency into the Executive Agency for Competitiveness and Innovation .

(14)  Commission implementing decision of 17 December 2013 establishing the ‘Executive Agency for Small and medium-sized Enterprises’ and repealing Decisions 2004/20/EC and 2007/372/EC (2013/771/EU).

(15)

Memorandum of Understanding between the Executive Agency for Small and medium-sized enterprises and DG Enterprise and Industry, DG Research and Innovation, DG Energy, DG Environment, DG Climate Action, DG Communication Networks, Content and Technology & DG Maritime Affairs and Fisheries of 5 June 2014.

(16) See table 16, Study supporting the evaluation of the operation of EASME, Final Report.

(17)  Horizon 2020 is the European Union’s Framework Programme for Research and Innovation.

(18)

Evaluation of the Executive Agency for Small and Medium-sized Enterprises (EASME), Valdani Vicari & Associati, Deloitte, April – 2019.

(19)   https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2017-5558220/public-consultation_en

(20) Due to the small proportion of the financial impact on the overall financial portfolio of the Agency, these reservations were lifted by the Agency as of its 2019 Annual Activity Report.

(21) EASME AAR 2016, https://ec.europa.eu/easme/en/about-easme

(22) EASME AAR 2015, https://ec.europa.eu/easme/en/about-easme

(23) Customer satisfaction report, 2.2.10.

(24) Please refer to the Cost Benefit Analysis attached to the Evaluation Report from the external supporting study.

(25) The categorisation of activities in operational, neutral and local activities follows DG HR’s typology used in their screening exercise of executive agencies’ staff screenings.

(26) Annual Activity Reports EASME 2014, 2015 and 2016.

(27)  Communication to the Commission on the delegation of the management of the 2014-2020 programmes to executive agencies SEC(2013)493.

(28) Results of interviews conducted with EASME staff.

(29)  Annual Activity Report, 2014, p.34.

(30)   https://ec.europa.eu/easme/sites/easme-site/files/easme-organigramme_en.pdf  

(31) According to the majority of EASME employees and parent DGs, information from interviews.

(32)  Annual Activity Report, 2014, p. 29.

(33) Annual Activity Report, 2014.

(34)  Commission Staff Working Document - In-Depth Interim Evaluation of Horizon 2020 (2017). SWD(2017) 220 final.

(35)  The Horizon 2020 INNOSUP programme aims to test new approaches for better innovation support through funding opportunities for innovation actors across Europe. https://ec.europa.eu/easme/en/innosup

(36)  Technopolis (2017). Interim evaluation of the COSME programme.

(37)  EASME 2016 Annual Activity Report.

(38)  The exercise covered data for the year 2015 and was completed in July 2016 for all operational units.