Table of contents

List of abbreviations2

1.Introduction3

2.Background on the intervention logic of the SRSP5

3.Implementation / state of Play9

4.Methodology16

5.Analysis and answers to the evaluation questions20

5.1.    Relevance    20

5.2.    Effectiveness    25

5.3.    Efficiency    32

5.4.    Coherence    42

5.5.    EU added value    50

6. Conclusions55

Annex 1: Procedural information59

Annex 2: Stakeholder consultation61

Annex 3: Evaluation framework66

 

List of abbreviations

Term or acronym	Meaning or definition
CSP	Cooperation and support plan
CSR	Country-specific recommendation
DG	Directorate-General
DG REFORM	Directorate-General for Structural Reform Support
ESIF	European Structural and Investment Funds
EU	European Union
EUR	Euro
ISSG	Interservice Steering Group
PA	Preparatory Action
SRSP	Structural Reform Support Programme
SRSS	Structural Reform Support Service
SWD	Staff working document
TAIEX	Technical Assistance and Information Exchange instrument

1.Introduction

Well-defined reforms are crucial for modernising economies, encouraging investment, creating jobs and raising living standards. To facilitate these reforms across the European Union, Regulation (EU) 2017/825 1 on the establishment of the Structural Reform Support Programme (‘the SRSP Regulation’) was adopted in May 2017. The objective of the Structural Reform Support Programme’s (the ‘programme’) is to contribute to institutional, administrative and growth-sustaining reforms in Member States by providing technical support to national authorities. The SRSP has a budget of EUR 222.8 million for the period 2017-2020. 

In accordance with Article 16 of the SRSP Regulation, the programme is subject to a mid-term evaluation. The overall goal of this evaluation is to assess the programme’s  relevance, effectiveness, efficiency, coherence and EU added value, in line with the Better Regulation guidelines, and to draw lessons that can serve possible future improvements 2 . The mid-term evaluation also ensures the transparency and accountability of the Commission’s activities.

To support the mid-term evaluation exercise, an external contractor was engaged to perform an evaluation report. This report was carried out over a period of 12 months, starting on 20 November 2018. It covers the technical support provided under the SRSP’s 2017 and 2018 budgetary cycles, as well as projects funded through the voluntary transfers of additional resources to the SRSP budget by Greece and Bulgaria in 2018 3 . It also covers technical support projects funded under the Preparatory Action of 2016 4 . In total, 303 technical support projects in 24 Member States fall within the scope of this evaluation.

In this staff working document (SWD), the Commission’s staff presents and reflects on the main outcomes and findings of the mid-term evaluation and provides evidence and data which could serve as a basis for further improving the functioning of the SRSP where relevant. The SWD also draws on other sources of information from the Directorate-General for Structural Reform Support, including its internal IT system and the annual monitoring reports, annual activity reports, and ex ante evaluation of the SRSP. It should be underlined that the evaluation was performed when the programme’s implementation was still at an early stage. This SWD outlines exploratory conclusions, proportionate to the scope of the evaluation.

The SWD has the following structure. Firstly, it describes the background of the SRSP and summarises the programme’s state of play. Subsequently, the SWD provides a brief overview of the methodology used for the evaluation, including associated limitations to the robustness of the findings. The main results of the evaluation report are presented in the next section and structured according to the five Better Regulation criteria. The last section of the SWD provides the overall conclusion and lessons learned.

2.Background on the intervention logic of the SRSP

The effective and efficient design and implementation of administrative and institutional reforms is essential for flexible economies that are able to adapt to changing circumstances. Fruitful and timely implemented reforms contribute to a favourable regulatory and institutional environment. This, in turn, reinforces competitiveness, supports job creation, and enhances investment and sustainable growth.

In the aftermath of the economic crisis, many EU Member States have been facing low potential growth, significant macroeconomic imbalances and obstacles to investment. Reforms were needed to address these challenges and enable governments to continue adapting to changing circumstances. It is an EU political priority to advance the implementation of growth-enhancing reforms in order to strengthen the capacity of Member States to adjust their economies. Not only does this help to boost economic growth, job creation and social welfare, but it also contributes to the process of convergence across the EU.

The process of designing and implementing institutional and growth enhancing reforms is complex and requires a high degree of knowledge and skills. Member States have different levels of in-house expertise and capacity to prepare and carry out much-needed reforms. Furthermore, they have to constantly keep up with changes, whether political or from the world around us. Often, the Member States most in need of reform are at the same time the ones least equipped. Some EU countries do not have sufficient capacity to initiate and implement reforms in response to the reform challenges that lie ahead. Others have the capacity, but their administrations are still struggling to adapt to constantly changing circumstances arising from globalisation and technological and international developments.

Experience in economic policy coordination through the European Semester framework shows that the implementation of some reforms is lagging behind. Although in the multiannual assessment Member States achieved at least ‘some progress’ with the implementation of around two thirds of the country-specific recommendations (CSRs), the implementation of the annual CSRs has a slower pace overall. Furthermore, Member States sometimes struggle with the adequate and timely implementation of EU legislation. Given that the economies of EU Member States are strongly intertwined, especially those that take part in the Economic and Monetary Union, inadequate implementation of reforms can have negative effects beyond the borders of the Member State concerned.

To address these challenges, the Commission decided to engage in closer cooperation with Member States to help them implement comprehensive reforms that foster common goals. The Commission already had positive experiences with supporting reforms in Greece and Cyprus, in light of the euro crisis that started in 2009. At the request of the national authorities, the Commission created the Task Force for Greece (2011) and the Support Group for Cyprus (2013). The objective was to help these countries make a series of reforms in a wide spectrum of public policy areas. These reforms formed part of their financial assistance packages from the EU. Given the success of these task forces, and seeing the benefits that reforms could have in making economies more flexible and competitive, the Commission decided to broaden the scope of the support provided. In 2015, the Structural Reform Support Service (SRSS) was set up to roll out support for reforms to all EU countries.

In 2017, a dedicated financing instrument was adopted: the Structural Reform Support Programme (SRSP). The SRSS, and subsequently, the Directorate-General for Structural Reform Support (DG REFORM), have been managing the programme since 5 . As laid down in the SRSP Regulation, the SRSP’s general objective is to contribute to institutional, administrative and growth-sustaining reforms in the Member States by providing support to national authorities. 6 The support may also help Member States who are preparing for participation in the euro area 7 . In order to achieve this general objective, the SRSP has set the following specific objectives 8 :

a)support the initiatives of national authorities to design their reforms according to their priorities, taking into account initial conditions and expected socio-economic impacts;

b)support the national authorities in enhancing their capacity to formulate, develop and implement reform policies and strategies and in pursuing an integrated approach ensuring consistency between goals and means across sectors;

c)support the efforts of national authorities to define and implement appropriate processes and methodologies by taking into account good practices of and lessons learned by other countries in addressing similar situations;

d)assist the national authorities in enhancing the efficiency and effectiveness of human-resource management, inter alia, by strengthening professional knowledge and skills and setting out clear responsibilities.

The programme’s general and specific objectives are also described in the intervention matrix ( Figure 1 ), visualising the relations between the different components of the intervention.

The SRSP is the main funding programme for projects providing technical support for reforms to EU countries. Support under the SRSP is available to any Member State facing challenges in designing and implementing reforms. This includes reforms to address challenges identified in the European Semester process of economic policy coordination (e.g. the CSRs), reforms to address EU priorities or reforms undertaken at the Member State’s own initiative. Furthermore, technical support can be provided to help Member States fully implement EU law. EU countries can also request technical support for economic adjustment programmes. Moreover, since the amendment to the SRSP Regulation in 2018, the SRSP can also finance reform-related actions and activities to help Members States prepare for their entry to the euro area 9 .

The technical support can be provided to help design and implement reforms that relate to a broad range of policy domains, including the following public policy areas:

·revenue administration and public financial management;

·governance and public administration;

·growth and business environment (including climate, energy and environment);

·labour market, education, health and social policy;

·financial services and access to finance.

The support provided through the programme is demand-driven and tailor-made to suit the specific situation of the country concerned. Actions eligible for funding include the formulation of policy strategies or roadmaps, the provision of long- and short-term experts, and capacity-building activities, such as workshops, training and working visits. More examples of eligible actions are presented in the logic of intervention (Figure 1).

The technical support to Member States is intended to help successfully design and/or implement the relevant reforms by tackling obstacles in the reform process. The results expected in the short term include the provision of strategies, reform roadmaps or drafts of new legislation. Other short-term results expected are new or improved procedures and methodologies, as well as organisational change and enhanced human resource management 10 . The programme’s long-term impacts are expected to be: (a) strengthened institutions, governance and public administration; (b) enhanced sustainable growth, job creation and social inclusion; and (c) enhanced cohesion, competitiveness, productivity and investment (see Figure 1).

Figure 1: Logic of intervention (source: on the basis of the SRSP Regulation, EY 2020)

Technical support projects can cover any part of the reform cycle – from identifying the needs to monitoring and evaluating the outcomes. At the same time, the effective follow-up of the actions, in terms of actual implementation of the reforms, remains the prerogative of the recipient Member State. Ownership of the reforms by the Member State is essential for their successful implementation.

As the SRSP is a newly established funding programme, the current mid-term evaluation is the first major evaluation exercise for the programme as a whole. In order to ensure that the technical support to Member States is effective and cost-efficient, the Commission monitors the programme’s performance, based on the indicators that are set out in Annex 1 of the SRSP Regulation. In addition to the mid-term evaluation, the Commission also uses other mechanisms to monitor the programme’s progress, including annual monitoring reports.

3.Implementation / state of Play

3.1.How the SRSP is implemented

The submission, analysis and selection of technical support requests under the SRSP is organised in annual cycles, in accordance with the provisions of the SRSP Regulation. The Member States have until the end of October to submit their technical support requests for the budgetary cycle of the upcoming year 11 . Once they have done so, the Commission assesses these requests, based on the criteria set out in Article 7 of the SRSP Regulation 12 . To avoid double funding and ensure complementarity, the Commission verifies whether requests overlap with actions implemented under other EU instruments or programmes. The Commission then selects the requests that are best suited for funding. Subsequent to this, the Commission agrees with the Member State concerned on the priority areas, the objectives, an indicative timeline, the scope of the support measures and the estimated financial contribution. This agreement between the Commission and the individual Member State is the cooperation and support plan. The cooperation and support plans are multiannual documents, signed and amended each year (if need be) by the Commission and Member States requesting support from the SRSP. The technical support projects financed through the programme do not require co-financing by the Member State concerned.

The SRSP is implemented on the basis of annual work programmes. The annual work programmes are adopted every year and serve as a Financing Decision. They set out the support measures to be implemented for the year concerned, the dedicated annual budget, and the expected results. The annual work programmes include the priority actions for the programme, as agreed between the Commission and the individual Member States in the respective cooperation and support plans.

To facilitate efficient communication with the Commission and the coordinated submission of technical support requests, each Member State has designated a national coordinating authority within its government structures. This coordinating authority functions as the Commission’s interlocutor for implementing the cooperation and support plans, and the SRSP in general. It is responsible for collecting all requests for the Member State in question, and subsequently submitting the requests to the Commission. The coordinating authority can also be responsible for prioritising the requests before submitting them to the Commission. The national beneficiary authorities are the actual recipients of the technical support, for example the thematic ministries that have requested assistance. The SRSP never gives money to the authorities, only technical support, such as strategic and legal advice, studies, training and expert visits on the ground. The coordinating authority is expected to coordinate the ensuing implementation of the support measures in the country and is also encouraged to take appropriate action if challenges arise during the technical support’s implementation.

3.2.Implementation of SRSP 2017 and SRSP 2018

The budget for SRSP 2017 and SRSP 2018 was EUR 53 million ( Figure 2 ) 13 . Under SRSP 2017, 159 requests from 16 Member States were selected to receive funding, out of the 271 requests submitted for a total of EUR 83 million – an amount almost four times the EUR 22.5 million budget available for 2017. In 2018, the SRSP’s second year of implementation, there was an increase in demand from Member States. In total, 146 requests from 24 Member States were selected for funding under SRSP 2018, out of the 444 support requests submitted. The submitted requests amounted to an estimated total value of EUR 152 million. This amount was five times the budget available for SRSP 2018 (EUR 30.5 million).

Funding source	Number requests submitted	Number of requests selected	Available budget
SRSP 2017	271	159	EUR 22.5 million
SRSP 2018	444	146	EUR 30.5 million
Total	715	305	EUR 53 million

Figure 2: Number of requests submitted and selected under SRSP 2017 and SRSP 2018 (source: Annual Monitoring Report 2017 and 2018)

Figure 3  shows the number of requests submitted per Member State under SRSP 2017 and SRSP 2018.

The high demand from Member States in both 2017 and 2018 indicates a significant interest in the instrument. In line with the principle of sound financial management, the Commission prioritised requests allowing for quick delivery of support and rapid implementation of reforms on the ground, and requests addressing defined objectives, with strong results expected on the ground.

Figure 4  shows the number of requests selected per Member State under SRSP 2017 and SRSP 2018.

Figure 4: Number of requests selected for funding under under SRSP 2017 and SRSP 2018 per Member State (source: SRSP Annual Monitoring Report, 2018)

Figure 5  shows the circumstances of the requests that have been selected under SRSP 2017 and SRSP 2018. Over half of the selected requests were related to implementing challenges identified in the context of economic governance processes, in particular to CSRs.

Circumstances of the selected requests	SRSP 2017	SRSP 2018
Economic governance process, including CSRs	51%	55%
EU priorities	27%	29%
EU law	7%	6%
Member States’ own initiative	11%	7%
Economic adjustment programmes	4%	3%

Figure 5: Circumstances of the selected requests under SRSP 2017 and SRSP 2018 (source: SRSP Annual Monitoring Report 2017 and 2018)

When the Commission decides to fund a request for technical support, the request becomes a project 14 . In the selection process, the Commission can decide to implement one large request through multiple projects. Similarly, the Commission can also decide to implement multiple related requests through one single project. Therefore, the number of selected requests can differ from the number of projects (i.e. support measures) that are implemented. The number of projects approved under SRSP 2017 and SRSP 2018 is provided in Figure 6 . 

Funding source	Number of approved projects
SRSP 2017	144
SRSP 2018	143
Total	287

Figure 6: Number of projects approved under SRSP 2017 and SRSP 2018 (source: Commission’s internal IT system, EY 2020)

In total, 246 technical support projects funded under SRSP 2017 and SRSP 2018 were evaluated as part of the mid-term evaluation, covering 90% of the allocated financial resources under the two budgetary cycles. More than 50% of the evaluated projects under SRSP 2017 and SRSP 2018 were implemented in 7 Member States (Romania, Croatia, Malta, Cyprus, Lithuania, Slovenia, and Poland). A detailed overview of the distribution of evaluated projects among EU countries is provided in  Figure 7 15 . 

Technical support under SRSP 2017 and SRSP 2018 covered all five thematic areas. As demonstrated in Figure 8 , out of the evaluated projects funded under SRSP 2017 and SRSP 2018 the largest share of the approved budget was allocated to projects in area of growth, business environment and sectoral issues (EUR 13.5 million), followed by labour market, education, health and social services (EUR 11.4 million) and revenue administration and public financial management (EUR 9.1 million).

In line with the SRSP Regulation, the measures of the programme are implemented either directly by the Commission or indirectly in accordance with the Financial Regulation. Under direct management this includes public procurement contracts, grants and reimbursement of costs incurred by external experts. The underlying procedures for concluding contracts/agreements with technical support providers under the SRSP are in line with the Financial Regulation and thereby in accordance with the principle of sound financial management.

In line with the Financial Regulation, some activities are implemented under the indirect management mode in view of the type of the implementing entity. Entities, such as international organisations, which have undergone an ex ante (/pillar-) assessment of their procedures and systems, guaranteeing a level of protection of the EU financial interests equivalent to that guaranteed by the Commission in direct management, can act under indirect management for the tasks they undertake to implement the actions.

The annual work programme contains an overview of the actions and the related budget that are implemented under direct (public procurement and grants) and indirect management (agreements with implementing organisations). For SRSP 2017 and SRSP 2018, around 98% of the evaluated budget was implemented through grants, procurement, indirect management and TAIEX 16  ( Figure 9 ).

For the projects evaluated in the mid-term evaluation, the types of providers most involved in implementing projects under SRSP 2017 and SRSP 2018 were:

·international organisations (37%)

·private companies (22%)

·government bodies (8%)

·public law bodies (7%)

·bodies governed by private law with a public service mission (4%).

These providers accounted for 78% of the evaluated budget allocated under SRSP 2017 and 2018 17 . Looking at the specific measures financed, the largest proportion of the support measures implemented concern the following sectors/actions:

SRSP 2017	SRSP 2018
·revenue administration (11%)	·tax policy (13%)
·capital markets union (9%)	·education and training (9%)
·healthcare system (9%)	·digital public administration (8%)
·energy, including climate change (6%)	·governance (8%)
·education and training (5%)	·healthcare system (7%)
·better regulation (5%) 18	·Capital Markets Union (5%) 19

Given that the evaluation took place in the middle of the implementation period of SRSP 2017 and SRSP 2018, the majority of the technical support projects were still in progress. At the time of the mid-term evaluation, 22% of the projects had been completed 20 .

4.Methodology

4.1.Fact-finding

The mid-term evaluation was carried out by an external contractor and coordinated by the Directorate-General for Structural Reform Support (DG REFORM), with the support of an inter-service steering group of other Commission departments. The contractor delivered an evaluation report, which served as a basis for this staff working document 21 .

The mid-term evaluation assessed projects funded under the SRSP’s budgetary cycles of 2017 and 2018, projects funded through the voluntary transfers of additional resources to the SRSP budget by Greece and Bulgaria in 2018, and projects funded under the Preparatory Action of 2016. Figure 10 provides an overview of the number of evaluated projects per funding source. The mid-term evaluation covered 89.4% of the allocated budget for 2016-2018, representing 303 implemented projects and 409 means of delivery 22 . The staff working document is focused primarily on the two budgetary cycles of the SRSP and refers to the Preparatory Action and the Article 11 only where relevant. The findings for evaluated projects funded under Article 11 and the Preparatory Action are similar to evaluated projects funded under SRSP 2017 and SRSP 2018.

Funding source	Number of projects evaluated	Approved financial resources (EUR)	% out of the allocated financial resources
PA (2016)	23	2 733 223	91.1%
SRSP out of which:	246	47 709 027	90.0%
·SRSP (2017)	120	19 608 927	87.2%
·SRSP (2018)	126	28 100 100	92.1%
Article 11 (2018)	34	18 830 000	87.6%
Total	303	69 272 250	89.4%

Figure 10: Approved financial resources for the evaluated projects (source: Commission’s internal IT system and targeted interviews, EY 2020)

The mid-term evaluation was divided into three main phases: (1) the inception phase, in which the structure of the evaluation was defined, based on a conceptual framework; (2) the data collection phase, in which the data was collected and structured, based on the data collection strategy defined in the first phase; and (3) the finalisation and feedback phase, in which the intermediary and final evaluation results were reported to stakeholders. The evaluation was carried out in line with the principles commonly applied for the evaluation of EU initiatives, as laid down in the Better Regulation guidelines.

The evaluation comprised different fact-finding tools and techniques, such as desk research, scoping interviews, targeted interviews, online consultation and case studies. Overall, the evaluation relied to a large extent on stakeholder consultation. The fact-finding exercises built on the evaluation framework that was designed during the inception phase of the report. The evaluation framework, including the evaluation questions and the corresponding sources of information, is provided in Annex 3 of this staff working document.

The desk research involved the review and analysis of relevant documentation. These can be grouped into six main types: (1) EU treaties and legal acts, i.e. EU legislation underpinning the SRSP, the Preparatory Action and Article 11; (2) Commission preparatory documents, i.e. staff working documents, cooperation and support plans; (3) other Commission resources, i.e. internal DG REFORM documentation; (4) online resources, including web pages of informative nature from official sources; and (5) other resources, including literature and studies.

A major source of information for the mid-term evaluation was the Commission (DG REFORM) internal IT system for monitoring technical support projects. Developed in 2018, this system provided a broad range of quantitative and qualitative information about the requests and projects under the SRSP. Data extracts from this system were provided to the contractor for the analyses.

In addition to the desk research and information from the Commission’s internal IT system, 16 semi-structured scoping interviews with DG REFORM officials were carried out (heads of unit/cluster, advisers and country coordinators). The aim of these interviews was to get a deeper understanding of the relevant issues for the different stakeholders. The intention was also to set up the detailed methodology and data collection tools. The scoping interviews were not project-specific.

As a follow-up to the scoping interviews, 31 semi-structured targeted interviews were carried out. This involved interviewing a sample of 25 DG REFORM project managers and a sample of staff members from six other Commission departments involved in SRSP assessment/implementation or in managing complementary EU funding programmes or policies. To ensure that the samples were representative, different elements were taken into consideration when selecting the interviewees, including the thematic areas of the technical support projects and their geographical coverage.

Furthermore, the evaluation included an online consultation targeting 185 stakeholders 23 . Feedback was collected from 9 coordinating authorities, 107 beneficiary authorities and 69 technical support providers 24 . The questions in the questionnaires covered all the five Better Regulation evaluation criteria. In order to optimise the results of this consultation, the questions were adapted to the types of stakeholders, according to the level and nature of engagement in the technical support funded. Both the targeted interviews and the online consultation were project-specific.

Additionally, the Commission carried out a public consultation between March and June 2019 on its official website. The goal was to collect information from the general public, businesses, trade associations and interest groups. However, given the very low number of respondents (11), the extent to which these results can be used is limited 25 . The number of respondents was low due to the fact that the SRSP specifically targets public administrations and bodies – not individuals – and that targeted interviews, targeted consultations and scoping interviews had already been carried out. Public administrations and bodies were largely covered by the external contractor’s evaluation report. Moreover, it is presumed that the programme was not widely known among the general public, given that it had only recently been established. Because the limited response to the public consultation cannot serve as solid evidence, the staff working document refers to the public consultation results only for illustrative and transparency purposes.

Lastly, the evaluation included case studies at Member State level. The case studies provided an analysis of how the SRSP projects were carried out on the ground in four countries (Croatia, Greece, Italy and Romania). Later in the process, the geographical coverage of the interviews was extended to five additional Member States (Ireland, Latvia, Malta, Portugal and Slovakia) so that the findings would be more relevant. The relevant Member States were selected on the basis of the number of approved projects, the allocated budget and the geographical coverage. The main purpose of the case studies was to assess in more detail the SRSP’s relevance, effectiveness, efficiency, coherence and EU added value in the particular Member States. The case studies included interviews with coordinating authorities (8), beneficiary authorities (22) and technical support providers (21). These interviews related to 28 evaluated projects, covering the five thematic areas of the SRSP. The case studies functioned as a method for gathering additional information in order to have more robust answers to the evaluation questions (i.e. triangulation).

4.2.Limitations and methodological issues

As required, the external evaluator contracted for this assignment carried out all tasks under the scrutiny of an inter-service steering group and the steer of DG REFORM. Overall, the quality and representativeness of the collected evidence (data, documentation, interviews, online consultation and case studies) are considered to be satisfactory. The contractor was able to collect extensive and meaningful data that allowed conclusions to be drawn. Nevertheless, there were some limitations and methodological issues. These were mitigated to the extent possible, with any limitations given due consideration.

Firstly, the most important constraint to the robustness of the findings was the fact that the SRSP had only been very recently established when the mid-term evaluation took place. The Regulation was adopted in May, and the first call was issued immediately after that. The fact that the mid-term evaluation was performed at a very early stage in the programme’s implementation limits the extent to which fixed conclusions on the SRSP’s success can be drawn. This staff working document provides first conclusions and lessons learned, which might evolve during the SRSP’s continuing implementation.

Secondly, in particular, the Commission’s (DG REFORM) new internal IT system was not in place yet for SRSP 2017 but was implemented only afterwards. The system was evolving significantly, and many questions and categories of data were added only at later stages and not necessarily retroactively completed. Therefore, information was to some extent incomplete. This shortcoming was overcome by triangulating the information based on other data sources, including the interviews, online consultation and case studies. Furthermore, the response rate to the online consultation was sufficient and ensured a balanced representation of stakeholders.

Thirdly, the breadth and diversity of the programme’s activities posed an important methodological challenge, in that it would not have been possible within the available resources and timeframe to cover all activities in the detail needed to draw robust conclusions. Moreover, much of the programme’s support plays a contributing role alongside other factors, such as the action of national administrations. This contributing role is difficult to assess without in-depth qualitative research.

Lastly, given the nature of a mid-term evaluation, many of the projects evaluated were still ongoing at the time of the exercise. Therefore, results and impact can only be measured to a limited extent. A certain share of the projects’ expected results were not yet delivered. In those cases, the likelihood of achieving the results was assessed only to the extent that information was available. The case studies confirmed that it is too early to expect considerable long-term impacts at this stage of the programme.

5.Analysis and answers to the evaluation questions

The mid-term evaluation assessed the performance of the SRSP against the following five criteria: relevance, effectiveness, efficiency, coherence and EU added value. The external study consisted of 22 specific evaluation questions, which were subsequently grouped according to their topic in order to provide a coherent framework. The overall analysis is based on evidence from both the external evaluation report and the Commission’s own sources. This chapter presents the analysis and provides the answers to the general evaluation questions.

5.1.Relevance

The overall assessment of the programme’s relevance focused on two aspects:

·the relationship between the needs of the Member States at the time of their request for technical support and the objectives and actions of the technical support provided;

·the continued suitability of the programme objectives.

5.1.1.Member State needs and the objectives of technical support

The representation of the programme’s logic of intervention ( Figure 1 ) illustrates the alignment of the SRSP design and specific objectives with the needs of the Member States at the time of their request for technical support.

The programme’s logic of intervention identifies three challenges facing Member States:

1)Limited administrative and institutional capacity;

2)Challenges in designing and implementing growth-sustaining reforms in line with the EU’s economic and social goals;

3)Limited application and implementation of EU law.

Both the beneficiary authorities and providers 26 of the technical support consulted during the mid-term evaluation confirmed these as the main challenges that Member States were facing at the time of the request for technical support. Figure 11 shows that ‘limited administrative/institutional capacity’ and ‘challenges in the design and implementation of structural reforms’ were the challenges most frequently mentioned by beneficiary authorities consulted.

Figure 11: 'Which of the following challenges was your institution facing at the moment of requesting technical support under the SRSP?' 112 beneficiary authority respondents (source: targeted online consultations, EY 2020).

With the case studies it was possible to specify these broad challenges. The case studies analysis shows that often the institutions responsible for designing and implementing structuralreforms in the relevant Member States had a limited capacity to implement reforms due to less efficient and effective human resource management. This in turn weakened the organisation, affecting also its capacity to implement enforcement measures. Moreover, these institutions face bottlenecks due to an institutional setting where competencies of sectoral and non-sectoral ministries overlap and responsibilities are unclear. Finally, they lack a systematic, analytical, and integrated approach to implementing reforms that would ensure a regular assessment of priorities, contextual conditions, expected socio-economic impacts and costs and benefits of different policy options 27 .

Analysis of the actions and activities 28 funded under the SRSP revealed that the Member States’ needs, as requested under the SRSP, have all been addressed to a large extent by the projects, and that on average, each project covered almost two of the needs mentioned above. The distribution of the projects according to the Member States’ needs is provided in  Figure 12 . Respondents to the public consultation were also positive about the extent to which the SRSP responds to the needs of Member States 29 .

Project funding source	# of projects related to strengthening administrative and institutional capacity	# of projects related to supporting the design and implementation of growth-sustaining structural reforms	# of projects related to application / implementation of EU law
SRSP 2017	82	88	37
SRSP 2018	83	102	51
Total	165	190	88
% (out of 246)	67.1%	77.2%	35.8%

Figure 12: Needs addressed by evaluated projects funded under SRSP 2017 and SRSP 2018 (source: Commission’s internal IT system and targeted interviews, EY 2020).

By pursuing its specific objectives 30 , the programme provided technical support to national authorities to develop the administrative and institutional capacity they were lacking and that is necessary for them to design, formulate, develop and implement the structural reforms they need, based on their priorities, conditions and expected socioeconomic impacts. The support also aimed at helping Member States to adopt an integrated approach to reforms, which ensures consistency across sectors, as well as coherence with EU priorities, programmes and legislation. Furthermore, the funded actions aimed at supporting national authorities to put in place new, more robust institutional processes and methodologies to enhance the efficiency and effectiveness of human-resource management, strengthening professional knowledge and skills and setting out clear responsibilities.

5.1.2.Continued suitability of the programme’s objectives

The SRSP’s objectives and actions were and continue to be relevant for Member States’ needs. The analysis showed that the SRSP is perceived as a suitable and appropriate instrument for supporting Member States in strengthening their administrative and institutional capacity. Almost all (97.3%) of the stakeholders 31  consulted agreed that the SRSP is a suitable instrument for providing technical support to design and implement growth-sustaining structural reforms ( Figure 13 ).

Figure 13: ‘The Structural Support Reform programme is a suitable instrument to provide technical support' (source: targeted online consultations, EY 2020)

Moreover, the the majority of the beneficiary authorities consulted (65%) consider that technical support is still needed 32 . The findings from the case studies 33 suggest that this is due to the long-term and broad nature of the Member States’ needs, their interconnectedness and the willingness of institutions to expand the application of the newly developed skills and capacity to implement reforms in other policy areas. The implementation of institutional, administrative and growth-sustaining reforms in the Member States is a long-term process linked to long-term needs. While the SRSP has supported the launch of this process, reforms require sustained effort over several years for economies to become more modern, flexible and competitive. From the case studies 34 it emerged that, due to the SRSP’s flexibility, the simple contracting and implementing procedures, and the absence of co-financing requirements, Member States have a tendency to consider SRSP as a relevant instrument to help design and implement national reform agendas. Moreover, the implementation of institutional, administrative and growth-sustaining reforms in the Member States is linked to wider needs. Therefore, the beneficiaries still need support for designing reforms both within the same thematic areas and in different ones. Furthermore, the incremental value of the skills and capacity developed by the beneficiary of technical support should also be considered. In other words, once the beneficiary gains experience with the instrument, they can identify other needs and potential areas of action, and thus plan to use the instrument to enlarge the scope and breadth of the reforms to implement.

In conclusion, the challenges faced by Member States as defined in the programme’s logic of intervention are aligned with the main challenges expressed by the consulted beneficiary authorities and providers. Almost all stakeholders consulted (beneficiary authorities, coordinating authorities and providers) indicated that the SRSP is a suitable instrument for providing technical support. The programme is perceived as a significant tool for improving the Member States’ capacity to define more efficient processes and methodologies and/or progress towards more effective human resource management. Since its second year of implementation, the SRSP has seen a regular increase in demand for the programme from Member States 35 . This is the first indication that the support provided by the Commission continues to be needed and is still relevant.

 

5.2.Effectiveness

For the criterion of effectiveness, the mid-term evaluation examined to what extent the objectives set were achieved and how this was linked to the EU intervention. The effectiveness of an intervention is measured in terms of both results and impacts. However, the mid-term evaluation of the SRSP was carried out while the programme’s implementation was still at a very early stage. Considering that impacts are expected only in the long-term, sometimes years after completion of the intervention, a solid assessment of the SRSP’s impacts was not possible at this stage. Therefore, the evaluation focused mainly on the SRSP’s effectiveness in terms of results 36 . 

The overall assessment of the programme’s effectiveness was composed of the following three sub-analyses:

·an examination of the extent to which the technical support projects were well-designed;

·an examination of the extent to which the planned activities and actions were feasible and expected results had been achieved;

·a preliminary exploration of the programme’s achieved impact on the ground.

In addition to these three specific areas of analysis, the evaluation also aimed to identify contextual factors that either encouraged or hindered progress towards the results and impact.

5.2.1.Design of the technical support

The evidence collected during the evaluation shows that the objectives defined at project level corresponded with the specific objectives of the SRSP. The majority of projects evaluated (around 60%) had links with more than one specific objective 37 . These outcomes demonstrate that the technical support projects are designed in such a way as to contribute to the objectives set out in the SRSP Regulation.

Furthermore, the evaluation shows that in most cases (85%), initial challenges faced by the beneficiary authorities when preparing the support requests were taken into account when the projects were designed and implemented 38 . At the same time, the findings show that when the SRSP actions were designed, the relevant socio-economic impacts were largely taken into account as well (72% of the cases evaluated) 39 . The socio-economic impacts mostly related to social inclusion, sustainable growth, competitiveness, productivity, investment, job creation and enhanced cohesion. 

In addition, the evaluation suggests that SRSP actions are often designed in a comprehensive manner, by paying attention to accompanying measures and/or the appropriate sequencing of reform measures – i.e. multiple projects that aim to address the same reform challenge, for instance from different angles or at different points in the reform design or implementation timeline. In Member States such as Cyprus, Czechia, Croatia, Greece, Romania and Slovenia, between 40% and 60% of the projects were implemented in connected reform areas, encompassing in-built synergies. This indicates that different SRSP-funded projects are linked to each other in order to achieve a comprehensive approach towards the reform challenge. Generally, in Member States where the total number of projects is higher, the degree of interconnection between projects also tends to be higher 40 . 

Lastly, the evidence collected shows that in the majority of the SRSP projects evaluated, the successful experiences and lessons learned from other EU countries are included in the design of the activities. Almost 60% of the projects evaluated included activities allowing for the exchange of best practices, such as short-term expert missions, study visits, workshops, seminars and conferences 41 . In addition to the exchange of EU Member State expertise, workshops and other training activities were also organised by experts from international organisations or other experts.

Overall, these findings suggest that the technical support projects are designed with satisfactory consideration of the specific situation of the individual beneficiary authority and the related socio-economic impacts. Technical support is generally provided in a comprehensive manner, while often incorporating in the project’s design the exchange of best practices between EU countries. All of these aspects can contribute to effective and targeted technical support activities.

5.2.2.Feasibility and achieving the expected results

To examine the feasibility of the programme’s activities, the evaluation focused on the extent to which the implementation of technical support projects is in accordance with the planned schedule. The collected evidence shows that the greatest share of the projects under SRSP 2017 and SRSP 2018 are on track (84.1%), i.e. the project’s implementation is aligned with the planned start and end date 42 . This indicates that the needs of the Member State and subsequent activities are very often translated into realistic time schedules. Some projects (around 5%) were faced with delays 43 . The targeted interviews with DG REFORM project managers suggested that these delays were often caused by political changes or changes in the beneficiary authority’s needs. Delays also occurred due to challenges in identifying the most suitable technical support provider.

The evaluation also assessed to what extent the projects have achieved their results 44 . In those cases where projects had not yet achieved their expected results, which was common given that many projects were still in progress at the time of evaluation, the likelihood of achieving the results was measured. 45 As outlined in Figure 14 , the evaluation findings show that most of the projects completed under the SRSP have achieved their expected results (70.4%). 46 The majority of the completed projects that had not yet achieved their results had a high likelihood of still doing so (63.6%) 47 . As anticipated, only a small number of ongoing projects has already achieved their results (4.2%). The evidence shows, too, that most of the ongoing projects have a high likelihood of achieving their results (86.4%) 48 . The likelihood of achieving the results is low for 13.2% of the completed and ongoing projects. 

Figure 14: Effectiveness of SRSP projects (source: Commission’s internal IT system and targeted interviews, EY 2020).

The evaluation also assessed the contextual factors influencing the delivery of project outputs. These contextual factors can either be positive or negative (see Figure 15 ).

Figure 15: Factors influencing the delivery of outputs (source: targeted online consultation, EY 2020)

The most common positively influencing factors are: the level of commitment from all stakeholders (45.6%); the provider’s level of expertise (15.2%); and the political circumstances (15.2%). The most common negatively influencing factors are: the level of collaboration among all actors involved (29.7%); resource constraints (29.7%); and political circumstances (21.7%) 49 . 

In conclusion, the evidence from the mid-term evaluation shows that the SRSP is progressing towards achieving its results, taking into account the evaluation’s limitations as many activities are still in progress. The Member States’s needs are to a good extent translated into feasible support actions under the SRSP. The majority of the projects adhere to the planned schedules, which implies that the designed activities are realistic. Meanwhile, the number of projects with delays is low. Furthermore, most of the projects are likely to achieve their results, or have already done so. A small number of completed projects did not achieve their results. A lack of collaboration among stakeholders is a challenging factor for delivering the projects’ outputs. Changes in the political structures of the beneficiary authority also pose an evident risk for the effective implementation of technical support. These and other negative factors should be avoided and mitigated to the extent possible. Optimising the exchange of good project management practices among DG REFORM project managers, for instance, through dedicated training or knowledge-sharing platforms, could help to further increase the number of projects that achieve their results. At the same time, it should be underlined that technical support for reforms is often most needed in institutions that are embedded in challenging political and governmental structures. Therefore, some degree of risk is inevitable. Reinforcing the commitment from all stakeholders in combination with effective collaboration and adequate technical expertise can increase the likelihood of achieving the projects’ results.

5.2.3.Achievement of impacts

As stated earlier, it is too early to measure the long-term impacts of the SRSP. The same goes for examining the programme’s unintended or unexpected effects. Nevertheless, the stakeholders’ perception at this early stage was used to assess the extent to which technical support projects helped the Member States to implement the relevant reforms, as planned in the cooperation and support plans. The evaluation also explored the contextual factors under which the results of SRSP actions would produce impacts on the ground.

Figure 16: The extent to which changes were observed in the beneficiary institutions between 2016 and 2018 and SRSP impacts (source: targeted online consultation, EY 2020).

The evidence collected from the targeted online consultation shows that stakeholders see signs of improvement with respect to the needs that the beneficiary authorities are facing ( Figure 16 ). The highest perception of improvement is with ‘increased administrative and institutional capacity’ (73.7%). Perceived improvements are the lowest for ‘improved application and implementation of EU law’ (42.1%). The majority of the stakeholders consider that these improvements are related to the technical support received through the programme 50 . The respondents to the public consultation were also positive in their view on the extent to which technical support has helped to achieve the SRSP’s specific objectives 51 .

The evaluation also looked closely into the specific case studies. Concrete examples of impact on the ground include:

·Support to the spending review process in Slovakia: the spending review resulted in the adoption of legislation, and Slovakia achieved EUR 83 million in savings in 2017 (0.09% of GDP);

·Education (curricular) reform in Croatia: the pilot phase of curricular reform was implemented in 74 schools in the 2018/2019 school year, and 400 tutors were trained under a follow-up project;

·A strategy for internal auditing and training internal auditors at local government level in Romania: the strategy for internal auditing was produced and adopted, and 524 city halls, prefectures and county councils were subject to an internal audit.

The evidence collected through the consultation of stakeholders shows that the most commonly mentioned factor positively influencing the achievement of impacts is the beneficiary authority’s commitment 52 . A lack of political support and stability is the most commonly mentioned negative factor for achieving the results 53 . 

In conclusion, taking into account the limitations of stakeholders’ perceptions and individual case study assessment, the achieved impacts are considered proportionate to the level of completion of the programme. However, it would be premature to draw vast conclusions on the SRSP’s impact when most of the projects are still in progress. Even achieving the project’s expected results does not guarantee that there will be an impact on the ground. Thus, a comprehensive evaluation at a later stage of the programme is essential to provide better substantiated evidence for measuring the extent to which an impact has been achieved in the long term.

5.3.Efficiency 

Efficiency considers the relationship between the resources used and the changes generated by the programme. Differences in the way the support is designed, implemented and monitored can have a significant influence on the effects. Therefore, assessing the programme’s efficiency is also related to verifying whether other choices could have maximised the use of resources for the greater benefit of the beneficiaries.

The overall assessment of the SRSP’s efficiency focused on three aspects:

·the operational efficiency of the SRSP governance, in terms of the time-effectiveness of the SRSP cycle, including its monitoring tools, as well as the quality of the requests and the cooperation with other Commission departments;

·the cost-effectiveness of the support provided, in terms of the correspondence between the results achieved and the budget allocated;

·the time-efficiency of the budget execution.

In parallel, the analysis looked at aspects of simplification, pinpointing areas where there is potential to reduce inefficiencies and simplify the intervention.

5.3.1.Operational efficiency of the SRSP governance

The assessment of the efficiency of the SRSP governance 54 focused on the time-effectiveness of the programme cycle, including the monitoring mechanisms in place, cooperation with other Commission departments and the improved quality of the requests for technical support.

Time-effectiveness was analysed by looking at the duration of the different phases of the programme and verifying whether their duration was aligned with the timing of the needs of the target group, i.e. the beneficiary authorities in the Member States. An important methodological limitation must be underlined. The analysis considered only technical support projects funded under SRSP 2018 (126 in total); due to the timing of the SRSP Regulation’s adoption, SRSP 2017 followed a considerably shorter cycle. Therefore, 2018 was the first year in which it was possible to adopt the planned timetable. Figure 17  shows the SRSP annual cycle.

Figure 17: The SRSP annual cycle 55

Based on the analysis of the Commission’s (DG REFORM) internal IT system and the results of the targeted interviews, it emerged that for the SRSP 2018 cycle, it took on average 5 months for the selection process to be completed: from the October 2017 deadline for submitting the technical support requests to the approval in March 2018 of the Financing Decision and related SRSP annual work programme containing the full list of projects selected for funding. This period comprised the phases of submission, analysis and selection of the technical support requests, as well as the adoption of the annual work programme and the Financing Decision and the conclusion of the cooperation and support plans. Once the Financing Decision was adopted, for most of the projects (81%) 56 it took at least 6 months before the planned technical support started to be delivered 57 . Figure 18 shows the distribution of evaluated projects under SRSP 2018 by starting date.

Figure 18: Distribution of projects under SRSP 2018 by starting date (source: Commission’s internal IT system and targeted interviews, EY 2020)

The average planned duration of the projects under SRSP 2018, from the procedures for concluding contracts/agreements with technical support providers to the completion of the project 58 , was 13 months. As demonstrated in Figure 19 , the planned duration of most projects (71%) 59 was between 6 and 18 months. Therefore, on average, the whole process could take 24 months, from the submission of the request by the Member States to the completion of the technical support project 60 . 

Figure 19: Planned duration of contracting and implementation phases under SRSP 2018 (source: Commission’s internal IT system and targeted interviews, EY 2020)

The targeted online consultations of both beneficiaries and providers confirmed the alignment of the SRSP timeline described above with the timing of the target group’s needs: 74.1% of the beneficiaries consulted and 66.6% of the national coordinating authorities consulted agreed entirely or to a large extent that the period between the submission of the technical support requests and the start of the delivery of the technical support was timely and efficient. Moreover, 84.6% of the beneficiaries, all coordinating authorities, and 86.2% of the providers agreed entirely or to a large extent that the implementation and delivery of the technical support was delivered on time and in accordance with the needs of the beneficiaries 61 . Respondents of the public consultation also positively assessed the programme’s efficiency 62 . Finally, 82.6% of the providers consulted agreed and strongly agreed that the duration of the the procedures for concluding contracts/agreements was aligned to the needs of the beneficiaries.

Although in general, stakeholders agreed that the current programme cycle is in line with the beneficiaries’ time needs, with the targeted interviews and case studies it was possible to identify some areas where there is room for improvement. For example, some of the providers 63 considered that the duration of the discussions on the details of the contracts/agreements was not always proportionate to the duration of the contracts/agreements. Some beneficiaries signalled that they would have wanted DG REFORM to involve them more in the process of selecting the providers, even though most beneficiaries 64  were pleased with the interaction with the provider. One in four considered that the alternatives offered were limited and did not represent the most adequate choice for their needs.

A specific area where it emerged that there could be room for improvement was the monitoring of the SRSP. Being a legal requirement, the systematic and regular review of progress and quality was continuously performed throughout the annual SRSP cycle, thus helping to ensure the programme’s efficiency. The majority of stakeholders had a largely positive perception of the procedures and tools used by the Commission to monitor the technical support’s implementation and saw them as effective. This was true for 79.8% of the beneficiaries, 66.6% of the coordinating authorities and 83% of the providers who took part in the targeted online consultation 65 . However, a more in-depth analysis of both the Commission’s practical monitoring arrangements (based mainly on the data stored in the Commission’s internal IT system) and Member State practices revealed that all the coordinating authorities consulted wanted additional support from DG REFORM in order to understand their monitoring prerogatives and to adapt their review capacity. The existence of different national approaches to monitoring activities showed the need for coordination and a procedural framework that could help coordinating authorities and beneficiaries to have a clear, strategic overview of the overall support received as well as progress and results achieved. This, in turn, would help to maximise the effects of the technical support, allowing national authorities to draw lessons and better define future needs.

It is important to underline that since the SRSP 2018 cycle, the Commission design of the monitoring mechanism has evolved significantly and continues to do so. Analysis of the information available in April 2019 on the Commission’s internal IT system revealed that improvements were needed, in particular to have more complete information on expected activities, expected results, state of play of activities and results. Hence, as a result of the internal feedback processes in DG REFORM, the Commission’s working methods have been continuously developed, reviewed and improved. Furthermore, additional tools have been used to monitor the progress of the projects. Progress reports have been used by the providers as a formal mechanism to inform the beneficiaries and DG REFORM of the progress of projects. At the same time, meetings and events, as well as calls and emails have served as informal tools to ensure that all stakeholders were updated on the progress of projects. From the targeted consultations, it also emerged that the good collaboration with and continuous support from the DG REFORM staff were strong determinants of the quality of the monitoring system.

Similarly, the stakeholders had a positive perception of the coordination between DG REFORM and the rest of the Commission. All the other Commission departments interviewed saw their cooperation with DG REFORM as efficient 66 .

The coordination mechanism was structured in such a way that potential overlaps and double financing were identified and avoided both at the technical level, through the feedback provided by the Commission departments on technical support requests, and at strategic level, through the High-Level Steering Committee meetings of representatives of different Commission departments. Dialogue between DG REFORM’s policy officers and their counterparts in the relevant Commission departments has also been strengthened. This system is a determining factor in making the SRSP action efficient and coherent with other EU programmes (for more information on the coherence of the SRSP, see Section  5.4 ). While the coordination mechanism was effective (i.e. no double funding), it was also pointed out that sharing information on the actual implementation of the projects, their outcomes and lessons learned could lead to more efficient cooperation between DG REFORM and the Commission departments. This, in turn, would result in better assessment of overlaps and complementarities and make it easier to assess progress with respect to the implementation of country-specific recommendations.

One of the most prominent factors that mitigates such considerations to a large extent is the use and continuous development of the Commission’s internal IT system, which fostered the exchange of information between DG REFORM and the other Commission departments, as pointed out by most of the Commission departments consulted (4 Directorates-General out of 6). The IT system facilitated the cooperation mechanisms because it enabled easier monitoring (i.e. a centralised approach, extraction of reports) and communication (i.e. direct contact at technical level between DG REFORM and other Commission departments).

The efficiency of the governance system, the SRSP cycle and cooperation with the other Commission departments can be reflected in the the growing number and increasingly good quality of the technical support requests and in the increasing number of Commission departments taking part in the procedures for the analysis and selection of technical support requests.  Figure 20 shows an increase in the number of requests of better quality analysed by DG REFORM under SRSP 2018 compared to SRSP 2017 and an increased number of Commission departments. The analysis showed that the quality of technical support requests 67  has considerably increased from one SRSP annual cycle to another.

	SRSP 2017		SRSP 2018
Budget	EUR 22.5 million		EUR 30.5 million
Requests received	271 (worth EUR 80 million)	444 (worth EUR 150 million )
Directorate-General that commented	15 Directorates-General		25 Directorates-General
Average score of selected requests	2.68		2.91
Average score of requests not selected	1.87		2.08
Requests selected	159		146
Member States	16 Member States		24 Member States
Average size of project	EUR 140,000		EUR 207,000

Figure 20: Comparison between SRSP 2017 and SRSP 2018 cycles

To conclude, the findings suggest that the governance of SRSP and the cooperation between DG REFORM and other Commission departments have been efficient. The duration of the design of the technical support, meaning both requests and projects, seems to have corresponded to the needs of the target group. At the same time, the quality of the technical support requests increased from one SRSP annual cycle to another. Furthermore, the programme’s implementation, including its monitoring, was aligned with the needs of the target groups. The analysis and the stakeholder feedback highlighted that the Commission’s current internal IT system is largely adequate for monitoring. However, there is potential for improvement to make the system more complete. Common standards for national monitoring practices could, for example, be considered in the future. Furthermore, that the governance system and cooperation between DG REFORM and the other Commission departments were efficient was reflected in the increased quality of the technical support requests and in the positive perception of the Commission departments interviewed. Overall, the governance of the SRSP has been continuously developing, duly reflecting the changing needs of the stakeholders involved and the learning curve that goes with the creation and management of a new programme.

5.3.2.Cost-effectiveness of the support 

The cost-effectiveness of the SRSP projects was assessed by verifying whether the level of achievement of the expected results was commensurate with the budget already allocated. Cost-effectiveness analyses usually look at the relationship between the costs of an action and the benefits obtained. While costs are in general more easily measurable in monetary terms, the benefits tend to emerge over a longer timeframe and are generally harder to measure objectively. Considering that the object of the mid-term evaluation was the implementation of a newly developed programme for the first 2 years, and given that only the results and not the benefits could be monitored at this point in time, the analysis considered project results as a proxy to benefits.

The mid-term evaluation assessed the correlation between financial resources allocated and the project status. From the data available in the Commission’s internal IT system, it emerged that on average, all projects were on track, as several steps had already been taken towards achieving the expected results. Figure 21 shows that the average score for the degree of achievement of the expected results for the completed projects was 3.8 out of 4 68 . The projects that have not yet achieved their expected results were mainly ongoing projects (110 out of 121). These ongoing projects had a high likelihood that they would still achieve their expected results by the end of their expected implementation period (score 2.8 out of 3) 69 . 

Funding source / Project status	Degree of achievement of the expected results (score 1 to 4)	Number of projects that have already achieved the expected results ( score = 4)	Number of projects that have not yet achieved the expected results (score =1, 2 or 3)	Total number of project subject to this analysis
Completed	3.8	38	11	49
Ongoing	2.7	8	110	118

Figure 21: Degree of achievement of the expected results of evaluated projects under SRSP 2017 and SRSP 2018 (source: Commission’s internal IT system and targeted interviews, EY 2020)

When comparing the status of implementation and achievement of results to the cost associated to the projects, it emerged that the financial resources of those projects having already achieved their results (30%) amounted to EUR 7.4 million, i.e. around 16% of the total budget allocated to the entire sample of projects evaluated. The cost associated to the projects that had not yet achieved the expected results was EUR 39.5 million, representing around 84% of the budget allocated to the entire sample of projects evaluated.

All the results have been achieved for almost 30% of the evaluated projects that correspond to 16% of the approved financial resources, most of which are linked to completed projects. At the same time, for the remaining 70% of the projects, there is a high likelihood that the results will be achieved, most of which are linked to ongoing projects. Therefore, the achieved results are considered commensurate to the resources allocated.

In conclusion, the SRSP actions can be considered cost-effective so far, while acknowledging the limited scope of the analysis. The achieved results are commensurate with the budget used. All the results have been achieved for almost 30% of the evaluated projects, which correspond to 16% of the approved financial resources, most of which are linked to completed projects. At the same time, for the remaining 70% of the projects (i.e. 84% of the amounts), there is a high likelihood that the results will be achieved, most of which are linked to ongoing projects.

5.3.3.Time-efficiency of the budget execution

The time efficiency of the budget execution refers to the capacity to contract the financial resource allocated (i.e. absorption capacity 70 ) and the capacity to make payments on time for the contracts/agreements that are signed. Given that information on payments made under the SRSP was not available at the time of the evaluation, the analysis focused on the alignment of the absorption capacity to the status of the projects, as well as on the status of the means of delivery and the existence of possible delays.

The analysis of the absorption capacity determined what percentage of the approved financial resources for 267 projects under SRSP 2017 and SRSP 2018 had been contracted. It emerged that 71% of the overall approved budget for the evaluated projects had already been contracted, i.e. EUR 37 million out of EUR 47 million. The findings suggest that the absorption rate is aligned with the two-year duration of the SRSP project cycle, as well as with the status of the contracting procedure under each funding source. Procedures under SRSP 2017 are at a more advanced stage of implementation (almost 34% have been completed), hence the higher absorption rate (95.8%). At the same time, most of the contracting procedures under SRSP 2018 are ongoing (more than 90%), which is correlated with a lower contracting rate, i.e. 67% ( Figure 22 ).

Funding source / Project status	# of contracting procedures evaluated	Approved amount EUR (in million)	Contracted amount EUR ((in million)	% of approved amount already contracted amount
SRSP 2017	146 (100%)	20	19.2	95.8%
Completed	49 (33.6%)	5.3	5.1	96.5%
Ongoing	97 (66.4%)	14.8	14.1	95.6%
SRSP 2018	158	27	17.9	66.5%
Completed	15 (9.5%)	1.2	1	84.1%
Ongoing	143 (80.5%)	25.9	17	65.7%
Total	303	47	37.1	81.1%

Figure 22: Financial resources approved and contracted by funding source (Commission’s internal IT system and targeted interviews, EY 2020)

Although the absorption rate is aligned with the SRSP cycle, there are signs that some delays might occur 71 , particularly as the share of means of delivery with risk of delays increased from 9% under SRSP 2017 to 17% under SRSP 2018. The distribution of the number of means of delivery with risks of delays is presented in  Figure 23 . Of these, indirect management, specific contracts under framework contracts and grant direct awards are the means of delivery with the highest shares of risks of delays. The rise in the share of procedures with risks of delays could be a symptom for the lower contracting rate registered under SRSP 2018 compared to SRSP 2017.

Figure 23: Means of delivery with the highest shares of risks of delays (source: Commission’s internal IT system and targeted interviews, EY 2020).

To conclude, although some minor risks of delays were noted, the financial resources were contracted in a time-efficient manner, as shown by the alignment of the absorption rate with the status of the projects.

5.4.Coherence

In line with the Better Regulation guidelines, the assessment of the programme’s coherence is divided into two parts: internal and external coherence. Internal coherence covers the design of an EU intervention. It looks at how the different components of the intervention interact, and whether they are logically linked to achieve the objectives set. The external coherence is part of an evaluation that examines how an EU intervention interacts (contradicts, overlaps, complements) with other programmes, instruments, policies, and actions.

Specifically, the SRSP evaluation comprised the following sub-analyses:

·an explanation of the rationale of the SRSP’s logic of intervention (internal coherence);

·an analysis of how the SRSP is integrated in the EU’s economic governance process (external coherence);

·an analysis of the extent to which the SRSP interacts efficiently with other EU interventions and Commission Directorates-General (DGs), particularly in preventing double funding (external coherence);

·an analysis of the extent to which beneficiary authorities are aware of other programmes and policies with similar objectives as the SRSP (external coherence).

5.4.1.Rationale of the logic of intervention

The rationale of the SRSP intervention is visualised in the programme’s logic of intervention ( Figure 1 ). The causal relations between the different components of the programme can be clearly identified. All the projects under the evaluation cover all types of eligible actions. These eligible actions delivered the expected outputs or are expected to still do so. Subsequently, the outputs are expected to deliver the results, which, in turn, are expected to trigger the defined impacts. For example, the experts in a specific policy area (eligible action) are expected to deliver a reform strategy (expected output). The strategy is then expected to be adopted by the relevant Member State (expected result) 72 . Subject to contextual factors, the correct adoption and implementation of the strategy, in turn, is expected to trigger enhanced sustainable growth, job creation and social inclusion, or greater cohesion, competitiveness, productivity and investment (expected impact).

At the level of individual projects, the expected activities, outputs and results/outcomes are defined in the description of the action or the terms of reference, depending on the means of delivery. These are drafted following extensive consultation with the beneficiary authority about the needs and expected results.

5.4.2.EU economic governance process

Naturally, the SRSP exists in the wider structures of EU policies and interventions. DG REFORM is also closely involved in the European Semester process, where it interacts with other Commission departments. The European Semester is a framework for the coordination of economic policies across the EU. Each year, the Commission carries out a detailed analysis of each Member State’s plans for budget, macroeconomic and structural reforms. It also provides EU governments with country-specific recommendations for the following 12-18 months.

Figure 24: Interaction between the annual cycles of the SRSP and the European Semester 73

The projects under the SRSP are supporting the European Semester. Both the SRSP and the European Semester are organised in annual cycles, which feed into one another ( Figure 24 ). As visualised in the figure above, European Semester country reports (including the CSRs) and the national reform programmes in a given year can serve as a guiding document for defining Member States’ requests for technical support under the SRSP in the following year.

The DG REFORM country coordinators participate in the European Semester country team meetings to ensure that the knowledge and experiences gained from the SRSP feed into the European Semester. These meetings have replaced the part of the coordination mechanism involving the technical support country teams 74 . The DG REFORM country coordinators inform the European Semester country teams about the requests submitted and about the state of implementation of the projects linked to the European Semester. In addition, SRSP projects can provide the European Semester country teams with information from the ground on the state of play of certain reforms in Member States and reasons behind obstacles in specific policy areas. This input optimises the quality of the country analysis. Moreover, the progress on the implementation of SRSP projects is included in the country report of each Member State.

The majority of the projects selected under SRSP 2017 (51%) and SRSP 2018 (55%) were related to the implementation of reforms in the context of the economic governance process, in particular to CSRs 75 . Of the projects evaluated that were linked to European Semester recommendations, 21.6% have already achieved their expected result 76 . Of the remaining projects that did not achieve their expected results, 82.8% have a high likelihood of doing so 77 . The fact that a significant share of the SRSP projects are related to the implementation of CSRs, as do the aligned governance structures, demonstrates that the SRSP contributes to the economic governance process. A potential for further improvement and even stronger synergies to optimise the integration in the economic governance process can be explored.

5.4.3.Interaction between DG REFORM and other Commission departments and instruments

As stated above, the European Semester is a process that allows for a comprehensive approach towards structural reforms and the coordination of different actions and policies, in which DG REFORM plays an integral role. DG REFORM has put in place a coordination mechanism in order to ensure that the technical support provided is aligned with the activities and policies of other Commission departments and to prevent any financing overlap.

DG REFORM has the mandate to steer and coordinate the technical support provided by the Commission to the Member States. The coordination mechanism has the following horizontal structure:

·A ‘High-Level Steering Committee’, which meets at least once a year (or per SRSP funding round) at senior management level, and makes recommendations and provides guidance on all aspects of the technical support provided to Member States, including funding priorities. The High-Level Steering Committee also endorses the list of requests proposed for funding under a funding round;

·A ‘Technical Support Working Group’, which meets quarterly to exchange information on the provision of technical support, optimise cooperation between Commission departments, and prepare the process of consultation on requests for support received under the SRSP.

As stated above, the Commission’s policy departments are consulted for the assessment of the technical support requests. Given that DG REFORM is itself not a policy department, the appropriate design and implementation of projects requires interaction with departments that are responsible for defining the Commission’s policies and that potentially fund actions with similar objectives. The consultation with policy departments allows these departments to inform DG REFORM of links between the request and their policy priorities. Moreover, this gives these departments the opportunity to signal if a request overlaps with projects funded under their programmes or instruments, potentially leading to double funding. Lastly, departments can identify synergies between the requests for technical support and the actions or projects that they themselves fund 78 . Hence, the consultation between departments allows the Commission as an organisation to timely identify synergies and overlaps.

The findings from scoping interviews and targeted interviews with departments confirm that the complementarities between the SRSP and other programmes are ensured to a large extent. The interviews point out that this is mainly due to DG REFORM’s efforts and to the high level of collaboration and communication between the Commission’s different departments. These findings are in line with the outcomes of the 2018 Annual Activity Report, which show that all Commission departments are satisfied with the efficiency of the coordination mechanism 79 . The interviews with Commission departments also confirm DG REFORM’s efforts to avoid funding overlaps through the checking system mentioned above. However, respondents also indicated that approving the SRSP in the middle of the programming period creates challenges for the other EU programmes with similar objectives.

SRSP support is directly complementary to many other Commission support programmes, most notably the European Structural and Investment Funds (ESIF) and Fiscalis 2020. To ensure close coherence and complementarity between the technical support and assistance provided under the ESIF funds and that provided under the SRSP, different terminology is used (See box below).

At the time of writing this staff working document, there were a number of checks in place to ensure that synergies and funding overlaps were being timely identified at Member State level. By submitting their technical support requests under the SRSP, Member States are obliged to inform the Commission immediately of other related ongoing actions financed out of the EU budget 80 . Subsequently, if a request is selected for funding, the relevant Member State is required to confirm to the Commission that there is no overlap between the selected request under the SRSP and concrete actions funded under other EU instruments, and that there is no double funding. The coordinating authority of a Member State is also obliged to send a signed letter to the Commission, confirming the absence of double funding for all approved technical support requests under the SRSP of that specific annual cycle, after having consulted the relevant national managing authorities of ESIF funds. The optimisation of synergies with other actions and projects and the declaration that there is no double funding and no duplication of efforts are integral parts of the guiding principles of the cooperation and support plans. By signing the plan, the Member State commits to comply with these principles.

At the time of the mid-term evaluation, the verification systems mentioned above were gradually being put in place. The case studies indicate that at the time of evaluation, the verification systems for double funding were relatively informal. Given this context, the general perception of the stakeholders consulted is that the process could be improved by further formalising the procedures for checking for double funding before the submission of requests and after their selection. The evaluation also suggests that Member States would need to increase their capacity in order to perform more in-depth checks. As described above, many improvements have already been made since then.

5.4.4.Awareness of other policies and programmes

Lastly, the evaluation assessed the extent to which the beneficiary authorities were aware of other policies and programmes whose objectives were similar to the SRSP’s. As indicated in  Figure 25 , 25.2% of the responding beneficiary authorities stated that they were indeed aware of these programmes and policies 81 . The EU programme similar to the SRSP that is most commonly known among beneficiary authorities is, according to the evidence from the targeted online consultation, the European Social Fund 82 . Two other EU programmes identified in the public consultation were the European Regional Development Fund and the Cohesion fund.

One of the reasons for requesting support under the SRSP could be the relatively low administrative burden, as DG REFORM carries out the contractual procedures 83 . The findings suggest, however, that beneficiary authorities sometimes lack an overview of all funding alternatives. From the targeted online consultation and case studies it also emerged that for some respondents the SRSP was appreciated because the entirety of the support is covered by the SRSP and that there is no need for co-financing at national level 84 .

Figure 25: 'Are you aware of other policies or programmes implemented with European/national funds that have similar objectives as the SRSP? (source: targeted online consultation of beneficiary authorities, EY 2020).

In conclusion, the SRSP can so far be considered internally and externally coherent to a high degree, taking into account the limited scope of the mid-term evaluation. Firstly, the different components of the logic of intervention have clear causal links. Furthermore, the SRSP is considered to be supportive of the EU economic governance process. The SRSP and the European Semester are adequately feeding into each other, providing in both directions input to enhance the coordinated approach towards achieving the objectives. The fact that a significant share of the SRSP projects are related to the implementation of CSRs demonstrates that the SRSP contributes to the economic governance process. In addition, the evaluation evidence shows that complementarities between the SRSP and other programmes are ensured to a large extent. DG REFORM has continued to optimise its coordination mechanism, which allows for the coordination of technical support with other Commission departments. Synergies and the absence of double funding are best ensured through consultation with other Commission departments during the assessment of the requests submitted under the SRSP, and through the declaration of absence of double funding for the selected request for support. At the level of the Member State, double funding checks are carried out in a relatively informal manner. These procedures have been further developed and formalised in subsequent years, in line with the recommendations of the SRSP mid-term evaluation. Lastly, the findings from the targeted online consultation suggest that beneficiary authorities are sometimes aware of other programmes and policies with objectives similar to the SRSP’s but lack an all-encompassing overview of these funding alternatives.

5.5.EU added value

In accordance with the principle of subsidiarity in areas of non-exclusive competence, the EU should only intervene when the objectives can be better achieved at EU level than at the level of the Member State. In this light, the evaluation assessed what resulted from the SRSP-funded activities in value that was above what could reasonably have been expected from Member States acting at national or regional levels.

Given that the programme was only set up recently, there are limitations as to how much EU added value can be measured in terms of impact. Taking this into consideration, the overall assessment of the programme’s EU added value was composed of the following two sub-analyses:

·an explanatory assessment of why SRSP support is managed at EU level and to what extent this support is requested by Member States;

·an examination of good practices exchange, cooperation among institutions from different Member States and cross-border technical support facilitated by the programme.

5.5.1.Rationale of the SRSP and Member States’ demand

All Member States should benefit from a resilient and flexible European economy. A strong European economy relies on Member State economies that function well and are adaptable. However, Member States sometimes lack the required expertise or administrative and institutional capacity to carry out reforms that the European Union recommends to them 85 . The same can be observed for the application of EU law. This inadequacy to reform could jeopardise sustainable growth across the EU. By providing EU-level technical support for reforms, it is possible to tackle knowledge and capacity constraints in Member States. Highly needed reforms could otherwise remain unaddressed at national level. In other words, in order to achieve the objective of economic growth shared by all Member States, technical support provided at Union level is needed to help those Member States and public institutions that cannot address the reform challenges with their own national means.

In addition, the relatively low bureaucratic and administrative burden of the SRSP for beneficiary authorities, together with the absence of co-financing, contributes to Member States making use of the available support. The evidence from the evaluation shows that this is particularly relevant for beneficiary authorities coping with limited financial resources or administrative capacity 86 . Often, these institutions are the ones most in need of reforms. 

The evidence on the submitted requests suggests a high demand for technical support managed at EU level. As for the period evaluated, every year, an increasing number of Member States request technical support under the SRSP 87 . Moreover, 80% of the Member States benefiting from support under SRSP 2017 and SRSP 2018 had increased their number of submitted requests from one year to the next 88 . In total, 25 Member States received technical support under the Preparatory Action (2016), SRSP 2017 and SRSP 2018.

5.5.2.Exchange of good practices and cross-border cooperation

The SRSP’s EU added value also lies in its ability to facilitate the exchange of good practices among different Member States. The evidence collected shows that 52.5% of the projects evaluated made use of good practices and lessons learned 89 . As outlined in  Figure 26 , the main types of outputs that include good practice sharing are: (1) recommendations, (2) analyses and reports and (3) workshops, training and training materials 90 . 

Figure 26: Main types of output developed based on activities that include sharing good practices (source: Commission’s internal IT system and targeted interviews, EY 2020).

Furthermore, from the evaluated means of delivery that used good practices and lessons learned, 28.2% was provided by a public sector entity, such as government bodies or public servant experts 91 . Naturally, support provided from one public entity to another enables the sharing of good practices among EU countries. Evidence from targeted online consultation shows that most beneficiary authorities (89%) perceive an increase of knowledge sharing between their institution and institutions from other Member States ( Figure 27 ). Likewise, 91% of the technical support providers perceived an increase in knowledge exchange 92 . The respondents of the public consultation were positive, too, about the programme’s EU added value in terms of sharing good practices and promoting cooperation. 93  

Figure 27: Beneficiary authorities’ degree of (dis)agreement with the statement: 'The technical support increased the sharing of knowledge and good practices between my institution and institutions from other countries' (source: targeted online consultation, EY 2020).

Technical support under the SRSP can also address reform challenges, which are common to several Member States. This can take two forms. In the first type, several Member States may make similar requests for support to address a common reform challenge, if they realise that their reform should be best tackled across national borders. All the interested Member States still need to submit their SRSP requests individually; one Member State cannot apply for support on behalf of another Member State. Second, if different Member States submit requests to tackle a similar issue, even though the relevant countries might be geographically far apart, the Commission can identify synergies between these individual support requests. In cases deemed feasible, the Commission can decide to implement requests from different Member States which are related to one another, through one project, with a view to achieve efficiency gains in terms of contracting. Instead of an individual technical support project for each request, one project would cover the requests of several Member States. Each Member State would in any case still receive individualised support, applicable to their local situation.

The findings from the evaluation demonstrate that the extent to which individual SRSP projects tackle cross-border challenges in multiple countries is low 94 . Only one out of all the projects evaluated (0.3%) combine independently requests submitted from different Member States into one multi-country project 95 . The reason for the limited joint technical support actions in different Member States is that the requests for support are country-specific, related to national priorities in line with EU goals. The needs of Member States often have their own unique characteristics, as challenges are embedded in the specific national context. The exchange of information among beneficiary authorities from different Member States during the design of the requests is also not advanced. This could be further enhanced in those cases where it is deemed feasible and of added value. The findings suggests that the Commission needs to optimise further the identification of synergies between individual requests from different Member States, to ensure that technical support efforts are complementary and closely related where relevant and practically achievable. Lastly, the sharing of solutions and knowledge after the project’s implementation among Member States seems to be limited. Although there are individual cases in which knowledge sharing took place after the closure of the project, this is not yet a common practice 96 . Nevertheless, these individual cases suggest that the programme can provide technical support for cross-border cooperation. A large share of the respondents of the public consultation positively assessed the programme’s added value in terms of addressing cross-border or Union-wide challenges 97 . 

In conclusion, the technical support provided under the SRSP is intended to target precisely those reform challenges that Member States cannot address with their own means and for which they seek expertise beyond the national level. The increasing number of Member States that requested support under the programme, as well as the increase in requests submitted, implies a high demand for technical support at EU level. Furthermore, many projects include activities that allow for the exchange of knowledge and good practices among Member States.The evidence suggests that the Commission is not yet sufficiently facilitating information exchange between beneficiary authorities in different Member States during the design phase of the project. Considering that the programme’s internal processes and procedures have by now taken their final shape, the exchange of information among beneficiary authorities should be reinforced. The Commission could also build on experiences from successful cases of cross-border cooperation, including by optimising the identification of synergies between individual SRSP requests. At the same time, Member States could seek cooperation for requests under SRSP that cope with common challenges.

6. Conclusions

The Structural Reform Support programme was adopted in 2017 to help Member States in the process of designing and implementing institutional, administrative and growth-sustaining reforms, by providing technical support to national authorities. This mid-term evaluation assessed the SRSP 2017 and 2018 budgetary cycles. The overall goal of the evaluation was to examine how the SRSP is performing against the evaluation criteria (relevance, effectiveness, efficiency, coherence and EU added value) and to draw lessons that can serve possible future improvements.

Overall, the quality and representativeness of the collected evidence is considered satisfactory. The evaluation covered over 300 technical support projects and included different evidence collecting techniques. The most considerable limitation to the robustness of the findings was the fact that the programme had only very recently been adopted at the time of the evaluation. Many projects were still in their initial phase, posing constraints on measuring the SRSP’s results, as well as impacts. At the same time, some internal procedures and processes, and the information system for monitoring projects, were only gradually being implemented. This limits the extent to which fixed conclusions on the SRSP’s success can be drawn in this mid-term evaluation. Nevertheless, the Commission’ staff considers that the findings have an adequate level of robustness to provide preliminary conclusions and lessons learned on the SRSP.

Regarding the relevance of the programme, the findings of the evaluation show that the design of the SRSP is aligned with Member States’s actual needs. The evidence demonstrates that the Member States’ needs, as identified in their request for support, have all been addressed to a large extent by the programme. Stakeholders indicate that the SRSP is a suitable instrument for providing technical support to Member States to design and implement growth enhancing reforms. Additionally, Member States especially appreciated the high degree of flexibility, the simplicity of the contracting and implementing procedures managed by the Commission and the absence of co-financing requirements. The findings suggest that the SRSP remains a relevant instrument for Member States. Technical support continues to be needed, given that reforms are linked to long-term and wide needs. The continuous need for support is also connected to the experience and capacity that the beneficiary authorities gain with the instrument, as the beneficiary authority might identify other needs and potential areas for support under the programme, after finishing one project. These outcomes correspond with the Commission’s expectations.

The analysis of the programme’s effectiveness shows that the projects funded under the SRSP are generally well designed. The way in which the Commission takes into account the specific needs of Member States, as well as the relevant socio-economic impacts, is considered a good and common practice according to the stakeholders. This makes it also possible to translate the needs into feasible actions with realistic timelines; only a few projects were delayed. The fact that technical support is generally provided in a comprehensive manner, with satisfactory consideration of the situation of the individual beneficiary authority, is a good practice and meets the expectations of delivering tailor-made technical support. Furthermore, the findings demonstrate that the SRSP is progressing towards achieving its objectives, taking into account that many projects were still ongoing at the time of the evaluation. Most of the technical support projects are likely to achieve their results, or have already done so. However, the exchange of good practices in terms of project management among Commission project managers could be reinforced in order to increase further the number of successful projects. Evidence also shows that changes in the political structure of the beneficiary authority, as well as a lack of collaboration among stakeholders, are important obstacles to the delivery of the projects’ outputs. These and other negative factors should be avoided and mitigated to the extent possible. As for the SRSP’s impact, it is premature to draw solid conclusions. Although the stakeholders’ perception of the impact is considered proportionate to the programme’s level of completion, a comprehensive evaluation of the long-term impacts of the supported interventions should be carried out at a later stage, as part of the ex-post evaluation.

As regards the programme’s efficiency, the evaluation findings indicate that governance of the SRSP is efficient. Overall, the programme’s cycles, including the design and implementation of the technical support, are aligned with the beneficiary authorities’ needs. The analysis also showed that the current Commission’s internal IT system is largely adequate as a monitoring arrangement, although it still needed further improvements at the time of evaluation. Additionally, all departments indicated that their cooperation with DG REFORM was efficient. Strong points highlighted by the stakeholders interviewed were: the efficiency of the governance system, the SRSP cycle and cooperation with the other Commission departments, the good quality of the technical support requests, as well as the increasing number of Commission departments taking part in the procedures for the analysis and selection of the technical support requests. In terms of cost-effectiveness, the analysis indicates that the achieved results are commensurate with the budget allocated. Given that the SRSP was established only very recently and therefore only results and not benefits could be monitored at this point, the SRSP-funded actions can be considered, so far, cost-effective. Finally, the evidence demonstrates that the programme’s absorption rate is aligned with the two-year duration of the SRSP project cycle. Therefore, it can be concluded that the SRSP is time-efficient in terms of its budget execution. When it comes to potential improvements, the evidence suggest that common standards as regards the monitoring practices at the level of the Member States could be considered in the future. Further attention could be paid to some minor risks of delays in the contracting procedure that were noted in the evaluated period.

The analysis of the programme’s coherence focused on both internal and external coherence. The findings demonstrate that the different components of the intervention logic of the SRSP are logically linked to one another. Furthermore, the SRSP is well integrated in the EU economic governance process. The annual cycles of the SRSP and the European Semester are adequately feeding into each other, which enables a comprehensive and coordinated approach towards structural reforms in Member States. The evidence also shows that DG REFORM has put in place an adequate coordination mechanism that allows for efficient coordination of technical support. The confirmation of the absence of double funding and the identification of potential synergies are carried out both by the Commission, through the consultation between DG REFORM and other departments, and by the Member States, through the support request template and the cooperation and support plans, and the regulatory provision of no double funding. At the time of the evaluation, the verification of double funding by the Member State was carried out in a relatively informal manner. Subsequently, these procedures have been further developed and formalised. The findings from scoping interviews with departments confirm that the complementarities between the SRSP and other programmes are ensured to a large extent. At the same time, the Commission departments stated that the fact that the SRSP support is not aligned to their programming period imposed challenges on other EU programmes with similar objectives. Lastly, the findings suggest that Member States would need to increase their capacity to perform more in-depth checks for double funding. It could be considered to look into this, in order to further optimise the safeguarding of synergies and the absence of funding overlaps.

Lastly, the evaluation assessed the programme’s EU added value. The technical support provided under the SRSP is intended to target those reform challenges that Member States cannot address at national level and for which authorities seek support at EU level due to capacity or knowledge constraints. Moreover, the individual projects very often provide a platform for exchanging good practices and lessons learned among Member States. These are very essential features that the SRSP was expected to deliver in terms of EU added value and that have materialised. At the same time, the findings demonstrate that the extent to which SRSP actions address cross-border challenges in multiple countries remains low. The exchange of information between beneficiary authorities in different Member States could also be further facilitated, where relevant. In addition, Member States could be invited to cooperate when they formulate requests for support that are related to shared challenges.

Lessons learned

Following the mid-term evaluation, the main lessons learned include:

·The high degree of flexibility, the simple contracting and implementing procedures, and the absence of co-financing requirements, make the SRSP an instrument relevant to Member States’ reform needs.

·A high degree of commitment from all stakeholders, a high level of expertise of the provider and favourable political circumstances are considered factors that positively influence the delivery of the projects’ outputs. A lack of collaboration among stakeholders and changing or uncertain political circumstances are considered factors that negatively influence the successful implementation of technical support projects.

·DG REFORM could provide further support to the coordinating authorities in order to understand their monitoring prerogatives and help them adapt their coordination and monitoring capacity.

·Improvements of the Commission internal IT system are needed, in particular on the level of completeness of information related to expected activities, expected results, state of play of activities and results.

·The good collaboration and the continuous support of the DG REFORM staff with beneficiary authorities, coordinating authorities, providers and other Commission departments were strong determinants of the quality of the monitoring system and the efficiency and coherence of the programme.

·The alignment of the annual cycles of the SRSP and the European Semester contributed to a coordinated approach towards support for the implementation of structural reforms. The high level of collaboration and communication between DG REFORM and other Commission departments were essential in ensuring complementarities between the SRSP and other programmes.

·Sharing good practices among Member States is of significant added value, which comes with managing technical support at EU level. DG REFORM could further facilitate the exchange of information between beneficiary authorities of different countries, where relevant, in order to optimise synergies and solution sharing. 

Annex 1: Procedural information

1.Lead DG, Decide Planning/CWP references

Directorate-General for Structural Reform Support.

Decide planning reference: PLAN/2018/3005.

2.Organisation and timing

The initiation of the mid-term evaluation of the SRSP was approved by the SRSS senior management 98 and the Cabinet in May 2018. This was followed by the appointment of  the members of the Inter-service Steering Group (ISSG). The evaluation roadmap was published on 20 June 2018, marking the evaluation’s official starting date. The procurement procedure for the external evaluation was initiated on 9 July 2018, and the contract with the external contractor was subsequently signed on 21 November 2018, with a kick-off meeting on 6 December 2018. The external evaluation lasted 12 months. The public consultation ran on the Commission’s website from March until June 2019. After the contractor submitted the final evaluation report, the Commission started drafting the staff working document, based on the external evaluation and the analysis of its own reporting and procedural documents.

DG REFORM chaired the ISSG, which comprised representatives of the Secretary General, REGIO, DIGIT, CNECT, EAC, HOME, ENV, GROW, SANTE, MOVE, AGRI, EMPL, TAXUD, JUST, FISMA, NEAR, ECFIN, ENER, COMP, ESTAT, RTD, HR, JRC and OLAF. The ISSG held three meetings.

3.Exceptions to the better regulation guidelines

N/A

4.Consultation of the RSB (if applicable)

N/A

5.Evidence, sources and quality

An external evaluation report served as a basis for the mid-term evaluation of the SRSP.

The contractor used a variety of research methods and sources, including: desk research, data extracts from the Commission’s internal IT system, scoping interviews, targeted interviews, targeted online consultation and case studies. In line with the Better Regulation guidelines, the evaluation also included a public consultation exercise.

The most important constraint to the robustness of the findings was the fact that the SRSP had only very recently been established at the time of the mid-term evaluation. The fact that the mid-term evaluation took place at a very early stage in the programme’s implementation limited the extent to which fixed conclusions on the SRSP’s success could be drawn.

Furthermore, at the time, the Commission’s new internal IT system was not in place for SRSP 2017, being implemented only afterwards. The system was evolving significantly and many questions / categories of data were added only at later stages and not necessarily completed retroactively. Therefore, information was to some extent incomplete. This shortcoming was overcome by triangulating the information based on other data sources, including the interviews, online consultation and case studies. Furthermore, the response rate to the targeted online consultation was sufficient and ensured a balanced representation of stakeholders.

Additionally, the breadth and diversity of the programme’s activities posed an important methodological challenge, in that it would not have been possible within the available resources and timeframe to cover all activities in the detail needed to draw robust conclusions. Moreover, much of the programme’s support plays a contributing role alongside other factors, such as the action of national administrations. This contributing role is difficult to assess without in-depth qualitative research.

Lastly, given the nature of a mid-term evaluation, many of the projects evaluated were still ongoing at the time of the exercise. Therefore, results and impact can only be measured to a limited extent. A certain share of the projects’ expected results were not yet delivered. In those cases, the likelihood of achieving the results was assessed only to the extent that information was available. The case studies confirmed that it is too early to expect considerable long-term impacts at this stage of the programme.

Despite the limitations mentioned above, the quality and representativeness of the evidence collected is considered satisfactory. The contractor was able to collect extensive and meaningful data that allowed conclusions to be drawn.

Annex 2: Stakeholder consultation

As part of the evaluation performed by the external contractor 99 , four types of stakeholders were consulted through three data collection methods:

·DG REFORM staff through scoping interviews and the targeted interviews, specifically:

oHeads of unit (HoU) / heads of cluster and deputy heads of unit

oDG REFORM advisers

oCountry coordinators

oProject managers

·Coordinating authorities through the targeted online consultation and the interviews for the case studies;

·Beneficiary authorities through the targeted online consultation and the interviews for the case studies;

·Technical support providers through the targeted online consultation and the interviews for the case studies.

The consultation activities were conducted between January and December 2019. The consultation activities are elaborated in the main text.

Public consultation

In addition to consulting stakeholders as part of the external evaluation, the Commission carried out a public consultation on the Commission’s official website between March and June 2019 . The goal of the public consultation was to collect information from the general public, businesses, trade associations and interest groups. The outcomes of the public consultation are outlined below.

Total respondents: 11 (5 public authorities, 4 members of the general public, 1 trade union and 1 ‘other’).

Geographical distribution: 5 from Latvia; 2 from Belgium; 2 from Spain; 1 from France; 1 from Italy.

(1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; and (5) to the fullest extent.

Scale : (1) not at all; (2) to a limited extent; (3) to some extent;  (4) to a high extent and (5) to the fullest extent

Scale : (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent and (5) to the fullest extent

Scale : (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent and (5) to the fullest extent

Scale : (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent and (5) to the fullest extent

Annex 3: Evaluation framework 100

Evaluation criteria	Evaluation question (EQ)	Details	Number of evaluated projects in the sample	Indicators	Source of information
Relevance	EQ.1.To what extent are the SRSP actions appropriate for supporting Member States in strengthening their administrative and institutional capacity, supporting the design of growth-sustaining structural reforms and the application / implementation of EU law?	Extent to which the SRSP projects, as they are designed, are expected to meet the identified Member State’s needs	303	·Percentage of projects whose expected results are related to: ·strengthening the administrative capacity ·strengthening the design of growth-sustaining structural reforms ·strengthening the application / implementation of EU law ·Perceived degree of support still needed	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers, the results of the targeted online consultation ·The results of the targeted online consultation
		Extent to which Member States still need technical support to perform structural reforms and the extent to which SRSP support remains relevant
Effectiveness - results	EQ.2.How effectively have SRSP actions and activities translated the Member States priorities / needs (conveyed in their requests for support) into feasible actions / activities?	Extent to which projects have been implemented in accordance with the planned schedule	303	·Percentage of projects whose implemented activities are aligned with the estimated start / end dates ·Percentage of projects whose expected results have been produced ·Percentage of projects whose expected results have a high likelihood of being achieved been achieved ·Perceived impacts of contextual factors on the projects’ implementation	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers ·The results of the targeted online consultation were used to confirm the results of the computation ·The results from the targeted online consultation
		Extent to which the SRSP actions and activities were feasible in terms of content thus ensuring that SRSP reaches its objectives effectively
		If and how contextual factors had any (positive or negative) effect on the implementation of actions and activities and the achievement of deliverables and results
Effectiveness - results	EQ.3.To what extent have SRSP actions and activities contributed to the design and implementation of institutional, administrative and structural reforms in the Member States?	Extent to which the provided technical support and its expected results are related to the design and implementation of institutional, administrative and structural reforms in the Member States	226 Number of projects whose expected results are related to the design and implementation of institutional, administrative and structural reforms in the Member States	·Percentage of projects whose expected results are related to the design and implementation of institutional, administrative and structural reforms in the Member States ·Percentage of projects whose expected results have been produced (related to the design and implementation of institutional, administrative and structural reforms in the Member States) ·Percentage of projects whose expected results (related to the design and implementation of institutional, administrative and structural reforms in the Member States) have a high likelihood of being achieved ·Perceived impacts of contextual factors on the projects’ implementation	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers ·The results of the targeted online consultation were used to confirm the results of the computation ·The results from the targeted online consultation
		The extent to which the expected results (of the projects related to the design and implementation of institutional, administrative and structural reforms in the Member States) have been achieved / are likely to be achieved
Effectiveness - results	EQ.4.To what extent have SRSP actions contributed to supporting national authorities in the design / implementation of their reforms according to their priorities?	Extent to which SRSP projects are addressing Member States’ own reform priorities	138 Number of projects within SRSP which are addressing Member States’ own reform priorities	·Percentage of projects addressing Member States’ own reform priorities ·Percentage of projects addressing Member States’ own reform priorities whose expected results have been achieved / have a high likelihood of being achieved	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers, Requests for technical support
		Extent to which SRSP projects addressing Member States’ own reform priorities have achieved / are likely to achieve their expected results
Effectiveness- results	EQ.5.To what extent SRSP actions were designed and implemented by taking into account initial conditions and expected socio-economic impacts?	Extent to which the designed and / or implemented SRSP projects considered the challenges beneficiary authorities were facing when requesting technical support	303	·Distribution of projects by specific objective ·Perception of the stakeholders of the extent to which the projects were designed and implemented by taking into account the challenges beneficiary authorities were facing when requesting technical support ·Perception of the stakeholders regarding the extent to which the projects were designed by taking into account the expected socio-economic impacts ·Perceived alignment of the projects with the initial conditions and expected socio-economic impacts of the support	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers ·The results from the targeted online consultation ·The results from the targeted online consultation
		Extent to which the expected socio-economic impacts of the support were considered when defining the content of the technical support requests
Effectiveness - results	EQ.6.Did the intervention pay due attention to accompanying measures and / or appropriate sequencing of reform measures?	Extent to which, within each Member State, SRSP has supported complementary projects on the same topic	303	·Percentage of complementary projects (related to the same topic at Member State level)	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers
		Extent to which, within each Member State, SRSP has supported successive projects addressing different aspects / phases of a structural reform
Effectiveness - results	EQ.7.To what extent have SRSP actions contributed to supporting the national authorities in enhancing their capacity to formulate, develop and implement reform policies and strategies?	Extent to which the provided technical support and its expected results were related to the formulation, development and implementation of reform policies and strategies	64 Number of projects that are related to enhancing Member States' capacity to formulate, develop and implement reform policies and strategies	·Percentage of projects whose expected results are related to enhancing Member States' capacity to formulate, develop and implement reform policies and strategies ·Degree of achievement of the expected results ·Perceived impact of contextual factors on the projects’ implementation	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers ·The results of the targeted online consultation
		Extent to which SRSP projects related to enhancing Member States' capacity to formulate, develop and implement reform policies and strategies have achieved / are likely to achieve their expected results
Effectiveness - results	EQ.8.To what extent have SRSP actions contributed to supporting the efforts of national authorities to define and implement appropriate processes and methodologies by taking into account good practices and lessons learned by other countries in addressing similar situations?	Extent to which the technical support provided under SRSP and its expected results are related to the definition of new processes and methodologies	110 Number of projects related to the definition of new processes and methodologies while taking into account successful experiences from other EU Member States	·Percentage of projects whose expected results are related to new processes and methodologies ·Percentage of projects whose expected results are related to new processes and methodologies defined while taking into account successful experiences from other EU Member States ·Percentage of projects whose expected results (related to new processes and methodologies while taking into account successful experiences from other EU Member States) have been achieved ·Percentage of projects whose expected results have a high likelihood of being achieved been achieved ·Perceived impact of contextual factors on the projects ‘implementation	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers ·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers  ·The results of the targeted online consultation
		Extent to which the definition (within SRSP) of the new processes and methodologies has taken into account successful experiences from others EU Member States
		Extent to which the expected results (of the projects related to the definition of new processes and methodologies, while taking into account successful experiences from other EU Member States) have been achieved / have a high likelihood of being achieved
Effectiveness - results	EQ.9.To what extent have SRSP actions contributed to assisting the national authorities in enhancing the efficiency and effectiveness of human-resource management?	Extent to which the technical support provided under SRSP and its expected results are related to the enhancement of the efficiency and effectiveness of human-resource management	40 Number of projects that are related to the enhancement of the efficiency and effectiveness of human-resource management	·Percentage of projects whose expected results are related to the enhancement of the efficiency and effectiveness of human-resource management ·Percentage of projects whose expected results (related to the enhancement of the efficiency and effectiveness of human-resource management) have been achieved ·Perceived impact of contextual factors on the projects’ implementation	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers ·The results from the targeted online consultation
		Extent to which the expected results (of the projects related to the enhancement of the efficiency and effectiveness of human-resource management) have been achieved / have a high likelihood of being achieved
Effectiveness - impacts	EQ.10.To what extent has the SRSP provided support to Member States in implementing the relevant reforms in the policy areas as planned in the CSP and enhance their impacts on the ground?	Extent to which technical support provided under SRSP is contributing to implementation of relevant reforms	303	·Percentage of relevant reforms implemented in the policy areas as planned in the cooperation and support plans ·Percentage of projects whose expected results have been produced ·The perception of stakeholders on the specific conditions under which reforms designed under SRSP will have impacts on the ground ·Types of technical support provided under SRSP contributing to implementation of relevant reforms for which impacts have been observed	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers ·Desk research (type of documents: EU legal acts, other Commission resources) ·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers ·Information collected through the targeted interviews and the interviews conducted in the context of the case studies
		Extent to which technical support provided under SRSP can have impacts on the ground under specific conditions
Effectiveness - impacts	EQ.11.To what extent can observed impacts be causally linked to the implementation of the reforms and to the support provided by the programme?	Extent to which observed impacts are casually linked to the implementation of the reforms and to the support provided by the programme	303	·The perception of the stakeholders regarding the impacts of the technical support ·Types of impacts produced of the technical support provided under SRSP	·The open results of the targeted online consultation ·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers; ·Information collected through the interviews conducted in the context of the case studies ·Desk research
Effectiveness - impacts	EQ.12.To what extent have the expected results set in the cooperation and support plans been achieved so far?	The extent to which the SRSP projects achieved the expected results	303	·Percentage of projects whose expected results have been produced ·Percentage of projects whose expected results have a high likelihood of being achieved ·The perception of the stakeholders regarding the factors that have an influence on the observed changes linked to the technical support	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers ·The open results of the targeted online consultation
		Factors that influence on the observed impacts of the technical support
Efficiency	EQ.13.To what extent is the governance of the SRSP efficient and how efficient is the cooperation with other Commission Services?	Allocation of responsibilities among all stakeholders involved ensures the achievement of the SRSP objectives together with an optimal use of the financial and human resources. This implies the absence of bottlenecks or needs for simplifications / streamlining of processes / procedures	126 The analysis related to this component considers only SRSP 2018, which was the first year that allowed for the planned timeline to be followed according to the SRSP Regulation (i.e. due to the late adoption of the SRSP Regulation, SRSP 2017 had to follow a shortened timeline). The assessment does not consider PA, SRSP 2017 and Article 11 given that, due to their specific characteristic as well as contextual factors, they followed non-standard timelines	·Existence of bottlenecks and needs for simplification / streamlining of processes / procedures and of roles and responsibilities ·Extent to which the SRSP monitoring procedures and processes, including tools, are adequate and efficiently undertaken / applied ·Perception of the stakeholders on the coordination between the SRSS and Commission departments ·The extent to which the analysis and the selection procedures allow the selection of good quality technical support requests	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers  ·Desk research (type of documents: EU legal acts, other Commission resources) ·The results from the targeted online consultation ·Information collected through the interviews conducted in the context of the case studies ·Targeted interviews with Directorates-General (those interviewed were: DG COMP, DG EMPL, DG JUST, DG REGIO, DG SANTE, DG TAXUD) ·Data provided by the Commission in February 2019
		Implementation of the programme, including its monitoring, is aligned with the needs of the target groups (i.e. beneficiary authorities)
		All relevant stakeholders are involved in an effective manner in order to ensure that all right types of expertise are available in all programme phases (i.e. effective cooperation)
Efficiency	EQ.14.To what extent have the actions been cost-effective so far?	Achieved results are commensurate to the budget	201 The cost-effectiveness analysis was possible only for those projects (201 out of the 303 in the sample of projects evaluated) for which information is available regarding both approved financial resources and progress achieved with the expected results	·The extent to which the achieved results are correlated to the approved financial resources	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers
Efficiency	EQ.15.How efficient is the support provided in terms of design, means of delivery and time taken from the reception of requests from Member States to the start of the action's implementation?	Design of the technical support was undertaken in the needed timeline	126 The analysis related to this component considers only SRSP 2018, which was the first year that allowed for the planned timeline to be followed according to the SRSP Regulation. The assessment does not consider the Preparatory Action, SRSP 2017 and Article 11 given that, due to their specific characteristic as well as contextual factors, they followed non-standard timelines	·Average duration of the design of the technical support requests ·Average duration of the design of the technical support projects ·Perception of the stakeholders on the timelines of the design of the technical support ·The extent to which the quality of the technical support requests increased from one SRSP annual cycle to another	·Desk research (type of documents: EU legal acts, EU preparatory documents, other European Commission resources, online resources) ·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers Desk research (type of documents: online resources) ·The results from the targeted online consultation ·Information collected through the interviews conducted in the context of the case studies ·Dataset with the scores of the technical support requests for SRSP 2017 and SRSP 2018, consulted on 14 February 2019
		Quality of the technical support requests increased over time
		Means of delivery included were the most appropriate to meet the objectives of the project(s) and to cover the needs of the beneficiary authorities
Efficiency	EQ.16.How time-efficient is budget execution from commitments to payments?	Financial resources have been contracted in a time-efficient manner	280 The analysis covers projects under SRSP 2017, SRSP 2018 and Article 11 (2018). Projects and means of delivery concerning the Preparatory Action (2016) are not addressed under this EQ due to limited availability of information.	·Alignment of the absorption capacity to the status of the projects ·The extent to which the means of delivery are marked by delays	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers
Coherence	EQ.17.To what extent is the SRSP coherent both internally with its own set of objectives, and externally with other Union programmes, instruments and policies at, Union and international level?	Extent to which the logic of intervention is internally coherent	303	·The extent to which the SRSP LoI is internally coherent ·The perception of the stakeholders with respect to overlaps, complementarities and synergies ·The extent to which there are other policies and programmes implemented with European funds that have similar scope and objectives with SRSP ·The extent to which the SRSS general / specific objectives and general / specific objectives of similar programmes and policies at regional, national, Union and international level are coherent	·Desk research (EU regulations, reports, decisions from European Commission) ·The results from the targeted online consultation ·The results from scoping interviews with HoU / Hoc, deputy HoU and advisers, country coordinators ·Targeted interviews with Directorates-General (those interviewed were: DG COMP, DG EMPL, DG JUST, DG REGIO, DG SANTE, DG TAXUD)
		The perception of the stakeholders with respect to overlaps, complementarities and synergies.
		The extent to which there are other policies and programmes implemented with European funds that have similar scope and objectives with SRSP.
		The extent to which the SRSS general / specific objectives and general / specific objectives of similar programmes and policies at regional, national, Union and international level are coherent.
Coherence	EQ.18.To what extent some actions supported by the SRSP could have also been supported by other EU Instruments?	Number of other EU instruments that have similar objectives like SRSP	303	·Number of other EU instruments that have similar objectives like SRSP ·Reasons for choosing SRSP over alternative EU instruments with similar objectives	·The results from targeted online consultation ·The results from scoping interviews HoU / Hoc, deputy HoU and advisers, country coordinators ·Targeted interviews with Directorates-General (those interviewed were: DG COMP, DG EMPL, DG JUST, DG REGIO, DG SANTE, DG TAXUD) ·The results of the case study interviews
		Reasons for choosing SRSP over alternative EU instruments with similar objectives
Coherence	EQ.19.To what extent is the SRSP integrated in the Economic governance processes of the EU, such as the European Semester?	Percentage of SRSP projects answering to recommendations identified / issued under the European Semester review mechanisms	257 The 46 projects implemented in Greece were excluded from this analysis, as Greece was not part of the European Semester until mid-2018, when it exited economic adjustment programmes.	·Percentage of SRSP projects answering to recommendations identified / issued under the European Semester review mechanisms ·Percentage of projects related to recommendations issued under the European Semester reviews mechanism whose expected results have been achieved / have a high likelihood of being achieved	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers, Requests for technical support ·
		Percentage of projects related to recommendations issued under the European Semester reviews mechanism whose expected results have been achieved / have a high likelihood of being achieved
EU added value	EQ.20.To what extent does the SRSP add value compared to interventions led at national level by Member State (or other key donors and partners)?	Variation in the extent to which support under SRSP is requested by Member States.	303	·Number of new Member States that submitted technical support requests for the first time under each budgetary cycle (PA (2016)	·Data provided by the EC in February 2019 regarding the technical support requests; ·Information available in the Commission’s internal IT system
		Perception of the beneficiary authorities of the added value provided by SRSP compared to alternative instruments
EU added value	EQ.21.To what extent has the programme produced cross border cooperation in the areas of intervention?	The extent to which the beneficiary authorities requested technical support to address cross-border needs	303	·Perception of beneficiary authorities and coordinating authorities of the extent to which the needs have had a cross-border component ·Percentage of projects designed based on the technical support requests of at least two neighbouring Member States ·Number of projects expected to bring added-value through an impact on cross-border or EU-wide challenges ·Perception of beneficiary authorities and coordinating authorities of the cross-border impacts	·Information collected through the interviews conducted in the context of the case studies ·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers ·
		The extent to which projects from different Member States have been implemented to address common regional challenges
		The extent to which the technical support produced cross-border impacts
EU added value	EQ.22.To what extent is the programme contributing to sharing good practices among Member States, also with a view to increasing the visibility of the reform programmes, better identifying the need for possible accompanying measures and / or sequencing of reforms, and to building a Union-wide platform and network of expertise?	The extent to which technical support was delivered by using good practices and exchange of information and experiences	303	·Percentage of projects that used good practices and lessons learned ·Percentage of means of delivery that used good practices and lessons learned ·Percentage of means of delivery that include short-term expert mission(s) ·Percentage of means of delivery that include study visit (of officials of the beneficiary Member State) ·Percentage of means of delivery that include workshops/conferences/seminars ·Types of outputs developed using good practices ·Percentage of means of delivery that have been implemented by government bodies, public servant expert, public law bodies and bodies governed by private law with a public service mission ·Percentage of respondents to the targeted online consultation who believe that the technical support increased the sharing of knowledge and good practices among countries ·The extent to which the monitoring database of SRSS collects information on the results/impacts of the projects ·Perception of the coordinating authorities of the extent to which some actions have been taken to ensure visibility of SRSP technical support projects	·Information available in the Commission’s internal IT system updated in accordance with the results of the targeted interviews with the project managers ·The results of the targeted online consultation
		The extent to which the technical support increased cooperation among institutions in different Member States	181 means of delivery The assessment was performed at means of delivery level because information about the type of provider was only available at this level. 181 is the number of means of delivery that used good practices and lessons learned
		The extent to which the exchange of good practices among Member States contributed to the visibility of the reform programmes	303

(1)  Regulation (EU) 2017/825 of the European Parliament and of the Council, OJ L 129, 19.5.2017, p. 1–16, as amended by Regulation (EU) 2018/1671, OJ L 284, 12.11.2018, p. 3–5 (the SRSP Regulation). 

(2) Article 16 requires the mid-term evaluation report to ‘include information on the achievement of the Programme's objectives, the efficiency of the use of resources and the Programme's European added value. It shall also address the continued relevance of all objectives and actions.’ All this is covered under the Commission’s Better Regulation criteria.

(3)  Article 11 (of the SRSP) refers to the voluntary contributions that a Member State can transfer from the structural funds to the SRSP for the use of that particular Member State.

(4)  The Preparatory Action was introduced in 2016, before the SRSP was adopted, to help certain Member States. Six Member States benefitted from this support (Bulgaria, Czechia, Croatia, Romania, Slovenia and Slovakia) for a value of EUR 3 million.

(5)

During the time period evaluated, the Structural Reform Support Service (SRSS) was responsible for managing the SRSP. On 1 January 2020, DG REFORM succeeded the SRSS. This SWD refers only to DG REFORM, even though during the implementation period the mid-term evaluation the SRSS was the service responsible.

(6)

 Although regional and local administrations can benefit from technical support under the SRSP, national administrations were the beneficiary authorities for the vast majority of the projects.

(7)

Article 4 of the SRSP Regulation.

(8)

Article 5(1) of the SRSP Regulation.

(9) New Article 5a of the SRSP Regulation.

(10)

The expected results are also commonly referred to as outcomes.

(11) The provision of support under Article 11 (support to be funded through additional voluntary contributions) follows a different timeline and is linked to the transfer of the additional contributions from the Member State concerned.

(12) These criteria are: urgency, breadth and depth of the problems identified, the support needs in the relevant policy area, the analysis of socioeconomic indicators, and the administrative capacity of the beneficiary authority concerned.

(13)  Under Article 11 of the SRSP Regulation, Member States can make an additional voluntary contribution to the programme, transferring their resources dedicated to technical assistance to the SRSP. A contribution from a Member State is used exclusively to support requests for technical support in that particular Member State. In 2018, Bulgaria and Greece made use of this possibility, which amounted to an additional EUR 21.5 million. Furthermore, EUR 3 million were allocated to the Preparatory Action (2016).

(14) Technically a project is a ‘support measure’ under the SRSP Regulation terminology.

(15) This figure excludes two multi-country projects.

(16)

 TAIEX provides short-term expertise exchange by public sector experts from Member States. It is not contracted, but utilised through a Service Level Agreement between DG REFORM and DG NEAR.

(17)

 The remaining share can be attributed to other types of providers or missing information. For projects funded under Article 11, the providers most involved were international organisations and government bodies, delivering approximately 81% of the projects evaluated.

(18)

SRSP Annual Monitoring Report, 2017.

(19)

SRSP Annual Monitoring Report, 2018.

(20)

Date of consultation: 30 April 2019. SRSP 2017: 36.7% (44 out of 120). SRSP 2018: 7.9% (10 out of 126). A completed project refers to a project whose final deliverables have been approved.

(21) Ernst & Young, ‘Mid-term Evaluation of the Structural Reform Support Programme (SRSP) 2017-2020’ (2020), available on https://data.europa.eu/doi/10.2887/656262 . Executive summaries are available on https://data.europa.eu/doi/10.2887/584399 (English), https://data.europa.eu/doi/10.2887/244815 (French), and https://data.europa.eu/doi/10.2887/530192 (German).

(22)

 Some of the projects (means of delivery) were excluded from the sample, as EY was a provider for some projects and some would be evaluated under Fiscalis (subject to a separate evaluation).

(23)

The mid-term evaluation targeted only beneficiary authorities that received technical support – authorities of which the technical support requests were not selected have not been consulted.

(24)

In total 429 stakeholders were contacted: 230 beneficiary authorities (response rate: 46%), 173 technical support providers (response rate: 39%) and 26 coordinating authorities (response rate: 34%). The avarage response rate of the targeted online consultation was 43%.

(25)  An overview of the results of the public consultation is provided in the annex of this document.

(26)  112 beneficiary authorities and 77 providers were consulted. Among the providers, 67.5% indicated ‘a) administrative and institutional capacity’ and 49.5% indicated ‘b) challenges in the design and implementation of structural reforms in line with the Union’s economic and social goals’ as main challenges faced by Member States requesting technical support.

(27)

 Based on the interviews with stakeholders of the relevant case studies. 

(28)

The analysis measured the percentage of projects whose actions and activities were addressing the Member States’ identified needs. 16.7% (41 out of 246) of the projects are related to all three of the needs mentioned above; 47.6% (117 out of 246) are related to two of the needs mentioned above; 35.0% (86 out of 246) addressed only one of the needs mentioned above.

(29)

 Four to five respondents (question subdivided per policy area) expressed the view that the SRSP responds ‘to a high extent’ or ‘to the fullest extent’ to the EU Member State’s needs. 

(30)

 See logic of intervention ( Figure 1 ). 

(31)

97.3% (180/185) of the stakeholders consulted – 99% of beneficiaries (106/107), 94.2% of providers (65/69), and all coordinating authorities (9/9) - strongly agreed or agreed that the SRSP is a suitable instrument to provide technical support.

(32)

When asked about the extent to which their institution still needed technical support, 65% (i.e. 69 out of 107) of the beneficiaries declared they need it to a large or to a very large extent.

(33)

 Based on interviews with 12 beneficiary authorities of the case studies.

(34) Based on interviews with 12 beneficiary authorities of the case studies.

(35)  Within the scope of this evaluation, data from SRSP2017 and SRSP2018 show a considerable increase. 24 Member States submitted 444 requests for support (the SRSP 2017 round saw 271 requests for support from 16 Member States).

(36) According to Article 16(3) of the SRSP Regulation, the evaluation of the long-term impact of the programme is subject to an ex post evaluation.

(37) More than 40% of the projects evaluated had as objective support to the design of reforms. More than 50% were linked either to supporting the development and implementation of reform policies or supporting the definition and implementation of processes and methodologies. Additionally, more than 10% of the projects evaluated had support to more efficient and effective human resource management as one of their objectives. Around 60% covered more than one specific objective. The evidence collected covers projects under SRSP 2017 and SRSP 2018 and was based on the Commission’s internal IT system, updated in accordance with targeted interviews with project managers.

(38)

Targeted online consultation of 113 replies from beneficiary authorities to this question: 85% confirm (completely taken into account (41%) or to a large extent taken into account (44%)); 11% to a small extent or not at all.

(39)

Online consultation of replies from beneficiary authorities: 72% (81 out of 113) said that socio-economic impacts had been taken into account when defining the content of the technical support requests. In cases in which socio-economic impacts had not been taken into account, respondents indicated in most cases (5 out of 7) that the project did not have socio-economic indicators.

(40)

The degree of comprehensiveness/complementarity was measured on the basis of the number of projects addressing a topic that is already covered by at least one other project, compared to the total number of projects implemented in that Member State.

(41)

58.5% of the projects evaluated (144 out of 246); 56.4% of the total value of all projects evaluated under SRSP 2017 and SRSP 2018. This conclusion is based on the assumption that ‘short-term expert missions’, ‘study visits’ and ‘workshops/conferences/seminars’ are eligible activities that imply the exchange of good practices and lessons learned from other EU countries. Evidence based on the Commission’s internal IT system, updated after interviews with project managers and the targeted online consultation were used to confirm the results.

(42)

84.1% of the evaluated projects under the SRSP 2017 and SRSP 2018 (207 out of 246).

(43)

 4.9% (16 out of 246) of the evaluated projects under SRSP 2017 and SRSP 2018 were marked as having delays.

(44)

The achievement of expected results refers to expected results that were defined in the Commission’s internal IT system or specified in the targeted interviews. The assessment of the extent to which the expected results of the evaluated projects had been achieved was based on an ad hoc scoring system. For each project for which information was available, a score was assigned on the basis of the percentage of expected results that were produced, based on the information available in the Commission’s internal IT system and following the targeted interviews with project managers.

(45)

 For projects where the results had not been achieved, a second assessment was carried out regarding the likelihood that the results would still be achieved, assessing the extent to which factors existed that could hinder or contribute to the achievement of expected results. 

(46)

 The achievement of results is not entirely dependent on the status of the implementation of the projects. It is possible to encounter completed projects that have not entirely achieved their results, as well as ongoing projects that have achieved their results.

(47)

These completed projects have gone beyond their planned end date, but activities are still ongoing.

(48)

95 projects out of 110.

(49)

Evidence from the targeted online consultation.

(50)

The number of answers and respondents per category is 38.

(51) Three to six respondents (question subdivided per objective) indicated that the technical support contributed ‘to a high extent’ or ‘to the fullest extent’ to achieving the SRSP’s objectives.

(52)

56.0% of respondents named this as a positively influencing factor (14 mentions; 25 respondents); respondents were given the possibility of providing more than one answer.

(53)

Lack of political support: 55.6% (5 mentions); lack of political stability: 33.3% (3 mentions; 9 respondents); respondents were given the possibility of providing more than one answer.

(54)  For this mid-term evaluation, SRSP governance refers to the formal and informal arrangements that determined how decisions were made and how SRSP projects were implemented between 2016 and 2018. These arrangements include processes, roles and responsibilities that have been established in order to help achieve the programme’s objectives.

(55)

The cooperation and support plans are being prepared before the adoption of the Financing Decision. After the Financing Decision is adopted, the cooperation and support plans are signed.

(56)

 102 out of 126 of the SRSP 2018 projects.

(57)

The start of the project is considered to be the launch of the procedures for concluding contracts/agreements with technical support providers.

(58)

Project completed includes the project’s evaluation, after the activities have been carried out.

(59)

89 out of 126 responses to this question.

(60)

The duration of each phase was calculated based on official documents or by computing averages based on data from the Commission’s internal IT system.

(61)

In numbers this corresponds to 88 of the 104 beneficiary authorities, 9 of the 9 national coordinating authorities, and 57 of the 69 providers who answered this question of the targeted online consultation.

(62)

 The majority of the respondents view the SRSP as being to some extent, to a high extent or to the fullest extent efficient (rapid, timely, cost-effective) compared to other EU, national or local interventions. 

(63)

2 of the 21 providers interviewed.

(64)

20 of the 21 beneficiary authorities stated that the interaction with the providers was good.

(65)

In response to the targeted online consultation question ‘Do you agree with the following statement: The procedures and tools used to monitor the implementation of the technical support are effective?’, 79.8% of the beneficiaries (83 out of 104) answered either ’agree’ or ‘strongly agree’ while 6.7% (7 out of 104) answered either ‘disagree’ or ‘strongly disagree’; 83% of the providers (54 out of 65) answered ‘strongly agree’ and ‘agree’; 66.6% of the coordinating authorities (6 out 9) answered ‘agree’ and ’strongly agree’ while 22.2% answered ‘disagree’ (2 out of 9).

(66)

The relevant question in the scoping interviews was: ‘How efficient do you find the cooperation with the SRSS and the other Commission Services within the coordination mechanism?’ 6 of the 6 Commission departments interviewed said the cooperation was efficient.

(67)

In the analysis and selection phase, each technical support request received several scores, which have been processed for the scope of the evaluation. Each request had a set of scores with the letters ‘A’, ‘B’ and ‘C’. The letters were converted to numbers (‘3’ for ‘A’, ‘2’ for ‘B’, ‘1’ for ‘C’,’0’ where there was no score) in order to compute the average of the scores for each project.

(68)

 The following score was applied to the projects: 1: No steps have been taken towards the results; 2: A few steps have been taken towards the results; 3: Several steps have been taken towards the results; 4: The results were achieved. In total, 167 out of the total sample of 246 projects (SRSP 2017 and SRSP 2018) were subject to this analysis. Projects for which information was not available on both approved financial resources and progress achieved with respect to the expected results were excluded. 

(69)

The following score was applied to the projects: 1: The results will certainly not be achieved; 2: There is a low likelihood that the results will be achieved; 3: There is a high likelihood that the results will be achieved. The score for the likelihood that completed projects which did not yet achieve their expected results (11 out of 49) would still do so was 2.6 out of 3.

(70)

Ratio between the financial resources contracted and the financial resources approved.

(71)

The population analysed comprises 386 means of delivery in the scope of this evaluation for SRSP 2017, SRSP 2018 and Article 11 (2018).

(72)  Subject to assumptions and contextual factors.

(73) NRP = national reform programme; CR = country report; ES = European Semester

(74) See the following section about the horizontal part of the coordination mechanism.

(75)

SRSP Annual Monitoring Report 2017; SRSP Annual Monitoring Report 2018.

(76)

25 out of 116 projects, based on an ad hoc scoring system.

(77)

48 out of 58 projects, based on an ad hoc scoring system.

(78)

Template for consultation with Commission departments.

(79)

100% satisfaction (15/15) European Commission, ‘Annual Activity Report 2018 – SRSS’.

(80)

Disclaimer in the SRSP request template; see Article 12 of the Regulation (EU) 2017/825 (SRSP Regulation).

(81)

27 out of 107 beneficiary authority respondents.

(82)

10 out of 17 beneficiary authority respondents mentioned the European Social Fund. 2 mentioned Fiscalis. 2 mentioned Erasmus+. 3 mentioned other EU programmes.

(83)

 8 out of 22 respondents.

(84) This was indicated by 5 out of 22 beneficiary authorities (23%) and 1 out of 5 respondents in the case study of Greece.

(85)

‘Reform design and implementation in Member States, as a response to the CSRs has not yet reached the desired level and outcome’ (SRSP ex ante evaluation, p.5).

(86)

36% of the beneficiary authorities (8 out of 22) mentioned in the interviews the low level of bureaucracy and the speedy technical support. 34% of the beneficiary authorities (9 out of 26) mentioned the simplified contracting and implementing procedures. 23% (5 out of 22) mentioned no need for co-financing as a reason for requesting support under SRSP instead of other policies or programmes.

(87)

As part of the Preparatory Action (2016), six Member States received support: Bulgaria, Czechia, Croatia, Romania, Slovenia and Slovakia. In the subsequent year (SRSP 2017), 10 new Member States submitted requests for the first time: Cyprus, Estonia, Greece, France, Italy, Lithuania, Latvia, Malta, Poland and Portugal. In the year after that (SRSP 2018), 9 new Member States requested support: Austria, Belgium, Germany, Spain, Finland, Hungary, Ireland, Luxembourg and Sweden.

(88)

13 out of 16 Member States.

(89)

159 out of 303 projects; 181 out of 409 means of delivery (44.3%).

(90)

Based on the assessed means of delivery that includes good practice sharing.

(91) 51 out of 181.

(92)

Targeted online consultation: 31 out of 34.

(93) 6 out of 9 respondents indicated that the SRSP adds value to a high extent or to the fullest extent in terms of sharing good practices; 7 out of 9 respondents indicated that the SRSP adds value to some extent or to a high extent in terms of promoting cooperation.

(94)

None of the 22 interviewed beneficiary authorities mentioned any past or future need for technical support for cross-border cooperation.

(95)

1 out of the 303 projects evaluated: ‘Strengthening capital markets in Estonia, Latvia, Lithuania and Poland’. The technical support provided for the creation and strengthening of a legal and regulatory framework with respect to structured products and hedging instruments aimed to contribute to the development of the pan-Baltic capital market. The expertise was ensured by the same provider in all the countries.

(96)

9% of the beneficiary authorities interviewed (2 out of 22) and 12% of the coordinating authorities interviewed (1 out of 8) could indicate the existence of cross-border impacts; Individual cases: Slovakia’s experience with the implementation of the spending review process prompted other countries to seek information from Slovakia about how the reform was addressed under the SRSP. Similarly, Croatia shared its experience of undertaking the functional integration of hospitals with SRSP support at an international workshop organised by the World Bank. At the same time, Italy was the first European country to have a national aid register, and it shared the instructions on how to draft this register with Greece as well.

(97) 5 out of 10 repondents indicated that the SRSP adds value to a high extent in terms of addressing cross-border or Union-wide challenges.

(98) At that time, DG REFORM was still SRSS.

(99) For more information see: Ernst&Young, 2020 “Mid-term Evaluation of the Structural Reform Support Programme (SRSP) 2017-2020”, Final report, May 2020 – ISBN 978-92-76-18436-2; https://op.europa.eu

(100) Drafted by the external contractor.