Table of contents

1.Introduction

2.What was the expected outcome of the intervention?

The ex-ante evaluation of the SRSP    

The mid-term evaluation of the SRSP    

3.How has the situation evolved over the evaluation period?

Current state of play    

Main outputs and results achieved during the implementation of the programme    

4.Evaluation findings (analytical part)

4.1.    To what extent was the intervention successful and why?    

4.1.1.    Effectiveness: How effective has the SRSP been in achieving its objectives?    

4.1.2 Efficiency: How efficient has the SRSP been in achieving its objectives?    

4.1.3 Coherence    

4.2 How did the EU intervention make a difference and to whom?    

4.3 Is the intervention still relevant?    

5.What are the conclusions and lessons learned?

5.1Conclusions

5.2Lessons learned

Annex I. Procedural information

1.Lead DG, Decide Planning/CWP references

2.Exceptions to the better regulation guidelines

3.Organisation and timing

4.Consultation of the RSB (if applicable)

5.Evidence, sources and quality – including external expertise

Annex II. Methodology and analytical models used

Annex III. Evaluation matrix and, where relevant, details on answers to the evaluation questions (by criterion)

1. Effectiveness (results)    

2. Impact    

3. Efficiency    

4. Relevance    

5. Coherence    

6. EU added value    

Annex IV. Overview of benefits and costs and, where relevant, table on simplification and burden reduction.

Annex V. Stakeholder consultation – synopsis report

1.1.    Summarised results of the Open Public Consultation (OPC)    

1.2.    Summarised results of the targeted survey    

1.3.    Summarised results of interviews    

1.4.    Results of focus group discussions    

2.1.    Main outputs and results achieved during the implementation of the programme    

Glossary

Structural reforms are essential for modernising economies, encouraging investment, creating jobs and raising living standards. Reforms are, by their very nature complex processes. To facilitate these reforms across the European Union, Regulation (EU) 2017/825 1 on the establishment of the Structural Reform Support Programme (“the SRSP Regulation”) was adopted in May 2017.

The objective of the Structural Reform Support Programme (“the programme” or “the SRSP”) was to contribute to institutional, administrative, and growth-sustaining structural reforms in Member States, by providing technical support to national authorities. The budget for the programme was EUR 222.8 million over 2017-20. The programme was managed by the Structural Reform Support Service (SRSS), which subsequently became the Directorate—General for Structural Reform Support (DG REFORM). 2

This staff working document provides an ex-post evaluation of the programme. In compliance with the Better Regulation guidelines, this evaluation is based on the following five criteria:

·Relevance – do the objectives of the SRSP appropriately address the needs in Member States?

·Effectiveness – how successful is the SRSP in achieving or progressing towards its objectives?

·Efficiency – are the effects of the SRSP achieved at a reasonable cost?

·Coherence – to what extent are the various components of the SRSP’s intervention logic coherent (internal coherence) and how does the SRSP interact with other EU interventions with similar objectives (external consistency)?

·EU added value – what is the additional value resulting from action at EU level, compared to what could be achieved by Member States acting alone at national and/or regional level?

This ex-post evaluation aims to help improve the provision of further technical support to Member States and to identify good practice for designing a future technical support programme in the next programming period (beyond 2027). This report is also intended as a tool to share the results of DG REFORM’s action with stakeholders and promote the use of evaluations as an evidence-based policy making tool.

In accordance with Article 16 of the SRSP Regulation, an external contractor was engaged to produce an independent evaluation study (“the study”).

This document presents and reflects on the main outcomes and findings of the Study, but also draws on other DG REFORM sources of information, including additional data from the internal IT system, annual monitoring reports, annual activity reports and the mid-term and ex-ante evaluation of the SRSP.

This document has the following structure:

-Section 2 describes the background of the programme and summarises its expected outcome.

-Section 3 covers the evolution of the situation during the evaluation period.

-Section 4 provides the main findings of the evaluation.

-Section 5 presents the conclusions and recommendations.

The general objective of the SRSP was to contribute to institutional, administrative and growth-sustaining structural reforms in the Member States by providing support to national authorities. The support may also help Member States preparing for participation in the euro area. 3  

To achieve this general objective, the SRSP Regulation has set the following specific objectives: 4

a)To support the initiatives of national authorities to design their reforms according to their priorities, considering initial conditions and expected socioeconomic impacts.

b)To support the national authorities in enhancing their capacity to formulate, develop and implement reform policies and strategies and in pursuing an integrated approach ensuring consistency between goals and means across sectors.

c)To support the efforts of national authorities to define and implement appropriate processes and methodologies by considering good practices of and lessons learned by other countries in addressing similar situations.

d)To assist the national authorities in enhancing the efficiency and effectiveness of human-resource management, inter alia, by strengthening professional knowledge and skills and setting out clear responsibilities.

The Programme intended to provide technical support for the design and implementation of reforms that relate to a broad range of policy domains, including:

·Revenue administration and public financial management.

·Governance and public administration.

·Growth and business environment (including climate, energy and environment).

·Labour market, education, health and social policy.

·Financial services and access to finance.

Designing and implementing institutional and structural reforms is complex and demanding and requires a high degree of knowledge and skills; the overall successful implementation of reforms requires efficient and effective public administrations. For this reason, addressing the structural challenges faced by public administrations – especially by developing adequate institutional and administrative capacity – is crucial for well-functioning and sustainable economies and the success of structural reforms.

The SRSP ex-ante evaluation 5  identified the following needs as more relevant with regard to Member States’ capacity to implement solid reform agendas:

1)Limited administrative and institutional capacity.

2)Challenges in the design and implementation of growth-sustaining structural reforms in line with the Union’s economic and social goals.

3)Limited application and implementation of EU law.

The SRSP provided demand-driven and tailor-made support to the specific situation of the concerned country. Actions that were eligible for funding included the formulation of policy strategies or roadmaps, the provision of long- and short-term experts, and capacity building activities, such as workshops, training and working visits.

The above actions were anticipated to lead to the short-term expected results of SRSP, also referred to as project outcomes. Achieving these outcomes was expected to lead to the achievement of the SRSP specific objectives. Outcomes included the adoption of strategies, reform roadmaps or new legislation; the adoption of new or improved procedures and methodologies; organisational change and enhanced human resource management.

The achievement of the outcomes depended not only on the breadth and depth of technical support provided to EU Member States, but also on many other factors – such as timing and sequencing of reforms, reform ownership by the Member State – that are outside the control of the technical support project.

Technical support projects could cover any part of the reform cycle, from identifying the needs to monitoring and evaluating the outcomes. At the same time, the actual implementation of the reforms remained the prerogative of the recipient Member State.

Figure 1 below presents the intervention logic of the Programme, in line with the main elements of the theory of change. 6  

Figure 1: SRSP Logic of intervention

Based on the results of the ex-ante and mid-term evaluations and of the Study, the ex-post evaluation considers two key types of impacts:

-Institutional Impact: structural changes in the institutional and administrative capacity 7 of Member States’ authorities to prepare and implement reforms and to apply EU law in an effective way.

-Policy Impact: achievement of relevant longer-term growth-sustaining reforms in EU Member States as indicated in the cooperation and support plans signed by DG REFORM and each Member State.

Clearly, external factors such as economic, social and institutional conditions; the election cycle and political support for reforms on all levels of government; the scope of reforms; and the COVID-19 crisis played a role in contributing to or constraining the achievement of the programme impacts.

The SRSP Regulation does not specifically refer to any of the Sustainable Development Goals (SDGs). Nevertheless, the SRSP was expected to contribute implicitly to the achievement of several SDGs. According to the Joint Research Centre (JRC) toolkit, the programme was expected to contribute mainly to SDG 8 (decent work and economic growth), SDG 16 (peace, justice, and stronger institutions) and SDG 10 (reduced inequalities).

2.2 Points of comparison

This ex-post evaluation aims to capture change that the programme has brought over time. To do so, it compares the actual performance of the programme against:

-The situation before the intervention as described in the ex-ante evaluation of the programme (published in November 2015).

-The situation during the mid-term evaluation of the programme as described in the Study (published in February 2020 using the monitoring data from 2019).

The ex-ante evaluation of the SRSP

The ex-ante evaluation of the SRSP noted that Member States may face a range of challenges in the design and implementation of structural and administrative reforms, such as the limited administrative and institutional capacity and the inadequate application and implementation of Union legislation. 8  Quantification of these problems/challenges that the SRSP was deemed to solve, was translated into the following two parameters:

-The implementation of Country Specific Recommendations (CSRs) of the European Semester, indicates the difficulties Member States face in adopting reforms. For example, Member States made at least “some progress” in 48.4 % the CSRs addressed to them in 2015. In 2019, the implementation rate of CSRs was lower compared to 2015, 9  thus suggesting the continued need for technical support.

-The inadequate application and implementation of EU legislation was quantified in terms of formal infringement procedures. 10  

The ex-ante evaluation of the programme, envisaged that the SRSP will accelerate these reforms and ensure their quality and sustainability, thus having a considerable positive impact on increasing economic and societal welfare in the Member States in the long term. 11 Increased welfare of individuals and societies through non-market benefits (e.g. improvements in education, skills, social protection, health, long-term care) or improved competitiveness of businesses through market efficiency can only be treated as indirect benefits of the SRSP, since the programme focuses on the reform process.

The mid-term evaluation of the SRSP

By the time of mid-term evaluation study, only 22% of the SRSP projects (54 out of 246) included in its scope were completed 12 , therefore, it was too early to measure the results and impact of the programme. However, the mid-term evaluation found general improvement in the administrative and institutional capacity (by 73.7% of the stakeholders from the completed projects), the quality of the design and implementation of structural reforms (60.5%) and the application and implementation of EU law (42.1%) based on the results of stakeholder consultation. 13  

The submission, analysis and selection of technical support requests under the SRSP was organised in annual cycles, in accordance with the SRSP Regulation.

Each year, Member States submitted their requests for technical support by the end of October, as specified in the SRSP Regulation. 14  Once they had done so, the Commission assessed these requests against the criteria set out in Article 7 of the SRSP Regulation. 15 The Commission also verified whether requests overlapped with actions implemented under other EU instruments or programmes, with a view to avoiding double funding and ensuring complementarity.

Thereafter, the Commission selected the requests that were assessed to be a highest quality in line with the set criteria. The technical support projects that were financed through the Programme did not require co-financing by the Member State concerned. The SRSP was implemented via annual work programmes that served as a financing decision. They set out the support measures to be implemented for the year concerned, the dedicated annual budget, and the expected results, as well as the priority actions.

Subsequently, the Commission came to an agreement, titled as “Cooperation and Support Plan” (CSP), with each Member State. The CSPs, co-signed annually, covered the priority areas, the objectives, an indicative timeline, the scope of the support measures and the estimated financial contribution.

To facilitate efficient communication with the Commission and the coordinated submission of technical support requests, each Member State designated a national Coordinating Authority within its government structures. The Coordinating Authority acted as the interlocutor of the Commission for the implementation of the CSP, and the SRSP in general. It was responsible for collecting all requests for the Member State in question, and subsequently submitting the requests to the Commission.

The Coordinating Authority had the opportunity to prioritise the requests before submitting them to the Commission. The national beneficiary authorities (such as ministries, public agencies, regulatory authorities, regional/local authorities and other public bodies that have requested assistance) were the actual recipients of technical support. In line with the Regulation, the SRSP never provided financial support to the authorities, but only technical support.

The Coordinating Authority was responsible for coordinating the effective implementation of the support measures in the country and was also encouraged to take appropriate action in case of challenges during the implementation of technical support in its country.

Technical support, such as strategic and legal advice, studies, training, and expert visits on the ground, was delivered by technical support providers contracted by the Commission. DG REFORM carried out all financial management tasks related to the contracts.

The Commission was responsible for monitoring the implementation of the actions financed by the programme and measuring the achievement of the general and specific objectives of the SRSP in accordance with indicators set out in the Annex of the SRSP Regulation and monitoring the achievement of EU added value. In addition to the mid-term and ex post evaluation, the Commission also followed other mechanisms to monitor the Programme’s progress, including annual monitoring reports, annual activity reports and the overall strategic planning and programming cycle.

To monitor the implementation of the programme, DG REFORM developed and effectively deployed an internal monitoring system 16 , using two types of feedback questionnaires: (1) satisfaction questionnaires that were sent to main stakeholders (beneficiary authorities, providers, DG REFORM policy officers) at the end of the project to assess the satisfaction level with the design and the implementation of the project and (2) outcome questionnaires that were sent to beneficiary authorities within 6, 12 or 18 months after the project completion to assess the achievement level of expected outcomes.

Current state of play

The demand for the SRSP was high and increased throughout SRSP implementation. In line with the principle of sound financial management, some of the criteria of selection were the focus and maturity of requests that allowed for quick delivery of support and rapid implementation of impactful reforms. Table 1 below provides the number of technical support requests received and selected per SRSP cycle.

Table 1: technical support requests received and selected

Source: DG REFORM internal monitoring system.

Once requests for technical support were approved, they became projects. 17  However, DG REFORM could decide to implement one large request through more than one project, or to implement multiple related requests through one single project. Therefore, the number of selected requests could – and usually did – differ from the number of projects that were implemented.

In addition to the four annual SRSP rounds, technical support projects have also been funded by the voluntary contributions of Greece (2018 and 2020) and Bulgaria (2018) under Article 11 of the SRSP Regulation.

A project could have one or more technical support providers, depending on the expertise that is required to address the needs of the Member State. The type of provider determined the contractual procedure. The type of contract/procedure used to deliver the technical support project was referred to as the means of delivery.

When defining the selected request/projects by their circumstances, the majority of the 826 18 projects covered the ex-post evaluation were related to implementing challenges identified in the context of economic governance processes (see table 2 below). 

Table 2: Circumstances of the selected requests under the SRSP

Source: SRSP annual monitoring reports 2017-2020.

With respect to the thematic areas covered by SRSP, the largest number of projects was implemented in the field of sustainable growth and business environment (which includes climate change and green issues), and labour market, education, health, and social services.

Figure 2: SRSP-funded projects by thematic area, 2017-2020

Source: DG REFORM internal monitoring system (N=826).

The SRSP covered projects in all Member States. Over the period of 2017-2020, Greece, Croatia and Romania had the highest numbers of SRSP-funded projects. Fewer technical support projects were implemented in the Nordic countries, Luxembourg, Germany and the Netherlands, which were consistently the countries with the lowest number of submitted requests.

Figure 3: number of requests selected per Member State under SRSP by year, 2017-2020

Source: DG REFORM internal monitoring system (N=826).

Note: the number of requests in the countries with the lowest number of requests is presented in the brackets. The first figure in the brackets stands for the number of requests selected in 2020.

Several delivery modes were used during the implementation of the SRSP, namely:

·Direct management, which includes public procurement contracts, grants and reimbursement of costs incurred by external experts.

·Indirect management, through entities that have undergone an ex-ante assessment of their procedures and systems, guaranteeing a level of protection of the EU financial interests equivalent to that guaranteed by the Commission in direct management.

·Technical Assistance and Information Exchange instrument (TAIEX), which provides short-term expertise exchange by public sector experts from the Member States, through a Service Level Agreement between DG REFORM and DG NEAR. 19  

A technical support project could consist of a mix of different delivery modes to tailor the support to the needs of the Member State; however, 78% of projects used only one specific delivery mode. In total, there were 1,037 individual delivery modes applied during the programme’s implementation in 2017-2020.

Figure 4: Use of main delivery modes during the implementation of the SRSP, 2017-2020

Source: DG REFORM internal monitoring system (N=1,037).

A growing number of procurement procedures were used during 2019 and 2020, following the substantial increase in the budget of the programme in these two years. The relative share of this delivery mode (out of all the delivery modes that year) was 37% in the 2017 round, 34% in 2018, 43% in 2019 and 43% in 2020.

The selection of an individual delivery mode or a support provider depended on the needs of Member States and the nature of activities/outputs planned in each project. Public procurement via framework contracts with private companies was mostly used when project activities required many experts, interdisciplinary teams and other resources on the ground, close contacts with national authorities and good knowledge of the national language. In contrast, grants and delegation agreements with international organisations were usually used when an exchange of international good practice and a transfer of knowledge from other countries was key to achieving the project objectives.

Main outputs and results achieved during the implementation of the programme

According to the project descriptions, the SRSP was expected to produce 1,462 outputs in 2017-2020.

Most of the outputs that the programme was expected to produce were ‘recommendations’, ‘analysis and reports’, ‘workshops’, ‘action plans and roadmaps’ (see Figure 5). 20  

Overall, there were 1,352 technical support activities planned for implementation under the four rounds of the SRSP. 21 The Programme most often planned the following activities: study, research, evaluation (313 such activities within 826 projects) and workshops, conferences and seminars (270 such activities). Long term expert missions (26 activities) and organisation of local operations (27 activities) were the least frequently used activities in the technical support projects under the 2017-2020 rounds. 76% of the SRSP projects included more than one activity.

The technical support projects became more complex and significant over time, as evidenced by an increasing number of activities and outputs per project, as they started addressing more important reform needs. According to the results of the interviews, the financial size of the projects was also matched by a substantial improvement in the quality of requests for technical support from EU Member States.

Figure 5: Total expected outputs of the SRSP, 2017-2020

Source: DG REFORM based on internal monitoring data (N=1,462). Note: the categories of ‘other outputs’ and ‘none of the outputs’ were excluded from the analysis.

In the monitoring system of DG REFORM, projects could be associated to four groups of pre-defined outcomes (results), which corresponded to the four specific objectives of the Programme. Most often the Programme supported the achievement of the following outcomes: ‘adoption of (new) procedures and actions to enhance the implementation of reforms’ and ‘improved internal working procedures, methodologies and processes, organisation’. A similar distribution was observed during the mid-term evaluation of the 2017-2018 Programme.  22

Figure 6: Breakdown of the pre-defined outcomes in the SRSP, 2017-2020

Source: DG REFORM internal monitoring system (N=1,338).

This ex-post evaluation assesses the performance of the SRSP against the following five criteria: effectiveness, efficiency, coherence, EU added value, and relevance. The overall analysis is based on evidence from both the external Study and the Commission’s own sources.

4.1.To what extent was the intervention successful and why?

4.1.1.Effectiveness: How effective has the SRSP been in achieving its objectives?

The overall assessment on the Programme’s effectiveness is analysed threefold:

·The extent to which the programme contributed to its specific objectives and achieved its expected results, as well as to the institutional and policy changes that occurred during the implementation of the programme.

·The extent to which the programme contributed to its expected impact, both in terms of structural reforms and capacity building.

·Identification of factors that either facilitated or stalled progress towards the results and impact.

Contribution of SRSP actions to the expected results

The SRSP had four specific objectives to be pursued in close cooperation with beneficiary Member States. According to the Study (see Figure 7), technical support projects most often contributed to specific objective 2 and 3. The programme less frequently contributed to objective 4 because this very specific objective was implemented mostly through the technical support in the single field of public administration and governance.

Figure 7: Contribution of technical support projects to the specific objectives of the programme in 2019 and 2021

Source: the mid-term evaluation of the SRSP (data from 2019), the ex-post evaluation of the SRSP (data from 2021). The 2021 data does not include 11 cases where the projects are contributing to none of the specific objectives (N=7) or contributed to other objectives (N=4).

Data from the DG REFORM evaluation system and the subsequent analysis of the external contractor, suggests that the specific outcomes of projects were to a large extent achieved. In particular, the average score for achieving the specific outcomes (results) of the SRSP-funded projects was 8.2 out of 10 (on a scale from 1 to 10). 23  Two thirds of technical support projects achieved their outcomes (based on average scores 8.0 and above per project). 24  

Beneficiary authorities also generally considered the technical support projects to be successful. According to the results of DG REFORM satisfaction questionnaires, both beneficiary authorities and DG REFORM policy officers assessed positively the success of the technical support projects (average score was 8.7 for both dimensions), with the success of the projects depending mostly on the effectiveness of the interactions with the support providers, beneficiary authorities and DG REFORM.

Overall, the programme’s contribution to the achievement of the expected results was possibly due to the successful implementation of technical support activities and the good achievement of the intended outputs (deliverables), according to the case study analysis.

The achievement of the project outputs and results depended on the quality of technical support provided on the ground, including the providers’ expertise and skills. 25  The Study 26 also hints that the quality of technical support depended on the thematic and national expertise of support providers, their skills in project management and communication, as well as their ability to adapt to any changes in the project environment. Conversely, the absence of these conditions, language barriers or fragmentation of responsibilities sometimes had a detrimental effect on the quality of technical support.

National authorities were closely involved in the project pre-implementation phase; the ex-post evaluation suggests that in future programmes, national authorities should continue to be closely involved in the pre-implementation phase, in terms of being consulted on a suitable type of provider and/or contributing to the drafting of project description/specifications. The latter is even more important when national priorities and needs undergo substantial changes following the selection of requests for technical support, e.g., due to election cycle or change in hierarchy in the beneficiary authority.

Contribution of SRSP actions to the expected impacts

The ex-post evaluation of SRSP’s contribution to its expected impacts faced a few limitations/challenges. In particular:

-A considerable share (about 30%) of projects were still ongoing during the external study 27 .

-Evidence also suggests 28 that most projects implemented under the SRSP concerned policy development. Transforming these types of outputs into impacts in terms of institutional and policy change takes a lot of time. In parallel, external factors affect more the achievement of longer-term growth-sustaining reforms, compared to short term reforms/results. Consequently, the external study finalised in October 2022 did not observe many impacts (especially long-term and policy impacts), because such impacts are inevitably expected to arise at a later stage and also because they are subject to many confounding factors.

The ex-ante evaluation of the SRSP concluded in 2015 that the provision of technical support for the design and implementation of reforms will accelerate their implementation and ensure their quality and sustainability. 29  

Consequently, the Study also explored SRSP’s contribution to the design and implementation of specific types of reforms and policies. The data 30 shows a substantial contribution of the programme to reforms of public administration; in particular, since DG REFORM is the competence centre on public administration and governance in the European Commission, it was able to exploit several synergies in providing support in this field (see box 2).

The Study concludes 31 that the SRSP improved all types of administrative capacities. It contributed more to the adoption of different administrative systems and tools (different instruments, methods, guidelines, manuals, systems, procedures, forms, etc.) compared to the other two types of administrative capacities (institutional structures, human resources). The report also observed that the programme strengthened human resources’ capacity to implement reforms more significantly in those beneficiary authorities that implemented three or more technical support projects.

The SRSP contributed to the implementation of administrative reforms and capacity building more than to other structural reforms in individual EU Member States. This is related to the fact that other structural reforms, which encompass different types of reforms (e.g., product market reforms, labour market reforms or fiscal, tax and pension reforms) 32 , are usually complex, challenging and far-reaching initiatives aimed at promoting sustainable longer-term growth, employment and competitiveness, especially compared to administrative changes or institutional capacity building whose benefits could occur over a shorter period of time.

Furthermore, the case study analysis 33 indicated that, at the time of the ex-post evaluation, for 73% of the technical support projects covered in the case studies expected changes had occurred totally or moderately and/or were probable to still occur (22 out of 30). The fact that the change did not fully occur under several projects during their implementation was related to the nature of these projects: most of them were preparatory projects intended to design roadmaps, prepare guidelines, introduce new procedures or develop target values for measurement. Therefore, it is possible that the use of these outputs and some follow-up activities (e.g., new legislation or additional projects) could produce additional policy or institutional changes in the future. 

Factors facilitating or constraining the achievement of expected results and impacts.

Designing and implementing reforms is a complex and long process whose success depends on a combination of external and internal factors and cannot be attributed only to the outputs of technical support projects.

The Study suggests that close cooperation between the European Commission, beneficiary authorities and providers of technical support was an important factor having a moderate or strong positive effect during implementation 34 . This is particularly significant when technical support was provided via public procurement, such as in the case of the of the projects dedicated for the preparation of territorial Just Transition Plans (TJTPs). Also, this cooperation was especially positive in those EU countries with the higher number of technical support projects during the 2017-2020 period. 35  

Positive cooperation was obtained through regular meetings of steering committees comprised of representatives of DG REFORM, beneficiary authorities and providers of technical support, as well as other ‘tripartite meetings’ involving these organisations. During the implementation of technical support projects, DG REFORM policy officers not only executed managerial tasks but were regularly involved in the activities to build a consensus on reforms at the national level.

At the same time, case study interviews suggest that the increasing workload of responsible DG REFORM policy officers and the staff turnover reduced their ability to provide tailor-made and swift expertise and support during the provision of technical support on the ground.

In addition, the Study suggests that consultation with different stakeholders during the reform process was an important factor having a moderate or strong positive effect 36 . The study also indicates 37 the positive effect of partnership among national, regional and local authorities during the reform process. For instance, evidence suggest to the importance of involving a broad range of stakeholders during the implementation of the TJTP projects.

The COVID-19 pandemic had a moderate or strong negative effect on the delivery of technical support, delaying the implementation of some activities or weakening the ownership of some projects because of a lack of in-person meetings and face-to-face interaction. 38  Generally, the providers of technical support maintained high levels of service to the extent possible, also moving activities online, thus providing flexibility and producing additional digital learning resources that could be used in follow-up activities.

Resistance to change at national level was one of the negative factors (having a moderate and strong negative effect) during the implementation of project activities.  39 It also indicated that some public sector employees and other stakeholders opposed the implementation of structural or administrative reforms in different EU countries.

According to the study, the resistance could however be alleviated by consulting and communicating with the key groups of stakeholders throughout the project and the reform process. This supports previous observations that effective consultation and communication with various reform stakeholders can help overcome resistance and increase the long-term sustainability of reforms. 40

The political cycle might also create opportunities for reforms. 41  Having the necessary political backing was identified as a pre-condition for the success of successful projects. Conversely, there were examples of political instability at the national level that had a negative effect on the delivery of technical support and the use of its results. 42  

A lack of domestic reform ownership and commitment was perceived to be a serious hurdle for planning, adopting and implementing complex reforms. 43 Therefore, the CSPs of the SRSP expect that national authorities maintain full ownership of their reform agenda and assume full political ownership of technical support provided by the programme. The supporting study indicates that the strong ownership of the reform in the beneficiary Member States was a positive factor in the delivery of projects’ outputs and results. 44  

It also underlined that since the CSPs were not legal documents, while the contracts for technical support were signed by the Commission and the providers of technical support. Therefore, national authorities were not bound by any contractual obligations regarding their participation in the SRSP-funded projects.

While receiving and flowing up the technical support, evidence indicates that the beneficiary authorities sometimes lacked adequate administrative and financial capacity. There is also some indication that the staff of beneficiary authorities were overburdened with conflicting/concurrent tasks, underestimated the effort required during the implementation of technical support and were thus unable to provide sufficient input or support during the provision of technical support on the ground.

Figure 8: Follow-up actions taken after the project end and building upon the project outputs and results

Source: the targeted survey of beneficiary authorities, May-June 2022 (N=152).

4.1.2 Efficiency: How efficient has the SRSP been in achieving its objectives?

The overall evaluation on the Programme’s efficiency is composed of the following three sub-analyses:

·To what extent was the scale of the SRSP-funded actions proportionate to the request for support submitted by the Member States and to the expected benefits from the reform projects? How cost-effective were the actions funded under the SRSP?

·Time-efficiency: how efficient is the SRSP as regards the process duration from request submission by Member States to the deployment of the technical support on the ground, and to completion of technical support? How time-efficient was the budget execution from commitments to payments?

·The adequacy of management arrangements for the implementation of the programme: To what extent was the governance of the SRSP efficient and how efficient was the cooperation with other Commission Services? 

Proportionality of the allocated resources and the cost-effectiveness of the SRSP

The demand for technical support funding among the Member States and the number of corresponding requests for technical support grew steadily during the 2017-2020 period. This increase in demand meant that the annual selection rate fell from 59% in 2017 to 33%, 45% and 37% in 2018-2020, despite an increase in the SRSP’s annual budget each year.

Figure 9: overall contracted budget by year (in EUR)

Source: internal DG REFORM monitoring system

The overall selection rate of proposals during this period was 42%, which demonstrated a sufficient level of competition and the selection of high-quality proposals. The collected evidence suggested that the quality of requests for technical support grew over this period.

Figure 10: Submitted and selected requests under the 2017-2020 SRSP

Source: DG REFORM based on desk research. Note: this figure does not include the 2020 dedicated call for the preparation of the TJTPs.

The average budget per selected request increased from EUR 142 thousand under the 2017 round to EUR 372 thousand under the 2020 round, which corresponds to an increase of 2.6 times. A higher average budget per selected request resulted in the number of selected requests growing slower compared to the increase in the SRSP annual budget 45 . 

During the first two years of implementing the programme, requests for support were often related to small studies or surveys, while in the following rounds requests for support typically aimed at much broader and more significant reforms.

Figure 11: Average budget per selected request for support (EUR thousand), 2017-2020

Source: DG REFORM based on desk research.

The SRSP tailor-made support requires the close involvement of DG REFORM staff in the management and implementation of technical support projects. Consequently, the number of technical support projects was also limited by the number of corresponding DG REFORM staff who could be involved in the management and implementation of such projects.

The evidence shows that beneficiary authorities and coordinating authorities 46  tended to agree or strongly agree that the budget/financial contribution of the SRSP was proportionate to the needs and estimated scope of technical support, especially requests approved after 2018.

Figure 12: Degree of agreement on the proportionality between budget and scope of technical support

Source: Targeted survey of beneficiary authorities, coordinating authorities and technical support providers, May-June 2022.

Further evidence 47 also indicates that SRSP actions were perceived as user-friendly, timely, and cost-effective to a moderate and high extent compared to other EU actions 48 and national actions by about two thirds of the respondents. 49  

According to the Study, DG REFORM’s overall cost of controls over total annual payments was estimated to be 6.7% of the payments executed in 2018, 4.8% – in 2019 and 4.5% – in 2020 50 . These costs of controls at DG REFORM were comparable with other EU instruments 51 .

The targeted survey of beneficiary authorities and technical support providers shows that effective cooperation with DG REFORM policy officers was one of the key factors related to the successful implementation of the SRSP projects.

Time-efficiency of the SRSP processes and the budget execution

DG REFORM achieved very good results in the timely selection of requests for support, according to the external evaluation report. Specifically: the average time from the application deadline to the financing decision (the adoption of the respective work programme) was around four months during the 2017-2020 period. This was substantially below the applicable time-to-inform target of six months set in Article 194.2(a) of the Financial Regulation 52 . 

As an element of comparison, the timeliness of the selection of requests for support under the SRSP compares favourably with other EU programmes. For example, the ‘time-to-inform’ within programmes managed by the Research Executive Agency in 2017 was 130 days 53 , and the corresponding indicator for the Education, Audiovisual and Culture Executive Agency was 125 days 54 , 55 .

Preparations for the implementation phase (i.e., time taken from the submission of technical support requests to the start of delivering technical support) were carried out in a timely and efficient manner (see the figure below), as evidence supports. In particular, the targeted survey carried out by the external contractor revealed that 87% of beneficiary authorities (101 out of 116) agreed to this statement. This was an improvement compared to the results of the survey carried out during the mid-term evaluation of the SRSP, where the corresponding result for the beneficiary authorities stood at 77% (N=100).

The element to be improved in the preparation of the implementation phase is the contracting process. Some interviewees indicated that the contracting process was the most frequently mentioned negative aspect of the SRSP’s administrative arrangements. In some cases, it took around one year from the conception of the project until the start of project activities; this was considered, since the reform priorities and the general context could have evolved in the meanwhile.

Conversely, the project duration was appropriate for carrying out the activities: 96% of beneficiary authorities (117 out of 122) and 90% of coordinating authorities (18 out of 20) agreed or strongly agreed with this statement. Only some interviewees considered the project implementation phase too short – especially compared to the period preceding the start of project activities – although nearly all projects managed to achieve their outputs in time.

Figure 13: The extent to which the respondents agreed that the following project phases have been implemented in a timely and efficient manner

Source: the targeted survey of beneficiary authorities and coordinating authorities, May-June 2022. Note: ‘BA’ stands for beneficiary authorities, and ‘CA’ – for coordinating authorities.

The targeted survey revealed that 93% of beneficiary authorities (108 out of 116) and 89% of coordinating authorities (17 out of 19) agreed that the implementation of technical support projects 56 was carried out in a timely and efficient manner, in line with the mid-term evaluation.

DG REFORM achieved good results in the execution of SRSP-related commitment appropriations and, starting from 2018, payment appropriations: DG REFORM managed to achieve nearly a full execution of SRSP-related commitment appropriations in 2017-2020. 57  Since the implementation of SRSP-funded projects usually takes more than one year, the amount of SRSP commitments to be settled at the end of 2020 constituted EUR 104.84 million.

Figure 14: Execution of SRSP related commitment and payment appropriations (EUR million), 2017-2020

Source: DG REFORM based on desk research.

The share of payments processed within the set time limits of DG REFORM stood at 79% in 2017 and gradually increased to 96% in 2020 and 98% in 2021. The average number of days taken to make a payment constituted around 30 days in 2018-2021. The targeted survey revealed that 88% of technical support providers agreed or strongly agreed that payments for the provision of technical support were received in due time.

Efficiency of the SRSP governance and administration procedures

The survey programme revealed that the beneficiary authorities, coordinating authorities and technical support providers were generally very positive concerning the clarity, transparency and user-friendliness of the procedures related to the application and selection of requests for funding and monitoring of the projects, including relevant IT tools (Figure 14).

Figure 15: The extent to which the respondents agreed that the SRSP administrative procedures were clear, transparent and user-friendly

Source: the targeted survey of beneficiary authorities, coordinating authorities and technical support providers, May-June 2022. Note: ‘BA’ stands for beneficiary authorities, ‘CA’ – for coordinating authorities and ‘TSP’ – for technical support providers.

Slightly lower scores were observed related to the clarity, transparency and user-friendliness of the evaluation procedures and processes after the completion of projects, based on the feedback mechanism set up by DG REFORM to evaluate the completed projects (see box 4 below).

In their open replies to the survey questions, several respondents from the coordinating authorities and technical support providers stated the need to improve the monitoring and evaluation of technical support outputs, and better use these monitoring/evaluation findings, considering that the projects and reforms are sometimes managed by small staff in the beneficiary institutions, while strong political commitment is not always in place and knowledge can be lost due to factors such as high staff turnover.

The administrative burden related to the application and project implementation processes were regarded as reasonable and proportionate by the beneficiary authorities and coordinating authorities, as revealed by the survey programme, by the analysis of the OPC data, 58 and by the results of the case studies 59  

Interviewees indicated that the administrative burden was comparable or lower compared to the management of other EU programmes and instruments (e.g., representatives of beneficiary authorities compared the implementation of the SRSP to the operational programmes under ESIF 60 ), describing arrangements and procedures as overall rather user-friendly and not too complex in terms of required documentation. 

Beneficiary authorities highlighted as a positive factor that much of the project management and implementation tasks were shifted to the Commission (e.g., in contracting procedures) and technical support providers (in terms of carrying out project activities, such as studies, research, evaluations, etc., and delivery of the project outputs, such as recommendations, analysis and reports, action plans, etc.)

While this limited the administrative burden on beneficiaries and allowed them to dedicate a smaller number of staff to technical support projects, it also to some extent limited their involvement and influence in some project phases (see section 4.1

.3 concerning the insufficient capacities of some administrations to provide support during the implementation of project activities).

Figure 16: The extent to which the respondents agreed that the administrative burden related to the SRSP administrative procedures was reasonable and proportionate

Source: the targeted survey of beneficiary authorities, coordinating authorities and technical support providers, May-June 2022. Note: ‘BA’ stands for beneficiary authorities, ‘CA’ – for coordinating authorities and ‘TSP’ – for technical support providers.

The satisfaction level, with respect to the administrative burden, among the technical support providers was also high although slightly lower than among the beneficiary authorities and coordinating authorities. A total of 84% of technical support providers (84 out of 100) agreed or strongly agreed that the administrative burden was reasonable and proportionate. However, these results varied substantially across different delivery modes, with 94% of support for public procurement and 67% of support for grants.

DG REFORM coordinated efficiently and effectively the technical support to Member States with other Commission DGs and services, as the results of the targeted survey evidence suggests, and actively participated in the European Semester framework for economic and employment policy coordination. 61  The study indicated that in the period 2017-2020 there could have been closer cooperation with other DGs and services of the Commission and disseminate and exploit the outputs of technical support after the completion of the projects. This was done as of mid-2020 onwards, also through the elaboration of flagships reforms in close collaboration with other DGs.

As with any EU programme, the Commission was required to establish reliable financial management and internal control processes. The main aim of such controls was to ensure that the residual error rate did not exceed the materiality threshold of 2% calculated on a cumulative multi-annual basis. Overall, the residual error rate estimated for the SRSP was below the materiality threshold for most parts of the Programme. 62 This indicates that suitable control arrangements were put in place, except for grants to non-pillar-assessed organisations for which a reservation was recorded.

The higher error rate in the non-pillar assessed grants was related primarily to the implementation modes of such grants, including the mechanism of grant reimbursement based on eligible actual costs and the related risk of errors in cost reimbursement claims submitted by respective technical support providers. To address this issue, DG REFORM improved its practices of instructing grant beneficiaries on the eligible actions and costs for final cost claim, reduced thresholds for audit certificates and increased ex-ante control checks on sampled final payments.

4.1.3 Coherence

This section addresses the internal and external coherence of SRSP actions where:

-Internal coherence refers to the extent to which the actions funded under the SRSP are coherent with the objectives as set out in the SRSP Regulation.

-External coherence refers to the extent to which the SRSP was externally coherent with other interventions at regional, national and EU level, such as technical assistance through structural and investment funds.

Internal coherence of the SRSP

No major gaps, inconsistencies or overlaps in the internal structure of SRSP actions were observed. There was a coherent internal logic of the SRSP structure, linking the programme’s general objectives, the specific objectives and the activities that should produce specific outputs and eventually lead to the achievement of reform results.

The programme contributed to a large extent to the implementation of the specific objectives set out in the SRSP Regulation, as indicated in section 4.1.1 above. However, the evaluation evidence shows that the project outputs could be better disseminated and utilised in all thematic areas (based on the example of good practices in the field of governance and public administration) and in communication or consultation activities could be added more frequently to the design of future projects.

The Programme developed a consistent internal project monitoring system, allowing a clear linkage at the level of individual projects between such key elements as: objective of the support; means of delivery, eligible activities, outputs, pre-defined groups of outcomes (results), and expected impact.

A variety of delivery modes ensures that the activities, type of support and expertise provided to beneficiary correspond to the project objectives and needs, as indicated by the evaluation report. The results of stakeholder consultations suggest that the internal structure and design of the SRSP allowed for a wide flexibility when selecting project activities that were best tailored to specific project objectives and beneficiary needs.

Over time the structure and activities of the SRSP were adjusted to better respond to newly emerging challenges and to achieve its objectives in a more effective way, as shown in some instances by the evaluation evidence. For example, a dedicated call was initiated to respond to the specific emerging needs of Member States or the implementation of new EU priorities.

External coherence of the SRSP with other programmes and interventions

Evidence suggests that the SRSP was complementary to support provided through the ESIF. 63 In particular, the targeted survey results show that 72% of national beneficiary authorities (35 out of 49) considered that SRSP to a moderate/high extent complemented the support provided through the ESIF. Similarly, 69% of the beneficiary authorities that offered an opinion on this issue (34 out of 49) considered that the SRSP complemented the interventions enhancing the institutional capacity of public authorities and stakeholders and efficient public administration supported by the ESF and the ERDF.

When it comes to employment policies, the respondents claimed that the SRSP complemented the modernisation of public and private employment services (financed by the ESF) and investments in employment infrastructure by the ERDF only to a limited extent. 64  However, there is mixed evidence regarding the SRSP’s complementarity with national/regional schemes (i.e., the schemes implemented at national/regional level and providing technical support similar to the SRSP) 65 .

Figure 17: Complementarity of the SRSP to other EU funds

Source: the targeted survey of beneficiary authorities, May-June 2022.

The Study shows also that the SRSP had strong complementarities and synergies with other programmes/interventions implemented at the level of Member States or the EU: For example, in addition to several SRSP-funded projects, the curriculum reform in Croatia was supported by an ESF-funded project that provided funding for materials, IT equipment, honoraria for mentors, etc.

The key difference between the ESIF and the SRSP lies in the requirement for co-financing from the Member States, which is present in case of the ESIF and absent in the SRSP. In addition, several interviews with Commission officials (including those from DG REFORM) supported the finding that while thematically the SRSP to some extent overlapped with other EU-level programmes (such as the ESF, Fiscalis or the programme for Competitiveness of Enterprises and Small and Medium-sized Enterprises (COSME)), the SRSP largely complemented these interventions primarily by providing tailor-made technical support for national authorities in a specific sectoral area. 66  

Evidence stemming from interviews and case studies also shows that in some cases, the SRSP had synergies with other EU-level programmes, where the instruments mutually reinforced each other and helped to achieve the reform objectives. More specifically, the technical support projects under SRSP often contributed to the preparation of the strategies and workplans for the reform actions foreseen to be funded under the ESF, ERDF or other funds. One example of such synergies is provided by Romania, where the methodologies and recommendations developed during the SRSP-funded project were incorporated and supported colorectal cancer screening project co-funded by the ESF 2014-2020.

Existing links and cooperation channels between DG REFORM and other Commission services contributed to synergies between different instruments: e.g., Directorate-General for Regional and Urban Policy (DG REGIO) experts cooperated with DG REFORM on a systemic basis by contributing to the design of SRSP-funded projects that have a direct link with the ERDF and Cohesion Fund. This support involved helping authorities draft terms of references, identifying project tasks and participating in project steering committees. In addition, DG REGIO provided DG REFORM with potential priority areas for intervention to be considered during bilateral meetings between DG REFORM and national authorities.

SRSP coherence with the European Semester process and the EU priorities

In general, the Study assessed very positively the contribution of the programme to addressing CSRs issued in the context of the European Semester process. According to the targeted survey results, most beneficiary authorities (61 out of 82, or around 74%) and coordinating authorities (18 out of 20, or 90%) indicated that their project addressed CSRs issued in the context of the European Semester to a moderate or high extent. In comparison, the SRSP mid-term evaluation of 2019 found that only 45% of projects addressed the European Semester recommendations 67 . 

A more in-depth analysis of the targeted survey data also revealed that most of the projects addressing CSRs were in the sectoral area of governance and public administration 68 , and to a somewhat smaller extent, in the area of labour market, education, health and social services.

In terms of the coherence of the SRSP with the EU priorities for 2019-2024:

·66% of beneficiary and coordinating authorities (67 out of 102) considered that their projects contributed to the EU priority of developing a strong and vibrant economic base. 

·58% of them considered that their projects contributed to the EU priority of building a climate-neutral, green, fair and social Europe (54 out of 93). 69  

·38% of them indicated that the projects moderately and highly contributed to the protection of citizens and freedoms (29 out of 77).

In general, the evidence from the interviews with stakeholders strongly shows a clear complementarity and synergies between the European Semester process and the SRSP. Since CSRs are often formulated in very general terms, the SRSP functioned as an instrument to better define, clarify, operationalise and provide specific guidelines for Member States to implement these recommendations. In other instances, the results of the SRSP-funded projects (guidelines, recommendations, etc.) were occasionally taken up and used when defining CSRs.

Evidence from the interviews with stakeholders also suggests that there were sufficient mechanisms to ensure the linkages and integration between the SRSP and the European Semester process/CSRs. For instance, the linkage and contribution to the fulfilment of CSRs was considered during the evaluation of requests for technical support. However, the study indicates that there was insufficient tracking of whether SRSP projects contribute to the fulfilment of specific CSRs.

While the fulfilment of CSRs is a responsibility of Member States and does not depend on the provision of technical support, in some cases CSRs could trigger the technical support under the SRSP and the latter could help Member States to address CSRs. The SRSP would therefore benefit from a more systematic monitoring of how and in which specific ways the technical support provided by the SRSP contributes to the implementation of CSRs in specific national contexts.

In addition, over the study period the European Semester country reports did not consistently provide references to the SRSP technical support and the contribution of this support to the progress that the countries achieved. The introduction of systematic referencing to the SRSP/TSI projects in the respective European Semester country reports should also be considered.

4.2 How did the EU intervention make a difference and to whom?

EU valued added of SRSP compared to what could be achieved at national/regional and local levels

The evidence from the targeted survey of beneficiary and coordinating authorities indicates that 80% of stakeholders (106 out of 132) considered that without the SRSP, beneficiary authorities could not have received similar technical support or could have received it only to a limited extent, be it at national, regional or local level. Similar results were provided during the open public consultation (OPC). 70  

Projects analysed in the case studies also demonstrated that, whilst the project benefits could have been realised without EU technical support, these would have been much smaller in scope and/or realised later in time.

The study also provided some more in-depth evidence on why the results of the SRSP-funded projects could not have been achieved at national/regional level and the added value of SRSP:

-The Programme’s funding provided access to high-level international expertise, perspective and knowledge, which was considered less likely to be available if the project was funded and managed exclusively at the national level.

-Technical support activities would have been unavailable without EU funding, mainly due to limited administrative and financial resources at the national level (this factor was particularly emphasised by the smaller Member States).

-The tailored solutions reflecting the needs of each Member State were offered by the SRSP and were unavailable or too costly at national and regional level.

-The involvement of the Commission in the design and implementation of reforms provided greater institutional legitimacy and political impetus for structural or administrative reforms that were contested or where there was insufficient political momentum to overcome institutional reluctance to change. The legitimising factor of EU-level support was especially relevant in countries overcoming political turbulence or in the case of the reforms focused on issues lacking a full consensus.

-Low administrative burden, clear procedures and the absence of co-financing requirements distinguished the SRSP from other national/regional and EU programmes. This was also identified as a source of the SRSP’s added value and one of the key reasons why Member States could not achieve similar results without the programme. The technical/administrative support provided by the European Commission at the stages of contracting, procurement and evaluation of tenders helped to reduce the administrative burden on beneficiary authorities.

-Based on the results of DG REFORM satisfaction questionnaires, guidance, help and supervision provided during the project design and implementation stages were the most commonly mentioned type of added value identified by beneficiary authorities. 71 The Commission’s guidance and support contributed to the quality of the support received by beneficiaries, compared to what is available at national/regional level.

Cross-border and Union-wide impacts of the SRSP

A large share of SRSP (2017-2020) projects was related to some cross-border/union wide impacts, as shown by the collected data: 720 out of 826 SRSP-funded projects (around 87%) covered in the evaluation study reported at least one such impact, with only 106 (or 13%) of the projects indicating no EU value added/cross-border impacts (see Figure 17).

Figure 18: Share of SRSP-funded projects (2017-2020) indicating specific European added value

Source: DG REFORM internal monitoring system (N=826).

The above SRSP’s cross-border and Union-wide impacts were also confirmed by the targeted survey of coordinating authorities 72 .

Beneficiary authorities not only shared good practices and lessons learned, but also quite actively used them after the end of technical support projects. According to the targeted survey results, 88% of beneficiary authorities (105 out of 119) used to a large or to some extent the good practices and lessons learnt from the specific technical support projects after their end. Some beneficiary authorities participating in the focus groups also reported that they benefitted from opportunities to collaborate and share knowledge between the SRSP-funded projects in the same Member State, particularly where the coordinating authority played a facilitation role (e.g., by organising annual meetings of the SRSP-funded projects).

In some cases, the SRSP-funded projects directly contributed to the implementation of EU priorities and policies, as showcased in Latvia where several different projects facilitated the internal restructuring of institutions and procedures and the implementation of the national reforms related to EU policies. Similarly, Croatia implemented SRSP-funded projects in the area of curriculum reform. This was done within the context of a broader EU reform policy framework in which the Commission provided support to EU Member States to upgrade their education systems in line with the European Education Area’s strategy.

In terms of sharing good practices among Member States, almost all the projects analysed in the case studies included the use of good practices from other Member States. Sharing of good practices either occurred through direct channels of communication amongst themselves developed during SRSP projects or were facilitated by the European Commission and occurred through specific dedicated platforms (e.g., in case of the TJTP projects, a Just Transition Platform was set up as a single access point to provide beneficiaries with opportunities to self-reflect and share good practices 73 ). In some cases, SRSP-funded projects contributed to the sharing of good practices among institutions of the same Member State.

In general, the case study evidence indicates that one of the key factors behind the establishment of cooperation ties among individual Member States was similarities in the specific national reform priorities and similar challenges faced by specific EU Member States.

The number of multi-country projects addressing the reform needs of at least two Member States reflects the cross-border or Union-wide impacts of the SRSP. The available evidence shows that the use of multi-country projects in the SRSP (2017-2020) was limited; 74 the SRSP did not provide explicitly the possibility of submitting multi-country requests from different Member States, this was rather a limiting factor in the programme’s practice. The results of both desk research and stakeholder consultations show a need to enhance the multi-country dimension and call for a more active role of the Commission in facilitating multi-country projects and sharing technical support outputs across borders.

Multi-country projects represent only one type of multi-beneficiary projects supported under the SRSP 2017-2020. The programme also provided the opportunity to form projects involving multiple beneficiary authorities from the same country (e.g., different ministries), as well as multi-regional projects (e.g., projects involving regional institutions from the same country or regional institutions across different countries). The SRSP thereby provided the framework not just for cross-border cooperation and good practice exchange, but also for cross-regional cooperation.

Some stakeholders suggested specific measures for a more active role of the Commission in facilitating collaboration between potential beneficiary authorities in different Member States with similar needs and priorities; for example by facilitating the networking and exchange of information between Member States about their common challenges, reform needs and good practice examples, by using the newly created coordinating authorities’ network, which could facilitate more direct exchange of experience and knowledge about common challenges between Member States.

These actions could create more favourable conditions for Member States to identify synergies and complementarities, leading to the set-up of multi-country technical support projects while fully respecting the demand driven SRSP approach. Cooperation between the Member States outside the programme’s framework could also be facilitated by further encouraging the establishment of communities of practice involving experts in interested Member States in specific areas, including after project completion.

The results and impacts of multi-country projects should also be closely monitored by the programme in parallel to the above efforts. The impacts of the multi-country projects (particularly, as compared to single-country projects) were not systemically assessed yet, as indicated by the evaluation report.

4.3 Is the intervention still relevant?

This section addresses the relevance of SRSP actions for enhancing the administrative and institutional capacity of Member States to design and implement the reforms needed to tackle the challenges faced; as well as the relevance of the actions funded under the SRSP to address the needs of beneficiary authorities and Member States.

Relevance of SRSP actions for enhancing the administrative capacity of Member States

The targeted survey results indicate that at the moment of requesting technical support beneficiary authorities faced the challenges that were also set out in the ex-ante evaluation of the SRSP: limited administrative and institutional capacity, challenges in the design and implementation of structural reforms in line with the Union’s economic and social goals and challenges in the application and implementation of Union law (see figure 18).

The incidence of these key challenges faced by the beneficiary authorities generally remained similar to the results of the mid-term evaluation.

Figure 19: Challenges faced by beneficiary authorities and addressed by technical support

Source: the targeted survey of beneficiary authorities, coordinating authorities and technical support providers, May-June 2022.

All stakeholders viewed very positively the relevance of the SRSP’s design and structure for achieving its goals, according to the targeted survey. 75  

94% of surveyed beneficiary authorities and coordinating authorities (126 out of 134) also agreed/strongly agreed that the objectives of technical support projects corresponded to the key reform goals of their country. There were no significant differences between different groups of stakeholders in terms of their views on the relevance of the SRSP’s design and structure. This highly positive assessment of the programme’s structure and design was very similar to the situation found during the mid-term evaluation of 2019, when 97% of survey respondents agreed that the SRSP is a suitable instrument to provide technical support.

Similarly, data from the satisfaction questionnaires of DG REFORM show that the beneficiary authorities very positively assessed the DG REFORM’s help in designing projects adequately to address their reform needs (identification of the problem, definition of the objectives and results to be achieved, definition of the activities to be carried out).

DG REFORM policy officers also positively assessed their own contribution in terms of designing projects in a manner that could adequately address the needs of the Member State concerned (average assessment score 8.6; N=382).

The relevance of project design for addressing the needs of the Member States depended on the quality of technical support requests submitted by the Member State, as suggested by the evaluation report (analysis based on the SRSP 2017-2020 monitoring data): the relevance was higher for projects where technical support requests were more mature to allow for speedier delivery of the support, as well as for the projects where measures requested by the Member States were targeted enough for addressing the reform needs. 76  

Looking at the relevance of the SRSP for addressing the specific types of reforms needed by the Member States, the analysis of the circumstances of the selected requests (see Table 2) indicated that the requests for technical support were mostly relevant to address the economic governance process, including CSRs, EU priorities and reforms initiated by Member States.

According to the results of the exploratory interviews, this achievement was made possible through the closer alignment of the SRSP with the European Semester process and by considering the technical support requests related to CSRs during the selection process. However, according to these interviews, there has been no assessment on how many CSRs have been addressed by Member States due to the provision of technical support.

Relevance of SRSP actions to address the needs of beneficiary authorities and Member States

All groups of stakeholders involved in SRSP-funded projects very positively assess the relevance of technical support to the needs of Member States. 96% of surveyed beneficiary authorities and coordinating authorities (145 out of 150) agreed or strongly agreed with the statement that their project addressed the key reform needs of the beneficiary institution. There were no significant variations in the distribution of answers among these groups of stakeholders.

Since the European Commission acted as a contracting authority during the implementation of the SRSP, it was responsible for the selection of technical support providers. However, in their requests for technical support, BAs were able to provide suggestions on the delivery mode and the technical support provider. 60% of surveyed BAs (78 out of 130) indicated that such suggestions were accepted to a large extent or to a moderate extent, with only 5 beneficiaries out of 130 (6% of surveyed beneficiary institutions) indicating that such suggestions were not considered at all. Nevertheless, some respondents highlighted the need for a closer involvement of beneficiary authorities during the selection of technical support providers.

Different sources of evidence also clearly indicate that the technical support provided by the SRSP remained highly relevant for the needs of both EU Member States and their beneficiary authorities. The evidence 77 on the increasing number of infringement cases demonstrated that technical support aiming at a more effective application and implementation of Union legislation could still be needed. However, since individual EU Member States did not initiate many technical support requests in this field during the implementation of the demand driven SRSP, the number of technical support projects addressing specific challenges in the application and implementation of EU law remained rather low.

Also, some deterioration in the annual implementation rate of CSRs during the period 2015-2019 indicates the continued relevance of the technical support aimed at addressing the challenges identified in the context of the economic governance processes. Although the increasing number of technical support projects addressed these challenges (e.g., 62% of such projects in 2019), such changes were expected to arise later due to the long logical chain between the activities of technical support and the expected results/impacts of the programme.

The growing need for the technical support provided by SRSP was showcased by the increasing number of requests from the Member States, as well as by the increasing budget of the projects, according to the evidence analysed in the evaluation report. These trends indicated a growing interest from the Member States, as well as the Programme’s lasting relevance for designing and implementing significant reforms.

The demand for technical support varied among different Member States, with the highest overall numbers of SRSP-funded projects found in some of the Southern (Greece), as well as Central and Eastern European countries (Romania and Croatia). Fewer projects were supported in the Nordic countries, Luxembourg, Germany, and the Netherlands.

Despite the smaller number of SRSP-funded projects, qualitative evidence shows that the support provided in these countries was highly relevant for their national reform needs: e.g., in the case of Luxembourg, technical support directly contributed to the modernisation of the national customs system.

Evidence from the interviews with stakeholders shows that one of the key reasons for a limited number of requests for technical support in some Member States was the timing of evaluation and selection procedures in the programme, which did not always match with the timing of the national reform agenda. Another reason for the limited number of requests in some Member States was a lack of awareness of the SRSP’s benefits and functioning among some of potential beneficiary institutions.

The interviews and focus group discussions generally confirmed the high relevance of technical support to the needs of Member States and beneficiary institutions. Some respondents, however, noted that the increased focus of the SRSP on larger projects limited access to the programme for smaller beneficiary authorities.

The case study evidence also supports the finding that the SRSP actions were highly relevant for the needs of beneficiary authorities, with a large majority of the projects analysed under the case study programme directly addressing the needs of the Member States beneficiary authorities. Overall, most of the projects covered under the case studies were highly relevant for addressing the most important national issues 78 . For example, all the SRSP-funded projects analysed in the Italian case study focused on important national reform priorities and structural challenges. These projects were highly relevant and specifically designed to address the needs of Italian beneficiary authorities. 79  

Finally, qualitative evidence also indicates that SRSP is perceived as highly relevant in the context of European-wide challenges and trends that will likely define the European policy agenda in the future. Overarching challenges – such as climate change, environmental degradation, digital hyperconnectivity, and technological transformation – pose a need for a closer strategic cooperation between the Member States, greater exchange of good practices and implementation of structural reforms (e.g., in the area of reducing dependencies on fossil fuels). In this context, the SRSP and its successor programme will sustain its high relevance for the Member States and the whole of the EU.

5.What are the conclusions and lessons learned?

5.1Conclusions

The Structural Reform Support Programme was adopted in 2017 to provide technical support to national authorities and help Member States design and implement institutional, administrative and growth-sustaining structural reforms.

Overall, the implementation of the 2017-20 SRSP was successful in terms of effectiveness, efficiency and coherence. The programme was also found to be highly relevant in addressing the needs of beneficiary authorities and EU Member States and had an EU added value (value above what the Member States could have achieved acting unilaterally).

Effectiveness

The programme successfully achieved its general objective, namely, to contribute to institutional, administrative and growth-sustaining structural reforms in the EU Member States. Furthermore, it has substantially advanced the reform agenda in EU Member States. The actual achievement of different types of reforms also depended on the wider efforts made by EU Member States and not only on the results of other EU programmes and schemes.

The SRSP contributed most to achieving its specific objectives concerning “the development and implementation of reform policies”, as well as “the definition and implementation of processes and methodologies”, followed by the specific objective concerning “the design of reforms according to the priorities of national authorities”.

Also, the programme fully achieved its expected results; 80 almost all programme stakeholders who participated in the consultation activities were highly satisfied with the results and the success of the technical support projects. For instance, according to the targeted survey of beneficiary authorities, 94% of respondents were very satisfied or satisfied overall with their participation in the SRSP and with the results of technical support projects.

The programme contributed substantially to the design and implementation of administrative, institutional and structural reforms in EU Member States, with potential benefits for European businesses and citizens/consumers.

Where the projects have not generated the full intended results and impacts, this tends to reflect wider challenges in the reform process rather than any inherent weaknesses in the projects or in the provision of technical support. And more policy impacts were expected to occur in future because some one third of all evaluated projects were still ongoing during the ex-post evaluation and more time was necessary for the expected results and impacts to occur, due to the long lag between the technical support activities and the expected results/impacts.

The SRSP contributed more to administrative reforms and capacity building than to the implementation of other structural reforms. In fact, administrative reforms are easier to implement: they require fewer actors (who usually come from the public sector), less time to be implemented and bring more direct results.

Other types of structural reform are more complex in nature, require the involvement of more stakeholders (including the private sector) and take more time for their results to bear fruits. In addition, these types of reforms are more dependent on external factors. Member State beneficiary authorities did not initiate many technical support projects aimed at advancing such reforms, especially in the initial rounds of the SRSP (2017 and 2018).

Overall, communication activities were rarely included in project design, despite an increase in the number of communication initiatives in the 2019 and 2020 rounds of the programme. However, engagement and communication with external stakeholders – when they happened – were crucial during the provision of technical support, because the effective implementation of such activities increased the legitimacy of the reforms and their quality and sustainability, and reduced organisational resistance to change.

Overall, the growing complexity of technical support projects (in terms of number of activities and outputs) might make the implementation of the projects and the achievement of their expected results more difficult in future. This risk should be taken into account when designing any future multi-country projects, which are by their nature more complex than single-country projects. 81

Almost all consulted stakeholders were highly satisfied with the quality of technical support provided on the ground. The choice of providers was crucial to the success of technical support projects due to the importance of having adequate professional skills and a good understanding of the national situation. This reflects the added value of the European Commission in choosing and contracting the best expertise on the market for each specific project.

The strong involvement of the beneficiary authorities and senior management ownership of the projects is key to achieving the results. Their involvement is essential not only during project implementation, but also in the pre-implementation phase, in terms of defining the project objectives and timelines and creating ownership and commitment to change.

While absorbing technical support and implementing the recommendations made, beneficiary authorities sometimes lacked adequate administrative capacity. This hampered the smooth running of the technical support projects and the implementation of resulting recommendations. While the provision of technical support does not involve any costs for the beneficiary authorities, they sometimes lacked the sufficient financial resources to implement project recommendations and relevant reforms.

Efficiency

The SRSP proved successful in attracting many high-quality requests relative to the budget available. Moreover, the programme was able to support projects of increasing size and budget, thus increasing the potential for projects to achieve greater impact. Growing demand for technical support among the EU Member States and a sufficient level of competition among the projects allowed high-quality proposals to be selected. 

The programme’s stakeholders – including beneficiary authorities, coordinating authorities, and providers – generally agreed that the financial contribution of the SRSP was proportionate to the needs and estimated scope of technical support, and they considered SRSP projects to be a user-friendly and cost-effective instrument. The costs of controls on total annual payments by DG REFORM were comparable with other EU programmes, such as Horizon 2020, under direct management mode. DG REFORM achieved good results in executing the SRSP budget.

Most notably, the assessment and selection of requests for technical support was completed on time and in line with the target set in the Financial Regulation, compared with other EU programmes. However, there is room for improvement in speeding up the preparation for the implementation phase, in particular the contracting process. As one of the selection criteria for funding a project under the SRSP was the urgency of the reform, Member States expect the project to be operational as early as possible to ensure the reforms remained relevant and were implemented on time.

The stakeholders involved in the programme were generally positive about the clarity, transparency and user-friendliness of the SRSP’s administrative procedures and the proportionality of the administrative burden. Areas for improvement are related to the need to further develop and refine project monitoring and evaluation practices, as well as communication activities.

DG REFORM efficiently and effectively coordinated the technical support given to the Member States with other Commission departments. According to the outcome of the study, in 2017-20, DG REFORM cooperated more closely with other DGs and Commission departments while disseminating and using the outputs from the technical support once projects had been completed.

Coherence

The SRSP is evaluated to be a coherent policy tool with a clear internal logic, well aligned with the objectives set out in the SRSP Regulation, with no major gaps, inconsistencies or overlaps in the internal structure of projects. The design of the programme allowed for considerable flexibility when selecting project activities that are best tailored to specific project objectives and beneficiary needs.

The structure and activities of the SRSP were adjusted over time to better respond to newly emerging challenges and to achieve its objectives in a more effective way, e.g., by introducing dedicated calls for requests that met the newly emerging needs of Member States. However, the discrete nature of the projects means that additional effort could be required to disseminate and exploit project outputs to wider audiences, so that greater impact can be generated.

Evidence also shows that the SRSP was highly consistent with other EU policies and actions (including ESF, ERDF, COSME, Fiscalis), as well as those of the Member States themselves. This was because it addressed different aspects of similar objectives, issues and target groups. Having common themes with other EU-level programmes, the SRSP-funded projects largely complemented these interventions by providing tailor-made technical support for national authorities in specific sectoral areas.

In terms of consistent objectives, the SRSP reinforced wider reform agendas of the Members States and the EU, including by helping address CSRs made as part of the European Semester process. Generally, the stakeholders of the SRSP highly valued the programme’s contribution to the Semester process. In some cases, the SRSP-funded projects helped better define, clarify and operationalise the CSRs, as well as providing specific guidelines for Member States to implement them. In other instances, the results of SRSP-funded projects were used to formulate these recommendations.

Although overall there were sufficient mechanisms to ensure the linkages and integration between the SRSP and the European Semester process/CSRs, the study showed that there is insufficient monitoring on how the SRSP-supported projects contribute to meeting the CSRs. There was also a lack of references to SRSP projects in the European Semester country reports.

EU added value 

The SRSP delivered this by providing Member States with better access to international expertise than they would have enjoyed in the absence of the SRSP. Even in the limited cases when project benefits could have been realised without EU technical support, these would have been much smaller in scope or realised later in time. As a result, EU added value was offered in terms of the strengthened reform processes in the Member States and strengthened contribution to achieving EU policy objectives, not least those of the European Semester.

The SRSP had significant cross-border, EU-wide impacts going beyond single Member States. Such impacts included helping the Member States to implement EU policies and priorities nationally, including the promotion of European values, as well as helping them to share good practice among themselves. The good practices and lessons learned were actively used by the beneficiaries after the end of the projects.

Relevance 

The SRSP was found to be a highly relevant instrument, well-tailored to improving the administrative capacities of Member States to design and implement the reforms needed to address the challenges they faced.

The beneficiary authorities and other stakeholders viewed positively the different aspects of the SRSP’s design and structure and its relevance for achieving the reform goals. At the same time, the objectives of technical support projects largely corresponded to the key reform goals and challenges faced by the Member States. The relevance of project design for achieving the reforms goals depended on the quality of technical support requests submitted by the Member States.

The technical support provided by the SRSP was highly relevant to addressing the needs of beneficiary authorities and Member States, both in terms of the implementation method (i.e. international technical support providers sourced by the Commission) and the nature of the support provided (i.e. being customised to the needs of each beneficiary authority). The programme – uniquely and in an effective and efficient manner – fulfils the need of Member States to access high-level international expertise when they are in the process of designing or implementing reforms.

At the same time, where the SRSP was used in an emergency capacity, it was not very suitable for coping with reform or capacity-building needs on the ground, since it was restricted by the timetable laid down in the calls for requests. Some stakeholders suggested that the programme introduce a possibility to draw on a pre-selected pool of experts to address urgent technical support needs in Member States.

Despite having received the SRSP’s contribution over many years, beneficiary authorities still need the technical support provided by the programme. The growing need for technical support is also showcased by the increasing number of requests from Member States, as well as by the increasing average budget for the projects. Furthermore, the number of Member States who requested support increased steadily during the programme: from 16 in 2017 to 23 in 2018, 26 in 2019 and eventually all 27 in 2020.

5.2Lessons learned

Following the ex-post evaluation, the main lessons learned include:

Engagement of stakeholders

·Active involvement of external stakeholders was crucial during the provision of technical support: it increased the legitimacy, quality and sustainability of the reforms, and reduced organisational resistance to change. Key stakeholders were consulted during the design and execution of successful projects. Effective communication strategies were used to reach many target groups during implementation.

·Engagement of beneficiary authorities was key in implementing technical support projects and in the reform process, to enable the smoother implementation of projects and their results.

·Close involvement of national coordinating authorities was important in monitoring the implementation of the technical support projects, discussing their progress with the Commission and national stakeholders and taking follow-up actions to use the outputs after project completion.

Cross-country dimension

·Providing access to international expertise for Member States that might not otherwise have had access to it was a key feature of the programme. Future programmes must retain this possibility.

·Exchanges of civil servants provided a basis for greater use of peer-to-peer advice within national administrations in future. In fact, the accumulated knowledge of civil servants in the Member States represented an enormous pool of expertise and skills.

·Disseminating the results of technical support projects within and across Member States encouraged the sharing and replication of good practice. DG REFORM was able to play a more active role in clustering similar projects or creating communities of practice and disseminating results effectively to the main relevant stakeholders.

·Transnational dimension of the programme – for example through more projects that support several Member States and through better dissemination of results to wider audiences – was instrumental in strengthening the ‘EU added value’.

Facilitator role of the European Commission

·Reducing the complexity of technical support projects (in terms of number of activities, outputs and outcome) made their implementation easier and more effective.

·The capacity to provide tailored solutions resulted in low administrative burdens for beneficiary authorities and ongoing guidance for them, and helped facilitate and legitimise the reforms.

·The Commission played a valuable role in facilitating collaboration between potential beneficiary authorities in different Member States who had similar needs and priorities – including though joint requests for multi-country projects. None of this interfered with the programme’s demand-driven logic.

Effective implementation of reforms and coordination with other EU processes

·Improved monitoring and evaluation contributed to a more efficient implementation of the programme and a better dissemination and exploitation of technical support outputs.

·When in place, the monitoring of the SRSP’s contribution to the European Semester process and to the fulfilment of specific CSRs in specific national contexts was significant. However, the provision of technical support by DG REFORM should not replace the Member States’ responsibility to address their CSRs.

·Technical support was crucial also in areas where the subject of the request is less frequent, such as assistance in applying and implementing EU legislation. 

Technical Support Instrument (TSI)

In 2021 the Technical Support Instrument replaced the SRSP as main programme for delivering technical support in 2021–27. The TSI was designed on the basis of several recommendations proposed in the mid-term evaluation of the SRSP and factoring in some of the conclusions of this ex-post evaluation, such as more active involvement of stakeholders, a tighter focus on the cross-country dimension and increased attention on monitoring and evaluation.

Annex I. Procedural information

1.Lead DG, Decide Planning/CWP references

Directorate-General for Structural Reform Support.

Decide planning reference: PLAN/2020/9192.

2.Exceptions to the better regulation guidelines

N/A

3.Organisation and timing

The initiation of the ex-post evaluation of the SRSP was approved by DG REFORM senior management and the Cabinet and the Inter-service Steering Group (ISSG) members were appointed. The evaluation roadmap was published on 17 December 2020, marking the official starting date of the evaluation. The procurement procedure for the external evaluation was initiated on 30 June 2021 and the contract with the external contractor was subsequently signed on 13 October 2021, for a duration of eleven months. The public consultation through the website of the European Commission took place between March and May 2022. After the contractor submitted the final evaluation report, the Commission started drafting the staff working document, based on the external evaluation and the analysis of its own reporting and procedural documents.

DG REFORM chaired the ISSG, which comprised of representatives of the Secretary General, REGIO, DIGIT, CNECT, EAC, HOME, ENV, GROW, SANTE, MOVE, AGRI, EMPL, TAXUD, JUST, FISMA, NEAR, ECFIN, ENER, COMP, ESTAT, RTD, HR, JRC and OLAF. The ISSG held four meetings.

4.Consultation of the RSB (if applicable)

N/A

5.Evidence, sources and quality – including external expertise

An external evaluation report served as a basis for the ex-post evaluation of the SRSP.

The contractor used a variety of research methods and sources, including: desk research, data extracts of the Commission internal IT system, scoping interviews, targeted interviews, online targeted consultation and case studies. In line with the Better Regulation guidelines, the evaluation also included a public consultation exercise.

DG REFORM (then SRSS) internal IT system was not in place at the start of the SRSP, but it was implemented only afterwards. The system was evolving significantly and many questions / categories of data were added only at later stages and not necessarily retroactively completed. Therefore, information was to some extent incomplete. This deficiency was overcome by triangulating the information based on other data sources, including the interviews, online consultation and case studies. Furthermore, the response rate to the online-targeted consultation was sufficient and ensured a balanced representation of stakeholders.

Additionally, the breadth and diversity of the Programme’s activities posed an important methodological challenge, in that it would not have been possible within the available resources and timeframe to cover all activities in the detail needed to draw robust conclusions. Moreover, much of the Programme’s support plays a contributing role alongside other factors, such as the action of national administrations. This contributing role is difficult to assess without an in-depth qualitative research.

Lastly, given that many of the evaluated projects were still ongoing at the time of the exercise, results and impact could only be measured to a limited extent. For projects that were still on the ground, the likelihood of achieving the results was assessed only to the extent that information was available. The case studies confirmed that it is too early to expect considerable long-term impacts at this stage of the Programme.

Despite the above-mentioned limitations, the quality and the representativeness of the collected evidence is considered satisfactory. The contractor was able to collect extensive and meaningful data that allowed drawing conclusions.

Annex II. Methodology and analytical models used

Evaluation framework

In line with the Better Regulation Guidelines, the ex-post evaluation provided the relevant evidence to assess the SRSP’s performance against the five criteria: effectiveness, efficiency, relevance, coherence and EU added value. The evaluation relied on quantitative and qualitative indicators. Measuring the first group of indicators was informed by the analysis of data from the internal monitoring system of DG REFORM, the satisfaction and outcome questionnaires, as well as responses to the targeted survey and the Open Public Consultation (OPC). In these cases, the judgement criterion for success was more than two thirds of technical support projects or responses supporting the expected outcome. Measuring the second group of indicators was informed by the analysis of literature and documents, results of the focus group discussions, interviews and case studies. This share of evidence was mostly used to provide specific examples, indicate good practices and lessons learned as well as to explain mechanisms of change and the impact of external conditions on the implementation of the SRSP-funded projects. In addition, the structured expert assessment of each project under the case study programme was employed with more than 50% of the projects supporting the expected outcome as the main judgement criterion for success. The evaluation matrix, which includes the evaluation questions pertaining to each of the evaluation criteria, the associated indicators/judgement criteria as well as methods and data sources used for their evaluation, is presented in Annex 2: Evaluation framework.

The adopted evaluation framework relied substantially on the theory of change presented in Annex 1: Theory of change of the SRSP. The theory of change was developed on the basis of desk research (especially the intervention logic of the programme’s mid-term evaluation, but also taking into account the results of recent academic research) and was validated during the implementation of stakeholder consultations. The vertical logic of the programme – a causal chain from policy/programme objectives to impacts – remained very similar to the one used during the mid-term evaluation, but our theory of change added external factors that might affect the performance of the programme (especially its expected results and impacts), considering the ex-post nature of this evaluation and the requirements of the Technical Specifications. 82

Furthermore, the ex-post evaluation referred to different types of results (e.g. those occurring at the individual level, organisation level and policy level) or impacts (in terms of changes in the institutional and administrative capacity of Member States’ authorities or achievement of relevant longer-term growth-sustaining reforms in EU Member States) (see Annex 1: Theory of change of the SRSP for more information). When possible, the ex-post evaluation used the results found in the prior ex-ante and mid-term evaluations as the points of comparison (see section 1.2 for more information).

Data collection and analysis

A complex methodology was deployed to collect solid evidence and provide well-informed answers to the evaluation questions. It consisted of extensive desk research, stakeholder consultation activities, in particular targeted survey, focus group discussions, a wide-ranging interview programme and the OPC, as well as in-depth case studies. To ensure the validity and reliability of evaluation findings, the evaluation relied on a combination of distinct methods and triangulation of different data sources, which was achieved by complementing perceptions of stakeholders (e.g., interview results) with objective data (e.g., monitoring information). Data-based, documentary and perception-based sources as well as quantitative and qualitative techniques were synthesised, depending on the evaluation question and the respective strengths of data and methods applied.

Desk research and literature review

The desk research exercise consisted of two key elements: (i) a literature review of publicly available documents and (ii) an analysis of internal statistical and monitoring data received from DG REFORM. The entire desk research programme was supported by a computer assisted qualitative data analysis (content analysis using NVivo 12 software), which helped to structure, manage and analyse the collected information as well as to make connections between different information sources and draw summarising conclusions.

The literature review served as a source of contextual information for the assessment of the study questions and for benchmarking and comparing the progress of the SRSP over time. It considered four key categories of publicly available sources of information:

-Documents directly related to the SRSP and its implementation process: the SRSP Regulation, annual work programmes and annexes; annual monitoring reports; the ex-ante evaluation report and the mid-term evaluation report; and the project-specific documents of the programme (e.g. descriptions of the projects, requests for services, deliverables of the projects, if available);

-National-level documents relevant for the analysis of the SRSP and its projects: national-level reform strategies and programmes, assessments/studies/reports on the implementation of the SRSP in individual Member States or the publicly available documents of beneficiary authorities;

-EU-level strategic, policy and legal documents relevant for contextualising the SRSP, including the documents on EU priorities and the European Semester process;

-Relevant academic research.

The documents focusing directly on the implementation of the SRSP (both previous mid-term evaluation and national level documents on the SRSP’s implementation) served as a basis for the descriptive analysis, were used for country case studies and as one of the sources for triangulation of evidence to answer some of the study questions. EU and national-level strategic documents, policy documents and legal acts were of key importance in contextualising the SRSP implementation (e.g. identifying the relevance of SRSP for national strategic reform priorities, coherence of SRSP with other EU funding programmes and their objectives). In addition, these documents contributed to the evaluation of the role of technical support to Member States achievement of the relevant long-term growth-sustaining reforms as indicated in the CSPs. Finally, recent academic research served as a source of evidence for assessing the SRSP’s implementation and identifying the relevant external factors that affect the success of structural reforms’ implementation.

During the evaluation, the best use of monitoring data already collected by DG REFORM was ensured (i.e. administrative data, satisfaction and outcome questionnaires filled in by the project stakeholders). A great deal of internal monitoring/administrative data sources were employed for the evaluation purposes, including:

-Data from the internal monitoring system for monitoring programme implementation. Initial descriptive statistics tables on the SRSP projects (2017-2020), disaggregated by such dimensions as lead unit, funding source, country, status, topic, eligible activities, delivery modes and expected outcome were prepared by the evaluation team, building on two internal monitoring datasets from JIRA. 83 The aggregated data facilitated the estimation of the state of play in the implementation of the programme during the evaluation period (see section 2. State of play in implementing the programme during 2017-2020 above).

-Ex-ante and mid-term evaluations. The evaluation drew on the results of the ex-ante evaluation (published in November 2015) and the mid-term evaluation 84 (published in 2020 using monitoring data from 2019) as points of comparison.

-Feedback mechanism: satisfaction and outcome questionnaires. The results of satisfaction questionnaires, filled in by the Member States beneficiary authorities, two types of technical support providers (providers of procurement and providers of grants) and policy officers of DG REFORM as well as outcome questionnaires 85 completed by Member States’ beneficiary authorities (January 2018 – August 2022) were employed to triangulate data on some of the evaluation questions. The questionnaire data consists of both qualitative responses and numeric scores, ranging from 1 (standing for “Completely disagree”) to 10 (standing for “Strongly agree”). The total number of responses to the satisfaction questionnaire was 1,000 (with the feedback provided by DG REFORM policy officers on 382 projects, by beneficiary authorities on 309 projects, and by technical support providers on 309 projects), while the total number of responses to the outcome questionnaire was 128. It was not possible to estimate specific response rates to these questionnaires because the evaluator did not have data on the total number of invitations sent out by DG REFORM.

-Data relevant for the cost-effectiveness analysis. Data regarding the SRSP’s budget, the number of DG REFORM staff involved in the management of the programme and contracting procedures were provided as inputs into our cost-effectiveness analysis.

Stakeholder consultation

To gather the views of key stakeholders and the data needed to inform responses to the evaluation questions, the stakeholder consultation combined a number of techniques for data collection and analysis (see Table 4). Given the wide scope of the SRSP, the consultation included different categories of stakeholders, including European Commission officials, Member State beneficiary and coordinating authorities, providers of technical support, representatives of industries, businesses, workers’ and other organisations, other public authorities, research centres and consultancies, research institutions as well as the general public. In addition, to grasp the varying experiences across the EU, each Member State that received technical support under the relevant SRSP annual cycles was represented in the consultation activities. Overall, the stakeholder consultation strategy enabled to effectively address a relevant breadth of stakeholders within varying geographical coverage.

Source: Independent evaluation study.

The targeted survey was launched in mid-May 2022 and was open for almost a month, which resulted in a reliable basis of 260 responses (compared to 185 in the mid-term evaluation). The survey design and questionnaires were customised to the expertise of different stakeholder categories, in particular the national coordinating authorities (22 responses), beneficiary authorities (132) and technical support providers (106). Taken together, representatives from all Member States expressed their opinions in the targeted survey. Both open-ended and closed survey questions were used to ensure comparability across groups of respondents and complementarity with the results of the mid-term evaluation, the satisfaction and outcome questionnaires, as well as the OPC.

In line with the Better Regulation Guidelines, the OPC was launched on 1 March 2022 and was open for 12 weeks. The total number of responses to the OPC was 26, which more than doubled the amount of responses collected in the OPC conducted for the mid-term report (11). The OPC attracted stakeholders from 16 Member States, including a large share of representatives of public authorities, as well as research institutions, business organisations and EU citizens. However, the results of the OPC are not statistically representative and, therefore, were only used to triangulate information collected during the evaluation process. A synopsis report summarising the consultation process and outcomes was published on the European Commission’s dedicated ‘Have your say’ website. 87

The design of the evaluation methodology allowed for continuous cross-checking and verification of findings. This was mostly achieved through the implementation of the interview programme, which was divided in four parts: exploratory interviews with the officials of the European Commission (17), exploratory and follow-up interviews with the national authorities (8), case study interviews (83) and validation interviews (2). The interview programme was designed to capture a wide variety of views, including those coming from the services of the European Commission, technical support providers and the national coordinating or beneficiary authorities. Depending on the question at hand, semi-structured interviews were used either to supplement other sources of evidence or to gain novel insights and validate findings in cases when other data sources were scarce.

Finally, two focus group discussions were implemented to get more detailed insights on the management and implementation of the SRSP and, in particular, the dedicated call for the preparation of the Territorial Just Transition Plans (TJTPs). Both taking place at the end of June 2022, the focus group discussions provided a rich set of insights from 12 Member States. Similarly to the interview programme, this information was used for validation and explanatory purposes and led to the collection of findings from the perspective of individual Member States.

In close cooperation with DG REFORM, the design of stakeholder consultation was slightly adjusted during the ex-post evaluation to ensure that a solid evidence base is created in the most efficient way, without overburdening participants. This resulted in a few changes of the interview programme and focus group discussions. First, we only interviewed the representatives of those national coordinating authorities who were not involved in the focus group discussions and/or represented the Member State that had lower numbers of requests for technical support. While maintaining the geographical scope of qualitative evidence, this decision reduced the number of national-level interviews. Second, the execution of validation interviews was combined with the extensive feedback from the Inter-service Steering Group on the Final Report to ensure a reliable verification of evaluation results, preliminary conclusions and lessons learned. Third, the initial selection criteria of participants of focus group discussions were modified to ensure the involvement of representatives from Member States that were not covered by the case studies and/or less represented in stakeholder consultation activities. To increase their participation rates, it was decided to organise a single session per focus group.

As per the Better Regulation Guidelines, the evaluation refers both to the percentage and the number of selections of a particular OPC or targeted survey response out of total responses to a particular question to which this percentage correspond. 88 While presenting results of the quantitative consultation activities, N indicates the total number of responses to a particular OPC or targeted survey question. In addition, N refers to the sample of responses to the question of satisfaction and outcome questionnaires at hand.

More in-depth methodological information and results of the stakeholder consultation activities are presented in Annex 3: Synopsis of the consultation activities.

Case studies and comparative analysis

The case study method was deployed to get an in-depth analysis of how technical support was provided in specific EU Member States. In addition, it provided an opportunity to better assess the medium- and long-term effects of SRSP-funded actions, the causal links between the actions and results, and the factors that have driven or hindered the achievement of the expected results and impact. A total of 8 country case studies (including Ireland, Cyprus, Finland, Croatia, Italy, Latvia, Romania, Czechia) and 1 horizontal case study of projects funded through the dedicated call for the preparation of TJTPs were selected for the case study programme (see Table 16 in Annex 4). The selection of countries for the case study programme integrated the feedback provided by DG REFORM while ensuring a representative sample, well-balanced in terms of geographical distribution and the thematic fields of technical support. Taken together, the nine case studies assessed the design and execution of 47 technical support projects.

Each of the case studies followed the structure of the case study template, including the background information on the SRSP-funded projects, information on the reform context, a description of the design and execution of SRSP-supported projects (based on the five evaluation criteria), conclusions and a standardized assessment of each project. Taken together, this approach ensured comparability of the case studies and facilitated the comparative analysis. The latter was built on two types of analysis stemming from the case studies: (i) a structured assessment of each project by the country expert, based on pre-defined assessment scales; (ii) a qualitative content analysis of the text of each case study, based on the key themes as per evaluation criteria. As a result, the cross-case analysis enabled the comparison of the SRSP-funded projects, their effectiveness, efficiency, relevance, coherence, EU added value and lessons learned, as well as extracted the key insights and generalisations from the case studies (see Annex 4: Comparative analysis of the case studies for a more in-depth methodology and the results of the comparative analysis).

Reliability of findings and limitations

Overall, the ex-post evaluation provides robust evaluation findings built on the high quality and representative data collected through desk research, stakeholder consultation activities and in-depth case studies. First, the evaluation was supported by DG REFORM, providing complete and comparable internal monitoring and evaluation data on the evaluation period. Second, the consultation activities were highly representative in terms of stakeholder types, while the choice of case studies was well grounded in terms of geographical distribution and thematic policy areas. Third, as indicated in the evaluation framework, the evaluation criteria were broken down the general/operational evaluations, indicators and judgement criteria. The majority of consultation activities were focused on gathering data related to all five evaluation criteria, thus allowing for data triangulation and providing the suitable mix of evidence for the evaluation questions at hand. Furthermore, different versions of the Report were reviewed by the DG REFORM team and the ISSG of the evaluation.

However, several limitations of the evaluation, while rather marginal or mitigated during the study as explained below, should be mentioned. First, only 31% (40 out of 129 responses) of beneficiary authorities indicated that their projects achieved all results by May-June 2022, according to the results of the targeted survey. The fact that some of the SRSP-funded projects were still ongoing at the time complicated the measurement of short-term, medium-term and longer-term results during the evaluation process. In addition, the nature of such project outputs as recommendations, roadmaps, action plans, handbooks and guidelines determined that the beneficiary authorities needed to take follow-up actions after the provision of technical support.

This situation was mitigated twofold. First, the study assessed the likelihood of achieving the intended results and impacts in the case of the still ongoing projects. Second, specific attention was paid to the projects that were already closed to find some “hard” evidence on any results and impacts. Organising ex post evaluation somewhat later in the implementation process could allow better capturing longer-term impacts of the successor programme in the future. 89  

Second, some drawbacks of the quantitative data should be highlighted. EU Member States that had the lowest numbers of requests for technical support and consequently the lowest number of the SRSP-funded projects were less represented in the targeted survey and OPC. However, their experiences were relevant to grasp possible drawbacks of the SRSP that discouraged these countries and their beneficiary authorities from requesting technical support. Therefore, the study put additional efforts into ensuring the inclusion of representatives of Member States with lower participation rates in the focus group discussions or interviews. Also, as indicated in the synopsis report, some OPC questions received a large share of “do not know/cannot answer” responses and limited background information on the respondents. Instead of treating them as a data limitation, the study used these responses to highlight issues that respondents were least aware of and cross-check them with the help of other sources of data.

Third, while the evaluation was carried out at the programme level (by treating the SRSP as the main unit of analysis), the analysis of the monitoring data, stakeholder consultations and case studies focused on the project level. The complete and comparable nature of the internal monitoring/evaluation data, relatively large samples of the targeted survey data and the in-depth analysis of 47 SRSP-funded projects under the case study programme made it however possible to draw reliable programme-level results and conclusions on the basis of project-focused information. The continuous triangulation of data was undertaken to ensure that the specific assessments are applicable beyond the individual project experience. Therefore, the overall findings can be well generalised to the whole programme.

Fourth, the thematic field of sustainable growth and business environment was represented in our case study programme through the horizontal case study on TJTP with 17 projects implemented 17 Member States, thus well addressing the sub-field of climate actions. However, during the case study programme we did not analyse in-depth any projects implemented in the sub-field of business environment where typical SRSP-funded projects included the actions to reduce market barriers, facilitate foreign investment and trade, develop the digital economy, stimulate research and innovation, manage public-private partnerships and improve the governance of state-owned enterprises (where more resistance to change is likely to occur during the reform process). Since the role of external factors is usually very important for achieving the intended results and impacts of the projects in the sub-field of business environment, the study assessed the role of such factors (including political support for reforms) by applying other evaluation methods (e.g., under all types of stakeholder consultations).

Annex III. Evaluation matrix and, where relevant, details on answers to the evaluation questions (by criterion)

1. Effectiveness (results)

Effectiveness refers to the extent to which the SRSP has been successful in achieving its short- and medium-term results. In addition, the assessment involves the underlying mechanisms of change and the external conditions/factors that encouraged or hindered the achievement of the expected results of the SRSP. The assessment of the SRSP results and the operationalisation of questions encompasses the individual, institutional and policy level results of the SRSP projects.

Table 4. The operationalisation of the evaluation questions for assessing the effectiveness (results) of the SRSP

Source: independent evaluation study.

2. Impact

Evaluation of the impact of the SRSP focuses on the assessment of longer-term outcomes of programme activities, including the key two types: 1) structural changes in the institutional and administrative capacity of Member States’ authorities to prepare and implement reforms, as well as to apply EU law in an effective way; and 2) achievement of relevant longer-term growth-sustaining reforms in EU Member States. The assessment focused not only on the impacts and the extent to which these impacts occurred as a result of SRSP, but also on the mechanisms of influence linking the SRSP outputs and results to the structural changes in Member States, as well as the external conditions that either facilitated the achievement of long-term SRSP impacts (success factors) or hindered their achievement (external obstacles/challenges).

Table 5. The operationalisation of the evaluation questions for assessing the impacts of the SRSP

Source: independent evaluation study.

3. Efficiency

Evaluation of efficiency refers to the extent to which the desired effects have been achieved at a reasonable cost or as the optimal balance between the resources employed and the results achieved. The concept of efficiency also concerns the adequacy of management arrangements for the implementation of the programme (e.g., institutional set-up, human and financial resources, processes and procedures, tools), the administrative and regulatory burden imposed by the intervention and the potential for simplifications of the processes. Among other evaluation and analysis methods and tools, the ex-post evaluation employed cost-effectiveness analysis for answering the efficiency questions. 

Table 6. The operationalisation of the evaluation questions for assessing the efficiency of the SRSP

Source: independent evaluation study.

4. Relevance

Evaluation of relevance concerns the relationship between the needs and problems in society and the objectives of the intervention and hence touches on aspects of project design.

Table 7. The operationalisation of the evaluation questions for assessing the relevance of the SRSP

Source: independent evaluation study.

5. Coherence

Evaluation of coherence refers to synergies or inconsistencies within the different elements of an EU intervention, or between different interventions (i.e., how well or not have different actions worked together). The ex post evaluation covered both the extent to which the actions funded under the SRSP were coherent with the objectives as set out in the SRSP Regulation (internal coherence) and the extent to which the SRSP was coherent with other interventions at regional, national and EU level (external coherence). For this purpose, the evaluation relied on the list of other EU programmes and instruments that could be combined with the SRSP to produce synergy effects during the implementation of specific reforms on the ground.

Table 8. The operationalisation of the evaluation questions for assessing the coherence of the SRSP

Source: independent evaluation study.

6. EU added value

Evaluation of the EU added value looked for changes which it can reasonably be argued are due to the EU intervention over and above what could have been expected from actions by the Member States either alone or cooperating together without an EU dimension. The evaluation of EU added value brought together the findings of the other criteria, presenting the arguments on causality and drawing conclusions, based on the evidence to hand, about the performance of the EU intervention and whether it was still justified. 

Table 9. The operationalisation of the evaluation questions for assessing the EU added value of the SRSP

Source: independent evaluation study.

Annex IV. Overview of benefits and costs and, where relevant, table on simplification and burden reduction.

Source: independent evaluation study.

Source: independent evaluation study.

Annex V. Stakeholder consultation – synopsis report

The synopsis report summarises the results of all consultation activities carried out during the ex-post evaluation of the SRSP by providing a qualitative and quantitative analytical overview of the main results. The purpose of this Annex is to inform policymaking on the outcome of all consultation activities and to inform stakeholders on how their input was considered.

The contributions received in the context of this public consultation cannot be regarded as the official position of the Commission and its services nor that the contributions can be considered as a representative sample of the EU population.

Outline of the consultation strategy

The consultation strategy involved the following three phases:

-Phase 1: Planning the stakeholder consultation and establishing the consultation strategy (setting out consultation objectives; mapping stakeholders; and creating consultation (OPC and targeted survey, interview and focus group) questionnaires that are linked to the specific evaluation questions and indicators);

-Phase 2: Conducting consultation work (announcing the launch of the OPC and the targeted survey; running their execution, the interview and focus group programmes);

-Phase 3: Informing policymaking (analysing the data collected during both the online public and targeted consultations; providing an analytical synopsis of the consultation results; triangulating the data with other sources; linking the data collected to specific indicators and judgement criteria of evaluation questions).

Figure 20: INTERACTING PHASES AND KEY STEPS OF THE STAKEHOLDER CONSULTATION PROCESS

Source: independent evaluation study. based on the Better Regulation Guidelines.

The purpose of the stakeholder consultation in this evaluation was to gather views of the key stakeholders and data needed to inform responses to the evaluation questions. Given the wide scope of the SRSP, the consultation sought to include different categories of stakeholders, in particular the European Commission officials, Member State beneficiary and coordinating authorities, providers of technical support, representatives of industries, businesses, workers’ and other organisations, other public authorities, research centres and consultancies, research institutions as well as the general public.

Source: independent evaluation study.

The consultation activities included both the online public consultation and the targeted consultation whose results are presented in the following chapters of this Annex:

1)Online public consultation (OPC). Duration: 1 March 2022 – 24 May 2022, 26 responses from 16 EU Member States. Results of the OPC are summarised in section 2.1 of the Annex 3.

2)Targeted consultation of specific stakeholders employed three methods: a targeted survey, interviews and focus group discussions.

-Targeted survey. Duration: 19 May 2022 – 7 June 2022, 260 responses from all EU Member States. Results are summarised in section 2.2 of the Annex 3.

-Interview programme. Duration: 26 January 2022 – 6 September 2022, 110 interviews with representatives from 17 Member States and other key stakeholders (see Table 13). Results are summarised in section 2.3 of the Annex 3.

-Two focus group discussions. Duration: 22 – 27 June 2022, a total of 17 participants from 12 EU Member States. Results are summarised in section 2.4 of the Annex 3.

Special attention was paid for the selection of participants of consultation activities. To grasp their varying experiences, at least one representative from each Member State that received SRSP funding was involved not only in the OPC or the targeted survey, but also in the interview programme or focus group discussions.

Source: independent evaluation study.

The activities of the online public and targeted consultation adhered to the principles (participation, openness and accountability, effectiveness and coherence) and standards (clear content of the consultation process, targeting and inclusiveness, adequate publication, time limits for participation and acknowledgement of feedback) of the Better Regulation Guidelines.

When referring to quantitative data, the synopsis report presents the summary of key questions analysed only in a descriptive way. As per the Better Regulation Guidelines, the evaluation refers both to the percentage and the number of selections of a particular OPC or targeted survey response out of total responses to a particular question to which this percentage correspond. While presenting results of the quantitative consultation activities, N indicates the total number of responses to a particular OPC or targeted survey question. In addition, N refers to the sample of responses to the question of satisfaction and outcome questionnaires at hand.

Main results of the consultation process

1.1.Summarised results of the Open Public Consultation (OPC)

Implementation of the OPC

The OPC is a web-based consultation that was launched on 1 March 2022 and closed on 24 May 2022 on the dedicated website of the European Commission ‘Have your say’. The consultation was running in all EU working languages for 12 weeks. It consisted of two main structural parts: the background information and SRSP-related questions regarding the key issues of effectiveness, efficiency, relevance, coherence and EU-added value of the programme. The OPC aimed to gather views of a broad spectrum of stakeholders who are interested in the evaluation of the SRSP.

The invitation to participate in the OPC was disseminated through social media platforms and websites of both DG REFORM and the contractor, directly contacting and informing stakeholders from all EU Member States that could be interested in this OPC 93 as well as sharing the invitation to the OPC with the registered participants of DG REFORM online events and country experts involved in the preparation of the case studies for further dissemination.

Taken together, these measures resulted in 26 responses to the OPC. Due to a limited number of responses and their quantitative nature, the data were manually checked to avoid cases of campaigning. No responses were discarded while preparing the synopsis report. Descriptive statistics was used as the main method for the data interpretation. However, it should be noted that the results of the OPC are not statistically representative and are only used to triangulate information collected during the evaluation process.

Participants of the OPC

Participants of the OPC represented four types of stakeholders: academic/research institution, public authority, company/business organisation and EU citizens (see Figure 36).

Figure 21: CATEGORIES OF THE OPC RESPONDENTS

Source: the results of OPC.

Public authorities were the most represented category among the four, covering 46% of respondents (12 out of 26). The vast majority (75%; 9 out of 12) of these public authorities were large-sized (more than 250 employees), 17% (2 out of 12) were medium-sized (50-249 employees) and 8% (1 out of 12) were small-sized (10-49). The sample included agencies (3 out of 12), local authorities (1 out of 12), regional authorities (1 out of 12) and other types of authorities (7 out of 12).

The second most represented category – EU citizens – covered 31% of respondents (8 out of 26), while academic/research institutions and companies/business organisations shared the smallest amount of responses (12% (6 out of 26)). Regarding academic/research institutions, all of the respondents in this category were representatives of large-sized (250+) universities. Variation within the companies/business organisations category was higher in terms of both size and type, including one small-sized (10-49) and two large-sized (250+) organisations.

Sixteen countries were represented (see Figure 37) in the OPC. However, their representation was not equal with the highest share of participants from Romania (19%; 5 out of 26) and Italy (12%; 3 out of 26). Two responses were received from Croatia, Latvia, Malta, and Portugal, while one response was provided from Austria, Bulgaria, Cyprus, Finland, Hungary, Ireland, Lithuania, Netherlands, Spain, and Sweden.

Figure 22: GEOGRAPHICAL DISTRIBUTION OF THE OPC RESPONDENTS

Source: the results of OPC.

Some participants (46%; 12 out of 26) were very familiar with the SRSP. Within this group, two respondents were not involved in the implementation of SRSP-funded projects, but the rest of them were involved in different countries (Croatia, Romania, Malta, Hungary, Austria, Latvia, and Bulgaria). Furthermore, 31% (8 out of 26) of participants were moderately familiar with the SRSP (8 out of 26). While three of them were not involved in the implementation of the SRSP-funded projects, the rest of them were involved in Portugal, Finland, Latvia, Ireland, and Spain. As little as 15% of participants were slightly familiar with the SRSP (4 out of 26). While half of them were not involved in the implementation of the SRSP-funded projects, the rest of them was involved in Lithuania and Sweden. Finally, only 8% of participants were not familiar at all with the SRSP (2 out of 26) – none of them were involved in the implementation of the SRSP-funded projects.

Relevance and coherence of the SRSP

More than a half of participants of the OPC considered the SRSP as the most appropriate means for identified challenges related to the EU`s economic governance processes and Member States` limited administrative and institutional capacity to design and implement structural reforms (20 and 21 respondents respectively found the SRSP appropriate to a moderate or high extent in these areas). However, the perception on the suitability of SRSP to address the challenge of inadequate application and implementation of EU legislation to achieve the EU`s fundamental goals was perceived somewhat lower (with 14 respondents finding the SRSP appropriate to a moderate or high extent in this area).

The majority of participants were not able to evaluate the SRSP’s suitability for addressing issues in such thematic areas as sustainable development, labour market, social protection, and migration, skills, education and training or financial sector and access to finance (i.e. nearly half of them responded ‘do not know/cannot answer’). However, a significant part of respondents recognised the SRSP as highly or moderately appropriate to address the needs of EU Member States in designing and implementing reforms in the horizontal areas of governance and public administration (77%, 20 out of 26) and digitalisation (58%, 15 out of 26). Additionally, eight participants followed up on this aspect in the comments, pointing out the specific challenges that should be addressed by SRSP (e.g. addressing specific national/sub-national challenges in EU Member States as well as doing it more quickly and in a more flexible way).

The vast majority of the OPC participants were at least to some extent familiar with the different EU programmes 94 . Among six of them, participants were highly familiar with the instruments of EU Cohesion policy (61%, 16 out of 26) and Horizon 2020 (42%, 11 out of 26), while to some smaller extent they were familiar with the LIFE programme (42%, 11 out of 26 including both moderate and high familiarity). In contrast, 30-38% of participants (8-10 out of 26) reported being not familiar at all with Fiscalis 2020, Customs 2020, and the Hercule III programme. 

When comparing the complementarity of the SRSP (by funding different aspects of similar activities, targeting different groups, etc.) with different EU programmes and instruments 95 , 62% of participants (16 out of 26) marked EU Cohesion policy as the most complementary. It was followed by Horizon 2020 (12 out of 26), the LIFE programme, Customs 2020, and Fiscalis 2020 (8 out of 26) with the Hercule III programme being the least complementary one according to participants (7 out of 26). Half of the participants (13 out of 26) were not aware of the complementarity of SRSP actions with similar national and regional programmes, while the largest share of the rest of participants considered SRSP actions moderately (9 out of 26) complementary in this regard.

Some participants shared specific examples of national and regional programmes similar to the SRSP (e.g., SAMA and POCI in Portugal, the Action Plan for joining the Exchange Rate Mechanism II (ERM II) and the Banking Union in Croatia). When answering an open question, one participant stated that complementarity depended on the Member State's strategy to use technical support and to coordinate the implementation of different initiatives at national level.

Efficiency of the SRSP

In general, the majority of participants saw SRSP actions as moderately efficient. More than half of them perceived SRSP actions as user-friendly, timely, and cost-effective to a moderate and high extent compared to other EU actions (17 out of 26 respondents) and national actions (18 out of 26). When it comes to comparison with the regional and local actions, half of respondents (13 out of 26) perceive SRSP actions as moderately and highly efficient.

Regarding efficiency in the implementation of technical support projects, the majority of participants considered the sufficient duration of the projects (20 out of 26) and the proportionality of administrative burden (19 out of 26) as moderately and highly suitable factors in making the SRSP`s implementation efficient. About half of participants perceived the proportionality of the budget scale of SRSP actions to the needs of technical support requested by the Member States (16 out of 26) and the appropriateness of the timeliness of administrative procedures (14 out of 26) moderately and highly suitable.

Effectiveness of the SRSP

The majority of respondents (62-73%, i.e. 16-19 out of 26) thought that SRSP actions were to a moderate or high extent successful in supporting national authorities in the design/implementation of reforms, enhancing their capacity to formulate, develop and implement reform policies and strategies, and their efforts to design and implement appropriate processes and methodologies by taking into account good practices of and lessons learned by other countries. While these objectives share a rather similar perception, the evaluation was rather different regarding enhancement the efficiency and effectiveness of human resource management – 50% (13 out of 26) of respondents thought SRSP actions were to a moderate or high extent successful, but 38% (13 out of 26) of them did not have an opinion on this objective of a more specific nature.

A total of six participants provided their comments on the usefulness of the outputs produced by the SRSP-funded projects. While most respondents saw these outputs as very useful, a few participants perceived the production of technical support outputs as insufficient to overcome long-standing and deep-rooted challenges (such as limited administrative capacity or even excessive bureaucracy).

The majority of respondents emphasised the high (38%, 10 out of 26) or moderate (38%, 10 out of 26) contribution of the SRSP to the improvement of internal working procedures, methodologies, and processes in beneficiary authorities. The programme’s contribution to other outcomes – the adoption of new strategies, new laws/acts, or amendments to current laws, procedures, and actions to improve the way reforms are implemented in the Member States, as well as on any organisational change in beneficiary authorities – was perceived to be somewhat smaller because some of the participants (from 8 to 11 out of 26) did not know about such outcomes.

More than a half of respondents recognised the high (35%, 9 out of 26) or moderate (23%, 6 out of 26) contribution of the SRSP to structural changes within Member States` beneficiary authorities in preparing and implementing reforms and adequately applying EU law. The programme’s contribution to the impact of achieving the relevant longer-term growth-sustaining reforms as indicated in the CSPs was slightly weaker (with 19% or 5 out of 26 respondents stating that the contribution was high and 35% or 9 out of them stating that it was moderate).

When it comes to the success of implementing SRSP-funded projects at national level, 56-60% (15-16 out of 26) of participants saw national ownership of reforms and/or cooperation between the European Commission, beneficiary authorities, and providers of technical support as the most influential factors. The following factors were less frequently mentioned, but still very important during the implementation process: political reform commitments (46%, 12 out of 26), national economic and social situation (46%, 12 out of 26), timing and sequencing of reforms at the national level (50%, 13 out of 26), national, regional, and local authorities working in partnership during the reform process (42%, 11 out of 26), consultation with different stakeholders during the reform process (38%, 10 out of 26) and communication of reforms to the general public or specific target groups (42%, 11 out of 26). Finally, four participants provided additional comments, e.g. mentioning the very high capability of SRSS/REFORM project officers, their networking skills and practical approach as a significant factor.

EU value added of the SRSP

A total of 38% of participants (10 out of 26) believed that Member states could have achieved similar results in the design and implementation of reforms only to a limited extent without the technical support they received through the SRSP, while 4% (1 out of 26) thought it could not be possible at all.

The perception of EU added value varied among the main groups of stakeholders. Public authorities stressed the relevance of tailored solutions, reflecting the needs of Member States as well as an opportunity to collaborate with valuable partners (e.g. the World Bank or OECD). Representatives of business organisations pointed to a comparatively low administrative burden and clear procedures for beneficiary authorities, ease of access to international expertise and absence of co-financing requirements for them. Representatives of academic/research institutions indicated that the technical support provided under the SRSP offered a continuity to the reform process, improved the development of administrative capacity and internal reorganisation of institutions. Finally, based on the views of EU citizens, the SRSP offered specialised assistance whose delivery would not be possible at national and regional level (e.g. developing new methodologies, roadmaps or guidelines).

Measures for the improvement of the future programme

Simplification and reduction of the administrative burden in the administration of the programme in the future as one of the most important measures by vast majority - 65% (17 out of 26) participants. Additionally, developing links between technical support projects and the European Semester/Recovery and Resilience Facility surveillance process, and developing flagship technical support projects that address top EU priorities and the common needs of Member States were marked as important by 54% each (14 out of 26). Furthermore, better support for the digital and green transition, as well as support for a broader range of technical support activities on the national level were considered important by 38-46% (10-12 out of 26 participants). In addition, involving more regional and local authorities in the design and implementation of the technical support and providing more support to multi-country projects were selected by 46% (12 out of 26) of participants. Finally, the least important measure according to participants was the better adaptation of delivery modes to Member States` needs, which was selected by only 27% (7 out of 27) of participants.

Representatives of public authorities pointed out that the SRSP should better take into account the country-specific recommendations and pay more attention to the involvement of grassroots organisations (e.g. NGOs). Additionally, the need for a more transparent management of funds was highlighted. Representatives of business organisations stressed that sharing of good practices and lessons learned/exchange of information between beneficiary authorities in different Member states could be further improved. Also, they added some recommendations regarding capacity-building activities for beneficiary authorities (workshops, seminars, etc.), the potential involvement of niche-oriented advisory firms in the implementation of projects (rather than using the already pre-selected companies) and the need for making a quicker project launch (especially in case of urgent reform needs). Finally, citizens pointed out that the on-demand nature of technical support be further improved, while beneficiary authorities should be involved in the selection of technical support providers to ensure their suitability for the project.

1.2.Summarised results of the targeted survey

Implementation of the targeted survey and data analysis

The targeted survey was running from 19 May 2022 to 7 June 2022 on the Alchemer survey tool. The survey questionnaire was divided into two parts: (i) an introduction and (ii) questions adjusted to the experience of national beneficiary authorities, coordinating authorities and technical support providers. Questions for these target groups were split into several blocks on the basis of key evaluation criteria: the effectiveness/impact, efficiency, relevance, coherence and EU-added value of the SRSP. The survey questions were carefully crafted to ensure comparability across the different groups of respondents and complementarity with the other consultation methods.

The main purpose of the survey was to collect opinions on the overall functioning of the SRSP, the quality of technical support received, the policy goals and objectives met, and the results and impacts achieved throughout the evaluation period (2017-2020). DG REFORM directly shared invitations to participate in the targeted survey with the representatives of beneficiary authorities, coordinating authorities as well as technical support providers, involved in the preparation, design and implementation of SRSP-funded projects.

To provide robust findings on the targeted survey results, the data was cleaned from empty and partial responses. The data analysis was based on two pillars: (i) a descriptive presentation of the targeted survey results and (ii) an estimation of statistical associations between key variables. Since most of the targeted survey responses were ordinal or nominal, Cramer’s V was used to measure the power of association, and Fisher’s exact test was employed to identify the statistical significance of the findings. Some nominal variables were transformed into binary ones when there was a need to report on individual answers to the multiple-choice question sets. While providing descriptive analysis of the targeted survey results, “Do not know/cannot answer” responses were omitted.

Participants of the targeted survey and scope of the projects involved

A total of 260 completed responses of the targeted survey were analysed. A total of 51% (132) of participants were representatives of the beneficiary authorities, 41% (106) – technical support providers, and 8% (22) submitted responses as the national coordinating authorities (Figure 38).

Figure 23: SHARE OF THE TARGETED SURVEY RESPONDENTS

Source: the targeted survey of beneficiary authorities, coordinating authorities and technical support providers, May-June 2022.

As presented in the figure below, the representatives of the beneficiary authorities from all EU Member States except France participated in the targeted survey, while the responses of national coordinating authorities from Austria, Belgium, Denmark, Ireland, Latvia, Lithuania, and Slovenia were absent. The number of respondents from different Member States closely reflected the distribution of SRSP-funded projects across the countries.

Figure 24: PARTICIPATION OF REPRESENTATIVES OF BENEFICIARY AND NATIONAL COORDINATING AUTHORITIES IN THE TARGETED SURVEY BY THE COUNTRY OF INSTITUTION

Source: the targeted survey of beneficiary authorities and coordinating authorities, May-June 2022. Note: numbers in brackets refer to the number of SRSP-funded projects, implemented in a particular Member State.

Regarding the previous experience of implementation of the SRSP-funded projects, 72% of technical support providers were involved in two and more (76 out of 106) projects, while more than a half of beneficiary authorities (51%, 67 out of 132) were involved only in one project.

As presented in the table below, the highest share of the targeted survey participants was involved in the SRSP 2019 projects. The distribution of beneficiary authorities and technical support providers in the SRSP-funded projects was rather equal through the period of evaluation (2017-2020), except for the SRSP 2017. There was only one technical support provider who took part in the Article 11 programme in 2018.

Source: the targeted survey of beneficiary authorities and technical support providers, May-June 2022.

A relatively equal share of beneficiary authorities and technical support providers, who responded to this targeted survey question delivered/received support in the fields of governance & public administration as well as financial sector & access to finance (Figure 40). A larger share of the contractors provided support in the fields of revenue administration & public financial management, sustainable growth & business environment, labour market, education, health & social services, and horizontal policies.

Figure 25: SECTORAL AREAS OF THE SRSP TECHNICAL SUPPORT, 2017-2020

 Source: the targeted survey of beneficiary authorities and technical support providers, May-June 2022.

The majority of respondents received or provided technical support through public procurement (57% of beneficiary authorities, 75 out of 108, 51% of technical support providers, 54 out of 98). A significantly lower share of beneficiaries (17%, 23 out of 108) and technical support providers (28%, 30 out of 98) used grants. No more than 15% of respondents from both groups used the TAIEX, delegation agreements, Fiscalis 2020 and direct expertise of the Commission.

Relevance and coherence of SRSP

Regarding specific types of reforms that beneficiary authorities targeted in their requests for technical support under the SRSP (2017-2020), the vast majority of reforms were undertaken at the own initiative of Member States (58%, 77 out of 132). However, a large share of reforms was also related with the context of economic governance (43%, 57 out of 132) or linked to the Union priorities (31% and 41 out of 132). A smaller number of respondents indicated reforms being related to the Union law (17%, 22 out of 132) or the implementation of economic adjustment programmes (3%, 4 out of 132).

The majority of beneficiary and coordinating authorities as well as technical support providers (84% – 98%) agreed or strongly agreed that the SRSP was a suitable instrument to provide technical support, addressed the key needs of beneficiary institutions, corresponded to the key reform goals of the Member State, the project design was appropriate, and the modes of delivery selected for the project(s) met their implementation needs.

When assessing the further need for technical support, the majority of respondents indicated that it is still relevant both for the particular institutions and the Member States. As many as 95% of representatives of national coordinating authorities (21 out of 22) reported that the support is needed to a large (45%, 10 out of 22) or to some extent (50%, 11 out of 22). This view was supported by 83% of beneficiary authorities (100 out of 121), the majority of which indicated that support is needed to some extent (61%, 74 out of 121). Finally, the views of technical support providers were more positive: 39% and 43% of them claimed that the support is still needed to a large or some extent respectively (38 and 42 out of 97 respectively).

There were no clear trends when assessing the complementarity of the SRSP with the similar national and regional programmes. According to the targeted survey results, the largest share of national beneficiary authorities (72%, 35 out of 49) considered that SRSP to a moderate/high extent complemented the support provided through the ESIF. Similarly, the majority of the beneficiary authorities that offered an opinion on this issue (69%, 34 out of 49) considered that the SRSP complemented the interventions enhancing the institutional capacity of public authorities and stakeholders and efficient public administration supported by the European Social Fund and the European Regional Development Fund. At the same time, the respondents claimed that the SRSP complemented the modernisation of public and private employment services (ESF) and investments in employment infrastructure by the ERDF only to a limited extent (40%, 14 out of 35).

When assessing the contribution of the SRSP-funded projects to the implementation of the EU priorities for 2019-2024, the majority of beneficiary and coordinating authorities agreed on their moderate and high contribution to the development of a strong and vibrant economic base (66%, 67 out of 116) and to a lesser extent – to building a climate-neutral, green, fair, and social Europe (58%, 54 out of 116). The smallest share of beneficiaries and coordinating authorities (38%, 29 out of 108) indicated that the projects moderately and highly contributed to the protection of citizens and freedoms.

Efficiency of the SRSP

The majority of respondents (over 80%) stated that the preparation, implementation, and evaluation of SRSP-funded projects were carried out in a timely and efficient manner (see the figure below), but their agreement concerning preparation for implementation was somewhat weaker.

Figure 26: Timeliness and efficiency of the SRSP-funded projects

Source: the targeted survey of beneficiary authorities, May-June 2022.

The absolute majority of respondents (92-97%, 74-77 out of 80) agreed or strongly agreed that the administration of technical support was efficient. However, 8% of beneficiaries disagreed that the evaluation procedures and processes after the completion of projects was clear, transparent, and user-friendly (10 out of 119). Moreover, 7% claimed that the financial contribution of the SRSP was not proportional to the needs of technical support (8 out of 115).

Regarding the efficiency of the SRSP funded projects implementation on the ground, the majority of beneficiary authorities and technical support providers (strongly) agreed that the support of DG REFORM officers was useful (89%, 118 out of 130 and 92%, 97 out of 102, respectively). Beneficiary authorities also (strongly) agreed that DG REFORM was able to provide support from the start of technical support to its end (91%, 120 out of 131) and risks of the project(s) have been effectively managed (85%, 112 out of 127). In addition, technical support were seen as having the required expertise and skills according to all delivery modes (88%, 109 out of 124), while the cooperation with them was treated as efficient (86%, 106 out of 122).According to the providers of technical support, bureaucratic resistance to change during the implementation of the project activities was one of their main inefficiencies (55%, 58 out of 94 (strongly) agreed on its occurrence).

Figure 27: The extent to which the respondents agreed that the providers of technical support has(ve) had the required expertise and skills according to the delivery modes

Source: the targeted survey of beneficiary authorities, May-June 2022.

Effectiveness of the SRSP

The respondents reported some differences in the stages of the policy cycle addressed by the SRSP-funded projects. Beneficiary authorities worked predominantly on policy development (76%, 100 out of 132) and policy implementation (51%, 67 out of 132). A fifth of them (19%, 25 out of 132) were involved in the evaluation phase of the cycle during their projects.

Both beneficiary authorities and technical support providers reported recommendations (73%, 96 out of 132 and 86%, 91 out of 106, respectively) being the most common outputs expected to be delivered through the technical support projects. The other outputs included analyses and reports (68% of beneficiaries, 90 out of 132, 83% of technical support providers, 88 out of 106), workshops, training, training material (65% of beneficiaries, 86 out of 132, 84% of technical support providers, 89 out of 106) as well as action plans and roadmaps (58% of beneficiaries, 77 out of 132, 69% of technical support providers, 73 out of 106). The least common outputs were legislative proposals (20% of beneficiaries, 27 out of 132, 29% of technical support providers, 31 out of 106) and terms of reference (9% of beneficiaries, 12 out of 132, 16% of technical support providers, 17 out of 106).

Similarly, the beneficiary authorities referred to the results of the analysis and reports used (90%, 114 out of 124), new knowledge and skills maintained (93%, 107 out of 120) as well as recommendations implemented (80%, 101 out of 121) as deliverables of the projects that were to some or large extent used in their work. In terms of the actual delivery of outputs, 80% of beneficiary authorities (106 out of 130) and 64% of technical support providers (68 out of 103) indicated that all project outputs were delivered.

Regarding the factors that positively or negatively affected the delivery of technical support outputs and results, answers of all the groups of respondents were analysed together (Figure 42) 96 . The cooperation between the EC, beneficiary authorities and technical support providers, consultation with different stakeholders during the reform process as well as partnership with national, regional and local authorities were named as factors with a moderate or strong positive impact. In terms of factors with the moderate or strong negative effect, the COVID-19 pandemic, time and sequencing of reforms as well as political reform commitments were distinguished.

Figure 28: FACTORS AFFECTING THE DELIVERY OF OUTPUTS AND RESULTS OF THE SRSP-FUNDED PROJECTS

Source: the targeted survey of beneficiary authorities, coordinating authorities and technical support providers, May-June 2022.

The results of the targeted survey highlight some changes observed in sectoral policy areas because of implementing the projects. According to beneficiary authorities, the projects resulted in the production of new information or knowledge (89%, 92 out of 103), improved strategies and reform/policy documents (79%, 81 out of 100) and issuing or modification of ‘soft’ instruments (67%, 68 out of 101) to some or a large extent. Such changes as better application of economic instruments (taxes, charges, fees, etc.) (31%, 23 out of 74) or improved application and implementation of EU law (43%, 35 out of 82) were less common. 

While responding to an open question on the contribution of the SRSP-funded projects to reforms and policies in their country, beneficiary and coordinating authorities highlighted the production of practically applicable tools (guidelines, roadmaps, action plans, etc). They mentioned the relevance of the received support for specific policy fields, resulting in an improvement of legislation, regulations and strategic documents, an increased administrative capacity and a better quality of public services.

The submission of the request for technical support for a follow-up-project or a new stage of the larger project was named as the most frequent activity (36%, 41 out of 114) among follow-up actions taken after the project end and building upon the project outputs and results. Overall, the beneficiary authorities were very satisfied with the SRSP-funded projects (65%, 79 out of 121 were very satisfied, 29%, 35 out of 121 were satisfied). Approximately a half of beneficiaries (49%, 54 out of 111) would consider requesting support from the TSI and 46% have already requested for it (51 out of 111). 

While answering to the open questions on the implementation of the SRSP and the possible improvements of the future programme, significant overlaps were noted in the opinion of beneficiary authorities, coordinating authorities and technical support providers. All these stakeholders raised the issue of project impact and sustainability that depend on national-level political commitment and continuous monitoring and provision of resources by the EC. In addition, the need for closer collaboration across countries and projects was stressed with an aim to learn and transfer professional experience. Looking from the administrative perspective, the period between the submission of requests for services and the start of project activities was named as too long. Furthermore, since the selection of technical support providers was seen as insufficiently transparent, a broader selection, a clearer criterion of selection and a greater inclusion of beneficiary authorities were seen as possible areas for improvement in this process.

EU value added of the SRSP

The SRSP-funded projects had a significant EU added value. A total of 88% of beneficiaries claimed that they used good practices and lessons learnt from the SRSP-funded projects to a large or to some extent (105 out of 122), while 12% (14 out of 122) indicated not using the good practice/lessons learned at all or using them only to a limited extent. At the same time, 80% of respondents (44-45 out of 111) indicated that a similar support would have been either not available or available to a limited extent without the SRSP funding (Figure 43).

Figure 29: EU VALUE-ADDED OF THE SRSP

Source: the targeted survey of beneficiary authorities, May-June 2022.

1.3.Summarised results of interviews

Implementation and participants of interviews

Interviews with EU- and national-level stakeholders were conducted to complement findings from desk research, the OPC and the targeted survey of stakeholders. The interview programme was divided into four parts: (i) exploratory interviews with the European Commission officials, (ii) exploratory and follow-up interviews with the national authorities, (iii) case study interviews, and (iv) validation interviews. The interviews were conducted following a standardised questionnaire, including questions on the evaluation criteria and adjusted to the experience of individual stakeholders. Each interview was recorded, interview notes and/or transcripts were prepared based on the recording. A total of 110 interviews were conducted (see the table below for the total number of interviews based on their type).

Source: independent evaluation study.

The interview programme was designed to embrace a wide variety of views, including those coming from the beneficiary or national coordinating authorities as well as from the policy officers and other officials of the EC. Depending on the question at hand, the interview programme was used either to supplement other sources of evidence or as the main information source in cases when other data sources were scarce.

Relevance and coherence of the SRSP

The initial results of the interview programme point to the importance of dedicated calls. According to the participants of exploratory and case study interviews, the introduction of these calls allowed providing more targeted support, addressing both the urgent needs of Member States as well as the EU priorities and country-specific recommendations of the European Semester. The opportunity to receive timely support (more often than through the annual call) was also highlighted.

The evaluation evidence also suggests that there are sufficient mechanisms to ensure the linkages and integration between the SRSP and the European Semester process/CSRs. For instance, the projects addressing CSRs are prioritised during the evaluation of requests for technical support. However, there is insufficient tracking if specific CSRs are fulfilled due to the results of SRSP projects implemented at national level.

Case study interviews revealed that the SRSP funded projects were complementary and had high level synergies with other projects, programmes, and/or interventions occurring at the level of the Member State or EU. In some Member States the SRSP funding was necessary to implement other projects or achieve access to funding mechanisms, while in the others follow-up actions were taken for the implementation of reforms or broadening of their scope.

Efficiency of the SRSP

Participants of the exploratory, follow-up and case study interviews highlighted that the administrative burden of the SRSP-funded projects was similar or even lower compared to the other EU programmes. The bilateral contract between the Commission and technical support providers was mentioned as reducing administrative burden for the beneficiary authorities and allowing them to dedicate necessary resources on the actual implementation of technical support projects. However, some drawbacks were mentioned, including a long time period between the submission of requests for technical support and the beginning of project implementation (sometimes taking up to one year of time).

Whereas the budget of the programme steadily increased over the evaluation period 2017-2020, the human resources in DG REFORM did not keep the pace with this development. As a consequence, DG REFORM staff needed to manage an increasing number and more complex projects. This increased the workload of responsible policy officers and, as noted by some of the participants of case study interviews, reduced their ability to provide tailor-made expertise during the provision of technical support on the ground.

The cooperation between DG REFORM and other Commission services was positively assessed by the majority of interviewees. In general, there were well-functioning formal and informal communication channels between DG REFORM units and corresponding Commission services working in specific sectoral areas. More specifically, all the parties involved considered the mechanism of consulting other DGs as very positive and productive when assessing technical support requests. Some of the stakeholders, however, indicated short cycles of evaluation and tight deadlines given for other DGs during this process, which added to their workload.

Some interviewees from beneficiary and national coordinating authorities highlighted the need for a closer involvement of beneficiary authorities into the selection of technical support providers, which would ensure a better match between the contractor, the aims of the project and the expectations of beneficiaries. Regarding the role of beneficiary authorities, the national ownership and commitment for the reforms were mentioned as key factors of successful project implementation, allowing to make timely decisions, facilitate engagement of relevant stakeholders and ensure smoother communication between the parties involved. It was noted that in cases without an effective involvement of political authorities, the projects required stronger involvement and steering by the contractor or policy officers of DG REFORM.

Effectiveness of the SRSP

The data collected during the evaluation evidenced the mixed effects of the COVID-19 crisis on the implementation of SRSP-funded projects during the evaluation period. On the one hand, some public health restrictions prevented organising some of the key project activities that previously worked as a platform for building trust-based relationships and networks of stakeholders. Also, the pandemic caused some changes to the project contracts as amendments to the budgets became necessary (e.g. due to the re-allocation of funds from missions or physical events to other activities). On the other hand, a sharp worsening of the economic conditions created more awareness on the need for reforms in individual EU Member States. Also, the crisis could have even increased the level of flexibility and made some project activities more inclusive for some groups of stakeholders due the organisation of online or hybrid events.

These interviews also provided some evidence on the key factors influencing the success of SRSP-funded projects in terms of achieving the project objectives and implementing reforms in Member States. Among the internal factors/mechanisms of change the following were identified by a number of stakeholders interviewed:

-Strong leadership and ownership by beneficiary authorities of the reforms and outputs developed during the technical support projects: leadership and pro-active involvement of beneficiary authorities is essential for the project outputs and results to be used in policy decisions;

-Proper cooperation and communication between different stakeholders involved in the project: it is crucial to ensure that there is constant communication and exchange of feedback between all the parties involved in the project implementation, e.g. between the beneficiary authority, the team of experts at national level and the support provider and is motivated. For this purpose it is important to develop and ensure the functioning of regular and effective communication channels – e.g. attending common meetings, exchanging comments, co-designing of project deliverables, setting common priorities and goals etc.;

-Adequate project design and planning: this includes development of a clear strategy of embedding the outputs, milestones for reforms, also timing of the key project deliverables and activities while considering the budget and allocation of resources;

-Presence of a well-balanced project team within the beneficiary Member States: this entails a project team that, on the one hand, involves technocrats/experts who would not leave their positions after changes related to political cycle. This helps maintain organisational continuity and expertise inside the institution. On the other hand, the project team should also include adviser(s) or other officers close to the Minister’s office who would ensure political support for the reform.

Furthermore, the evaluation evidence also confirmed some of the key external factors that influence the success of technical support projects – most importantly, the electoral cycle and political support for the reforms planned and designed in these projects. A technical support project is more likely to be successful if it addresses reforms or particular measures that are an integral part of a wider reform programme within a specific sector or when it addresses a specific EU policy priority in a particular sector. Since the political cycle might create incentives for reforms or might disrupt their implementation, the implementation of reforms depends much on the role of civil servants.

EU value added of the SRSP

The majority of participants of the interview programme referred to a few examples of the EU added value. Sharing of good practices, exchange of know-how and other activities allowing to exploit the synergies across the projects were highlighted. Some interviewees even indicated that learning from the experience of other Member States significantly improved the implementation of reforms in their institution or country. Our evaluation results suggest that there were no significant differences in this regard between the annual and dedicated calls. However, the role of policy officers and technical support providers as facilitators of this exchange was stressed, especially due to the lack of communication platforms (e.g., Just Transition Platform) and limited cooperation experience of the Member States. On the other hand, some of our interviewees assessed the SRSP-funded projects as facilitating cooperation among Member States. Key factors that encouraged this process included similarities in specific country challenges and national priority reforms as well as incentivising role of the policy officers of DG REFORM.

In addition, participants of case study interviews were rather sceptical about the possibility of their institution or country to implement foreseen reforms without the technical support of SRSP. The programme provided valuable financial resources to achieve high-level expertise, which was considered less likely to be available if the project was funded and managed exclusively at the national level. Furthermore, the role of the EC as a legitimising actor was mentioned. It was especially relevant in countries overcoming political turbulences or projects focused on contested topics, when the endorsement of the European Commission allowed to gain support as well as ensure involvement of the relevant stakeholders.

1.4.Results of focus group discussions

Implementation and participants of focus group discussions

Two focus groups were carried out on 22 June 2022 (on management and implementation of the SRSP) and 27 June 2022 (on the management and implementation of the dedicated call for the preparation of TJTPs).

-22 June: focus group with national coordinating authorities (Cas). Nine representatives of eight national coordinating authorities participated in the focus group (Belgium, Estonia, Spain, Croatia, Hungary, Malta, Portugal, Slovakia) in addition to the evaluation team.

-27 June: focus group with eight representatives of six beneficiary authorities (Bas) of projects under the dedicated call (Bulgaria, Greece, Czechia, Croatia, Latvia, Slovakia) in addition to the evaluation team.

Both focus groups shared the same agenda, including introduction, presentation of emergent findings, structured discussion and its summary. The purpose of the focus groups was to test the findings emerging from the research with a cohort of individuals directly involved in implementing the programme. The participants were asked to comment on the validity of the findings, provide examples from their experience and suggest recommendations for the future.

Efficiency of the SRSP

The majority of participants agreed that dedicated calls helped to address the needs of specific geographical areas and achieve a better understanding of new funding instruments. In addition, some participants emphasised that dedicated calls were very helpful when focused on new areas where Member States had no prior experience. For instance, since the regulation establishing the Just Transition Fund was new to Member States, the support of the Commission provided under the dedicated call on the preparation of TJTPs was key to understand it. This was also the case with the dedicated call related to the implementation of the Recovery and Resilience Plans giving access to the Recovery and Resilience Facility.

Transparency and clarity in the prioritisation and selection of technical support requests submitted by the Member Sates’ authorities could facilitate the role of coordinating authorities. Member States, particularly those with a decentralised system of government, have experienced challenges in the prioritisation of technical support projects under SRSP. Coordinating authorities from these countries faced competing requests from different bodies both at national and regional levels and therefore needed to state the basis on which requests have been put forward to the Commission and accepted by it. One possible solution advised during the focus group discussion was to have a regulatory basis for the prioritisation. This could also encourage regional governments of Member States to join forces and collaborate on projects.

Some coordinating authorities have created specific systems to prioritise the technical support projects. Such systems assess the projects against their compliance and coherence with the EU and national priorities and define different priority categories. In addition, the coordinating authorities are also usually involved in the programming of other funds (e.g. Cohesion Policy programmes) and therefore have a strategic overview of programming priorities for the country. Sometimes, there are discrepancies between the priorities defined by the coordinating authorities and the technical support projects selected by the Commission and the justification provided is not always sufficient. Reconciling these two approaches would help programme implementation.

Overall, the submission and selection of technical support requests was viewed positively by coordinating authorities. Simplifying and ensuring more coherence is crucial especially for coordinating authorities with limited resources dedicated to the programme. Coordinating authorities have a key role as they facilitate cooperation with the beneficiary authorities and provide peer to peer review and support. Participants of the focus groups indicated some areas for improvement of the technical support that beneficiaries receive, including the use of some pre-filled forms for the submission of requests and the further simplification of the current procedures.

Coordinating authorities and beneficiary authorities reported a high level of satisfaction with the selected technical support providers. The Commission was responsible for the selection of technical support providers and overall, participants agreed that the contractors selected by the Commission had the right competencies. There was a general agreement that this is crucial, as beneficiary authorities required the high-quality expertise.

In practice, the selected international experts had a smaller role during the project implementation compared to what was initially foreseen in the project proposal. This might have been caused by subsequent economic considerations by the technical support provider or by a wrong interpretation of the terms of reference. Therefore, the Commission was expected to maintain oversight over the selected providers in order to avoid subcontracting of certain services to third entities that are not able to provide an equivalent level of expertise and quality.

On the other hand, there were several advantages associated with the involvement of local experts. Above all, local experts can discuss technical aspects of projects in their native language. The involvement of local experts can also facilitate the engagement with local stakeholders. However, several participants argued that having national experts is helpful provided that they are connected internationally and ensure access to international experience.

Furthermore, coordinating authorities reported that the process of agreeing the CSPs is straightforward, however, some challenges might arise from the non-binding nature of these Plans. The only binding document is the contract between the Commission and the providers.

Collaboration between the Commission, technical support providers and beneficiary authorities was reported to be positive, especially during the preparation of the TJTPs. Organising ‘tripartite meetings’ between these actors was mentioned as key to effective preparation and implementation of projects, because of input and views provided by other DGs of the Commission (in case of the dedicated call, they were helpful for the preparation of TJTPs). Overall, the Commission supported the MSs from the preparation of the request for technical support, to the finalisation and submission of the same.

Effectiveness of the SRSP

Generally, the need to continuously involve beneficiary authorities in decision-making was stressed by the participants. Involving beneficiary authorities in the drafting of the terms of reference, especially as regards the description of the data collection activities, might help ensure that the work of the provider meets the expectations of the beneficiary authorities. The beneficiary authorities reported that the preliminary collection of data and statistics was useful and important for the preparation of the TJTPs and the selection of the projects that should be part of it. However, some beneficiary authorities shared concerns about the performance of the contractors on this specific activity. In some cases, the technical support providers did not conduct additional interviews or in-depth research to fill some data gaps. One factor which could explain this misapprehension – according to a participant – is a wrong interpretation of the terms of reference.

One more issue raised by the coordinating authorities was the possibility for the Commission to make changes to the project during implementation without sufficient involvement and consultation with the beneficiary authorities. In some cases, this approach has determined a slippage in the timing allocated during the project implementation. In addition, this approach might reduce the value of the technical support for the beneficiary authorities.

Events can be a useful means by which beneficiary authorities can raise awareness among stakeholders about the results of their projects and also about the TJTP in general. Some beneficiary authorities reported benefits from organising workshops to do this. Some countries adopted a more structured approach to stakeholder engagement than others. For instance, one participant from Greece reported that the contractor was responsible for identifying key stakeholders of the TJTP (including ministries and other relevant stakeholders). The beneficiary authorities set up a formal partnership with these stakeholders and organised a series of online meetings and workshops to explain the rationale and value of this plan and how it would benefit the different regions.

Overall, involving a broad range of stakeholders was seen as key to ensure the effectiveness of TJTP projects. In Croatia, for instance, an ad hoc team was set up at the central government level, but also regional stakeholders were involved in several aspects of this process (including the preparation of the projects and overall TJTP structure and priorities). Private entities were also involved in working groups dedicated to the preparation of TJTP projects.

It is important to mention that the SRSP-funded projects were able to move much of their activity online and thus minimise, although not completely overcome, the adverse impact of the pandemic on their implementation. The COVID-19 pandemic caused a shift in the project delivery mode, from in person to online for most activities in most projects. From the design perspective, most of the projects were not affected. In the implementation phase, instead, the ownership of the projects has been weakened in some cases by the restrictions and the lack of in person meetings. This particularly affected projects which foresaw trainings and conferences. In particular, the majority of the events were held online, and it was not easy to get people involved and share data and information effectively through virtual communication channels. The online exchange had some drawbacks compared to the in-person interaction and required more coordination efforts. However, coordinating authorities reported that technical support providers had generally maintained equivalent levels of service to the extent possible.

Measures for the improvement of the future programme

Focus group participants made a number of suggestions for measures that could improve the future programme.

-Multi-country projects: the coordinating authorities were supportive of the idea of multi-country projects. According to them, the Commission could take a more active role and encourage collaboration between potential beneficiary authorities in different Member States with similar needs and priorities, in order to support the preparation of a joint request for a multi-country project. To this end, the Commission could set up a platform/website where beneficiary authorities could share information and experiences on different projects and learn from each other;

-Preselected pool of experts: Member States could benefit from being able to draw on a pool of experts to address urgent technical support needs;

-Involving regional and local authorities in the project implementation: some participants of the focus group emphasised that, based on their experience, involving regional and local authorities as part of Steering Committees and in the exchanges with the technical support providers was beneficial for the implementation of projects;

-More dedicated calls on specific themes: the coordinating authorities reported that the design of dedicated calls is beneficial to help Member States address their new or evolving needs. Such calls could be launched as and when it is needed to address the urgent needs or priorities similar across the Member States;

-Increased flexibility in implementation: beneficiary authorities would benefit from the possibility to make changes during the implementation of technical support projects to address the major changes in their needs or respond to changing circumstances. At the same time, the need for a possibility of project duration extension was expressed. With hindsight, this might have been beneficial when the JTF regulation was introduced and some of the projects under the dedicated call have already been finalised;

-Collaboration and knowledge exchange between the projects: the coordinating authorities can play a key role in facilitating this collaboration and sharing of knowledge. Beneficiary authorities highlighted benefits provided by annual meetings that were organised by the coordinating authorities to encourage mutual learning and informal exchanges between SRSP-funded projects.

Project status, activities and delivery modes of the programme

Out of 826 projects included in the scope of this evaluation, in terms of the circumstances of the selected requests, most of them were related to implementing challenges identified in the context of economic governance processes, with the gradual increase of their share from 51% in 2017 to 62% in 2019, followed by some decrease of their share to 58% in 2020.

Table 15. Circumstances of the selected requests under the SRSP, 2017-2020

Source: SRSP annual monitoring reports 2017-2020.

Overall, there were 1,352 technical support activities planned for implementation under the four rounds of the SRSP (in 2017, 2018, 2018 and 2020). The number of annual activities increased from 281 under the 2017 round to 339 under the 2018 round and to 432 under the 2019 round, with some decrease in the volume of activities (to 316) under the 2020 round due to the selection of fewer larger projects. Similarly, the average number of activities per project increased from 2017 to 2020 and this increase continued across all the rounds – around 2.4, 2.8, 3.1 and 3.2 activities per project respectively in each round.

The monitoring system of DG REFORM contains 12 eligible activities of the SRSP. The analysis of the SRSP monitoring data indicates that the programme in 2017-2020 most often planned the following activities: study, research, evaluation (313 such activities out of 826 projects) and workshops, conferences and seminars (270 such activities). Long term expert mission(s) (26 activities) and organisation of local operations (27 activities) were the least frequently used activities in the technical support projects under the 2017-2020 rounds.

Figure 30: Number of activities planned for the implementation of the SRSP, 2017-2020

Source: DG REFORM internal monitoring system (N=1,352). Note: the categories of ‘other activities’ and ‘none of the activities’ were excluded from the analysis.

The technical support projects in 2017-2020 often planned more than one specific activity. A total of 24% of projects used only one activity, 21% of projects used two different activities, 22% of projects – three activities, 18% of projects – four activities, and 15% of projects – five or more different activities. The number of projects that planned the implementation of one specific activity decreased gradually over the annual rounds, while the number of projects having four, five or more activities increased substantially from 2017 to 2019. If the most common SRSP-funded project in 2017 often involved a single activity (usually ‘study, research, evaluation’ or ‘study visit’) or a combination of two activities (e.g. ‘workshop/conference/seminar’ and ‘study, research, evaluation’), the most common projects of the 2020 round combined three different activities besides the common activity of ‘study, research and evaluation’. Also, the 2020 projects became more complex in terms of outputs most frequently involving the combinations of 3-4 outputs (compared to 1-3 outputs in the 2017 projects) (see the table below).

Therefore, the increasing number of activities and their larger combinations per project show that the technical support projects became more complex over time. This is because the average size of the budget increased from the 2017 round to the 2020 round and, based on the interview data, the technical support projects became more comprehensive and ambitious in their design during the implementation of the SRSP. Also, the 2019 and 2020 rounds of the programme saw an increase in the volume of such operational activities as ‘collection of statistics’, ‘communication activities’ or ‘compilation/dissemination of material’. In contrast, the frequency of such specific capacity-building activities as ‘study visits’ or ‘short-term expert mission(s)’ remained at the similar levels in the 2019 and 2020 rounds compared to the previous rounds. 97 The growing complexity of technical support projects combined with an increased focus on operational activities in their design can make the implementation of these projects more difficult in the future, despite their possible relevance to the needs of Member States and beneficiary authorities.

Table 16. the most COMMON SRSP-funded projects in terms of activities and outputs during 2017-2020

Source: DG REFORM internal monitoring system (N=1,352).

Note: unique cases and combinations refer to the number of unique (the only one of its kind) activities and outputs found in the design of technical support projects. For instance, a total of 6 SRSP-funded projects only combined the specific activities of ‘workshop/conference/seminar’ and ‘study, research, evaluation’ in the 2017 round (with other 2017 projects having other types of activities in their design).

Furthermore, several delivery modes were used during the implementation of the SRSP. According to the SRSP Regulation, the specific mode of implementation is done either directly by the Commission or indirectly in accordance with the Financial Regulation 98 . Direct management includes public procurement contracts, grants and reimbursement of costs incurred by external experts. Fiscalis is also implemented in direct management by the Commission (Directorate-General for Taxation and Customs Union, DG TAXUD). Entities, such as international organisations, which have undergone an ex-ante assessment of their procedures and systems, guaranteeing a level of protection of the EU financial interests equivalent to that guaranteed by the Commission in direct management, can act under indirect management for the tasks they undertake to implement the actions. Technical Assistance and Information Exchange instrument (TAIEX), which provides short-term expertise exchange by public sector experts from the Member States, is not contracted but utilised through a Service Level Agreement between DG REFORM and Directorate-General for Neighbourhood and Enlargement Negotiations (DG NEAR). 99  

In total, there were 1,037 individual delivery modes applied during the programme’s implementation in 2017-2020. Procurement (416 cases; mostly implemented by private companies) and grants (332; often awarded to international organisations) were most often used to provide technical support to EU Member States. A technical support project can consist of a mix of different delivery modes to tailor the support to the needs of the Member State. A total of 78% of projects used one specific delivery mode, with 19% of projects applying two different delivery modes, and 3% of projects applying three or more different delivery modes. In addition, a total of 9 multi-country projects were implemented during the 2017-2020 rounds. 100

Figure 31: Use of main delivery modes during the implementation of the SRSP, 2017-2020

Source: DG REFORM internal monitoring system (N=1,037).

A substantial increase in the budget of the programme in 2019 and 2020 produced more procurements whose number rose from 80 in 2018 to 136 in 2019 and to 131 in 2020. For comparison, the relative share of this delivery mode (out of all the delivery modes that year) was 37% in the 2017 round, 34% in 2018, 43% in 2019 and 43% in 2020. The delivery mode of Fiscalis was applied only under the 2017 and 2018 rounds (15 times each cycle) in the sectoral category of revenue administration and public financial management.

Evidence from the interviews indicates that the selection of an individual delivery mode or a support provider depends on the needs of Member States and the nature of activities/outputs planned in each project. In cases when project activities require significant operational capacities (many experts and other resources on the ground) in specific countries, close contacts with national authorities and good knowledge of the national language, the framework contract involving the consortia of private sector companies is often used for the delivery of technical support. In contrast, when an exchange of international good practice and a transfer of knowledge from other countries is key to achieving the project objectives, international organisations are frequently selected as providers of technical support.

2.1.Main outputs and results achieved during the implementation of the programme 

The monitoring system of DG REFORM contains a list of outputs whose achievement is planned during the execution of technical support projects. In total, the programme was expected to produce 1,462 outputs in 2017-2020. The number of outputs per annual round grew from 300 (under the 2017 round) to 344 (under the 2018 round) and to 470 (under the 2019 round), with a decrease to 354 outputs (coinciding with a similar decrease of activities) under the 2020 round.

The analysis of the SRSP monitoring data indicates that the programme in 2017-2020 was most often planned to produce ‘recommendations’ (334 times), ‘analysis and reports’ (330 times), ‘workshops’ (234 times), as well as ‘action plans and roadmaps’ (207 times) as a result of implementing different technical support activities. The outputs of ‘recommendations’ and ‘analysis and reports’ were most frequently expected to be produced as a result of implementing the following two activities: (i) ‘study, research and evaluation’ and (ii) ‘workshop/conference/seminar’. The activity of ‘study, research and evaluation’ was also often expected to lead to the production of ‘action plans and roadmaps’. The least expected output in 2017-2020 was the ‘production of terms of reference’ (16 times). Similar types of outputs were found to be most frequently produced during the mid-term evaluation of the 2017-2018 SRSP in terms of design of action plans, roadmaps, guidelines, recommendations and strategies (88 times), seminars, conferences and workshops (69 times) and reports (including working visit reports) (64 times). 101 The biggest change in the number of planned outputs occurred in ‘communication strategy and other events’ whose volume tripled from 6 such outputs in each 2017 and 2018 round to 18 and 15 such outputs under the 2019 and 2020 rounds respectively.

The technical support projects were often expected to achieve more than one output in 2017-2020. In total, 20% of projects were expected to achieve only one output, 19% of projects – two outputs, 26% of projects – three outputs, 16% of projects – four outputs, and 19% of projects – five or more different outputs. The number of projects that were expected to produce one output gradually decreased over the annual rounds, while the number of projects expected to achieve five or more outputs substantially increased in 2019 and exceeded other categories of projects according to their outputs in 2020. Similarly to the number of technical support activities (see section 1.4 of this annex), this suggests an increase in the complexity and significance of projects in the SRSP’s implementation over time as they started addressing more important reform needs of the beneficiary Member States and they were expected to produce more value to the formulation and implementation of reforms with the availability of more financial resources per project. According to the results of the interviews, the financial size of the projects was also matched by a substantial improvement in the quality of requests for technical support from EU Member States, which was illustrated by the share of high-score requests among the selected requests.

Figure 32:Total expected outputs of the SRSP, 2017-2020

Source: DG REFORM internal monitoring system (N=1,462). Note: the categories of ‘other outputs’ and ‘none of the outputs’ were excluded from the analysis.

Furthermore, the monitoring system of DG REFORM contained the four groups of pre-defined outcomes (results). The analysis of the SRSP monitoring data indicates that the programme in 2017-2020 most often supported the achievement of the following outcomes: ‘adoption of (new) procedures and actions to enhance the implementation of reforms’ (447 times) and ‘improved internal working procedures, methodologies and processes, organisation’ (447 times). The outcome of ‘organisational change, change management, improved human resource management’ received somewhat less technical support from the programme (119 times). A similar trend was observed during the mid-term evaluation of the 2017-2018 programme, which found ‘adoption of new internal working procedures, methodologies and processes’ (29 times) to be the most frequently sought outcome, with the outcome of ‘improving the efficiency and effectiveness of human-resource management’ (9 times) sought least frequently. 102

Figure 33: Breakdown of the pre-defined outcomes in the SRSP, 2017-2020

Source: DG REFORM internal monitoring system (N=1,338).

In total, the number of expected outcomes grew from 193 under the 2017 round to 417 under the 2020 round. The number of expected outcomes increased in all pre-defined groups of outcomes over time, except for the outcomes related to ‘organisational change, change management, improved human resource management’ whose volume somewhat decreased from 2019 to 2020 (from 41 to 33). The technical support projects often had more than one pre-defined outcome. A total of 49% of projects pursued one pre-defined outcome, 32% of projects – two pre-defined outcomes, 11% of projects – three outcomes, and only 2% of projects – four outcomes.

More technical support was provided to achieving the outcome of ‘organisational change, change management, improved human resource management’ in the sectoral field of governance and public administration. The ‘adoption of a strategy or a new law’ was more frequent in the sectoral categories of sustainable growth and business environment, as well as financial sector and access to finance, which is expected, given that these projects were related to public interventions into the private sector.

Finally, there were some differences across the pre-defined outcomes in terms of their links to the specific objectives of the SRSP. For instance, technical support was more often provided to the ‘adoption of internal working procedures, methodologies and processes’ when these projects were contributing to the specific objective of ‘defining and implementing processes and methodologies’ (objective 3). Also, the ‘adoption of a strategy, a new law/act or modification of an existing one’ was more likely when the projects were pursuing specific objectives 1 and 2 related to ‘design of reforms’ or ‘enhancing capacity to implement reforms’. The ‘adoption of improved internal working procedures, methodologies and processes, organisation’ was frequently associated with ‘the production of guidelines and handbooks’ and the ‘delivery of training activities’ during the technical support projects.

Figure 34: Breakdown of groups of outcomes according to Lead Units, 2017-2020

Source: DG REFORM internal monitoring system (N=1,280). Note: the categories of ‘other outcomes’ and ‘none of the outcomes’ were excluded from the analysis.

(1)  Regulation (EU) 2017/825 of the European Parliament and of the Council, OJ L 129, 19.5.2017, p. 1–16, as amended by Regulation (EU) 2018/1671, OJ L 284, 12.11.2018, p. 3–5 (the SRSP Regulation). 

(2)

On 1 January 2020, DG REFORM succeeded the SRSS. For reasons of simplicity, this document refers only to DG REFORM.

(3)

The general objective as laid down in Article 4 of the SRSP Regulation reads as follows: ‘to contribute to institutional, administrative and growth-sustaining structural reforms in the Member States by providing support to national authorities for measures aimed at reforming and strengthening institutions, governance, public administration, and economic and social sectors in response to economic and social challenges, with a view to enhancing cohesion, competitiveness, productivity, sustainable growth, job creation, investment and social inclusion and to contributing to real convergence in the Union, which may also prepare for participation in the euro area, in particular in the context of economic governance processes, including through assistance for the efficient, effective and transparent use of the Union funds’.

(4)

Article 5(1) of the SRSP Regulation.

(5)  Commission staff working document Ex-ante Evaluation Accompanying the document Proposal for a Regulation of the European Parliament and of the Council on the establishment of the Structural Reform Support Programme for the period 2017 to 2020 and amending Regulations (EU) No 1303/2013 and (EU) No 1305/2013. The Ex-Ante Evaluation identified specifically two challenges/needs: (a) Limited administrative and institutional capacity (b) Inadequate application and implementation of Union legislation towards achieving the Union's fundamental goals.

(6)  According to the Commission’s Better Regulation Guidelines, a theory of change is oriented towards the objectives that are intended to be achieved under specific conditions, showing the ways and mechanisms that connect individual objectives, activities, and outcomes.

(7) Administrative capacity is considered in terms of structures, human resources, systems and tools, in line with European Commission (2020), Roadmaps for Administrative Capacity Building: Practical Toolkit. https://ec.europa.eu/regional_policy/sources/policy/how/improving-investment/roadmap_toolkit.pdf.

(8) European Commission (2015), Ex-ante Evaluation Accompanying the document Proposal for a Regulation of the European Parliament and of the Council on the establishment of the Structural Reform Support Programme for the period 2017 to 2020 and amending Regulations (EU) No 1303/2013 and (EU) No 1305/2013, p. 6. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52015SC0750&from=EN 

(9)  No/limited progress was achieved for 60.2% of CSRs, some progress was achieved for 38.7% of CSRs, and full/substantial progress was achieved for 1.1% of CSRs. European Parliament (2020), Country-specific recommendations: An overview – September 2020, p. 7. https://www.europarl.europa.eu/RegData/etudes/BRIE/2018/624404/IPOL_BRI(2018)624404_EN.pdf  

(10)  The number of total infringement cases stated in the ex-ante evaluation report was 1,347 at the end of 2014. Ibid, p. 6.

(11) European Commission (2015), Ex-ante Evaluation Accompanying the document Proposal for a Regulation of the European Parliament and of the Council on the establishment of the Structural Reform Support Programme for the period 2017 to 2020 and amending Regulations (EU) No 1303/2013 and (EU) No 1305/2013. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52015SC0750&from=EN 

(12) Ernst & Young (2018), Mid-term Evaluation of the Structural Reform Support Programme (SRSP) 2017- 2020 (2016 ECFIN 009/A). Publications Office of the European Union, p. 13, 17. https://doi.org/10.2887/656262 This analysis was based on the information available on Jira, consulted on 30 April 2019.

(13) The consultation conducted during the mid-term evaluation involved an online targeted consultation and an in-depth analysis of 30 implemented projects. Opinions were collected from the beneficiaries and the providers of technical support during the consultation process. Ibid, p. 17. 

(14) The provision of support under Article 11 (support to be funded through additional voluntary contributions) follows a different timeline, linked to the transfer of the additional contributions from the Member State concerned.

(15) Urgency, breadth and depth of the problems identified, the support needs in the relevant policy area, the analysis of socioeconomic indicators, and the administrative capacity of the beneficiary authority concerned.

(16)  The external contractor used data from the internal monitoring system of DG REFORM (JIRA) to compile this Report.

(17) Technically a project is a ‘support measure’ under the SRSP Regulation terminology.

(18)  The present figure of 826 used in the ex-post evaluation refers to the actual number of funded projects (i.e., the total number of the projects funded during 2017-2020 after merging some of the selected requests), excluding the projects implemented by the contractor.

(19)  European Commission (2020), Commission Staff working document: Mid-term evaluation of the Structural Reform Support Programme. https://ec.europa.eu/info/sites/default/files/report_en_final.pdf 

(20) Within the scope of the SRSP, recommendations are non-binding suggestions that are addressed to the beneficiary authorities to tackle the problems they highlighted in their request for support, or to promote their reform agenda.

(21) The number of annual activities increased from 281 under the 2017 round to 339 under the 2018 round and to 432 under the 2019 round, with some decrease in the volume of activities (to 316) under the 2020 round due to the selection of fewer larger projects. Similarly, the average number of activities per project increased from 2017 to 2020 and this increase continued across all the rounds – around 2.4, 2.8, 3.1 and 3.2 activities per project respectively in each round. The monitoring system of DG REFORM contains 12 kinds of SRSP eligible activities. 

(22) Ernst & Young (2018). Mid-term Evaluation of the Structural Reform Support Programme (SRSP) 2017-2020 (2016 ECFIN 009/A). Publications Office of the European Union, p. 159. https://doi.org/10.2887/656262. Since this analysis was based on the results of a survey, the formulations of the expected outcomes do not fully correspond to those provided in the monitoring system of DG REFORM. 

(23) N=128; with the total number of outcomes being 237.

(24)  Interestingly, the achievement of the first project outcome was above the average (the score of 8.45), while other outcomes came below the average (ranging from 8.07 to 7.55), pointing to potential difficulties in the implementation of more complex projects.

(25) According to the targeted survey of the external study, 88% of beneficiary authorities strongly agreed or agreed (109 out of 124) that the providers of technical support had the required expertise and skills, and 86% of them accordingly agreed that they had delivered according to the expected quality (106 out of 123). These findings were also supported by those coordinating authorities and beneficiary authorities who participated in the focus group discussions, who agreed that the international experts selected by the Commission had the right competencies.

(26)  According to the results of the case studies.

(27)  According to the targeted survey of beneficiary authorities, as many as 30% (39 out of 131) of all evaluated projects were still ongoing during the ex-post evaluation. Only 31% (40 out of 129) of all surveyed projects achieved all results by May-June 2022. This is in line with the monitoring data of DG REFORM that 79% of all SRSP projects were finished in September 2022.

(28)

Based on the targeted survey of beneficiary authorities 76% (100 out of 129) of projects implemented under the SRSP concerned policy development (analysis of the situation, identification of best practices, definition of a strategy, advice on policy options and recommendations, etc.)

(29)  European Commission (2015), Ex-ante Evaluation Accompanying the document Proposal for a Regulation of the European Parliament and of the Council on the establishment of the Structural Reform Support Programme for the period 2017 to 2020 and amending Regulations (EU) No 1303/2013 and (EU) No 1305/2013. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52015SC0750&from=EN 

(30)  46% (41 out of 89) beneficiary authorities agreeing to a large extent or to some extent.

(31)  According to the targeted survey data

(32)

Based on the Commission’s (from DG ECFIN) methodologies for assessing the impact of structural reforms, we drew a difference between product market reforms (addressing cost of starting a business; administrative burden; sectorial regulation; and access to finance); labour market (including education) reforms (unemployment benefit reforms; reforms to active labour market policies; education reforms; etc.); as well as fiscal, tax and pension reforms. European Commission (2016), The Economic Impact of Selected Structural Reform Measures in Italy, France, Spain and Portugal, European Economy. Institutional Paper 23. https://doi.org/10.2765/195899

(33) See Annex II: Methodology and analytical models used

(34)

A total of 78% (91 out of 116) of beneficiary authorities and 89% of technical support providers (87 out of 98) from the survey programme of the external contractor.

(35)  Greece, Croatia, Romania, Italy, Cyprus, Latvia, Poland, Lithuania; each of these countries implemented more than 40 projects during this period.

(36)  A total of 79% (89 out of 113) of beneficiary authorities and 83% of technical support providers (79 out of 95)

(37) 63% (65 out of 104) of beneficiary authorities and 73% (66 out of 91) of technical support providers, stated a moderate or strong positive effect respectively.

(38)  A total of 51% of beneficiary authorities (55 out of 107) and 63% of technical support providers (55 out of 87) suggested that the COVID-19 pandemic was an important factor having a moderate or strong negative effect on the delivery of technical support. The focus group discussions and interviews conducted with national stakeholders indicated that the pandemic delayed the implementation of some activities or weakened the ownership of some projects because of a lack of in-person meetings and face-to-face interaction.

(39)  38% (40 out of 105) of beneficiary authorities from the targeted survey

(40) European Commission (2020), Understanding the Political Economy of Reforms: Evidence from the EU: Technical note for the Eurogroup, Brussels, p. 4-5. https://www.consilium.europa.eu/media/45511/ares-2020-4586969_eurogroup-note-on-political-economy-of-reforms.pdf

(41)  According to the targeted survey data, a total of 56% (50 out of 89) of technical support providers recognised the positive effect of political reform commitments, while 52% (47 out of 91) of them admitted the relevance of timing and sequencing of reforms (having a moderate and strong positive effect).

(42)  For instance, due to the unstable political situation, Bulgaria did not submit its TJTP to the Commission by May 2022, despite the fact that the finalisation of this plan was the expected specific outcome of its technical support project (‘Support for the Preparation of Territorial Just Transition Plans’ under the 2020 dedicated call).

(43) European Commission (2020), Understanding the Political Economy of Reforms: Evidence from the EU: Technical note for the Eurogroup. https://www.consilium.europa.eu/media/45511/ares-2020-4586969_eurogroup-note-on-political-economy-of-reforms.pdf

(44) A total of 64% of technical support providers (61 out of 96), 62% of coordinating authorities (13 out of 21), as well as 55% (53 out of 96) of beneficiary authorities (with 36% of these authorities, 35 out of 96, indicating a neutral effect) agree and strongly agree that national ownership of reforms was a positive factor in the delivery of outputs and results.

(45)

 The budget of SRSP annual calls increased 3.8 times from 2017 (EUR 22.5 million) to 2020 (EUR 84.9 million), while the number of selected requests grew only 1.4 times (from 159 in 2017 to 228 in 2020). 

(46)

From the targeted survey of beneficiary authorities, coordinating authorities and technical support providers revealed that 93% of beneficiary authorities (107 out of 115) and 89% of national coordinating authorities (17 out of 19) although the corresponding share of technical support providers was somewhat lower but still high (80%, 79 out of 99). However, only one technical support provider (out of 99) from this survey indicated that the project budget was too small, which had a strong negative effect on the delivery of the expected outputs and results.

(47)

The analysis of the OPC data revealed that more than half of OPC respondents perceived SRSP actions as user-friendly, timely, and cost-effective to a moderate and high extent compared to other EU actions (17 out of 26 respondents) and national actions.

(48)  17 out of 26 respondents

(49)  18 out of 26 respondents

(50) Note that the estimated costs of controls over the total annual payments progressively decreased over the evaluation period considering the progressively increasing value of the total payments made.

(51)

 For instance, the programme management costs (the ratio between their administrative and operational budgets in terms of executed payment appropriations) of the Commission’s Executive Agencies on average constituted 4.25% in 2020 (based on the Annual activity reports 2020 of INEA, CHAFEA, REA, EACEA, ERCEA and EASME). The programme management costs in the agencies ranged from 0.75% (in the case of the Innovation and Networks Executive Agency) to 8.65% (in the case of the Consumers, Health, Agriculture and Food Executive Agency); these differences were related to the specific characteristics of the programmes delegated to the respective agencies (such as the average project size, the competition rate in the application and evaluation stage, the heterogeneity of applicants and beneficiaries, the complexity and diversity of the agency’s programme portfolio, etc.) Similarly, the Commission's administrative expenditure (programme management costs) can constitute up to 5% of the Horizon 2020 overall budget.

(52) Regulation (EU, Euratom) 2018/1046 of the European Parliament and of the Council of 18 July 2018 on the financial rules applicable to the general budget of the Union, amending Regulations (EU) No 1296/2013, (EU) No 1301/2013, (EU) No 1303/2013, (EU) No 1304/2013, (EU) No 1309/2013, (EU) No 1316/2013, (EU) No 223/2014, (EU) No 283/2014, and Decision No 541/2014/EU and repealing Regulation (EU, Euratom) No 966/2012. Official Journal of the European Union, L 193/1, 30.7.2018. http://data.europa.eu/eli/reg/2018/1046/oj.

(53) PPMI and CSES (2020), Study supporting the evaluation of the Research Executive Agency (2015-2018): final report. Publications Office, https://data.europa.eu/doi/10.2777/299516.

(54)  PPMI (2020), Study supporting the evaluation of the Education, Audiovisual and Culture Executive Agency (2015 – 2017).

(55) Other examples: The ‘time-to-inform’ for the main calls in 2015-2018 within programmes managed by the European Research Council Executive Agency varied from 133 days to 237 days (PPMI and CSES (2020), Study supporting the evaluation of the European Research Council Executive Agency (2015-2018): final report. Publications Office, https://data.europa.eu/doi/10.2777/487469). Within the Urban Innovative Actions, the ‘time-to-inform’ was between 157 and 187 days (CSES (2021), Assessment Study of the Urban Innovative Actions 2014-2020: final report. Publications office, https://data.europa.eu/doi/10.2776/5314.

(56) Time taken from the conclusion of contracts/agreements with technical support providers to the completion of projects.

(57) The amount of SRSP payment appropriations grew constantly during 2017-2020, and the execution level of payment appropriations rose from 67% in 2017 to 97% in 2018 and nearly full execution in 2019-2020

(58)  The majority of OPC respondents (19 out of 26) considered the proportionality of the administrative burden as a moderately or highly suitable factor in making the SRSP`s implementation efficient.

(59) The case studies confirmed that the administrative arrangements under the SRSP were overall adequate and did not impose an excessive administrative burden on either technical support providers or beneficiaries.

(60) It is important to note that the SRSP and ESIF fall under different implementation modes. The SRSP is implemented in a direct management mode, and the European Commission is directly responsible for all steps in the programme's implementation. The ESIF operational programmes are implemented in a shared management mode where the European Commission and national authorities jointly manage the funding. Under the ESIF, Member States' administrations select which projects to finance and take responsibility for the day-to-day management; the final beneficiaries are usually responsible for contracting and management of the specific ESIF projects. Further, the ESIF projects generally require co-financing by the Member States, whereas SRSP does not require co-financing. Therefore the instruments are not directly comparable.

(61)  The efficient and effective coordination activities with other DGs showed that 100% of the other DGs were satisfied with the coordination process in 2018 (15/15) and 2019 (10/10). The corresponding result for 2020 is 95%, however, it relates to the coordination process for the TSI round.

(62) DG REFORM – Directorate-General for Structural Reform Support (2021), Annual Activity Report 2020.https://ec.europa.eu/info/sites/default/files/annual-activity-report-2020-structural-reform-support_en.pdf.

(63) It must be noted that the technical assistance under the ESIF was mainly focused on supporting the measures for the implementation of funds, while the support aimed to enhance the institutional capacity of public authorities and stakeholders and efficient public administration was provided under thematic objective 11.

(64)

This evidence was also supported by the OPC results where 18 out of 26 respondents indicated that the SRSP complemented (by funding different aspects of similar activities, targeting different groups, etc.) most EU cohesion policy instruments (the ERDF, the Cohesion Fund, the ESF). 13 out of 26 respondents indicated that SRSP complemented the Horizon 2020 programme.

(65)  According to the targeted survey results, more than half (54% of those who offered an opinion on the issue, N=44) of surveyed beneficiary authorities indicated that the programme to a high/moderate extent complemented the actions of similar national and regional programmes (e.g., by addressing the same issues, target groups, aiming at the same objectives, etc.). Similarly, 11 out of 26 OPC respondents indicated that the SRSP complemented the actions of similar national and regional programmes to a high/moderate extent (in addition, 13 stakeholders indicated they had no opinion/cannot answer the question, with only 2 respondents indicating there was no complementarity).

(66)  For example, the Fiscalis programme featured multilateral and joint actions, including sharing good practices and training for national tax administrations to fight against tax fraud, tax evasion and tax avoidance. However, Fiscalis did not feature tailored support provided for addressing the specific needs of individual Member States. Therefore, the SRSP complemented Fiscalis by providing Member States with such tailored support, e.g. in the area of tax administration reforms.

(67)

It must be noted that in the SRSP mid-term evaluation, the assessment of the share of projects related to European Semester recommendations was based on the contractors’ analysis of the administrative data from the Jira system available at the time, i.e. not the evidence from beneficiary authorities’ targeted survey. 

(68)  30 out of 61 beneficiary authorities that indicated that their project addressed CSRs at least to a moderate extent, also indicated that their technical support project(s) were delivered in this thematic area.

(69) Moreover, the dedicated call on the preparation of the TJTPs particularly strengthened the programme’s contribution to the EU priority of building a climate-neutral, green, fair and social Europe because these plans fall under the Just Transition Mechanism (JTM) that is an integral part of the European Green Deal.

(70)

The majority of OPC respondents (11 out of 26) indicated that Member States could have achieved similar results in their design and implementation of national reforms without the technical support they received through the SRSP only to a limited extent or not at all. In addition, 9 out 26 stakeholders indicated they had no opinion/cannot answer the question, with 6 out 26 stakeholders considering that Member States could have achieved similar results in their design and implementation of reforms without the technical support they received through the SRSP.

(71)  Answers to open-ended questions indicate that beneficiary authorities perceived the following as some of the most important sources of added value provided by DG REFORM: (i) expertise and support when assessing the needs, (ii) defining project objectives and terms of references, (iii) planning the timing of project activities and deliverables, reviewing and supervision of the quality of project deliverables, and (iv) setting requirements for experts.

(72) The majority (18 out of 21, or 86%) of coordination authorities who responded (respondents) agreed/strongly agreed that the technical support increased the sharing of knowledge and good practices between beneficiary authorities in their country and institutions from other countries. Similarly, 77% (17 out of 22) of the respondents agreed/strongly agreed that the technical support increased mutual trust between beneficiary authorities in their country and the Commission, with another 88% (15 out of 17) agreeing/strongly agreeing that the technical support helped to address cross-border or Union-wide challenges in their country.

(73)  Interviews with stakeholders also indicate that some delivery modes (e.g. TAIEX focusing on peer-to-peer cooperation) were particularly well-suited for the sharing of good practices and the exchange of information between different national authorities.

(74)  During the period 2017-2020, there were only 9 multi-country SRSP-funded projects out of 826 total projects covered in the evaluation (i.e., around 1%).

(75)  Around 98% of the surveyed beneficiary authorities and coordinating authorities (149 out of 152) agreed or strongly agreed that the SRSP was a suitable instrument to provide technical support, around 96% (144 out of 150) agreed that their project design (identification of the problem, definition of the objectives and results to achieve, definition of the activities to carry out) was appropriate. Similarly, technical support providers very positively assessed the design and structure of the SRSP, with 98% of them (101 out of 104) agreeing the SRSP was a suitable instrument to provide technical support and 97% of them (101 out of 105) agreeing that the project design was appropriate.

(76) There is a significant statistical relation between the assessment of the project's design in terms of its relevance for addressing the needs of the Member State and the assessment of the quality for technical support requests by the policy officers. More specifically, there is a strong positive correlation between the assessment by policy officers of the project design's relevance for the Member States’ needs and the assessment of the maturity (to allow a quick delivery of the support) of the requests for the technical support submitted by the Member States. Similarly, there is a strong correlation between the assessment of the project design's relevance for the Member States needs and the assessment of the measures requested by the Member States (i.e., whether they were targeted enough for addressing the reform needs).

(77)  1,786 cases at the end of 2020 compared to 1,347 cases at the end of 2014, European Commission (2021), Commission Staff Working Document: General Statistical Overview Accompanying the document ‘Report from the Commission Monitoring the application of European Union law 2020 Annual Report’. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52021SC0212&rid=10 

(78) Case studies’ projects were not chosen on the basis of circumstances of requests.

(79)  For instance, Italy’s project (‘Support for the Implementation of the Accrual IPSAS/EPSAS-based Accounting in the Italian Public Administration’) on accounting reforms related to the European Public Sector Accounting Standards (EPSAS) and International Public Sector Accounting Standards (IPSAS) was closely in line with national reform priorities. Similarly, the SRSP-funded project in Ireland (‘Financial Reporting Reforms’) closely corresponded to the pending reform needs of the country.

(80)

Based on the responses to the outcome questionnaires by the beneficiary authorities, the average rating for having achieved the specific outcomes of the SRSP-funded projects was 8.2 out of 10 (with 66% of technical support projects achieving their outcomes, based on average scores 8.0 and above). In the targeted survey of beneficiary authorities, 31% of respondents achieved all expected results and 65% of respondents achieved some expected results, both during the implementation of the technical support projects and after their completion.

(81) On the other hand, multicountry projects were able to increase efficiency by providing synergies between the authorities involved.

(82) European Commission, ‘Better Regulation’ toolbox 2021, Chapter 6 – How to carry an evaluation and fitness check, p. 389.

(83) Since the provider of the study received a separate monitoring dataset on the application of different delivery modes (sub-tasks) per project, it became necessary to merge it with the main monitoring dataset.

(84) European Commission, Ex-ante Evaluation Accompanying the document Proposal for a Regulation of the European Parliament and of the Council on the establishment of the Structural Reform Support Programme for the period 2017 to 2020 and amending Regulations (EU) No 1303/2013 and (EU) No 1305/2013; Ernst & Young (2018), Mid-term Evaluation of the Structural Reform Support Programme (SRSP) 2017- 2020 (2016 ECFIN 009/A). Publications Office of the European Union, p. 13, 17. https://doi.org/10.2887/656262

(85) The data excludes information on projects implemented by PPMI or CSES.

(86) While some interviews involved a few participants, the count refers to the total of number of interviews carried out.

(87) Factual Summary Report, summarising results of the Open Public Consultation “Structural reform support programme – final evaluation“. https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/12790-Structural-reform-support-programme-final-evaluation/public-consultation_en

(88) For example, in the statement ‘According to the targeted survey of beneficiary authorities, as many as 30% (39 out of 131) of all evaluated projects were still ongoing during the ex post evaluation’ we refer to the results of the targeted survey in the following way: 131 (out of 132, excluding “Do not know/cannot answer” responses) beneficiary authorities responded to this targeted survey question, with 39 of them (or 30%) claiming that their projects were still ongoing during the ex post evaluation.

(89)  According to the SRSP Regulation, the European Commission should provide an independent ex post evaluation report by 31 December 2021. Planned start and finish dates of this evaluation were moved forward in the evaluation roadmap. The ex post evaluation was completed in the beginning of October 2022.

(90)  Where there is a prior impact assessment, the table should contain as a minimum the costs/benefits identified in the IA with the information gathered on the actual cost/benefit. As available, the table should include the monetisation (€) of the costs/benefits based on any quantitative translation of the data (time taken, person days, number of records/equipment/staff etc. affected or involved represented in monetary value – see Standard cost model, for example). For all information presented, it should be included in the comments section whether it relates to all Member States or is drawn from a subset. An indication of the robustness of the data should be provided in Annex II on Methodology and analytical models used.

(91) Each simplification/saving should be included on a separate line.

(92) This assessment is without prejudice to a possible future Impact Assessment.

(93) A total of 126 stakeholders were contacted and informed about the OPC.

(94) EU cohesion policy instruments, Horizon 2020, LIFE programme, Customs 2020, Fiscalis 2020 and Hercule III programme.

(95) EU cohesion policy instruments (the European Regional Development Fund, the Cohesion Fund, the European Social Fund), Horizon 2020, LIFE programme (the EU’s funding instrument for the environment and climate action), Customs 2020 (enables national customs administrations to create and exchange information and expertise), Fiscalis 2020 (enables national tax administrations to create and exchange information and expertise), Horizon 2020 (the EU framework programme for research and innovation) and Hercule III programme (combats irregularities, fraud and corruption affecting the EU budget).

(96) Three answer options specific for different groups of participants, namely, objectives and the scope of the reforms initially planned, the quality of the technical support request design and bureaucratic resistance to change during the implementation of the project(s) activities, were excluded to ensure comparability of responses across the groups.

(97) It is possible that the providers of technical support (both under public procurement and grants) more frequently contracted the services of individual experts in the 2019 and 2020 rounds, making the execution of such individual activities as ‘short-term expert mission(s)’ less clearly visible in the design of technical support projects. This issue could be assessed during the evaluation of the TSI in the future.  

(98) Regulation (EU, Euratom) 2018/1046 of the European Parliament and of the Council of 18 July 2018 on the financial rules applicable to the general budget of the Union, amending Regulations (EU) No 1296/2013, (EU) No 1301/2013, (EU) No 1303/2013, (EU) No 1304/2013, (EU) No 1309/2013, (EU) No 1316/2013, (EU) No 223/2014, (EU) No 283/2014, and Decision No 541/2014/EU and repealing Regulation (EU, Euratom) No 966/2012. Official Journal of the European Union, L 193/1, 30.7.2018. http://data.europa.eu/eli/reg/2018/1046/oj

(99)  European Commission (2020), Commission Staff working document: Mid-term evaluation of the Structural Reform Support Programme. https://ec.europa.eu/info/sites/default/files/report_en_final.pdf

(100) Since the SRSP Regulation did not explicitly provide for the possibility of submitting multi-country requests from EU Member States, this way of delivering technical support was underdeveloped during the period 2017-2020.

(101) Ernst & Young (2018). Mid-term Evaluation of the Structural Reform Support Programme (SRSP) 2017-2020 (2016 ECFIN 009/A). Publications Office of the European Union, p. 96, 158. https://doi.org/10.2887/656262 This analysis was based on the results of a survey.

(102) Ernst & Young (2018). Mid-term Evaluation of the Structural Reform Support Programme (SRSP) 2017-2020 (2016 ECFIN 009/A). Publications Office of the European Union, p. 159. https://doi.org/10.2887/656262. Since this analysis was based on the results of a survey, the formulations of the expected outcomes do not fully correspond to those provided in the monitoring system of DG REFORM.