52023SC0071

Glossary

Term or acronym	Meaning or definition
CCI	Character Code Identifier
EC	European Commission
ERDF	European Regional Development Fund
ESF	European Social Fund
ESFRI	European Strategy forum on Research Infrastructures
ESIF	European Structural and Investment Funds
EQ	Evaluation Question
EU	European Union
EU13	The following Member States: Czechia, Estonia, Cyprus, Latvia, Lithuania, Hungary, Malta, Poland, the Slovak Republic, Slovenia, Romania, Bulgaria, and Croatia.
EU15	The following Member States: Austria, Belgium, Denmark, Finland, France, Germany, Greece, Ireland, Italy, Luxembourg, the Netherlands, Portugal, Spain, Sweden, and the United Kingdom.
EUR	Euro (currency)
FP7	Seventh Framework Programme
GDP	Gross Domestic Product
H2020	Horizon 2020
HEI(S)	Higher Education Institution(s)
ICT	Information and Communication Technology
I3 Instrument	Interregional Innovation Investments Instrument.
MA(s)	Managing Authority(ies)
MS	Member State(s)
NUTS	Nomenclature des Unités Territoriales Statistiques
OP(s)	Operational Programme(s)
R&D	Research and Development
RDI	Research Development and Innovation
RTD	Research and Technological Development
RTO	Research and Technology Organisation
S3	Smart Specialisation Strategies
SME	Small and Medium-Sized Enterprise
SWD	Staff working document
ToC(s)	Theory of Change(s)

1.Introduction

1.1.Context and purpose

The general objective of this SWD is to report on the ex-post evaluation of investment in Research and Technological Development (RTD) infrastructures and activities supported by the European Regional Development Fund (ERDF) in the period 2007-2013. To carry out the evaluation, the European Commission, Directorate-General for Regional and Urban Policy, contracted CSIL (Centre for Industrial Studies), in partnership with Prognos AG and Technopolis Consulting Group Belgium. The contract started on 1 September 2019 and the final report was published on 10 December 2021 1 . This SWD is based on that work.

1.2.Scope of the evaluation

In total, for the period 2007-2013 around EUR17 billion of ERDF resources (5% of the total allocation) were invested through 215 Operational Programmes in projects supporting RTD 2 infrastructure, competence centres and activities in Member States and regions. Based on reporting by Managing Authorities on categories of expenditure, around EUR11 billion of that sum went to support for infrastructure and around EUR6 billion to support for activities.

This evaluation focused on 53 Operational Programmes (OPs) selected by the European Commission out of the total of 215 OPs funded by the ERDF. These cover 18 of the 28 Member States involved, and a substantial amount of RTD investments (with EUR14.64 billion in funding, i.e. about 85% of the EU total funding for the relevant themes) under diverse contexts. Within the selected OPs, the evaluation study focused on two categories of ERDF expenditure (01 – RTD activities in research centres, and 02 – RTD infrastructure and centres of competence in a specific technology) 3 . Public support for investment in firms directly linked to research and innovation was not covered by the evaluation 4 .

Different levels of analysis were considered:

·OP level: this level analyses the strategies and policy mixes put forward by the OPs, their linkage with specific contexts and their linkages with other policies. This was done in a more general way for the 53 OPs and in a more specific way for the OPs analyses within seven case studies

·Country-level: this level reviews, in detail, the use of policy instruments for RTD in different national contexts (including relevant OPs), as well as the rationale underpinning the policy mix. It was addressed in seven case studies;

·Instrumental level: this level analyses the theories of change (chains of effect and mechanisms) of selected types of intervention, allowing the identification of ‘lessons learnt’ and providing evidence to support debates and future policy considerations;

·Project and beneficiary level: this level of analysis explores and describes in detail the diversity of RTD projects funded under Cohesion Policy in the 53 representative OPs, covering 18 Member States. It also classifies them according to well-defined typologies.

Looking at different levels of analysis yielded complementary benefits, providing different perspectives of analysis.

Figure 1.Representative sample of 53 Operational Programmes and Member States

Source: Ex-post evaluation report (2021)

1.3.Structure of the SWD

In compliance with Better Regulation Guidelines, the remaining sections are organised as follows:

·Section 2 describes the expected outcome of the intervention. In particular, 2.1 focuses on ERDF priorities, while 2.2 describes the intervention logic.

·Section 3 outlines how the situation evolved over the evaluation period. More specifically, 3.1 describes the actual funding allocation, 3.2 reports on the key achievements, 3.3 reconstructs the causal links, and 3.4 examines external factors.

·Section 4 reports on the evaluation findings. The section is organised by evaluation criteria (effectiveness, efficiency, coherence, EU added value, relevance).

·Section 5 summarises the main conclusions, policy considerations, and open questions.

·The Annexes provide complementary information on procedures (Annex I), methodology (Annex II), evaluation matrix (Annex III), costs and benefits (Annex IV), and stakeholder consultation (Annex V).

2.What was the expected outcome of the intervention?

2.1.Background and needs

ERDF priorities in the RTD field

The Europe 2020 strategy adopted in 2010 put research, development and innovation at the top of the EU agenda for smart, sustainable and inclusive growth. In that period, R&D expenditure was increasing across the EU but was still considerably below the 3% target of investment in R&D as a share of gross domestic product (in 2007, this share ranged from a maximum of 3.35% for Finland to less than 1% in most Central and Eastern European countries, the only exceptions being Czechia with 1.31% and Estonia with 1.07%) 5 .

Since then, despite a yearly increase of 1% since 2000, R&D expenditure in the EU has remained below the set target. Its share of world expenditure on R&D has declined compared with the EU’s global competitors. Only Germany, Denmark and Cyprus reached their targets, with the greatest progress reported by Poland, Greece, Estonia, Hungary and Czechia (European Commission, 2020).

To contribute to the overarching goal of making the EU a leading knowledge-based economy, the Community strategic guidelines for 2007-13 emphasised the need to increase and improve investment in research and technological development (RTD). 6 Priority was allocated to the support of existing poles of excellence, making better use of existing potential and avoiding the excessive spatial dispersion of resources. Increasing private and public investment in RTD and innovation also encouraged partnerships across the different regions of the EU. Meanwhile, creating and exploiting a larger pool of high-quality research talent in Europe was identified as a key strategy. In line with the systemic approach to innovation, RTD infrastructures, competence centres, and activities were considered necessary to help structure the scientific community and contribute to the construction of an efficient research and innovation ecosystem. The role of research infrastructures as key to enhancing national and regional RTD capacities was also emphasised. The development of pan-European research infrastructures and their impact on the regional ecosystems were seen as a key driver of economic growth.

R&D investments were closely linked to the objective of fostering regional innovation. The report on ‘Creating an Innovative Europe’ 7 stressed the key role of the regional level to foster an innovative Europe and underlined the choice to focus a large share of Cohesion Policy resources for 2007-2013 on the Innovation priority. About EUR86.4 billion, nearly 25% of the total allocated budget, went towards innovation in the broader sense, including research centres and infrastructure, technology transfer and innovation in firms, the development and diffusion of information and communication technologies, and human capital development. These investments represented more than a tripling of absolute financial resources dedicated to innovation and R&D compared with the previous period (2000-2006). The amount also greatly exceeded the budget of the 7th Framework Programme for Research (EUR50.5 billion) and the Framework Programme for Competitiveness and Innovation (EUR3.6 billion).

Although still significant, looking at only the RTD component of the overall financial resources, the figure is much lower. In the period 2007-2013, around EUR17 billion of ERDF resources (almost 5% of the total ERDF allocation) was invested through 215 OPs in projects supporting RTD infrastructure, competence centres and activities in EU Member States and regions (codes 01 and 02). More than EUR11 billion (65.5% of the total) was allocated to research infrastructure support (code 02) and around EUR5.8 billion (34.5% of the total) to research activities support (code 01).

Investments needs and regional differences

The analysis of the 53 OPs (documentary review complemented by direct interviews with the Managing Authorities) shows that the 53 OPs reviewed generally did not refer to specific, individual rationales of RTD investments, but they recognised multiple investment needs.

Although different formulations were used in programming documents, two consisted of the key barriers identified by policymakers. On the one hand, there was the lack of a critical mass of infrastructure endowments and research capacities to enable the production of top-class research. On the other, there was the need to increase the industrial relevance of the regional science base by linking existing or emerging poles of scientific excellence to areas of industrial strength. While the lack of density of research capacities seemed more relevant for the EU13 countries, the asserted needs of the EU15 MSs related more commonly to better science-industry linkages.

Figure 2.Rationale for RTD investment by type of country – share of EU13 and EU15 OPs mentioning ‘need’

Note: Each OP can be associated with more than one need. The percentage was calculated as the number of EU13 (or EU15) OPs mentioning need(s) over the total number of EU13 (or EU15) OPs. The total number of EU13 OPs was 16, while the total number of EU15 OPs was 37.

Source: Ex-post evaluation report (2021), based on a review of the OP programming documents

The diagnosis reflected in the programming documents is not surprising. It mirrors fairly accurately the situation at the beginning of the programming period, as highlighted by the cluster analysis and the literature and policy documents (European Commission, 2009; Radosevic and Lepori, 2009). For all the Central and Eastern European countries that joined the EU in 2004 and 2007, the 2007-2013 Operational Programmes represented the first systematic set of interventions addressed in the research field. They were facing the most considerable challenges: the public-research system was fragmented, with research and higher education systems being split between academies of science and universities; research technologies were obsolete; collaboration with industry was minimal, due to a lack of strategic awareness of the importance of innovation among companies, and due to insufficient orientation of research towards industry needs. The main problem was linked to education and research infrastructures, which were in poor condition and outdated. Therefore, they failed to contribute to top-level research and educational activities, thereby limiting their attractiveness for companies and international research networks.

Compared with EU13 Member States, in the selected Western European countries, regional innovation systems were, overall, relatively more mature. They had a stronger network of universities and research centres and some already existing structures to favour technological diffusion and science-industry collaboration, such as clusters, poles of excellence, and science-and-technology parks. Nonetheless, there were numerous differences across both countries and regions, both in terms of perceived RTD needs and designed strategic approaches. On the one hand, Germany was one of the 10 most research-intensive economies worldwide, with only limited development needs in certain areas of R&D in convergence regions. Here, traditionally strong SMEs played a notable role in innovation, and there was intensive historical cooperation between the science and enterprise sectors. On the other hand, Portugal and Italy were still follower countries. Italy, in particular, suffered from significant regional disparities beyond a generalised weak research system at the national level. This problem also stemmed from a long tradition of scarce mobilisation of financial resources for research.

Figure 3.RTD regional contexts: results of a cluster analysis - 2007

Source: Ex-post evaluation report (2021)

These agglomeration effects were not surprising. Indeed, widespread empirical evidence shows that research and innovation activities tend physically and spatially to agglomerate in specific geographical areas. Some studies (e.g., Falk, Hölzl and Leo, 2007) highlight how the nationally fragmented nature of the public research system can contribute to innovation gaps between regions. This fragmentation does not enable the exploitation of scale effects and/or efficiency in RTD investments (particularly for large infrastructures) that would otherwise derive from the concentration of resources on selected priorities.

2.2.Expected outcome

Infrastructure investments

Infrastructure investments were selected as a result of a top-down approach, with national authorities undertaking needs assessments, and national or EU roadmap exercises, as part of the implementation of the European Research Area (ERA). The extent to which the needs assessment was reflected in the selection of funded infrastructures varied. The roadmap exercises followed a largely top-down approach and identified national infrastructure of pan-European relevance, contributing to the construction of the ERA and the identification of European investment priorities by the European Strategy forum on Research Infrastructures (ESFRI). For example, the ELI Beamlines Facility was planned as part of the European roadmap of next-generation major research facilities that the ESFRI Forum prepared (ESFRI Roadmap)

Exceptions in infrastructure planning and selection were represented by the ICT-based infrastructures in higher education institutions (HEIs), and the Major Project analysed in the German case study. The former was the result of a thorough bottom-up and top-down analysis.

The funded infrastructure projects differed in their underlying rationales. A more in-depth analysis of funded projects and relative beneficiaries showed a wide variation in their design and implementation, reflecting different objectives.

In the Seventh Framework Programme of the European Community for research, technological development and demonstration activities (FP7) 8 , and later within Horizon 2020 Regulation (EU) No 1291/2013 of 11 December 2013, the term ‘research infrastructure’ referred to ‘facilities, resources, and services used by the research communities to conduct research and foster innovation in their fields. They include major scientific equipment (or sets of instruments), knowledge-based resources such as collections, archives and scientific data, e-infrastructures, such as data- and computing systems and communication networks, and any other tools that are essential to achieve excellence in research and innovation’ 9 .

This category of interventions accounted for the largest share of ERDF resources in the OPs under consideration, in line with the fact that most OPs indicated infrastructural failures as one of the key investment needs to be tackled by the OP in question. The specific nature of the projects funded ranged from support for new or reconstructed infrastructure, such as buildings, plants or laboratories, to investment in research-related equipment, such as lab instruments, machinery or highly specialised apparatus, as well as supporting infrastructure. These instruments addressed the lack of sufficient or modern physical and technological infrastructure, an essential component in fostering knowledge creation.

Among many of the newer Member States of the EU, there was a research infrastructure gap that impacted the effectiveness of their R&D capabilities. As highlighted by some of the case studies, research infrastructures were underfunded. They were thus not providing sufficient capacity and quality for researchers, as research equipment was outdated or not in line with modern research standards. 10 This policy intervention aimed at upgrading existing infrastructure and equipment and replacing obsolete or outdated instances of these. In so doing, it sought to develop new research capacities that aimed to match the level of quality and research excellence at European and international levels. The intervention was not seen as an end in itself but rather regarded as a mechanism that would improve the quality of research and the innovative capacity of economies.

In terms of the intentions and expectations declared by the OP or the Major Project applications, the establishment of new research infrastructures was expected to contribute to three categories of impact:

·Scaling up research. The goal of many Major Projects was to achieve more ambitious research projects with respect to research status in the region at that time. In some cases, the objective was even more ambitious, i.e., to promote world-class research with international standards of excellence in a specific topic. The goal of scaling up research was complemented by the necessity of scaling up facilities (new research buildings and equipment) and supporting their cost. In response to these costs, several Major Projects were expected to attract increasing volumes of international research grants (both private and public), increase the number of research contracts, and to build up knowledge-sharing networks.

·Increasing collaboration with industry. Most of the research infrastructure projects funded in the 2007-2013 period were conceived with clear attention being given to facilitating a potentially positive spillover to the business environment and thus (also) possible benefits in terms of knowledge transfer, patenting and commercialisation of innovation.

·Seeding early-stage researchers. Some major infrastructural projects indicated educational activities and the training of early-career researchers as a priority.

Figure 4.Generalised (hypothesised) Theory of Change of infrastructure investments for research

Source: Ex-post evaluation report (2021)

ICT-based infrastructures are another identified subgroup of the broader infrastructure investment for research. Under the influence of the increasing importance of data digitalisation and open science as means to improve the accessibility, interoperability and re-use of scientific publications and data, ICT-based infrastructures provide digital-based services and tools for data and computing-intensive research in virtual and collaborative environments. These environments offer services and tools that support whole research cycles and ease or foster the movement of scientific data across scientific disciplines. Moreover, open-data spaces can be created, and scientific workflows can be improved by connecting data sets from diverse disciplines. Researchers also have the opportunity to link with high-performance computational systems, and they can therefore improve the overall capacity and scope of their research (Thanos, 2010).

By their nature, ICT-based infrastructures are meant to support researchers in their work and serve various research communities. Generally, the literature suggests that ICT-based infrastructure investments positively affect innovation capacities. This is because this infrastructure enables researchers to handle big data sets efficiently through the use of high-performance computational systems. Together with the improved dissemination of scientific output and exchange with other researchers, this means that innovative capacities increase (Thanos, 2010).

Figure 5.Generalised (hypothesised) Theory of Change of infrastructure investments for research

Source: Ex-post evaluation report (2021)

The ICT-based infrastructures identified across the OPs can be categorised as follows:

·rojects responding to advanced computing needs: computing grids and infrastructures/centres, supercomputers, computing servers, and similar phenomena;

·Projects to manage more effectively data creation, storage and access: these include data digitisation, data storage centres, open-data infrastructures, and ICT network systems;

·E-infrastructures to deliver e-services (mainly in the health domain) and to connect existing resources to a central hub for research and development; and

·Projects related to investments in other IT infrastructure or equipment for research, which do not fall under any of the previous subtypes.

R&D projects and related activities

Collaborative R&D projects support research activities carried out jointly by either research institutions/HEIs themselves or in collaboration with partners in private industry (referred to as science-industry collaborations). The latter is widely considered an essential driver of knowledge-based economies and societies.

A collaboration project is defined as a project where at least two partners participate in the design of the project itself, contribute to its implementation, and share the risk and output of the project (European Commission, 2006). Collaborative efforts funded by this policy intervention had various aims, ranging from addressing industrially relevant or societal challenges to stimulating technological advancement in specific areas to boosting international cooperation by conducting internationally competitive high-quality R&D activities.

Figure 6.Generalised (hypothesised) Theory of Change of collaborative R&D projects

Source: Ex-post evaluation report (2021)

The immediate outcomes of the policy intervention were aimed at increasing the number of joint projects and activities, both between R&D institutions themselves and in collaboration with private-sector partners. Moreover, researchers across supported research institutions, and those in private enterprises, were expected to develop skills and competencies that would increase their scientific-technological knowledge. The research pursued by these collaborative entities would be expected to lead to technological advancement and thereby facilitate the improvement or invention of new products, processes or services. These immediate outcomes (during the project) were expected to be followed by more intermediate outcomes (after the project), such as the economic benefits stemming from the commercial valorisation of R&D results. Moreover, an enhanced knowledge transfer between science and industry partners was to be expected.

Individual research projects were supported in various scientific fields to strengthen the region’s scientific and technological capacity in question. This policy intervention consisted of support for existing research fields, for which applications were investigated, and support for more ‘exploratory’ or ‘foundational’ research, which targeted areas that had great potential for innovation but were untapped. To some extent, the intervention also sought to improve knowledge and technology transfer into the industry, which would involve the economic valorisation of new scientific or technological products and processes.

The focus of ERDF support shifted significantly in the last two programming periods. However, this kind of R&D support was a central feature of the Lisbon National Reform Programmes (2006) since large disparities between the EU Member States and regions were observed, and a persistent gap existed at the global level compared with competitors. Early-stage (foundational) and exploratory research often do not have specific, predetermined commercial applications and rather serve to generate new knowledge and further develop innovative skills in research institutions; this is crucial for long-term, rather than immediate, R&D results. The fact that this type of research carries high risks and is therefore of reduced interest to private-sector investment is well documented in the literature (European Commission, 2017a). The uncertainty relating to the return of investment, and the sunk costs involved in ensuring a critical mass in terms of knowledge and skills accumulation, which is often a precondition for any meaningful R&D results, would otherwise induce underinvestment in such forms of research and innovation: this, therefore, underscores the significance of sufficient public investment in these areas.

Figure 7.Generalised (hypothesised) Theory of Change of individual R&D projects

Source: Authors on the cross-case study analysis performed in Task 4

The classification process identified a residual category of interventions (accounting for less than 3% of the total ERDF expenditure for RTD) that could not be defined as either R&D projects or infrastructure investments but were still relevant in the RTD field. They included the following.

·Internationalisation of research, mainly including projects related to promoting international collaboration among HEIs and RTOs, or international mobility programmes for scientists and students.

·Capacity building for research 11 , including projects addressing the development of researchers involved in R&D activities (including support for PhD programmes) or activities improving the capacity of institutions to raise funds, increase international visibility, or better manage the research process.

·Science dissemination to the general public, including projects to increase public engagement in and awareness of science.

·Intellectual Property Protection Instruments, a very specific policy instrument implemented by the Polish OP for the Innovative Economy, benefiting those RTOs and HEIs that had previously implemented an industrial R&D project, supported by the same OP.

·Operating subsidy, providing generic support for the functioning of research and technology centres (either science and technology parks or RTOs), with no specific reference to implementing any RTD activity or infrastructure. These responded to a very broad and ‘unspecified’ logic, with no precise outputs and results expected beyond the operation of the research institute itself.

3.How has the situation evolved over the evaluation period?

3.1.Projects and beneficiaries

The analysis of expenditure mapped more than 20,000 projects, half of which occurred in Spain alone 12 . Altogether, they accounted for almost EUR14.9 billion, which is 2% higher than the allocated amount reported in the Final Implementation Report. The ERDF contribution was typically provided in the form of non-repayable grants. Private co-funding was provided only for 15.7% of the almost 8 000 projects for which this information is available. The average duration of projects was three years.

Projects funded infrastructure investments, R&D activities and other types of activity 13 . A residual share was either not classifiable or should not have been classified under codes 01 and 02. To further distinguish the logic of interventions among the different projects, these main types were further split into 10 different types of intervention. The largest share of ERDF expenditure was concentrated on support for infrastructure investment (72% of total expenditure), with infrastructure investment for research absorbing more than half of ERDF expenditure (57%).

In terms of numbers, the most common types of intervention were R&D projects (55% of the total). A smaller share of projects (6%) and expenditure (3%) was allocated to implementing other RTD activity. A residual portion (3% of the projects and ERDF expenditure) fell into a fourth category, namely activities that were not strictly related to RTD and should instead have been classified under different codes.

Figure 8.Types of funded projects, number and million EUR

Source: Ex-post evaluation report (2021), based on monitoring data from the 46 OPs (excluding Spain)

Funded projects are, on average, of a significant financial scale: they received an average ERDF contribution of EUR1.3 million, while 24 Major Projects accounted for 10% of total ERDF expenditure in the reported OPs (i.e., EUR1.49 billion). Infrastructure investments for education activities cost, on average, much more than other types of investment.

The funded research was predominantly conducted in the fields of Engineering and Technology, Natural Sciences and Medical and Health Sciences. The vast majority of projects and expenditure pertained to applied research, generally with a possible industrial application.

Figure 9.Type of research, total ERDF contribution (million EUR)

Source: Ex-post evaluation report (2021), based on monitoring data from the 46 OPs (excluding Spain)

Most beneficiaries (almost 77%) are publicly owned organisations. Higher Education Institutes (HEIs) and Research and Technological Organisations (RTOs) account for nearly 88% of the sample of lead beneficiaries and received nearly 83% of total ERDF contribution. Enterprises comprised 4.5% of total direct beneficiaries, 70% of these being SMEs. They played the role of partners in collaborating R&D projects, typically with HEIs and RTOs. There were recurrent beneficiaries: on average, each body/institution received funding under four or five projects.

Figure 10.Type of lead beneficiaries, number and million EUR

Source: Ex-post evaluation report (2021), based on monitoring data from the 46 OPs (excluding Spain)

A total of about 4 000 different institutions (including businesses) can be identified among the almost 24 000 lead beneficiaries (9 973 for the 46 OPs and about 13 000 for the 7 Spanish OPs). When excluding the 7 Spanish OPs, the total falls to about 2 000 different institutions (including almost 580 HEIs, more than 720 RTOs, and nearly 470 businesses). Nonetheless, data on the total ERDF contribution shows that more than 13% of the ERDF support for RTD provided to lead beneficiaries 14 was concentrated on 10 institutions, and more than 20% on 20 institutions, with the Fraunhofer-Gesellschaft Institute in Munich (Germany) receiving more than 2% of the total. The other institutions where the largest share of ERDF funding was provided were, conversely, located mainly in Czechia and Poland, the largest recipient countries alongside Germany.

Figure 11.ERDF expenditure on RTD investments by NUTS2 regions in the targeted OPs

Note: expenditure of national or multi-regional OPs was regionalised based on the location of project beneficiaries

Source: Ex-post evaluation report (2021), based on monitoring data from the 46 OPs (excluding Spain)

Overall, evidence on expenditure on RTD infrastructures and activities highlights patterns of territorial concentration as a result of eligibility rules on the one hand and territorial concentration of RTD capacities on the other. More than 50% of ERDF expenditures in the selected OPs was concentrated in Poland, Germany and the Czechia, while 70% was concentrated in convergence regions and 64% in urban areas.

3.2.Key achievements

The monitoring system set up by the European Commission provided a first set of data on ERDF investment achievements. In compliance with Council Regulation (EC) No 1083/2006 laying down general provisions on the ERDF, the ESF and the Cohesion Fund 15 , indicators were reported by the Managing Authorities to assess the achievements of each OP. Annex II(4) presents the core and common indicators associated with RTD investments across the 53 OPs, along with their initial target values. As confirmed by the analysis carried out in the case studies, monitoring systems, in general, provided a fair amount of quantitative evidence regarding instrument activity and output performance. They were much weaker in the context of long-term achievements.

The monitoring indicators, which were not specific for the categories of expenditure 01 and 02 but referred to the overall OP, showed that the total number of research jobs created across the 53 OPs amounted to 20 500. Meanwhile, the total number of RTD projects supported was over 51 000, and the number of cooperation projects between businesses and research institutions was 15 000. As compared with the initial target, the degree of achievement was mixed, and most of the OPs demonstrated an overachievement for some indicators and underachievement for others. Nonetheless, as already noted in previous evaluation studies 16 , monitoring indicators present some shortcomings when it comes to assessing OP effectiveness.

In general terms, the quality and robustness of ERDF RTD performance measurement systems and accompanying indicators were found to be weak. The amount of data available for evaluators to assess selected instruments at this level was limited.

In some cases, the evidence from the monitoring indicators could be complemented by mid-term and ex-post evaluations conducted by the Managing Authorities or by external independent studies. Impact evaluations on individual policy instruments or programmes carried out at the national/regional level are not implemented on a systematic basis. Nonetheless, these studies, when available, can provide richer and more qualitative information on effectiveness. The scope of the evaluation studies varies to a great extent (e.g. they may focus only on selected policy interventions and not on the overall policy mix for RTD). Equally variable are the adopted methodologies (counterfactual impact-evaluation studies, qualitative case studies, macroeconomic modelling, etc.). Overall, this evidence is not sufficient to provide conclusive information about overall OP effectiveness in addressing pre-existing RTD barriers.

Finally, stakeholder perceptions regarding achievements were collected during the seminar held as part of the evaluation. Although reflecting the views of a limited number of invited programme managers and representatives of beneficiaries, they provide an initial picture of the relevant achievements of the 2007-2013 ERDF support for RTD.

Therefore, the 2021 evaluation filled the gaps by making use of other sources, as described below and detailed in the annexes to this SWD. The analysis led to identifying the following four categories of achievement.

R&D personnel and researchers

The availability of more and better-skilled R&D personnel and researchers in a region improves its R&D capacities and makes it possible to increase both the volume and quality of knowledge production. Between 2007 and 2017, the average growth rate in the number of R&D personnel and researchers in the target regions was 40%, and the average regional ERDF expenditure on infrastructures and R&D projects in HEIs was about EUR35 million. The econometric analysis shows that ERDF investments in research infrastructures and individual R&D projects in HEIs contributed to an increase in the number of R&D personnel and researchers at the regional level. Such a relationship was valid on average, i.e. without any statistically significant differences between the EU13 and EU15 or between convergence and competitiveness regions 17 .

The mechanisms through which this achievement was made possible are described in the case studies. One channel of effects was that triggered by individual R&D projects. Evidence indicates how funded projects facilitated an increase in the qualification levels of researchers and enabled the training of young scientists, as evidenced by the completion of around 70 PhDs as a direct result of the funded projects in Saxony.

The main channel of effects, however, lay through infrastructure development. Evidence illustrates how support for infrastructure investment in research contributed to the creation or modernisation of public R&D facilities and essential scientific equipment; the evidence suggests that newly purchased equipment created more respectable research environments and increased research operating standards. Infrastructure projects, such as buildings, needed time to be constructed before they could generate useful outcomes, while acquiring research equipment and simpler infrastructure (such as laboratories) had more immediate effects. Beneficiaries confirmed that the access to modern research equipment contributed to the elimination of previous handicaps that had prevented research institutions from participating in research endeavours. Consequently, the research potential and capacities of beneficiaries institutions were perceived to have increased significantly, rendering them more attractive and competitive and gaining increased interest from students and researchers.

Increased research capacities in universities also led to better awareness of, and interest in, industrial research needs and opportunities. In Poland, the number of enterprise R&D personnel with doctoral degrees increased significantly. The ERDF policy intervention supporting cross-sectoral R&D collaboration played a crucial role in redirecting the career pathways of many scientists, who became aware of industrially oriented R&D research and the possibility of pursuing such research in private enterprises 18 .

The creation or modernisation of ICT-based infrastructure tackled the urgent need for updated ICT infrastructure. For some institutions, basic ICT infrastructure elements, such as passive network components or phone systems, remained at the level of the 1990s. In other cases, some network elements (especially active components) were so outdated that no service was available from the suppliers. In Poland, a total of 59 projects funded 2 167 new or rebuilt laboratories. In Czechia, more than EUR24 million of ERDF funding was directed at new, developed, or modernised ICT information infrastructure for R&D, including repositories and storage capacities, optical networks, network optical elements, licences or databases. These outputs ensured a qualitative shift in existing services or, in some cases, the establishment of new ones.

Although ICT-based infrastructure projects need time to be constructed before generating useful outcomes, certain ICT infrastructures could be swiftly improved. Evidence from the case studies suggests that the building or modernisation of ICT-based infrastructure created more professional research environments and increased the quality of education.

Scientific output

Scientific output is traditionally measured by the number of scientific publications. In the period 2007-2017, scientific publications almost doubled in volume across the EU as a whole. However, the EU13 regions experienced a higher growth rate as compared with regions in the EU15 (145% against 96%), highlighting an ongoing catching-up process. In particular, some regions in Romania, Slovakia and Czechia experienced values higher than 400% 19 . Investments in research infrastructure and individual projects in HEIs have contributed to this achievement.

Evidence indicates that the ERDF investments significantly contributed to the catching-up process of the EU13 regions in terms of scientific output. This was partially due to the magnitude of the investments in HEIs (EUR43 million in the EU13 against EUR29 million in the EU15 regions) and certain other factors (national public expenditure in particular). The analysis also suggests that lagging regions performed better than more developed regions in terms of growth in the number of publications, but this happened regardless of the ERDF investments in the type of instrument under scrutiny 20 . The higher growth rate recorded by the EU13 regions was mainly explained by their poorer starting positions with respect to more developed regions. For instance, the Romanian region of Centru recorded 101 scientific publications in 2007 and 1 116 in 2017, with an extraordinary growth rate in the period under assessment. Indeed, the regression analysis shows a negative and statistical significance of the coefficient associated with the initial number of scientific publications in 2007, pointing to ‘diminishing returns to scale’ concerning the evolution of the number of publications over time.

The regression analysis indicates a positive and statistically significant relationship between ERDF support for research infrastructure and HEIs, on the one hand, and the growth rate in the number of scientific publications on the other: on average, the higher the ERDF expenditure, the higher the growth rate in the number of scientific publications in the period 2007-2017.

Evidence from the case studies helps to qualify the findings on scientific output. One channel of effects stemmed from individual R&D projects consolidating existing knowledge or expanding it to new research fields. Individual R&D projects played a significant role in contributing to the increase of scientific and technological output since, in some regions, they were the main source of support for individual R&D projects during this period. The immediate outcomes of individual R&D projects included an increase in the scientific and technological capacity of researchers, allowing them to increase existing expertise and develop new areas of inquiry. The reinforcement of teams with more human resources, and the acquisition of essential materials and services, ensured that the beneficiaries could improve conditions to develop their main research lines. In Portugal, almost all projects (over 99%) funded under the assessed policy instrument supporting individual R&D projects produced publications, in contrast to Germany, where it was estimated that around 76% of the funded projects did so. These results echoed broader national trends. In Portugal, for instance, from 2005 to 2014, the contribution to published knowledge (i.e. in the form of publications) more than doubled, with an average annual growth rate of 11%.

Another channel of effects arose from investments in upgraded research facilities and equipment. Beneficiaries confirmed that new infrastructure, modern equipment, and first-class instruments attracted students, researchers, and professors, both from the country and from abroad 21 . The analysed ICT-based infrastructure project, for example, led to a profound shift in quality in provided services. It operated and developed the national e-infrastructure for science, research and education, encompassing a computer network, computational grids, data storage and a collaborative environment in a ‘radical’ way. The newly developed infrastructure lifted the system towards the wider European level and opened a gateway to European cooperation.

Another positive implication of increased scientific expertise and international visibility was that funded entities were more empowered to apply for ambitious projects and also demonstrated improved chances of obtaining third-party funds from both federal and international (EU) sources.

However, the econometric analysis found that there was no statistically significant relationship between ERDF investments in HEIs and scientific excellence. When looking at the 2007-2017 rate of growth in the share of regional publications in both the top 25% and the top 10% by citations worldwide, the analysis suggested that scientific excellence was mainly driven by the level of R&D development in the region in the initial year (2007), measured by the level of regional Gross domestic expenditure on R&D in that year, and by long-term investments in R&D in the following decade. This finding is in line with the analysis presented by the European Commission (2020), indicating evidence of a catch-up process among the eastern regions in terms of quantity of scientific output, but not yet in terms of quality, a process that may take longer. The production of high-quality publications currently remains concentrated in some western and northern EU regions.

Technological development and innovation

The reconstructed theories of change indicated that, among the expected impacts of some policy instruments, there was the ambition to support the development of innovation and technological production capacity in the region as a consequence of improved scientific capacity and production. Policy instruments more directly addressing this objective were those implemented by RTOs, science parks, competence centres, cluster organisations (both infrastructure and R&D projects) and science-industry collaborative projects.

When looking at the hard technological output, as measured by patenting and public-private co-publications 22 , the econometric analysis confirms the results already highlighted in the case studies. In the period under scrutiny, there is no statistically significant relationship between the ERDF support provided to policy instruments targeting applied scientific activities and science-industry collaborations (either infrastructure investments or activities implemented by RTOs, enterprises or centre of excellence) and the growth rate of technological output. This holds true even when considering different model specifications with different context variables and controlling for the average ERDF investment in that policy instrument (e.g. amount of ERDF expenditure per number of beneficiaries). This finding emerges against a more general pattern of dynamic patenting activity in EU regions in the period under assessment and against some convergence patterns of eastern regions (European Commission, 2020). The average value of the growth rate in the number of patents in the period 2008-16 in the sample of regions was 48%: four regions experienced an increase in the number of patents higher than 500% (RO12 – Centru; PL34 – Podlaskie; RO22 – Sud-est; PL43 – Lubuskie). By contrast, a set of regions, located mainly in Germany, Italy, the UK, Czechia and Belgium, recorded negative growth rates. The case studies on selected policy instruments confirmed the anecdotal evidence that pointed to limited uptake of the scientific results of supported R&D projects, with poor results in terms of improved innovation and technological development.

Implementation issues were, however, limited, and overall, the projects were successfully completed. Immediate outcomes included an increased number of joint projects between R&D institutions and collaborations with private-sector partners. The evidence suggests that the funded R&D projects played a significant role in increasing scientific and technological knowledge and competencies among beneficiaries since many private companies gained access to new ideas and became aware of new technologies. At the same time, through their collaboration with industry partners, research centres were also able to explore business partners’ needs and develop skills needed for industrially oriented and applied R&D.

Results, however, may remain far too distant from an industrial application. This issue may apply in a number of cases when fundamental research was the focus of funded activities. Although this applies to a limited share of projects, it can be part of a broader situational explanation. In many cases, research results were simply not relevant for the industry, or no follow-up projects had been intended. In the Polish case study, scientific partners reported limited interest among industry partners due to changes in business strategies that made the R&D results useless; alternatively, limitations related to insufficient technological readiness were combined with a lack of funding necessary to continue the technological development. Consequently, as of 2013, R&D results had been successfully implemented by companies in only four cases for the policy instrument under assessment (Re-Source, 2014).

Another explanation for the lack of relationship between ERDF support and technological output is that of time lag. As the literature points out (see, for example, Finardi, 2011 and Bastianin et al., 2021), it may take some time for research activities to generate a technological output, so the results of research activities implemented in the 2007-2013 programming period may be visible far beyond 2017. This may be even truer regarding infrastructure development. Infrastructure investments supported by the ERDF may have facilitated the setup of a suitable environment, enabling more intensive and productive exchanges between science and industry. This phenomenon, however, takes time to be consolidated and to deliver results 23 . While an increase in collaboration between scientific institutions and commercial partners can be observed from the evidence, the transfer of results from R&D projects to external users for economic and social valorisation occurred to a more limited degree. -

The most convincing explanation regarding the lack of relationship between ERDF support and the growth of patents is, however, the consideration that ERDF support alone may not have been enough: other supporting factors were probably necessary for a noticeable shift in technological capacity. Rodriguez Pose (2020) noted that innovation in the EU regions is linked to four main factors: investment in R&D; population density; a higher share of population with tertiary education; and governmental quality. The econometric analysis confirms this hypothesis and indicates that the main drivers of patent growth were the R&D investments carried out by firms and the maturity of the R&I industry system in the region, as suggested by the initial level (in 2008) of regional R&D investments in the business sector. Thus, the ERDF contribution to technological production may be more indirect. One viable hypothesis is that the ERDF policy instrument positively influenced the R&D expenditure variation in the business sector and, through this channel, had an indirect impact on patents. Overall, private R&D expenditure still increased in all convergence regions, reaching the target as set by the OP. In 2007, private R&D as a proportion of GDP amounted to 0.22%, while in 2015, it reached 0.32%, and by 2017, it had fallen back slightly to 0.31%.

Thus, the econometric analysis also tested whether ERDF policy instruments showed an indirect link with the growth rate in the number of patents through the R&D expenditure in the business sector. Nonetheless, the regression analysis rejects any statistically significant correlation between ERDF policy instruments and increases in business R&D expenditure in the target regions during the decade 2007-2017. While this is more disappointing than the previous results regarding patents, there are also explanations available here. Other factors can play a more direct and significant role in triggering business R&D expenditure, including ERDF support to business R&D (outside the scope of the present evaluation), other forms of direct support to business R&D, or more contextual factors. Among the latter, the role of the economic crisis should not be overlooked, and this will be discussed in the next section. In addition, building trust and a positive relationship between science and industry might not be achieved via a single project; more effort is needed. In Poland, the case study describes that few follow-up R&D contracts or joint R&D initiatives between industry and scientific partners were pursued upon completion of the funded projects. Most interviewed representatives of both scientific and industrial organisations confirmed that no joint R&D projects were carried out with the same partner following the conclusion of projects. Polish businesses gradually reduced their funding of external R&D projects performed by scientific organisations for reasons concerning legal challenges related to intellectual property and technology transfers – challenges that themselves resulted from overly bureaucratic frameworks. Scientific partners also pursued different objectives upon project completion, focusing more on publications and broad knowledge dissemination rather than commercial developments.

Hence, the contribution of ERDF support for R&D collaborative projects to the competitiveness of the regions may occur through more indirect and, especially, ‘softer’ effects than through an observable shift in patent or business R&D. Improvements in the competitiveness of local industry may come from improvements in R&D management, increased interest in the pursuit of innovation strategies and a greater understanding of new technologies and international technological trends, rather than from the successful commercialisation of the R&D results of projects 24 . Another more indirect link to innovation may have occurred through the effect on the education sector and the share of the population with tertiary education, as discussed earlier.

The role of ERDF support in fostering behavioural change among the target beneficiaries has already been highlighted by previous evaluations (European Commission, 2016b). The Romanian case study points to a ‘soft’ impact, such as ‘paving the connecting road’ between large and powerful industries operating in Romania, alongside profound changes in the organisational culture and management style of scientific research organisations and higher education institutions inherited from the previous economic regime.

Following this line, the econometric analysis tested correlations between the ERDF and ‘softer’ innovation outcomes. The analysis suggests that ERDF support 25 positively and significantly correlated with the growth rate of EU trademark applications in the period 2007-2015. Interestingly, in addition to ERDF support for RTD investments, additional drivers were the level of ERDF expenditure on business support and R&D expenditure in the business sector (as a percentage of GDP) 26 . Although not conclusive evidence in itself, this finding seems to confirm the impression that the role of the ERDF is more related to behavioural changes and less to technologically intensive innovation.

The ERDF role in triggering behavioural changes among its beneficiaries is supported by evidence from the case studies. It is indicated in some instances as, perhaps, one of the most lasting effects of ERDF support in the field of RTD. There is widespread consensus that the ERDF played a key role in strengthening the capacities of beneficiaries and that this has lasted well beyond the period during which they benefited directly from ERDF support. These effects have also rippled into others, such as the capacity to engage in collaborative research activities or participate in international research programmes. Still, it is worth noting that these factors mainly concern public organisations such as Higher Education Institutions or Public Research Organisations.

Students and tertiary attainments

Even though this was not the primary objective of the policy, a selected number of countries implemented policy instruments supporting infrastructure and investment for education to enhance Research and Innovation capacities by attracting more students and improving tertiary attainments in the region. The observed growth rate in the share of tertiary-educated people in the target regions was, on average, 7% in the period 2007-2017, from a minimum of -5% in the German region of Chemnitz (DED1) to a maximum of 18% in the region of Prague (CZ01).

In the evaluation sample of NUTS2 regions, only 29 regions (out of 104) invested in R&D infrastructure for education. Of these 29, 21 were in the EU13. The Polish regions of Mazowieckie (PL12) and Pomorskie (PL63), together with Estonia (EE00), were noteworthy for allocating a total of more than EUR100 million to that instrument. Evidence shows that ERDF investments in education infrastructures contributed to increasing the RTD capacity of beneficiary regions by supporting the increase in tertiary population attainment and the increase in people employed in science and technology.

The econometric analysis points to a positive and statistically significant correlation between the ERDF policy instrument and the growth rate in the number of tertiary-educated people (as a percentage of the population aged 25-64) in the period 2007-2017. The same positive relation holds for the growth rate in the number of tertiary-educated people employed in science and technology in the same period.

The evidence of the case studies offers a description of the mechanisms underpinning this achievement. Beneficiaries underscored the high degree of satisfaction about the intervention, as it allowed universities, for the first time, to invest in the development of a holistic teaching and learning environment. The renovated and newly constructed buildings enabled institutions to accommodate new equipment, creating a better environment and attracting new students and researchers.

Improved infrastructure had an impact on the quality of life at supported universities. For example, in Czechia, thanks to numerous investments, the city of Brno became a valuable alternative to Prague in terms of opportunities to gain a high-quality tertiary education, especially in certain fields. Investments helped dramatically transform the appearances of HEIs in some regional university cities, with positive effects spreading to local communities (e.g., further upgrades of the physical environment in particular city districts) 27 .

Poland invested the highest amount in terms of ERDF funding to the development of educational facilities. In the supported region, there was a steady and significant growth in student choices of strategic (i.e. OP-supported) faculties from the academic year 2008/2009 until 2015/2016, while the total number of students decreased in the same period. One evaluation (EGO, 2013) indicated that the new infrastructure was used by a significant share of students attending HEIs who participated in the programme (32% of students overall and 83% of students of faculties directly supported by the project). Moreover, the evidence suggests that the improved conditions and teaching environments helped attract students and researchers from abroad, as evidenced by the increase in international students in Poland and higher levels of cooperation with international partners. This has contributed to an increase in the internationalisation index of Polish HEIs, from 0.5% in 2005 to 5.63% in 2017.

The degree of achievement from this type of instrument also depends on factors other than the ERDF. The econometric analysis shows that a positive and statistically significant contribution to the growth in the number of tertiary-educated people (and employment in science and technology) was achieved as compared with the initial level (in 2007). As for the growth rate (2007-2017) of R&D in the business sector, regions with a more advanced industrial fabric and higher R&D in the business sector experienced higher growth rates in the number of tertiary-educated people employed in science and technology. While ERDF support for RTD can increase the supply of researchers, demand-side effects related to the absorption capacity of local labour markets are crucial for the final success of such measures.

While there is plenty of evidence pointing to the fact that education has been improved, it is uncertain whether these improvements are in line with the needs of the labour market and potential employers. There are also questions regarding the extent to which improved education has led to higher employability and increased recruitment. In Poland, for instance, a recent peer review of Poland’s Higher Education and Science system (European Commission, 2017b) suggests that the employment of recent higher education graduates is above the EU average, but there are growing concerns about labour market mismatches. In 2015, the employment of recent tertiary graduates in Poland stood at 85.1%, compared with the EU average of 81.9%. Nonetheless, a substantial and increasing number of tertiary education graduates are in medium or low-skilled jobs, pointing to labour-market skills mismatches (European Commission, 2016a). Nonetheless, the extent of ‘over-qualification’ remains significantly below the EU average, as evidenced by several studies (Cedefop, 2015).

Important issues in the research labour market were also found in Romania, although it should be borne in mind that the country did not invest in education facilities but rather in infrastructure for research in universities. Even though there was a slight increase from 25.4% in 2008 to 28.2% in 2016, human resources in science and technology remained lower than the EU average. One reason for this development, as illustrated by a recent JRC report (2018), was that staff in HEIs, as well as in the education, research and medical systems, remained strikingly underpaid in comparison with EU28 averages and with other categories of personnel (e.g. judges, local administration, police, army) in Romania.

3.3.Contribution analysis

Impacts and causality

Evidence from the case studies shows that the great majority of instruments completed their planned activities and delivered both intended outputs and immediate outcomes (e.g. projects and grants were provided, buildings and new research infrastructures were developed, collaborative projects were completed). This was also reflected in the high level of disbursements of ERDF funds, which indicated that activities and work originally scheduled under Operational Programmes were successfully executed.

The great majority of intended outputs and immediate outcomes identified in the reconstructed Theories of Change were duly observed by the case study teams. Instances in which projects originally selected to receive ERDF support were then cancelled due to poor performance or other implementation challenges were extremely limited. Italy is perhaps the most notable exception to this general finding, given that various output and immediate-outcome goals have been only partially met, given the delays in ERDF funding execution.

The causal link is then diluted along the pathway towards impacts. When one considers effects that were observed far beyond the life cycle of the policy intervention, it must be acknowledged that various other factors – not directly related to the policy intervention of 2007-2013 – exerted a significant influence. These other factors combined their effects with the ones produced by the ERDF in many ways. Such contributing factors, preconditions and risks, and the way they influenced the causal pathways, are discussed in the next section. This dilution effect may also be the reason why evidence regarding intermediate outcomes and impacts is more limited.

The main positive outcomes identified in the case studies relate to improvements in the scientific capacity and performance of funded beneficiaries. This is visible through an increase in students and graduates, an increase in R&D personnel and researchers, and the higher production of research outputs such as publications. It is also confirmed, however, by more qualitative aspects not immediately captured by indicators. These aspects relate to the increase in scientific standing and visibility (also internationally) of the beneficiaries, increased participation in international research collaboration networks and projects, and reported increased capacities to plan, execute and manage research projects and infrastructures in accordance with international best practices and standards.

Most important, however, is the relatively limited existence of evidence and data pointing to any uptake in private-sector innovation as a result of the work performed based on ERDF support for RTD investments. The case studies only identified a handful of instances in which data, knowledge or infrastructures stemming from ERDF projects either directly or indirectly benefited the private sector, thereby developing innovations that were later introduced to the market. Even though private-sector innovation was a common intended outcome of many of the instruments analysed, in hindsight, this does appear to be one of the major blind spots of ERDF support for RTD during the 2007-2013 period.

External factors

More than 90% of OPs saw a reprogramming of the planned expenditures. The reprogramming did not necessarily imply a change in the overall intervention logic but potentially pointed to a financial reshuffle among interventions with the same logic. Interestingly, there was a sharp difference between the reasons mentioned by the EU15 and EU13 OPs, with the former being mainly related to the external shock of the economic crisis. Conversely, the latter were more influenced by the need to better target the funds to the instruments considered more attractive by potential beneficiaries, thus showing a better performance in terms of funds absorption. Since this was the first full programming period for those countries, it is reasonable to conclude that planning and administrative capacities were still less developed in the EU13 than in the EU15.

In the period under assessment, the economic crisis negatively affected investments in RTD infrastructure and activities. The impact of the crisis across EU regions was uneven. It is possible to identify a core continental area wherein RTD efforts were affected only to a limited extent by the crisis or where they could recover relatively quickly. This was true of Germany 28 , most of Poland 29 , and neighbouring regions such as Slovakia. By contrast, one can discern a more peripheral area that was impacted the most: Ireland, Spain, parts of Italy, Greece, Cyprus, Lithuania, Latvia and Estonia.

In 14 of the 53 OPs analysed, not all of which were hit by the economic recession, the crisis determined a change in the OP policy mix for RTD. In most of the observed cases, the Managing Authorities decided to increase the resources allocated to RTD investment (both R&D projects and infrastructures), considering this as a means to sustain business development and innovation and ensure job creation and economic competitiveness in the long run (two Spanish OPs, the three Portuguese OPs and the Slovenian OP). This is in line with what the literature suggests, indicating that maintaining or even increasing investment levels remains crucial during economic downturns 30 . In a minority of cases (especially the Italian OP for the southern regions, the Latvian OP and, to a lesser extent, the UK West Wales and the Valleys OP), resources allocated to RTD interventions were cut and moved to direct business support to mitigate the impact of economic recession, especially on SMEs, in a countercyclical role.

Although different reactions to the crisis were adopted in different Member States, there was a generalised decrease in national public expenditure for RTD investments, which the ERDF partially compensated 31 .

Looking at broader RTD investments beyond the ERDF, public R&D expenditure during the period remained highly concentrated in those countries where most regions with high R&D intensity were located (i.e. Finland, Germany and France). Nevertheless, a generally positive trend was recorded, primarily in the newest Member States, such as Czechia, Estonia, Lithuania and Slovakia.

In terms of the regional strategies that were potentially complementary to the ERDF OPs, special mention should be made of the Smart Specialisation Strategies (S3). Even if these strategies were drafted towards the end of the programming period and examined in a recent study 32 covering 2014-2020, clear synergies and mechanisms to ensure coherence between the S3 and the ERDF 2007-2013 OPs were detected in some OPs. Full synergy between the ERDF and the S3 was ensured when they adopted coherent sectoral priorities. In some countries, especially the EU13, the investment carried out in the 2007-2013 period directly informed the selection of priorities for the next S3. In other cases, the S3 influenced the design and implementation of the OP.

Generally speaking, linkages between ERDF and non-ERDF support for RTD were visible through the co-financing requirements established by the ERDF. Beneficiaries, and public entities, in particular, made use of regional or national financial resources to ensure they complied with ERDF co-financing rules and obligations.

Evidence collected from the case studies indicated a significant degree of coordination across interventions carried out under the ERDF. This applied to coordination across different ERDF OPs (national and/or regional) and the different axes, measures, and instruments implemented under individual OPs.

Within individual OPs, beyond investments for RTD infrastructure and activities, other interventions potentially contributed to strengthening the regional RTD systems. This was particularly so as concerns the objective of improving the systematic relations of regional actors. In the context of those additional resources, expenditure for these interventions represented a significant portion of the ERDF contribution in the period under evaluation (more than EUR26 billion) 33 .

Case studies show that there was no overlapping or duplication of efforts between the two types of instruments in terms of design and implementation. A typical distinction was made in terms of target beneficiaries: while instruments supported by 01 and 02 categories of expenditure mainly targeted research providers to improve their capacities, the other codes primarily reflected the targeting of SMEs, with the principal aim of supporting innovation processes.

Beyond ERDF measures funded under the same OP, expenditures were also planned in other OPs but targeted the same territories, notably when regional ones complemented national OPs. In many cases, the case studies found that sufficient complementarities across ERDF support actions and programmes in the field of RTD played a key role in achieving the expected results, either as necessary preconditions or as key supporting factors 34 .

Synergies with the European Social Fund (ESF) consisted in funding training activities directly related to RTD projects by applying the principle of complementarity between Structural Funds (Article 34 of Regulation (EC) 1083/2006). This allowed the use of the ERDF to finance actions that fell within areas of intervention established by the ESF, up to a maximum of 10% of the Community contribution of the priority axis.

The analysis of regional strategies shows positive examples of the combination of ERDF and ESF funding. Not surprisingly, synergy was particularly strong in the regions where the ERDF emphasised infrastructure investment in tertiary education, as in Poland, Estonia and Slovakia. Complementary training programmes funded by the ESF were implemented primarily in Poland.

The in-depth analysis carried out with the case studies further expands on this point. Several instances have been identified in which national and regional authorities developed coordination to ensure that ERDF and ESF operations contributed to achieving common goals.

4.Evaluation findings

4.1.To what extent was the intervention successful and why?

Effectiveness

The main achievement of ERDF support for RTD investment in the period 2007-2013 is a positive and significant contribution to the observed improvement of R&D capacities in the target regions, particularly in EU13 regions. Evidence shows that ERDF investments aimed at modernising education facilities are positively correlated with the growth rate in the number of tertiary-educated people and the growth rate of tertiary-educated people employed in science and technology, in 2007-2017 and within the target regions. The growth rate in the share of tertiary-educated people in the target regions was, on average, 7% in the period under assessment. Investments supported by the ERDF have contributed to this trend by improving the conditions and teaching environments of the target universities, which have attracted more students, not only nationally but also from abroad.

Evidence from the case studies shows that renovated and newly constructed buildings enabled institutions to accommodate new equipment, which created a better environment to attract new students and researchers. Evidence further shows that regions with a more advanced industrial fabric and higher R&D in the business sector experienced higher growth rates in the number of tertiary-educated people employed in science and technology. Problems in absorption capacity of the labour market and labour market mismatches were specifically reported for Poland but may also apply to other countries.

R&D capacities were further improved in terms of the number of R&D personnel and researchers, with an average growth rate in the target regions of 40% between 2007 and 2017. ERDF investments in research infrastructures and individual R&D projects in HEIs positively contributed to the increase in the number of R&D personnel and researchers at the regional level. Infrastructure investments also contributed to the creation or modernisation of public R&D facilities, including ICT-based infrastructures, which increased the potential and capacity of the beneficiary institutions and created more attractive and better performing research and education environments. Individual projects allowed researchers both to increase existing expertise and to develop new areas of inquiry.

Evidence also points to a positive and statistically significant relationship between ERDF support and the growth rate in the number of scientific publications. While a catching-up process in scientific production is particularly visible in the EU13 (145% growth in the volume of publications between 2007 and 2017), it is reasonable to suppose that ERDF investments in HEIs have significantly contributed to this process. Conversely, no relationship is found regarding the quality of scientific production (as evidenced by the growth rate in the number of regional scientific publications in the top 25% of most cited publications), which may take longer to catch up.

More limited, however, was the capacity of funded projects to generate economic benefits from the commercial valorisation of R&D results and improve the knowledge transfer capacities and mechanisms from scientific to industry partners. No statistically significant relationships are found in the econometric analysis between ERDF support and the growth rate of technological outputs, which confirms the limited uptake of research results observed in the case studies.

The ERDF was not successful in stimulating business R&D, which is the main driver of technological outputs. Evidence from the case studies shows that some implementation issues were reported for collaborative R&D instruments and projects, despite reports of generally high interest from beneficiaries. Evidence collected in the field shows that science-industry collaborations did not lead to systematic follow-up projects. This was because of a lack of trust or resources, or because of legal problems related to intellectual property rights and technology transfer procedures. Expected results in terms of consolidation of research partnerships showed limited sustainability in the long run. Nonetheless, some positive results were reported in terms of softer innovation aspects measured by the growth rate of EU trademark applications, especially in those regions with higher ERDF expenditures in business support. Positive effects were also reported regarding the managerial capacity of research institutions and the enhancement of their research and innovation capacities.

Overall, there is evidence that the ERDF contributed to building and modernising R&D infrastructure in EU regions, especially those lagging behind. This process of upgrading and improving RTD capacities is especially evident in the EU13, where the ERDF contributed to filling the chronic investment gap they had suffered. ERDF investments in 2007-2013 supported the creation of favourable conditions to conduct research of international quality, helping less developed EU regions to bring their R&I systems closer to EU standards. Evidence from the cluster analysis of RTD performance in the EU regions shows that half of these regions did not change their relative RTD performance in the ten years since the start of the programming period. Although performance improvements were concentrated in economically stronger regions, some transition regions also saw a catch-up dynamic. Evidence of this evaluation suggests that, particularly in certain EU13 regions, the ERDF has positively contributed to this catch-up process of RTD capacities, thus contributing to reducing disparities between EU regions in performing quality research. Thanks to the combined effects of ERDF investments and favourable framework conditions (including national support to RTD investments), some of the EU13 regions were well equipped to conduct more and better-quality research, contributing to strengthening the EU’s RTD capacity.

The ERDF was less effective in facilitating the coordination and interactions between all the actors involved in the innovation ecosystem, thus adding to system failures. In particular, there is no evidence of an improvement in the science-industry relationship, which is one of the possible explanations of the observed limited knowledge transfer and innovation uptake. Moreover, while there is a dominant scale effect of the policy, where existing systems performed better or maintained a stable performance, limited or even no evidence is available about a transformation of the regional system, for example in the extent and nature of the science-industry links. Indeed, the policy did not move towards a structural transformation in how knowledge is produced, disseminated and exploited. Ultimately, the ERDF was less effective in transferring the increased research capacity into more competitive territories and regional economies that were addressing system failures. If this may have come true in a longer time horizon, capitalising on the investments made with the Smart Specialisation Strategies, it should be verified in future evaluation studies. However, it is important that future evaluations take a more systemic point of view in the assessment: first, by mapping regional systems and their investment needs; and second, by assessing the appropriateness of the observed trajectory of systemic change.

Finally, the 2021 evaluation on which this SWD is based highlighted the importance of underlying factors for impact generation in the implementation of R&D funding - a point widely discussed in the literature. In particular, synergy and complementarity with existing funding sources were not always well exploited. Moreover, administrative failures and legal constraints exposed the implementation to delays, uncertainties, rejections and, indeed, financial stress in a field where timing, long-term commitments and clear rules are crucial incentives for the collaboration of engaged actors.

Efficiency

Financial concentration is often seen as a desirable outcome of policy action and an indication of efficiency. Evidence collected on funded projects and beneficiaries highlights a concentration pattern on economically stronger territories, sectors and leading institutions. Reflecting existing agglomeration effects of R&D activities and capacities, RTD investments funded by the ERDF followed concentration patterns: more than 50% of mapped funds were invested in Poland, Germany and Czechia, while 70% were directed to convergence regions and 64% to urban areas. ERDF support for RTD was overall sufficiently concentrated to lead to upgrades in both the quality of research infrastructure and research management capacities in most of the countries under investigation. However, its role as ‘game-changer’ or ‘needle mover’ of RTD performance in beneficiary countries and regions was strongly related to the importance of the ERDF in the overall national and regional RTD policy mix. As already highlighted, fund concentration on increasing efficiency may lead to a ‘winner-take-all’ dynamic that needs to be addressed with policy action to reduce gaps between winners and losers.

The evaluation also looked at the efficient use of financial resources in the management and implementation of interventions, strictly linked to the administrative capacity issue. The administrative and managerial capacities of both programme managers and beneficiaries are crucial for effective public spending. Some implementation issues, mainly related to limited administrative capacity or an unclear legal framework, were reported especially for collaborative R&D. Uncertainties in the interpretation and application of rules, especially for what concerns State aid rules, caused delays and generated confusion and adjustments during the implementation process.

It is not surprising that administrative capacity can improve the effectiveness of supported instruments. However, specifically for RTD, the capacity to ensure selected projects’ high scientific quality, and their timely selection and funding, are identified as key elements. Ensuring a timely and smooth project assessment and selection procedure can improve the quality at entry of funded projects and increase the probability of success. A successful regional RTD system requires intensive and successful interactions among many different players, alongside multiple stakeholders and behavioural incentives.

Wider aspects of value for money were outside the scope of this evaluation, given the scale and heterogeneity of the funded interventions, making it unpractical to measure and value the produced output in a systematic and comprehensive way.

Coherence

ERDF support for RTD was implemented as part of a wider policy mix, including other ERDF policies and other national, regional and EU initiatives. They all somehow contributed to improving R&D performance in the EU regions. Thus, their respective roles and potential synergy were carefully considered. The role of the ERDF differed significantly across regions and countries in terms of financial weight and strategic coherence.

One of the key factors affecting the long-term sustainability of projects was revealed to be the long-lasting strategic and financial commitment to investment priorities, both for private and public organisations. In this regard, the ERDF played a countercyclical role in many regions, representing a ‘safety belt’ for many beneficiaries. One of the evaluation findings is that, in some countries, ERDF funding in the period 2007-2013 prevented the erosion of R&D systems in a time of severe cuts in public funding for education and research, given the economic downturn of 2008. Thus, it played a significant substituting role in those countries that were most severely hit by the crisis. Conversely, this may have prevented or delayed the restructuring of some national R&D systems, thereby losing the opportunity specifically to promote the most relevant systems or to achieve excellent growth.

Above all, the crisis impacted firms’ financial capacity and resources to undertake investments, with a risk of limited fund absorption, especially for those OPs that allocated large resources to collaborative research projects. This underscored the need for adaptive strategies to cope with changing socio-economic contexts. Evidence shows that continued public investment in research institutions is key. It enables for follow-up projects to take place that strengthen existing capacities and encourage the development of new ones. Hence, the long-term alignment of the ERDF with national and regional RTD strategies becomes a crucial element of success. Resilience in strategy design and implementation is also a crucial element of success in combining different policy instruments and funding schemes.

Coherence with other forms of ERDF support (internal coherence, i.e. coherence with other ERDF measures in the same OP, or ERDF support for RTD by other OPs targeting the same territories) was generally high. There was robust coordination among different OPs and between different priority axes within the same OP, clearly considering possible synergies and complementarity of respective roles.

The ERDF policy mix for RTD was also generally coherent with regional and national RTD strategies, especially regarding a strategic alignment of priority sectors and scientific fields. In some countries, when the prevailing rationale was to improve science-industry collaboration, RTD strategies supported by the ERDF were often closely linked to objectives of industrial competitiveness. The ERDF role in shaping national and regional policies was stronger in those countries where it represented a significant share of national or regional R&D expenditure, and therefore mainly in convergence regions.

Despite a high strategic alignment, however, there was often a tacit division of goals between local and ERDF policies and instruments in more operational terms. Coordination was mainly driven by co-financing obligations in these cases, and there was a notable effort to avoid overlapping. In some cases, a lack of political stability and related long-term commitment, and the absence of financial predictability for national RTD strategies, prevented a stronger alignment. In some regions, in the aftermath of the economic crisis in 2008 and subsequent years and shrinking national public expenditures, this also meant the substitution of national funds by ERDF resources.

Good synergy was reported with the ESF, with specific reference to support in the higher education sector. Here, the combination of ERDF and ESF funding resulted in a limited number of positive examples that nevertheless showed significant promise.

ERDF and EU Research and Innovation Framework Programmes were seen as serving related but essentially different purposes. The former mainly provided funds to ensure the enabling conditions to carry out excellent scientific work (through infrastructure investment) and to support applied research benefiting local R&I systems; the latter provided funds for excellent, EU-wide research activities, primarily in fundamental research. Nonetheless, despite ambitions to build on relative strengths and implement projects with coordination between the two funds regarding selected target areas or beneficiaries, no specific arrangements were implemented to facilitate or promote active synergy. No special coordination mechanisms were put in place to implement programmes and specific instruments, mainly because the two funds still followed different implementation mechanisms (e.g. the selection method for the interventions and the object of these interventions). It is not yet clear if this lack of coordination has had some adverse impact on the overall performance of the programmes.

Clarity about the rules of the game, shared within the common RTD space by science and industry partners and regulating their respective roles and responsibilities while providing the most appropriate incentives for successful partnerships, proved to be decisive. Here, the perception of State aid rules was reported as the most problematic factor in many countries. They are among the main challenges hampering a more intensive and effective involvement of businesses in the funded projects and follow-up activities. The role and influence of State aid were more evident in the implementation of policy instruments rather than in their design. According to interview respondents, unclear interpretation of State aid rules generated uncertainty regarding the eligibility of business enterprises to benefit directly from the ERDF and the extent to which the private sector could be involved as users of ERDF-funded infrastructures 35 . This likely had implications on the effectiveness of RTD instruments, especially those that targeted science-industry collaboration. Although State aid provisions have been later adjusted and revised to cope with perceived challenges, the question of coherence between cohesion and competition policy remains open.

4.2.How did the EU intervention make a difference?

EU added value

In the broader policy mix, MAs recognise that the main EU added value was a scale effect produced by accessing a considerable quantity of financial resources. This holds true especially in the EU13, where ERDF 2007-2013 programmes represented the first systematic set of interventions addressed to the research field after years of underinvestment and limited political priority.

A missed opportunity was the lack of the systematic promotion of interregional or international research collaborations as a potential EU added value. Partnerships of collaborative R&D projects were mainly regional or, albeit only in selected cases, multi-regional within the same country.

EU-wide effects were not among the intended effects of funded instruments. Thus, the contribution of ERDF support to them was more indirect. It occurred through the development of EU-level research communities in specific fields, enabling the construction or upgrading of strategic infrastructures of pan-European relevance (as confirmed by the later inclusion into the ESFRI roadmap) and also supporting the internationalisation of research communities. It helped structure and consolidate a European Research Area by promoting the achievement of EU standards in RTD capacities and production, and this can be claimed to have been the main EU added value of ERDF support to RTD investments in the period 2007-2013.

4.3.Is the intervention still relevant?

Relevance

The evidence collected through the case studies indicates that, in most cases, the ERDF support for RTD was relevant, meaning that it addressed the most pressing needs of expansion and modernisation of national RTD systems. It addressed, in particular, the huge infrastructure gap of Central and Eastern European countries. However, the ERDF support also reflected the need to improve science-industry collaboration, mainly in more advanced regions. Overall, ERDF support to RTD investments concentrated on interventions on the supply side, mainly focused on strengthening RTD capacities rather than on improving the performance of regional RTD systems as a whole.

Project selection was conducted by managing authorities based on a mixed approach. While infrastructure investments were typically the result of a top-down approach guided by national road mapping exercises, research projects followed a more bottom-up approach, responding to the needs of regional scientific communities within a well-identified set of scientific and technological priority fields.

Looking at the targeting strategies, it is important to highlight that the dominant approach was more functional than place-based. ERDF prioritisation strategies targeted territories/institutions/sectors with significant potential or comparative advantages. When the distribution of ERDF support was not geographically driven by eligibility criteria, such as in national OPs in Central and Eastern Europe, ERDF expenditure was mainly concentrated in urban areas, economically stronger sectors and more competitive institutions and organisations. This approach was justified by the need to ensure critical mass, fund absorption and knowledge externalities in more mature territories. Only in a few cases was there a balance between the choice of international or national excellence and territorial cohesion. However, overall, the evaluation found a lack of precise context-specific considerations, including territorial imbalances, in the design of the policy. This evaluation however could not provide conclusive evidence about whether the pursuit of excellence objectives was made at the expense of cohesion and whether may have contributed to increasing the territorial divide.

The observed targeting strategy based on excellence confirms what is already noted in the literature as ‘innovation paradox’, i.e. that regions with a stronger need to invest in RTD seem to be those with a comparatively lower capacity to absorb funding than more mature regions. The adoption of the Smart Specialisation Strategy in the period 2014-2020 is expected to have offered a workable solution to the innovation paradox and a platform to guide the design of RTD policies in lagging regions. It also possibly offered a more solid theoretical framework to assess the relevance of the adopted policy mix in the different territories, which should be based on a thorough mapping and prioritisation of regional vocation and potential. Future evaluations should shed light on the extent to which this new approach effectively supported the design and implementation of place-based RTD investments, especially in less developed regions. It should also point to the extent to which this approach has facilitated the understanding of needs, capacities, motivations and interests of the different actors in the system, an aspect that appeared to be weak in the design of the 2007-2013 ERDF programmes.

5.What are the conclusions and lessons learned?

5.1.Conclusions

The findings of this study indicate that the selected 2007-2013 ERDF programmes, and their associated strategies, were well justified overall and backed by a relatively solid rationale. Positive achievements were reported, in particular, by policy instruments targeting HEIs. Overall ERDF support has positively contributed to the catch-up process of RTD capacities in certain regions, thus contributing to reducing disparities between EU regions in performing quality research. This was mainly delivered by infrastructure investments targeting the upgrade and modernisation of existing facilities and the construction of new ones. This applied especially to those EU13 regions lagging behind in terms of scientific and technological capacities and standards. Support for R&D projects, both individual and collaborative, was important in developing know-how in new scientific fields or existing areas with great potential. As a result, there were observable effects on scientific production and capacity, as witnessed by some key RTD performance indicators: the increase in the volume of scientific publications; the increase in the number of people with a tertiary education employed in science and technology; and the growth rate in the number of R&D personnel and researchers.

The main drawback reported by the study relates to the lack of observable long-term impacts, as far this concerns the use of research results for technological development and innovation. Evidence shows that the observed, improved scientific knowledge did not translate into technological development and innovation and ultimately did not increase regional competitiveness. Certainly, the economic crisis played a role in reducing the capacities of both public and private organisations to maintain long-term commitments to research and innovation strategies. Still, this was not the only reason. The analysis of the complex causal packages underpinning both the success of individual instruments and the full policy mixes shows that a combination of factors had to be guaranteed. The evidence shows, in particular, that synergy and complementarity with existing funding sources were not always well exploited, which led to specific regulatory measures to overcome the issue in the 2014-2020 and 2021-2027 programming periods. Moreover, administrative failures and legal constraints exposed implementation to delays, uncertainties, rejections and, for some beneficiaries, financial stress in a field where timing, long-term commitments and clear rules were crucial factors for the successful collaboration of engaged actors.

Based on this, the evaluation identified a list of issues that should be considered when designing and implementing RTD policy.

The preparatory phase includes the needs assessment for the RTD landscape and the prioritisation process. It should be based on an in-depth understanding of existing RTD actors (i.e. their capacities and expectations, their willingness and incentive to engage in know-how transfer, and their territorial distribution) and the national institutional and legal framework (i.e. administrative capacities, legal constraints, and policy framework). Specific points of attention should be the following.

·Long-term commitment to public and private investments benefits from clarity regarding the legal framework. In line with their recent efforts, national and EU authorities should guarantee that legislation regarding public procurement, State aid, and other important regulations is sufficiently clear and perceived as such, so that they lead to a smooth implementation of RTD investments. Clear and effective State aid rules are important in ensuring that businesses are eligible for public funding and encouraged to participate in collaborative R&D projects. Administrative burdens related to public procurement should be minimised, and rule changes should be avoided to reduce project delays. Legal constraints and other framework conditions that may prevent appropriate pay levels for researchers are also important obstacles to consider.

·Equally importantly, a clear policy strategy delineating a long-term commitment to public investment in R&D should be established, communicated and maintained over time, thereby reducing fluctuations, particularly in times of crisis. This should include a plan to coordinate the various support programmes in the field of RTD in the region and country, to ensure an effective alignment and complementarity of all funding mechanisms. In this manner, the logic of supporting RTD investments in the long journey for research and innovation, with appropriate coordination, instead of a clear separation of competencies that may lead to fragmentation, can facilitate follow-up investments. A specific role for the ERDF in the broader policy mix should be designed and acknowledged.

·Programme-management units within MAs should be appropriately staffed and trained. Implementing RTD investment support is a demanding task that requires managerial and entrepreneurial capacity. When these are not already in place, especially in less developed regions benefiting from large funding, swift restructuring within responsible administrations should be carried out, with dedicated units equipped with the necessary staff and competencies.

·Investment prioritisation and targeting should be informed by an in-depth understanding of the system failures affecting the regional RTD ecosystem. This should look particularly at existing relationships between science-industry partners in the region and the drivers that can foster an environment enhancing their collaboration and better diffusion of R&D results. RTD supply-side investments should be combined with due consideration of demand-side absorption capacities and constraints. The absorption capacity of the local labour market or the business sector of trained researchers and advanced technological services offered as a result of the planned investments should be considered. Technology transfer offices, or permanent collaboration platforms such as competence centres or cluster organisations, can be promoted in those cases where there are possible mismatches between the research supply and actual local demand. Demographic change can have an impact on the territorial absorption potential of RTD capacities. For example, the emigration of students and researchers due to unfavourable general conditions can dilute the expected local impact of RTD investments and result in a brain drain.

·To improve the sustainability of supported investments, in the design phase, there is the need to enhance the focus on market orientation of research activities to support smart economic transformation.

·Possible trade-offs between excellence and territorial inequality can emerge in the targeting strategy. In a place-based approach, such trade-offs should be addressed by better considering the local relevance of RTD investments to the territorial context, avoiding the promotion of investments motivated by the pursuit of scientific excellence but unrelated to the local business sector and technological capacities.

During programme implementation, it is necessary to ensure a transparent and timely selection procedure, and clear and prompt funds disbursement to prevent delays and ensure that high-quality projects are implemented and produce sustainable results. Positive conditions should be guaranteed and consistently and robustly maintained.

·Instrumental support from advisory and support services may be useful in improving the engagement of stakeholders and ensuring that good-quality projects are prepared. This would also avoid that selection criteria promoting excellence concentrate the funds in a few leading institutions. MAs and implementing bodies are encouraged to guarantee a high level of commitment and willingness to assist in the application process. Capacity building is also essential in developing awareness of industry needs and providing the capacity to transfer knowledge. Communication channels can be activated to present and explain R&D results that may have commercial potential.

·Administrative procedures for fund disbursement should be kept as simple as possible to reduce the administrative burden on beneficiaries and the impact on timely beneficiary payments.

·In selecting infrastructure investments, due attention should be paid to ensure that sufficient, highly qualified R&D and ICT staff can be employed in the new infrastructure.

·For long-term financial sustainability, any beneficiary infrastructure must develop a business model specifying a balanced source of funds without relying unduly on an individual source. It is also essential to maximise the revenue-generating capacity arising from services offered to industrial partners.

·Private partners’ lasting commitment and interest regarding research activities and collaboration with science partners should be promoted and maintained. Measures should be taken to prevent this commitment and interest from being undermined by external shocks or unfavourable contextual conditions.

5.2.Policy response in the 2014-2020 & 2021-2027 programming periods

The 2014-2020 programming period brought forward a new legislative framework for the five ESI Funds. For instance, the ‘thematic concentration of funds’ and the delineation of 11 thematic objectives represented a key element in the reform of the ESI Funds. This principle and the related regulatory requirements sought to align the ESI Funds with the Europe 2020 strategy to integrate the ESI Funds in the broader EU policy framework. It also aimed to increase added value by concentrating the funds on fewer priorities across all Member States and regions. The first four of these thematic objectives 36 constituted key priorities for the ERDF and helped guide ERDF investments in research and innovation. As such, these priorities aimed at addressing some of the shortcomings outlined in this evaluation.

Furthermore, special provisions were developed to encourage Member States to pursue the complementary use of ESI Funds and other EU instruments, and in particular Horizon funding, thereby emphasising synergy. These conditions ensured that Member States pursued the strategic identification of priorities and made better use of the possibility to combine support from different instruments to finance individual operations.

Moreover, Member States were tasked to develop Smart Specialisation Strategies (S3) in entrepreneurial discovery processes so that ESI Funds could be used in a more targeted way, taking into account different challenges, resources and capacities of each region and Member State. For 2014-2020, over 180 smart specialisation strategies were developed, with both the Council of the EU and the European Parliament highlighting the need to further build on this approach 37 . A recent study 38 confirms that such concentration and prioritisation efforts have largely been effective and are based on an objective data-driven identification process that involves key stakeholders from the private, public and research sectors.

Among the most ambitious innovations, a notable one is also the Interregional Innovation Investments (I3) Instrument. It aims at supporting interregional innovation projects in their commercialisation and scale-up phases giving them the tools to overcome regulatory and other barriers and bring their project to investment level.

Upon starting a new programming period (2021-2027), the 11 thematic objectives were reduced to five policy objectives (PO). While investments in all objectives will be supported, the ERDF will pursue the twin transition and prioritise spending that promotes a more competitive and smarter Europe (PO1), in particular under specific objectives 1.1 and 1.4, and a greener, low-carbon economy (PO2). This will also guide investments that will support development of research and innovation capacities and uptake of advanced technologies, foster innovation in small and medium-sized enterprises, and promote digitalisation and digital connectivity and development of skills for smart specialisation, entrepreneurship and industrial transition 39 .

The achievement of the goals under these new priorities will benefit from guidance and synergies with the broader EU strategic context and in particular the European Green Deal. The policy framework for the smarter Europe objective is set out in a number of strategic documents such as the Smart Specialisation Communication , A renewed European Agenda for Research and Innovation , and A New Industrial Strategy for Europe 40 . The policy consequence for ERDF investments is that a stronger emphasis will be put on market orientation and capability building to help less developed region catch up, specifically by concentrating investments both geographically and in terms of beneficiaries and focusing on higher technology readiness levels for more immediate commercial application. Specific focus will also need to be paid to digitalisation as an innovation enabler cutting across all areas, the specific location of SMEs and start-ups, interregional cooperation, and access to high-capacity telecommunication networks.

5.3.Questions for further studies

One of the key aspects of the SWD was the presentation and discussion of the rationales underpinning ERDF interventions, concerning both individual policy instruments and the whole RTD set of investments within a broader policy mix. The 2021 evaluation on which this SWD is based was descriptive on this point, reporting the various claimed justifications, usually based more on an ex-post reconstruction of interviewed programme managers. A more comprehensive and conclusive judgement about the most credible rationales of the different strategies cannot be provided.

To this end, it would be particularly interesting to check to what extent introducing the Smart Specialisation approach in the 2014-2020 programming period may have provided a solid logic to assess the rationale of ERDF support to RTD investments. A longer-term analysis or a back-to-back approach of evaluations of different programming periods would better account for longer-term effects, including the link between infrastructure investment and innovation, and policy development of cohesion policy. It would also help assessing whether the stand-alone approach to investments has effectively been overcome, absorbing research infrastructure into the ecosystem approach.

Linked to the previous point, there is the need to reflect better a ‘system perspective’ in evaluating RTD investments under cohesion policy. One of the main barriers for RTD investments recognised by the literature and indicated by MAs was system failures, particularly the suboptimal interaction of the main actors of regional RTD systems. The study highlighted the importance of understanding the needs, capacities, motivations and interests belonging to the different actors of the system. In this sense, an analysis of the broader system involved in, and affected by, RTD policies in the different regions is necessary to understand the impact or desired impact of both individual instruments and overall investment strategies.

Moreover, very limited evidence could be gathered by this evaluation about the capacity of the ERDF to transform the regional/national systems of actors structurally. The evaluation could instead offer conclusions about a dominant scale effect of the policy, where existing systems performed better or maintained a stable performance but did not move towards a structural transformation in how knowledge is produced, disseminated and exploited. The problem may be due to the fact that systems needed different interventions or that there were not enough synergies between the ones designed. The Smart Specialisation Strategy approach may have provided already an improvement on this point in the 2014-2020 programming period. The strategy focused on bringing together different stakeholders and on the importance of mapping needs, for both enterprises and research providers as part of RTD regional ecosystems, as a starting point in the formulation of the regional strategy.

Also, because of the smart specialisation approach, future evaluation should possibly take a more systemic point of view, first, by mapping regional systems and their investment needs and, second, by assessing the appropriateness of the trajectory of systemic change to which the ERDF has contributed or facilitated. A starting point is provided by the categorisation of activities by theme performed in this evaluation, which proved successful for subsequent aggregated analysis.

The policy response for 2014-2020 and 2021-2027 introduced further aspects that will need specific assessment. Among these, particular importance will be played by the evaluation of effectiveness and efficiency for the full range of financial instruments implemented and for investments in different technology readiness levels s.

As highlighted in the conclusions, implementation capacity remains a problem hampering the smooth and efficient delivery of public investment programmes, especially in countries such as Italy and Romania. This issue is particularly relevant when dealing with RTD investments where the efficient engagement of different actors and stakeholders of the regional ecosystems is a determinant success factor. Administrative capacity has been addressed for a long time as one of the main areas for improvement in the delivery of cohesion policy, but it will deserve further attention in the 2021-2027 period. Future evaluations should dedicate enough attention to considerations on the extent to which there is noticeable long-term improvement in the administrative capacity of regions and the role that the ERDF may have had to facilitate such improvements. In addition, any evaluation should consider the extent to which the design of regional/national strategies and individual policy instruments were taken into account and whether they anticipated possible implementation failures due to administrative capacity issues.

The SWD illustrates how the added value of cohesion policy in the RTD field, intended as the production of EU-wide effects and catching-up phenomena of regional RTD performances to EU standards, comes as a sort of unexpected and unplanned effect of funded interventions. In particular, added value does not appear in the reconstructed ToCs as an explicitly intended effect. Despite this, it constitutes a relevant and visible effect of the policy that MAs should better recognise and pursue as from the programming phase. MAs seem to have a better awareness of the financial scale effect of the contribution of EU funds to regional and national RTD investments rather than the more strategic effect, which has to do with setting standards of performance and promoting common approaches to RTD policies. It would be interesting to investigate to what extent, in the following programming period, the EU added value of RTD policies was an integral aspect of the intervention logic of both individual policy instruments and the policy mix as a whole.

The above questions will be addressed in the ex post evaluation of Cohesion policy programmes 2014- 2020 financed by the European Regional Development Fund (ERDF) and the Cohesion Fund (CF).

A relatively blind spot of this evaluation remained the relation between ERDF support and the European Research Framework Programmes’ support to RTD investments. This evaluation described the existing alignment of policy objectives and implementation procedures, concluding on minor synergy between the two funding streams, which was tackled through specific regulatory provisions for the following programming periods and will be evaluated in the future. For data quality, however, information on beneficiaries collected by this evaluation could not be systematically matched with data on beneficiaries of the Research Framework Programmes. The relation with ERDF support is tackled by a study on the ‘Analysis of key parameters of Smart Specialisation Strategies (S3)’ 41 , which examines the synergies between the ERDF and Horizon 2020. This will not only include a matching of Horizon 2020 funding areas and S3 priorities, but also an analysis of data covering individual operations and beneficiaries, building on the more comprehensive dataset available for the 2014-2020 ERDF and a newly built project database for Horizon 2020.

Annex I. Procedural Information

1.Lead DG and Decide reference

The evaluation was led by the European Commission's Directorate-General for Regional and Urban Policy, DG REGIO Unit B2: Evaluation and European Semester.

Decide entry: PLAN/2018/4614.

2.Organisation and timing

The evaluation roadmap was published on 8t November 2018 for a four-week period until 6 December 2018 for stakeholders and the general public to provide feedback.

An Interservice Steering Group was set up comprising EC services: SG, BUDG, CNECT, COMP and RTD. Four meetings of the Interservice Steering Group were held on: 23 October 2019, 23 April 2020, 18 January 2021, and 13 July 2021.

3.Evidence and sources

Evidence was gathered from a variety of sources, including: operational programmes, implementation reports by the Member States, Major Projects documentation, monitoring systems of the managing authorities, previous evaluations undertaken by the Member States, literature review, interviews with managing authorities, stakeholders, final beneficiaries and independent experts, data sources at NUTS2 level including Eurostat, Patstat and Web of Science.

Annex VI includes a full list of bibliographic references.

4.External expertise

The evaluation was supported by a study run by a consortium led by CSIL – Centre for Industrial Studies, in partnership with Prognos AG and Technopolis Consulting Group Belgium. The service contract started on 1st of September 2019 and the final report 42 was delivered on the 1st of December 2021.

Annex II. Methodology and Analytical models used

1.General methodology

The proposed methodological approach, which is further detailed in the ex-post evaluation report 43 at the basis of this SWD, stems from the ambition to build a theory-based impact evaluation in the specific context of RTD infrastructures and activities. In particular, the role of the “theory” behind the supported interventions (i.e. the overall rationale and expected supporting factors, pre-conditions and risks) was the starting point informing all the evaluation activities, from the literature review to the projects and beneficiary mapping until the case studies and the econometric analysis. In this way, the study went beyond assessing what had happened (i.e., the direct effects of the ERDF support for RTD), and it also tried to provide answers about why and how the observed effects had occurred.

For the assessment of the degree of effectiveness of selected policy instruments, the study followed the approach of Contribution Analysis (CA) (Mayne, 2011), a specific form of theory-based evaluation that focuses on ‘causal relationships and explanatory conclusions between observed changes and specific interventions’ (European Commission, 2013). ‘Theories of Change’ (ToC) are central to this approach. The aim is to provide evidence to reduce uncertainty rather than to define links between interventions and effects. This approach relies on assumptions that should be made visible as both requirements for and limits to our evaluation.

The principles of CA were used to guide the collection and processing of evidence to provide a judgment on the effectiveness of selected policy instruments in the seven case studies. However, the adopted approach went beyond the standard CA approach in several ways, adapting to the specificity of the evaluation and the need to be operationalised in a manageable yet rigorous way that could be implemented consistently by all country experts performing the field analysis. One specific element expanding beyond the standard CA was considering that an intervention works as part of a broader ‘causal package’, comprising the intervention outputs, a set of support factors, preconditions and possible risks or threats. The assessment of the effectiveness of individual policy instruments was therefore designed along with three main steps:

·Assessing what has changed in the performance of the beneficiaries of ERDF;

·Assessing the extent to which the ERDF has contributed to the observed changes;

·Assessing how and via which mechanisms or contextual factors, outcomes and results materialise.

This evaluation was guided by a set of Evaluation Questions corresponding to several evaluation criteria (see Annex III). The evaluation approach relied on a combination of different methods to provide comprehensive answers to the questions. In particular:

·The selection of 53 representative OPs by the European Commission out of the total of 215 OPs funded by the ERDF and covering 18 Member States out of 28 as well as a substantial amount of RTD investments (with EUR 14.64 billion of contribution, i.e., about 85% of the EU total funding for the relevant themes). Within the selected OPs, the evaluation study focused on two categories of ERDF expenditures (01 – RTD activities in research centres, and 02 – RTD infrastructure and centres of competence in a specific technology) 44 .

·A mapping of ERDF expenditures by types of projects and beneficiaries funded by 53 OPs provided a fine-grained description of what was funded on the ground and the logic underpinning the mix.

·An analysis of strategies and objectives pursued by the 53 OPs to understand the funded instruments' motivations and rationales.

·A cluster analysis described the RTD performance of EU regions in the period under assessment.

·A literature review identified prior theories of change regarding ERDF support for RTD investments and their expected results.

·Seven case studies at Member State (MS) level assessed, in detail, a selected number of OPs and their most representative policy instruments (a total of 21 policy instruments were assessed with a contribution analysis). This was done to build more precise theories about the implemented instruments and collect evidence on observed outcomes and results and the conditions for their materialisation. Primary evidence collected via the case studies came from 200 interviews with managing authorities, stakeholders, final beneficiaries and independent experts.

·A cross-case analysis, at the levels of four types of intervention, aggregated and generalised the theories, and results duly crystallised in terms of what works, where, and according to which mechanisms.

·A seminar with stakeholders and experts allowed the discussion of preliminary results;

·An econometric analysis, employing a multivariate regression and using the data collected from funded projects and beneficiaries, allowed the testing of a set of hypotheses about the contribution of the various types of instruments to a set of regional outcomes. The multivariate analysis permits one to isolate the contribution of specific ERDF types of instrument to specific regional outcomes from other potential factors (these ‘other’ factors might include, e.g., regional socio-economic start conditions, other R&D policies beyond the ERDF interventions, etc.). The analysis was not meant to provide an estimation of the size of the observed effects, which was not conceivable given the available data 45 , but to offer a complement to the qualitative analysis of the case studies. In combination with the understanding of the causal mechanisms explored in the case studies, the estimated statistical significance expanded and corroborated their findings on the contribution of ERDF interventions to some observed changes.

Figure 1 - Map of selected case studies and policy instruments

Source: Ex-post evaluation report (2021)

The combination of different methods was instrumental in collecting a comprehensive set of evidence. Their interconnection was built at a sequential level, building on the evidence gained from the previous step and expanding in the directions indicated by the evaluation questions. While some analysis methods were selected because of their appropriateness in answering some of the evaluation questions (case studies, cross-cases analysis, econometric analysis), others were necessary as a starting point for further analysis (mapping of projects and beneficiaries, literature review, analysis of OP strategies). The overall logic went from a general overview to the specific assessment of individual cases, then expanded to a more general level.

Figure 2 - Methodological framework: a combination of methods

Source: Ex-post evaluation report (2021)

2.Limitations and mitigation strategy

Despite the huge amount of data and evidence collected via the evaluation activities, some limitations remain. Among these, the most significant are as follows:

·The scope of the evaluation is limited to investments supported under expenditure codes 01 and 02, which were not included in the ex-post evaluation 46 . Nevertheless, within the target OPs, other codes of expenditure also synergistically contributed to the overarching objective of strengthening the regional RTD systems. This was true, in particular, as far as the contribution of ERDF concerns the objective of bridging the gap between the provision and the use of research results. The distinction between ERDF expenditure codes in this area is rather artificial, also prone to possible miscoding on the part of the MA. Focusing merely on codes 01 and 02 may have limited the capacity of the present evaluation to explore the contribution of ERDF to this strategic objective. Nonetheless, this evaluation could partially address this gap by building on the results of a previous evaluation focused on ERDF support for SME research and innovation (European Commission, 2016b).

·As discussed in section 3 of the SWD and in Annex II.4, the monitoring indicators collected by MAs have limitations both in terms of coverage of the actions supported and in terms of quality (i.e. consistency and gaps). Therefore, the evaluation filled the gaps by making use of additional indicators. Expanding the list of such indicators in further evaluations could further increase the strength of similar analyses.

·Evidence on outcome and impact at the level of individual policy instruments is mostly qualitative and based on stakeholder perception. Respondents' possible optimism biases were mitigated by triangulating among different stakeholders and by complementing primary evidence with secondary data sources.

·A theory-based impact evaluation was performed on 21 individual instruments, as implemented in seven MS. This was, therefore, a limited sample of all implemented instruments. By construction, the contribution analysis aimed to provide evidence to reduce uncertainty regarding the mechanisms underpinning the observed changes, rather than to define links between interventions and observed changes. This approach relies on certain assumptions as both requirements for and limits to our evaluation. A standardised approach to producing the contribution analyses and assessments and to reporting on preconditions, supporting factors and risks, was designed and promoted among the case-study authors. Nevertheless, some degree of discretion may remain in the way judgements are derived, both in terms of the evidential base (particularly, the balance between evidence generated by the case studies and the existing literature) and the interpretation of such evidence and causal mechanisms.

·The poor quality of some beneficiary data (lack of partner beneficiaries for collaborative projects, duplication or missing data) prevented a more in-depth analysis, and it also obstructed systematic matching with external databases (for instance, matching with the CORDIS database could be carried out only for the selected OPs of the case studies).

·The evaluation did not collect information on compliance costs (nor direct or indirect) and enforcement costs due to the fact that this information is not relevant in case of the evaluated interventions which are investments. Similarly, no data was collected on simplification and burden reduction which was not in the scope of this evaluation. Consequently, this SWD does not contain “Annex IV. Overview of benefits and costs and, where relevant, table on simplification and burden reduction” as envisaged by Tool#49 of the Better Regulation Toolbox.

Despite these limitations, the triangulation of data sources and extensive discussions with country experts, senior advisors, and stakeholders enabled the team to gather robust conclusions for most evaluation questions. Some open questions remain; they are discussed in the last chapter of the report.

3.List of the sample of 53 operational programmes

The list of the 53 Operational Programmes covered by the study is presented in the Table below.

Table 1 - List of 53 Operational Programmes

Country	CCI	Name of the OP
BE	2007BE161PO001	Programme opérationnel 'Convergence' Hainaut - FEDER
BE	2007BE162PO003	Programme opérationnel 'Compétitivité régionale et emploi' - Wallonie (hors Hainaut) - FEDER
CZ	2007CZ161PO004	OP Podnikání a inovace
CZ	2007CZ161PO012	OP Výzkum a vývoj pro inovace
CZ	2007CZ162PO001	OP Praha Konkurenceschopnost
DE	2007DE161PO001	Operationelles Programm EFRE Thüringen 2007 bis 2013
DE	2007DE161PO002	Operationelles Programm EFRE Brandenburg 2007-2013
DE	2007DE161PO003	Operationelles Programm EFRE 2007 - 2013 Mecklenburg- Vorpommern
DE	2007DE161PO004	Operationelles Programm EFRE Sachsen 2007-2013
DE	2007DE161PO007	Operationelles Programm EFRE Sachsen-Anhalt 2007-2013
DE	2007DE162PO001	Operationelles Programm EFRE Bayern 2007 - 2013
DE	2007DE162PO003	Operationelles Programm EFRE Schleswig-Holstein 2007-2013
DE	2007DE162PO004	Operationelles Programm EFRE Berlin 2007-2013
DE	2007DE162PO007	Operationelles Programm EFRE Nordrhein-Westfalen 2007-2013
EE	2007EE161PO001	Operational Programme for the Development of Economic Environment
ES	2007ES162PO002	Programa Operativo FEDER del País Vasco
ES	2007ES162PO004	Programa Operativo FEDER de Madrid
ES	2007ES162PO005	Programa Operativo FEDER de La Rioja
ES	2007ES162PO006	Programa Operativo FEDER de Cataluña
ES	2007ES162PO010	Programa Operativo FEDER de la Comunitat Valenciana
ES	2007ES16UPO001	Programa Operativo FEDER de Investigación, Desarrollo e innovación por y para el beneficio de las Empresas - Fondo Tecnológico
ES	2007ES16UPO003	Programa Operativo FEDER de Economía basada en el Conocimiento
FI	2007FI162PO001	Itä-Suomen EAKR-toimenpideohjelma 2007-2013
FR	2007FR162PO001	Programme opérationnel FEDER AQUITAINE
FR	2007FR162PO011	Programme opérationnel FEDER HAUTE-NORMANDIE
FR	2007FR162PO015	Programme opérationnel FEDER LORRAINE
FR	2007FR162PO016	Programme opérationnel FEDER PAYS DE LA LOIRE
FR	2007FR162PO017	Programme opérationnel FEDER NORD PAS-DE-CALAIS
FR	2007FR162PO020	Programme opérationnel FEDER PROVENCE ALPES COTE D´AZUR
FR	2007FR162PO021	Programme opérationnel FEDER MIDI-PYRENEES
HU	2007HU161PO001	Economic Development Operational Programme
IE	2007IE162PO002	Southern and Eastern Operational Programme
IT	2007IT161PO006	Pon Ricerca e competitivita' - Riprogrammazione - 30 ottobre 2014
IT	2007IT162PO002	Por Emilia Romagna FESR Versione approvata dal Comitato di Sorveglianza 2015
LT	2007LT161PO002	2007-2013 m. Ekonomikos augimo veiksmų programa
LV	2007LV161PO001	Entrepreneurship and Innovations
PL	2007PL161PO001	Program Operacyjny Innowacyjna Gospodarka, 2007-2013
PL	2007PL161PO002	Program Operacyjny Infrastruktura i Środowisko
PL	2007PL161PO003	Program Operacyjny Rozwój Polski Wschodniej 2007-2013
PL	2007PL161PO010	Małopolski Regionalny Program Operacyjny na lata 2007-2013
PL	2007PL161PO011	Regionalny Program Operacyjny Województwa Mazowieckiego
PL	2007PL161PO013	Regionalny Program Operacyjny Województwa Podkarpackiego
PT	2007PT161PO001	PO Factores de Competitividade 2007-2013
PT	2007PT161PO002	PO Regional do Norte 2007-2013
PT	2007PT161PO003	PO Regional do Centro 2007-2013
RO	2007RO161PO002	Sectoral Operational Programme Increase of Economic Competitiveness
SI	2007SI161PO001	Operativni program krepitve regionalnih razvojnih potencialov za obdobje 2007 - 2013
SK	2007SK16UPO001	OP Research and Development
UK	2007UK161PO002	West Wales and the Valleys ERDF Convergence programme
UK	2007UK161PO003	Cornwall and the Isles of Scilly ERDF Convergence programme
UK	2007UK162PO001	Lowlands and Uplands of Scotland ERDF Regional Competitiveness and Employment programme
UK	2007UK162PO008	Northwest England ERDF Regional Competitiveness and Employment Operational Programme
UK	2007UK162PO009	Yorkshire and Humberside England ERDF Regional Competitiveness and Employment programme

Source: Author

4.Overview of core and common indicators on RTD

In the framework of the past 2007-2013 Cohesion Policy, monitoring and evaluation activities were asked to be performed by regulation 1083/2006 (Art 37.c). The relevance of the indicators is “to make it possible to measure the progress of each Priority Axis of the Operational Programme in relation to the baseline situation and the achievement of the targets”. For each result and output indicator, the baseline and target values were required to be established by the Managing Authorities. The progress towards achieving targets established for each indicator has to be reported by the Member States in the Annual Implementation Report (as required by Art 29 of the EC Reg. 1083/2006) and the financial execution tables.

While result and output indicators were mandatory, impact indicators were strongly encouraged but not formally required. Member States were free to identify and use the most appropriate indicators according to Programme objectives and the programming period's strategic focus. While providing the Member States with this responsibility, the European Commission also strongly encouraged them to use a limited number of “core indicators” (output and result), aggregated and compared at the EU level. 47 Out of the full list of 56 core indicators indicated by the European Commission, three are directly and explicitly referred to research and technological development (RTD), i.e.:

·‘number of RTD projects (4)’

·‘number of cooperation projects enterprises – research institutions (5)’

·‘research jobs created (preferably 5 years after projects start) (6)’.

Additionally, during the 2014-2020 programming period, the Commission recommended using an additional 46 common indicators, some of which had already been used by many OPs in the previous programming period. The common indicators relevant to the RTD investments are:

·‘number of new researchers in supported entities (24)’,

·‘number of researchers working in improved research infrastructure facilities (25)’,

·‘number of enterprises cooperating with research institutions (26)’

·‘private investments matching public support in innovation or R&D projects (27)’.

In addition to these, each OP included various programme-specific indicators selected by the Managing Authorities to keep track of the achievements concerning the specific objectives set in the programme.

Previous evaluation studies have stressed that the achievement indicators of the regional monitoring systems have some limitations: they are generally not available at the level of individual projects or policy instruments and could refer to projects funded under other categories of expenditure than 01 and 02; the target indicators can be flawed making the comparison with the actual achievement indicators not fully reliable; it is not possible to compare the programme specific indicators across different OPs; being focused on the programme outputs and results (and not on the impact), the achievement indicators are not sufficient for a complete evaluation of effectiveness.

Despite this, these indicators are a source of evidence (even the only one for some OPs) to have some signs of what has been reported as having been achieved by the OPs.

5.Econometric analysis

The ex-post evaluation report employed multivariate regression analysis to support the findings in the main report. The multivariate analysis permits one “to isolate” the contribution of the ERDF types of instruments in the scope of the study to specific regional outcomes from the other potential factors (e.g., regional socio-economic conditions, other R&D policies beyond the ERDF instruments, etc.) influencing those outcomes (see box below).

The regression equation estimated is as follows:

Y_i=α+βX_i+γZ_i+ε_i

Where:

·Y_i indicates the outcome (dependent) variable we want to explain in the NUTS2 region i (for instance, the growth rate in the number of scientific publications);

·X_i is the ERDF type of instrument (policy variable) we are interested in;

·Z_i is a vector of controls, i.e. variables that can influence the outcome variable beyond the ERDF instrument (for instance, the Gross domestic expenditure on R&D in the region);

·ε_i is an i.i.d error term.

·α,β, and γ represent the parameters (coefficients) of the model to be estimated and measure the correlation between the regressors (X_i,Z_i) and the outcome variable. If the ERDF policy variable had an impact on the outcome variable, then the coefficient β in the equation above is expected to be positive and statistically significant. α is the constant of the model.

Specifically, the ex-post evaluation tested a set of hypotheses about the contribution of three ERDF types of instruments implemented in the 2007-2013 programming period to a set of regional outcomes. The ERDF types of instruments examined (see the ex-post evaluation report for the rationale behind their selection) were:

·ERDF expenditure in infrastructures for research and individual R&D projects in universities / Higher Education Institutions (HEIs);

·ERDF expenditure in infrastructures and investments for education;

·ERDF expenditure in infrastructures for research and in private-public collaborative R&D projects that involve enterprises. This instrument is targeted to Research and Technology Organisations (RTOs), Enterprises, Clusters; Science Parks, Consortia, Competence and/or excellence centres.

As explained in detail in the ex-post evaluation report, each ERDF type of instrument had its specific instrument logic, its policy objectives and targeted different beneficiaries. Accordingly, depending on the instrument and related hypotheses, several regression analyses with different dependent (outcome) and control (context) variables were run.

Building on the findings of the case studies, the following hypotheses were selected for testing:

·HP1: Regions, especially lagging ones, improved their basic research capacities by supporting universities' infrastructure investments and research projects. This contributed to an increase in the R&D personnel in the region and an improvement in scientific production. This process has been further supported by an increase in public expenditures in R&D but did not immediately improve scientific excellence.

·HP2: Lagging regions investing in infrastructure for education attracted more students and improved the tertiary attainment in the region. This contributed to an increase in employees in science and technology, but only in those regions with an already advanced industrial fabric.

·HP3: Regions investing a larger share of funds in RTO, science-industry collaborations or centres of excellence (either infrastructures or activities) experienced an increase in the level of patent applications and/or other intellectual property rights and public-private co-publications. This holds true in those regions with a high concentration of funds among beneficiaries and regions with a more mature R&I system.

The ex-post evaluation assembled a database at the regional NUTS2 level from several sources, including Eurostat, Patstat, Web of Science to test the hypotheses. The sample includes 104 EU regions (NUTS2) covered by selected Operational Programmes (OPs) in the scope of the evaluation: 46 OPs with the highest ERDF expenditure in the Cohesion Policy themes of expenditures 01 and 02. 48

Hypotheses are verified employing regression analysis. The main variables entering the models vary according to the specific hypothesis and type of instrument under scrutiny 49 .

Annex IV to the ex-post evaluation reports on the full results. It is organised into three main sections, each one devoted to the analysis of one hypothesis. Tables with related statistics and econometric results are reported, and, at the end of each section, the main findings are summarised.

Annex III. Evaluation matrix

RELEVANCE

The relevance criterion has to do with the relationship between the needs to be tackled and the objectives of the overall ERDF strategies and related policy instruments. More specifically, it touched on aspects of the design of the programmes and identified whether there was a mismatch between the ERDF mix of policy instruments for RTD and the barriers to research and technological development identified by the programmes.

Questions/sub-questions

Conclusions

Source of evidence

Level of analysis

EQ 1: What are the interventions supported by ERDF?

› ERDF support in RTD investments funded mainly infrastructure construction and modernisation, including ICT, to improve education and research activities. These investments took the lion’s share of ERDF contributions. In total, excluding Spain, 55% of projects financed R&D projects, both individual and collaborative. Other interventions were internationalisation of research and capacity building for research, investments for science and dissemination, intellectual property rights instruments and operating subsidy (see Section 3.1).

› Mapping of projects and beneficiaries (Task 1)

› OP and MS level (46 OPs and 17 MS)

EQ 2 How is ERDF support divided between the different types of intervention and forms of financing?

› The largest share of ERDF expenditure, more than EUR 9 billion (72% of total), was concentrated on support for infrastructure investments, with infrastructure investments for research absorbing more than half of ERDF expenditure (57%). R&D projects, both individual and collaborative, represented the most common type of intervention in terms of the number of projects, but only 23% of the total ERDF expenditure. A smaller share of projects (6%) and expenditure (3%) was allocated to the implementation of other sorts of RTD activity, while a residual portion (3% of ERDF expenditure) funded activities that were not strictly related to RTD and should instead have been classified under different codes (see Section 3.1).

› The selected 53 OPs supported more than 20,000 projects, almost half of which were in Spain. Almost all projects were funded through non-repayable grants (see Section 3.1).

› The majority of beneficiaries were publicly owned organisations. About 4,000 different institutions can be identified among the almost 24,000 lead beneficiaries (about 2,000 different institutions including almost 580 HEIs, more than 720 RTOs, nearly 470 enterprises, when excluding Spain) (see Section 3.2).

› Mapping of projects and beneficiaries (Task 1)

› OP analysis (Task 1)

› Project level (46 OPs)

› OP level and MS (53 OPs and 18 MS)

EQ 3: What is the underlying rationale, also considering the role of ERDF support in the policy mix?

Overall:

› The rationale for public intervention in RTD stems from the need to tackle multiple investment needs. According to the literature, such needs refer to specific market mechanisms preventing long-term investments in RTD because of the indivisible, inappropriable and uncertain nature of research (market failures) as well as key deficiencies in the actors producing research and in overall systems (systemic failures) (see Section 2.1).

› According to the OP analysis, some systemic failures were the main rationale for RTD support. The need to tackle infrastructure gaps and failures and to facilitate improvements in science-industry collaboration comprised the logical foundation of the interventions of most of the programmes (87% and 79% respectively), despite the wide variety of territorial contexts and research systems. Differences were instead observed in the way the OPs translated their strategic approaches into policy mixes. Similar territorial contexts saw the adoption of different combinations of instruments (see Section 2.4).

By policy instrument:

› Infrastructure investments for research aimed at addressing the lack of sufficient or modern physical and technological infrastructure, an essential component in fostering knowledge creation (see Section 3.3.1).

› Infrastructure investments for education were geared more towards improving university education facilities than towards RTD laboratories (see Section 3.3.2).

› ICT-based infrastructures aimed at providing digital-based services and tools for data and computing-intensive research in virtual and collaborative environments (see Section 3.3.3).

› Collaborative projects had various aims, ranging from addressing industrially relevant or societal challenges, stimulating technological advancement in specific areas, and boosting international cooperation by conducting internationally competitive high-quality R&D activities (see Section 3.4.1).

› Individual research projects had the objective to strengthen the scientific and technological capacity of the supported region (see Section 3.4.2).

› Documentary and literature review (Task 2)

› OP analysis (Task 1)

› Case studies (Task 3)

› OP level and MS (53 OPs and 18 MS)

› PI level (21 PIs)

EQ 4: Is ERDF support based on research demand (bottom-up), or does it focus on the availability of support services and infrastructure and gaps in these (top-down)?

› In project selection, infrastructure investments largely followed a top-down approach guided by national road-mapping exercises (see Section 3). R&D projects, conversely, primarily followed a bottom-up approach within well-identified priority scientific or technological fields (see Section 4).

› OP analysis (Task 1)

› Case studies (Task 3)

› OP level and MS (53 OPs and 18 MS)

EQ 5: How was investment targeted in respect of geographical areas and sectors: to those with significant potential or comparative advantage or those in difficulty or lagging behind?

› While broader territorial targeting strategies were driven by eligibility criteria targeting lagging territories, targeting strategies of specific OPs (especially nationally) or instruments did not include an explicit geographical component and were rather ‘territorially agnostic’. Overall, the funds were mainly addressed to strengthen existing territorial excellence, even in countries with strong regional disparities in RTD (such as Romania) (see Sections 3.7 and 5.2.1).

› In many cases, ‘target priority’ sectors and technologies reflected regional specialisation and were identified by existing policy strategies and documents, either at national or regional levels (see Sections 3.7 and 5.2.1).

› In most cases, the target OPs funded beneficiaries with a competitive advantage, with a high concentration within individual beneficiary organisations leading institutions in their field (see Sections 3.7 and 5.2.1).

› Mapping of projects and beneficiaries (Task 1)

› Case studies (Task 3)

› OP level and MS (9 OPs and 7 MS)

EQ 6: Did the ERDF interventions match, or respond to, the policy challenges?

Overall:

› Despite differences in terms of policy challenges between Central and Eastern European countries on the one hand and Western EU countries on the other, the key strategic objectives pursued by the selected OPs and related PIs were to fill the infrastructure gap and to improve the systematic interaction among regional actors. This was in line with the main systemic failures identified by the different OPs (see Section 3.6).

› ERDF support for RTD responded to the main policy challenges and external challenges, such as the economic crisis. It helped public research infrastructures and businesses to withstand the crisis by providing a significant source of funds, sometimes palliating a decrease in national public support. This holds true particularly in those countries and regions most severely affected by the crisis (see Section 5.1.1).

By policy instrument:

› Infrastructure investments for research and education upgraded existing infrastructure and equipment and replaced obsolete or outdated instances in Higher Education Institutions and RTD organisations (see Section 3.3.1 and Romania, Czechia, Estonia and Poland Case Study).

› ICT-based infrastructures established or improved computing grids, data-storage centres, open-data infrastructures, ICT network systems and e-infrastructures (see Section 3.3.1 and the Czechia and Germany Case Study).

› Collaborative projects consisted of projects between R&D institutions themselves and with private-sector partners carrying out research activities mainly with technological and innovation potential (see Section 3.4.1. and Romania, Poland and Italy Case Study).

› Individual research projects consisted of both early-stage (foundational) and exploratory research aimed to generate new knowledge and develop innovative skills in research institutions and projects with a predetermined commercial application (see Section 3.4.2 and the Portugal and Germany Case Study).

› Mapping of projects and beneficiaries (Task 1)

› OP analysis (Task 1)

› Case studies (Task 3)

› Cross-case studies analysis (Task 4)

› Seminar (Task 5)

› OP level and MS (53 OPs and 18 MS)

› PI level (21 PIs)

Overall assessment on RELEVANCE

ERDF support for RTD was overall relevant. The combinations of policy instruments were designed to respond to a wide range of needs, mainly related to RTD capacities, identified in the programming documents and confirmed by the literature and the cluster analysis. More specifically, the ERDF supported massive investments to address infrastructural gaps and, to a lesser extent, tackled difficulties in the interactions of the innovation system actors.

The majority of RTD interventions were geared at supporting excellence objectives, targeting more advanced territories, stronger sectors and best performing institutions within eligible territories. Although this approach was justified by the need to ensure critical mass, fund absorption and knowledge externalities in more mature territories, the question of whether this approach may have contributed to increasing the territorial divide vis-à-vis lagging regions remains open.

EFFECTIVENESS

Effectiveness assesses the extent to which selected policy instruments have successfully achieved or progressed towards the stated objectives and delivered the expected outputs, outcomes, and impacts. Effectiveness assessed the extent to which:

·selected policy instruments have been successful, also in combination with other EU and national support for RTD, in achieving or progressing towards the stated objectives and delivered the expected results;

·the ERDF policy mix for RTD has been effective in improving RTD performances of funded regions.

The criterion also analysed the main factors influencing the effectiveness of RTD interventions.

Questions/sub-questions

Conclusions

Source of evidence

Level of analysis

EQ 7: Have research projects achieved their intended objectives?

› Infrastructure investments for research and education contributed to the creation or modernisation of public R&D facilities (including universities), which in turn increased the potential and capacity of the beneficiary institutions and created more ‘respectable’ research and education environments, thus attracting new students and researchers (see Section 4.2, 4.3 and the Czechia, Estonia, Poland and Romania Case Study).

› ICT-based infrastructures enabled the higher storage, computational and information capacities of R&D institutions, thus improving the availability of scientific information resources and keeping up with the always faster progressing digitisation (see the Czechia and Germany Case Study).

› Collaborative projects generally boosted cooperation between science and industry actions, thus favouring a knowledge exchange process, but more limited evidence is available regarding the capacity of funded projects to generate economic benefits obtained from the commercial valorisation of R&D results (see Section 4.5 and Romania, Poland and Italy Case Study).

› Individual research projects helped develop high-level scientific activities and consolidate relevant knowledge in the scientific and technological system (see Section 4.4. and the Portugal and Germany Case Study).

› Mapping of projects and beneficiaries (Task 1)

› Case studies (Task 3)

› Cross-case studies analysis (Task 4)

› Seminar (Task 5)

› Econometric analysis (Task 6)

› PI level (21 PIs)

EQ 8: How effective were the different groups of ERDF interventions for RTD infrastructure and activities, and how they were combined with other RTD support?

› Infrastructure investments targeting HEIs had a significant positive effect on the number of tertiary-educated persons and employment in science and technology, especially in those regions with more advanced industrial fabric and with higher R&D in the business sector (see Section 4.2 and 4.3) and where continued public investment in science, technology and innovation in combination with ERDF support was ensured (see the Estonia and Portugal Case Study).

While the level of collaboration has generally improved or remained stable, the role of industry actors in the uptake of RTD results has not significantly changed. › Collaborative R&D projects were less effective in improving the interactions between the different actors of the RTD system. This happened although collaborative R&D projects were combined with infrastructure investments which should improve the attractiveness of RTD institutions and lay the groundwork for science-industry collaborations (see Section 4.4 and Italy, Romania and Poland Case Study).

› Mapping of projects and beneficiaries (Task 1)

› Case studies (Task 3)

› Cross-case studies analysis (Task 4)

› Seminar (Task 5)

› Econometric analysis (Task 6)

› PI level (21 PIs)

EQ 9: Is there a specific impact associated with certain types of interventions?

› Infrastructure investments and individual projects contributed to an increase in the number of R&D personnel and researchers at the regional level; infrastructure investments for education were also key to increasing the number of students and tertiary attainments. Those infrastructure investments targeting HEIs also contributed to an increase in scientific outputs. Specifically, ERDF infrastructure investments significantly contributed to the catching-up process of the EU13 regions in terms of scientific output. This was partially due to the magnitude of the investments in HEIs and certain other factors (national public expenditures in particular). Lagging regions performed better than more developed regions in terms of growth in publications (see Section 4.3).

› Collaborative R&D projects contributed to the increase of scientific and technological knowledge and competencies among beneficiaries. However, even when results were produced, they generally remained unfeasibly distant from an industrial application (see Section 4.4).

› Mapping of projects and beneficiaries (Task 1)

› Case studies (Task 3)

› Cross-case studies analysis (Task 4)

› Seminar (Task 5)

› Econometric analysis (Task 6)

› PI level (21 PIs)

EQ 10: To what extent did the support generate additional innovation or output in the supported entities and growth and development in the regions?

› Overall, while ERDF support played a role in the modernisation of RTD systems by supporting the creation of more advanced and competitive research systems, it was less effective in translating this increased research capacity into more competitive territories and regional economies. The cluster analysis highlighted that a decade after the launch of the 2007-2013 programming period, half of supported regions had not changed their relative RTD performance. Performance improvements were concentrated in stronger regions, but some transition regions also saw a catch-up dynamic. In those regions where the economic crisis hit more profoundly, the ERDF support for RTD did not result in a leap in RTD performance; however, it played a countercyclical role, representing a ‘safety belt’ for many beneficiaries (see Section 7.3).

› Cluster analysis (Task 1)

› Case studies (Task 3)

› Cross-case studies analysis (Task 4)

› Seminar (Task 5)

› OP level and MS (53 OPs and 18 MS)

EQ 11: Which were the underlying factors for impacts generation?

› A number of contextual factors played a crucial role (as pre-conditions, supporting factors or risks) in explaining what worked and how. Long-lasting strategic and financial commitment to investment priorities, both for private and public organisations, was key as it allowed for follow-up projects to take place. Clarity about the ‘rules of the game’, shared within the common RTD space by science and industry partners, was decisive for successful partnerships and effective implementation. Administrative and managerial capacities were crucial for effective public spending as they ensured the high scientific quality of selected projects and their timely selection and funding (see Section 5).

› Case studies (Task 3)

› Cross-case studies analysis (Task 4)

› Seminar (Task 5)

› OP level and MS (9 OPs and 7 MS)

Overall assessment on EFFECTIVENESS

ERDF support for RTD effectively contributed to the consolidating and modernisation of existing RTD systems, also favouring a catch-up process of EU13 countries on EU RTD standards. It was less effective in transforming the knowledge base of regional economic systems and ultimately improving target territories' long-term competitiveness. In less developed regions, the consequences of the economic crisis were more severe, and ERDF support has certainly played a countercyclical role. Regarding the effectiveness of specific policy instruments, while infrastructure investments and individual R&D projects generally matched their intended objectives and intermediate results, collaborative projects were not always effective in consolidating the role of industrial partners in the RTD system and their uptake of research results. In this regard, the lack of continued public funding and administrative and managerial capacities issues have had a negative impact on effectiveness.

COHERENCE

Coherence was assessed from three perspectives:

·internal coherence, which focused on assessing whether different interventions under the same ERDF OP or across different ERDF OPs within the same region/country were coherent and complementary;

·external coherence, which focused on determining whether ERDF support to RTD infrastructures and activities were coherent and complementary with respect to other EU and regional/national policies (including the EU Research Framework Programmes);

·influence of EU State Aid rules on the choice of interventions under ERDF.

Sub-questions

Conclusions

Source of evidence

Level of analysis

EQ 12: Did EU State Aid rules influence the choice of interventions?

› The influence of State Aid was more evident in the implementation of policy instruments rather than in their design. Managing Authorities adapted their instruments to avoid any potential conflicts with, or infringement of, such rules, for instance, by limiting the involvement of the private sector or by selecting projects that could not produce factual findings that would result in a competitive advantage for certain companies. This limited the possibility of involving the private sector either as a direct beneficiary or as users of funded infrastructures, with negative impacts on the uptake of research results and strengthening of science-industry partnerships (see Section 5.2.2).

› The need to ensure coherence with State aid rules proved to be a challenge in some countries because of the unclear interpretation and changes in legislation over the period. The limited degree of alignment and even conflicts between competition and cohesion policy was the weakest aspect of coherence in the 2007-2013 programming period (see Section 5.2.2).

› Documentary and literature review (Task 2)

› Case studies (Task 3)

› OP level and MS (9 OPs and 7 MS)

EQ 13: What was the role of the policy mix's links with the Research Framework Programmes (FP7 and Horizon2020)?

› Despite a very high level of coherence in terms of overall policy goals, the level of synergy between ERDF RTD support for RTD investments and the European Research and Innovation framework programme was found to be limited. The two funds were conceived as being highly complementary, but they followed different rationales and operational arrangements that somehow hampered a systematic and intended combination of funds (see Section 5.1.4).

› Despite there was no active strategy for combining the two sources of funds, matching data from the ERDF beneficiaries mapped by this study with FP7/H2020 beneficiaries from the CORDIS database shows that the proportion of ERDF beneficiaries that also benefited from FP7/H2020, at least in some countries, was rather significant (see Section 5.1.4).

› OP analysis (Task 1)

› Documentary and literature review (Task 2)

› Case studies (Task 3)

› OP level and MS (53 OPs and 18 MS)

EQ 14: What is the mix of RTD policy measures that MS implemented in the period (including regulatory incentives or national schemes if they play a role in the programmes)? What is the role of ERDF in this policy mix?

› ERDF support for RTD was generally coherent with national strategies, both RTD and industrial competitiveness strategies. Strategic coherence in priority sectors and scientific fields was particularly strong, but there was a lack of long-term commitment regarding specific RTD strategies and the related political stability and predictability of national policies. In more operational terms, there was a general tacit division of goals between local and ERDF policies and instruments, with coordination mainly driven by co-financing obligations and a great effort towards avoiding overlapping (see Section 5.1.2).

› There was a significant degree of coordination across interventions carried out in the framework of ERDF. This applied to coordination across different ERDF OPs (national and/or regional) and the different axes, measures, and instruments implemented in the framework of individual OPs (see Section 5.1.3).

› There are also positive examples of the combination of ERDF and ESF funding. Synergies were particularly strong in the regions where the ERDF emphasised infrastructure investment in tertiary education, as in Poland, Estonia and Slovakia (see Section 5.1.5).

› Documentary and literature review (Task 2)

› Case studies (Task 3)

› Cross-case studies analysis (Task 4)

› OP level and MS (9 OPs and 7 MS)

EQ 15: To what extent are interventions organised to maximise their combined effects, considering the different underlying goals?

› Interventions funded with different EU and national/regional funds were mainly implemented in light of a separation of objectives and approaches, with more attention to avoiding overlapping than building on relative strengths and maximising the joint effects.

› While instruments supported by 01 and 02 ERDF categories of expenditure mainly targeted research providers to improve their capacities, the other expenditure codes primarily reflected the targeting of SMEs, with the principal aim of supporting innovation processes (see Section 5.1.3).

› EU-level cooperation on research projects was undertaken mainly through the FP7/H2020 programme, while ERDF was essentially seen as an instrument for regional and national cooperation (see Section 5.1.4).

› RTD programmes supported by the ERDF were often closely linked to objectives of industrial competitiveness, as indicated by the strict links between the ERDF interventions for RTD and those of national and regional strategies for cluster development, business innovation, support. The ERDF was also instrumental in supporting national and regional strategies of economic conversion or transition from an industrial economy towards a diversified economy. Despite a high strategic alignment, however, there was often an implicit division of goals between local and ERDF policies and instruments (see Section 5.1.2).

› Documentary and literature review (Task 2)

› Case studies (Task 3)

› OP level and MS (9 OPs and 7 MS)

Overall assessment on COHERENCE

ERDF support for RTD was strongly in line with other support policies, such as Framework Programmes and other ERDF and national/regional support. Therefore, the coherence of policy strategic objectives was relatively high with the FP7, ERDF support for business innovation, ESF support, and national RTD support. However, synergies and coordination between ERDF support for RTD and other types of RTD funding were not always ensured in practice. No active strategy for combining different sources of funds was generally implemented, except for ERDF and ESF OPs. Significant challenges were reported in terms of coherence between cohesion and competition policies. Uncertainties in the rules or risk-aversion interpretation of the legal framework limited the involvement of private businesses in implemented projects and hampered their capacity to use the services provided by funded research infrastructure.

EFFICIENCY Efficiency did not address the wider aspect of value for money considerations but concentrated on two aspects. Firstly, it assessed the scale of funding and the use of financial resources under the angle of the concentration of ERDF funds to ensure critical mass. Second, it also explored the issue of administrative capacity and speed in funds absorption in selected case studies.
Questions/sub-questions	Conclusions	Source of evidence	Level of analysis
EQ 16.1: Was the funding sufficiently concentrated on making a perceptible difference to pursuing policy objectives (including when combined with other instruments or sources of support)?	› There was a concentration pattern on stronger territories, sectors and leading institutions (see Section 5.2.1). › ERDF support for RTD was overall sufficiently concentrated to lead to upgrades in both the quality of research infrastructure and research management capacities in most of the countries under investigation. Its role as “game-changer” or “needle mover” in terms of RTD performance in beneficiary countries and regions was strongly related to the importance of ERDF in the overall national and regional RTD policy mix. In cases where ERDF represented a limited share of the total national R&D spending, but where there was a high regional concentration of this spending (e.g., in convergence regions), its role was crucial to develop critical mass in specific areas, sectors and types of beneficiary organisations (see Section 5.2.1 and the Czechia, Romania, Poland, Italy Case Study).	› Case studies (Task 3) › Cross-case studies analysis (Task 4)	› OP level and MS (9 OPs and 7 MS)
EQ 16.2: Were there sources of inefficiencies in the way funds were managed and disbursed?	Some implementation issues, mainly related to limited administrative capacity or unclear legal framework, were reported especially for collaborative R&D. Delays in project selection and funds disbursements, especially in Italy and Romania, which impacted the capacity of funded projects to generate benefits. Uncertainties in the interpretation and application of rules, especially regarding State-aid rules, caused delays and generated confusion and adjustments during the implementation process.	› Case studies (Task 3) › Cross-case studies analysis (Task 4)	› OP level and PI level (9 OPs and 21PIs)
Overall assessment on EFFICIENCY	ERDF support for RTD helped reach critical mass in most of the target territories, especially in convergence regions where it represented the main funding source of RTD investments. It should also be highlighted that some inefficiencies in the implementation of ERDF interventions impacted their effectiveness.

EU ADDED VALUE The EU added value criterion analysed the beneficial impacts attributed to EU intervention, over and above what could reasonably have been expected and achieved from the action of Member States at the national and/or regional level. Moreover, it provided evidence on the potential EU- wide effects of ERDF interventions in the field of RTD.
Sub-questions	Conclusions	Source of evidence	Level of analysis
EQ 18: What additional value results from the EU intervention compared to what could have been achieved by MS at a national, regional and local level?	› The main EU added value recognised by MAs was a scale effect produced by the access of a considerable quantity of financial resources, especially in the EU13, where ERDF 2007-2013 programmes represented the first systematic set of interventions addressed to the research field after years of underinvestment and limited political priority (see Section 7.1). › Another aspect highlighting the EU added value effect of ERDF support lies in the ambition of supporting research infrastructures of EU scientific and research relevance operating at EU level standards (see Section 7.2).	› OP analysis (Task 1) › Case studies (Task 3) › Seminar (Task 5)	› OP level and MS (53 OPs and 18 MS)
EQ 19: What was the impact of the interventions on cooperation between regions and Member States within the EU?	› A missed opportunity was the lack of the systematic promotion of interregional or international research collaborations as a potential EU added value. Partnerships of collaborative R&D projects were mainly regional or, albeit only in selected cases, multi-regional within the same country (see Section 7.2).	› Case studies (Task 3) › Seminar (Task 5)	› OP level and MS (53 OPs and 18 MS)
EQ 20: Did the interventions achieve any other EU-wide effects?	› EU-wide effects were not among the directly intended effects of funded instruments. Thus, the contribution of ERDF support to them was more indirect, and it occurred through the development of EU-level research communities in specific fields, enabling the construction or upgrading of strategic infrastructures of pan-European relevance (as the later inclusion into the ESFRI roadmap confirms) and also supporting the internationalisation of research communities (see Section 7.2).	› Case studies (Task 3) › Seminar (Task 5)	› OP level and MS (53 OPs and 18 MS)
Overall assessment on EU ADDED VALUE	ERDF support for RTD produced a scale effect that would not have been achieved by national sources alone, especially in those countries and regions generally suffering from underinvestment in the field of RTD policy (all EU13 countries and convergence regions). Despite not specifically intended at producing EU-wide effects, ERDF support for RTD indirectly contributed to the development of EU-level research communities. It helped structure and consolidate a European Research Area by promoting EU standards in RTD capacities and production. This can be claimed to have been the main EU added value of the ERDF support to RTD investments in the period 2007-2013.

Annex IV. Stakeholders consultation - Synopsis report

In compliance with the Better Regulation guidelines, this Synopsis Report outlines the delivery and results of the consultation activities organised in the context of the ex-post evaluation of investments in Research and Technological Development (RTD) infrastructures and activities supported by the European Regional Development Funds (ERDF) in the period 2007-2013.

The first section of this Synopsis Report outlines the objectives, tools, and stakeholder participation. The second section analyses the delivery and results by type of consultation activity: roadmap consultation, interviews, and seminar.

1.Strategy

Objectives

The general objectives of the consultation activities were to:

·Disseminate the activity and stimulate stakeholder participation in the evaluation process;

·Ensure transparency of the process; enhance accountability and transparency of the European Commission activities;

·Stimulate stakeholders to respond to the enquiries and the other knowledge gathering activities to fill information gaps;

·Gather general public end expert stakeholders opinions on the principal evaluation findings; and to

·Publicise the main evaluation findings.

Tools

The stakeholders targeted by the consultation activities were first mapped by and then matched to different consultation tools. In particular, three sets of activities with different purposes were organised:

·The evaluation roadmap was published by the EC in order to gather feedback from the general public;

·Semi-structured and in-depth interviews were implemented by the contractor in order to gather OP-level evidence; and

·A seminar, divided into an expert workshop and a validation webinar, was organised by the contractor in order to discuss.

Stakeholder participation

A summary of the final stakeholder participation figures is provided in the table below.

Table 2 – Stakeholder reach by activity

Activity	Key numbers
Roadmap consultation	1 respondent, Wallonia (Belgium) government
Interviews	103 stakeholders (mainly Managing Authorities and Intermediate Bodies) – Semi-structured interviews
	200 stakeholders (43 Managing Authorities, 135 direct and final beneficiaries, 14 EU/national/regional authorities involved in the oversight of ERDF funds and 8 other relevant stakeholders, such as business associations, etc.) – In-depth interviews
Seminar	65 participants, including EC officers, representatives of Managing Authorities and direct/final beneficiaries – In-depth expert workshop
	200 participants from a broader audience of interested RTD policy stakeholders from Member States, pan-European institutions, and European Commission officials – Validation workshop
TOT	569 stakeholders reached (individual stakeholders might have taken part in more than one activity)

Source: Author

2.Delivery and results

Roadmap consultation

After internal consultation of EC services, the evaluation roadmap was published for feedback on the official EC website. 50 The roadmap aimed to inform citizens and stakeholders about the Commission's work to allow them to provide feedback and to participate effectively in planned consultation activities.

The roadmap was open for public feedback over the period 08 November 2018 - 06 December 2018. However, there only was one reply, coming from the Wallonia regional government of Belgium.

Interviews

Semi-structured interviews

A total number of 103 interviews with the Managing Authorities were to collect further information and critically discuss the OP logic, beyond the intentions expressed in the programming documents.

The interviewees were selected in order to insure a balanced representation of the 53 Ops included in the analysis. A list of the interviewees is provided in the table below, reporting only the OP code and name in order to preserve anonymity.

Table 3 - List of semi-structured interviews conducted

OP code and name

2007BE161PO001 -
Programme opérationnel 'Convergence' Hainaut -
FEDER

2007BE162PO003 -
Programme opérationnel 'Compétitivité régionale et emploi' -
Wallonie (hors Hainaut ) -
FEDER

2007CZ161PO004 -

OP Podnikání a inovace

2007CZ161PO012 -
OP Výzkum a vývoj pro inovace

2007CZ162PO001 -
OP Praha Konkurenceschopnost

2007DE161PO001 -
Operationelles Programm EFRE Thüringen 2007 bis 2013

2007DE161PO002 -
Operationelles Programm EFRE Brandenburg 2007-2013

2007DE161PO003 -

Operationelles Programm EFRE 2007 -2013 Mecklenburg- Vorpommern

2007DE161PO004 -

Operationelles Programm EFRE Sachsen 2007-2013

2007DE161PO007 -
Operationelles Programm EFRE 2007-2013 in Sachsen-Anhalt

2007DE162PO001 -
Operationelles Programm EFRE Bayern 2007 -
2013

2007DE162PO003 -
Operationelles Programm EFRE Schleswig-Holstein 2007-2013

2007DE162PO004 -
Operationelles Programm EFRE Berlin 2007-2013

2007DE162PO007 -
Operationelles Programm EFRE Nordrhein-Westfalen 2007-2013

2007EE161PO001

Operational Programme 'Development of Economic Environment'

2007ES16UPO001 -

Programa Operativo FEDER de Investigación, Desarrollo e innovación por y para el beneficio de las Empresas - Fondo Tecnológico

2007ES16UPO003 -

Programa Operativo FEDER de Economía basada en el Conocimiento

2007ES162PO002 –

Pais Vasco – PO FEDER Pais Vasco

2007ES162PO004 –

Programa Operativo FEDER de Madrid

2007ES162PO004 -

Programa Operativo FEDER de Madrid

2007ES162PO005 -

Programa Operativo FEDER de La Rioja

2007ES162PO006 -

Programa Operativo FEDER de Cataluña

2007FI162PO001 -
Itä-Suomen EAKR-toimenpideohjelma 2007-2013

2007ES162PO010 -
PO FEDER Comunitat Valenciana

2007FR162PO001 -
Programme opérationnel FEDER AQUITAINE

2007FR162PO011 -

Programme opérationnel FEDER HAUTE-NORMANDIE

2007FR162PO015 -

Programme opérationnel FEDER LORRAINE

2007FR162PO016 -

Programme opérationnel FEDER PAYS DE LA LOIRE

2007FR162PO017 -

Programme opérationnel FEDER NORD PAS-DE-CALAIS

2007FR162PO020 -

Programme opérationnel FEDER PROVENCE ALPES COTE D´AZUR

2007FR162PO021 -

Programme opérationnel FEDER MIDI-PYRENEES

2007HU161PO001-

Gazdaságfejlesztési Operatív Program (GOP) 2007-2013

2007IE162PO002 -

Southern and Eastern Operational Programme

2007IT162PO002 -

Por Emilia Romagna FESR 2007-2013

2007IT161PO006 -

PON Ricerca e Competitività

2007LT161PO002 -

2007-2013 m. Ekonomikos augimo veiksmų programa

2007LV161PO001 -

Operational Programme Entrepreneurship and Innovations“

2007PL161PO001 -

Program Operacyjny Innowacyjna Gospodarka, 2007-2013

2007PL161PO002 -
Program Operacyjny Infrastruktura i Środowisko

2007PL161PO003 -
Program Operacyjny Rozwój Polski Wschodniej 2007-2013

2007PL161PO010 -
Małopolski Regionalny Program Operacyjny na lata 2007-2013

2007PL161PO011 -
Regionalny Program Operacyjny Województwa Mazowieckiego

2007PL161PO013 -
Regionaln Program Operacyjny Województwa Podkarpackiego

2007PT161PO001 -
Programa Operacional Fatores de Competitividade (COMPETE) 2007-2013

2007PT161PO002 -
PO Regional do Norte 2007-2013

2007PT161PO003 -
PO Regional do Centro 2007-2013

2007RO161PO002 -
Sectoral Operational Programme Increase of Economic Competitiveness

2007SI161PO001 -
Operativni program krepitve regionalnih razvojnih potencialov za obdobje 2007 -
2013

2007SK16UPO001 -
Operačný program Výskum a vývoj

2007UK161PO002 -
West Wales and the Valleys ERDF Convergence programme

2007UK161PO003 -
Cornwall and the Isles of Scilly ERDF Convergence programme

2007UK162PO001 -
Lowlands and Uplands of Scotland ERDF Regional Competitiveness and Employment programme

2007UK162PO009 -
Yorkshire and the Humber Regional Competitiveness and Employment Programme

2007UK162PO008 -
North West England ERDF Regional Competitiveness and Employment Operational Programme

Source: Author

In-depth interviews

The following table presents the types of stakeholders we contacted and interviewed for each of the selected case studies.

Table 4 - Stakeholders interviewed as part of the case studies

	European/ national/ regional authorities involved in the oversight of ERDF funds	Managing authorities	Direct and final beneficiaries	Other relevant stakeholders (e.g. business associations, other experts)	Total interviewees
Poland	/	2	18	/	20
Czech Republic	4	5	21	5	35
Romania	2	6	23	/	31
Estonia	/	9	6	/	15
Germany	4	2	24	2	32
Italy	3	8	19	1	31
Portugal	1	11	24	/	36
Total	14	43	135	8	200

Seminar

In-depth expert workshop

The first of two seminar days took the form of an in-depth expert workshop, which virtually hosted 65 participants – ranging from representatives of Managing Authorities, R&D and evaluation experts, as well as European Commission officials. The workshop took place on May 4th, 2021 from 09.30 AM to 1.30 PM and was virtually hosted on Microsoft Teams.

The purpose of the workshop was to discuss, deepen and validate the emerging findings from the study. It also served as an opportunity to reflect upon possible recommendations and improvements of future RTD support under ERDF in the upcoming programming period.

To begin, the participants were invited to share what type of RTD projects or policy instruments were supported by the ERDF (2007 – 2013) in their region. In the form of a WordCloud, the responses can be observed in Figure 1. The text size of the response indicates the share of participants that provided that answer (i.e., the larger the size, the more participants provided that response).

Figure 3 – Responses of participants: What type of RTD projects or policy instruments were supported by ERDF (2007 – 2013) in your region?

Source: Author

The first presentation – Mapping the funding of RTD infrastructure and activities under the ERDF 2007-2013: Taking stock of funded projects and beneficiaries (CSIL) – provided an overview of the objects of inquiry of this ex-post evaluation, by comprehensively mapping and describing the funded RTD infrastructure and activities under the 2007 – 2013 ERDF (key findings of Task 1 of the project). As can be observed in Figure 2 below, the presentation highlighted the types of RTD projects funded, which ranged from individual and collaborative R&D projects, to infrastructure investments for research or education, to other types of projects, such as internationalisation of research and capacity building for research.

Figure 4 – Types of RTD projects funded

Source: Author

After taking stock of the funded projects and beneficiaries, the second presentation – RTD support from ERDF and policy mix in selected case studies (Technopolis) – outlined the case studies pursued, which covered 21 policy instruments across seven Member States. The presentation discussed the key insights from Task 3 of this evaluation by highlighting the performance of ERDF support for RTD activities based on the relevant evaluation criteria. A graphic overview of the findings can be observed in Figure 3 below.

Figure 5 – Overall performance of ERDF support for RTD activities for the 2007 – 2013 period

Source: Author

A plenary discussion began with an initial polling of participants’ views regarding significant achievements from RTD support under ERDF. The achievements under inquiry, as well as the results of the poll, can be observed in the figures below.

Figure 6 – Rating of achievements from RTD support under ERDF (2007-2013)

Source: Author

Upon the backdrop of the above polling, RTD contribution stories, which entailed detailed accounts of the functioning of RTD policy, were shared by participants of the expert workshop. This allowed for participants to explain in more depth how the RTD funding functioned in their settings, highlighting both the achievements gained and challenges encountered. The reflections were documented live using the online documentation tool, MIRO, an example of which can be observed in Figure 5 below.

Figure 7 – Documentation of plenary discussion: RTD Contribution Stories

Source: Author

The insights gained from this exercise, which reflect stakeholders’ perception about supported RTD efforts in three Member States, are displayed below.

Figure 8 - RTD Contribution Stories: Some insights from participants

A third presentation – Preparation for the group discussion: The four types of policy interventions and the respective theories of change in Czechia, Estonia, Germany, Italy, Poland, Portugal, and Romania (Prognos AG) – followed, which outlined the four overarching RTD policy interventions under inquiry and described the associated generalised theories of change. Moreover, the presentation highlighted certain important contextual factors (pre-conditions, supporting factors, risks) that either contributed to the effective functioning of the RTD policy intervention or limited its effectiveness. These factors were further deepened upon and discussed in the group discussions that followed.

Group discussions on important contextual factors for individual and collaborative R&D projects as well as for R&D and ICT-based infrastructure investments were held in virtual breakout rooms. Using MIRO, some key learnings were shared by the stakeholders and concurrently documented, which can be viewed below.

Figure 9 - Documentation of group discussion: important contextual factors

Source: Author

Some findings of the respective group discussions in regard to salient contextual factors for the implementation of RTD policy were as follows.

Group 1: Collaborative & Individual R&D Projects

·Quality of project selection procedure: in Portugal, National Science Foundation helped identify RTD actors most capable to pursue rigorous R&D. This ensured that highly skilled actors are involved, with better chances for internationalisation efforts and further acquisition of funding (and: higher impacts compared to other selection procedures).

·Important to create permanent collaboration platforms for exchange between industry and science actors, in order to foster sustainable relationships (e.g., competence centres or clusters)

·Capacity building: develop more of an awareness for industry needs. Not only about technology development, but capacities to foster knowledge transfer. Currently more a “science push” approach. Suggestions for better commercialisation pathways:

oOrganise calls around challenges expressed by companies and use micro-grants for testing

oInclude representative of a Technology Transfer Office in each research team from the start of the R&D process

Group 2: RTD & ICT Infrastructure Projects

·Road-mapping process before project call & selection helps ensure that projects matching strategic priorities get funds and reduces complexity in selection.

·Top-down vs bottom-up prioritisation: mixed approach observed in multiple countries – indication of priorities is followed by proposals.

oIn Estonia: small-scale investments bottom-up, big investments top-down → good compromise

·Mission-oriented infrastructure is often public-private: potential conflict with state aid rules. The more application oriented, the greater the conflict.

oUncertainty hampers business engagement in R&D.

·Qualified human resources to operate new ICT infrastructure? Yes, relevant skills are available, but they need a sufficient salary.

·Without proper IT experts running ICT systems, infrastructure yields limited results.

After a review of the key findings of the morning, the in-depth workshop finished with closing words by the European Commission, referring to the succeeding and complementary validation webinar that was to take place on May 7th, 2021.

Validation webinar

The second event in the seminar series took the form of a validation webinar, which was intended for a broader audience of interested RTD policy stakeholders from Member States, pan-European institutions, and European Commission officials. The workshop took place on May 7th, 2021 from 09.30 AM to 12.30 PM and was virtually hosted on Microsoft Teams.

In total, more than 200 participants took part in the validation webinar. The purpose of the webinar was to present, reflect and validate the emerging findings of the ex-post evaluation. It was also used to reflect upon possible recommendations and improvements of future RTD support under ERDF. While the in-depth expert workshop (see above) provided an opportunity to assess in detail the operation of ERDF supported projects during the 2007 – 2013 funding period, the validation webinar sought to encourage more high-level strategic discussions on RTD policy.

The participants were welcomed to the validation webinar by Nicola De Michelis (former Director DG REGIO), who set the scene and objective of the morning. The participants were then invited to share their thoughts on what strategic objectives future RTD policy under ERDF should address. In the form of a WordCloud, the responses can be observed in Figure 9. The text size of the response indicates the share of participants that provided that answer (i.e., the larger the size, the more participants provided that response).

Figure 10 – What strategic objectives should future RTD policy under ERDF address?

Source: Author

Upon this basis, the first presentation – Big Data - Mapping 2007-2013 ERDF support to RTD infrastructure and activities: Beneficiary and project dashboard (CSIL and DG REGIO) – offered a detailed mapping of the RTD infrastructure and activities under inquiry in the ex-post evaluation. This was complemented by a presentation of the ERDF R&I Database on the ESIF Open Data Portal, which is maintained by DG Regio and gives access to data on financing and achievements of ESIF funds.

The second presentation – Success Stories and Challenges in ERDF RTD support – experiences from seven EU MS (Technopolis) – outlined in some more depth the empirical basis and methodological approach for the ex-post evaluation and highlighted key findings from the seven Member State case studies pursued, according to the pertinent evaluation criteria (relevance; coherence; effectiveness; efficiency; EU-added value). The presentation engendered some lively discussion via the chat function. For instance, one participant pointed to the fact that for some Member States this was the first programming period and considered to what extent this may have had an influence on the choice of R&D investments. Another set of comments underscored the concept of coherence and the relationship between ERDF support for RTD and other EU Framework Programmes, reflecting on the objectives that each funding source holds. Another participant reflected on the challenge of assessing effective interventions since the evaluation criteria remains at national levels in many cases, suggesting that an inter-state perspective could better inform the evaluation process. Further comments and questions on how to improve the measurement of ERDF effectiveness followed, as well as reflections on the challenges of assessing long-term impacts in a contribution analysis approach. Regarding the latter point, it was underscored that especially infrastructure investments are to be regarded as long-term investments. As such, a time-lag is to be expected and improved RTD performance should be evaluated with this in mind.

The third presentation – The RTD cookbook: Success factors and important considerations when designing RTD instruments (Prognos AG) – presented the RTD Cookbook, which serves as a decision-aid tool for policy practitioners at the start of the upcoming programming period. Upon a comprehensive analysis of contextual factors (pre-conditions, supporting factors, risks), the RTD Cookbook identifies key ingredients (depicted in Figure 10) that can help avoid common pitfalls in the design and implementation of RTD policy.

Figure 11 - RTD Cookbook: key ingredients for effective RTD policies

Source: Author

A comprehensive discussion of these salient factors was followed by live interaction with the webinar participants, who were asked to provide their opinion on the significance of the identified contextual factors. The result of the polling is displayed in Figure 11 below.

Figure 12 – Significance rating: contextual factors for effective RTD policies

Source: Author

It is important to once again underscore that the webinar was particularly enriched by the active participation of the participants via the chat function, continuously offering insightful and important reflection questions or comments. Serving the purpose of a validation webinar, these helpful insights have been captured and will be considered and reflected upon when drafting the final report of this ex-post evaluation.

The validation webinar concluded with a policy roundtable - “What works, why, and where in RTD policy?” – and a final discussion of the results. Facilitated by CSIL, the three academic experts - Prof. Slavo Radosevic, Prof. Dirk Czarnitzki and Prof. Elliot Stern – focused on the following key topics:

·Main achievements of ERDF

·Substitution role of ERDF during the economic crisis

·Excellence vs. territorial cohesion

·Coherence with other EU policies, in particular State Aid

·Coherence with national policies

·Administrative capacity and implementation failures

·Methodological considerations about impact evaluation of ERDF support to RTD investments

Some key insights from this discussion related to the tension between “local relevance” and “scientific excellence” of RTD support under ERDF, limited institutional capacities in many regions, and ongoing uncertainties of stakeholders around “State Aid”. On the first point, it was underscored the crucial policy challenge to ensure the balance between R&D excellence and local relevance. RTD policy under ERDF should prioritise locally relevant but internationally excellent R&D. As this often may not be possible, the priority seems to be given to the second-best area, to islands of excellence but locally not necessarily relevant research. Considering the increasing interconnectivity through digital technologies, however, may also allow us to think about more distributed RTD infrastructures in the future. With regards to institutional capacities, it was stressed that both administrative capacities of public authorities and also the institutional capacities of the various stakeholders in the innovation system (universities, enterprises, regional development authorities, cluster organisations) need to be considered and that sufficient institutional capacities are a major pre-condition for effective policies – especially in complex policy domains such as RTD. Finally, regarding ongoing uncertainties around State Aid it was put forward that – especially in the 2007-2013 period – it was often a very significant factor in generating incoherence due to the missing link between State Aid rules and the selection of RTD instruments implemented in the framework of ERDF. In particular due to the perceived complexities, policymakers have deterred in advance from considering the issue of science-industry collaboration and the role of industry commons or joint public-private RDI infrastructure. Funding rules limit private sector organisations' involvement in newly developed infrastructure and prevent commercialisation of knowledge stemming from ERDF-supported research projects that defy the purpose – ultimate application of knowledge in innovation processes. Thus, effectively the State Aid rules hindered the creation of 'industry commons' or RTD activities that cross public-private knowledge domains.

Regarding methodological considerations it was emphasised by the experts that the approach to this ex-post evaluation is quite novel in the field of RTD support, mainly due to its scale, its cross-case analysis, and the strong emphasis on the role of contextual factors. At the same time, it was highlighted that further methodological advances are needed, incorporating agent-based modelling approaches, or combining a system-dynamics approach with theory-based evaluation techniques. This is particularly important in the field of RTD support, as the unit of analysis typically changes throughout the evaluation exercise, starting from individual operations and ending in innovation ecosystems towards the end of the evaluation. A final thought put forward by the experts was on the future use of theory-based evaluations and the learnings from such evaluations. Thus, it was suggested to use the concept of theories of change more frequently in the design phase of RTD policy instruments (i.e., in an ex-ante fashion) and to build on the learnings from such types of evaluations to strengthen our “foresight” capacities. The latter could be a cornerstone in our thinking about increasing resilience and prepare us on upcoming crisis.

The webinar was closed by the European Commission, who reflected on the insights gained from the webinar and offered some concluding remarks.

Annex V. List of references

Aghion, P. (2006). "A primer on innovation and growth," Policy Briefs 233, Bruegel.

Arnold, E. (2004). “Evaluating Research and Innovation Policy: A Systems World Needs Systems Evaluations”, Research Evaluation 13 (1): 3–17. https://doi.org/10.3152/147154404781776509 .

Arrow, K.J. (1962). “Economic Welfare and the Allocation of Resources for Innovation”, in Nelson R.R. (ed.), “The Rate and Direction of Inventive Activity”, Princeton, Princeton University Press, pp. 609-662.

Barnes, T.A., Pashby I.R. and Gibbons A.M. (2006). “Managing Collaborative R&D Projects Development of a Practical Management Tool”, International Journal of Project Management 24 (5): 395–404. https://doi.org/10.1016/j.ijproman.2006.03.003 .

Bastianin, A., Castelnovo, P., Florio, M. et al. (2021), “Big science and innovation: gestation lag from procurement to patents for CERN suppliers”. They Journal of Technology Transfer (2021). https://doi.org/10.1007/s10961-021-09854-5

Benneworth, P. & Dahl Fitjar, R. (2019). “Contextualizing the role of universities to regional development: introduction to the special issue”, Regional Studies, Regional Science, 6:1, 331-338, DOI: 10.1080/21681376.2019.1601593.

Bleda, M., & Del Río P. (2013). “The Market Failure and the Systemic Failure Rationales in Technological Innovation Systems.” Research Policy 42 (5): 1039–52. https://doi.org/10.1016/j.respol.2013.02.008.

Boschma, R. & Gianelle, C. (2013). “Regional branching and smart specialization policy”, JRC technical reports, S3 Policy Brief Series, issue 06/2104.

Cedefop (2015). “Skills, qualifications and jobs in the EU: the making of a perfect match? Evidence from Cedefop’s European skills and jobs survey”, Luxembourg: Publications Office. Cedefop reference series; No 103. http://dx.doi.org/10.2801/606129.

Cooke, P., Gómez Uranga M. and Etxebarria G. (1997). “Regional Innovation Systems: Institutional and Organisational Dimensions”, Research Policy 26(4-5):475-491.

Crescenzi, R., de Blasio, M., Giua, M., (2018). “Cohesion Policy Incentives for Collaborative Industrial Research. The Evaluation of a Smart Specialisation Forerunner Programme”, Spatial Economics Research Centre, Discussion paper 231.

Dahlin, K. & Behrens, D. (2005). “When Is an Invention Really Radical? Defining and Measuring Technological Radicalness”, Research Policy 34(5):717-737.

Daigneau, W. A., Valenti, M.S., Ricciarini, S. Bender, S.O., Alleyne, N., Di Grappa, M., Duart, J.M., Lupiáñez, F., Ehrenzweig Sanchez, M.A. (2005). “Planning, Designing and Managing Higher Education Institutions”, PEB Exchange, Programme on Educational Building, https://doi.org/10.1787/541370867752 .

Davey, T., Galan Muros, V. Meerman, A., Orazbayeva, B. and Baaken, T. for European Commission, Directorate-General for Education, Youth, Sport and Culture (2018). “The State of University-Business Cooperation in Europe”, Final Report.

De Bruijn, P. & Lagendijk, A. (2005). « Regional Innovation Systems in the Lisbon strategy”, European Planning Studies, 13, (8), 1153-1172

Di Cataldo, M., Monastiriotis, V., and Rodríguez-Pose, A. (2020). “How ‘Smart’ Are Smart Specialization Strategies?”, JCMS: Journal of Common Market Studies, https://doi.org/10.1111/jcms.13156 .

EACE – Evaluation Advisory Central Europe (2018). “Ex post evaluation of the programming period 2007-2013 in the R&D sector”, Final report (in Czech), National Coordination Authority, Ministry of Regional Development.

Ebbekink, M. and Lagendijk, A. (2013). “What’s next in researching cluster policy: place-based governance for effective cluster policy”, European Planning Studies, Vol. 21, No. 5, pp.735–753.

Edquist, C. (1997). “Systems of innovation: Technologies, Institutions and Organizations”.

EGO (2013). “Badanie podsumowujące realizację Priorytetu XIII Infrastruktura szkolnictwa wyższego Programu Operacyjnego Infrastruktura i Środowisko (pakiet 2)".

European Strategy Forum on Research Infrastructures (ESFRI) (2017). “Long-Term Sustainability Working Group. Long-Term Sustainability of Research Infrastructures”, Milano: Università degli studi di Milano, Dipartimento di fisica.

European Commission (2006). “Community Framework for State Aid for Research and Development and Innovation”.

European Commission (2008). “European Competitiveness Report”, DOI 10.2769 / 65417.

European Commission (2009). “European Innovation Scoreboard 2009: Comparative analysis of Innovation Performance”.

European Commission, Directorate General for Regional Policy (2013). EVALSED: The resource for the evaluation of Socio-Economic Development.

European Commission (2016a). "Employment and Social Developments in Europe 2015", http://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=7859&furtherPubs=yes .

European Commission, Directorate General for Regional Policy (2016b). “Ex post evaluation of Cohesion Policy programmes 2007-2013, focusing on the European Regional Development Fund (ERDF) and the Cohesion Fund (CF)”, Final Report Work Package 2.

European Commission (2017a). “The Economic Rationale for Public R&I Funding and Its Impacts”.

European Commission, Policy Support Facility (2017b). “Peer Review of Poland’s Higher Education and Science System”, https://rio.jrc.ec.europa.eu/sites/default/files/report/PSF-Peer_review_Poland__FINAL%20REPORT.pdf

European Commission (2020). “Science, research and innovation performance of the EU 2020 - A fair, green and digital Europe”.

European Commission, DG Regional and Urban Policy (2021). “Study on prioritisation in Smart Specialisation Strategies in the EU”.

European Commission, DG Regional and Urban Policy (2021). “Evaluation of investments in Research and Technological Development (RTD) infrastructures and activities supported by the European Regional Development Funds (ERDF) in the period 2007-2013”.

European Strategy Forum on Research Infrastructures (ESFRI) (2017). “Long-Term Sustainibility Working Group. Long-Term Sustainability of Research Infrastructures”, Milano: Università degli studi di Milano, Dipartimento di fisica.

Finardi, U. (2011). Time relations between scientific production and patenting of knowledge: The case of nanotechnologies. Scientometrics, 89(1), 37–50.

Florio, M. (2019). “Investing in Science: Social Cost-Benefit Analysis of Research Infrastructures”, Cambridge, MA: The MIT Press.

Florio, M. and Sirtori, E. (2016). “Social Benefits and Costs of Large Scale Research Infrastructures”, Technological Forecasting and Social Change, 112: pp. 65-78.

Foray, D. (2014). “Smart Specialisation: Challenges and Opportunities for Regional Innovation Policies: Opportunities and Challenges for Regional Innovation Policy”, London: Routledge.

Foray, D. and Van Ark, B. (2007). “Smart specialisation in a truly integrated research area is the key to attracting more R&D to Europe”, Knowledge Economists Policy Brief n° 1, October 2007.

Foray, D., David, P.A. and Hall, B. (2009). “Smart specialisation – The concept”, Knowledge Economists Policy Brief n° 9, June 2009.

Foray, D., Morgan, K. and Radosevic, S. (2018). “The Role of Smart Specialisation in the EU Research and Innovation Policy Landscape”, Brussels: European Commission.

Freeman, C. (1987). “Technology Policy and Economic Performance: Lesson from Japan, Pinter Publisher.

Freeman, C. (1988). “Japan: A New National System of Innovation?” in G. Dosi, C. Freeman, R. Nelson, G. Silverberg, L. Soete (eds.), Technical Change and Economic Theory. London and New York: Pinter Publisher (pp. 330–348).

Georghiou, L. (1993). “National Systems of Innovation. Towards a Theory of Innovation and Interactive Learning Edited by Bengt-Ake Lundvall (Pinter Publishers, London. 1992) Pp. Xiii + 342, £45, ISBN 1-85567-063-1.” Prometheus 11 (2): 291–291. https://doi.org/10.1080/08109029308629360.

Griliches, Z. (1979). "Issues in Assessing the Contribution of Research and Development to Productivity Growth," Bell Journal of Economics, The RAND Corporation, vol. 10(1), pages 92-116, Spring.

Guy, K. (2007). “Policy Mix Peer Reviews: Synthesis Report,” 79.

Hallonsten, O. (2014). “How scientists may ‘benefit from the mess’: A resource dependence perspective on individual organizing in contemporary science”, Social Science Information, 53(3):341-362. doi:10.1177/0539018414524037.

Hospers, G-J. and Beugelskijk, S. (2002). “Regional cluster policies: learning by comparing?”, KYKLOS, Vol. 55, No. 3, pp.381–402.

Jacoby, N. (2010). “An Evolutionary Theory of Economic Change, Nelson & Winter (1982).” In De Taylor à Aujourd’hui (2010), l’Encyclopédie Multimédia En Sciences de l’organisation, edited by E. Friedberg, CD-Rom. R&O multimédia. https://hal.archives-ouvertes.fr/hal-00605363.

JIIP (2017). “Synergies between Framework Programmes for Research and Innovation and European Structural and Investment Funds”.

Joint Research Centre (2018). “RIO Country Report 2017: Romania”, https://rio.jrc.ec.europa.eu/sites/default/files/riowatch_country_report/RIO_CR_RO_2017_PUBSY_IDF.pdf

Keraminiyage, K., Amaratunga, D., and Haigh, R. (2009). “Achieving Success in Collaborative Research: The Role of Virtual Research Environments”, Journal of Information Technology in Construction, 14, 59-69.

Landabaso, M., Oughton, C., and Morgan, K. (2002). ‘The innovation paradox’, Journal of Technology Transfer, 27, pp. 97-110.

Laudel, G. (2002). “What Do We Measure by Co-Authorships?", Research Evaluation, Oxford University Press, vol. 11(1), pages 3-15, April.

Lundvall, B-Å. (1992). “National Systems of Innovation. Towards a Theory of Innovation and Interactive Learning”. Pinter Publisher.

Malerba, F. (1996). “Public Policy and Industrial Dynamics: An Evolutionary Perspective”, 30.

Martin, B. R. (1996). “The Relationship between Publicly Funded Basic Research and Economic Performance: A SPRU Review”. https://www.academia.edu/17547694/The_relationship_between_publicly_funded_basic_research_and_economic_performance_a_SPRU_review.

Mayne, J. (2011). “Contribution analysis: Addressing cause and effect”, Evaluating the Complex, pp. 53–96.

Nelson, R. (1993). “National Innovation Systems: a Comparative Analysis”, Oxford University Press.

OECD (1998). “Redefining Tertiary Education”, OECD Publishing, Paris, https://doi.org/10.1787/9789264163102-en.

OECD (2014). Science, Technology and Industry Outlook 2014, OECD Publishing, Paris.

O’Kane, M. (2008). “Review of the National Innovation System”, Canberra, A.C.T.: Dept. of Innovation, Industry, Science and Research.

Perspektywy (2020). “Foreign students in Poland 2020 conference”, Educational Foundation.

Porter, M. E. (1998). "Clusters and the New Economics of Competition", Harvard Business Review 76, no. 6: 77–90.

Radosevic, S. & Lepori B. (2009). “Public Research Funding Systems in Central and Eastern Europe: Between Excellence and Relevance: Introduction to Special Section”, Science and Public Policy 36 (9), 659-666.

Re-Source (2014). “Ewaluacja procesu komercjalizacji wyników prac B+R oraz współpracy jednostek naukowych z przedsiębiorcami w ramach I osi priorytetowej Programu Operacyjnego Innowacyjna Gospodarka (Poddziałanie 1.1.1 oraz Poddziałanie 1.3.1)”, Raport końcowy. http://www.ewaluacja.gov.pl/media/24739/2_092.pdf

Robin, S. & Schubert, T. (2013). “Cooperation with Public Research Institutions and Success in Innovation: Evidence from France and Germany”, Research Policy 42 (1): 149–66. https://doi.org/10.1016/j.respol.2012.06.002 .

Rodríguez‐Pose, A. (2020). "Institutions and the fortunes of territories," Regional Science Policy & Practice, Wiley Blackwell, vol. 12(3), pages 371-386, June.

Romer, P.M. (1990). “Endogenous Technological Change”, Journal of Political Economy, 98 (5), Part 2: 71-102.Salter, A. J., & Martin B. R. (2001). “The Economic Benefits of Publicly Funded Basic Research: A Critical Review”, Research Policy 30 (3): 509–32. https://doi.org/10.1016/S0048-7333(00)00091-3 .

Simmonds, P. (2016). “ISIS Lifetime Impact Study”.

Smith, K. (2000). “Innovation as a Systemic Phenomenon: Rethinking the Role of Policy”, Enterprise and Innovation Management Studies 1 (1): 73–102. https://doi.org/10.1080/146324400363536 .

Technopolis Group & MIOIR (2012). “Evaluation of Innovation Activities. Guidance on methods and practices”, Study funded by the European Commission, Directorate for Regional Policy.

Technopolis Group, Praxis and Institute of Baltic Studies. (2011). “Euroopa Liidu tõukefondide perioodi 2007‐2013 teadus‐ ja arendustegevuse ning kõrghariduse meetmete rakendamise vahehindamine”, Tallinn: Government of Estonia.

Thanos, C. (2010). “Global Research Data Infrastructures: The GRDI2020 Vision”.

World Bank (2014). “SABER — Systems Approach for Better Education Results Education Management Information Systems”.

(1) https://ec.europa.eu/regional_policy/en/information/publications/evaluations/2021/evaluation-of-investments-in-research-and-technological-development-rtd-infrastructures-and-activities-supported-by-the-european-regional-development-funds-erdf-in-the-period-2007-2013 .

(2) The expression ‘Research and Technological Development (RTD) is a reference to the 2007-2013 categories of intervention, reflected in the title of the evaluation and more generally to EC policy lines for the funding period under analysis. By contrast, the expression ‘Research and Development’ (R&D) derives from the EC 2020 strategy, Eurostat indicators definition, and the most common international statistical and policy reference for the sector. Therefore, despite partial overlap, the two expressions are used throughout the SWD with two distinct meanings.

(3) Categories of ERDF expenditure 01 and 02 were deliberately analysed later than the main ex-post evaluation 2007-2013, as further explained in Annex II.2. More specifically, they include: support for RTD activities in research centres (e.g., scientific R&D activities; collaborative research activities; support for the internationalisation of research activities; development of researchers and other personnel involved in R&D activities; support for technology-transfer activities; and the valorisation of research results), support for RTD infrastructures, and support for competence centres in a specific technology.

(4) Nonetheless, evidence from the evaluation revealed that, in a limited number of cases, support for RTD activities in SMEs was mistakenly encoded under codes 01 or 02.

(5) Source: Eurostat.

(6) https://eur-lex.europa.eu/EN/legal-content/summary/cohesion-policy-in-support-of-growth-and-jobs-community-strategic-guidelines-2007-13.html .

(7) http://ec.europa.eu/invest-in-research/pdf/download_en/aho_report.pdf .

(8) Decision No 1982/2006/EC of the European Parliament and the Council of 18 December 2006 concerning the Seventh Framework Programme of the European Community for research, technological development and demonstration activities (2007-2013) (OJ L 412, 30.12.2006, p. 1).

(9) Article 2 (6) of Regulation (EU) No 1291/2013 of 11 December 2013 of the European Parliament and of the Council of 11 December 2013 establishing Horizon 2020 - the Framework Programme for Research and Innovation (2014-2020) and repealing Decision No 1982/2006/EC (OJ L 347, 20.12.2013, p. 104-173).

(10) Case Study Report: Czechia - Evaluation of investments in Research and Technological Development (RTD) infrastructures and activities supported by the European Regional Development Funds (ERDF) in the period 2007-2013.

(11) Individual and collaborative R&D projects could also include a capacity-building component, although this was not the main objective of the project.

(12) Spain had an untypical monitoring system that lacked any project-level identification code. This prevented the aggregation of all expenditure data at project level. In the report, approximate data and information on the Spanish programmes are provided. However, to preserve the accuracy and reliability of the rest of the data, they are not considered when producing aggregate project-level statistics.

(13) The comparison between the types of intervention attributed to individual projects, and the official category of expenditure under which the same project was coded by the Managing Authorities, reveals that a certain share of projects had been miscoded, and that it was indeed necessary to reclassify the projects under a new taxonomy to have a more precise distinction between investments for RTD activities and RTD infrastructures. More specifically, around 9% of projects and 19% of ERDF contributions coded as 01 referred, in fact, to infrastructure investments. When one considers the projects coded as 02, it is found that 83% of these projects and 94% of the respective ERDF contribution had been properly coded, with the remaining share being research projects, other RTD activities (e.g. capacity-building activities delivered to research centres) or other non-RTD-related investments (e.g., infrastructure investment in incubator centres to support entrepreneurship and innovation, but not RTD activities).

(14) Excluding the seven Spanish OPs, the total ERDF contribution to lead beneficiaries amounted to EUR 11.7 billion.

(15) https://eur-lex.europa.eu/legal-content/en/ALL/?uri=CELEX%3A32006R1083 .

(16) See for instance WP0 of the 2007-2013 Ex Post Evaluation of ERDF and CF https://ec.europa.eu/regional_policy/policy/evaluations/ec/2007-2013_en and WP1 of the 2014-2020 Ex Post Evaluation of ERDF and CF https://ec.europa.eu/regional_policy/policy/evaluations/ec/2014-2020_en.

(17) In order to test the validity of this result, the analysis tested a set of interaction terms (first) between the dummy variable ‘EU13’ and the ERDF policy variable, (second) between the dummy variable ‘Convergence region’ and the ERDF policy variable, and (third) between the Type of Territory dummy variables and the ERDF policy variable. Interaction terms were never statistically significant.

(18) See the Polish case study.

(19) The t-test suggests that the mean difference between the EU13 and EU15 was statistically significant at 1% level. In our sample, there were four NUTS2 regions with an extremely high rate of growth in the number of scientific publications. They were: RO12 – Centru (growth rate=1,005%); SK03 - Stredné Slovensko (883%); RO31 - Sud – Muntenia (723%); and CZ08 – Moravskoslezsko (549%).

(20) To test whether the ERDF policy variable was more effective in lagging regions, we tested a set of interaction terms between the dummy variable ‘EU13’ and the ERDF policy variable, between the dummy variable ‘Convergence region’ and the ERDF policy variable, and between the Type of Territory dummy variables and the ERDF policy variable. Interaction terms were never statistically significant.

(21) See the Czech case study.

(22) The evaluation used i) the growth rate in the number of patents between 2008 and 2016; (ii) the growth rate in the number of public - private co-publications in the period 2008-2015.

(23) See for example the Italian, Czech and Romanian case studies.

(24) See the Polish case study.

(25) The analysis considered the expenditure on infrastructure for research and in collaborative R&D projects.

(26) Other context variables such as the type of R&D, type of territory or membership of EU13/EU15 played no role in shaping the relationship between the ERDF support and the growth rate in the number of EU trademark applications.

(27) See the Czechia case study.

(28) Compared with other EU MS, Germany showed a relatively fast economic recovery and was nicknamed an ‘engine of growth’ in the years after the financial and economic crisis.

(29) Poland’s economic growth was not as significantly affected by the slowdown as the other EU countries to which it was linked, as the country benefited from an influx of foreign investments, the internationalisation of businesses, changes in skills and human-resource availability, the growing importance of innovations in the economy, and reforms of the science sector.

(30) Pellens, Maikel, et al. (2018). ‘Public investment in R&D in reaction to economic crises - a longitudinal study for OECD countries.’

(31) See in particular the Italy, Czechia and Estonia case studies.

(32) https://ec.europa.eu/regional_policy/en/information/publications/studies/2021/study-on-prioritisation-in-smart-specialisation-strategies-in-the-eu .

(33) It is worth noting that Work Package 13, ‘Geography of Expenditure’, of the Ex-post evaluation of Cohesion Policy programmes 2007-2013, in the analysis of the ERDF contribution to RTD, actually referred not only to codes 01 and 02, but also to codes 03, 04, 07 and 09. Similarly, the categorisation of ERDF expenditure for the 2014-2020 programming period aggregated these six codes under the same umbrella category of ‘Research and development and innovation’, because of the strong linkages between them.

(34) See the Czechia and Estonia case studies.

(35) State aid rules are per definitions applicable to undertakings to ensure a level-playing field in the market and in order not to crowd-out private investment. On the other hand, public support for infrastructure used for scientific/educational purposes can and in many cases does fall outside the control of State aid rules.

(36) 1) Strengthening research, technological development, and innovation; 2) Enhancing access to, and use and quality of information and communication technologies (ICT); 3) Enhancing the competitiveness of small and medium-sized enterprises (SMEs); 4) Supporting the shift towards a low-carbon economy in all sectors.

(37) Council Conclusions 2016 on 'A more R&I friendly, smart and simple Cohesion Policy and the European Structural and Investment ESI Funds more generally' (24 June 2016), and European Parliament resolution of 13 September 2016 on Cohesion Policy and R&I Strategies for Smart Specialisation (2015/2278(INI)).

(38) https://ec.europa.eu/regional_policy/en/information/publications/studies/2021/study-on-prioritisation-in-smart-specialisation-strategies-in-the-eu .

(39) European Regional Development Fund – Funding Priorities. European Commission: https://ec.europa.eu/regional_policy/en/funding/erdf/#2 .

(40) Key references also include: An SME strategy for a sustainable and digital Europe, European Skills Agenda for sustainable competitiveness, social fairness and resilience, the Renewed EU agenda for higher education, Shaping Europe’s digital future and the Communication on a new ERA for research and innovation, and the Strategy on Connectivity for a European Gigabit Society.

(41) https://op.europa.eu/en/publication-detail/-/publication/3026007b-8be2-11ed-999b-01aa75ed71a1 .

(42) See: https://ec.europa.eu/regional_policy/en/information/publications/evaluations/2021/evaluation-of-investments-in-research-and-technological-development-rtd-infrastructures-and-activities-supported-by-the-european-regional-development-funds-erdf-in-the-period-2007-2013

(43) https://ec.europa.eu/regional_policy/en/information/publications/evaluations/2021/evaluation-of-investments-in-research-and-technological-development-rtd-infrastructures-and-activities-supported-by-the-european-regional-development-funds-erdf-in-the-period-2007-2013

(44) More specifically, they include: support for RTD activities in research centres (e.g., scientific R&D activities; collaborative research activities; support for the internationalisation of research activities; development of researchers and other personnel involved in R&D activities; support for technology-transfer activities; and the valorisation of research results), support for RTD infrastructures, and support for competence centres in a specific technology.

(45) In particular, lacking a control group of non-treated regions, a pure counterfactual method, such as propensity score matching, regression discontinuity design or difference in difference, could not be carried out.

(46) The Commission, in work package 2 "Small and medium sized enterprises, innovation, ICT", assessed the effects of ERDF support on research and innovation in micro, small and medium-sized enterprises (SMEs) and outlined the mechanisms and context features underlying these effects. However it was limited to SME support and did not include specific investments on Research and Technological Development (RTD) infrastructures and activities, as these investments, due to their scope and nature, take longer to implement (many of them were not finalised by the time when the ex-post evaluations of 2007-2013 period were carried out) and yield results even later.

(47) The concept of core indicators was first used in the EC Working Document No. 2, providing guidelines on the setting up of indicator systems for monitoring and evaluation. European Commission (2006), Working Document No. 2: Indicative guidelines on evaluation methods: monitoring and evaluation indicators, DG Regio, Brussels.

(48) Depending on the way how the regional and national expenditure monitoring systems are organised, project-level data were not available for all the OPs. In some cases, they had to be derived from processing the list of beneficiaries or data on individual tranches of payments. Significant efforts were devoted to this activity, and it was eventually possible to build a consistent database of projects for 46 out of 53 OPs. In the specific case of Spain, the peculiarities of the monitoring systems and particularly the lack of any project-level identification code, prevented from aggregating all expenditure data at the level of projects. In the report, approximate data and information on the Spanish programmes are provided. Still, they are not considered when producing aggregate project-level statistics to preserve the accuracy and reliability of the rest of the data.

(49) In the analysis, the variable of interest – i.e., the ERDF instrument- is labelled as “ERDF policy variable”.

(50) https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/1983-R&D-infrastructure-&-activities-investment-2007-13-evaluation_en

Choose the experimental features you want to try