EUROPEAN COMMISSION

Brussels, 3.10.2024

SWD(2024) 470 final

COMMISSION STAFF WORKING DOCUMENT

EVALUATION

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND TO THE COUNCIL

Final Evaluation of Eurostars-2

{COM(2024) 436 final}

1. INTRODUCTION

The Eurostars-2 programme was a public-public partnership funded under the Horizon 2020 Framework Programme for Research and Innovation (R&I) covering the 2014-2020 period. Eurostars-2 aimed at supporting small and medium-sized enterprises (“SMEs”) performing research and development (R&D) activities through international cooperation. It was the second iteration of a programme that started in 2008.

1.1 Purpose and scope of the evaluation

Eurostars-2 is jointly undertaken by 33 Eurostars-2 Participating States and Partner countries and the European Union (EU) based on Article 185 of the Treaty on the Functioning of the European Union (TFEU) (ex-Art. 169 TEC). The participation of the EU was formally acted through the Decision No 553/2014/EU of the European Parliament and of the Council of 15 May 2014 on the participation of the Union in a Research and Development Programme jointly undertaken by several Member States aimed at supporting R&D performing SMEs 1 ("Eurostars-2 Decision"). The decision entered into force on 27 June 2014.

The European Union supports financially the Eurostars-2 Programme, with maximum EUR 287 million for the period 2014-2020, coming from the Horizon 2020 budget allocated to "Innovation in SMEs" (Industrial Leadership pillar).

According to Article 15 of the Eurostars-2 Decision, the European Commission had to conduct a final evaluation of Eurostars-2 by 31st of December 2022.

This Staff Working Document accompanies the Commission Report on the final evaluation of the Eurostars-2 programme, covering the period between the inception of Eurostars-2 in 2014 until March 2022. 2 It builds on the interim evaluation of Eurostars-2 performed in 2017 3 , a call for evidence launched in 2022, and available monitoring data. The performance of Eurostars-2 was also covered as part of the external ex-post evaluation of Horizon 2020 4 .

The interim evaluation of Eurostars-2 performed in 2017 informed the impact assessment of its successor initiative “European Partnership on Innovative SMEs” supported under Horizon Europe. It highlighted that the major benefits of Eurostars rest in the niche features of the programme, such as its bottom up approach, its strong trans-national focus, the division of work between the central structure and the decentralised structures of national funding bodies. It also supports beneficiaries to introduce new products within two years of projects’ completion and allows SMEs to obtain funds without any prior experience in transnational R&I collaboration.

On the other hand, the interim evaluation pointed out that, due to the decentralised structure, times to grant have shown a high level of heterogeneity and the lack of synchronisation of procedures often hampered the smooth implementation of the programme. The low number of active participating States and the uneven concentration 5 of beneficiaries in them limited the impact of the programme. A certain level of uncertainty had manifested in terms of getting funding for selected projects when the contribution of participating States had been exhausted by other projects.

1.2 Methodology

The final evaluation of Eurostars-2 builds on the findings of a supporting evaluation study (ISBN 978-92-68-01803-3), conducted by external contractors, which is part of the Research and Innovation Framework Programme Impact Monitoring, Analysis and Evaluation strategy for the period 2019-2024 covering the overall ex-post evaluation of Horizon 2020 and the interim evaluation of Horizon Europe, including partnerships. More particular the Eurostars-2 final evaluation is covered under the Evaluation study of the European Framework Programmes for Research and Innovation for an Innovative Europe. This study includes two case studies, using data collected through surveys and approximately 30 interviews with programme participants and a wider variety of stakeholders (including beneficiaries, national funding bodies, the Eureka Secretariat and Commission services).

This is complemented with statistical information about the Eurostars-2 programme provided by the Eureka Secretariat. The call for evidence published in September 2022 gathered replies from 62 entities from nine different countries and largely confirmed the conclusions that were drawn from the interviews conducted with stakeholders.

Monitoring data provided (by 25 August 2022) include:

- List of contact details for some National Funding Bodies

- Data on the number of applications for each cut-off

- Data on the countries of origin of the applicants

- Data on the number of funded projects for each cut-off
- Data on the total projects budget for each cut-off

- Data on the consortium composition for the 2014-2020 funding period
- List of contact details for 10 Eurostars-2 beneficiaries
- List of all approved Eurostars-2 projects
- Data on the committed budget for each Participating State and Partner country for each cut-off & the committed EU contribution

- List of Eurostars-2 projects with their evaluation status
- Data on the time-to-contract per country for cut-off 1-15
- Excel list with answers from the final reports of all projects from cut-off 1-5
- Annual Reports of the Eurostars-2 programme from 2014-2021 1
- Excel list with answers from the Market Impact Reports from 2019
- Excel list with answers from the final reports from 2020

Since around half of the Eurostars-2 projects were still running at the time of the evaluation (August 2022), insights from final reports are mainly projected from cut-offs 1-5, that are completed. Another important source for the information collected on the Eurostars-2 programme has been the stakeholder interviews. The interviews followed a semi-structured approach and were conducted via videoconferences. The interviews with representatives of the European Commission, the Eureka Secretariat and the National Funding Bodies allowed the project team to learn more about the administrative functioning of Eurostars-2. By speaking with programme beneficiaries such as SMEs, experiences on the actual outputs and results of the programme were collected.

More information on the methodology and process to carry out this evaluation is available in Annex II.

2. WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION?

2.1 Description of the intervention and its objectives

The rationale behind Eurostars-2 is that SMEs need access to market, finance, skills and knowledge in order to thrive and compete globally. They often lack in-house capabilities and capacity to achieve the innovative breakthroughs that are needed for their scale up and integration into global value chains. Collaboration between enterprises and with public research-performing organisations are key for faster knowledge diffusion and exploitation. Whilst many national programmes and instruments exist to facilitate participation of SMEs in R&I projects, most do not explicitly support or focus on international R&I collaboration.

Under the Eurostars-2 programme, 33 participating countries, four additional non-EU partner countries and the EU, focused on the development of synergies of their R&I programmes and on improved cooperation between national and regional R&I programmes for the benefit of R&D-performing SMEs.

‘Eurostars-2’ aligned with the Europe 2020 strategy, its flagship initiative ‘Innovation Union’ and the Commission Communication of 17 July 2012 entitled ‘A Reinforced European Research Area Partnership for Excellence and Growth’, aimed to supporting R&D performing SMEs by co-financing their market oriented research projects in any field. As such, and in combination with the activities under the ‘Leadership in Enabling and Industrial Technologies’ objective set out in Horizon 2020, it aimed to contribute to the goals of the Industrial Leadership part of that programme to speed-up development of the technologies and innovations that were to underpin tomorrow’s businesses and help innovative European SMEs to grow into world-leading companies.

The Eurostars-2 programme built on the experience of its predecessor, the Eurostars-1 programme implemented between 2008 and 2013. The objective of Eurostars-1 was to support transnational market-oriented research and innovation projects initiated and driven by R&D-performing SMEs in order to improve their competitive position. Findings from the final evaluation of the Eurostars-1 programme 6 showed that the financial support successfully helped Eurostars-1 beneficiaries to develop new or improved products and services with which they could improve their competitive position and that 88% of the analysed Eurostars-1 projects developed their planned innovations.

Similar to Eurostars-1, Eurostars-2 had no specific scientific or technological focus but supported cross-border R&I collaboration among SMEs, large firms, research organisations and universities with a strong focus on internationalisation through R&D-driven projects with up to four consortium members. As part of the improvements from the previous Eurostars-1 programme, Eurostars-2 aimed to head towards shorter time-to-grant, stronger integration, and lean, transparent, and more efficient administration to the ultimate benefit of research and development performing SMEs.

Following the previous practice under Eurostars-1, the Eureka Secretariat was designated as the implementation structure for Eurostars-2. The Eureka Secretariat has been created as the operational office of Eureka, an intergovernmental initiative established in 1985 with the objective of promoting cooperation in industrial research. 7

The overall objectives of the Eurostars-2 programme were mainly to provide financial support to SMEs to:

•Support transnational market-oriented research projects initiated and driven by R&D- performing SMEs

•Encourage the development of new products, processes, and services by SMEs

•Promote technological and business development

•Boost the internationalisation of SMEs

To achieve these set goals, the programme's activities took a bottom-up approach; the project partners were free to generate innovation in any technological areas and address any (civil) market areas. The regular calls were bottom-up and easily accessible. One central feature of the Eurostars-2 programme was the mix of a centralized and decentralized application and evaluation approach. Whilst the evaluation of applications was performed by the Eureka Secretariat, the funding and its monitoring for each beneficiary were implemented by the National Funding Bodies (NFBs).

The financial support of the Eurostars-2 programme aimed at leading to the following outputs (short-term):

•Development of international partnerships between SMEs and other stakeholders to work on R&D-driven projects to commercialise new products, processes, and services within two years after project completion.

•Extended international networks of stakeholders within the European and international innovation ecosystems.

•In the medium-term, the following results should be facilitated:

•Improved SME innovation performance.

•Development of new market-ready products/services.

•Leverage of EU structural (ESIF - European structural and investment funds), national, regional funds and private investment.

In the longer term, the Eurostars-2 programme aims to contribute to the following impacts:

•Industrial leadership of EU and associated countries.

•Improved business environment of supported SMEs;

•Acceleration of European product, process, and service innovation.

The following Figure summarises the intervention logic of the Eurostars-2 programme.

Figure 1 Intervention logic of Eurostars-2

2.2 Point(s) of comparison

Overall, building on the lessons from the Eurostars-1 programme, Eurostars-2 targeted to demonstrate clear progress towards further alignment and synchronisation of the national research and innovation programmes as a truly joint programme featuring stronger scientific, management and financial synchronisation. Stronger scientific integration had to be achieved through the common definition and implementation of activities and should ensure the excellence and the high impact of the projects selected. Management integration aimed to ensure further improvement of operational excellence and accountability for the programme. Stronger financial integration was to be based on overall and yearly adequate financial contribution by the States participating in Eurostars-2 and a high degree of national synchronisation. This was to be achieved through a progressive harmonisation of national funding rules.

General objective

To tackle the room for improvement identified above, the following general objective for Eurostars-2 has been set:

•Stimulate economic growth and job creation by enhancing the competitiveness of R&D performing SMEs through transnational R&D collaboration.

Specific objectives

To achieve the general objective above, two specific objectives have been set.

1.SO1. Promotion of transnational research activities for R&D performing SMEs 'close to the market'

2.SO2. Contributing to the completion of the ERA and increasing the accessibility, efficiency, and effectiveness of public funding for R&D performing SMEs in Europe by aligning, harmonising and synchronising the national funding mechanisms

Operational objectives

To reach the specific objectives above, the following operational objectives have been identified.

1.OO1.Scientific integration of national programmes: Ensure excellence and impact of the projects selected through international (EUREKA wide) competition and the application of a single evaluation and selection process.

2.OO2.Management integration of national programmes: Further improve operational excellence and accountability for the programme by reducing the time to contract while maintaining an optimal frequency of calls per year.

3.OO3.Financial integration of national programmes: Harmonisation of national funding rules and application of a binding ranking list.

4.OO4. Facilitate the participation of R&D performing SMEs without previous experience in transnational R&D activities.

3. HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD?

3. 1 Current state of play

Based on the framework of its predecessor programme, the Eurostars-2 programme was implemented as jointly undertaken between EUREKA and the EU in the 2014-2020 funding period by 33 Eurostars-2 participating states and four partner countries. Besides the participation of all EU Member States and COSME 8 countries such as the countries from the European Free Trade Association - EFTA (Iceland, Norway, Switzerland), other countries outside of the European continent such as Canada, the Republic of Korea, and South Africa have been partnering countries in the Eurostars-2 programme.

Between 2014 and 2021, 15 joint calls (cut-offs at least twice per year) took place, and 5891 projects were submitted in total. The number of funded projects in the period from January 2014 to December 2020 amounted to 1546. The Figure 2 below provides for an overview of the number of approved projects for the Eurostars-2 programme between 2014-2020 for each cut-off. While the number of applications differed significantly for each cut-off, the average number of approved projects per cut-off was 103 projects. The countries with the highest number of applications were Germany (2602 applications), the Netherlands (1766 applications) and Denmark (1542 applications). The countries with the lowest number of applications were Greece (eight applications), Malta (14 applications) and South Africa (40 applications). The average success rate defined as the percentage of applicants actually receiving funding of Eurostars-2, was 27%. The average project duration was 30 months and project consortia mostly consisted of an average of two to three organisations.

Figure 2 - Number of approved projects for the Eurostars-2 programme between 2014 - 2020

Source: PPMI/Idea Consult/Prognos (2022), based on data delivered by Eureka Secretariat.

When looking at the implementation status of approved projects, projects from the cut-offs 2- 9 which started between 2014 and 2018 are mostly terminated while most of the projects from the cut-offs 10-15 which started between 2018 and 2021 were still ongoing. As of August 2022, 601 Eurostar-2 projects were still running (39%) and 800 projects were completed (52%). The remaining 9% of the Eurostars-2 projects have been either withdrawn or are on hold because of ethical conflicts. According to the Annual Eurostars-2 report (2021), withdrawals are mainly due to bankruptcy issues, feasibility issues as well as project changes. Main project changes in 2021 were related to prolongation requests (62%).

With regards to the budget committed in the 2014-2020 period, the Eurostars-2 programme had a total committed public budget of EUR 1.14 billion distributed over the Annual Work Plans during that period. While the EU committed EUR 287 million (33.3%) to the financing of the projects, the remaining EUR 856 million (66.6%) has been committed by the Eurostar participating countries themselves through national funding resources. The average project costs have been around EUR 1.4 million, out of which a maximum of 50% was funded through Eurostars-2.

The committed national budget by participating states and partner countries for all 15 cut-offs in the 2014-2020 funding period shows strong variances. The committed grant amounts by the 33 Participating countries and the four Eurostars-2 partner countries differed significantly for each country. While the highest grant amounts have been committed by Germany (EUR 112 million), the Netherlands (EUR 102 million) and France (EUR 74 million), no grants have been committed by Greece. The committed budget per participating state and partner country shows that five countries (Germany, Netherlands, France, Sweden, and Norway) committed almost half of the total committed national budget showing that the intensity of financial commitment by participating country is differing significantly.

Figure 3 - Committed national budget by participating states and partner countries* for cut-off 1-15 in EUR million

Source: PPMI/Idea Consult/Prognos (2022), based on data delivered by Eureka Secretariat. Switzerland, South Korea, Canada and South Africa were Eurostars-2 partner countries in the 2014-2020 funding period.

With regards to the committed budget of the Eurostars-2 programme, the total committed amount for the cut-offs 1-15 was EUR 1.074 billion composed of EUR 891.33 million of official funding committed for the cut-offs 1-15 through the official declaration of commitments by the participating countries and partner countries. Out of this amount, EUR 6.94 million was committed as alternative funding (loan). The committed budget of the partner countries consisting of Switzerland, South Korea, Canada and South Africa was around EUR 84.12 million. The total expected EU contribution for the cut-offs 1-15 was around EUR 281 million.

4. EVALUATION FINDINGS (ANALYTICAL PART)

4.1. To what extent was the intervention successful and why?

Effectiveness and efficiency

The main beneficiaries of the Eurostars-2 programme were R&D-performing SMEs (66%) 9 , 30% of which had no prior experience in international collaboration before having participated in the Eurostars-2 programme. By supporting transnational consortia, the Eurostars-2 programme aimed at increasing the number of cross-regional research activities and partnerships of SMEs with other organisations such as other SMEs or research organisations. The most common transnational collaborations were e.g., for the cut-offs 11 and 12 between project partners from Germany-Switzerland (9% of projects), Germany-Netherlands (9% of projects) and Switzerland-Netherlands (8% of projects).

The transnational partnerships followed a bottom-up approach meaning that no specific thematic nor technological focus was required under the Eurostars-2 programme. Different technological areas were thus addressed in the 2014-2020 funding period. In terms of thematic focus, a strong focus of the Eurostars-2 projects has been on biological sciences and technologies (35%), followed by electronics, IT and telecoms technologies (22%). In terms of markets, while around 38% of the projects focused on the medical and health-related market, other different markets were addressed by the Eurostars-2 programme such as industrial products/manufacturing, computer-related markets and biotechnology markets.

Eurostars-2 as a partnership aims to add value by being open to new beneficiaries as a niche programme that was also, or even in particular, interesting for SMEs with no previous experience in securing public funding. By attracting around 50% of newcomers to the Horizon programme, the Eurostars-2 programme achieved to attract inexperienced SMEs and other organisations to participate in an EU-wide funding programme. Moreover, findings from the Market Impact Reports from 2020 show that 30% of SMEs had no prior experience in international collaboration before having participated in the Eurostars-2 programme. However, based on findings from interviews with NFBs and beneficiaries, the openness of partnerships as well as the flexibility of introducing amendments to the project depends significantly on national rules and differs therefore by each NFB.

The partnerships’ budget leverage factor, in mobilising additional resources, on top of contribution from partners could not be fully established since the NFBs do not follow up on completed projects in that respect. However, findings from the interviews with beneficiaries suggest that the successful implementation of a Eurostars-2 project is perceived as a quality label by private investors.

A lesson learned regarding partnership-specific criteria concerns the divergence in national rules and on how to increase transparency to Eurostars-2 applicants. The uniqueness of Eurostars-2 rests on the funding of transnational collaborative projects led by R&D performing / innovative SMEs. However, consortium members for a given project face different rules for application depending on their country of origin. In addition, funding is allocated based not only on the position on the ranking list but also according to the availability of national funding. There is scope for further convergence of the rules and for better explanations to applicants of the link between the availability of national funding and the provision of grants.

Findings from the final reports of completed Eurostars-2 projects of the cut-off 1-5 indicate that 94% of beneficiaries of completed Eurostars-2 projects perceived the programme as effective and only 6% saw potential for improvement. Interviews with beneficiaries indicate that the Eurostars-2 funding was relevant to taking the first steps to further elaborate their innovation which would have been difficult for smaller enterprises without the public funding support. Findings from the interviews suggest that the elements of cross-regional cooperation between different organisations combined with the bottom-up approach are one of the central drivers of the Eurostars-2 programme giving interested organisations the flexibility to find adequate collaboration partners outside of their country. The initiated knowledge transfer between partnerships worked out very well in many cases. In addition, some SMEs explained that the Eurostars-2 support was perceived as a de-risking factor and success label for private investors making the projects more attractive for private investments.

Coherence

According to the interim evaluation of the Eurostars-2 programme (2017), the programme was one of several funding programmes aiming at boosting growth, jobs and innovation in Europe implemented by the EU. By having been implemented under Article 185 TFEU, the programme is considered a niche programme that addresses SME needs which are not covered by national or regional funding programmes. By strengthening the transnational cooperation between the Member States, the Eurostars-2 programme supported the establishment of the European Research Area on two levels. On the one hand, NFBs cooperated with the Eureka Secretariat while putting into place cross-regional funding schemes which would not be possible in this broad scope by national funding programmes. On the other hand, based on the limited information available, it appears that SMEs successfully transferred their knowledge across borders within Europe and even in some cases worldwide, contributing to support research excellence and leadership of industry stakeholders in Europe.

Overall, Eurostars-2 supported synergies in terms of cross-border learning between NFBs and SMEs. There are strong indications that the EU co-funding was an incentive for NFBs to take part in the Eurostars-2 programme and improve their own national funding processes.

While NFBs and the Eureka Secretariat endeavoured to further align national and European funding rules, NFBs reported that there were still delays in the projects’ start as the time-to-contract differs from each participating state and partner country. According to the interim evaluation (2017), the Eurostars-2 programme belongs with the EIC Accelerator (former SME Instrument) to the most known funding instruments of Horizon 2020. Findings from the 15 interviews show that NFBs and beneficiaries also perceive a high level of coherence and sustainability among partnerships. This is also confirmed in the final reports of cut-offs 1-5.

Communication and coherence within the partnerships are perceived by interviewed beneficiaries as good.

Moreover, several interviewed SMEs indicated that the Eurostars-2 programme has been a good starting point to develop a product, process, or service through public funding support as the success rate (27%) has been relatively high compared to the EIC Accelerator. Several interviewed beneficiaries have been successful in a second step in receiving funding from the EIC Accelerator and see synergies between both programmes.

4.2. How did the EU intervention make a difference?

The Eurostars-2 programme was an Article 185 initiative consisting of national funding efforts that have been topped up by funding contributions of the EU. From the perspective of interviewed beneficiaries, this mixed centralised and decentralised approach is unique and perceived as an EU added value. The Eurostars-2 programme is regarded as a well-established support programme for SMEs in the European Research Area allowing different types of organisations to collaborate with international partners while having the well-known local NFBs as a national contact institution. For many SMEs, this programme offers the possibility to go international for the first time and to learn more about other EU markets. The Eurostars-2 programme is thus not only offering the opportunity to strengthen the relationship with the consortium partners but to discover and reach out to new markets and contacts. Especially companies that do not have high resources have the possibility through the programme to collaborate with other companies and develop new intellectual properties and leverage skills that they do not have in their home countries.

Another important feature of the programme is the open innovation and bottom-up approach giving SMEs with different sectoral backgrounds the possibility to apply for funding. The Eurostars-2 programme has been perceived by beneficiaries as an important support instrument to strengthen the European Research Area and beyond. As national and regional support programmes do often not allow cross-border cooperation, the Eurostars-2 programme seems to have been in the 2014-2020 funding period a successful niche programme for SMEs and other organisations wishing to engage in transnational collaboration.

Compared to other funding programmes, the Eurostars-2 programme offered extended possibilities to initiate a cooperative learning process with successful international partners with competencies that would have been not available on the regional or national level. Moreover, interviewed beneficiaries emphasize that the granting amount offered at the EU level is generally higher than at the national or regional level. Therefore, allowing them for comparatively more activities performed through the Eurostars-2 programme.

With regards to the project results, one central objective of the Eurostars-2 programme was to support organisations to develop market-ready products, services, and processes. The limited number of Market Impact Reports from 2020 suggest that most commercialised results were products (53%), followed by services (28%) and processes (19%).

Interviews with beneficiaries suggest that the Eurostars-2 programme was overall successful in its activities to support R&D performing SMEs to develop new processes, products or services. Beneficiaries were capable to develop patents and prototypes that led to the growth of the company and other beneficiaries received private investments after the project's completion to further develop their project.

With regards to the rating of the overall technological achievements of beneficiaries of the cut-offs 1-5, 89% of the beneficiaries either ranked their technological achievement as good or even excellent. Reasons indicated in the final reports on why some technological achievements were not satisfying were e.g., changes in the regulatory environment requiring a technical change and a different approach to market entry or technological problems that could not be solved during the project implementation. Project results were mainly commercialised in Europe, followed by North America and Asia.

Findings from the final reports of the cut-offs 1-5 show that 75% of the Eurostars-2 beneficiaries were satisfied with the outcome of their project and 86% stated that they planned to continue the collaboration of their partnership after the end of their funding. Overall, based on preliminary results, the number of full-time equivalents that resulted from Eurostars-2 partnerships after the completion of the projects reached the overall programme objective of 1500 FTEs.

According to interviews performed with SMEs, the Eurostars-2 projects seem to have contributed to the EU policy priorities such as the twin transition (green and digital transition) as well to the Sustainable Development Goals (SDGs). Data from final reports provided by the Eureka Secretariat show that 29% of beneficiaries confirmed that their project contributed to one or more of the objectives of the SDGs.

4.3. Is the intervention still relevant?

By targeting R&D performing SMEs, the Eurostars-2 programme aimed at boosting innovation, growth, and competitiveness of the EU innovation ecosystem through applied research and development. The financial support of the Eurostars-2 programme contributed to the bottom-up and market-oriented development of new, innovative products, processes, and services.

The nearly constantly increasing number of applications to the 15 cut-offs is a good indicator showing that the programmes’ objectives and support schemes were and are still relevant for the main target group, namely interested R&D-driven SMEs and other SMEs. While the average success rate of Eurostars-2 applicants for all 15 cut-offs (final submission dates for the calls) of the calls was 27% of proposals awarded, the success rate has been significantly lower during the last four cut-offs with success rates between 19-25% showing that more and more SMEs and other organisations have applied for Eurostars-2 funding support in the 2014-2020 funding period. Compared to the Eurostars-1 programme (26%), the success rate remained similar. The 7 interviews with National Funding Bodies pointed to the existing need and demand for financial support for R&D-driven companies.

5. WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED?

5.1. Conclusions

Target group and scope of the programme

The evaluation faced certain limitations. Between 2014-2020, 15 Eurostars-2 calls took place and around 1546 Eurostars-2 projects were selected. However, the final evaluation only shows a limited picture of the results and impact of completed Eurostars-2 projects. As Eurostars-2 projects have an implementation duration of 36 months and many projects were extended due to the impact of the COVID-19 pandemic, only 52% of all Eurostars-2 projects were completed by August 2022, at the time of completion of the external evaluation study. In addition, the methodology and questions asked in the final reports (FIR) were changed after cut-off 5 to further harmonise report templates for all EUREKA programmes. However, based on interviews with 6 beneficiaries, 7 NFBs and 1 EU official and the feedback following the ‘call for evidence’, the Eurostars-2 programme is regarded as a relevant support programme for SMEs to develop new and innovative products, processes, and services. While a strong focus of the funded projects has been on biological science and technologies (35%) and electronics, IT, and telecoms technologies (22%), the technological openness and the bottom-up approach are perceived by interviewees as one of the major benefits of the Eurostars-2 programme. According to interviewed SMEs, the geographical openness, and the possibility to collaborate with partner organisations from a broad range of the 33 participating countries is one of the EU added values of the Eurostars-2 programme compared to regional or national funding programmes which often have also lower funding rates. The increasing number of applications throughout the 15 cut-offs indicates the existing relevance of the programme for the target groups. For some interviewed stakeholders, the Eurostars-2 programme is an attractive but fewer known niche programme giving SMEs unexperienced in going international for the first time the chance to collaborate with international partners.

Governance and management

Overall, findings from the final evaluation indicate that the governance structure of Eurostars-2 by the Eureka Secretariat together with the NFBs is adequate. According to interviewees, the governance structure is complex but adapted to the needs of the Eurostars-2 programme and cost-efficient. The Eureka Secretariat confirms that the governance model has been re-established in terms of risk management policies for the organisations.

According to available information, the Eurostars-2 programme seems to have had a positive impact. As around half of the projects were still running at the time of the evaluation, and the monitoring and reporting system did not allow for a proper tracking of results, it is not possible to draw detailed conclusions on final project results and their impact.

The external evaluation points to the role of the Eureka Secretariat which is broadly confirmed to be relevant for the successful implementation of the programme because of its broad, international network and expert database with broad knowledge in different technological fields. The Eureka Secretariat did, however, by the time of the evaluation, not provide more detailed monitoring and reporting information on project results allowing to fully assess the effects of the programme nor the efficiency of its implementation.

Interviews with NFBs show that application and participation rates differ in the different participating states and partner countries. As outlined in more depth in the case study on participation this is largely explained by the design of the programme, which means that the budget allocated at the national level is a strong determinant of the number of projects funded with beneficiaries from a given country. The countries that provide the strongest input have the most participants.

The centralised evaluation process of applications is generally confirmed to be well structured, however it is also sometimes criticised for being non-transparent to applicants. Complicated application processes lead to sub-business of external firms helping with the application process and writing the actual project application according to interviewees. In addition, beneficiaries emphasised that the lack of user-friendliness of the Eureka Secretariat website was perceived in some cases as a challenge for applicants to understand specific national eligibility criteria applied in the context of an EU level funding programme.

Results and impacts of Eurostars-2 projects

Due to IT and database issues in the Eureka secretariat at the time of the study the study team faced a lack of information and data on project results and impacts of Eurostars-2 projects. Only insights from interviews and desk research are integrated into the final evaluation giving only selective and limited insights into the actual results and impacts. With regards to the project implementation and results, interviews with beneficiaries indicate that Eurostars-2 projects seem to be successful to develop new processes, products or services. To follow up on the achievements and success of the projects, the final report and market impact reports are requested periodically for three years from the Eurostars-2 projects by the Eureka Secretariat. However, these reports have certain limitations as some organisations participating in the Eurostars-2 programme do not have any market impact (e.g., universities) and beneficiaries stop replying after having received their final payment. Overall, interviews with beneficiaries and feedback from various SMEs in the Call for Evidence show that SMEs have been successful in developing new patents and prototypes and received private investments after the project completion. In this context, Eurostars-2 projects are perceived as quality labels by private investors.

5.2 Lessons Learned

Overall, the Eurostars-2 programme has been successful in boosting innovations developed by international project consortia with a focus on SMEs. Based on the recommendations given in the interim evaluation (2017) and progress and improvements made since 2017, the following lessons can be drawn.

Monitoring and data collection for the partnership

Due to a lack of more detailed information and data on final project results and impacts of Eurostars-2 projects, mainly insights from interviews and desk research are integrated into this final evaluation giving only limited information on the actual results and impacts. Although the Eureka Secretariat has changed its monitoring system in response to the Interim Evaluation (2017), which noted the insufficient accuracy and lack of up-to-date information in the Eureka Secretariat database, it was not yet possible to assess at the time of the final evaluation (2022) if the new monitoring has improved data timeliness and availability, in particular due to the ongoing changes in the IT system of the Eureka Secretariat. Nevertheless, changes in IT systems, staff replacement and clearer guidance in the project agreement are expected to lead to an improved situation in the successor programme of Eurostars-2, which is subject to reporting obligations related to co-funded partnerships under Horizon Europe, with an increased focus on impact.

Uncertainty to get funding for selected projects when the contribution of participating states has been exhausted by other projects

This was a key weakness identified in the interim evaluation. Due to the lack of information and data noted above this aspect was not fully examined in this final evaluation. However, since the selection process for the partnership did not change in the interim period, it is likely that the issue persisted throughout Eurostars-2.

Divergence in national rules and transparency to applicants

Consortium members for a given project face different rules for application depending on their country of origin. In addition, funding is allocated based, not only on the position on the ranking list, but also according to availability of national funding. The link between availability of national funding and provision of grants is not always clearly explained to applicants.

Synergies between the Eurostar programme with other EU funding instruments

The Eurostars-2 programme has been perceived as an interesting funding programme for SMEs applying for the first time for EU funding apart from their national or regional funding programmes, with potential to further strengthen its synergies with other EU funding programmes (e.g., EIC Accelerator).

Geographic diversity of participation

The findings indicate that five out of the 33 participating states and four partner countries contribute around 49% of the total committed national budget of the Eurostars-2 programme. Due to the design of the programme there is a strong correlation between budget allocation and participation at national level.

Visibility of the Eurostars-2 programme in some participating states:

While some participating countries offer different types of support services and promotional activities that increase the visibility of the Eurostars-2 programme on the national level, other NFBs do not have the financial and human resources to promote the programme on the national level. The lack of certain minimum level of promotional activities to be performed at the national level results in an uneven visibility of the Eurostars-2 programme in all participating states. The lack of more centralised communication activities and better coordinated communications and social media activities by Eureka Secretariat also leads to a suboptimal visibility of the programme.

Time-to-contract

The time-to-contract is the time between the cut-off date and the date of signature of the grant agreement. In the bilateral agreements between the NFBs and the Eureka Secretariat, the involved parties agreed to keep the time-to-contract within seven months meaning that the Eureka Secretariat would communicate funding results to the applicants of selected Eurostars-2 projects and their respective NFBs within four months and that NFBs would finalise the grant agreement within three months after the communication of the funding results by the Eureka Secretariat. Statistics show that the average time-to-contract decreased from cut-off 1 with an average time-to-contract of 9,7 months to 6,6 months for the cut-off 14. However, divergences of national rules in the different participating states and partner countries appear to be a barrier to the operational performance of the Eurostars-2 programme and further synchronisation of national rules is a key factor for improvement. The adaptions of the time-to-contract in the 2021-2027 funding period are perceived by interviewed beneficiaries and the SME’s that supplied feedback in the call for evidence as an important step.

Absorption of designated budget

The design of the programme, where national funding is topped up with the European Commission contribution is unique. Committed national funding was distributed as shown in the figure below and subsequently topped up with the Commission portion.

Figure 4 - Committed national funding in Eurostars-2 per country, EUR million

Source: PPMI/Idea Consult/Prognos (2022), based on data delivered by ESE. Switzerland, South Korea, Canada and South Africa as Eurostars-2 partner countries in the 2014-2020 funding period.

A comparison of the committed funding with the actual funding provided is only possible to a limited extent. This is because at the time of the report only the first six cut-offs had been closed. For all other cut-offs, the Eureka Secretariat still receives information on expenditures from NFBs, so a final comparison of the pre-committed budget to the actual funding provided is not possible. Nevertheless, from the data of the closed first six cut-offs, there is a considerable discrepancy between the pre-committed budget and the actual funding provided. The reasons for this discrepancy are manifold and were answered in different ways in the interviews. Besides others, one reason relates to the design of the programme. The fact that the ranking list is created independently means that it does not exactly match the NFBs' commitments, so not all funds are allocated. In addition, NFBs confirm that it is difficult for countries to set a fixed amount for Eurostars-2 in advance, as these are checked (and possibly re-allocated) in the national budget right before the cut-off. So if the funding is needed elsewhere, it may well be that less budget is available for Eurostars-2 than originally committed. Hence, partly because of the matching needs between the national budgets, calls and available EC budget the programme failed to absorb the full budget dedicated to it. As compared to other instruments, such as the EIC Accelerator where absorption ratios are around 100%, Eurostars-2 lacked tools and instruments to enhance flexibility of call management to ensure full budget absorption of the programme. Still, towards the end-phase of the programme the European Commission facilitated the creation of an extra call during the Covid outbreak, which ensured continuity of the programme. The extra call was funded using the substantial underspend over the lifetime of the programme. A total of EUR 16.6 million EU funding was dedicated to the 15th call, meaning that the total current EU underspending for the lifetime of the programme was reduced to EUR 31.4 million, which however were not lost but could be used in other parts of Horizon 2020.

ANNEX I: PROCEDURAL INFORMATION

Lead DG

The European Commission’s Directorate-General (DG) for Research and Innovation is the lead DG for this evaluation (PLAN/2022/1345).

Organisation and timing

The Commission published a ‘call for evidence’ on the final evaluation of the ‘Eurostars-2 programme (EU partnership on innovative SMEs)’ on 23 September 2022 that was open for feedback until 21 October 2022.

Three partnerships based on Article 185 TFEU (the Active and Assisted Living Research and Development Programme (AAL2), Eurostars-2 and the Partnership for Research and Innovation in the Mediterranean Area (PRIMA)) have evaluations coming up by the end of 2022. In this context, DG Research and Innovation set up one inter-service group (ISG) to oversee the three evaluations.

The ISG was established on 4 July 2022 involving representatives from the Secretariat-General, DG for Research and Innovation, DG for Communications Networks, Content and Technology, DG for Agriculture and Rural Development, DG for Informatics, DG for Budget, DG for Competition, DG for Education, Youth, Sport and Culture, DG for Environment, DG for Migration and Home Affairs, DG for Maritime Affairs and Fisheries, DG for Structural Reform Support, the Joint Research Centre and the Executive Agency for Small and Medium-sized Enterprises. The ISG contributed to the evaluation and ensured that it met the necessary standards for approval of the final report. Two meetings were held.

Evidence, sources, and quality

This evaluation report drew on the following sources of evidence:

• Beck et al. (2019): Eurostars. The international programme for Research Intensive SMEs. A joint Swiss Danish Impact Study. Available under: https://innovationsfonden.dk/sites/default/files/2019-12/eurostars-a-joint-swiss-danish-impact-study_0.pdf .

• ERA-LEARN (2020): Annual Report on Public-Public Partnerships 2020. Available under: https://www.era-learn.eu/news-events/news/era-learn-annual-report-on-public-public-partnerships-2020 .

• ERA-LEARN (2020): Workshop report. “Supporting the preparation of future European Partnerships" Brussels, 9-10 March 2020. Available under: https://www.era-learn.eu/documents/era-learn-ws-report-supporting-the-preparation-of-future-european-partnerships.pdf/view .

• European Commission (2014) Final evaluation of the Eurostars Joint Programme. Available under: https://www.researchgate.net/publication/323644226_Final_Evaluation_of_the_Eurostars_Joint_Programme_Expert_group .

• European Commission (2017): Interim Evaluation of the Eurostars-2 Joint Programme. Available under: https://op.europa.eu/en/publication-detail/-/publication/e6bbaa13-b867-11e7-ac8e-01aa75ed71a1/language-en .

• European Commission (2017): Impact assessment of EUREKA network projects and cluster projects. Available under: https://www.researchgate.net/publication/319162491_Impact_Assessment_of_EUREKA_Network_Projects_and_Cluster_Projects .

• European Commission (2017): Commission staff working document on in-depth interim evaluation of Horizon 2020. Available under: https://op.europa.eu/de/publication-detail/-/publication/33dc9472-d8c9-11e8-afb3-01aa75ed71a1/language-en .

• European Commission (2017): Meta-evaluation of Article 185 initiatives – report of the expert group. Available under: https://op.europa.eu/en/publication-detail/-/publication/3966c4a7-b47c-11e7-837e-01aa75ed71a1/language-en .

• European Commission (2019): European Partnerships under Horizon Europe: results of the structured consultation of Member States. Available under: https://www.era-learn.eu/news-events/news/european-partnerships-under-horizon-europe-results-of-the-structured-consultation-of-member-states-1 .

• European Commission (2021). Study on the effectiveness of public innovation support for SMEs in Europe. Available under: https://op.europa.eu/de/publication-detail/-/publication/d031aa03-9295-11eb-b85c-01aa75ed71a1/language-en .

• EUREKA (2018): Eurostars-Guidelines for project progress reports 2018-2020. Available under: https://www.eurekanetwork.org/programmes/eurostars/guidelines .

• EUREKA (2019): Eurostars – Eligibility guidelines for applications. Available under: https://www.eurekanetwork.org/programmes/eurostars/guidelines .

• EUREKA (2019): Annual report. Available under: https://issuu.com/eurekaassociation/docs/annual_report_2019 .

• EUREKA (2020): Eurostars - Submitting your Project Progress Report and Final Report online. Available under: https://www.eurekanetwork.org/dA/9b6b673323/Guidelines+for+the+online+submission+of+PPRs+and+FiRs+(Nov+2020).pdf?language_id=1 .

• EUREKA (2014): Annual Report 2014. Eurostars-2.

• EUREKA (2015): Annual Report 2021. Eurostars-2.

• EUREKA (2016): Annual Report 2021. Eurostars-2.

• EUREKA (2017): Annual Report 2021. Eurostars-2.

• EUREKA (2018): Annual Report 2021. Eurostars-2.

• EUREKA (2019): Annual Report 2021. Eurostars-2.

• EUREKA (2020): Annual Report 2021. Eurostars-2.

• EUREKA (2021): Annual Report 2021. Eurostars-2.

External expertise

Expert advice has been widely used to prepare the Commission Staff Working Document. It mainly includes the Study report on the final evaluation of Eurostars-2, commissioned by the European Commission. The contractor is PPMI and more specifically Prognos. The study, Eurostars-2 final evaluation – Evaluation study of the European Framework Programmes for Research and Innovation for an innovative Europe, Publications Office of the European Union, 2023, has been published in 2023 by Directorate-General for Research and Innovation.

ANNEX II. METHODOLOGY AND ANALYTICAL MODELS USED

Study design

Prognos, as part of a consortium led by PPMI, carried out a support study to provide input for this evaluation. The study was delivered over a period of nine months in 2022.

Limitations and reliability of data

The feedback to the ‘call for evidence’ was extensive and largely confirmed satisfaction with the programme and the key findings of the study.

The study report on the final evaluation of Eurostars-2 was largely conducted by using desk research and the analyses of administrative data and stakeholder interviews. With the exception of the interviews, it did not collect any new data but relied in its analyses on the data that had already been collected and presented elsewhere.

The evaluation builds on a broad set of qualitative and quantitative data such as desk research, stakeholder interviews, an intervention logic, and secondary data such as reports and earlier evaluations. The EUREKA secretariat provided secondary data in the form of Annual Reports and monitoring data of the Eurostars-2 programme. In addition, the evaluation is enriched by findings from two case studies on the Eurostars-2 programme performed as part of the wider study on the support to the innovation of the EU Research & Innovation Framework programme (‘Framework Programme’).

The evaluation faces certain limitations. Between 2014-2020, 15 Eurostars-2 calls took place and 1546 Eurostars-2 projects were selected. However, the final evaluation only shows a limited picture of the results and impact of completed Eurostars-2 projects. As Eurostars-2 projects have an implementation duration of 36 months and many projects were extended due to the impact of the COVID-19 pandemic, only 52% of all Eurostars-2 projects were completed by the 24th of August. In addition, the methodology and questions asked in the final reports (FIR) were changed after cut-off no. 5 to further harmonise report templates for all EUREKA programmes. Several questions were therefore not available anymore in the FIR from cut-off no. 6 onwards. In addition, the market impact of Eurostars-2 projects is followed up through two market impact reports (MIR) which are requested from beneficiaries after the end of the project completion. MIRs from 2019 and 2020 have been available for the final evaluation.

As interviews were meant to primarily cover the content of the case studies, but also contribute to the final evaluation, the list of questions covering two case studies and the evaluation was extensive. This meant that not all questions could be given the same priority during the interviews and questions were selected that were most pertinent to interviewees.

Methodology, sources of information and data analysis

The methodology for the support study was based on:

•Desk-based research;

•Interviews;

•Case studies;

•Analysis of funding and administrative data;

•Other quantitative methods.

The final Eurostars-2 evaluation follows a mixed-method approach by combining quantitative and qualitative data collection and analysis methods. The desk research was a starting point to get the first insights on the Eurostars-2 joint programme. In addition, information was enriched by expert interviews with relevant stakeholders. Monitoring data provide an additional source of evidence on the actual results and impact of the Eurostars-2 programme. Moreover, findings from the interim evaluation and the Eurostars-2 case studies, which are also part of the study on the support to the innovation of the EU Research & Innovation Framework programme (‘Framework Programme’), were giving additional insights.

The evaluation is based on 12 evaluation questions linked to the EU evaluation criteria. The following Table gives an overview of the six evaluation criteria of this evaluation.

EVALUATION CRITERIA AND RELEVANT QUESTIONS

Evaluation criteria	Judgement criteria
Relevance	This refers to the assessment of the relationship between the needs of society /target groups – and the objectives of the initiative.
Coherence	Coherence covers the assessment of the initiative compared to other EU initiatives and policies and if possible, to relevant national and regional policies. internal coherence is considered as coherence with the evaluated cluster support initiatives. external coherence is considered as coherence with national/regional support and other EU-level programmes.
Efficiency	This part assesses the relationship between the resources used (i.e., inputs) and the outputs achieved.
Effectiveness	The effectiveness criterion assesses how successful the different initiatives have been in terms of achieving or making progress towards the set objectives.
EU added value	Assessment of whether the achievements of the initiative could have been achieved without EU intervention (by national actions by the member states) and why action on the EU level is required.
Partnership	The partnership criterion deals with all questions related to the partnership of entities within the frame of the Eurostars-2 programme such as private funding and openness of partnerships

Desk research

The desk research provided a first, comprehensive overview of the Eurostars-2 programme and allowed to learn more about the developments of the Eurostars-2 programme since the 2007-2013 funding period and the interim findings and recommendations of the interim evaluation (2017).

Critical assessment of work carried out by external contractor

The work carried out by the contractors is of good quality. The Commission services agree with the conclusions presented as this address the key issues arising from the evaluation.

ANNEX III. EVALUATION MATRIX AND, WHERE RELEVANT, DETAILS ON ANSWERS TO THE EVALUATION QUESTIONS (BY CRITERION)

Evaluation Matrix

Criterion	Guiding questions	Evidence-based answers
Evaluation criteria defined by the Better Regulation Guidelines
Relevance	To what extent have the objectives of the partnership been, and are still relevant vis-à-vis of the needs and problems addressed by the Framework Programme? How flexible has the partnership been?	The analysis confirms that the Eurostars-2 programme was successful in reaching its main target group consisting of R&D-performing SMEs and other SMEs. Out of the total number of 12.968 participants who were and are part of consortia applying for Eurostars-2 funding in 15 Cut-offs, 68% of all participants involved were either R&D-performing SMEs or other SMEs. For the participating SMEs it is still relevant to also involve other beneficiaries such as universities and other research-performing organisations to benefit from research findings and to further develop their products, processes or services. Within the funding period of 2014-2020, the Eurostars-2 programme had to face the consequences of the outbreak of the COVID-19 pandemic which started in the beginning of 2020. While 12 out of 15 calls had already taken place at that moment and most of the Eurostars-2 projects had already started their activities (77.2%), the pandemic had mostly an impact on the cross-border project implementation. With regards to the impact of the general work processes such as on-spot evaluation processes, IEP (Independent Evaluation Panel) sessions, and meetings of the Eurostar Ethics Panel, new collaborative ways were put into practice to continue activities normally. Call management processes were flexible and adapted to the circumstances of the COVID-19 pandemic and internal as well as external physical meetings such as promotional events changed to online meetings. Constantly increasing numbers of submitted project applications during the cut-offs 13-15 and the highest number of approved projects with a total of 123 projects show that the COVID-19 pandemic did not influence the relevance market needs of R&D-performing SMEs targeted by the Eurostars-2 programme.
Coherence	How well do the different actions work together, internally (i.e., to the partnership, with other partnerships and with other Framework Programme activities), and with other EU interventions/policies (complementarities, synergies, overlaps)? Is Eurostars-2 more effective in achieving synergies, compared to other modalities of the programme?	By having been implemented under Article 185 TFEU, the programme combines national and EU rules and is considered a niche programme that addresses SME needs which are not covered by national or regional funding programmes. By strengthening the transnational cooperation between the Member States, the Eurostars-2 programme focused on the establishment of the European Research Area on two levels. On the one hand, NFBs cooperated with the ESE while putting into place a cross-regional funding schemes which would not be possible in this broad scope by national funding programmes. On the other hand, SMEs successfully transferred their knowledge across borders within Europe and even in some cases worldwide. The high satisfaction rate collected throughout the interviews in the study indicates that the Eurostars-2 programme was a successful programme within Horizon 2020 to further support research excellence and leadership of industry stakeholders in Europe. Overall, Article 185 TFEU supported synergies in terms of cross-border learning between NFBs and SMEs. There are strong indications that the EU top-up contribution is an incentive for NFBs to take part in the Eurostars-2 programme. Eurostars-2 programme has been a good starting point to develop a product, process, or service through public funding support as the success rate has been relatively high compared to the EIC Accelerator. Several interviewed beneficiaries have been successful in a second step in receiving funding from the EIC Accelerator and see synergies between both programmes.
Efficiency	What is the relationship between the resources used by the partnership and the changes it is generating? How did processes cater for flexibility needs in implementation? How cost-effective has Eurostars-2 been? How proportionate were the costs of application and participation borne by different stakeholder groups, taking into account the associated benefits?	Insights shared by beneficiaries and NFBs confirm that the Eurostars-2 programme is overall perceived as efficient in terms of its implementation processes to achieve its main objective. The mix of national and European funding is seen as efficient and funding amounts were high enough for successful project implementations. The Eurostars-2 programme is perceived as straightforward when it comes to the application and evaluation process compared to other central EU funding programmes. Information events organised by national funding bodies (e.g. in Denmark or Sweden) are perceived as helpful guidance in the simultaneous dual-application process on the European and national levels. The Eurostars-2 programme is perceived by those SMEs as a good starting point for start-ups and SMEs to get familiar with EU funding rules and processes, especially for interested SMEs that have not applied beforehand for EU funding programmes. The centralised evaluation process of applications is generally confirmed to be well structured and transparent for applicants. Nevertheless, findings from the interim evaluation (2017) and insight from interviews with NFBs show that the mix of centralized and decentralized implementation structures is also leading to different funding rules and rates that apply in the different participating states. Even though the participating states are constantly trying to align their national rules as reported by interviewed NFBs, differing numbers of applications and participants in the participating states show that the full potential of the Eurostars-2 programme is not exploited in all participating states. In this context, the case study on varying participation rates of Eurostars-2 participating states confirms that the main reason for the lower number of applications in some participating states is due to different programme budgets and the lack of visibility of the Eurostars-2 programme. Also, findings from interviews with NFBs show that different supportive and promotional activities that are performed by some NFBs can be a central factor to boost application and participation rates. For instance, the Swedish, Dutch and German NFBs explained during the interviews that they have implemented several successful promotional activities such as dedicated websites or dedicated informative events for interested SMEs. However, promotional activities, like participation rates, are closely linked to budget amounts by the participating states which differed in the 2014-2020 funding from EUR one million (South Africa) to EUR 112 million (Germany).
Effectiveness	What is the progress made towards the objectives of the partnership and those of the Framework programme, including the contribution to EU priorities and Sustainable Development Goals? Were adequate systems put in place to produce and share lessons learnt from implementation and results achieved, for policy making and between Framework Programme interventions? To what extent does the programme communication/valorisation strategy allow identifying, capitalising upon and (possibly) transferring good practices/results? Includes also the partnership-specific question of how the partnership has helped foster the international positioning and visibility of the European R&I system, and an assessment of the level of international cooperation at partnership and project level	The main beneficiaries of the Eurostars-2 programme were R&D-performing SMEs (66%). By supporting transnational consortia, the Eurostars-2 programme aimed at increasing the number of cross-regional research activities and partnerships of SMEs with other organisations such as other SMEs or research organisations. The transnational partnerships followed a bottom-up approach meaning that no thematic and technological focus was required under the Eurostars-2 programme. Different technological areas were addressed through the Eurostars-2 programme in the 2014-2020 funding period. In terms of thematic focuses of the Eurostars-2 projects, a strong focus has been on biological sciences and technologies (35%), followed by electronics, IT and telecoms technologies (22%). While around 38% of the projects focused on the medical and health-related market, other different markets were addressed by the Eurostars-2 programme such as industrial products/manufacturing, computer-related markets and biotechnology markets. Findings from the final reports of completed Eurostars-2 projects of the cut-off nos. 1-5 show 94% of beneficiaries of completed Eurostars-2 projects perceived the programme as effective and only 6% saw the potential for improvement. Interviews with beneficiaries confirm that the Eurostars-2 funding was relevant to taking the first steps to further elaborate their innovation which would have been difficult for smaller enterprises without the public funding support. Key benefits of participation in the Eurostars-2 programme mentioned in the final impact reports of the cut-offs nos. 1-5 were mainly the possibility for SMEs to collaborate with academia (28%) as well as the increased visibility and reputation (25%). Moreover, the findings from the conducted interviews indicate that the elements of cross-regional cooperation between different organisations combined with the bottom-up approach were regarded as one of the central drivers of the Eurostars-2 programme giving interested organisations the flexibility to find adequate partners outside of their country to collaborate. The initiated knowledge transfer between partnerships worked out very well in many cases. In addition, some SMEs explained that the Eurostars-2 support was perceived as a de-risking factor and success label for private investors making the projects more attractive for private investments.
EU added value	What is the value resulting from the partnership that is additional to the value that could result from interventions carried out at regional or national level or with other forms of implementation?	The Eurostars-2 programme is perceived by beneficiaries as an important support instrument to strengthen the European Research Area and beyond. As national and regional support programmes do often not allow cross-border cooperation, the Eurostars-2 programme has been the 2014-2020 funding period a successful niche programme for SMEs and other organisations wishing to engage in transnational collaboration. Compared to other funding programmes, the Eurostars-2 programme offered the possibility to initiate a cooperative learning process with successful international partners with competencies that would have been not available on the regional or national level. Moreover, interviewed beneficiaries emphasize that the granting amount offered at the EU level is generally higher than at the national or regional level. Therefore, more activities could be performed through the Eurostars-2 programme.
Additional partnership-specific criteria
Additionality	How much additional private and/or public R&I investments on EU priorities have been mobilised thanks to the partnership, under the partnership and on top of contribution of partners, both at national and European level? How do partnerships facilitate the creation and expansion of R&I networks that bring together relevant and competent actors from across Europe, thus contributing to the realisation of the ERA? To what extent has Eurostars-2 created conditions for the competitiveness of the Union's industry, aimed at a better exploitation of the industrial potential of policies of innovation, research and technological development (H2020), in particular with regards to ICT based products and services for active and healthy ageing?	The successful implementation of a Eurostars-2 project is perceived as a quality label by private investors. Evidence suggests that investors perceive the previous public funding support as a de-risking factor and a proof of quality making the project more attractive for private investments. Eurostars-2 funding helped companies to develop patents and attract further private investments. To further boost successful Eurostars projects, the Eureka Secretariat InnoVest Programme was introduced in 2017. The new programme was set up in cooperation with European Business Angels Network (EBAN), the European Business and Innovation Centre Network (EBN) and Tech Tour. Its objectives have been to facilitate the matching of investment-ready companies with investors and to increase the investment awareness of SMEs having participated in the Eureka Secretariat projects. The programme has only been open to SMEs coming from the Eureka Secretariat national authorities financing the programme.
Transparency & openness	How open is the partnership to new participants (incl. procedures / mechanisms to involve new members at partnership and project level, as well as gradually engage a broader set of stakeholders across Europe)? How transparent are the processes for consulting all relevant stakeholders and constituent entities in the identification of priorities? How accessible is the partnership to SMEs?	NFBs and beneficiaries perceived Eurostars-2 as a niche programme that was also interesting for SMEs with no previous experience in securing public funding. By attracting around 50% of newcomers to Horizon 2020 programme, the Eurostars-2 programme achieved to attract inexperienced SMEs and other organisations to participate in an EU-wide funding programme. Moreover, findings from the Market Impact Reports from 2020 show that 30% of SMEs had no prior experience in international collaboration before having participated in the Eurostars-2 programme. Based on findings from interviews with NFBs and beneficiaries, the openness of partnerships as well as the flexibility of introducing amendments to the project depends significantly on national rules and differs therefore by each NFB.

ANNEX IV. OVERVIEW OF BENEFITS AND COSTS

Due to lacking information and data on project results and impacts of Eurostars-2 projects, only insights from interviews and desk research are integrated into the final evaluation giving only selective and limited insights into the actual results and impacts. With regards to the project implementation and results, interviews with beneficiaries show that the Eurostars-2 programme indicates that Eurostars-2 projects seem to be successful to develop new processes, products or services. To follow up on the achievements and success of the projects, the final report and market impact reports are requested periodically for three years from the Eurostars-2 projects by the ESE. However, these reports have certain limitations as some organisations participating in the Eurostars-2 programme do not have any market impact (e.g., universities) and beneficiaries stop replying after having received their final payment. Overall, interviews with beneficiaries show that SMEs have been successful in developing new patents, and prototypes and received private investments after the project completion. In this context, Eurostars-2 projects are perceived as quality labels by private investors.

Overview of costs and benefits identified in the evaluation
	Citizens/Consumers			Businesses		Administrations
	Quantitative		Comment	Quantitative	Comment	Quantitative	Comment
Costs on research programme level
Administrative		recurrent				4%	Average percentage of total annual Programme budget
Support Actions		recurrent				N/A
Research projects		recurrent				96%	Average % of top-up payments from allocated budget.
Benefits
Direct benefits
Indirect benefits
Better quality of life
Strengthening the industrial base in Europe
Increase efficiency and sustainability of support and care systems

ANNEX V. STAKEHOLDERS CONSULTATION - SYNOPSIS REPORT

Overview of consultation activities

An important source for the information collected on the Eurostars-2 programme has been the conduction of stakeholder interviews. The interviews followed a semi-structured approach and were conducted via videoconferences. The interviews with representatives of the European Commission, the ESE and the National Funding Bodies allowed the project team to learn more about the administrative functioning of the Eurostars-2. By speaking with beneficiaries of the programme such as SMEs, experiences on the actual outputs and results of the programme were collected. Overall, 15 interviews have been conducted with beneficiaries of different cut-offs in the Eurostars-2 programme coming from different participating states, contact persons of National funding bodies and the ESE secretariat. An overview of the performed interviews and the interview guide is attached in the following table.

Stakeholders consulted

The following table provides an overview of the stakeholders consulted.

	Type of stakeholder	Role in the EUrostars-2 programme	Country	Date of the interview
1	EU official	DG RTD	Belgium	May, 18 2022
2	EUREKA secretariat	EUREKA	Belgium	June, 13 2022
3	Lead partner	National funding body	Germany	April, 29 2022
4	National investment bank	National funding body	France	April, 27 2022
5	Innovation agency	National funding body	Netherlands	May, 10 2022
6	Innovation agency	National funding body	Sweden	May, 23 2022
7	National ministry	National funding body	Italy	April, 27 2022
8	Innovation agency	National funding body	Ireland	April, 26 2022
9	Innovation agency	National funding body	Croatia	April, 27 2022
10	SME	Beneficiary	Denmark	May, 9 2022
11	SME	Beneficiary	Denmark	June, 10 2022
12	SME	Beneficiary	Netherlands	May, 13 2022
13	SME	Beneficiary	Sweden	May, 9 2022
14	SME	Beneficiary	Spain	May, 17 2022
15	SME	Beneficiary	Austria	June, 1 2022

The call for evidence was open for feedback between 23 September 2022 and 21 October 2022. Feedback received is extensive, largely positive and in confirmation of the key findings of the study. The feedback did therefore not lead to extra research for the study, nor to changes in key findings. The feedback was submitted from 62 entities from 9 different countries and represented both academic spinoffs as well as startups and regular companies. Mostly mentioned were the programme’s facilitation for engaging into international partnerships, financing of R&D that would have not been done otherwise and development of new products and services to the market. Overall, there was a high level of satisfaction with the management of the programme and the relatively easiness of participating. It is worth mentioning that in several comments it was pointed out that the ES-2 program. was the first engagement in international cooperation and funding. Finally, it was mentioned that the international partnership and cooperation developed in the ES-2 programme tended to outlive the project duration and became of a more permanent nature.

(1)

Decision No 553/2014/EU of the European Parliament and of the Council of 15 May 2014 on the participation of the Union in a Research and Development Programme jointly undertaken by several Member States aimed at supporting research and development performing small and medium-sized enterprises Text with EEA relevance (europa.eu) .

(2)

Although there are no further calls under the programme, projects supported by the programme and related top-up payments will continue until 2025.

(3)

European Commission, Directorate-General for Research and Innovation, Shaton, M., Pando, E., Vicini, I. et al., Interim evaluation of the Eurostars-2 Joint Programme, Publications Office, 2017, https://data.europa.eu/doi/10.2777/357102

(4)

European Commission, Directorate-General for Research and Innovation, Eurostars-2 final evaluation – Evaluation study of the European Framework Programmes for Research and Innovation for an innovative Europe, Publications Office of the European Union, 2023, https://data.europa.eu/doi/10.2777/333838

(5)

9 participating countries concentrate more than half of all participating R&D SMEs: Germany, The Netherlands, United Kingdom, France, Spain, Denmark, Switzerland, Sweden and Norway.

(6)

COM(2015)479 - Final evaluation of the Eurostars Joint Programme (2008-2013)

(7)

Home - Eureka (eurekanetwork.org)

(8)

https://single-market-economy.ec.europa.eu/smes/cosme_en

(9)

With the rest of the funding going to other stakeholders, such as research and technology organisations.