EUROPEAN COMMISSION

Brussels, 22.3.2022

SWD(2022) 58 final

COMMISSION STAFF WORKING DOCUMENT

EVALUATION

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS

assessing the implementation and achievements of the 2014-2020 rights, equality and citizenship programme

{COM(2022) 118 final} - {SWD(2022) 59 final}

       Table of contents

1.Introduction

Purpose and scope    

2.Background to the intervention

Description of the intervention and its objectives    

Baseline and points of comparison    

3.Implementation / state of Play

Description of the current situation    

Progress in indicators    

4.Method

Short description of methodology    

Limitations and robustness of findings    

5.Analysis and answers to the evaluation questions

6.Conclusions

Annex 1: Procedural information

Annex 2: Stakeholder consultation

Annex 3: Methods and analytical models

Annex 4: Progress on indicators

Annex 5: Results of the case studies

Glossary

1.Introduction

Purpose and scope

This staff working document accompanies the report from the European Commission to the European Parliament and the Council, in line with the obligation specified in Article 13.2(c) of the legal base of the rights, equality and citizenship programme (hereinafter, 'REC programme') 1 .

The Regulation requires the European Commission to provide the European Parliament and the Council with an ex post evaluation report for the programme by 31 December 2021. The report must assess the long-term impact and the sustainability of the programme’s effects, in order to inform a decision on a subsequent programme. In 2021, however, a considerable number of projects (more than 40%) remained ongoing due to their multiannual life-span. This situation was also affected by the COVID pandemic, which resulted in the duration of several projects being extended. Therefore, it may only be possible to comprehensively evaluate the programme’s long-term results or impacts at a later stage.

For these reasons, the ex post evaluation will be carried out in two parts. This report represents the first part. It is based on the data currently available, provides an overview of how the funding is distributed and assesses the programme’s achievements to date. While the new citizens, equality, rights and values programme for 2021-2027 (hereinafter ‘CERV programme’) has already been adopted 2 , based on, among other things, the results of the interim evaluation of the 2014-2020 REC programme, this first part of the ex post evaluation of the previous programme will be of added value in shedding light on potential areas for improvement as regards implementing the new CERV programme.

The second part of the evaluation will be carried out at a later stage once all final data are available and in conjunction with the interim evaluation of the succeeding CERV programme. This second part will assess the long-term impacts and the sustainability of the programme’s effects. It will provide recommendations for the new multiannual financial framework after 2027, as appropriate.

The document provides an overview of the programme’s objectives and the needs addressed, the intervention logic, and the programme’s implementation, providing a backdrop to the evaluative work. It then provides the replies to the evaluation questions on the effectiveness, efficiency, coherence, EU added value, equity and scope for simplifying the programme. Finally, it also explains the conclusions of the analysis of eight case studies, in which a number of specific questions have been studied in detail.

Although the analysis covered the call for proposals and procurement activities financed through all annual work programmes from 2014 to 2020, not all grant agreements and contracts had already been signed by the time the evaluation began. Therefore, a cut-off date of 31 December 2020 was set, so to define the scope of this evaluation. The second part of the ex post evaluation will complete the overview with the contracts and grant agreements signed after the cut-off date.

In terms of geographical scope, the evaluation covers all the participating countries, namely all EU Member States and, when relevant, Liechtenstein, Iceland, and Serbia.

2.Background to the intervention

Description of the intervention and its objectives

The REC programme aimed at further developing a Europe of rights and equality in line with the Europe 2020 strategy. It promoted the key values upon which the European Union is founded and helped further develop of an area where equality and the rights of people, as enshrined in the Treaty of the European Union, the EU Charter of Fundamental Rights and international human rights conventions, are promoted, protected and implemented effectively. The EU’s overall financial contribution the programme over its duration (2014 to 2020) was EUR 439 473 000.

When the REC programme’s focus and priorities were being determined, the following key problems and needs were identified:

·limited effectiveness in the implementation of actions against discrimination on the grounds of sex, racial or ethnic origin, religion or belief, disability, age or sexual orientation, and more Roma integration and gender equality;

·insufficient promotion and protection of the rights enshrined in the EU Charter of Fundamental Rights, including those of the child and persons with disabilities, and in the UN Convention on the Rights of Persons with Disabilities;

·too high a level of gender-based violence and too little support provided to victims of violence;

·insufficient protection of EU citizens’ personal data;

·limited knowledge and implementation of the rights deriving from EU citizenship and EU law.

Specific objectives

The REC programme aimed to tackle the problems explored above. It attempted this challenging task by providing funding to help achieve the overall strategic objective of developing a Europe of freedom, security, and justice in conjunction with the Europe 2020 strategy. In particular, it contributed to i) the EU justice agenda for 2020 3 ; ii) the 2016-2020 strategic engagement for gender equality 4 ; iii) the European disability strategy 5 ; iv) the list of actions by the Commission to advance LGBTI equality 6 ; and v) the EU framework for national Roma integration strategies up to 2020 7 . It also strengthened the implementation of EU law, such as the General Data Protection Regulation, the Racial Equality Directive, Equal Treatment Directives, the Work-life Balance Directive and Victims’ Rights Directives.

The REC programme focused on the following nine specific objectives addressing fundamental rights.

·Specific objective 1 – ‘RDIS’: Promote the effective implementation of the principle of non-discrimination on grounds of sex, racial or ethnic origin, religion or belief, disability, age or sexual orientation, and to respect the principle of non-discrimination on the grounds provided for in Article 21 of the Charter

The programme supported projects aiming to prevent and combat discrimination on the grounds of sex, racial or ethnic origin, religion or belief, disability, age or sexual orientation. In particular, promoting Roma integration is a key aspect of this specific objective. In this regard, the REC programme supported the national Roma platforms and worked closely with EU Member States and their national Roma contact points, and with civil society organisations active in the area of non-discrimination and Roma integration. The programme also financed the activities of the European network of equality bodies. This network’s core task is strengthening the cooperation and exchange of information on topics related to non-discrimination between the national equality bodies designated by the Member States.

·Specific objective 2 – ‘RRAC’: Prevent and combat racism, xenophobia, homophobia and other forms of intolerance

Projects financed in this area helped improve the implementation of existing EU legislation in the Member States and assist the victims of hate crime and hate speech. In particular, the activities funded supported the protection of minorities against a surge of populism, extremism and intolerance.

·Specific objective 3 – ‘RDIB’: Promote and protect the rights of persons with disabilities

The aim of this specific objective was to increase awareness and improve implementation of the rights of people with disabilities leading to a reduction of barriers, their full participation in society and enjoyment of their rights. This was mostly done through operating grants supporting the activities of European level networks advocating for the rights of people with disabilities and through procurement activities focusing on data collection, training and awareness-raising activities. Under the 2014 annual work programme, the European Disability Card 8 was launched with support from the programme to facilitate cross-border travelling for persons with disabilities.

·Specific objective 4 – ‘RGEN’: Promote equality between women and men and to advance gender mainstreaming

The programme promoted projects to support participating countries and relevant stakeholders in improving gender balance in all spheres of life, especially in economic decision-making, and in promoting equal economic independence for women and men. Funding under this objective was also focused on reducing the gender gaps in pay, earnings and pensions.

·Specific objective 5 – ‘RDAP’: prevent and combat all forms of violence against children, young people and women, as well as violence against other groups at risk, in particular groups at risk of violence in close relationships, and to protect victims of such violence (DAPHNE)

The programme primarily financed projects which helped to protect and support victims of violence, such as women and children, as well as activities to raise awareness and prevent violence. This involved engaging with civil society organisations working on the ground. It also supported the treatment of perpetrators of violence. Strengthening child protection systems is one of the Commission's priorities and the programme supported the rights of migrant children by funding international organisations working directly for and with these children.

·Specific objective 6 – ‘RCHI’: Promote and protect the rights of the child

The rights of the child are promoted through projects focusing on child-friendly justice and on protecting vulnerable children (such as those leaving foster care systems and those in conflict with the law), by supporting, in particular, training activities of civil servants, civil society organisations and lawyers dealing with vulnerable children.

·Specific objective 7 – ‘RDAT’: contribute to ensuring the highest level of protection of privacy and personal data 

The programme is the main source of EU funding for data protection. The activities carried out within this specific objective were strongly linked with the data protection reform 9 , adopted in 2016. Under this objective, the programme aimed to finance activities to support Member States, specifically in transposing and implementing the new EU data protection legislation and in training national data protection authorities and data protection officers.

·Specific objective 8 – ‘RCIT’: promote and enhance the exercise of rights deriving from citizenship of the Union

The programme financed projects, mostly awareness-raising activities, aimed at including EU citizens in the civic and political life of the EU. It supported projects aimed at helping citizens to become more aware of their rights deriving from EU citizenship. Information campaigns on EU citizenship rights were financed under this specific objective, with a particular focus on their electoral rights in view of the 2019 European elections.

·Specific objective 9 – ‘RCON’: enable individuals in their capacity as consumers or entrepreneurs in the internal market to enforce their rights deriving from Union law, having regard to the projects funded under the consumer programme

The programme also financed activities which support individuals in their capacity as consumers or entrepreneurs in the internal market in order to enforce their rights deriving from EU law. EU funds in this area aim to help increase the knowledge and awareness of consumers' rights, especially in the digital market.

Funding delivery mechanisms

The REC programme has three main funding mechanisms: action grants, operating grants and procurement actions.

·Action grants are addressed mainly to civil society organisations and Member States’ authorities. 

·Operating grants (‘support to networks’) fund mainly European networks active in: i) non-discrimination; ii) preventing racism, xenophobia, homophobia or other forms of intolerance; iii) disabilities; iv) equality between women and men; v) preventing and combating all forms of violence; and vi) promoting the rights of the child.

·Procurement actions (‘Commission initiatives’) mostly fund conferences, seminars, studies, surveys and awareness-raising activities.

Expected outputs, results and impact

The implementation of these activities was meant, in the short term, to lead to the following specific outputs: (i) training and capacity-building activities, conferences and seminars, mutual learning and exchange of good practices, which can also be measured in terms of the respective number of participants; (ii) tools being developed such as training kits/material/curricula, brochures, publications, information and dissemination material; (iii) analytical studies; (iv) databases; (v) policy recommendations.

These outputs were expected to lead in the longer term to results, which can be linked back to the specific objectives identified above, namely:

·the design, development and monitoring of policies aimed at tackling discrimination and promoting equality; and the legislation on non-discrimination being implemented more effectively, including an increased number of national Roma platforms which are actively engaged in activities of the national Roma contact points;

·national and local authorities as well as civil society organisations and communities having increased capacity to address issues related to racism, xenophobia and other forms of intolerance, including hate speech over the internet;

·increased awareness and better implementation of the rights of people with disabilities leading to barriers being reduced and these people being able to fully participate in society and enjoy their rights;

·increased awareness of gender equality gaps and their underlying mechanisms, and related policies towards equal economic independence of women and men, also in regard to tackling the pensions gap;

·improved support being provided to children, especially the most vulnerable, when they are preparing to leave care, and better protection being provided including support for the rights of all children in migration on EU territory;

·increased awareness and reporting of violence, with appropriate mechanisms of response in place; and improved protection and support standards for victims of gender-based violence and violence against children, including strengthened cooperation between European/national/regional/local authorities;

·increased awareness of data protection and increased capacity for national experts and civil society organisations;

·increased capacity for national, regional and local experts to enable mobile EU citizens to participate in their host Member States, including the exercise of their democratic rights;

·increased knowledge and awareness of consumers’ rights, especially in the digital internal market.

These results are instrumental to achieving the specific objectives and were expected to promote, protect, and implement effectively the long-term equality and the rights of people.

The intervention logic of the REC programme displays the causal links between the different levels.

Figure 1. Intervention logic of the REC programme

Source: Interim evaluation of the REC programme

Stakeholders and target groups

The stakeholders eligible for support were public or private organisations (usually non-profit), duly established in one or more countries participating in the programme, or international organisations (such as the UN High Commissioner for Refugees, UNICEF and the International Organization for Migration).

In terms of target groups, intended as the groups that can benefit directly or indirectly from the programme, these were mainly public authorities and civil society organisations. Target groups were also the general population of participating countries, since the programme’s objectives and initiatives aimed at promoting, protecting and implementing effectively the equality and the rights of people, in particular those subjected to discrimination, intolerance or violence, including migrants and minorities.

Baseline and points of comparison

The 2018 interim evaluation used the 2011 impact assessment 10  and the 2015 ex post evaluation of the three predecessor programmes (Daphne III, fundamental rights and citizenship and progress) 11  as a baseline for measuring the achievements of the 2014-2020 REC programme. These documents will also remain the baseline for the ex post evaluation of this programme. 

The impact assessment outlined both a ‘baseline scenario’ against which the policy options for the REC programme were benchmarked, and a series of expected outcomes connected to the programme’s development. The baseline analysis showed that, if no changes had been made between the 2007-2013 and 2014-2020 programming periods, the predecessor programmes would have continued to be successfully implemented, but with reduced potential. The following issues were reported as critical regarding continuation of the predecessor programmes:

·the lack of flexibility in the funding instruments would have represented a difficulty given the pace of change and reform in the policy areas;

·fragmentation of funding, especially in an area where funding would be needed to support cross-cutting issues, could have reduced the capacity of the programmes to deliver results;

·the high number of different funding instruments, in the context of rising numbers of applicants, could have increased the administrative burden, potentially resulting in delays in procedures for the beneficiaries.

Given these difficulties, the 2011 impact assessment suggested the option of consolidating the three programmes in the new comprehensive REC programme, outlining the following expected outcomes.

·For relevance and scope, a reduction in fragmentation and overlaps was expected, as policies with close links to each other would be addressed under the same programme. In addition, the broader scope of the REC programme was expected to bring more flexibility in supporting policies in an ever-evolving policy context.

·For effectiveness, it was expected that the new comprehensive REC programme would foster concentration of funds and development of stronger partnerships. The applicants would coordinate their efforts in wider partnerships with more funding available. A more balanced geographical spread among beneficiaries was also expected thanks to larger partnerships.

·For efficiency, the REC programme was expected to improve compared with preceding funding instruments, chiefly by simplifying the procedures for applicants and beneficiaries. In particular, merging several predecessor programmes under the umbrella of the REC programme was expected to end ‘programme-shopping’ among applicants, who previously had to identify the most appropriate calls among six different funding instruments with diverse rules and procedures. Increases in efficiency were also expected thanks to the number of calls being reduced, which was expected to enable them to become more focused.

·For complementarity, it was expected that merging several predecessor programmes into one would ensure a coordinated approach in identifying annual priorities, to be applied also vis-à-vis programmes in other EU policy areas related to justice and rights. Coordination at the level of annual programming would ensure that there is no duplication of funding and that funds are used in a complementary way, to the best interest of the public.

·In terms of administrative burden, considerable improvements were expected, in terms of reducing the number of annual work programmes and the number of calls. Harmonisation of the management procedures was also expected to reduce the administrative burden faced by applicants and beneficiaries.

All the 2015 ex post evaluation reports on the previous funding programmes confirmed their overall effectiveness and highlighted that their specific objectives and priorities were largely specific, attainable and realistic, but not always measurable. The problems identified in 2011 were also confirmed for the three programmes.

3.Implementation / state of Play

Description of the current situation

Programme management

The REC programme was implemented via direct centralised management by the European Commission. This ensured a close relationship between the programme management and EU policy-making. This type of management allowed the Commission to tailor funded activities to policy priorities and needs and to directly target the relevant groups of stakeholders. It also enabled close contact with the programmes' beneficiaries and better knowledge of the needs on the ground.

State of play

The purpose of this section is to present the state of play on the programme’s implementation based on the information available at the time of writing. As explained in the introduction, a considerable number of projects are still ongoing, and therefore, the ex post evaluation will be carried out in two parts. The first part was being prepared in 2021, based on the available data at the time, and it will assess the REC programme’s achievements so far. In 2024, the second part will complete the evaluation of all the projects and will assess the long-term impact and sustainability of the programme’s effects.

Annual work programmes

Annual work programmes over the programming period have varied their focus on specific objectives depending on the wider social, political and economic context. This ensured the programme continued to be relevant so as to meet the challenges in the year that followed, such as the migrant crisis, rising hate crime and hate speech, European elections or data protection reform.

The first annual work programme of the programming period covered 2014 and, among other priorities, it gave special focus to protecting children through the call to combat bullying of minors in public institutions. Furthermore, following the Council’s Recommendation 2013/C 378/01 on effective Roma integration, a call was launched to promote Roma integration, with continuous focus on this throughout recent years. On combating racism and intolerance, activities to improve monitoring and criminal proceedings were highlighted. Following the proposal for an EU disability card to make travel for people with disabilities easier, funds were provided to help Member States introduce the card. Calls to fund projects that involve training public officials were launched in the context of the tail end of the European strategy for equality between women and men.

In 2015, specific attention was given to preventing violence against children. The calls focused on preventing sexual violence against girls and young women in the workplace, schools, and universities through new technologies. Under the same objective, a call was also launched to help fund national missing children’s hotlines. In addition, calls for proposals were launched to set up and strengthen a national consultation process by national Roma contact points following the Council’s Recommendation on effective Roma integration mentioned above. A conference in the first half of the year under the Latvian Presidency was funded to help implement the UN Convention on the Rights of Persons with Disabilities, in order to help achieve the specific objective of promoting and protecting the rights of people with disabilities.

In 2016, the programme focused in particular on the ongoing migration crisis. This involved providing direct grants to the International Organization for Migration, UNICEF, UNHCR and Save the Children to finance capacity-building for professionals. Among other priorities, the annual work programme aimed at raising awareness and training professionals in light of the Data Protection Reform taking effect in 2018. Following the first Annual Fundamental Rights Colloquium, two calls for proposals were launched to address hate crime, hate speech and radicalisation, with a focus on antisemitism and islamophobia.

The 2017 annual work programme supported the national data protection authorities in their efforts to raise awareness among individuals and businesses, in particular SMEs, about the entry into application of the General Data Protection Regulation (GDPR) in 2018. It also provided for an EU communication campaign, carried out in 2017-2018, to raise overall awareness of the GDPR. In 2017, there was also increasing concern over hate crime and hate speech with particular attention being paid to antisemitism and anti-Muslim hatred and discrimination. That same year, there was a call for action grants aimed at supporting the Commission’s LGBTI list of actions as well as diversity promotion and Roma inclusivity. Two calls specifically sought to address gender-based violence and violence against children. With European elections taking place in 2019, there were also calls for proposals to strengthen democratic participation and electoral rights.

In 2018, when the new data protection legislative framework entered in force, the funding focused on helping Member States to successfully implement it. With the European elections due the following year, funding was allocated to projects promoting democratic participation and awareness of electoral rights. In 2018, the following areas were also specifically targeted: protecting minorities and countering extremism, populism and intolerance. As a result, child protection funding was aimed specifically at migrant children. Countering hate speech online was also targeted. Equal opportunities for women and men were promoted in public roles (politics, corporate sector, or civil society), in line with the 'New Start to Support Work-Life Balance for Parents and Carers' initiative. Particular attention was paid to overseeing the national Roma integration strategies through a restricted call for a national Roma platform.

The 2019 annual work programme highlighted the need to promote the Commission’s priority of ‘an area of Justice and Fundamental Rights based on mutual trust’. Data policy was promoted through the framing of a ‘Connected Digital Single Market’ and gender equality was promoted in line with boosting jobs and investment. Support to the Roma community was also highlighted, in particular through building national platforms. 2019 was the first year in which the work programme offered a monetary prize for the winners of the Access City Award, which rewards efforts to increase urban accessibility to people with disabilities. The application of the new General Data Protection Regulation since 2018 was accompanied by a new wave of grants to national data protection authorities to help them reach out to individuals and businesses, in particular SMEs. Due to the 2019 European Parliament elections, special attention was paid to ensuring the successful inclusion and political participation of all EU citizens. This involved funding a study on the elections and data gathering for the next EU citizenship report.

In 2020, particular attention was paid to ensuring children were included in decision-making processes as well as on the ‘worrying increase of intolerance in our societies’. On the latter, the work programme sought to develop a common methodology on collecting data on hate crimes with continued special focus on antisemitism and anti-Muslim hatred and discrimination. Due to the Digital Content Directive being adopted in 2019, the 2020 work programme focused on funding activities which helped Member States implement the required changes. Focus was also concentrated on promoting the rights of EU citizens. This included a new wave of grants to support the activities of the national data protection authorities in raising awareness on the GDPR.

The table below shows the budget amount per annual work programme amounting to EUR 426.8 m for the entire programming period. The yearly planned amounts increased every year from EUR 54.2 m in 2014 to EUR 67.9 m in 2020.

Table 1. Budget amount planned by year

Source: Annual work programmes 2014-2020

Calls for proposals

A total of 81 calls for proposals 12 were planned in the 2014-2020 annual work programmes. The yearly number of calls peaked in the initial years of the period before dropping after 2017 as a result of being merged in the consolidation effort to increase the efficiency of the programme’s management. Most of these calls covered the specific objective RDAP - ‘prevent violence (Daphne)’, with 18 calls. As the specific objective RCON - ‘Consumers' rights’ was funded through procurement only, there were no calls for proposals related to it.

Table 2. Calls for proposals by specific objective and year

Source: Annual work programmes 2014-2020

The biggest share of the programme (25%) was planned for the specific objective RDAP - ‘prevent violence (Daphne)’. The specific objective with the second largest budget allocation was that dedicated to promoting non-discrimination (22%).

Table 3. Specific objectives and the amounts planned

Source: annual work programmes 2014-2020

For the specific objectives on ‘rights of the child’, ‘prevent violence (Daphne)’, ‘non-discrimination’, ‘racism and other forms of intolerance’ and ‘gender equality’, the biggest share of the funding was earmarked for action grants. For the remaining specific objectives (data protection, rights of persons with disabilities, EU citizenship, consumer rights) the activities were implemented mainly through procurement. Prizes were introduced in the final 2 years of the REC programme and were used for the specific objective on ‘rights of persons with disabilities’.

Action grants and operating grants

The table below shows the number of action grants (AGs) and operating grants (OGs) awarded by specific objective and year. Overall, 942 AGs and OGs were awarded throughout the programme’s duration.

Table 4. Number of AGs and OGs awarded by specific objective and by year

Source: Data for 2014-2015 are retrieved from the interim evaluation of the REC programme. Data for 2016-2020 are retrieved from the project database (AG and OG excluding direct grants).

Types of project activities

Figure 2 provides an overview of the different types of activities implemented by funded projects. Up to three main types of activities could be flagged for each project 2016-2020. Most of activities funded by REC projects between 2016 and 2020 can be classified as ‘training activities’ (35%).

Figure 2. Types of activities funded by year (2016-2020)

Note: Projects can be assigned to more than one type of activity.

Source: Project documents by DG JUST (AG and OG).

Procurement

Procurement was an important funding mechanism for a number of specific objectives and amounted to 345 contracts over the entire period. The specific objective which most heavily relied on this funding option was ‘Racism and other forms of intolerance’ with 112 procurement contracts over the period.

In total, approximately EUR 81 million was allocated to procurement contracts and services. Contracts under the specific objective on ‘non-discrimination’ accounted for 38% of the total amount.

Table 5. Number of procurement contracts awarded by year and specific objective

Source: Procurement contracts by DG JUST (n=345).

Type of procurement activities

The graph below details the distribution of procurement activities by type. The most common activity was categorised as ‘Analytical and monitoring’. Many contracts incorporated more than one of these components; for example, many would have had both ‘Analytical and monitoring’ and ‘Awareness-raising’ activities.

Figure 3. Number of procurement contract signed by type (2014-2020)

Source: Procurement data by DG JUST (n=305). For 40 procurement contracts managed by DG EMPL, no information is available on activities and are therefore not included.

Applications by Member States

The country with the most grant applications was Italy with 975 applications (coordinator) over the programming period. This was by far the largest number of applications from any country, with Spain in second place (353 applications).

Figure 4. Number of applications by participating states (2014-2020)

Source: Interim evaluation (2014-2015) and project database (2016-2020)

However, once the population size of participating countries is taken into account (by looking at applications per 100 000 people), some smaller countries do actually have a large number of applications compared to their populations. Cyprus stands out in this regard with the highest ratio of applications to population (6.31 per 100 000 people). Malta and Slovenia are second and third with 4.28 and 3.48, respectively.

Distribution of resources by type of beneficiary

Non-profit organisations accounted for the majority (57%) of beneficiaries throughout 2016-2020 (Figure 5 – OTH). In this period, the share of beneficiaries in the research sector (REC) fell from 12% in 2016 to 9% in 2020. The share of higher or secondary education sector (HES) also fell from 14% in 2016 to 8% in 2020. This was mirrored by a slight increase of the share of private for-profit companies (PRC). Overall, the annual changes remained insignificant and did not affect the general trend.

Figure 5 - Share of resources by type of beneficiary (2016-2020)

Source: monitoring data by DG JUST

Target groups

In 2014-2015 the most mentioned target group was young people (including children), followed by women and students. If we look at the distribution of target groups in 2016-2020, their granularity increases compared to 2014-2015. In 2016-2020 overall, professionals are the most mentioned target group, followed by women and children. Citizens, young people and communities appear also to be important target groups for the REC programme in those same years.

Source: Interim evaluation and text mining on project documents.

Note: Two specific objectives, RCIT and RDAT, were excluded from the representation as their representation is too disaggregated.

Progress in indicators

Legal base indicators

Article 14 of the REC Regulation lists indicators which will serve as a basis for monitoring and evaluating. Assessing the extent to which the legal basis indicators have been met is hindered by the difficulties in calculating them and by the lack of a baseline or benchmark against which to measure progress. In many cases, the conclusion on these indicators has been reached indirectly and based on partial information. The informative value of these figures, therefore, should be taken with caution.

(a)The number and percentage of persons in a target group reached by the awareness-raising activities funded by the programme

This indicator provides information on the number of people reached by awareness-raising activities funded by the programme. It should also provide information about the coverage of such activities with respect to the reference population. Awareness-raising activities are a key output of the REC programme. They play a key role in empowering target groups as they help them gain a better knowledge of their own rights and provide them with basic tools to report and fight any violation of such rights. Awareness-raising activities are also key to promoting awareness of fundamental EU values and changes of attitudes in society, particularly when directed at tackling stereotypes.

Analysis of 211 project documents (out of more than 900 projects funded by the programme) shows that, between 2016 and 2020, at least 80 million people were reached by information and dissemination campaigns or took part in awareness-raising events. This figure includes various target groups, of which the most numerous are the general public (ca. 46 million); followed by EU citizens (ca. 19 million); women (ca. 4 million); and the scientific community, civil society, policymakers, media (ca. 3 million); individuals belonging to the LGBTI community (ca. 1 million) and who represent businesses/companies (ca. 1 million).

(b)The number of stakeholders participating in, inter alia, training activities, exchanges, study visits, workshops and seminars funded by the programme

This indicator provides an overview of how the programme helped increase the capacities of stakeholders, including private and public authorities and professionals working with the target groups and advancing their rights. As such, it provides important information about the programme’s capacity to help improve the quality of support services aimed at target groups.

The analysis of 211 projects shows that between 2016 and 2020 13 approximately 85 000 individuals participated in exchange and mutual learning events. In addition, the documents report that approximately 1.5 million individuals took part in training activities.

(c)The improvement in the level of knowledge of Union law and policies and, where applicable, of rights, values and principles underpinning the Union, in the groups participating in activities funded by the programme compared to with the entire target group

This indicator assesses how the REC programme has helped increase the level of knowledge of EU law and policies, another key goal of the REC programme.

Based on the evidence at hand, when looking at the key types of results reported by the projects, stakeholders and end beneficiaries increasing their knowledge of EU law and policies is by far the most reported result, mentioned by 78% of projects for which a technical report was available. This goes in line with the programme’s intervention logic: it intends to trigger a behavioural change in society, for which awareness-raising activities are crucial.

(d)The number of cases, activities, and outputs of cross-border cooperation

Cross-border cooperation is a distinctive feature of the REC programme, with most calls and projects being directed at transnational consortia and actions. Only a limited number of calls did not require transnational partnerships.

A total of 645 cross-border tools and mechanisms were sustained after the projects ended, which include formal advisory groups, memorandums of cooperation/cooperation agreements, and networks. The results obtained in this area and the strengths reported in the REC programme are similar to those obtained from predecessor programmes.

(e)Participants' assessment of the activities in which they participated and of their (expected) sustainability

It is not possible to calculate this indicator due to the lack of a systematic reporting by projects on this theme.

(f)The geographical coverage of the activities funded by the programme, (g) the number of applications and grants related to each specific objective, and (h) the level of funding requested by applicants and granted in relation to each specific objective

During its implementation period, the REC programme received 4903 applications corresponding to 5.4 applications per grant awarded. Overall, the average success rate of receiving a grant in the REC programme was 17%. Comparing relevant specific objectives with their predecessors, the success rate of ‘prevent violence’ (Daphne) was 14.3%, lower than the success rate of the Daphne III programme (16%). On the other hand, the fundamental rights and citizenship programme had a much higher number of applications per project granted. Consequently, its success rate ranged between 0.4% and 2% and was much lower than the success range of relevant specific objectives of the REC programme: rights of the child, racism and other forms of intolerance, EU citizenship, and data protection.

The specific objectives ‘rights of persons with disabilities’ and ‘data protection’ recorded the highest success rates (40% and 37%, respectively). ‘Gender equality’ followed with 28%.

Applications were not equally distributed across the Member States. The balance of the REC programme’s resources and activities were skewed towards a few countries that made many applications. Nevertheless, the operating grants, which mainly fund EU-wide networks, guarantee fair geographic coverage at EU level.

Indicators for the REC programme’s general objective and the specific objectives

Along with the indicators of the legal base, a number of additional indicators were established, which aimed to measure the progress of the programme from the baseline situation. Annex 4 presents the 2020 target, the baseline, the milestone and the actual results for each indicator, where available.

Although the data is only partially available at this point in time, achieving most of the indicators’ 2020 targets seems challenging as regards the programme’s general objective 14 , given that Europe is exiting a long financial recession, exacerbated in 2020 by the COVID-19 crisis. Only the target for the indicator on ‘the percentage of Europeans who consider themselves as “well or “very well” informed of the rights they enjoy as citizens of the Union’ is presumed to be achieved, well before 2020.

Several indicators have been used to evaluate the programme’s performance at the level of individual specific objectives. Indicators are a valuable source of information, despite the bias and the external factors influencing the results. The data available so far indicate that the targets for several indicators have been achieved or exceeded. However, over the period of the programme it was proven difficult to collect data for some indicators. These shortcomings were addressed when the performance monitoring framework for the new CERV programme was designed.

4.Method

Short description of methodology

During the evaluation process for this first part of the ex post evaluation of the REC programme, the Commission was supported by a consortium of contractors (Ernst & Young Consulting BV/SRL and Fondazione Giacomo Brodolini Foundation S.r.l. SB).

This assessment is based on a mixed methods approach. To this end, the evaluation integrates and compares quantitative and qualitative approaches, data collection, data analysis and interpretation. Its purpose is to strengthen the reliability of data, and validity of the findings and recommendations. It does this by triangulating multiple sources of information, to broaden and deepen the understanding of the processes through which the programme’s outcomes are achieved, and how these are affected by the context within which the programme is implemented.

Ultimately it provides answers to the evaluation questions on effectiveness, efficiency, relevance, coherence, EU added value, equity and simplification as well as provides results on the eight case studies.

The evaluation relied on both desk 15 and field research.

The field research aimed at collecting relevant data and input from project beneficiaries and stakeholders, filling gaps and validating information collected during the desk research.

The field research focused on different categories of stakeholders, namely:

·Beneficiaries of the support provided under the programme, including i) national, regional and local authorities in all participating countries, ii) international organisations, iii) EU-level umbrella organisations, iv) civil society organisations, v) academic and research institutions, vi) networks, vii) representatives of target groups and other organisations; in some instances, potential beneficiaries were also consulted;

·Commission staff (such as programme managers and policy officers);

·The REC programme committee members representing participating countries.

Field research relied on the following techniques:

·semi-structured interviews (in total, 53 interviews were carried out);

·focus groups (4 focus groups were organised);

·E-Survey to programme committee members (17 responses in total).Additional survey to beneficiaries to feed the case studies.

Table 6. Distribution of stakeholders consulted

This evaluation covered the call for proposals and procurement activities financed in 2014-2020 with a cut-off date of 31 December 2020 for signature of the grant agreements and the contracts. The second part of the ex post evaluation will complete the overview with the contracts and grant agreements signed after the cut-off date.

The collected data were used to carry out a series of quantitative and qualitative analyses. On the quantitative side, the analysis helped to provide perspective both at the programme level, assessing, for example, how and where the programme deployed the available financial resources, and at the project level, giving a granular view of quantitative indicators provided by individual projects. Concerning the qualitative analysis, automated text mining and mixed techniques were deployed to help sift through the large amounts of data on individual projects. This was done because of the need to give structure and find meaning within a large and mostly unstructured dataset (namely project applications and final reports).

Following the collection, extraction, and systematisation of the project-related data and documents, additional data collection activities took place to feed the response to the evaluation questions. This included field research on a sample of projects.

A total of 36 projects, beneficiaries both action (28) and operating grants (8), plus six procurement contracts, were sampled in order to carry out a qualitative analysis based on comments and insights from key stakeholders concerning the REC programme’s implementation. The projects were sampled taking into account geographical distribution, specific objective, and financial amount.

Table 7. Projects sampled by specific objective and form of support

Case studies

The main purpose of the case studies was to provide solid, triangulated evidence, in order to support preparation of replies to evaluation questions as well as suggestions for improvements, with a focus on specific issues mentioned in the tender specifications. For each case study a customised methodological approach was developed. Please see Annex 3 for an overview of the focus of the eight case studies and Annex 5 for the results.

Limitations and robustness of findings

The evaluation’s findings and conclusions are deemed to be credible and consistent according to the methodology. There were, however, some challenges which could not always be mitigated.

·The change in the project information base and migration to eGrants in the middle of the programming period led to differences in completeness and consistency of the data used for the monitoring of the programme. While a detailed and extensive database has existed for the projects since 2016, information on projects that ran before 2016 is fragmented. In certain cases, the indicators could be computed only for projects granted from 2016 onwards, due to unavailability of information before 2016. To overcome these hurdles, the contractors worked on linking the different data sources (for funded projects), assessing what represents the common ‘nucleus’ of information that is shared between them, and creating a unique dataset of projects.

·Weaknesses in the existing monitoring framework have been identified as the main limitation in relation to assessing the programme’s effectiveness. To overcome this problem, in many cases, the conclusion on indicators had to be reached indirectly and with additional input from interviews and project document analysis.

·There was a limited response rate to the e-surveys launched as part of the data collection activities. To address low response rates in surveys, the response time was prolonged.

·There were issues surrounding the feasibility of using readily available information. When automatic data extraction techniques were applied (to semi-structured information, e.g. in the case of monitoring information reported in the annexes to the technical reports), the format in which data were reported proved sometimes to be problematic for the extraction.

5.Analysis and answers to the evaluation questions

Effectiveness

Based on qualitative evidence collected, the REC programme’s implementation was effective and this helped to achieve its overall objectives. In particular, this success was thanks to the quality of most of the projects implemented with REC funding, enabling targeted actions to be implemented such as awareness campaigns, training activities, and publication of materials, research and relevant data.

Progress was also recorded in terms of the legal base indicators (identified by the Regulation), concerning in particular:

·the number of people that were reached by information and dissemination campaigns or that took part in awareness-raising events (at least 80 million people); 

·the number of stakeholders participating in training activities (at least 1.5 million individuals) and in exchanges, study visits, workshops and seminars (at least 85 000 individuals) funded by the programme.

Overall, the evidence gathered suggests that the REC programme has helped increase the knowledge of EU soft and hard law.

Key activities implemented match to a large extent the specific objectives. Furthermore, the activities and achievements of all projects sampled were directly relevant to the calls’ objectives. The calls’ priorities helped the beneficiary organisations to align their actions with the REC programme’s objectives. At the same time, difficulties were also highlighted in identifying projects with the most promising impact, as some organisations were adept at writing proposals and passing the selection process without necessarily being best placed to deliver on the expected results.

Overall, both desk research and fieldwork show that the projects were largely successful in implementing the planned activities. Qualitative information collected shows that almost all finalised projects have achieved the expected results.

Figure 7. Comparison between expected and achieved results by specific objective 

Source: Project documents 2016-2020.

Note: The result categories are based on the main typologies of results reported and on the main expected results of the REC programme 16 .

A complex interplay of internal and external factors supported and hindered the programme’s performance. Internal factors mainly relate to the difficulties linked with the application process, internal capacity of project partners, and the type and quality of consortium. Common external factors relate to the impact of external shocks such as the COVID-19 pandemic and the challenge of involving public entities and stakeholders in the activities planned.

Figure 8. External factors that influenced the implementation of the funded projects (sample)

Source: Project documents 2016-2020.

Figure 9. Internal factors that influenced the implementation of the funded projects (sample)

Source: Project documents 2016-2020.

The evaluation found that the success factors in REC projects include the quality of partnerships, bringing together organisations with complementary expertise, covering managerial and coordination skills with experience of working with target groups on the ground. Another element of success is the capacity of the projects to respond to the aims of the calls while also pursuing the longer-term strategies of the implementing organisations.

According to the data collected, the application process was communicated effectively to applicants. It was complex but fair, even though several factors hindering the participation of certain types of beneficiaries (mainly public institutions) were detected, such as low administrative capacity, lack of experience in applying for and implementing EU-funded projects, and difficulties in covering the resources required for co-financing.

Funding a second (continuation) project with the same beneficiary organisations (case study 1) 17 does create significant synergies in terms of ensuring sustainability, geographical spread and improving the way REC funded activities are implemented. At the same time, the programme needs to address the potential risk of closing the door to new applicants and restricting the space for new inputs and perspectives. This is particularly important since the application process may lead to the creation of barriers to participation in the REC programme as the process could be perceived as being resource intensive, competitive, and skewed by the capacity to ‘tick the boxes’.

Efficiency

Existing qualitative evidence demonstrates that the benefits of projects’ implementation outweigh the costs and, in this sense, efficiency was achieved (case study 4) 18 . However, the costs linked to the administrative burden are still considered an issue by beneficiaries, especially in relation to the application and reporting procedures. While most beneficiaries identified the administrative burden as a significant cost component, they also recognised that burdensome administrative requirements are legitimate when transparency and fairness in the use of public funding is at stake.

REC administrative costs are perceived on average to be higher compared with other non-EU alternatives or with national programmes. National and international funding instruments are considered to be less demanding when it comes to the application and reporting requirements.

In terms of benefits, the REC programme offers the possibility of working on a transnational level, sharing best practices and building relationships with consortium partners, and strengthening internal capacity and expertise. There are also intangible gains such as a better reputation and increased trustworthiness of their organisation as a result of participation in EU-funded projects.

Key aspects that contributed to the projects’ efficiency included:

·good planning and organisation, especially financial planning;

·anticipation of any over or underspending in order to take prompt corrective steps;

·the in-house experience and capacity of participating organisations;

·the support received from the Commission, including the possibility for reallocating funds.

The evaluation identified underspending in 82% of the projects sampled (case study 5 19 ). While underspending is widespread in the sample analysed, most projects interviewed reported underspending of around 10-12%, which was mainly due to the COVID-19 outbreak and the consequent halt in physical meetings and activities which resulted in savings in terms of travel and subsistence costs for physical meetings, workshops and other events.

Compared to its predecessors (case study 6 20 ), there has been an improvement in the prompt adoption/publication of calls published under the REC programme and the predictability of these. Information collected through interviews with recurring beneficiaries shows that the merging of the predecessor programmes considerably simplified the management and implementation procedures.

Relevance

The needs identified at the time the REC programme was adopted are still relevant to ensuring that equality and the rights enshrined in the EU Charter of Fundamental Rights and other legally binding instruments continue to being promoted across the EU. This is supported by the fact that the programme receives many more applications than it can fund, showing its continued relevance to the needs of its target groups. Furthermore, the REC programme throughout the years has continued to evolve in order to adapt to the emergence of new needs, such as, for example, the need to tackle the increase in hate speech.

The analysis also confirmed that the actions implemented under the REC programme are relevant to the needs of its stakeholders, such as improving the knowledge base, sharing best practices through mutual learning, capacity building and structural support for specific organisational structures. Furthermore, the REC programme was relevant in targeting the intended end beneficiaries – ultimately EU public. This has been confirmed through several strands of evaluation evidence, such as the text mining analysis, a sampling of projects and stakeholder consultations.

The analysis of the overview of the main target groups according to needs addressed is presented in the Figure below. More than one target group and need can be associated with each project. Figures below show the links between target groups and needs covered by projects.

Figure 10. Matching needs and target groups (2016-2020)

Source: Text mining on 295 projects with available documentation.

In terms of the programme’s ability to select the most relevant beneficiaries, two types of beneficiaries were identified as being particularly relevant but less likely to be involved: (i) public authorities, for whom obstacles to participation can be mainly traced to a lack of administrative capacity and dedicated human resources; (ii) small grassroots organisations, for whom the application process is very complex and the requirement to secure a co-financing is challenging (case studies 3 and 7) 21 .

Coherence

In the REC programme, the calls for proposals are guided by the specific strategic developments and the EU priorities. In this way, the REC programme acted as a link across the EU priorities, in line and consistent with other EU actions. The REC programme is aligned with several EU policies, such as the EU justice agenda for 2020, the 2016-2020 strategic engagement for gender equality, the 2010-2020 European disability strategy, the list of actions by the Commission to advance LGBTI equality, the EU framework for national Roma integration strategies up to 2020. The analysis also indicates a certain level of consistency with national policies and developments at the project level.

EU Added Value

The EU added value of the REC programme was assessed according to the following four criteria:

1)Supporting implementation of EU law and awareness about the rights deriving from it;

2)increasing mutual trust among participating countries and cross-border cooperation;

3)developing and disseminating best practices;

4)Creating standards, practical tools and solutions that address EU-wide challenges.

The programme was successful in all of these aspects, thanks to its inherent transnational nature. The evaluation identified additional aspects of the EU added value of the programme.

·It represents an important source of financing for the beneficiaries, enabling them to implement additional projects addressing the challenges the REC programme was created to tackle.

·Receiving EU funding and implementing activities under the umbrella of the REC programme had positive consequences for the projects in terms of improved reputation and increased visibility that no national input could have developed.

·The REC programme is an important driver for innovation since little national funding is available for testing innovative and transnational actions.

·The REC programme generated a scope effect, by broadening the actions implemented by the beneficiaries in a way that the national level interventions would find difficult to sustain.

Beneficiaries highlight cross-border cooperation and transnational partnerships and networks, and mutual learning among partners as key results of their participation in the programme. The transnational aspect is particularly evident for beneficiaries of operating grants due to their European remit; their activities could not have been similarly developed through national or international funding.

Equity

The REC programme’s resources and support were not distributed in a balanced manner across participating states, and the distribution of REC resources was focused on a few countries – namely Italy, Belgium, Greece and Spain, from where the applicants apply more often. However, this picture changes once the population of participating countries is taken into account, as some smaller countries have a large number of beneficiaries compared to their population (e.g. Malta and Cyprus). Furthermore, there was quite a high level of participation in Romania and Bulgaria. In addition, the countries, where the actual activities take place, are not always the same as the ‘nationality’ of the beneficiary.

Figure 11. Number of beneficiaries by country (2016-2020)

Source: 2016-2020 projects

Overall, programmed funding reaches different target groups in an equitable way, despite the concentration of REC programme resources on three specific objectives. Often the same target groups were targeted across different specific objectives. The REC programme does not fully capture the needs of people suffering from multiple disadvantages.

Significant financial resources were dedicated to promoting the cross-cutting priorities of gender equality, children’s rights and the rights of people with disabilities. In the implementation phase, the calls which focused on children’s rights, gender rights and the rights of people with disabilities were those organised under the dedicated specific objectives. While there is a good awareness of the importance of gender mainstreaming, there is a general lack of knowledge of good practice among the approved REC projects on the best approaches to address gender mainstreaming in the activities funded outside dedicated projects. Beneficiaries have difficulty in formulating a gender approach when developing their application (case study 8 22 ).

Simplification

The recently introduced system for submitting applications (the funding & tenders opportunities portal) simplified the application process and improved the collection and aggregation of monitoring data. In general, the beneficiaries positively assess the new IT tools developed and find them effective and easy to use. The possibility of directly communicating with project officers has been particularly appreciated. Some beneficiaries still believe the system is not sufficiently user-friendly, e.g. slow uploading of documents, unclear navigation and the lack of information on the number of characters allowed.

A case study carried out in the context of this evaluation on the ‘Changes in management of the programmes by DG JUST (case study 6 23 )’ concluded that the trend towards increased digitalisation is clearly viewed as a positive development, although there could still be room for further simplifying the process particularly in relation to the administrative documentation required from applicants.

6.Conclusions

The 2014-2020 REC programme was launched when the effects of the economic crisis had led to a general reduction in the amount of national resources and funding available for social and fundamental rights issues.

In this critical political and economic context, the programme has proven its EU added value and its crucial role in developing a European area of equality and rights.

The REC programme continues to be relevant to the (changing) needs of stakeholders and the EU public. This was ensured via its flexible programming nature, including the annual prioritisation of emerging policy needs and issues, and the dialogue with civil society organisations.

Overall, based on the qualitative evidence collected, the implementation of the REC programme was successful and helped achieve the overall objectives. In particular, this success was due to the quality of most of the projects implemented with REC funding, enabling the implementation of concrete actions, directly relevant to the evolving needs on the ground.

Both desk research and fieldwork show that the projects were largely successful in implementing the planned activities and almost all finalised projects have achieved the expected results.

Qualitative evidence collected by the study suggests that the benefits of the REC programme outweigh the costs.

As regards the baseline situation, as described in the 2011 impact assessment 24 and the 2015 ex post evaluation of the three predecessor programmes (Daphne III, fundamental rights, and citizenship and progress) 25 , the evaluation evidence shows that almost all difficulties identified at that time have been overcome in the REC programme, except for the geographical imbalance which still persists.

The analysis also identified a number of areas for improvement which are being addressed for the REC programme’s successor, the CERV programme, which are described below.

The weaknesses in the existing monitoring framework have been identified as the main limitation in relation to assessing the programme’s effectiveness. Therefore, an important lesson learned for the next programming period is the need to improve the monitoring system for the successor programme in order to systematically collect information on achieved outputs, results and achieved targets. Such a comprehensive monitoring system would be key to assessing the programme’s effectiveness and efficiency (and more generally its performance). The system should be proportional to the investment of the programme and not be too burdensome for beneficiaries and the Commission. It should identify a limited number of output and result indicators against which performance will be assessed and measured.

The geographical coverage and allocation of programme resources across participating states are significantly concentrated on a limited number of (larger) countries. While such concentration is partly linked to the size of recipients’ countries, it may highlight a potential problem from an equity perspective. The following lessons learned were highlighted during the evaluation:

1) To encourage participation in calls for proposals and to increase the geographical coverage of participants, the need to increase the capacity of organisations in some Member States could be addressed. This could be achieved through the Civil Dialogue Group (CDG) set up under the new CERV programme. The CDG is an open and informal forum aiming to provide information about the programme to representatives from civil society organisations, promote the exchange of experience, and provide a platform for sharing information and networking between projects beneficiaries. In this context, information sessions could be continued for work programmes and could target Member States where the level of participation is lower than expected. Capacity-building actions targeting civil society organisations in countries where the level of participation is typically weaker could also lead – in the medium to long term - to a more proactive CSO sector, willing to invest in EU-funded projects.

2) The national contact points under the new CERV programme can contribute to a more balanced participation, especially from smaller, grassroots organisations. As they work in national languages, they can play a positive role particularly in less-represented Member States to guide and support organisations interested in the programme through the application process. These contact points should receive sufficient resources to carry out these tasks.

3) To encourage a wide range of beneficiaries to participate, particularly organisations that are new to the programme, simplifying the application process and the IT tools used for applying could motivate potential beneficiaries that are reluctant to apply due to their low administrative capacity.

The tendency to implement different types of activities (training, research, awareness raising) under the same project, while not being a negative element in itself, makes projects less focused and increases the managerial burden for coordinating different tasks. In future, consideration could be given to encouraging projects in particular policy areas to focus on a limited number of key activities. The possibility of funding the continuation of projects through recurring or multiple grants could help beneficiaries to focus on a limited set of activities, as well as provide opportunities to implement more systematic and continuous actions.

Concerning the evaluation process, no significant differences in the characteristics of projects when clustered according to their evaluation score have been detected. Projects sampled that obtained a score below 80 still managed to implement all activities and achieved outputs and results that help fulfil the objectives of the REC programmes (case study 2 26 ). Therefore, the 70-points threshold 27  can also be maintained in future programmes. Meanwhile, the process for integrating the observations received during the negotiation phase for grant agreements could be simplified: only the major comments from evaluators could be taken into account at the preparation stage for grant agreements in order to streamline and shorten the process.

On the administrative burden, feedback from beneficiaries suggests that there is scope for decreasing the administrative burden or at least for increasing its perceived usefulness. This could be achieved by shifting the focus of the requirements from a compliance perspective to a more result-oriented one.

At the programme level, the REC programme’s management has improved over time and compared with its predecessors, notably due to the new portal, improved communication with the Commission, offered more flexibility in budget modifications and clearer access to information.

To improve gender equality mainstreaming in the future programmes, the importance of ‘gender mainstreaming’ could be communicated more extensively, and support in the form of guidance could be provided to beneficiaries. Furthermore, the future programmes could lead by example and continue promoting gender mainstreaming as a key tenet of their intervention.

The promotion of mainstreaming of other equality grounds (‘equality mainstreaming’) could be strengthened by requesting applicants to carry out a needs assessment from the perspective of equality, which could also cover the intersection with other grounds of discrimination. The monitoring data must be broken down by participant sex.

Finally mainstreaming at the level of the future programmes could be strengthened by adopting a ‘gender budgeting approach’ to the programming of resources by keeping track of how resources are allocated and spent on actions that not only directly target gender equality, but also indirectly promote this cross-cutting objective.

Annex 1: Procedural information

1.Lead DG, Decide Planning/CWP references

Lead Directorate-General: Directorate-General for Justice and Consumers

Decide planning reference: PLAN/2020/9646 - JUST - Report to the EP and the Council assessing the 2014-2020 REC Programme

2.Organisation and timing

The evaluation was carried out between March and September 2021. The Commission was supported by a consortium of contractors (Ernst & Young Consulting BV/SRL and Fondazione Giacomo Brodolini Foundation S.r.l. SB), commissioned under the framework contract No JUST/2020/PR/03/0001-01 Lot 1 with a specific contract JUST/2021/JADM/FW/JU04/0007.

As this is not a fully-fledged ex post evaluation, no Inter-service Steering Group was established. However, several DG JUST policy units (JUST 01, JUST 03, JUST A1, JUST B1, JUST B2, JUST C3, JUST D2) as well as colleagues from DG EMPL were involved and contributed to the evaluation.

Chronology of the evaluation:

3.Exceptions to the better regulation guidelines

As this ‘first-step’ report only provides a first assessment of the REC programme’s achievements, it is not fully aligned with better regulation guidelines. In particular, while taking guidance from the ‘better regulation’ regarding the evaluation methodology to be applied, no roadmap, no public consultation and no formal inter-service steering group were created at this stage. These ‘better regulation’ requirements will be fully followed for the ‘second-step’ reports as the benefits from these would be more pertinent when preparing the fully-fledged ex post evaluation.

4.Consultation of the RSB (if applicable)

Not applicable

5.Evidence, sources and quality

The evaluation was based on evidence from different sources. The complete set of documents that were consulted for this evaluation is listed below:

Legal base of the programme:

·Regulation (EU) No 1381/2013 of 17 December 2013 establishing a Rights, Equality and Citizenship Programme for the period 2014 to 2020 (OJ L 354, 28.12.2013).

Commission Implementing Decisions:

·Commission Implementing Decision C(2014)2556 concerning the adoption of the work programme for 2014 and the financing for the implementation of a Rights, Equality and Citizenship Programme

·Commission Implementing Decision C(2015)1997 concerning the adoption of the work programme for 2015 and the financing for the implementation of a Rights, Equality and Citizenship Programme

·Commission Implementing Decision C(2016)1677 concerning the adoption of the work programme for 2016 and the financing for the implementation of a Rights, Equality and Citizenship Programme

·Commission Implementing Decision C(2017)1316 concerning the adoption of the work programme for 2017 and the financing for the implementation of a Rights, Equality and Citizenship Programme

·Commission Implementing Decision C(2017) 8518 concerning the adoption of the work programme for 2018 and on the financing of the Rights, Equality and Citizenship Programme

·Commission Implementing Decision C(2018) 7916 concerning the adoption of the work programme for 2019 and on the financing of the Rights, Equality and Citizenship Programme

·Commission Implementing Decision C(2019) 7824 concerning the adoption of the work programme for 2020 and on the financing of the Rights, Equality and Citizenship Programme

Desk research focused on available documents, both at programme, policy and project level. The documents at programme level included the REC programme’s programming and implementation documents (such as annual implementation reports, programme statements, impact assessment), as well as the interim evaluation of the REC programme.

Documents at project level comprised:

·files with extracts of key monitoring data;

·monitoring data derived from the interim evaluation;

·grant agreements and project proposals (covering projects funded from 2016 onwards);

·technical periodic reports (final reports), including monitoring annexes and financial reports (only for finalised projects funded from 2016 onwards);

·amendment of grant agreement (‘amendment notarised’), providing information about possible changes made after the signature of the agreement;

·proposal evaluation form (‘evaluation summary – ESR’): providing information on the evaluation process of the proposal (scores, ranking, qualitative assessment);

·technical report and annexes;

·evaluation of the final technical reports, providing a final assessment of the project by policy officers;

·final financial statements;

·list of final deliverables;

·list of milestones.

The documents at policy level included the 2016-2020 strategic engagement for gender equality, the Commission’s list of actions to advance LGBTI equality, the EU Framework for national Roma integration strategies up to 2020.

In addition, the following documents were used for this evaluation:

Annex 2: Stakeholder consultation

Consultations with the main stakeholders of the REC programme were conducted over several stages:

Interviews

Semi-structured interviews were organised with both project beneficiaries/contractors and Commission staff. Interviews consisted of semi-structured video interviews. Questions were customised according to the different categories of stakeholders targeted, taking into account their different answers for preparing replies to the evaluation questions.

In-depth interviews had a two-fold objective:

-collecting information to feed the analysis alongside the evaluation criteria, filling gaps (e.g. in terms of quantifying costs and benefits) and going more in-depth with specific aspects,    such as identifying good practices;

-gathering insight and input to draw conclusions and formulate recommendations on how to improve the programme’s design and operation.

A total of 53 interviews were carried out. These comprised:

-5 scoping interviews (April – May 2021);

-7 interviews with policy officers (May – July 2021);

-41 interviews with project beneficiaries (28 related to action grants, 8 related to operating grants, and 5 related to procurement contracts) (May – July 2021).

Focus groups

To help draft the case studies, in addition to monitoring data and information collected through interviews with project beneficiaries and Commission staff, three focus groups were organised with a sub-sample of project beneficiaries selected for in-depth analysis. Additional potential applicants were also invited to the focus groups.

-Focus Group 1 – Case study 1: ‘Recurring beneficiaries/continuations of projects’.

-Focus Group 2 – Case study 3: ‘Why some institutional bodies are reluctant to apply?’ Case study 7: ‘Are the beneficiaries selected best fit to help the target groups?’

-Focus Group 3 – Case study 8: ‘How has the gender mainstreaming been promoted within the funded intervention and what were the strengths of your approach?’

One additional focus group was organised with DG JUST project officers. Its aim was to collect additional and complementary information, focusing on management and procedural aspect of project implementation.

Programme Committee Members (PCM) E-Survey

The goal of this survey was to collect comprehensive and specific information on the application processes, main features and results of projects and activities funded by the programmes, as well as to capture insights and expert views. Survey results made it possible to gain a preliminary understanding of how the REC programme performed according to the evaluation criteria.

A first survey was sent to PCMs. Its goal was to collect information that could feed the analysis and the answers to the evaluation questions. Questions were tailored to PCMs experience and knowledge.

The survey was launched though an online tool (Qualtrics) on 23 June 2021 and remained open until 14 July. The survey was kept open 4 weeks longer than originally planned to allow PCMs more time to answer the questions.

A total of 17 responses were collected for the REC programme, covering the following countries – Austria, Bulgaria, Czechia, France, GR, Croatia, Latvia, Romania, Sweden, Slovenia, and Serbia).

Additional survey to feed the case study 4

A second survey was launched aimed at collecting information to feed Case Study 4. The survey was sent to 200 project beneficiaries, and 11 relevant responses were collected. 

The different field research activities together with the main stakeholders involved and their relationship with the methodological steps are shown in the table below.

Distribution of stakeholders consulted

Annex 3: Methods and analytical models

Starting from a deep understanding of the programme, its intervention logic and underlying theory of change, the evaluation is built upon three main interconnected methodological steps.

-Task 1 – Extraction, collection and analysis of the information monitored providing an overview of the state of play on implementation and of the programme’s performance and main achievements.

-Task 2 – Collection of additional information, through:

ofield research (interviews with beneficiaries, stakeholders and Commission officials, focus groups, e-survey to programme committee members and additional documentary review); and

oin-depth analysis of a sample of projects and services funded through the programme, providing a more nuanced and detailed picture of the key features and achievements of a limited set of projects:

§types of actions implemented;

§how the actions are linked with the specific objectives;

§outcomes and results of the actions.

-Task 3 – Drafting case studies and answers to the evaluation questions. Based on the information and findings derived from the two tasks mentionned above and their triangulation:

a.eight case studies were drafted, focusing on a set of operational and procedural components covering both the REC and the justice programmes and ultimately contributing to feeding the answers to the evaluation questions

b.the evaluation questions were answered and suggestions were identified as regards improvement.

Data collection tools and activities

The study relied on both desk and field research.

Desk research focused on available documents, both at programme, policy and project level. It was mostly carried out under Task 1, but additional desk research was carried out for projects selected for in-depth analysis under Task 2 and for answering the evaluation questions.

These documents were mainly used for the following steps:

·building a retrospective overview of the programme against which the achievement and performance of the projects were assessed;

·reviewing and integrating, where relevant, the Intervention Logic and the underlying theory of change 28 ;

·collecting additional information on programmes’ performance.

Information collected was fed into the evidence grids, the case studies and ultimately the evaluation report. Part of the information was also used to finalise Task 1 data collection (e.g. Block 4 indicators covering projects sampled and Block 2 indicators covering the ‘legal base indicators’).

Distribution and coverage of project documents reviewed

The field research aimed at collecting relevant data and input from project beneficiaries and stakeholders, filling gaps and validating information collected during the desk research. It was mostly carried out as part of the in-depth project analysis under Task 2 and during the inception phase, but additional field information was collected under Task 3 for drafting the case studies and answering the evaluation questions. For more information, please also see Annex 2: Stakeholder consultation.

Task 1 - Extraction, categorisation, and aggregation of monitoring data

For this task, the following steps were undertaken.

A.Creation of the dataset for action grants and operating grants 

The information retrieved from monitoring files was cleaned, systematised, and transferred to a dataset specifically created for this evaluation. Since information available from monitoring files and project documents covered mostly 2016-2020 projects and applications, resulting in the dataset still having gaps and missing information for older projects, the dataset was integrated with information from the interim evaluation. Similarly, information on applicants and participants covered mostly 2016 onwards. Information was further completed from different sources: e.g. information on the type of organisation applicants represented – which was missing – was extracted from the list of participants and linked to those applicants that participated in the granted project too.

Further information was obtained from the following.

·New extractions on projects and applications – both updating the list and adding new types of information from proposals (e.g. on the names, country, and type of beneficiary) and reports (e.g. on dissemination activities carried out). The new data were either added to the existing dataset or used to update it.

·Annex 3 to the technical reports. The new data were first extracted automatically and then manually cleaned and systematised to reduce the ‘noise’ and prevent it from being lost’.

B.Extraction of key qualitative texts/unstructured information

To extract data on qualitative descriptors, the research team adopted a mix of qualitative data analysis approaches and tools (word search, categorisation, text mining and semantic tools). The starting base for implementing these approaches involved firstly identifying the document sources and the exact text fields within these sources to be screened. Next, the team organised the manual extraction (copy/paste) of these relevant text fields into an excel sheet on which the various qualitative analysis tools could then be implemented. Based on the qualitative information extracted, the following analyses were carried out:

Text mining/semantic tool

The documents were analysed using named entity recognition (NER) tools. NER is a subtask of information extraction that seeks to locate and classify named entities mentioned in unstructured text into pre-defined categories such as names of people, organisations, locations, medical codes, time expressions, quantities, monetary values, and percentages. In the case of analysing project proposals, these entities are words relevant for the content of the call. In our case, we focused on: target groups, dissemination activities and needs.

Screening of qualitative information and categorisation

For a number of descriptors, the research team adopted a manual categorisation approach, involving the screening of specific text fields and categorising the information for each project based on a drop-down list of categories.

Mixed methods (word search + categorisation):

For a number of descriptors, a mixed approach to qualitative analysis was followed. First, the relevant information (when available) was identified through a simple word search; it was then manually extracted. This approach was followed for the following descriptors:

·number of interactions (likes, shares, retweets, comments) generated by online awareness-raising, information and dissemination activities;

·Number of fans and followers acquired through online awareness-raising, information and dissemination activities

C.Creation of a list of indicators/descriptors

The monitoring data that were collected as per the previous steps were then categorised and aggregated – to support a narrative around key topics – as follows:

·indicators and descriptors related to applicants/participants;

·indicators and descriptors related to the projects’ budget allocation and distribution;

·indicators and descriptors related to the types of activity funded;

·indicators and descriptors related to target groups/stakeholders;

·indicators and descriptors related to other features (such as geographical coverage, project duration);

indicators and descriptors related to the programme’s performance/the evaluation criteria.

Task 2 - Additional collection and systematisation of information

After the project-related data and documents were collected, extracted, and systematised as described in the previous section, the Research Team carried out further data collection activities to gather additional input and systematise all information collected to feed the response to the evaluation questions.

This included both desk and field research on a sample of projects.

A total of 36 projects, involving beneficiaries of both action (28) and operating grants (8), plus six procurement contracts, were sampled in order to carry out a qualitative analysis based on comments and insights from key stakeholders on the REC programme’s implementation. The projects were sampled taking into account geographical distribution, specific objective, and financial amount.

D.Sampling of action grants

This sample of projects aimed to be as representative as possible of the overall project ‘population’ at Member State/specific objective level, which was:

·proportional to the amount and form of funding per Member State/specific objective;

·comprehensive, thus including projects that:

ocovered all specific objectives;

ocovered all forms of support;

ocovered different years of the programme’s period’.

These criteria ensured that the sample of projects identified was suitable enough to demonstrate how the programme was implemented by each of its specific objectives.

In order for this distribution to be representative not only in terms of specific objectives and forms of support, but also geographically, attention was also paid to the beneficiary’s nationality. Therefore, the sample of projects ensured that Member state representation (in terms of beneficiaries’ nationalities) was proportionate to the overall geographical distribution of projects (in terms of participation in the calls and success rate for action grants and in terms of grants awarded for operating grants).

E.Sampling of operating grants

On selecting the operating grants, this was first informed by criteria such as: i) the overall financial allocation to a given organisation/network within the specific objective; or ii) the relevance of the organisation to the programmes objectives and the specific objective. Sampling of procurement contracts

The selection of the procurement contracts was based on the relevance of the contracted service to the programmes’ objectives and to the European Commission’s needs as regards information. In selecting contracts, care was taken to cover different types of procurement activities

Task 3 Drafting case studies and answering the evaluation questions

F.Case studies

The evaluation included eight case studies. Their main purpose was to provide solid, triangulated evidence to support preparation of replies to evaluation questions as well as suggestions for improvements, with a focus on several specific issues. The case studies were developed, based on a customised methodological approach.

In addition to data and information collected under Tasks 1 and 2, the case studies involved additional field and desk research, in particular:

·further in-depth analysis of sampled projects’ documents; 

·an online survey regarding all projects sampled (for Case Study 4);

·three focus groups, which included beneficiaries from projects that were not part of the sample (for case study 1, 3 and 7, 8).

The table below provides an overview of what the eight case studies focused on and shows the evaluation questions they refer to.

Overview of the eight case studies

G.Answering the Evaluation questions

Information gathered through desk and field research was used to answer the following evaluation questions:

1.To what extent have the objectives been achieved considering the set indicators? 

2.How fit for purpose were the actions funded under the programme to the objectives of the programme?

3.What factors influenced the achievements observed?

4.How effective were the communication activities as to informing the potential applicants about upcoming calls and in increasing the visibility of funded projects?

5.Have the projects delivered the results envisaged in the applications?

6.To what extent have the interventions been cost-effective?

7.Has the management of the Programme by the Commission become more efficient?

8.How relevant were the actions and results achieved to the needs of the different stakeholders?

9.How well do the (original) objectives (still) correspond to the needs within the EU?

10.How relevant for achieving the objectives of the Programme were the groups targeted by the intervention?

11.To what extent are these interventions coherent/ complementary with other EU and/or national policies and funding Programmes which have similar objectives and Union bodies' work? Are the conclusions on coherence and complementarity in the interim evaluation still valid?

12.What is the added value resulting from the EU intervention(s), compared to what could be achieved by Member States?

13.What would be the most likely consequences of limiting the level of the existing EU intervention or completely stopping/withdrawing from it?

14.How fairly are the different activities distributed across the different target groups, and participating countries?

15.How has the gender mainstreaming been promoted within the funded interventions?

16.How have the rights of the child been promoted within the funded interventions?

17.How have the rights of people with disabilities been promoted within the funded interventions?

18.How can the programme management, with focus in particular on the grant management, be further simplified to alleviate administrative burden of the Commission and of the applicants and the beneficiaries?

Annex 4: Progress on indicators

Indicators concerning the general objective

Indicators concerning the specific objectives

Annex 5: Results of the case studies

(1) REGULATION (EU) No 1381/2013 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 17 December 2013 establishing a Rights, Equality and Citizenship Programme for the period 2014 to 2020.

(2) REGULATION (EU) 2021/692 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 28 April 2021 establishing the Citizens, Equality, Rights and Values Programme and repealing Regulation (EU) No 1381/2013 of the European Parliament and of the Council and Council Regulation (EU) No 390/2014.

(3)

     COM(2014) 144 Communication from the Commission to the European Parliament, the Council and the European Economic and Social Committee and the Committee of the Regions, The EU Justice Agenda for 2020 - Strengthening Trust, Mobility and Growth within the Union:
http://ec.europa.eu/justice/effective-justice/files/com_2014_144_en.pdf  

(4)      SWD(2015) 278 Strategic Engagement for Gender Equality 2016-2019:
https://ec.europa.eu/anti-trafficking/sites/antitrafficking/files/strategic_engagement_for_gender_equality_en.pdf  

(5)      COM(2010) 636 European Disability Strategy 2010-2020: A Renewed Commitment to a Barrier-Free Europe:
https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=LEGISSUM%3Aem0047  

(6)      List of actions by the Commission to advance LGBTI equality (2015-2019):
https://ec.europa.eu/info/sites/info/files/lgbti-actionlist-dg-just_en.pdf  

(7)      COM/2011/0173 An EU Framework for National Roma Integration Strategies up to 2020:
https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1444910104414&uri=CELEX:52011DC0173  

(8)  For more info on the European Disability Card, see http://ec.europa.eu/social/main.jsp?catId=1139  

(9)  For more info on the Data Protection Reform, see https://ec.europa.eu/info/law/law-topic/data-protection_en . 

(10) SEC(2011) 1364 Final, ibid.

(11)  European Commission (2015), Ex-post evaluation of five programmes implemented under the 2007-2013 financial perspective – Final Report, ibid. 

(12)  This includes restricted all explicit ‘Call for proposals’ and ‘Calls for fixed-year Framework Partnership Agreements’ (operating grants), but excludes the direct grants, yearly calls for operating grants as well as procurement, prizes, and awards.

(13) The analysis counts only finalised projects and thus presumably covers projects which started, at the latest, in the first quarter of 2019.

(14)  ‘To contribute to the further development of an area, where equality and the rights of persons as enshrined in the Treaty on European Union, in the Treaty on the Functioning of the European Union, in the Charter of Fundamental Rights of the European Union and in the international human rights conventions to which the Union has acceded, are promoted, protected and effectively implemented’. 

(15) For the list of documents used during the desk research, see Annex 1.

(16) Art 4 (2 a, b, c, d) of the REC Regulation.

(17)  For more details, see Annex 5.

(18)  For more details, see Annex 5.

(19)  For more details, see Annex 5.

(20)  For more details, see Annex 5.

(21)  For more details, see Annex 5.

(22)  For more details, see Annex 5.

(23)  For more details, see Annex 5.

(24) SEC(2011) 1364 Final, ibid.

(25)  European Commission (2015), Ex-post evaluation of five programmes implemented under the 2007-2013 financial perspective – Final Report, ibid. 

(26)  For more details, see Annex 5.

(27) Currently, project proposals that receive at evaluation stage 70/100 points or more can be funded.

(28)  Theory of change implies logical explanation of the ways and assumptions through which allocation of funds will produce outputs through which intended results are to be achieved (the expected change). This theory of change has to take on board the economic and political context of the programme as well as other factors (social, cultural, institutional…) that may influence the mechanisms leading to the results.

(29) Programme statement 2021–  document .

(30) Programme statement 2021–  document .