EUROPEAN COMMISSION

Brussels, 29.6.2018

SWD(2018) 358 final

COMMISSION STAFF WORKING DOCUMENT

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS

on the interim evaluation of the implementation of the Rights, Equality and Citizenship Programme 2014-2020

{COM(2018) 508 final}
{SWD(2018) 359 final}

Table of contents

1.Introduction

2.Background to the intervention

3.Implementation / state of Play

4.Method

5.Analysis and answers to the evaluation questions

6. Conclusions and way forward

Annex 1: Procedural information

Annex 2: Stakeholders consultation

Annex 3: Methods

1.Introduction

Purpose and scope of the evaluation

This staff working document sets out the results of the interim evaluation of the Rights Equality and Citizenship (REC) Programme for the period 2014-2020.

The evaluation was carried out under the provisions of Article 13(2)(b) of the European Parliament and Council Regulation establishing the Rights Equality and Citizenship Programme for the period 2014-2020 1 .

It aimed mainly to assess the Programme’s outputs and results compared to its objectives and to assess qualitative and quantitative aspects of its implementation. It also assessed whether the current Programme is on track to achieving its objectives and the extent to which the recommendations of the previous 2007-2013 ex-post evaluation have been followed. The ex-post evaluation carried out in 2015 concerned the three predecessor programmes which were merged in the current Programme (namely, Daphne III, Fundamental Rights and Citizenship and Progress) 2 .

The current interim evaluation has informed the Report from the Commission to the European Parliament, the Council of the European Union, the European Economic and Social Committee and the Committee of Regions which this document accompanies. It also helped the reflection on the Programme's future by providing evidence-based information.

Under the Regulation, the Commission is requested to present an ex-post evaluation report for the Programme by 31 December 2021.

The Programme was assessed based on the following main evaluation criteria:

1. Relevance: whether and to what extent the Rights Equality and Citizenship Programme addresses needs and problems of the target groups identified in the 2011 Impact Assessment 3 and in the legal basis of the Programme, as well as emergent issues, and whether its objectives are still relevant for the needs and problems of the beneficiaries;

2. Efficiency: whether and to what extent the costs of the Programme were proportionate given the benefits achieved and which parameters/factors participated in these results;

3. Effectiveness: whether and to what extent the Rights Equality and Citizenship Programme has achieved its general objective, as well as its nine specific objectives, and which are the factors that have contributed to these achievements;

4. Coherence/Complementarity/Synergies: whether and to what extent the Programme is coherent with other interventions at the EU and international level, such as with the predecessor EU programmes in the field 4 , with activities supported by other Union instruments and, in general, with the European priorities in the fields covered by the Programme;

5. EU added-value: to what extent the effects from the EU action are additional to the value that would have resulted from action at the national level only;

6. Equity: whether and to what extent the Programme has distributed the available resources fairly among beneficiaries in different Member States, took into consideration the needs of target groups, promoted gender mainstreaming, the rights of the child and the rights of people with disabilities;

7. Scope for simplification: whether and to what extent the management of the Rights Equality and Citizenship Programme could be further simplified.

Geographically, the Programme is open to all EU Member States, but also to the European Free Trade Association States that are party to the European Economic Area, candidate countries, potential candidates and countries acceding to the Union, provided that they conclude an agreement with the Union laying down the details of their respective participation in the Programme. Iceland joined for all specific objectives of the Programme, while Liechtenstein participates only in the specific objectives related to non–discrimination, racism, xenophobia, the rights of persons with disabilities and equality between women and men.

The reference period for this interim evaluation is the first half time of the Programmes’ implementation from 2014 to mid-2017 5 .

2.Background to the intervention

Since 1948, when the United Nations General Assembly adopted the Universal Declaration of Human Rights, an elaborate body of international human rights law has developed through state practice, the work of international courts and agreement of multilateral treaties. Human rights law is currently enshrined in dozens of human rights treaties and protected by organisations, such as the UN, the Council of Europe and the African Union.

Within EU borders, human rights are embedded in the EU treaties and reinforced by the EU Charter of Fundamental Rights. These rights include protection of individual rights for EU citizens as well as promotion of equality and non-discrimination for all individuals in the EU.

Since 2014, the Rights, Equality and Citizenship Programme aims at further developing a Europe of rights and equality in accordance with the Europe 2020 Strategy 6 .

At the time of its definition, the following key problems and needs were identified in the Impact assessment of the three predecessor programmes 7 : 1. Limited effectiveness in the implementation of activities against discrimination on the grounds of sex, racial or ethnic origin, religion or belief, disability, age or sexual orientation, and more Roma integration and gender equality; 2. Insufficient promotion and protection of the rights enshrined in the EU Charter of Fundamental Rights, including the rights of the child and the rights of persons with disabilities, and in the UN Convention on the Rights of Persons with Disabilities; 3. Too high levels of gender based violence and too little support to victims of violence; 4. Insufficient protection of EU citizen’s personal data; 5. Limited knowledge and implementation of the rights deriving from the citizenship and law of the Union.

Description of the programme

The introduction of the Rights, Equality and Citizenship Programme was meant to address all these challenges with the general objective of contributing to the strengthening of equality and the rights of persons, as enshrined in the Treaties, the EU Charter of Fundamental Rights and in international human rights conventions to which the European Union has acceded.

In order to do so, the Programme addresses fundamental rights in nine areas, corresponding to its specific objectives:

·Specific objective 1: Promoting the effective implementation of the principle of non-discrimination on grounds of sex, racial or ethnic origin, religion or belief, disability, age or sexual orientation, and to respect the principle of non-discrimination on the grounds provided for in Article 21 of the Charter

The Programme supports projects aiming towards preventing and combatting discrimination on the grounds of sex, racial or ethnic origin, religion or belief, disability, age or sexual orientation. In particular, the promotion of Roma integration is a key topic of this specific objective. In this regard, the Rights, Equality and Citizenship Programme supports the National Roma Platforms and works in close cooperation with EU Member States and their National Roma Contact Points and with civil society organisations active on non-discrimination and Roma integration. The Programme finances also the activities of the European Network of Equality Bodies. The core task of this network includes strengthening the cooperation and exchange of information between the National Equality Bodies designated by the Member States on topics related to non-discrimination.

·Specific objective 2: Prevent and combat racism, xenophobia, homophobia and other forms of intolerance

Projects financed in this area contribute to better implementation of existing EU legislation in the Member States and to assisting the victims of hate crime and hate speech. In particular, the funded activities support the protection of refugees and migrants against the surge of populism, extremism and intolerance.

·Specific objective 3: Promote the rights of persons with disabilities

The operational aim of this specific objective is to increase awareness and better implementation of the rights of people with disabilities leading to a reduction of barriers, to their full participation in society and enjoyment of their rights. This is mostly done through operating grants supporting the activities of European level networks advocating for the rights of people with disabilities and through procurement activities focusing on data collection, training and awareness-raising activities. Under the 2014 annual work programme, the European Disability Card 8 was launched with support from the Programme in order to facilitate cross-border travelling for persons with disabilities.

·Specific objective 4: Promote equality women and men

The Programme promotes projects to support Member States and relevant stakeholders in increasing gender balance in all spheres of life, especially in economic decision-making, and promoting equal economic independence of women and men. Funding under this objective also aims to contribute to reducing the gender gaps in pay, earnings and pensions.

·Specific objective 5: Prevent and combat violence against children, young people and women, and other groups at risk

The Programme finances primarily projects which help to protect and support the most vulnerable victims of violence, such as women and children, as well as activities to raise awareness and prevent violence, engaging with non-governmental organisations working on the ground. It also supports the treatment of perpetrators of violence. Strengthening child protection systems is one of the Commission's priorities and the Programme supports the rights of migrant children by funding international organisations working directly for and with these children.

·Specific objective 6: Promote the rights of the child

The rights of the child are promoted through projects focusing on child-friendly justice and on the protection of vulnerable children (like children leaving foster care systems and children in conflict with the law), by supporting, in particular, training activities of civil servants, non-governmental organisations and lawyers dealing with them.

·Specific objective 7: Protect privacy and personal data

With regard to data protection, the Programme is the EU main funding source. The activities carried out within this specific objective are strongly linked with the Data Protection Reform 9 , adopted in 2016. Through this objective, the Programme aims at financing activities to support, in particular, the transposition and implementation of the new data protection legislation by the Member States and the training of data protection authorities and data protection officers.

·Specific objective 8: EU citizenship

The Programme finances projects, mostly awareness raising activities, aimed at fostering the inclusion of EU citizens in the civic and political life of the EU. Therefore, it supports projects which can help citizens to be more aware of their rights deriving from the citizenship of the Union. Information campaigns on EU citizenship rights are financed through this specific objective and a particular focus is on their electoral rights in view of the 2019 European elections.

·Specific objective 9: Consumers' or entrepreneurs' rights

The Programme finances also projects which support individuals in their capacity as consumers or entrepreneurs in the internal market in order to enforce their rights deriving from UE law. EU funds in this area aim at contributing to increase the knowledge and awareness of consumers' rights, especially in the digital market.

Budget

The total Programme's budget for the period 2014-2020 is EUR 439 473 000 (see Table 1). The highest commitment rate is observed in 2016, while, according to available sources, the total committed EU contribution is lower for the previous two years.

Table 1: Total annual amount planned and committed in 2014-2016

Sources: Annual monitoring reports (reports on the implementation of the annual work programmes and Project Database for 2016)

Procurement activities represent about 23% of the budget, though the distribution between the specific objectives is not homogeneous. Indeed, procurement activities represent the total planned spending of the specific objective on the rights of consumers, 65% of the specific objective on citizenship and less than 5% of the total planned budget on the specific objective on the prevention of racism. All other specific objectives allocate to procurement activities between 11% and 55% of the planed budget.

In order to allow the achievement of its specific objectives, the Programme identifies a wide range of activities to be implemented. In particular, according to Article 5 of the Regulation establishing the Rights, Equality and Citizenship Programme for the period 2014 to 2020, the Programme can finance:

·Analytical activities, in particular: collection of data and statistics; development of common methodologies and, where appropriate, indicators or benchmarks; studies, researches, analyses, surveys and evaluations; elaboration and publication of guides, reports and educational material; workshops, seminars, experts' meetings and conferences;

·Mutual learning, cooperation, awareness raising and dissemination activities, in particular: identification of, and exchanges concerning, good practices, innovative approaches and experiences; organisation of peer reviews and mutual learning; organisation of conferences, seminars, media campaigns, including in the online media, information campaigns, including institutional communication on the political priorities of the Union as far as they relate to the objectives of the Programme; compilation and publication of materials to disseminate information about the Programme and its results; development, operation and maintenance of systems and tools using information and communication technologies;

·Training activities, for instance: staff exchanges, workshops, seminars, train-the-trainer events and the development of online training tools or other training modules;

·Actions to support the main actors whose activities contribute to the implementation of the objectives of the Programme, such as support for Non-governmental organisations in the implementation of actions with European added value; support for key European actors, European-level networks and harmonised services of social value; support for Member States in the implementation of Union law and policies and support for networking activities at European level among specialised bodies and entities as well as national, regional and local authorities and non-governmental organisations, including support by way of action grants or operating grants 10 .

The stakeholders eligible for support are public or private organisations (usually non-profit-oriented), duly established in one of the countries participating in the Programme, or international organisations (such as the UN High Commissioner for Refugees, UNICEF and the International Organization for Migration).

In terms of target groups, intended as the groups that can benefit directly or indirectly from the Programme, public authorities, non-governmental organisations and other research entities, concerned with pursuing its objectives, are included. Target groups are also all citizens, since its objectives and initiatives aim at promoting, protecting and effectively implementing equality and the rights of persons, in particular those subjected to discrimination, intolerance or violence, including migrants and minorities. Specific objectives focus on support for people with disabilities, Roma, women, EU citizens, consumers and entrepreneurs and children 11 .

The Rights, Equality and Citizenship Programme has three main funding mechanisms: action grants, operating grants and procurement actions 12 .

·Action grants are addressed mainly to civil society organizations, Member States authorities and universities;

·Operating grants ("support to Networks") fund mainly European networks active in the following areas: non-discrimination; racism, xenophobia, homophobia or other forms of intolerance; disabilities; equality between women and men; preventing and combating all forms of violence and promoting the rights of the child.

·Procurement actions ("Commission initiatives") fund mostly conferences, seminars, studies, surveys, awareness-raising activities, but also specific IT projects. 

The Programme intervention logic is outlined in the Figure 1.

Figure 1: Intervention Logic of the Rights Equality and Citizenship Programme 2014-2020

External factors could influence the Programme's outcomes and make challenging to identify changes as a result of its intervention. In particular, the economic crisis has led to a lack of interest on the part of public sector stakeholders (more interested in economic issues) and, therefore, to a related reduction of national funding for social and human rights issues. This is more accentuated in certain Member States where the political climate is less supportive 13 .

Baseline and points of comparison

The interim evaluation assessed the Rights, Equality and Citizenship Programme's performance starting from the situation described in the 2011 Impact Assessment 14 and in the ex-post evaluation of 2007-2013, concerning the three predecessor programmes (Daphne III, Fundamental Rights and Citizenship and Progress), carried out in 2015 15 .

These documents contain the baselines and main points of comparison for measuring the Programme’s achievements during the reporting period.

The baseline analysis showed that, if no changes had been made between the 2007-13 and 2014-20 programming period, the three previous programmes would have continued to be successfully implemented, but at a reduced potential, in particular due to: 1. The lack of flexibility in the funding instruments which did not reflect the pace of change and reform in this policy area; 2. The fragmentation of funding that reduced the capacity of the programmes to deliver results in horizontal and cross-cutting issues; and 3. The elevated number of different funding instruments which increased the administrative burden.

Given these difficulties, the Impact Assessment suggested the option to consolidate the three programmes in the current Rights, Equality and Citizenship Programme.

This possibility was assessed in terms of its relevance/scope, effectiveness, complementarity, European added value, efficiency and potential for simplification. Since the option to proceed with only one programme was undertaken, the findings of the Impact Assessment, together with the recommendations of the ex-post evaluations of 2007-2013, provide a useful baseline to verify whether the Rights, Equality and Citizenship Programme achieved its expected goals.

Follow-up on the ex post evaluation report of the period 2007-2013

All the ex-post evaluation reports concerning the previous funding programmes confirmed their overall effectiveness and highlighted that their specific objectives and priorities were largely specific, attainable and realistic, but not always measurable. Indicators, allowing the measuring of progress toward the attainment of the specific objectives, were not included.

Moreover, the reports pointed out the need to make improvements on other issues which could have increased the impact, added value, effectiveness and efficiency in delivering result of the programmes.

In particular, the reports have identified the following needs common to the three programmes:

1. Better definition of the priorities in order to ensure that they can be adequately achieved within an earmarked budget;

2. Realistic assessments of project risks and better risk mitigation strategies throughout the projects duration;

3. Increase focus on assessment of impacts at all levels and not merely on outputs, as regards monitoring and evaluation;

4. Explore ways of enhancing the uptake of project outputs, results and best practices by other organisations, including in other Member States;

5. More dissemination/use of results and outputs of the funded activities.

The ex-post evaluation reports of the period 2007-2013 have also highlighted the following problems: 1. The dilution of funds amongst many small-scale projects with limited impact and EU dimension; 2. No balanced geographical spread among the organisations which receive funding; 3. The complex and bureaucratic procedures for the applicants; 4. The high administrative burden on the Commission and an increase of the length of procedures due to the multiplication of procedures for the different programmes.

These issues were taken into consideration and were either integrated into the Regulation establishing the current Rights, Equality and Citizenship Programme and in the annual work programmes or translated into technical implementation measures and included in the Guides for applicants to projects calls.

In particular, the merger of the three funding programmes has provided a significant positive impact on the identified problems of scope, effectiveness, fragmentation and efficiency (see further details in section 5 “Analysis and answer to the evaluation questions”).

3.Implementation / state of Play

Programme management

The Rights, Equality and Citizenship Programme is implemented via direct centralised management by the European Commission. This was meant to ensure a close relationship between the Programme management and EU policy-making and to contribute to the alignment of budget implementation with EU policy priorities in line with the "budget for result" approach. This management mode allows the Commission to tailor funded activities to policy priorities and policy needs and to target directly the relevant groups of stakeholders. According to the evaluation carried out, it also allows a close contact with the programmes' beneficiaries and better knowledge of the needs on the ground. Indeed, most of the stakeholders interviewed referred to improvements in the management systems, as well as good communication with the Commission and contact persons. In this respect, several stakeholders agreed that the policy staff in the current Programme is much more engaged with civil society 16 .

State of play

The purpose of this section is to present the state of play of the Programme’s implementation and its key initiatives in the period 2014-2017. It will also provide the qualitative and quantitative results achieved in the same period, in general terms and for each specific objective.

In order to measure the progress from the baseline situation, a series of Programme indicators (see below in this section 17 ) have been devised for the current Programme. The introduction of a system of indicators has proved to be adequate for measuring the achievements of the Programme since the selected indicators show if the targets are close to being achieved or have already been achieved and this contributes to better focus the Programme's outcomes.

Concerning the general objective of the Programme, the 2020 target appears difficult to be achieved for three of the five selected indicators 18 .

In particular, the increase of the employment rate of women and people with disabilities is hampered by an external factor, i.e. the European financial recession. Also the percentage of women among non-executive directors on boards of listed companies is not reaching the 2020 target. However, the Rights, Equality and Citizenship Programme is contributing to the achievement of these objectives through projects aimed at promoting the exchange of good practices, developing tools to collect data on women and men in decision making and lobbying at the EU and national level for promoting the labour market inclusion of women.

Concerning the employment rate of people with disabilities, the target is difficult to be achieved since the participation in employment of people with disabilities is lower than people without disabilities and the recession has only worsened it. The Programme is, however, contributing to the achievement of the target by funding projects empowering organisations representing people with disabilities towards an effective lobbying at the EU and national level for designing policies favouring the labour market inclusion of people with disabilities.

Instead, the gender pay gap is showing decreasing trends and the target is likely to be reached. Projects financed by the Programme promote wage transparency and awareness-raising activities that can enable employees and employers to reduce the gender pay gap.

Finally, the percentage of Europeans who consider themselves as “well or “very well” informed of the rights they enjoy as citizens of the Union is showing increasing trends and the target is likely to be reached if awareness raising activities are adequately tailored to different targets (i.e. primary school students, migrants, rural communities) (see Table 2 below). However, it's challenging to gauge the specific contribution of the Rights, Equality and Citizenship Programme to such results since this topic is addressed by other EU level initiatives (i.e. Erasmus+) 19 .

Table 2: Result indicators, targets and actual values

Concerning the specific objectives of the Programme, the following indicators have been used to evaluate its performance:

·Specific objective 1: To promote the effective implementation of the principle of non-discrimination on the grounds of sex, racial or ethnic origin, religion or belief, disability, age or sexual orientation, and to respect the principle of non-discrimination on the grounds provided for in Article 21 of the Charter

The Programme's performance concerning this specific objective is measured through two main indicators: the number of Europeans aware of their rights if they fall victims of discrimination and the number of Member States that set up structural coordination mechanisms with all stakeholders, including Roma, on the implementation of the National Roma Integration Strategies 20 .

For both of these indicators, it seems likely that the stated goals for 2020 will be reached (see Table 3). Concerning the first indicator, the target will be attained faster if the increased awareness of the rights of victims is combined with training activities of stakeholders in direct contact with them (i.e. social assistants, police officers, lawyers).

Concerning the second indicator, the Rights, Equality and Citizenship Programme has launched ad hoc calls on the implementation of the National Roma Integration Strategies. Therefore, the achievement of the target depends largely on the investments of the Programme in this regard.

Table 3: Result indicators, targets and actual values

·Specific objective 2: To prevent and combat racism, xenophobia, homophobia and other forms of intolerance

For this specific objective, no specific indicator was set up, but the same indicator on non-discrimination could be used.

·Specific objective 3: To promote and protect the rights of people with disabilities

Also for this specific objective, no specific indicator was set up. However, the operating grants funded under this specific objective also cover the promotion of social and labour market inclusion of people with disabilities. The Programme's performance concerning this specific objective is, therefore, measured through the indicator on the employment rate of people with disabilities (see above Table 2, indicator 2).

·Specific objective 4: To promote equality between women and men and to enhance gender mainstreaming

No specific indicator was set up also for this specific objective. However, the indicators used for the general objective are adequate for measuring the achievements under this specific objective. As a matter of fact, these indicators capture the expected results of the projects funded by the Programme that aim at promoting the increased gender balance in economic decision-making positions at all levels (see above Table 2).

·Specific objective 5: To prevent and combat all forms of violence against children, young people and women, as well as violence against other groups at risk, in particular groups at risk of violence in close relationships, and to protect victims of such violence

The Programme's performance concerning this specific objective is measured through one indicator: the percentage of people that consider that domestic violence against women is unacceptable. Until 2015 there were no progresses in this regard, but it is likely that the awareness raising actions of the projects funded by the Programme, which represent a significant portion of the activities funded in this area, will lead to getting closer to the target value (see Table 4). However, this indicator is rather extensive, and many external factors such as economic growth, national policies (including the Istanbul Convention ratification) influence its value.

Table 4: Result indicators, targets and actual values

·Specific objective 6: To promote and protect the rights of the child

The Programme's performance concerning this specific objective was, at first, measured through one indicator: the percentage of children aware that they enjoy specific rights. However, it soon emerged that this specific indicator is not fully adequate to capture the achievements of the Programme within this specific objective. The Commission is currently developing a more suitable indicator for measuring and reporting the progress achieved by the Programme on the promotion and protection of the rights of the child.

·Specific objective 7: To contribute to ensuring the highest level of protection of privacy and personal data

The Programme's performance concerning this specific objective is measured through one indicator: the number of complaints received by data protection authorities from individuals relating to data protection. The contribution of the Rights, Equality and Citizenship Programme to this specific objective is expressed by the capacity of the Programme to raise awareness among public authorities and companies about the content and technical requirements of the 2016 Data Protection Reform. However, no data were available so far (see Table 5).

Table 5: Result indicators, targets and actual values

·Specific objective 8: To promote and enhance the exercise of rights deriving from the citizenship of the Union

The Programme's performance concerning this specific objective is measured through three indicators: 1. The share of the population that knows the meaning of “Citizenship of the Union”; 2. The share of the population considering themselves as well or very well informed of the rights they enjoy as citizens of the Union; 3. The percentage of population aware of the right to vote and to stand as candidate in European election in the Member State of residence, without having the nationality of that Member State.

All the stated goals of the three indicators are likely to be achieved in 2020 (see Table 6).

Concerning the first indicator, the target was almost achieved in 2015 and it is likely that has been achieved meanwhile. However, it is difficult to gauge the specific contribution of the Rights, Equality and Citizenship Programme to such results since this topic is addressed by other EU level initiatives (i.e. Erasmus+). This last consideration holds also for the second indicator.

Concerning the third indicator, despite the drop in 2015 compared to the baseline, the target may still be achieved by 2020 since projects and Commission awareness raising campaigns financed under this specific objective aimed at encouraging the political participation of mobile EU citizens and, therefore, to increase their participation to European elections.

Table 6: Result indicators, targets and actual values

·Specific objective 9: To enable individuals in their capacity as consumers or entrepreneurs to fully benefit from the European Single Market by removing remaining legal obstacles and ensuring enforcement of their rights deriving from Union law, having regard to the projects funded under the Consumer Programme

The Programme's performance concerning this specific objective is measured through four indicators: 1. The perception of consumers of being protected; 2. The percentage of consumers who are aware of their right to keep the unordered product sent to them together with an invoice; 3. The percentage of retailers in the EU who know that including invoices with marketing material (for unordered products) is prohibited; 4. The level of consumer confidence in cross-border shopping, as measured by the percentage of consumers who have at least equal level of confidence in sellers from their own country as from another EU country.

The targets set for the first and the fourth indicators have already been achieved. However, the Programme, through the funded awareness raising activities, can surely contribute to further increasing these achievements. The target of the second indicator may not be achieved, and of the third indicator is likely to be achieved (see Table 7).

Concerning, in particular, the second indicator, the Programme is contributing to achieving the target by implementing training and awareness raising actions targeted directly at companies selling products online in a way that they provide clearer information to their customers.

Table 7: Result indicators, targets and actual values

Moreover, a series of Programme-related indicators are also mentioned in the legal basis of the Rights, Equality and Citizenship Programme 21 .

They provide a basis for monitoring and evaluating the extent to which each of the Programme's specific objectives has been achieved. These are: 1. The number and percentage of persons in a target group reached by the awareness-raising activities funded by the Programme; 2. The number of stakeholders participating in, training activities, exchanges, study visits, workshops and seminars funded by the Programme; 3. The improvement in the level of knowledge of Union law and policies and, where applicable, of rights, values and principles underpinning the Union, in the groups participating in activities funded by the Programme compared to with the entire target group; 4. The number of cases, activities and outputs of cross-border cooperation; 5. Participants' assessment of the activities in which they participated and of their (expected) sustainability; 6. The geographical coverage of the activities funded by the Programme; 7. The number of applications and grants related to each specific objective; 8. The level of funding requested by applicants and granted in relation to each specific objective.

Programme indicators 1, 2 and 4 relate to the outputs and outcomes of funded activities and can be calculated through the final reports of the projects. At the moment, there are not sufficient available data to come to consolidated conclusion on these indicators; Programme indicators 3 and 5 relate to the perception of target groups concerning the activities they participated in and level of knowledge they acquired. These indicators are currently not measured. Programme indicator 6 speaks to the equity in distribution of activities funded by the Programme amongst beneficiaries in the participating Member States, while indicator 7 and indicator 8 are interrelated since they speak of the relationship between the demand for grants and for funding and the extent to which such demand is being satisfied.

The assessment of these indicators is provided in the section on the Implementation state of play and in the section dedicated to answering evaluation questions on the effectiveness and equity of the Programme. However, in general, there is no baseline available for these indicators, as they relate directly to the specific actions of the Programme, and there are also no defined targets, except for the geographical coverage.

Other data, collected during the Interim evaluation, show the current state of play of the Programme’s implementation and its key initiatives in the period 2014-2017.

In relation to the type of partnership developed under the Programme, the data collected in the interim evaluation suggest that, on average, the number of participating organisations per project, between 2016 and 2017, is equal to six-seven for the action grants (including the coordinator). Within the Programme network there are 236 organisations connected out of 440 organisations reported in the database, i.e. 204 are isolated, meaning that one organisation out of two, within the Programme collaboration network, is isolated 22 . This shows that the Programme network is characterised by the presence of a set of actors that has no connection at all (isolated) or belong to very small groups and tend to participate in a small number of projects.

Transnational partnerships, thanks to their networking and interconnectedness between, are essential contributors to the effectiveness of the Programme. Indeed, according to the majority of respondents, the partnerships developed under the Programme have improved their organisational structures in terms of project implementation, sustainability of results and overall capacities. A smaller number of beneficiaries reported also that the partnerships had a positive impact on their fund-raising capacities.

The evaluation of the partnerships formed has shown also an appropriate representation of beneficiaries by type of organisation, but an uneven geographical composition of the participating organisations.

As regards the type of beneficiaries of the Programme, the available data showed that 39% of the total beneficiaries are non-profit organisations, while only 5% are private for-profit entities. As for the Higher or Secondary Education entities category, 92% are universities and 17% of them are projects coordinators. More than half of the universities involved as coordinators or as partners in approved projects come from only three Member States (Italy, Belgium and Spain). The category of Public Bodies is constituted for 14% by Belgian organisations and to the same extent by Spanish organisations, 12% by the Dutch, 10% by French and to a lesser extent by organisations from other Member States. The research organisations, the private for-profit entities and no-profit organisations are distributed fairly evenly throughout Europe, even if, in the last category, there is still a greater presence of two Member States (Belgium and Italy).

According to the evaluation, about 20% of beneficiaries participated in multiple projects within the Programme and the organisations involved in multiple projects tend to be based in Member States with relatively low participation rates, such as Sweden, Latvia, Cyprus and Denmark. This distribution reflects the fact that the Rights, Equality and Citizenship network is more “periphery-based” compared, for example, to the Justice Programme.

Taking into account the involvement of partner organisations, it appears clearly that the bulk of participants to the Programme come only from few Member States and that a balanced geographical spread has still to be achieved (as was the case with the predecessor programmes) (see Figure 2).

Figure 2: Number of applying organisations per country (main applicant or partners)

Source: Sygma and Priamos data

Consequently, this affected the allocation of funds. For example, the largest share of available resources for action grants is attributed to Italian organisations, followed by organisations in the United Kingdom, Spain, Belgium and Greece. Instead, the vast majority of resources available for yearly work programmes of funded operating grants is distributed to organisations based in Belgium 23 .  

With regard to the kind of projects/activities funded, in the period covered by the 2014-2016 Annual Work Programmes, 52 calls for proposals were launched. Most calls covered the specific objectives “prevent and combat all forms of violence against children, young people and women” (15 calls) and “promote the effective implementation of the principle of non-discrimination” (10 calls). This distribution is in line with what was programmed in the annual work programmes. Under these calls, 352 projects have been funded.

The two specific objectives with the highest number of awarded grants are the ones on promoting non-discrimination and preventing all forms of violence, which cover more than half of total grants approved (26 % of the Programme's committed budget). Through them, the Programme mainly financed grants focused on training and mutual learning activities. In relation to the implementation of procurements (which represent about the 23% of the budget), the Programme finances activities focused mostly on the specific objective on the rights of people with disabilities, followed by the specific objective on non-discrimination. Most funded activities concerned the organisation of conferences, workshops, the provision of analytical activities and the support to expert networks.

On average, by looking at the awarding rate of calls for proposals in 2014 and 2015 24 , this ranges between 7% and 27% for almost all the specific objectives, which is quite low. Only under the specific objective on disability, almost all the applications submitted were awarded a grant. The second highest awarding rate, almost 58 %, is observed for 2014 in relation to the specific objective on gender equality. As with the case of the Justice Programme, a drop in the grant applications received (coinciding with the rollout of the Participant Portal 25 ) increased the awarding rate of calls for proposals.

The specific objectives that attracted the highest number of applications are the ones related to the promotion of non-discrimination and prevention of violence that are, therefore, amongst the most competitive ones.

As shown in Figure 3 below, the most covered target groups in the project proposals are young people (including children), women and students.

Figure 3: Text mining of target groups of the Rights, Equality and Citizenship Programme, by specific objective

Source: Text mining analysis conducted on 256 application forms; RCHI (rights of the child), RDAP (To prevent and combat all forms of violence against children, young people and women), RDIS (non-discrimination), RGEN (gender equality), RRAC (to prevent and combat racism), RDIB (rights of the people with disabilities)

The most covered categories to which the Programme’s activities are addressed are children (above all missing children and child victims of violence), (young) women and law-related professionals (in particular lawyers).

4.Method

Short description of methodology

The evaluation process was supported by an external evaluator (Ernst & Young Financial-Business Advisors) 26 , under Commission's responsibility. The criteria used for the evaluation include: relevance, effectiveness, efficiency, coherence, equity, scope of simplification of the Programme and EU added value (see Table 8).

Table 8: Evaluation criteria and corresponding evaluation questions 

The evaluation deployed a series of quantitative and qualitative tools, which were devised to provide detailed responses to specific evaluation questions associated with each evaluation criteria. The evaluation methodology relied on a Mixed Methods approach which integrates and compares quantitative and qualitative approaches, data collection, data analysis and interpretation in order to strengthen the reliability of data, the validity of the findings and recommendations by triangulating multiple sources of information.

The analysis covered the call for proposals and procurement activities financed in 2014, 2015 and 2016 annual work programmes. The 2017 work programme, on the other hand, has been analysed chiefly in terms of design and structure, not in terms of execution.

One key method to analyse the Programme was desk research, which focused on available documents at programme, policy and project level. The desk research was complemented by fieldwork, which was leveraged to collect relevant data and input from stakeholders. This helped to fill knowledge gaps and validate information which was retrieved during the course of the desk research. The following fieldwork activities were performed: 1. A web-based survey was launched to gauge and compare the views of relevant stakeholders (i.e. project applicants, project beneficiaries, Programme Committee Members and additional relevant stakeholders); 2. 33 interviews were conducted with project beneficiaries and Commission officials; 3. Finally, a webinar was held, which provided a chance to discuss preliminary findings with a number of beneficiaries.

The collected data was then used to perform a series of quantitative and qualitative analyses. On the quantitative side, the analysis performed helped provide perspective both at the Programme level, assessing for instance how and where the Programme has deployed the available financial resources, and at the project level, giving a granular view of quantitative indicators provided by individual projects. Also, a Social Network Analysis was performed; this helped achieve a “bird’s-eye” view of the collaborations and transnational networks that were developed as a result of projects financed by the 2014-2020 Rights, Equality and Citizenship Programme. Finally, concerning the qualitative analysis, automated text mining techniques reinforced with machine learning algorithms were deployed to help sift through the large amounts of data concerning individual projects. This was done in light of the need to give structure and find meaning within a large and mostly unstructured dataset (namely project applications and final reports), so as to create taxonomies of the information present in free text fields. Crucially, text mining helped in the analysis to find commonalities and correlations in the data that might not been found had solely manual techniques been deployed.

Robustness and limitations of findings

The evaluation’s findings and conclusions are deemed as credible and consistent according to the methodology. There were, however, some challenges which by their nature could not always be mitigated. 

First of all, only 5 projects had both monitoring and survey data were available which means that no statistical inference has proved feasible and that the triangulation of quantitative data at the project level has been limited. Nevertheless, the analysis of a series of output indicators allowed to have a description of the actions’ features as well as progress in terms of target groups reached and activities carried out.

Moreover, another challenge was that in evaluations of EU legislation “it can be difficult to identify a robust counterfactual situation (i.e. what the situation would be if EU laws had not been adopted)”, which made quantitative analysis problematic 27 .

In view of these limitations, a counterfactual analysis, built on qualitative data, has been considered to be the more suitable method. The analysis of direct questions on behavioural change assumes that the respondents are able to reflect on their behaviour in hypothetical, counterfactual situations and that they are telling the truth to the best of their knowledge. However, as respondents have an interest in the continuation of public support, they might be tempted to over-emphasize the merits thereof by answering strategically 28 . To address the positive bias in answers by beneficiaries and the self-selection bias in answers from applicants who have not received funding, a similar set of questions was asked to applicants having had their proposal rejected. As the features of all respondents (both successful and unsuccessful applicants) are known, this helped triangulate findings gathered from beneficiaries.

Concerning the indicators, those set in the Regulation represent a mixture of output and result indicators which conceptually are separate. In general, each project establishes its own outputs indicators. This means that output data reported is based on applicants' indications in the application form and not on beneficiaries’ actual achievements. Furthermore, since as no targets are set for these indicators, their performance (measured as the distance between the target and actuals achieved) cannot be assessed. Moreover, no targets are set in several specific objectives, such as the specific objective on “preventing and combating racism, xenophobia, homophobia and other forms of intolerance”, on “promoting and protecting the rights of people with disabilities” (apart from the employment rate of persons with disability) and on “promoting and protecting the rights of the child.

In addition, even though the 9 case studies proved to be a useful tool to get a better picture of the Programme’s activities, their representativeness of the Programme as a whole was somewhat limited.

Despite these limitations, the evaluation carried out is supported by a robust evidence base.

5.Analysis and answers to the evaluation questions

The Rights, Equality and Citizenship Programme is performing generally well at mid-term with regard to its specific objectives, in terms of effectiveness, efficiency, relevance, coherence/complementarity/synergies and EU added value, but should do better, in particular, in terms of equity 29 .

5.1 Effectiveness

As the progress in achieving the Programme’s outputs and outcomes has been good so far, it is reasonable to expect that the Programme is making good progress towards achieving also its general objective (namely “the further development of an area where equality and the rights of persons as enshrined in the TEU, in the TFEU, in the Charter and in the international human rights conventions to which the Union has acceded, are promoted, protected and effectively implemented).

However, sometimes is challenging to trace and attribute the changes in the global indicators to the interventions of the Programme. Its contribution to the achievement of the outcomes on awareness of EU citizenship rights is likely to be substantial, as there is the direct impact of the Programme through its focus on awareness-raising measures aimed at enabling change. In contrast, in the achievement of targets in relation to female employment rates, the representation of women in the higher decision-making levels and the employment rate of people with disabilities, the contribution of the Programme activities is more indirect as change in these areas requires a more structural change in the labour markets and economic structures which are outside the direct sphere of influence of the Programme. The programme can be expected to play an important role in supporting such structural changes, but the actual change will depend on many factors external to the Programme, including the business cycle.

Concerning its specific objectives, the progress in the achievement of targets in relation to the nine objectives is uneven. With regard to three specific objectives (namely the ones on promoting non-discrimination, promoting the rights deriving from the citizenship of the Union and promoting consumer rights), the Programme’s contribution is direct and substantial and their targets are likely to be achieved.

The evidence on the impact of the Programme on the achievement of other specific objectives showed that the Programme activities are contributing to the achievement of their targets, but this impact is moderate, due both to the limited amount of Programme resources (especially in the areas of fighting and preventing violence, anti-discrimination and combating racism), as well as change being influenced by many other different factors and policies (such as the differences across national contexts, the national unsupportive policy and legislative framework, the lack of cooperation with government bodies and institutions, the public budget cuts following the economic crises and the consequent austerity measures).

In any case, the key result achieved across the specific objectives is the improved skills and competences of professionals. This is also the key aim of the Programme: to build capacity of the key stakeholder groups to instigate change in the specific policy areas.

Another important achievement of the Programme is the contribution to systemic change by project results, including better tools, procedures, services and policies developed across the specific objectives.

The monitoring indicators selected for measuring the achievements of the Programme are overall adequate, but some improvements could be implemented, for example data collection and processing procedures for the indicators set in the Regulation could be clearer.

Furthermore, the evaluation carried out suggests that the system of indicators could be further enriched to capture the whole range of expected results of the Programme. For instance, indicators capturing the improvement brought about by capacity building activities of public institutions’ staff and non-governmental organisations in preventing non-discrimination or in tackling the cases of violence could be added, as well as indicators measuring the level of awareness of the changes introduced by the 2016 Data Protection Reform. Moreover, there are different specific objectives that do not have set targets or measurement indicators and this should be an area for further improvement.

The indicators relating to the physical and financial progress in the achievement of the outputs of the Programme show improvements. In particular, the reach of participants by the awareness-raising activities can be expected to be above the figures achieved in the predecessor programmes. Similarly, the number of stakeholders participating in the funded activities can be expected to exceed the achievements of the predecessor programmes. This is due to the fact that the communication activities undertaken by the Commission to promote the Programme have been largely successful and the popularity of the Programme has, therefore, increased compared to the predecessor programmes. According to the evaluation, the majority of the beneficiaries considered that the communication activities were clear and understandable, information was relevant and complete and the right audience was targeted to a very high and high extent 30 .

However, efforts need to be maintained to ensure all potential applicants in all Member States are reached. Indeed, one area where the Programme did not improve compared to the predecessor programmes is the geographical coverage of its activities. As in the previous period, the Rights, Equality and Citizenship Programme has been so far dominated by the participants from few Member States. However, a positive development in this respect is the increased participation of Romania and Bulgaria.

The benefit of the Programme has been high compared to the predecessor programmes also in terms of applications received and awards granted. The award rate has also increased slightly compared to the predecessor programmes (from average 6-7 applications per award to average 8 applications per award).

An additional positive development has been the high level of financial implementation of the Programme since the level of funding requested by applicants and granted in relation to each specific objective in years 2014-2015 has been around 90-95% and is slightly higher than in the predecessor programmes. However, stakeholders’ view was less positive about the capacity to target all potential applicants. This confirms that the geographical coverage of the different typologies of beneficiaries of the different Member States is very uneven in the Programme and also the communication activities have not been equally successful from a geographical point of view.

Moreover, based on several indicators and according to the surveyed and interviewed stakeholders 31 , the evaluation showed positive progress in delivering improvements in the level of knowledge of EU law and EU policies, rights and values (given the number and the extent of awareness activities carried out at project level), as well as in the organisational structures, skills and knowledge amongst the groups participating in the activities funded by the Programme.

As mentioned above, the partnerships formed during the Programme activities have had positive impacts on beneficiaries and Programme participants’ skills and capacities. According to the results of the evaluation, the most frequently reported result was the improved skills and competences of professionals, which is highly appropriate given the objective of the Rights, Equality and Citizenship Programme. Important and appropriate have been also the frequent results reported in projects promoting the policy development and quality of services. In addition, 74% of the responding beneficiaries reported having benefitted from the increased capacity of the organisation and 41% from creating new infrastructures/departments within the organisation as a result of participation in the Programme.

Concerning the sustainability of the Programme’s activities after the end of the funding, the majority of surveyed stakeholders considered that actions are very highly likely to be sustained in terms of acquiring new skills and knowledge, increased awareness and the creation of tools and procedures outlasting the projects. Slightly lower was the expectation of the training activities lasting after the project completion. However, only a third of respondents expected the partnerships which were funded during the project to last beyond the funding period (despite the majority view that they have improved the organisation’s capacity building and contributed to the sustainability of the project results). Similarly, low is the proportion of the stakeholders who considered that the partnerships formed during the projects financed by the Programme have increased their fundraising capabilities which is one of the factors to ensure the organisational sustainability. In general, there is very little evidence about the sustainability of project results after the end of funding. There seems to be little systematic planning, monitoring and implementation of activities which would support the sustainability of project results.

Finally, more than 80 % of survey respondents state that the Programme is (very) highly effective in meeting the needs of the relevant target groups. Over 70% of the Programme beneficiaries surveyed in the evaluation also considered that the Programme has been highly and very effective in targeting the right policy areas and the most relevant target groups. This is an important improvement perceived amongst these beneficiaries, particularly as they have been active in both programming periods and are therefore in a good situation to provide a direct comparative analysis.

5.2 Relevance

The Programme is highly relevant to the needs of its stakeholders and beneficiaries, e.g. in terms of knowledge development, training, awareness raising and structural support. Indeed, the stakeholders surveyed in the evaluation were very positive about the relevance of the Programme actions in meeting their needs.

However, the evaluation also found some gaps in relation to the needs of some stakeholders and citizens that could receive greater attention by the Programme, such as women subject to multiple discrimination (e.g. migrant, disabled, older women), children with disabilities, family carers of elderly with degenerative diseases, refugees with psychological problems, children left behind by migrant families in eastern European countries, EU citizens outside the EU.

The needs identified at the time of the Programme’s adoption are still relevant, since, as demonstrated by the evaluation carried out, the achievement of several targets, relating, in particular, to gender equality or rights of disabled people is unlikely as progress has been slow. This means that the overall support needs as identified at the start of the Programme have not been met and continue to be relevant. Also all beneficiaries interviewed agreed that the Programme’s objectives and funded activities are still relevant for the current needs of the groups they target.

The Programme is also flexible enough to incorporate emerging needs, thanks to some calls responding to the real and urgent stakeholders’ and citizens’ needs, other calls formulated in a sufficiently open way to allow a margin of definition of specific activities and others targeting emerging new needs.

Overall, the Programme is appropriately addressing the relevant groups it is expected to target. Indeed, given that the Programme is expected to engage the civil society organisations, its focus on providing funding to non-governmental organisations was appropriate, as well as the involvement of other types of stakeholders to provide analytical activities (especially relevant to the universities and research organisations) or direct awareness raising activities (pertinent to the public bodies).

Another indication of the continuing relevance of the Programme to the current needs is the increase of demand for grants funded compared to previous programmes, especially concerning the specific objectives relating to the violence against children, non-discrimination and combating racism. On average the Rights, Equality and Citizenship Programme is receiving around 900 applications per year. In comparison, on average the predecessor programmes have received around 400 applications per year. For example, the number of applications to the specific objective relating to the violence against children was around 380, whereas the average number of applications to the predecessor Daphne III programme was around 270.

5.3 Efficiency

The analysis of the efficiency is based on a qualitative evaluation of the beneficiaries’ perception of the cost-effectiveness of the actions since the limited data available in relation to the costs and benefits of the Programme’s intervention do not allow for the application of standard quantitative methodologies for a cost-benefit assessment.

The costs relate to the management of the programme at EU level, as well as costs incurred by beneficiaries when applying for funding and when implementing the actions for which funding was received. The costs include direct costs (such as costs of labour, travel costs, subsistence costs, subcontracting) and indirect costs (such as water, electricity/gas and heating, telephone, fax, internet and e-mail).

The information obtained from the survey of beneficiaries was not sufficient to calculate the average amount for each cost category, due to the heterogeneity of the answers provided. For 60% of respondents the main typology of costs relates to staff costs (i.e. accounting and reporting, including financial reporting, and proposal preparation).

The benefits of the Programme relate to development of a European area where equality and the rights of persons are promoted and protected. At this stage, an assessment of the benefits in terms of the result of the Programme can be derived from the results that beneficiaries expect for the specific objectives of the Programme 32 . Examples of concrete benefits derive from the results of the evaluation: 74% of the responding beneficiaries reported having benefitted from the increased capacity of the organisation and 41% from creating new infrastructures/departments within the organisation as a result of participation in the Programme. Moreover, the performance of the projects in terms of training activities is beyond expectations for several specific objectives. In particular, this is true for the specific objective on preventing violence against children, young people and women and this is also more remarkable in the light of the reduced incurred expenditure (at around 60% of the estimated).

In conclusion, according to the majority of beneficiaries and Member States representatives, the costs of the Programme are either proportionate to benefits or outweighing them. The interviews furthermore show that an important reason for this positive assessment is that the Programme allows to experiment innovative approaches and to show national governments the return of the activities performed.

A key achievement of the Programme, compared to its predecessors, has been also the lower demand on beneficiaries in terms of time and financial resources. More specifically, 62% of the responding beneficiaries saved time in preparing proposals because of simpler procedures, fewer overlaps and duplications in procedures, requirements and necessary tasks; 44% reported saving material resources thanks to such provisions in the Programme. In addition, 52% of the responding beneficiaries reported having spent less resources in implementing activities and 54% reported spending less time.

Moreover, according to stakeholders, the changes in the design of the Programme, compared to 2007-2013, are the most important factor for the efficiency of projects in achieving the results. According to the 32% beneficiaries, these changes influence the efficiency of projects to a moderate or high extent.

Beneficiaries consider current funding instruments as adequate for the Programme 33 . However, they consider that there can be scope for using alternative measures, such as microcredit and small loans, as a solution to increase the capacity of participating organisations and potential participants (which is considered as the most influencing factors for the efficiency of projects in achieving their results).

Finally, several beneficiaries pointed out to the issue of the relative long time span between the application and the start of project activities as problematic for their efficiency, since, in this long period, contextual factors may change, requesting an additional effort in adapting the project to the new context 34 .

5.4 Coherence, Complementarity, Synergies

According to beneficiaries, the Programme presents a high level of internal coherence (between the specific objectives and interventions) and a high level of external coherence and complementarity with other EU instruments, programmes and actions, such as the Charter of Fundamental rights, the European Agenda for Justice and the programmes/funds “Europe for citizens”, Horizon 2020, the European Structural and Investment Funds, the Employment and Social Innovation Programme, Creative Europe Programme (in particular MEDIA) and Erasmus+. Beneficiaries especially highlighted a high degree of coherence between the Programme and the European Structural and Investment Funds in the area of anti-violence, rights of the child and non-discrimination. This is probably explained by the fact that the European Social Fund and the European Regional Development Fund (which are part of the European Structural and Investment Funds) can finance projects aimed at the social integration of people at risk of discrimination and of most vulnerable people, such as victims of violence. 73% of respondents reported having established synergies with projects within the Employment and Social Innovation Programme, the European Social Fund, including Youth Guarantee, and Erasmus+. The evaluation showed also that, according to 70% of beneficiaries surveyed, these synergies were established on the occasions of meetings organised by the European Commission. In addition, the webinar confirmed that EU level events represent useful opportunities for beneficiaries to meet EU level policy makers and to network with peer organisations operating in different countries on similar issues. Furthermore, it seems that synergies between beneficiaries of action grants and beneficiaries of operating grants are developed, especially in the field of rights of the child and rights of people with disabilities 35 . Still, coherence with other EU initiatives and a clearer streamlining of concepts and messages could have been strengthened through a more intense consultation and cooperation among the Commission's Directorates-General in the preparation of the calls.

No significant overlaps with other EU funded programmes have been found in the evaluation. This would seem to confirm that specific objectives are more streamlined and clearly defined if compared to the predecessor programmes.

The Programme is highly coherent with the internationally agreed principles, such as the 2030 Agenda for Sustainable Development. The United Nations launched the 2030 Agenda for Sustainable Development (“Transforming Our World”) in 2015. It presents the main international obligations as regards sustainable development. Three goals in the Agenda, in particular, correspond to policy priorities in the 2014-2020 Rights, Equality and Citizenship Programme: 1. Achieve gender equality and empower all women and girls (Goal 5); 2. Reduce inequality within and among countries (Goal 10); 3. Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels (Goal 16). 60% of the applicants and 65% of the beneficiaries surveyed consider the Rights, Equality and Citizenship Programme coherent with the Agenda, mainly through three of its specific objectives, namely on gender equality, non-discrimination, prevention of violence against children, young people and women .

According to almost 90% of beneficiaries, the Rights, Equality and Citizenship Programme is also overall consistent, in its objectives, targets and types of actions, with national policies in the same field. Unsuccessful applicants and Programme Committee Members report that there is an overall consistency between the Programme and national policies, especially in the area of gender equality.

The reason for inconsistencies, specifically reported in relation to the areas of hate crime and hate of speech and non-discrimination, seem to relate to lower Member States’ commitment in these areas covered by the Programme compared to the European Commission’s commitment.

5.5 EU added value

The inherent “European” and transnational dimension is at the core of the EU added value of the Programme. According to the majority of beneficiaries, in the absence of the Programme activities, similar projects would not have taken place or not be possible with the same coverage in terms of beneficiaries and target groups, due to lack of available resources at Member State level. This is particularly relevant for the training actions in the field of prevention of all forms of violence and in the field of data protection. Similarly, EU funding for EU level organisations is particularly vital as national funding for a transnational network would be even more difficult to obtain. In this regard, the Programme supports Member States in progressing towards common standards and practices by helping to enforce EU legislation and policies (such as in the field of data protection).

Analytical activities funded through procurement actions are key to the production of EU added value as these serve to identify gaps in Member States capacities and inspire capacity building activities to fill in those gaps. Moreover, projects financed by the Programme, according to surveyed stakeholders 36 , seem to have higher quality and to be more innovative than those funded at the national level. In particular, these projects, by allowing innovative approaches to be experimented with, can be used to show national governments the benefits of the activities undertaken. For example, trainings at schools in the field of bullying would not apply innovative methods, such as those involving both perpetrators and victims and another example are the projects aiming to pilot the EU disability card. These were supported under the specific objective on the rights of people with disabilities and aimed at setting up the respective EU Disability Card national organisations (governmental or nongovernmental entities) and establishing the respective national package of benefits that Member States are ready to mutualise. In addition to funding the piloting of this innovative action, the national engagement in these projects brought more attention and strength to the issue, triggering the initiative, in terms of impact and visibility. Another example is funding the National Roma Platforms that contribute to increase their number and to strengthen their capacity to play active roles in the coordination and monitoring activities of the National Roma Coordination points. They shall be empowered to participate in the dialogue, cooperation and coordination of stakeholders. 37

Finally, the demand for Programme funding has continued to be high throughout the implementation of the Programme. All specific objectives have a higher number of applications than the number of grants awarded, especially in the areas of fighting and preventing violence, anti-discrimination and combating racism. Indeed, as already mentioned in the section on the "state of play", the most competitive specific objectives are the ones related to the promotion of non-discrimination and prevention of violence.

All these findings suggest that there is sustained demand for EU-level action in the fields covered by the Programme.

5.6 Equity

The Programme has provided specific support to the promotion of the cross-cutting priorities of gender equality, rights of child and rights of people with disabilities.

The issue of the gender mainstreaming is promoted through a dedicated specific objective. In total, the actions under this specific objective received around 12% of the Programme resources. This makes it the third largest specific objective in terms of allocated funding. Furthermore, gender equality specific actions have also been funded under other specific objectives, including the ones on non-discrimination and on combatting violence against children, young people and women. This means that significant financial resources have been dedicated to the promotion of gender equality and gender mainstreaming directly. Gender mainstreaming principles and mechanisms are reflected in all Programme phases of programming, implementation and monitoring but still there is scope for strengthening the protection of women facing multiple disadvantages (since gaps in addressing their needs have been detected) and also for involving more the equality bodies in the setting of Programme priorities.

Concerning the protection of the rights of the child, the Programme has two specific objectives directly targeting the promotion of the rights of the child (the more general one on the promotion and protection of the rights of child and the other to prevent and combat violence against children and other vulnerable groups). Furthermore, also other specific objectives could fund actions potentially relevant for children (e.g. concerning children belonging to ethnic minorities at risk of discrimination or to vulnerable groups such as children with disabilities or protect the personal data of children). However, formally the child-centred perspective is not reflected in other relevant specific Programme objectives and the Regulation also did not establish a formal need to mainstream the rights of the child across all Programme activities (in contrast to the requirement for gender mainstreaming). Nevertheless, the respect of the rights of the child is enhanced by the fact that organisations applying for funding (and any of their partners), which will work directly with children during the project implementation, must provide the Commission with a description of their child protection policy. In general, the mainstreaming of rights of child across all Programme phases could be further improved. In particular, through the involvement of bodies representing the interest of children in the setting of Programme priorities and, according to the majority of the stakeholders, the rights of child should be more present in the Programme calls and be promoted (around 70% of respondents considering this to a very high and high extent).

The Programme envisages a specific objective for the promotion of the rights of persons with disabilities (that accounts for 10% of the Programme funding) and one on the promotion of the effective implementation of the principle of non-discrimination, including based on disability. However, formally the disability perspective is not reflected in other relevant specific objectives and the Regulation did not establish a formal need to mainstream rights of people with disabilities across all Programme activities (in contrast to the requirement for gender mainstreaming).

All applications submitted under the specific objective dedicated to the rights of people with disabilities have been awarded, although it should be noted that the number of applications was relatively small. The mainstreaming of rights of rights of disabled people across all Programme phases needs to be increased, in particular through a direct link with the EU Disability Strategy 2010-2020 38 and a more involvement of bodies representing the interests of disabled people in the setting of Programme priorities. According to stakeholders, the rights of people with disability should be more present in the Programme calls and be promoted (around 70% of respondents considering this to a very high and high extent).

Always in relation to equity, the Programme does not seem to capture the needs of people suffering from multiple disadvantages, as the specific objectives do not support synergies and actions for groups falling under two or more of them.

In addition, co-financing seems to be a barrier for small non-governmental organisations wishing to participate in the Programme.

Finally, as already mentioned, in the future, the Programme should try to distribute its resources in a more balanced manner across the different target groups of beneficiaries and Member States. The unequal geographical representation of Member States was detected also in relation to the predecessor programmes. This means that further effort should be placed in promoting a more equal participation to the Programme funding from partners in Member States that are less likely to participate: such effort shall take the form of increased communication and information activities, as well as better guidance and support to potential partners’ organisations in applying for EU funding and managing EU-funded programmes.

Moreover, the Programme has concentrated its resources in two specific objectives (non-discrimination and preventing violence against children), both of which, so far, have taken up around 50% of overall Programme resources. Comparatively, other specific objectives have received fewer resources (for example, the specific objectives on Union citizenship, data protection and consumer rights have received allocations below 5%). In the absence of the detailed needs assessments by specific objective at the Member State level, it is challenging to draw conclusions on the equity of such allocations. However, the evaluation already shows that the funding does reach Member States with a high concentration of the groups in need of support, but also that various Member States with similar needs are receiving no funding.

To understand better how the Programme promotes equity through the funded activities, participants' data broken down by sex, disability status or age, as required by the Regulation, shall be collected. This is however not yet done.

5.7 Scope for simplification

According to the evaluation, the current direct management mode is adequate given the size and objectives of the Programme.

However, according to the majority of beneficiaries, there is scope for further simplification of the implementation, management and design of the Programme. Among the reasons provided, beneficiaries indicated, in particular, that the financial reporting is too detailed, especially compared to the ones applied within other EU Programmes (e.g. Horizon 2020 and Erasmus+) and underlined the possibility to reduce the current administrative burden in terms of proposal drafting and monitoring and reporting requirements.

Another current key suggestion from beneficiaries concerns the operating grants, whose duration in their view could be extended to cover at least two years (instead of one) in order to reduce the administrative burden in terms of project application and management procedures. However, the annual operating grants allow the Commission to have oversight of the activities of the funded organisations and allow more flexibility to adapt the work of beneficiaries to the emerging needs in their respective fields of expertise.

Furthermore, the absence of standard costs implies sometimes lengthy negotiations among project partners on the costs of the different activities. The introduction of standard costs, at least for some typologies of activities (e.g. training), could significantly reduce the administrative and accounting duties on beneficiaries and on Commission officials involved in management and control.

On the positive side, beneficiaries and Commission officials agreed that the recently introduced system for applications (Participant Portal) goes in the direction of simplifying the application process and improving the collection and aggregation of monitoring data. However, there could be still room for further simplifying the process, particularly in relation to the administrative documentation to be provided and details on work packages.

Moreover, as mentioned above, another key issue according to the beneficiaries, is that most of the funding available in the Programme is relatively inaccessible to small civil society non-governmental organisations, due to the co-financing amounts required for the set size of grants. For small non-governmental organisations providing such amounts is very challenging and, as a consequence, most Programme funding goes to larger non-governmental organisations, even though, within the EU, relevant grass-root activities are also delivered by smaller non-governmental organisations. It is to be noticed, however, that the average amount of grants has been increased to the current level following the ex-post evaluation of the predecessor programmes that found that the dilution of funds amongst many small-scale projects had had limited impact and EU dimension.

Finally, as mentioned above, an element of administrative burden, as widely shared by all beneficiaries, is the excessive length of the period from the application until the start of project activities (which is reported to range between six months and one year).

6. Conclusions and way forward

The Rights, Equality and Citizenship Programme 2014-2020 was launched when the effects of the economic crisis had resulted in a general reduction in the amount of national resources and funding available for social and human rights issues.

In this critical political and economic context, the Programme has proved its EU added value and crucial role in the development of a European area of equality and rights.

The Programme's interim evaluation confirms its relevance for:

·contributing to the promotion of relevant EU values, such as the respect for human dignity, fundamental rights, non-discrimination, tolerance, solidarity and gender equality;

·promoting the exchange of information at EU level as well as mutual learning and the dissemination of practices in the area of protection of rights, citizenship and anti-discrimination;

·reinforcing and supporting the development of a stronger EU citizenship.

The activities and projects financed effectively contributed to its nine specific objectives, as set out in the Regulation. The problems underlying the general and specific objectives of the Programme are still relevant and its intervention strategy appears still adequate to achieving the desired objectives and the needs of the beneficiaries. However, some new priorities emerged and improvement could be made in specific areas of the Programme.

Moreover, the geographic balance is still not good overall, since the Programme is dominated by beneficiaries from only a small number of Member States. This balance could be improved by strengthening the Programme's visibility and improving communication activities in those Member States with relatively low participation rates. Dissemination could equally be enhanced through mutual collaboration between funded projects.

Regarding the efficiency, the current Programme has reduced the administrative burden on beneficiaries compared to its predecessors. Both action grants and operating grants are used efficiently for the separate functions they fulfil and in pursuing the Programme’s objectives. Nevertheless, there is still room for improvements to make the Programme even more efficient in its implementation.

The simplification carried out, thanks to the merger of the three previous programmes, has produced improvements in the quality of the evaluation and monitoring process. However, the revision of the monitoring indicators has been identified as an area for improvement.

At EU level, the Rights, Equality and Citizenship Programme is coherent and complementary with other EU instruments, programmes and actions, with limited risks of duplications or incoherence 39 .

Lessons learned and way forward for the next programming period 40 :

Annex 1: Procedural information

1.Lead DG, Decide Planning/CWP references

Lead Directorate-General: Directorate-General for Justice and Consumers

Decide planning reference:

2.Organisation and timing

The evaluation was carried out between October 2017 and April 2018.

An external contractor, Ernst & Young Financial-Business Advisors, was commissioned under the Framework Contract for evaluation, Impact Assessment and Related Services (JUST/2016/JCOO/FW/JU04/0178) to carry out a interim evaluation of the Rights, Equality and Citizenship Programme (2014-2020). The final report was completed in April 2018. An inter service Steering group led by Directorate-General for Justice and Consumers was set up specifically for this evaluation. The representatives of the following Commission services were members of the group:

Secretariat General, Directorate-General for Budget, Directorate-General for Regional and Urban Policy, Directorate General for Education, Youth, Sport and Culture, Eurostat, Directorate-General for Communications Networks, Content and Technology, Directorate-General for Employment, Social Affairs and Inclusion, Directorate-General for Migration and Home Affairs, Directorate-General for Health and Food Safety, and European Anti-Fraud Office.

Chronology of the interim evaluation:

3.Exceptions to the better regulation guidelines

A public consultation was not carried out specifically under this evaluation, but under the Impact Assessment for the post-2020 Multiannual Financial Framework.

4.Consultation of the Regulatory Scrutiny Board (if applicable)

N/A

5.Evidence, sources and quality

The evaluation was based on evidence from different sources. The complete set of documents that were consulted for this evaluation is listed below:

Legal basis of the Programme:

·Arts. 19(2), 21(2), 114, 168, 169, 197 TFUE;

·Regulation (EU) No 1381/2013 of 17 December 2013 establishing a Rights, Equality and Citizenship Programme for the period 2014 to 2020 (OJ L 354, 28.12.2013).

Commission Implementing Decisions:

·Commission Implementing Decision C(2014)2556 concerning the adoption of the work programme for 2014 and the financing for the implementation of a Rights, Equality and Citizenship Programme;

·Commission Implementing Decision C(2015)1997 concerning the adoption of the work programme for 2015 and the financing for the implementation of a Rights, Equality and Citizenship Programme;

·Commission Implementing Decision C(2016)1677 concerning the adoption of the work programme for 2016 and the financing for the implementation of a Rights, Equality and Citizenship Programme;

·Commission Implementing Decision C(2017)1316 concerning the adoption of the work programme for 2017 and the financing for the implementation of a Rights, Equality and Citizenship Programme.

EU documents :

·Policy-specific reports published by Directorate-General for Justice and Consumers, such as on Roma, gender equality, non-discrimination, Fundamental Rights Agency and European Institute for Gender Equality

·Policy-specific reports and resolutions by the European Parliament

·Political and policy documents, including Council Conclusions by the Council of the EU, and by the EU Member States

·Policy-specific reports published by other stakeholders.

·Ex-post evaluation report of Fundamental Rights Programme (2007-2013)

http://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1488141558521&uri=CELEX:52017DC0069  

·Ex-post evaluation report of Daphne Programme (2007-2013)

http://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1488141666866&uri=CELEX:52017DC0055  

·Ex-post evaluation of the Programme for employment and social solidarity – PROGRESS 2007-2013 and recommendations for the successor programmes to PROGRESS 2014-2020

  http://ec.europa.eu/social/BlobServlet?docId=12610&langId=en

·The Commission Impact Assessment which accompanied the Commission proposal to establish the Rights, Equality and Citizenship Programme 2014-2020, SEC(2011) 1364 final

http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=SEC:2011:1364:FIN:EN:PDF  

·The interim evaluation report of Fundamental Rights Programme, 2011 http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2011:0249:FIN:EN:PDF

·The interim evaluation report of Daphne Programme, 2011

http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52011DC0254&from=EN

·Policy-specific reports published by other DGs of the Commission, specifically by DG Regional and Urban Policy, and DG Employment, Social Affairs and Inclusion

·Monitoring of and reporting by selected beneficiaries receiving operating grants, such as the European Judicial Training Network, the European network of equality bodies and Framework Partners.

·European Citizenship Reports

RIGHTS EQUALITY AND CITIZENSHIP Programme documentation:

·Programming documents: annual monitoring reports (2014, 2015), annual work programmes (2014-2017), programme statements (2014-2018), annual management reports (2014-2017), annual activity reports (2014-2016)

·Project documents (all action grants and operating grants): proposals, grant agreements, interim and, if available, final project reports, project websites, proposals evaluations, projects evaluations, Guide for Applicants, Guide for Evaluators

·Procurements: signed contracts, delivered goods and services

·Responses to the surveys, interviews, and all other consultation activities

·Analysis of public consultation

·The Commission Impact Assessment which accompanied the Commission proposal to establish the Justice Programme 2014-2020 and the Rights, Equality and Citizenship Programme 2014-2020

·EU Justice Agenda for 2020 (COM(2014) 144final

·  http://ec.europa.eu/justice/effective-justice/files/com_2014_144_en.pdf .

Annex 2: Stakeholders consultation

Consultations with the main stakeholders of the Rights, Equality and Citizenship Programme were conducted over several steps:

·A web-based survey targeting the following stakeholder groups:

oprojects applicants;

oprojects beneficiaries;

ounsuccessful applicants;

oEU28 Member States Programme Committee Members;

oOther relevant stakeholders involved directly by the Commission.

This consultation was launched by the external consultant via the EU Survey tool and was open from 23 December 2017 until 6 March 2018. It received 173 responses 41 ;

·33 interviews with 9 Commission officials and 24 selected projects beneficiaries and representatives of target groups were performed between the 8th of January 2018 and the 26th of March 2018. Interviews had a twofold objective: 1. Collecting additional information (from the survey and the desk research) to feed the analysis alongside the evaluation criteria, filling gaps (especially in terms of quantification of costs and benefits) and going more in-depth with specific aspects, such as the identification of best practices; 3. Gathering insight and input to draw conclusions and recommendations on how to improve the design and functioning of the Programmes. Interviews consisted of semi-structured phone interviews. Questions were customised according to the different categories of stakeholders targeted, taking into account their different contribution to the evaluation questions;

·1 webinar, organized in March 2018, to discuss preliminary findings with few selected project beneficiaries. The webinar aimed at assessing at a higher level results achieved by the Programme, testing and discussing preliminary findings and conclusions, also identifying key success factors and areas for improvement in view of the design of operational recommendations. It was organised using Zoom (an online tool);

·A broad public consultation undertook by the Commission in the area of "values and mobility" was available online in 23 official EU languages for a mandatory period between 10 January 2018 and 9 March 2018. The purpose of this consultation was to collect the views of all interested parties on how to make the most of every euro of the EU budget. Consultations have taken place in the context of evaluations of existing EU financial programmes covering several policy areas, including the Rights, Equality and Citizenship Programme. In total, the public consultation received 1839 replies from all over Europe. The respondents had experience with the following EU programmes: 1. Europe for Citizens Programme and /or 2. Rights, Equality and Citizenship Programme and /or 3. Creative Europe Programme and /or 4. Justice Programme. The results of this public consultation have shown that, according to the majority of respondents, "promote European identity and common values", as well as "promote rights and equality", are important common policy challenges to be addressed in each of the four programmes. "Support active citizenship, democratic participation in society and the rule of law" and "promote social inclusion and fairness" appear to be important challenges to be addressed in the concerned programmes as well. Instead, "support innovation", "foster European cultural diversity and cultural heritage", "promote European identity and common values" are considered as policies that fully or fairly well address the challenges by half or more of the respondents. Around 80% the respondents agree that these programmes add value to a large extent or to a fairly good extent to what Member States could achieve at national, regional and/or local levels. The main obstacles identified by the respondents, that could prevent the current programmes/funds from achieving their objectives, are very similar regardless of the programme concerned: "lack of budget of the programmes to satisfy demand", "insufficient support provided to small-scale stakeholders" and "lack of support to first-time applicants" are identified as the main three obstacles. Finally, the respondents agree that "the use of more simplified application forms", "facilitating structured network and partnerships", "facilitating funding for actions cutting across the sectors of action", as well as "better coordination between different programmes/funds", are the main steps to be taken to simplify and reduce the administrative burdens for beneficiaries.

Annex 3: Methods

The following methodological tools have been used:

 1. Data collection 42 ;

·desk research of relevant documents (at programme, policy and project level);

·field research relied on a web-based survey with relevant stakeholders, i.e. (unsuccessful) project applicants, project beneficiaries, Programme Committee members, Commission staff and additional relevant stakeholders, 33 interviews (with 24 selected project beneficiaries and representatives of target groups and 9 with Commission officials) and one webinar to test and discuss preliminary findings and conclusions with few selected project beneficiaries.

·9 case studies of projects connected to the specific objectives of the Programme 43 ;

·analysis of the results of a broad public consultation undertook by the Commission between 10 January and 8 March 2018 and done in the framework of the proposal for the post 2020 Multiannual Financial Framework in the area of "European values and mobility".

2. Data analysis

The contribution of the Rights, Equality and Citizenship Programme to its objectives was assessed in quantitative and qualitative terms.

·quantitative data analysis 44 : in addition to monitoring and survey data, descriptive statistics have been used to analyse the effectiveness, efficiency and EU added value of the Programme; self-reported counterfactual analysis have been used for the analysis of its effectiveness and added value 45 ; exploratory data analysis and econometric modelling have been used to analyse its effectiveness and EU added value 46 ; cost-benefit analysis to analyse its efficiency 47 and Social Network Analysis to analyse its effectiveness and equity 48 ;

·qualitative data analysis 49 : a "text mining" methodology 50 and machine learning algorithms.

Based on this methodology, output indicators (as monitoring data at project level) and results indicators (based on Eurobarometer and other sources) were used. The analysis of the performance at the project level has mainly relied on the answers to the survey.

(1) Regulation (EU) No 1381/2013 of 17 December 2013 establishing a Rights, Equality and Citizenship Programme for the period 2014 to 2020 (OJ L 354, 28.12.2013).

(2) European Commission (2015), Ex-post evaluation of five programmes implemented under the 2007-2013 financial perspective – Final Report. Specific programme evaluation: Daphne Programme, http://ec.europa.eu/justice/grants1/files/expost_evaluations_2007_2013/daphne_programme_evaluation__final_report.pdf ; Specific programme evaluation: Fundamental Rights and Citizenship, http://ec.europa.eu/justice/grants1/files/expost_evaluations_2007_2013/frc_programme_evaluation_final_report.pdf ; Ex-post evaluation of the Programme for employment and social solidarity – PROGRESS 2007-2013 and recommendations for the successor programmes to PROGRESS 2014-2020 http://ec.europa.eu/social/BlobServlet?docId=12610&langId=en  

(3) European Commission. 'Commission Staff Working Paper – Impact Assessment - Accompanying the document Proposal for a Regulation of the European Parliament and of the Council establishing for the Period to 2020 the Rights and Citizenship Programme. Impact Assessment’, SEC (2011) 1364 Final.

(4)

As already mentioned, the Rights, Equality and Citizenship Programme has replaced three programmes in the field of security and citizenship which were in force during the 2007-2013 Programming period.

(5) Given its relatively early stage of implementation, the 2017 annual work programme have been analysed only in terms of design and structure, not in terms of its execution.

(6) For more info on the Europe 2020 Strategy, see  https://ec.europa.eu/info/business-economy-euro/economic-and-fiscal-policy-coordination/eu-economic-governance-monitoring-prevention-correction/european-semester/framework/europe-2020-strategy_en .

(7) European Commission (2015), Ex-post evaluation of five programmes implemented under the 2007-2013 financial perspective – Final Report, ibid.

(8) For more info on the European Disability Card, see http://ec.europa.eu/social/main.jsp?catId=1139 .

(9) For more info on the Data Protection Reform, see https://ec.europa.eu/info/law/law-topic/data-protection_en .

(10)

To see the types of activities funded by the Programme in 2014, 2015 and 2016, see the Mid-term evaluation of the Rights, Equality and Citizenship Programme 2014-2020, Ernst & Young Financial-Business Advisors, Final Report, April 2018, p. 64-65.

(11) For more information, see below p. 19 and see also section 6.2 on "Target groups and beneficiaries" in the Interim report, p. 71 et seq.

(12) For more information on the number of action grants and operating grants awarded per year and on the number of procurement activities, see section 6 "Implementation state of play" in the Interim report, in particular p. 61-63.

(13)  Interim report, ibid.

(14) SEC(2011) 1364 Final, ibid.

(15)  European Commission (2015), Ex-post evaluation of five programmes implemented under the 2007-2013 financial perspective – Final Report, ibid.

(16) Although, the beneficiaries interviewed were not specifically asked to describe their experience with other kinds of Programme management.

(17) In particular, see below pages 15-16.

(18)

See Interim report, ibid.

(19) There are also other EU level initiatives which contribute to such result, such as the Erasmus+ Programme.

(20) The European Commission invited EU Member States to adopt national strategies aim at improving the economic and social situation of Roma and reducing existing disparities with the rest of the population by 2020. For more info, see https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=LEGISSUM%3Aem0049 .

(21) Art. 14 of the Regulation (EU) 1381/2013 of 17 December 2013 establishing a Rights, Equality and Citizenship Programme for the period 2014 to 2020.

(22) Within the Justice Programme only one organisation out of five is isolated. See Interim report, ibid.

(23)  This is due to the fact that operating grants resources are made available to organisations representing European networks, which tend to be headquartered in Belgium (thanks to the proximity with other EU institutions ).

(24) The awarding rate cannot be calculated for 2016 on the basis of the information available.

(25) The Participant Portal is an electronic platform used to manage the applications received for calls published in the framework of the Rights, Equality and Citizenship Programme. According to the evaluation results, the new Portal could be further simplified, in particular in relation to the administrative documentation to be provided, in order to encourage the participation from smaller or less experienced organisations.

(26)

Interim evaluation of the Rights Equality and Citizenship Programme 2014-2020, Final Report, April 2018.

(27) European Commission, Better Regulation Toolbox, p. 344, https://ec.europa.eu/info/sites/info/files/better-regulation-toolbox_0.pdf .

(28)

In interpreting the results, it should be considered that there may be two sources of bias to the answers: 1. Strategic behaviour (beneficiaries may tend to overestimate the effect of the treatment to further legitimise their receipt of funding, while applicants may tend to say to be well off also in absence of funding – also to qualify for the next round); 2. Estimation bias: both beneficiaries and applicants don’t know exactly what the counterfactual situation would be like, so answers are based on their educated guess. This applies especially to applicants who can base their judgment only on the knowledge they acquired in the application phase with no actual exposure to the implementation of the measures.

(29) For more information and evidence, see section 7 "Responses to evaluation questions" in the Interim report, p. 84 et seq.

(30) See Interim report, ibid.

(31) See Interim report, ibid.

(32) For more info on the specific objectives and expected results, see Interim report, ibid.

(33) See Interim report, ibid.

(34) See Interim report, ibid.

(35) For more detailed information on the complementarities of the specific objectives of the Programme with other EU funding, see the Interim report, ibid.

(36) See Interim report, ibid.

(37) There are: national, regional, local authorities, Roma and non-Roma communities including Roma youth and Roma women, Roma and pro-Roma non-governmental organisations (NGOs), private entrepreneurs, trade unions, professional associations, academia, equality bodies, etc.

(38) See http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2010:0636:FIN:en:PDF .

(39) For additional information on the conclusions, see section 8 "Conclusions" in the Interim report, p. 156 et seq.

(40) For more information, see section 9 "Recommendations" in the Interim report, p. 161 et seq.

(41) For more information on the outcomes of the web survey, see Annex 2 "Analysis of surveys and quantitative analysis" in the Interim report, ibid, p. 168 et seq.

(42) For more information, see section 5.3 "Data collection tools and activities" in the Interim report, p. 47 et seq.

(43) For more information, see the Annexe 4 of the Interim report, ibid.

(44) For more information, see section 5.3.1 "Quantitative data analysis" in the Interim report, p. 50 et seq.

(45) A counterfactual analysis is a tool that can be used to distil changes that are caused by a specific policy intervention from changes that would have occurred even in absence of the given intervention (see Interim report, ibid).

(46) Different tools of econometric analysis have been employed in order to examine the results of the self-reported counterfactual questions, as well as to test the responses of different stakeholders to the main question of the survey linked to effectiveness/relevance (see Interim report, ibid).

(47) The Cost-Benefit analysis is as an economic tool applied to public decision−making to quantify the advantages (in terms of benefits) and disadvantages (in terms of costs) associated with a particular project or policy (see Interim report, ibid).

(48) The Social Network analysis enabled to better understand the explicit and implicit links and interrelations among the different beneficiaries and hence the value of the partnership approach that is a distinguishing feature of the Programme (see Interim report, ibid).

(49) For more information, see section 5.3.2 "Qualitative data analysis through text mining" in the Interim report, p. 57 et seq.

(50) Text mining is an analytical methodology that has the aim of structuring large amounts of unstructured texts into categories or clusters of information by broad themes (e.g. project objectives, weaknesses) (see Interim report, ibid).