EUROPEAN COMMISSION

Brussels, 4.12.2018

SWD(2018) 480 final

COMMISSION STAFF WORKING DOCUMENT

Evaluation of the EU Framework for National Roma Integration Strategies up to 2020

Accompanying the document

COMMUNICATION FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

Report on the evaluation of the EU Framework for National Roma Integration Strategies up to 2020

{COM(2018) 785 final}

TABLE OF CONTENTS

1.Introduction

2.Background to the intervention

3.Implementation / state of play

4.Method

5.Analysis and answers to the evaluation questions

5.1 Effectiveness

5.2. Coordination

5.3. Relevance

5.4. Equity

5.5. Coherence

5.6. Efficiency

5.7. EU added value

5.8 Sustainability

6.Conclusions

Annex 1: Procedural information

Annex 2: Stakeholder consultation

Annex 3: Evaluation criteria and questions

Annex 4: Methods and analytical models

Annex 5: Costs and benefits

Annex 6: Objectives and Measures 2011 and 2013

Annex 7a: Intervention logic I

Annex 7b: Intervention logic II-zoom on the EU Framework

Annex 8: Thematic mapping of instruments and processes

Glossary

1.Introduction         

This staff working document (SWD) is a midterm evaluation of the ‘EU framework for national Roma 1 integration strategies up to 2020’ (EU framework), adopted as a Communication 2 by the Commission on 5 April 2011 and welcomed by the Council 3 in May 2011 and the European Council in June 2011 4 .

This midterm evaluation responds to the request to the Commission 5 made in the EPSCO Council conclusions of 8 December 2016 entitled ‘Accelerating the Process of Roma Integration 6 ’. It aims to provide information to the Commission’s departments and to stakeholders to help steer the work for the remaining implementation period of the EU framework (2020) and to prepare for the period thereafter.

The evaluation assesses the relevance, effectiveness, efficiency, coherence and EU added value of the EU framework. It also assesses three further evaluation criteria, namely coordination, equity and sustainability.

In terms of geographic scope, the evaluation mainly focuses on EU Member States, with the exception of Malta, which does not have a Roma community. However, it also covers the enlargement region (Bosnia and Herzegovina, the former Yugoslav Republic of Macedonia, Serbia, Turkey, Albania, Kosovo 7*and Montenegro 8 ).

Regarding the thematic scope, the evaluation focuses on how the EU framework and its objectives are working, notably in the areas of poverty, discrimination, education, employment, health and housing. The evaluation covers in particular:

·changes in the situation of Roma in the areas of poverty, discrimination and access to education, employment, healthcare and housing since the EU framework was adopted and the extent to which changes are attributable to this initiative;

·the existence of preconditions and structures for Roma integration at national level as provided for in the EU framework, such as national Roma integration strategies (NRIS) and national Roma contact points (NRCPs);

·alignment of EU level and national policies and mobilisation of legal and financial instruments (e.g. the European Semester and EU funding programmes) to help meet the objectives of the EU framework;

·the role of the EU framework and NRIS in integrating Roma inclusion into mainstream policies and in stimulating cooperation and dialogue between stakeholders, including civil society;

·the use of EU and national funding to implement NRIS and Roma integration measures;

·the impact of the EU framework on promoting Roma integration in the enlargement region; and

·lessons learnt for the remaining implementation period and for the period after 2020.

The thematic scope does not include an evaluation of other legal, policy or funding instruments that are directly or indirectly relevant for Roma integration, such as the Racial Equality Directive, the European Semester or the European structural funds. For these instruments, only their alignment with the EU framework’s objectives and their mobilisation to advance Roma integration and fight discrimination is assessed 9 . Where available and relevant, the SWD draws on existing evaluations 10 of these instruments.

The evaluation covers the period April 2011 (start of the intervention) to the end of 2017. Comparable 11 data on the living conditions of Roma for the period 2011-2016 for EU Member States and 2011-2017 for enlargement countries have been used.

The evaluation has been informed by an external study requested by the Commission and prepared by ICF Consulting Services Ltd and Milieu (hereafter ‘ICF’). 12

2.Background to the intervention         

2.1. Description of the intervention and its objectives

Wider policy developments at the time of adoption of the EU framework include:

From an economic perspective, the financial and economic crisis wiped out years of economic and social progress, leaving millions of people unemployed, exposing structural weaknesses in Europe’s economy and putting new pressure on social cohesion 14 . In response, in its Europe 2020 strategy adopted in 2010, the Commission proposed five measurable EU targets for 2020 to be converted into national targets. Of these targets, the three related to employment, education and poverty are directly relevant for Roma integration.

From an equality perspective, a decade had passed since adoption of the two anti-discrimination Directives 15 which Member States were required to implement by 2003 and which, among other grounds, prohibit discrimination on grounds of race and ethnic origin. The Framework Decision combating racism and xenophobia 16 was adopted in 2008 and implementation of all three pieces of legislation was being monitored 17 . With regard to other equality policies, the Commission adopted its 2010-2015 strategy for equality between women and men in September 2010. This strategy underlined the cumulative effect of discrimination on two or more grounds, for example, for being Roma and a woman 18 .

Concerning the free movement of people, the dialogue between the Commission and some national authorities on how the relevant EU law should be applied to address the situation of the Roma had intensified in 2010. The Commission had called upon Member States to respect the rules laid down in the 2004 Directive on free movement 19 with respect to the fundamental rights of EU citizens and avoiding discrimination, notably on the grounds of nationality or ethnic group 20 . In addition, transitional arrangements on the free movement of workers that had been agreed in the Accession Treaties of Bulgaria and Romania still allowed the remaining Member States to temporarily restrict workers from Bulgaria and Romania from working in their country. This restriction had an effect on Roma from these two countries, as the last restrictions on the free movement of workers from Bulgaria and Romania were only lifted on 1 January 2014.

International, European and national policies and initiatives for Roma integration started from 2005 and continued in the years preceding the EU framework. These include:

·the Decade of Roma Inclusion 2005-2015 21 ;

·the 2008 Commission Communication 22 highlighting, among other things, that the tools to advance the social inclusion of Roma should be better applied;

·the common basic principles on Roma inclusion attached to the EPSCO Council conclusions on Roma inclusion 23 of June 2009;

·a 2010 Commission Communication focusing on the social and economic integration of Roma in Europe 24 ; and

·the European Parliament’s Resolution of 9 March 2011 on the EU strategy on Roma inclusion 25 .

Despite these efforts, the economic and social marginalisation of Roma persisted and many of the estimated 10-12 million Roma in Europe 26 continued to face prejudice, intolerance, discrimination and social exclusion in their daily lives. EU countries were (and are) characterised by different sizes of the Roma population living in their territories and different starting points 27 in terms of Roma integration policies. The estimated shares of Roma in the EU countries in 2012 28 range from 10.3% in Bulgaria, 9.1% in Slovakia, 8.3% in Romania, 7% in Hungary, 2.5% in Greece, 2% in the Czech Republic, 1.6% in Spain to less than 1% in most of the other countries.

The fact that Roma were marginalised and lived in very poor socio-economic conditions 29 was increasingly seen as an economic disadvantage, resulting in welfare dependency and limiting the labour markets and tax revenues 30   31 , in addition to being a question of European values and human rights

The Commission’s policy response was to adopt the EU framework. This was the first EU initiative specifically concerning Roma which included a follow-up mechanism 32 . The Commission proposed that Member States, in proportion to the size of the Roma population living in their territories and taking into account their different starting points, design national Roma integration strategies (NRIS) and set achievable national goals for Roma integration or, where they already exist, adapt them to meet four EU Roma integration goals:

The framework did not set a specific EU Roma integration goals in the area of non-discrimination. Instead, it underlined that Member States are already under an obligation to ensure that Roma are not discriminated against, as laid down in the Racial Equality Directive, i.e. to give Roma non-discriminatory access to education, employment, vocational training, healthcare, social protection and housing. Its conclusions stress that the framework is complementary to existing EU legislation and policies in the areas of non-discrimination, fundamental rights, the free movement of persons, and the rights of the child.

The framework underlines that non-discrimination is not sufficient to combat the social exclusion of Roma. Member States have the primary responsibility as well as the competence to improve the situation of marginalised populations. Action to support Roma integration lies first and foremost in their hands, taking into account specific national circumstances, needs and required solutions. The EU framework acknowledges this, but due to shared values (such as equality and fundamental rights) and common objectives (such as economic prosperity, social cohesion and solidarity between Member States) 33 , it calls for a European role in policies for Roma integration. A key driving force for the EU framework was transnational mobility of Roma in the context of freedom of movement, making Roma integration a joint EU-level objective.

To achieve each of the four Roma integration goals mentioned above (the specific objectives), the EU framework took a non-binding approach, listing a set of measures Member States should take to make progress. These measures take account of different situations and starting points in terms of national strategies already in place and the size of the Roma population. The EU framework also requests that when developing their NRIS Member States:

·allocate sufficient funding from national budgets, complemented where appropriate by international and EU funding;

·include robust monitoring methods to evaluate the impact of Roma integration actions, and a review mechanism so that the strategy can be adapted if necessary;

·design, implement and monitor the strategy in close cooperation and continuous dialogue with Roma civil society and regional and local authorities; and

·appoint a national contact point for the NRIS, with the authority to coordinate the development and implementation of the strategy or, where relevant, rely on suitable existing administrative structures.

Overall, the EU framework was ambitious in terms of EU guidance and new features (such as EU Roma integration goals and national goals, development of the National Roma Integration Strategies, appointment of National Roma Contact Points). And it was flexible in terms of concrete approaches and responses expected from Member States, taking into account the size of the Roma population and different starting points and having a non-binding approach for national measures.

In addition, the EU framework: (i) encouraged Member States to make use of available EU funds to address the needs of Roma; (ii) stressed the importance of making European and national policies more sensitive to Roma needs, including through empowering civil society; and (iii) highlighted the need to put in place a robust monitoring system. With regard to the enlargement region, the EU framework emphasised that the EU Roma integration goals are equally relevant to enlargement countries. It included three main goals: i) improve the delivery of pre-accession assistance, ii) strengthen the involvement of the civil society, iii) enhance the monitoring.

2.2 Intervention logic and baseline

The EU framework’s theory of change began with the observation that discrimination on the basis of racial or ethnic origin in education, employment, health and housing and other areas was already prohibited by EU law, but that non-discrimination alone is not sufficient to combat the social exclusion of Roma 34 . Therefore, to fight discrimination and combat poverty (general objectives), countries need to develop and implement an integrated and sustainable approach, with efforts spanning different policy areas — education, employment, health and housing 35 in particular (Roma integration goals specific objectives). The expectation was that in the long term this approach would lead to social and economic benefits for Roma and non-Roma 36 alike.

When the EU framework was adopted, no intervention logic had been prepared. It has therefore been developed for this evaluation and has two elements: Annex 7a (intervention logic I 37 ) provides an overview of key instruments, governance structures and activities available to advance Roma integration at EU, national and local levels. Inputs and activities directly related to the EU framework are highlighted by colouring them in red. This annex illustrates that in working towards the Roma integration goals, the EU framework does not operate independently of other legal, policy and financial instruments but mobilises and aligns these instruments and their concrete activities to reach its specific objectives. It highlights that outputs, outcomes and impacts of the EU framework depend on the smooth running of these other instruments as well as on commonly shared external factors. The annex also illustrates that the expected key outputs included increased funding for Roma integration, a higher number of Roma beneficiaries, and more effective targeted and mainstream policy measures. It shows that the key expected outcomes of the intervention were the achievement of the Roma integration goals in 2020, increased capacity of those involved at national and local level to implement Roma integration measures as well as strengthened cooperation between stakeholders; and that the expected key impacts include socio-economic inclusion, less discrimination of Roma and economic, fiscal and societal benefits such as a rise in GDP through better education outcomes and a more skilled workforce. Some external factors influencing effectiveness were already apparent when the framework was adopted. These include the effect of the economic crisis on the labour market, high levels of discrimination preventing some goals from being quickly or fully achieved and the political will required to achieve the EU framework’s objectives. Annex 7b (intervention logic II) focuses more specifically on the EU framework, setting out its objectives, its interaction with Member States and enlargement countries and its role in mobilising and aligning other EU and national policy, legal and financial tools for Roma integration. The expected outputs, outcomes and impacts remain the same.

Information about the baseline situation of Roma in the EU and the enlargement region is still incomplete, with gaps in most Member States, at least for certain subgroups such as migrant or EU-mobile Roma 38 . The fact that no impact assessment was carried out when the EU framework was prepared also affects this evaluation. Furthermore, early studies and reports by European institutions showed a lack of systematic and in-depth knowledge of the situation of the Roma 39 . Most information was collected and made available thanks to efforts of the United Nations Development Programme (UNDP) in central and eastern Europe 40 . The baselines sources mentioned in the 2011 Communication therefore refer to partial studies covering a limited number of Member States 41 . The EU institutions’ first systematic (but still incomplete) mapping of the situation of the Roma (in Member States where this could be done using quantitative methods) followed the adoption in 2011 of the EU framework. This first attempt was the survey of 11 EU Member States 42 by the Fundamental Rights Agency (FRA), which found that in 2011, Roma people were significantly discriminated against across the EU. Eurobarometer surveys from 2012 also provide insights into the prevalence and nature of discrimination faced by Roma. When asked how comfortable citizens in their country would feel if their children had Roma schoolmates, 34% thought that citizens in their country would feel uncomfortable with such situation (28% fairly comfortable and 31% comfortable). The 2011 UNDP/World Bank/EC Regional Roma survey 43  revealed significant gaps between Roma and non-Roma in Western Balkan countries.

The table below shows the situation of EU Roma in 2011, versus the general population, for a number of indicators. As explained above, such data on Roma did not yet exist at the time of adoption of the EU framework. Eurobarometer findings on perception of Roma in the general public were not included as questions in the 2012 and 2015 surveys were different and findings thus not comparable.

Table 1: Comparing the situation of Roma and the general population in 2011

For the Roma integration goals more specifically, this translates into the following reconstructed baseline not available at the time of adoption of the EU framework:

Table 2: 2011 reconstructed baseline for the Roma integration goals

3.Implementation / state of play         

The EPSCO Council conclusions from May 2011 64 welcomed the EU framework and ‘invited Member States to set or continue working towards their goals, in accordance with the Member States’ policies, in the fields of education, employment, healthcare and housing with a view to closing the gaps between marginalised Roma communities and the general population’. As such, the conclusions did not explicitly endorse the EU Roma integration goals. The political importance of Roma integration was recognised by the European Council in June 2011 65 which called for the Council’s conclusions to be implemented rapidly. It particularly requested that Member States prepare, update or develop NRIS or integrated sets of policy measures for improving the situation of Roma within their broader social inclusion policies, by the end of 2011.

By 2012, all Member States except for Malta 66 had submitted their strategies or integrated sets of policy measures 67 . That same year, the Commission presented 68 the results of a first assessment of all NRISs and invited Member States to consider a number of adjustments for the future. The assessment concluded that much more needed to be done at national level. In particular, more concrete measures, explicit targets for measurable deliverables, clearly earmarked funding at national level and a robust national monitoring and evaluation system 69 would be necessary. In its Communication of 26 June 2013 entitled ‘Steps forward in implementing NRIS’ 70 , the Commission stressed the need for further action on the preconditions necessary to promptly carry out measures to speed up Roma integration.

In response, in December 2013 the Council Recommendation 71 on effective Roma integration measures in the Member States was adopted unanimously. The Recommendation, which was the first EU soft law instrument explicitly targeting Roma:

·provided guidance to Member States to make their measures for Roma integration more effective and strengthen implementation of their NRIS;

·recommended that Member States take effective policy measures to ensure equal treatment for Roma and the respect of their fundamental rights, including equal access to education, employment, healthcare and housing;

·indicated how each of the four goals — ensuring equal access to education, employment, healthcare and housing — could be met;

·set out that these goals could be achieved either through mainstream or targeted measures, including specific ones to prevent or compensate for disadvantages, or by a combination of both, paying special attention to the gender dimension;

·reinforced the focus on anti-discrimination, referring explicitly to antigypsyism 72 , and extended the work on Roma integration to new horizontal and structural areas 73 beyond employment, education, health and housing; and

·called on Member States to communicate to the Commission on an annual basis, starting from 2016, the measures taken in line with the Recommendation and progress achieved in implementing the strategies.

For the enlargement countries it is important to note that in 2013, the Communication on the enlargement strategy and main challenges 2013-2014 identified access to civil documentation 74 as a priority for enlargement countries, in addition to the four areas of employment, education, housing and health 75 .

Individual assessments of Member States’ NRIS were presented by the Commission in 2012, 2014 76 and 2016. The 2016 assessment identified some positive trends, such as the growing focus on early childhood education, but underlined that — overall — efforts had not prevented further deterioration in living conditions of Roma or widespread hostility among the general population. In its Communication, the Commission called on Member States to reaffirm political commitment to Roma integration so that the legal, policy and financial instruments in place could be fully used to bring tangible results on the ground 77 78 . In response, the Council adopted Conclusions in December 2016 on accelerating the process of Roma integration 79 . Member States reconfirmed their commitment to the Roma integration process and their determination to ensure that all policy, legal and financial instruments put in place at European and national level be used to close the gap between Roma and non-Roma. The Commission’s 2017 midterm review of the EU framework 80 took stock of progress achieved 81 since 2011 by presenting the findings of the Roma integration scoreboard. It urged Member States to intensify efforts and provided guidance on how to prioritise actions. It announced that a midterm evaluation would be launched in order to analyse the EU framework's achievements and challenges in more depth, consult stakeholders and use external expertise.

The EU framework led to the adoption of NRIS and the establishment of NRCPs at national level (see table below). In total, 15 Member States 82 have adopted strategies and the remaining 12 83 have adopted integrated sets of policy measures within their broader social inclusion policies 84 . The reasons provided by these 12 Member States for not having developed strategies are: (i) the small size of the Roma population 85 ; (ii) the legal impediments to adopting a strategy that targets one specific ethnic group 86 ; and (iii) insufficient evidence of discrimination against Roma, which deemed a strategy unnecessary 87 .

Most Member States that adopted NRIS did so in response to the EU framework. As of 2016, all enlargement countries had adopted an NRIS and action plans and had put in place NRCPs 88 . Some Member States had bodies in place prior to the EU framework to coordinate matters on ethnic minority issues, including Roma issues but they were formally nominated as NRCPs following adoption of the EU framework 89 .

Table 3: Which of the following instruments and governments structures were established at Member State in response to the EU framework for NRIS 90 ?

Source: Interviews with NRCPs; *The table is based on interviews with NRCPs — not all NRCPs responded to these questions and therefore not all Member States are covered. In addition, enlargement countries were only partially covered by interviews. DG NEAR provided additional information.

4.Method                

An external evaluation carried out by a team of consultants from October 2017 to May 2018 provided information for the current midterm evaluation. The methodological approach chosen for the external study is described in detail in Annex 4. In this SWD, the Commission has built on the findings and conclusions of the external evaluation, unless highlighted otherwise in the text. In addition, to prepare the midterm evaluation, the Commission put together a set of draft evaluation questions relating to the five main criteria of effectiveness, relevance, coherence, efficiency and EU added value, as per the evaluation design and road map 91 . The evaluation framework developed by the external evaluator under the guidance of an inter-service steering group (ISSG) built on and further elaborated these questions, also covering data collection and analysis methods. The evaluation also addresses the criteria of coordination, sustainability and equity as the ISSG deemed them relevant for this evaluation 92 .

Data collection and analysis

Evaluation findings are based on the following range of data sources in particular:

·literature review (175 sources 93 )

·phone and face-to-face interviews of key stakeholders in 27 Member States (191 interviews)

·phone and face-to-face interviews of 10 key stakeholders in three enlargement countries

·8 phone and face-to-face interviews at EU level (Commission staff, MEP, EU-level NGO)

·country analysis studies (11 Member States with a total share of 90.2 % of the EU Roma population) pulling together quantitative and qualitative country-specific information

·an open public consultation (OPC), which generated 240 responses and 28 position papers

·a targeted NGO survey, which generated 65 responses

·data provided by Member States in the online reporting tool for annual reporting to the Commission (years covered: 2015 and 2016)

·ESIF programming documents 94 , including through the use of dedicated portals provided by the Commission (SFC 95 2014)

·2011 FRA Roma Survey and 2016 FRA EU-MIDIS II survey

·2011 UNDP/ World Bank/ EC regional Roma survey and 2017 UNDP/World Bank/EC survey on the Western Balkans

·discussions at the evaluation workshop on preliminary findings of this evaluation, at an event discussing the evaluation at the EU Roma Week, at the 2017 European Platform for Roma Inclusion (focusing on the transition from education and employment) and other stakeholder and consultation events.

Method strengths, limitations and mitigation actions

The methodological approach chosen for this evaluation (see Annex 4) ensured the gathering and triangulation of a substantial amount of qualitative data, in particular through the stakeholder interviews, country analyses, the OPC and the NGO survey. This was important given the various limitations which made it impossible to present a quantified counter-factual scenario, i.e. the consequences of not having an EU framework.

a) General lack of Roma-specific quantitative data

The lack of solid Roma specific performance indicators and quantitative data affected this evaluation in several ways, in particular the ability to assess the effectiveness and efficiency criteria. Other limiting factors included the relative unreliability of the data from the online reporting tool (used by Member States to report to the Commission on NRIS implementation) and reduced availability of data on quantifiable costs and benefits (see Annex 4 for a detailed description and the mitigation actions taken).

b) Limitations regarding attribution of outputs, outcomes and impacts to the EU framework

A number of social and economic factors (economic developments, political priorities etc.) unrelated to the EU framework, directly or indirectly impact efforts for Roma integration. In addition, given that the policy areas covered by the EU framework fall largely under the competence of Member States, the aim of the EU framework was to provide guidance for the NRISs and their implementation. It was expected that the EU framework would enable exchange and encourage cooperation of relevant stakeholders as well as mobilise other EU level policy, legal and funding instruments for Roma integration. In turn, it was for the NRISs to mobilise and align the use of national legal, policy and funding instruments, and provide guidance for local-level action plans and strategies, which would interact with local-level policies (see Annex 8 96 ). Therefore, the results and impacts of the EU framework on the situation of Roma can thus only be considered to be indirect and as being influenced by the instruments and actors at national and local levels.

Consequently, a prudent approach was chosen for this evaluation, particularly with regard to efficiency criterion. The evaluation focuses primarily on costs and benefits attributable to the EU framework that have so far been identified. On the other hand, it does not attempt to assess the extent to which EU and national funding in the area of Roma integration is directly justified and proportionate to the long-term benefits from Roma integration in inclusive, quality mainstream education, employment, healthcare and housing 97 .Such an assessment would have needed more solid Roma-specific indicators as well as a clear relationship between cause and effect of interventions.

Validity and reliability of the findings

Despite the lack of a quantified counter-factual scenario and the limitations set out above, the evaluation findings are valid and reliable. Mitigation measures were taken and the evaluation was underpinned by a large body of qualitative evidence which provides a solid basis for drawing conclusions. In particular the workshop disseminating and discussing preliminary findings helped to further verify the correctness of the findings and fine-tune the findings.

5.Analysis and answers to the evaluation questions    

5.1 Effectiveness

The analysis of effectiveness considers the extent to which the EU framework has made progress towards its objectives at the midterm stage, using appropriate points of comparison and assessing the role of the EU framework in delivering the observed changes. The effectiveness of the EU framework so far has been assessed as being largely limited regarding progress towards the Roma integration goals. The assessment is more positive with respect to the setting up and strengthening of instruments and structures and increased cooperation among and capacity of stakeholders.

The assessment has been informed by 14 evaluation questions that looked into the extent to which:

·progress has been made so far on the expected outcomes, in particular towards reaching the Roma integration goals in the areas of access to education, employment, housing and health, as well as the reasons for this progress;

·changes in the situation of Roma in the four areas can be attributed to the EU framework and NRIS;

·the EU framework contributed to the setting up of instruments and structures for Roma integration at EU and national level, increasing cooperation among stakeholders and capacity of national and local actors;

·the EU framework has indeed served as a framework for NRIS;

·the EU framework and NRIS contributed to an increase in the number of and funding for Roma integration measures as well as an increase in the number of Roma beneficiaries.

5.1.1. Progress towards the Roma integration goals, attribution and outlook

Attributing progress towards the attainment of the Roma integration goals is difficult in light of: (i) numerous external factors such as the economic and financial crisis (and the resulting economic and labour market challenges in various countries), the 2015 refugee crisis (which in some Member States led to lowering the priority of Roma integration in the national policy agenda) 98 or the rise of nationalist parties and spread of discriminatory rhetoric; and (ii) the distribution of competences between the EU and the Member States. The policy areas covered by the EU framework fall largely under the competences of Member States. The EU framework therefore provided guidance for NRIS implementation, enabled exchange and cooperation between relevant stakeholders and mobilised other policy, legal and funding instruments for Roma integration. These aspects are elaborated in detail below and in the coherence section. In turn, it was for the NRIS to mobilise and align the use of national legal, policy and funding instruments, and provide guidance for local-level action plans and strategies, which would interact with local-level policies. The effects of the EU framework on the situation of Roma can therefore only be indirect, and mitigated by the instruments and actors at play at national and local levels.

The area of education observed the most pronounced progress 99 . Participation of Roma in early childhood education and care (ECEC) has improved since 2011 in most countries 100 , from 47 % in 2011 to 53 % in 2016. School attendance at compulsory schooling age has improved, from 86 % to 90 % and more specifically in four out of the nine countries surveyed (CZ, EL, HR, PT); early leaving from education and training has decreased from 87 % to 68 % and more specifically in seven out of the nine countries (BG, CZ, ES, HU, PT, RO, SK). However, substantial gaps remain both between Roma and the general population and between countries (ranging from a 28 % participation rate of Roma children in Greece to 94 % in Spain). Attendance of education when in compulsory schooling age is not far from the general population (90 % Roma compared to 96 % non-Roma). The share of Roma students attending classes where ‘all classmates are Roma’ has increased (from 10 % to 15 %) and one to two thirds of Roma children attend schools where most or all children are Roma. For the Western Balkan countries, the situation shows improvement between 2011 and 2017 especially as regards enrolment in compulsory education (ages 7-15). However,, pre-primary school enrolment rates slightly decreased in 3 out of 6 countries (Albania, Bosnia and Herzegovina, the former Yugoslav Republic of Macedonia). Substantial improvement was registered in Montenegro only and appears to be directly linked with the implementation of a UNICEF project. Importantly, in most of the cases, there was no reduction of the gap in education between e Roma and non-Roma 101 .

FRA survey data shows a lack of observable improvement in access to employment across the analysed period. The proportion of Roma who are in paid work as their main activity — including self-employment and occasional work — remains similar (around 25 % of Roma aged 16+). The share of young Roma aged 16 to 24 whose current main activity is cited as being neither in employment, education or training (NEET) has increased from 56 % to 63 %. The gap with the general population remains wide. Until now, the improvements observed in education have barely resulted in higher employment rates, and the transition from education to employment remains a challenge. In respect to the Western Balkan countries, there has been has been a general worsening of the situation since 2011; only the former Yugoslav Republic of Macedonia has shown an improvement related to reduction of unemployment and informal employment of Roma. Most likely reasons for this lack of improvement in employment rates include: (i) the consequences of the economic crisis; (ii) existing prejudices and discrimination; (iii) the absence of adapted support measures; (iv) insufficient qualifications 102 . Country analysis studies and interviews 103 mention that low educational status, the lack of basic skills and discrimination are not tackled effectively. Also, mainstream measures for active employment typically lack explicit references to Roma and do not provide individualised support. Therefore, coupled with adverse economic conditions, the employment measures have failed to effectively reduce the employment gap between Roma and non-Roma. However, given the duration of the educational cycle and considering its long-term qualitative effects, improvement in the employment situation is likely in the medium-to long-term.

Health self-perception has improved but medical health insurance coverage is still limited 104 . The reported average self-perceived health status of Roma is equal to that of the general population. As compared with previous years, more Roma consider that their health is good or very good (from 55 % to 68 %). Nevertheless, healthcare coverage has not increased (around 75 % of Roma aged 16+), with differences between countries ranging significantly from 45 % in Bulgaria to 98 % in Spain 105 . Access to health insurance and health services, including health promotion and disease prevention, poses special problems for many Roma and results in difficulties particularly in the area of sexual and reproductive health services and vaccination 106 . In the Western Balkan countries, general improvements in health insurance coverage and reduced inequality 107 were observed. However, Montenegro had poorer outcomes for both of these indicators and health insurance coverage reduced in Albania. Over 70 % of OPC respondents claimed that their country had not made progress on the implementation of Roma integration measures on healthcare. Most likely reasons for this limited progress over the short time frame are a range of barriers such as language and literacy, a lack of trust, a lack of identification documents and physical distance to healthcare facilities 108 .

There has been very little change in Roma housing conditions. The housing situation for many Roma families continues to be difficult. In 2016, the average number of rooms per person was 0.7 (0.6 in 2011), 30 % of households did not have tap water inside the dwelling (29 % in 2011) and the share of people living in households having neither a toilet, shower, nor bathroom inside the dwelling was reportedly 38 % (36 % in 2011). There is still a gap between Roma and the general population in these areas 109 . However, most households reportedly have an electricity supply (92 % to 96 %).

Most likely reasons identified for the lack of progress are discrimination and lack of funding. The data does not indicate any decrease in discrimination in access to housing. On the contrary, in some countries a notable increase is observed 110 . Many housing conditions are substandard and a gap remains in access to public utilities. Many respondents to the NGO survey (31 out of 65 111 ) believe that since 2011, the NRIS of their Member State did not contribute to ensuring national funding for Roma integration in the area of housing. Forced evictions of Roma from their homes continue to be reported across many Member States and enlargement countries 112 . Regarding the Western Balkan enlargement region, the housing situation of Roma varies considerably between countries, with improvements in Serbia and Bosnia and Herzegovina, and a deterioration in Albania and Montenegro 113 .

The following table presents a shortened and simplified 'nutshell assessment' of progress towards the specific objectives of the EU framework:

Table 4: Progress towards the Roma integration goals

Source: Roma: Based data quoted in FRA (2018) — A persisting concern: anti-Gypsyism as a barrier to Roma inclusion; General Population (Eurostat); Data: Percentage point difference between Roma and General Population indicators for nine Member States covered by FRA 2011 Roma / 2016 EU-MIDIS II surveys.

Modest progress has been observed for the general objective of fighting poverty. According to the 2011 FRA Roma survey, on average 86 % of the Roma population surveyed were at risk of poverty. The situation had improved somewhat by 2016, when 80 % of Roma surveyed were found to be at risk 120 .

Table 5: FRA Roma survey results on poverty

Source: Roma: for 2011, 2016 — EU Roma and EU-MIDIS II survey data quoted in FRA (2018); General population: Own calculations (weighted average) based on Eurostat (t2020_52) EU-SILC 2014,; Eurostat, EU-SILC 2011

Discrimination continues to be of high concern in European societies, despite evidence of some improvement in a few areas. The enforcement of anti-discrimination law regarding Roma in the period 2011-2016 shows a scattered, mixed picture among Member States. However, in most countries the situation has not changed in this five-year period 121 . According to surveys the overall share of Roma who felt discriminated against when in contact with schools (as a parent or student) has not decreased significantly (17 % to 14 %). There has, however, been a significant decrease when accessing health services (20 % to 9 % in the case of men and 19 % to 7 % in the case of women) 122 . In the past 5 years, the share of Roma who felt they were discriminated against when looking for housing has dropped slightly (from 45 % to 41 %) but the extent of discrimination felt by Roma ‘when looking for a job’ and ‘at work’ remains high (40 % and 17 % respectively), even though for the former there has been a considerable drop of 10 %. In 2016, almost every third Roma (30 %) had experienced some form of harassment in the past 12 months 123 that they felt was due to their ethnicity 124 . For 76 % of the respondents, harassment due to ethnicity is a recurring experience. Instances of antigypsyism in the form of hate crime against Roma diminishes Roma people’s trust in their public institutions, in particular in law enforcement and justice, thus undermining social inclusion efforts 125 . Antigypsyism is reported to have increased, fuelled by the stigmatisation of the Roma community by some mainstream political parties 126 . The majority of the OPC respondents identify rising discrimination and antigypsyism as main challenges to Roma inclusion at both EU and national level 127 .

The OPC respondents confirmed that the most significant progress was achieved in education. In health, employment and housing, most respondents see no change. Moreover, while on health more people see improvement than deterioration, opinions are divided on employment, whereas on housing those who consider the situation to be worsening outnumber those that see improvement. On discrimination, the biggest response is ‘no change’, and many more people refer to a worsening situation than to improvements.

Figure 1: OPC— Perceived changes in the situation of Roma since 2011

Overall, 6 years is too short a period to make significant improvements. Structural changes require time and the real impact of progress (including a change in employment trends, resulting from positive steps in education) may only be seen in a generation. However, the evaluation indicates that further progress can still be made in the remaining period provided that efforts to combat discriminatory attitudes and tackle antigypsyism increase. Favourable economic developments, falling unemployment rates and increasing labour shortages in the EU could provide some conditions for progress in terms of improving the employment situation of Roma.

5.1.2 Alignment between the EU framework and NRIS

All Member States 128 and enlargement countries have a NRIS as well as a National Roma Contact Point with the authority to coordinate the development and implementation of NRIS in place. A total of 71 % of OPC respondents stated that one of the main achievements of the EU framework had been that Member States developed NRISs. 

Member States make a differentiated use of these instruments, depending on country specificities. In total, 15 Member States 129 have adopted strategies, while the remaining 12 Member States 130 have adopted integrated sets of policy measures within their broader social inclusion policies 131 . The human and financial resources, as well as the mandate and tasks of NRCPs vary. While with few exceptions they are in charge of coordination and monitoring of NRIS and almost all of them also facilitate civil involvement, fewer are involved in development of other policies and the planning of the use of EU funds, and less than half have regular contacts with equality bodies. The placement of NRCPs within the national government structures also vary: Some are placed in institutions with a wider social inclusion agenda with relatively strong mandate and influence on mainstream policies, while others are in well-established institutions responsible for a specific narrower policy agenda (such as housing). Some NRCPs rather have a weaker technical or policy communication role with other institutions in place coordinating social inclusion. 132

The EU framework called on Member States to set achievable national targets for Roma integration — addressing the four EU Roma integration goals as a minimum — to bridge the gap with the general population. The below analysis shows that in particular those countries with the highest share of Roma fully addressed the Roma integration goals in their NRIS (Bulgaria, Spain, Hungary, Romania, Slovakia), complemented by Finland.

Table 6: Inclusion of national goals explicitly addressing the Roma integration goals

Source: ICF Research based on NRIS and NRCP interviews; complemented with a contribution from DG NEAR

The cases of incomplete alignment with the Roma integration goals above can be explained primarily by the fact that the Council used the following wording to endorse the Roma integration goals: ‘four goals of ensuring equal access in education, employment, healthcare and housing 133 ’. The scope was therefore broader, providing more flexibility to Member States in shaping their national goals, taking into account the size of their Roma population, different starting points and country-specific features, for example in the composition of the Roma population.

The same reasoning applies to the measures taken by the Member States in the four key areas. As shown in the comparison between the 2013 Council Recommendation and the EU framework set out in Annex 6 to this SWD, the Recommendation has widened the set of possible measures per area but strengthened their voluntary nature by stating that the Roma integration goals could be attained by means of measures such as those listed.

5.1.3. Setting up other instruments and structures for Roma integration and increasing capacity of those involved at national and local level

At EU level, this evaluation finds that that the EU framework was effective in introducing a number of instruments (in particular annual reporting by Member States and the Commission, recent reporting by civil society) and EU-level governance structures (for example the NRCP network — see coordination section). Mirroring the Member States’ practices, similar instruments have been introduced in the enlargement region. With regard to instruments, the 2013 Council Recommendation introduced comprehensive reporting by Member States from 2016. The reporting has been put in place but focuses on policy measures taken by Member States, rather than on outcomes. Independent monitoring, in particular through FRA surveys, is still hampered by the relatively low coverage of Member States (although the nine Member States covered in both 2011 and 2016 surveys make up more than 80 % of the EU Roma population). In 2017 the Commission piloted a project to monitor the implementation of the NRIS by civil society in all Member States but Malta 134 . As for the structures, even if some of them predated the EU framework (the Roma task force was created in 2010, while the European Platform for Roma Inclusion met for the first time in 2009 135 ), they were strengthened and made permanent following the adoption of the EU framework. Notably, 66.7 % of OPC respondents recognised that the development of these structures at EU level was one of the key achievements of the EU framework.

136 137 138 139 140 141 142 With regard to the capacity of national actors to implement Roma integration measures, the evaluation shows that today there is better awareness and more knowledge about Roma integration. During the interviews with NRCPs, nine Member States in total reported an increase in the number of government officials working on Roma integration in response to the EU framework. Closer engagement with Roma stakeholders either at local or central level, including via formal Roma platform structures at national level, has helped increase understanding of Roma issues among government officials. However, some NGO representatives voiced the criticism that this increased understanding had not sufficiently been translated into action. Roma participation has increased at national level and most NRISs have systems of consultation with Roma civil society in place. However, there remains a need for more effective Roma engagement at all levels, including from the gender perspective. The quality of civil society participation is still considered inadequate in terms of stakeholder involvement in the monitoring progress. Furthermore, cooperation between Roma NGOs and broader social NGOs rarely occurs in practice, and Roma issues are barely present in the mainstream social inclusion agenda.

5.1.4. Increase in number of and funding for Roma integration measures and in Roma beneficiaries

Finally, the evaluation finds that there is insufficient quantitative data available at EU and national levels to assess whether the EU framework has contributed to an increase in the number of Roma integration measures, the funding attributed and the number of Roma beneficiaries.

As described in the EU added value and coherence sections, the EU framework and EU structural funds have been successfully aligned for the 2014-2020 programming period. In total, 12 Member States (AT, BE, BG, CZ, ES, FR, EL, HU, IT, PL, RO and SK) selected investment priority (IP) 9.2 on socio-economic integration of marginalised communities such as the Roma, allocating it a total of EUR 1.5 billion, or 3.6 % of European Social Fund (ESF) funding. However:

Øthe Investment Priority covers marginalised communities in general (and not exclusively Roma) thus making it more difficult to quantify concrete amounts spent on Roma integration;

Øthe ESF output indicator for participants ‘migrants, participants with a foreign background, minorities (including marginalised communities such as the Roma)’ is not specific enough to provide a reliable assessment of the number of Roma beneficiaries 143 ; and particularly

Øno comparison can be made with the 2007-2013 programming period, which took a mainstream approach with no such IP and indicator.

The evaluation could nevertheless provide some qualitative insights. For example:

Øregarding EU funding in 10 Member States, the consensus among interviewed stakeholders was that it has increased 144 , in particular from the ESF;

Øin the enlargement countries, all respondents expressed a similar view on the impact of IPA funding 145 ;

Øin seven Member States 146 , there was consensus that there had been no increase, while in another eight Member States 147  the government officials interviewed did not know if and how funding had changed;

Øthe majority of OPC respondents (53.5 %) also agreed that the EU framework had contributed to an (increased) allocation of EU funding under ESIF (2014-2020) for Roma integration;

Øregarding national funding, the consensus from the in-depth interviews in 13 Member States was that there had been no increase 148 ;

ØOPC respondents held similar views: 52 % stated that the EU framework had not contributed to (increased) allocation of national funding for Roma integration;

Øin the enlargement region, respondents argued that prior to the financial crisis, budgets for Roma integration were generally on the rise, except in Turkey and Kosovo; and

Øthe crisis provoked a drop in the budgets in most countries in the region that neither the Roma Decade nor the EU framework reversed 149 .

5.2. Coordination

The analysis of coordination considers to what extent actions carried out under the EU framework have been coordinated to maximise effects. It looks at coordination at EU and national level and at cooperation with stakeholders. Coordination is closely linked to effectiveness but treated separately as a key precondition for delivering the EU framework. Coordination of the EU framework has been assessed as being positive overall at EU level and limited at national level.

The assessment has been informed by five evaluation questions that looked into the extent to which:

·coordination mechanisms at EU and national levels have been established in response to the EU framework and whether they are effective; and to which

·stakeholders — in particular civil society — both at EU and national level, are involved in implementing, planning and monitoring the EU framework and the NRIS.

5.2.1. Coordination mechanisms

At the EU level, several coordination mechanisms supporting the EU framework have been set up at EU level.

Box 1: Coordination mechanisms at EU level and in the enlargement region

These mechanisms provide a permanent system of cooperation. During the workshop, several NGOs highlighted the stronger cooperation between all relevant stakeholders resulting from the EU Roma framework, and acknowledged the role of the Commission in achieving it. The network of NRCPs is considered by the NRCPs interviewed as a good opportunity for establishing contacts with colleagues in other Member States dealing with similar situations and for exchanging practices 154 .

At the national level, NRCPs are the main coordinators, having the mandate to coordinate the development and implementation of the NRISs. In addition, the national Roma platforms established in most Member States and all enlargement countries aim to ensure inclusive involvement of and coordination with all relevant stakeholders (such as civil society, public authorities, Roma and non-Roma communities and business). In the enlargement region, since 2011 in all countries except Turkey, the government and the Commission organise a national Roma integration seminar every 2 years. The objective is for all partners (central and local authorities, civil society, international organisations, donors) to monitor progress and agree on the Roma integration priorities to be implemented in the next 2 years.

Many of the NRCPs interviewed in the EU and enlargement countries (11 out of the 19 that provided an opinion) say they have sufficient administrative capacity to effectively coordinate NRIS implementation 155 .

The NRCPs have improved cooperation among public administrations and between public administrations and other stakeholders. However, given their mandate and power, they have a limited influence on the design and implementation of mainstream policies and, more generally, on policies implemented by other institutions at the national and local level 156 . Coordination is not yet well integrated into the policy cycle of planning, budgeting, implementation, monitoring and evaluation. Moreover, NRISs are not well known, even in some cases by key administrative departments in the government. NRCPs have limited power in influencing decision-making processes across policies, which is a major obstacle for their NRIS coordination capacity. In other cases, insufficient coordination seems to stem from a lack of commitment by ministries, the political context or the lack of participation at local and regional levels 157 .

5.2.2. Stakeholder involvement

At the EU level, stakeholder cooperation includes the Council of Europe, the Economic and Social Committee, EU agencies such as the FRA, UN agencies, the OSCE, EEA/Norway Grants, the World Bank, civil society associations and foundations, representatives of municipalities, thematic umbrella organisations and others. In enlargement countries, partners include a number of bilateral and multilateral donors. Stakeholder cooperation and dialogue has been developed through regular events such as the EU Roma Summit, Roma platforms, and consultation meetings with civil society and international organisations. Synergies were achieved through the cooperation with the Council of Europe in different joint programmes 158 . With regard to ensuring their participation in decision-making at EU-level, Roma organisations think that 159 : (i) dialogue with the Roma team and other Commission departments could be more permanent; (ii) information and transparency could be improved; and (iii) the Commission should help facilitate dialogue with the NRCPs.

At national level, stakeholder participation — including of Roma representatives and civil society — has been boosted through the EU framework. New mechanisms introduced are diverse and include: (i) national Roma platforms; (ii) annual national progress reports that are shared with stakeholders; (iii) dedicated advisory committees; (iv) projects to further strengthen cooperation with Roma NGOs; (v) regular consultation meetings with Roma representatives and other stakeholders (e.g. municipalities); and (vi) expert groups including both government and NGOs representatives 160 .

Despite these efforts, weaknesses have been identified with regard to the involvement of civil society in the practical implementation of the NRIS and in the monitoring and evaluation processes. As regards planning and implementation of the ESIF, it is suggested that the degree and quality of stakeholder participation varies across countries, and ‘in many cases the mechanisms and processes for involvement are weak (e.g. informal)’ or limited to specific phases of the policy cycle 161 .

The following chart 162 illustrates this mixed assessment.

Figure 2: Reply to: ‘Please indicate the extent to which you agree with the following statement: The existing mechanisms for coordination and implementation of the NRIS of my Member State allow for effective cooperation and/or consultation with all key stakeholders.’ 

5.3. Relevance

The analysis of relevance looks at whether the EU framework objectives set in 2011 were appropriate to the needs at the time and whether they continue to be relevant today. Relevance of the EU framework has been assessed 163 as positive overall, with some limitations as set out below.

The analysis has been informed by seven evaluation questions that looked into the extent to which:

·the EU framework’s original objectives were appropriate in view of the original needs; and

·the EU framework’s original objectives are appropriate in view of the current needs.

Annex 6 provides an overview of the objectives of the EU framework and the proposed measures for Member States, together with an overview of how these have been adapted and extended by the 2013 Council Recommendation. In general, as explained above, while respecting Member State competences, the EU framework provides flexibility 164 so that its objectives can be adapted to specific national contexts and changes that take place over time. This flexibility implies that the relevance of the EU framework hinges closely on the appropriateness and ambition of the concrete objectives and measures set out in the individual NRISs. The flexibility and the non-binding nature of the EU framework have allowed Member States to follow a tailored approach taking account of specific national contexts when selecting measures. This evaluation therefore assesses the relevance of the EU framework’s general and specific objectives but not the relevance of the individual measures proposed.

5.3.1. Relevance in light of the original needs

The evaluation finds that the original objectives set in 2011 were appropriate in view of the Roma integration needs at the time, both in the Member States and in the enlargement countries. Overall, taking into account the baseline situation described above and the persistent socio-economic exclusion of and discrimination against the Roma, the general objectives on poverty and discrimination and the specific objectives on employment, education, health and housing have been confirmed as relevant for Roma inclusion. This view was clearly shared by the workshop participants when discussing the preliminary findings of this evaluation. During the OPC, an overwhelming majority of respondents (between 86 and 95 % depending on the thematic area) agreed that targeted public interventions are needed in these areas.

The evaluation finds, however, that for education, housing and non-discrimination, relevance of the EU framework could have been stronger.

Education: while relevant, the Roma integration goal in the area of education was considered to be insufficiently ambitious by several civil society organisations and interviewees 165 . Indeed, primary school completion rates of Roma children were already quite high at the time the EU framework was introduced. The relevance of the EU Roma framework could have been increased by other objectives such as: (i) the transition of Roma from lower secondary to upper secondary education; (ii) reducing the share of early leavers from education and training; and (iii) focusing on reducing gaps between Roma and the rest of the population in the other areas.

Housing: the Roma integration goal to close the housing and essential services gap between Roma and the rest of the population is a key factor for improving the integration of Roma into society. While acknowledging that the EU framework’s specific objective was appropriate in view of the original needs, the civil society stakeholders consulted 166 expressed regret that there were no tailored housing objectives across the Member States to address country-specific needs. The 2011 FRA Roma survey and the data collected for the country analysis reports indicate that deprived living conditions, lack of basic infrastructure and segregation among Roma, was worst in Eastern European countries — particularly so in Hungary, Romania and Slovakia. In countries such as Austria and Finland, the problems faced by Roma mainly related to access to housing due to discrimination, rather than to housing conditions or housing segregation 167 .

Non-discrimination: a key conclusion of the workshop was that the EU framework’s relevance was reduced because non-discrimination received insufficient attention. Indeed, the EU framework did not set a specific non-discrimination goal but referred to the existing legal obligations of Member States.

The evaluation also finds that the following factors reduced the relevance of the EU framework in light of the original needs.

a) The scope of the Roma integration goals: Stakeholders consider that some issues that were highly relevant to the needs of the Roma at the time the EU framework was adopted merited more attention. These include gender equality, Roma empowerment and active participation, including civil and political participation 168 . The 2013 Council Recommendation reinforced these objectives by inviting Member States to pay special attention to the gender dimension and by introducing the empowerment of Roma as a horizontal policy measure.

b) The EU framework gives a broad definition 169 of its target population but does not highlight specific groups to be addressed, for example travellers or other subgroups. Despite the political context at the time of adoption of the EU framework (see chapter 2), questions concerning the freedom of movement of Roma EU citizens were not included in the EU framework. Similarly, with regard to non-EU Roma nationals legally residing in the EU, only a short reference to EU policies promoting integration of this group was included in the EU framework. For the enlargement region, access to civil documentation was only identified at a later stage as an area requiring full attention 170 .

5.3.2. Relevance in light of the current needs

The evaluation finds that the original general and specific objectives continue to be relevant today, both in Member States and in enlargement countries. As set out in the effectiveness section, although there is visible progress on education, progress in other areas is slower. Despite better economic conditions, Roma continue to have much higher rates of being at risk of poverty, having lower employment and higher numbers of NEETs, substandard housing conditions and unequal access to healthcare services. The survey of national NGOs also shows that the objectives remain largely relevant to the current needs 171 .

a) With regard to the four Roma integration goals (see effectiveness section for more detail on the 2016 situation of Roma in these areas):

Education: this goal would be more relevant to current needs if it did not focus exclusively on primary education but was extended to aspects such as transition from lower to upper secondary school attendance, early school leaving (ESL) and transition from education to employment. These are areas were the gaps between Roma and non-Roma remain high.

Health: despite some improvements, the analysis finds that the Roma integration goal in the area of health is still appropriate today. Although the gap between Roma and non-Roma largely disappeared for self-reported health status, it remained practically unchanged for medical insurance coverage since 2011 (see effectiveness section). The actual (not self-reported) health status of Roma remains unknown, but EU-wide studies reveal ‘indirect evidence’ such as shorter life expectancy versus the population as a whole, that clearly indicates a health disparity 172 . The Roma population is also disproportionately affected by communicable diseases 173 . Persistently poor living conditions often result in a higher probability of serious illnesses and chronic diseases, even when access to healthcare is provided 174 . Furthermore, little progress has been made on preventative healthcare, as Roma continue to have consistently lower child vaccination rates compared to non-Roma 175 .

Employment: the original objective still corresponds to the current needs, as the employment gap between the Roma and the general population remains very high. Furthermore, nearly half of the respondents to the NGO survey considered that the needs of Roma youth and Roma women were not addressed in employment measures and programmes 176 .

Housing: this Roma integration goal also continues to be relevant to the needs of the Roma. About half of the country analysis studies identify housing as one of the most important areas for further work 177 — a view that is shared by several stakeholders consulted in the Member States and enlargement countries. The OPC confirms the lack of progress between 2011 and 2016, with more than half (57.3 %) of respondents indicating the housing situation has worsened due to higher levels of housing discrimination.

b) In light of the current needs, the evaluation finds that the general objective of non-discrimination needs updating.

Already in 2016 the Commission 178 highlighted the importance of more systematic approaches and greater political will on the part of Member States in fighting antigypsyism. Levels of discrimination remain high (see section on effectiveness) and antigypsyism is a critical issue as recently highlighted by the European Parliament and the Fundamental Rights Agency 179 . Stakeholders, including workshop participants 180 , found that compared to the general objective of combating poverty and the four specific objectives, giving more weight to the anti-discrimination objective is critical to address antigypsyism 181  and its manifestations in hate speech and crime 182 . The OPC results show that, in addition to noting a rise in discrimination, respondents find that little has been done to actively fight discrimination within the four main policy areas over the course of the current EU framework.

Figure 3: Perception of progress/implemented measures to fight discrimination in the four main policy areas 2011-2016

Source: OPC — Public consultation on the Evaluation of the EU framework for national Roma integration strategies up to 2020, 2017.

In order to make more progress towards the Roma integration goals, more efforts are thus needed to fight discrimination and anti-gypsyism if equal access to education, employment, health and housing is to be ensured.

c) The EU framework’s objectives remain relevant also from an economic perspective

The effects of Roma exclusion on welfare dependency, the labour market and tax revenues 183 were already clear at the time the EU framework was adopted. Since then, for the five Member States with the highest share of Roma (BG, CZ, HU, RO and SK) the country reports prepared in the context of the European Semester have repeatedly underlined the risks of adverse demographic trends. Low fertility rates are expected to cause the working-age population to shrink, leading to difficulties with labour supply, shortages of skilled and unskilled workers. These factors are projected to hamper potential growth in the next years 184 .

In this climate, the economic rationale for providing Roma with skills and integrating them in the open labour market is clear in ageing Central Eastern European societies. Recent research carried out for the European Parliament illustrates the economic impacts of discrimination on grounds of race and ethnicity, ranging from GDP and tax revenue loss to direct costs in mental health provision 185 . Roma integration is also relevant for attaining the Sustainable Development Goals 186 of the United Nations 2030 Agenda for Sustainable Development, as many countries cannot do this without making progress on Roma inclusion 187 .

5.4. Equity

For this evaluation, the analysis of equity considers whether the EU framework has the capacity to address the needs of subgroups among Roma. Equity is closely related to effectiveness and relevance (hence its inclusion here directly after the effectiveness and relevance chapters), but looks more deeply into the needs of Roma subgroups. This analysis is important as ‘the Roma’ targeted in the EU framework may experience different forms of disadvantage which need to be addressed. The analysis is therefore presented separately. Equity of the EU framework has been assessed as limited overall.

The assessment was informed by four evaluation questions that looked into the extent to which the EU framework addresses the needs of subgroups among Roma. The evaluation concludes that the EU framework, as adopted in 2011, does not sufficiently address the needs of specific subgroups. The country studies and the targeted stakeholder consultations reveal that whenever special attention is given to vulnerable Roma subgroups at national level, Roma children, youth and women are most likely to be included 188 , and Roma with disabilities to a much lesser extent 189 . Various sources, including contributions during the workshop, the country studies and interviews with national stakeholders, indicate that several other subgroups currently receive insufficient attention, if any at all, in the EU framework and at national level. These include Roma migrants 190 , LGBTI Roma 191 and Muslim Roma 192 . Insufficient attention is also given to the distinction between urban and rural Roma 193 .

With regard to the three subgroups, the EU framework only very briefly refers to the situation of Roma women, pointing out their lower employment rates and the need for better access to quality healthcare. This was partly rectified by the new orientation given in the 2013 Council Recommendation, which also speaks about the need to fight violence, including domestic violence, violence against women and girls, trafficking in human beings, underage and forced marriages, and begging involving children. However, the absence of an explicit focus on gender equality in the EU framework has meant that the NRISs lack indicators and targets to tackle Roma women’s specific situation, and also lack the political commitment at national level, to take a more proactive approach to addressing gender needs 194 . Many NRIS either take a gender-neutral approach 195 or they lack explicit gender measures in nearly all areas except for health 196 . Some enlargement countries provide a positive example, such as the former Yugoslav Republic of Macedonia, where a national plan for strengthening the position of Roma women includes the goal of Roma women and girls’ enjoyment of their human rights without discrimination 197 .

Among the four areas of the EU framework, it is in the employment area that the gap between Roma men and Roma women has increased between 2011 and 2016, both in terms of employment rates and NEET rates.

Table 7: Difference between male and female experiences in employment (2011-2016)

Source: Unpublished information from FRA Roma Survey (2011) and EU-MIDIS II survey presented by FRA to Commission services in February 2018

As set out in the relevance section, stakeholders consider that gender equality was highly relevant to the needs of the Roma at the time the EU framework was adopted and thus merited more attention 198 . The above findings in the area of employment confirm this view.

Both Roma EU-mobile citizens and Roma third-country nationals are barely considered in the EU framework and NRIS of reception countries 205 . We must remember that restrictions on the free movement of workers from Bulgaria and Romania after EU accession were lifted in 2014, and while this was not specific to Roma, it also affected them. Since that date, a significant number of people from these countries moved to EU-15 Member States 206 . The 2013 Council Recommendation already highlighted the need to ‘respect the right to free movement’ of EU citizens and ‘the conditions for its exercise, including the possession of sufficient resources and of a comprehensive sickness insurance cover, in accordance with Directive 2004/38/EC of the European Parliament and of the Council, while also seeking to improve the living conditions of Roma and pursuing measures to promote their economic and social integration in their Member States of origin as well as their Member States of residence.’ 207 With regard to Roma who are third-country nationals staying legally in a Member State, the 2013 Council Recommendation states that they can also be put in a vulnerable position, particularly when they share the same poor living conditions as many Roma who are citizens of the EU, while also facing the challenges of many migrants coming from outside the EU. Country studies confirm the high level of discrimination faced by Roma migrants and EU-mobile Roma for being Roma and migrants at the same time, their precarious living conditions and issues related to their residence status 208 . Several interviews with stakeholders have highlighted concerns about Roma migrants and the need to recognise this group as a specific area for action under the EU Roma framework 209 .

5.5. Coherence

The analysis of coherence focuses on how well the EU framework works together with other EU and national instruments. The coherence of the EU framework has been assessed as positive at EU level and as limited at national level.

The assessment has been informed by nine evaluation questions that looked into the extent to which 210 :

·the EU framework is consistent with other EU policies;

·policy, legal and financial instruments at EU level were mobilised and aligned to accomplish the objectives of the EU framework; and

·policy, legal and financial instruments at national level were mobilised and aligned to meet the objectives of the EU framework.

5.5.1. EU level: internal coherence 211

The EU framework was the first comprehensive EU approach to Roma integration across the four policy areas. However, the formulation of the Roma integration goals was not consistent or in line with the SMART criteria (specific, measurable, achievable, relevant and time-bound 212 ). While the goal for education was not particularly ambitious, considering the situation in 2011 213 the goal for housing 214 was very ambitious. The goals were not easy to measure 215 given issues of comparability of data for Roma and for the general population, nor were they adequately specific. The fact that three goals aim to bridge gaps between Roma and non-Roma while the education goal does not relate to gaps, is another inconsistency.

The evaluation also finds it less coherent that in contrast to having Roma integration goals in the areas of employment, education, housing and health, the EU framework did not include a specific non-discrimination goal 216 . Non-discrimination was rather set as a cross-cutting theme 217 , which was identified as a significant shortcoming of the existing EU framework by the European Court of Auditors 218 and others.

However, consistency between the EU framework and the work of the FRA is found to be strong. The FRA has generated valuable evidence and data on living conditions and discrimination of Roma, in particular through its 2011 Roma pilot survey, the 2016 second EU minorities and discrimination survey (EU-MIDIS II) and the Roma integration indicators scoreboard (2011-2016) 219 . In 2012, the FRA, in close cooperation with the European Commission, established a working party on Roma integration indicators to help EU Member States develop indicators to monitor the progress of the NRIS. 18 Member States participated in the process, represented by their NRCPs, which consulted with a variety of national stakeholders including competent ministries, statistical agencies, structural funds managing authorities, regional and local authorities, Roma civil society and communities, and other institutions that have a role in implementing the NRIS 220 . This process resulted in the development of the online reporting system used by the Commission to collect annual information on implementation of the NRISs.

5.5.2 EU level: external coherence

Intervention logic Annex 7a illustrates that progress on Roma integration depends on consistency between the EU framework’s objectives and those of other EU and national policy, legal and financial instruments.

Table 8: Coherence at EU level (partly based on ICF, Chapter 3.3)

Mainstreaming into policies

Overall, progress has been made in mainstreaming Roma integration in EU policies. On employment, education and poverty, 3 out of 5 Europe 2020 headline targets are directly linked to the EU framework targets for Roma integration: (i) the fight against poverty and social exclusion; (ii) raising employment levels; and (iii) reducing school dropout rates while increasing attendance in tertiary education.

Making sufficient progress towards these three Europe 2020 targets requires addressing the situation of Roma 221 . The 2013 Council Recommendation on Roma integration therefore called on Member States to ‘… depending on the size and social and economic situation of their Roma populations, consider making Roma integration an important issue within their national reform programmes or their national social reports in the context of the Europe 2020 strategy.’

Country-specific recommendations (CSRs) have been addressed to Bulgaria, the Czech Republic, Hungary, Romania and Slovakia in the context of the European Semester since 2012. In addition, the ESIF regulations established a link between the European Semester and EU funding in the 2014-2020 ESIF programming period. Member States which at the time of programming had received CSRs on Roma integration were to include the above-mentioned IP 9.2 in their operational programmes (OP) and use EU funds for necessary reforms. This in turn made the Roma-related thematic ex ante conditionality 222 and its fulfilment criteria applicable, closely linked to the objectives of the EU framework.

Other policies also include an explicit link to the EU framework.

ØRoma integration has been fully mainstreamed into European enlargement policy and explicitly identified as one of the key priority areas. Progress on Roma integration is now also fully included in the negotiations of enlargement Chapter 23 ‘Judiciary and Fundamental Rights’. Furthermore, the Commission analyses the state of play by country in the context of the annual enlargement package.

ØRoma integration has been reflected in the EU urban agenda, allowing for a better mainstreaming of Roma integration into urban policies 223 .

ØThe framework is coherent with EU anti-trafficking policies which often address the situation of Roma and their particular risks and vulnerability to trafficking in human beings 224 .

ØIn the area of health, the European Commission’s recent proposal for a recast of the Drinking Water Directive 225 refers directly to the 2011 EU framework.

There are also policy initiatives with no explicit link to the objectives of the EU framework. Roma integration is not explicitly considered in social policy and education initiatives such as the Youth Guarantee 226 , the Skills Agenda, the Alliance for Apprenticeships and the Pillar of Social Rights. These initiatives take a mainstream approach to vulnerable and disadvantaged groups, mentioning Roma specifically either fleetingly or not at all. The non-explicit reference to Roma in initiatives such as the ones mentioned may be the reason why a large share of the OPC respondents (88.9 %) identified the lack of effective mainstreaming of Roma integration in policy as one of the main challenges at EU level in the 2011-2016 period.

Legislation on fighting discrimination and racism

The Racial Equality Directive, together with the Council Framework Decision on combating racism and xenophobia of 2008 and other instruments, has provided a normative ground for monitoring the fundamental rights situation of Roma 227 . The Directive’s implementation has been monitored by the Commission 228 And a review of the enforcement of anti-discrimination law in relation to Roma found that they do not benefit equally from the non-discrimination legal instruments, and a lack of enforcement is particularly evident in the areas of education and housing. Consequently, the Commission initiated infringement procedures over the school segregation of Roma children against three countries 229 and put in place several policy tools to help national authorities implement anti-hate speech and hate crime legislation to combat antigypsyism 230 among other issues.

The EU framework’s socio-economic inclusion standards and the provisions of the EU legislation complement each other significantly in terms of aims and scope. The Commission’s enforcement of the EU legislation can strengthen the EU framework’s policy objectives, while the monitoring and implementation of the EU framework informs the Commission on the state of play of the EU legislation and supports its practical application. About 83 % of the OPC respondents considered the measures relating to the monitoring and enforcement of EU anti-discrimination laws as very useful for increasing political commitment to ambitious public policies for Roma inclusion 231 . However, as set out under the ‘internal coherence’ section, the lack of specific objectives on non-discrimination is seen as a missed opportunity for full coherence.

EU funding

In the 2007-2013 programming period, EU funding regulations targeted disadvantaged groups in general, but not explicitly Roma.

For the 2014-2020 period, however, substantial changes were made. The Rights, Equality and Citizenship Programme (REC) as well as the instrument for pre-accession assistance (IPA), made funding for Roma integration a priority. Going further than this, the ESIF regulations 232 established an explicit link between the EU framework and EU funding.

ØIP 9.2 of the ESF entitled ‘Socio-economic integration of marginalised communities such as the Roma’ was introduced.

ØThe thematic ex ante conditionality 233 ‘a national Roma inclusion strategic policy framework is in place’ was linked to the above IP and three European Regional Development Fund (ERDF) IPs dealing with investment in health and social infrastructure, support for regeneration of deprived communities and investment in education and training infrastructure 234 . The fulfilment criteria 235 for this thematic ex ante conditionality were fully aligned with the EU framework’s objectives.

ØIn 2011 and 2015, the Commission issued thematic guidance on the use of ESIF in tackling educational and spatial segregation 236 . For its internal organisation, it then prepared in 2014, thematic guidance for ESF desk officers on how to operationalise the objectives of the EU framework for the ESI funds 237 .

These measures have been considered to be an 'element of progress' 238 , 'a promising practice' 239 , as 'having provided additional opportunities to align Roma needs with mainstream policies' 240 and as being 'instrumental in mobilising public administrations to work together and connecting Roma integration goals to mainstream policies' 241 . In total, 12 Member States selected this IP (AT, BE, BG, CZ, ES, FR, EL, HU, IT, PL, RO and SK) allocating a total of EUR 1.5 billion, or 3.6 % of ESF funding. Across all ESIF, six Member States reported action plans related to the Roma-specific ex ante conditionality (BG, EL, HU, LT, RO and SK) in their partnership agreements. Hungary and Romania succeeded in meeting the conditions between the submission of the partnership agreement and the OP. Examples of actions related to the Roma strategy are: (i) developing strategies (e.g. RO) or action plans (e.g. LT); (ii) adopting revised strategies (e.g. HU); and (iii) developing an adequate monitoring system (e.g. EL, CZ and BG 242 ).

5.5.3 National level

At national level, limited progress has been made overall.

Table 9: Coherence at national level (partly based on ICF, Chapter 3)

Mainstreaming

The evaluation finds that the EU framework has contributed to incorporating Roma integration into all policies to a limited extent at national level. Evidence of coherence is particularly limited in the health sector. In terms of employment, objectives were generally part of mainstream policies, but often without specific targeted measures or a specific reference to Roma inclusion into the labour market. On housing, Roma inclusion measures have been short-term actions rather than longer-term programmes. There is some reluctance to include specific references to Roma and their specific needs into mainstream policies to avoid singling out a specific vulnerable group 243 . Thematic policies often address vulnerable or disadvantaged groups in general, without referring to the particular needs of specific groups, including Roma. Indeed, a large share of the OPC respondents (88.9 %) identified the lack of effective inclusion of Roma in policies as one of the main challenges at national level.

National funding

In the 11 countries 244 analysed in depth for this evaluation, national funding was allocated for implementing the NRIS, in addition to or topping up EU funding. In the enlargement countries, all but Turkey 245 have allocated specific funding (national/EU) to implement the strategy. In some countries, a consistent approach has developed over time between NRIS and funding mechanisms at national level, either through co-financing or an increase in national budget allocations for Roma integration measures. Quantification of amounts is difficult despite Member States providing the Commission with information on the implementation of their NRIS, including financial data, since 2015, through the Roma integration measures reporting tool. The information is incomplete due to the limitations of the tool explained in Annex 4.

Fighting discrimination and racism

The evaluation finds that the NRISs have contributed to fighting discrimination, hate speech and hate crime against Roma in some Member States, while in others their contribution was minor or non-existent. Overall, combating discrimination has not been at the core of the NRIS. The NGO survey respondents considered that the NRIS of their respective Member State contributed to some extent to reducing discrimination against Roma (40 %), reducing hate speech (50 %), reducing hate crimes (53 %) and improving the enforcement of anti-discrimination legislation (44 %) during 2011-2017 246 . Targets and measures to combat trafficking in human beings are included in the NRIS of only two of the countries researched for this evaluation (BG and HU). However, other countries have taken measures against trafficking in Roma people outside the scope of the NRIS.

5.6. Efficiency

The analysis of efficiency assesses the relationship between the resources used by an intervention and the changes generated by it. The efficiency of the EU framework has been assessed as limited with regard to monitoring and reporting systems. Furthermore, costs and benefits could not be conclusively evaluated.

Efficiency was analysed through five evaluation questions covering the following areas in particular:

·efficiency of the monitoring and reporting system, including scope for simplification; and

·costs and benefits in the context of the EU framework and NRIS.

5.6.1. Monitoring and reporting systems

The EU framework called for a robust monitoring system to be set up based on detailed and complete data as well as benchmarks 247 and announced that the Commission would report annually to the European Parliament and to the Council on progress 248 . It highlighted the importance of ‘Member States contributions based on their own monitoring systems of Roma integration’, assuming that ‘the in-depth monitoring by Member States and stakeholders of the implementation of NRIS is a sound method for enhancing transparency and accountability in order to ensure the most effective impacts of Roma integration.’

The 2013 Council Recommendation stressed that ‘Member States should choose their own monitoring methods, including appropriate methods for any data collection, and possible indicators.’ 249 It recommended that Member States provide their NRCPs with ‘an adequate mandate and resources appropriate to their role so that they can effectively coordinate the cross-sectoral monitoring of Roma integration policies with a view to their implementation, while respecting the division of responsibilities within each Member State 250 ’. Importantly, the Recommendation introduced a more explicit reporting obligation whereby the Member States should communicate to the Commission any measures taken in accordance with the Recommendation by 1 January 2016, and thereafter ‘any new measures taken on an annual basis, at the end of each year, along with information on the progress achieved in implementing their NRIS 251 ’. The Recommendation asked the Commission to ensure that the information provided by the Member States would serve as a basis for the preparation of its annual reports to the European Parliament and to the Council 252 .

The evaluation finds that monitoring and reporting systems have gradually been set up, but they are still weak and need improvement. In particular, countries have monitored and reported on process indicators, rather than results and outcome indicators which would allow the overall effectiveness of the interventions to be measured. An evaluation commissioned by the European Parliament in 2015 253 noted that almost none of the Member States provided details on their progress in implementing the strategy and, wherever references were made, targets, baselines and indicators set by most Member States were generally poor and the monitoring mechanisms weak.

To accompany the new reporting requirement for Member States, the Commission introduced a new online reporting tool (based on the work of the FRA-coordinated working party on Roma integration indicators) to be used by the NRCPs as of 2016 for their annual reporting 254 . The data collected in the online tool includes a description of targeted and mainstream policy measures implemented across 12 thematic areas set out in the Council Recommendation. The tool also gathers data related to those involved in implementation, funding and beneficiaries. The evaluation came to the following conclusions:

·The tool enables a variety of actors and levels of public administration working on Roma integration to collaborate online. This reportedly decreases the amount of time taken by NRCPs to collect and compile data from those involved, and reduces the variety in the responses they receive.

·There is insufficient information relating to the pre-2016 system to allow for a comparison of efficiency.

·NRCPs reported some difficulties including in particular: (i) unavailability of Roma-specific data requested by the tool; and (ii) regional municipal and local levels not feeding the tool appropriately. As a result, the information provided has limited validity.

·NRCPs reportedly attempted to fill data gaps with information from small-scale evaluations and research projects. However, this was considered a time-consuming and inefficient way to collect data as it did not guarantee coverage at population level and comparability of data across Member States.

·Some NRCPs also found that the reporting tool relied too much on quantitative information, whereas additional qualitative and contextual information was also considered to be important.

In addition to the EU framework’s specific reporting requirements, other related EU instruments have reporting requirements. Examples include ESIF reporting 255 , reporting in the context of the European Semester and implementation reports for Directive 2000/43/EC 256 . Member States also have a national reporting requirement linked to the implementation of the NRIS. As in some Member States the NRIS is broader in scope than the EU framework, the national implementation reports are sometimes more comprehensive than the reports submitted to the Commission. The evaluation observes a general absence of coordination of reporting times and formats.

With regard to monitoring systems, as demonstrated in the ‘coherence’ section, the analysis and data collection by FRA is a positive development at EU level.  At national level, the EU framework does not provide any specific guidelines on the kind of monitoring system that should be put in place. The national systems serve the purpose of monitoring NRIS, but their use may be limited in terms of comparability or assessing progress on the EU framework objectives in cases of limited alignment between NRIS and the EU framework 257 .

During the stakeholder interviews, those involved in reporting to the EU expressed the opinion that there could be a reduction in the administrative costs 258 . Nevertheless, substantiating these claims with comprehensive evidence on the actual costs (in terms of full-time equivalents) has not been possible as: (i) only nine NRCPs 259 were able to provide the number of staff and/or dedicate the amount of time required to complete the reporting and monitoring tasks; and (ii) the key difficulty in estimating the time involved in the reporting process is not assessing the time taken by individuals to complete the online reporting tool forms 260 , but the time it takes to collect the required information and the time spent by other public servants not in directly reporting, but in summarising and presenting the information to be included in the report.

According to the NRCPs, the main areas for improvement are:

·the timelines for EU reporting could be better aligned with Member States’ reporting deadlines;

·given that several Member States do not have Roma-specific data, the efforts required to supplement the reporting with qualitative information are disproportionate, without clear added value to the monitoring process;

·the frequency and level of reporting could be linked to the size of the Roma population in the Member State, meaning that Member States with small Roma populations and few policies could report less frequently;

·a ‘materiality’ threshold that eliminates reporting on small projects that do not contribute significantly to the objectives of the EU framework could be introduced, setting a different format or frequency for small projects whose only added value is the ‘exchange of experience’; and

·reducing reporting frequency or replacing annual reporting with a strategic cycle reporting would reduce the administrative burden of NRCPs and be more appropriate considering that the vast majority of NRIS policies are long-term ones. For example, a staggered reporting model could provide more meaningful and comparable data.

5.6.2 Costs and benefits in the context of the EU framework

It was not possible to conclusively evaluate costs and benefits for a number of reasons. Firstly, most known or potential costs and benefits cannot exclusively be attributed to the EU framework or they are not quantifiable. Secondly, where costs are quantifiable in principle (for example in the case of ESIF investment), information about the share of funding spent exclusively on Roma is not sufficiently available. This is particularly due to the absence of Roma-specific indicators because of the broad consensus that the correct approach is to target funding at Roma explicitly, but not exclusively 261 (i.e. not excluding non-Roma in a similar situation). Lastly, output values in terms of Roma beneficiaries and participants are not systematically available, also because of the lack of specific indicators.

With regard to benefits, in the longer term, progress made in education, employment, housing and health can potentially not only further reduce poverty among Roma, but is also likely to have fiscal benefits such as contributions to national budgets (increased tax payment, social security, indirect taxes, such as VAT or excise tax 262 ) or impact the use of public goods and services (reduced take-up of social welfare, unemployment or child support benefits). The integration of Roma could be positive for the labour market (improved efficiency through a decrease in labour and skills shortages 263 ) and the economy 264 , in particular GDP (albeit with a significant delay to labour gains stemming e.g. from educational progress or desegregating housing policies).

In the light of the above limitations, this evaluation:

I)assesses those costs and benefits that are directly attributable to the EU framework (see below);

II)provides information about quantifiable benefits gained so far through having a higher share of Roma in paid employment and a lower share of Roma NEETs where applicable (as these changes are not directly attributable, they are not presented here but in Annex 5); and

III)does not assess the extent to which EU and national funding in the area of Roma integration is justified and proportionate to the long-term benefits from Roma integration in quality inclusive mainstream education, employment, healthcare and housing (original evaluation question 265 ).

With regard to the costs and benefits identified and attributable to the EU framework, the evaluation finds that there are administrative costs (staff) for both the EU and Member States:

ØAt EU level the administrative costs relate in particular to the Commission’s Roma team with its current staffing of six full-time equivalents. Additional input and contributions are made via the Commission’s internal Roma task force involving a number of Commission departments.

ØOther costs directly stemming from the implementation of the EU framework at EU level are estimated to be around EUR 800 000 per year, notably covering the organisation of bi-annual NRCP meetings and annual EU Roma platform meetings, co-financing national Roma platforms and development and maintenance of the reporting tool.

ØAt national level, administrative costs are estimated at around EUR 3.3 million annually 266 , covering staff required to implement the EU framework and report on the NRIS.

In terms of benefits, the evaluation found that the EU framework provides EU added value (see section on EU added value) — such as: (i) putting Roma inclusion on the EU and national political agendas; (ii) raising awareness of Roma issues including in countries with smaller Roma populations; and (iii) ensuring a continuous focus on Roma integration through its multiannual character. This EU added value brings a benefit to the EU as it is being recognised as promoting equality and Roma integration. The evaluation also identifies benefits for EU, national and local stakeholders working on Roma integration. It found that increased cooperation between and capacity of stakeholders, in particular through the setting up and strengthening of instruments and structures for Roma integration, but also through their increased awareness of Roma issues. Increased awareness and cooperation, including with Roma civil society, can in turn improve the efficiency of policies.

Other costs and benefits of Roma integration are not directly or exclusively attributable to the EU framework or not yet measurable because they are long-term costs and benefits. These are presented in detail Annex 5.

5.7. EU added value

Analysis of EU added value looks at changes triggered by the EU framework over and above what could reasonably have been expected from stakeholders alone or from no action at all. The evaluation highlights the positive EU added value of the framework 267 .

The assessment of the EU added value of the EU framework has been informed by four evaluation questions covering the extent to which:

·EU action is necessary to stimulate, complement, leverage and create synergies with national action;

·EU action in areas such as policy guidance and knowledge exchange bring added value; and

·there would be consequences of stopping targeted EU action.

This evaluation finds that EU action is necessary and has provided added value to national Roma policies and their implementation at the political level and in terms of governance and financial support.

At political level, the EU framework has stimulated political action and encouraged political will to improve Roma integration. In some countries, Roma integration has found its place on the national political agenda thanks to the steer from the EU political agenda 268 . The framework also raised attention on Roma issues in countries with a smaller Roma population, mainly in northern and western European countries, and in countries with a Roma population including high shares of EU-mobile and third-country Roma. In this sense it laid the foundation of an incipient European agenda. In those countries with larger Roma populations or previous engagement in the Roma Decade 269 , the EU framework strengthened existing structures. The perspective of a multiannual policy framework ensures a certain stability at national level, especially in the face of changing political priorities 270 . More than three quarters of the 242 OPC respondents recognised that thanks to the EU framework Roma inclusion was higher on the EU policy agenda and that more funding was available for projects promoting Roma inclusion 271 . The view that the EU framework has been ‘the key driver forcing national governments to act’ 272 is confirmed by civil society. Many NGO participants in the workshop stated that none of the Member States would be where they are now without the EU framework 273 . This was also confirmed in a number of — but not all — national-level interviews, pointing to the importance of EU action in fostering political ownership and will to act on Roma inclusion 274 .

In terms of governance, the EU framework: (i) provided policy guidance: (ii) supported greater accountability and transparency; and (iii) created opportunities for mutual exchange and coordination, in particular through the meetings of the NRCP network and the EU Roma platform. The framework encouraged a move away from a compartmentalised approach to an integrated approach covering all core policy areas 275 . The framework’s more comprehensive approach allowed Roma integration to be addressed from various policy perspectives and for central, regional and local authorities to work together on Roma issues, thus helping to spread ownership of the Roma inclusion issue 276 . The EU framework also encouraged Roma representatives to become more actively involved in the policy and decision-making structures directly affecting Roma 277 . Monitoring and reporting was highlighted as being important to regularly take stock of progress and evaluate which measures are working better than others 278 . The monitoring and reporting system also allows easier comparisons with other EU countries, giving a certain competitive and comparative aspect to the process 279 .

At the financial level, EU added value was created through the close link made between the European Semester, ESIF in the 2014-2020 programming period and NRIS (see section on coherence). In many countries, much of the funding for Roma integration has indeed been ESIF or IPA funding, while there seems to be more reticence to invest domestic money for Roma integration, at least beyond national co-funding required for ESIF. However, this is difficult to assess due to the limitations set out above. In many cases, ESIF helped scale up existing projects, intensify actions and improve the quality of interventions 280 .

Consequences of stopping targeted EU action

The evaluation finds 281 that stopping targeted EU action on Roma integration is likely to result in reduced political commitment and focus on Roma integration in both the EU and enlargement countries and a significant decrease in available funding. Stopping EU action is also likely to result in less and looser monitoring and reporting of the situation of Roma. Less policy focus, funding and monitoring is likely to worsen the living and working conditions of Roma, not only stopping but also potentially reversing the current trend towards an improvement of the Roma situation.

For a majority of the OPC respondents (almost 60 %), the EU still has a major role to play in supporting national, regional and local authorities as they cannot effectively improve the situation of Roma on their own. This opinion is shared by the public institutions, NGOs/think tanks and citizens who expressed their views. The main reasons provided relate to the perceived role of EU funding programmes 282 and to the importance of monitoring the Roma situation as a critical driver for reform and political commitment at national level. National-level interviews and country analysis studies showed that political focus on Roma inclusion risks being reduced in the absence of an EU requirement for reporting and monitoring 283 . Another possible consequence would be that Roma inclusion could become more dependent on national political parties in government and their political priorities. Interviews and country analysis studies confirm the likely consequences of discontinuing EU action in terms of fewer measures funded and less focus on Roma inclusion 284 . There is a risk that stopping EU action could result in a less comprehensive and coordinated approach to Roma inclusion, narrowing the scope, reaching fewer beneficiaries and moving backwards towards small-scale and short-term projects and initiatives  285 .

5.8 Sustainability

The analysis of sustainability assesses the extent to which effects stemming so far from the EU framework are likely to last after it ends. This separate evaluation criterion is intended to feed into a potential post-2020 initiative.

The sustainability of the EU framework is assessed as limited overall. The assessment has been informed by four evaluation questions that looked into the extent to which:

·measures have been taken at EU and national level to ensure continuity and sustainability; and

·outcomes are expected to continue after 2020.

The evaluation finds that, at present, the effects of the EU framework are unlikely to last after 2020. Most of the current national policies and structures created around the NRIS (NRCP, systems of coordination, monitoring and reporting, systems of consultation with civil society and Roma organisations) would stop or would be less operational and become more symbolic if the EU framework did not continue after 2020 286 . A longer duration is needed to ensure the sustainability and long-term impact of policies and to consolidate working structures. At national level, substantial and continued political commitment, public action and funding of NRIS are still needed to maintain the improvements in integration and lower the risk of some programmes disappearing 287 .

The evaluation was based on three main prerequisites for sustainability: (i) sustainable funding; (ii) incorporating Roma integration into policy-making; and (iii) community engagement (involvement of Roma, NGOs, regional and local authorities).

Sustainable funding

The EU framework stated that ‘Member States should allocate sufficient funding from national budgets, which can be supplemented, where appropriate, with EU and international funding’ 288 .

The evaluation notes that funding is not yet sustainable. Many stakeholders questioned the extent to which resources allocated to support the implementation of NRIS are sustainable in the long-term 289 . The framework encouraged the use of national funding for implementing measures supporting Roma integration, but Member States and enlargement countries still largely relied on European funding to implement the NRIS 290 .

In particular, phasing out Roma integration from ESIF would result in the discontinuation of many programmes, as many of the new initiatives addressing Roma inclusion have been developed due to the EU framework and its support by ESIF, especially the ESF. In the countries where the majority of Roma are living, many programmes and projects are highly dependent on the ESIF fundign, with many receiving modest co-funding from national budgets (around 20%). Consequently, much of the action consists of transitory, one-off projects rather than long-term and sustainable programmes. Changes in political leadership, new priorities or budgetary constraints could put the sustainability of such projects at risk if not supported by an EU framework. Indeed, many programmes and projects initiated prior to the EU framework were only short-term and relied on annual extensions, with interruptions in some cases 291 .

Mainstreaming Roma integration

To incorporate Roma integration into general education, employment, housing and health programmes, Roma need to benefit from general policies. This includes removing their specific barriers in gaining access to public services. A policy of inclusive mainstream services can help ensure that measures continue in the future (even if specific strategies on Roma inclusion are discontinued). Projects with a mainstream approach are more likely to be sustainable than targeted ones. Regular monitoring and periodic evaluations are the main means available to ensure that sustainability is checked at policy implementation level. While in principle Member States have reporting obligations on the implementation of the EU framework, in practice there are no mechanisms in place to ensure a sustainability check at policy implementation level because of data and information gaps as detailed in the reporting and monitoring section above.

The evaluation finds that insufficient measures have been taken so far to ensure that Roma integration is included in mainstream measures in the four key areas. Positive outcomes may be long-lasting in education, where inclusive reform of mainstream policies have been initiated 292 . However, long-term benefits in employment, health and housing are more questionable as initiatives are usually short-term, ad hoc and have not brought the institutional or cultural changes required to be sustainable 293 .

Community engagement

Effective community engagement can help to identify funding priorities 294 , empower local communities, provide critical feedback and increase accountability for Roma inclusion policies. However, to date, not enough effort has been made to provide sustainable support for building the capacity of Roma grassroots organisations — a factor that in practice is reducing their participation 295 .

6.Conclusions         

Effectiveness of the EU framework has been assessed as limited regarding progress towards the Roma integration goals. While there is some progress in the area of education, progress was limited in the area of health and lacking in the areas of employment and housing. Overall, attributing the progress/lack of progress to the EU framework is difficult in light of external factors such as the economic and financial crisis or the 2015 refugee crisis and given the distribution of competences between the EU and the Member States. Also, structural changes require time and the real impact of progress (such as positive initial changes in the area of education) will only be seen later. Monitoring progress in Roma integration continuous to be hampered by a lack of reliable data running over time. The EU framework's objectives could have been designed in a more specific and measurable manner for them to serve as both political signals and quantifiable policy goals.

The EU framework has been more effective in setting up and strengthening instruments and structures for Roma integration, increasing cooperation and capacity of people and institutions working on Roma integration. This is particularly true at EU level, where coordination has been assessed as positive overall. At national level, an obstacle is the limited influence of the National Roma Contact Points (NRCPs) on the design and implementation of mainstream policies and decision-making processes across policies. Nevertheless, at both EU and national level the EU framework was found to have stepped up stakeholder cooperation. NRISs have systems of consultation in place with Roma civil society. However, the need remains for more effective, transparent and inclusive Roma engagement and participation at all levels 296 .

Relevance of the EU framework’s original objectives in view of the original and current needs has been assessed as overall positive, however with limitations. The framework was and remains relevant regarding the needs of Roma, and also in terms of social and economic needs due to increasing labour shortages in countries with the largest shares of Roma. However, some limitations were identified, namely: (i) the relatively low relevance and narrow focus of the Roma integration goal on education, with its emphasis on completing primary school; (ii) the absence of a specific non-discrimination goal alongside the four Roma integration goals and insufficient attention to antigypsyism; (iii) the limited attention to the specific disadvantages of Roma women, children and youth and other groups within the Roma population, including EU-mobile Roma and Roma who are non-EU nationals (equity).

Overall, while respecting Member State competences, the EU framework provides the flexibility 297 to adapt its objectives to specific national contexts and to changes that take place over time. This suggests that the relevance of the EU framework hinges closely on the appropriateness and ambition of the concrete objectives and measures set out in the NRISs. The flexibility and the non-binding nature of the EU framework have allowed Member States to follow a tailored approach taking account of specific national contexts. While such an approach has the potential to make actions more relevant, the evaluation also found that it contributed to a generally fragmented implementation and incomplete alignment between the Roma integration goals and the NRIS.

Coherence has been assessed as generally positive at EU level and as limited at national level. At EU level, progress has been made in aligning and mobilising legal, policy and financial instruments for the support of Roma integration. EU funding such as ESIF, the Europe 2020 strategy and the use of CSRs were found to be particularly instrumental to achieve the EU framework's objectives. Other areas of coherence include the European enlargement policy, the EU urban agenda and EU anti-trafficking policies or recent proposals such as the recast Drinking Water Directive 298 . Roma integration is not explicitly included in key social policy initiatives such as the Youth Guarantee or the European Pillar of Social Rights. The evaluation found that the EU framework has contributed to incorporating Roma integration into legal, policy and financial instruments at national level to a more limited extent, and this across Member States and the enlargement countries.

Efficiency has been assessed as limited with regard to monitoring and reporting systems and inconclusive on the costs and benefits, mainly due to limited data availability. Improvements are necessary in aligning domestic and European reporting requirements as well as coordination between the EU framework’s reporting times and formats and related reporting such as for ESIF or the European Semester. The online reporting tool provided by the Commission for reporting from the national to the European level is a positive step, but several weaknesses (data availability to feed the tool; cooperation between the NRCPs and other stakeholders in gathering data; development of more meaningful indicators) need to be addressed. Existing comparable data (e.g. from the FRA) could be integrated into the tool to provide outcome and impact indicators. Costs and benefits could not be conclusively evaluated for a number of reasons, including: (i) most identified or potential costs and benefits cannot exclusively be attributed to the EU framework or are not quantifiable; (ii) where costs are in principle quantifiable (for example ESIF investment), information about the proportion of funding spent exclusively on Roma is not sufficiently available, this being in particular due to the absence of Roma-specific indicators; and (iii) output values in terms of Roma beneficiaries and participants are not systematically available.

EU added value has been assessed as positive. EU action has provided added value to national Roma policies and their implementation at the political level and in terms of governance and financial support. In particular, EU action has: (i) put Roma inclusion on the political EU and national agendas; (ii) raised attention to Roma issues in countries with smaller Roma populations; (iii) strengthened existing structures for Roma integration and contributed to the creation of new ones; (iv) ensured stability through its multiannual character; (v) provided policy guidance, monitoring and reporting frameworks supporting greater accountability and transparency; (vi) provided opportunities for mutual exchange and cooperation; (vii) enabled — through its comprehensive approach — Roma inclusion to be addressed from various policy perspectives as well as collaboration between different political and governmental levels; and viii) ensured that ESIF supports implementation of the NRIS. Stopping targeted EU action on Roma integration is likely to reduce political commitment and focus on Roma integration in both the EU Member States and enlargement countries and lead to a sharp decrease in available funding. Stopping EU action is also likely to result in less and looser monitoring and reporting about the situation of Roma.At present, the effects of the EU framework are unlikely to last after 2020 if there is no further EU support (sustainability). Most of the current national policies and structures created around the NRIS (NRCPs, systems of coordination, monitoring and reporting, systems of consultation with civil society and Roma organisations) would stop or would be less operational and become more symbolic, if the EU framework did not continue after 2020 299 . Increased efforts at national level are needed as well as more time to consolidate working structures, to further align and mobilise other policy, legal and financial instruments and to better monitor the impact of policies.

A number of the above lessons learnt can be addressed in the longer term but not in the context of the current EU framework, its objectives and its set up. These lessons learnt include in particular:

·The need for a revision of the framework’s objectives to make them more specific, measurable and realistic while ambitious enough to bring about changes;

·The importance of enabling Member States to follow a more tailored approach taking account specific national contexts, for example through individually adaptable Roma integration objectives;

·The need for complementing the four priority areas of employment, education, health and housing with a specific focus on fighting non-discrimination and antigypsyism;

·The importance of addressing the limited attention to the specific disadvantages of Roma women, children and youth and other groups within the Roma population, including EU-mobile Roma and Roma who are non-EU nationals;

·The need to strengthen the national Roma contact points in terms of mandate and capacity for mainstreaming Roma inclusion into all relevant policy areas.

However, in the remaining implementing period until 2020 the following corrective measures could be prioritised:

·Some improvements to the monitoring and reporting systems associated with the EU framework and NRIS can be initiated swiftly. This includes in particular the weaknesses accompanying the current online Roma reporting tool provided by the Commission (data availability to feed the tool; cooperation between the NRCPs and other stakeholders in gathering data; development of more meaningful indicators) which should be addressed. Equally, suggestions made during the evaluation for reducing administrative burden (such as regarding frequency and level of reporting) should be further explored. Existing comparable data (e.g. from the FRA) could be integrated into the tool to provide outcome and impact indicators. Also, on-going efforts of the Member States towards reliable, ethnically disaggregated data collection should continue, in line with national practices, as the improvements regarding data availability to feed the tool will not solve the problem of lack of disaggregated data in some cases.

·Effective Roma participation in policy-making can be further encouraged, both at national and EU level. This includes capacity building of Roma and pro-Roma civil society as well as their involvement in the monitoring of NRIS implementation. At EU-level, events similar to the workshop organised for this evaluation, bringing together NRCPs and Roma civil society for a discussion, are useful to stimulate increased dialogue.

·Continued efforts at EU and national level are needed to halt and break down the rising levels of discrimination and antigypsyism. Building on efforts made in the past (at EU level for example the prioritisation of the fight against discrimination of Roma, antigypsyism, hate speech and hate crime in calls under the Rights, Equality and Citizenship programme), Member States should be encouraged to implement more measures tackling discrimination and antigypsyism (such as for example awareness raising targeting employers, teachers, police).

·Roma needs could be better reflected in mainstream policies, in particular at national level. Effective inclusion of Roma happens when mainstream policies are sensitive and responsive to their needs. In this context, awareness about he existence of NRIS and NRCP could be increased. Also, based on the generally positive evaluation of the link between the EU framework and ESIF, Member States should make full use of the ESF investment priority 9.2 ‘Socio-economic integration of marginalised communities such as the Roma’ as well as of other relevant ESF and ERDF investment priorities. In their programming for the post-2020 period, Member States should make full use of the opportunities offered by the post-2020 ESIF regulations 300 and enlargement countries should make use of IPA assistance. At EU level, consistency between the objectives of the EU framework and other key policies such as the European Semester, the European Pillar of Social Rights and the UN 2030 Agenda on Sustainable Development should be ensured.

·While consolidating its achievements in terms of EU added value, the Commission could start to prepare a programme of EU-level good practice exchanges and mutual learning, taking into account transferability of good practices in light of differences between the countries.

·Overall, Member States could be encouraged to already take steps to make the positive results more sustainable. Key conditions for sustainability presented in the evaluation are: (i) a consistent EU and national funding mix; (ii) the development of inclusive public services that recognise the needs of the most vulnerable, including Roma; and (iii) involving Roma in a structured policy-making process.

Annex 1: Procedural information

Lead DG, Decide planning/CWP references

• DG Justice and Consumers (DG JUST)

• PLAN/2017/830

Organisation and timing

This evaluation has been steered by DG Justice and Consumers since May 2017 under the scrutiny of an inter-service group (ISSG) comprising representatives of DGs EAC, EMPL, HOME, REGIO, AGRI, JRC, NEAR, SG and SANTE, and the Fundamental Rights Agency. The ISG was consulted at each stage of the evaluation process and reviewed each deliverable produced by the contractor as well as this staff working document (SWD). ISSG meetings took place on:

• 5 May 2017 (consultation strategy)

• 14 June 2017 (terms of reference evaluation study)

• 29 November 2017 (inception report)

• 24 January 2018 (interim report)

• 25 April 2018 (final report).

The ISG was consulted on the draft SWD on 28 June 2018 and informed about the Regulatory Scrutiny Board’s opinion on 05 October 2018.

Exceptions to the better regulation guidelines

None

Consultation of the Regulatory Scrutiny Board (RSB) (if applicable)

The RSB was consulted on 29 August 2018 and met on 26 September 2018. The Board gave a positive opinion 301 , with comments to improve the document.

The following changes were made to this SWD, in response to the Board’s main considerations:

Evidence, sources and quality

This SWD has been drawn up according to a roadmap published in March 2017 302 , to which no feedback was received.

Sources of evidence

The following box provides an overview of the data sources from which the evidence was drawn. A detailed description of the individual methods is provided in Annex 2.

Expert advice

A workshop was organised on European Commission premises in Brussels on 15 March 2018. The workshop brought together 88 stakeholders who discussed the initial findings of the evaluation. Participants of the workshop represented a wide range of stakeholders and experts. Present were 29 national Roma contact points (across EU Member States and enlargement countries), 28 representatives of NGO or civil society organisations, 14 representatives of the European Commission, 4 representatives of international organisations, 2 representatives of the European Parliament, 1 representative of the Fundamental Rights Agency and 1 representative of Equinet.

External consultant

The external evaluator contracted for this assignment (ICF/Milieu) has carried out since October 2017 all tasks as required, under the scrutiny of an inter-service group (ISSG) set up for this evaluation and the steer of DG Justice and Consumers. Primary data were mainly collected from January to March 2017. During the inception phase, the ISSG clarified the thematic scope of the evaluation compared to the wording in the roadmap, which was found to be insufficiently clear. It was clarified that the evaluation would focus on assessing the EU framework but abstain from evaluating any other financial, legal or policy initiative mobilised for Roma integration or the NRIS itself. This clarification was important in light of the request made by the Council to evaluate the EU framework itself. The external evaluation study was finalised in July 2018 306 . Its overall quality was assessed as satisfactory by the ISSG.

Annex 2: Stakeholder consultation

1. Introduction

This annex presents an overview of all activity conducted with stakeholders as part of the ‘Midterm evaluation of the EU framework for national Roma integration strategies (NRIS) up to 2020’. The general objectives of the stakeholder consultations, as outlined in the consultation strategy prepared for the evaluation, were to collect stakeholders’ views on:

·results achieved and challenges faced during the 5 years of implementation of the EU framework in the key areas of education, employment, healthcare and housing as well as on discrimination;

·the alignment, relevance and effectiveness of European and national policy, legal and funding instruments.

The specific objectives of the stakeholder consultations were to obtain targeted feedback on:

·the use and results of the EU framework;

·the use, impact and alignment of European policy, legal and funding instruments put in place in support of Roma integration;

·the impact on Roma of the implementation of the NRISs and of mainstream policies.

The consultation strategy specified that stakeholder views should be ensured by facilitating targeted stakeholder consultations, and through an open public consultation (OPC). The stakeholder consultations that were ultimately pursued as part of the evaluation covered both of these activities. The final types of stakeholder consultation that took place are as follows:

Ÿan OPC, which featured a set of questions for a range of stakeholder groups;

Ÿtargeted stakeholder consultations, which took the form of interviews with a variety of stakeholder groups from across Member States and enlargement countries;

Ÿan online survey specifically targeted at NGOs; the survey enabled the views of NGOs involved in Roma integration across Member States to be incorporated, as their insight might not have been adequately captured through the other consultation methods planned;

Ÿa workshop, which brought together a wide variety of expert stakeholders to address different dimensions of the preliminary findings of the external evaluation study.

The following table summarises the range of stakeholders consulted as part of the evaluation, in line with the consultation strategy.

Table 10: Stakeholder type and data collection method

2. Overview of consultation activities

While all the stakeholder types included in Table 10 were approached, representatives of organisations representing business and professional associations did not respond to the OPC. The other stakeholder groups were effectively reached through the data collection methods outlined above 307 .

2.1. Open public consultation

The OPC carried out by the Commission aimed to compile the opinions of these stakeholders on the achievement and challenges of the EU framework between 2011 and 2016, in order to identify specific areas which would need prioritising during the remaining implementation period. Additionally, the OPC took stock of the various European and national policies, and legal and funding instruments which had so far been mobilised for Roma integration. The OPC questionnaire 308 consisted of 16 questions 309 that covered:

Ÿintroductory questions on the background of the respondent;

Ÿgeneral questions on social exclusion and discrimination and expectations for future priority areas at the European and national level. This set of questions did not require specific knowledge of European or national instruments used to further Roma inclusion;

Ÿspecific questions on: (i) European and national efforts at Roma inclusion; (ii) relevant policy developments; (iii) achievements and challenges pertaining to the EU framework and NRISs; and (iv) specific measures taken across the four main policy areas of education, employment, health and housing.

The online OPC ran between 19 July and 25 October 2017 on the website of the European Commission. A total of 240 responses were received to the survey 310 . 165 of these came from organisations, while 75 were from individual citizens. Of those 165 organisations, 106 indicated that they represented a NGO or think tank, 44 represented public administration, and 15 answered on behalf of other organisations (such as equality bodies). Of the 240 respondents, 202 specified their ethnicity: 91 identified themselves as Roma and 111 as non-Roma.

Additionally, 28 position papers were received as part of the OPC. The majority of these were from NGOs, although UN agencies, universities and the World Health Organization also provided submissions. Some of the submissions were tailored responses to the OPC, while others were research or advocacy papers going back as far as 2010.

2.2 Targeted stakeholder consultations

Interviews with a range of relevant stakeholders across Member States and enlargement countries were carried out. Interview consultations that were undertaken were categorised into three groups:

Ÿ138 interviews with stakeholders (NRCPs, officials in employment, housing, education and health ministries, NGOs, equality bodies, regional authorities, experts) across 11 Member States selected for country analysis studies 311 ;

Ÿ53 interviews with stakeholders (NRCPs, officials in employment, housing, education and health ministries, NGOs) across 16 Member States not selected for country analysis studies 312 ;

Ÿ10 interviews with stakeholders in three enlargement countries (Serbia, the Former Yugoslav Republic of Macedonia, and Bosnia and Herzegovina).

The final number of stakeholders interviewed was less than the number originally intended, as it was not always possible to arrange an interview with all stakeholders 313 . The main reasons for this included: (i) some of those contacted were unable to take part in the interview, but also unable to suggest another possible contact to replace them; (ii) some ceased responding; and (iii) others did not reply to the initial invitation to interview. In addition to the above interviews, eight interviews were conducted with experts operating at the European or international level 314 .

2.3. NGO survey

A targeted online survey was opened on 16 January 2018 and ran until 23 February, with the aim of giving NGOs an additional opportunity to provide comments. This was considered important as not in all countries were NGOs part of the targeted stakeholder consultation described below. The survey drew 65 full responses. Respondents to the survey represented 19 Member States, with the largest share of respondents representing Greece, Slovenia and Sweden (each Member State individually accounting for 13 % of all respondents). A substantial proportion of organisations (47 %) reported that they operate at the national level, while 16 % of respondents represented a regional-level organisation, 17 % a community- or local-level organisation, while represented 14 % an international organisation.

The survey questions comprised 24 multiple choice questions, organised in accordance with the different evaluation criteria explored in the study: relevance, coherence, effectiveness, equity, coordination, efficiency, sustainability and EU added value. They consisted of statements for which the respondents had to state whether they agreed/disagreed/neither agreed nor disagreed/had no opinion/did not know.

2.4. Workshop

A workshop was organised on European Commission premises in Brussels on 15 March 2018. The workshop brought together 88 stakeholders who discussed the preliminary findings of the evaluation. The workshop gave participants the opportunity to respond specifically to the findings on the effectiveness, EU added value, relevance and coherence of the EU framework. Participants at the workshop represented a wide range of stakeholders. Present were 29 national Roma contact points (from across EU Member States and enlargement countries), 28 representatives of NGO or civil society organisations, 14 representatives of the European Commission, 4 representatives of international organisations, 2 representatives of the European Parliament, 1 representative of the Fundamental Rights Agency, 1 representative of Equinet and 9 members of the ICF/Milieu evaluation team.

3. Methodology

3.1 Open public consultation

The survey data was downloaded by the Commission in excel format and processed using the following excel functions: tables, bar charts, filters and cross tabulations. The analytical work involved the breaking down of the results by:

Ÿrespondent type (citizens vs organisations);

Ÿorganisation type (public administrations, NGOs/think tanks, other);

ŸRoma vs non-Roma background (i.e. respondents identifying as Roma vs respondents identifying as non-Roma);

Ÿthe following country clusters: EU-15, EU-13, enlargement counties.

A separate analysis of the survey results was done for the five countries with relatively sizeable Roma communities: Bulgaria, the Czech Republic, Hungary, Romania and Slovakia. The results were broken down based on the above characteristics to identify any differences in perception or opinion among certain respondent categories or in certain (groups of) countries. A report analysing the results of the OPC was submitted by the external contractor to the Commission as a separate deliverable.

3.2 Targeted stakeholder consultations

Stakeholders were selected depending on the type of organisation or institution that they represented, following provisional sampling agreed at inception report level. For each country covered, it was imperative to consult with a representative of the national Roma contact point 315 . Other categories of stakeholders interviewed included equality bodies, representatives of government entities at national or regional level (dealing with health, employment, education, housing and trafficking), and also NGOs. For each of the 11 country analysis studies, local country correspondents carried out an initial stakeholder analysis to also identify local government representatives responsible for the four policy areas. The full list of stakeholders to be consulted per country was agreed with the ISSG.

All responses from the targeted consultations were processed using NVivo qualitative data analysis computer software 316 . The interview guidelines that were used to support the individual consultations had a specific structure, which grouped certain questions together in correspondence with the evaluation criteria being assessed. This meant that as a whole, the interview responses could be effectively analysed per evaluation criteria, using NVivo software to isolate those responses relevant to the evaluation criteria.

3.3 NGO survey

A list of 135 national NGOs from across the EU was developed and the survey was sent on 15 January to these NGOs. National Roma contact points were also invited to distribute the survey further. The NGO survey was comprised of a series of multiple choice questions. The data received showed how many people responded to each question and the percentage share of respondents that answered a certain question. The content of the response fed into each evaluation criterion.

3.4. Workshop

The workshop presented the preliminary evaluation findings and enabled participants to provide feedback in particular to the specific evaluation criteria questions on effectiveness, EU added value, relevance and coherence. This feedback was given through an open discussion, facilitated by members of the evaluation team. Similarly, participants were invited to share recommendations openly at the workshop or to send them in writing later.

4. Results of stakeholder consultations per activity and how they fed into the evaluation

The results of the consultation activities were used according to the evaluation framework agreed at inception report level. For each of the evaluation questions, the evaluation framework clarified which of the consultation activities would be relevant for data collection.

4.1 Open public consultation (OPC)

A full summary of results stemming from the OPC was published on EUROPA 317 . Findings include:

·An overwhelming majority of the respondents (between 86 and 95 % depending on the thematic area) agreed that targeted public interventions are needed in the fields of discrimination, employment, education, housing and healthcare 318 .

·For a majority of the respondents (almost 60 %), the EU has a major role to play in supporting national, regional and local authorities because alone they cannot effectively improve the situation of Roma 319 .

·Respondents consistently stated that both EU institutions and national authorities should work together to develop measures to improve Roma inclusion. They see a stronger role for the EU than for national authorities in: (i) monitoring and enforcing European non-discrimination and anti-racism legislation; and (ii) making access to funding conditional on developing and implementing ambitious Roma policies 320 .

·National authorities are expected to play a bigger role in measures such as: (i) community building between Roma and non-Roma; (ii) non-discrimination; (iii) training for public officials on how to achieve Roma inclusion; (iv) making Roma history and culture part of school curricula; and (v) providing policy guidance to authorities 321 .

·Key challenges identified by the respondents include: (i) the insufficient incorporation of Roma inclusion into other policies and instruments at both European and national level; (ii) rising discrimination and antigypsyism, especially at European level; and (iii) insufficient funding allocated to Roma inclusion at the national level 322 .

·With regard to suggested priorities at European and national level, respondents confirmed that successful Roma inclusion strategies need to be comprehensive. Access to education came out as a clear priority (67 % at European level and 76 % at national level), while access to employment, healthcare and housing, fighting discrimination and addressing antigypsyism were also selected as a priority by at least one third of respondents at both European and national levels 323 .

4.2. Targeted stakeholder consultations

Targeted stakeholder consultations consisted of 201 324 interviews as explained above. The replies to the interviews were so numerous and varied, and covered such a wide geographical scope, that summarising their results as a whole for the purpose of this report is not realistic. However, the outcomes of these targeted stakeholder interviews are fully reflected in the external evaluation study 325 and this SWD, using the following referencing of sources:

Table 11: Referencing of sources

4.4. NGO survey

The results of the NGO survey are published in the external evaluation study 326 . Key findings include:

·With regard to contributions made by NRISs to effective changes on the ground, survey participants considered that the NRIS of their respective Member State had contributed to some extent to reducing discrimination against Roma (40 %), reducing hate speech against Roma (50 %), reducing hate crimes against Roma (53 %) and improving the enforcement of anti-discrimination legislation (44 %) during 2011-2017 327 .

·Nearly half of the respondents considered that the Roma’s access to employment has not changed since 2011. 43 % of the respondents felt that Roma children’s access to and integration into education systems has improved since 2011, while 20 % believed this has worsened over the years 328 .

·Around half of the NGOs (53.8 %) are involved in mechanisms for coordination, implementation or monitoring of the NRIS. However, most of them support the opinion that the existing mechanisms for coordination and implementation of the NRIS do not allow for effective cooperation and/or consultation with all key stakeholders 329 .

·Asked whether both EU and national funding per Member State was sufficient and proportionate to meet the needs of Roma across the four policy areas, over half of all respondents thought that this was not the case. 63 % of respondents did not think that EU or national funding for Roma inclusion reaches the Roma beneficiaries it was intended for, and did not think that it provides them with long-term benefits 330 . 60 % of NGO respondents believe that EU funding has provided added value in terms of addressing the national funding gap 331 .

4.5. Workshop

Consultation with workshop participants focused on the evaluation criteria of relevance, coherence, effectiveness and EU added value.

With regard to relevance, the participants agreed that the original objectives of the EU framework remained relevant today but that the ambition in some areas should be increased. Given the deep-rooted nature of discrimination against Roma, several participants noted that explaining and addressing institutionalised racism was of great importance, as was generally increasing the focus on antigypsyism.

For the coherence criterion, several workshop participants highlighted the need for better policy mainstreaming. On budget allocation and funding for Roma inclusion, it was recommended that specific Roma indicators be identified and developed to effectively implement NRISs. Participants noted problems due to the NRCPs feeling isolated within their governments and that the lack of financial capacity can prevent the effective implementation of NRISs.

On effectiveness, several participants noted the important role played by the Commission in prioritising Roma issues and in particular by adopting the EU framework in 2011. Other participants highlighted how the availability of EU funding for Roma inclusion helped Member States commit to the cause. The rise of populism and far-right political parties was also cited by participants as a worrying aspect that would affect the objectives for Roma inclusion.

With regard to EU added value, following a question asked through an online tool (SLIDO), 97 % of the participants agreed that the EU had provided added value for Roma inclusion in the Member States. NGO participants added that none of the Member States would be where they were without the EU framework. On the other hand, several participants stated that while certain tools are in place, a clear connection between EU funding and the indicators in the NRIS should be made and monitored.

5.    Overall results from the consultations

Across the results of all the consultations, a number of common messages can be identified. In particular, there was a broad consensus that since 2011:

·The EU framework has been necessary to help realise positive changes related to Roma inclusion across the policy areas of education, employment, health and housing at national level.

·Without the framework and EU direction and support, it is unlikely that Member States would be able to effectively improve the situation of the Roma.

More specifically, multiple stakeholders raised the aspects set out below.

·Improving access to education must remain an absolute priority for all actors involved in the implementation of NRIS.

·Roma inclusion has become a higher priority on the EU policy agenda.

·Mainstreaming of Roma inclusion in policies remains to be effectively implemented in particular at national levels.

·Political commitment at national level to policies that ensure Roma inclusion must be increased.

·Measures at national level to tackle antigypsyism were insufficient.

·National funds are often deemed insufficient to implement Roma inclusion measures.

The results of the stakeholder consultations generally demonstrate a range of common aspects and shared areas of concern. While there were differences in the opinions of the multiple stakeholders consulted, these differences are normal given the backgrounds of the interviewees, for example when discussing the functioning and influence of the NRCPs with NGOs vs with NRCPs themselves; or when discussing mainstreaming at national level with ministries vs NGO experts. Such differences were reflected in the analysis and do not challenge the above overall results of the consultations.

Annex 3: Evaluation criteria and questions

Relevance (relationship between needs and objectives):

To what extent have the (original) objectives proven to be appropriate in view of the (original) needs?

To what extent were the original objectives of the EU framework appropriate in view of the needs?

To what extent was the combination of social inclusion and anti-discrimination objectives sufficiently balanced to address the needs?

To what extent is the EU framework still relevant?

How well do the original objectives still correspond to the current needs? How relevant is the EU framework considering the current levels of discrimination and disadvantages faced by Roma?

How relevant are the goals of the EU framework for Roma and for European societies?

Do the objectives need to be updated, when and how?

Coherence (relationship between different European/national instruments mobilised to promote Roma inclusion):

To what extent did the EU framework contribute to mainstreaming Roma integration into European and national policy?

To what extent did it contribute to mainstreaming Roma integration into policies?    

To what extent did the EU framework contribute to linking Roma integration priorities with EU funding?

To what extent did the EU framework contribute to fighting discrimination against Roma under European legislation (such as the Racial Equality Directive, the Council Framework Decision on Racism and Xenophobia and Directive 2011/36/EU of 5 April 2011 on preventing and combating trafficking in human beings and protecting its victims)?

To what extent did the monitoring and guidance provided by the European Commission (under the EU framework, the European Semester, European legislation such as the Racial Equality Directive) identify relevant points for improving effectiveness of Roma integration efforts?

To what extent is the EU framework coherent with other EU policies?

To what extent did National Roma Integration Strategies (NRIS) or integrated sets of policy measures contribute to mainstreaming Roma integration under public policies in the fields of education, employment, healthcare and housing?

To what extent did they contribute to linking Roma integration priorities with EU and national funding?

To what extent did they contribute to fighting discrimination, hate speech and hate crime against Roma, enforcing EU anti-discrimination and anti-racism legislation at national and local levels and addressing prevention and awareness-raising regarding trafficking in human beings?

Effectiveness (degree of progress towards achieving objectives and role of the EU action in observed changes):

To what extent have the objectives set out in the EU framework as defined in 2011 and extended by the 2013 Council Recommendation been achieved from 2011-2016?

What have been the changes in discrimination patterns in education, employment, housing and health?

What have been the quantitative and qualitative effects of Roma integration measures implemented in Member States in education?

What have been the quantitative and qualitative effects of Roma integration measures implemented in Member States in employment?

What have been the quantitative and qualitative effects of Roma integration measures implemented in Member States in healthcare?

What have been the quantitative and qualitative effects of Roma integration measures implemented in Member States in housing?

To what extent can changes in discrimination patterns and quantitative and qualitative effects of Roma integration measures in employment, education, housing and healthcare be credited to the EU framework? What other factors have influenced the achievements observed?

Why were certain objectives not reached?

To what extent did the EU framework contribute to setting up and mobilising the necessary instruments (such as the NRIS and annual reports by Member States, civil society and the Commission) and governance structures at European and national levels (National Roma Contact Points and its network, National Roma Platforms, European Platform for Roma Inclusion, civil society consultation meetings, EU Roma Summits)?

Has the EU framework served as a framework for NRIS?

To what extent did National Roma Integration Strategies (NRIS) lead to effective Roma integration measures and inclusive reform of mainstream policies?

To what extent did they contribute to increased European and national funding for Roma integration and inclusive reforms?

To what extent did the number of Roma beneficiaries of integration measures increase?

3.4.4 To what extent did national/local/civil capacity to implement Roma integration measures improve?

Coordination (synergies between interventions):

Have coordination mechanisms at EU/national level been effective in coordinating the policy making, funding, implementation, monitoring and evaluation)?

To what extent effective methods of coordination of the implementation engage the relevant stakeholders? The extent of participation of civil society, in planning and monitoring inclusion policies and programmes

What is the level of dialogue and cooperation with Roma representatives?

Existence and clarity of the communication measures of the NRIS?

Is there sufficient administrative capacity at EU, national, sub- national and local level to organise effective coordination of the implementation of the EU framework and the National Roma Integration Policies?

Efficiency (relationship between resources invested and benefits):

What factors influenced efficiency of EU and national interventions in the context of the EU framework, in particular regarding the instruments (such as NRIS and annual reports by MS, civil society and Commission) and governance structures built at EU and national levels (NRCPs and its network, National Roma Platforms, European Platform for Roma Inclusion, civil society consultation meetings, EU Roma Summit)?

To what extent are the reporting and monitoring systems in place adequate and what are the respective strengths and points for improvement on EU and national levels?

Is there a need to simplify or reduce administrative burdens on NRCPs and MS public officials involved in the reporting and information gathering process?

What have been the costs and benefits of Roma inclusion in the context of the EU framework and NRIS?

To what extent are EU and national funding in the area of Roma inclusion justified and proportionate to the long-term benefits from Roma inclusion in quality inclusive mainstream education, employment, healthcare and housing?

EU added value (changes due to EU intervention):

To what extent is EU action necessary to stimulate, complement, leverage and create synergies with national action to promote Roma integration?

To what extent do the EU actions in areas such as monitoring, reporting, policy guidance, funding and knowledge exchange bring added value?

To what extent does Roma inclusion continue to require EU level action?

What would be the consequences of stopping targeted EU action?

Equity (have results been achieved in a proportional and fair manner with respect to vulnerable subgroups groups):

Has there been sufficient level of awareness-raising efforts for fighting discrimination and addressing antigypsyism 332 ?

To what extent do the Framework and the National Roma Integration Strategies address the risk of double discrimination among Roma?

To what extent the Framework results address the specific needs of Roma women?

To what extent the Framework results address the specific needs of Roma youth?

To what extent the Framework results address the specific needs of Roma children?

To what extent are mainstream and (non Roma-specific) targeting programmes available to Roma 333 ?

Sustainability (extent to which effects are likely to last after the intervention ends):

What measures were adopted by the European Commission and the Member State to ensure the sustainability of the results of the policy actions implemented within the scope of the NRIS / EU framework (both at planning and implementation stage)?

What measures were adopted to ensure the continuity of the activities carried out thanks to the EU funding?

Are there any mechanisms in place to ensure a sustainability check at policy implementation level?

To what extent are the outcomes/benefits of the policies implemented via the NRIS and EU framework expected to continue post 2020?

Annex 4: Methods and analytical models

This annex describes the following aspects in detail:

1.Methodological approach of the external evaluation study

2.Reasons for and consequences of the scarcity of ethnic data

3.Description of key data sources, their limitations and mitigation measures

1. Methodological approach of the external evaluation study

The methodological approach taken for the external evaluation study can be summarised as follows 334 :

Phase 1 included more specifically:

·An initial desk review of the NRISs and mapping of existing secondary literature of interest to the evaluation;

·A review of: (i) the 2011 FRA Roma survey and 2016 FRA MIDIS II survey; (ii) the Roma integration scoreboard; and (iii) the 2011 UNDP/World Bank/EC regional Roma survey;

·A review of data available in the online Roma integration measures reporting tool;

·Eight EU level scoping telephone and face-to-face interviews;

·Development of the evaluation framework, including the approach to the evaluation questions and corresponding judgement criteria.

·Selection of 11 country analysis studies.

·Development of data collection instruments: (i) interview guidelines and template for country analysis studies; and (ii) interview guidelines for other stakeholder consultations.

·Development of the analytical approaches for assessing costs of Roma exclusion/benefits of inclusion.

Phase 2 included more specifically:

·A comprehensive literature review;

·Analysis of the use of EU funds for Roma inclusion, using in particular ESIF programming documents 335 , including through the use of dedicated portals provided by the Commission (SFC 2014);

·Targeted stakeholder consultations: a large-scale programme of in-depth semi-structured stakeholder interviews and surveys was conducted (see Annex 2 for details). The interviews fell into three groups:

o138 interviews with relevant stakeholders across 11 Member States, which fed directly into corresponding country analysis studies;

o53 interviews with relevant stakeholders across 16 Member States, which did not feed into country analysis studies;

o8 interviews with relevant stakeholders in 3 enlargement countries.

·NGO targeted survey: the survey specifically targeted NGO representatives that had not taken part in the OPC. In total, the survey was fully completed by 65 respondents (see Annex 2 for detail).

·Preparation of 11 country analysis studies (AT, BG, CZ, ES, EL, FR, FI, HU, IT, RO, SK). In each country between 15 and 20 face-to-face and phone interviews were conducted (see Annex 2 for details). Secondary national literature was reviewed, as were NRIS implementation reports. The country studies were undertaken mostly by local experts with detailed knowledge of the local situation and prior experience with Roma issues. The 11 countries were selected so as to cover a maximum of Roma living in the EU (90.2 %) while ensuring a mix of countries with different approaches to Roma integration. The selection was agreed with the ISSG. The purpose was not to assess the NRISs, but rather to collect sufficient evidence to draw conclusions for the evaluation of the EU framework.

Phase 3 included more specifically:

·Analysis of replies to the OPC: analysis of the OPC carried out by the Commission (see Annex 2) was carried out following the principles of the Better Regulation Guidelines. Closed-ended questions, open-ended questions and written responses submitted outside of the framework of the questionnaire were analysed using qualitative analytical techniques. The quantitative analysis of the OPC involved descriptive analysis of OPC respondents and a univariate analysis of other numerical data included in the OPC questionnaire.

·Triangulation and synthesis: this task involved the systematic organisation of all quantitative and qualitative evidence collected from various secondary and primary sources. This involved coding and collating data collected with the aid of NVivo 336 , performing a sense check to ensure its reliability, and transforming data into usable formats.

·Cost-benefit analysis: a simplified form of multi-criteria analysis was carried out. Applying a full CBA to an EU level intervention such as the EU framework is impossible. Measures deliver a range of indirect and direct benefits, not all of which can be monetised. Cost effectiveness analysis can be used as an alternative a CBA to compare different interventions when the benefits (outcomes and impacts) of an intervention cannot be credibly monetised. It is best used when all the expected effects have been defined and are homogeneous and/or can be measured in terms of a key outcome indicator. For this evaluation, a partial monetisation or quantification of costs and benefits combined with a qualitative assessment of costs and benefits that cannot be monetised or quantified was carried out. This corresponds to a simplified form of multi-criteria analysis.

·Sensitivity analysis for financial information.

·Standard cost model approach for administrative costs: using the ‘standard cost model approach’, the interviews carried out by the external consultant with NRCPs and analysis of information in the online reporting tool were intended to provide clarity about the time required to carry out all activities related to the EU framework (for example reporting to the EU level) and implementing the NRIS in each Member State.

·A stakeholders workshop was organised on 15 March 2018 to discuss the preliminary findings of the evaluation.

2. Reasons for and consequences of the scarcity of ethnic data

Following the adoption and transposition of the Racial Equality Directive, racial and ethnic minorities are now protected by anti-discrimination legislation across the EU. The focus has turned to implementation and monitoring and in turn the need has arisen for data on (in)equalities based on racial and ethnic origin. Such data are essential to measure the level of implementation and monitor the impact of policies, but there are serious shortcomings with regard to the availability of data on the situation of racial and ethnic minorities 337 .

Many of the conclusions in the present report are based on data from surveys, in particular FRA’s 2011 Roma survey and 2016 EU-MIDIS and the WB/UNDP 2011 and 2017 surveys on the Western Balkans. This is necessary because official government statistics on Roma are generally not collected due to a number of ethical, political and/or legal considerations in Member States. If data are collected at Member State level, they are not comparable across Member States. In all Member States the collection of ethnic data takes place in accordance with the EU Data Protection Directive, which prohibits the processing of personal data in relation to certain special categories, including data concerning ethnic origin. This prohibition is, however, subject to exceptions. In particular, it does not prevent the gathering of such data, either with the data subject’s consent, or if it is rendered anonymous. Most Member States have chosen to follow the wording of the Directive and interpretation includes racial as well as ethnic origin among the sensitive data 338 ..

At EU level, with the exception of the above surveys, there has been a general reluctance to collect ethnically disaggregated data. Eurostat’s labour force survey and the EU statistics on income and living conditions (EU-SILC 339 ) do not include Roma ethnicity as a marker.

The lack of ethnically disaggregated data leaves the following issues without a clear answer for most of the EU Member States:

·the baseline situation of the level of discrimination and social exclusion 340 ;

·the number of Roma beneficiaries reached through policy measures, especially mainstream measures;

·the funds allocated specifically to measures for Roma inclusion;

·the effects of policy measures in countries not covered by the 2016 FRA survey and overall uncertainty about attributing the effects to the policy measures.

Illustrations of limitations in the quantification of funding and benefits

·There is a lack of data availability on the proportions of EU and national funding specifically spent on Roma integration. While amounts of funding can be identified if measures are targeting Roma only, funding cannot be identified if measures are designed for several groups or disadvantaged groups in general or are mainstream measures, i.e. designed for the general population, without Roma-specific indicators. The evaluation found that there was a general absence of Roma-specific indicators in national and EU funding programmes.

·There is scarce reporting on effects. This is primarily due to a lack of information about the precise number of Roma beneficiaries that benefit from a particular intervention (for example the number of Roma participating in a training session). This is due in turn to a lack of data collection and relevant Roma-specific indicators. Where contextualising a non Roma-specific indicator would have been possible in principle, such as for the ESF 2014-2020, the low level of implementation of the operational programmes so far has been an obstacle for quantification 341 .

·While there is research that demonstrates economic returns on general social investment in the long term, no conclusive argument can be made about the Roma population due to a lack of ethnically disaggregated data.

·Where quantification of benefits was possible (for example on higher employment rates), it was not possible to demonstrate convincingly that this could be attributed to the EU framework. It was therefore also not possible to establish whether benefits would have been smaller without the EU framework.

3. Description of key data sources, their limitations and mitigation measures

FRA surveys 342

The Roma integration indicators scoreboard (2011-2016) accompanying the midterm review of the EU framework presents changes in the situation of Roma in nine EU Member States, as recorded by two FRA surveys in 2011 and in 2016. In 2016, the second European Union minorities and discrimination survey (EU-MIDIS II 343 ) collected information on the situation of Roma in Bulgaria, Croatia, Czech Republic, Greece, Hungary, Portugal, Romania, Slovakia and Spain, in total covering more than 80 % of the EU Roma population. The 2011 Roma survey 344 covered the same countries apart from Croatia. However, information on the situation in Croatia was collected in the UNDP/World Bank/EC 2011 regional Roma survey 345 . The Member States included in the FRA surveys represent approximately between 80 % and 85 % of the EU’s Roma population.

The surveys were carried out using a similar methodology, applying a multi-stage selection of respondents. To optimise the sampling approach, EU-MIDIS II refined the methodology applied in 2011. Despite the similar approaches, the surveys have some limitations as to their direct comparability. In 2017, FRA addressed the limitations on the comparability of the surveys. Given the relative similarity of the unweighted samples of the 2011 and 2016 surveys for the nine Member States, the 2011 sample was weighted to reflect the differences between the two surveys as regards regional coverage and the urban nature of surveyed localities. For Croatia, the same approach was applied to the dataset from the UNDP/World Bank/EC survey.

The scoreboard presents 18 indicators in four main thematic areas (education, housing, employment and health) and the crosscutting area of poverty. It also presents average values for the Member States in question. For 2011, the average does not include Croatia, which at that time was not a Member State. The caveats that need to be considered when analysing values for 2011 and 2016 are provided alongside each indicator. All sample surveys are affected by sampling error, as the interviews cover only a fraction of the total population. Therefore, all results presented are point estimates underlying statistical variation. Small differences of a few percentage points between groups of respondents are to be interpreted within the range of statistical variation and only more substantial divergence between population groups should be considered as evidence of actual differences. A difference of a few percentage points between the 2011 and 2016 values may be assessed as ‘no change’.

Survey data on Roma comes with many caveats. Sampling of Roma is difficult due to lack of reliable data on the actual demographic composition or geographic distribution from census data 346 . Capturing migrant/mobile Roma populations with an adequate sample is even more challenging. Even in Member States where Roma constitute significant parts of the population sampling is a challenge: similarly to census data collection, Roma do not always declare their ethnicity but rather that of the majority population or of other minority groups. The survey language may also be an issue, as the variety of Roma languages or dialects makes it difficult to conduct a single survey in ‘Romani’. Surveys are typically in the national language, which may limit the responses from Roma who face more exclusion due to inadequate language skills. Phone surveys may also lead to under-representation of the poorest Roma, who do not have access to a phone. As a result. the scale of certain problems may be overestimated or underestimated (depending on the issue 347 ).

Comparability of FRA data with official government statistics or Eurostat data on the majority population also is an issue. The 2011 FRA Roma survey collected data for the general population from non-Roma living nearby the areas surveyed to obtain the Roma data. While from a sociological point of view this approach is correct (as it makes it possible to better compare gaps by accounting for regional or local social and economic disparities), it makes comparison of data more difficult.

The above limitations required certain mitigation measures to be adopted when presenting or analysing the data:

·Data based on FRA surveys were only commented on if changes to three percentage points outside of the statistical margin of error were apparent;

·Information from the FRA’s ‘online data explorer’ presenting data from the 2011 Roma survey is not weighted and therefore cannot be used for comparison with information from the 2016 EU-MIDIS II, which used weighted data.

UNDP/WB/EC regional Roma surveys 2011 and 2017 348

A survey carried out by UNDP, the World Bank, and the European Commission in 2011 interviewed 750 Roma and 350 non-Roma households living in or close to Roma communities in 12 countries of central and south-eastern Europe. The survey collected basic socio-economic data on households and on individual household members, as well as perception data of selected adult members from each household. The 2017 regional Roma survey 349 was the first major collection of data on marginalised Roma in Albania, Bosnia and Herzegovina, the former Yugoslavia Republic of Macedonia, Montenegro,  Serbia and Kosovo* since 2011. By following a similar procedure to the 2011 survey, the 2017 regional Roma survey allows for a level of comparability over time.

Data on EU and national funding dedicated to Roma integration, in particular ESIF

Information about exact amounts of funding specifically spent on Roma integration is scarce. With regard to ESIF 2014-2020, investment priority (IP) 9.2 350  covers marginalised communities in general (not exclusively Roma), so the exact amounts spent on Roma integration cannot be identified. Equally, the ESF output indicator for participants encompasses ‘migrants, participants with a foreign background, minorities (including marginalised communities such as the Roma)’ and is therefore not sufficiently specific to provide a clear picture. The 2007-2013 programming period provides even less information, taking a mainstream approach with no such investment priority and indicator.

Online Roma integration reporting tool

Since the 2015 reporting year, annual reporting by Member States on the NRIS implementation has been carried out online. The data in the online Roma integration reporting tool in particular include a description of targeted and mainstream policy measures implemented across 12 thematic areas in total. It also attempts to gather data on funding and beneficiaries. At the time of the evaluation, data for 2015 and 2016 were available from 24 Member States.

To have a comprehensive view of the cost and benefits of individual measures implemented in all Member States, it would have been necessary to rely fully on the information available from the reporting tool. However, serious data limitations were identified around the information provided in the reporting tool. These are reported below, in turn for costs and benefits.

Costs

·There are gaps in the data, as some Member States have not provided any financial information.

·Mainstream measures included in the reporting generally provide no estimate of the share of funding spent on Roma. Nor can it be assumed that the amount spent on Roma is proportionate to the share of the Roma population compared to the general population.

·There are limitations and inconsistencies as to how Member States report on the costs when they choose to report, in particular regarding which costs they include. For example: (i) do they include ‘implementation’ or ‘management’ costs of policy measures?; (ii) are all costs quantified? Often policy measures that have long-term financial impact are immediately quantified — e.g. an educational measure that may require more teachers or more schools at some future point in time.

Benefits

·The number of Roma beneficiaries was indicated by the Member States only for some of the measures implemented in 2015 and 2016. This is particularly the case for mainstream measures.

·Outcome indicators are not reported in the standard reporting form for 2015 and 2016.

·There are numerous social and economic exogenous factors affecting direct and indirect benefits; it is not possible to attribute them directly to the individual measures.

In light of these limitations, this evaluation took the approach not to use the reporting tool data 351 .

CF to cot information under sensitivity analysi

Annex 5: Costs and benefits

a) The table below provides an overview of the costs and benefits identified during the evaluation and attributable to the EU framework, followed by an explanation of the calculations of administrative costs at national level:

Table 12: Costs and benefits identified and directly attributable to the EU framework

Explanation and limitations of the calculation of administrative costs at national level:

The tasks undertaken by the NRCPs are varied. They include policy coordination, monitoring how NRIS are implemented, compiling reports, coordinating and meeting with local authorities and other actors supporting Roma integration, running Roma platforms and engaging with the Roma community. Member States with larger populations of Roma tend to have a higher number of staff (and costs) devoted to the NRIS. Using the standard cost model approach, the interviews carried out by the external consultant with NRCPs and analysis of information in the online reporting tool were intended to provide clarity about the time required to carry out all activities related to the EU framework. These include reporting to the EU level and implementing the NRIS in each Member State.

During the interviews, only nine Member States could provide information on the of number of staff and time spent on NRIS implementation and monitoring. Based on the responses and information provided in the reporting tool, in total, the number of people performing this task was estimated to be 164 individuals. The number of people was multiplied by an average labour cost for public administrative and support services in Member States (hourly labour cost) and the average number of hours worked, also calculated on the basis of the interviews. In total, the administrative cost was estimated to be EUR 3.3 million annually, primarily driven by a high number of staff reported in three Member States (SK, HU and IT). Overall, however, the number of staff contributing to implementation and monitoring of NRISs is likely to be an underestimate, mostly because additional individuals and organisations are involved, in particular at regional and local level.

b) In addition to the above, the following table provides and overview of costs and benefits of Roma integration not directly or exclusively attributable to the EU framework or not yet identified because they are long-term costs and benefits.

Table 13: Costs and benefits of Roma integration not attributable or not identified

Explanations:

a) Quantification of EU and national funding for Roma integration

To ensure that the evidence on funding has been explored as comprehensively as possible, the following information sources have been investigated:

·The online reporting tool, in which Member States provide information on EU (ESIF only) and national funding for Roma integration.

·ESF/ERDF Member State operational programmes implementation reports (2007-2013 and 2014-2020) to identify projects and funding targeting the Roma and the number of beneficiaries targeted. This included examining the project descriptions to identify financial information outside programme indicators, which were found to be mostly incomplete.

·The Instrument for Pre-Accession Assistance (IPA) evaluation, to establish funding through IPA for Roma.

·Other European funding programmes, such as PROGRESS, ERAMUS+, the Fund for European Aid to the Most Deprived (FEAD) and the Rights, Equality and Citizenship (REC) programme, Horizon 2020.

·National research to identify any further funding streams.

·A review of interviews and survey responses, where some funding information is provided.

Despite examining all these different information sources, it was not possible to robustly identify the current or previous levels of expenditure on Roma integration due to the limitations set out above. However, as highlighted in the effectiveness, EU value added and coherence sections, EU funding has overall been well aligned with the objectives of the EU framework. This is particularly the case for the ESF, the IPA, the REC Programme and the European Instrument for Democracy and Human Rights (EIDHR). This is acknowledged in this evaluation by presenting the data that follows even if insufficient to draw overall conclusions.

ESF allocation to IP 9.2 targeting Roma explicitly but not exclusively

Table 14: Total 2014-2020 ESF and IP 9.2 allocations and Roma population

Source: European Court of Auditors (2016) EU policy initiatives and financial support for Roma integration: significant progress made over the last decade, but additional efforts needed on the ground

Instrument for Pre-Accession Assistance (IPA) 355

The EU provides funding to countries which are candidates and potential candidates to join the EU. The funding can be used to fund projects aiming to improve the following topics, which may include spending on the Roma:

- regional development (transport, environment, regional and economic development)

- human resources (strengthening human capital and combating exclusion)

- rural development.

Table 15: IPA I — period 2007-2013 (actual amount spent, subject to update based on final reporting)

Source: Commission, July 2018

*) Actual amount spent 2007-2013, continuously updated based on implementation reports

**) The estimates are based on the Thematic Evaluation on IPA Support to Roma Communities. The report was published in June 2015 as the final deliverable of Contract No 2014/344098/1. The evaluation referred to the 2007-2013 IPA programme and estimated the contribution to Roma inclusion in eight enlargement countries, including Croatia which at the time was a candidate country. In 2017, the figures on the approx. 80 identified Roma inclusion IPA I projects were updated based on reporting on actual project implementation in the enlargement countries. Some of the IPA I Roma integration projects are still being implemented and the figures will be updated pending implementation reports; ***) These figures are subject to updates based on IPA implementation reports.

Table 16: IPA II — indicative allocations for assistance to Roma integration during the first half of IPA II 2014-2016*) **) ***)

Rights, Equality and Citizenship Programme (REC)

Since 2014, the REC programme has funded 43 projects specifically addressing the Roma. Their main focus was on promoting non-discrimination and on promoting cooperation at national level through co-funding of national Roma platforms. These projects received EUR 6.9 million of EU funding (with total funding of EUR 8.4 million). REC represents both a higher number of projects and a higher average annual spending on Roma compared to the previous programming period and the related spending for Roma projects under PROGRESS.

European Instrument for Democracy and Human Rights 356

The European Instrument for Democracy and Human Rights (EIDHR) is a thematic funding instrument for EU external action aiming to support projects on human rights, fundamental freedoms and democracy in non-EU countries. This instrument is designed to support civil society to become an effective force for political reform and defence of human rights.

Table 17: EIDHR allocations for assistance to Roma integration 2007-2013*) and 20142016 **) ***)

b) Quantification of benefits resulting from actual progress made in between 2011 and 2016 in education, employment, housing and health

Even if not directly attributable to the EU framework, an attempt was made to quantify benefits resulting from actual progress between 2011 and 2016 for the following indicators used in the FRA surveys:

·early childhood education and care (4 to compulsory schooling age);

·early leaving from education and training;

·self-declared main activity status ‘paid work’;

·share of young people aged 16-24 years neither in employment, education or training (NEETs).

The evaluation found that research has been done on the economic returns from social investment in general, but that there is scant research on returns of investment in Roma integration more specifically. Transferability of this research is thus restricted, in particular as socio-economically marginalised groups such as the Roma are considered to be further from the labour market than other groups in the population. Potential sources were identified and their transferability for the evaluation assessed, as set out below.

Education

The positive effects of early childhood education and care (ECEC) have been explored in several studies 357 . Children who remain in education longer have an increased probability of being in employment and of higher average earnings once they are employed. Research has estimated the effect of an individual attending pre-primary education to be equivalent to an increase in the PISA score in mathematics of 51 points. This is equivalent of more than a year of formal schooling (PISA 2013 358 ).

In turn, the effect of an individual’s extra year in education is estimated to be an additional 10 % of earnings 359 . Using this increase in earnings would assume that the quality and quantity of ECEC provided for Roma children is comparable to the quality and quantity of ECEC for the general population. This evaluation finds that there is limited transferability of the above quantification to the particular situation of Roma and therefore refrains from using it to quantify benefits 360 . Other benefits of early childhood education according to the specialist literature in the field include: (i) lower dropout rates, improved health and social wellbeing; (ii) increased labour force participation; (iii) reduced welfare dependency; (iv) increased tax receipts; and (iv) reduced crime rates 361 .

Early school leavers face barriers in entering the labour market (i.e. obtaining a job) and on average have lower earnings once they are employed. Additionally, their levels of health, wellbeing and social participation are lower than those who remain in school. This in turn generates additional social costs 362 . Research into early school leaving (ESL) and its cost sometimes uses estimates of the cost of being NEET as a proxy 363 . Following this approach, the change in the number of Roma who are early school leavers could have been multiplied by the annual cost of being NEET to estimate the costs or benefits of changes in ESL in the Roma population. Again, however the mentioned research is not Roma-specific and the transferability of the data is questionable.

Employment

The quantifications below are based on developments in the two Member States with positive changes. They assume stability in the size of the specific age cohorts between 2011 and 2016 and have been carried out to illustrate benefits to be gained if more progress was made, including in other countries.

a) Changes in the share of Roma in paid employment

By increasing the share of Roma in paid employment, there are benefits to be gained in terms of increased earnings of Roma and increased tax receipts for the state budget.

364 365 The increase in the share of Roma in paid employment between 2011 and 2016 in Hungary (from 25 % to 36 %) and Portugal (from 14 % to 34 %) was multiplied by the Roma working-age population in the two countries to estimate the difference in the numbers of Roma workers. This resulted in an increase of 55 500 in Hungary and 6 800 in Portugal. This number was then multiplied by the average annual earnings for Roma (taken from UNDP-WB-European Commission 2011 data and inflated to 2016 prices using GDP deflators, resulting in EUR 3 800 for Roma in HU and EUR 8 300 for Roma in Portugal). The additional wages (EUR 209 300 000 for Hungary and EUR 56 200 000 for Portugal) were multiplied by the average tax rate for the two Member States to estimate the increase in tax receipts (EUR 59 300 000 for Hungary and 14 200 000 for Portugal).

b) Changes in the share of young people aged 16-24 years neither in employment, education or training

According to Eurofound estimations the cost to the economy of NEETs was over EUR 150 billion in Europe in 2011 (more than 1.2 % of EU GDP). The countries most affected were Bulgaria and Greece 366 , where there are significant Roma populations. The Eurofound research also presents annual costs of NEETs by Member State. These costs, which are presented in the table below, relate to the costs to the public purse (benefit payments) and private costs (lack of income generated).

Table 18: Cost of NEETs

Eurofound (2012) Young people not in employment, education or training: Characteristics, costs and policy responses in Europe

The only Member State with a decrease in the share of NEETs among the Roma between 2011 and 2016 is Portugal (from 79 % to 52 %), resulting in 1 500 fewer NEETs in that country. This decrease has been multiplied by the cost per NEET to estimate a benefit of around EUR 12 000 000.

Long-term benefits from Roma inclusion

This SWD does not attempt to quantify the long-term benefits resulting from closing the education, employment and earnings gaps between Roma and non-Roma. This is because long-term projections fall outside the scope of the evaluation. Attempts to quantify this have been made in the past 367 but are based on a number of assumptions including equal educational attainment levels, equal access to employment, equal pay and equal productivity levels.

 Annex 6: Objectives and Measures 2011 and 2013

Annex 7a: Intervention logic I

Annex 7b: Intervention logic II-zoom on the EU Framework

Annex 8: Thematic mapping of instruments and processes

(1) Here, the term ‘Roma’ is used in line with the terminology of European institutions and international organisations, to refer to a number of different groups (such as Roma, Sinti, Kale, Gypsies, Romanichels, Boyash, Ashkali, Egyptians, Yenish, Dom, Lom, Rom, Abdal) and also includes travellers, without denying the unique features and varieties of lifestyles and situations of these groups.

(2) COM(2011) 173 final.

(3)   Council Conclusions (EPSCO) of 19 May 2011 on an EU Framework for national Roma integration strategies up to 2020 .

(4) https://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/ec/123075.pdf.

(5) The Council conclusions urge the Commission to ‘carry out a mid-term evaluation of the EU framework for national Roma integration strategies up to 2020 and to propose a post 2020 strategy on Roma integration, and include therein a proposal for a revision of the Council Recommendation.’.

(6) Council Conclusions (EPSCO) of 8 December 2016: http://data.consilium.europa.eu/doc/document/ST-15406-2016-INIT/en/pdf

(7) * This designation is without prejudice to positions on status, and is in line with UNSCR 1244/1999 and the ICJ Opinion on the Kosovo declaration of independence.

(8) The last four countries through desk research only.

(9) The scope therefore also excludes an assessment of efficiency at project level, which does not allow for reporting on policy indicators at population level.

(10) In particular, ex post and midterm evaluations of EU funding programmes and implementation reports on the Racial Equality Directive.

(11) With the limitations explained in Annex 4.

(12) Add link once published.

(13) Speech of European Commission Vice-President Viviane Reding, 8 April 2011, Budapest.

(14) COM(2010) 2020 final; EUROPE 2020 — A strategy for smart, sustainable and inclusive growth.

(15) Council Directive 2000/43/EC and Council Directive 2000/78/EC of 29 June 2000.

(16) In addition to the above: Council Framework Decision 2008/913/JHA of 28 November 2008 on combating certain forms and expressions of racism and xenophobia by means of criminal law, OJ L 328, 6.12.2008.

(17) See in particular Commission Staff Working Document ‘Roma in Europe: The Implementation of European Union Instruments and Policies for Roma Inclusion — Progress Report 2008-2010’; SEC(2010) 400 final of 7 April 2010 for detail.

(18) COM(2010) 491 final.

(19)

 Directive 2004/38/EC of 29 April 2004 on the right of citizens of the Union and their family members to move and reside freely within the territory of the Member States.

(20)   http://europa.eu/rapid/press-release_IP-10-1207_en.htm

(21) The Decade of Roma Inclusion 2005-2015 was an initiative that brought together governments, intergovernmental and nongovernmental organisations, as well as Romani civil society. Formally established in February 2005 in Sofia, it brought together the governments of Albania, Bosnia and Herzegovina, Bulgaria, Croatia, the Czech Republic, Hungary, Macedonia, Montenegro, Romania, Serbia, Slovakia and Spain (Slovenia, the United States, Norway and Moldova had observer status) in a joint effort to eradicate discrimination and ‘close gaps between Roma and the rest of society’. The initiative was formally closed in 2015.

(22) COM(2008) 420 final.

(23) http://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/lsa/108377.pdf.

(24) COM(2010) 133 final .

(25) European Parliament, Resolution of 9 March 2011 on the EU strategy on Roma inclusion, 2010/2276(INI).

(26) Council of Europe estimates included as an Annex to COM(2011) 173 final.

(27) In particular in terms of participation in the Decade of Roma Inclusion since 2005

(28) Council of Europe estimates

(29) COM(2011) 173 final.

(30) InGrid — Integrating expertise in inclusive growth; Methodological and Data Infrastructure Report on Roma Population in Europe; August 2016.

(31) World Bank (2010) — Roma Inclusion: An economic opportunity for Bulgaria, Czech Republic, Romania and Serbia.

(32)  European Court of Auditors, Special Report: EU policy initiatives and financial support for Roma integration: significant progress made over the last decade, but additional efforts needed on the ground, 2016, https://www.eca.europa.eu/Lists/ECADocuments/SR16_14/SR_ROMA_EN.pdf .

(33) COM(2012) 226 final.

(34) COM(2011) 173 final.

(35) COM(2012) 226 final.

(36) COM(2011) 173.

(37) This intervention logic was included in the terms of reference agreed by the ISSG.

(38) Roma from the EU using their right to freedom of movement within the EU

(39) E.g. European Commission Directorate-General for Employment and Social Affairs, Unit D3, (2004) ‘The Situation of Roma in an Enlarged European Union’; FRA (November 2009) The Situation of Roma EU Citizens moving to and Settling in other EU Member States.

(40) UNDP, Bratislava 2002 ‘Avoiding the Dependency trap — the Roma in Central and Eastern Europe’. UNDP later conducted two rounds of quantitative data collection (in 2004 and 2011).

(41)

E.g. Open Society Institute, international comparative data set on Roma education, 2008. World Bank, Roma Inclusion: An Economic Opportunity for Bulgaria, the Czech Republic, Romania and Serbia, September 2010. Fundación Secretariado Gitano, Health and the Roma community, analysis of the situation in Europe, 2009.

(42) European Union Agency for Fundamental Rights, The Situation of Roma in 11 EU Member States — Survey results at a glance , Publications Office of the European Union, 2012, p. 26.

(43)   http://www.eurasia.undp.org/content/rbec/en/home/ourwork/sustainable-development/development-planning-and-inclusive-sustainable-growth/roma-in-central-and-southeast-europe/roma-data.html

(44) Based on SWD (2017) 458 — FRA Roma integration scoreboard and FRA (2018) A Persisting Concern: anti-Gypsyism as barrier to Roma inclusion

(45) Source is Eurostat, covering the same countries as FRA for Roma and weighted.

(46) General population: Own calculations (weighted average) based on Eurostat (t2020_52) EU-SILC 2014; Eurostat, EU-SILC 2011

(47) General population: Own calculations (weighted average) based on Eurostat: 2011 - educ_ipart (downloaded 07/06/2018)

(48) General population: Own calculations (weighted average) based on Eurostat, Labour Force Survey (LFS) 2011 ((edat_lfse_14, downloaded 06/07/2018 )

(49)  FRA (2018) A Persisting Concern: anti-Gypsyism as barrier to Roma inclusion

(50) FRA uses paid work as a proxy for employment rates.

(51) General population: Own calculations (weighted average) based on Eurostat, Labour Force Survey (LFS) 2011 ([lfsi_emp_a, downloaded 06/07/2018); and Eurostat edat_lfse_20 (downloaded 06/07/2018).

(52) Comparability between EU-MIDIS II/Roma Survey and Eurostat NEET rate is restricted due to the different age bands. Taking 15 year-olds into account would show values lower by a few percentage points for those who are not in employment, training or education. The Eurostat NEET rate is based on the ILO concept, which refers to having worked at least 1 hour in the past week, whereas EU-MIDIS II asked about the self-declared main activity and any paid work in the past 4 weeks (FRA, 2018).

(53) idem

(54) General population: Own calculations (weighted average) based on Eurostat, EU-SILC 2011 ([hlth_silc_01] (download 06/07/2018);

(55)

 OECD Health Database; Health at Glance: Europe reports 2010; (download 06/07/2018)

(56)  General population: Own calculations (weighted average) based on Eurostat **2011 [env_wat_pop]  (download 06/07/2018)

(57)  General population: Own calculations (weighted average) based on Eurostat EU-SILC ilc_lvho04d.

(58) Based on SWD (2017) 458 — FRA Roma integration scoreboard and FRA (2018) A persisting concern: antigypsyism as barrier to Roma inclusion for Roma; based on Eurostat for general population

(59)  Shown value (participation rate) is for education attendance when in compulsory schooling age (5-17 depending on the country); Source: SWD (2017) 458 — FRA Roma integration scoreboard

(60) Gap in the share of people 16+ who self-declared main activity status "paid work"

(61) Gap in the share of people 16+ in 'very good' or 'good health' (self reported)

(62) Gap in the share of people living in households having neither a toilet, nor shower, nor bathroom inside the dwelling.

(63) Gap in the average number of rooms per person in the household.

(64)   Council Conclusions (EPSCO) of 19 May 2011 on an EU Framework for National Roma Integration Strategies up to 2020 .

(65) https://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/ec/123075.pdf.

(66) Malta does not have a Roma community.

(67) Or integrated sets of policy measures in line with the Council Conclusions (EPSCO) of 19 May 2011 on an EU Framework for National Roma Integration Strategies up to 2020 . Strategies and integrated sets of policy measures are referred to as NRIS.

(68) COM(2012) 226 final.

(69) COM(2012) 226 final.

(70) COM(2013) 454 final.

(71)   Council Recommendation (EPSCO) of 9 December 2013 on effective Roma integration measures in the Member States

(72) The Council of Europe’s European Commission against Racism and Intolerance (ECRI) defines antigypsyism as a ‘specific form of racism, an ideology founded on racial superiority, a form of dehumanisation and institutional racism nurtured by historical discrimination, which is expressed, among others, by violence, hate speech, exploitation, stigmatisation and the most blatant kind of discrimination’.

(73) Examples of ‘horizontal areas’ are: the protection of Roma women and children, including human trafficking, poverty reduction, empowerment; examples of ‘structural areas’ are: bodies for the promotion of equal treatment, transnational cooperation.

(74)

Such as ID cards,birth, marriage and death certificates

(75) COM(2013) 700 final.

(76) COM(2014) 209 final.

(77) COM(2016) 424 final.

(78) The 2016 Communication also included for the first time detailed information on Roma integration for the enlargement region. The report included country fiches presenting the state of play in the four key priority areas, the EU dialogue and monitoring structures as well as the funds allocated under IPA I for Roma integration projects.

(79)

14294/16.

(80) COM(2017) 458 final.

(81) For a short summary, see press release: http://europa.eu/rapid/press-release_IP-17-2961_en.htm

(82) BG, CZ, IE, EL, ES, HR, IT, HU, PL, PT, RO, SI, SK, FI, SE.

(83) BE, DK, DE, EE, FR, CY, LV, LT, LU, NL, AT, UK.

(84)

In line with the 2011 Council conclusions, both approaches stimulate the implementation of inclusion policies. Therefore, the remaining analysis does not differentiate between countries with strategies and those with integrated sets of policy measures. Both are referred to as NRIS.

(85) DK, DE, LU.

(86) DE, FR.

(87) CY.

(88) European Commission, ‘Communication from the Commission to the European Parliament and the Council on the Midterm review of the EU framework for national Roma integration strategies’ COM(2017) 458 final.

(89) BG, SI, SK.

(90) ICF, chapter 2.4.2.

(91)

europ.eu/info/law/better-regulation/initiatives/ares-2017-1716166_en.

(92) See Annex 3 for all evaluation questions.

(93) Bibliography is provided in the external evaluation study; in addition, around 200 national sources were included in the country analysis studies.

(94) Operational Programmes; synthesis reports of annual implementation reports such as for the ESF 2017: https://ec.europa.eu/esf/transnationality/content/synthesis-report-esf-2017-annual-implementation-reports.

(95) System for fund management in the European Union.

(96) This mapping was included in the evaluation roadmap.

(97)

This was one of the original evaluation questions; the external study did not take this approach.

(98) In the 2016 Council conclusions on accelerating the process of Roma integration Member States recognise the new challenges posed by the refugee crisis; see recital (11);.

(99) These statements are based on FRA survey data for 9 EU MS, for which data were collected in a comparable way for both 2011 and 2016. These countries account for over 80 % of the overall estimated Roma and travellers population in the EU (as per the Council of Europe).

(100) 6 out of the 9 countries surveyed (BG, EL, ES, HR, HU, SK).

(101)   UNDP, Regional Roma Survey 2017: Country fact sheets

(102) Gatti, Roberta et al. 2016. Being Fair, Faring Better — Promoting Equality of Opportunity for Marginalised Roma. (World Bank). Washington D.C.; O’Higgins, Niall. 2012. Roma and Non-Roma in the Labour Market in Central and South Eastern Europe. edited by A. Ivanov and J. Kling. (ILO) Bratislava; ILO. 2016. Promoting Decent Work Opportunities for Roma Youth in Central and Eastern Europe: An ILO Resource Guide. Geneva. http://www.ilo.org/budapest/what-we-do/publications/WCMS_535448/lang--en/index.htm.

(103) RO-07, CS-RO, CS-AT, EL-01, BE-05, El-06, CZ-12, El-16, ES-09.

(104) FRA (2018) A persisting concern: anti-Gypsyism as a barrier to Roma inclusion

(105)  SWD (2017) 458, FRA Roma integration scoreboard and FRA (2018) A persisting concern: anti-Gypsyism as a barrier to Roma inclusion

(106) Sándor, János et al. 2017. ‘The Decade of Roma Inclusion: Did It Make a Difference to Health and Use of Health Care Services?’ International Journal of Public Health 62(7); European Public Health Alliance 2017: Health and Early Childhood Development in Roma Communities — A document analysis of European Union and national policy commitments in the national Roma integration strategies; European Public Health Alliance 2016. ‘ The European Semester and Roma Health .’; European Commission, Roma Health Report — Health status of the Roma population. Data collection in the Member States of the European Union , 2014,; Fésüs, Gabriella, Piroska Östlin, Martin McKee, and Róza Ádány. 2012. ‘Policies to Improve the Health and Well-Being of Roma People: The European Experience.’ Health Policy 105(1):25-32 .

(107) UNDP Survey. Overview at a Glance.

(108) Matrix, 2014. Roma Health Report: Health status of the Roma population: Data collection in the Member States of the European Union. DG SANCO, http://eige.europa.eu/resources/2014_roma_health_report_en.pdf

(109) FRA (2018) A persisting concern: antigypsyism as a barrier to Roma inclusion

(110)  SWD (2017) 458, FRA Roma integration scoreboard and FRA (2018) A persisting concern: antigypsyism as a barrier to Roma inclusion

(111) S-NGO. Of the remaining respondents, 19 indicated ‘Contribution to some extent’, 6 a ‘Significant contribution’, 1 a ‘Very significant contribution’, and 8 stated ‘Not applicable’.

(112) UN-OHCHR 2017: Lessons Learned: Views in the Context of Mid-Term Review of Implementation of the EU Framework for National Roma Integration Strategies 2012-2020 , Brussels;.

(113) Regional overview at First Glance of the UNDP/WB/EC survey. WB3; WB6; WB8.

(114) Shown value (participation rate) is for share of children attending education when in compulsory schooling age, 5-17 (depending on the country). No comparable data for primary school completion are available..

(115) Used as a proxy for the employment rates of the general population.

(116) The gap between Roma and non-Roma has even increased from 18 % in 2011 to 22 % in 2016.

(117) Self-reported.

(118) General population: Own calculations (weighted average) based on Eurostat,EU-SILC 2016, General population; EU-SILC 2011 ([hlth_silc_01] (download 06/07/2018); **OECD Health Database; Health at Glance: Europe reports 2010, 2016; (download 06/07/2018); also FRA (2018) A persisting concern: anti-Gypsyism as a barrier to Roma inclusion ; p. 13 states that self-reported health status is on average similar to the general population.

(119) Gap in people living without toilet, shower or bathroom inside the dwelling.

(120) At-risk-of-poverty (national threshold (€): 60 % of median equivalent income after social transfers), FRA (2018) A persisting concern: antigypsyism as a barrier to Roma inclusion

(121) European network of legal experts in gender equality and non-discrimination, Roma and the enforcement of anti-discrimination law (2017)

(122) Second European Union minorities and discrimination survey (EU-MIDIS II).

(123) Offensive or threatening comments in person; threats of violence in person; offensive gestures; offensive or threatening e-mails or text messages; offensive comments made online.

(124) FRA(2018), A persisting concern: antigypsyism as a barrier to Roma inclusion

(125) idem.

(126) Antigypsyism as an underlying cause for discrimination has been consistently mentioned in the consultations with stakeholders for this evaluation, such as in the 15 March workshop. It has been highlighted by submissions to the open consultation as well as in specific contributions, see: Carrera, Sergio, Iulius Rostas, and Lina Vosyliūtė. 2017. Combating Institutional Antigypsyism Responses and Promising Practices in the EU and Selected Member States. Brussels Alliance against Antigypsyism. 2017. Antigypsyism — a Reference Paper. Berlin / Budapest. http://antigypsyism.eu/ ;.

(127) When asked whether rising discrimination and antigypsyism was one of the main challenges (in the 2011-2016 period) in respect of Roma inclusion at the European level (Q11.1), 86.3 % of the respondents rather agreed, 10.3 % rather disagreed, and 3.4 % did not have an opinion. When asked about the perceived increase in antigypsyism since 2011 in the EU-15 and EU-13 (Q16.2), 72.6 % of EU-13 and 62.2 % of EU-15 respondents agreed that antigypsyism was more severe today than it was in 2011, while 19.8 % of EU-13 and 23.3 % of EU-15 respondents disagreed.

(128) except MT which does not have a Roma community

(129) , BG, CZ, IE, EL, ES, FR, HR, IT, HU, PL, PT, RO, SI, SK, FI, SE.

(130) BE, DK, DE, EE, FR, CY, LV, LT, LU, NL, AT, UK.

(131)

In line with the 2011 Council conclusions. Both approaches can stimulate the implementation of inclusion policies. Therefore, the remaining analysis does not differentiate between countries with strategies and those with integrated sets of policy measures.

(132) Roma integration measures reporting tool, 2017. Roma Civil Monitor 2018.

(133) Both in the 2011 Council Conclusions and the 2013 Council Recommendation.

(134) European Commission, Description of the EU pilot project for civil society capacity building and monitoring of the implementation of national Roma integration strategies , 2017,.

(135) European Commission ‘ EU Platform for Roma inclusion: Frequently asked questions ’ [website], 2011,.

(136) AT-11, EUI-1, EUI-6, HR-3, NL-4, NL-5.

(137) AT; BE; DE; ES; FI; HR; LT; PL; SE.

(138) National Roma platforms exist in 18 Member States.

(139) AT-6, PT-1, UK-5.

(140) FI-10, FI-13, SK-14.

(141) Roma Civil Monitor 2018: Synthesis Report on the Implementation of the EU Framework for National Roma Integration Strategies in Bulgaria, Czech Republic, Hungary, Romania and Slovakia . Brussels / Budapest. cps.ceu.edu/roma-civil-monitor.

(142) An exception seems to be the work of the European Public Health Alliance (EPHA), which integrates the Roma health issues within a wider agenda of public health and universal health care. See EPHA 2017: Health and Early Childhood Development in Roma Communities — A document analysis of European Union and national policy commitments in the national Roma integration strategies, Brussels.

(143) In countries without significant migrant inflows and larger Roma populations one could in principle assume that a large majority of the participants are actually Roma, which would in such cases be confirmed through the content of the specific investment priority in an operational programme. However, the low level of implementation of the operational programmes so far is an obstacle to this approach.

(144) AT, BG, EE, EL, ES, HR, IT, PT, RO, SK.

(145) In the enlargement region, all countries have used their national IPA envelops to finance Roma integration projects. They used different approaches: Roma integration-specific projects, projects related to vulnerable parts of population benefiting also Roma, mainstream projects including specific Roma integration targets. Furthermore, multi-country IPA projects have been used to deal with questions relevant to all the enlargement region: set up a Roma integration support team in the Regional Cooperation Council (RCC), support Roma civil society and actions in the field of education, boost Roma integration at local level, organise the Roma survey and the returnee study (information provided by DG NEAR).

(146) CZ, FI, IE, LT, LU, LV, PL.

(147) BE, CY, DE, DK, NL, SE, SI, UK.

(148) AT, CY, CZ, EE, EL, ES, IT, LU, NL, PL, RO, SI, SK, and enlargement countries.

(149)  WB-3, WB-8.

(150)   https://ec.europa.eu/info/policies/justice-and-fundamental-rights/combatting-discrimination/roma-and-eu/european-platform-roma-inclusion_en

(151) For more information, see: http://www.rcc.int/romaintegration2020/home.

(152) euromanet.eu.

(153)  (JUST, EMPL, EAC, SANTE, NEAR, HOME, REGIO, AGRI) Other DGs — e.g. SG, BUDG, RTD, ECFIN, COMM — are part of the ‘extended taskforce’, which meets less frequently. The Roma taskforce was created in 2010 (i.e. before the adoption of the EU Roma framework) with the goal to assess to use of EU funds for Roma inclusion, and its conclusions led to the setting up of the EU framework..

(154) AT1, BE5, DE3, LU1, LV1, NL6.

(155) Interviews conducted with 26 NRCPs in the EU and enlargement countries, question ‘Is the administrative capacity of the NRCP sufficient to organise effective coordination of the implementation of the NRIS? What about in ministries or institutions responsible for the implementation of specific inclusion policies?’ In relation to the capacity of NRCPs: out of 26, 11 replied yes, 7 did not reply, and 8 provided comments pointing to the need to reinforce capacity. In relation to other ministries and institutions: in 7 cases the NRCPs explicitly replied ‘yes’.

(156) Open Society European Policy Institute, 2017, Revisiting the EU Roma framework: Assessing the European dimension for the post-2020 future; European Parliament, 2017, Report on fundamental rights aspects in Roma integration in the EU: fighting antigypsyism; CEPS, 2017, Combating institutional antigypsyism responses and promising practices in the EU and selected Member States.

(157) Sources for these statements include BE4, DE1, DE2, DE3, HR2; CS-RO, CS-HU, CS-SI, CS-ES, CS-BG.

(158) https://www.coe.int/en/web/portal/roma.

(159)  Views expressed at the evaluation workshop

(160) CY1 and LV1;DE3; EE2; NL6 and PT3; WB2 (Serbia).

(161) EURoma, 2016, Promoting the use of ESI Funds for Roma Inclusion .

(162) Source: NGO survey.

(163) Using the following criteria: very positive/positive/limited/negative/very negative.

(164) See Section 2, description of the initiative; in particular: designing NRIS taking into account the size of the Roma population and different starting points; also non-binding approach to measures to be implemented by Member States.

(165)  Workshop; EUI1, and B1.

(166) Central Council of German Sinti and Roma, ERGO Network and ENAR.

(167) CS-AT; CS-FI; CS-HU; CS-RO; CS-SK. European Union Agency for Fundamental Rights, Poverty and employment: the situation of Roma in 11 EU Member States . Roma survey — Data in focus, Publications Office of the European Union, 2014.

(168) Workshop participants. EUI-1. European Parliament, Study on the Empowerment of Roma Women within the European Framework of National Roma Inclusion Strategies , 2013, p. 10.

(169)  The term ‘Roma’ is used, in line with the terminology of European institutions and international organisations, to refer to a number of different groups (such as Roma, Sinti, Kale, Gypsies, Romanichels, Boyash, Ashkali, Egyptians, Yenish, Dom, Lom, Rom, Abdal) and also includes travellers, without denying the unique features and varieties of lifestyles and situations of these groups.

(170) This issue was first explicitly addressed as a problem for Roma by the 2013 enlargement strategy, COM(2013) final.

(171)  The share of respondents perceiving the objectives as relevant accounted for 60 % (strongly — 20 % or partially — 40 %) while 20 % disagreed (partially — 14 % or strongly — 6 %).

(172) European Commission (2014) Roma Health Report, http://eige.europa.eu/resources/2014_roma_health_report_en.pdf.

(173) Ibid.

(174) CS-EL and CS-AT.

(175) European Commission, Roma Health Report — Health status of the Roma population . Data Collection in the Member States of the European Union, 2014,.

(176) NGO survey.

(177) CS-BG; CS-EL; CS-FI; CS-FR; CS-IT; CS-RO. BE3; BE4; DE1; IE1; IE2; LT1; LT2: LT4; NL3; PT2; PT3; SE3; SE4; UK4; WB3 (Regional Cooperation Council).

(178) COM(2016) 424.

(179)   European Parliament Report on fundamental rights aspects in Roma integration in the EU: fighting anti-Gypsyism, 11 October 2017/2038(INI) . Alliance against Antigypsyism. 2017. Antigypsyism — a Reference Paper. Berlin / Budapest. http://antigypsyism.eu/ ; FRA 2018: A persisting concern: anti-Gypsyism as a barrier to Roma inclusion , Vienna.

(180) Expressed by the Central Council of German Sinti and Roma, ERGO Network, Open Society Foundations and ENAR.

(181) Centre for European Policy Studies, Research Report: Combating Institutional Anti-Gypsyism. Responses and promising practices in the EU and selected Member States , No 2017/08, 2017, p. 26, European Parliament, Resolution of 9 March 2011 on the EU strategy on Roma inclusion , P7_TA(2011)0092,.

(182) Open Society Foundations, European Roma and Travellers Forum, Roma Women’s Network; EU-13.

(183) InGrid — Integrating expertise in inclusive growth; Methodological and data infrastructure report on Roma population in Europe; August 2016; World Bank (2010) — Roma Inclusion: An economic opportunity for Bulgaria, Czech Republic, Romania and Serbia.

(184) See for example the European Semester 2017 country reports

(185) European Parliamentary Research Service, March 2018, cost of non-Europe report, Equality and the Fight against Racism and Xenophobia; based on conceptual framework for the impacts of discrimination, page 28.

(186) Among the sustainable development goals the following are of highest relevance for Roma inclusion: 1.3 social protection floors, 2.1 hunger and food security, 3.8 universal health coverage, 4.1-6 access to quality education, 6.1 access to safe drinking water, 11.1 affordable housing. Gender equality is transversal to all goals (and indicators) and specifically mentioned in goal 5. Equal treatment is covered in a cross-cutting manner and specifically in target 10.3. Also relevant are: 16.1 protection from violence and 16.7 political participation. Target 17.18 calls for ethnically disaggregated data collection.

(187) FRA 2018; A persisting concern: antigypsyism as a barrier to Roma inclusion

(188) CS-BG; CS-EL; CS-FI; CS-HU; CS-RO; CS-SK; HR2; SE2.

(189) CS-BG; CS-FI; HR2.

(190) Two NRCP participants in the workshop; CS-FR; CS-ES; CS-FI; WB6 (former Yugoslav Republic of Macedonia).

(191) CS-SK.

(192) IE5; SE2. The first Finnish national policy on Roma did include religion (CS-FI).

(193) NRCP participant in the workshop; CS-FR; LT3; PT4.

(194)   European Roma and Travellers Forum, Report: National Roma Integration Strategies. Evaluating Gender , 2015, p. 7; European Parliament, Evaluation of the EU Framework for National Roma Integration Strategies , 2015.

(195) European Parliament, Study on the Empowerment of Roma Women within the European Framework of National Roma Inclusion Strategies , 2013, p. 10 and 41.

(196) Per the country analysis studies, FR, ES, RO and SK do put a focus on women in the area of health.

(197) WB11 (former Yugoslav Republic of Macedonia).

(198) Workshop participants. EUI-1. European Parliament, Study on the Empowerment of Roma women within the European framework of national Roma inclusion strategies, 2013, p. 10.

(199) CS-EL; CS-HU; SI1; SE3; BE4; HU4.

(200) CS-IT.

(201) CS-ES.

(202) CS-BG.

(203)  European Parliament, Resolution of 15 April 2015 on the occasion of International Roma Day — antigypsyism in Europe and EU: recognition of the memorial day of the Roma genocide during World War II, P8_TA(2015)009, http://www.europarl.europa.eu/sides/getDoc.do?type=TA&language=EN&reference=P8-TA-2015-0095 .

(204)  COM(2016) 424, p. 9 and 17, available at http://ec.europa.eu/justice/discrimination/files/roma-report-2016_en.pdf .

(205) EUROCITIES (2017), Roma Inclusion in cities — Mapping of the situation of Roma in cities in Europe ).

(206)

  http://ec.europa.eu/social/BlobServlet?docId=14000&langId=en . Open Migration, 2015, From Morocco to Romania: how immigration to Italy has changed over 10 years, http://openmigration.org/en/analyses/from-morocco-to-romania-marocco-how-immigration-to-italy-changed-in-10-years/ ; CBS, 2017, More Eastern Europeans working in the Netherlands, https://www.cbs.nl/en-gb/news/2017/05/more-eastern-europeans-working-in-the-netherlands ; The Guardian, 2017, Number of Romanians and Bulgarians in UK rises to 413 000, https://www.theguardian.com/uk-news/2017/oct/11/number-of-romanians-and-bulgarians-in-uk-rises-413000

(207) Recital (12).

(208) CS-AT and CS-FR.

(209) EUI7, EUI4, UK 4, IE1.

(210) The external evaluation study did not look into internal coherence.

(211) The findings on internal coherence are mainly based on Commission analysis.

(212) Better Regulation principles.

(213) ‘ensure that all Roma children complete at least primary school’ — much more ambitious would have been a goal on completion of secondary education for example.

(214) ‘close the gap between the share of Roma with access to housing and to public utilities and that of the rest of the population’.

(215) ‘reduce the gap in health status’ and ‘cut the employment gap’.

(216) European Roma Information Office, position paper on the EU framework for national Roma integration strategies and a post-2020 strategy, 2017; UN-OHCHR 2017: Lessons Learned: Views in the context of midterm review of implementation of the EU framework for national Roma integration strategies 2012-2020

(217) Which was more explicit under education, employment and housing than under health.

(218) European Court of Auditors, Special report: EU policy initiatives and financial support for Roma integration: significant progress made over the last decade, but additional efforts needed on the ground, 2016, https://www.eca.europa.eu/Lists/ECADocuments/SR16_14/SR_ROMA_EN.pdf .

(219) See the FRA website dedicated to Roma issues here fra.europa.eu/en/theme/roma/

(220) FRA paper ‘Measuring Roma inclusion strategies — a fundamental rights based approach to indicators’, June 2016.

(221) COM(2012) 226 final.

(222) Ex ante conditionalities are preconditions that Member States must fulfil when submitting their programming documents.

(223) Eurocities. 2017: Roma in Cities in Europe. Brussels .

(224)

Directive 2011/36/EU and the related EU policy framework take an integrated and holistic, victim-centred, gender-specific, child sensitive, human rights-based approach. Article 5 of the Charter on Fundamental Rights addresses human trafficking. Study on high-risk groups for trafficking in human beings (2015); Study on comprehensive policy review of anti-trafficking projects funded by the European Commission (2016); first progress report (COM(2016) 267); 2012-2016 EU strategy towards the eradication of trafficking in human beings (COM(2012) 286); European Commission Communication on Reporting on the follow-up to the EU strategy towards the eradication of trafficking in human beings and identifying further concrete actions , COM(2017) 728 final.

(225) COM(2017) 753 final.

(226) Compare ERGO network (2016). Youth Guarantee — Opportunities for young Roma. Findings of a small scale field research in six EU countries. http://ergonetwork.org/wp-content/uploads/2017/05/Youth-Guarantee-%E2 %80 %93-Synthesis-note_ERGO-Network.pdf.

(227) Chopin, Isabelle, Catharina Germaine, and Judit Tanczos 2017: Roma and the Enforcement of Anti-Discrimination Law, Brussels. http://ec.europa.eu/newsroom/just/document.cfm?action=display&doc_id=47560 ; FRA and EC. 2018. Handbook on European Non-Discrimination Law 2018 Edition. Vienna / Brussels: European Union Agency for Fundamental Rights. http://fra.europa.eu/en/publication/2018/handbook-european-law-non-discrimination ; EC. 2016. European Handbook on Equality Data — 2016 Revision. Brussels (European Commission — DG Justice) http://ec.europa.eu/newsroom/just/item-detail.cfm?item_id=54849

(228) EC 2014: Joint Report on the application of Racial Equality Directive and of the Employment Equality Directive — Com(2014)2, Brussels https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52014DC0002&from=EN

(229) HU, CZ and SK.

(230) The Code of Conduct on countering illegal hate speech online and the High Level Group on combating racism and xenophobia and other forms of intolerance.

(231) OPC, p. 18.

(232)  Council Regulation (EU) No 1303/2013 and Council Regulation (EU) No 1304/2013 of 17 December 2013.

(233) In total there are 36 ex ante conditionalities; their overarching aim is to help achieve an effective use of EU funds. Ex ante conditionalities are preconditions that Member States must fulfil when submitting their programming documents. If a Member State did not fulfil an ex ante conditionality, it had to present an action plan and a timetable for its implementation to the Commission.

(234) Regulation (EU) No 1303/2013 of 17 December 2013.

(235) These fulfilment criteria are: ‘It sets achievable national goals for Roma integration to bridge the gap with the general population. These targets should address the four EU Roma integration goals relating to access to education, employment, healthcare and housing. It identifies disadvantaged micro-regions or segregated neighbourhoods where communities are most deprived. It includes strong monitoring methods to evaluate the impact of Roma integration actions and a review mechanism for the adaptation of the strategy. It is designed, implemented and monitored in close cooperation and continuous dialogue with Roma civil society, regional and local authorities.’.

(236) EC 2015: Guidance for Member States on the use of European Structural and Investment Funds in tackling educational and spatial segregation, Brussels ; EC 2011: Guidance note on the implementation of integrated housing interventions in favour of marginalised communities under the ERDF , Brussels;.

(237) This guidance also stressed the partnership principle in ESIF and that representatives of marginalised groups should be involved in the partnership organised by Member States around partnership agreements and operational programmes.

(238) EURoma 2014. Promoting the Use of Structural Funds for Roma Inclusion in the European Union — A Glance at EURoma’s 8 years of work and how Roma inclusion is considered in the 2014-2020 programming period, 2014.

(239) Centre for European Policy Studies, Research Report: Combating Institutional Antigypsyism. Responses and promising practices in the EU and selected Member States , No 2017/08, 2017,.

(240) EURoma 2014. Promoting the Use of Structural Funds for Roma Inclusion in the European Union — A Glance at EURoma’s 8 years of work and how Roma inclusion Is considered in the 2014-2020 programming period, 2014.

(241) Roma Civil Society Monitoring Initiative (RCM): Summary of findings year 1 country reports; unpublished; April 2018.

(242)

 The implementation of the provisions in relation to the ex ante conditionalities during the programming phase of the European Structural and Investment (ESI) funds’, DG REGIO, 2016.

(243)  CS-ES, CS-AT, CS-BG.

(244) AT, BG, CZ, EL, ES, FI, FR, IT, HU, RO, SK.

(245) Turkey has allocated funds from the annual budgets of the relevant ministries to implement some of the actions provided for in the national action plan.

(246) NGO survey report, p. 15.

(247) Chapter 8 of COM(2011) 173 final.

(248)

Reporting by the Commission has been carried out since 2012 in the form of country-by-country and horizontal guidance. For the enlargement countries, country-specific descriptions of the situation of the Roma were added in 2016.

(249) Recital (19) of 2013 Council Recommendation.

(250) Paragraph 3.8 of 2013 Council Recommendation (National Contact Points for Roma integration).

(251) Paragraph 4 of 2013 Council Recommendation (reporting and follow-up).

(252) Idem.

(253) European Parliament, Evaluation of the EU framework for national Roma integration strategies, 2015, http://www.lse.ac.uk/business-and-consultancy/consulting/assets/documents/evaluation-of-the-EU-Framework-for-National-Roma-Integration-Strategies.pdf . The evaluation focused on the following countries: Bulgaria, France, Hungary, Italy, Romania, Slovakia and Spain.

(254) For the 2015 reporting year.

(255) In SFC.

(256)  Council Directive 2000/43/EC of 29 June 2000.

(257) See the chapter on effectiveness, in particular alignment between the EU framework and NRIS.

(258) AT, CY, CZ, DE, EE, EL, FI, FR, HR, IE, LT, NL, PT, SE, SI, UK. Additionally, a West Balkan regional body and Bosnia and Herzegovina.

(259) CY, CZ, EL, ES, FI, HR, LT, LV, SI.

(260) This could be assessed fairly straightforwardly by examining the number of reported measures (939 in 2016, and 937 in 2015) and trying to assess the average time to complete the information on one measure.

(261) In line with the common basic principles on Roma inclusion already agreed in 2009 by the Council: http://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/lsa/108377.pdf

(262) Kertesi, Gábor, and Gábor Kézdi. 2006. Expected long-term budgetary benefits to Roma education in Hungary. No. BWP-2006/5. Budapest working papers on the labour market, http://www.romaeducationfund.org/sites/default/files/publications/kertesi-kezdi-budgetarybenefits.pdf

(263) de Laat, J., Bodewig, C., Arnhold, N., Linden, T., Dulitzky, D., Kosko, S. and Torracchi, T., 2010. Roma Inclusion: An Economic Opportunity for Bulgaria, Czech Republic, Romania and Serbia. Policy note, World Bank, Human Development Sector Unit, Washington, DC.

(264) Marcinčin, A. and Marcinčinová, L., 2009. The Cost of Non-inclusion. The key to integration is respect for diversity. http://childhub.org/sites/default/files/library/attachments/the_cost_of_non_inclusion.pdf

(265) The question was replied to by the external study; this evaluation finds that data limitations are too strong for the findings to be reliable.

(266) See information about calculations in Annex 5.

(267) In particular for those countries having a sizeable Roma population.

(268) See ICF, Chapter 3.7.1 for country-specific examples.

(269) AL, BiH, BG, CZ, ES, HU, HR, former Yugoslav Republic of Macedonia, ME, RO, RS, SK.

(270) Confirmed for example in CS-SK, ‘added value’ section.

(271)

 When broken down by respondent type, the response trends (public institutions, NGOs/think tanks, citizens) or and country cluster (EU-15, EU-13, enlargement) were overall consistent with the overall response trends.

(272) Open Society European Policy Institute, Report: Revisiting the EU Roma Framework: Assessing the European Dimension for the Post-2020 Future 2017, https://www.opensocietyfoundations.org/reports/revisiting-eu-roma-framework-assessing-european-dimension-post-2020-future .

(273) In the workshop discussion, 97 % of all participants agreed that there is EU added value with regard to Roma inclusion in the Member States.

(274) BE1, BG6, BG8, DE1, FR2, HR1, IE1, LT1, NL1.

(275) CS-AT, CS-ES, CS-IT.

(276) CS-ES, CS-IT.

(277) For example, in Spain the EU framework has reinforced the participation of the Roma population through the National Roma Council.

(278) BE5, FR3, LT1, EU-level interviews EU1, EU2.

(279) IE2, LV1, PT3, SE3, WB3.

(280) For example, in Austria EU action has allowed for targeted projects to be implemented at a (monetary) scale that far exceeded previous targeted projects, thus allowing for projects to be scaled up and greater numbers of Roma to be reached. In Bulgaria, the scale of the programmes, especially in the areas of education and employment, is unlikely to have been the same as the one enabled via the EU funds. In Finland also, the measures could not have been accomplished by Finland alone, without the support of EU structural funds, with the exception of the education measures carried out using Finnish State funds. Similarly in Greece, almost all measures in the areas of education, employment, housing, health and anti-discrimination (excluding those aimed at combating poverty) are EU co-funded projects, the implementation of which would be difficult if these funds were not available, especially during a period of economic crisis and structural and financial adjustments. In Italy, without the EU framework there would be no NRIS, and without the NRIS the situation would be much more similar to the one seen in 2009 (IT-2, IT-3). Source: country analysis studies, ‘added value’ section. However, these views could not be quantitatively confirmed due to an overall lack of robust data on ESIF funding.

(281)

ICF, Chapter 3.7.4 underlines that this overall conclusion was confirmed by evidence gathered though the case studies, stakeholder workshop discussion, national- and EU-level interviews and the NGO survey.

(282)

 The important role of EU funding programmes was confirmed by 60 % of the respondents of the NGO survey, who highlighted a lack of sufficient funding at national level.

(283) DE3, FR6, IE4, NL1, SI1.

(284) CZ4, AT11, ES2, ES10, EUI1, FI2, HR1, IE4, LT1, PT4, SK8, RO1, UK1.

(285) CZ9, CZ4, AT11, ES2, ES10, EUI1, FI2, HR1, IE4, LT1, PT4, SK8, RO1, UK1. BE4, ES1.

(286) ICF, Chapter 8, as documented in country analysis studies and interviews.

(287) Open Society European Policy Institute 2017: Revisiting the EU Roma Framework: Assessing the European Dimension for the Post-2020 Future , Budapest/Brussels.

(288) COM(2011) 0173 final.

(289) ICF, Chapter 3.8.2; this was stressed by several interviewees, e.g. BE-3; BE-5; DK1; ES1; ES10; SK5 and in country analysis studies AT, BG, FI, FR, EL.

(290) Open Society European Policy Institute, Report. Revisiting the EU Roma framework: Assessing the European dimension for the post-2020 future, 2017,.

(291) See the successful programming principles outlined in EC 2012: What works for Roma inclusion in the EU — Policies and model approaches , Brussels.

(292) EPRD, 2015 ‘ Thematic Evaluation on IPA Support to Roma Communities’ :.

(293) Ibid.; ICF, Chapter 3.8 and country analysis studies BG, FR, EL.

(294) IES (2014) The Missing Piece: Empowerment of Roma Grassroots Organisations in EU Roma Integration Policies .

(295) ERIO (2017), Position paper on the EU Framework for National Roma Integration Strategies and a post-2020 strategy as a contribution to the Mid-term review of the European Commission ; Open Society European Policy Institute (2017), Revisiting the EU Roma Framework: Assessing the European Dimension for the Post-2020 Future . To be noted that the Commission financially supports the European Roma Grassroot Network.

(296) Roma Civil Monitor 2018: Synthesis Report on the Implementation of the EU Framework for National Roma Integration Strategies in Bulgaria, Czech Republic, Hungary, Romania and Slovakia . Brussels / Budapest.

(297) See Section 2, description of the initiative; in particular: Designing NRIS taking into account the size of the Roma population and different starting points; also non-binding approach to measures to be implemented by Member States.

(298) COM(2017) 753 final.

(299) ICF, Chapter 8, as documented in country analysis studies and interviews.

(300)   https://ec.europa.eu/commission/publications/investing-people_en for ESF+ proposal adopted by the Commission on 30 May 2018; https://ec.europa.eu/commission/publications/regional-development-and-cohesion_en for Common Provisions Regulation proposal adopted by the Commission on 29 May 2018.

(301) ARES(2018)4995826

(302)   https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2017-1716166_en

(303) Bibliography is provided in the external evaluation study; in addition around 200 national sources were included in the country analysis studies.

(304) Operational programmes; synthesis reports of annual implementation reports such as for the ESF 2017

(305) System for fund management in the European Union.

(306) Add link once published.

(307) It cannot be ascertained whether representatives of organisations representing churches and religious communities were reached, as the identity categories of the OPC were: non-governmental organisation, public, administration, business, employer organisation, trade union, association, academia/research/think tank and other.

(308) The online consultation form was published on the EU survey page: https://ec.europa.eu/eusurvey/runner/EvaluationEUFrameworkforNRIS .

(309) All questions were optional except those on self-identification.

(310)

The results of the consultation are published on the European Commission website and are available here: https://ec.europa.eu/info/consultations/public-consultation-evaluation-eu-framework-national-roma-integration-strategies-2020_en.

(311) AT, BG, CZ, ES, EL, FR, FI, IT, HU, SK and RO.

(312) BE, CY, DE, DK, EE, IE, HR, LV, LT, LU, NL, PL, PT, SE, SI and UK.

(313) It was intended that 64 telephone interviews would be conducted with stakeholders in the 16 Member States not covered by a country analysis study and in enlargement countries, while up to 20 interviews would be conducted in each of the 11 Member States covered by the country analysis studies.

(314) It was intended that 10 European- or international-level stakeholders would be interviewed; despite repeated efforts, it was only possible to interview eight.

(315) Contacts provided by the Commission.

(316) https://en.wikipedia.org/wiki/NVivo.

(317) OPC results available here: https://ec.europa.eu/info/consultations/public-consultation-evaluation-eu-framework-national-roma-integration-strategies-2020_en.

(318) EU added value.

(319) EU added value.

(320) Coordination; EU added value.

(321) Coordination.

(322) Effectiveness.

(323) Relevance.

(324) Comprising 138 stakeholder consultations across 11 Member States covered by a country analysis study; 53 stakeholder consultations across 16 Member States not covered by a country analysis study; and 10 stakeholder consultations with stakeholders from enlargement countries.

(325) Link to external evaluation study once published.

(326) Add footnote once published.

(327)

 Effectiveness.

(328) idem.

(329) Coordination.

(330) Effectiveness.

(331) EU added value.

(332) Results as presented in the external study not relevant for this criterion.

(333) Results as presented in the external study not relevant for this criterion.

(334) ICF, inception report.

(335) Operational Programmes; synthesis reports of annual implementation reports.

(336)  https://en.wikipedia.org/wiki/NVivo.

(337) European Commission (2017); Network of legal experts ‘Data collection in the field of ethnicity’; Lila Farkas; European Commission (2017) Network of legal experts ‘Legal framework and practice in the EU Member States’.

(338) European Commission (2017); Network of legal experts ‘Data collection in the field of ethnicity’; Lila Farkas.

(339) Known exceptions on the national level for SILC are BG and HU, which collect ethnically disaggregated statistics in their national rounds of SILC.

(340) Except for those covered by the 2011 FRA survey (BG, CZ, EL, ES, HR, HU, PT, RO, SK).

(341) For the ESF 2014-2020 for example, the common output indicator is ‘migrants, participants with a foreign background, minorities (including marginalised communities such as the Roma)’. In countries without significant migrant inflows one can assume that a large majority of the participants are actually Roma, to be confirmed through the content of the specific investment priority in an operational programme.

(342) See also FRA Roma integration scoreboard, SWD(2017) 286 final and FRA 2018 ‘ A persisting concern: anti-Gypsyism as a barrier to Roma inclusion ’.

(343)       http://fra.europa.eu/en/publication/2016/eumidis-ii-roma-selected-findings

(344)       http://fra.europa.eu/en/survey/2012/roma-pilot-survey

(345)       http://www.eurasia.undp.org/content/rbec/en/home/ourwork/sustainable-development/development-planning-and-inclusive-sustainable-growth/roma-in-central-and-southeast-europe/roma-data.html

(346) During national censuses, typically more than a third of Roma across the EU do not declare their ethnicity.

(347) For instance, some more ‘integrated’ Roma or others aspiring upward social mobility may self-identify with the ethnicity of the majority population. In other cases, respondents who are more sensitive to discrimination or have experienced more discrimination may also identify as the majority population to avoid stigmatisation.

(348)   http://www.eurasia.undp.org/content/rbec/en/home/ourwork/sustainable-development/development-planning-and-inclusive-sustainable-growth/roma-in-central-and-southeast-europe/roma-data.html

(349)   http://www.eurasia.undp.org/content/rbec/en/home/library/roma/regional-roma-survey-2017-country-fact-sheets.html

(350)  ‘Socio-economic integration of marginalised communities such as the Roma’.

(351)

This was in contrast to the external study which, however, also concluded that there is a high degree of uncertainty about the values. One of the lessons learnt from this evaluation is the need to improve data collection and comparability in the reporting tool (see conclusions).

(352) ICF, Chapter 3.6.2 — see limitations in Annex 4 to this SWD.

(353) Compare European Parliamentary Research Service; Cost on Non-Europe Report ‘Equality and the Fight against Racism and Xenophobia’, March 2018; based on conceptual framework for the impacts of discrimination presented by Milieu (see below).

(354) See explanations and limitations below

(355) Data provided by Commission departments.

(356) Data provided by Commission departments.

(357) Magnuson and Duncan (2014), ‘Can early childhood interventions decrease inequality of economic opportunity?’; Heckman, J.J (2006), Skill formation and the economics of investing in disadvantaged children.

(358) Janna van Belle (2011) Early Childhood Education and Care (ECEC) and its long-term effects on educational and labour market outcomes

(359) West, A (2016) L’école maternelle à la source de la réduction des inégalités sociales: une comparaison internationale; Unesco (2011), Education Counts, Toward the Millennium Development Goals, p. 6.

(360) In contrast to the external evaluation study.

(361) Janna van Belle (2011) Early Childhood Education and Care (ECEC) and its long-term effects on educational and labour market outcomes

(362)

Brunello, G and De Paola, M (2013) The costs of early school leaving in Europe; the study found that these costs are private, fiscal and social. Costs due to lost private benefits include the expected gains in earnings and wealth, improved health and life expectancy and higher lifetime satisfaction. Costs related to lost fiscal benefits include increased tax payments, lower reliance on government transfers and reduced expenditures on criminal justice. Social costs related to lost social benefits include productivity externalities, the social value of better health and the gains from reduced crime.

(363) See for example European Commission: Overview and examples of costs of early school leaving in Europe

(364) Figures are for Roma aged 16+

(365) Age group 16-64; Source: World Bank

(366) Eurofound (2015), Young People and ‘NEETs’.

(367) See in particular World Bank (2010) — Roma Inclusion: An economic opportunity for Bulgaria, Czech Republic, Romania and Serbia; The World Bank (2010) used survey data from 6 EU MS to compare average working-age Roma and non-Roma. It built a model that draws on: (i) the probability of employment; and (ii) the average wage, conditional on employment of both groups. The difference between the average expected earnings for Roma and non-Roma is the average earnings gap per working-age individual. Total productivity losses are then calculated using the estimated number of Roma working-age individuals. Using the same data on wages and probability of employment, the World Bank estimated (partially) the fiscal contribution opportunity costs of Roma exclusions, i.e. — the forgone income tax and social security payments.