EUROPEAN COMMISSION

Brussels, 29.6.2018

SWD(2018) 356 final

COMMISSION STAFF WORKING DOCUMENT

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS

on the interim evaluation of the implementation of the Justice Programme 2014-2020

{COM(2018) 507 final}
{SWD(2018) 357 final}

Table of contents

1.Introduction

2.Background to the initiative

3.Implementation / state of Play

4.Method

5.Analysis and answers to the evaluation questions

6.Conclusions and way forward

Annex 1: Procedural information

Annex 2: Stakeholders consultation

Annex 3: Methods

1.Introduction

Purpose and scope of the evaluation

This staff working document sets out the results of the interim evaluation of the Justice Programme for the period 2014-2020.

The evaluation was carried out under the provisions of Article 14(2)(b) of the European Parliament and Council Regulation establishing the Justice Programme for the period 2014-2020 1 .

It aimed mainly to assess the Programme’s outputs and results compared to its objectives and to assess qualitative and quantitative aspects of its implementation. It also assessed whether the current Programme is on track to achieving its objectives and the extent to which the recommendations of the previous 2007-2013 ex-post evaluation have been followed. The ex-post evaluation carried out in 2015 concerned the three predecessor programmes which were merged in the current Justice Programme (namely the Civil justice programme, the Criminal justice programme and the Drug prevention and information programme) 2 .

The current interim evaluation informed the Report from the Commission to the European Parliament, the Council of the European Union, the European Economic and Social Committee and the Committee of Regions which this document accompanies. It also helped the reflection on the Programme's future by providing evidence-based information.

Under the Regulation, the Commission is requested to present an ex-post evaluation report for the Justice Programme by 31 December 2021.

The Programme was assessed based on the following main evaluation criteria:

1. Relevance: whether and to what extent the Justice Programme addresses needs and problems of the target groups identified in the 2011 Impact Assessment 3 and in the legal basis of the Programme (as well as emergent needs related to the creation of a European Area of Justice) and whether its objectives are still relevant for the needs and problems of the beneficiaries;

2. Efficiency: whether and to what extent the costs of the Programme were proportionate given the benefits achieved and which parameters/factors participated in these results;

3. Effectiveness: whether and to what extent the Justice Programme has achieved its general objective, as well as its four specific objectives, and which are the factors that have contributed to these achievements;

4. Coherence/Complementarity/Synergies: whether and to what extent the Programme is coherent with other interventions at the EU and international level, such as with the predecessor EU programmes in the field 4 , with activities supported by other Union instruments and, in general, with the European priorities in the fields covered by the Programme;

5. EU added-value: to what extent the effects from the EU action are additional to the value that would have resulted from action at the national level only;

6. Equity: whether and to what extent the Justice Programme has distributed the available resources fairly among beneficiaries in different Member States, took into consideration the needs of target groups, promoted gender mainstreaming, the rights of the child and the rights of people with disabilities 5 ;

7. Scope for simplification: whether and to what extent the management of the Justice Programme could be further simplified.

Geographically, the Programme is open to all EU Member States 6 , but also to the European Free Trade Association States that are party to the European Economic Area, candidate countries, potential candidates and countries acceding to the Union, provided that they conclude an agreement with the Union laying down the details of their respective participation in the programme. Albania joined the Programme as of 2017.

The reference period for this interim evaluation is the first half time of the Programme’s implementation from 2014 to mid-2017 7 .

2.Background to the initiative

The Treaty on the Functioning of the European Union provides for the creation of a European area of freedom, security and justice based on mutual recognition of judicial decisions and mutual trust among Member States, in which persons are free to move and can rely on the respect of fundamental rights as well as of common principles, such as non-discrimination, gender equality, effective access to justice for all, the rule of law and well-functioning independent judicial systems.

These ambitious goals, set by the Treaty, have also been reaffirmed by the European Council in the Stockholm Programme 8 . The achievement of a Europe of law and justice is one of the political priorities of the EU and the 2014-2020 Justice Programme is one of the EU instruments that contribute achieving this objective.

The Programme was designed to overcome the obstacles in the functioning of an effective European Area of Justice 9 and to encourage national judicial systems to have faith in each other’s standards of fairness and justice, as prerequisite for the realisation of a European Area of Justice. The Programme also contributes directly to the Juncker priority aimed at establishing an area of justice built on respect of fundamental rights and mutual trust, and digital single market.

Description of the programme

The general objective of the Justice Programme is to contribute to the development of a European area of justice based on mutual recognition and mutual trust, in particular by promoting judicial cooperation in civil and criminal matters.

In order to do so, the Justice Programme provides for four specific objectives which aim to:

·facilitate and support judicial cooperation in civil and criminal matters. Through this objective, the Programme supports activities that contribute to:

§the effective and coherent application of the EU acquis relating to judicial cooperation in civil and criminal matters also by building-up and/or improving data collection and statistics on the application of the EU acquis;

§the enforcement of EU instruments and judicial decisions, in particular resulting from cross-border disputes;

§the improvement of the exchange of information among professionals in order to enhance the operational cooperation and mutual trust in the EU. 

Under this specific objective, the Programme supports the participation of the EU to the Hague Conference of Private International Law 10 and activities related to the setting up and strengthening of national networks under the European Judicial Network in civil and commercial areas. Moreover, specific calls for proposals to prevent radicalisation leading to terrorism and violent extremism have also been launched in the wake of terrorism episodes in the EU. In the context of this specific objective, the EU carries-out its cooperation with the Council of Europe on two particular issues: the SPACE report on prison statistics and the setting up of the EU network of prison monitoring bodies in the Member States.

·support and promote judicial training of professionals such as judges, prosecutors, notaries, prison staff and lawyers on civil and criminal law EU instruments, fundamental rights, judicial ethics and the rule of law, including language training on legal terminology, with a view to fostering a common legal and judicial culture. Judicial training can involve “basic” components, such linguistic skills and terminology, and more specialised aspects, such as seminars on specific aspects of both civil and criminal law, e-learning and exchanges of staff and experience. 11 For example, specialised training is made available in the field of competition law, which typically requires a significant amount of specialisation. The Programme supports, in particular, the training of the members of the judiciary and judicial staff, but also other legal practitioners associated with the judiciary. It also funds the development of tools for training providers and co-finances the expenditure associated with the work of the European Judicial Training Network.

Certain areas represent priorities for the training activities: EU civil, criminal and fundamental rights law, legal systems of the Member States, judicial ethics and the rule of law, knowledge of cross-border IT tools and linguistic skills.

Moreover, the Programme finances projects which aim at achieving the objectives of the European e-Justice Strategy 2014-2018 12 and that contribute to the effective and coherent application of EU criminal law in the areas of the rights of persons suspected or accused of crime (procedural rights priority) and of the rights of victims of crime;

·facilitate effective access to justice for all, including to promote and support the rights of victims of crime as well as the procedural rights of suspects and accused persons in criminal proceedings. Actions financed in this area by the Programme aim at providing EU citizens with effective remedies in case of violation of EU law, especially where national procedures are too difficult for citizens to be enabled. In particular, the Programme aims to promote the use of other types of remedies and non-remedies developed in the EU that can provide a quick, efficient and less costly solution to disputes, as supported, for example, by the e-Justice Portal. 13 It also aims to encourage a close cooperation between national authorities or administrative bodies, which is particularly important for the effectiveness of certain EU rights.

·promote initiatives in the field of drug policy as regards judicial cooperation and crime prevention aspects 14 . In this field, the Justice Programme promotes initiatives which focus on judicial cooperation and crime prevention. The activities under the Programme mainly aim to promote practical application of drug-related research, to support civil society organisations and key stakeholders and to expand the knowledge base and develop innovative methods of addressing the phenomenon of new psychoactive substances.

Budget

The initial total Justice programme's budget for the period 2014-2020 is EUR 377 604 000. According to available sources, the total requested and committed EU contribution still not reached the amount planned, despite the improvements made in 2016 (see Table 1). In general, most resources allocated to action grants and operating grants have been committed (with a commitment rate of close to 90%), but the commitment rate for procurement activities was much lower, hovering around 60%-70%.

Table 1: Total amount planned and committed in the Justice Programme 2014-2016

Sources: Annual monitoring reports (reports on the implementation of the Annual working programmes and data extracted from Sygma for 2016)

In order to allow the achievement of its objectives, the Programme identifies a wide range of activities to be implemented. In particular, according to Article 6 of the Regulation establishing the Justice Programme for the period 2014 to 2020, the Programme can finance:

·Analytical activities, in particular: collection of data and statistics; development of common methodologies and, where appropriate, indicators or benchmarks; studies, researches, analyses and surveys; evaluations; elaboration and publication of guides, reports and educational material; workshops, seminars, experts meetings and conferences;

·Mutual learning, cooperation, awareness raising and dissemination activities, in particular: identification of, and exchanges concerning good practices, innovative approaches and experiences; organisation of peer reviews and mutual learning; organisation of conferences, seminars, information campaigns; compilation and publication of materials to disseminate information about the Programme and its results; development, operation and maintenance of systems and tools, using information and communication technologies, including the further development of the European e-Justice portal as a tool to improve citizens' access to justice;

·Training activities, for instance: staff exchanges, workshops, seminars, train-the-trainer events, including language training on legal terminology, and the development of online training tools or other training modules for members of the judiciary and judicial staff;

·Actions to support the main actors whose activities contribute to the implementation of the objectives of the Programme, such as: support for Member States in the implementation of Union law and policies; support for key European actors and European-level networks, including in the field of judicial training; support for networking activities at European level among specialised bodies and entities as well as national, regional and local authorities and non-governmental organisations 15 .

The main stakeholders eligible for support are: European networks active in the area of judicial cooperation; justice professionals; public or private organisations (usually non-profit-oriented, but in particular cases also profit-oriented organisations 16 ); national, regional and local authorities in EU Member States; non-governmental organisations; national judicial schools of individual Member States; universities and research institutions as well as international organisations active in the areas covered by the Programme. 

In terms of target groups, intended as the groups that can benefit directly or indirectly from the Programme, potentially all EU citizens are included, since the Programme aims at creating a European Area of Justice where all citizens are aware of and can exercise their rights. However, in particular, the Justice Programme covers: judicial practitioners, public authorities, universities, international organisations, non-governmental organisations and other research entities; companies undergoing insolvency or pre-insolvency proceedings; citizens accused or victim of a crime, families, consumers, minors, groups in need of specialised legal protection and subjects at risk of radicalisation. Additionally, specific calls support entrepreneurs in the single market and national drug awareness and prevention agencies 17 .

The Justice Programme has three main funding mechanisms: action grants, operating grants and procurement actions 18 .

·Action grants are addressed to civil society organizations, to Member States authorities and universities;

·Operating grants ("support to Networks") fund mainly European networks active in the area of facilitating and supporting judicial cooperation in civil and/or criminal matters that have signed Framework Partnership Agreements with the Commission;

·Procurement actions ("Commission initiatives") fund mostly conferences, expert meetings, seminars, awareness-raising activities and studies, but also translation services and IT projects.

The Programme intervention logic is outlined in the Figure 1.

External factors could influence the Programme's outcomes and make challenging to identify changes as a result of its intervention. In particular, the economic crisis has led to a lack of interest on the part of public sector stakeholders in the areas covered by the Programme (particularly in the field of access to justice) and, therefore, to a related reduction of national funding for the justice systems. This is more accentuated in certain Member States where the political climate is less supportive 19 . Also the uneven pace at which Member States incorporate the EU acquis in their national legislations is a factor influencing the performance of the Programme 20 . 

Baseline and points of comparison

The interim evaluation assessed the Justice Programme performance starting from the situation described in the 2011 Impact Assessment 21 and in the ex-post evaluation of 2007-2013, concerning the three predecessors programmes (Civil justice programme, the Criminal justice programme and the Drug prevention and information programme), carried out in 2015 22 .

These documents contain the baselines and main points of comparison for measuring the Programme’s achievements during the reporting period.

The baseline analysis showed that, if no changes had been made between the 2007-13 and 2014-20 programming period, the three previous programmes would have continued to be successfully implemented, but at a reduced potential in particular due to: 1. The lack of flexibility in the funding instruments which did not reflect the pace of change and reform in this policy area; 2. The fragmentation of funding that reduced the capacity of the programmes to deliver results in horizontal and cross-cutting issues; and 3. The elevated number of different funding instruments which increased the administrative burden.

Given these difficulties, the Impact Assessment suggested the option to consolidate the three programmes in the current Justice Programme.

This possibility was assessed in terms of its relevance/scope, effectiveness, complementarity, European added value, efficiency and potential for simplification. Since the option to proceed with only one programme was undertaken, the findings of the Impact Assessment, together with the recommendations of the ex-post evaluations of 2007-2013, provide a useful baseline to verify whether the current Justice Programme achieved its expected goals.

Follow-up on the ex-post evaluation report of the period 2007-2013

All the ex-post evaluation reports concerning the previous funding programmes confirmed their overall effectiveness and highlighted that their specific objectives and priorities were largely specific, attainable and realistic, but not always measurable. Indicators, allowing the measuring of progress toward the attainment of the specific objectives, were not included.

Moreover, the reports pointed out the need to make improvements on other issues which could have increased the impact, added value, effectiveness and efficiency in delivering result of the programmes. In particular, the reports have identified the following needs common to the three programmes:

1. Better definition of the priorities in order to ensure that they can be adequately achieved within an earmarked budget; 

2. Realistic assessments of project risks and better risk mitigation strategies throughout the projects duration;

3. Increase focus on assessment of impacts at all levels and not merely on outputs, as regards monitoring and evaluation;

4. Explore ways of enhancing the uptake of project outputs, results and best practices by other organisations, including in other Member States;

5. More dissemination/use of results and outputs of the funded activities.

The ex-post evaluation reports of the period 2007-2013 have also highlighted the following problems: 1. The dilution of funds amongst many small-scale projects with limited impact and EU dimension; 2. No balanced geographical spread among the organisations which receive funding; 3. The complex and bureaucratic procedures for the applicants; 4. The high administrative burden on the Commission and an increase of the length of procedures due to the multiplication of procedures for the different programmes.

These issues were taken into consideration and were either integrated into the Regulation establishing the current Justice Programme and in the Annual Work Programmes or translated into technical implementation measures and included in the Guides for applicants to projects calls.

In particular, the merger of the three previous programmes has provided a significant positive impact on the identified problems of scope, effectiveness, fragmentation and efficiency (see further details in section 5 “Analysis and answer to the evaluation questions”). 

3.Implementation / state of Play

Programme management

The Justice Programme is implemented via direct centralised management by the Commission. This was meant to ensure a close relationship between the Programme management and EU policy-making and to contribute to the alignment of budget implementation with EU policy priorities in line with the "budget for result" approach. This management mode allows the Commission to tailor funded activities to policy priorities and policy needs and to target directly the relevant groups of stakeholders. According to the evaluation carried out, it also allows a close contact with the programmes' beneficiaries and better knowledge of the needs on the ground. Indeed, all beneficiaries interviewed, who also participated in predecessor programmes, agreed that the current Justice Programme represented an improvement in terms of capacity to engage with European Commission officials (i.e. having a reliable point of contact to answer their questions and help them address any project-related issues that may arise during the implementation phase) 23 .

State of play

The purpose of this section is to present the state of play of the Programme’s implementation and its key initiatives in the period 2014-2017. It will also provide the qualitative and quantitative results achieved in the same period, in general terms and for each specific objective.

In order to measure the progress from the baseline situation, a series of Programme indicators have been devised for the current Justice Programme. The introduction of a system of indicators has proved to be adequate for measuring the achievements of the Programme since the selected indicators show if the targets are close to being achieved or have already been achieved and this contributes to better focus the Programme's outcomes.

·General objective: The development of a European area of justice based on mutual recognition and mutual trust

The Regulation establishing the Justice Programme does not provide for any indicator concerning its general objective. However, the Commission reports via the Programme Statements, accompanying the Draft General Budget of the EU 24 , use the percentage of legal practitioners trained (not only through this Programme) as the main indicator for measuring the Justice Programme’s impact, as shown in Table 2.

Table 2: Result indicator, target and actual value

Source: Annual reports on European Judicial Training

The 2020 target was almost reached already in 2017 25 . The graphic below (Figure 2) shows clearly that the number of legal practitioners trained has constantly increased between 2013 and 2016.

Figure 2: number of legal practitioners trained

Concerning the Specific objectives of the Programme, the following indicators have been used to evaluate its performance:

·Specific objective 1: Judicial cooperation in civil and criminal matters

The Programme's performance under this specific objective is measured through two main indicators (see Table 3): the average time of the surrender procedure under the European Arrest Warrant in cases where the person consents to the surrender and the number of exchanges of information in the European Criminal Records Information System. The European Arrest Warrant is the most important EU legal instrument developed in the area of mutual recognition of judicial decisions. It consists in a simplified cross-border judicial surrender procedure for the purpose of prosecuting or executing a custodial sentence or detention order (a warrant issued by one EU Member State's judicial authority is valid in the entire territory of the EU) 26 .

The European Criminal Records Information System is a database established to improve the exchange of information on criminal records throughout the EU. All EU Member States are currently connected to this system 27 . 

Concerning the first indicator, the data appears to be highly influenced by exogenous factors (such as the different degree of incorporation of EU directives at national level, as well as the different degree of judicial system reforms within EU Member States that have a significant impact on the level of enforcement of the EU acquis 28 ) that make estimating the contribution of the Justice Programme to the realization of this indicator difficult. Concerning the second indicator, after a rapid increase post the 2012 baseline, the number of exchanges appears to have stabilised and it is likely that this objective will be reached.

Table 3: Result indicators, target and actual values

·Specific objective 2: Judicial training

The Programme's performance concerning this specific objective is measured through one main indicator: the number and percentage of members of the judiciary and judicial staff that participated in training activities, staff exchanges, study visits, workshops and seminars funded by the Programme.

This indicator appears well-adapted to the goals of the specific objective and of the Programme, as it helps track both the overall contribution of the EU funding instruments of the European Commission and the specific contribution of the Justice Programme. It is clearly measurable and it is showing that, looking at the entire Commission, the goal of training 20 000 practitioners yearly by 2020 has been already achieved (see Table 4). The drop in 2016 below this threshold is likely misleading, as data from the European Social Fund-funded judicial training is currently missing 29 . As ca. 3 000 legal practitioners were trained in 2015 with European Social Fund funding, it is quite likely that once this data are reported, the objective will have been achieved for 2016 as well.

Table 4: Result indicator, target and actual value

·Specific objective 3: Access to justice

The Programme's performance concerning this specific objective is measured through two main indicators: the number of hits on the e-justice portal and the number of Victim Support Organisations with national coverage (see Table 5). The e-justice portal provides information on justice systems and improves and facilitates the access to justice throughout the EU, in 23 languages 30 . The number of Victim Support Organisations is an important indicator to assess the implementation of Directive 2012/29/EU establishing minimum standards for the rights, support and protection of victims of crime. This directive contains important procedural provisions for victims of crime regarding, in particular, their right to be heard and have access to support services (i.e. Victim Support Organisations).

Concerning the first indicator, it has already exceeded the stated goals for 2020. Concerning the second indicator, it seems likely that it will be reached, though it partly depends on whether Member States/Victim Support Organisations will implement quality standards, which is partially out of the control of the Justice Programme and could be influenced by exogenous factors, such as the uneven pace at which Member States implement and enforce the EU acquis or other political priorities or emergencies in the Member States (in particular the economic crises has weakened the civil society engagement for this specific objective).

Table 5: Result indicators, target and actual values

·Specific objective 4: Drugs policy

The Programme's performance concerning this specific objective is measured through two main indicators: the number of new psychoactive substances assessed (including through testing, if necessary) to enable the EU or the Member States to take appropriate action to protect consumers, depending on the type and level of risk that they may pose when consumed by humans and the % of problem opioid users that are in drug treatment.

Given the stability observed over time of indicator 2 (which has remained at the 50% for the last five years), it seems difficult that this percentage could reach 60% in 2020. Concerning the first indicator, the reverse is true, as figures appear to fluctuate significantly. The 2020 yearly goal was reached in 2015, but in 2016 the number returned near to the 2012 baseline (see Table 6). This indicator can be influenced by different exogenous factors, such as macroeconomic conditions, rate of development of new psychoactive substances, etc.

Table 6: Result indicator, target and actual value

Moreover, a series of Programme-related indicators are also mentioned in the legal basis of the Justice Programme 32 . They provide a basis for monitoring and evaluating the extent to which each of the Programme's specific objectives has been achieved. These are: 1. The number of persons reached by awareness-raising activities funded by the Programme; 2. The improvement in the level of knowledge of Union law and policies in the groups participating in activities funded by the Programme compared to the entire target group; 3. The number of cases, activities and outputs of cross-border cooperation, including cooperation by means of information technology tools and procedures established at Union level; 4. Participants' assessment of the activities in which they participated and of their (expected) sustainability; and 5.The geographical coverage of the activities funded by the Programme.

Programme indicators 1 and 3 relate to the outputs and outcomes of funded activities and can be calculated through the final reports of the projects. At the moment, there are not sufficient available data to come to consolidated conclusion on these indicators. Programme indicators 2 and 4 relate to the perception of target groups concerning the activities they participated in and the level of knowledge they acquired. The indicator 4 is not measurable at this stage, instead for the indicator 2, no surveys on the perception of target groups were deployed and, therefore, it cannot be measured. Programme Indicator 5 speaks to the equity in distribution of activities funded by the Programme amongst beneficiaries in the 26 participating Member States. For projects financed by means of applications connected to the Annual Working Programmes of 2014 and 2015, beneficiaries from all Member States benefitted in some capacity (either as coordinators or as participants), while for 2016 no beneficiaries from Estonia received funding. The goal is to ensure 100% coverage through 2020 and it has been largely respected so far (Member States coverage of participating organisations in 2016 was of 96%). However, this indicator does not capture two essential elements connected to the equity dimensions, namely: (a) the unfulfilled ‘demand’ of applications (i.e. ratio of beneficiaries / applications for each Member State) and (b) the distribution of programme resources.

In general, there is no baseline available for all these indicators, as they relate directly to the specific actions of the Justice Programme, and there are also no defined targets, except for the geographical coverage.

Other data, collected during the interim evaluation, show the current state of play of the Programme’s implementation and its key initiatives in the period 2014-2017.

As regards the type of target groups, the Justice Programme focuses mainly on research institutions/universities, victims, young people and families (see Figure 3).

Figure 3: Text mining of target groups of the Justice Programme, by Specific objective

Source: Evaluation team’s elaboration based on the text mining exercise conducted on Grant Application Forms of the Justice Programme

In relation to the types of partnership, the data collected in the interim evaluation suggest that the Programme is successful in attracting more transnational partnerships 33 . Indeed, on average the number of participating organisation per project is equal to six-eight, a slight increase compared to the average of four-five observed in the predecessor programmes. Within the Justice Programme network there are 292 organisations connected out of 336 organisation reported in the database, i.e. 45 are isolated, meaning that only one organisation out of five within the Justice Programme collaboration network is isolated. Public bodies (mainly judicial training organisations) and higher education institutions (which also provide judicial training) are particularly well represented within the networks of the Justice Programme, reflecting the judicial training priority of the Programme.

The evaluation showed that one of the key vectors through which the Programme supports the development of an area of justice in Europe is by catalysing the development of transnational partnerships. Transnational partnerships are essential contributors to the effectiveness of the Programme. According to the 78% of respondents, this has increased their capacity to implement the projects, 70% reported an increase in sustainability of results and 75% mentioned increased capacities.

Moreover, these partnerships appear to be quite robust, with many interlinkages amongst participants 34 .

In addition, 50 organisations participating in the Justice Programme are also active in the Rights, Equality and Citizenship Programme. It thus appears that the Programme has facilitated the growth of durable European networks in the area of justice policy and that structured networks of collaborations exist also in relation to other EU funding programmes.

However, taking into account the involvement of partner organisations, it appears that the bulk of participants to the Programme come only from few Member States and, therefore, a balanced geographical spread has still to be achieved (as was the case with the predecessor programmes). For example, about 25% of all the beneficiaries come from either Italy or Belgium and about half of the partner organisations come from five countries and the remaining 50% are organisations from the remaining 21 Member States. Consequently, this affected the allocation of funds. For example, the largest share of available resources for action grants is attributed to Italian organisations, followed by organisations in the Netherlands, France, Belgium and Germany. Instead, the vast majority of resources available for Annual Work Programmes allocated to main actors via operating grants is distributed to organisations based in Belgium 35 .

With regard to the kind of projects/activities funded, the two specific objectives with the highest number of awarded grants are the ones on "Judicial Training" and "Access to Justice". Through them, the Programme mainly financed grants focused on training activities, but also analytical activities and actions to support main actors. Instead, in relation to the implementation of procurements, the Programme finances activities focused mostly on the specific objective "Judicial Cooperation", followed by the specific objective on "Access to Justice". In 2014, most funded activities concerned the organisation of conferences and studies, whereas in 2015 activities focused more on Information Technology interventions. 

The awarding rate of calls for proposal (in 2014 and 2015 36 ), of almost all specific objectives, ranged between around 16% and 47% . With respect to the specific objectives "Judicial Cooperation" and "Judicial Training", almost half of the applications submitted were awarded; instead, in the field of drug policies, where very few calls were launched in the period covered by the 2014-2016 Annual Work Programmes (5%), the demand for funding outstripped the supply significantly.

In 2016, a drop in the grant applications received, coinciding with the rollout of the Participant Portal 37 , was registered. This drop in applications notwithstanding, in general, the number of awarded projects remained stable across the four specific objectives and even increased significantly in the case of judicial cooperation projects. The drop also led to an increase of the awarding rate.

4.Method

Short description of methodology

The evaluation process was supported by an external evaluator (Ernst & Young Financial-Business Advisors) 38 , under the Commission's responsibility. The criteria used for the evaluation include: relevance, effectiveness, efficiency, coherence, equity, scope of simplification of the Programme and EU added value (see Table 7).

Table 7: Evaluation criteria and corresponding evaluation questions

The evaluation deployed a series of quantitative and qualitative tools, which were devised to provide detailed responses to specific evaluation questions associated with each evaluation criteria. The evaluation methodology relied on a Mixed Methods approach which integrates and compares quantitative and qualitative approaches, data collection, data analysis and interpretation in order to strengthen the reliability of data, the validity of the findings and recommendations by triangulating multiple sources of information.

The analysis covered the call for proposals and procurement activities financed in 2014, 2015 and 2016 Annual Work Programmes. The 2017 Work Programme on the other hand has been analysed chiefly in terms of design and structure, not in terms of execution.

One key method used to analyse the Programme was desk research, which focused on available documents at programme, policy and project level. The desk research was complemented by fieldwork, which was leveraged to collect relevant data and input from stakeholders. This helped to fill knowledge gaps and validate information which was retrieved during the course of the desk research. The following fieldwork activities were performed: 1. A web-based survey was launched to gauge and compare the views of relevant stakeholders (i.e. project applicants, project beneficiaries, Programme Committee Members and additional relevant stakeholders); 2. Some interviews were conducted with project beneficiaries and Commission officials; 3. Finally, a webinar was held, which provided a chance to discuss preliminary findings with a number of beneficiaries.

The collected data was then used to perform a series of quantitative and qualitative analyses. On the quantitative side, the analysis performed helped provide perspective both at the Programme level, assessing for instance how and where the Programme has deployed the available financial resources, and at the project level, giving a granular view of quantitative results indicators provided by individual projects. Also, a Social Network Analysis was performed. This analysis helped achieve a “bird’s-eye” view of the collaborations and transnational networks that were developed as a result of projects financed by the 2014-2020 Justice Programme. Finally, concerning the qualitative analysis, automated text mining techniques reinforced with machine learning algorithms were deployed to help sift through the large amounts of data available on individual projects. This was done in light of the need to give structure and find meaning within a large and mostly unstructured dataset (namely project applications and final reports), so as to create taxonomies of the information present in free text fields. Crucially, text mining helped in the analysis to find commonalities and correlations in the data that might not been found had solely manual techniques been deployed.

Robustness and limitations of findings

The evaluation’s findings and conclusions are deemed as credible and consistent according to the methodology. There were, however, some challenges which by their nature could not always be mitigated.

First of all, only 8 projects had both monitoring and survey data available which means that no statistical inference has proved feasible and that the triangulation of quantitative data at the project level has been limited. Nevertheless, the analysis of a series of output indicators allowed to have a description of the actions’ features as well as progress in terms of target groups reached and activities carried out.

Moreover, another challenge was that in evaluations of EU legislation “it can be difficult to identify a robust counterfactual situation (i.e. what the situation would be if EU laws had not been adopted)”, which made purely quantitative analysis problematic 39 .

In view of these limitations, a counterfactual analysis, built on qualitative data, has been considered to be the more suitable method. The analysis of direct questions on behavioural change assumes that the respondents are able to reflect on their behaviour in hypothetical, counterfactual situations and that they are telling the truth to the best of their knowledge. However, as respondents have an interest in the continuation of public support, they might be tempted to over-emphasize the merits thereof by answering strategically 40 . To address the positive bias in answers by beneficiaries and the self-selection bias in answers from applicants who have not received funding, a similar set of questions was asked to applicants having had their proposal rejected. As the features of all respondents (both successful and unsuccessful applicants) are known, this helped triangulate findings gathered from beneficiaries.

In addition, even though the four case studies proved to be a useful tool to get a better picture of the Programme’s activities, their representativeness of the Programme as a whole was somewhat limited.

Despite these limitations, the evaluation carried out is supported by a robust evidence base.

5.Analysis and answers to the evaluation questions

The Justice Programme is performing generally well at midterm with regard to its specific objectives, in terms of effectiveness, efficiency, relevance, coherence/complementarity/synergies and EU added value, but should do better in particular in terms of equity 41 .

5.1 Effectiveness

The analysis of indicators has shown that significant progress has been achieved on several fronts since some targets are close to being achieved and others have already been achieved. Therefore, the Justice Programme contributes positively to the general objective of the Programme through the progress under its specific objectives.

The Programme-specific indicators are adequate to monitor progress towards the objectives of the Programme, but sometimes they are difficult to measure due to a lack of adequate tools (e.g. there is no satisfaction survey to help gauge the perception of participants that took part in training activities) or to the influence of exogenous factors (such as macroeconomic conditions, rate of development of new psychoactive substances). Moreover, some indicators, concerning for instance the geographical coverage of the Programme, do not capture certain underlying dynamics, such as the fact that the Programme, so far, has been dominated by beneficiaries from a small number of Member States.

Overall, the 2014-2020 Justice Programme is broadly perceived by applicants, beneficiaries and stakeholders as an improvement compared to the three predecessor programmes (namely the Civil justice programme, the Criminal justice programme and the Drug prevention and information programme), both in terms of better policy targeting and better involvement of the right stakeholder groups. A better targeting is observed in the sense that calls tend to tackle specific issues which are considered priorities 42 . The merging of predecessor programmes has been particularly effective in the field of European judicial training since this has reduced the overlaps and enhanced the capacity of training providers by reducing the complications associated with having to deliver training across multiple programmes with different rules.

Given the wide scope of the general objective of the Programme, a large number of external factors influenced its effectiveness, such as the migration crisis (concerning, in particular, the specific objective on “access to justice”) and, in general, the uneven pace at which Member States transpose and enforce the EU acquis. This, however, did not undermine its intervention logic and its operational flexibility: the annual work programmes can be easily adapted to emerging needs in the area of justice (see more under “Relevance”).

Finally, compared to the 2007-2013 period, the sustainability of projects (with their results and outputs 43 ) beyond the end of their life-cycle has become an increasingly important factor to consider in the evaluation process. However, projects focused on creating tools/outputs might have a more difficult sustainability landscape since, in these cases, sustainability depends on whether the organisations that implemented the projects can receive sufficient resources to maintain the tools once the EU funding is over.

5.2 Efficiency

The Justice Programme has been cost-effective relative to the actions it has funded so far. Indeed, the results of the evaluation have shown that the beneficiaries’ perception of the efficiency of the Programme is positive. 60% of respondents to the relevant survey question consider the benefits of the intervention greater than costs. This is true for the Programme as a whole, but, in particular, for the specific objective on judicial training where more projects than expected have reported to have carried out training events and activities and more people than expected have been involved in the activities, both within the categories foreseen and beyond. The increase is quite significant also in terms of the types of modules of training provided (live events/remote training/tools and guidelines). Moreover, the average expenditure on judicial training is on average lower than expected for the projects identified.

A key achievement of the Programme, compared to its predecessors, has been the lower burden on beneficiaries in terms of time and financial resources. Nevertheless, there is still room for improvement in terms of easing requirements and obligations to make the Programme even more efficient in its implementation (see more under “Scope for simplification”).

According to beneficiaries, the current instruments (action grants, operating grants and procurement activities) are adequate for the needs of the Programme and, therefore, using alternative and innovative funding instruments is not necessary. However, the efficiency of their implementation should continue to be improved, notably with regard to procurement actions, to ensure that the allocated funding is indeed used. Indeed, since the forward planning for procurement actions occurs, usually, many months before the action itself, this leads to a tendency to overestimate procurement needs (to ensure that sufficient budget is allocated) and to a subsequent low implementation rate of the allocated funding. The efficiency of such actions must be increased by ensuring a greater degree of connection between the planning of procurement actions and their implementation.

5.3 Relevance

According to all interviewed beneficiaries, the Programme is highly relevant to address the needs of selected target groups. Indeed, one of the key features of the Programme has been its ability to adapt and modify the priorities in light of emerging needs. For instance, following a string of terrorist attacks in Europe, two calls were issued to combat the radicalisation of inmates in prison. This can be achieved thanks to the unique structure of the Justice Programme, as the Programme has been devised with broad specific objectives, due to the fact that it has aggregated the three programmes from the previous programming period. The general and specific objectives allowed the Commission to adapt the Programme to evolving needs within the EU, especially in terms of the judicial cooperation. However, the specific objective concerning the initiatives in the field of drug policies is sometimes difficult to reconcile with other Programme priorities, such as judicial cooperation and access to justice, as drug-related prevention policies tend to be broader in scope.

In general, however, the needs identified at the time of the Programme’s adoption are still actual and relevant, in particular the general objective of further developing a European Area of Justice based on mutual recognition and mutual trust.

With respect to stakeholder needs, there is still scope to further increase the relevance of the Programme through systematic analyses per main stakeholder type, as well as per specific objective and Member States, so as to ensure that priorities related to each specific objective for each call, as set out in the annual work programmes, are in line with the current key needs of the stakeholders. Moreover, the Programme could include additional target groups that are relevant to the achievement of its general objective. These groups include staff in regulatory agencies, young judicial professionals and students and judicial professionals in candidate countries and countries within the scope of the European Neighbourhood policy. This is however currently not possible under the legal basis of the Justice Programme.

5.4 Coherence, Complementarity, Synergies

The Programme presents a good level of coherence and complementarity with other EU instruments, programmes and actions (for instance, there is a very high coherence with the European Agenda for Justice 2020 44 ) and the risk of duplications or incoherence is very low. Especially in the case of judicial training, the merging of the predecessor programmes has increased the coherence with other EU initiatives and among different training objectives and reduced the possibility of duplications, both in scope and in funding. A strong coherence and complementarity in the access to justice specific objective exist with Connecting Europe Facility Telecommunications Programme that also contributed to the development of e-Justice Portal and e-Codex.

Some exceptional overlaps in terms of objectives, target groups and actions still exist as a natural consequence of the broad objectives and target groups covered by the Programme, ranging from judicial cooperation in civil and criminal matters (thus essentially the full spectrum of judicial activity) to drug policy and judicial training, as well as since it targets all EU citizens.

In any case, there is still potential for strengthening the synergies with other EU funding programmes and initiatives. This is the case, for instance, of the drugs policy area, where the coordination with the Health for Growth Programme could be increased.

According to interviewed stakeholders, the coherence with national policies and initiatives with similar objectives and/or targeting the same areas is high. The Programme fills the gaps left by national actions, while existing national projects and initiatives complement the Programme rather than being in contrast with it or a mere duplication. Indeed, when both the Justice Programme and national initiatives have the same (or similar) objectives and target groups, there are always differences between them, in terms of size of geographical scope (with enlarged scope in terms of target groups), available resources and number of projects funded.

The Justice Programme is coherent also with international obligations, such as the UN 2030 Sustainable Development Agenda. Indeed, some general principles of the UN agenda can be linked to objectives and areas targeted by the Justice Programme. This is the case of the UN objective of ensuring peace and security, to be achieved through just and inclusive societies that provide equal access to justice, effective rule of law, as well as transparent and effective judicial institutions. All these elements can be found as integrative parts of the Justice Programme, both in terms of access to justice and judicial training. Moreover, the EU is party to The Hague Conference on Private International Law and pursues its international action in relation to civil justice mainly through this international organisation.

5.5 EU added value

All evidence collected confirms the high added value of the Programme intervention, which is considered de facto instrumental for a good and effective achievement of objectives in the area of justice.

The results of the evaluation show that the survey respondents agree in saying that not only the funded activities would have not been possible without the EU intervention, but also confirmed that the same results would have not been achieved with Member States intervention only, in terms of creation of partnerships, realisation of outputs, funding of innovative actions, sustainability of results, and, especially, implementation of projects of transnational size and scale. Indeed, the EU added value of the Justice Programme is evident, above all, in the promotion of transnational projects with a European dimension to tackle cross-border issues and in the provision of financial resources to fund activities in key areas that are not necessarily high on the agenda of Member States, due to lack of political will (according to stakeholders, this is particularly true for the specific objective on access to justice 45 ).

According to beneficiaries interviewed, the Programme can influence and align, at least to a moderate extent, national actions in the targeted areas.

The Programme makes also beneficiaries able to work with partners in other Member States, a possibility that increased their knowledge and understanding of the issues covered by the Programme, widened their approach and range of skills, and provided them with access to good practice examples and tools developed in other Member States. As already mentioned above, partnerships are a key element for the success of the Justice Programme and are present in the majority of projects.

The lack of national funding is one of the main reasons why the Programmes’ funded activities would not have been possible through single Member States action. This is particularly true for the initiatives in the field of drug policies, where national initiatives are often missing a crucial transnational dimension that is pivotal in this field, and for EU-level judicial training that, usually, is not provided at national level. Moreover, the Justice Programme ensures the continued existence of European networks such as the European Judicial Training Network.

The perceived importance of the Programme can be explained also by the high number of projects awarded after just three years, compared to the three predecessor programmes 46 and the increase in the number of yearly applications received from potential beneficiaries indicates that the Programme remains a key source of funding across the EU.

Moreover, judicial training figures more prominently in the intervention logic of the Justice Programme, compared to the predecessor programmes. This is an important step forward for the EU added value of the Programme since judicial training is central to build mutual trust, improve cooperation between judicial authorities and practitioners in the Member States and increase the coherence in the application of the EU legislation.

The results of the evaluation show that the issues and areas addressed by the Justice Programme would require further action and involvement at EU level. Indeed, the demand of EU action in these fields and the fact that the number of applications to the Programme are still higher than the number of awarded grants demonstrate a clear interest in the priorities addressed by the Programme.

5.6 Equity

The promotion of the cross-cutting priorities of gender equality, rights of child and rights of people with disabilities is highly significant for the Justice Programme and it’s enshrined in its legal basis. In particular, the principles of gender and child rights mainstreaming are evaluated during the evaluation process under the quality of proposals. The issue of gender mainstreaming is specifically mentioned in the “part B” of the application form. However, the evaluation showed that, in terms of projects developed, gender issues and equality are hardly a major theme in most of them. Reference to women and gender equality are nonetheless found in five-six projects, which, still not focusing directly on the matter, take into account such element in their implementation.

The Justice Programme supports the rights of the child both in the programming phase (design of the calls for proposals) and by means of the activities of projects selected for funding. The respect of the rights of the child is therefore enhanced by the fact that organisations applying for funding (and any of their partners), which will work directly with children during the project implementation, must provide the Commission with a description of their child protection policy. Moreover, calls for proposals funded under the Justice Programme contain topics that are relevant to the subject. The results of the evaluation show that the majority of respondents agree to further mainstream and promote the rights of the child in the Programme.

Concerning the priority of rights of people with disabilities, it seems more in the background compared to the previous two. Indeed, among the projects funded, only one directly targeted at people with disabilities, under the specific objective access to justice 47 . In terms of the need to further promote the rights of people with disability in the Programme, stakeholders have quite heterogeneous opinions, but all categories tend to think there is at least a “moderate” need to do it.

To understand how the Programme promotes equity through the funded activities, participants' data broken down by sex, disability status or age, as required by the Regulation, shall be collected. This is however not yet done.

Finally, as already mentioned, in the future, the Programme should try to distribute its resources in a more balanced manner across the different target groups of beneficiaries and Member States.

5.7 Scope for simplification

The evaluation did not identify clear scope for further simplifications concerning the management mode of the Programme. The current direct management mode appears adequate given the size of the Programme.

However, notwithstanding the improvements made with the current Programme, both Commission officials and beneficiaries interviewed expressed critical opinions on the implementation process, with specific regard to budget management and reporting duties. Among the reasons provided, beneficiaries indicate that the financial reporting is too detailed and inflexible when compared to the ones applied within other EU Programmes (e.g. Horizon 2020 and Erasmus+).

In terms of administrative burden, nearly 70% of beneficiaries and applicants perceived as still burdensome the proposal drafting, the provisions of necessary administrative and financial information to participate in the calls and the monitoring and reporting requirements. In general, beneficiaries agreed that the application and reporting procedures were complex and long, but recognised that it was probably hard to simplify them further, given the need to ensure accountability of European financial resources.

According to beneficiaries, the duration of projects financed by the Programme could be extended, especially with regards to action grants. In particular, an extension to three years was considered ideal, as it would be a timeframe that is aligned with the average duration of scientific partnerships between higher education institutions. Moreover, beneficiaries mentioned that also operating grants could be extended to cover at least two years (instead of one) in order to reduce the administrative burden, in terms of application and reporting. However, the annual operating grants allow the Commission to have oversight of the activities of the funded organisations and allow for more flexibility to adapt the work of beneficiaries to the emerging needs in their respective fields of expertise.

Another key issue, according to small civil society organisations, is the difficulty to find the co-financing amounts required; therefore, small non-governmental organisations could be more supported by the Justice Programme 48 .

Even if, at first, the rollout of the new Participant Portal created some difficulties, currently stakeholders consider the submission of proposals via the Participant Portal an improvement, compared to the previous IT system, since the number of documents required for the eligibility check has decreased and, therefore, also the related administrative burden. In addition, once registered, these documents will not be required any more, only in case of changes. However, since the Participant Portal has been designed with research institutions and voluminous grants in mind, in its current form, the Portal appears ill-adapted to typical Justice Programme applicants. There is, therefore, scope for some improvements in this regard.

Finally, monitoring requirements and indicators, both at Programme and project level, could be simplified and streamlined.

6.Conclusions and way forward

Without the Programme's support, there is broad agreement among beneficiaries, legal professionals and Commission officials that a lot of activities, above all cross-border activities (such as the judicial training and the judicial cooperation in both civil and criminal matter) would not be provided by the Member States and the existence of relevant European networks, such as the European Judicial Training Network (a key player in the field of judicial training), would be at risk. Indeed, the areas covered by the Justice Programme are often not prioritised at national level and this makes the Programme unique since its positive impacts are not likely to be felt by other means or by single Member States' actions. In this sense, the EU added value of the Justice programme is undeniable and this is true for both its general objective and specific objectives. The activities financed help to tackle cross-border issues by creating partnerships, networking opportunities and occasions to build trust among judicial staff of different Member States. This fosters the acquisition of durable legal knowledge and competencies. Moreover, the broader scope of the current Justice Programme makes possible to identify each year the adequate priorities and to adapt them to emerging needs and threats.

The Programme's interim evaluation confirms its relevance for:

·contributing to upholding EU values such as the rule of law, the independence of the judiciary, the effectiveness of the justice system and its equal access to all;

·supporting Member States to achieve more effective justice systems, where legal practitioners know how and when to apply EU acquis, where cross-border cooperation is thriving and the rights of victims and persons accused of crime are respected;

·supporting to collect comparative information on the efficiency, quality and independence of national justice systems and national rule of law safeguards.

Some new priorities emerged and improvement could be made in specific areas of the Programme. In particular, the basin of potential recipients of the actions could be expanded to increase its relevance and the achievement of its general objective. Moreover, the geographic balance is still not good overall, since the Justice Programme is dominated by beneficiaries from only a small number of Member States.

Regarding the efficiency of the Programme, the administrative burden on beneficiaries has been reduced but there is still room for improvements to make the Programme even more efficient in its implementation.

The simplification carried out, thanks to the merger of the three previous programmes, has produced improvements in the quality of the evaluation and monitoring process. However, the development of workable indicators, allowing for precise monitoring of the quality and the impact of projects, should be better taken into consideration.

The Programme’s current structure works well by reducing the risks of duplications and allowing the Programme to be sufficiently flexible to adapt to emerging needs, even though coordination and information exchange between EU programmes and projects could be strengthened in order to guarantee a more coherent and efficient allocation of resources according to the most relevant priorities.

At EU level, the Justice Programme is coherent and complementary with other EU instruments, programmes and actions. However, in the future, there could be scope for further centralisation in terms of Judicial Training for legal practitioners 49 .

Lessons learned and way forward for the next programming period 50 :

Annex 1: Procedural information

1.Lead DG, Decide Planning/CWP References

Lead Directorate-General: Directorate-General for Justice and Consumers (DG JUST)

Decide planning reference:

2.Organisation and timing

The evaluation was carried out between October 2017 and April 2018.

An external contractor, Ernst & Young Financial-Business Advisors, was commissioned under the Framework Contract for evaluation, Impact Assessment and Related Services (JUST/2016/JCOO/FW/JU04/0178) to carry out a interim evaluation of the Justice programme (2014-2020). The final report was completed in April 2018. An Inter-service Steering Group led by DG JUST was set up specifically for this evaluation. The representatives of the following Commission services were members of the group:

Secretariat General, Directorate-General for Budget, Directorate-General for International Cooperation and Development, Legal Service, Directorate-General for Communications Networks, Content and Technology, Directorate-General for Migration and Home Affairs, Directorate-General for Neighbourhood and Enlargement Negotiations, Directorate-General for Health and Food Safety, and European Anti-Fraud Office.

Chronology of the interim evaluation:

3.Exceptions to the better regulation guidelines

A public consultation was not carried out specifically under this evaluation, but under the Impact Assessment for the post-2020 Multiannual Financial Framework.

4.Consultation of the RSB (if applicable)

N/A

5.Evidence, sources and quality

The evaluation was based on evidence from different sources. The complete set of documents that were consulted for this evaluation is listed below:

Legal basis of the Justice Programme:

·Arts. 81(1) and 82(1) TFUE;

·Regulation (EU) No 1382/2013 of the European Parliament and of The Council of 17 December 2013 establishing a Justice Programme for the period 2014 to 2020.

Commission Implementing Decisions 2014, 2015, 2016:

·Commission Implementing Decision C(2014)2556 concerning the adoption of the work programme for 2014 and the financing for the implementation of the Justice Programme;

·Commission Implementing Decision C(2015)1997 concerning the adoption of the work programme for 2015 and the financing for the implementation of the Justice Programme;

·Commission Implementing Decision C(2016)1677 concerning the adoption of the work programme for 2016 and the financing for the implementation of the Justice Programme.

EU documents:

·Council conclusions on the European Judicial Network in civil and commercial matters (15349/16) on 8 and 9 December 2016;

·Conclusions of the European Council on 26-27 June 2014 in the area of freedom, security and justice (including on judicial cooperation in civil matters)

·The European Agenda for Justice for 2020, adopted in 2014;

·The European Agenda for Security, adopted in 2015;

·The 2030 Sustainable Development Agenda;

·The “Juncker Priorities” for 2015-2019;

·Conclusions of the European Council on 26-27 June 2014 in the area of freedom, security and justice;

·Multiannual European e-Justice Action Plan 2014-2018.

Justice Programme documentation:

·Ex-post evaluation report of Civil Justice Programme (2007-2013)

http://eur-lex.europa.eu/legal-content/en/TXT/?uri=CELEX:52017DC0059  

·Ex-post evaluation report of Criminal Justice Programme (2007-2013)

http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52017DC0115  

·Ex-post evaluation report of Drugs Prevention and Information Programme (2007-2013)

·The mid-term evaluation report of Civil Justice Programme, 2011

http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2011:0351:FIN:EN:PDF  

·The mid-term evaluation report of Criminal Justice Programme, 2011

http://ec.europa.eu/justice/funding/jpen/interim_evaluation_report_2011_en.pdf  

·The mid-term evaluation report of Drugs Prevention and Information Programme, 2011

http://eur-lex.europa.eu/legal content/EN/TXT/PDF/?uri=CELEX:52011DC0246&qid=1467122450426&from=EN

·Policy-specific reports published by DG Justice and Consumers, such as the annual report on European judicial training

https://e-justice.europa.eu/fileDownload.do?id=5799faaa-b4ce-498c-aeb1-bba4b81d900e , https://e-justice.europa.eu/fileDownload.do?id=ae7d84cc-2bd0-4eb3-9c31-94b753a20bdb

·EU Justice Scoreboard COM (2017) 167 final

https://ec.europa.eu/info/strategy/justice-and-fundamental-rights/effective-justice/eu-justice-scoreboard_en  

Annex 2: Stakeholders consultation

Consultations with the main stakeholders of the ‘Justice Programme’ were conducted over several steps:

·A web-based survey targeting the following stakeholder groups:

oprojects applicants (also unsuccessful applicants ) 51 ;

oprojects beneficiaries 52 ;

oEU26 Member State Programme Committee Members 53 ;

oOther relevant stakeholders involved directly by the Commission;

This consultation was launched by the external consultant via the EU Survey tool and was open from 23 December 2017 until 6 March 2018. It received 132 responses 54 .

·33 interviews with 12 Commission officials and 21 selected projects beneficiaries and representatives of target groups were performed between the 8th of January 2018 and the 26th of March 2018. Interviews had a twofold objective: 1. Collecting additional information (from the survey and the desk research) to feed the analysis alongside the evaluation criteria, filling gaps (especially in terms of quantification of costs and benefits) and going more in-depth with specific aspects, such as the identification of best practices; 3. Gathering insight and input to draw conclusions and recommendations on how to improve the design and functioning of the Programmes.

Interviews consisted of semi-structured phone interviews. Questions were customised according to the different categories of stakeholders targeted, taking into account their different contribution to the evaluation questions;

·1 webinar, organized in March 2018, to discuss preliminary findings with few selected project beneficiaries. The webinar aimed at assessing at a higher level results achieved by the Programme, testing and discussing preliminary findings and conclusions, also identifying key success factors and areas for improvement in view of the design of operational recommendations. It was organised using Zoom (an online tool);

·A broad public consultation undertook by the Commission in the area of "values and mobility" was available online in 23 official EU languages for a mandatory period between 10 January 2018 and 9 March 2018. The purpose of this consultation was to collect the views of all interested parties on how to make the most of every euro of the EU budget. Consultations have taken place in the context of evaluations of existing EU financial programmes covering several policy areas, including the Justice Programme. In total, the public consultation received 1839 replies from all over Europe. The respondents had experience with the following EU programmes: 1. Europe for Citizens Programme and /or 2. Rights, Equality and Citizenship Programme and /or 3. Creative Europe Programme and /or 4. Justice Programme. The results of this public consultation have shown that, according to the majority of respondents, "promote European identity and common values", as well as "promote rights and equality", are important common policy challenges to be addressed in each of the four programmes. "Support active citizenship, democratic participation in society and the rule of law" and "promote social inclusion and fairness" appear to be important challenges to be addressed in the concerned programmes as well. Instead, "support innovation", "foster European cultural diversity and cultural heritage", "promote European identity and common values" are considered as policies that fully or fairly well address the challenges by half or more of the respondents. Around 80% the respondents agree that these programmes add value to a large extent or to a fairly good extent to what Member States could achieve at national, regional and/or local levels. The main obstacles identified by the respondents, that could prevent the current programmes/funds from achieving their objectives, are very similar regardless of the programme concerned: "lack of budget of the programmes to satisfy demand", "insufficient support provided to small-scale stakeholders" and "lack of support to first-time applicants" are identified as the main three obstacles. Finally, the respondents agree that "the use of more simplified application forms", "facilitating structured network and partnerships", "facilitating funding for actions cutting across the sectors of action", as well as "better coordination between different programmes/funds", are the main steps to be taken to simplify and reduce the administrative burdens for beneficiaries.

Annex 3: Methods

The following methodological tools have been used:

1. Data collection 55 ;

·desk research of relevant documents (at programme, policy and project level);

·field research relied on a web-based survey with relevant stakeholders, i.e. (unsuccessful) project applicants, project beneficiaries, Programme Committee members, Commission staff and additional relevant stakeholders, 33 interviews with 21 selected project beneficiaries and representatives of target groups and 12 with Commission officials and one webinar to test and discuss preliminary findings and conclusions with few selected project beneficiaries.

·four case studies of projects connected to the specific objectives 56 ;

·analysis of the results of a broad public consultation undertaken by the Commission between 10 January and 8 March 2018 in the framework of the proposal for the post 2020 Multiannual Financial Framework in the area of "European values and mobility".

2. Data analysis:

·quantitative data analysis 57 : in addition to monitoring and survey data, descriptive statistics was used to analyse the effectiveness, efficiency and EU added value of the Programme; self-reported counterfactual analysis have been used for the analysis of its effectiveness and added value 58 ; exploratory data analysis and econometric modelling were used to analyse its effectiveness and EU added value 59 ; cost-benefit analysis to analyse its efficiency 60 and Social Network Analysis to analyse its effectiveness and equity 61 ;

·qualitative data analysis 62 : a "text mining" methodology was used 63 .

Based on this methodology, output indicators (as monitoring data at project level) and results indicators (based on Eurobarometer and other sources) were used. The analysis of the performance at the project level has mainly relied on the answers to the survey.

(1)  Regulation (EU) No 1382/2013 of the European Parliament and of the Council of 17 December 2013 establishing a Justice Programme for the period 2014 to 2020 (OJ L 354 of 28.12.2013), available online at: http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32013R1381&from=EN .

(2)  Ex-post evaluation of five programmes implemented under the 2007-2013 financial perspective – Final Report. Specific programme evaluation: Civil Justice, available here http://ec.europa.eu/justice/grants1/files/expost_evaluations_reports_2007_2013/civil_justice_programme.pdf . Ex-post evaluation of five programmes implemented under the 2007-2013 financial perspective - specific programme evaluation: Criminal Justice Support Programme, available here http://ec.europa.eu/justice/grants1/files/expost_evaluations_reports_2007_2013/criminal_justice_programme.pdf . Ex-post evaluation of five programmes implemented under the 2007-2013 financial perspective Specific programme evaluation: Drug Prevention and Information Programme, available here http://ec.europa.eu/justice/grants1/files/expost_evaluations_reports_2007_2013/dpip_programme.pdf .

(3) SEC (2011) 1364 Final.

(4) As already mentioned, the Justice Programme has replaced three programmes that were in force during the 2007-2013 Multi-annual Financial Framework.

(5) In this context, “equity” does not mean necessarily equality (i.e. that target groups and Member States receive the same level of support in terms of projects funded and resources allocated). Instead, the concept of equity relates to the allocation of resources mirroring the different intensity of, and urgency for, support of each target group/Member States rather than for example their capacity for advocacy or in accessing the Programme funds/managing projects.

(6) The United Kingdom and Denmark do not participate in the Programme.

(7) Given its relatively early stage of implementation, the 2017 Annual Working Programme has been analysed in terms of design and structure, not in terms of execution.

(8)

OJ C 115, 4.5.2010, p. 1.

(9) Such as, in particular, insufficient judicial cooperation in civil and criminal matters, wide differences in national procedures and laws across the EU, lack of trust from EU citizens in the functioning of judicial systems of other Member States, lack of awareness and difficulties in the exercise of EU citizenship rights by people, limited improvements in preventing and combating drug-related crimes.

(10) The EU is party to The Hague Conference on Private International Law since in 2007. The purpose of this international intergovernmental organisation is to work for the progressive unification of the rules of private international law in the participating countries. Most of the conventions developed by The Hague Conference fall within exclusive or partial external competence of the EU and are part of the EU acquis.

(11) European Commission, Directorate-General for Justice and Consumers, European Judicial Training 2016. Available online at the following address: http://ec.europa.eu/justice/criminal/files/final_report_2015_en.pdf .

(12)

Council of the European Union, Multiannual European e-Justice Action Plan 2014-2018, 2014/C 182/02, available here https://publications.europa.eu/en/publication-detail/-/publication/2a2f84b9-f3a4-11e3-831f-01aa75ed71a1/language-en .

(13) Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, The EU Justice Agenda for 2020 - Strengthening Trust, Mobility and Growth within the Union, Strasbourg, 11.3.2014, COM(2014) 144 final.

(14)

 Insofar as this kind of initiatives are not covered by the Internal Security Fund for financial support for police cooperation, preventing and combating crime and crisis management (for more info, see https://ec.europa.eu/home-affairs/financing/fundings/security-and-safeguarding-liberties/internal-security-fund-police_en ), or by the Health for Growth Programme (for more info, see https://ec.europa.eu/health/funding/programme/2014-2020_en ).

(15) To see the types of activities funded by the Justice Programme in 2014, 2015 and 2016, see the Mid-term evaluation of the Justice Programme 2014-2020, Ernst & Young Financial-Business Advisors, Final Report, April 2018, p. 71-72.

(16) Bodies and entities which are profit-oriented can be funded only in conjunction with non-profit or public organisations.

(17) For more information, see below p. 15 and see also section 5.4 on "Target groups and beneficiaries" in the Interim report, p. 77 et seq.

(18) For more information on the number of action grants and operating grants awarded per year and on the number of procurement activities, see section 5 "Implementation state of play" in the Interim report, in particular p. 69 et seq.

(19) Mid-term evaluation of the Justice Programme 2014-2020, Ernst & Young Financial-Business Advisors, Final Report, April 2018, published.

(20) Ibid.

(21) European Commission, 'Commission Staff Working Paper – Impact Assessment - Accompanying the document Proposal for a Regulation of the European Parliament and of the Council establishing for the Period 2014-2020 the Justice Programme. Impact Assessment’. SEC (2011) 1364 Final.

(22)  Ex-post evaluation of five programmes implemented under the 2007-2013 financial perspective – Final Report, ibid.

(23) Although, the beneficiaries interviewed were not specifically asked to describe their experience with other kinds of Programme management.

(24) COM (2017) 400 - May 2017, Draft Budget of the European Union 2017. Information on indicators related to the general objectives of the Programmes and indicators related to the specific objectives of the Programmes are provided by this document.

(25) Moreover, gaps in data described in the training reports lead to surmise that the number of trained judicial staff is somewhat underreported.

(26)

The European arrest warrant has been operational since 1 January 2004. It has replaced the extradition procedure that used to exist between EU countries. At present, a person who has committed a serious crime in a Member State, but who lives in another can be returned to the first country to face justice quickly and with little administrative burden. For more info, see Council Framework Decision of 13 June 2002 on the European arrest warrant and the surrender procedures between Member States (2002/584/JHA).

(27)  This database ensures that information on convictions is exchanged between EU Member States in a uniform, fast and compatible way, provides judges and prosecutors with easy access to comprehensive information on the criminal history of persons concerned, including in which EU Member States that person has previously been convicted and removes the possibility for offenders to escape convictions by moving from one Member State to another. On 28 June 2017, the European Commission published its first report on the use of this database by EU countries.

(28) These obstacles were mentioned by several projects and shown also by the "Justice Scorecards".

(29) For more info on the European Social Fund, see http://ec.europa.eu/social/main.jsp?catId=325&langId=en .

(30) More info available at the following link: https://e-justice.europa.eu/home .

(31) Source of data: DG Justice and Consumers.

(32)

Art. 15 of the Regulation (EU) 1382/2013 of 17 December 2013 establishing a Justice Programme for the period 2014 to 2020.

(33) Interim evaluation of the Justice Programme 2014-2020, Ernst & Young Financial-Business Advisors, Final Report, April 2018, published.

(34) See the results of the Social Network analysis of the networks of beneficiaries within the Programme in the Interim report, ibid, p. 56.

(35)  This is due to the fact that operating grants resources are made available to organisations representing European networks, which tend to be headquartered in Belgium (thanks to proximity with other EU institutions ).

(36) This information is not available for 2016 yet.

(37) See ‘The Justice Programme - implementation in 2016 & 2017’, Directorate-General for Justice and Consumers, 2017. The Participant Portal is an electronic platform used to manage the applications received for calls published in the framework of the Justice Programme. According to the evaluation results, the new Participant Portal is not adapted to typical Justice Programme applicants (such as training organisations), as the Portal was initially designed with research institutions and voluminous grants in mind.

(38) Interim evaluation of the Justice Programme 2014-2020, Ernst & Young Financial-Business Advisors, Final Report, April 2018, published.

(39) European Commission, Better Regulation Toolbox, p. 344, https://ec.europa.eu/info/sites/info/files/better-regulation-toolbox_0.pdf .

(40)

In interpreting the results, it should be considered that there may be two sources of bias to the answers: 1. Strategic behaviour (beneficiaries may tend to overestimate the effect of the treatment to further legitimise their receipt of funding, while applicants may tend to say to be well off also in absence of funding – also to qualify for the next round); 2. Estimation bias: both beneficiaries and applicants don’t know exactly what the counterfactual situation would be like, so answers are based on their educated guess. This applies especially to applicants who can base their judgment only on the knowledge they acquired in the application phase with no actual exposure to the implementation of the measures.

(41) For more information and evidence, see section 6 "Responses to evaluation questions" in the Interim report, p. 94 et seq.

(42) However, it should be noted that better targeting makes sense as long as there is enough demand. If the targeting is excessively specific the risk is that of receiving a small number of high quality proposals, with the risk of not allocating the entire available budget.

(43)    Sustainability of results and outputs, achieved through the implementation of activities that generate increased awareness, knowledge and skills and through the development of tools that can be used also after the project ends, by beneficiaries and target groups alike, (including transferred to other organisations).

(44) The objectives and addressed areas of the European Agenda for Justice for 2020 are very much in line with the Justice Programme, especially in the area of judicial cooperation, even if the Agenda prioritises areas such as terrorism and cybercrime that are less present under the Justice Programme.

(45) See Interim report, ibid.

(46) The Civil Justice Programme, the Criminal Justice Programme and the Drug Prevention and Information Programme awarded 806 projects over seven years (2007-2013). The Justice Programme awarded 418 just in the three-year period 2014-2016.

(47) Project “Enhancing Procedural Rights of Persons with Intellectual and/or Psychiatric Impairments in Criminal Proceedings: Exploring the Need for Actions”.

(48)  It is to be noticed that the average amount of grants has been increased to the current level following the ex-post evaluation of the predecessor programmes that, as already said above, found that the dilution of funds amongst many small-scale projects had had limited impact and EU dimension.

(49) However, it should be noted that each Directorate-General is responsible for the enforcement of the legislation it is in charge of and the centralisation of training on all EU topics would require additional financial resources.For additional information on the conclusions, see section 7 "Conclusions" in the Interim report, p. 161 et seq.

(50) For more information, see section 8 "Recommendations" in the Interim report, p. 167 et seq.

(51) From 16 Member States.

(52) From 16 Member States.

(53) From 8 Member States.

(54) For more information on the outcomes of the web survey, see Annex 2 "Analysis of surveys and quantitative analysis" in the Interim report, ibid, p. 172 et seq.

(55) For more information, see section 4.1.1 "Data collection tools and activities" in the Interim report, p. 49 et seq.

(56) For more information, see the Annexe 4 of the Interim report, ibid.

(57) For more information, see section 4.2.1 "Quantitative data analysis" in the Interim report, p. 51 et seq.

(58) A counterfactual analysis is a tool that can be used to distil changes that are caused by a specific policy intervention from changes that would have occurred even in absence of the given intervention (see Interim report, ibid).

(59) Different tools of econometric analysis have been employed in order to examine the results of the self-reported counterfactual questions, as well as to test the responses of different stakeholders to the main question of the survey linked to effectiveness/relevance (see Interim report, ibid).

(60) The Cost-Benefit analysis is as an economic tool applied to public decision−making to quantify the advantages (in terms of benefits) and disadvantages (in terms of costs) associated with a particular project or policy (see Interim report, ibid).

(61) The Social Network analysis enabled to better understand the explicit and implicit links and interrelations among the different beneficiaries and hence the value of the partnership approach that is a distinguishing feature of the (see Interim report, ibid).

(62) For more information, see section 4.2.2 "Qualitative data analysis through text mining" in the Interim report, p. 58 et seq.

(63) Text mining is an analytical methodology that has the aim of structuring large amounts of unstructured texts into categories or clusters of information by broad themes (e.g. project objectives, weaknesses) (see Interim report, ibid).